[GH-ISSUE #117 ] Syntax Error on Babel? #44

Author

Owner

@Yusufkulcu commented on GitHub (Sep 25, 2024):

I have the same problem, but no clear solution has been found

@Yusufkulcu commented on GitHub (Sep 25, 2024): I have the same problem, but no clear solution has been found

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 25, 2024):

Thank you for the detailed issue! I'll look into it. It seems that this line is indeed missing a ,:

static Qp(_static calculateQualityParameter, qualityParameter) {

Tha does seem super strange to me, as there should definitely not be a space there. I'll have a look at it a bit later.

@jehna commented on GitHub (Sep 25, 2024): Thank you for the detailed issue! I'll look into it. It seems that this line is indeed missing a `,`: ``` static Qp(_static calculateQualityParameter, qualityParameter) { ``` Tha does seem super strange to me, as there should definitely not be a space there. I'll have a look at it a bit later.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 25, 2024):

Thank you for the detailed issue! I'll look into it. It seems that this line is indeed missing a ,:
static Qp(_static calculateQualityParameter, qualityParameter) {
Tha does seem super strange to me, as there should definitely not be a space there. I'll have a look at it a bit later.

If you have time, could you also look at my problem, I would be very grateful
https://github.com/jehna/humanify/issues/111

@Yusufkulcu commented on GitHub (Sep 25, 2024): > Thank you for the detailed issue! I'll look into it. It seems that this line is indeed missing a `,`: > > ``` > static Qp(_static calculateQualityParameter, qualityParameter) { > ``` > > Tha does seem super strange to me, as there should definitely not be a space there. I'll have a look at it a bit later. If you have time, could you also look at my problem, I would be very grateful https://github.com/jehna/humanify/issues/111

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 25, 2024):

It seems that this line is indeed missing a ,:

@jehna @Anooxy17 Looking at the attached obfuscated.txt, on what appears to be that same lines of code:

# ..snip..

        static Qp(a, b) {
            let c = new ib;
            var d = w.J(a.d0, Ob)
              , e = w.J(a.d1, Ob);
            let f = a.color
              , g = a.strength;

# ..snip..

It looks like the problem is less that there should be a ,, and more that that something is adding the _static prefix to the variable where it presumably shouldn't be.

I can see that a lot of the function names have a static prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking.

I wonder if running it again with --verbose or similar would provide better info for debugging this further? May help narrow down where/why the _static prefix is being introduced to things?

Another theory is that the _ is being prefixed onto static to avoid it clashing with a JS keyword. I still don't know that it should be present for this param at all in general though.

@0xdevalias commented on GitHub (Sep 25, 2024): > It seems that this line is indeed missing a `,`: @jehna @Anooxy17 Looking at the attached `obfuscated.txt`, on what appears to be that same lines of code: ```js # ..snip.. static Qp(a, b) { let c = new ib; var d = w.J(a.d0, Ob) , e = w.J(a.d1, Ob); let f = a.color , g = a.strength; # ..snip.. ``` It looks like the problem is less that there should be a `,`, and more that that something is adding the `_static` prefix to the variable where it presumably shouldn't be. I can see that a lot of the function names have a `static` prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking. I wonder if running it again with `--verbose` or similar would provide better info for debugging this further? May help narrow down where/why the `_static` prefix is being introduced to things? Another theory is that the `_` is being prefixed onto `static` to avoid it clashing with a JS keyword. I still don't know that it should be present for this param at all in general though.

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

I have the same problem, but no clear solution has been found

@Yusufkulcu Your problem in https://github.com/jehna/humanify/issues/111 (which is a duplicate of https://github.com/jehna/humanify/issues/54) isn't the same as the problem in this issue.

If you have time, could you also look at my problem, I would be very grateful

@Yusufkulcu Spamming other issues trying to draw more attention to an issue you raised is generally considered rude and bad practice in open-source.

@0xdevalias commented on GitHub (Sep 26, 2024): > I have the same problem, but no clear solution has been found @Yusufkulcu Your problem in https://github.com/jehna/humanify/issues/111 (which is a duplicate of https://github.com/jehna/humanify/issues/54) isn't the same as the problem in this issue. > If you have time, could you also look at my problem, I would be very grateful @Yusufkulcu Spamming other issues trying to draw more attention to an issue you raised is generally considered rude and bad practice in open-source.

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

Looking at the attached obfuscated.txt, on what appears to be that same lines of code:

# ..snip..

        static Qp(a, b) {
            let c = new ib;
            var d = w.J(a.d0, Ob)
              , e = w.J(a.d1, Ob);
            let f = a.color
              , g = a.strength;

# ..snip..

I tried running this locally with the --verbose flag. I didn't let it complete, but I did let it get past the above Qp code, without running into any errors:

..snip..

[2024-09-26 03:46:49]  Processing: 5%
[2024-09-26 03:46:49]  Renaming a
[2024-09-26 03:46:49]  Context:  static Qp(a, b) {
  let c = new ib();
  var d = w.J(a.d0, Ob);
  var e = w.J(a.d1, Ob);
  let f = a.color;
  let g = a.strength;
  a = a.length;
  if (d >= b.length || d < 0) {
    throw v.C(null);
  
[2024-09-26 03:46:49]  Renamed to qpInstance
[2024-09-26 03:46:49]  Processing: 5%
[2024-09-26 03:46:50]  Renaming b
[2024-09-26 03:46:50]  Context:  static Qp(qpInstance, b) {
  let c = new ib();
  var d = w.J(qpInstance.d0, Ob);
  var e = w.J(qpInstance.d1, Ob);
  let f = qpInstance.color;
  let g = qpInstance.strength;
  qpInstance = qpInstance.
[2024-09-26 03:46:50]  Renamed to strengthValue
[2024-09-26 03:46:50]  Processing: 5%
[2024-09-26 03:46:50]  Renaming c
[2024-09-26 03:46:50]  Context:  static Qp(qpInstance, strengthValue) {
  let c = new ib();
  var d = w.J(qpInstance.d0, Ob);
  var e = w.J(qpInstance.d1, Ob);
  let f = qpInstance.color;
  let g = qpInstance.strength;
  qpInstance =
[2024-09-26 03:46:51]  Renamed to createIbInstance
[2024-09-26 03:46:51]  Processing: 5%
[2024-09-26 03:46:51]  Renaming d
[2024-09-26 03:46:51]  Context:  static Qp(qpInstance, strengthValue) {
  let createIbInstance = new ib();
  var d = w.J(qpInstance.d0, Ob);
  var e = w.J(qpInstance.d1, Ob);
  let f = qpInstance.color;
  let g = qpInstance.strength;
[2024-09-26 03:46:52]  Renamed to distanceCalculation
[2024-09-26 03:46:52]  Processing: 5%
[2024-09-26 03:46:52]  Renaming e
[2024-09-26 03:46:52]  Context:  static Qp(qpInstance, strengthValue) {
  let createIbInstance = new ib();
  var distanceCalculation = w.J(qpInstance.d0, Ob);
  var e = w.J(qpInstance.d1, Ob);
  let f = qpInstance.color;
  let g = qp
[2024-09-26 03:46:53]  Renamed to distanceCalculationD1
[2024-09-26 03:46:53]  Processing: 5%
[2024-09-26 03:46:53]  Renaming f
[2024-09-26 03:46:53]  Context:  static Qp(qpInstance, strengthValue) {
  let createIbInstance = new ib();
  var distanceCalculation = w.J(qpInstance.d0, Ob);
  var distanceCalculationD1 = w.J(qpInstance.d1, Ob);
  let f = qpInstance
[2024-09-26 03:46:53]  Renamed to currentQpInstance
[2024-09-26 03:46:53]  Processing: 5%
[2024-09-26 03:46:53]  Renaming g
[2024-09-26 03:46:53]  Context:  static Qp(qpInstance, strengthValue) {
  let createIbInstance = new ib();
  var distanceCalculation = w.J(qpInstance.d0, Ob);
  var distanceCalculationD1 = w.J(qpInstance.d1, Ob);
  let currentQpInsta
[2024-09-26 03:46:54]  Renamed to calculateDistance

..snip..

@Anooxy17 Are you able to run this again with --verbose and see if you get the error again? And if so, can you paste the relevant logs (or maybe the full logs as an attachment)? That may help narrow down where the issue might be coming from.

It looks like the problem is less that there should be a ,, and more that that something is adding the _static prefix to the variable where it presumably shouldn't be.

I can see that a lot of the function names have a static prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking.

This is still my main theory though, particularly given the same _static calculateQualityParameter 'variable name' can be seen on lines 1327, 1329, 1330:

SyntaxError: Unexpected token, expected "," (1327:23)
  1325 |       return dataPointDetails;
  1326 |     }
> 1327 |     static Qp(_static calculateQualityParameter, qualityParameter) {
       |                       ^
  1328 |       let qualityCalculator = new InvestmentBalance();
  1329 |       var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil);
  1330 |       var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil);
    at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472)
    at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798)

I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use static calculateQualityParameter as a variable name, and then humanify prefixed it with _; or similar based on something like this:

https://github.com/jehna/humanify/issues/20

@0xdevalias commented on GitHub (Sep 26, 2024): > Looking at the attached `obfuscated.txt`, on what appears to be that same lines of code: > > ```js > # ..snip.. > > static Qp(a, b) { > let c = new ib; > var d = w.J(a.d0, Ob) > , e = w.J(a.d1, Ob); > let f = a.color > , g = a.strength; > > # ..snip.. > ``` I tried running this locally with the `--verbose` flag. I didn't let it complete, but I did let it get past the above `Qp` code, without running into any errors: ``` ..snip.. [2024-09-26 03:46:49] Processing: 5% [2024-09-26 03:46:49] Renaming a [2024-09-26 03:46:49] Context: static Qp(a, b) { let c = new ib(); var d = w.J(a.d0, Ob); var e = w.J(a.d1, Ob); let f = a.color; let g = a.strength; a = a.length; if (d >= b.length || d < 0) { throw v.C(null); [2024-09-26 03:46:49] Renamed to qpInstance [2024-09-26 03:46:49] Processing: 5% [2024-09-26 03:46:50] Renaming b [2024-09-26 03:46:50] Context: static Qp(qpInstance, b) { let c = new ib(); var d = w.J(qpInstance.d0, Ob); var e = w.J(qpInstance.d1, Ob); let f = qpInstance.color; let g = qpInstance.strength; qpInstance = qpInstance. [2024-09-26 03:46:50] Renamed to strengthValue [2024-09-26 03:46:50] Processing: 5% [2024-09-26 03:46:50] Renaming c [2024-09-26 03:46:50] Context: static Qp(qpInstance, strengthValue) { let c = new ib(); var d = w.J(qpInstance.d0, Ob); var e = w.J(qpInstance.d1, Ob); let f = qpInstance.color; let g = qpInstance.strength; qpInstance = [2024-09-26 03:46:51] Renamed to createIbInstance [2024-09-26 03:46:51] Processing: 5% [2024-09-26 03:46:51] Renaming d [2024-09-26 03:46:51] Context: static Qp(qpInstance, strengthValue) { let createIbInstance = new ib(); var d = w.J(qpInstance.d0, Ob); var e = w.J(qpInstance.d1, Ob); let f = qpInstance.color; let g = qpInstance.strength; [2024-09-26 03:46:52] Renamed to distanceCalculation [2024-09-26 03:46:52] Processing: 5% [2024-09-26 03:46:52] Renaming e [2024-09-26 03:46:52] Context: static Qp(qpInstance, strengthValue) { let createIbInstance = new ib(); var distanceCalculation = w.J(qpInstance.d0, Ob); var e = w.J(qpInstance.d1, Ob); let f = qpInstance.color; let g = qp [2024-09-26 03:46:53] Renamed to distanceCalculationD1 [2024-09-26 03:46:53] Processing: 5% [2024-09-26 03:46:53] Renaming f [2024-09-26 03:46:53] Context: static Qp(qpInstance, strengthValue) { let createIbInstance = new ib(); var distanceCalculation = w.J(qpInstance.d0, Ob); var distanceCalculationD1 = w.J(qpInstance.d1, Ob); let f = qpInstance [2024-09-26 03:46:53] Renamed to currentQpInstance [2024-09-26 03:46:53] Processing: 5% [2024-09-26 03:46:53] Renaming g [2024-09-26 03:46:53] Context: static Qp(qpInstance, strengthValue) { let createIbInstance = new ib(); var distanceCalculation = w.J(qpInstance.d0, Ob); var distanceCalculationD1 = w.J(qpInstance.d1, Ob); let currentQpInsta [2024-09-26 03:46:54] Renamed to calculateDistance ..snip.. ``` @Anooxy17 Are you able to run this again with `--verbose` and see if you get the error again? And if so, can you paste the relevant logs (or maybe the full logs as an attachment)? That may help narrow down where the issue might be coming from. --- > It looks like the problem is less that there should be a `,`, and more that that something is adding the `_static` prefix to the variable where it presumably shouldn't be. > > I can see that a lot of the function names have a `static` prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking. This is still my main theory though, particularly given the same `_static calculateQualityParameter` 'variable name' can be seen on lines 1327, 1329, 1330: ``` SyntaxError: Unexpected token, expected "," (1327:23) 1325 | return dataPointDetails; 1326 | } > 1327 | static Qp(_static calculateQualityParameter, qualityParameter) { | ^ 1328 | let qualityCalculator = new InvestmentBalance(); 1329 | var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil); 1330 | var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil); at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472) at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798) ``` I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use `static calculateQualityParameter` as a variable name, and then `humanify` prefixed it with `_`; or similar based on something like this: - https://github.com/jehna/humanify/issues/20

kerem commented

Author

Owner

@Anooxy17 commented on GitHub (Sep 26, 2024):

So should I open it on openai again or try it locally? With which model?

@Anooxy17 commented on GitHub (Sep 26, 2024): So should I open it on openai again or try it locally? With which model?

kerem commented

Author

Owner

@Anooxy17 commented on GitHub (Sep 26, 2024):

@0xdevalias
errors.txt
It was made in OpenAI, not even went into over 70% now

@Anooxy17 commented on GitHub (Sep 26, 2024): @0xdevalias [errors.txt](https://github.com/user-attachments/files/17148386/errors.txt) It was made in OpenAI, not even went into over 70% now

kerem commented

github.com/jehna/humanify@c2e0be679d/package.json (L61)

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

errors.txt
It was made in OpenAI, not even went into over 70% now

This error seems to be related to the attempt to assign a value to an object within a for loop, which isn't allowed:

SyntaxError: Unexpected token (3562:43)
  3560 |       if (this.M == null) {
  3561 |         this.M = new CustomAnimationHandler();
> 3562 |         for (var _________index = 0, _this.kLength = this.K; _________index < _this.kLength.length;) {
       |                                           ^
  3563 |           let kLengthValue = _this.kLength[_________index];
  3564 |           ++_________index;
  3565 |           kLengthValue.I = null;
    at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472)
    at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798)
    at Object.parse (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:15:2885)
    at parse4 (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:21902:24)
    at async coreFormat (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:22355:7)
    at async formatWithCursor (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:22557:14)
    at async Object.format2 (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:23932:25)
    at async unminify (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:214:27)
    at async Command.<anonymous> (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:56814:3) {
  loc: { start: { line: 3562, column: 43 } },

While I haven't looked deeply into it, this part makes me wonder if it's an issue with webcrack and how it's choosing to inline certain variables/etc within for loops.

It seems humanify is currently using webcrack 2.13.0:

Looking at the releases we can see that the latest version is 2.14.1:

https://github.com/j4k0xb/webcrack/releases

Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue:

https://github.com/j4k0xb/webcrack/releases/tag/v2.14.0
- Extract sequence from for-of: for (let value of (a = 1, array)) {} -> a = 1; for (let value of array) {}

It also seems to change how variables/parameters are named by default, which might give better starting names to pass to the LLM:

Mangle: names are inferred based on type (variable, function, parameter, class) and some expressions

@Anooxy17 You could potentially try cloning the humanify repo, bumping webcrack to the latest version in package.json, and then seeing if that resolves this issue.

I'll also note that this seems like it might actually be a different issue to what was reported initially, so even if bumping webcrack to the latest version helps solve it, it may still be worth looking deeper into the cause of the original issue.

@0xdevalias commented on GitHub (Sep 26, 2024): > [errors.txt](https://github.com/user-attachments/files/17148386/errors.txt) > It was made in OpenAI, not even went into over 70% now This error seems to be related to the attempt to assign a value to an object within a `for` loop, which isn't allowed: ``` SyntaxError: Unexpected token (3562:43) 3560 | if (this.M == null) { 3561 | this.M = new CustomAnimationHandler(); > 3562 | for (var _________index = 0, _this.kLength = this.K; _________index < _this.kLength.length;) { | ^ 3563 | let kLengthValue = _this.kLength[_________index]; 3564 | ++_________index; 3565 | kLengthValue.I = null; at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472) at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798) at Object.parse (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:15:2885) at parse4 (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:21902:24) at async coreFormat (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:22355:7) at async formatWithCursor (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:22557:14) at async Object.format2 (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:23932:25) at async unminify (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:214:27) at async Command.<anonymous> (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:56814:3) { loc: { start: { line: 3562, column: 43 } }, ``` While I haven't looked deeply into it, this part makes me wonder if it's an issue with `webcrack` and how it's choosing to inline certain variables/etc within `for` loops. It seems `humanify` is currently using `webcrack` `2.13.0`: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/package.json#L61 Looking at the releases we can see that the latest version is `2.14.1`: - https://github.com/j4k0xb/webcrack/releases Looking at the release notes for `2.14.0`, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue: - https://github.com/j4k0xb/webcrack/releases/tag/v2.14.0 - > Extract sequence from for-of: `for (let value of (a = 1, array)) {}` -> `a = 1; for (let value of array) {}` It also seems to change how variables/parameters are named by default, which might give better starting names to pass to the LLM: - > Mangle: names are inferred based on type (variable, function, parameter, class) and some expressions --- @Anooxy17 You could potentially try cloning the `humanify` repo, bumping `webcrack` to the latest version in `package.json`, and then seeing if that resolves this issue. I'll also note that this seems like it might actually be a different issue to what was reported initially, so even if bumping `webcrack` to the latest version helps solve it, it may still be worth looking deeper into the cause of the original issue.

kerem commented

Author

Owner

@Anooxy17 commented on GitHub (Sep 26, 2024):

@0xdevalias
Yeees, it exactly looks like different error now, im crazy right now, doesnt matter how much I try, I cant really deobfuscate this file 😆
I will try when I came back to home and let u know then, ty

@Anooxy17 commented on GitHub (Sep 26, 2024): @0xdevalias Yeees, it exactly looks like different error now, im crazy right now, doesnt matter how much I try, I cant really deobfuscate this file 😆 I will try when I came back to home and let u know then, ty

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 27, 2024):

Looks like @Yusufkulcu is hitting a similar sort of error now, so perhaps they can try the workaround steps above and see if that helps in their case too:

SyntaxError: Unexpected token, expected "," (74:70)
72 |      ｝）；
73 |    };
74 |    subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) {
   |                                                                       ^
75 |      var sourceObject = this.source;
76 |      return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown);
77 |    ｝；

Thanks for the suggestion, it doesn't give that error anymore, but now I'm getting another error. I get the following error in file 63. I've added the original file and the file where I got the error below. It's a pretty long mistake, so I added some more pictures.

6blF.txt

main.txt

..snip..

..snip..

@Yusufkulcu I'm glad that workaround helped get around the first issue :)

That new error you're running into seems to now be the same/similar to the errors being discussed in this issue:

https://github.com/jehna/humanify/issues/117

You might like to try the potential solution I described in this comment, RE: cloning the humanify repo and bumping the webcrack version in package.json to the latest version (2.14.1):

https://github.com/jehna/humanify/issues/117#issuecomment-2378098164

Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/111#issuecomment-2378407904

@0xdevalias commented on GitHub (Sep 27, 2024): Looks like @Yusufkulcu is hitting a similar sort of error now, so perhaps they can try the workaround steps above and see if that helps in their case too: ```shell SyntaxError: Unexpected token, expected "," (74:70) 72 | ｝）； 73 | }; 74 | subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) { | ^ 75 | var sourceObject = this.source; 76 | return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown); 77 | ｝； ``` > > Thanks for the suggestion, it doesn't give that error anymore, but now I'm getting another error. I get the following error in file 63. I've added the original file and the file where I got the error below. It's a pretty long mistake, so I added some more pictures. > > > [6blF.txt](https://github.com/user-attachments/files/17158987/6blF.txt) > > > > [main.txt](https://github.com/user-attachments/files/17158990/main.txt) > > > ![image](https://private-user-images.githubusercontent.com/69112573/371389851-6518b0bb-7bad-4578-9e8a-29cb31cb96c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc0MTMzNzksIm5iZiI6MTcyNzQxMzA3OSwicGF0aCI6Ii82OTExMjU3My8zNzEzODk4NTEtNjUxOGIwYmItN2JhZC00NTc4LTllOGEtMjljYjMxY2I5NmM4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI3VDA0NTc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM1MTE1ZmQ1M2E1ZjdlMzc4N2UxYmFiODUzMjRhODVmMjRhNGExYzY1NWY3ODkxMjIxYjVhZjQwZDQzNDk2MWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Egw90ef2egH2t60qtkMxxIIo2bJOck2g5OHu3DW1yyA) > > > > ..snip.. > > > > ![image](https://private-user-images.githubusercontent.com/69112573/371390037-3f3ce8f4-85fc-4752-a306-e4f06709fed8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc0MTMzNzksIm5iZiI6MTcyNzQxMzA3OSwicGF0aCI6Ii82OTExMjU3My8zNzEzOTAwMzctM2YzY2U4ZjQtODVmYy00NzUyLWEzMDYtZTRmMDY3MDlmZWQ4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI3VDA0NTc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0MDNlNzM2MzBiN2MwOGVmOGRkN2MxODUwYmEyZjRmYmVkZjNmMGJhOGY3MWIzMDZkZTU0M2ZkYTdmMDU0NDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.PRBPC8-RLSWIAkdj4HZg4lEpahI-XvNOx1D9Zkb6H4E) > > > > ..snip.. > > @Yusufkulcu I'm glad that workaround helped get around the first issue :) > > That new error you're running into seems to now be the same/similar to the errors being discussed in this issue: > > - https://github.com/jehna/humanify/issues/117 > > You might like to try the potential solution I described in this comment, RE: cloning the `humanify` repo and bumping the `webcrack` version in `package.json` to the latest version (`2.14.1`): > > - https://github.com/jehna/humanify/issues/117#issuecomment-2378098164 > > _Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/111#issuecomment-2378407904_

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 27, 2024):

Looks like @Yusufkulcu is hitting a similar sort of error now

SyntaxError: Unexpected token, expected "," (74:70)
72 |      ｝）；
73 |    };
74 |    subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) {
   |                                                                       ^
75 |      var sourceObject = this.source;
76 |      return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown);
77 |    ｝；

6blF.txt

main.txt

I made a few patches to my locally checked out version of humanify to help provide better output info + skip through to the relevant files that seem to be causing errors:

src/unminify.ts:

diff --git a/src/unminify.ts b/src/unminify.ts
index 948b4df..ff95ae0 100644
--- a/src/unminify.ts
+++ b/src/unminify.ts
@@ -9,22 +9,42 @@ export async function unminify(
   plugins: ((code: string) => Promise<string>)[] = []
 ) {
   ensureFileExists(filename);
+
+  console.log(`Reading input file: ${filename}`);
   const bundledCode = await fs.readFile(filename, "utf-8");
+
+  console.log('Running webcrack to extract/decrypt bundled code...');
   const extractedFiles = await webcrack(bundledCode, outputDir);
 
+  console.log(`Unbundled ${extractedFiles.length} files from ${filename} into ${outputDir}`);
+
   for (let i = 0; i < extractedFiles.length; i++) {
-    console.log(`Processing file ${i + 1}/${extractedFiles.length}`);
-
     const file = extractedFiles[i];
+
+    console.log(`Processing file ${i + 1}/${extractedFiles.length} (${file.path})`);
+
+    // Skip extracted files till we find the one we actually want to try and process
+    if (file.path !== `${outputDir}/6blF.js`) {
+      console.log('  [hack] Skipping files that are not the one we want to process');
+      continue;
+    }
+
     const code = await fs.readFile(file.path, "utf-8");
-    const formattedCode = await plugins.reduce(
-      (p, next) => p.then(next),
-      Promise.resolve(code)
-    );
+    if (code) {
+      const formattedCode = await plugins.reduce(
+        (p, next) => p.then(next),
+        Promise.resolve(code)
+      );
 
-    verbose.log("Input: ", code);
-    verbose.log("Output: ", formattedCode);
+      verbose.log("  Input: ", code);
+      verbose.log("  Output: ", formattedCode);
 
-    await fs.writeFile(file.path, formattedCode);
+      await fs.writeFile(file.path, formattedCode);
+    }
+    else {
+      verbose.log(`  Skipping empty file ${i + 1} (${file.path})`);
+    }
   }
+
+  console.log(`Unminified (or skipped) ${extractedFiles.length} files from ${filename} into ${outputDir}`);
 }

Then tried running it against main.txt, configured to skip all of the extracted files except for output/6blF.js; but I didn't seem to run into an error when I ran it (for reference, I hadn't bumped my version of webcrack to the latest before running this):

This is how the lines that originally caused the error seemed to look like after being extracted with webcrack:

  // ..snip..
  _0x223285.prototype._subscribe = function (_0x3becc8) {
    var _0x2e0ffb = this.source;
    return _0x2e0ffb && _0x2e0ffb.subscribe(_0x3becc8);
  };
  // ..snip..

And this is how the equivalent code looked after my run of humanify had finished with it:

  // ..snip..
  createSubscription.prototype._subscribe = function (___dataProcessor) {
    var sourceSubscription = this.source;
    return sourceSubscription && sourceSubscription.subscribe(___dataProcessor);
  };
  // ..snip..

Which we can contrast against the version that @Yusufkulcu got the error with, which seemed to end up as:

  // ..snip..
  subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) {
    var sourceObject = this.source;
    return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown);
  };
  // ..snip..

In the original code, the _0x3becc8 seems to only exist in that functions parameter + the usage of it within the function. That is the same for ___dataProcessor within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name _subscribeHandler.syncErrorThrown as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment https://github.com/jehna/humanify/issues/111#issuecomment-2378375379 seem to not mention what variable renames were being applied; which should have been shown in the output from openaiRename (Ref) (assuming they were running humanify openai, and not humanify local or similar?)

@jehna probably knows the answer to this better.. but from memory of skimming through the humanify code I don't think there is any context shared between the files being processed; so I wouldn't have expected the 'skip other files' hack I implemented above should have impacted the outcome.. so I'm even more confused as to why I can't reproduce the error that @Yusufkulcu was seeing.

Edit: One thing I noticed is that when @Yusufkulcu was unpacking it output/6blF.js was file 63, yet when I was running unpacking it was file 66, so it seems webcrack's file order may not be the same each time it's run (even though the filenames seem consistent, at least based on this one example)

@0xdevalias commented on GitHub (Sep 27, 2024): > Looks like @Yusufkulcu is hitting a similar sort of error now > > ```shell > SyntaxError: Unexpected token, expected "," (74:70) > 72 | ｝）； > 73 | }; > 74 | subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) { > | ^ > 75 | var sourceObject = this.source; > 76 | return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown); > 77 | ｝； > ``` > > [6blF.txt](https://github.com/user-attachments/files/17158987/6blF.txt) > > [main.txt](https://github.com/user-attachments/files/17158990/main.txt) I made a few patches to my locally checked out version of `humanify` to help provide better output info + skip through to the relevant files that seem to be causing errors: `src/unminify.ts`: ```diff diff --git a/src/unminify.ts b/src/unminify.ts index 948b4df..ff95ae0 100644 --- a/src/unminify.ts +++ b/src/unminify.ts @@ -9,22 +9,42 @@ export async function unminify( plugins: ((code: string) => Promise<string>)[] = [] ) { ensureFileExists(filename); + + console.log(`Reading input file: ${filename}`); const bundledCode = await fs.readFile(filename, "utf-8"); + + console.log('Running webcrack to extract/decrypt bundled code...'); const extractedFiles = await webcrack(bundledCode, outputDir); + console.log(`Unbundled ${extractedFiles.length} files from ${filename} into ${outputDir}`); + for (let i = 0; i < extractedFiles.length; i++) { - console.log(`Processing file ${i + 1}/${extractedFiles.length}`); - const file = extractedFiles[i]; + + console.log(`Processing file ${i + 1}/${extractedFiles.length} (${file.path})`); + + // Skip extracted files till we find the one we actually want to try and process + if (file.path !== `${outputDir}/6blF.js`) { + console.log(' [hack] Skipping files that are not the one we want to process'); + continue; + } + const code = await fs.readFile(file.path, "utf-8"); - const formattedCode = await plugins.reduce( - (p, next) => p.then(next), - Promise.resolve(code) - ); + if (code) { + const formattedCode = await plugins.reduce( + (p, next) => p.then(next), + Promise.resolve(code) + ); - verbose.log("Input: ", code); - verbose.log("Output: ", formattedCode); + verbose.log(" Input: ", code); + verbose.log(" Output: ", formattedCode); - await fs.writeFile(file.path, formattedCode); + await fs.writeFile(file.path, formattedCode); + } + else { + verbose.log(` Skipping empty file ${i + 1} (${file.path})`); + } } + + console.log(`Unminified (or skipped) ${extractedFiles.length} files from ${filename} into ${outputDir}`); } ``` Then tried running it against `main.txt`, configured to skip all of the extracted files except for `output/6blF.js`; but I didn't seem to run into an error when I ran it (for reference, I hadn't bumped my version of `webcrack` to the latest before running this): - [main-filter-for-6blF-js-output-devalias.txt](https://github.com/user-attachments/files/17159535/main-filter-for-6blF-js-output-devalias.txt) - [6blF.js.txt](https://github.com/user-attachments/files/17159615/6blF.js.txt) This is how the lines that originally caused the error seemed to look like after being extracted with `webcrack`: ```js // ..snip.. _0x223285.prototype._subscribe = function (_0x3becc8) { var _0x2e0ffb = this.source; return _0x2e0ffb && _0x2e0ffb.subscribe(_0x3becc8); }; // ..snip.. ``` And this is how the equivalent code looked after my run of `humanify` had finished with it: ```js // ..snip.. createSubscription.prototype._subscribe = function (___dataProcessor) { var sourceSubscription = this.source; return sourceSubscription && sourceSubscription.subscribe(___dataProcessor); }; // ..snip.. ``` Which we can contrast against the version that @Yusufkulcu got the error with, which seemed to end up as: ```js // ..snip.. subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) { var sourceObject = this.source; return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown); }; // ..snip.. ``` In the original code, the `_0x3becc8` seems to only exist in that functions parameter + the usage of it within the function. That is the same for `___dataProcessor` within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name `_subscribeHandler.syncErrorThrown` as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment https://github.com/jehna/humanify/issues/111#issuecomment-2378375379 seem to not mention what variable renames were being applied; which should have been shown in the output from `openaiRename` ([Ref](https://github.com/jehna/humanify/blob/81ca7069b5235ba319e1927c6e5533db3aac1af5/src/plugins/openai/openai-rename.ts#L6-L38)) (assuming they were running `humanify openai`, and not `humanify local` or similar?) --- @jehna probably knows the answer to this better.. but from memory of skimming through the `humanify` code I don't think there is any context shared between the files being processed; so I wouldn't have expected the 'skip other files' hack I implemented above should have impacted the outcome.. so I'm even more confused as to why I can't reproduce the error that @Yusufkulcu was seeing. --- **Edit:** One thing I noticed is that when @Yusufkulcu was unpacking it `output/6blF.js` was file 63, yet when I was running unpacking it was file 66, so it seems `webcrack`'s file order may not be the same each time it's run (even though the filenames seem consistent, at least based on this one example)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

Looks like @Yusufkulcu is hitting a similar sort of error now, so perhaps they can try the workaround steps above and see if that helps in their case too:
SyntaxError: Unexpected token, expected "," (74:70)
72 |      ｝）；
73 |    };
74 |    subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) {
   |                                                                       ^
75 |      var sourceObject = this.source;
76 |      return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown);
77 |    ｝；
Thanks for the suggestion, it doesn't give that error anymore, but now I'm getting another error. I get the following error in file 63. I've added the original file and the file where I got the error below. It's a pretty long mistake, so I added some more pictures.

6blF.txt
main.txt

..snip..

..snip..

@Yusufkulcu I'm glad that workaround helped get around the first issue :)
That new error you're running into seems to now be the same/similar to the errors being discussed in this issue:

Syntax Error on Babel? #117

You might like to try the potential solution I described in this comment, RE: cloning the humanify repo and bumping the webcrack version in package.json to the latest version (2.14.1):

Syntax Error on Babel? #117 (comment)

Originally posted by @0xdevalias in #111 (comment)

Thank you very much for your reply. I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error.

@Yusufkulcu commented on GitHub (Sep 27, 2024): > Looks like @Yusufkulcu is hitting a similar sort of error now, so perhaps they can try the workaround steps above and see if that helps in their case too: > > ```shell > SyntaxError: Unexpected token, expected "," (74:70) > 72 | ｝）； > 73 | }; > 74 | subscribeHandler.prototype._subscribe = function (_subscribeHandler.syncErrorThrown) { > | ^ > 75 | var sourceObject = this.source; > 76 | return sourceObject && sourceObject.subscribe(_subscribeHandler.syncErrorThrown); > 77 | ｝； > ``` > > > > Thanks for the suggestion, it doesn't give that error anymore, but now I'm getting another error. I get the following error in file 63. I've added the original file and the file where I got the error below. It's a pretty long mistake, so I added some more pictures. > > > > > > > [6blF.txt](https://github.com/user-attachments/files/17158987/6blF.txt) > > > [main.txt](https://github.com/user-attachments/files/17158990/main.txt) > > > > > > > ![image](https://private-user-images.githubusercontent.com/69112573/371389851-6518b0bb-7bad-4578-9e8a-29cb31cb96c8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc0MTMzNzksIm5iZiI6MTcyNzQxMzA3OSwicGF0aCI6Ii82OTExMjU3My8zNzEzODk4NTEtNjUxOGIwYmItN2JhZC00NTc4LTllOGEtMjljYjMxY2I5NmM4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI3VDA0NTc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM1MTE1ZmQ1M2E1ZjdlMzc4N2UxYmFiODUzMjRhODVmMjRhNGExYzY1NWY3ODkxMjIxYjVhZjQwZDQzNDk2MWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Egw90ef2egH2t60qtkMxxIIo2bJOck2g5OHu3DW1yyA) > > > ..snip.. > > > ![image](https://private-user-images.githubusercontent.com/69112573/371390037-3f3ce8f4-85fc-4752-a306-e4f06709fed8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc0MTMzNzksIm5iZiI6MTcyNzQxMzA3OSwicGF0aCI6Ii82OTExMjU3My8zNzEzOTAwMzctM2YzY2U4ZjQtODVmYy00NzUyLWEzMDYtZTRmMDY3MDlmZWQ4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI3VDA0NTc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0MDNlNzM2MzBiN2MwOGVmOGRkN2MxODUwYmEyZjRmYmVkZjNmMGJhOGY3MWIzMDZkZTU0M2ZkYTdmMDU0NDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.PRBPC8-RLSWIAkdj4HZg4lEpahI-XvNOx1D9Zkb6H4E) > > > ..snip.. > > > > > > @Yusufkulcu I'm glad that workaround helped get around the first issue :) > > That new error you're running into seems to now be the same/similar to the errors being discussed in this issue: > > > > * [Syntax Error on Babel? #117](https://github.com/jehna/humanify/issues/117) > > > > You might like to try the potential solution I described in this comment, RE: cloning the `humanify` repo and bumping the `webcrack` version in `package.json` to the latest version (`2.14.1`): > > > > * [Syntax Error on Babel? #117 (comment)](https://github.com/jehna/humanify/issues/117#issuecomment-2378098164) > > > > _Originally posted by @0xdevalias in [#111 (comment)](https://github.com/jehna/humanify/issues/111#issuecomment-2378407904)_ Thank you very much for your reply. I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error. ![Screenshot_20240927_110444_Chrome](https://github.com/user-attachments/assets/1b496c05-1daa-4058-bb9b-e0eb755d94ad) ![Screenshot_20240927_110512_Chrome](https://github.com/user-attachments/assets/609aba2c-6c20-45ec-84fa-84cf06fb9db4) ![Screenshot_20240927_110742_Chrome](https://github.com/user-attachments/assets/e6e79902-47aa-4d4d-8d7f-d3afad8db70c)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

@0xdevalias I am also grateful for your detailed Explanations

@Yusufkulcu commented on GitHub (Sep 27, 2024): @0xdevalias I am also grateful for your detailed Explanations

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

Instead of writing it on my own record, I mixed it up and wrote it here. Sorry

@Yusufkulcu commented on GitHub (Sep 27, 2024): Instead of writing it on my own record, I mixed it up and wrote it here. Sorry

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 27, 2024):

I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error.

@Yusufkulcu Are you running humanify openai or one of the other commands?

Are you running with --verbose?

It looks like the screenshots of your latest output are still missing the critical detail I mentioned above; of what the before and after names are when the variables are renamed.

Instead of writing it on my own record, I mixed it up and wrote it here. Sorry

@Yusufkulcu Thats fine; now that you're getting the same types of errors; this is a good place to centralise the discussion of these types of error.

@0xdevalias commented on GitHub (Sep 27, 2024): > I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error. @Yusufkulcu Are you running `humanify openai` or one of the other commands? Are you running with `--verbose`? It looks like the screenshots of your latest output are still missing the critical detail I mentioned above; of what the before and after names are when the variables are renamed. --- > Instead of writing it on my own record, I mixed it up and wrote it here. Sorry @Yusufkulcu Thats fine; now that you're getting the same types of errors; this is a good place to centralise the discussion of these types of error.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error.

@Yusufkulcu Are you running humanify openai or one of the other commands?

Are you running with --verbose?

It looks like the screenshots of your latest output are still missing the critical detail I mentioned above; of what the before and after names are when the variables are renamed.

Instead of writing it on my own record, I mixed it up and wrote it here. Sorry

@Yusufkulcu Thats fine; now that you're getting the same types of errors; this is a good place to centralise the discussion of these types of error.

I'm using OpenAI. Right now - I ran it with -verbose . How can I post the Verbose log here

@Yusufkulcu commented on GitHub (Sep 27, 2024): > > I updated the "webcrack" package, but the same problem persists. The problem is another file this time. Interestingly, I can't find the file with the error. > > @Yusufkulcu Are you running `humanify openai` or one of the other commands? > > Are you running with `--verbose`? > > It looks like the screenshots of your latest output are still missing the critical detail I mentioned above; of what the before and after names are when the variables are renamed. > > > Instead of writing it on my own record, I mixed it up and wrote it here. Sorry > > @Yusufkulcu Thats fine; now that you're getting the same types of errors; this is a good place to centralise the discussion of these types of error. I'm using OpenAI. Right now - I ran it with -verbose . How can I post the Verbose log here

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 27, 2024):

Right now - I ran it with -verbose . How can I post the Verbose log here

@Yusufkulcu Cool, does it have the log lines that talk about 'Renaming VAR' and 'Renamed to NEWVAR' or similar? Did it still get the error?

It should look something like this:

..snip..

Processing file 66/559 (output/6blF.js)

[2024-09-27 05:35:05]  Renaming _0x29881a
[2024-09-27 05:35:05]  Context:  var _0x29881a = require("./FFOo.js");
var _0x21aff1 = require("./L/V9.js");
var _0x332379 = require("./6ahw.js");
var _0x51037e = require("./xTla.js");
var _0x456d11 = require("./y3By.js");
var _0x2ad
[2024-09-27 05:35:06]  Renamed to FFOoModule

[2024-09-27 05:35:06]  Processing: 2%
[2024-09-27 05:35:06]  Renaming _0x21aff1
[2024-09-27 05:35:06]  Context:  var FFOoModule = require("./FFOo.js");
var _0x21aff1 = require("./L/V9.js");
var _0x332379 = require("./6ahw.js");
var _0x51037e = require("./xTla.js");
var _0x456d11 = require("./y3By.js");
var _0x2a
[2024-09-27 05:35:06]  Renamed to versionChecker

..snip..

I usually copy/paste all of the text output from my terminal into a .txt file (eg. With notepad if you're on windows); then you should be able to attach that to a comment here.

@0xdevalias commented on GitHub (Sep 27, 2024): > Right now - I ran it with -verbose . How can I post the Verbose log here @Yusufkulcu Cool, does it have the log lines that talk about 'Renaming VAR' and 'Renamed to NEWVAR' or similar? Did it still get the error? It should look something like this: ``` ..snip.. Processing file 66/559 (output/6blF.js) [2024-09-27 05:35:05] Renaming _0x29881a [2024-09-27 05:35:05] Context: var _0x29881a = require("./FFOo.js"); var _0x21aff1 = require("./L/V9.js"); var _0x332379 = require("./6ahw.js"); var _0x51037e = require("./xTla.js"); var _0x456d11 = require("./y3By.js"); var _0x2ad [2024-09-27 05:35:06] Renamed to FFOoModule [2024-09-27 05:35:06] Processing: 2% [2024-09-27 05:35:06] Renaming _0x21aff1 [2024-09-27 05:35:06] Context: var FFOoModule = require("./FFOo.js"); var _0x21aff1 = require("./L/V9.js"); var _0x332379 = require("./6ahw.js"); var _0x51037e = require("./xTla.js"); var _0x456d11 = require("./y3By.js"); var _0x2a [2024-09-27 05:35:06] Renamed to versionChecker ..snip.. ``` I usually copy/paste all of the text output from my terminal into a `.txt` file (eg. With notepad if you're on windows); then you should be able to attach that to a comment here.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

Right now - I ran it with -verbose . How can I post the Verbose log here

@Yusufkulcu Cool, does it have the log lines that talk about 'Renaming VAR' and 'Renamed to NEWVAR' or similar? Did it still get the error?

It should look something like this:
..snip..

Processing file 66/559 (output/6blF.js)

[2024-09-27 05:35:05]  Renaming _0x29881a
[2024-09-27 05:35:05]  Context:  var _0x29881a = require("./FFOo.js");
var _0x21aff1 = require("./L/V9.js");
var _0x332379 = require("./6ahw.js");
var _0x51037e = require("./xTla.js");
var _0x456d11 = require("./y3By.js");
var _0x2ad
[2024-09-27 05:35:06]  Renamed to FFOoModule

[2024-09-27 05:35:06]  Processing: 2%
[2024-09-27 05:35:06]  Renaming _0x21aff1
[2024-09-27 05:35:06]  Context:  var FFOoModule = require("./FFOo.js");
var _0x21aff1 = require("./L/V9.js");
var _0x332379 = require("./6ahw.js");
var _0x51037e = require("./xTla.js");
var _0x456d11 = require("./y3By.js");
var _0x2a
[2024-09-27 05:35:06]  Renamed to versionChecker

..snip..
I usually copy/paste all of the text output from my terminal into a .txt file (eg. With notepad if you're on windows); then you should be able to attach that to a comment here.

@Yusufkulcu commented on GitHub (Sep 27, 2024): > > Right now - I ran it with -verbose . How can I post the Verbose log here > > @Yusufkulcu Cool, does it have the log lines that talk about 'Renaming VAR' and 'Renamed to NEWVAR' or similar? Did it still get the error? > > It should look something like this: > > ``` > ..snip.. > > Processing file 66/559 (output/6blF.js) > > [2024-09-27 05:35:05] Renaming _0x29881a > [2024-09-27 05:35:05] Context: var _0x29881a = require("./FFOo.js"); > var _0x21aff1 = require("./L/V9.js"); > var _0x332379 = require("./6ahw.js"); > var _0x51037e = require("./xTla.js"); > var _0x456d11 = require("./y3By.js"); > var _0x2ad > [2024-09-27 05:35:06] Renamed to FFOoModule > > [2024-09-27 05:35:06] Processing: 2% > [2024-09-27 05:35:06] Renaming _0x21aff1 > [2024-09-27 05:35:06] Context: var FFOoModule = require("./FFOo.js"); > var _0x21aff1 = require("./L/V9.js"); > var _0x332379 = require("./6ahw.js"); > var _0x51037e = require("./xTla.js"); > var _0x456d11 = require("./y3By.js"); > var _0x2a > [2024-09-27 05:35:06] Renamed to versionChecker > > ..snip.. > ``` > > I usually copy/paste all of the text output from my terminal into a `.txt` file (eg. With notepad if you're on windows); then you should be able to attach that to a comment here. ![Screenshot_20240927_132354_Chrome](https://github.com/user-attachments/assets/d182b303-f1c1-4f29-bcc8-f659d000feda) ![Screenshot_20240927_132521_Chrome](https://github.com/user-attachments/assets/6f8bf83a-5055-42da-ba80-c55eee37aea8)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

@0xdevalias

I noticed that the syntax error appears in a different file every time. This is something I don't understand.

@Yusufkulcu commented on GitHub (Sep 27, 2024): @0xdevalias I noticed that the syntax error appears in a different file every time. This is something I don't understand.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 27, 2024):

It's really strange, the error occurs in a different file each time. I am adding the error logs below. "main.txt" is the original file, "+Branch.txt" is the file from which the error was received.

I use the "humanify openi" command

+Dal.txt

main.txt

verbose-log.txt

@Yusufkulcu commented on GitHub (Sep 27, 2024): It's really strange, the error occurs in a different file each time. I am adding the error logs below. "main.txt" is the original file, "+Branch.txt" is the file from which the error was received. I use the "humanify openi" command ![image](https://github.com/user-attachments/assets/b0c438e2-28c0-49e3-b466-7674fa6f376c) ![image](https://github.com/user-attachments/assets/29bc54dc-7cb5-450c-b662-724f8d1e56b6) ![image](https://github.com/user-attachments/assets/409f52e0-dbde-40ac-afe8-a27a2622eb33) [+Dal.txt](https://github.com/user-attachments/files/17168644/%2BDal.txt) [main.txt](https://github.com/user-attachments/files/17168649/main.txt) [verbose-log.txt](https://github.com/user-attachments/files/17168691/verbose-log.txt)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 28, 2024):

@0xdevalias @jehna I wonder if you have any ideas or suggestions.

@Yusufkulcu commented on GitHub (Sep 28, 2024): @0xdevalias @jehna I wonder if you have any ideas or suggestions.

kerem commented

Author

Owner

@j4k0xb commented on GitHub (Sep 28, 2024):

I think I found the root issue:

The LLM generates this.kLength as a name -> detected as invalid and changed to _this.kLength:
github.com/jehna/humanify@64a1b9511d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L44-L50)

The proper fix would be to use toIdentifier from @babel/types

I noticed that the syntax error appears in a different file every time. This is something I don't understand.

LLM use a random sampling technique

Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue:

That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅
edit: https://github.com/jehna/humanify/pull/38 it's already the newest, just not in package.json for some reason

@j4k0xb commented on GitHub (Sep 28, 2024): I think I found the root issue: The LLM generates `this.kLength` as a name -> detected as invalid and changed to `_this.kLength`: https://github.com/jehna/humanify/blob/64a1b9511d2e42b705df0858674eab1f5e83dd0d/src/plugins/local-llm-rename/visit-all-identifiers.ts#L44-L50 The proper fix would be to use `toIdentifier` from `@babel/types` > I noticed that the syntax error appears in a different file every time. This is something I don't understand. LLM use a random sampling technique > Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue: That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅 edit: https://github.com/jehna/humanify/pull/38 it's already the newest, just not in package.json for some reason

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 28, 2024):

toIdentifier

Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem

@Yusufkulcu commented on GitHub (Sep 28, 2024): > `toIdentifier` Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem

kerem commented

Author

Owner

@j4k0xb commented on GitHub (Sep 28, 2024):

Gonna make a PR in a few minutes

@j4k0xb commented on GitHub (Sep 28, 2024): Gonna make a PR in a few minutes

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 28, 2024):

Gonna make a PR in a few minutes

I am waiting

@Yusufkulcu commented on GitHub (Sep 28, 2024): > Gonna make a PR in a few minutes I am waiting

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 28, 2024):

@j4k0xb

This fix seems to have worked, the files that previously gave errors are now progressing smoothly. I will inform you when the process is completed.

@Yusufkulcu commented on GitHub (Sep 28, 2024): @j4k0xb This fix seems to have worked, the files that previously gave errors are now progressing smoothly. I will inform you when the process is completed.

kerem commented

github.com/jehna/humanify@64a1b9511d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L44)

Author

Owner

@0xdevalias commented on GitHub (Sep 29, 2024):

verbose-log.txt

@Yusufkulcu Thanks for that verbose log, it seems to confirm that the bug is what I thought it might have been, with the LLM choosing an invalid variable name:

It looks like the problem is less that there should be a ,, and more that that something is adding the _static prefix to the variable where it presumably shouldn't be.

I can see that a lot of the function names have a static prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking.

This is still my main theory though, particularly given the same _static calculateQualityParameter 'variable name' can be seen on lines 1327, 1329, 1330:
SyntaxError: Unexpected token, expected "," (1327:23)
  1325 |       return dataPointDetails;
  1326 |     }
> 1327 |     static Qp(_static calculateQualityParameter, qualityParameter) {
       |                       ^
  1328 |       let qualityCalculator = new InvestmentBalance();
  1329 |       var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil);
  1330 |       var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil);
    at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472)
    at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798)
I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use static calculateQualityParameter as a variable name, and then humanify prefixed it with _; or similar based on something like this:

https://github.com/jehna/humanify/issues/20

Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/117#issuecomment-2375810553

In the original code, the _0x3becc8 seems to only exist in that functions parameter + the usage of it within the function. That is the same for ___dataProcessor within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name _subscribeHandler.syncErrorThrown as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment https://github.com/jehna/humanify/issues/111#issuecomment-2378375379 seem to not mention what variable renames were being applied; which should have been shown in the output from openaiRename (Ref) (assuming they were running humanify openai, and not humanify local or similar?)

Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/117#issuecomment-2378447998

While the context it's showing doesn't make a lot of sense to me as to where _0x53274b fits into it, we can see that the variable name chosen by the LLM was setChangeDetection.prototype.updateValue, which obviously isn't valid for a single variable name:

[2024-09-27 18:56:10]  Renaming _0x53274b
[2024-09-27 18:56:10]  Context:  setChangeDetection.prototype.writeValue = function (initModules) {
    if (initModules % 1 !== initModules) {
      this.value = Math.round(initModules);
      this.preValue = initModules;
      this.changeDet
[2024-09-27 18:56:11]  Renamed to setChangeDetection.prototype.updateValue

I noticed that the syntax error appears in a different file every time. This is something I don't understand.

@Yusufkulcu This is happening because the bug is related to the LLM suggesting an invalid variable name. Because the LLM can come up with different answers every time it's run (related: https://github.com/jehna/humanify/issues/97), this is why the error ends up being in a different file each time.

I think I found the root issue:

The LLM generates this.kLength as a name -> detected as invalid and changed to _this.kLength:

@j4k0xb Agreed, that's the bug as I see it too. Thanks for pointing out the safeRenamed / isValidIdentifier part. I knew I read something about prefixing variables somewhere (eg. https://github.com/jehna/humanify/issues/8, https://github.com/jehna/humanify/issues/20), but hadn't looked deeply into the code to figure out where that was happening.

The proper fix would be to use toIdentifier from @babel/types

@j4k0xb @jehna From a 2sec search I couldn't find rendered docs, but here's the relevant source for each:

isValidIdentifier: github.com/babel/babel@a2025d7d69/packages/babel-types/src/validators/isValidIdentifier.ts (L7-L25)
toIdentifier: github.com/babel/babel@a2025d7d69/packages/babel-types/src/converters/toIdentifier.ts (L4-L26)

toIdentifier definitely seems like a more robust approach than the current 'prefix with _' approach for sure.

Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Like instead of just forcing an invalid suggestion to be valid (with toIdentifier), we could detect that it's invalid (with isValidIdentifier) and then provide that feedback to the LLM, asking it to give a new suggestion; probably with some max retry limit; after which we could fall back to using the invalid suggestion run through toIdentifier, or log a warning and leave it un-renamed or similar.

Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue:

That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅

@j4k0xb Haha, yeah, that's fair enough. I didn't look too deeply + we didn't have a lot more information at the time so was just throwing out ideas/potentials 😅

Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem

@Yusufkulcu Basically on this line:

Since toIdentifier calls isValidIdentifier within itself, the basic workaround change could be as simple as:

- let safeRenamed = isValidIdentifier(renamed) ? renamed : `_${renamed}`;
+ let safeRenamed = toIdentifier(renamed);

Gonna make a PR in a few minutes

@j4k0xb Thanks for making a PR with the workaround fix!

https://github.com/jehna/humanify/pull/133

Will add some thoughts to it (though realistically they can probably be handled as a separate PR)

@0xdevalias commented on GitHub (Sep 29, 2024): > verbose-log.txt @Yusufkulcu Thanks for that verbose log, it seems to confirm that the bug is what I thought it might have been, with the LLM choosing an invalid variable name: > > It looks like the problem is less that there should be a `,`, and more that that something is adding the `_static` prefix to the variable where it presumably shouldn't be. > > > > I can see that a lot of the function names have a `static` prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking. > > This is still my main theory though, particularly given the same `_static calculateQualityParameter` 'variable name' can be seen on lines 1327, 1329, 1330: > > ``` > SyntaxError: Unexpected token, expected "," (1327:23) > 1325 | return dataPointDetails; > 1326 | } > > 1327 | static Qp(_static calculateQualityParameter, qualityParameter) { > | ^ > 1328 | let qualityCalculator = new InvestmentBalance(); > 1329 | var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil); > 1330 | var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil); > at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472) > at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798) > ``` > > I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use `static calculateQualityParameter` as a variable name, and then `humanify` prefixed it with `_`; or similar based on something like this: > > - https://github.com/jehna/humanify/issues/20 > > _Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/117#issuecomment-2375810553_ > In the original code, the `_0x3becc8` seems to only exist in that functions parameter + the usage of it within the function. That is the same for `___dataProcessor` within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name `_subscribeHandler.syncErrorThrown` as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment https://github.com/jehna/humanify/issues/111#issuecomment-2378375379 seem to not mention what variable renames were being applied; which should have been shown in the output from `openaiRename` ([Ref](https://github.com/jehna/humanify/blob/81ca7069b5235ba319e1927c6e5533db3aac1af5/src/plugins/openai/openai-rename.ts#L6-L38)) (assuming they were running `humanify openai`, and not `humanify local` or similar?) > > _Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/117#issuecomment-2378447998_ While the context it's showing doesn't make a lot of sense to me as to where `_0x53274b` fits into it, we can see that the variable name chosen by the LLM was `setChangeDetection.prototype.updateValue`, which obviously isn't valid for a single variable name: ``` [2024-09-27 18:56:10] Renaming _0x53274b [2024-09-27 18:56:10] Context: setChangeDetection.prototype.writeValue = function (initModules) { if (initModules % 1 !== initModules) { this.value = Math.round(initModules); this.preValue = initModules; this.changeDet [2024-09-27 18:56:11] Renamed to setChangeDetection.prototype.updateValue ``` --- > I noticed that the syntax error appears in a different file every time. This is something I don't understand. @Yusufkulcu This is happening because the bug is related to the LLM suggesting an invalid variable name. Because the LLM can come up with different answers every time it's run (related: https://github.com/jehna/humanify/issues/97), this is why the error ends up being in a different file each time. --- > I think I found the root issue: > > The LLM generates `this.kLength` as a name -> detected as invalid and changed to `_this.kLength`: @j4k0xb Agreed, that's the bug as I see it too. Thanks for pointing out the [`safeRenamed` / `isValidIdentifier`](https://github.com/jehna/humanify/blob/64a1b9511d2e42b705df0858674eab1f5e83dd0d/src/plugins/local-llm-rename/visit-all-identifiers.ts#L44) part. I knew I read something about prefixing variables somewhere (eg. https://github.com/jehna/humanify/issues/8, https://github.com/jehna/humanify/issues/20), but hadn't looked deeply into the code to figure out where that was happening. --- > The proper fix would be to use `toIdentifier` from `@babel/types` @j4k0xb @jehna From a 2sec search I couldn't find rendered docs, but here's the relevant source for each: - `isValidIdentifier`: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/validators/isValidIdentifier.ts#L7-L25 - `toIdentifier`: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/converters/toIdentifier.ts#L4-L26 `toIdentifier` definitely seems like a more robust approach than the current 'prefix with `_`' approach for sure. Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Like instead of just forcing an invalid suggestion to be valid (with `toIdentifier`), we could detect that it's invalid (with `isValidIdentifier`) and then provide that feedback to the LLM, asking it to give a new suggestion; probably with some max retry limit; after which we could fall back to using the invalid suggestion run through `toIdentifier`, or log a warning and leave it un-renamed or similar. --- > > Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue: > > That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅 @j4k0xb Haha, yeah, that's fair enough. I didn't look too deeply + we didn't have a lot more information at the time so was just throwing out ideas/potentials 😅 --- > Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem @Yusufkulcu Basically on this line: https://github.com/jehna/humanify/blob/64a1b9511d2e42b705df0858674eab1f5e83dd0d/src/plugins/local-llm-rename/visit-all-identifiers.ts#L44 Since [`toIdentifier`](https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/converters/toIdentifier.ts#L4-L26) calls [`isValidIdentifier`](https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/validators/isValidIdentifier.ts#L7-L25) within itself, the basic workaround change could be as simple as: ```diff - let safeRenamed = isValidIdentifier(renamed) ? renamed : `_${renamed}`; + let safeRenamed = toIdentifier(renamed); ``` --- > Gonna make a PR in a few minutes @j4k0xb Thanks for making a PR with the workaround fix! - https://github.com/jehna/humanify/pull/133 Will add some thoughts to it (though realistically they can probably be handled as a separate PR)

kerem commented

Author

Owner

@j4k0xb commented on GitHub (Sep 29, 2024):

Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well.

Yeah there's for sure a way to improve (also mentioned in other issues afaik) if you're willing to do the work.
That PR is just a quick fix to prevent crashing.

@j4k0xb commented on GitHub (Sep 29, 2024): > Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Yeah there's for sure a way to improve (also mentioned in other issues afaik) if you're willing to do the work. That PR is just a quick fix to prevent crashing.

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 29, 2024):

also mentioned in other issues afaik

@j4k0xb Yup, seems I mentioned a similar pattern in this comment:

https://github.com/jehna/humanify/issues/8#issuecomment-1853022637

Made a reference back to the suggestion here in this followup comment for continuity:

https://github.com/jehna/humanify/issues/8#issuecomment-2381065685

@0xdevalias commented on GitHub (Sep 29, 2024): > also mentioned in other issues afaik @j4k0xb Yup, seems I mentioned a similar pattern in this comment: - https://github.com/jehna/humanify/issues/8#issuecomment-1853022637 Made a reference back to the suggestion here in this followup comment for continuity: - https://github.com/jehna/humanify/issues/8#issuecomment-2381065685

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 29, 2024):

I had a question. The corresponding file size in the image is too large (134881 lines). And the processing of the file seems to take a very long time. Is there a faster way to do this? If the terminal screen closes the process will reset and that would be very sad

@Yusufkulcu commented on GitHub (Sep 29, 2024): I had a question. The corresponding file size in the image is too large (134881 lines). And the processing of the file seems to take a very long time. Is there a faster way to do this? If the terminal screen closes the process will reset and that would be very sad ![image](https://github.com/user-attachments/assets/71795558-83eb-4dc4-b8ea-f349089f5438)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 29, 2024):

verbose-log.txt

@Yusufkulcu Thanks for that verbose log, it seems to confirm that the bug is what I thought it might have been, with the LLM choosing an invalid variable name:
It looks like the problem is less that there should be a ,, and more that that something is adding the _static prefix to the variable where it presumably shouldn't be.
I can see that a lot of the function names have a static prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking.

This is still my main theory though, particularly given the same _static calculateQualityParameter 'variable name' can be seen on lines 1327, 1329, 1330:
SyntaxError: Unexpected token, expected "," (1327:23)
  1325 |       return dataPointDetails;
  1326 |     }
> 1327 |     static Qp(_static calculateQualityParameter, qualityParameter) {
       |                       ^
  1328 |       let qualityCalculator = new InvestmentBalance();
  1329 |       var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil);
  1330 |       var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil);
    at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472)
    at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798)
I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use static calculateQualityParameter as a variable name, and then humanify prefixed it with _; or similar based on something like this:

Expand reserved Keywords list for renaming variables or functions #20

Originally posted by @0xdevalias in #117 (comment)
In the original code, the _0x3becc8 seems to only exist in that functions parameter + the usage of it within the function. That is the same for ___dataProcessor within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name _subscribeHandler.syncErrorThrown as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment #111 (comment) seem to not mention what variable renames were being applied; which should have been shown in the output from openaiRename (Ref) (assuming they were running humanify openai, and not humanify local or similar?)
Originally posted by @0xdevalias in #117 (comment)

While the context it's showing doesn't make a lot of sense to me as to where _0x53274b fits into it, we can see that the variable name chosen by the LLM was setChangeDetection.prototype.updateValue, which obviously isn't valid for a single variable name:
[2024-09-27 18:56:10]  Renaming _0x53274b
[2024-09-27 18:56:10]  Context:  setChangeDetection.prototype.writeValue = function (initModules) {
    if (initModules % 1 !== initModules) {
      this.value = Math.round(initModules);
      this.preValue = initModules;
      this.changeDet
[2024-09-27 18:56:11]  Renamed to setChangeDetection.prototype.updateValue
I noticed that the syntax error appears in a different file every time. This is something I don't understand.

@Yusufkulcu This is happening because the bug is related to the LLM suggesting an invalid variable name. Because the LLM can come up with different answers every time it's run (related: #97), this is why the error ends up being in a different file each time.

I think I found the root issue:
The LLM generates this.kLength as a name -> detected as invalid and changed to _this.kLength:

@j4k0xb Agreed, that's the bug as I see it too. Thanks for pointing out the safeRenamed / isValidIdentifier part. I knew I read something about prefixing variables somewhere (eg. #8, #20), but hadn't looked deeply into the code to figure out where that was happening.

The proper fix would be to use toIdentifier from @babel/types

@j4k0xb @jehna From a 2sec search I couldn't find rendered docs, but here's the relevant source for each:

isValidIdentifier: github.com/babel/babel@a2025d7d69/packages/babel-types/src/validators/isValidIdentifier.ts (L7-L25)

toIdentifier: github.com/babel/babel@a2025d7d69/packages/babel-types/src/converters/toIdentifier.ts (L4-L26)

toIdentifier definitely seems like a more robust approach than the current 'prefix with _' approach for sure.

Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Like instead of just forcing an invalid suggestion to be valid (with toIdentifier), we could detect that it's invalid (with isValidIdentifier) and then provide that feedback to the LLM, asking it to give a new suggestion; probably with some max retry limit; after which we could fall back to using the invalid suggestion run through toIdentifier, or log a warning and leave it un-renamed or similar.

Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue:

That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅

@j4k0xb Haha, yeah, that's fair enough. I didn't look too deeply + we didn't have a lot more information at the time so was just throwing out ideas/potentials 😅

Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem

@Yusufkulcu Basically on this line:

github.com/jehna/humanify@64a1b9511d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L44)

Since toIdentifier calls isValidIdentifier within itself, the basic workaround change could be as simple as:
- let safeRenamed = isValidIdentifier(renamed) ? renamed : `_${renamed}`;
+ let safeRenamed = toIdentifier(renamed);
Gonna make a PR in a few minutes

@j4k0xb Thanks for making a PR with the workaround fix!

fix: invalid identifiers #133

Will add some thoughts to it (though realistically they can probably be handled as a separate PR)

Thank you for your help, my problem has been solved

@Yusufkulcu commented on GitHub (Sep 29, 2024): > > verbose-log.txt > > @Yusufkulcu Thanks for that verbose log, it seems to confirm that the bug is what I thought it might have been, with the LLM choosing an invalid variable name: > > > > It looks like the problem is less that there should be a `,`, and more that that something is adding the `_static` prefix to the variable where it presumably shouldn't be. > > > I can see that a lot of the function names have a `static` prefix; not sure if that ended up confusing the LLM when it was trying to come up with useful param renames or similar; but that is where I would start looking. > > > > > > This is still my main theory though, particularly given the same `_static calculateQualityParameter` 'variable name' can be seen on lines 1327, 1329, 1330: > > ``` > > SyntaxError: Unexpected token, expected "," (1327:23) > > 1325 | return dataPointDetails; > > 1326 | } > > > 1327 | static Qp(_static calculateQualityParameter, qualityParameter) { > > | ^ > > 1328 | let qualityCalculator = new InvestmentBalance(); > > 1329 | var qualityParameterValue = _UtilityClass.J(_static calculateQualityParameter.d0, getPeWithEmptyStringUtil); > > 1330 | var qualityData = _UtilityClass.J(_static calculateQualityParameter.d1, getPeWithEmptyStringUtil); > > at vr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15472) > > at Lr (file:///C:/Users/ktepe/AppData/Roaming/npm/node_modules/humanifyjs/dist/babel-C_YYVHCf.mjs:14:15798) > > ``` > > > > > > > > > > > > > > > > > > > > > > > > I haven't looked deeply at the code that handles variable names returned from openai, but I am assuming OpenAI said to use `static calculateQualityParameter` as a variable name, and then `humanify` prefixed it with `_`; or similar based on something like this: > > > > * [Expand reserved Keywords list for renaming variables or functions #20](https://github.com/jehna/humanify/issues/20) > > > > _Originally posted by @0xdevalias in [#117 (comment)](https://github.com/jehna/humanify/issues/117#issuecomment-2375810553)_ > > > In the original code, the `_0x3becc8` seems to only exist in that functions parameter + the usage of it within the function. That is the same for `___dataProcessor` within my version. So my best guess is that with @Yusufkulcu's version, the LLM must have chosen the invalid variable name `_subscribeHandler.syncErrorThrown` as a replacement or similar, but I can't really validate that as the screenshots/output shown in this comment [#111 (comment)](https://github.com/jehna/humanify/issues/111#issuecomment-2378375379) seem to not mention what variable renames were being applied; which should have been shown in the output from `openaiRename` ([Ref](https://github.com/jehna/humanify/blob/81ca7069b5235ba319e1927c6e5533db3aac1af5/src/plugins/openai/openai-rename.ts#L6-L38)) (assuming they were running `humanify openai`, and not `humanify local` or similar?) > > _Originally posted by @0xdevalias in [#117 (comment)](https://github.com/jehna/humanify/issues/117#issuecomment-2378447998)_ > > While the context it's showing doesn't make a lot of sense to me as to where `_0x53274b` fits into it, we can see that the variable name chosen by the LLM was `setChangeDetection.prototype.updateValue`, which obviously isn't valid for a single variable name: > > ``` > [2024-09-27 18:56:10] Renaming _0x53274b > [2024-09-27 18:56:10] Context: setChangeDetection.prototype.writeValue = function (initModules) { > if (initModules % 1 !== initModules) { > this.value = Math.round(initModules); > this.preValue = initModules; > this.changeDet > [2024-09-27 18:56:11] Renamed to setChangeDetection.prototype.updateValue > ``` > > > I noticed that the syntax error appears in a different file every time. This is something I don't understand. > > @Yusufkulcu This is happening because the bug is related to the LLM suggesting an invalid variable name. Because the LLM can come up with different answers every time it's run (related: #97), this is why the error ends up being in a different file each time. > > > I think I found the root issue: > > The LLM generates `this.kLength` as a name -> detected as invalid and changed to `_this.kLength`: > > @j4k0xb Agreed, that's the bug as I see it too. Thanks for pointing out the [`safeRenamed` / `isValidIdentifier`](https://github.com/jehna/humanify/blob/64a1b9511d2e42b705df0858674eab1f5e83dd0d/src/plugins/local-llm-rename/visit-all-identifiers.ts#L44) part. I knew I read something about prefixing variables somewhere (eg. #8, #20), but hadn't looked deeply into the code to figure out where that was happening. > > > The proper fix would be to use `toIdentifier` from `@babel/types` > > @j4k0xb @jehna From a 2sec search I couldn't find rendered docs, but here's the relevant source for each: > > * `isValidIdentifier`: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/validators/isValidIdentifier.ts#L7-L25 > * `toIdentifier`: https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/converters/toIdentifier.ts#L4-L26 > > `toIdentifier` definitely seems like a more robust approach than the current 'prefix with `_`' approach for sure. > > Though I wonder if a 'proper fix' should also involve tweaking how we prompt for/filter the suggestions coming back from the LLM itself as well. Like instead of just forcing an invalid suggestion to be valid (with `toIdentifier`), we could detect that it's invalid (with `isValidIdentifier`) and then provide that feedback to the LLM, asking it to give a new suggestion; probably with some max retry limit; after which we could fall back to using the invalid suggestion run through `toIdentifier`, or log a warning and leave it un-renamed or similar. > > > > Looking at the release notes for 2.14.0, there is a line that talks about extracting sequences from for loops; I wonder if that would also fix this issue: > > > > > > That's a different type of for-loop. But I'd recommend updating either way because of other bugs 😅 > > @j4k0xb Haha, yeah, that's fair enough. I didn't look too deeply + we didn't have a lot more information at the time so was just throwing out ideas/potentials 😅 > > > Where should the "toIdentifier" function be used? I don't fully understand the solution to the problem > > @Yusufkulcu Basically on this line: > > https://github.com/jehna/humanify/blob/64a1b9511d2e42b705df0858674eab1f5e83dd0d/src/plugins/local-llm-rename/visit-all-identifiers.ts#L44 > > Since [`toIdentifier`](https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/converters/toIdentifier.ts#L4-L26) calls [`isValidIdentifier`](https://github.com/babel/babel/blob/a2025d7d695b0f3b0506f66225d6b15bcfa1cf6a/packages/babel-types/src/validators/isValidIdentifier.ts#L7-L25) within itself, the basic workaround change could be as simple as: > > ```diff > - let safeRenamed = isValidIdentifier(renamed) ? renamed : `_${renamed}`; > + let safeRenamed = toIdentifier(renamed); > ``` > > > Gonna make a PR in a few minutes > > @j4k0xb Thanks for making a PR with the workaround fix! > > * [fix: invalid identifiers #133](https://github.com/jehna/humanify/pull/133) > > Will add some thoughts to it (though realistically they can probably be handled as a separate PR) Thank you for your help, my problem has been solved

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 29, 2024):

I had a question. The corresponding file size in the image is too large (134881 lines). And the processing of the file seems to take a very long time. Is there a faster way to do this? If the terminal screen closes the process will reset and that would be very sad

@0xdevalias Do you have any ideas on this?

@Yusufkulcu commented on GitHub (Sep 29, 2024): > I had a question. The corresponding file size in the image is too large (134881 lines). And the processing of the file seems to take a very long time. Is there a faster way to do this? If the terminal screen closes the process will reset and that would be very sad > > ![image](https://private-user-images.githubusercontent.com/69112573/371833876-71795558-83eb-4dc4-b8ea-f349089f5438.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc1OTUyNDIsIm5iZiI6MTcyNzU5NDk0MiwicGF0aCI6Ii82OTExMjU3My8zNzE4MzM4NzYtNzE3OTU1NTgtODNlYi00ZGM0LWI4ZWEtZjM0OTA4OWY1NDM4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI5VDA3MjkwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNlODI1ZGY0MTRkNjViOGQ0MTU4MTQ1ZDgwZGM5NWE0NDA2NmY2ZDZlYjE1MDQ5MmNiZjc4NzhkNDdkZjA2ZjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.9cdParlcvqzGrnUHPDs1mdIWC0A9kh_4lybQalBGPXo) @0xdevalias Do you have any ideas on this?

kerem commented