[GH-ISSUE #111 ] Error: Failed to stringify code #40

Owner

Originally created by @Yusufkulcu on GitHub (Sep 21, 2024).
Original GitHub issue: https://github.com/jehna/humanify/issues/111

I trade using chatgpt. I am getting the error shown in the screenshot. The file size is large (7mb on average) and the number of lines is high (134881 lines). What is the problem? Can you help me? This is very important to me

Originally created by @Yusufkulcu on GitHub (Sep 21, 2024). Original GitHub issue: https://github.com/jehna/humanify/issues/111 I trade using chatgpt. I am getting the error shown in the screenshot. The file size is large (7mb on average) and the number of lines is high (134881 lines). What is the problem? Can you help me? This is very important to me ![image](https://github.com/user-attachments/assets/4684e355-ff6e-4fa4-a524-637eeb08f74c)

kerem closed this issue

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

@jehna please help me

@Yusufkulcu commented on GitHub (Sep 22, 2024): @jehna please help me

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 22, 2024):

Can you run humanify using --verbose flag and post the output? It gives a lot more debug data to work with

@jehna commented on GitHub (Sep 22, 2024): Can you run humanify using `--verbose` flag and post the output? It gives a lot more debug data to work with

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

Can you run humanify using --verbose flag and post the output? It gives a lot more debug data to work with

I'm trying right now

@Yusufkulcu commented on GitHub (Sep 22, 2024): > Can you run humanify using `--verbose` flag and post the output? It gives a lot more debug data to work with I'm trying right now

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

@jehna

@Yusufkulcu commented on GitHub (Sep 22, 2024): @jehna ![image](https://github.com/user-attachments/assets/dbf7bc3b-61cd-464c-95a0-01d4593ec195)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

@jehna

Do you have any suggestions on what I can do to fix the problem?

@Yusufkulcu commented on GitHub (Sep 22, 2024): @jehna Do you have any suggestions on what I can do to fix the problem?

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 22, 2024):

Hmm. Seems like an issue with some of the dependencies, at least that error does not originate from humanify's code.
In hindsight it seems that using pkgroll was a bad idea, as it seems to mess with the dependencies' stack traces.
I think this would need some extra debugging to solve.

@jehna commented on GitHub (Sep 22, 2024): Hmm. Seems like an issue with some of the dependencies, at least that error does not originate from humanify's code. In hindsight it seems that using pkgroll was a bad idea, as it seems to mess with the dependencies' stack traces. I think this would need some extra debugging to solve.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

Hmm. Bazı bağımlılıklarla ilgili bir sorun gibi görünüyor, en azından bu hata humanify'ın kodundan kaynaklanmıyor. Geriye dönüp bakıldığında pkgroll'u kullanmanın kötü bir fikir olduğu anlaşılıyor, çünkü bağımlılıkların yığın izlerini bozuyor gibi görünüyor. Bunun çözülmesi için biraz daha hata ayıklamaya ihtiyaç olduğunu düşünüyorum.

@jehna

So what can I do as a solution? Solving this code is very important to me

@Yusufkulcu commented on GitHub (Sep 22, 2024): > Hmm. Bazı bağımlılıklarla ilgili bir sorun gibi görünüyor, en azından bu hata humanify'ın kodundan kaynaklanmıyor. Geriye dönüp bakıldığında pkgroll'u kullanmanın kötü bir fikir olduğu anlaşılıyor, çünkü bağımlılıkların yığın izlerini bozuyor gibi görünüyor. Bunun çözülmesi için biraz daha hata ayıklamaya ihtiyaç olduğunu düşünüyorum. @jehna So what can I do as a solution? Solving this code is very important to me

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

How can I send you error details

@Yusufkulcu commented on GitHub (Sep 22, 2024): How can I send you error details

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 22, 2024):

Whoops, closed this by accident. You could check which part of the bundle throws the error, that would help

@jehna commented on GitHub (Sep 22, 2024): Whoops, closed this by accident. You could check which part of the bundle throws the error, that would help

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

Whoops, closed this by accident. You could check which part of the bundle throws the error, that would help

@jehna

When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem. I would be grateful if you could help

@Yusufkulcu commented on GitHub (Sep 22, 2024): > Whoops, closed this by accident. You could check which part of the bundle throws the error, that would help @jehna When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem. I would be grateful if you could help ![image](https://github.com/user-attachments/assets/64d59b6f-de1b-41b5-bad6-4756f4464a72)

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

@jehna

Please help me

@Yusufkulcu commented on GitHub (Sep 22, 2024): @jehna Please help me

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

@jehna

Will you help?

@Yusufkulcu commented on GitHub (Sep 22, 2024): @jehna Will you help?

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 22, 2024):

@Yusufkulcu thanks for the additional info, but unfortunately I'm not able to respond to bigger work immediately (not being paid for doing ppen source etc)

@jehna commented on GitHub (Sep 22, 2024): @Yusufkulcu thanks for the additional info, but unfortunately I'm not able to respond to bigger work immediately (not being paid for doing ppen source etc)

kerem commented

Author

Owner

@jehna commented on GitHub (Sep 22, 2024):

I think I'll be able to investigate more within a few weeks when I have some time

@jehna commented on GitHub (Sep 22, 2024): I think I'll be able to investigate more within a few weeks when I have some time

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 22, 2024):

I understood. I would be very happy if I know when the study is concluded. We hope that it will be concluded as soon as possibleI understood. I would be very happy if I know when the study is concluded. Hoping to conclude it as soon as possible 😊

@Yusufkulcu commented on GitHub (Sep 22, 2024): I understood. I would be very happy if I know when the study is concluded. We hope that it will be concluded as soon as possibleI understood. I would be very happy if I know when the study is concluded. Hoping to conclude it as soon as possible 😊

kerem commented

github.com/jehna/humanify@2ac5d25d1d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L57-L62)

Author

Owner

@0xdevalias commented on GitHub (Sep 25, 2024):

When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem.

@Yusufkulcu I haven't looked too deeply into this, but I suspect the issue is likely not going to be directly within transformFromAstAsync within @babel/core; but more with how it is being used by either humanify's AST parsing code, or how the webcrack project it depends on is using it.

Looking up that "Failed to stringify code" error leads to this source within humanify:

It looks like it throws that when the result from transformFromAstAsync has a falsy value for stringified?.code

Depending on your skill at debugging, it might be useful to see what the value of ast being passed in is, as well as the full value of stringified that is being returned.

Depending on the sensitivity of the code you're running this on, it could also be useful if you're able to attach the code that consistently triggers this area; even better if you're first able to reduce it to a minimal example that triggers the error; as that would make it a lot easier for others to help debug/understand what might be causing it.

@0xdevalias commented on GitHub (Sep 25, 2024): > When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem. @Yusufkulcu I haven't looked too deeply into this, but I suspect the issue is likely not going to be directly within `transformFromAstAsync` within `@babel/core`; but more with how it is being used by either `humanify`'s AST parsing code, or how the `webcrack` project it depends on is using it. Looking up that `"Failed to stringify code"` error leads to this source within `humanify`: https://github.com/jehna/humanify/blob/2ac5d25d1d6a314043e57d06bf4d57e61e03f649/src/plugins/local-llm-rename/visit-all-identifiers.ts#L57-L62 It looks like it throws that when the result from `transformFromAstAsync` has a falsy value for `stringified?.code` Depending on your skill at debugging, it might be useful to see what the value of `ast` being passed in is, as well as the full value of `stringified` that is being returned. Depending on the sensitivity of the code you're running this on, it could also be useful if you're able to attach the code that consistently triggers this area; even better if you're first able to reduce it to a minimal example that triggers the error; as that would make it a lot easier for others to help debug/understand what might be causing it.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 25, 2024):

When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem.

@Yusufkulcu I haven't looked too deeply into this, but I suspect the issue is likely not going to be directly within transformFromAstAsync within @babel/core; but more with how it is being used by either humanify's AST parsing code, or how the webcrack project it depends on is using it.

Looking up that "Failed to stringify code" error leads to this source within humanify:

github.com/jehna/humanify@2ac5d25d1d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L57-L62)

It looks like it throws that when the result from transformFromAstAsync has a falsy value for stringified?.code

Depending on your skill at debugging, it might be useful to see what the value of ast being passed in is, as well as the full value of stringified that is being returned.

Depending on the sensitivity of the code you're running this on, it could also be useful if you're able to attach the code that consistently triggers this area; even better if you're first able to reduce it to a minimal example that triggers the error; as that would make it a lot easier for others to help debug/understand what might be causing it.

I didn't understand exactly what I was supposed to do:(

@Yusufkulcu commented on GitHub (Sep 25, 2024): > > When I check, the "transformFromAstAsync" function in the "@babel/core" package gives an error. Since I'm not familiar with the package, I have no idea how to fix the problem. > > @Yusufkulcu I haven't looked too deeply into this, but I suspect the issue is likely not going to be directly within `transformFromAstAsync` within `@babel/core`; but more with how it is being used by either `humanify`'s AST parsing code, or how the `webcrack` project it depends on is using it. > > Looking up that `"Failed to stringify code"` error leads to this source within `humanify`: > > https://github.com/jehna/humanify/blob/2ac5d25d1d6a314043e57d06bf4d57e61e03f649/src/plugins/local-llm-rename/visit-all-identifiers.ts#L57-L62 > > It looks like it throws that when the result from `transformFromAstAsync` has a falsy value for `stringified?.code` > > Depending on your skill at debugging, it might be useful to see what the value of `ast` being passed in is, as well as the full value of `stringified` that is being returned. > > Depending on the sensitivity of the code you're running this on, it could also be useful if you're able to attach the code that consistently triggers this area; even better if you're first able to reduce it to a minimal example that triggers the error; as that would make it a lot easier for others to help debug/understand what might be causing it. I didn't understand exactly what I was supposed to do:(

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 25, 2024):

I didn't understand exactly what I was supposed to do:(

@Yusufkulcu At a minimum, if you are able to attach the code you're trying to decrypt, that would be really useful to help narrow things down.

@0xdevalias commented on GitHub (Sep 25, 2024): > I didn't understand exactly what I was supposed to do:( @Yusufkulcu At a minimum, if you are able to attach the code you're trying to decrypt, that would be really useful to help narrow things down.

kerem commented

https://drive.google.com/file/d/1H9_g6L32xMchrpi-dIRksz7MyQkmWm2a/view?usp=drivesdk

Author

Owner

@Yusufkulcu commented on GitHub (Sep 25, 2024):

Tam olarak ne yapmam gerektiğini anlamadım :(

@YusufkulcuEn azından, şifresini çözmeye çalıştığınız kodu ekleyebilirseniz, bu, işleri daraltmanıza yardımcı olmak için gerçekten yararlı olacaktır.

Yes, you are right, I have attached the file below. Github I uploaded it to my Google drive account because it doesn't allow javascript file uploading.

@jehna
@0xdevalias

@Yusufkulcu commented on GitHub (Sep 25, 2024): > > Tam olarak ne yapmam gerektiğini anlamadım :( > > @YusufkulcuEn azından, şifresini çözmeye çalıştığınız kodu ekleyebilirseniz, bu, işleri daraltmanıza yardımcı olmak için gerçekten yararlı olacaktır. Yes, you are right, I have attached the file below. Github I uploaded it to my Google drive account because it doesn't allow javascript file uploading. https://drive.google.com/file/d/1H9_g6L32xMchrpi-dIRksz7MyQkmWm2a/view?usp=drivesdk @jehna @0xdevalias

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 25, 2024):

I uploaded it to my Google drive account because it doesn't allow javascript file uploading.

@Yusufkulcu Thanks. If you just rename the .js file to .txt that would usually let you upload it directly to GitHub as well.

@0xdevalias commented on GitHub (Sep 25, 2024): > I uploaded it to my Google drive account because it doesn't allow javascript file uploading. @Yusufkulcu Thanks. If you just rename the .js file to .txt that would usually let you upload it directly to GitHub as well.

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 25, 2024):

Javascript dosya yüklemeye izin vermediği için Google Drive hesabıma yükledim.

@YusufkulcuTeşekkürler. .js dosyasının adını .txt olarak değiştirirseniz genellikle onu doğrudan GitHub'a da yükleyebilirsiniz.

I uploaded it to my Google drive account because it doesn't allow javascript file uploading.

@Yusufkulcu Thanks. If you just rename the .js file to .txt that would usually let you upload it directly to GitHub as well.

Thanks for the info. I reloaded it as txt. I will be waiting for your help. It's an important project for me
main.txt

@Yusufkulcu commented on GitHub (Sep 25, 2024): > > Javascript dosya yüklemeye izin vermediği için Google Drive hesabıma yükledim. > > @YusufkulcuTeşekkürler. .js dosyasının adını .txt olarak değiştirirseniz genellikle onu doğrudan GitHub'a da yükleyebilirsiniz. > > I uploaded it to my Google drive account because it doesn't allow javascript file uploading. > > @Yusufkulcu Thanks. If you just rename the .js file to .txt that would usually let you upload it directly to GitHub as well. > > I uploaded it to my Google drive account because it doesn't allow javascript file uploading. > > @Yusufkulcu Thanks. If you just rename the .js file to .txt that would usually let you upload it directly to GitHub as well. Thanks for the info. I reloaded it as txt. I will be waiting for your help. It's an important project for me [main.txt](https://github.com/user-attachments/files/17126700/main.txt)

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

Thanks for the info. I reloaded it as txt.

@Yusufkulcu Thanks for that. I ran it through the web version of and it seemed to unpack correctly there, which makes it a higher likelihood that the issue is withinhumanify`:

https://webcrack.netlify.app/

Depending on your use case, I wonder if you might be able to use that unpacked/unmangled version as a starting point for your analysis; even though it doesn't have the 'nice variable renames' that humanify adds on top? You might also be able to run humanify against individual files from that extracted version in the interim as well.

@0xdevalias commented on GitHub (Sep 26, 2024): > Thanks for the info. I reloaded it as txt. @Yusufkulcu Thanks for that. I ran it through the web version of ` and it seemed to unpack correctly there, which makes it a higher likelihood that the issue is within `humanify`: - https://webcrack.netlify.app/ Depending on your use case, I wonder if you might be able to use that unpacked/unmangled version as a starting point for your analysis; even though it doesn't have the 'nice variable renames' that `humanify` adds on top? You might also be able to run `humanify` against individual files from that extracted version in the interim as well.

kerem commented

github.com/jehna/humanify@c2e0be679d/src/unminify.ts (L6-L30)

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

@Yusufkulcu Looking at your screenshot from https://github.com/jehna/humanify/issues/111#issuecomment-2365764329, it looks like the error is occuring on file 18:

Looking at the code where this happens, it seems to be within unminify, just after webcrack has run to extract the files, and in the main loop where those files are being processed:

Since even with --verbose enabled, humanify doesn't output the filenames it's processing, it makes it a bit harder to debug; but running it through a debugger we can see what 'file 18' of extractedFiles corresponds with anyway (assuming webcrack consistently unpacks the code the same way); which since the output log shows i + 1, that makes it actually entry 17 in the array (because it's 0-indexed); which seems to be the extracted file 1.js:

Looking at the contents of output/1.js, we can see that it's an empty file:

⇒ cat output/1.js | wc -m
       0

As part of the main unminify loop, the code then calls plugins.reduce on the plugins passed in to unminify:

github.com/jehna/humanify@c2e0be679d/src/commands/openai.ts (L26-L30)

When run as humanify openai, these plugins are:

babel (Ref)
openaiRename({ apiKey, model: opts.model }) (Ref)
prettier (Ref)

The Failed to stringify code error you're seeing comes from the visitAllIdentifiers function, just after calling babel's transformFromAstAsync:

github.com/jehna/humanify@c2e0be679d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L57-L61)

https://babeljs.io/docs/babel-core#transformfromastasync

Which is called by the openaiRename plugin that is passed to unminify:

github.com/jehna/humanify@c2e0be679d/src/plugins/openai/openai-rename.ts (L6-L38)

transformFromAstAsync accepts a parsed ast, and returns an object with code, map, and ast.

Presumably, with an empty file as input, that would result in an empty ast, which when passed to transformFromAstAsync, would result in an empty string returned in the code field.

Given the check that ends up throwing the error you're seeing is looking for a falsy value in stringified.code; the empty string would definitely trigger that:

github.com/jehna/humanify@c2e0be679d/src/plugins/local-llm-rename/visit-all-identifiers.ts (L57-L62)

@Yusufkulcu @jehna So the above seems to be the debugging as to the 'why' of what is going on; though I'm not familiar enough with the codebase to suggest where the best point of implementing the fix is.

If I was to suggest the most minimal/localised fix, it would be to the if (!stringified?.code) check within visitAllIdentifiers (Ref)

Though an argument could also be made that filtering out empty/similar files at the unminify level could have it's merits (Ref); though that would then make it less generic/flexible, as theoretically a plugin in the chain could make it a non-empty file before the openaiRename actually needs to process it.

So I still think the best fix is probably within visitAllIdentifiers.

In fact, in looking through related issues, this just seems to be a variation of the same root cause that I raised in this issue:

https://github.com/jehna/humanify/issues/54

I also think the general CLI output should be improved to explain what is happening better. eg. a message before running webcrack, outputting the actual filenames (not just the file number), etc.

These issues also seem to tangentially relate to some aspect of this latter suggestion:

@0xdevalias commented on GitHub (Sep 26, 2024): @Yusufkulcu Looking at your screenshot from https://github.com/jehna/humanify/issues/111#issuecomment-2365764329, it looks like the error is occuring on file 18: ![image](https://github.com/user-attachments/assets/48fcd461-2e3c-466d-a4ce-9230c8d8bf3a) Looking at the code where this happens, it seems to be within `unminify`, just after `webcrack` has run to extract the files, and in the main loop where those files are being processed: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/unminify.ts#L6-L30 Since even with `--verbose` enabled, `humanify` doesn't output the filenames it's processing, it makes it a bit harder to debug; but running it through a debugger we can see what 'file 18' of `extractedFiles` corresponds with anyway (assuming `webcrack` consistently unpacks the code the same way); which since the output log shows `i + 1`, that makes it actually entry 17 in the array (because it's 0-indexed); which seems to be the extracted file `1.js`: ![image](https://github.com/user-attachments/assets/2d0d066f-e113-405f-bbe4-c5bfa30a6d75) Looking at the contents of `output/1.js`, we can see that it's an empty file: ```shell ⇒ cat output/1.js | wc -m 0 ``` As part of the main `unminify` loop, the code then calls `plugins.reduce` on the `plugins` passed in to `unminify`: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/commands/openai.ts#L26-L30 When run as `humanify openai`, these plugins are: - `babel` ([Ref](https://github.com/jehna/humanify/blob/main/src/plugins/babel/babel.ts)) - `openaiRename({ apiKey, model: opts.model })` ([Ref](https://github.com/jehna/humanify/blob/main/src/plugins/openai/openai-rename.ts)) - `prettier` ([Ref](https://github.com/jehna/humanify/blob/main/src/plugins/prettier.ts)) The `Failed to stringify code` error you're seeing comes from the `visitAllIdentifiers` function, just after calling `babel`'s `transformFromAstAsync`: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/plugins/local-llm-rename/visit-all-identifiers.ts#L57-L61 - https://babeljs.io/docs/babel-core#transformfromastasync Which is called by the `openaiRename` plugin that is passed to `unminify`: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/plugins/openai/openai-rename.ts#L6-L38 `transformFromAstAsync` accepts a parsed `ast`, and returns an object with `code`, `map`, and `ast`. Presumably, with an empty file as input, that would result in an empty ast, which when passed to `transformFromAstAsync`, would result in an empty string returned in the `code` field. Given the check that ends up throwing the error you're seeing is looking for a falsy value in `stringified.code`; the empty string would definitely trigger that: https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/plugins/local-llm-rename/visit-all-identifiers.ts#L57-L62 --- @Yusufkulcu @jehna So the above seems to be the debugging as to the 'why' of what is going on; though I'm not familiar enough with the codebase to suggest where the best point of implementing the fix is. If I was to suggest the most minimal/localised fix, it would be to the `if (!stringified?.code)` check within `visitAllIdentifiers` ([Ref](https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/plugins/local-llm-rename/visit-all-identifiers.ts#L57-L62)) Though an argument could also be made that filtering out empty/similar files at the `unminify` level could have it's merits ([Ref](https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/unminify.ts#L6-L30)); though that would then make it less generic/flexible, as theoretically a plugin in the chain could make it a non-empty file before the `openaiRename` actually needs to process it. So I still think the best fix is probably within `visitAllIdentifiers`. In fact, in looking through related issues, this just seems to be a variation of the same root cause that I raised in this issue: - https://github.com/jehna/humanify/issues/54 I also think the general CLI output should be improved to explain what is happening better. eg. a message before running `webcrack`, outputting the actual filenames (not just the file number), etc. These issues also seem to tangentially relate to some aspect of this latter suggestion: - https://github.com/jehna/humanify/issues/92 - https://github.com/jehna/humanify/issues/58

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 26, 2024):

Thanks for the info. I reloaded it as txt.

@Yusufkulcu Thanks for that. I ran it through the web version of and it seemed to unpack correctly there, which makes it a higher likelihood that the issue is withinhumanify`:

https://webcrack.netlify.app/

Depending on your use case, I wonder if you might be able to use that unpacked/unmangled version as a starting point for your analysis; even though it doesn't have the 'nice variable renames' that humanify adds on top? You might also be able to run humanify against individual files from that extracted version in the interim as well.

Thanks for the suggested solution. It works largely without "beautifying variable" values. But my goal is to change it completely. It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this?

@Yusufkulcu commented on GitHub (Sep 26, 2024): > > Thanks for the info. I reloaded it as txt. > > @Yusufkulcu Thanks for that. I ran it through the web version of `and it seemed to unpack correctly there, which makes it a higher likelihood that the issue is within`humanify`: > > * https://webcrack.netlify.app/ > > Depending on your use case, I wonder if you might be able to use that unpacked/unmangled version as a starting point for your analysis; even though it doesn't have the 'nice variable renames' that `humanify` adds on top? You might also be able to run `humanify` against individual files from that extracted version in the interim as well. Thanks for the suggested solution. It works largely without "beautifying variable" values. But my goal is to change it completely. It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this?

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this?

@Yusufkulcu Do you mean that the process takes a long time in general? Or that having to manually run it against each file takes a long time?

If the latter, you could probably extract them with webcrack, then write a bash script/similar that runs humanify on each of those files individually. I'm not 100% sure how the renames works across module boundaries/for exports/similar.. but I think it already reads/parses each of the split files individually anyway, so I don't think it would be much worse to run it this way..

Actually.. it will probably still be a bit slower because of the startup boilerplate, and then trying to run webcrack against the file each time (even though they are already split)..

While it's not currently a feature, if you're going to need to be re-running the variable renaming process on future versions of the same code, this feature request would probably also be relevant to you:

https://github.com/jehna/humanify/issues/97

@jehna It might be a neat/useful feature to be able to run humanify against a source file (or multiple files/directory of files/etc) while bypassing the webcrack step; that way it could be used on files that had already been unpacked/etc with webcrack (or another tool like wakaru) previously.

Edit: I split this last part into its own issue:

https://github.com/jehna/humanify/issues/129

@0xdevalias commented on GitHub (Sep 26, 2024): > It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this? @Yusufkulcu Do you mean that the process takes a long time in general? Or that having to manually run it against each file takes a long time? If the latter, you could probably extract them with `webcrack`, then write a bash script/similar that runs `humanify` on each of those files individually. I'm not 100% sure how the renames works across module boundaries/for exports/similar.. but I think it already reads/parses each of the split files individually anyway, so I don't think it would be much worse to run it this way.. Actually.. it will probably still be a bit slower because of the startup boilerplate, and then trying to run `webcrack` against the file each time (even though they are already split).. While it's not currently a feature, if you're going to need to be re-running the variable renaming process on future versions of the same code, this feature request would probably also be relevant to you: - https://github.com/jehna/humanify/issues/97 --- @jehna It might be a neat/useful feature to be able to run `humanify` against a source file (or multiple files/directory of files/etc) while bypassing the `webcrack` step; that way it could be used on files that had already been unpacked/etc with `webcrack` (or another tool like `wakaru`) previously. **Edit:** I split this last part into its own issue: - https://github.com/jehna/humanify/issues/129

kerem commented

Author

Owner

@Yusufkulcu commented on GitHub (Sep 26, 2024):

It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this?

@Yusufkulcu Do you mean that the process takes a long time in general? Or that having to manually run it against each file takes a long time?

If the latter, you could probably extract them with webcrack, then write a bash script/similar that runs humanify on each of those files individually. I'm not 100% sure how the renames works across module boundaries/for exports/similar.. but I think it already reads/parses each of the split files individually anyway, so I don't think it would be much worse to run it this way..

Actually.. it will probably still be a bit slower because of the startup boilerplate, and then trying to run webcrack against the file each time (even though they are already split)..

While it's not currently a feature, if you're going to need to be re-running the variable renaming process on future versions of the same code, this feature request would probably also be relevant to you:

More deterministic renames across different versions of the same code #97

@jehna It might be a neat/useful feature to be able to run humanify against a source file (or multiple files/directory of files/etc) while bypassing the webcrack step; that way it could be used on files that had already been unpacked/etc with webcrack (or another tool like wakaru) previously.

Edit: Split this last part into its own issue:

Allow running humanify against one or more source files that have already been unpacked (eg. skip re-running through webcrack) #129

Thank you for your detailed explanation. I'm going to try the parsed source code separately by writing a script that implements the humanify command. I have one more question.

What is the reason why the "Webcrack" package breaks the code into pieces. I divide it into many files in the Outputs folder and I don't know what they are. I wonder if you can explain that. Thank you in advance

@Yusufkulcu commented on GitHub (Sep 26, 2024): > > It seems to take a long time to decode the files in the "outputs" folder one by one with Humanify. Is there a shortcut to this? > > @Yusufkulcu Do you mean that the process takes a long time in general? Or that having to manually run it against each file takes a long time? > > If the latter, you could probably extract them with `webcrack`, then write a bash script/similar that runs `humanify` on each of those files individually. I'm not 100% sure how the renames works across module boundaries/for exports/similar.. but I think it already reads/parses each of the split files individually anyway, so I don't think it would be much worse to run it this way.. > > Actually.. it will probably still be a bit slower because of the startup boilerplate, and then trying to run `webcrack` against the file each time (even though they are already split).. > > While it's not currently a feature, if you're going to need to be re-running the variable renaming process on future versions of the same code, this feature request would probably also be relevant to you: > > * [More deterministic renames across different versions of the same code #97](https://github.com/jehna/humanify/issues/97) > > @jehna It might be a neat/useful feature to be able to run `humanify` against a source file (or multiple files/directory of files/etc) while bypassing the `webcrack` step; that way it could be used on files that had already been unpacked/etc with `webcrack` (or another tool like `wakaru`) previously. > > **Edit:** Split this last part into its own issue: > > * [Allow running humanify against one or more source files that have already been unpacked (eg. skip re-running through webcrack) #129](https://github.com/jehna/humanify/issues/129) Thank you for your detailed explanation. I'm going to try the parsed source code separately by writing a script that implements the humanify command. I have one more question. What is the reason why the "Webcrack" package breaks the code into pieces. I divide it into many files in the Outputs folder and I don't know what they are. I wonder if you can explain that. Thank you in advance

kerem commented

Author

Owner

@0xdevalias commented on GitHub (Sep 26, 2024):

Thank you for your detailed explanation. I'm going to try the parsed source code separately by writing a script that implements the humanify command.

@Yusufkulcu It might actually be quicker for you to just clone the humanify repo and temporarily patch the code to skip trying to run against empty files.

I haven't tried it, but a quick/dirty hack fix could just be to add an if check to skip empty files within unminify:

Something like:

  export async function unminify(
    filename: string,
    outputDir: string,
    plugins: ((code: string) => Promise<string>)[] = []
  ) {
    ensureFileExists(filename);
    const bundledCode = await fs.readFile(filename, "utf-8");
    const extractedFiles = await webcrack(bundledCode, outputDir);
  
    for (let i = 0; i < extractedFiles.length; i++) {
      console.log(`Processing file ${i + 1}/${extractedFiles.length}`);
  
      const file = extractedFiles[i];
      const code = await fs.readFile(file.path, "utf-8");
+     if (code) {
      const formattedCode = await plugins.reduce(
        (p, next) => p.then(next),
        Promise.resolve(code)
      );
  
      verbose.log("Input: ", code);
      verbose.log("Output: ", formattedCode);
  
      await fs.writeFile(file.path, formattedCode);
+     }
+     else {
+       verbose.log(`Skipping empty file ${i + 1} (${file.path})`);
+     }
    }
  }

While this wouldn't really be suited for a proper fix, it should hopefully be enough to do what you need to work around this bug in the meantime.

What is the reason why the "Webcrack" package breaks the code into pieces. I divide it into many files in the Outputs folder and I don't know what they are. I wonder if you can explain that.

@Yusufkulcu The original source file you provided is a webpack bundle of code (or similar), where the original many code files have been combined into a single file. When webcrack runs on it, it 'undoes' this bundling, and so splits the single file back into many files similar to how it was originally.

@0xdevalias commented on GitHub (Sep 26, 2024): > Thank you for your detailed explanation. I'm going to try the parsed source code separately by writing a script that implements the humanify command. @Yusufkulcu It might actually be quicker for you to just clone the `humanify` repo and temporarily patch the code to skip trying to run against empty files. I haven't tried it, but a quick/dirty hack fix could just be to add an `if` check to skip empty files within [`unminify`](https://github.com/jehna/humanify/blob/c2e0be679dcab43ec684668202d709f41452c0e9/src/unminify.ts#L19-L28): Something like: ```diff export async function unminify( filename: string, outputDir: string, plugins: ((code: string) => Promise<string>)[] = [] ) { ensureFileExists(filename); const bundledCode = await fs.readFile(filename, "utf-8"); const extractedFiles = await webcrack(bundledCode, outputDir); for (let i = 0; i < extractedFiles.length; i++) { console.log(`Processing file ${i + 1}/${extractedFiles.length}`); const file = extractedFiles[i]; const code = await fs.readFile(file.path, "utf-8"); + if (code) { const formattedCode = await plugins.reduce( (p, next) => p.then(next), Promise.resolve(code) ); verbose.log("Input: ", code); verbose.log("Output: ", formattedCode); await fs.writeFile(file.path, formattedCode); + } + else { + verbose.log(`Skipping empty file ${i + 1} (${file.path})`); + } } } ``` While this wouldn't really be suited for a proper fix, it should hopefully be enough to do what you need to work around this bug in the meantime. --- > What is the reason why the "Webcrack" package breaks the code into pieces. I divide it into many files in the Outputs folder and I don't know what they are. I wonder if you can explain that. @Yusufkulcu The original source file you provided is a [webpack](https://webpack.js.org/) bundle of code (or similar), where the original many code files have been combined into a single file. When `webcrack` runs on it, it 'undoes' this bundling, and so splits the single file back into many files similar to how it was originally.

kerem commented