[GH-ISSUE #1513] zstd compression support

kerem commented

2026-03-03 19:52:01 +03:00

Owner

Originally created by @aexvir on GitHub (Jan 25, 2023).
Original GitHub issue: https://github.com/ProxymanApp/Proxyman/issues/1513

Originally assigned to: @NghiaTranUIT on GitHub.

Description

we're using zstd compression between some of our services, and currently in proxyman I'm only able to visualize the binary data for the response body; other content encodings like gzip seem to be supported, so I'm guessing this is a feature request and not a bug

Why this feature/change is important?

currently I'm forced to always download the request body to disk and then use the zstd cli to obtain plain text that I can use to debug what's going on, which slows me down

thanks for developing this app! 🙂

Originally created by @aexvir on GitHub (Jan 25, 2023). Original GitHub issue: https://github.com/ProxymanApp/Proxyman/issues/1513 Originally assigned to: @NghiaTranUIT on GitHub. ## Description we're using [zstd](https://github.com/facebook/zstd) compression between some of our services, and currently in proxyman I'm only able to visualize the binary data for the response body; other content encodings like gzip seem to be supported, so I'm guessing this is a feature request and not a bug ## Why this feature/change is important? currently I'm forced to always download the request body to disk and then use the `zstd` cli to obtain plain text that I can use to debug what's going on, which slows me down thanks for developing this app! 🙂

kerem added the

enhancement

✅ Done

labels

2026-03-03 19:52:01 +03:00

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Jan 25, 2023):

Thanks for opening the ticket.

Does the zstd is defined in the Header Content Encoding?

@NghiaTranUIT commented on GitHub (Jan 25, 2023): Thanks for opening the ticket. Does the zstd is defined in the Header Content Encoding?

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@aexvir commented on GitHub (Jan 25, 2023):

yep! Content-Encoding: zstd

@aexvir commented on GitHub (Jan 25, 2023): yep! `Content-Encoding: zstd`

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Jan 26, 2023):

You can do a quick hack, by removing the Accept-Encoding in the Request -> Your server doesn't compress with the zstd -> You can preview it on Proxyman -> Your client can get the uncompressed data since the Content-Encoding is omit.

You can use the Scripting with On Request mode:

function onRequest(context, url, request) {
    delete request.headers["Accept-Encoding"];
    return request;
}

@NghiaTranUIT commented on GitHub (Jan 26, 2023): You can do a quick hack, by removing the Accept-Encoding in the Request -> Your server doesn't compress with the `zstd` -> You can preview it on Proxyman -> Your client can get the uncompressed data since the Content-Encoding is omit. You can use the Scripting with On Request mode: ```js function onRequest(context, url, request) { delete request.headers["Accept-Encoding"]; return request; } ```

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@aexvir commented on GitHub (Feb 1, 2023):

nice! I haven't used the scripting feature on proxyman yet, so I didn't think about this use case
I got it working, thanks for the suggestion, but if at some point zstd support could be added I think that would be nice 🙂

@aexvir commented on GitHub (Feb 1, 2023): nice! I haven't used the scripting feature on proxyman yet, so I didn't think about this use case I got it working, thanks for the suggestion, but if at some point zstd support could be added I think that would be nice 🙂

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@rockmandash commented on GitHub (Jun 22, 2024):

You can do a quick hack, by removing the Accept-Encoding in the Request -> Your server doesn't compress with the zstd -> You can preview it on Proxyman -> Your client can get the uncompressed data since the Content-Encoding is omit.

You can use the Scripting with On Request mode:
function onRequest(context, url, request) {
    delete request.headers["Accept-Encoding"];
    return request;
}

I spent whole day finally found this!
This is pro tip!
It should be directly baked into example starter code!

@rockmandash commented on GitHub (Jun 22, 2024): > You can do a quick hack, by removing the Accept-Encoding in the Request -> Your server doesn't compress with the `zstd` -> You can preview it on Proxyman -> Your client can get the uncompressed data since the Content-Encoding is omit. > > You can use the Scripting with On Request mode: > > ```js > function onRequest(context, url, request) { > delete request.headers["Accept-Encoding"]; > return request; > } > ``` I spent whole day finally found this! This is pro tip! It should be directly baked into example starter code!

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Jun 23, 2024):

Glad it helps you @rockmandash 👍

@NghiaTranUIT commented on GitHub (Jun 23, 2024): Glad it helps you @rockmandash 👍

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@pavelhiq commented on GitHub (Oct 3, 2024):

Is there any update on this one?

While the tip is indeed helpful, it does introduce some (in my opinion) unnecessary difference in behaviour between the 'prod' app and the app under investigation. Might also be less convenient for projects and teams of larger scale.

@pavelhiq commented on GitHub (Oct 3, 2024): Is there any update on this one? While the tip is indeed helpful, it does introduce some (in my opinion) unnecessary difference in behaviour between the 'prod' app and the app under investigation. Might also be less convenient for projects and teams of larger scale.

kerem commented

2026-03-03 19:52:02 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 4, 2024):

Let me try to add it, and send a Beta build to you 👍

@NghiaTranUIT commented on GitHub (Oct 4, 2024): Let me try to add it, and send a Beta build to you 👍

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 4, 2024):

@pavelhiq @rockmandash @aexvir if you don't mind, let's try this Beta build: https://download.proxyman.io/beta/Proxyman_5.9.0_Support_zstd.dmg

Proxyman now tries to decompress your zstd body if this header Content-Encoding: zstd exist 👍
Work with Request and Response Body
Use the latest zstd lib

@NghiaTranUIT commented on GitHub (Oct 4, 2024): @pavelhiq @rockmandash @aexvir if you don't mind, let's try this Beta build: https://download.proxyman.io/beta/Proxyman_5.9.0_Support_zstd.dmg - Proxyman now tries to decompress your zstd body if this header `Content-Encoding: zstd` exist 👍 - Work with Request and Response Body - Use the latest zstd lib

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@pavelhiq commented on GitHub (Oct 4, 2024):

@NghiaTranUIT that was soo quick, thank you!

I've now tried the beta you provided, however I could not get it working.

The response does contain the content-encoding: zstd (lowercased!) header. But I still see Preview Not Support! in the Body tab. Lowercasing should be fine according to RFC 7540, section 8.1.2. Could that be the reason for not showing decoded content?

I tried to verify that guess myself with the help of scripting, similar to what you provided above:

function onResponse(context, url, request, response) {
  delete response.headers["content-encoding"];
  response.headers["Content-Encoding"] = "zstd";
  return response;
}

Had to remove async from the template though. What I can see is:

apparently some decoding takes place
the Body tab now shows an array of bytes
the Content-Encoding header is not showing any longer in the Header tab

Is that expected or am I doing smth wrong?

@pavelhiq commented on GitHub (Oct 4, 2024): @NghiaTranUIT that was soo quick, thank you! I've now tried the beta you provided, however I could not get it working. The response does contain the `content-encoding: zstd` (lowercased!) header. But I still see `Preview Not Support!` in the `Body` tab. Lowercasing should be fine according to [RFC 7540, section 8.1.2](https://www.rfc-editor.org/rfc/rfc7540#section-8.1.2). Could that be the reason for not showing decoded content? I tried to verify that guess myself with the help of scripting, similar to what you provided above: ```javascript function onResponse(context, url, request, response) { delete response.headers["content-encoding"]; response.headers["Content-Encoding"] = "zstd"; return response; } ``` Had to remove `async` from the template though. What I can see is: - apparently some decoding takes place - the `Body` tab now shows an array of bytes - the `Content-Encoding` header is _not_ showing any longer in the `Header` tab Is that expected or am I doing smth wrong?

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 5, 2024):

Hm, it's odd, I tested with my local ExpressJS server, and it works fine with simple-zstd.

If you don't mind @pavelhiq , please share your sample request with nghia@proxyman.io (Right click on the request -> Export -> ProxymanLog). I'd like to investigate it 👍

@NghiaTranUIT commented on GitHub (Oct 5, 2024): Hm, it's odd, I tested with my local ExpressJS server, and it works fine with [simple-zstd](https://www.npmjs.com/package/simple-zstd). If you don't mind @pavelhiq , please share your sample request with nghia@proxyman.io (Right click on the request -> Export -> ProxymanLog). I'd like to investigate it 👍

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 5, 2024):

Second question:

Does your Response body compress by zstd (with Content-Type: application/zstd), or a text-based (but use ztd compression when transferring the data (Content-Encoding)?)

content-encoding: zstd

It's not matter, I handle both case

@NghiaTranUIT commented on GitHub (Oct 5, 2024): Second question: Does your Response body compress by zstd (with `Content-Type: application/zstd`), or a text-based (but use ztd compression when transferring the data (Content-Encoding)?) > content-encoding: zstd It's not matter, I handle both case

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@pavelhiq commented on GitHub (Oct 7, 2024):

sorry, I dont have a suitable request that I could raw-export. but I do my best to provide details as much as possible)

or a text-based (but use ztd compression when transferring the data (Content-Encoding)?)

to me this reflects how it works, yes. So first the body needs to be decompressed with zstd (because of the content-encoding: zstd header), then the Content-Type: application/json steps in, the body to be interpreted as json.

Btw, with no scripting, if I just export the binary response and then decompress with the cli tool - it works. Like this

% zstd --version                                 
*** Zstandard CLI (64-bit) v1.5.6, by Yann Collet ***
% zstd -d api_5_main_shortcuts -o shortcuts.json
api_5_main_shortcuts: 27289 bytes

@pavelhiq commented on GitHub (Oct 7, 2024): sorry, I dont have a suitable request that I could raw-export. but I do my best to provide details as much as possible) > or a text-based (but use ztd compression when transferring the data (Content-Encoding)?) to me this reflects how it works, yes. So first the body needs to be decompressed with zstd (because of the `content-encoding: zstd` header), then the `Content-Type: application/json` steps in, the body to be interpreted as json. Btw, with no scripting, if I just export the binary response and then decompress with the cli tool - it works. Like this ``` % zstd --version *** Zstandard CLI (64-bit) v1.5.6, by Yann Collet *** % zstd -d api_5_main_shortcuts -o shortcuts.json api_5_main_shortcuts: 27289 bytes ```

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 8, 2024):

@pavelhiq I'm able to reproduce this issue. Tested with facebook.com, some requests are decompressed successfully (zst), and some won't. Not sure why it's.

I'm fixing and will send you new update 👍

@NghiaTranUIT commented on GitHub (Oct 8, 2024): @pavelhiq I'm able to reproduce this issue. Tested with facebook.com, some requests are decompressed successfully (zst), and some won't. Not sure why it's. I'm fixing and will send you new update 👍

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 8, 2024):

@pavelhiq it's fixed 👍

Beta build: https://download.proxyman.io/beta/Proxyman_5.9.0_Support_zstd_v2.dmg
Turn out some zstd packages don't include the Content Size header, so it needs to decode with a Streaming Mode
Tested with www.facebook.com, it uses zstd a lot 👍

@NghiaTranUIT commented on GitHub (Oct 8, 2024): @pavelhiq it's fixed 👍 - Beta build: https://download.proxyman.io/beta/Proxyman_5.9.0_Support_zstd_v2.dmg - Turn out some zstd packages don't include the Content Size header, so it needs to decode with a Streaming Mode - Tested with www.facebook.com, it uses zstd a lot 👍

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@pavelhiq commented on GitHub (Oct 8, 2024):

Yeah, streaming mode was something I could have mentioned earlier.. myself I had an issue with it too.

Now checked your latest beta update and it worked. Brilliant! 🎉

Should we expect this enhancement in 5.10, correct? Any estimate perhaps?

@pavelhiq commented on GitHub (Oct 8, 2024): Yeah, streaming mode was something I could have mentioned earlier.. myself I had an issue with it too. Now checked your latest beta update and it worked. Brilliant! :tada: Should we expect this enhancement in 5.10, correct? Any estimate perhaps?

kerem commented

2026-03-03 19:52:03 +03:00

Author

Owner

@NghiaTranUIT commented on GitHub (Oct 8, 2024):

yes, will release in 5.10.0, maybe next week 👍

@NghiaTranUIT commented on GitHub (Oct 8, 2024): yes, will release in 5.10.0, maybe next week 👍

Rows
Columns

[GH-ISSUE #1513] zstd compression support #1505

Description

Why this feature/change is important?