[GH-ISSUE #1594] Bulk save many responses to files #1587

Open
opened 2026-03-03 19:52:40 +03:00 by kerem · 26 comments
Owner

Originally created by @drzazga3009 on GitHub (Apr 9, 2023).
Original GitHub issue: https://github.com/ProxymanApp/Proxyman/issues/1594

I am a new user and like Proxyman so far. My main use is to capture live streams directly from my iPhone (hundreds of ts segments) without a computer.
When I have access to my PC I just use Fiddler to capture all traffic from my phone, then filter out all requests that are not ts files, select all requests that are left, and save response bodies to files. Fiddler is not perfect for this since it asks for each file name, but at least I don't need to click on every request separately to export.
When I don't have access to my PC I want to use MITM Proxyman installed on the phone. It captures all traffic, then I export the whole file to my PC and open it with the desktop version of Proxyman. Then, I can filter only ts files but I don't know of any way to bulk save responses to files. I need to separately click on every request and there are hundreds of them.

I'm looking for a way to just save all bodies, the name of the file is not relevant, it could be just the ID of the request captured. Not sure if this is possible, maybe this should be a feature request.

Originally created by @drzazga3009 on GitHub (Apr 9, 2023). Original GitHub issue: https://github.com/ProxymanApp/Proxyman/issues/1594 I am a new user and like Proxyman so far. My main use is to capture live streams directly from my iPhone (hundreds of ts segments) without a computer. When I have access to my PC I just use Fiddler to capture all traffic from my phone, then filter out all requests that are not ts files, select all requests that are left, and save response bodies to files. Fiddler is not perfect for this since it asks for each file name, but at least I don't need to click on every request separately to export. When I don't have access to my PC I want to use MITM Proxyman installed on the phone. It captures all traffic, then I export the whole file to my PC and open it with the desktop version of Proxyman. Then, I can filter only ts files but I don't know of any way to bulk save responses to files. I need to separately click on every request and there are hundreds of them. I'm looking for a way to just save all bodies, the name of the file is not relevant, it could be just the ID of the request captured. Not sure if this is possible, maybe this should be a feature request.
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 9, 2023):

@drzazga3009 if you're using Proxyman for macOS, you can export multiple requests at the same time.

Screenshot 2023-04-09 at 21 46 58

<!-- gh-comment-id:1501147907 --> @NghiaTranUIT commented on GitHub (Apr 9, 2023): @drzazga3009 if you're using Proxyman for macOS, you can export multiple requests at the same time. ![Screenshot 2023-04-09 at 21 46 58](https://user-images.githubusercontent.com/5878421/230780006-7fab24f3-33eb-4a1e-99c6-a2a6ba7434df.jpg)
Author
Owner

@drzazga3009 commented on GitHub (Apr 9, 2023):

Thanks. I forgot to include this, I use Proxyman 1.5.0 for Windows. Previously, I tried exactly this approach as shown and it exports only the first response.

Just looking at your screenshot I can see the windows version is very limited compared to the macos version. So I guess it is possible to do what I want on macos, but not on windows?

<!-- gh-comment-id:1501211344 --> @drzazga3009 commented on GitHub (Apr 9, 2023): Thanks. I forgot to include this, I use Proxyman 1.5.0 for Windows. Previously, I tried exactly this approach as shown and it exports only the first response. Just looking at your screenshot I can see the windows version is very limited compared to the macos version. So I guess it is possible to do what I want on macos, but not on windows?
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 10, 2023):

I'm going to create a ticket and support this feature on Windows 👍

<!-- gh-comment-id:1501264494 --> @NghiaTranUIT commented on GitHub (Apr 10, 2023): I'm going to create a ticket and support this feature on Windows 👍
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 10, 2023):

Here is the ticket: https://github.com/ProxymanApp/proxyman-windows-linux/issues/216

<!-- gh-comment-id:1501274319 --> @NghiaTranUIT commented on GitHub (Apr 10, 2023): Here is the ticket: https://github.com/ProxymanApp/proxyman-windows-linux/issues/216
Author
Owner

@drzazga3009 commented on GitHub (Apr 10, 2023):

Thank you very much! Should this ticket be closed after you created a new one?

<!-- gh-comment-id:1501500531 --> @drzazga3009 commented on GitHub (Apr 10, 2023): Thank you very much! Should this ticket be closed after you created a new one?
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 11, 2023):

Just keep it opens 👍

<!-- gh-comment-id:1502853893 --> @NghiaTranUIT commented on GitHub (Apr 11, 2023): Just keep it opens 👍
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 13, 2023):

@drzazga3009 Bulk export feature is released on the latest Windows update v1.6.0 👍

<!-- gh-comment-id:1506354096 --> @NghiaTranUIT commented on GitHub (Apr 13, 2023): @drzazga3009 Bulk export feature is released on the latest Windows update v1.6.0 👍
Author
Owner

@drzazga3009 commented on GitHub (Apr 13, 2023):

Thanks, I just tested it and it works for JSON responses. Unfortunately, when I choose to export many ts files it only creates an empty directory. Exporting a single ts file works though, proxyman creates a .bin file.

<!-- gh-comment-id:1507100976 --> @drzazga3009 commented on GitHub (Apr 13, 2023): Thanks, I just tested it and it works for JSON responses. Unfortunately, when I choose to export many ts files it only creates an empty directory. Exporting a single ts file works though, proxyman creates a .bin file.
Author
Owner

@drzazga3009 commented on GitHub (Apr 18, 2023):

@NghiaTranUIT not sure if this ticket should be closed, are there any plans to handle all types of response bodies?

<!-- gh-comment-id:1512822601 --> @drzazga3009 commented on GitHub (Apr 18, 2023): @NghiaTranUIT not sure if this ticket should be closed, are there any plans to handle all types of response bodies?
Author
Owner

@NghiaTranUIT commented on GitHub (Apr 18, 2023):

@kics223w1 please fix this issue 👋

<!-- gh-comment-id:1512826314 --> @NghiaTranUIT commented on GitHub (Apr 18, 2023): @kics223w1 please fix this issue 👋
Author
Owner

@kics223w1 commented on GitHub (Apr 18, 2023):

Sure I got it 👍🏻

<!-- gh-comment-id:1512830149 --> @kics223w1 commented on GitHub (Apr 18, 2023): Sure I got it 👍🏻
Author
Owner

@kics223w1 commented on GitHub (Apr 18, 2023):

  • Hi @drzazga3009, I have tried and successfully export response bodies which are TS files.
  • Here is my proxymanlogv2 file, you can try it. Sorry, I haven't found the smaller sample TS file, so this file is quite large.
  • I guess the problem is when Proxyman exports hundreds of response bodies that were too large and causing the error.
  • Can you try to export 2 or 3 response bodies at the same time, if it works then the error is due to large response body instead of the response body type 👍🏻

Notes :

  • I'm using MapLocal Tool to mock the API to return a TS file. However, I have found a bug in MapLocal Tool with the TS file.
  • Therefore, I recommend not using MapLocal Tool with the TS file until the bug is fixed.
  • I'm fixing the bug 👍🏻
<!-- gh-comment-id:1513022098 --> @kics223w1 commented on GitHub (Apr 18, 2023): - Hi @drzazga3009, I have tried and successfully export response bodies which are TS files. - Here is my [proxymanlogv2 file](https://drive.google.com/file/d/1HzGCzkqv9rmzWq2qwLQSkvxDLykBIBJo/view?usp=sharing), you can try it. Sorry, I haven't found the smaller sample TS file, so this file is quite large. - I guess the problem is when Proxyman exports hundreds of response bodies that were too large and causing the error. - Can you try to export 2 or 3 response bodies at the same time, if it works then the error is due to large response body instead of the response body type 👍🏻 Notes : - I'm using MapLocal Tool to mock the API to return a TS file. However, I have found a bug in MapLocal Tool with the TS file. - Therefore, I recommend not using MapLocal Tool with the TS file until the bug is fixed. - I'm fixing the bug 👍🏻
Author
Owner

@drzazga3009 commented on GitHub (Apr 18, 2023):

Thank you for working on this.

I downloaded your file and exporting both response bodies at the same time works. It doesn't work on my real file, no matter if I want to export only 2 at the same time or 20. These responses take less than 100 Kb each. Here is my file, you can try it link

If you need more input like logs just please let me know where they are located.

<!-- gh-comment-id:1513361185 --> @drzazga3009 commented on GitHub (Apr 18, 2023): Thank you for working on this. I downloaded your file and exporting both response bodies at the same time works. It doesn't work on my real file, no matter if I want to export only 2 at the same time or 20. These responses take less than 100 Kb each. Here is my file, you can try it [link](https://drive.google.com/file/d/1uVseW4C7KUum_-IVGW6hJgHUlxDbcoiy/view?usp=sharing) If you need more input like logs just please let me know where they are located.
Author
Owner

@kics223w1 commented on GitHub (Apr 20, 2023):

  • Thank you for providing the proxymanlogv2 file. It really helps me to solve the issue 👍🏻 @drzazga3009
  • The issue has been solved and it'll be in the next release 👍🏻
  • The bug is the length of the file path exceeds the limit length on Windows (256 characters) and that causes the action writing file is failed.
  • In feature bulk export, we build the file name from the URL, and in some URL the length is too long which can cause the length of the file path exceeds the limit length. That's why you see an empty directory because all the actions writing files have failed.
<!-- gh-comment-id:1515780168 --> @kics223w1 commented on GitHub (Apr 20, 2023): - Thank you for providing the proxymanlogv2 file. It really helps me to solve the issue 👍🏻 @drzazga3009 - The issue has been solved and it'll be in the next release 👍🏻 - The bug is the length of the file path exceeds the limit length on Windows (256 characters) and that causes the action writing file is failed. - In feature bulk export, we build the file name from the URL, and in some URL the length is too long which can cause the length of the file path exceeds the limit length. That's why you see an empty directory because all the actions writing files have failed.
Author
Owner

@drzazga3009 commented on GitHub (May 23, 2023):

Hi, sorry it took me so long to verify the fix.
I confirm, exporting many requests body is working now for me, thank you very much.

If may I ask here, what is the file naming convention/the order? The exported files are [1] Response and so on with incremented numbers. But it seems these files are not named in the order.

<!-- gh-comment-id:1558963974 --> @drzazga3009 commented on GitHub (May 23, 2023): Hi, sorry it took me so long to verify the fix. I confirm, exporting many requests body is working now for me, thank you very much. If may I ask here, what is the file naming convention/the order? The exported files are `[1] Response` and so on with incremented numbers. But it seems these files are not named in the order.
Author
Owner

@NghiaTranUIT commented on GitHub (May 23, 2023):

Can you show me the screenshot how it looks like @drzazga3009

<!-- gh-comment-id:1558984203 --> @NghiaTranUIT commented on GitHub (May 23, 2023): Can you show me the screenshot how it looks like @drzazga3009
Author
Owner

@drzazga3009 commented on GitHub (May 23, 2023):

This is a live stream captured. Every other segment is 32 kB, the requests are in order in Proxyman. However, after exporting the requests are not in order.
image

<!-- gh-comment-id:1558996665 --> @drzazga3009 commented on GitHub (May 23, 2023): This is a live stream captured. Every other segment is 32 kB, the requests are in order in Proxyman. However, after exporting the requests are not in order. ![image](https://github.com/ProxymanApp/Proxyman/assets/28488586/b7e1fb21-b106-49bf-8d00-1c781ddc4bfd)
Author
Owner

@NghiaTranUIT commented on GitHub (May 23, 2023):

@kics223w1 please investigate this ☝️ issue 👍

<!-- gh-comment-id:1559081303 --> @NghiaTranUIT commented on GitHub (May 23, 2023): @kics223w1 please investigate this ☝️ issue 👍
Author
Owner

@kics223w1 commented on GitHub (May 23, 2023):

Sure I will double check this.

<!-- gh-comment-id:1559087308 --> @kics223w1 commented on GitHub (May 23, 2023): Sure I will double check this.
Author
Owner

@kics223w1 commented on GitHub (Jun 5, 2023):

@drzazga3009 Sorry for the late response but the issue is fixed.
I changed the code to wait for the file write action to complete before writing the next file, so the exported files should be in the order of the selected requests.

<!-- gh-comment-id:1576507828 --> @kics223w1 commented on GitHub (Jun 5, 2023): @drzazga3009 Sorry for the late response but the issue is fixed. I changed the code to wait for the file write action to complete before writing the next file, so the exported files should be in the order of the selected requests.
Author
Owner

@standinga commented on GitHub (Jun 7, 2023):

@drzazga3009 if you're using Proxyman for macOS, you can export multiple requests at the same time.

Screenshot 2023-04-09 at 21 46 58

I'm using Proxyman Version 2.35.4 (23540) (OSX) and when I select multiple requests (all returning JSON,
the option to export the response body is not available.

Screenshot 2023-06-07 at 16 04 18
<!-- gh-comment-id:1580893647 --> @standinga commented on GitHub (Jun 7, 2023): > @drzazga3009 if you're using Proxyman for macOS, you can export multiple requests at the same time. > > ![Screenshot 2023-04-09 at 21 46 58](https://user-images.githubusercontent.com/5878421/230780006-7fab24f3-33eb-4a1e-99c6-a2a6ba7434df.jpg) I'm using Proxyman Version 2.35.4 (23540) (OSX) and when I select multiple requests (all returning JSON, the option to export the response body is not available. <img width="166" alt="Screenshot 2023-06-07 at 16 04 18" src="https://github.com/ProxymanApp/Proxyman/assets/11987427/ca7455f8-eb2a-495a-92fa-e65e56570bc0">
Author
Owner

@NghiaTranUIT commented on GitHub (Jun 7, 2023):

@standinga please use the latest version, your build 2.35.4 is 2 years ago, when this feature wasn't implemented yet

<!-- gh-comment-id:1581027063 --> @NghiaTranUIT commented on GitHub (Jun 7, 2023): @standinga please use the latest version, your build 2.35.4 is 2 years ago, when this feature wasn't implemented yet
Author
Owner

@standinga commented on GitHub (Jun 7, 2023):

@standinga please use the latest version, your build 2.35.4 is 2 years ago, when this feature wasn't implemented yet

Wow, I wonder how it could happen, I thought I was keeping Proxyman updated (I have the premium version). Sorry for the confusion!

<!-- gh-comment-id:1581043519 --> @standinga commented on GitHub (Jun 7, 2023): > @standinga please use the latest version, your build 2.35.4 is 2 years ago, when this feature wasn't implemented yet Wow, I wonder how it could happen, I thought I was keeping Proxyman updated (I have the premium version). Sorry for the confusion!
Author
Owner

@drzazga3009 commented on GitHub (Jul 14, 2023):

Still, not in order
image
Using Proxyman Version 2.0.0

<!-- gh-comment-id:1636082272 --> @drzazga3009 commented on GitHub (Jul 14, 2023): Still, not in order ![image](https://github.com/ProxymanApp/Proxyman/assets/28488586/4f88876f-b7b2-4a55-bf17-258986e6b040) Using Proxyman Version 2.0.0
Author
Owner

@NghiaTranUIT commented on GitHub (Jul 15, 2023):

@drzazga3009 can you take a screenshot with the ID column again? Maybe the table doesn't sort by ID column. Currently, when exporting to file, it exports sequentially by the ID column

<!-- gh-comment-id:1636613785 --> @NghiaTranUIT commented on GitHub (Jul 15, 2023): @drzazga3009 can you take a screenshot with the ID column again? Maybe the table doesn't sort by ID column. Currently, when exporting to file, it exports sequentially by the ID column
Author
Owner

@drzazga3009 commented on GitHub (Jul 15, 2023):

It's sorted by id
image

<!-- gh-comment-id:1636738886 --> @drzazga3009 commented on GitHub (Jul 15, 2023): It's sorted by id ![image](https://github.com/ProxymanApp/Proxyman/assets/28488586/14bcf956-b83c-41ac-8448-7809a1dd476a)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/Proxyman#1587
No description provided.