mirror of
https://github.com/ProxymanApp/Proxyman.git
synced 2026-04-26 00:25:56 +03:00
[GH-ISSUE #236] Node.js requests don't display in Proxyman #233
Labels
No labels
Discussion
Feature request
In Progress...
Plugins
Waiting response
Windows
Windows
bug
duplicate
enhancement
feature
good first issue
iOS
macOS 10.11
question
wontfix
✅ Done
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/Proxyman#233
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fishcharlie on GitHub (Aug 4, 2019).
Original GitHub issue: https://github.com/ProxymanApp/Proxyman/issues/236
Originally assigned to: @NghiaTranUIT on GitHub.
Proxyman version? (Ex. Proxyman 1.4.3)
Proxyman 1.4.7
macOS Version? (Ex. mac 10.14)
macOS 10.15 Beta (19A526h)
Steps to reproduce
Expected behavior
Request from Node.js to display in Proxyman
Screenshots (optional)
NA
@NghiaTranUIT commented on GitHub (Aug 5, 2019):
Hey, thank for the report. I will check it out today 👍
@NghiaTranUIT commented on GitHub (Aug 5, 2019):
Hey @fishcharlie, I've checked and the answer is that we have to explicitly point the
axiosto Proxyman.then, the request will appear in Proxyman
It's relevant to https://github.com/ProxymanApp/Proxyman/issues/175#issuecomment-515282923, since some CLI doesn't use System Proxy until we explicitly override it.
Additionally, we have to do same for Charles Proxy too.
Hope that could help you 😄 🌮
@NghiaTranUIT commented on GitHub (Aug 5, 2019):
If you don't want to explicitly point the axios to Proxyman, we can use
Then all CLIs (curl, nodejs, ...) will automatically proxy to Proxyman, but make sure we remove it when we don't use Proxyman. Or the app will be failed 😄
@shirshak55 commented on GitHub (Aug 17, 2019):
actually its working on mine. If we use terminal it wont work directly we need to explicitly say axios to use proxy and that is how it work in other cli apps too :) Every cli supports proxy like curl etc.. And as u have already given implicit way using environment variable its fine.
But please add it in request so other don't get same problem.
@alnorris commented on GitHub (Nov 17, 2020):
I'm getting this when ever I try both the environmental vars and the axios proxy. Any ideas?
@shirshak55 commented on GitHub (Nov 17, 2020):
@alnorris it can be your server.
@NghiaTranUIT commented on GitHub (Nov 17, 2020):
@alnorris it might be your port is different than the listening port of Proxyman ( default is 9090)
You can go to proxyman Preference and see the port, then setting it correctly
@NghiaTranUIT commented on GitHub (Nov 17, 2020):
If you don’t mind, pls do the same config with other proxy tools to see where the bug is 😄 because there are many hidden factors could affect
@dvcrn commented on GitHub (Nov 29, 2020):
Is there a workaround to enable the proxy for everything? I sometimes have no influence on what the node app is doing, so having an option to transparently proxy everything would be nice.
Right now I am falling back to mitmproxy which has a workaround using
pfctland transparent mode to route traffic system-wide to the proxy. Works pretty good, but if Proxyman could do that out of the box, it would be reaaaally nice. Setup like this: https://docs.mitmproxy.org/stable/howto-transparent/#macos@shirshak55 commented on GitHub (Nov 29, 2020):
@dvcrn it still wont work if they enable certs pinning (Like zoom app enable cert pinning). And transparent proxy is different thing than http/https proxy. And regarding node app you will have source code right so you can change proxy easily from source code? But yes transparent proxy can help in many situations like some app that don't obey proxy but it still isn't easy today due to certs pinning, android certs being hard to place, and we can forget ios app as most of them use certs pinning anyway. I know we can use frida but thats not easy.
@NghiaTranUIT commented on GitHub (Nov 29, 2020):
I haven't tested
pfctl, but wondering what happens if we use 9090 in the following script9090 is where Proxyman serves
@dvcrn commented on GitHub (Nov 29, 2020):
I don't always have access to the source. For example I was pentesting one mac app but it just wasn't showing up in proxyman/charles and I had no idea why. It took me a while to figure out that the app was spawning a node worker in the background that does the connection and that's why nothing was showing up, which got me wondering what else is happening that I'm missing with Proxyman.
Using mitmproxy + pfctl worked fine even for SSL after trusting the certificate globally
@shirshak55 commented on GitHub (Nov 29, 2020):
@dvcrn did u use pfctl to forward to proxyman? BSD uses pf firewall which is a bit different than iptables unfortunately.
@NghiaTranUIT commented on GitHub (Nov 29, 2020):
He forwarded it to mitmproxy @shirshak55. Btw, it's a cool idea to use pfctl to do transparent Proxy.
From what I see, if pfctl can work with Proxyman, we can fix the VPN issue 😄
@shirshak55 commented on GitHub (Nov 29, 2020):
@NghiaTranUIT pfctl is just a firewall :) And you should have transparent proxy support on proxyman (I think at the moment proxyman supports Http/Https proxy only). pfctl isn't any magic its like iptables for osx. Buy the way if you land transparant proxy soon let me know I can help you with docs regarding how to connect andriod with that transparent proxy.
@NghiaTranUIT commented on GitHub (Nov 29, 2020):
There is a workaround by using proxychains to swamp your app under proxychain umbrella , and all traffic will appear on Proxyman. Read more at https://github.com/ProxymanApp/Proxyman/issues/544
It's quite difficult to set up and requires to disable SIP on your mac, so I would not recommend this approach.
@shirshak55 commented on GitHub (Dec 1, 2020):
@dvcrn I don't know why you are saying charles proxy doesn't work. Charles proxy has transparent proxy so it should have covered your usecase. Its hidden in ssl settings second tab. I haven't used it for year probably but charles proxy does support transparent proxy.
@Cykelero commented on GitHub (Mar 25, 2021):
Seems like Node.js doesn't respect the
http_proxy/https_proxyvariables; instead, you need to use something like global-agent. This is what ended up working in my case.Step by step:
npm install global-agentrequire("global-agent/bootstrap")(or see their docs for how to inject it from the CLI)export GLOBAL_AGENT_HTTP_PROXY=http://127.0.01:9090This should be it!
For those interested, there are lengthy issues on the subject of adding support for the standard proxy vars to Node: https://github.com/nodejs/node/issues/8381, https://github.com/nodejs/node/issues/15620
@NghiaTranUIT commented on GitHub (Mar 25, 2021):
Thank you so much for the contribution 🎉 I will add your comment to official Proxyman doc
@CyberMew commented on GitHub (May 20, 2021):
Has this been fixed yet such that even javascript requests in browsers or some apps which I don't have code access to will work, and have the network requests show up?
@NghiaTranUIT commented on GitHub (May 21, 2021):
@CyberMew it's not a bug, so I can't fix it. It's how NodeJS works.
If you could not access the source code to change the HTTP Proxy, please try to override on your terminal.
It is not 100% work if your app doesn't respect the env http_proxy
Ref: https://www.serverlab.ca/tutorials/osx/administration-osx/configuring-a-network-proxy-for-osx/
@djpowers commented on GitHub (Jul 28, 2022):
I wanted to check if people have since found a workaround for this. I had tried exporting HTTP(S) proxy variables, and using the global-agent approach, but didn't have any luck. Have people been successful with any other techniques?
Also it looks like the issues referenced above about adding proxy support to Node have been closed, so I'm wondering if people are aware of any other discussions about making that happen.
@NghiaTranUIT commented on GitHub (Jul 28, 2022):
The solution is depended on what network library you're using on NodeJS. Can you share with me what the lib is? I might google to find a solution 👍
@djpowers commented on GitHub (Jul 28, 2022):
Thanks @NghiaTranUIT! I'm specifically working on a Raycast extension using their
useFetchhook. It's not currently open source, but the team says the API follows the node-fetch API.@NghiaTranUIT commented on GitHub (Jul 28, 2022):
Google shows that we can pass the HTTPAgent to tell
node-fetchto use the Proxy, like this code:However, useFetch wraps the
node-fetchlib, so I'm not sure how to pass theHttpsProxyAgent. Maybe you should ask Raycast devs for hints.@shirshak55 commented on GitHub (Jul 30, 2022):
@NghiaTranUIT if its react, shouldn't it be running in Browser? So, setting global proxies from windows etc. should work?
@NghiaTranUIT commented on GitHub (Jul 31, 2022):
@shirshak55 If it's a React, which access from the web browser or using NodeJS to serve a static file (Build from ReactJS). Proxyman can capture it since the web browser would use the system HTTP Proxy. (Localhost would pass the proxy by default, but there is a solution to fix it).
@djpowers case is different, he is making a request from NodeJS. Thus, it doesn't respect the global/system HTTP proxy.
@sonic1981 commented on GitHub (Dec 11, 2022):
I also had to run
@gplusdotgr commented on GitHub (Apr 14, 2023):
set the shell env vars,
Could someone explain why would someone buy this product which is supposed to work out of the box but doesnt work even after 3-4 workarounds/hacks?
I'm super disappointed about the effort this requires to get to work. Who is this product for? If its for developers then you're clearly doing something wrong.
@NghiaTranUIT commented on GitHub (Apr 14, 2023):
Sorry @gkatsanos, the problem is from the NodeJS, it doesn't respect the system HTTP Proxy, so it doesn't work out of the box. If you develop iOS/Android app, it works out of the box.
Your /etc/host looks like doesn't have the following line:
Thus, http://proxyman.local:3000/ doesn't work.
Ref: https://docs.proxyman.io/troubleshooting/couldnt-see-any-request-from-localhost-server#solution-2-map-localhost-to-the-domain-name-in-etc-hosts
@gplusdotgr commented on GitHub (Apr 14, 2023):
But then why recommend to people to set the HTTP_PROXY environment variables in their shell?
I feel everytime I try to use Proxyman I'm ping ponged from hack to hack..
@NghiaTranUIT commented on GitHub (Apr 14, 2023):
It's because NodeJS development is really messy. Some libraries require the HTTP_PROXY env, some don't respect the System Proxy -> Needs to manually config in their source code, and some libs do respect the proxy -> Proxyman can work out of the box.
There is no silver bullet to fix all NodeJS problems automatically.
Proxyman helps you a lot behind the scene that you don't notice, such as auto install the certificate on your Mac, auto overriding/reverting the system proxy, or auto-scripts for iOS, and Android, ...
You can try other tools, Charles Proxy, Fiddler, Wireshark, and you have to google a lot to make it works with NodeJS too.
@gplusdotgr commented on GitHub (Apr 14, 2023):
I understand. Concluding, could we have maybe a dedicated NodeJS-specific docs/FAQ page on how to fix it since it's probably a big chunk of the audience?
@NghiaTranUIT commented on GitHub (Apr 15, 2023):
@gkatsanos it depends on your setup. If you don't mind, what is the NodeJS web framework you're using?
Here is a simple sample:
http://proxyman.local:3000-> Proxyman will capture the traffic as usual.If you;re using fetch or axios to call to 3rd-server, and you'd like to capture these traffic. Please check this doc: https://docs.proxyman.io/debug-devices/nodejs
@NghiaTranUIT commented on GitHub (Apr 15, 2023):
@gkatsanos I've updated the NodeJS Documentation to make it clears what the Proxyman & solution: https://docs.proxyman.io/debug-devices/nodejs
@gplusdotgr commented on GitHub (Apr 15, 2023):
thank you , that worked.
@NghiaTranUIT commented on GitHub (Apr 16, 2023):
@gkatsanos thanks for your suggestion. I've found the solution to Automatically capture all traffic that is called from the Terminal app.
It can work with NodeJS, cURL, Ruby, and Python without any manual config 💯
For example: Just open Proxyman -> Click on "Terminal" -> New Terminal/iTerm shows up -> Run your NodeJS server here -> Traffic is automatically captured and send to Proxyman 👍
I'm going to play around and support this feature now. It will resolve all pain we have 😄
@gplusdotgr commented on GitHub (Apr 16, 2023):
Sounds awesome! is it possible to start a terminal already now from the current Proxyman version or is this a future feature? :)
@NghiaTranUIT commented on GitHub (Apr 19, 2023):
@gkatsanos here is a sneak peek of what I'm doing 😄
--proxyflag), and NodeJS (includingfetch,axios,request, andsuperagentlibraries).Video Demo
fetchto call to an external API=> ✅ Proxyman captures it automatically
=> ✅ User doesn't need to modify any source code.
https://user-images.githubusercontent.com/5878421/232966680-20327230-2818-49a2-b671-5b8f2b6c3f38.mp4
@NghiaTranUIT commented on GitHub (Apr 20, 2023):
@gkatsanos just wondering: What's the NodeJS network library you're using? Currently, in the Beta build, It supports axios, got, superagent, fetch (node v18), node-fetch, and a built-in NodeJS
httpspackage.It works out of the box ✅ (No Proxy config, no certificate)
@gplusdotgr commented on GitHub (Apr 20, 2023):
@NghiaTranUIT https://docs.astro.build/en/guides/data-fetching/ I think it uses some form of native fetch
@NghiaTranUIT commented on GitHub (Apr 22, 2023):
Good news: @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero
Proxyman can:
out of the box✅Download
Beta: https://download.proxyman.io/beta/Proxyman_4.6.1_Feature:_Automatic_Setup.dmg
Automatic Setup
Manual Setup
Documentation:
https://docs.proxyman.io/automatic-setup/automatic-setup
Demo video:
aiohttplibrary). NodeJS is similar.@NghiaTranUIT commented on GitHub (Apr 22, 2023):
@gkatsanos if you don't mind, please give it a try. I've tested with NodeJS (fetch). Proxyman can see traffic from
fetchrequests 👍@Geczy commented on GitHub (Apr 25, 2023):
trying this now, i think its working to capture requests, but it's not actually forwarding the request to the original server from what i can tell. im getting a
HTTP/1.1 301 Moved Permanentlyon all outgoing requests from inside the electron app. i started the app from the terminal viaopen ....xdyz/Title.appi don't have this issue in httptoolkit
it looks like maybe proxyman is not following the location (
Lparameter in curl)so the original request is
httpand it's not following tohttps@NghiaTranUIT commented on GitHub (Apr 25, 2023):
@Geczy can you share with me how your setup your Electron app 🤔
You have the source code, which allows you to start the Electron app from the Terminal, right? Or you'd like to capture traffic from other Electron app?
From what I know, I need to support ElectronJS. The above Beta build is only for NodeJS, Ruby, and Python.
@Geczy commented on GitHub (Apr 25, 2023):
well previously in the production release, the electron app did not even display in proxyman as an "app" or any of the domains or anything. it's only captured if i use the terminal that proxyman opens to open the app.
i start this app by doing
open /System/Volumes/Data/Users/matt/Applications/xyz.appin the spawned terminalin this build, it does display. the electron app uses axios under the hood, if that matters. i don't have the source code for it.
@NghiaTranUIT commented on GitHub (Apr 25, 2023):
I see, if you don't mind, can you share with me the name of your Electron app via nghia@proxyman.io ? I'd like to download production build, test it and fix it 👍
@Geczy commented on GitHub (Apr 25, 2023):
sure, i sent you the email. thanks for taking a look. super sexy app
@NghiaTranUIT commented on GitHub (Apr 26, 2023):
New Beta build 🎉
https://github.com/ProxymanApp/Proxyman/issues/1611#issuecomment-1521044841
@NghiaTranUIT commented on GitHub (Apr 29, 2023):
Just a friendly reminder that we've released the automatic / manual setup 🎉
cc @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero
You can checkout the latest build or from this comment: https://github.com/ProxymanApp/Proxyman/issues/236#issuecomment-1518572201
@gplusdotgr commented on GitHub (Apr 29, 2023):
Thank you!
@NghiaTranUIT commented on GitHub (Apr 29, 2023):
@Geczy I've fixed the ElectronJS issue, please get the Beta build from https://github.com/ProxymanApp/Proxyman/issues/1618#issuecomment-1528565327
@Geczy commented on GitHub (Apr 30, 2023):
Can confirm it's fixed !
@ChiuMungZitAlexander commented on GitHub (May 17, 2023):
My question is how can I disable it?
@NghiaTranUIT commented on GitHub (May 17, 2023):
@ChiuMungZitAlexander Automatic Script doesn't start if you don't open the Terminal app. It only affects the current Terminal Session, not permanently change your OS. So, no need to disable it.
@gplusdotgr commented on GitHub (Jun 14, 2023):
Hey @NghiaTranUIT :
I get this error when trying to open up a new 'automatic setup' terminal. I use
@gplusdotgr commented on GitHub (Jun 14, 2023):
another issue I have (unrelated?) is even though I see one request, I dont see another one which happens to an external API endpoint but which passes through my VPN client.. could this be the reason?
@NghiaTranUIT commented on GitHub (Jun 14, 2023):
@gkatsanos yes it can be a problem. If you're using
fish, when opening the Automatic Setup Terminal -> Can you print theecho $PATH, to see if it contains the Proxyman path?It's where Proxyman overrides the NodeJS env.
The reason is: there is a small warning from your log.
@gplusdotgr commented on GitHub (Jun 14, 2023):
@leohxj commented on GitHub (Sep 25, 2023):
add
export NODE_TLS_REJECT_UNAUTHORIZED=0after manual setup.@NghiaTranUIT commented on GitHub (Sep 25, 2023):
@leohxj a Beta build for you: https://download.proxyman.io/beta/Proxyman_4.11.0_Improve_UI_v1.dmg
Changelog
@leohxj commented on GitHub (Sep 26, 2023):
thanks @NghiaTranUIT , it works well
@shirshak55 commented on GitHub (Sep 27, 2023):
actually I don't recommend this. Last week , I think from Egypt government was hacked because of http. Although, you may not be the target, I don't recommend messing with security.
I think proxy man should explicitly ask if they want to reduce this security?
@NghiaTranUIT commented on GitHub (Sep 27, 2023):
@shirshak55 it doesn't globally modify your entire system. It only takes effect on the current Terminal app session when you need to debug your NodeJS project.
@leohxj commented on GitHub (Sep 27, 2023):
yes, only for develop and use it locally
@tonyxiao commented on GitHub (Nov 3, 2023):
Does this work with the built in node.js fetch now? Or is the
node-fetch3rd party lib required to use with proxyman?@NghiaTranUIT commented on GitHub (Nov 3, 2023):
It works for both. Native fetch and node-fetch
@tonyxiao commented on GitHub (Nov 3, 2023):
Which env variable does native fetch use? When I look at the node docs it seems that fetch doesn't support proxy / agent agt all.
@NghiaTranUIT commented on GitHub (Nov 3, 2023):
Auto Setup from Proxyman does more things than just the env. Proxyman uses global-agent to auto override the proxy of NodeJS Fetch in runtime.
You simply:
@tonyxiao commented on GitHub (Nov 3, 2023):
Hmm but the
global-agentlib doesn't actually seem to support node 18 fetch. @see https://github.com/gajus/global-agent/issues/52 Does proxyman use the undici workaround too?@NghiaTranUIT commented on GitHub (Nov 3, 2023):
Yes, it's.
@NghiaTranUIT commented on GitHub (Sep 26, 2024):
FYI: We can capture all HTTPS Traffic from NodeJS with the Auto Setup (https://docs.proxyman.io/debug-devices/nodejs)
No need to config the Proxy or manually trust the self-signed certificate on your NodeJS code 👍