mirror of
https://github.com/benbusby/whoogle-search.git
synced 2026-04-25 12:15:50 +03:00
[GH-ISSUE #1211] Google no longer supports javascript-free search requests #726
Labels
No labels
Fixed (Pending PR Merge)
Stale
bug
enhancement
enhancement
good first issue
help wanted
keep-open
needs more info
pull-request
question
theme
unfortunate
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/whoogle-search#726
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @buzzert on GitHub (Jan 16, 2025).
Original GitHub issue: https://github.com/benbusby/whoogle-search/issues/1211
Describe the bug
As of late, I'm starting to get empty search results when searching for something.
Screenshot:
It actually appears the HTML returned from Whoogle contains a fragment of an error message from Google (I presume):
To Reproduce
Steps to reproduce the behavior:
Deployment Method
runexecutableVersion of Whoogle Search
Desktop (please complete the following information):
@vkoaala commented on GitHub (Jan 16, 2025):
I'm having the same issues, but I'm being rate limited as per my docker logs : ERROR:app:503 (CAPTCHA)
@buzzert commented on GitHub (Jan 16, 2025):
Not seeing any rate limitations mentioned in my logs.
@LupinThidr commented on GitHub (Jan 16, 2025):
Not directly related to this project, but I've been getting these same redirects for the past couple weeks (on my browser with JS disabled). I've been able to "bypass" it by just simply re-querying, but as of today, they've become more persistent. After enabling Javascript, I'm redirected to the usual "google.com/sorry/index" captcha page that used to regularly happen to me on VPNs.
Relevant HN discussion from today ("Google.com search now refusing to search for FF esr 128 without JavaScript"): https://news.ycombinator.com/item?id=42719865
@mon5termatt commented on GitHub (Jan 16, 2025):
Getting this as well, temporarily redirecting my engine to searxng but would love to get this resolved as it's too cluttered :D
was putting off making an issue while i verified it on a new install....
@scottmando2000 commented on GitHub (Jan 16, 2025):
Also experiencing this behavior, started happening a few days ago and has entirely stopped working today.
@philebosxxx commented on GitHub (Jan 16, 2025):
I am too having exactly the same issue
@gitshipdone commented on GitHub (Jan 16, 2025):
+1
@blackletum commented on GitHub (Jan 16, 2025):
I am also having the same issue, on latest.
@skaboy71 commented on GitHub (Jan 16, 2025):
Same here -was able to refresh but now no-worky. Running in docker on Unraid
@damien-crypto commented on GitHub (Jan 16, 2025):
Same on Umbrel OS but only in the "ALL" option.
The rest work, news, images etc.
@jbhorner commented on GitHub (Jan 16, 2025):
+1
News, Images, etc. works for me as well. I tried to use by disabling tor through environment variable, but that didn't change anything.
@gads83 commented on GitHub (Jan 16, 2025):
+1 here as well
@abhandal commented on GitHub (Jan 16, 2025):
+1 same issue
@patienttruth commented on GitHub (Jan 16, 2025):
Same here since this morning. Relieved for a moment that it's not on my end, but that's actually more concerning...
@mitanshu7 commented on GitHub (Jan 16, 2025):
+1
@conundrum741 commented on GitHub (Jan 16, 2025):
+1
@jersam commented on GitHub (Jan 16, 2025):
+1
@Nick4154 commented on GitHub (Jan 16, 2025):
+1 for me, Docker or Debian
@Pootch commented on GitHub (Jan 16, 2025):
Same issue here. No change from my side at the docker compose.
Getting this page instead:
@mooglestiltzkin commented on GitHub (Jan 16, 2025):
+1 truenas docker container.
only recently occured, empty results.
@qcdesign commented on GitHub (Jan 16, 2025):
+1 for me, Docker. Same as everyone. I get the 503 captcha in the docker log but blank results, Images/News etc. work fine.
@stdestro commented on GitHub (Jan 16, 2025):
same here on docker. search results is blank, images/videos/news/maps works
@mztiq commented on GitHub (Jan 16, 2025):
+1
@Tubeman-cell commented on GitHub (Jan 16, 2025):
+1
@dsvdglenn commented on GitHub (Jan 16, 2025):
+1
@bruvv commented on GitHub (Jan 16, 2025):
The +1 comments are pretty useless, please there is a
button, use that instead.
@foegra commented on GitHub (Jan 16, 2025):
maybe would be good to add backup search engine?
@mztiq commented on GitHub (Jan 16, 2025):
@bruvv the +1 is not about subscribing it's more for showcasing how many people are affected by this issue.
@bruvv commented on GitHub (Jan 16, 2025):
@mztiq please use emoticons then:
This is just spamming @benbusby i think he now knows about this issue.
@inittux111 commented on GitHub (Jan 16, 2025):
@mztiq Wouldn't be better to start a poll then or a thumbs up emote which doesn't cause a mail notification? It's annoying to get a "+1" notification mail for every person that has the problem, I would rather be notified when someone actually adds a comment with an actual reply rather than "me too" notifications.
@mooglestiltzkin commented on GitHub (Jan 16, 2025):
" You’ve been unsubscribed from this thread. "
problem solved ^-^;
@inittux111 commented on GitHub (Jan 16, 2025):
That does not solve the problem because I want to be notifiied when something useful is actually said that's about the open issue.
@mooglestiltzkin commented on GitHub (Jan 16, 2025):
well u can check in every now and then at your own time. but if u simply want to complain without a solution, well.. *rolls eyes
@mztiq commented on GitHub (Jan 16, 2025):
I understood and accepted the politely expressed criticism of @bruvv and @Hydranet. IMHO there is no need for more discussion around this which is producing even more spam that is not related to this issue.
@loudfm commented on GitHub (Jan 16, 2025):
Same on Synology docker but only in the "ALL" option.
The rest work, news, images etc.
@misa1313 commented on GitHub (Jan 16, 2025):
Experiencing the issue on Debian 12, docker 27.5.0, latest whoogle release.
@benbusby commented on GitHub (Jan 16, 2025):
I haven't read everyone's comments yet, but yeah I'm aware of the issue. It looks like Google has removed the ability to fetch search results from their javascript-free URL (which is what Whoogle uses for everything). This was a feature they supported for over a decade, so I'm kinda surprised they seemingly killed it with no warning. Usually they only do that after a year or so of supporting a product.
At this point I'm not sure what the best option would be. I've tried a few methods using the "If this page doesn't look correct, use this link: ", but the link they provide still requires javascript to load anything. All of Whoogle is built around access to Google's javascript-free results so that it can perform the filtering on the backend, and rearchitecting everything to scrape results from these changes would probably mean an entirely new project (which atm is not something I think I have time for unfortunately).
I'll continue to monitor the changes they're making and post an update if I find a workaround or if the change seems permanent. In the meantime, I'd highly recommend checking out Kagi. It's not quite the same level of privacy preservation that Whoogle was going for, but the results are typically a lot better than Google/Whoogle results anyways. It is unfortunately a paid service, however, so not a good solution for all.
Apologies to all who were surprised by this like I was. It's not fun to wake up and not have a working search engine
Also see: https://news.ycombinator.com/item?id=42719865 (particularly this comment which mirrors what I saw as of this morning: https://news.ycombinator.com/item?id=42720062)
@inittux111 commented on GitHub (Jan 16, 2025):
@benbusby While looking for an alternative I came across a search engine done by Mullvad.
https://alternativeto.net/software/mullvad-leta/about
https://leta.mullvad.net
Mullvad Leta uses the Google Search API as a proxy, caching each search. These cached results are shared amongst all users, reducing costs and improving privacy. This service is user-supported and doesn't rely on ads or data selling.They use the Google api as a proxy too and it still works so it would be interesting to find out how they implemented it, only downside of Mullvad Leta is you have to be connected to Mullvad vpn to use it. Unless it's just by pure coinicidence and that it's cached searches that I'm getting now. I guess we''ll find out in a few days.
@benbusby commented on GitHub (Jan 16, 2025):
Interesting, thanks for sharing that. My assumption is that they're probably paying Google for access to their search API, which is something that other companies do as well (I think at least Startpage, but I'm sure some others do too). If not, I'll also be interested to see if they start having issues over the next few days.
@JMiahMan1 commented on GitHub (Jan 16, 2025):
Looks like the Lynx browser still works with Google Search not using Javascript (I tested it) is there a way you can spoof it?
@benbusby commented on GitHub (Jan 16, 2025):
Yeah looks like using the Lynx user agent works (at least for now). The styling is a bit weird when using that UA, but better than nothing! Will push a quick fix now. Thanks for finding that @JMiahMan1
@jon6fingrs commented on GitHub (Jan 17, 2025):
Hi just tried your fix and working great. Thank you.
@LupinThidr commented on GitHub (Jan 17, 2025):
If Lynx ever gets blocked (probably will, if the current game of whack-a-mole that's been going on against YouTube is anything to go by), a few more UA's worth trying are the Nintendo ones (3DS / Wii) and Opera Mini's. There's still millions of users in poorer countries reliant on Opera Mini and its serverside rendering.
@patienttruth commented on GitHub (Jan 17, 2025):
Forgive my lack of understanding.
How are you able to try the fix? I'm using the docker compose environment, is it possible, or do I have to build something?
@jbhorner commented on GitHub (Jan 17, 2025):
From the command prompt in the environment you're using to host Docker:
(Run these in the directory where your Docker Compose file is located.)
docker compose down
docker compose pull
docker compose up -d
The above are in a Linux environment using a standard docker implementation. I know there are some variants where it is "docker-compose" instead, but I do not have that and therefore haven't tested that.
In order what this does:
Number 2 ensures a pull. I've seen Docker not pull images even if the tag in the docker compose file is :latest
@JMiahMan1 commented on GitHub (Jan 17, 2025):
I wanted to thank you as well, great work on this project. Thank you so much for the time and effort you put into it. Hopefully, someone at Google has a love for Lynx (https://news.ycombinator.com/item?id=21626995) and leaves you alone for a while.
@Don-Swanson commented on GitHub (Jan 17, 2025):
@benbusby I spent a little time tonight (A little past my bedtime, but for a worthy cause) and made a PR to enable User Agent Switching (Including adding the ability for users to enter a custom user agent)
https://github.com/benbusby/whoogle-search/pull/1212
As a side note, in playing around, I did find at least 2 other UA strings that work. The ones I remember are:
Mozilla/1.22 (compatible; MSIE 2.0; Windows 3.1)
BlackBerry7100/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1
@rstefko commented on GitHub (Jan 17, 2025):
Thanks guys, just tried LYNX UA and it works, I don't mind the format is different. I might like it more :)
For others you may just change the existing Environment variables:
Environment=WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0" Environment=WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"and make sure
Environment=WHOOGLE_USE_CLIENT_USER_AGENT=0@jniggemann commented on GitHub (Jan 17, 2025):
Direct install, not using docker. I confirm setting the 3 aforementioned ENV variables (and restarting the service) as a working workaround.
@xircon commented on GitHub (Jan 17, 2025):
Which UA is "better"? @Don-Swanson
@pezz commented on GitHub (Jan 17, 2025):
I'm guessing containered versions have no problem with cssutils at 2.6.0?
@patienttruth commented on GitHub (Jan 17, 2025):
I can do that, thank you very much!
What I didn't realize was that docker images were current, since the release # in GitHub hadn't changed for a few months. I assumed new docker images weren't being generated.
@arhcoder commented on GitHub (Jan 17, 2025):
Same problem. Not with this project, but with an own Python SERP tool. Suddenly stop working, and the problem seems to be that Google now requires enabled Javascript. Lynx seems to work using as User-Agent on the request like this:
But for my case it ask to solve a Captcha, so Selenium with visible driver is my only solution, it is not scalable.
@Don-Swanson commented on GitHub (Jan 17, 2025):
Whatever works is “better”. They all seemed to provide the same experience as the Lynx one.
@anoo2niem commented on GitHub (Jan 17, 2025):
While using the lynx one, i get results on 'all', but not under the 'images' tab.
I've found one which returns results either in 'all' & 'images':
Environment=WHOOGLE_USER_AGENT="Mozilla/3.0 (compatible; MSIE 3.0; Windows NT 5.0)"@benbusby commented on GitHub (Jan 17, 2025):
Just merged the PR from @Don-Swanson -- thank you again!
I'm going to keep this issue open, however, as I'm unsure how reliable this new User Agent hopping is going to be. With the latest changes, you can enter a custom User Agent string if the one that you're using (either the new default Lynx one, or one you set yourself) gets blocked. My hope is that this is a longer term fix, but it's impossible to know for sure.
One downside to this new approach is that although results seem to be returned correctly now, they are styled and/or formatted slightly differently depending on the configured User Agent. This could impact the backend filtering that Whoogle performs for each search, so you may or may not see things like sponsored content, ads, etc returning for some searches. Open an issue if that happens and I'll do what I can to address it.
In the meantime, I've also added a new Wiki page for tracking which User Agents are working for returning results -- see: https://github.com/benbusby/whoogle-search/wiki/User-Agents. If you find another User Agent that works for you but isn't in this list, let me know!
@xircon commented on GitHub (Jan 17, 2025):
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"
@benbusby - Vivaldi UA works - I use Vivaldi.
@talentedmrripley commented on GitHub (Jan 17, 2025):
This one isn't working for me. I've tested several of the others and they all return results but this one doesn't.
@xircon commented on GitHub (Jan 17, 2025):
Maybe because I am using vivaldi???
The above is my docker-compose.yml snipped.
@jeffbl commented on GitHub (Jan 17, 2025):
Looks like this is not an accident:
https://techcrunch.com/2025/01/17/google-begins-requiring-javascript-for-google-search/
@calderjh commented on GitHub (Jan 18, 2025):
Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!
@cauliflowerdoughnuts commented on GitHub (Jan 18, 2025):
I'm assuming you used the Proxmox VE Helper-Scripts? Just re-run their install command and it will pull the update via pip.
@VenomousRage commented on GitHub (Jan 18, 2025):
I am in a similar situation. I just tried the in-built "update" command, as well as then deleting the LXC and spinning up a fresh new one with the install script, and neither worked. The issue remains the same. The "all" tab is empty. The other tabs all populate results.
Perhaps when the community script is ran, it pulls a cached copy of Whoogle to install and not a fresh one?
@calderjh commented on GitHub (Jan 18, 2025):
No didnt work for me - also tried "update" - nothing - same results as VenomousRage
@calderjh commented on GitHub (Jan 18, 2025):
I did try to search for the modified file so I could just overwrite it but there are multiple versions and I didnt know which to replace - this is via "filebrowser" - also installed as a proxmox script into the LXC whoogle lives in.
@cauliflowerdoughnuts commented on GitHub (Jan 18, 2025):
@calderjh if the update worked you should have a user agent drop down option in the config on the whoogle landing page and it should default to lynx which works. if you dont have that drop down then the update didnt work.
@calderjh commented on GitHub (Jan 18, 2025):
OK that's not there and version still showing as v0.9.1 (current) - from what I can see, just one file has been updated, do you know where this is located exaclty? I could just drag-drop the new one over?
@cauliflowerdoughnuts commented on GitHub (Jan 18, 2025):
try running
pip3 install whoogle-search --upgradein your LXC container@calderjh commented on GitHub (Jan 18, 2025):
Gives me this:
Last login: Sat Jan 18 00:58:20 GMT 2025 on tty1
root@whoogle:
# pip3 install whoogle-search --upgrade#Requirement already satisfied: whoogle-search in /usr/local/lib/python3.11/dist-packages (0.9.1)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (4.12.3)
Requirement already satisfied: brotli in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.1.0)
Requirement already satisfied: cssutils in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (2.11.1)
Requirement already satisfied: cryptography in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (44.0.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (0.7.1)
Requirement already satisfied: Flask in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (3.1.0)
Requirement already satisfied: python-dotenv in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.0.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from whoogle-search) (2.28.1)
Requirement already satisfied: stem in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.8.2)
Requirement already satisfied: validators in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (0.34.0)
Requirement already satisfied: waitress in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (3.0.2)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.11/dist-packages (from beautifulsoup4->whoogle-search) (2.6)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.11/dist-packages (from cryptography->whoogle-search) (1.17.1)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.11/dist-packages (from cssutils->whoogle-search) (10.5.0)
Requirement already satisfied: Werkzeug>=3.1 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (3.1.3)
Requirement already satisfied: Jinja2>=3.1.2 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (3.1.4)
Requirement already satisfied: itsdangerous>=2.2 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (2.2.0)
Requirement already satisfied: click>=8.1.3 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (8.1.7)
Requirement already satisfied: blinker>=1.9 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (1.9.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.11/dist-packages (from cffi>=1.12->cryptography->whoogle-search) (2.22)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/dist-packages (from Jinja2>=3.1.2->Flask->whoogle-search) (3.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@whoogle:
But still not working
@dave0003 commented on GitHub (Jan 18, 2025):
For what it's worth, "pip install whoogle-search --upgrade" also doesn't seem to be pulling any updates for me, still at 0.9.1.
I also usually get the latest requirements and update them.
It did get some new cssutils.
But I did get it working by manually adding the user agent to the environment:
whoogle/lib/python3.11/site-packages
@calderjh commented on GitHub (Jan 18, 2025):
Thanks so much but I can't find site-packages for the life of me. In that location I have sitecusomize.py but not sure i should be meddling with this if its the wrong one!
@bullfrog-au commented on GitHub (Jan 18, 2025):
New install using docker-compose and the user agent env variables, I get an error on creating the container.
[+] Running 0/1⠋ Container whoogle-search Creating 0.0sError response from daemon: invalid mount path: 'WHOOGLE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"' mount path must be absoluteDocker compose yaml
- WHOOGLE_USE_CLIENT_USER_AGENT=0#- WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"- WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"- WHOOGLE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0Safari/537.36 Vivaldi/7.0.3495.29"@dave0003 commented on GitHub (Jan 18, 2025):
site-packages is a directory. I should have added the filename.
whoogle/lib/python3.11/site-packages/whoogle.envYou might need to add the file. But your setup might be different, too. I'm still not sure why it's not pulling any update.
@calderjh commented on GitHub (Jan 18, 2025):
Totally different set-up I think as nothing resembling this exists. The version number has not been incremented so i would assume that the old version is still being dished out? But I am ignorant of how GitHub works so could be wrong.
@bullfrog-au commented on GitHub (Jan 18, 2025):
Looking at the current main branch on github the active version is '0.9.1'
@calderjh commented on GitHub (Jan 18, 2025):
0.91 was released on 24th October last year. Which, I guess, is why we have no fix in that version - I just wasn't sure if the code was a "living entity" as such whereby any requests to download automatically fed from an updated version.
Seems not.
@golesscom commented on GitHub (Jan 18, 2025):
Programmatically decode and decipher the code provided by Google to determine if JavaScript is enabled, and then save this code in cookies or local storage as requested.
@xaabi6 commented on GitHub (Jan 18, 2025):
For all of those using Whoogle LXC container by Proxmox VE Helper-Scripts, I've updated requirements as @dave0003 suggests:
Then, adding environment variables was quite different. I achieved it by editing the service using
systemctl edit whoogle.servicecommand:Then restart the service issuing
systemctl restart whoogle.serviceand you're done. Hope it helps someone!@dave0003 commented on GitHub (Jan 18, 2025):
That makes sense. In my whoogle.service file I've got:
Environment=WHOOGLE_DOTENV=1I think that tells it to use the whoogle.env file, but it looks like you can add them all there, too. I use it to set some other settings, too, so it's handy.
@xaabi6 commented on GitHub (Jan 18, 2025):
You're right, it can also be done with a dedicated file for environment variables, but as the service needs to be edited for both types of configuration, I prefered to put all of them in the same place 😉
@bitchecker commented on GitHub (Jan 18, 2025):
I've been experiencing the same issue!
I've also configured
WHOOGLE_FALLBACK_ENGINE_URL=https://duckduckgo.com/?k1=-1&q=to try to work around, but it doesn't seem to work (maybe my config is wrong, but it what is reported on readme).Maybe a configuration to change the backend search engine could be helpful.
@patienttruth commented on GitHub (Jan 19, 2025):
This worked great though now I'm seeing mention of TOR in the logs. I don't believe that was there before.
@xaabi6 commented on GitHub (Jan 19, 2025):
Your configuration is fine but I'm not sure about the query string, please, check this recent commit. You're probably not seeing the fallback engine because Whoogle is not failing, it just doesn't show any result because of the recent change Google did.
@bitchecker commented on GitHub (Jan 19, 2025):
So, instead of
https://duckduckgo.com/?k1=-1&q=I should usehttps://duckduckgo.com/search?q=?I think that having a completely empty result can be an error just because when you search something on a search engine and no results are found, at least a string with something like "your search terms are not producing any result!"
@xaabi6 commented on GitHub (Jan 19, 2025):
Based on that commit, it seems so.
I admit that now that we know where the problem cames from, it could be treated as an error, so if it happens again, at least we still can continue using Whoogle despite it always redirects to another engine.
@Alifoss commented on GitHub (Jan 19, 2025):
So sad tbh, Invidious is kinda broken at the moment luckily piped still works routed through a vpn, sometimes... Now whoogle, which i used so much and feel honestly sad now it isn't what it used to be.
Changing user agent works, but i can't cope with the formatting. I've spinned up a SearXNG instance since it fetches results from different sources, even if for example google return nothing i'll still get results from brave, duckduckgo, startpage etc.
To my suprise, my searXNG instance can still fetch results from google. Did they already use a specific user agent? Or does searxng fetches results from google with js enabled? Could anybody more knowledgable on this matter explain the differences backend wise?
@inittux111 commented on GitHub (Jan 20, 2025):
I remember reading about it somewhere so I looked it up, searXNG uses the Google search mobile site to fetch results, they implemented it in 2021.
https://github.com/searxng/searxng/issues/159
https://github.com/searxng/searxng/pull/160
Then in 2023 they ran into the issue of getting rate-limited by Google after several searches, they found a fix for that as well.
https://github.com/searxng/searxng/issues/2515
@MadWalnut commented on GitHub (Jan 20, 2025):
With the new user agent has anybody found a way to color visited links? I don't mind the different layout but for me all links are gray, even if I have visited them in the past.
@aibosss commented on GitHub (Jan 20, 2025):
+1
@iMartyn commented on GitHub (Jan 21, 2025):
I don't think I have seen this mentioned, but using the lynx user agent version (which I'm so grateful for!) seems to do regional searching based on geoip, even though I have the
WHOOGLE_CONFIG_LANGUAGEandWHOOGLE_CONFIG_SEARCH_LANGUAGEset toen. I'm getting lots of German results which is frustrating as there doesn't seem to be a way to switch it either. :(@CAL77 commented on GitHub (Jan 22, 2025):
I don't know what I'm doing and I could use a little help. My WHOOGLE is at ver 0.8.1. Using Docker, I run this command to update:
docker run -p 5000:5000 -d --name whoogle-search benbusby/whoogle-search:latest
Although it seems to execute, I'm still at v0.8.1. Please help me with the right steps to upgrade to the latest. Thank you.
@iMartyn commented on GitHub (Jan 22, 2025):
@CAL77 this is not really related to this issue, but docker caches your image and because you're using the
:latesttag, it thinks nothing has changed and doesn't check for a newer version. Using:latestis a thorny topic that is way out of scope. To ensure you get the latest version you can either rundocker pull benbusby/whoogle-search:latestto force download the latest version of:latestor remove the image (making sure you've removed any containers using it first) withdocker rmi benbusby/whoogle-search:latest, then docker run will use the new image (either because it doesn't exist locally or because you've manually pulled it).@benbusby commented on GitHub (Jan 22, 2025):
I've made some changes that should improve the visual styling of results when using the Lynx User Agent (including fixing the visited link coloring).
Between the styling fixes and the seeming stability of using the new user agent(s), it looks like this is at least a short term fix that isn't being immediately blocked. And since the issue comments have gotten a little off topic, I'm going to close the issue for now.
The latest version 0.9.3 includes both the styling fixes as well as the user agent fixes from 0.9.2 -- if there are still issues with either fetching results or styling problems, open a new issue and let me know!
@bitchecker commented on GitHub (Jan 22, 2025):
Is this fix solving the "empty result set" issue related to JavaScript-free search requests blocked by google? 🤔
@benbusby commented on GitHub (Jan 22, 2025):
@bitchecker yes. The user agent changes address that.
@xircon commented on GitHub (Jan 22, 2025):
Thanks Ben, can confirm the link highlighting is now good.
@bitchecker commented on GitHub (Jan 22, 2025):
great job! 👍🏻
@hobby-grade commented on GitHub (Sep 10, 2025):
I am not 100% sure of this, but this user agent change may now be causing this error when performing searches:
@scottmando2000 commented on GitHub (Sep 10, 2025):
Experiencing the same behavior
@inittux111 commented on GitHub (Sep 10, 2025):
@hobby-grade @scottmando2000 That's not the same issue, someone already started a new issue about this here.
https://github.com/benbusby/whoogle-search/issues/1239
@hobby-grade commented on GitHub (Sep 10, 2025):
I understand that. I am simply mentioning that the issue in #1239 is likely a product of the solution to this issue. I figured it should be considered a sub issue of this one.