starred/soccerdata

Fork 0

mirror of https://github.com/probberechts/soccerdata.git synced 2026-04-26 10:35:53 +03:00

[GH-ISSUE #398] [Whoscored] headless error with WSL #76

New issue

Closed

opened 2026-03-02 15:55:34 +03:00 by kerem · 9 comments

kerem commented

2026-03-02 15:55:34 +03:00

Owner

Originally created by @mhd0528 on GitHub (Oct 5, 2023).
Original GitHub issue: https://github.com/probberechts/soccerdata/issues/398

Hi,

I have come across the following errors with setting headless to either True or False.
Set to False:

Set to True:

I noticed similar issues with the error when setting it to True, which might be due to undetected-chrome. But I don't know what's the cause of the error with setting headless to False...
I have installed undetected-chromedriver==3.5.3 and chromium-browser 1:85.0.4183.83-0ubuntu0.20.04.3 amd64. I noticed most of the posts saying they have chrome 116 or 117, so I'm wondering if this could be the reason since I'm using the package on WSL subsystem in Windows.

Thanks.

Originally created by @mhd0528 on GitHub (Oct 5, 2023). Original GitHub issue: https://github.com/probberechts/soccerdata/issues/398 Hi, I have come across the following errors with setting headless to either True or False. Set to False: ![image](https://github.com/probberechts/soccerdata/assets/22436513/6111a496-79a0-4ea4-b478-a807c41833a1) Set to True: ![image](https://github.com/probberechts/soccerdata/assets/22436513/de44679d-a22e-44bb-9789-ff9e5b8f338d) I noticed similar issues with the error when setting it to True, which might be due to undetected-chrome. But I don't know what's the cause of the error with setting headless to False... I have installed `undetected-chromedriver==3.5.3` and `chromium-browser 1:85.0.4183.83-0ubuntu0.20.04.3 amd64`. I noticed most of the posts saying they have chrome 116 or 117, so I'm wondering if this could be the reason since I'm using the package on WSL subsystem in Windows. Thanks.

kerem

2026-03-02 15:55:34 +03:00

closed this issue
added the
bug

WhoScored
labels

kerem commented

2026-03-02 15:55:35 +03:00

Author

Owner

@probberechts commented on GitHub (Oct 5, 2023):

This SO post explains the problem and gives a few solutions: https://stackoverflow.com/questions/77191221/undetected-chromedriver-attributeerror-chromeoptions-object-has-no-attribute

I haven't tested it yet, but downgrading selenium to 4.12 seems the most straightforward solution.

Ultimately, this has to be fixed in undetected-chromedriver (see https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1584); not in soccerdata.

@probberechts commented on GitHub (Oct 5, 2023): This SO post explains the problem and gives a few solutions: https://stackoverflow.com/questions/77191221/undetected-chromedriver-attributeerror-chromeoptions-object-has-no-attribute I haven't tested it yet, but downgrading selenium to 4.12 seems the most straightforward solution. Ultimately, this has to be fixed in undetected-chromedriver (see https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1584); not in soccerdata.

kerem commented

2026-03-02 15:55:35 +03:00

Author

Owner

@Gibranium commented on GitHub (Oct 6, 2023):

Sorry to say it Pieter, but to me it doesn't work downgrading selenium - tried 4.12, 4,11, 4.10 - neither using the selenium base import SB method at the link.
Here I've used the SB method and I have tor active, don't know what's wrong at this point.

@Gibranium commented on GitHub (Oct 6, 2023): Sorry to say it Pieter, but to me it doesn't work downgrading selenium - tried 4.12, 4,11, 4.10 - neither using the selenium base import SB method at the link. Here I've used the SB method and I have tor active, don't know what's wrong at this point. ![Screenshot 2023-10-06 alle 22 58 43](https://github.com/probberechts/soccerdata/assets/147209202/f14e8b98-9f0c-4b4c-800d-16ccd12d53da)

kerem commented

2026-03-02 15:55:35 +03:00

Author

Owner

@probberechts commented on GitHub (Oct 7, 2023):

@Gibranium Your issue seems related to #395. WhoScored can detect that you are a bot when you are in headless mode. You'll have to run in non-headless mode for now.

@probberechts commented on GitHub (Oct 7, 2023): @Gibranium Your issue seems related to #395. WhoScored can detect that you are a bot when you are in headless mode. You'll have to run in non-headless mode for now.

kerem commented

2026-03-02 15:55:35 +03:00

Author

Owner

@Gibranium commented on GitHub (Oct 7, 2023):

Ok, I've tried again with selenium 4.12, no headless, no cache and tor, but still got the same identical error.
Do I have to give up for now, and wait improvements from undetected or there's something else I can try?

@Gibranium commented on GitHub (Oct 7, 2023): Ok, I've tried again with selenium 4.12, no headless, no cache and tor, but still got the same identical error. Do I have to give up for now, and wait improvements from undetected or there's something else I can try?

kerem commented

2026-03-02 15:55:35 +03:00

Author

Owner

@mhd0528 commented on GitHub (Oct 7, 2023):

I have it worked with the same setting as yours. Are you setting everything up in a Linux system? Or maybe try the latest version of undetected-chromedriver (3.5.2 I think) as mentioned in another issue?

@mhd0528 commented on GitHub (Oct 7, 2023): I have it worked with the same setting as yours. Are you setting everything up in a Linux system? Or maybe try the latest version of `undetected-chromedriver` (3.5.2 I think) as mentioned in another issue?

kerem commented

2026-03-02 15:55:36 +03:00

Author

Owner

@Gibranium commented on GitHub (Oct 10, 2023):

@mhd0528 nope, I'm on Mac M2. I've tried with undetected on 3.5.2, but it requires Chrome on 114 and I don't think it's supported on Apple silicon. Maybe Apple silicon it's the problem? Because until May I used an Intel Mac and all worked perfectly.

@Gibranium commented on GitHub (Oct 10, 2023): @mhd0528 nope, I'm on Mac M2. I've tried with undetected on 3.5.2, but it requires Chrome on 114 and I don't think it's supported on Apple silicon. Maybe Apple silicon it's the problem? Because until May I used an Intel Mac and all worked perfectly.

kerem commented

2026-03-02 15:55:36 +03:00

Author

Owner

@mhd0528 commented on GitHub (Oct 11, 2023):

Yea, I think that's possible. I saw other issues about the version of Chrome can also lead to version problems.

@mhd0528 commented on GitHub (Oct 11, 2023): Yea, I think that's possible. I saw other issues about the version of Chrome can also lead to version problems.

kerem commented

2026-03-02 15:55:36 +03:00

Author

Owner

@marcjbaron commented on GitHub (Oct 14, 2023):

Sorry if this is not the place for it, but using the most recent release (#412) still gives the same issues using Selenium ( >=4.11) and undetected-chromedriver (3.5.3).

Running with headless==True gives the expected error message:

ValueError: Headless mode is not supported for Selenium 4.13.0 and above. Please downgrade to a lower version of Selenium or set 'headless=False'.

Running with headless==False gives a similar error message as a previous comment from Gibranium when running in headless mode; to be specific, running the command

ws = sd.WhoScored(leagues=["USA-MLS"], seasons = ["2023"], no_cache=True, headless=False, proxy='tor')

gives the following error:

After 5 attempts of this, the typical existing error is given:

FOLDER/.analytics-env/lib/python3.10/site-packages/soccerdata/_common.py", line 463, in _download_and_save raise ConnectionError("Could not download %s." % url)

ConnectionError: Could not download https://www.whoscored.com.

@marcjbaron commented on GitHub (Oct 14, 2023): Sorry if this is not the place for it, but using the most recent release ([#412](https://github.com/probberechts/soccerdata/pull/412)) still gives the same issues using Selenium ( >=4.11) and undetected-chromedriver (3.5.3). Running with `headless==True` gives the expected error message: `ValueError: Headless mode is not supported for Selenium 4.13.0 and above. Please downgrade to a lower version of Selenium or set 'headless=False'.` Running with `headless==False` gives a similar error message as [a previous comment from Gibranium](https://github.com/probberechts/soccerdata/issues/398#issuecomment-1751397366_) when running in headless mode; to be specific, running the command `ws = sd.WhoScored(leagues=["USA-MLS"], seasons = ["2023"], no_cache=True, headless=False, proxy='tor')` gives the following error: ![Screenshot from 2023-10-14 12-42-55](https://github.com/probberechts/soccerdata/assets/68666330/515b7c00-ddf1-472a-9692-771123aceb9f) After 5 attempts of this, the typical existing error is given: `FOLDER/.analytics-env/lib/python3.10/site-packages/soccerdata/_common.py", line 463, in _download_and_save raise ConnectionError("Could not download %s." % url)` ` ConnectionError: Could not download https://www.whoscored.com.`

kerem commented

2026-03-02 15:55:36 +03:00

Author

Owner

@probberechts commented on GitHub (Oct 14, 2023):

@marcjbaron There were two different issues discussed in this thread:

Selenium 4.13 no longer supports headless mode. As a consequence, running the whoscored scraper with "headless=True" resulted in a crash. This is fixed in the latest release of soccerdata.
The undetected-chromedriver library is used to patch selenium such that it does not trigger anti-bot services. Currently, this does not seem to work / sufficient for everyone (works fine for me though). The only thing I can do here is to wait until undetected-chromedriver gets updated. But one thing you could try is not using tor. Some IPs of tor nodes might have been blacklisted.

@probberechts commented on GitHub (Oct 14, 2023): @marcjbaron There were two different issues discussed in this thread: 1. Selenium 4.13 no longer supports headless mode. As a consequence, running the whoscored scraper with "headless=True" resulted in a crash. This is fixed in the latest release of soccerdata. 2. The undetected-chromedriver library is used to patch selenium such that it does not trigger anti-bot services. Currently, this does not seem to work / sufficient for everyone (works fine for me though). The only thing I can do here is to wait until undetected-chromedriver gets updated. But one thing you could try is not using tor. Some IPs of tor nodes might have been blacklisted.