[GH-ISSUE #175] Dismiss cookie permission, GDPR, and subscription popups before archiving

kerem commented

2026-03-01 14:40:47 +03:00

Owner

Originally created by @pirate on GitHub (Mar 14, 2019).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/175

Might be able to graft some code from: https://www.i-dont-care-about-cookies.eu for hiding modals and overlays.

Also:
https://greasyfork.org/de/scripts/34815-anti-adblock-killer-reek-

Originally created by @pirate on GitHub (Mar 14, 2019). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/175 Might be able to graft some code from: https://www.i-dont-care-about-cookies.eu for hiding modals and overlays. Also: https://greasyfork.org/de/scripts/34815-anti-adblock-killer-reek-

kerem

2026-03-01 14:40:47 +03:00

closed this issue
added the
size: hard

touches: data/schema/architecture

expected: next release
labels

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@pirate commented on GitHub (Mar 15, 2019):

This is blocked by:

#177 (we need pyppeteer to run user scripts to dismiss popups)
#51 support for user scripted puppeteer flows

@pirate commented on GitHub (Mar 15, 2019): This is blocked by: - #177 (we need pyppeteer to run user scripts to dismiss popups) - #51 support for user scripted puppeteer flows

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@pirate commented on GitHub (Nov 18, 2022):

Useful scripts for evading bot detection in the future: https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection

@pirate commented on GitHub (Nov 18, 2022): Useful scripts for evading bot detection in the future: https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@datoslabs commented on GitHub (Jan 7, 2025):

Hi,

I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.

Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!

The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:

In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.

@datoslabs commented on GitHub (Jan 7, 2025): Hi, I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue. Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up! The test I choose was [artnet.com](https://www.artnet.com/), it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio: ![Image](https://github.com/user-attachments/assets/817dc0af-0d44-4f69-a1f4-637dcb7c8bde) ![Image](https://github.com/user-attachments/assets/ec7bc1a3-793b-48f2-b60d-05050f3964f1) In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, [https://github.com/browser-use/browser-use](https://github.com/browser-use/browser-use) have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@GlassedSilver commented on GitHub (Jan 21, 2025):

Hi,

I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.

Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!

The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:

In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.

This has major privacy implications, maybe optionally this could be fine, but I think a local AI model would suit this purpose better if you want to involve AI at all.

Personally I use Consent-o-Matic in my browsers and it not only auto okays all cookies, it even accepts or denies based on what the site lets you control and what you desire. For example allowing functional cookies for things like embeds but keeping trackers out.

It's truly set and forget, rarely misses and could be a fantastic consideration to source from. It even clicks your choices in a hidden manner.

https://youtu.be/LtgyVYwjlGE

@GlassedSilver commented on GitHub (Jan 21, 2025): > Hi, > > I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue. > > Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up! > > The test I choose was [artnet.com](https://www.artnet.com/), it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio: > > ![Image](https://github.com/user-attachments/assets/817dc0af-0d44-4f69-a1f4-637dcb7c8bde) > > ![Image](https://github.com/user-attachments/assets/ec7bc1a3-793b-48f2-b60d-05050f3964f1) > > In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control. This has major privacy implications, maybe optionally this could be fine, but I think a local AI model would suit this purpose better if you want to involve AI at all. Personally I use Consent-o-Matic in my browsers and it not only auto okays all cookies, it even accepts or denies based on what the site lets you control and what you desire. For example allowing functional cookies for things like embeds but keeping trackers out. It's truly set and forget, rarely misses and could be a fantastic consideration to source from. It even clicks your choices in a hidden manner. https://youtu.be/LtgyVYwjlGE

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@pirate commented on GitHub (Jan 21, 2025):

Yes for paid clients I'm already using a mixture of local LLMs and hosted models to detect and solve CAPTCHAs, hide cookie popups, and do QA analysis of captures to determine if there's any obsctructions to seeing the content.

None of it is open source right now because it's the only thing I can charge for to sustain the rest of the project.

@pirate commented on GitHub (Jan 21, 2025): Yes for [paid clients](https://docs.monadical.com/s/archivebox-consulting-services) I'm already using a mixture of local LLMs and hosted models to detect and solve CAPTCHAs, hide cookie popups, and do QA analysis of captures to determine if there's any obsctructions to seeing the content. None of it is open source right now because it's the only thing I can charge for to sustain the rest of the project.

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@pirate commented on GitHub (Dec 29, 2025):

ok this is implemented on dev using archivebox/plugins/modalcloser and archivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.

@pirate commented on GitHub (Dec 29, 2025): ok this is implemented on `dev` using `archivebox/plugins/modalcloser` and `archivebox/plugins/istilldontcareaboutcookies`, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@GlassedSilver commented on GitHub (Jan 7, 2026):

ok this is implemented on dev using archivebox/plugins/modalcloser and archivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.

This is great news and makes me very happy to see!
Very thankful for this, just one note that if anyone else extends on this in the future or maybe you yourself might I suggest using Consent-o-Matic?

@GlassedSilver commented on GitHub (Jan 7, 2026): > ok this is implemented on `dev` using `archivebox/plugins/modalcloser` and `archivebox/plugins/istilldontcareaboutcookies`, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed. This is great news and makes me very happy to see! Very thankful for this, just one note that if anyone else extends on this in the future or maybe you yourself might I suggest using [Consent-o-Matic](https://consentomatic.au.dk/)?

kerem commented

2026-03-01 14:40:48 +03:00

Author

Owner

@pirate commented on GitHub (Jan 27, 2026):

I have tested both extensively and unfortunately consent-o-matic breaks a lot of browser automations and makes the headless browsers flakyer in general (e.g. sometimes it'll open a new features/config page on first browser launch that steals foreground tab focus away from the page we're trying to archive). I've stuck with istilldontcareaboutcookies for now but if there is enough demand we could add content-o-matic as an opt-in choice as well.

@pirate commented on GitHub (Jan 27, 2026): I have tested both extensively and unfortunately consent-o-matic breaks a lot of browser automations and makes the headless browsers flakyer in general (e.g. sometimes it'll open a new features/config page on first browser launch that steals foreground tab focus away from the page we're trying to archive). I've stuck with `istilldontcareaboutcookies` for now but if there is enough demand we could add content-o-matic as an opt-in choice as well.

kerem referenced this issue

2026-03-01 17:52:00 +03:00

[GH-ISSUE #122] DOM Dump of the HTML of Pocket format page #1591

kerem referenced this issue

2026-03-14 21:02:34 +03:00

[GH-ISSUE #122] DOM Dump of the HTML of Pocket format page #3100

Rows
Columns

[GH-ISSUE #175] Dismiss cookie permission, GDPR, and subscription popups before archiving #122