[GH-ISSUE #643] Questions: Alternate title source, regex woes, updating single links, blacklisting updates, and piping lists to delete/remove links in Docker #400

New issue

Open

opened 2026-03-01 14:43:15 +03:00 by kerem · 2 comments

kerem commented

2026-03-01 14:43:15 +03:00

Owner

Originally created by @drpfenderson on GitHub (Feb 1, 2021).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/643

Y'all have been so amazingly helpful with bugs and errors, but every once and a while I find a tidbit that my perusing of the issues/documentation can't seem to meet. I'm sure I'm also overlooking some things. Each issue never seems big enough to ask about on their own, so I've been compiling a few of my queries from the past year into one post. These are definitely not feature requests, more "I think I'm missing something" or not understanding how a command works. Thanks for any clarification you can provide!

How to pull title/favicon from singlefile, or other source, instead of wget due to Cloudflare/blocking errors?

I'm having a recurring problem with Cloudflare/whatever blocking wget as a method, so it fails to pull the title of a link instead just giving me "Access denied | www.example.com used Cloudflare to restrict access". However, singlefile and other methods seem to pull the information and page just fine. Is there a way to force archivebox to pull that information from one of the other capture sources without just disabling all wget captures? Or possibly blacklisting WGET on certain urls?

Is something wrong with my regex for blacklisting, or is this a Docker issue? I'm passing the environmental variables through docker-compose.yml. This is my code:

        environment:
            - USE_COLOR=True
            - SHOW_PROGRESS=True
            - TIMEOUT=120
            - SAVE_MEDIA=False
            - SAVE_GIT=False
            - URL_BLACKLIST='.*.exe,.*.pinboard.in/.*'

I usually pipe my list to it with archivebox add < ~/urls.txt, as well as attempted with archivebox add --depth=1 https://pinboard.in/u:username, but it still adds the blacklisted url to the archive:

[+] [2020-09-30 17:26:33] "feed.pinboard.in/rss"
    https://feed.pinboard.in/rss
    > ./archive/1601486785.704537
      > title
      > favicon
      > wget

I've tried a couple variations that I have validated (using regex101), but either run into ERROR: Invalid interpolation format for "environment" option in service error, or it just ignores it and adds the urls from that domain anyways. Another one that validates, but fails to prevent the url from being archived: URL_BLACKLIST='http(s)?:\/\/(.+)?(pinboard\.in)\/.*' Is this a bug that I should report, a docker-compose limitation, or something wrong with my regex? I would love to just point this at my private pinboard feed with a cron task, without picking up those extra links to RSS spec or pinboard homepage.

Is there a way to update a specific, single link from the command-line? (would that be utilizing the web-interface?) There are some 404s or missing pages sprinkled through my archive, some of them temporary, and I would love to update just those specific links without hitting all errored/unfinished links.
I guess that leads to the question, is it possible to set a page to not be checked/rearchived, even if it has an error? I have certain older links that have partial archives or something, so I don't want to remove them, but the links don't exist on the web anymore and will just indefinitely 404.
Is it possible to pipe list of urls to remove/delete from docker? I've tried using various stdout methods, like < or cat, with no luck. When it gets to "Do you want to process with removing these? y/n", it just pops back over to the command-line without letting me choose. I've tried using the program yes as well, but no dice.

Originally created by @drpfenderson on GitHub (Feb 1, 2021). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/643 Y'all have been so amazingly helpful with bugs and errors, but every once and a while I find a tidbit that my perusing of the issues/documentation can't seem to meet. I'm sure I'm also overlooking some things. Each issue never seems big enough to ask about on their own, so I've been compiling a few of my queries from the past year into one post. These are definitely not feature requests, more "I think I'm missing something" or not understanding how a command works. Thanks for any clarification you can provide! 1) How to pull title/favicon from singlefile, or other source, instead of wget due to Cloudflare/blocking errors? I'm having a recurring problem with Cloudflare/whatever blocking wget as a method, so it fails to pull the title of a link instead just giving me "Access denied | www.example.com used Cloudflare to restrict access". However, singlefile and other methods seem to pull the information and page just fine. Is there a way to force archivebox to pull that information from one of the other capture sources without just disabling _all_ wget captures? Or possibly blacklisting WGET on certain urls? 2) Is something wrong with my regex for blacklisting, or is this a Docker issue? I'm passing the environmental variables through `docker-compose.yml`. This is my code: ``` environment: - USE_COLOR=True - SHOW_PROGRESS=True - TIMEOUT=120 - SAVE_MEDIA=False - SAVE_GIT=False - URL_BLACKLIST='.*.exe,.*.pinboard.in/.*' ``` I usually pipe my list to it with `archivebox add < ~/urls.txt`, as well as attempted with `archivebox add --depth=1 https://pinboard.in/u:username`, but it still adds the blacklisted url to the archive: ``` [+] [2020-09-30 17:26:33] "feed.pinboard.in/rss" https://feed.pinboard.in/rss > ./archive/1601486785.704537 > title > favicon > wget ``` I've tried a couple variations that I have validated (using regex101), but either run into `ERROR: Invalid interpolation format for "environment" option in service` error, or it just ignores it and adds the urls from that domain anyways. Another one that validates, but fails to prevent the url from being archived: `URL_BLACKLIST='http(s)?:\/\/(.+)?(pinboard\.in)\/.*'` Is this a bug that I should report, a docker-compose limitation, or something wrong with my regex? I would love to just point this at my private pinboard feed with a cron task, without picking up those extra links to RSS spec or pinboard homepage. 3) Is there a way to update a specific, single link from the command-line? (would that be utilizing the web-interface?) There are some 404s or missing pages sprinkled through my archive, some of them temporary, and I would love to update just those specific links without hitting all errored/unfinished links. 4) I guess that leads to the question, is it possible to set a page to not be checked/rearchived, even if it has an error? I have certain older links that have partial archives or something, so I don't want to remove them, but the links don't exist on the web anymore and will just indefinitely 404. 5) Is it possible to pipe list of urls to remove/delete from docker? I've tried using various `stdout` methods, like `<` or `cat`, with no luck. When it gets to "Do you want to process with removing these? y/n", it just pops back over to the command-line without letting me choose. I've tried using the program `yes` as well, but no dice.

kerem commented

2026-03-01 14:43:16 +03:00

Author

Owner

@pirate commented on GitHub (Feb 2, 2021):

How to pull title/favicon from singlefile, or other source, instead of wget due to Cloudflare/blocking errors?

The title is actually fetched using requests.get, not with wget, but I can see why you want to switch it to Singlefile. If we switch over to playwright and get the headless browser scripting side of things worked out, title will be one of the first extractors we'll switch over to JS. That probably wont be for at least 4+ months though.

Is something wrong with my regex for blacklisting, or is this a Docker issue?

You can test your regex works for ArchiveBox by using the python3 shell:

$ python3
>>> import re
>>> URL_BLACKLIST = r'http(s)?:\/\/(.+)?(youtube\.com)|(amazon\.com)\/.*'  # replace this with your regex to test
>>> test_url = 'https://test.youtube.com/example.php?abc=123'
>>> bool(re.compile(URL_BLACKLIST, re.IGNORECASE).match(test_url))

If it works in the shell but not in archivebox, then it's a Docker/env escaping issue, try setting it with docker-compose run archivebox config --set URL_BLACKLIST='...' instead of using environment variables.

Is there a way to update a specific, single link from the command-line? (would that be utilizing the web-interface?) There are some 404s or missing pages sprinkled through my archive, some of them temporary, and I would love to update just those specific links without hitting all errored/unfinished links.

archivebox update 'https://example.com/some/url'
# or to update + overwrite any existing snapshot with the latest
archivebox add --overwrite 'https://example.com/some/url'

The Archive button in the UI is also equivalent to running archivebox update on it

is it possible to set a page to not be checked/rearchived, even if it has an error?

Not yet, follow these:

Is it possible to pipe list of urls to remove/delete from docker?

docker-compose run archivebox remove --yes --delete < urls.txt

# to see more usage + examples:
docker-compose run archivebox remove --help

@pirate commented on GitHub (Feb 2, 2021): > 1. How to pull title/favicon from singlefile, or other source, instead of wget due to Cloudflare/blocking errors? The title is actually fetched using `requests.get`, not with wget, but I can see why you want to switch it to Singlefile. If we switch over to playwright and get the headless browser scripting side of things worked out, `title` will be one of the first extractors we'll switch over to JS. That probably wont be for at least 4+ months though. > 2. Is something wrong with my regex for blacklisting, or is this a Docker issue? You can test your regex works for ArchiveBox by using the python3 shell: ```python $ python3 >>> import re >>> URL_BLACKLIST = r'http(s)?:\/\/(.+)?(youtube\.com)|(amazon\.com)\/.*' # replace this with your regex to test >>> test_url = 'https://test.youtube.com/example.php?abc=123' >>> bool(re.compile(URL_BLACKLIST, re.IGNORECASE).match(test_url)) ``` If it works in the shell but not in archivebox, then it's a Docker/env escaping issue, try setting it with `docker-compose run archivebox config --set URL_BLACKLIST='...'` instead of using environment variables. > 3. Is there a way to update a specific, single link from the command-line? (would that be utilizing the web-interface?) There are some 404s or missing pages sprinkled through my archive, some of them temporary, and I would love to update just those specific links without hitting all errored/unfinished links. ```bash archivebox update 'https://example.com/some/url' # or to update + overwrite any existing snapshot with the latest archivebox add --overwrite 'https://example.com/some/url' ``` The `Archive` button in the UI is also equivalent to running `archivebox update` on it ![image](https://user-images.githubusercontent.com/511499/106545851-1c925e00-64d8-11eb-9abb-b9fbee72eaaf.png) > 4. is it possible to set a page to not be checked/rearchived, even if it has an error? Not yet, follow these: - https://github.com/ArchiveBox/ArchiveBox/issues/109 - https://github.com/ArchiveBox/ArchiveBox/issues/566 > 5. Is it possible to pipe list of urls to remove/delete from docker? ```bash docker-compose run archivebox remove --yes --delete < urls.txt # to see more usage + examples: docker-compose run archivebox remove --help ```

kerem commented

2026-03-01 14:43:16 +03:00

Author

Owner

@drpfenderson commented on GitHub (Feb 2, 2021):

Good to know! Now that I've discovered the new title-editing feature in the web GUI, I can fix the very few that have this issue. Wonderful feature. That, and the tag editor, are saving my brain with these last handful of older links I have as holdover from previous version. Also, some more recent change fixes this issue, it seems, as I added a number of new links from sites that errored-out on previous versions. They correctly pull the title now.
The regex matched properly using your method of testing, and after a number of different tests on my server - turns out that adding it to the docker-compose.yml file isn't sufficient. I set it in config by passing it like you recommended, it wrote it to my .conf file, and that seemed to work! Though I did have to put quotes around the env variable, like docker-compose run archivebox config --set "URL_BLACKLIST='...'", otherwise it threw an error.
Amazing! I had tried this with the browser in the past, but for some reason it ignored my SAVE_GIT=False and other env variables when running from the webui. It respects those now, so either of these methods is perfect. Fantastic!
Followed. Though the fact that I can manually "fix/update" single links now means that I won't run into this very often, hopefully.
Aha! That --yes tag is precisely what I was missing. Worked!

Thanks so much for the time/energy you've taken to help me. Y'all are the best.

@drpfenderson commented on GitHub (Feb 2, 2021): 1. Good to know! Now that I've discovered the new title-editing feature in the web GUI, I can fix the very few that have this issue. Wonderful feature. That, and the tag editor, are saving my brain with these last handful of older links I have as holdover from previous version. Also, some more recent change fixes this issue, it seems, as I added a number of new links from sites that errored-out on previous versions. They correctly pull the title now. 2. The regex matched properly using your method of testing, and after a number of different tests on my server - turns out that adding it to the `docker-compose.yml` file isn't sufficient. I set it in `config` by passing it like you recommended, it wrote it to my `.conf` file, and that seemed to work! Though I did have to put quotes around the env variable, like `docker-compose run archivebox config --set "URL_BLACKLIST='...'"`, otherwise it threw an error. 3. Amazing! I had tried this with the browser in the past, but for some reason it ignored my `SAVE_GIT=False` and other env variables when running from the webui. It respects those now, so either of these methods is perfect. Fantastic! 4. Followed. Though the fact that I can manually "fix/update" single links now means that I won't run into this very often, hopefully. 5. Aha! That `--yes` tag is precisely what I was missing. Worked! Thanks so much for the time/energy you've taken to help me. Y'all are the best.