[GH-ISSUE #290] Bug: FileExistsError when archiving snapshot with PyPI prerelease on NixOS

kerem commented

2026-03-01 14:41:32 +03:00

Owner

Originally created by @gjabell on GitHub (Oct 26, 2019).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/290

Describe the bug

On a brand new install / initialization, running archivebox add fails because the static directory already exists.

Steps to reproduce

Run archivebox add https://singleservingchef.com/chicken-and-mushroom-skillet-recipe
Archiving fails after the Writing links to main index

Screenshots or log output

Full log:

[*] [2019-10-25 22:01:17] Downloading https://singleservingchef.com/chicken-and-mushroom-skillet-recipe
    > ./sources/singleservingchef.com-1572040877.txt

[*] [2019-10-25 22:01:20] Parsing new links from output/sources/singleservingchef.com-1572040877.txt...
    > Parsed 100 links as Plain Text (80 new links added)

[*] [2019-10-25 22:01:20] Writing 80 links to main index...
    √ /home/gjabell/sync/archive/index.sqlite3
    √ /home/gjabell/sync/archive/index.json
Traceback (most recent call last):
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/bin/..archivebox-wrapped-wrapped", line 11, in <module>
    sys.exit(main())
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/__main__.py", line 10, in main
    archivebox.main(args=sys.argv[1:], stdin=sys.stdin)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/archivebox.py", line 58, in main
    pwd=pwd or OUTPUT_DIR,
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/__init__.py", line 55, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/archivebox_add.py", line 55, in main
    out_dir=pwd or OUTPUT_DIR,
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function
    return func(*args, **kwargs)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/main.py", line 509, in add
    write_main_index(links=all_links, out_dir=out_dir)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function
    return func(*args, **kwargs)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/index/__init__.py", line 239, in write_main_index
    write_html_main_index(links, out_dir=out_dir, finished=finished)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function
    return func(*args, **kwargs)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/index/html.py", line 58, in write_html_main_index
    copy_and_overwrite(join(TEMPLATES_DIR, STATIC_DIR_NAME), join(out_dir, STATIC_DIR_NAME))
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function
    return func(*args, **kwargs)
  File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/system.py", line 104, in copy_and_overwrite
    shutil.copytree(from_path, to_path)
  File "/nix/store/dxk1vqyhfmjk2ivxcrfw85k8kqsrzhhg-python3-3.7.4/lib/python3.7/shutil.py", line 324, in copytree
    os.makedirs(dst)
  File "/nix/store/dxk1vqyhfmjk2ivxcrfw85k8kqsrzhhg-python3-3.7.4/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/gjabell/sync/archive/static'

Software versions

OS: NixOS 19.09
ArchiveBox version: v0.4.2 (downloaded from Pypi)
Python version: v3.7
Chrome version: v77.0.3865.120

Originally created by @gjabell on GitHub (Oct 26, 2019). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/290  #### Describe the bug  On a brand new install / initialization, running `archivebox add` fails because the static directory already exists. #### Steps to reproduce  1. Run `archivebox add https://singleservingchef.com/chicken-and-mushroom-skillet-recipe` 2. Archiving fails after the `Writing links to main index` #### Screenshots or log output  Full log: ``` [*] [2019-10-25 22:01:17] Downloading https://singleservingchef.com/chicken-and-mushroom-skillet-recipe > ./sources/singleservingchef.com-1572040877.txt [*] [2019-10-25 22:01:20] Parsing new links from output/sources/singleservingchef.com-1572040877.txt... > Parsed 100 links as Plain Text (80 new links added) [*] [2019-10-25 22:01:20] Writing 80 links to main index... √ /home/gjabell/sync/archive/index.sqlite3 √ /home/gjabell/sync/archive/index.json Traceback (most recent call last): File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/bin/..archivebox-wrapped-wrapped", line 11, in <module> sys.exit(main()) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/__main__.py", line 10, in main archivebox.main(args=sys.argv[1:], stdin=sys.stdin) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/archivebox.py", line 58, in main pwd=pwd or OUTPUT_DIR, File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/__init__.py", line 55, in run_subcommand module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/cli/archivebox_add.py", line 55, in main out_dir=pwd or OUTPUT_DIR, File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function return func(*args, **kwargs) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/main.py", line 509, in add write_main_index(links=all_links, out_dir=out_dir) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function return func(*args, **kwargs) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/index/__init__.py", line 239, in write_main_index write_html_main_index(links, out_dir=out_dir, finished=finished) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function return func(*args, **kwargs) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/index/html.py", line 58, in write_html_main_index copy_and_overwrite(join(TEMPLATES_DIR, STATIC_DIR_NAME), join(out_dir, STATIC_DIR_NAME)) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/util.py", line 104, in typechecked_function return func(*args, **kwargs) File "/nix/store/8zja7yh31hrs1097f5c1s62mdwq8ncll-archivebox-0.4.2/lib/python3.7/site-packages/archivebox/system.py", line 104, in copy_and_overwrite shutil.copytree(from_path, to_path) File "/nix/store/dxk1vqyhfmjk2ivxcrfw85k8kqsrzhhg-python3-3.7.4/lib/python3.7/shutil.py", line 324, in copytree os.makedirs(dst) File "/nix/store/dxk1vqyhfmjk2ivxcrfw85k8kqsrzhhg-python3-3.7.4/lib/python3.7/os.py", line 221, in makedirs mkdir(name, mode) FileExistsError: [Errno 17] File exists: '/home/gjabell/sync/archive/static' ``` #### Software versions - OS: NixOS 19.09 - ArchiveBox version: v0.4.2 (downloaded from Pypi) - Python version: v3.7 - Chrome version: v77.0.3865.120

kerem closed this issue

2026-03-01 14:41:32 +03:00

kerem commented

2026-03-01 14:41:33 +03:00

Author

Owner

@pirate commented on GitHub (Oct 28, 2019):

The pypi version is not ready for public use yet, you're in uncharted territory there 😉

If you want to keep living on the cutting edge, can you try either the v0.4 branch or the v0.4.3 branch. (git clone ...; git checkout v0.4.3; pip install -e .)

@pirate commented on GitHub (Oct 28, 2019): The pypi version is not ready for public use yet, you're in uncharted territory there 😉 If you want to keep living on the cutting edge, can you try either the `v0.4` branch or the `v0.4.3` branch. (`git clone ...; git checkout v0.4.3; pip install -e .`)

kerem commented

2026-03-01 14:41:33 +03:00

Author

Owner

@gjabell commented on GitHub (Oct 28, 2019):

All good, I switched to the v0.4.3 branch but it has the same issue. Looks to be an issue with the static folder having read-only permissions, so the shutil.rmtree call in system.py line 103 silently fails, and then the shutil.copytree right after fails because the destination must not exist.

Manually changing the permissions on the static folder to allow writing lets ArchiveBox get past this error, although it then fails on something else (don't think it's related) :P

@gjabell commented on GitHub (Oct 28, 2019): All good, I switched to the v0.4.3 branch but it has the same issue. Looks to be an issue with the `static` folder having read-only permissions, so the `shutil.rmtree` call in `system.py` line 103 silently fails, and then the `shutil.copytree` right after fails because the destination must not exist. Manually changing the permissions on the static folder to allow writing lets ArchiveBox get past this error, although it then fails on something else (don't think it's related) :P

kerem commented

2026-03-01 14:41:33 +03:00

Author

Owner

@cdvv7788 commented on GitHub (Jul 17, 2020):

@gjabell can you please check if you still have this issue when running the django branch? I tested and it worked as expected in this branch:

archivebox init
archivebox add https://singleservingchef.com/chicken-and-mushroom-skillet-recipe

@cdvv7788 commented on GitHub (Jul 17, 2020): @gjabell can you please check if you still have this issue when running the `django` branch? I tested and it worked as expected in this branch: ``` archivebox init archivebox add https://singleservingchef.com/chicken-and-mushroom-skillet-recipe ```

kerem commented

2026-03-01 14:41:33 +03:00

Author

Owner

@gjabell commented on GitHub (Jul 17, 2020):

This seems to be fixed now, thanks for the update!

@gjabell commented on GitHub (Jul 17, 2020): This seems to be fixed now, thanks for the update!

kerem referenced this issue

2026-03-01 14:48:42 +03:00

[PR #438] [CLOSED] feat: Check page responsiveness before trying to archive it #1167

kerem referenced this issue

2026-03-01 17:52:32 +03:00