[GH-ISSUE #690] Bug: Installing missing 3 recommended dependencies never completes

kerem commented

2026-03-14 23:00:29 +03:00

Owner

Originally created by @rxw1 on GitHub (Apr 3, 2021).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/690

$ npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"
npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated request-promise@4.2.6: request-promise has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated core-js@2.6.12: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
[                  ] / reify:single-file: timing reifyNode:node_modules/core-js Completed in 2177ms

npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" does never complete and hangs at the last line above.

1 archivebox bla
2 cd bla
3 npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"

ArchiveBox version

ArchiveBox v0.5.4
Cpython Linux Linux-5.11.10-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-glibc2.2.5 x86_64 (not in Docker)

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.5.4          valid     /home/asdf/.python/bin/archivebox
 √  PYTHON_BINARY         v3.8.8          valid     /usr/bin/python3.8
 √  DJANGO_BINARY         v3.1.3          valid     /home/asdf/.python/lib/python3.8/site-packages/django/bin/django-admin.py
 √  CURL_BINARY           v7.74.0         valid     /usr/bin/curl
 √  WGET_BINARY           v1.20.3         valid     /usr/bin/wget
 √  NODE_BINARY           v15.11.0        valid     /usr/bin/node
 X  SINGLEFILE_BINARY     ?               invalid   single-file
 X  READABILITY_BINARY    ?               invalid   readability-extractor
 X  MERCURY_BINARY        ?               invalid   mercury-parser
 √  GIT_BINARY            v2.26.3         valid     /usr/bin/git
 √  YOUTUBEDL_BINARY      v2021.04.01     valid     /home/asdf/.python/bin/youtube-dl
 √  CHROME_BINARY         v90.0.4430.51   valid     /usr/bin/google-chrome-beta
 √  RIPGREP_BINARY        v0.10.0         valid     /usr/local/bin/rg

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /home/asdf/.python/lib/python3.8/site-packages/archivebox
 √  TEMPLATES_DIR         3 files         valid     /home/asdf/.python/lib/python3.8/site-packages/archivebox/templates

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled
 -  COOKIES_FILE          -               disabled

[i] Data locations:
 √  OUTPUT_DIR            9 files         valid     /mnt/unsort/archive/www
 √  SOURCES_DIR           108 files       valid     ./sources
 √  LOGS_DIR              0 files         valid     ./logs
 √  ARCHIVE_DIR           87 files        valid     ./archive
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf
 √  SQL_INDEX             892.0 KB        valid     ./index.sqlite3

[!] Warning: Missing 3 recommended dependencies
    ! SINGLEFILE_BINARY: single-file (unable to detect version)
      Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"
            or archivebox config --set SAVE_SINGLEFILE=False to silence this warning

    ! READABILITY_BINARY: readability-extractor (unable to detect version)
      Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"
            or archivebox config --set SAVE_READABILITY=False to silence this warning

    ! MERCURY_BINARY: mercury-parser (unable to detect version)
      Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"
            or archivebox config --set SAVE_MERCURY=False to silence this warning

thx

Originally created by @rxw1 on GitHub (Apr 3, 2021). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/690 ``` $ npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142 npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated request-promise@4.2.6: request-promise has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142 npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart() npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated core-js@2.6.12: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3. [ ] / reify:single-file: timing reifyNode:node_modules/core-js Completed in 2177ms ``` `npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"` does never complete and hangs at the last line above. 1 `archivebox bla` 2 `cd bla` 3 `npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"` ![screenshot-20210403-203038](https://user-images.githubusercontent.com/154026/113488030-ccc21a00-94bb-11eb-8e1b-bb2997ff2915.png) #### ArchiveBox version ``` ArchiveBox v0.5.4 Cpython Linux Linux-5.11.10-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-glibc2.2.5 x86_64 (not in Docker) [i] Dependency versions: √ ARCHIVEBOX_BINARY v0.5.4 valid /home/asdf/.python/bin/archivebox √ PYTHON_BINARY v3.8.8 valid /usr/bin/python3.8 √ DJANGO_BINARY v3.1.3 valid /home/asdf/.python/lib/python3.8/site-packages/django/bin/django-admin.py √ CURL_BINARY v7.74.0 valid /usr/bin/curl √ WGET_BINARY v1.20.3 valid /usr/bin/wget √ NODE_BINARY v15.11.0 valid /usr/bin/node X SINGLEFILE_BINARY ? invalid single-file X READABILITY_BINARY ? invalid readability-extractor X MERCURY_BINARY ? invalid mercury-parser √ GIT_BINARY v2.26.3 valid /usr/bin/git √ YOUTUBEDL_BINARY v2021.04.01 valid /home/asdf/.python/bin/youtube-dl √ CHROME_BINARY v90.0.4430.51 valid /usr/bin/google-chrome-beta √ RIPGREP_BINARY v0.10.0 valid /usr/local/bin/rg [i] Source-code locations: √ PACKAGE_DIR 23 files valid /home/asdf/.python/lib/python3.8/site-packages/archivebox √ TEMPLATES_DIR 3 files valid /home/asdf/.python/lib/python3.8/site-packages/archivebox/templates [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled - COOKIES_FILE - disabled [i] Data locations: √ OUTPUT_DIR 9 files valid /mnt/unsort/archive/www √ SOURCES_DIR 108 files valid ./sources √ LOGS_DIR 0 files valid ./logs √ ARCHIVE_DIR 87 files valid ./archive √ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf √ SQL_INDEX 892.0 KB valid ./index.sqlite3 [!] Warning: Missing 3 recommended dependencies ! SINGLEFILE_BINARY: single-file (unable to detect version) Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" or archivebox config --set SAVE_SINGLEFILE=False to silence this warning ! READABILITY_BINARY: readability-extractor (unable to detect version) Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" or archivebox config --set SAVE_READABILITY=False to silence this warning ! MERCURY_BINARY: mercury-parser (unable to detect version) Hint: npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" or archivebox config --set SAVE_MERCURY=False to silence this warning ``` thx

kerem closed this issue

2026-03-14 23:00:34 +03:00

kerem commented

2026-03-14 23:00:40 +03:00

Author

Owner

@pirate commented on GitHub (Apr 3, 2021):

Strange, possibly a network issue, upstream dependency issue in one of the packages, or something hanging the npm audit process? OR maybe it's the new postinstall step I added recently that tries to run python3 -m pip install archivebox after the npm install completes if it doesn't detect the archivebox python package already installed.

npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git" is a handy shortcut for installing the 3 actual JS dependencies, can you try installing them individually instead to see which one is causing the hang?

npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git
npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git
npm install --no-save --no-audit --no-fund --loglevel=error @postlight/mercury-parser

@pirate commented on GitHub (Apr 3, 2021): Strange, possibly a network issue, upstream dependency issue in one of the packages, or something hanging the npm audit process? OR maybe it's the new `postinstall` step I added recently that tries to run `python3 -m pip install archivebox` after the npm install completes if it doesn't detect the archivebox python package already installed. `npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"` is a handy shortcut for installing the 3 actual JS dependencies, can you try installing them individually instead to see which one is causing the hang? ```bash npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git npm install --no-save --no-audit --no-fund --loglevel=error @postlight/mercury-parser ```

kerem commented

2026-03-14 23:00:45 +03:00

Author

Owner

@dunnno commented on GitHub (Apr 5, 2021):

That's the problem I face as well : https://github.com/ArchiveBox/ArchiveBox/discussions/687#discussioncomment-553225

I ran all the 3 commands but only the last one remain valid. ==> I ran the SingleFile install command, it gets installed, I ran the readability one, it gets installed, but now SingleFile becomes invalid.

@dunnno commented on GitHub (Apr 5, 2021): That's the problem I face as well : https://github.com/ArchiveBox/ArchiveBox/discussions/687#discussioncomment-553225 I ran all the 3 commands but only the last one remain valid. ==> I ran the SingleFile install command, it gets installed, I ran the readability one, it gets installed, but now SingleFile becomes invalid.

kerem commented

2026-03-14 23:00:50 +03:00

Author

Owner

@pirate commented on GitHub (Apr 5, 2021):

Delete any package.json or package-lock.json present in your data folder and re-run those commands. It's still trying to install the broken archivebox npm package after those commands because it saved it into a lockfile earlier.

@pirate commented on GitHub (Apr 5, 2021): Delete any package.json or package-lock.json present in your data folder and re-run those commands. It's still trying to install the broken archivebox npm package after those commands because it saved it into a lockfile earlier.

kerem commented

2026-03-14 23:00:56 +03:00

Author

Owner

@dunnno commented on GitHub (Apr 5, 2021):

I ran again the single file line in order to get it installed. Readability and mercury aren't installed.

Unless I look at the wrong place, I don't see any readability or mercury folders in the node_modules folder :/

@dunnno commented on GitHub (Apr 5, 2021): I ran again the single file line in order to get it installed. Readability and mercury aren't installed. Unless I look at the wrong place, I don't see any readability or mercury folders in the node_modules folder :/

kerem commented

2026-03-14 23:01:01 +03:00

Author

Owner

@pirate commented on GitHub (Apr 5, 2021):

Just to confirm, you tried running exactly these commands in this order?

rm -Rf package.json package-lock.json node_modules
npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git
npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git
npm install --no-save --no-audit --no-fund --loglevel=error @postlight/mercury-parser

And afterwards node_modules only contains one of them?

@pirate commented on GitHub (Apr 5, 2021): Just to confirm, you tried running exactly these commands in this order? ```bash rm -Rf package.json package-lock.json node_modules npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git npm install --no-save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git npm install --no-save --no-audit --no-fund --loglevel=error @postlight/mercury-parser ``` And afterwards `node_modules` only contains one of them?

kerem commented

2026-03-14 23:01:06 +03:00

Author

Owner

@dunnno commented on GitHub (Apr 5, 2021):

Yep, tried it again, singlefile installs fine, readability installs fine, but singlefile got uninstalled, and so on with readability when doing mercury

@dunnno commented on GitHub (Apr 5, 2021): Yep, tried it again, singlefile installs fine, readability installs fine, but singlefile got uninstalled, and so on with readability when doing mercury

kerem commented

2026-03-14 23:01:11 +03:00

Author

Owner

@pirate commented on GitHub (Apr 5, 2021):

Arg, it turns out this is a difference in npm behavior between mac and linux. On mac it will install all 3, on linux it will overwrite the previous node_modules each time. 😡

On linux you'll have to use --save instead of --no-save:

rm -Rf package.json package-lock.json node_modules
npm install --save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git
npm install --save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git
npm install --save --no-audit --no-fund --loglevel=error @postlight/mercury-parser

@pirate commented on GitHub (Apr 5, 2021): Arg, it turns out this is a difference in `npm` behavior between mac and linux. On mac it will install all 3, on linux it will overwrite the previous node_modules each time. 😡 <img width="400px" src="https://user-images.githubusercontent.com/511499/113625666-33b41000-962f-11eb-81f2-cf1ad261074a.png"> <img width="400px" src="https://user-images.githubusercontent.com/511499/113625716-47f80d00-962f-11eb-9894-9d97aa8cedfb.png"> On linux you'll have to use `--save` instead of `--no-save`: ```bash rm -Rf package.json package-lock.json node_modules npm install --save --no-audit --no-fund --loglevel=error git+https://github.com/gildas-lormeau/SingleFile.git npm install --save --no-audit --no-fund --loglevel=error git+https://github.com/ArchiveBox/readability-extractor.git npm install --save --no-audit --no-fund --loglevel=error @postlight/mercury-parser ``` <img width="400px" src="https://user-images.githubusercontent.com/511499/113626490-3e22d980-9630-11eb-8670-b40fcdfd81cb.png">

kerem commented

2026-03-14 23:01:16 +03:00

Author

Owner

@dunnno commented on GitHub (Apr 5, 2021):

Well, that was it, that did the trick, thanks a bunch :)

@dunnno commented on GitHub (Apr 5, 2021): Well, that was it, that did the trick, thanks a bunch :)

Rows
Columns

[GH-ISSUE #690] Bug: Installing missing 3 recommended dependencies never completes #3451

ArchiveBox version