[GH-ISSUE #581] Move all vendored code into git submodules to make upgrading easier #3384

New issue

Closed

opened 2026-03-14 22:34:59 +03:00 by kerem · 2 comments

kerem commented

2026-03-14 22:34:59 +03:00

Owner

Originally created by @pirate on GitHub (Dec 11, 2020).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/581

Currently in we have all vendored libraries as copy pasted python files under archivebox/vendor/*.py.

This makes it difficult to upgrade these libraries, instead we should move them into git submodules that are clones of their original sources on github.

E.g. instead of vendor/base32_crockford.py it should be vendor/base32_crockford/.git and vendor/base32_crockford/base32_crockford.py.

base32-crockford: https://github.com/jbittel/base32-crockford
pocket-api: https://github.com/tapanpandita/pocket
taggit: https://github.com/jazzband/django-taggit

In the future to upgrade these vendored libs it should be as simple as cd archivebox/vendor/base32_crockford; git pull and then committing the updated submodule commit to the main repo.

Originally created by @pirate on GitHub (Dec 11, 2020). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/581 Currently in we have all vendored libraries as copy pasted python files under `archivebox/vendor/*.py`. This makes it difficult to upgrade these libraries, instead we should move them into git submodules that are clones of their original sources on github. E.g. instead of `vendor/base32_crockford.py` it should be `vendor/base32_crockford/.git` and `vendor/base32_crockford/base32_crockford.py`. - base32-crockford: https://github.com/jbittel/base32-crockford - pocket-api: https://github.com/tapanpandita/pocket - taggit: https://github.com/jazzband/django-taggit In the future to upgrade these vendored libs it should be as simple as `cd archivebox/vendor/base32_crockford; git pull` and then committing the updated submodule commit to the main repo.

kerem

2026-03-14 22:34:59 +03:00

closed this issue
added the
touches: data/schema/architecture

status: done

size: easy
labels

kerem commented

2026-03-14 22:35:16 +03:00

Author

Owner

@mAAdhaTTah commented on GitHub (Dec 11, 2020):

I would also be open to just using requests directly instead of the Pocket API package. Didn't realize dealing with dependencies for deb packages was going to be such a chore. That said, the Pocket API package hasn't changed in a long time so may not need to be submoduled either (and all my experience w/ submodules has been terrible lol).

@mAAdhaTTah commented on GitHub (Dec 11, 2020): I would also be open to just using `requests` directly instead of the Pocket API package. Didn't realize dealing with dependencies for deb packages was going to be such a chore. That said, the Pocket API package hasn't changed in a long time so may not need to be submoduled either (and all my experience w/ submodules has been terrible lol).

kerem commented

2026-03-14 22:35:21 +03:00

Author

Owner

@pirate commented on GitHub (Dec 11, 2020):

Pocket is not the only vendored dependency, and we're likely to want others in the future. We already use submodules for many other parts of the ArchiveBox packaging process, so it's familiar to us (git pull --recurse-submodules fixes almost all submodules headaches btw). The main benefit is that updating a vendored dependency goes from being a manual copy-paste process to just cd vendor/<submodule>; git pull.

This article has a good explanation:
https://gist.github.com/datagrok/8577287
https://gist.github.com/datagrok/8577287#use-git-submodules

@pirate commented on GitHub (Dec 11, 2020): Pocket is not the only vendored dependency, and we're likely to want others in the future. We already use submodules for many other parts of the ArchiveBox packaging process, so it's familiar to us (`git pull --recurse-submodules` fixes almost all submodules headaches btw). The main benefit is that updating a vendored dependency goes from being a manual copy-paste process to just `cd vendor/<submodule>; git pull`. This article has a good explanation: https://gist.github.com/datagrok/8577287 https://gist.github.com/datagrok/8577287#use-git-submodules

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3384

No description provided.

Rows
Columns