starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 09:06:02 +03:00

[GH-ISSUE #679] Feature Request: subdivide archive/ directory into subdirectories #3449

New issue

Open

opened 2026-03-14 22:56:45 +03:00 by kerem · 2 comments

kerem commented

2026-03-14 22:56:45 +03:00

Owner

Originally created by @karlicoss on GitHub (Mar 27, 2021).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/679

Type

Request modification of existing behavior or design

What is the problem that your feature request solves

I've tried to do some processing on my archive which had 100K+ items and it was really slow.

To be fair, it was also on HDD, not SSD (although I'd imagine it's also natural to store something like web archives on HDDs)!
However I believe another potential problem could be the large number of entries in archive/ directory. Currently the disk layout is such that you get a archive/<timestamp> directory for each new link. This is often problematic on various filesystem, starting from being super slow on some, and ending with not working at all.

You can read this stackexchange answer for more info.

Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes

It might be somewhat relevant to https://github.com/ArchiveBox/ArchiveBox/issues/74 -- if we used hashes then could adopt a similar format that .git/objects dir uses! Wonder what's the status of this?

How badly do you want this new feature?

It would be nice to have eventually

I'm willing to contribute dev time / money to fix this issue
I like ArchiveBox so far / would recommend it to a friend

Originally created by @karlicoss on GitHub (Mar 27, 2021). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/679 ## Type - [x] Request modification of existing behavior or design ## What is the problem that your feature request solves I've tried to do some processing on my archive which had 100K+ items and it was really slow. To be fair, it was also on HDD, not SSD (although I'd imagine it's also natural to store something like web archives on HDDs)! However I believe another potential problem could be the large number of entries in `archive/` directory. Currently the disk layout is such that you get a `archive/<timestamp>` directory for each new link. This is often problematic on various filesystem, starting from being super slow on some, and ending with not working at all. You can read [this](https://softwareengineering.stackexchange.com/questions/301400/why-is-the-git-git-objects-folder-subdivided-in-many-sha-prefix-folders) stackexchange answer for more info. ## Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes It might be somewhat relevant to https://github.com/ArchiveBox/ArchiveBox/issues/74 -- if we used hashes then could adopt a similar format that `.git/objects` dir uses! Wonder what's the status of this? ## How badly do you want this new feature? - [x] It would be nice to have eventually --- - [x] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue - [x] I like ArchiveBox so far / would recommend it to a friend

kerem added the

size: hard

status: idea-phase

why: performance

labels

2026-03-14 22:56:45 +03:00

kerem commented

2026-03-14 22:56:56 +03:00

Author

Owner

@pirate commented on GitHub (Mar 29, 2021):

Have you tried the v0.6/dev branch out of curiosity? It doesn't solve the directory entries problem in particular, but it should massively improve performance in a lot of areas by reducing disk reads where unecessary.

@pirate commented on GitHub (Mar 29, 2021): Have you tried the v0.6/dev branch out of curiosity? It doesn't solve the directory entries problem in particular, but it should massively improve performance in a lot of areas by reducing disk reads where unecessary.

kerem commented

2026-03-14 22:57:01 +03:00

Author

Owner

@karlicoss commented on GitHub (Mar 29, 2021):

Not yet!
I guess I was 'processing' it in shell, so in that sense the performance of archivebox itself wouldn't matter here.

@karlicoss commented on GitHub (Mar 29, 2021): Not yet! I guess I was 'processing' it in shell, so in that sense the performance of archivebox itself wouldn't matter here.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3449

No description provided.

Rows
Columns