[PR #542] [CLOSED] Settings refactor using dynaconf #2724

New issue

Closed

opened 2026-03-01 18:00:34 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 18:00:34 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/ArchiveBox/ArchiveBox/pull/542
Author: @cdvv7788
Created: 11/19/2020
Status: ❌ Closed

Base: master ← Head: dynaconf

📝 Commits (3)

f485f5d feat: install dynaconf in the project
68689a5 feat: Move wget to a module with its own config
41b23f6 feat: Very rough draft of settings re-structuring for the wget extractor

📊 Changes

7 files changed (+69 additions, -37 deletions)

View changed files

📝 .gitignore (+2 -0)
📝 archivebox/config.py (+8 -0)
📝 archivebox/extractors/wget/__init__.py (+16 -37)
➕ archivebox/extractors/wget/config.py (+31 -0)
➕ archivebox/extractors/wget/settings.toml (+1 -0)
➕ archivebox/settings.toml (+10 -0)
📝 setup.py (+1 -0)

📄 Description

Summary

Replace the current config structure with a tiered architecture to ease readability and composability of the configurations around the application.
In this case there are several options:

Load the main configuration and override values that are found in the local configuration
Load the main configuration and use it as the basis to calculate values in the local configuration

I went with the second option, because it is less traumatic at this point. There are a lot of values that are calculated at runtime, so it is not as easy as dropping some config files and get done with it. If we are still going to require some logic to calculate some args, at least now we can keep them contained in the config files (mostly). For example, to calculate the version of some binaries, that is done at the upper level. We cannot just inherit it as a plain setting. We could move all of the configurations that require these calculations down to the extractors, but that would make this refactor way bigger.

So far, the wget extractor seems to work with this new structure. It is now on its own module with its own settings, and the logic dedicated to generate the flags to pass to the command is in the configuration script now. The actual extractor looks simpler now. There are some hardcoded values that will need to be replaced if we decide we want to keep going this way.

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Snapshot data layout on disk

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/ArchiveBox/ArchiveBox/pull/542 **Author:** [@cdvv7788](https://github.com/cdvv7788) **Created:** 11/19/2020 **Status:** ❌ Closed **Base:** `master` ← **Head:** `dynaconf` --- ### 📝 Commits (3) - [`f485f5d`](https://github.com/ArchiveBox/ArchiveBox/commit/f485f5dc901e0e99c52cba5f4328c0da57c51f44) feat: install dynaconf in the project - [`68689a5`](https://github.com/ArchiveBox/ArchiveBox/commit/68689a5ec28a017632ff4c15a4033afabf3b5c5a) feat: Move wget to a module with its own config - [`41b23f6`](https://github.com/ArchiveBox/ArchiveBox/commit/41b23f6b9198d23c6dc7f552685686c1f0964165) feat: Very rough draft of settings re-structuring for the wget extractor ### 📊 Changes **7 files changed** (+69 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+2 -0) 📝 `archivebox/config.py` (+8 -0) 📝 `archivebox/extractors/wget/__init__.py` (+16 -37) ➕ `archivebox/extractors/wget/config.py` (+31 -0) ➕ `archivebox/extractors/wget/settings.toml` (+1 -0) ➕ `archivebox/settings.toml` (+10 -0) 📝 `setup.py` (+1 -0) </details> ### 📄 Description # Summary Replace the current config structure with a tiered architecture to ease readability and composability of the configurations around the application. In this case there are several options: - Load the main configuration and override values that are found in the local configuration - Load the main configuration and use it as the basis to calculate values in the local configuration I went with the second option, because it is less traumatic at this point. There are a lot of values that are calculated at runtime, so it is not as easy as dropping some config files and get done with it. If we are still going to require some logic to calculate some `args`, at least now we can keep them contained in the config files (mostly). For example, to calculate the version of some binaries, that is done at the upper level. We cannot just inherit it as a plain setting. We could move all of the configurations that require these calculations down to the extractors, but that would make this refactor way bigger. So far, the `wget` extractor seems to work with this new structure. It is now on its own module with its own settings, and the logic dedicated to generate the flags to pass to the command is in the configuration script now. The actual extractor looks simpler now. There are some hardcoded values that will need to be replaced if we decide we want to keep going this way. # Changes these areas - [ ] Bugfixes - [ ] Feature behavior - [ ] Command line interface - [X] Configuration options - [X] Internal architecture - [ ] Snapshot data layout on disk --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-01 18:00:34 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#2724

No description provided.

Rows
Columns