[GH-ISSUE #1126] Discussion: Sponsoring the REST API development + applying for grants to fund ArchiveBox development

kerem commented

2026-03-01 14:45:40 +03:00

Owner

Originally created by @djkemmet on GitHub (Mar 22, 2023).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1126

I would like to sponsor (pay a developer cash monies in USD) for the development of the API for ArchiveBox. I need this for my digital preservation and ML/NLP work and I suspect a lot of people will end up doing a lot of cool things with it. I would like to talk to @pirate and get his blessing and solicit feedback from the community on how we could work together to accomplish this as well as what a reasonable rate and time line would be. @pirate is this something you have the bandwidth and inclination to do? if not could you recommend anyone as tribute? :P

Ideally I see the community coming to consensus on what an initial API would look like in the form of an Issue, that would serve as the scope of work for the commission, a $ amount would be set against that issue and once complete and the PR is accepted the money would be paid to the developer from money in escrow for this task.

Thank you all for your time with this and I look forward to hearing from you.

Originally created by @djkemmet on GitHub (Mar 22, 2023). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1126 I would like to sponsor (pay a developer cash monies in USD) for the development of the API for ArchiveBox. I need this for my digital preservation and ML/NLP work and I suspect a lot of people will end up doing a lot of cool things with it. I would like to talk to @pirate and get his blessing and solicit feedback from the community on how we could work together to accomplish this as well as what a reasonable rate and time line would be. @pirate is this something you have the bandwidth and inclination to do? if not could you recommend anyone as tribute? :P Ideally I see the community coming to consensus on what an initial API would look like in the form of an Issue, that would serve as the scope of work for the commission, a $ amount would be set against that issue and once complete and the PR is accepted the money would be paid to the developer from money in escrow for this task. Thank you all for your time with this and I look forward to hearing from you.

kerem

2026-03-01 14:45:40 +03:00

closed this issue
added the
why: functionality

touches: API/CLI/Spec
labels

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pirate commented on GitHub (Mar 28, 2023):

Hey @djkemmet, this is definitely something I'm interested in and I'm excited that you're willing to contribute towards seeing it happen.

This is good timing as I'm getting ready to write some grant applications for ArchiveBox and I'm hoping to fund my full-time development on it for the next few years. The API is definitely one of the main things I want to build in the next year, and I want to work on ArchiveBox as my main job if I can pull off finding a big sponsor.

The grants I'm aiming for are $150k+ so if your budget for this task is <$2k I wouldn't bother sponsoring this if you cant wait a few months, as I'll likely get around to it anyway. If your needs are more urgent than that or your budget is higher then lets discuss further via chat/call.

I agree either way we should finalize an API spec and framework choice, the work so far toward that is here: https://github.com/ArchiveBox/ArchiveBox/issues/496#issuecomment-1097310582

You can comment your thoughts on that draft API spec in that issue ^, and we can continue the discussion about timeline/sponsorship/and your specific needs for ML/NLP/etc. here.

Places I'm considering applying for grants:

https://longnow.org
https://www.arweave.org/funding (was offered grant in 2020, but was too busy then)
https://grants.filecoin.io (have friends here, and they fund similar archiving tools)
https://www.opentech.fund / https://www.opentech.fund/funds/internet-freedom-fund/ (specifically interested in censorship circumvention and internet freedom tooling)
https://www.ripe.net/support/cpf (May 1st deadline)
https://www.python.org/psf/grants/
https://nlnet.nl/news/2023/20230201-call.html (April 1st deadline)
https://www.nsf.gov/pubs/2023/nsf23556/nsf23556.htm (Sept 2023 deadline)
Open Source Peer Bonus Program | Google Open Source
MIECO: Advancing the future of the internet - Mozilla Innovations
https://sfconservancy.org/projects/apply/ or https://opencollective.com/collectives (help with non-profit administration, legal stuff, and getting funding)
more: https://github.com/ralphtheninja/open-funding, https://itsfoss.com/open-source-funding-platforms/, https://www.oss.fund

@pirate commented on GitHub (Mar 28, 2023): Hey @djkemmet, this is definitely something I'm interested in and I'm excited that you're willing to contribute towards seeing it happen. This is good timing as I'm getting ready to write some grant applications for ArchiveBox and I'm hoping to fund my full-time development on it for the next few years. The API is definitely one of the main things I want to build in the next year, and I want to work on ArchiveBox as my main job if I can pull off finding a big sponsor. The grants I'm aiming for are $150k+ so if your budget for this task is <$2k I wouldn't bother sponsoring this if you cant wait a few months, as I'll likely get around to it anyway. If your needs are more urgent than that or your budget is higher then lets discuss further via chat/call. I agree either way we should finalize an API spec and framework choice, the work so far toward that is here: https://github.com/ArchiveBox/ArchiveBox/issues/496#issuecomment-1097310582 You can comment your thoughts on that draft API spec in that issue ^, and we can continue the discussion about timeline/sponsorship/and your specific needs for ML/NLP/etc. here. Places I'm considering applying for grants: - https://longnow.org - https://www.arweave.org/funding (was offered grant in 2020, but was too busy then) - https://grants.filecoin.io (have friends here, and they fund similar archiving tools) - https://www.opentech.fund / https://www.opentech.fund/funds/internet-freedom-fund/ (specifically interested in censorship circumvention and internet freedom tooling) - https://www.ripe.net/support/cpf (May 1st deadline) - https://www.python.org/psf/grants/ - https://nlnet.nl/news/2023/20230201-call.html (April 1st deadline) - https://www.nsf.gov/pubs/2023/nsf23556/nsf23556.htm (Sept 2023 deadline) - [Open Source Peer Bonus Program | Google Open Source](https://opensource.google/documentation/reference/growing/peer-bonus) - [MIECO: Advancing the future of the internet - Mozilla Innovations](https://future.mozilla.org/mieco/%23newsletter-section) - https://sfconservancy.org/projects/apply/ or https://opencollective.com/collectives (help with non-profit administration, legal stuff, and getting funding) - more: https://github.com/ralphtheninja/open-funding, https://itsfoss.com/open-source-funding-platforms/, https://www.oss.fund

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@aaronmayer108 commented on GitHub (Mar 28, 2023):

WOO! So excited to see that grant money might be a possibility! Btw, grant writing is becoming trivially easy with GPT.

@aaronmayer108 commented on GitHub (Mar 28, 2023): WOO! So excited to see that grant money might be a possibility! Btw, grant writing is becoming trivially easy with GPT.

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@djkemmet commented on GitHub (May 1, 2023):

heyy @pirate,

yeah my budget for this was $1000 USD If there's anyone in your network who might be suited for this work and assuming $1000 is a reasonable amount of money to get the API off the ground I would definitely be happy to sponsor someone based on your recommendation. that said, as far as grants go, have you looked at futo? https://futo.org/fellows/

@djkemmet commented on GitHub (May 1, 2023): heyy @pirate, yeah my budget for this was $1000 USD If there's anyone in your network who might be suited for this work and assuming $1000 is a reasonable amount of money to get the API off the ground I would definitely be happy to sponsor someone based on your recommendation. that said, as far as grants go, have you looked at futo? https://futo.org/fellows/

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pe1uca commented on GitHub (Sep 28, 2023):

Hello guys!
I can step up to work on this.

@pirate let me know if there's anything to consider while doing grant work for the project and what things to have in mind (like what you mentioned of grants already being discussed with other parties, and guidelines for the code to be more likely to be merged).

@pe1uca commented on GitHub (Sep 28, 2023): Hello guys! I can step up to work on this. @pirate let me know if there's anything to consider while doing grant work for the project and what things to have in mind (like what you mentioned of grants already being discussed with other parties, and guidelines for the code to be more likely to be merged).

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pirate commented on GitHub (Oct 2, 2023):

Hey @pe1uca thanks for the offer! I'd definitely accept help with grant applications, want to chat on a call sometime? lets find a calendar time on Zulip

A quick update for everyone here, I am currently in the process of setting up a contract with the United Nations to help with their internet archiving needs! That contract will focus directly on ArchiveBox development (keeping it all open source), as well as enhancing it for their specific needs. It's a slow moving bureaucratic process so it may not start for a few more months until we get all the approvals, but it's a major step forward in securing real money for part/full time ArchiveBox development going forward.

@pirate commented on GitHub (Oct 2, 2023): Hey @pe1uca thanks for the offer! I'd definitely accept help with grant applications, want to chat on a call sometime? lets find a calendar time on Zulip A quick update for everyone here, I am currently in the process of setting up a contract with the United Nations to help with their internet archiving needs! That contract will focus directly on ArchiveBox development (keeping it all open source), as well as enhancing it for their specific needs. It's a slow moving bureaucratic process so it may not start for a few more months until we get all the approvals, but it's a major step forward in securing real money for part/full time ArchiveBox development going forward.

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pirate commented on GitHub (Oct 9, 2023):

Just a call with @pe1uca and we discussed a good approach to get some API functionality before my event sourcing refactor is complete:

create a new app under archivebox/api (separate from core and CLI)
create an endpoint for each archivebox CLI subcommand (e.g. add, remove, list, schedule) and allow it to take config/env variables and CLI parameters as the POST request parameters
call the CLI code directly using the requests parameters, and return the stdout/stderr as the response

Notes:

for long-running requests (e.g. an add that takes 20min), we can return a 200 and use this hack to run the job after the response is returned. later on we will use a proper job queue system, but because I am working on this in my refactor we don't want to depend on this just yet otherwise it will block the entire API development process. if the job is cancelled or times out early, it can be picked up automatically and resumed by an archivebox schedule ONLY_NEW=False job in the background, acting as a janky but cheap/easy background job system until my refactor is finished
some off-the-shelf django library or snippet should be used to allow users to create and manage API tokens from the UI/CLI to be used with the new API endpoints. I think allowing the user to create named API tokens (that inherit the same permissions as that user) is enough for now, no need to do OAuth2 or JWT

We're also discussing this further on the ArchiveBox chat server here: https://zulip.archivebox.io/#narrow/stream/158-development/topic/REST.20API.20development/near/1166232

@pe1uca is going to take a look at doing this, @djkemmet if you're still willing to sponsor development it would be great to discuss it further between the three of us :)

@pirate commented on GitHub (Oct 9, 2023): Just a call with @pe1uca and we discussed a good approach to get some API functionality before my event sourcing refactor is complete: 1. create a new app under archivebox/api (separate from core and CLI) 2. create an endpoint for each archivebox CLI subcommand (e.g. `add`, `remove`, `list`, `schedule`) and allow it to take config/env variables and CLI parameters as the POST request parameters 3. call the CLI code directly using the requests parameters, and return the stdout/stderr as the response Notes: - for long-running requests (e.g. an `add` that takes 20min), we can return a 200 and use [this hack](https://github.com/pirate/django-http2-middleware?tab=readme-ov-file#bonus-material) to run the job after the response is returned. later on we will use a proper job queue system, but because I am working on this in my refactor we don't want to depend on this just yet otherwise it will block the entire API development process. if the job is cancelled or times out early, it can be picked up automatically and resumed by an `archivebox schedule` `ONLY_NEW=False` job in the background, acting as a janky but cheap/easy background job system until my refactor is finished - some off-the-shelf django library or snippet should be used to allow users to create and manage API tokens from the UI/CLI to be used with the new API endpoints. I think allowing the user to create named API tokens (that inherit the same permissions as that user) is enough for now, no need to do OAuth2 or JWT We're also discussing this further on the ArchiveBox chat server here: https://zulip.archivebox.io/#narrow/stream/158-development/topic/REST.20API.20development/near/1166232 @pe1uca is going to take a look at doing this, @djkemmet if you're still willing to sponsor development it would be great to discuss it further between the three of us :)

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@rcarmo commented on GitHub (Jan 29, 2024):

Will this cover, say, returning a JSON payload of document content via a search? That would be the key thing for me.

@rcarmo commented on GitHub (Jan 29, 2024): Will this cover, say, returning a JSON payload of document content via a search? That would be the key thing for me.

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pirate commented on GitHub (Jan 30, 2024):

Yes, certainly @rcarmo

@pirate commented on GitHub (Jan 30, 2024): Yes, certainly @rcarmo

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@Brandl commented on GitHub (Apr 9, 2024):

I made a step towards an API by making the current CLI command available via API:

See https://github.com/ArchiveBox/ArchiveBox/pull/1397

@Brandl commented on GitHub (Apr 9, 2024): I made a step towards an API by making the current CLI command available via API: ![Screenshot from 2024-04-10 01-45-11](https://github.com/ArchiveBox/ArchiveBox/assets/3877188/2927b793-2769-466b-97b9-fb0a1333bd27) See https://github.com/ArchiveBox/ArchiveBox/pull/1397

kerem commented

2026-03-01 14:45:41 +03:00

Author

Owner

@pirate commented on GitHub (May 1, 2024):

The new REST API is rapidly evolving on dev! Big thanks to @Brandl for the first PR, keep an eye out for new releases!

https://github.com/ArchiveBox/ArchiveBox/releases/v0.8.0-rc

I'm going to close this issue for now and move these discussions to:

REST API bugs and feature requests here: https://github.com/ArchiveBox/ArchiveBox/issues/496
discuss ArchiveBox grants and funding here: https://zulip.archivebox.io/#narrow/stream/178-funding/topic/grants.20to.20apply.20to

@pirate commented on GitHub (May 1, 2024): The new REST API is rapidly evolving on `dev`! Big thanks to @Brandl for the first PR, keep an eye out for new releases! https://github.com/ArchiveBox/ArchiveBox/releases/v0.8.0-rc ![image](https://github.com/ArchiveBox/ArchiveBox/assets/511499/0adff6db-b602-4898-9d4b-a15d80fe8d34) --- I'm going to close this issue for now and move these discussions to: - REST API bugs and feature requests here: https://github.com/ArchiveBox/ArchiveBox/issues/496 - discuss ArchiveBox grants and funding here: https://zulip.archivebox.io/#narrow/stream/178-funding/topic/grants.20to.20apply.20to

kerem referenced this issue

2026-03-01 14:49:05 +03:00

[PR #708] [MERGED] fix bookmarklet script #1266

kerem referenced this issue

2026-03-01 18:00:43 +03:00

[PR #708] fix bookmarklet script #2774

kerem referenced this issue

2026-03-15 01:36:04 +03:00

[PR #708] [MERGED] fix bookmarklet script #4281

Rows
Columns

[GH-ISSUE #1126] Discussion: Sponsoring the REST API development + applying for grants to fund ArchiveBox development #708