[GH-ISSUE #906] feature request: allow for customizing the worker user-agent #595

New issue

Closed

opened 2026-03-02 11:51:11 +03:00 by kerem · 1 comment

kerem commented

2026-03-02 11:51:11 +03:00

Owner

Originally created by @meysam81 on GitHub (Jan 19, 2025).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/906

Hi @MohamedBassem,

Great project.

I have started using it for a week and I love it. Its philosophy is exactly as I always wanted for myself.

I wonder if you'd be willing to allow for customizing the user-agent header in the worker process, allowing for bypassingthe usual bot blockers from detecting that we're not doing harmful crawling, but rather scraping the usual content for our own self-hosted Hoarder instance; for example by passing the Firefox or Chrome user agent, or ideally by filling it randomly every single time!

github.com/hoarder-app/hoarder@219e16a0bd/apps/workers/feedWorker.ts (L96-L102)

An already implemented version in another part of the code:

github.com/hoarder-app/hoarder@219e16a0bd/apps/workers/crawlerWorker.ts (L282-L284)

Is that something you'd be OK with?

Cheers. 🥂

Originally created by @meysam81 on GitHub (Jan 19, 2025). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/906 Hi @MohamedBassem, Great project. I have started using it for a week and I love it. Its philosophy is exactly as I always wanted for myself. I wonder if you'd be willing to allow for customizing the `user-agent` header in the worker process, allowing for bypassingthe usual bot blockers from detecting that we're not doing harmful crawling, but rather scraping the usual content for our own self-hosted Hoarder instance; for example by passing the Firefox or Chrome user agent, or ideally by filling it randomly every single time! https://github.com/hoarder-app/hoarder/blob/219e16a0bd8e0aaa7cbba24fc5e758e9f719907c/apps/workers/feedWorker.ts#L96-L102 An already implemented version in another part of the code: https://github.com/hoarder-app/hoarder/blob/219e16a0bd8e0aaa7cbba24fc5e758e9f719907c/apps/workers/crawlerWorker.ts#L282-L284 Is that something you'd be OK with? Cheers. 🥂

kerem

2026-03-02 11:51:11 +03:00

closed this issue
added the
feature request
label

kerem commented

2026-03-02 11:51:12 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Jan 19, 2025):

@meysam81 I think I'm ok with using the same user agent as the one used in the crawler worker 👍 Happy to accept a PR if you want to send one.

@MohamedBassem commented on GitHub (Jan 19, 2025): @meysam81 I think I'm ok with using the same user agent as the one used in the crawler worker 👍 Happy to accept a PR if you want to send one.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/karakeep#595

No description provided.

Rows
Columns