[GH-ISSUE #1748] Bug: High memory usage on karakeep worker during large imports (and crawlings in general)

kerem commented

2026-03-02 11:54:55 +03:00

Owner

Originally created by @Garrett-R on GitHub (Jul 18, 2025).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/1748

Describe the Bug

I imported ~1,500 links from a Pocket export CSV and it's exhausting memory. I'm running Karakeep on PikaPods.

Steps to Reproduce

Import large Pocket .csv
Observe it swamps Karakeep

It exhausted the 2GB memory, so I upped it to 4GB of memory, and it exhausted the new limit about 20m later:

Here's the most recent logs: pikapods_logs.txt

There is perhaps a memory leak somewhere.

Expected Behaviour

Memory doesn't keep growing.

Screenshots or Additional Context

No response

Device Details

Firefox 140.0.4

Exact Karakeep Version

v0.25.0

Have you checked the troubleshooting guide?

I have checked the troubleshooting guide and I haven't found a solution to my problem

Originally created by @Garrett-R on GitHub (Jul 18, 2025). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/1748 ### Describe the Bug I imported ~1,500 links from a Pocket export CSV and it's exhausting memory. I'm running Karakeep on PikaPods. ### Steps to Reproduce 1. Import large Pocket .csv 2. Observe it swamps Karakeep It exhausted the 2GB memory, so I upped it to 4GB of memory, and it exhausted the new limit about 20m later: <img width="952" height="839" alt="Image" src="https://github.com/user-attachments/assets/181fa045-fe9b-415f-8c0d-793f9a562f68" /> Here's the most recent logs: [pikapods_logs.txt](https://github.com/user-attachments/files/21322883/pikapods_logs.txt) There is perhaps a memory leak somewhere. ### Expected Behaviour Memory doesn't keep growing. ### Screenshots or Additional Context _No response_ ### Device Details Firefox 140.0.4 ### Exact Karakeep Version v0.25.0 ### Have you checked the troubleshooting guide? - [x] I have checked the troubleshooting guide and I haven't found a solution to my problem

kerem added the

bug

pri/high

status/approved

labels

2026-03-02 11:54:55 +03:00

kerem commented

2026-03-02 11:54:55 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Jul 19, 2025):

There's a high chance that this is meillisearch eating your memory as it by defaults sets a limit to itself at 2/3 of the memory. Does pikapods allow you to set env variables on the meillisearch container? Can you try setting MEILI_MAX_INDEXING_MEMORY=1Gb and see if the memory grows as much?

@MohamedBassem commented on GitHub (Jul 19, 2025): There's a high chance that this is meillisearch eating your memory as it by defaults sets a limit to itself at 2/3 of the memory. Does pikapods allow you to set env variables on the meillisearch container? Can you try setting `MEILI_MAX_INDEXING_MEMORY=1Gb` and see if the memory grows as much?

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@Garrett-R commented on GitHub (Jul 20, 2025):

Does pikapods allow you to set env variables on the meillisearch container

Seems you have to email them to add an env var. They provide some common ones used by the app it seems, but MEILI_MAX_INDEXING_MEMORY is not there:

I can go ahead and email them.

But are you saying there's a separate meillisearch container? I just have a single pod that I would think is running a container built from the repo's Dockerfile.

BTW, going to 4GB and waiting a while a day or two seems to have worked, it's down to 2.5GB of memory usage and seems to be working nicely now.

@Garrett-R commented on GitHub (Jul 20, 2025): > Does pikapods allow you to set env variables on the meillisearch container Seems you have to [email them](https://docs.pikapods.com/faq#missing-environment-variables) to add an env var. They provide some common ones used by the app it seems, but `MEILI_MAX_INDEXING_MEMORY` is not there: <img width="1568" height="797" alt="Image" src="https://github.com/user-attachments/assets/7ce0a4da-f64b-4014-a337-eed40705f48c" /> I can go ahead and email them. But are you saying there's a separate `meillisearch` container? I just have a single pod that I would think is running a container built from the repo's [Dockerfile](https://github.com/karakeep-app/karakeep/blob/main/docker/Dockerfile). --------- BTW, going to 4GB and waiting a while a day or two seems to have worked, it's down to 2.5GB of memory usage and seems to be working nicely now.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@maxtheaxe commented on GitHub (Jul 22, 2025):

This is the same issue I'm (still) experiencing. My pod has 16gb of RAM allocated to it and it still hasn't finished processing my 12,000 link export from pocket after two weeks. It locks up at least three times a day, at which point I need to manually log in and restart it. Quite painful to do every time I want to save a new link.

@maxtheaxe commented on GitHub (Jul 22, 2025): This is [the same issue I'm (still) experiencing](https://github.com/karakeep-app/karakeep/issues/1721). My pod has 16gb of RAM allocated to it and it still hasn't finished processing my 12,000 link export from pocket after two weeks. It locks up at least three times a day, at which point I need to manually log in and restart it. Quite painful to do every time I want to save a new link.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@maxtheaxe commented on GitHub (Jul 22, 2025):

Side note, @Garrett-R, you may want to update your pod URL now that you've shared it publicly. At a minimum, you should block new account creation.

@maxtheaxe commented on GitHub (Jul 22, 2025): Side note, @Garrett-R, you may want to update your pod URL now that you've shared it publicly. At a minimum, you should block new account creation.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@Garrett-R commented on GitHub (Jul 22, 2025):

@Garrett-R, you may want to update your pod URL now that you've shared it publicly

Ah right, thanks, will do!

@Garrett-R commented on GitHub (Jul 22, 2025): > @Garrett-R, you may want to update your pod URL now that you've shared it publicly Ah right, thanks, will do!

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@maxtheaxe commented on GitHub (Aug 22, 2025):

I finished importing the backlog a long time ago and even in regular usage, karakeep eventually still uses all available memory and inevitably crashes.

@maxtheaxe commented on GitHub (Aug 22, 2025): I finished importing the backlog a long time ago and even in regular usage, karakeep eventually still uses all available memory and inevitably crashes.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@Garrett-R commented on GitHub (Aug 22, 2025):

Ah strange, for me, once it finished importing and retrieving thumbnails and stuff, it was never an issue again. Took multiple days though for ~1,500 links IIRC. Are you sure it's finished retrieving all the thumbnails and what not?

@Garrett-R commented on GitHub (Aug 22, 2025): Ah strange, for me, once it finished importing and retrieving thumbnails and stuff, it was never an issue again. Took multiple days though for ~1,500 links IIRC. Are you sure it's finished retrieving all the thumbnails and what not?

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Sep 7, 2025):

I have seen this issue as well that during crawling, the worker container sometimes spikes in memory usage up to 2.5GB. This will require more debugging from my side, but I've marked it as a high/pri item to fix.

@MohamedBassem commented on GitHub (Sep 7, 2025): I have seen this issue as well that during crawling, the worker container sometimes spikes in memory usage up to 2.5GB. This will require more debugging from my side, but I've marked it as a `high/pri` item to fix.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Sep 28, 2025):

I merged one potential fix, will test is in the cloud deployment and report back if it helps.

@MohamedBassem commented on GitHub (Sep 28, 2025): I merged one potential fix, will test is in the cloud deployment and report back if it helps.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Oct 27, 2025):

I merged one more fix (github.com/karakeep-app/karakeep@40d548bd25). And have had reasonable success by setting NODE_OPTIONS=--max-old-space-size=1024 on the workers. The assumption is that the memory ballooning during imports are caused by the GC not running frequently enough, and this flag should help. What's the correct value to use, I still don't know. Experimenting with 1GB and will report back.

@MohamedBassem commented on GitHub (Oct 27, 2025): I merged one more fix (https://github.com/karakeep-app/karakeep/commit/40d548bd25141e9aba6c145911d44ebc16ac0e61). And have had reasonable success by setting `NODE_OPTIONS=--max-old-space-size=1024` on the workers. The assumption is that the memory ballooning during imports are caused by the GC not running frequently enough, and this flag should help. What's the correct value to use, I still don't know. Experimenting with 1GB and will report back. <img width="1174" height="436" alt="Image" src="https://github.com/user-attachments/assets/1a4303f0-39b5-44b8-8792-897c7cdcd2da" />

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@maxtheaxe commented on GitHub (Oct 27, 2025):

Thanks for your persistence, Mohamed! I'll send pikapods an email and ask that they add that environment variable on my end.

@maxtheaxe commented on GitHub (Oct 27, 2025): Thanks for your persistence, Mohamed! I'll send pikapods an email and ask that they add that environment variable on my end.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Nov 9, 2025):

Ok, went through a couple more intensive imports with NODE_OPTIONS=--max-old-space-size=1024:

The memory peaks (at least ignoring the small spikes) at ~1.3GB. So I'm confident that the fixes plus the flags are helping. Now the next step is for me to find a nice way to pass --max-old-space-size=1024 to the app via a karakeep provided env variable.

@MohamedBassem commented on GitHub (Nov 9, 2025): Ok, went through a couple more intensive imports with `NODE_OPTIONS=--max-old-space-size=1024`: <img width="2324" height="1334" alt="Image" src="https://github.com/user-attachments/assets/caf764ed-0abd-43f0-8c97-ae27b1bb1669" /> The memory peaks (at least ignoring the small spikes) at ~1.3GB. So I'm confident that the fixes plus the flags are helping. Now the next step is for me to find a nice way to pass `--max-old-space-size=1024` to the app via a karakeep provided env variable.

kerem commented

2026-03-02 11:54:56 +03:00

Author

Owner

@gusvd commented on GitHub (Nov 16, 2025):

Hi all! I'm new to Karakeep, so please bear with me, but I'm encountering memory issues on a fresh installation.

The Chromium instance is taking so much memory that it's freezing my VPS. Here are some details:

My setup:

VPS: 2 vCPU, 4GB RAM (Hetzner)
Self-hosting with Coolify v4, installed via the official Karakeep docker-compose template
Karakeep version: ghcr.io/karakeep-app/karakeep:release (latest)

Issues:

Chrome processes attempting to allocate 3-3.4GB RAM each, causing repeated OOM kills
Python3 processes (likely the orchestrator) also getting killed
OOM (Out of Memory) killer terminating processes
VPS becomes completely unresponsive

What fixed it:
Adding Docker resource limits to the compose file, for example:

chrome:
  deploy:
    resources:
      limits:
        cpus: '0.75'
        memory: 768M
      reservations:
        memory: 128M
  shm_size: '256m'

Maybe the official docker-compose template could include default resource limits? This would help users on modest VPS setups avoid these issues out of the box.
Happy to provide more details if needed!

@gusvd commented on GitHub (Nov 16, 2025): Hi all! I'm new to Karakeep, so please bear with me, but I'm encountering memory issues on a fresh installation. The Chromium instance is taking so much memory that it's freezing my VPS. Here are some details: **My setup:** - VPS: 2 vCPU, 4GB RAM (Hetzner) - Self-hosting with Coolify v4, installed via the official Karakeep docker-compose template - Karakeep version: ghcr.io/karakeep-app/karakeep:release (latest) **Issues:** - Chrome processes attempting to allocate 3-3.4GB RAM each, causing repeated OOM kills - Python3 processes (likely the orchestrator) also getting killed - OOM (Out of Memory) killer terminating processes - VPS becomes completely unresponsive **What fixed it:** Adding Docker resource limits to the compose file, for example: ``` chrome: deploy: resources: limits: cpus: '0.75' memory: 768M reservations: memory: 128M shm_size: '256m' ``` Maybe the official docker-compose template could include default resource limits? This would help users on modest VPS setups avoid these issues out of the box. Happy to provide more details if needed! <img width="2048" height="1222" alt="Image" src="https://github.com/user-attachments/assets/02bafec6-d21f-4c21-b0d3-8499b7995459" />

Rows
Columns

[GH-ISSUE #1748] Bug: High memory usage on karakeep worker during large imports (and crawlings in general) #1091

Describe the Bug

Steps to Reproduce

Expected Behaviour

Screenshots or Additional Context

Device Details

Exact Karakeep Version

Have you checked the troubleshooting guide?