[GH-ISSUE #224] Add configurable timeout for LLM task #161

Closed
opened 2026-03-02 11:47:12 +03:00 by kerem · 4 comments
Owner

Originally created by @jbvioix on GitHub (Jun 14, 2024).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/224

I've successfully tried Ollama on GPU to generate keywords. However, when I use it on a CPU, I get no results. I've done a few tests in Python, the calculation time on CPU is much longer with correct results. I think there's a timeout somewhere that stops the Ollama task. Is it possible to configure it so that the CPU can be used (on a single-user lightweight server) for labelling?

Originally created by @jbvioix on GitHub (Jun 14, 2024). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/224 I've successfully tried Ollama on GPU to generate keywords. However, when I use it on a CPU, I get no results. I've done a few tests in Python, the calculation time on CPU is much longer with correct results. I think there's a timeout somewhere that stops the Ollama task. Is it possible to configure it so that the CPU can be used (on a single-user lightweight server) for labelling?
kerem 2026-03-02 11:47:12 +03:00
Author
Owner

@MohamedBassem commented on GitHub (Jun 14, 2024):

This makes a lot of sense. Can you share with me the "timeout" logs that you're getting? I want to know where exactly we're timing out to make it configurage. Is it that the background job itself timesout, or is it the call to ollama that timesout

<!-- gh-comment-id:2168653049 --> @MohamedBassem commented on GitHub (Jun 14, 2024): This makes a lot of sense. Can you share with me the "timeout" logs that you're getting? I want to know where exactly we're timing out to make it configurage. Is it that the background job itself timesout, or is it the call to ollama that timesout
Author
Owner

@jbvioix commented on GitHub (Jun 15, 2024):

With GPU enabled, I've got these logs:

workers-1      | 2024-06-15T07:16:31.518Z info: [inference][99] Starting an inference job for bookmark with id "ouhm96clwfw25pkbmdcrlj3o"
ollama-1       | [GIN] 2024/06/15 - 07:17:16 | 200 | 44.993621295s |      172.25.0.7 | POST     "/api/chat"
workers-1      | 2024-06-15T07:17:16.537Z info: [inference][99] Inferring tag for bookmark "ouhm96clwfw25pkbmdcrlj3o" used 1656 tokens and inferred: Python,History,ProgrammingLanguage,ComputerScience,DevelopmentEnvironment
workers-1      | 2024-06-15T07:17:16.584Z info: [inference][99] Completed successfully

Perfect job, no problem.
If I disabled GPU, I've got this:

workers-1      | 2024-06-15T07:19:46.715Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj"
ollama-1       | [GIN] 2024/06/15 - 07:24:47 | 200 |          5m1s |      172.25.0.7 | POST     "/api/chat"
workers-1      | 2024-06-15T07:24:47.971Z error: [inference][100] inference job failed: TypeError: fetch failed
workers-1      | 2024-06-15T07:29:50.926Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj"
...
ollama-1       | [GIN] 2024/06/15 - 07:29:49 | 200 |          5m1s |      172.25.0.7 | POST     "/api/chat"
workers-1      | 2024-06-15T07:29:49.832Z error: [inference][100] inference job failed: TypeError: fetch failed
workers-1      | 2024-06-15T07:29:50.926Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj"
...
ollama-1       | [GIN] 2024/06/15 - 07:34:52 | 200 |          5m1s |      172.25.0.7 | POST     "/api/chat"
workers-1      | 2024-06-15T07:34:52.254Z error: [inference][100] inference job failed: TypeError: fetch failed
...

After the first fail, a new inference job is automatically launched. There are 5 minutes between each job events. I think it's a timeout somewhere...

<!-- gh-comment-id:2169234088 --> @jbvioix commented on GitHub (Jun 15, 2024): With GPU enabled, I've got these logs: ``` workers-1 | 2024-06-15T07:16:31.518Z info: [inference][99] Starting an inference job for bookmark with id "ouhm96clwfw25pkbmdcrlj3o" ollama-1 | [GIN] 2024/06/15 - 07:17:16 | 200 | 44.993621295s | 172.25.0.7 | POST "/api/chat" workers-1 | 2024-06-15T07:17:16.537Z info: [inference][99] Inferring tag for bookmark "ouhm96clwfw25pkbmdcrlj3o" used 1656 tokens and inferred: Python,History,ProgrammingLanguage,ComputerScience,DevelopmentEnvironment workers-1 | 2024-06-15T07:17:16.584Z info: [inference][99] Completed successfully ``` Perfect job, no problem. If I disabled GPU, I've got this: ``` workers-1 | 2024-06-15T07:19:46.715Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj" ollama-1 | [GIN] 2024/06/15 - 07:24:47 | 200 | 5m1s | 172.25.0.7 | POST "/api/chat" workers-1 | 2024-06-15T07:24:47.971Z error: [inference][100] inference job failed: TypeError: fetch failed workers-1 | 2024-06-15T07:29:50.926Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj" ... ollama-1 | [GIN] 2024/06/15 - 07:29:49 | 200 | 5m1s | 172.25.0.7 | POST "/api/chat" workers-1 | 2024-06-15T07:29:49.832Z error: [inference][100] inference job failed: TypeError: fetch failed workers-1 | 2024-06-15T07:29:50.926Z info: [inference][100] Starting an inference job for bookmark with id "uxr3yjtfke0tu2u800jbh9rj" ... ollama-1 | [GIN] 2024/06/15 - 07:34:52 | 200 | 5m1s | 172.25.0.7 | POST "/api/chat" workers-1 | 2024-06-15T07:34:52.254Z error: [inference][100] inference job failed: TypeError: fetch failed ... ``` After the first fail, a new inference job is automatically launched. There are 5 minutes between each job events. I think it's a timeout somewhere...
Author
Owner

@kirincorleone commented on GitHub (Aug 23, 2024):

Hi,

Loving Hoarder, thanks for this app!

I am using ollama on my Synology DS920+
CPU: J4125
GPU: None
Text Model: tinydolphin
Image Model: None

Workers Docker Environment Variables:

OLLAMA_BASE_URL: http://[address]
INFERENCE_TEXT_MODEL: tinydolphin

#INFERENCE_IMAGE_MODEL: llava

Just saying Hi on Open-webUI takes many minutes for a reply. So, I definitely need longer time for Hoarder Inference to do its job.

Here are my logs, I hope they help:

stdout 2024-08-22T19:37:34.312Z error: [inference][197] inference job failed: Error: Timeout
stdout 2024-08-22T19:37:35.033Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf"
stdout Getting text from response
stdout 2024-08-22T19:38:05.913Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf"
stdout Getting text from response
stdout 2024-08-22T19:38:36.580Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf"
stdout Getting text from response
stdout 2024-08-22T19:39:07.237Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf"
stdout Getting text from response
stdout 2024-08-22T19:39:37.233Z error: [inference][199] inference job failed: Error: Timeout
stdout 2024-08-22T19:39:37.945Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x"
stdout Getting text from response
stdout 2024-08-22T19:40:08.519Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x"
stdout Getting text from response
stdout 2024-08-22T19:40:38.922Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x"
stdout Getting text from response
stdout 2024-08-22T19:41:09.717Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x"
stdout Getting text from response
stdout 2024-08-22T19:41:39.712Z error: [inference][201] inference job failed: Error: Timeout
stdout 2024-08-22T19:41:40.875Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime"
stdout Getting text from response
stdout 2024-08-22T19:42:11.337Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime"
stdout Getting text from response
stdout 2024-08-22T19:42:42.016Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime"
stdout Getting text from response
stdout 2024-08-22T19:43:12.583Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime"
stdout Getting text from response
stdout 2024-08-22T19:43:42.581Z error: [inference][203] inference job failed: Error: Timeout
stdout 2024-08-22T19:43:44.484Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38"
stdout Getting text from response
stdout 2024-08-22T19:44:15.008Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38"
stdout Getting text from response
stdout 2024-08-22T19:44:46.119Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38"
stdout Getting text from response
stdout 2024-08-22T19:45:16.759Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38"
stdout Getting text from response
stdout 2024-08-22T19:45:46.757Z error: [inference][205] inference job failed: Error: Timeout
stdout 2024-08-22T19:45:47.462Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7"
stdout Getting text from response
stdout 2024-08-22T19:46:18.110Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7"
stdout Getting text from response
stdout 2024-08-22T19:46:48.969Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7"
stdout Getting text from response
stdout 2024-08-22T19:47:19.313Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7"
stdout Getting text from response
stdout 2024-08-22T19:47:49.305Z error: [inference][207] inference job failed: Error: Timeout
<!-- gh-comment-id:2307085533 --> @kirincorleone commented on GitHub (Aug 23, 2024): Hi, Loving Hoarder, thanks for this app! I am using ollama on my Synology DS920+ CPU: J4125 GPU: None Text Model: tinydolphin Image Model: None Workers Docker Environment Variables: OLLAMA_BASE_URL: http://[address] INFERENCE_TEXT_MODEL: tinydolphin > #INFERENCE_IMAGE_MODEL: llava Just saying Hi on Open-webUI takes many minutes for a reply. So, I definitely need longer time for Hoarder Inference to do its job. Here are my logs, I hope they help: ``` stdout 2024-08-22T19:37:34.312Z error: [inference][197] inference job failed: Error: Timeout stdout 2024-08-22T19:37:35.033Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf" stdout Getting text from response stdout 2024-08-22T19:38:05.913Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf" stdout Getting text from response stdout 2024-08-22T19:38:36.580Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf" stdout Getting text from response stdout 2024-08-22T19:39:07.237Z info: [inference][199] Starting an inference job for bookmark with id "hf59xhijya0mxxka9jxvr9rf" stdout Getting text from response stdout 2024-08-22T19:39:37.233Z error: [inference][199] inference job failed: Error: Timeout stdout 2024-08-22T19:39:37.945Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x" stdout Getting text from response stdout 2024-08-22T19:40:08.519Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x" stdout Getting text from response stdout 2024-08-22T19:40:38.922Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x" stdout Getting text from response stdout 2024-08-22T19:41:09.717Z info: [inference][201] Starting an inference job for bookmark with id "qcgxclnnymm0n2lb8eks5u2x" stdout Getting text from response stdout 2024-08-22T19:41:39.712Z error: [inference][201] inference job failed: Error: Timeout stdout 2024-08-22T19:41:40.875Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime" stdout Getting text from response stdout 2024-08-22T19:42:11.337Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime" stdout Getting text from response stdout 2024-08-22T19:42:42.016Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime" stdout Getting text from response stdout 2024-08-22T19:43:12.583Z info: [inference][203] Starting an inference job for bookmark with id "smevi67ztewic14e3qv1aime" stdout Getting text from response stdout 2024-08-22T19:43:42.581Z error: [inference][203] inference job failed: Error: Timeout stdout 2024-08-22T19:43:44.484Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38" stdout Getting text from response stdout 2024-08-22T19:44:15.008Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38" stdout Getting text from response stdout 2024-08-22T19:44:46.119Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38" stdout Getting text from response stdout 2024-08-22T19:45:16.759Z info: [inference][205] Starting an inference job for bookmark with id "vgynzse8zf56agjkbywu1d38" stdout Getting text from response stdout 2024-08-22T19:45:46.757Z error: [inference][205] inference job failed: Error: Timeout stdout 2024-08-22T19:45:47.462Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7" stdout Getting text from response stdout 2024-08-22T19:46:18.110Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7" stdout Getting text from response stdout 2024-08-22T19:46:48.969Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7" stdout Getting text from response stdout 2024-08-22T19:47:19.313Z info: [inference][207] Starting an inference job for bookmark with id "ziwptxbaqofot2yud3qth5e7" stdout Getting text from response stdout 2024-08-22T19:47:49.305Z error: [inference][207] inference job failed: Error: Timeout ```
Author
Owner

@MohamedBassem commented on GitHub (Sep 15, 2024):

This is going to be available in the next release. Sorry for how long it took me to get to this :)

<!-- gh-comment-id:2351234169 --> @MohamedBassem commented on GitHub (Sep 15, 2024): This is going to be available in the next release. Sorry for how long it took me to get to this :)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/karakeep#161
No description provided.