[GH-ISSUE #20] Inference Failed with Ollama

kerem commented

2026-03-02 11:45:39 +03:00

Owner

Originally created by @joshuapeterson on GitHub (Mar 27, 2024).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/20

I added ollama support per instructions. Some of the requests seemed to work fine but I found the following in the logs:
2024-03-27T14:01:40.649Z error: [inference][17] inference job failed: Error: Expected a completed response. 2024-03-27T14:06:40.997Z error: [inference][18] inference job failed: TypeError: fetch failed

At least one of my bookmarks does not have any tags added to it.

I'm running this on a docker container on linux.

Originally created by @joshuapeterson on GitHub (Mar 27, 2024). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/20 I added ollama support per instructions. Some of the requests seemed to work fine but I found the following in the logs: `2024-03-27T14:01:40.649Z error: [inference][17] inference job failed: Error: Expected a completed response. 2024-03-27T14:06:40.997Z error: [inference][18] inference job failed: TypeError: fetch failed` At least one of my bookmarks does not have any tags added to it. I'm running this on a docker container on linux.

kerem

2026-03-02 11:45:39 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-02 11:45:40 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Mar 27, 2024):

Ollama is a new addition (a couple of hours old), so it's not as battle tested as using openAI. I'll probably add Experimental beside it in the documentation.

I managed to reproduce some weird behaviors at least with the llama2 model locally. Sometimes it doesn't ignores the instructions and outputs random jsons, etc. I'll try to dig deeper in what's happening. Thanks for the report!

@MohamedBassem commented on GitHub (Mar 27, 2024): Ollama is a new addition (a couple of hours old), so it's not as battle tested as using openAI. I'll probably add `Experimental` beside it in the documentation. I managed to reproduce some weird behaviors at least with the `llama2` model locally. Sometimes it doesn't ignores the instructions and outputs random jsons, etc. I'll try to dig deeper in what's happening. Thanks for the report!

kerem commented

2026-03-02 11:45:40 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Mar 27, 2024):

Added some warnings in github.com/MohamedBassem/hoarder-app@5cbce67fda.

@MohamedBassem commented on GitHub (Mar 27, 2024): Added some warnings in https://github.com/MohamedBassem/hoarder-app/commit/5cbce67fdae7ef697dd999b0f1e3cc6ed9c53e3f.

kerem commented

2026-03-02 11:45:40 +03:00

Author

Owner

@joshuapeterson commented on GitHub (Mar 27, 2024):

For sure these are not high priority issues. Just trying to add issues as I see them to help with your development. Liking what I'm seeing so far.

I just got a new one that had more data in the response:
2024-03-27T14:52:32.188Z error: [inference][25] inference job failed: Error: [inference][25] Failed to parse JSON response from inference client: [ { "code": "invalid_type", "expected": "array", "received": "undefined", "path": [ "tags" ], "message": "Required" } ]

@joshuapeterson commented on GitHub (Mar 27, 2024): For sure these are not high priority issues. Just trying to add issues as I see them to help with your development. Liking what I'm seeing so far. I just got a new one that had more data in the response: `2024-03-27T14:52:32.188Z error: [inference][25] inference job failed: Error: [inference][25] Failed to parse JSON response from inference client: [ { "code": "invalid_type", "expected": "array", "received": "undefined", "path": [ "tags" ], "message": "Required" } ]`

kerem commented

2026-03-02 11:45:40 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Mar 27, 2024):

Keep the bug reports coming, they are really helpful!

Yes, that's the error I got as well. As I mentioned, the model sometimes decides to ignore my instruction and outputs either an invalid json or a random json :D This is what casues this error. Btw, I'm curious, which model are you using? llama2?

@MohamedBassem commented on GitHub (Mar 27, 2024): Keep the bug reports coming, they are really helpful! Yes, that's the error I got as well. As I mentioned, the model sometimes decides to ignore my instruction and outputs either an invalid json or a random json :D This is what casues this error. Btw, I'm curious, which model are you using? `llama2`?

kerem commented

2026-03-02 11:45:40 +03:00

Author

Owner

@joshuapeterson commented on GitHub (Mar 27, 2024):

Yes, using llama2. No other reason except that's what was in your documentation. I've used Ollama for about 2 hours at this point... :)

@joshuapeterson commented on GitHub (Mar 27, 2024): Yes, using llama2. No other reason except that's what was in your documentation. I've used Ollama for about 2 hours at this point... :)

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Mar 27, 2024):

I sent github.com/MohamedBassem/hoarder-app@8156e8ce3e to solve the Expected a completed response problem in ollama, but the model still sometimes outputs JSONs not confronting to the format I requested. So it'll have a better success rate than what it used to have, but still not 100% as that of openai.

@MohamedBassem commented on GitHub (Mar 27, 2024): I sent https://github.com/MohamedBassem/hoarder-app/commit/8156e8ce3e2198da4e4b29f0857f4f5b6df7e1d0 to solve the `Expected a completed response` problem in ollama, but the model still sometimes outputs JSONs not confronting to the format I requested. So it'll have a better success rate than what it used to have, but still not 100% as that of openai.

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@dyslexicjedi commented on GitHub (Mar 28, 2024):

Not to jump onto your thread. But I've had two issues with this as well.

First:
2024-03-28T00:23:22.223Z info: [Crawler][12] Will crawl "https://css-tricks.com/creating-an-editable-textarea-that-supports-syntax-highlighted-code/" for link with id "g6avp8icpxw0w9q4tiyjl2sk"
2024-03-28T00:23:22.233Z info: [search][27] Completed successfully
2024-03-28T00:23:26.825Z info: [Crawler][12] Completed successfully
2024-03-28T00:23:26.838Z info: [search][28] Completed successfully
2024-03-28T00:23:31.077Z error: [inference][14] inference job failed: Error: [inference][14] Failed to parse JSON response from inference client: SyntaxError: Bad control character in string literal in JSON at position 243 (line 3 column 162)

Second:
2024-03-28T00:24:49.026Z info: [Crawler][13] Will crawl "https://docs.docker.com/engine/install/ubuntu/" for link with id "mjucq8li26nhh71ll97pf8id"
2024-03-28T00:24:49.033Z info: [search][29] Completed successfully
2024-03-28T00:24:51.975Z info: [Crawler][13] Completed successfully
2024-03-28T00:24:51.982Z info: [search][30] Completed successfully
2024-03-28T00:24:54.865Z error: [inference][15] inference job failed: Error: [inference][15] Failed to parse JSON response from inference client: [
{
"code": "invalid_type",
"expected": "array",
"received": "undefined",
"path": [
"tags"
],
"message": "Required"
}
]

@dyslexicjedi commented on GitHub (Mar 28, 2024): Not to jump onto your thread. But I've had two issues with this as well. First: 2024-03-28T00:23:22.223Z info: [Crawler][12] Will crawl "https://css-tricks.com/creating-an-editable-textarea-that-supports-syntax-highlighted-code/" for link with id "g6avp8icpxw0w9q4tiyjl2sk" 2024-03-28T00:23:22.233Z info: [search][27] Completed successfully 2024-03-28T00:23:26.825Z info: [Crawler][12] Completed successfully 2024-03-28T00:23:26.838Z info: [search][28] Completed successfully 2024-03-28T00:23:31.077Z error: [inference][14] inference job failed: Error: [inference][14] Failed to parse JSON response from inference client: SyntaxError: Bad control character in string literal in JSON at position 243 (line 3 column 162) Second: 2024-03-28T00:24:49.026Z info: [Crawler][13] Will crawl "https://docs.docker.com/engine/install/ubuntu/" for link with id "mjucq8li26nhh71ll97pf8id" 2024-03-28T00:24:49.033Z info: [search][29] Completed successfully 2024-03-28T00:24:51.975Z info: [Crawler][13] Completed successfully 2024-03-28T00:24:51.982Z info: [search][30] Completed successfully 2024-03-28T00:24:54.865Z error: [inference][15] inference job failed: Error: [inference][15] Failed to parse JSON response from inference client: [ { "code": "invalid_type", "expected": "array", "received": "undefined", "path": [ "tags" ], "message": "Required" } ]

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Mar 30, 2024):

@dyslexicjedi yeah, that one is the model not respecting the prompt. Sometimes the models just forgets about the prompt (I guess with long contexts?) and starts just outputing random JSONs. I wonder if there are some techniques that I can use for the model not to lose context. For example, Maybe I share the content first, and then ask the model to give me the tags? 🤔

@MohamedBassem commented on GitHub (Mar 30, 2024): @dyslexicjedi yeah, that one is the model not respecting the prompt. Sometimes the models just forgets about the prompt (I guess with long contexts?) and starts just outputing random JSONs. I wonder if there are some techniques that I can use for the model not to lose context. For example, Maybe I share the content first, and then ask the model to give me the tags? 🤔

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@kamtschatka commented on GitHub (May 16, 2024):

@dyslexicjedi Seems like I had the same idea as MohamedBassem. I tried it out before reading his answer and it seems to work fine.

Basically I split the existing prompt into 2 and added the "respond with json and 3-5 tags" part at the end:
github.com/MohamedBassem/hoarder-app@031f025bb1

For the samples above, the llama3 model responded properly:

"Install Docker Engine on Ubuntu": docker, installation, linux
"Creating an Editable Textarea That Supports Syntax-Highlighted Code | CSS-Tricks": code editor, javascript, css, html, syntax highlighting

I also tried it with the ~20 links I have in my hoarder instance so far and all of them were tagged (previously 3 weren't).
So if anybody has more links that did not work, please let me know, so I can try it out.

@kamtschatka commented on GitHub (May 16, 2024): @dyslexicjedi Seems like I had the same idea as [MohamedBassem](https://github.com/MohamedBassem). I tried it out before reading his answer and it seems to work fine. Basically I split the existing prompt into 2 and added the "respond with json and 3-5 tags" part at the end: https://github.com/MohamedBassem/hoarder-app/commit/031f025bb16f64167972917da3e681ce7b5fe124 For the samples above, the llama3 model responded properly: * "Install Docker Engine on Ubuntu": docker, installation, linux * "Creating an Editable Textarea That Supports Syntax-Highlighted Code | CSS-Tricks": code editor, javascript, css, html, syntax highlighting I also tried it with the ~20 links I have in my hoarder instance so far and all of them were tagged (previously 3 weren't). So if anybody has more links that did not work, please let me know, so I can try it out.

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@MohamedBassem commented on GitHub (May 16, 2024):

@kamtschatka the change you shared in general looks good to me. I'm happy to accept it in a PR if this improves ollama's accuracy :) I can also give it a try a bit with openAI.

@MohamedBassem commented on GitHub (May 16, 2024): @kamtschatka the change you shared in general looks good to me. I'm happy to accept it in a PR if this improves ollama's accuracy :) I can also give it a try a bit with openAI.

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@roadkingvrod commented on GitHub (May 17, 2024):

Hi,

I just set up Hoarder with OLLAMA. When it runs the inference jobs, i get errors such as:

2024-05-17T18:13:08.088Z info: [inference][68] Starting an inference job for bookmark with id "dzrjut9qb81couvfepd5pr7g"
2024-05-17T18:13:08.090Z info: [search][134] Attempting to index bookmark with id dzrjut9qb81couvfepd5pr7g ...
2024-05-17T18:13:08.099Z error: [inference][68] inference job failed: TypeError: fetch failed
2024-05-17T18:13:08.162Z info: [search][134] Completed successfully

I don't know where else to look for deeper log data, so any input is greatly appreciated. I'm running the models: llama3 and llava-llama3 for image.

Thanks!

@roadkingvrod commented on GitHub (May 17, 2024): Hi, I just set up Hoarder with OLLAMA. When it runs the inference jobs, i get errors such as: 2024-05-17T18:13:08.088Z info: [inference][68] Starting an inference job for bookmark with id "dzrjut9qb81couvfepd5pr7g" 2024-05-17T18:13:08.090Z info: [search][134] Attempting to index bookmark with id dzrjut9qb81couvfepd5pr7g ... 2024-05-17T18:13:08.099Z error: [inference][68] inference job failed: TypeError: fetch failed 2024-05-17T18:13:08.162Z info: [search][134] Completed successfully I don't know where else to look for deeper log data, so any input is greatly appreciated. I'm running the models: llama3 and llava-llama3 for image. Thanks!

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@kamtschatka commented on GitHub (May 17, 2024):

Sounds like it is not reachable, or running on the wrong port or something like that.
When I change the port to something that is incorrect i get the same output:

2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny"
2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed

@kamtschatka commented on GitHub (May 17, 2024): Sounds like it is not reachable, or running on the wrong port or something like that. When I change the port to something that is incorrect i get the same output: ``` 2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny" 2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed ```

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@roadkingvrod commented on GitHub (May 17, 2024):

Sounds like it is not reachable, or running on the wrong port or something like that. When I change the port to something that is incorrect i get the same output:
2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny"
2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed

Thank you! That gave me enough direction to figure out what was going on. Getting successes now.

@roadkingvrod commented on GitHub (May 17, 2024): > Sounds like it is not reachable, or running on the wrong port or something like that. When I change the port to something that is incorrect i get the same output: > > ``` > 2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny" > 2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed > ``` Thank you! That gave me enough direction to figure out what was going on. Getting successes now.

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@roadkingvrod commented on GitHub (May 22, 2024):

As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3.

I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions?

2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt"
2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging"

@roadkingvrod commented on GitHub (May 22, 2024): As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3. I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions? 2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt" 2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging"

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@roadkingvrod commented on GitHub (Jun 2, 2024):

As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3.

I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions?

2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt" 2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging"

Anything I can do to help troubleshoot this?

@roadkingvrod commented on GitHub (Jun 2, 2024): > As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3. > > I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions? > > 2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt" 2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging" Anything I can do to help troubleshoot this?

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@kamtschatka commented on GitHub (Jun 2, 2024):

Provide instructions on what you are doing/have configured and a sample image.
Are you on the latest version of everything?

I just tried to upload an image and got proper tagging, so it can't be a general issue.

@kamtschatka commented on GitHub (Jun 2, 2024): Provide instructions on what you are doing/have configured and a sample image. Are you on the latest version of everything? I just tried to upload an image and got proper tagging, so it can't be a general issue.

kerem commented

2026-03-02 11:45:41 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Jul 13, 2024):

It seems that ollama has been mostly stable with folks so far. So I'm closing this for now.

@MohamedBassem commented on GitHub (Jul 13, 2024): It seems that ollama has been mostly stable with folks so far. So I'm closing this for now.

Rows
Columns

[GH-ISSUE #20] Inference Failed with Ollama #18