starred/karakeep

Fork 0

mirror of https://github.com/karakeep-app/karakeep.git synced 2026-04-25 16:06:04 +03:00

[GH-ISSUE #1267] FR: A more lenient JSON parsing (allow markdown parsing) in inference structured calls #816

New issue

Closed

opened 2026-03-02 11:52:58 +03:00 by kerem · 3 comments

kerem commented

2026-03-02 11:52:58 +03:00

Owner

Originally created by @Nixellion on GitHub (Apr 15, 2025).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/1267

Describe the Bug

Can't get Ollama to work.

Ollama itself runs on a separate server that's accessible on LAN. Other services can reach it and are using it fine.

However in KaraKeep I get this:

2025-04-15T07:27:17.160Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
    at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3)
    at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25)
    at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271)
    at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305)
    at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133)
    at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410)
    at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868)
    at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656)
2025-04-15T07:27:17.171Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w"
2025-04-15T07:27:17.176Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
    at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3)
    at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25)
    at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271)
    at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305)
    at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133)
    at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410)
    at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868)
    at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656)
2025-04-15T07:27:17.189Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w"
2025-04-15T07:27:17.193Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
    at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3)
    at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25)
    at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271)
    at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305)
    at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133)
    at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410)
    at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868)
    at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656)
2025-04-15T07:27:17.206Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w"
2025-04-15T07:27:17.210Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string
    at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3)
    at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25)
    at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271)
    at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305)
    at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133)
    at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410)
    at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868)
    at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656)
2025-04-15T07:27:17.441Z info: [search][5] Completed successfully

Here's my compose:

version: "3.8"
services:
  web:
    image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release}
    restart: unless-stopped
    volumes:
      # By default, the data is stored in a docker volume called "data".
      # If you want to mount a custom directory, change the volume mapping to:
      # - /path/to/your/directory:/data
      - data:/data
    ports:
      - 3355:3000
    environment:
      MEILI_ADDR: http://meilisearch:7700
      BROWSER_WEB_URL: http://chrome:9222
      KARAKEEP_VERSION: release
      NEXTAUTH_URL: http://192.168.17.19:3355
      NEXTAUTH_SECRET: xxxxxxx
      MEILI_MASTER_KEY: xxxxxxx
      OLLAMA_BASE_URL: http://192.168.17.192:11434
      INFERENCE_TEXT_MODEL: qwen2.5-coder:7b-instruct-q4_0
      INFERENCE_IMAGE_MODEL: minicpm-v:latest
      INFERENCE_CONTEXT_LENGTH: 2048
      # OPENAI_API_KEY: ...

      # You almost never want to change the value of the DATA_DIR variable.
      # If you want to mount a custom directory, change the volume mapping above instead.
      DATA_DIR: /data # DON'T CHANGE THIS
  chrome:
    image: gcr.io/zenika-hub/alpine-chrome:123
    restart: unless-stopped
    command:
      - --no-sandbox
      - --disable-gpu
      - --disable-dev-shm-usage
      - --remote-debugging-address=0.0.0.0
      - --remote-debugging-port=9222
      - --hide-scrollbars
  meilisearch:
    image: getmeili/meilisearch:v1.13.3
    restart: unless-stopped
    environment:
      MEILI_NO_ANALYTICS: "true"
    volumes:
      - meilisearch:/meili_data

volumes:
  meilisearch:
  data:

Steps to Reproduce

Set OLLAMA up as per documentation
No worky

Expected Behaviour

Ollama worky

Screenshots or Additional Context

No response

Device Details

No response

Exact Karakeep Version

0.23.2

Have you checked the troubleshooting guide?

I have checked the troubleshooting guide and I haven't found a solution to my problem

Originally created by @Nixellion on GitHub (Apr 15, 2025). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/1267 ### Describe the Bug Can't get Ollama to work. Ollama itself runs on a separate server that's accessible on LAN. Other services can reach it and are using it fine. However in KaraKeep I get this: ``` 2025-04-15T07:27:17.160Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3) at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25) at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271) at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305) at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133) at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410) at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868) at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656) 2025-04-15T07:27:17.171Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w" 2025-04-15T07:27:17.176Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3) at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25) at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271) at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305) at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133) at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410) at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868) at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656) 2025-04-15T07:27:17.189Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w" 2025-04-15T07:27:17.193Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3) at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25) at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271) at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305) at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133) at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410) at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868) at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656) 2025-04-15T07:27:17.206Z info: [inference][4] Starting an inference job for bookmark with id "d9608fk7q3tigncah1osfn7w" 2025-04-15T07:27:17.210Z error: [inference][4] inference job failed: ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string ResponseError: json: cannot unmarshal object into Go struct field ChatRequest.format of type string at checkOk (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:77:9) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async post (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:141:3) at async Ollama.processStreamableRequest (/app/apps/workers/node_modules/.pnpm/ollama@0.5.14/node_modules/ollama/dist/browser.cjs:252:25) at async OllamaInferenceClient.runModel (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:4271) at async OllamaInferenceClient.inferFromText (/app/apps/workers/node_modules/.pnpm/@hoarder+shared@file+packages+shared_better-sqlite3@11.3.0/node_modules/@hoarder/shared/inference.ts:2:5305) at async inferTagsFromText (/app/apps/workers/openaiWorker.ts:6:3133) at async inferTags (/app/apps/workers/openaiWorker.ts:6:3410) at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868) at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656) 2025-04-15T07:27:17.441Z info: [search][5] Completed successfully ``` Here's my compose: ``` version: "3.8" services: web: image: ghcr.io/karakeep-app/karakeep:${KARAKEEP_VERSION:-release} restart: unless-stopped volumes: # By default, the data is stored in a docker volume called "data". # If you want to mount a custom directory, change the volume mapping to: # - /path/to/your/directory:/data - data:/data ports: - 3355:3000 environment: MEILI_ADDR: http://meilisearch:7700 BROWSER_WEB_URL: http://chrome:9222 KARAKEEP_VERSION: release NEXTAUTH_URL: http://192.168.17.19:3355 NEXTAUTH_SECRET: xxxxxxx MEILI_MASTER_KEY: xxxxxxx OLLAMA_BASE_URL: http://192.168.17.192:11434 INFERENCE_TEXT_MODEL: qwen2.5-coder:7b-instruct-q4_0 INFERENCE_IMAGE_MODEL: minicpm-v:latest INFERENCE_CONTEXT_LENGTH: 2048 # OPENAI_API_KEY: ... # You almost never want to change the value of the DATA_DIR variable. # If you want to mount a custom directory, change the volume mapping above instead. DATA_DIR: /data # DON'T CHANGE THIS chrome: image: gcr.io/zenika-hub/alpine-chrome:123 restart: unless-stopped command: - --no-sandbox - --disable-gpu - --disable-dev-shm-usage - --remote-debugging-address=0.0.0.0 - --remote-debugging-port=9222 - --hide-scrollbars meilisearch: image: getmeili/meilisearch:v1.13.3 restart: unless-stopped environment: MEILI_NO_ANALYTICS: "true" volumes: - meilisearch:/meili_data volumes: meilisearch: data: ``` ### Steps to Reproduce 1. Set OLLAMA up as per documentation 2. No worky ### Expected Behaviour 1. Ollama worky ### Screenshots or Additional Context _No response_ ### Device Details _No response_ ### Exact Karakeep Version 0.23.2 ### Have you checked the troubleshooting guide? - [x] I have checked the troubleshooting guide and I haven't found a solution to my problem

kerem

2026-03-02 11:52:58 +03:00

closed this issue
added the
bug

status/approved

pri/medium
labels

kerem commented

2026-03-02 11:52:59 +03:00

Author

Owner

@Nixellion commented on GitHub (Apr 15, 2025):

I was able to work around it by using OpenAI API endpoint instead, but changing config to:

      OPENAI_BASE_URL: "http://192.168.17.192:11434/v1"
      OPENAI_API_KEY: "none"
      INFERENCE_TEXT_MODEL: "qwen2.5-coder:7b-instruct-q4_0"
      INFERENCE_IMAGE_MODEL: "minicpm-v:latest"
      INFERENCE_CONTEXT_LENGTH: 2048

However I am now getting this:

2025-04-15T08:45:24.197Z error: [inference][24] inference job failed: Error: [inference][24] The model ignored our prompt and didn't respond with the expected JSON: {}. Here's a sneak peak from the response: ```json

{

  "tags": 

Error: [inference][24] The model ignored our prompt and didn't respond with the expected JSON: {}. Here's a sneak peak from the response: ```json

{

  "tags": 

    at inferTags (/app/apps/workers/openaiWorker.ts:6:4346)

    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

    at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868)

    at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656)

I might suggest making response parser that is more flexible and lenient to possible errors and formats that models may reply. While I understand that your prompts might be requesting a clean json in response, if you plan on supporting local models you should account for possible formatting inconsistencies.

In my own LLM library (python) I am handling it by using multiple regex expressions and checking each one. It's not looking for any specific expected output, but instead adapts to support all common ways models output JSON.

@Nixellion commented on GitHub (Apr 15, 2025): I was able to work around it by using OpenAI API endpoint instead, but changing config to: ``` OPENAI_BASE_URL: "http://192.168.17.192:11434/v1" OPENAI_API_KEY: "none" INFERENCE_TEXT_MODEL: "qwen2.5-coder:7b-instruct-q4_0" INFERENCE_IMAGE_MODEL: "minicpm-v:latest" INFERENCE_CONTEXT_LENGTH: 2048 ``` However I am now getting this: ``` 2025-04-15T08:45:24.197Z error: [inference][24] inference job failed: Error: [inference][24] The model ignored our prompt and didn't respond with the expected JSON: {}. Here's a sneak peak from the response: ```json { "tags": Error: [inference][24] The model ignored our prompt and didn't respond with the expected JSON: {}. Here's a sneak peak from the response: ```json { "tags": at inferTags (/app/apps/workers/openaiWorker.ts:6:4346) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async Object.runOpenAI [as run] (/app/apps/workers/openaiWorker.ts:6:6868) at async Runner.runOnce (/app/apps/workers/node_modules/.pnpm/liteque@0.3.2_better-sqlite3@11.3.0/node_modules/liteque/dist/runner.js:2:2656) ``` I might suggest making response parser that is more flexible and lenient to possible errors and formats that models may reply. While I understand that your prompts might be requesting a clean json in response, if you plan on supporting local models you should account for possible formatting inconsistencies. In my own LLM library (python) I am handling it by using multiple regex expressions and checking each one. It's not looking for any specific expected output, but instead adapts to support all common ways models output JSON.

kerem commented

2026-03-02 11:52:59 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Apr 15, 2025):

@Nixellion I'll check the ollama thing when I'm in front of a laptop. As for the json parsing error, I agree with you that we can be more lenient with the parsing. For the particular problem you're facing I've seen people having success with explicitly instructing the llm in the AI setting to output without any markdown formatting.

@MohamedBassem commented on GitHub (Apr 15, 2025): @Nixellion I'll check the ollama thing when I'm in front of a laptop. As for the json parsing error, I agree with you that we can be more lenient with the parsing. For the particular problem you're facing I've seen people having success with explicitly instructing the llm in the AI setting to output without any markdown formatting.

kerem commented

2026-03-02 11:52:59 +03:00

Author

Owner

@Nixellion commented on GitHub (Apr 15, 2025):

Ah, haven't found that setting yet. Thanks

@Nixellion commented on GitHub (Apr 15, 2025): Ah, haven't found that setting yet. Thanks

kerem referenced this issue

2026-03-02 11:58:47 +03:00

[PR #816] [MERGED] Fix openAI pricing link #1703

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/karakeep#816

No description provided.

Rows
Columns