starred/karakeep

Fork 0

mirror of https://github.com/karakeep-app/karakeep.git synced 2026-04-25 07:56:05 +03:00

[GH-ISSUE #667] Gateway Timeouts when running with Traefik Reverse Proxy #431

New issue

Closed

opened 2026-03-02 11:49:47 +03:00 by kerem · 7 comments

kerem commented

2026-03-02 11:49:47 +03:00

Owner

Originally created by @PhilipJonasFranz on GitHub (Nov 17, 2024).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/667

Describe the Bug

Im currently trying to setup the service using the following compose file:

version: "3.8"
services:
  web:
    image: ghcr.io/hoarder-app/hoarder:${HOARDER_VERSION:-release}
    restart: unless-stopped
    volumes:
      - ./data/web:/data
    env_file:
      - .env
    environment:
      MEILI_ADDR: http://meilisearch:7700
      BROWSER_WEB_URL: http://chrome:9222
      # OPENAI_API_KEY: ...
      DATA_DIR: /data
    networks:
      - internal
      - proxy
    labels:
      - "traefik.enable=true"
      
      - "traefik.http.routers.hoarder.entrypoints=http"
      - "traefik.http.routers.hoarder.rule=Host(`hoarder.mydomain.tld`)"
      - "traefik.http.routers.hoarder.priority=1000"
      - "traefik.http.middlewares.hoarder-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.hoarder.middlewares=hoarder-https-redirect"
      
      - "traefik.http.routers.hoarder-secure.entrypoints=https"
      - "traefik.http.routers.hoarder-secure.rule=Host(`hoarder.mydomain.tld`)"
      - "traefik.http.routers.hoarder-secure.priority=1000"
      - "traefik.http.routers.hoarder-secure.tls=true"
      - "traefik.http.routers.hoarder-secure.service=hoarder"
      - "traefik.http.services.hoarder.loadbalancer.server.port=3000"
      - "traefik.http.routers.hoarder-secure.middlewares=local-whitelist@file"
  
  chrome:
    image: gcr.io/zenika-hub/alpine-chrome:123
    restart: unless-stopped
    command:
      - --no-sandbox
      - --disable-gpu
      - --disable-dev-shm-usage
      - --remote-debugging-address=0.0.0.0
      - --remote-debugging-port=9222
      - --hide-scrollbars
    networks:
      - internal
  
  meilisearch:
    image: getmeili/meilisearch:v1.11.1
    restart: unless-stopped
    env_file:
      - .env
    environment:
      MEILI_NO_ANALYTICS: "true"
    volumes:
      - ./data/meilisearch:/meili_data
    networks:
      - internal

networks:
  proxy:
    external: true
  internal:

The reverse proxy setup works for a bunch of services, but i cannot get it to work for this one. I keep getting 504 Gateway Timeouts. I have set the NEXTAUTH_URL to point to the base-url of the service.

Strangely, after some time of trying to connect to the service, it randomly works, and i get this error message in the web-container log:

[next-auth][error][CLIENT_FETCH_ERROR] 
https://next-auth.js.org/errors#client_fetch_error Unexpected token 'F', "Forbidden" is not valid JSON {
  error: {
    message: `Unexpected token 'F', "Forbidden" is not valid JSON`,
    stack: `SyntaxError: Unexpected token 'F', "Forbidden" is not valid JSON\n` +
      '    at JSON.parse (<anonymous>)\n' +
      '    at parseJSONFromBytes (node:internal/deps/undici/undici:5682:19)\n' +
      '    at successSteps (node:internal/deps/undici/undici:5663:27)\n' +
      '    at fullyReadBody (node:internal/deps/undici/undici:4561:9)\n' +
      '    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n' +
      '    at async consumeBody (node:internal/deps/undici/undici:5672:7)',
    name: 'SyntaxError'
  },
  url: 'https://hoarder.mydomain.tld/api/auth/providers',
  message: `Unexpected token 'F', "Forbidden" is not valid JSON`
}
}

I can then connect to the dashboard, sign in and navigate pages, until it suddenly stops working and i get a gateway timeout again. Then im stuck with gateway timeouts again and cannot access the dashboard.

Steps to Reproduce

Deploy the service with the provided compose file
Open Firefox and heat to https://hoarder.mydomain.tld
Wait a few seconds to get gateway timeout

Expected Behaviour

The service should load correctly

Screenshots or Additional Context

No response

Device Details

Firefox 132.0.2 (64-bit), Windows 10

Exact Hoarder Version

v.0.19.0

Originally created by @PhilipJonasFranz on GitHub (Nov 17, 2024). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/667 ### Describe the Bug Im currently trying to setup the service using the following compose file: ```yml version: "3.8" services: web: image: ghcr.io/hoarder-app/hoarder:${HOARDER_VERSION:-release} restart: unless-stopped volumes: - ./data/web:/data env_file: - .env environment: MEILI_ADDR: http://meilisearch:7700 BROWSER_WEB_URL: http://chrome:9222 # OPENAI_API_KEY: ... DATA_DIR: /data networks: - internal - proxy labels: - "traefik.enable=true" - "traefik.http.routers.hoarder.entrypoints=http" - "traefik.http.routers.hoarder.rule=Host(`hoarder.mydomain.tld`)" - "traefik.http.routers.hoarder.priority=1000" - "traefik.http.middlewares.hoarder-https-redirect.redirectscheme.scheme=https" - "traefik.http.routers.hoarder.middlewares=hoarder-https-redirect" - "traefik.http.routers.hoarder-secure.entrypoints=https" - "traefik.http.routers.hoarder-secure.rule=Host(`hoarder.mydomain.tld`)" - "traefik.http.routers.hoarder-secure.priority=1000" - "traefik.http.routers.hoarder-secure.tls=true" - "traefik.http.routers.hoarder-secure.service=hoarder" - "traefik.http.services.hoarder.loadbalancer.server.port=3000" - "traefik.http.routers.hoarder-secure.middlewares=local-whitelist@file" chrome: image: gcr.io/zenika-hub/alpine-chrome:123 restart: unless-stopped command: - --no-sandbox - --disable-gpu - --disable-dev-shm-usage - --remote-debugging-address=0.0.0.0 - --remote-debugging-port=9222 - --hide-scrollbars networks: - internal meilisearch: image: getmeili/meilisearch:v1.11.1 restart: unless-stopped env_file: - .env environment: MEILI_NO_ANALYTICS: "true" volumes: - ./data/meilisearch:/meili_data networks: - internal networks: proxy: external: true internal: ``` The reverse proxy setup works for a bunch of services, but i cannot get it to work for this one. I keep getting 504 Gateway Timeouts. I have set the NEXTAUTH_URL to point to the base-url of the service. Strangely, after some time of trying to connect to the service, it randomly works, and i get this error message in the web-container log: ``` [next-auth][error][CLIENT_FETCH_ERROR] https://next-auth.js.org/errors#client_fetch_error Unexpected token 'F', "Forbidden" is not valid JSON { error: { message: `Unexpected token 'F', "Forbidden" is not valid JSON`, stack: `SyntaxError: Unexpected token 'F', "Forbidden" is not valid JSON\n` + ' at JSON.parse (<anonymous>)\n' + ' at parseJSONFromBytes (node:internal/deps/undici/undici:5682:19)\n' + ' at successSteps (node:internal/deps/undici/undici:5663:27)\n' + ' at fullyReadBody (node:internal/deps/undici/undici:4561:9)\n' + ' at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n' + ' at async consumeBody (node:internal/deps/undici/undici:5672:7)', name: 'SyntaxError' }, url: 'https://hoarder.mydomain.tld/api/auth/providers', message: `Unexpected token 'F', "Forbidden" is not valid JSON` } } ``` I can then connect to the dashboard, sign in and navigate pages, until it suddenly stops working and i get a gateway timeout again. Then im stuck with gateway timeouts again and cannot access the dashboard. ### Steps to Reproduce - Deploy the service with the provided compose file - Open Firefox and heat to https://hoarder.mydomain.tld - Wait a few seconds to get gateway timeout ### Expected Behaviour - The service should load correctly ### Screenshots or Additional Context _No response_ ### Device Details Firefox 132.0.2 (64-bit), Windows 10 ### Exact Hoarder Version v.0.19.0

kerem

2026-03-02 11:49:47 +03:00

closed this issue
added the
question
label

kerem commented

2026-03-02 11:49:48 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Nov 17, 2024):

hmmm, the error message indicates that something is responding with forbidden to nextauth's request (and it's probably not hoarder). What's this middleware that you have configured? What does it do? Have you tried commenting this middleware out as an experiment?

@MohamedBassem commented on GitHub (Nov 17, 2024): hmmm, the error message indicates that something is responding with forbidden to nextauth's request (and it's probably not hoarder). What's this middleware that you have configured? What does it do? Have you tried commenting this middleware out as an experiment?

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@PhilipJonasFranz commented on GitHub (Nov 17, 2024):

Hi,

thanks for the quick reply. The Middleware is a simple IP-Whitelist Filter:

    local-whitelist:
      ipWhiteList:
        sourceRange:
        # Local IP Ranges
        - "10.0.0.0/8"

I have tried commenting it out but sadly it does not change the behaviour. Its also worth noting that if i run the service without traefik and expose port 3000, i at least can connect to the dashboard without issues via IP:Port. The SSL Cert that Traefik serves is valid and not expired.

@PhilipJonasFranz commented on GitHub (Nov 17, 2024): Hi, thanks for the quick reply. The Middleware is a simple IP-Whitelist Filter: ```yml local-whitelist: ipWhiteList: sourceRange: # Local IP Ranges - "10.0.0.0/8" ``` I have tried commenting it out but sadly it does not change the behaviour. Its also worth noting that if i run the service without traefik and expose port 3000, i at least can connect to the dashboard without issues via IP:Port. The SSL Cert that Traefik serves is valid and not expired.

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Nov 18, 2024):

hmmm, ok so it working without traefik means that it's a traefik configuration issue.

I'm not familiar with traefik, but what does hoarder in this line supposed to point to?

traefik.http.routers.hoarder-secure.service=hoarder

@MohamedBassem commented on GitHub (Nov 18, 2024): hmmm, ok so it working without traefik means that it's a traefik configuration issue. I'm not familiar with traefik, but what does `hoarder` in this line supposed to point to? ``` traefik.http.routers.hoarder-secure.service=hoarder ```

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@PhilipJonasFranz commented on GitHub (Nov 18, 2024):

In the setup, hoarder-secure is the router for the SSL-secured endpoint. Every incoming request that matches the Host(hoarder.mydomain.tld) rule gets forwarded to the hoarder service, and traefik performs SSL termination. hoarder-secure is implicitly defined. But as stated previously, i use this traefik configuration on about 30 other services without issues, so im fairly confident the configuration in itself is not the issue, rather than some weird interaction between traefik and the service. Maybe the SSL termination?

Edit: same issue without SSL via Traefik.

@PhilipJonasFranz commented on GitHub (Nov 18, 2024): In the setup, `hoarder-secure` is the router for the SSL-secured endpoint. Every incoming request that matches the `Host(hoarder.mydomain.tld)` rule gets forwarded to the `hoarder` service, and traefik performs SSL termination. `hoarder-secure` is implicitly defined. But as stated previously, i use this traefik configuration on about 30 other services without issues, so im fairly confident the configuration in itself is not the issue, rather than some weird interaction between traefik and the service. Maybe the SSL termination? Edit: same issue without SSL via Traefik.

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Nov 18, 2024):

did you confirm you can ping the hoarder container from inside the traefik container btw?

@MohamedBassem commented on GitHub (Nov 18, 2024): did you confirm you can ping the hoarder container from inside the traefik container btw?

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@MohamedBassem commented on GitHub (Nov 18, 2024):

Also found this: https://community.traefik.io/t/traefik-gateway-timeout/17889

Maybe you need to tell traefik which network to use with: providers.docker.network

@MohamedBassem commented on GitHub (Nov 18, 2024): Also found this: https://community.traefik.io/t/traefik-gateway-timeout/17889 Maybe you need to tell traefik which network to use with: providers.docker.network

kerem commented

2026-03-02 11:49:49 +03:00

Author

Owner

@PhilipJonasFranz commented on GitHub (Nov 18, 2024):

Hello again,

I think it is now working correctly, no issues so far. I added the traefik.docker.network=proxy label and it started working. I have no idea why to be honest. For all other services it worked flawlessly without it. But many thanks for your answer, it helped me a lot!

Cheers!

@PhilipJonasFranz commented on GitHub (Nov 18, 2024): Hello again, I think it is now working correctly, no issues so far. I added the `traefik.docker.network=proxy` label and it started working. I have no idea why to be honest. For all other services it worked flawlessly without it. But many thanks for your answer, it helped me a lot! Cheers!

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/karakeep#431

No description provided.

Rows
Columns