[GH-ISSUE #657] [Bug] Timezone issue when using DB=postgres

kerem commented

2026-02-25 23:42:35 +03:00

Owner

Originally created by @RaphMad on GitHub (May 27, 2022).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/657

When running a dockerized healthchecks instance set to a timezone different than UTC, using DB=postgres with the DB instance running in the same non-UTC timezone and setting the monitored jobs to that same timezone:

Values for last_ping are consistently off "in the direction" of that timezone
E.g. when using GMT+2 in the case described above, manually performing a ping will register as 2 hours from now (so they register "in the future", and all grace timeout handling etc. seems to be also based on that future value)

Observations

When using DB=sqlite or DB=mysql the problem does not occur
Manually examining the last_ping values within the api_check table in postgres shows a correct value
So it seems the interpretation when reading back postgres' api_check.last_ping value is off compared to the other supported DB types (maybe a problem related to postgres' timestamptz type?)

(I hope the above makes sense, interpreting timezone issues just is soo confusing...)

Originally created by @RaphMad on GitHub (May 27, 2022). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/657 When running a dockerized healthchecks instance set to a timezone different than UTC, using `DB=postgres` with the DB instance running in the same non-UTC timezone and setting the monitored jobs to that same timezone: * Values for `last_ping` are consistently off "in the direction" of that timezone * E.g. when using `GMT+2` in the case described above, manually performing a ping will register as `2 hours from now` (so they register "in the future", and all grace timeout handling etc. seems to be also based on that future value) #### Observations * When using `DB=sqlite` or `DB=mysql` the problem does _not_ occur * Manually examining the `last_ping` values within the `api_check` table in postgres shows a correct value * So it seems the interpretation when reading back postgres' `api_check.last_ping` value is off compared to the other supported DB types (maybe a problem related to postgres' `timestamptz` type?) (I hope the above makes sense, interpreting timezone issues just is soo confusing...)

kerem closed this issue

2026-02-25 23:42:35 +03:00

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 28, 2022):

How and where are you setting the timezone?

Values for last_ping are consistently off "in the direction" of that timezone

Where do you see this? In the web UI, for example, in the ping log?

Can you show a specific example (wall clock time of making a ping, contents of the database row, screenshot of the web UI with the wrong time displayed)?

@cuu508 commented on GitHub (May 28, 2022): How and where are you setting the timezone? > Values for last_ping are consistently off "in the direction" of that timezone Where do you see this? In the web UI, for example, in the ping log? Can you show a specific example (wall clock time of making a ping, contents of the database row, screenshot of the web UI with the wrong time displayed)?

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 28, 2022):

How and where are you setting the timezone?

By mounting /etc/localtime and /etc/timezone in the respective containers.
Output inside both the healtchecks and the DB container (no timedatectl, so date seems like the next best thing):

_Where do you see this? In the web UI, for example, in the ping log?

Can you show a specific example (wall clock time of making a ping, contents of the database row, screenshot of the web UI with the wrong time displayed)?_

Scenario:

date output for both healthchecks as well as postgres is as described above
Manually triggering a ping (by opening the ping URL) right now (11:12 CEST)
Ping log will show this ping as though it had been received at _11:12 UTC (so 13:12 CEST, or 2 hours in the future)
Also all grace handling etc. treats this ping as "2 hours in the future", and will not trigger even for very strict grace periods, its not only visual

In the DB the stored value looks correct:

What strikes me so odd about this problem is that it will not occur in the exact same setup when switching to sqlite or mysql!

@RaphMad commented on GitHub (May 28, 2022): _How and where are you setting the timezone?_ By mounting `/etc/localtime` and `/etc/timezone` in the respective containers. Output inside both the healtchecks and the DB container (no `timedatectl`, so `date` seems like the next best thing): ![image](https://user-images.githubusercontent.com/5588470/170819130-a7395c30-d951-4f10-bb30-b5344fb19a2f.png) _Where do you see this? In the web UI, for example, in the ping log? Can you show a specific example (wall clock time of making a ping, contents of the database row, screenshot of the web UI with the wrong time displayed)?_ #### Scenario: * date output for both healthchecks as well as postgres is as described above * Manually triggering a ping (by opening the ping URL) _right now_ (11:12 CEST) * Ping log will show this ping as though it had been received at _11:12 _UTC_ (so 13:12 CEST, or 2 hours in the future) * Also all grace handling etc. treats this ping as "2 hours in the future", and will not trigger even for very strict grace periods, its not only visual ![image](https://user-images.githubusercontent.com/5588470/170819260-ed1d55af-a343-4922-9138-c1e9d05dd49c.png) * In the DB the stored value looks correct: ![image](https://user-images.githubusercontent.com/5588470/170819343-723ea299-e6a6-4437-a0ce-1e45d14a8e1a.png) What strikes me so odd about this problem is that it **will not occur in the exact same setup when switching to _sqlite_ or _mysql_**!

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 30, 2022):

By mounting /etc/localtime and /etc/timezone in the respective containers.

To investigate this, I'd like to be able to reproduce the issue myself. Can you please show me the steps (a Dockerfile, a docker-compose.yml?) on how you're setting it up. Specifically, how do you mount the /etc/localtime and /etc/timezone, and do you do it on the web container, on the database container or on both?

I wonder if it's a Healthchecks-specific issue. One thing I'd like to check would be to make a minimal Django project with a single model with a datetime field, and see if the problem can be reproduced there too.

@cuu508 commented on GitHub (May 30, 2022): > By mounting /etc/localtime and /etc/timezone in the respective containers. To investigate this, I'd like to be able to reproduce the issue myself. Can you please show me the steps (a Dockerfile, a docker-compose.yml?) on how you're setting it up. Specifically, how do you mount the `/etc/localtime` and `/etc/timezone`, and do you do it on the web container, on the database container or on both? I wonder if it's a Healthchecks-specific issue. One thing I'd like to check would be to make a minimal Django project with a single model with a datetime field, and see if the problem can be reproduced there too.

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 30, 2022):

I prepared 2 somewhat minimal compose files:

hc_mariadb

version: '3.9'

services:
  healthchecks:
    image: healthchecks/healthchecks
    container_name: hc_mariadb
    restart: unless-stopped
    environment:
      DB: mysql
      DB_HOST: mariadb
      DB_PORT: 3306
      DB_USER: healthchecks_user
      DB_NAME: healthchecks_db
      DB_PASSWORD: healthchecks_pass
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    ports:
      - 8000:8000
    depends_on:
        - mariadb

  mariadb:
    image: mariadb
    container_name: mariadb
    restart: unless-stopped
    environment:
      MARIADB_DATABASE: healthchecks_db
      MARIADB_USER: healthchecks_user
      MARIADB_PASSWORD: healthchecks_pass
      MARIADB_ROOT_PASSWORD: healthchecks_pass
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - healthchecks_mariadb:/var/lib/mysql/

volumes:
  healthchecks_mariadb:

hc_postgres

version: '3.9'

services:
  healthchecks:
    image: healthchecks/healthchecks
    container_name: hc_postgres
    restart: unless-stopped
    environment:
      DB: postgres
      DB_HOST: postgres
      DB_PORT: 5432
      DB_USER: healthchecks_user
      DB_NAME: healthchecks_db
      DB_PASSWORD: healthchecks_pass
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    ports:
      - 9000:8000
    depends_on:
        - postgres

  postgres:
    image: postgres
    container_name: postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: healthchecks_db
      POSTGRES_USER: healthchecks_user
      POSTGRES_PASSWORD: healthchecks_pass
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - healthchecks_postgres:/var/lib/postgresql/data/

volumes:
  healthchecks_postgres:

Preparation

Start on separate terminal with docker-compose up (no -d to see log output and potential errors)
(unrelated sidenote: Even with the depends_on the auto DB init can be finicky, on first start DBs may not get recognized and healthchecks stops retrying the DB connect, CTRL-C and run again in that case - https://github.com/vishnubob/wait-for-it or just programmatically waiting for DBs to become ready is the preferred solution for this)
Run docker exec -it hc_mariadb /opt/healthchecks/manage.py createsuperuser + docker exec -it hc_postgres /opt/healthchecks/manage.py createsuperuser to set up an initial user
Navigate to :8000, :9000 to see the different behaviour

Procedure

Output of cat /etc/timezone on docker host: Europe/Vienna (this may be crucial, but unfortunately also the only part not easily reproducable even with docker compose)
Navigate to :8000
Copy the "My first Healthcheck" link into a separate browser tab, trigger it
Observe the correct incoming ping received only some seconds ago:

Navigate to :9000
Copy the "My first Healthcheck" link into a separate browser tab, trigger it (care about the port, needs to be changed to 9000)
Observe the incorrect incoming ping, registers "one hour from now":

In the details it becomes clear that the last_ping is interpreted as UTC (it is 16:47 in Vienna at the point of time of this screenshot):

Setting to browser timezone shows that the ping is thought to be received "2 hours in the future":

@RaphMad commented on GitHub (May 30, 2022): I prepared 2 somewhat minimal compose files: #### hc_mariadb ``` version: '3.9' services: healthchecks: image: healthchecks/healthchecks container_name: hc_mariadb restart: unless-stopped environment: DB: mysql DB_HOST: mariadb DB_PORT: 3306 DB_USER: healthchecks_user DB_NAME: healthchecks_db DB_PASSWORD: healthchecks_pass volumes: - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro ports: - 8000:8000 depends_on: - mariadb mariadb: image: mariadb container_name: mariadb restart: unless-stopped environment: MARIADB_DATABASE: healthchecks_db MARIADB_USER: healthchecks_user MARIADB_PASSWORD: healthchecks_pass MARIADB_ROOT_PASSWORD: healthchecks_pass volumes: - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro - healthchecks_mariadb:/var/lib/mysql/ volumes: healthchecks_mariadb: ``` #### hc_postgres ``` version: '3.9' services: healthchecks: image: healthchecks/healthchecks container_name: hc_postgres restart: unless-stopped environment: DB: postgres DB_HOST: postgres DB_PORT: 5432 DB_USER: healthchecks_user DB_NAME: healthchecks_db DB_PASSWORD: healthchecks_pass volumes: - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro ports: - 9000:8000 depends_on: - postgres postgres: image: postgres container_name: postgres restart: unless-stopped environment: POSTGRES_DB: healthchecks_db POSTGRES_USER: healthchecks_user POSTGRES_PASSWORD: healthchecks_pass volumes: - /etc/localtime:/etc/localtime:ro - /etc/timezone:/etc/timezone:ro - healthchecks_postgres:/var/lib/postgresql/data/ volumes: healthchecks_postgres: ``` #### Preparation * Start on separate terminal with `docker-compose up` (no `-d` to see log output and potential errors) * (unrelated sidenote: Even with the `depends_on` the auto DB init can be finicky, on first start DBs may not get recognized and healthchecks stops retrying the DB connect, CTRL-C and run again in that case - `https://github.com/vishnubob/wait-for-it` or just programmatically waiting for DBs to become ready is the preferred solution for this) * Run `docker exec -it hc_mariadb /opt/healthchecks/manage.py createsuperuser` + `docker exec -it hc_postgres /opt/healthchecks/manage.py createsuperuser` to set up an initial user * Navigate to `:8000`, `:9000` to see the different behaviour #### Procedure * Output of `cat /etc/timezone` on docker host: `Europe/Vienna` (this may be crucial, but unfortunately also the only part not easily reproducable even with docker compose) * Navigate to `:8000` * Copy the "My first Healthcheck" link into a separate browser tab, trigger it * Observe the correct incoming ping received only some seconds ago: ![image](https://user-images.githubusercontent.com/5588470/171019418-e2ef4be1-449a-4e54-952d-2127cd04d535.png) * Navigate to `:9000` * Copy the "My first Healthcheck" link into a separate browser tab, trigger it (care about the port, needs to be changed to 9000) * Observe the _incorrect_ incoming ping, registers "one hour from now": ![image](https://user-images.githubusercontent.com/5588470/171019579-354c0035-5eee-4d99-bb11-1fc84cf60597.png) * In the details it becomes clear that the `last_ping` is interpreted as UTC (it is 16:47 in Vienna at the point of time of this screenshot): ![image](https://user-images.githubusercontent.com/5588470/171019730-e0d8cc98-1f10-4b82-854b-d5249bc6a7e6.png) * Setting to browser timezone shows that the ping is thought to be received "2 hours in the future": ![image](https://user-images.githubusercontent.com/5588470/171019893-27ec0bd0-5c35-498b-bf45-bfe169d7fb3e.png)

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 30, 2022):

Thanks for the detailed instructions! I can reproduce the issue and will see if I can track down what's going on.

My timezone is Europe/Riga, and I made a ping request at 18:14 local time. psql shows it (I think) correctly:

docker exec -it postgres psql -U healthchecks_user healthchecks_db
psql (14.3 (Debian 14.3-1.pgdg110+1))
Type "help" for help.

healthchecks_db=# select created from api_ping;
            created            
-------------------------------
 2022-05-30 18:14:16.792406+03
(1 row)

Django, somehow, gets it wrong:

docker exec -it hc_postgres /opt/healthchecks/manage.py shell
Python 3.9.9 (main, Dec 21 2021, 10:35:05) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from hc.api.models import Ping
>>> p = Ping.objects.get()
>>> p.created
datetime.datetime(2022, 5, 30, 18, 14, 16, 792406, tzinfo=datetime.timezone.utc)

@cuu508 commented on GitHub (May 30, 2022): Thanks for the detailed instructions! I can reproduce the issue and will see if I can track down what's going on. My timezone is `Europe/Riga`, and I made a ping request at 18:14 local time. psql shows it (I think) correctly: ``` docker exec -it postgres psql -U healthchecks_user healthchecks_db psql (14.3 (Debian 14.3-1.pgdg110+1)) Type "help" for help. healthchecks_db=# select created from api_ping; created ------------------------------- 2022-05-30 18:14:16.792406+03 (1 row) ``` Django, somehow, gets it wrong: ``` docker exec -it hc_postgres /opt/healthchecks/manage.py shell Python 3.9.9 (main, Dec 21 2021, 10:35:05) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from hc.api.models import Ping >>> p = Ping.objects.get() >>> p.created datetime.datetime(2022, 5, 30, 18, 14, 16, 792406, tzinfo=datetime.timezone.utc) ```

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 30, 2022):

Here's one observation. Postgres running directly on host:

hc=# set session timezone to 'Europe/Riga';
SET
hc=# select now();
              now              
-------------------------------
 2022-05-30 20:52:45.248653+03
(1 row)

hc=# set session timezone to 'UTC';
SET
hc=# select now();
              now              
-------------------------------
 2022-05-30 17:52:58.161483+00
(1 row)

And Postgres running inside the container:

healthchecks_db=# set session timezone to 'Europe/Riga';
SET
healthchecks_db=# select now();
              now              
-------------------------------
 2022-05-30 20:45:38.705525+03
(1 row)

healthchecks_db=# set session timezone to 'UTC';
SET
healthchecks_db=# select now();
              now              
-------------------------------
 2022-05-30 20:45:46.378756+03
(1 row)

Notice how it returns a timestamp with the same timezone both times.

When Django opens a database connection, the first thing it runs is SET TIME ZONE 'UTC'. I'm guessing after that point it assumes the datetimes will be in UTC, but they are not, and hence the problems.

@cuu508 commented on GitHub (May 30, 2022): Here's one observation. Postgres running directly on host: ``` hc=# set session timezone to 'Europe/Riga'; SET hc=# select now(); now ------------------------------- 2022-05-30 20:52:45.248653+03 (1 row) hc=# set session timezone to 'UTC'; SET hc=# select now(); now ------------------------------- 2022-05-30 17:52:58.161483+00 (1 row) ``` And Postgres running inside the container: ``` healthchecks_db=# set session timezone to 'Europe/Riga'; SET healthchecks_db=# select now(); now ------------------------------- 2022-05-30 20:45:38.705525+03 (1 row) healthchecks_db=# set session timezone to 'UTC'; SET healthchecks_db=# select now(); now ------------------------------- 2022-05-30 20:45:46.378756+03 (1 row) ``` Notice how it returns a timestamp with the same timezone both times. When Django opens a database connection, the first thing it runs is `SET TIME ZONE 'UTC'`. I'm guessing after that point it assumes the datetimes will be in UTC, but they are not, and hence the problems.

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 30, 2022):

In docker-compose.yml, if I comment out /etc/localtime and /etc/timezone mounts for the postgres container, then SET TIMEZONE calls work correctly, and Healthchecks indeed show the correct datetimes in the UI. To me it looks like the issue is with the database container being confused about what timezone to use.

I'd either ask for help at https://github.com/docker-library/postgres/ or stick with UTC and call it a day :-)

@cuu508 commented on GitHub (May 30, 2022): In `docker-compose.yml`, if I comment out `/etc/localtime` and `/etc/timezone` mounts for the postgres container, then `SET TIMEZONE` calls work correctly, and Healthchecks indeed show the correct datetimes in the UI. To me it looks like the issue is with the database container being confused about what timezone to use. I'd either ask for help at https://github.com/docker-library/postgres/ or stick with UTC and call it a day :-)

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 31, 2022):

I now found it to also work when when setting TZ=Europe/Vienna for the environment of the postgres container (and removing /etc/localtime / /etc/timezone).

This fixes the mentioned bug and allows things like date output, log timestamps etc. to still have the local timezone in the postgres container.

Mounting /etc/localtime / /etc/timezone usually is my default approach over the TZ env var because it automatically adapts to the docker host it is running on, it also is the first time I see different behaviour between the 2 approaches.

Anyway I think this can be closed because the problem was identified to be caused the the interpretation of timezones by postgres itself.

@RaphMad commented on GitHub (May 31, 2022): I now found it to also work when when setting `TZ=Europe/Vienna` for the environment of the postgres container (and removing `/etc/localtime` / `/etc/timezone`). This fixes the mentioned bug and allows things like `date` output, log timestamps etc. to still have the local timezone in the postgres container. Mounting `/etc/localtime` / `/etc/timezone` usually is my default approach over the `TZ` env var because it automatically adapts to the docker host it is running on, it also is the first time I see different behaviour between the 2 approaches. Anyway I think this can be closed because the problem was identified to be caused the the interpretation of timezones by `postgres` itself.

Rows
Columns

[GH-ISSUE #657] [Bug] Timezone issue when using DB=postgres #473

Observations

Scenario:

hc_mariadb

hc_postgres

Preparation

Procedure