[GH-ISSUE #649] Check API works Intermittently #469

New issue

Closed

opened 2026-02-25 23:42:34 +03:00 by kerem · 4 comments

kerem commented

2026-02-25 23:42:34 +03:00

Owner

Originally created by @texasbobs on GitHub (May 6, 2022).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/649

During our evaluation of this tool, we wanted to see how it would scale. We will have about 2200 checks that need to be created. I wrote a basic script to hit the check API 2200 times. It worked at first but then started returning 403 errors and an occasional success message.

[06/May/2022 15:39:54] "POST /api/v1/checks/ HTTP/1.1" 403 0
Forbidden: /api/v1/checks/
[06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0
Forbidden: /api/v1/checks/
[06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0
[06/May/2022 15:39:55] "GET /projects/515cfd68-c098-44d1-9fc5-b2076850eb4c/checks/status/ HTTP/1.1" 200 61609
Forbidden: /api/v1/checks/
[06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0
Forbidden: /api/v1/checks/
[06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0
Forbidden: /api/v1/checks/
[06/May/2022 15:39:56] "POST /api/v1/checks/ HTTP/1.1" 403 0

The script just does this:

for storelist in $(ansible-inventory --export --list -i stores.yml | jq -r .stores.children  | cut -c 4-9)
do
    curl http://10.x.x.x:8000/api/v1/checks/ \
    --header "X-Api-Key: ,redacted" \
    --data '{"name": "'$storelist'", "tags": "test store monarch-run '$storelist'", "timeout": 3600, "grace": 60}'
done

In the end, only 500 checks were created.

Originally created by @texasbobs on GitHub (May 6, 2022). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/649 During our evaluation of this tool, we wanted to see how it would scale. We will have about 2200 checks that need to be created. I wrote a basic script to hit the check API 2200 times. It worked at first but then started returning 403 errors and an occasional success message. ``` [06/May/2022 15:39:54] "POST /api/v1/checks/ HTTP/1.1" 403 0 Forbidden: /api/v1/checks/ [06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0 Forbidden: /api/v1/checks/ [06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0 [06/May/2022 15:39:55] "GET /projects/515cfd68-c098-44d1-9fc5-b2076850eb4c/checks/status/ HTTP/1.1" 200 61609 Forbidden: /api/v1/checks/ [06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0 Forbidden: /api/v1/checks/ [06/May/2022 15:39:55] "POST /api/v1/checks/ HTTP/1.1" 403 0 Forbidden: /api/v1/checks/ [06/May/2022 15:39:56] "POST /api/v1/checks/ HTTP/1.1" 403 0 ``` The script just does this: ``` for storelist in $(ansible-inventory --export --list -i stores.yml | jq -r .stores.children | cut -c 4-9) do curl http://10.x.x.x:8000/api/v1/checks/ \ --header "X-Api-Key: ,redacted" \ --data '{"name": "'$storelist'", "tags": "test store monarch-run '$storelist'", "timeout": 3600, "grace": 60}' done ``` In the end, only 500 checks were created.

kerem closed this issue

2026-02-25 23:42:34 +03:00

kerem commented

2026-02-25 23:42:35 +03:00

Author

Owner

@cuu508 commented on GitHub (May 6, 2022):

On self-hosted instances, the default check limit for each account is 500 checks. You can increase it if you log into Django admin (/admin/), go to "Profiles", look up the profile you are testing with, and edit its "Check limit" field.

@cuu508 commented on GitHub (May 6, 2022): On self-hosted instances, the default check limit for each account is 500 checks. You can increase it if you log into Django admin (`/admin/`), go to "Profiles", look up the profile you are testing with, and edit its "Check limit" field.

kerem commented

2026-02-25 23:42:35 +03:00

Author

Owner

@texasbobs commented on GitHub (May 6, 2022):

That seems to have solved the problem. It is interesting that in the middle of the 403 errors, one would work every once in a while. Also, the 403 might not be the best message for reaching that limit. But in any case, I was able to add all of the checks. Thanks.

@texasbobs commented on GitHub (May 6, 2022): That seems to have solved the problem. It is interesting that in the middle of the 403 errors, one would work every once in a while. Also, the 403 might not be the best message for reaching that limit. But in any case, I was able to add all of the checks. Thanks.

kerem commented

2026-02-25 23:42:35 +03:00

Author

Owner

@cuu508 commented on GitHub (May 6, 2022):

It is interesting that in the middle of the 403 errors, one would work every once in a while.

That is unexpected. Can you confirm the 200 response were also for POST /api/v1/checks/? Perhaps you had Healthchecks open in a browser tab, and the page was auto-refreshing?

Also, the 403 might not be the best message for reaching that limit.

Any suggestions what HTTP response status code to use instead?

PS. The response codes are documented in Management API docs: https://healthchecks.io/docs/api/#create-check

@cuu508 commented on GitHub (May 6, 2022): > It is interesting that in the middle of the 403 errors, one would work every once in a while. That *is* unexpected. Can you confirm the 200 response were also for `POST /api/v1/checks/`? Perhaps you had Healthchecks open in a browser tab, and the page was auto-refreshing? > Also, the 403 might not be the best message for reaching that limit. Any suggestions what HTTP response status code to use instead? PS. The response codes are documented in Management API docs: https://healthchecks.io/docs/api/#create-check

kerem commented

2026-02-25 23:42:35 +03:00

Author

Owner

@texasbobs commented on GitHub (May 6, 2022):

Any suggestions what HTTP response status code to use instead?

PS. The response codes are documented in Management API docs: https://healthchecks.io/docs/api/#create-check

Thanks for the link.

In most places, the 403 indicates a problem with the token. It's fine though, just not what I was intuitively expecting.

@texasbobs commented on GitHub (May 6, 2022): > Any suggestions what HTTP response status code to use instead? > > PS. The response codes are documented in Management API docs: https://healthchecks.io/docs/api/#create-check Thanks for the link. In most places, the 403 indicates a problem with the token. It's fine though, just not what I was intuitively expecting.