[PR #868] [MERGED] Add jitter on the processor fetch backoff sleep #901

New issue

Closed

opened 2026-03-02 06:10:22 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 06:10:22 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/868
Author: @pior
Created: 4/22/2024
Status: ✅ Merged
Merged: 10/19/2024
Merged by: @kamikazechaser

Base: master ← Head: jitter

📝 Commits (1)

61d5816 Add jitter on the processor fetch backoff sleep

📊 Changes

1 file changed (+2 additions, -1 deletions)

View changed files

📝 processor.go (+2 -1)

📄 Description

Changes

This PR adds some jitter to the backoff sleep of the processor fetch loop to avoid workers synchronization, and thus improve worker efficiency and job latency.

Context

We run dozens of workers, with an execution rate of up to 5K jobs/sec.

We started to suspect that the job workers were significantly inefficient.

Changing the TaskCheckInterval from 1s (default) to 100ms has reduced the latency from ~500ms to 40ms on a high traffic worker pool, and 10ms to a low traffic worker pool.
Then, adding the jitter has further reduced the latency to 2 ms on both high and low traffic worker pool.

Synchronization on a distributed system is well known, and adding jitter helped avoid it.

Impact of adding a jitter: Queue size:
Screenshot 2024-04-15 at 19 05 32

Latency:
Screenshot 2024-04-15 at 19 05 51

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/868 **Author:** [@pior](https://github.com/pior) **Created:** 4/22/2024 **Status:** ✅ Merged **Merged:** 10/19/2024 **Merged by:** [@kamikazechaser](https://github.com/kamikazechaser) **Base:** `master` ← **Head:** `jitter` --- ### 📝 Commits (1) - [`61d5816`](https://github.com/hibiken/asynq/commit/61d581648dc38b2a5b271b534ae91a22e78e7120) Add jitter on the processor fetch backoff sleep ### 📊 Changes **1 file changed** (+2 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `processor.go` (+2 -1) </details> ### 📄 Description ## Changes This PR adds some jitter to the backoff sleep of the processor fetch loop to avoid workers synchronization, and thus improve worker efficiency and job latency. ## Context We run dozens of workers, with an execution rate of up to 5K jobs/sec. We started to suspect that the job workers were significantly inefficient. 1. Changing the TaskCheckInterval from 1s (default) to 100ms has reduced the latency from ~500ms to 40ms on a high traffic worker pool, and 10ms to a low traffic worker pool. 2. Then, adding the jitter has further reduced the latency to 2 ms on both high and low traffic worker pool. Synchronization on a distributed system is well known, and adding jitter helped avoid it. Impact of adding a jitter: Queue size: <img width="567" alt="Screenshot 2024-04-15 at 19 05 32" src="https://github.com/hibiken/asynq/assets/60219/5b2fc86e-5e13-42cf-9e2a-0e9dd8efe05c"> Latency: <img width="570" alt="Screenshot 2024-04-15 at 19 05 51" src="https://github.com/hibiken/asynq/assets/60219/a8e634f6-6224-455d-8a08-6743a2d81b20"> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>