[GH-ISSUE #549] Improve performance of enqueueing tasks #261

New issue

Closed

opened 2026-03-02 05:20:00 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 05:20:00 +03:00

Owner

Originally created by @yousifh on GitHub (Sep 25, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/549

When load testing the library at high load, there is opportunity to optimize the enqueue code path. It's currently doing two things

Add queue name to the set "asynq:queues"
Enqueue the task

The first call is only necessary the first time, after that the extra network is just wasting resources. We can optimize that by caching which queues we enqueued to and only run the first call whenever the cache expires. On aggregate this will reduce the network calls, Redis commands, and CPU usage of the service and Redis itself.

On a load test of 100k tasks/min and 16 producer instances

	Before	After
Producer CPU	195 mcores	173 mcores
Redis CPU	290 ms	222 ms
Redis cmds/sec	8300	6600

The PR will highlight the change but it's essentially maintaining a map of queue names to their expiration and refreshing each item every 10 secs. On aggregate this heavily reduces the number of network calls needed as shown above.

The cache has to expire entries because if a queue is deleted, it will be removed from "asynq:queues" set so next time we enqueue to that queue, we will need to add the queue name there.

Originally created by @yousifh on GitHub (Sep 25, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/549 When load testing the library at high load, there is opportunity to optimize the enqueue code path. It's currently doing two things - Add queue name to the set "asynq:queues" - Enqueue the task The first call is only necessary the first time, after that the extra network is just wasting resources. We can optimize that by caching which queues we enqueued to and only run the first call whenever the cache expires. On aggregate this will reduce the network calls, Redis commands, and CPU usage of the service and Redis itself. On a load test of 100k tasks/min and 16 producer instances | | Before | After | | --- | ---- | --- | | Producer CPU | 195 mcores | 173 mcores | | Redis CPU | 290 ms | 222 ms | | Redis cmds/sec | 8300 | 6600 | The PR will highlight the change but it's essentially maintaining a map of queue names to their expiration and refreshing each item every 10 secs. On aggregate this heavily reduces the number of network calls needed as shown above. The cache has to expire entries because if a queue is deleted, it will be removed from "asynq:queues" set so next time we enqueue to that queue, we will need to add the queue name there.