[PR #973] Feat/dynamic queue with concurrency #958

New issue

Open

opened 2026-03-02 06:10:36 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 06:10:36 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/973
Author: @pcmid
Created: 11/20/2024
Status: 🔄 Open

Base: master ← Head: feat/dynamic_queue_with_concurrency

📝 Commits (4)

be15ef6 feat: concurrency control queue
2bace4c feat: dynamic queue concurrency
8d2955e fix test
a05cbd2 fix: add queue after server stopped

📊 Changes

10 files changed (+457 additions, -27 deletions)

View changed files

📝 aggregator.go (+5 -0)
📝 heartbeat.go (+1 -0)
📝 internal/base/base.go (+5 -3)
📝 internal/rdb/rdb.go (+47 -16)
📝 internal/rdb/rdb_test.go (+89 -1)
📝 internal/testbroker/testbroker.go (+6 -1)
📝 processor.go (+15 -3)
📝 server.go (+93 -3)
📝 server_test.go (+193 -0)
📝 subscriber.go (+3 -0)

📄 Description

Implemented #250, based on #879, cc @kanzihuang, with support for dynamically adding queues and controlling concurrency levels.

~~Not fully verified, but~~ it works perfectly for my use case: a distributed spider with concurrency control.

Known issues:

We need a global lock to protect the server’s restart operation.
~~When the server is running, setting the queue concurrency does not take effect.~~

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/973 **Author:** [@pcmid](https://github.com/pcmid) **Created:** 11/20/2024 **Status:** 🔄 Open **Base:** `master` ← **Head:** `feat/dynamic_queue_with_concurrency` --- ### 📝 Commits (4) - [`be15ef6`](https://github.com/hibiken/asynq/commit/be15ef61d63e30cacf2df431de8e407899cf959e) feat: concurrency control queue - [`2bace4c`](https://github.com/hibiken/asynq/commit/2bace4cce4c4835c850e8a58122588730edeca28) feat: dynamic queue concurrency - [`8d2955e`](https://github.com/hibiken/asynq/commit/8d2955efc17b9eec3ca8240656ba073081048ecd) fix test - [`a05cbd2`](https://github.com/hibiken/asynq/commit/a05cbd2e387186d3e12b93d939a9e6b7be8a6b5c) fix: add queue after server stopped ### 📊 Changes **10 files changed** (+457 additions, -27 deletions) <details> <summary>View changed files</summary> 📝 `aggregator.go` (+5 -0) 📝 `heartbeat.go` (+1 -0) 📝 `internal/base/base.go` (+5 -3) 📝 `internal/rdb/rdb.go` (+47 -16) 📝 `internal/rdb/rdb_test.go` (+89 -1) 📝 `internal/testbroker/testbroker.go` (+6 -1) 📝 `processor.go` (+15 -3) 📝 `server.go` (+93 -3) 📝 `server_test.go` (+193 -0) 📝 `subscriber.go` (+3 -0) </details> ### 📄 Description Implemented #250, based on [#879](https://github.com/hibiken/asynq/pull/879), cc @kanzihuang, with support for dynamically adding queues and controlling concurrency levels. ~~Not fully verified, but~~ it works perfectly for my use case: a distributed spider with concurrency control. Known issues: - [x] We need a global lock to protect the server’s restart operation. - [x] ~~When the server is running, setting the queue concurrency does not take effect.~~ --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>