[GH-ISSUE #998] [BUG] Some question about priority queue #2503

New issue

Closed

opened 2026-03-15 20:43:53 +03:00 by kerem · 1 comment

kerem commented

2026-03-15 20:43:53 +03:00

Owner

Originally created by @FelixSeptem on GitHub (Dec 30, 2024).
Original GitHub issue: https://github.com/hibiken/asynq/issues/998

Originally assigned to: @hibiken, @kamikazechaser on GitHub.

Describe the bug
When I choose to use the multi queue with priority, I was curious about the implementation mechanism. So I dig the code in, and seek it in the queues method of the processor, which is https://github.com/hibiken/asynq/blob/master/processor.go#L399

// queues returns a list of queues to query.
// Order of the queue names is based on the priority of each queue.
// Queue names is sorted by their priority level if strict-priority is true.
// If strict-priority is false, then the order of queue names are roughly based on
// the priority level but randomized in order to avoid starving low priority queues.
func (p *processor) queues() []string {
	// skip the overhead of generating a list of queue names
	// if we are processing one queue.
	if len(p.queueConfig) == 1 {
		for qname := range p.queueConfig {
			return []string{qname}
		}
	}
	if p.orderedQueues != nil {
		return p.orderedQueues
	}
	var names []string
	for qname, priority := range p.queueConfig {
		for i := 0; i < priority; i++ {
			names = append(names, qname)
		}
	}
	rand.Shuffle(len(names), func(i, j int) { names[i], names[j] = names[j], names[i] })
	return uniq(names, len(p.queueConfig))
}

I wondering priority variable only affects the composition of the names list and does not affect the final returned queue list. The final queue list is unduplicated and the queues in the configuration are selected with equal probability, rather than increasing the probability of some queues being selected by priority.

Suggestion
Modify uniq function to conclude priority infactors, I will happy to create a pull request if need.

Originally created by @FelixSeptem on GitHub (Dec 30, 2024). Original GitHub issue: https://github.com/hibiken/asynq/issues/998 Originally assigned to: @hibiken, @kamikazechaser on GitHub. **Describe the bug** When I choose to use the multi queue with priority, I was curious about the implementation mechanism. So I dig the code in, and seek it in the `queues` method of the `processor`, which is https://github.com/hibiken/asynq/blob/master/processor.go#L399 ```golang // queues returns a list of queues to query. // Order of the queue names is based on the priority of each queue. // Queue names is sorted by their priority level if strict-priority is true. // If strict-priority is false, then the order of queue names are roughly based on // the priority level but randomized in order to avoid starving low priority queues. func (p *processor) queues() []string { // skip the overhead of generating a list of queue names // if we are processing one queue. if len(p.queueConfig) == 1 { for qname := range p.queueConfig { return []string{qname} } } if p.orderedQueues != nil { return p.orderedQueues } var names []string for qname, priority := range p.queueConfig { for i := 0; i < priority; i++ { names = append(names, qname) } } rand.Shuffle(len(names), func(i, j int) { names[i], names[j] = names[j], names[i] }) return uniq(names, len(p.queueConfig)) } ``` I wondering `priority` variable only affects the composition of the names list and does not affect the final returned queue list. The final queue list is unduplicated and the queues in the configuration are selected with equal probability, rather than increasing the probability of some queues being selected by priority. **Suggestion** Modify `uniq` function to conclude priority infactors, I will happy to create a pull request if need.

kerem

2026-03-15 20:43:53 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-15 20:44:09 +03:00

Author

Owner

@ganlvtech commented on GitHub (Jul 7, 2025):

shuffle and uniq will sort the names by probabilities. It got random order by the priority. And the Dequeue operation is done one by one.

github.com/hibiken/asynq@c327bc40a2/internal/rdb/rdb.go (L240-L274)

If there are 3 names "a", "b", "c", and probabilities are a🅱️c, then you will get P(a_before_b):P(b_before_a) = a:b

I don't think there is a bug here.

@ganlvtech commented on GitHub (Jul 7, 2025): shuffle and uniq will sort the names by probabilities. It got random order by the `priority`. And the `Dequeue` operation is done one by one. https://github.com/hibiken/asynq/blob/c327bc40a28e4db45195cfe082d88faa808ce87d/internal/rdb/rdb.go#L240-L274 If there are 3 names "a", "b", "c", and probabilities are a:b:c, then you will get P(a_before_b):P(b_before_a) = a:b I don't think there is a bug here.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2503

No description provided.

Rows
Columns