[GH-ISSUE #198] [PERF] High CPU consumption when load testing #1081

New issue

Closed

opened 2026-03-07 22:05:23 +03:00 by kerem · 1 comment

kerem commented

2026-03-07 22:05:23 +03:00

Owner

Originally created by @sujit-baniya on GitHub (Sep 18, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/198

Originally assigned to: @hibiken on GitHub.

I tried to do load testing and CPU for worker (above 100%), redis and clients were above 60%

https://www.dropbox.com/s/o38apit2ulb44cp/Screen%20Shot%202020-09-18%20at%2018.07.29.png?dl=0

// client.go
func main() {
	r := asynq.RedisClientOpt{Addr: "localhost:6379"}
	client := asynq.NewClient(r)
	start := time.Now()
	for i := 0; i < 1000000; i++ {
		// Create a task with typename and payload.
		t1 := asynq.NewTask("email:welcome", map[string]interface{}{"user_id": i})

		// Process the task immediately.
		_, err := client.Enqueue(t1)
		if err != nil {
			log.Fatal(err)
		}
	}
	fmt.Println(fmt.Sprintf("%s", time.Since(start)))
}

// workers.go
func main() {
	r := asynq.RedisClientOpt{Addr: "localhost:6379"}
	srv := asynq.NewServer(r, asynq.Config{
		Concurrency: 10,
		Logger: nil,
	})

	mux := asynq.NewServeMux()
	mux.HandleFunc(WelcomeEmail, HandleWelcomeEmailTask)
	mux.HandleFunc(ReminderEmail, HandleReminderEmailTask)
	if err := srv.Run(mux); err != nil {
		log.Fatal(err)
	}
}

Originally created by @sujit-baniya on GitHub (Sep 18, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/198 Originally assigned to: @hibiken on GitHub. I tried to do load testing and CPU for worker (above 100%), redis and clients were above 60% https://www.dropbox.com/s/o38apit2ulb44cp/Screen%20Shot%202020-09-18%20at%2018.07.29.png?dl=0 ```go // client.go func main() { r := asynq.RedisClientOpt{Addr: "localhost:6379"} client := asynq.NewClient(r) start := time.Now() for i := 0; i < 1000000; i++ { // Create a task with typename and payload. t1 := asynq.NewTask("email:welcome", map[string]interface{}{"user_id": i}) // Process the task immediately. _, err := client.Enqueue(t1) if err != nil { log.Fatal(err) } } fmt.Println(fmt.Sprintf("%s", time.Since(start))) } ``` ```go // workers.go func main() { r := asynq.RedisClientOpt{Addr: "localhost:6379"} srv := asynq.NewServer(r, asynq.Config{ Concurrency: 10, Logger: nil, }) mux := asynq.NewServeMux() mux.HandleFunc(WelcomeEmail, HandleWelcomeEmailTask) mux.HandleFunc(ReminderEmail, HandleReminderEmailTask) if err := srv.Run(mux); err != nil { log.Fatal(err) } } ```

kerem

2026-03-07 22:05:23 +03:00

closed this issue
added the
performance
label

kerem commented

2026-03-07 22:05:25 +03:00

Author

Owner

@hibiken commented on GitHub (Sep 20, 2020):

Thank you for opening this issue.

CPU usage above 100% was most likely due to the worker process using more than one core on the machine.
With this simulated load, your Redis server is being slammed by many requests. Using redis cluster instead to balance the load across multiple instances may help.

I will look into a way to minimize the CPU usage by the package.

@hibiken commented on GitHub (Sep 20, 2020): Thank you for opening this issue. CPU usage above 100% was most likely due to the worker process using more than one core on the machine. With this simulated load, your Redis server is being slammed by many requests. Using redis cluster instead to balance the load across multiple instances may help. I will look into a way to minimize the CPU usage by the package.

kerem referenced this issue

2026-03-07 22:11:02 +03:00

[GH-ISSUE #1081] how to delete Recent History Enqueued in scheduler? #1533

kerem referenced this issue

2026-03-15 20:49:41 +03:00

[GH-ISSUE #1081] how to delete Recent History Enqueued in scheduler? #2544

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1081

No description provided.

Rows
Columns