starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 07:25:56 +03:00

[GH-ISSUE #309] [BUG] asynq client failed when use command `stats` #1144

New issue

Closed

opened 2026-03-07 22:06:24 +03:00 by kerem · 5 comments

kerem commented

2026-03-07 22:06:24 +03:00

Owner

Originally created by @duyhungtnn on GitHub (Aug 7, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/309

Originally assigned to: @hibiken on GitHub.

Describe the bug

After connect asynq with our Redis server, run a command asynq stats. It take so long to complete and end up with

ryan@ ~ asynq stats

Using config file: /Users/ryan/.asynq.yml

UNKNOWN: UNKNOWN: redis command error: MEMORY USAGE failed: redis: nil

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):
0. Setup redis with docker-compose

version: '3.1'

services:
  redis:
    container_name: abc_redis
    image: redis:6.2.5-alpine
    command: redis-server --requirepass ${REDIS_PASSWORD}
    environment:
      - ALLOW_EMPTY_PASSWORD=no
      - REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL
    ports:
      - '6379:6379'
    volumes:
      - './data:/data'

networks:
  default:
    external:
      name: abc

Setup background processing ...
Enqueue tasks ...
See Error ...

Expected behavior

Need to see the stats of the system

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

OS: [Ubuntu 18.4]
Version of asynq package [v0.18.2]

Additional context

The worker and scheduler work as normal
The data on Redis is okay too

Originally created by @duyhungtnn on GitHub (Aug 7, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/309 Originally assigned to: @hibiken on GitHub. **Describe the bug** - After connect asynq with our Redis server, run a command `asynq stats`. It take so long to complete and end up with ``` ryan@ ~ asynq stats Using config file: /Users/ryan/.asynq.yml UNKNOWN: UNKNOWN: redis command error: MEMORY USAGE failed: redis: nil ``` **To Reproduce** Steps to reproduce the behavior (Code snippets if applicable): 0. Setup redis with docker-compose ``` version: '3.1' services: redis: container_name: abc_redis image: redis:6.2.5-alpine command: redis-server --requirepass ${REDIS_PASSWORD} environment: - ALLOW_EMPTY_PASSWORD=no - REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL ports: - '6379:6379' volumes: - './data:/data' networks: default: external: name: abc ``` 1. Setup background processing ... 2. Enqueue tasks ... 3. See Error ... **Expected behavior** - Need to see the stats of the system **Screenshots** If applicable, add screenshots to help explain your problem. ![Screen Shot 2021-08-07 at 11 07 54](https://user-images.githubusercontent.com/2597710/128587604-587a417d-cb33-4949-881b-791416c1d40d.png) **Environment (please complete the following information):** - OS: [Ubuntu 18.4] - Version of `asynq` package [v0.18.2] **Additional context** - The worker and scheduler work as normal ![Screen Shot 2021-08-07 at 11 20 15](https://user-images.githubusercontent.com/2597710/128587715-82bfc73d-fff7-4d7c-852f-8fa03dfae42f.png) - The data on Redis is okay too ![Screen Shot 2021-08-07 at 11 19 05](https://user-images.githubusercontent.com/2597710/128587694-d1a9a7ea-3781-4e25-bdaa-089179a3f369.png)

kerem

2026-03-07 22:06:24 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@crossworth commented on GitHub (Aug 7, 2021):

Hello, I have encoutered this problem as well, but using asynqmon, I could not yet determine why (for this reason I havent create a issue or pr), but I know that I can cause it by enqueuing a lot of tasks and processing they very fast (no code, no printing, just a empty handler).

Probably the MEMORY USAGE command is returning nil and since we dont handle it we get this error message.

I will investigate it.

@crossworth commented on GitHub (Aug 7, 2021): Hello, I have encoutered this problem as well, but using `asynqmon`, I could not yet determine why (for this reason I havent create a issue or pr), but I know that I can cause it by enqueuing a lot of tasks and processing they very fast (no code, no printing, just a empty handler). Probably the [MEMORY USAGE](https://redis.io/commands/memory-usage) command is returning [nil](https://redis.uptrace.dev/#redisnil) and since [we dont handle it](https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/internal/rdb/inspect.go#L261) we get this error message. I will investigate it.

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 7, 2021):

@duyhungtnn thank you for reporting this!

Could you double check the CLI version by running asynq version?

@hibiken commented on GitHub (Aug 7, 2021): @duyhungtnn thank you for reporting this! Could you double check the CLI version by running `asynq version`?

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 7, 2021):

The error message in the screenshot is coming from here: github.com/hibiken/asynq@95c90a5cb8/internal/rdb/inspect.go (L197)

My initial guess is that the for loop for the scan command is taking too long (github.com/hibiken/asynq@95c90a5cb8/internal/rdb/inspect.go (L183)) since we recently changed each task to have its only redis key.
Since redis only executes one command at a time, since other clients are sending commands to redis while this for loop is executing scan commands to collect all keys, keys may have been deleted (i.e. task processed successfully), you'll see the key not found error when memeorUsage command is invoked.

If this above analysis is correct, then we should make this operation into a Lua script and run the script with EXECUTE so that all of this operation becomes atomic (i.e. other clients commands won't be executing between multiple calls to scan and memoryusage). But now that we can have a lot more redis-keys than before (each task has its own key), maybe getting memory usage within a short timeframe is not feasible anymore.

@hibiken commented on GitHub (Aug 7, 2021): The error message in the screenshot is coming from here: https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/internal/rdb/inspect.go#L197 My initial guess is that the for loop for the scan command is taking too long (https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/internal/rdb/inspect.go#L183) since we recently changed each task to have its only redis key. Since redis only executes one command at a time, since other clients are sending commands to redis while this for loop is executing scan commands to collect all keys, keys may have been deleted (i.e. task processed successfully), you'll see the key not found error when `memeorUsage` command is invoked. If this above analysis is correct, then we should make this operation into a Lua script and run the script with EXECUTE so that all of this operation becomes atomic (i.e. other clients commands won't be executing between multiple calls to scan and memoryusage). But now that we can have a lot more redis-keys than before (each task has its own key), maybe getting memory usage within a short timeframe is not feasible anymore.

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 7, 2021):

@duyhungtnn thank you for reporting this!

Could you double check the CLI version by running asynq version?

@hibiken yeah my asynq is asynq version 0.18.2

@crossworth I am facing that issue with asynqmon too. The web dashboard display nothing when viewing the queue info.
Yes, I was using a scheduler to trigger a single task every 2h. On that single task will enqueue 5000 tasks to feed the workers. It may be the cause.

@duyhungtnn commented on GitHub (Aug 7, 2021): > @duyhungtnn thank you for reporting this! > > Could you double check the CLI version by running `asynq version`? @hibiken yeah my asynq is `asynq version 0.18.2` @crossworth I am facing that issue with `asynqmon` too. The web dashboard display nothing when viewing the queue info. Yes, I was using a scheduler to trigger a single task every 2h. On that single task will enqueue 5000 tasks to feed the workers. It may be the cause.

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 9, 2021):

@duyhungtnn The fix is in 👍
Let me know if you continue to see issues with new versions (asynq: v0.18.3, asynqmon v0.2.2)

@hibiken commented on GitHub (Aug 9, 2021): @duyhungtnn The fix is in 👍 Let me know if you continue to see issues with new versions (asynq: v0.18.3, asynqmon v0.2.2)

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1144

No description provided.

Rows
Columns

[GH-ISSUE #309] [BUG] asynq client failed when use command stats #1144

[GH-ISSUE #309] [BUG] asynq client failed when use command `stats` #1144