starred/asynqmon
1
0
Fork 0
mirror of https://github.com/hibiken/asynqmon.git synced 2026-04-26 17:45:50 +03:00
Code Issues 211 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #210] [BUG] redis eval error: "this Redis command is not allowed from scripts" #51

New issue
Closed
opened 2026-03-02 05:18:23 +03:00 by kerem · 15 comments
kerem commented 2026-03-02 05:18:23 +03:00
Owner
Copy link

Originally created by @momentumtools on GitHub (Jan 27, 2022).
Original GitHub issue: https://github.com/hibiken/asynqmon/issues/210

Hi,

locally the asynqmon works perfectly, however now that I just deployed it to my heroku setup (using a redis enterprise cloud instance the chrome console log returns these errors:

UNKNOWN: UNKNOWN: redis eval error: ERR Error running script 
(call to f_7574d838a4ddc18d626139a5521903c904f1404a): 
@user_script:19: @user_script: 19: This Redis command is not allowed from scripts

I googled a bit and this is most likely due to the REDIS server being relatively strict.
Any idea how to circumvent this through an asynqmon configuration (because on the redis enterprise cloud I don't see any EVAL related configurations)

Greetings,
PJ

PS: love your queuing tool and have been in the game for quite some time now :)!

Originally created by @momentumtools on GitHub (Jan 27, 2022). Original GitHub issue: https://github.com/hibiken/asynqmon/issues/210 Hi, locally the asynqmon works perfectly, however now that I just deployed it to my heroku setup (using a [redis enterprise cloud](https://elements.heroku.com/addons/rediscloud) instance the chrome console log returns these errors: ``` UNKNOWN: UNKNOWN: redis eval error: ERR Error running script (call to f_7574d838a4ddc18d626139a5521903c904f1404a): @user_script:19: @user_script: 19: This Redis command is not allowed from scripts ``` I googled a bit and this is most likely due to the REDIS server being relatively strict. Any idea how to circumvent this through an asynqmon configuration (because on the redis enterprise cloud I don't see any EVAL related configurations) Greetings, PJ PS: love your queuing tool and have been in the game for quite some time now :)!
kerem closed this issue 2026-03-02 05:18:24 +03:00
kerem commented 2026-03-02 05:18:24 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jan 28, 2022):

@momentumtools Thank you for opening this issue!

Would you mind providing additional info?

  • Redis version number
  • Which page (e.g. URL path) in the Web UI triggers this error
<!-- gh-comment-id:1023766840 --> @hibiken commented on GitHub (Jan 28, 2022): @momentumtools Thank you for opening this issue! Would you mind providing additional info? - Redis version number - Which page (e.g. URL path) in the Web UI triggers this error
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@momentumtools commented on GitHub (Jan 28, 2022):

Hi @hibiken ,

thanks for the quick reply, here are the parameters

  • Redis 6.2.3
  • paths with errors: /jobs/api/queues
  • paths without errors: /jobs/api/servers, /jobs/api/redis_info, /jobs/api/scheduler_entries, /jobs/api/queue_stats

PS: the support folks at redis enterprise ask me the specific LUA script that we are trying to execute to help me figure out which command it is that is blocked on my DB

<!-- gh-comment-id:1024456882 --> @momentumtools commented on GitHub (Jan 28, 2022): Hi @hibiken , thanks for the quick reply, here are the parameters - Redis 6.2.3 - paths with errors: /jobs/api/queues - paths without errors: /jobs/api/servers, /jobs/api/redis_info, /jobs/api/scheduler_entries, /jobs/api/queue_stats PS: the support folks at redis enterprise ask me the specific LUA script that we are trying to execute to help me figure out which command it is that is blocked on my DB
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jan 28, 2022):

Thanks!

I haven't confirmed it on my side, but my first guess is this MEMORY USAGE command we are calling in this Lua script . Specifically these two lines
github.com/hibiken/asynq@dff2e3a336/internal/rdb/inspect.go (L236)
github.com/hibiken/asynq@dff2e3a336/internal/rdb/inspect.go (L253)

I need to do a bit more research but my initial thought is that the issue is due to the restriction around non-deterministic commands in script; non-deterministic commands are not allowed in Lua script when replication is enabled. As documented in https://redis.io/commands/eval, we may want to disable replication when calling MEMORY USAGE from that script (https://redis.io/commands/eval#selective-replication-of-commands) Or use read-only EVAL command instead (this EVAL_RO is only supported from redis v7+ though, so maybe not realistic to use this). I will look into this issue in the next few days but if you have any updates, please let me know in this thread :)

<!-- gh-comment-id:1024754563 --> @hibiken commented on GitHub (Jan 28, 2022): Thanks! I haven't confirmed it on my side, but my first guess is this MEMORY USAGE command we are calling in [this Lua script ](https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L225-L266). Specifically these two lines https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L236 https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L253 I need to do a bit more research but my initial thought is that the issue is due to the restriction around non-deterministic commands in script; [non-deterministic commands are not allowed in Lua script when replication is enabled](https://stackoverflow.com/questions/40013598/why-redis-cannot-execute-non-deterministic-commands-in-lua-script/49764136). As documented in https://redis.io/commands/eval, we may want to disable replication when calling MEMORY USAGE from that script (https://redis.io/commands/eval#selective-replication-of-commands) Or use [read-only EVAL](https://redis.io/commands/eval_ro) command instead (this EVAL_RO is only supported from redis v7+ though, so maybe not realistic to use this). I will look into this issue in the next few days but if you have any updates, please let me know in this thread :)
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jan 29, 2022):

Actually I take that back, since the redis version is above v4 in this case, script effect replication is used and since the Lua script is read-only, it should be allowed. Not sure where this error is coming from. Let me know if you hear back from the support team.

<!-- gh-comment-id:1024768745 --> @hibiken commented on GitHub (Jan 29, 2022): Actually I take that back, since the redis version is above v4 in this case, [**script effect replication**](https://redis.io/commands/eval#replicating-commands-instead-of-scripts) is used and since the Lua script is read-only, it should be allowed. Not sure where this error is coming from. Let me know if you hear back from the support team.
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@momentumtools commented on GitHub (Jan 29, 2022):

Hi @hibiken , thanks again for such fast support. I'm amazed :)!

I've forwarded github.com/hibiken/asynq@dff2e3a336/internal/rdb/inspect.go (L225-L266) to the redis enterprise support, maybe it will ring a bell for them.

Is there a way for us to be surer which LUA script is triggering the error? a kind of debug mode to enable?

<!-- gh-comment-id:1024905283 --> @momentumtools commented on GitHub (Jan 29, 2022): Hi @hibiken , thanks again for such fast support. I'm amazed :)! I've forwarded https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L225-L266 to the redis enterprise support, maybe it will ring a bell for them. Is there a way for us to be surer which LUA script is triggering the error? a kind of debug mode to enable?
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jan 30, 2022):

This is a good opportunity to find a bug so thank you for providing the info!

There's no debug mode to quickly figure out the problematic lua script unfortunately, but based on the fact that the error is returned when we make API calls to /api/queues endpoint, it tells me that this handler is causing the error. And if you look at the handler definition, it's most likely this call to inspector.GetQueueInfo is a culprit, which in turn calls this CurrentStat and this function executes two lua scripts this one and this one.

<!-- gh-comment-id:1025015160 --> @hibiken commented on GitHub (Jan 30, 2022): This is a good opportunity to find a bug so thank you for providing the info! There's no debug mode to quickly figure out the problematic lua script unfortunately, but based on the fact that the error is returned when we make API calls to `/api/queues` endpoint, it tells me that [this handler](https://github.com/hibiken/asynqmon/blob/609b319a9e7daf221cfbae978fd03953cce3c0de/handler.go#L115) is causing the error. And if you look at the handler definition, it's most likely this call to [`inspector.GetQueueInfo`](https://github.com/hibiken/asynqmon/blob/609b319a9e7daf221cfbae978fd03953cce3c0de/queue_handlers.go#L27) is a culprit, which in turn calls this [`CurrentStat`](https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L124) and this function executes two lua scripts [this one](https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L88) and [this one](https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L225).
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@momentumtools commented on GitHub (Jan 31, 2022):

Hi @hibiken , I'll pass the first script to the redis enterprise support team as well, as I only pasted them the 2nd one you thought the error was in when you wrote https://github.com/hibiken/asynqmon/issues/210#issuecomment-1024754563

<!-- gh-comment-id:1025627105 --> @momentumtools commented on GitHub (Jan 31, 2022): Hi @hibiken , I'll pass [the first script](https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L88) to the redis enterprise support team as well, as I only pasted them the 2nd one you thought the error was in when you wrote https://github.com/hibiken/asynqmon/issues/210#issuecomment-1024754563
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Feb 8, 2022):

@momentumtools Any update on this?

<!-- gh-comment-id:1032159044 --> @hibiken commented on GitHub (Feb 8, 2022): @momentumtools Any update on this?
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@momentumtools commented on GitHub (Feb 8, 2022):

Nope, redis enterprise support is dead silent...
most likely I'll try with another redis provider and see if the same error occurs, if not it's redis enterprise their loss sadly...

Long story short, this EVAL error did not pop up yet with any other users, right?

<!-- gh-comment-id:1032328809 --> @momentumtools commented on GitHub (Feb 8, 2022): Nope, redis enterprise support is dead silent... most likely I'll try with another redis provider and see if the same error occurs, if not it's redis enterprise their loss sadly... Long story short, this EVAL error did not pop up yet with any other users, right?
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@momentumtools commented on GitHub (Feb 17, 2022):

@hibiken ok I got back from their support

The issue is in the script you mentioned, there are some MEMORY commands. Most MEMORY commands are blocked on Redis Enterprise Cloud due to the architecture differences between Enterprise and Open Source Redis.

I'll thus have to pick another redis provider, but so now when other people bump into this they'll know where to look ;)

<!-- gh-comment-id:1043361044 --> @momentumtools commented on GitHub (Feb 17, 2022): @hibiken ok I got back from their support > The issue is in the script you mentioned, there are some MEMORY commands. Most MEMORY commands are blocked on Redis Enterprise Cloud due to the architecture differences between Enterprise and Open Source Redis. I'll thus have to pick another redis provider, but so now when other people bump into this they'll know where to look ;)
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Feb 18, 2022):

@momentumtools Thank you for the update! Switching to another redis provider sounds like a non-trivial change, especially for some tooling issues. Maybe we should support turning off memory usage command in the script via a flag; For example, we could add a flag --disable_memory_usage_command to this Web UI binary. Let me know what you think!

<!-- gh-comment-id:1043661471 --> @hibiken commented on GitHub (Feb 18, 2022): @momentumtools Thank you for the update! Switching to another redis provider sounds like a non-trivial change, especially for some tooling issues. Maybe we should support turning off memory usage command in the script via a flag; For example, we could add a flag `--disable_memory_usage_command` to this Web UI binary. Let me know what you think!
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@Soul-Killer-Ky commented on GitHub (Apr 16, 2024):

image

In the api/queues interface, eval will be called twice and an error will be reported the second time

<!-- gh-comment-id:2058397935 --> @Soul-Killer-Ky commented on GitHub (Apr 16, 2024): ![image](https://github.com/hibiken/asynqmon/assets/26269381/b421c421-c160-458a-9cac-567feceb8144) In the api/queues interface, eval will be called twice and an error will be reported the second time
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@Soul-Killer-Ky commented on GitHub (Apr 16, 2024):

image

I rewrote the eval of redisClient, so I can print the logs

<!-- gh-comment-id:2058399673 --> @Soul-Killer-Ky commented on GitHub (Apr 16, 2024): ![image](https://github.com/hibiken/asynqmon/assets/26269381/fc31a45b-e7e5-45ce-95ea-c39e7da10871) I rewrote the eval of redisClient, so I can print the logs
kerem commented 2026-03-02 05:18:25 +03:00
Author
Owner
Copy link

@brummetj commented on GitHub (May 6, 2024):

hey @hibiken how easy can you get that flag in to disable the check? Im running asynqmon on AWS with elastic cache and getting

UNKNOWN: UNKNOWN: redis eval error: ERR This Redis command is not allowed from script script: 0c80fc1868e3aae205155ae2017ce995a3a7cec0, on @user_script:19.
<!-- gh-comment-id:2096707979 --> @brummetj commented on GitHub (May 6, 2024): hey @hibiken how easy can you get that flag in to disable the check? Im running asynqmon on AWS with elastic cache and getting ``` UNKNOWN: UNKNOWN: redis eval error: ERR This Redis command is not allowed from script script: 0c80fc1868e3aae205155ae2017ce995a3a7cec0, on @user_script:19. ```
kerem commented 2026-03-02 05:18:26 +03:00
Author
Owner
Copy link

@unix2dos commented on GitHub (Aug 22, 2024):

@momentumtools Thank you for the update! Switching to another redis provider sounds like a non-trivial change, especially for some tooling issues. Maybe we should support turning off memory usage command in the script via a flag; For example, we could add a flag --disable_memory_usage_command to this Web UI binary. Let me know what you think!

@hibiken can this flag be added?

<!-- gh-comment-id:2304676620 --> @unix2dos commented on GitHub (Aug 22, 2024): > @momentumtools Thank you for the update! Switching to another redis provider sounds like a non-trivial change, especially for some tooling issues. Maybe we should support turning off memory usage command in the script via a flag; For example, we could add a flag `--disable_memory_usage_command` to this Web UI binary. Let me know what you think! @hibiken can this flag be added?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
dependabot/npm_and_yarn/ui/react-redux-8.0.5
dependabot/npm_and_yarn/ui/typescript-4.9.5
dependabot/npm_and_yarn/ui/testing-library/user-event-14.4.3
dependabot/npm_and_yarn/ui/testing-library/jest-dom-5.16.5
dependabot/npm_and_yarn/ui/react-syntax-highlighter-15.5.0
refactor/generic-tasks-table
refactor/generic-tasks
v0.7.2
v0.7.1
v0.7.0
v0.6.1
v0.6.0
v0.5.0
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1-beta2
v0.1-beta1
v0.1-alpha2
v0.1-alpha
Labels
Clear labels   
documentation

   
enhancement

   
enhancement

   
good first issue

   
pull-request

Mirrored from GitHub Pull Request

No labels
documentation
enhancement
enhancement
good first issue
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/asynqmon#51
No description provided.