[GH-ISSUE #1671] Regression of issue 1199 with uvicorn: Retrying websockets connection with an invalid token #2987

Open
opened 2026-03-14 06:09:55 +03:00 by kerem · 1 comment
Owner

Originally created by @NiceGuyIT on GitHub (Nov 8, 2023).
Original GitHub issue: https://github.com/amidaware/tacticalrmm/issues/1671

Server Info (please complete the following information):

  • OS: Ubuntu 20.04.4 LTS
  • Browser: Firefox 115.3.1esr
  • RMM Version (as shown in top left of web UI): v0.17.1

Installation Method:

  • Standard
  • Docker

Agent Info (please complete the following information):

  • Agent version (as shown in the 'Summary' tab of the agent from web UI): N/A
  • Agent OS: N/A

Describe the bug
This is a regression of #1199 with the uvicorn implementation. After upgrading the server, I kept getting the error below. The browser kept retrying the connection even though it received an error. Every retry caused an exception in the server logs. This will happen continuously as long as the browser is running. I.e. there is no way for the server to stop the client from retrying.

To Reproduce
Steps to reproduce the behavior:

  1. Log in to the RMM.
  2. Upgrade the server. I don't know if restarting will cause the same problem.
  3. After the server is upgraded, follow the logs. journalctl --follow
  4. See the error below.

Expected behavior
The client (browser) should not keep retrying if it receives an invalid status code from the server. In this case, it was 403 Forbidden

Screenshots
If applicable, add screenshots to help explain your problem.

image

Additional context
Note: The IP and token were changed and do not reflect the actual IP and token.

journalctl --follow

Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO:     ('9.9.9.9', 0) - "WebSocket /ws/dashinfo/?access_token=5f37238aa54cc9c03ea6c640c9bec237bce126c4bc834c35fea21269db485841" 403
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO:     connection failed (403 Forbidden)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ERROR:    Exception in ASGI application
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: Traceback (most recent call last):
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 247, in run_asgi
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.app(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/routing.py", line 62, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await application(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/tacticalrmm/tacticalrmm/utils.py", line 268, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.app(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 47, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.inner(dict(scope, cookies=cookies), receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 263, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.inner(wrapper.scope, receive, wrapper.send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/auth.py", line 185, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await super().__call__(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/middleware.py", line 24, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.inner(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/routing.py", line 116, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await application(
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 94, in app
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await consumer(scope, receive, send)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 58, in __call__
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await await_many_dispatch(
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/utils.py", line 50, in await_many_dispatch
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await dispatch(result)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 73, in dispatch
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await handler(message)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/generic/websocket.py", line 173, in websocket_connect
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await self.connect()
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/tacticalrmm/core/consumers.py", line 22, in connect
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await self.accept()
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/generic/websocket.py", line 186, in accept
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await super().send({"type": "websocket.accept", "subprotocol": subprotocol})
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 81, in send
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     await self.base_send(message)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 226, in send
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     return await self.real_send(message)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:   File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 338, in asgi_send
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]:     raise RuntimeError(msg % message_type)
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: RuntimeError: Unexpected ASGI message 'websocket.accept', after sending 'websocket.close'.
Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO:     connection closed
Originally created by @NiceGuyIT on GitHub (Nov 8, 2023). Original GitHub issue: https://github.com/amidaware/tacticalrmm/issues/1671 **Server Info (please complete the following information):** - OS: Ubuntu 20.04.4 LTS - Browser: Firefox 115.3.1esr - RMM Version (as shown in top left of web UI): v0.17.1 **Installation Method:** - [x] Standard - [ ] Docker **Agent Info (please complete the following information):** - Agent version (as shown in the 'Summary' tab of the agent from web UI): N/A - Agent OS: N/A **Describe the bug** This is a regression of #1199 with the `uvicorn` implementation. After upgrading the server, I kept getting the error below. The browser kept retrying the connection even though it received an error. Every retry caused an exception in the server logs. This will happen continuously as long as the browser is running. I.e. there is no way for the server to stop the client from retrying. **To Reproduce** Steps to reproduce the behavior: 1. Log in to the RMM. 2. Upgrade the server. I don't know if restarting will cause the same problem. 3. After the server is upgraded, follow the logs. `journalctl --follow` 4. See the error below. **Expected behavior** The client (browser) should not keep retrying if it receives an invalid status code from the server. In this case, it was `403 Forbidden` **Screenshots** If applicable, add screenshots to help explain your problem. ![image](https://github.com/amidaware/tacticalrmm/assets/7763429/170a72a4-175e-4fc6-9755-a9767e8dcc82) **Additional context** Note: The IP and token were changed and do not reflect the actual IP and token. `journalctl --follow` ```text Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO: ('9.9.9.9', 0) - "WebSocket /ws/dashinfo/?access_token=5f37238aa54cc9c03ea6c640c9bec237bce126c4bc834c35fea21269db485841" 403 Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO: connection failed (403 Forbidden) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ERROR: Exception in ASGI application Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: Traceback (most recent call last): Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 247, in run_asgi Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: result = await self.app(self.scope, self.asgi_receive, self.asgi_send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.app(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/routing.py", line 62, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await application(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/tacticalrmm/tacticalrmm/utils.py", line 268, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.app(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 47, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.inner(dict(scope, cookies=cookies), receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 263, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.inner(wrapper.scope, receive, wrapper.send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/auth.py", line 185, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await super().__call__(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/middleware.py", line 24, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.inner(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/routing.py", line 116, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await application( Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 94, in app Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await consumer(scope, receive, send) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 58, in __call__ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await await_many_dispatch( Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/utils.py", line 50, in await_many_dispatch Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await dispatch(result) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 73, in dispatch Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await handler(message) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/generic/websocket.py", line 173, in websocket_connect Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await self.connect() Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/tacticalrmm/core/consumers.py", line 22, in connect Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await self.accept() Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/generic/websocket.py", line 186, in accept Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await super().send({"type": "websocket.accept", "subprotocol": subprotocol}) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/consumer.py", line 81, in send Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: await self.base_send(message) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/channels/sessions.py", line 226, in send Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: return await self.real_send(message) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: File "/rmm/api/env/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 338, in asgi_send Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: raise RuntimeError(msg % message_type) Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: RuntimeError: Unexpected ASGI message 'websocket.accept', after sending 'websocket.close'. Nov 08 08:44:22 ns-v35-tactical uvicorn[25990]: INFO: connection closed ```
Author
Owner

@Trapulo commented on GitHub (Apr 22, 2024):

I have same problem, any way to fix?

<!-- gh-comment-id:2070039093 --> @Trapulo commented on GitHub (Apr 22, 2024): I have same problem, any way to fix?
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/tacticalrmm#2987
No description provided.