[GH-ISSUE #280] Busy loop calling futex, 100% CPU usage #166

Open
opened 2026-02-28 14:25:57 +03:00 by kerem · 7 comments
Owner

Originally created by @markferry on GitHub (Jan 4, 2026).
Original GitHub issue: https://github.com/devgianlu/go-librespot/issues/280

Intermittent but seen on two occasions recently.

futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                    [55/432]
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 6                                                                                                                                                                                                  
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 6                                                                                                                                                                                                  
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 824637841104                                                                                                                                                                                       
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 824637841104                                                                                                                                                                                       
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 6                                                                                                                                                                                                  
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 824637841104                                                                                                                                                                                       
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} ---                                                                                                                                                                  
rt_sigreturn({mask=[]})                 = 824637841104                                                                                                                                                                                       
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1                                                                                                                                                                                               
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                                                                                                                                                            
futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0                                                         
Originally created by @markferry on GitHub (Jan 4, 2026). Original GitHub issue: https://github.com/devgianlu/go-librespot/issues/280 Intermittent but seen on two occasions recently. ``` futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 [55/432] futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 6 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 6 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 824637841104 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 824637841104 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 6 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 824637841104 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000100848, FUTEX_WAKE_PRIVATE, 1) = 1 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=2446, si_uid=128} --- rt_sigreturn({mask=[]}) = 824637841104 futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x1255a00, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 ```
Author
Owner

@markferry commented on GitHub (Jan 4, 2026):

busy-lsof.txt

busy-objdump.txt.gz

busy-pmap.txt

<!-- gh-comment-id:3707936415 --> @markferry commented on GitHub (Jan 4, 2026): [busy-lsof.txt](https://github.com/user-attachments/files/24421115/busy-lsof.txt) [busy-objdump.txt.gz](https://github.com/user-attachments/files/24421163/busy-objdump.txt.gz) [busy-pmap.txt](https://github.com/user-attachments/files/24421142/busy-pmap.txt)
Author
Owner

@markferry commented on GitHub (Jan 4, 2026):

$ sudo netstat -pnt  | grep 2446
tcp        0      0 192.168.7.5:34330       34.158.255.62:443       ESTABLISHED 2446/go-librespot   
tcp6       0      0 127.0.0.1:24807         127.0.0.1:34484         ESTABLISHED 2446/go-librespot   
$ host 34.158.255.62
62.255.158.34.in-addr.arpa domain name pointer 62.255.158.34.bc.googleusercontent.com.
<!-- gh-comment-id:3707938974 --> @markferry commented on GitHub (Jan 4, 2026): ``` $ sudo netstat -pnt | grep 2446 tcp 0 0 192.168.7.5:34330 34.158.255.62:443 ESTABLISHED 2446/go-librespot tcp6 0 0 127.0.0.1:24807 127.0.0.1:34484 ESTABLISHED 2446/go-librespot ``` ``` $ host 34.158.255.62 62.255.158.34.in-addr.arpa domain name pointer 62.255.158.34.bc.googleusercontent.com. ```
Author
Owner

@devgianlu commented on GitHub (Jan 4, 2026):

I'd say the most useful thing here would be a goroutine dump. You can trigger it by sending SIGABRT to the process.

<!-- gh-comment-id:3708027783 --> @devgianlu commented on GitHub (Jan 4, 2026): I'd say the most useful thing here would be a goroutine dump. You can trigger it by sending SIGABRT to the process.
Author
Owner

@markferry commented on GitHub (Jan 4, 2026):

Had to kill the process. I'll catch it next time.

<!-- gh-comment-id:3708366282 --> @markferry commented on GitHub (Jan 4, 2026): Had to kill the process. I'll catch it next time.
Author
Owner

@markferry commented on GitHub (Jan 20, 2026):

7 out of 8 instances all hung. Here are goroutine dumps from three of them:

An interesting snippet prior to the SIGABRT:

Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=error msg="failed reconnecting dealer" error="failed obtaining dealer access token: failed renewing login5 access token: failed authenticating with login5: UNKNOWN_ERROR"
Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=warning msg="failed sending dealer ping" error="failed to write msg: WebSocket closed: received close frame: status = StatusGoingAway and reason = \"Service is going to be redeployed\""
Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=error msg="did not receive last pong from dealer, 859s passed"

Looks like those messages appeared for each of the hanging instances starting from ~11AM GMT today.

A more complete dump of all go-librespot instance output for today including timestamps:

<!-- gh-comment-id:3773782470 --> @markferry commented on GitHub (Jan 20, 2026): 7 out of 8 instances all hung. Here are goroutine dumps from three of them: - [dump.outside.txt](https://github.com/user-attachments/files/24741542/dump.outside.txt) - [dump.ballroom-patio.txt](https://github.com/user-attachments/files/24741543/dump.ballroom-patio.txt) - [dump.study.txt](https://github.com/user-attachments/files/24742078/dump.study.txt) An interesting snippet prior to the SIGABRT: ``` Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=error msg="failed reconnecting dealer" error="failed obtaining dealer access token: failed renewing login5 access token: failed authenticating with login5: UNKNOWN_ERROR" Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=warning msg="failed sending dealer ping" error="failed to write msg: WebSocket closed: received close frame: status = StatusGoingAway and reason = \"Service is going to be redeployed\"" Jan 20 15:47:51 (study-spotify) time="2026-01-20T15:47:51Z" level=error msg="did not receive last pong from dealer, 859s passed" ``` Looks like those messages appeared for each of the hanging instances starting from ~11AM GMT today. A more complete dump of all go-librespot instance output for today including timestamps: - [dump-golibrespot.txt.gz](https://github.com/user-attachments/files/24742061/dump-golibrespot.txt.gz)
Author
Owner

@markferry commented on GitHub (Jan 20, 2026):

So could it be triggered by Spotify doing (segmented) restarts of their services?

<!-- gh-comment-id:3773787951 --> @markferry commented on GitHub (Jan 20, 2026): So could it be triggered by Spotify doing (segmented) restarts of their services?
Author
Owner

@devgianlu commented on GitHub (Feb 1, 2026):

So could it be triggered by Spotify doing (segmented) restarts of their services?

Yes, this seems like it. However from the goroutine dumps I cannot figure out what is causing this issue, I may need to try and reproduce this in some way.

<!-- gh-comment-id:3831371390 --> @devgianlu commented on GitHub (Feb 1, 2026): > So could it be triggered by Spotify doing (segmented) restarts of their services? Yes, this seems like it. However from the goroutine dumps I cannot figure out what is causing this issue, I may need to try and reproduce this in some way.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/go-librespot#166
No description provided.