mirror of
https://github.com/spotipy-dev/spotipy.git
synced 2026-04-27 08:35:49 +03:00
[GH-ISSUE #913] Spotipy doesnt doesnt respond but no error #548
Labels
No labels
api-bug
bug
dependencies
documentation
duplicate
enhancement
external-ide
headless-mode
implicit-grant-flow
invalid
missing-endpoint
pr-welcome
private-api
pull-request
question
spotipy3
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/spotipy#548
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @nickhir on GitHub (Dec 4, 2022).
Original GitHub issue: https://github.com/spotipy-dev/spotipy/issues/913
Hello,
I have been using spotipy for the last year and never ran into any problems.
Today, I realized that something is not working anymore and looked into it.
I can initializing the
sp.Spotifyinstance as usual without any problems, but if I then call a function (for examplespotify.me()orspotify.devices(), it simply hangs and doesnt return anything.This is the code I used for the last months:
I checked my Spotify dashboard, and noticed, that noticed that the number of daily requests dropped to zero just when December started:

Do you have any idea what might cause the issue? I already tried to regenerate a new client secret, but it didnt work.
I am using the version 2.21.0 on a raspberry pi (Raspbian version 10 [Buster])
@JohnWhy commented on GitHub (Mar 8, 2023):
I believe something occurs that spams the Spotify API and they essentially blacklist your app. I wrote my own requests function to query the API using my existing credentials I used with spotipy and it was giving me a 429 error for "Too Many Requests" even when waiting a long time between search requests.
I made a new Spotify app on the website (new client/secret) and used my custom requests function and it worked perfectly.
Made another Spotify app on the website (new client/secret) and tried plugging those into the existing spotipy application and was still getting an infinite hanging problem. So I think what's happening is spotipy is getting a 429 error and not handling/reporting it. It must be spamming requests at some point causing the issue, notice how you have a bump in API activity for a brief moment, my guess is spotipy unintentionally spammed the API with thousands of requests in a second, and Spotify auto-blocked your application from sending more.
My app had a similar spike (although more pronounced)

@Rsalganik1123 commented on GitHub (Mar 17, 2023):
Hi! I had a similar issue. I found that once a request gets denied, with each additional call you make following this, the wait time grows exponentially. Since I had a lot of calls to make (I'm doing a dataset augmentation task), I added an error catch for time out that looks like this:
Normally, it makes me wait like 8 hours and then I can come back.
Also, a few tips I have for making it less likely to time out:
Not sure whether you setting allows for these changes, but in the case that it does - hope it helps :)
@stephanebruckert commented on GitHub (Mar 30, 2023):
spotipy should already be waiting/sleeping when the Spotify API response requests to wait (retry-after value). Maybe there is a scenario where this does not apply.
Would anyone have a Minimal reproducible example that would allow us to reproduce this rate limiting? This would need to work with a new Spotify app.
Feel free to include logs about the current retry-after value, maybe with inspiration from https://github.com/spotipy-dev/spotipy/issues/766#issuecomment-1005102900?
@CalColistra commented on GitHub (Mar 31, 2023):
Hey, I was just curious if Rsalgnakit1223's solution with the try catch blocks would work even if my spotipy api doesn't throw an error. Mine just gets stuck in sleep_for_rety() and hangs forever. Would the try catch block work in my case if an error is never thrown to trigger the catch?
@stephanebruckert commented on GitHub (Mar 31, 2023):
@CalColistra, sorry Rsalgnakit1223's solution is incomplete, here is a better example which will raise an exception https://github.com/spotipy-dev/spotipy/issues/766#issuecomment-1005102900
@CalColistra commented on GitHub (Apr 6, 2023):
@stephanebruckert Hey, I am sorry if I missed anything obvious in any of the issues made for this sleep_for_retry() problem with the spotipy api. But are there any working solutions that we know of? I see you are keeping up with a lot of the discussions about this and really appreciate you taking the time to do so.
@stephanebruckert commented on GitHub (Apr 7, 2023):
@reflexiveprophecy commented on GitHub (Apr 11, 2023):
It gives me a 22 hour timeout the first time of bumping into 429 in this fresh run after 23k search requests, does the API assign rate limit based on IP too? I read on previous issues that the timeout would go up exponentially every time you hit the limit. How does the timeout get determined for search under Web API?
[main][INFO ]processing the 23210th title
[main][ERROR ]Rate limit hit, please try after:81654 The program now needs to rest for 81654 seconds
I thought I could catch the retry_after and make the program sleep for the value, but 22 hours timeout renders that not very useful...
@stephanebruckert commented on GitHub (Apr 11, 2023):
@reflexiveprophecy can you please provide a Minimal reproducible example? Something that we could just run once to reproduce the problem with a new Spotify app?
The code above doesn't look complete as it doesn't include any loop.
I agree but the app is supposed to start sleeping from the very first 429s, not after 23k requests and 22 hours wait. Could your code show that as well?
My assumption is that at one point, it's not "respecting" the given
retry-aftervalue. For exampleretry-afterrequires a 10 seconds wait, but another request is started before that, which leads to a violation and therefore exponential wait times as a "punishment". See https://stackoverflow.com/a/70311787/1515819I imagine examples of such violation could be occurring when:
retry-aftervalue as it's not aware of it,retry-aftervalue of say 10s, but the program is stopped and re-started after 5s, which would lead to a violation since it should have waited 5 more seconds.@reflexiveprophecy commented on GitHub (Apr 11, 2023):
@stephanebruckert Okay, please see the following self-contained example. You shouldn't have to do much else to run this example besides pip install requests and embedding CLIENT_ID and CLIENT_SECRET as env variables. This example would produce 100k artist names in a list for the search request to run through in a loop. I tested the script today with a new set of credentials and was able to run through 24.9k search requests this time without a 429 error and it gives me a 23 hour timeout this time after bumping into the error this time...I don't see other 429 messages as I searched through the entire log. And I don't think it's particularly blocking my ip address either as I was able to run through again by switching to new "non-timeout" account credentials. Also I didn't use Spotipy client as the client silences 429 messages for some reasons as previous issues also noted. Looking forward to some solutions :)
@spotifyr1 commented on GitHub (Jul 3, 2023):
Thanks for posting this response, it was very helpful! I'm more of a beginner, but I'm trying to collect some tracks metadata. I'm getting a 429 response, which I expected (I'm making many requests to the API), but the header comes back empty. So when I use
if int(e.headers['retry-after']) > 60to exit the loop, it doesn't work because there is no 'retry-after' key in the headers. Do you know why that could happen? I'm printing out the values returned by the exception, and they all look fine except the header which is empty{}.@reflexiveprophecy commented on GitHub (Jul 5, 2023):
@spotifyr1 Hey, regarding this question "So when I use if int(e.headers['retry-after']) > 60 to exit the loop, it doesn't work because there is no 'retry-after' key in the headers. Do you know why that could happen? I'm printing out the values returned by the exception, and they all look fine except the header which is empty {}", I don't think spotipy has this returned. This is one of the reasons why I didn't use spotipy client as the api, I directly called the REST API with the requests module, hence you don't see me importing spotipy and just the requests module. With the requests module, you should be able to catch the retry-after value.
@Drag-3 commented on GitHub (Jul 6, 2023):
@reflexiveprophecy I took your advice, but I have the same issue with the requests module directly (The get command blocks forever even with a timeout applied) So I am unsure as to whether this is a problem with Spotipy.
@reflexiveprophecy commented on GitHub (Jul 6, 2023):
Hey @Drag-3, got it, happy to test it out if you could share your code, thank you!
@Drag-3 commented on GitHub (Jul 7, 2023):
@reflexiveprophecy Here's my code! Hopefully you can find something I am missing.
According to curl, I should be getting an Error 429, but the code blocks instead.
@dnoegel commented on GitHub (Dec 8, 2023):
I run into the same problem today with this reduced example with spotipy 2.23.0:
My mistake was: I was in headless mode, and the "open the browser" part obviously can not work there. In the example section of spotify I found that you need to set
open_browser=Falseas a param inSpotifyOAuthlike this:With this setup, the oauth flow will be handled in terminal. Hope this is helpful for some 👍
@eleqtrizit commented on GitHub (Jan 18, 2024):
I found the cause of the lack of response, and this is mentioned in https://github.com/urllib3/urllib3/issues/2911. In module urllib3/connectionpool.py, line #943 is code
This is honoring the Retry-After that Spotify's API is sending back. And if you're like me, who somehow got a long retry time (mine is currently 20k+ seconds), it is going to hang.
A potential fix is simply doing the following...
..so Spotify doesn't try the retry. But if you do this, it just raises an Exception and doesn't report back what the Retry-After value was. This is where the improvement can be made, and perhaps build Spotipy's own retry feature.
@dieser-niko commented on GitHub (May 23, 2024):
If you create your own
requests.Session()and pass it toSpotify, you can actually get the retry-after header yourself like this:Sometimes a timeout occurs, so catching (and ignoring) timeout errors would probably be necessary to run this.
You can also speed it up by using
threading.Thread.I'm honestly not sure how much we can improve this. Perhaps a warning to inform the user of the current situation, as there are a lot of issues being opened stating that spotipy has just started to hang randomly.
I'm currently monitoring the
retry-aftervalue and it seems to behave just like expected. It doesn't seem to be counting down any faster or anything. And I now have to wait 11 hours. (The waiting time could also be included in the warning message)Edit: My application is still rate limited, but I've already made one observation. Every hour or two, a single request is successfully executed, and then the countdown continues as if nothing had happened.
@chaffinched commented on GitHub (Jun 5, 2024):
There seems to be some misunderstanding of this error message and very large 'retry-after' value, I get that error while fetching sp.album_tracks, regardless of the time between requests. It is not what people are understanding to be a rate limit where you just have to slow down. I think it means that there is a limit to the number of requests you can make of a specific type in any 24 hour period. You don't get any prior warning, it is a brick wall limit and lasts for multiple hours.
Testing over the last few days suggests that the limit is somewhere in excess of 2000 sp.album_tracks queries in a (presumably) 24 hour period. I'll do more tests and see if the quota reset time is the same time each day, or is dependent on the time of your first / last query. There may be similar errors for other query types, but I only realised this week that this was happening as I was getting a fairly consistent set of results each day. I can't see anything on the Spotify documentation or forum that confirms my suspicion though.
I'm seeing this too.
@dieser-niko commented on GitHub (Jun 5, 2024):
I doubt it is possible to get only 2000 successful requests in over 24 hours. The time between requests is probably also important.
I've run a test script similar to my previous example. The script makes a request every 0.5 seconds (not exactly). If a request takes a bit longer, it doesn't matter because each request is done with a separate `threading.Thread'.
I've also included a counter and timestamp to make sure the timing is at least somewhat correct.
The script has been running for almost 45 minutes and has made 4986 successful requests. Only one request had a bad gateway error. I don't run it anymore because I just got rate limited for 23 hours and 30 minutes (83848).
Well it seems to be. You just have to slow down. Spotify probably doesn't want to encourage people to bombard the API because it wouldn't make a difference.
But maybe it's just the fact that you're using a different endpoint or that I'm only requesting the "name" field, but I'll try that out tomorrow as well.
@chaffinched commented on GitHub (Jun 5, 2024):
I've done the sp.album_tracks query so that there is a second between each query, I've also done it so there is a minute between queries. Neither got a 429 rate limit until they had done 2000+ album queries, at which point both got 60k+ second waits.
Telling me that I'm wrong, and then adding "maybe it's just the fact that you're using a different endpoint" seems unproductive.
@chaffinched commented on GitHub (Jun 7, 2024):
More info on this, again specifically with the album_tracks query. Rate limiting works as expected, and gives the "429 due to API rate limit exceeded" message, but... I get the message "Too Many Requests" after about 2300 queries in a day. So, as I tried to explain above, this means that there have been (you'll never guess!) too many requests. It does not mean the rate limit has been exceeded.
Quite why my comment above was marked as resolved, I don't know, but then I'm not sure why niko's was marked as spam.
Understanding the error messages you are being presented with is essential, particularly when Spotify haven't mentioned them in their documentation, otherwise you have issues like this one that have been open for years without a resolution.
@dieser-niko commented on GitHub (Jun 10, 2024):
I don't want to draw too much attention to the hidden comments, but to make a long story short, when I wrote my comment I wasn't at my best. I realized this afterwards and as a result have hidden the comment. Your comment was hidden because it was related to mine and wouldn't make much sense on its own.
Back to the main topic.
Quick question: when does this issue count as resolved? The only things I can think of that could be improved would be either adding tips to the docs and/or (as mentioned before) adding warnings when a rate limit occurs.
Other solutions such as throttling, caching, lazy loading or anything else mentioned on Spotify's site about rate limits should, in my opinion, be managed by the programmer and not by the library.
@chaffinched commented on GitHub (Jun 10, 2024):
Fair enough.
As we are talking about an undocumented "feature", I don't it can be resolved by the library, but reading this thread (and other similar threads about blocking code and long 429 retry times) there have been various answers and suggestions that completely miss the point of the error.
That's why I commented, to highlight that the "Too Many Requests" error is exactly what it says, and not a rate-limit issue (in the usual sense).
Hopefully that will mean that people who search for "429" or "Too Many Requests" will understand that this isn't an issue with Spotipy and that there isn't much you can do to avoid the error other than limit the number of queries you make each day. The limit for album_tracks seems to be around 2300 per day (I've got 2300-2450 each day over the last couple of weeks). There may be limits for other endpoints, but I haven't hit them. Reset for me always appears to be 20:00 UTC, give or take an hour.
I'll leave it to you to decide whether this needs to be documented or an comment in Issues is sufficient.
@dieser-niko commented on GitHub (Jun 11, 2024):
I've started working on a draft which would print a warning: #1134.
Also another PR for adding a section to the FAQ (I can't add it directly): #1133.
I probably could have done it in one PR, but too late for that now.
Thank you for that. I've tried to highlight this in the FAQ too.
@dieser-niko commented on GitHub (Jun 25, 2024):
Alright, both PRs are now merged. The warning should be included in the next version of Spotipy and then we can finally close this issue for good.
@ejalal commented on GitHub (Oct 22, 2024):
I beleive this is an issue with the Spotify API itself, I converted my original script that used spotipy to use the regular "requests" lib instead, and here are my findings.
After a few hours or processing, I got a retry after 79173 seconds without any reason, or any other 429 status code in between.... my script sends one request per second. And as soon as it gets the 429 status code, my code stops and sleeps for the "retry after" amount of seconds by design.
Well, based on the api documentation, they count the number of requests on a shifting 30s period, if you send too many (without giving exact limit) they send a 429 status code back and tell you to retry after a precise number of seconds. Honestly I don't know where the 79k seconds come from in my case.
If they are counting in a shifting period of 30s, which I understand as 30s between current request and 30s back in time, and are supposed to send 429 as soon as you hit the limit, logically the "retry after" period should never exceed 30s !!
@ejalal commented on GitHub (Oct 22, 2024):
Well after a few tests, there's definitely a "Retry-After" header with response code 200, I think that has to be respected too.
@humcrush-joe commented on GitHub (Dec 23, 2024):
Any chance of this getting resolved? Getting a message noting the problem, including the time to wait, would be infinitely more helpful than having a script just silently spin.
@dieser-niko commented on GitHub (Jan 3, 2025):
We have just released a new version with the change that I mentioned in late June. Make sure you upgrade to 2.25.0 to get the retry warning.
@humcrush-joe commented on GitHub (Jan 3, 2025):
@dieser-niko Many thanks!