mirror of
https://github.com/spotipy-dev/spotipy.git
synced 2026-04-27 08:35:49 +03:00
[GH-ISSUE #525] Search through all markets at once #317
Labels
No labels
api-bug
bug
dependencies
documentation
duplicate
enhancement
external-ide
headless-mode
implicit-grant-flow
invalid
missing-endpoint
pr-welcome
private-api
pull-request
question
spotipy3
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/spotipy#317
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @stephanebruckert on GitHub (Jun 30, 2020).
Original GitHub issue: https://github.com/spotipy-dev/spotipy/issues/525
Is your feature request related to a problem? Please describe.
Describe the solution you'd like
Add options to the
searchfunction to allow searching multiple markets at once:nresults are foundAdditional context
This will not be performant due to the amount of requests, appropriate warnings should be given
@nleroy917 commented on GitHub (Jul 1, 2020):
I can give this a shot - I agree this might make a lot of requests, but in my experience, Spotify is pretty generous with API rate limiting
@shillshocked commented on GitHub (Oct 15, 2020):
How do we use this experimental functionality to search through all markets at once? I tried:
sp.search(search_str, offset=off, limit=50, type='album', market='ALL')
And got:
HTTP Error for GET to https://api.spotify.com/v1/search returned 400 due to Invalid market code
@stephanebruckert commented on GitHub (Oct 15, 2020):
@shillshocked look for
search_marketshttps://spotipy.readthedocs.io/en/2.16.0/#spotipy.client.Spotify.search_markets@shillshocked commented on GitHub (Oct 15, 2020):
OK, I try sp.search_markets(search_str, offset=off, limit=50, type='album') and get the same error. Also same results with:
sp.search_markets(search_str, offset=off, limit=50, type='album', market='ALL')
EDIT: Sorry, user error. Working now.
@shillshocked commented on GitHub (Oct 16, 2020):
I noticed that there are only 60 markets in client.py, but Spotify actually has at least 92 country codes. It should be updated to include them all, or you can just specify markets=['AL','DZ','AD','AR','AU','AT','BH','BY','BE','BO','BA','BR','BG','CA','CL','CO','CR','HR','CY','CZ','DK','DO','EC','EG','SV','EE','FI','FR','DE','GR','GT','HN','HK','HU','IS','IN','ID','IE','IL','IT','JP','JO','KZ','XK','KW','LV','LB','LI','LT','LU','MY','MT','MX','MD','MC','ME','MA','NL','NZ','NI','MK','NO','OM','PS','PA','PY','PE','PH','PL','PT','QA','RO','RU','SA','RS','SG','SK','SI','ZA','ES','SE','CH','TW','TH','TN','TR','UA','AE','GB','US','UY','VN'] in the search.
Not sure why some were excluded. Perhaps they some countries have identical results?
Anyways, I gave it a shot, and then tried combining the results via:
albums = {**results['AL']['albums'], **results['DZ']['albums'], **results['AD']['albums'], **results['AR']['albums'], **results['AU']['albums'], **results['AT']['albums'], **results['BH']['albums'], **results['BY']['albums'], **results['BE']['albums'], **results['BO']['albums'], **results['BA']['albums'], **results['BR']['albums'], **results['BG']['albums'], **results['CA']['albums'], **results['CL']['albums'], **results['CO']['albums'], **results['CR']['albums'], **results['HR']['albums'], **results['CY']['albums'], **results['CZ']['albums'], **results['DK']['albums'], **results['DO']['albums'], **results['EC']['albums'], **results['EG']['albums'], **results['SV']['albums'], **results['EE']['albums'], **results['FI']['albums'], **results['FR']['albums'], **results['DE']['albums'], **results['GR']['albums'], **results['GT']['albums'], **results['HN']['albums'], **results['HK']['albums'], **results['HU']['albums'], **results['IS']['albums'], **results['IN']['albums'], **results['ID']['albums'], **results['IE']['albums'], **results['IL']['albums'], **results['IT']['albums'], **results['JP']['albums'], **results['JO']['albums'], **results['KZ']['albums'], **results['XK']['albums'], **results['KW']['albums'], **results['LV']['albums'], **results['LB']['albums'], **results['LI']['albums'], **results['LT']['albums'], **results['LU']['albums'], **results['MY']['albums'], **results['MT']['albums'], **results['MX']['albums'], **results['MD']['albums'], **results['MC']['albums'], **results['ME']['albums'], **results['MA']['albums'], **results['NL']['albums'], **results['NZ']['albums'], **results['NI']['albums'], **results['MK']['albums'], **results['NO']['albums'], **results['OM']['albums'], **results['PS']['albums'], **results['PA']['albums'], **results['PY']['albums'], **results['PE']['albums'], **results['PH']['albums'], **results['PL']['albums'], **results['PT']['albums'], **results['QA']['albums'], **results['RO']['albums'], **results['RU']['albums'], **results['SA']['albums'], **results['RS']['albums'], **results['SG']['albums'], **results['SK']['albums'], **results['SI']['albums'], **results['ZA']['albums'], **results['ES']['albums'], **results['SE']['albums'], **results['CH']['albums'], **results['TW']['albums'], **results['TH']['albums'], **results['TN']['albums'], **results['TR']['albums'], **results['UA']['albums'], **results['AE']['albums'], **results['GB']['albums'], **results['US']['albums'], **results['UY']['albums'], **results['VN']['albums']}Weird thing is, I didn't see any change of results from just using a couple countries. Maybe they overwrote each other?
@nleroy917 commented on GitHub (Oct 16, 2020):
Here is conversation I had with @stephanebruckert about which market codes to include. Spotify (to my knowledge) doesn't say which market codes they support and that was the most reliable and recent list I could find.
@shillshocked commented on GitHub (Oct 16, 2020):
I got those country codes directly from http://everynoise.com/new_releases_by_genre.cgi which is run by a Spotify employee. They should be the most recent. I even modified my client.py in my home and /usr/lib/python3.8 site packages to use them by default.
Anyways, the challenge I am having using this functionality is that I cannot merge all the albums from each country. I'm scraping Spotify new releases through a recursive alphanumeric search, but it doesn't pick stuff outside whichever region I use. With this, I can get them all, but I have no idea how to put all the ['CC']['albums'] into just a deduped (or even not deduped) singular ['albums'] so I can print them all out to file. I could parse the json manually in this case except that I have to run through each level of next albums when it goes past the 50 album api limit. Because each country has different total albums, this makes it all the more difficult.
I have tested all the country codes and they each deliver their own total if you calculate it, it's done through this:
albumtotals = [ results['AL']['albums']['total'], results['DZ']['albums']['total'], results['AD']['albums']['total'], results['AR']['albums']['total'], results['AU']['albums']['total'], results['AT']['albums']['total'], results['BH']['albums']['total'], results['BY']['albums']['total'], results['BE']['albums']['total'], results['BO']['albums']['total'], results['BA']['albums']['total'], results['BR']['albums']['total'], results['BG']['albums']['total'], results['CA']['albums']['total'], results['CL']['albums']['total'], results['CO']['albums']['total'], results['CR']['albums']['total'], results['HR']['albums']['total'], results['CY']['albums']['total'], results['CZ']['albums']['total'], results['DK']['albums']['total'], results['DO']['albums']['total'], results['EC']['albums']['total'], results['EG']['albums']['total'], results['SV']['albums']['total'], results['EE']['albums']['total'], results['FI']['albums']['total'], results['FR']['albums']['total'], results['DE']['albums']['total'], results['GR']['albums']['total'], results['GT']['albums']['total'], results['HN']['albums']['total'], results['HK']['albums']['total'], results['HU']['albums']['total'], results['IS']['albums']['total'], results['IN']['albums']['total'], results['ID']['albums']['total'], results['IE']['albums']['total'], results['IL']['albums']['total'], results['IT']['albums']['total'], results['JP']['albums']['total'], results['JO']['albums']['total'], results['KZ']['albums']['total'], results['XK']['albums']['total'], results['KW']['albums']['total'], results['LV']['albums']['total'], results['LB']['albums']['total'], results['LI']['albums']['total'], results['LT']['albums']['total'], results['LU']['albums']['total'], results['MY']['albums']['total'], results['MT']['albums']['total'], results['MX']['albums']['total'], results['MD']['albums']['total'], results['MC']['albums']['total'], results['ME']['albums']['total'], results['MA']['albums']['total'], results['NL']['albums']['total'], results['NZ']['albums']['total'], results['NI']['albums']['total'], results['MK']['albums']['total'], results['NO']['albums']['total'], results['OM']['albums']['total'], results['PS']['albums']['total'], results['PA']['albums']['total'], results['PY']['albums']['total'], results['PE']['albums']['total'], results['PH']['albums']['total'], results['PL']['albums']['total'], results['PT']['albums']['total'], results['QA']['albums']['total'], results['RO']['albums']['total'], results['RU']['albums']['total'], results['SA']['albums']['total'], results['RS']['albums']['total'], results['SG']['albums']['total'], results['SK']['albums']['total'], results['SI']['albums']['total'], results['ZA']['albums']['total'], results['ES']['albums']['total'], results['SE']['albums']['total'], results['CH']['albums']['total'], results['TW']['albums']['total'], results['TH']['albums']['total'], results['TN']['albums']['total'], results['TR']['albums']['total'], results['UA']['albums']['total'], results['AE']['albums']['total'], results['GB']['albums']['total'], results['US']['albums']['total'], results['UY']['albums']['total'], results['VN']['albums']['total'] ]You can calculate the highest total through: total=max(albumtotals)
But I don't know if this would require one to go through each country individually to obtain all the results, or if it could be done all at once. Any ideas on this?
@shillshocked commented on GitHub (Jan 22, 2025):
Interestingly, I tried using the Spotify API directly, and then compared the results to a Spotipy search with a default installation (it normally has 60 countries in client.py):
it's the same track, the difference is spotify api checking all 250 markets one at a time, and spotipy checking all at once.
With spotipy checking available markets, it found Ed Sheeran's "Shape of You" available in 184 markets. With the web api checking 250 markets individually, it found it available in each.
Spotify API with Spotipy (available_markets() all at once): This method gives you a list of available markets for a specific track in a single request, but it may return fewer markets due to specific track availability data that includes licensing restrictions, regional differences, or a limited set of markets.
Spotify API Checking All 250 Markets Individually: This approach checks each of the 250 markets individually, which might not be restricted by licensing agreements and could show more markets where the track is available.
The key reason for the difference is that when checking markets one by one, you're likely querying availability for the track in all Spotify markets, without the restrictions that available_markets() imposes. On the other hand, when you use available_markets(), the response is limited to those markets where the track is explicitly available, which reflects licensing and distribution rights.
And an update for client.py:
(copy client.py to client.bak first)
for 184 markets, change to:
or for all 250 markets, change to:
@shillshocked commented on GitHub (Jan 22, 2025):
Also, in theory, you might want to also update this:
to this (for 184 markets):
or to this (for 250 markets):
@shillshocked commented on GitHub (Feb 6, 2025):
By the way, in the experiments I have done, I found Spotify seems to be using these particular 185 country regions regularly (different from the previous 184 list by some). The other ones are probably unnecessary.
Here is the updated list:
First change (for coding in python using spotipy):
def search_markets(self, q, limit=10, offset=0, type="track", markets=['AD','AE','AG','AL','AM','AO','AR','AT','AU','AZ','BA','BB','BD','BE','BF','BG','BH','BI','BJ','BN','BO','BR','BS','BT','BW','BY','BZ','CA','CD','CG','CH','CI','CL','CM','CO','CR','CV','CW','CY','CZ','DE','DJ','DK','DM','DO','DZ','EC','EE','EG','ES','ET','FI','FJ','FM','FR','GA','GB','GD','GE','GH','GM','GN','GQ','GR','GT','GW','GY','HK','HN','HR','HT','HU','ID','IE','IL','IN','IQ','IS','IT','JM','JO','JP','KE','KG','KH','KI','KM','KN','KR','KW','KZ','LA','LB','LC','LI','LK','LR','LS','LT','LU','LV','LY','MA','MC','MD','ME','MG','MH','MK','ML','MN','MO','MR','MT','MU','MV','MW','MX','MY','MZ','NA','NE','NG','NI','NL','NO','NP','NR','NZ','OM','PA','PE','PG','PH','PK','PL','PR','PS','PT','PW','PY','QA','RO','RS','RW','SA','SB','SC','SE','SG','SI','SK','SL','SM','SN','SR','ST','SV','SZ','TD','TG','TH','TJ','TL','TN','TO','TR','TT','TV','TW','TZ','UA','UG','US','UY','UZ','VC','VE','VN','VU','WS','XK','ZA','ZM','ZW'], total=None):Second change, in client.py in your (pip) spotipy folder: