starred/zotify
1
0
Fork 0
mirror of https://github.com/Googolplexed0/zotify.git synced 2026-04-24 22:05:53 +03:00
Code Issues 21 Projects Releases 10 Packages Wiki Activity

[GH-ISSUE #41] [Major Feature Proposal & Discussion] High Efficiency API Request System #36

New issue
Open
opened 2026-02-27 04:57:12 +03:00 by kerem · 20 comments
kerem commented 2026-02-27 04:57:12 +03:00
Owner
Copy link

Originally created by @Googolplexed0 on GitHub (Aug 1, 2025).
Original GitHub issue: https://github.com/Googolplexed0/zotify/issues/41

UPDATE

New efficient-api branch is live, give it a shot by running pipx install git+https://github.com/Googolplexed0/zotify.git@efficient-api.

Goals:

  • Significantly reduce / entirely prevent API timeouts (failed to fetch audio key! error)
  • Do not require idling / waiting / cooldowns
  • Write concise, easily readable and maintainable code

Paths:

  • Reduce API calls as much as possible
    • Always use bulk calls if available
    • Carry down sub-item metadata included in parent metadata request
  • Use custom class storage objects for API responses
  • Alternate between fastest-to-download and slowest-to-download items
    • Time downloading == Time not calling API -> decrease average APIs/sec over total run

Benefits:

  • Fewer API calls per object (see Expected # Reduction in API Calls chart)
  • Overall faster execution
  • Rewritten logic (hopefully) ensures no missed edge cases
  • Fewer errors/bugs into the future
  • More "pythonic" -> More likely to receive contributions from community members
    • Better community support and upkeep likelihood

Table: Expected Reduction in API Calls for Metadata (shown in ~big O notation)

(n = # objects needed to fetch, t = # contained tracks, a = # contained albums, n, t, a > 0)
(*** = container metadata arrives with 20 fully-populated items)

Object Type # of Calls (Currently) # of Calls (Theoretical) Approximate Reduction
Tracks n 1 + (n - 1) // 100 100:1
Playlists n(2 + t) n[2 + (t - 101)*** // 100] 100:1
Albums n(2 + t) [1 + (n - 1) // 20 ][2 + (t - 21)*** // 50 ] 100:1
Artists 2n + 2na + nat [1 + (n - 1) // 50 ] + n[1 + (a - 1) // 50 ] + na[1 + (t - 1) // 50 ] 75:1
Example Reductions # of Calls (Currently) # of Calls (Theoretical)
60 Tracks 60 calls 1 + 0 = 1 call
180 Tracks 180 calls 1 + 1 = 2 calls
1 Playlist, 10 Tracks 1(2 + 10) = 12 calls 1(2 + -1) = 1 call
1 Playlist, 50 Tracks 1(2 + 50) = 52 calls 1(2 + -1) = 1 calls
1 Playlist, 150 Tracks 1(2 + 150) = 152 calls 1(2 + 0) = 2 calls
10 Playlist, 250 Tracks Each 10(2 + 250) = 2,520 calls 10(2 + 5) = 70 calls
3 Albums, 15 Tracks Each 3(2 + 15) = 51 calls (1 + 0)(2 + -1) = 1 call
30 Albums, 150 Tracks Each 30(2 + 150) = 4,560 calls (1 + 1)(2 + 2) = 8 calls
2 Artists, 5 Albums Each, 30 Tracks Each 2(2) + 2(2)(5) + (2)(5)(30) = 324 calls (1 + 0) + 2(1 + 0) + 2(5)(1 + 0) = 13 calls
10 Artists, 35 Albums Each, 20 Tracks Each 2(10) + 2(10)(35) + (10)(35)(20) = 7,720 calls (1 + 0) + 10(1 + 0) + 10(35)(1 + 0) = 361 calls

Tradeoffs:

  • Users cannot choose download order N/A, this will be optional as a new config option.
    • Required for Path #3
  • Metadata classes heavily rely on inheritance, which can be hard to read / understand / maintain
    • Base classes may be hard to understand, but the metadata-holding-classes are straightforward
    • Future changes should be metadata-holding-class specific
  • Others I may have missed (comment them below)

Discussion:

Is this something you guys believe would be worth coding and integrating? It would require a major rework similar to the original repo's v1.0-dev branch. I don't mind the effort investment, but I don't want to waste my time if no one would find it useful. Are rate limits a frequent pain point for you? Does download order being semi-random bother you? Is there some other tweak that should be considered to further reduce API requests usage? Please discuss with me and each other in the comments below.

Originally created by @Googolplexed0 on GitHub (Aug 1, 2025). Original GitHub issue: https://github.com/Googolplexed0/zotify/issues/41 # UPDATE # New [efficient-api branch](https://github.com/Googolplexed0/zotify/tree/efficient-api) is live, give it a shot by running `pipx install git+https://github.com/Googolplexed0/zotify.git@efficient-api`. ## Goals: ## - Significantly reduce / entirely prevent API timeouts (failed to fetch audio key! error) - Do not require idling / waiting / cooldowns - Write concise, easily readable and maintainable code ## Paths: ## - Reduce API calls as much as possible - Always use bulk calls if available - Carry down sub-item metadata included in parent metadata request - Use custom class storage objects for API responses - Alternate between fastest-to-download and slowest-to-download items - Time downloading == Time not calling API -> decrease average APIs/sec over total run ## Benefits: ## - Fewer API calls per object (see Expected # Reduction in API Calls chart) - Overall faster execution - Rewritten logic (hopefully) ensures no missed edge cases - Fewer errors/bugs into the future - More "pythonic" -> More likely to receive contributions from community members - Better community support and upkeep likelihood ### Table: Expected Reduction in API Calls for Metadata (shown in ~big O notation) ### *(`n` = # objects needed to fetch, `t` = # contained tracks, `a` = # contained albums, `n, t, a > 0`)* *(`***` = container metadata arrives with 20 fully-populated items)* | Object Type | # of Calls (Currently) | # of Calls (Theoretical) | Approximate Reduction | |-------------|------------------------|--------------------------|-------------------| | Tracks | `n` | `1 + (n - 1) // 100` | 100:1 | | Playlists | `n(2 + t)` | `n[2 + (t - 101)*** // 100]` | 100:1 | | Albums | `n(2 + t)` | `[1 + (n - 1) // 20 ][2 + (t - 21)*** // 50 ]` | 100:1 | | Artists | `2n + 2na + nat`| `[1 + (n - 1) // 50 ] + n[1 + (a - 1) // 50 ] + na[1 + (t - 1) // 50 ]` | 75:1 | | Example Reductions | # of Calls (Currently) | # of Calls (Theoretical) | |-----------|----------|------------------------------------------------------------------| | 60 Tracks | **60 calls** | 1 + 0 = **1 call** | | 180 Tracks | **180 calls** | 1 + 1 = **2 calls** | | 1 Playlist, 10 Tracks | 1(2 + 10) = **12 calls** | 1(2 + -1) = **1 call** | | 1 Playlist, 50 Tracks | 1(2 + 50) = **52 calls** | 1(2 + -1) = **1 calls** | | 1 Playlist, 150 Tracks | 1(2 + 150) = **152 calls** | 1(2 + 0) = **2 calls** | | 10 Playlist, 250 Tracks Each | 10(2 + 250) = **2,520 calls** | 10(2 + 5) = **70 calls**| | 3 Albums, 15 Tracks Each | 3(2 + 15) = **51 calls** | (1 + 0)(2 + -1) = **1 call**| | 30 Albums, 150 Tracks Each | 30(2 + 150) = **4,560 calls** | (1 + 1)(2 + 2) = **8 calls**| | 2 Artists, 5 Albums Each, 30 Tracks Each | 2(2) + 2(2)(5) + (2)(5)(30) = **324 calls** | (1 + 0) + 2(1 + 0) + 2(5)(1 + 0) = **13 calls**| | 10 Artists, 35 Albums Each, 20 Tracks Each | 2(10) + 2(10)(35) + (10)(35)(20) = **7,720 calls** | (1 + 0) + 10(1 + 0) + 10(35)(1 + 0) = **361 calls**| ## Tradeoffs: ## - ~~Users cannot choose download order~~ N/A, this will be optional as a new config option. - ~~Required for `Path #3`~~ - Metadata classes heavily rely on inheritance, which can be hard to read / understand / maintain - Base classes may be hard to understand, but the metadata-holding-classes are straightforward - Future changes should be metadata-holding-class specific - Others I may have missed (comment them below) ## Discussion: ## Is this something you guys believe would be worth coding and integrating? It would require a major rework similar to the original repo's `v1.0-dev` branch. I don't mind the effort investment, but I don't want to waste my time if no one would find it useful. Are rate limits a frequent pain point for you? Does download order being semi-random bother you? Is there some other tweak that should be considered to further reduce API requests usage? Please discuss with me and each other in the comments below.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@chr15t0ph commented on GitHub (Aug 1, 2025):

This is a VERY good idea. And would be a MAJOR improvement.

(And: Yes, rate limits are a frequent pain point for me. Your calculations are also related to the time needed for skipping existing files in case of restarting downloads, as described in my other ticket.)

The only downside seems the significant implementation effort for refactoring lots of existing code? "Chapeau" that you not mind the effort for investment.

<!-- gh-comment-id:3142762983 --> @chr15t0ph commented on GitHub (Aug 1, 2025): This is a VERY good idea. And would be a MAJOR improvement. (And: Yes, rate limits are a frequent pain point for me. Your calculations are also related to the time needed for skipping existing files in case of restarting downloads, as described in my other ticket.) The only downside seems the significant implementation effort for refactoring lots of existing code? "Chapeau" that you not mind the effort for investment.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@DerKO9 commented on GitHub (Aug 1, 2025):

Download order does not matter to me. As long as the exported playlist files have the correct order of songs in them, I'm happy. If not, it is not a big deal, the improved performance would still be welcome.

I'm not a python guy and have been hesitant to help make PRs of features id like, but if the code ends up being more readable, it may make the project more approachable for people like me to help out

<!-- gh-comment-id:3145530421 --> @DerKO9 commented on GitHub (Aug 1, 2025): Download order does not matter to me. As long as the exported playlist files have the correct order of songs in them, I'm happy. If not, it is not a big deal, the improved performance would still be welcome. I'm not a python guy and have been hesitant to help make PRs of features id like, but if the code ends up being more readable, it may make the project more approachable for people like me to help out
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@trustosas commented on GitHub (Aug 6, 2025):

I don't mind rate limits and audio key whatnots. I have 3 different credentials for 3 different accounts for that.
Rate limit = Switch credentials. Audio key errors gone.

<!-- gh-comment-id:3159924605 --> @trustosas commented on GitHub (Aug 6, 2025): I don't mind rate limits and audio key whatnots. I have 3 different credentials for 3 different accounts for that. Rate limit = Switch credentials. Audio key errors gone.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@rcMarty commented on GitHub (Aug 13, 2025):

Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help?

<!-- gh-comment-id:3183113379 --> @rcMarty commented on GitHub (Aug 13, 2025): Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help?
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Aug 14, 2025):

Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help?

I finished an alpha version of this proposal and added the help wanted tag so I could request bug testing from users here. Unfortunately, while in the process of getting everything ready to publish, the librespot OAuth/keymaster flow broke (see #48). As a result, I have refrained from releasing the new alpha branch and will wait to do so until everything in main is back in working order.

Anyone who wants to get the new backend published faster can help by working towards an OAuth login5 implementation for librespot-python (again, see #48).

Since #48 has been fixed, the new efficient-api branch will be released soon!

<!-- gh-comment-id:3186679349 --> @Googolplexed0 commented on GitHub (Aug 14, 2025): > Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help? I finished an alpha version of this proposal and added the `help wanted` tag so I could request bug testing from users here. Unfortunately, while in the process of getting everything ready to publish, the librespot OAuth/keymaster flow broke (see #48). As a result, I have refrained from releasing the new alpha branch and will wait to do so until everything in main is back in working order. ~~Anyone who wants to get the new backend published faster can help by working towards an OAuth login5 implementation for [librespot-python](https://github.com/kokarare1212/librespot-python) (again, see #48).~~ Since #48 has been fixed, the new efficient-api branch will be released soon!
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@anbe108 commented on GitHub (Aug 22, 2025):

Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.

<!-- gh-comment-id:3212938133 --> @anbe108 commented on GitHub (Aug 22, 2025): Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Aug 23, 2025):

Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.

Would be a good add for those using it repeatedly/interactively like an application, but may also detract from those using it as a command line util. Finding a middle ground there may be difficult and I am fairly certain that login calls do not make up a significant number of API requests anyways.

<!-- gh-comment-id:3216244757 --> @Googolplexed0 commented on GitHub (Aug 23, 2025): > Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls. Would be a good add for those using it repeatedly/interactively like an application, but may also detract from those using it as a command line util. Finding a middle ground there may be difficult and I am fairly certain that login calls do not make up a significant number of API requests anyways.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@anbe108 commented on GitHub (Aug 24, 2025):

Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.

Would be a good add for those using it repeatedly/interactively like an application, but may also detract from those using it as a command line util. Finding a middle ground there may be difficult and I am fairly certain that login calls do not make up a significant number of API requests anyways.

In that case a CLI toggle using --persist or something can be a good option. In my use case i have seen if I call login too fast there is (I suspect) some kind of rate limit which is not reproducible.

<!-- gh-comment-id:3217682628 --> @anbe108 commented on GitHub (Aug 24, 2025): > > Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls. > > Would be a good add for those using it repeatedly/interactively like an application, but may also detract from those using it as a command line util. Finding a middle ground there may be difficult and I am fairly certain that login calls do not make up a significant number of API requests anyways. In that case a CLI toggle using --persist or something can be a good option. In my use case i have seen if I call login too fast there is (I suspect) some kind of rate limit which is not reproducible.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Aug 25, 2025):

In that case a CLI toggle using --persist or something can be a good option.

This is a good idea. An extra add-on arg so that it doesn't change default behavior. Will implement something like this soon. Feel free to make a feature request for this if you want it ASAP.

<!-- gh-comment-id:3218623296 --> @Googolplexed0 commented on GitHub (Aug 25, 2025): > In that case a CLI toggle using `--persist` or something can be a good option. This is a good idea. An extra add-on arg so that it doesn't change default behavior. Will implement something like this soon. Feel free to make a feature request for this if you want it ASAP.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Aug 25, 2025):

New efficient-api branch is live, give it a shot by running pipx install git+https://github.com/Googolplexed0/zotify.git@efficient-api.

<!-- gh-comment-id:3218629599 --> @Googolplexed0 commented on GitHub (Aug 25, 2025): New [efficient-api branch](https://github.com/Googolplexed0/zotify/tree/efficient-api) is live, give it a shot by running `pipx install git+https://github.com/Googolplexed0/zotify.git@efficient-api`.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@JarrodDoyle commented on GitHub (Sep 7, 2025):

What's the current state of this? Using the efficient-api branch I still hit "failed to fetch audio key" after a couple of albums (setting a bulk wait of 30 doesn't seem to help). Unsure if that's still expected at this point.

<!-- gh-comment-id:3263593889 --> @JarrodDoyle commented on GitHub (Sep 7, 2025): What's the current state of this? Using the `efficient-api` branch I still hit "failed to fetch audio key" after a couple of albums (setting a bulk wait of 30 doesn't seem to help). Unsure if that's still expected at this point.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@chr15t0ph commented on GitHub (Sep 7, 2025):

Hi, I am back from a vacation. I have tested it. (currently on https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu).
I am not sure how many refactorings you did, and there might be errors/observations unrelated to your high-efficiency-API request systems. However, I did run your zotify to the same playlists before my vacation and these errors did not appear. So from my perspective they are "new".

== Observation 1 ==
The "[●∙∙] Fetching playlist information..." for the 946 files of the example playlist is VERY fast!! Well done!

== Observation 2 ==
Before the vacation I already downloaded 906 of 946 files from https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu. When recovering such a big playlist, there seems still an approximately one second (or at least a half second) delay between several of these outputs:
[●∙∙] Preparing download...
[●∙∙] Preparing download...

I would have expected a much faster skipping. There seems to be applied some of the zotify delays... This might not be ideal.

== Observation 3 ==
There is now a problem with writing metadata.
For the sake of completeness, I have a ffmpeg in my path! As said before, only the zotify-behaviour is new, not my environment.
The mp3 files ARE created! But as the error says, they have no metadata.

zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu

[∙∙∙] Logging in...
[∙∙●] Fetching playlist information...
[●∙∙] Parsing playlist information...
[∙∙∙] Parsing playlist information...
[∙∙∙] Fetching bulk track/disc total information...
[●∙∙] Preparing download...
[●∙∙] Converting file...
###   ERROR:  FAILED TO WRITE METADATA   ###
###   Ensure FFMPEG is installed and added to your PATH   ###

Traceback (most recent call last):
  File "C:\Program Files\Python313\Lib\site-packages\zotify\api.py", line 860, in download
self.set_audio_tags(path)
~~~~~~~~~~~~~~~~~~~^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\zotify\api.py", line 739, in set_audio_tags
tags[TOTALDISCS] = self.album.total_discs
~~~~^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 603, in __setitem__
self.set(norm_key, val)
~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 500, in set
val = MetadataItem(md_type, md_sanitizer, val)
  File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 67, in __init__
self.values = val
^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 92, in values
v = self.type(v)
ValueError: invalid literal for int() with base 10: ''

== Epilogue ==
I have some more biggie playlists. I could do some more tests. But I will wait for your feedback. (Hope the observations are helpful and related to the refactoring and, thus, not distracting.)

<!-- gh-comment-id:3263683126 --> @chr15t0ph commented on GitHub (Sep 7, 2025): Hi, I am back from a vacation. I have tested it. (currently on https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu). I am not sure how many refactorings you did, and there might be errors/observations unrelated to your high-efficiency-API request systems. However, I did run your zotify to the same playlists before my vacation and these errors did not appear. So from my perspective they are "new". == Observation 1 == The "[●∙∙] Fetching playlist information..." for the 946 files of the example playlist is VERY fast!! Well done! == Observation 2 == Before the vacation I already downloaded 906 of 946 files from https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu. When recovering such a big playlist, there seems still an approximately one second (or at least a half second) delay between several of these outputs: [●∙∙] Preparing download... [●∙∙] Preparing download... I would have expected a much faster skipping. There seems to be applied some of the zotify delays... This might not be ideal. == Observation 3 == There is now a problem with writing metadata. For the sake of completeness, I have a ffmpeg in my path! As said before, only the zotify-behaviour is new, not my environment. The mp3 files ARE created! But as the error says, they have no metadata. ``` zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu [∙∙∙] Logging in... [∙∙●] Fetching playlist information... [●∙∙] Parsing playlist information... [∙∙∙] Parsing playlist information... [∙∙∙] Fetching bulk track/disc total information... [●∙∙] Preparing download... [●∙∙] Converting file... ### ERROR: FAILED TO WRITE METADATA ### ### Ensure FFMPEG is installed and added to your PATH ### Traceback (most recent call last): File "C:\Program Files\Python313\Lib\site-packages\zotify\api.py", line 860, in download self.set_audio_tags(path) ~~~~~~~~~~~~~~~~~~~^^^^^^ File "C:\Program Files\Python313\Lib\site-packages\zotify\api.py", line 739, in set_audio_tags tags[TOTALDISCS] = self.album.total_discs ~~~~^^^^^^^^^^^^ File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 603, in __setitem__ self.set(norm_key, val) ~~~~~~~~^^^^^^^^^^^^^^^ File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 500, in set val = MetadataItem(md_type, md_sanitizer, val) File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 67, in __init__ self.values = val ^^^^^^^^^^^ File "C:\Program Files\Python313\Lib\site-packages\music_tag\file.py", line 92, in values v = self.type(v) ValueError: invalid literal for int() with base 10: '' ``` == Epilogue == I have some more biggie playlists. I could do some more tests. But I will wait for your feedback. (Hope the observations are helpful and related to the refactoring and, thus, not distracting.)
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Sep 7, 2025):

What's the current state of this?

Current state is that I am still making changes and improving things daily. Just finished a huge upgrade to v0.10.10 with some major speed boosts.

I still hit "failed to fetch audio key" after a couple of albums (setting a bulk wait of 30 doesn't seem to help). Unsure if that's still expected at this point.

Honestly, no, that isn't supposed to happen. Can you give a ballpark number of how many tracks you made it before the failures started? I was having some really good luck with it previously and getting through well over a hundred songs at 0 BULK_WAIT_TIME with no issues. Lately it hasn't seemed to be getting as far. Maybe stronger API restrictions have been rolled out?

I have reduced metadata calls as low as feasible without making major functionality sacrifices. At this point, I think we are hitting an API limits when requesting the actual song bytes. Not sure if there is even a path to optimize there. Hopefully I can find something there soon.

<!-- gh-comment-id:3263912266 --> @Googolplexed0 commented on GitHub (Sep 7, 2025): > What's the current state of this? Current state is that I am still making changes and improving things daily. Just finished a huge upgrade to v0.10.10 with some major speed boosts. > I still hit "failed to fetch audio key" after a couple of albums (setting a bulk wait of 30 doesn't seem to help). Unsure if that's still expected at this point. Honestly, no, that isn't supposed to happen. Can you give a ballpark number of how many tracks you made it before the failures started? I was having some really good luck with it previously and getting through well over a hundred songs at 0 BULK_WAIT_TIME with no issues. Lately it hasn't seemed to be getting as far. Maybe stronger API restrictions have been rolled out? I have reduced metadata calls as low as feasible without making major functionality sacrifices. At this point, I think we are hitting an API limits when requesting the actual song bytes. Not sure if there is even a path to optimize there. Hopefully I can find something there soon.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Sep 7, 2025):

The "[●∙∙] Fetching playlist information..." for the 946 files of the example playlist is VERY fast!! Well done!

Yes! It actually should be even faster now after v0.10.10.

there seems still an approximately one second (or at least a half second) delay between several of these outputs

Is that at least faster than the legacy branch? Most of the delay might be caused by print statements since all of the metadata fetching is done ahead of time so we aren't waiting on the network.

invalid literal for int() with base 10: ''

Should be fixed with v0.10.11 (a9d6b57e6a). An old bug I fixed a while ago that has to do with sharing metadata across duplicate albums came back when I implemented the 1000x speedup for duplicate checks.

<!-- gh-comment-id:3263973040 --> @Googolplexed0 commented on GitHub (Sep 7, 2025): > The "[●∙∙] Fetching playlist information..." for the 946 files of the example playlist is VERY fast!! Well done! Yes! It actually should be even faster now after v0.10.10. > there seems still an approximately one second (or at least a half second) delay between several of these outputs Is that at least faster than the legacy branch? Most of the delay might be caused by print statements since all of the metadata fetching is done ahead of time so we aren't waiting on the network. > invalid literal for int() with base 10: '' Should be fixed with v0.10.11 (a9d6b57e6af9a352d84780c27a9fdbbe867c50d4). An old bug I fixed a while ago that has to do with sharing metadata across duplicate albums came back when I implemented the 1000x speedup for duplicate checks.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@chr15t0ph commented on GitHub (Sep 8, 2025):

Perfect. Metadata issue is fixed. Now, that the error text blocks are out of the way, we can test the actual functionality better.

The 1sec delay is there (with -rt and without -rt). In detail, for the sake of completeness: When there are existing files, that are skipped, each existing file produces a "[●∙∙] Preparing Download..." line (I have --print-skips false). Between each printout, there is a significant 1 sec delay. (It seems to be far more than a print statement; actually, it seems not faster than the legacy branch, it seems to be an identical delay.) It should be very easy to reproduce. (The only tricky thing is, that the download order seems to have changed between 9.x and 10.x, thus, one has to begin "clean-sleave": Begin downloading a new playlist, with three to five songs downloaded. Stop. Restart. (If you do it with an old half-downloaded playlist, it will just start downloading songs, without skipping anything, because the new order does not yet trigger skipping..., thus, use a clean sleave test setup.) If the skipping delays are not reproducable for you, then please do not hesitate to tell me that "I see ghosts".

Again my parameter list (this one without -rt true, because fast reproducability is focused on):

zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu

Interim question: Has also the standard naming pattern changed between 9.x and 10.x? It seems to me that way. I observed duplicates of my 906 of 946 previously downloaded songs. But here I am not absolutely sure, and I might see ghosts...

I have now started a real-time download, again clean-sleave, and will report whether there will be bans:

zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -rt true -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu
<!-- gh-comment-id:3266388338 --> @chr15t0ph commented on GitHub (Sep 8, 2025): Perfect. Metadata issue is fixed. Now, that the error text blocks are out of the way, we can test the actual functionality better. The 1sec delay is there (with -rt and without -rt). In detail, for the sake of completeness: When there are existing files, that are skipped, each existing file produces a "[●∙∙] Preparing Download..." line (I have --print-skips false). Between each printout, there is a significant 1 sec delay. (It seems to be far more than a print statement; actually, it seems *not* faster than the legacy branch, it seems to be an identical delay.) It should be very easy to reproduce. (The only tricky thing is, that the download order seems to have changed between 9.x and 10.x, thus, one has to begin "clean-sleave": Begin downloading a *new* playlist, with three to five songs downloaded. Stop. Restart. (If you do it with an old half-downloaded playlist, it will just start downloading songs, without skipping anything, because the new order does not yet trigger skipping..., thus, use a clean sleave test setup.) If the skipping delays are not reproducable for you, then please do not hesitate to tell me that "I see ghosts". Again my parameter list (this one without -rt true, because fast reproducability is focused on): ``` zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu ``` Interim question: Has also the standard naming pattern changed between 9.x and 10.x? It seems to me that way. I observed duplicates of my 906 of 946 previously downloaded songs. But here I am not absolutely sure, and I might see ghosts... I have now started a real-time download, again clean-sleave, and will report whether there will be bans: ``` zotify -rp . --creds . --print-skips false --download-lyrics false --retry-attempts 4 --codec mp3 -q high --ffmpeg-log-level warn -rt true -u «MYUSERID» --url https://open.spotify.com/playlist/1A000r9CutJcTGMVwCk1Xu ```
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@DamianRyse commented on GitHub (Sep 10, 2025):

This is a great improvement. I'm currently testing it as well. What I personally don't understand is why Zotify still enumerates over tracks that have been downloaded already. When I want to download a playlist, it fetches the metadata in bulk and can then compare it to the .song_ids file and filter the already present files out before starting to try to download files. But instead, it still goes file by file.

<!-- gh-comment-id:3274759185 --> @DamianRyse commented on GitHub (Sep 10, 2025): This is a great improvement. I'm currently testing it as well. What I personally don't understand is why Zotify still enumerates over tracks that have been downloaded already. When I want to download a playlist, it fetches the metadata in bulk and can then compare it to the `.song_ids` file and filter the already present files out before starting to try to download files. But instead, it still goes file by file.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@chr15t0ph commented on GitHub (Sep 10, 2025):

After some days of testing. I used my standard account. And had some >24h cool-down. And I additionally used a secondary account. In conclusion:

The connection closings / "failed to fetch audio key" have unfortunately NOT improved. That is my experience.

<!-- gh-comment-id:3275991342 --> @chr15t0ph commented on GitHub (Sep 10, 2025): After some days of testing. I used my standard account. And had some >24h cool-down. And I additionally used a secondary account. In conclusion: The connection closings / "failed to fetch audio key" have unfortunately NOT improved. That is my experience.
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Sep 11, 2025):

What I personally don't understand is why Zotify still enumerates over tracks that have been downloaded already. When I want to download a playlist, it fetches the metadata in bulk and can then compare it to the .song_ids file and filter the already present files out before starting to try to download files. But instead, it still goes file by file.

This was done to maintain code readability, logical flow, and all previous functionality. I wanted to get every boost I could without tradeoffs. You are very correct that filtering out tracks to skip can be done in bulk before iterating, resulting in a significant speed-up, but this will supersede the ALWAYS_CHECK_LYRICS config.

In all honesty the improvements offered by OPTIMIZED_DOWNLOADING after bulk skipping will be worth it and can always be toggled off if a user truly cares about all lyrics. Going to do my best to find a way to minimize the impact though. TL;DR bulk skipping will be implemented in v0.10.12.

When there are existing files, that are skipped, each existing file produces a "[●∙∙] Preparing Download..." line (I have --print-skips false). Between each printout, there is a significant 1 sec delay.

The above change in v0.10.12 will make this a non-issue.

EDIT: v0.10.12 implemented in 2d3c82bcd3

<!-- gh-comment-id:3277662394 --> @Googolplexed0 commented on GitHub (Sep 11, 2025): > What I personally don't understand is why Zotify still enumerates over tracks that have been downloaded already. When I want to download a playlist, it fetches the metadata in bulk and can then compare it to the `.song_ids` file and filter the already present files out before starting to try to download files. But instead, it still goes file by file. This was done to maintain code readability, logical flow, and all previous functionality. I wanted to get every boost I could without tradeoffs. You are very correct that filtering out tracks to skip can be done in bulk before iterating, resulting in a significant speed-up, but this will supersede the `ALWAYS_CHECK_LYRICS` config. In all honesty the improvements offered by `OPTIMIZED_DOWNLOADING` after bulk skipping will be worth it and can always be toggled off if a user truly cares about _all_ lyrics. Going to do my best to find a way to minimize the impact though. TL;DR bulk skipping will be implemented in v0.10.12. > When there are existing files, that are skipped, each existing file produces a "[●∙∙] Preparing Download..." line (I have --print-skips false). Between each printout, there is a significant 1 sec delay. The above change in v0.10.12 will make this a non-issue. **EDIT: v0.10.12 implemented in 2d3c82bcd3e0da5d3f76076d77daf76b9478245e**
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@KyrillosL commented on GitHub (Oct 12, 2025):

There is no PR opened yet so I'm posting here. L1752 in api, shouldn't be tuple[list[list[Playlist]] instead of tuple[list[list[Artist]]?
def select_user_playlists(self, user_playlist_resps: list[None | dict]) -> tuple[list[list[Artist]], list[list[dict]]]:

<!-- gh-comment-id:3394117825 --> @KyrillosL commented on GitHub (Oct 12, 2025): There is no PR opened yet so I'm posting here. L1752 in api, shouldn't be `tuple[list[list[Playlist]]` instead of `tuple[list[list[Artist]]`? [ def select_user_playlists(self, user_playlist_resps: list[None | dict]) -> tuple[list[list[Artist]], list[list[dict]]]:](https://github.com/Googolplexed0/zotify/blob/77da3ffd599d6720206d5aab5012157f72908d82/zotify/api.py#L1752)
kerem commented 2026-02-27 04:57:13 +03:00
Author
Owner
Copy link

@Googolplexed0 commented on GitHub (Oct 30, 2025):

shouldn't be tuple[list[list[Playlist]] instead of tuple[list[list[Artist]]?

You were correct, but that function and a lot of similar boilerplate code became unnecessary after the introduction of the UserItem extension of the Query. Good catch nonetheless.

<!-- gh-comment-id:3465904976 --> @Googolplexed0 commented on GitHub (Oct 30, 2025): > shouldn't be `tuple[list[list[Playlist]]` instead of `tuple[list[list[Artist]]`? You were correct, but that function and a lot of similar boilerplate code became unnecessary after the introduction of the `UserItem` extension of the `Query`. Good catch nonetheless.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.17.0
v0.10.26
v0.9.21
v0.9.20
v0.9.14
v0.9.11
v0.9.10
v0.9.0
v0.8.30
v0.8.20
v0.8.10
v0.8.6
v0.8.0
v0.7.15
v0.7.14
v0.7.10
v0.7.5
v0.7.0
Labels
Clear labels   
bug

   
considering

   
discussion

   
documentation

   
enhancement

   
enhancement

   
good first issue

   
help wanted

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
stale

   
wontfix
No labels
bug
considering
discussion
documentation
enhancement
enhancement
good first issue
help wanted
pull-request
question
stale
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/zotify#36
No description provided.