mirror of
https://github.com/sigma67/ytmusicapi.git
synced 2026-04-26 07:46:00 +03:00
[GH-ISSUE #196] Is get_streaming_data still working? #153
Labels
No labels
a/b
bug
documentation
enhancement
good first issue
help wanted
invalid
pull-request
question
wontfix
yt-error
yt-update
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/ytmusicapi#153
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @KoljaWindeler on GitHub (May 17, 2021).
Original GitHub issue: https://github.com/sigma67/ytmusicapi/issues/196
Hi, this is one of those "some user have a problem but not all" things.
Original issue:
https://github.com/KoljaWindeler/ytube_music_player/issues/97
I can't reproduce the issue myself but quite a few user suddenly lost the ability to listen to music.
My code would call get_streaming_data but (according to the logs) would only get an empty response {}.
As of now I use a backup method for those users but wonder why get_streaming_data isn't working anymore for so many users.
This is the interesting part:
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] No adaptiveFormat and no formats found
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] get_streaming_data(SQyu9vfW9Bk)
2021-05-17 12:46:11 ERROR (MainThread) [custom_components.ytube_music_player.media_player] {}
github.com/KoljaWindeler/ytube_music_player@dc06c8b1c3/custom_components/ytube_music_player/media_player.py (L1118)@sigma67 commented on GitHub (May 17, 2021):
It might be related to the x-goog-visitor-id. Can you have the troubled users try again with the latest version? Perhaps that already fixed the issues.
@KoljaWindeler commented on GitHub (May 18, 2021):
hah, interesting .. now I have the same issue with 17.0 ... what can i do to get you the needed infos?
@sigma67 commented on GitHub (May 18, 2021):
Thanks for being persistent, it seems the flow has changed on the YouTube side. There is now a YouTube Music specific endpoint, it no longer uses the YouTube get_video_info.
The new endpoint is
https://music.youtube.com/youtubei/v1/player. BesidesstreamingDatait also returns some other things likevideoDetails,playerConfig,storyboardsand anattestation, to prevent bots.I propose we replace the existing call with that one, but keep the method name.
@KoljaWindeler commented on GitHub (May 18, 2021):
I can totally adopt as the current result is empty anyway ;)
@sigma67 commented on GitHub (May 18, 2021):
I pushed a simple implementation, feel free to try and provide feedback :)
The method is now
get_songonly which also provides the streaming data.@KoljaWindeler commented on GitHub (May 18, 2021):
hmm ok .. i was able to install the latest version from github and test the get_song interface. It look fine but I can't decode a link from these data anymore .. so the format or the way how the data was coded seems to have changed .. damn ..
Edit: ok this is really a problem for me .. i used to load your api and the pytube lib (https://pytube.io/en/latest/).
Your code provided the streaming data (ciphered_signature + url) and the pytube.cipher.get_signature() methode decoded the ciphered_signature. Joining them together resulted in the direct streaming link that I'd forward to the various players. It turns out that those links aren't valid any more. So I'd assume that the cipher for YouTube.music and YouTube now differ. Bummer.
Any idea how I could get the direct streaming link now?
@sigma67 commented on GitHub (May 18, 2021):
I think the url is right there in the adaptiveFormats, signatureCipher. If you plug the string into urldecode you get this
But I have no idea how your code works so I might be missing something.
@KoljaWindeler commented on GitHub (May 18, 2021):
ok this is really a problem for me .. i used to load your api and the pytube lib (https://pytube.io/en/latest/).
Your code provided the streaming data (ciphered_signature + url) and the pytube.cipher.get_signature() methode decoded the ciphered_signature. Joining them together resulted in the direct streaming link that I'd forward to the various players. It turns out that those links aren't valid any more. So I'd assume that the cipher for YouTube.music and YouTube now differ. Bummer.
Any idea how I could get the direct streaming link now?
github.com/KoljaWindeler/ytube_music_player@8aea65d2c5/custom_components/ytube_music_player/media_player.py (L1164)@KoljaWindeler commented on GitHub (May 18, 2021):
some more info... the cipher of youtube and youtube music are indeed identical ... so even if I point Pytube to the Youtube music js i get the same decoding schema ... and the same decoded signature ... and the same error ... so it feels like something is of with the data that get_song returns .. any clue?
@sigma67 commented on GitHub (May 18, 2021):
get_songis currently simply passing on the response that is being returned by theplayerendpoint without modification. You can verify easily by using the cipher in the response data of theplayerrequest in your browser instead. As mentioned above, have you tried runningurllib.parse.unquoteon the cipher before passing it to the other library?@KoljaWindeler commented on GitHub (May 18, 2021):
Hi, yep I unquote the url. The s parameter isn't quoted or at least wouldn't change (tested ;))
@KoljaWindeler commented on GitHub (May 19, 2021):
good morning, I got some news .. there is definitely some off :D
I've searched for 'player' in the network tab and found the strreamingData. I've copied those data to a simple script and decoded the url without issues (horray)
after that grabbed the data for the same videoId via ytmusicapi and failed. so something is happening between browser and ytmusicapi. The issue is that the url is almost completely of .. so it's hard to tell where the issue is ..
here is the script (I guess both links won't work for you as they are mapped to my IP but the way should be fairly clear)
https://pastebin.com/FDqXu6W7
what can I do to help?
Edit1:
the format of the URL look the 'same' .. at least both (webbrowser vs ytmusicapi) have the same fields / format .. sure different server, different Id, different timestamp but the webbrowser link changes as well everytime I refresh (and all of them work)
same for the signature .. about the same length, same 'coding' ..
edit2:
not sure if this is relevant but i get two player responses in the browser .. selecting the first will give me the streaming data to the video I want .. the second might be some pre-buffering? anyway .. both provide decodable streaming data
@sigma67 commented on GitHub (May 19, 2021):
Thanks for the script, I tried it and it worked fine for me. No errors and both URLs get printed at the end. Can you point out where you're getting errors with the script?
The POST data of the two in-browser requests differs only with regard to the
isPrefetchflag, which is true for the first one. So I think we can ignore that.Other things that are missing in the ytmusicapi request POST data are the following (disregarding the context key, which shouldn't matter as it's used in all requests):
CPN is a nonce computed clientside for debug purposes (Client Playback Nonce). Not sure if the playbackContext is required, but seems not as your script was working for me.
@KoljaWindeler commented on GitHub (May 19, 2021):
So both urls printed at the end work for you? The non_browser url Always results in a error 403 for me
@sigma67 commented on GitHub (May 19, 2021):
Yes. There might be something wrong with your headers as 403 is returned if you're not authorized. Can you access your library with those headers?
@KoljaWindeler commented on GitHub (May 19, 2021):
Library works 100% correct
@KoljaWindeler commented on GitHub (May 19, 2021):
Would it help if I send you my header file via email?
@sigma67 commented on GitHub (May 19, 2021):
You could try doing another setup run, as the setup procedure has changed in the latest release. If that doesn't work you can send it to ytmusicapi@gmail.com
@KoljaWindeler commented on GitHub (May 19, 2021):
still didn't work .. email is on the way .. lets see if that works for you
@sigma67 commented on GitHub (May 19, 2021):
No errors at all using your code and headers. Not sure what I should be looking for. I literally copy pasted the file in your email and ran it. I only changed the filepath to a file in the local directory (not /tmp/). Full output below:
@sigma67 commented on GitHub (May 19, 2021):
Oh I think there's a misunderstanding. You're doing another step by calling the googlevideo.com URL and getting 403 on that?
Besides id and signature differences the only thing that stands out to me is that the URLs have different IPs. The browser result is an IPv4 IP and the ytmusicapi result is an IPv6 ip. Strangely enough, the IPv4 IP is a foreign IP with ping around 40ms, while the IPv6 IP is my very own IP. What do those IPs look like for you?
@KoljaWindeler commented on GitHub (May 19, 2021):
Exactly .. I open the resulting googlemusic url .. one leads to .. well the MP4 the other to a 403.
The 31.18. ... Is my ip
Edit: I guess none of the urls are working for you right? The browser based link is connected to my ip and the other one won't work due to the bug. But you should be able to fetch a signatureCiphered from the website which would then generate a valid link for a given videoId. At least that's how I came up with the data
@sigma67 commented on GitHub (May 19, 2021):
Sorry I got a bit confused in your script there...
It seems it's indeed related to the missing POST data I included above. Will commit a fix shortly
@sigma67 commented on GitHub (May 19, 2021):
Unfortunately I'm not sure what the
signatureTimestampis based on, but it was always18766for me, even when not signed in.@KoljaWindeler commented on GitHub (May 19, 2021):
So the additional information fixes the issue for you? In other words:
Can you open the generated link and will it play music for you? Will test right after dinner!!
@sigma67 commented on GitHub (May 19, 2021):
Yes, I verified that now :)
@KoljaWindeler commented on GitHub (May 19, 2021):
here as well .. great. could you publish a new version so we can roll this out to all others?
Thanks a thousand times! great work!
@sigma67 commented on GitHub (May 20, 2021):
Closed in
8c4b857@simu commented on GitHub (May 21, 2021):
Playback (I'm using ytmusicapi via mopidy_ytmusic) broke again for me today (song URLs result in 403), and inspecting the requests on the web app showed that
signatureTimestamphas changed to18767for me now. Adjusting the timestamp in the code fixes playback.@KoljaWindeler commented on GitHub (May 21, 2021):
Same here
@sigma67 commented on GitHub (May 21, 2021):
Incrementing by one after a day seems very suspicious. Turns out it is derived from the UNIX datestamp! The datestamp is the number of days since 01-01-1970, which is currently
18768. If you decrement that by one you get thesignatureTimestamp18767that's currently being used.Strangely enough, about 43 hours ago (time of my commit), the timestamp was
18766, so that would imply that the timestamp had just incremented in the 5 hours before that time. That would mean that the increment time is not UTC midnight, as it would be if it was derived from the UNIX timestamp. Perhaps we can monitor in the coming hours when thesignatureTimestampchanges, so we can avoid similar issues in the future.@simu commented on GitHub (May 21, 2021):
I figured this out in the mean time as well, but I haven't been able to pinpoint the increment time yet. I'm currently running a patched version of
get_song()which usessignatureTimestamp = (date.today() - date.fromtimestamp(0)).days - 1(dateisfrom datetime import date).@KoljaWindeler commented on GitHub (May 21, 2021):
UTC15:55 still 18767
UTC16:34 still 18767
@impliedchaos commented on GitHub (May 21, 2021):
I'm not sure how viable of a strategy it is to do it based on time. The signatureTimestamp is set in the base.js version. This doesn't change at the same time everyday. My application tries to fetch the base.js url every 15 minutes:
(times are UTC+4)
@sigma67 commented on GitHub (May 21, 2021):
Seems it isn't a fixed interval but instead depends on when the server script gets updated.
The
signatureTimestampvalue is hardcoded (probably dynamically generated by the server) in the base.js player script retrieved from music.youtube.com. Looking at other apps like YoutubeExplode they get thesignatureTimestampby parsing the player script retrieved from the watch page. That's at least two additional requests though, one to get the page with the URL of the player script, another to get the player script, from which you can then retrieve thesignatureTimestamp.Another alternative could be to try the
datestamp - 1method by default and fall back todatestampif it fails. Then we'd need to verify the correctness of the URL though, by checking for 403 when calling the response URL. Or makesignatureTimestampan optional parameter toget_song.This is under the assumption that the server will only accept the
signatureTimestampin the most current player script, which seems to be the case from what we've seen.Edit: @impliedchaos seems we commented at the same time but have a similar assessment of the situation 👍
@KoljaWindeler commented on GitHub (May 21, 2021):
Well .. grabbing the signatureTimestamp wouldn't be necessary every single time. One could grab it, set it, decode the url, check the header and refetch if the header returns a 403 ...
@impliedchaos commented on GitHub (May 21, 2021):
This would be my personal preference. And if the parameter isn't passed, then use the
datestamp - 1logic.@KoljaWindeler commented on GitHub (May 21, 2021):
jep, something like this https://pastebin.com/sxbwa8Ee just with 37 instead of 38
edit: just because it's quite fancy .. during testing I get most of the time 18767 but I've already seen 18768 twice ..
@sigma67 commented on GitHub (May 21, 2021):
I've added a quick fix along the lines of what was suggested above. I'd gladly accept a PR for a
get_signatureTimestamphelper method since you guys seem to know more about that part than me@KoljaWindeler commented on GitHub (May 21, 2021):
the current change totally works for me .. just tested .. i guess adding the lookup into the library wouldn't really help because it would heavily slow down the function .. would you consider to release this change?
@sigma67 commented on GitHub (May 22, 2021):
Published a new release.
I was thinking more of making a separate method that can be called to get the signatureTimestamp, so others don't have to reimplement that part
@KoljaWindeler commented on GitHub (May 22, 2021):
Fair enough. Will push a PR based on my stuff tomorrow
@KoljaWindeler commented on GitHub (May 25, 2021):
works just perfect here .. I guess we can close this once more
Thanks a lot guys!