[GH-ISSUE #1502] Unable to decode dealer user collection (liked song collection) events #681

Closed
opened 2026-02-27 19:31:55 +03:00 by kerem · 3 comments
Owner

Originally created by @BenJeau on GitHub (May 27, 2025).
Original GitHub issue: https://github.com/librespot-org/librespot/issues/1502

Description

I'm creating a Spotify client to listen passively on events from Spotify. It works great for most of the events, except the hm://collection/collection/:username one, which is used whenever a user adds/remove a song from their liked songs.

Version

Using the latest commit, github.com/librespot-org/librespot@8b729540f4

How to reproduce

  1. Create a new project with the following dependencies + code
  2. Execute cargo run
  3. Login to Spotify in the browser window it opens up
  4. Add/Remove a song from your saved library
  5. View error in the logs

Cargo.toml

[dependencies]
futures-util = "0.3.31"
librespot = { git = "https://github.com/librespot-org/librespot", rev = "8b729540f4ad1e7f8c94ff3bba33095878b24d02" }
protobuf = "3.7.2"
tokio = { version = "1.45.1", features = ["full"] }
tracing = "0.1.41"
tracing-subscriber = "0.3.19"

main.rs - the protobuf message used for hm://collection/collection/:username is certainly not the right one, but it would give another error if it reached that part (I'm also unsure which protobuf schema this would relate to):

use futures_util::stream::StreamExt;
use librespot::{
    core::{Error, Session, SessionConfig, dealer::protocol::Message},
    discovery::Credentials,
    oauth::OAuthClientBuilder,
    protocol::connect::{ClusterUpdate, MemberType, PutStateReason, PutStateRequest},
};
use protobuf::EnumOrUnknown;

const HTML: &str = r#"<!DOCTYPE html>
<html>
  <head>
    <script>
      window.close();
    </script>
  </head>
</html>
"#;

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::DEBUG)
        .init();

    let client = OAuthClientBuilder::new(
        "65b708073fc0480ea92a077233ca87bd",
        "http://127.0.0.1:8898/login",
        vec!["streaming"],
    )
    .open_in_browser()
    .with_custom_message(HTML)
    .build()
    .unwrap();

    let access_token = client.get_access_token_async().await.unwrap();
    let refresh_token = client
        .refresh_token_async(&access_token.refresh_token)
        .await
        .unwrap();
    let credentials = Credentials::with_access_token(refresh_token.access_token);
    let session = Session::new(SessionConfig::default(), None);

    fn extract_connection_id(msg: Message) -> Result<String, Error> {
        let connection_id = msg.headers.get("Spotify-Connection-Id").unwrap();
        Ok(connection_id.to_owned())
    }

    let mut connection_id_update = session
        .dealer()
        .listen_for("hm://pusher/v1/connections/", extract_connection_id)
        .unwrap();
    let mut user_collection_update = session
        .dealer()
        .listen_for::<ClusterUpdate>(
            &format!("hm://collection/collection/{}", session.username()),
            |message| {
                println!("user collection update:\n{message:?}");
                Message::from_raw(message)
            },
        )
        .unwrap();

    session.connect(credentials, true).await.unwrap();
    session.dealer().start().await.unwrap();

    println!("---connected!!!---");
    loop {
        tokio::select! {
            user_collection_update = user_collection_update.next() => {
                println!("user collection update:\n{user_collection_update:?}");
            }
            Some(Ok(connection_id)) = connection_id_update.next() => {
                session.set_connection_id(&connection_id);
                let request = PutStateRequest {
                    member_type: EnumOrUnknown::new(MemberType::CONNECT_STATE),
                    put_state_reason: EnumOrUnknown::new(PutStateReason::NEW_DEVICE),
                    ..Default::default()
                };
                session.spclient().put_connect_state_request(&request).await.unwrap();
            }
        }
    }
}

Log

My logs are extremely verbose, but I've captured where the parsing issues are logged. I removed/added/removed/added the same song link=https://open.spotify.com/track/1kuPgsWuwfNVHTPDBGMMj4, uri="spotify:track:1kuPgsWuwfNVHTPDBGMMj4", uid="ea0019b856d6d45d2cc4"

// removed
Message { headers: {"MC-ETag": "64941807882", "Collection-Update-Id": "66dce80f5c4d7c20", "Collection-Source-Revision": "55320748569", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:23.652329Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. }    


// liked
Message { headers: {"Collection-Update-Id": "9ee4a66d1fa26263", "MC-ETag": "46685679324", "Collection-Source-Revision": "64941807882", "Content-Type": "application/octet-stream"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:31.498298Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. }    

 
// removed
Message { headers: {"MC-ETag": "47316492062", "Collection-Source-Revision": "46685679324", "Collection-Update-Id": "ee516cc0498e02dd", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:40.601061Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. }   


// liked
Message { headers: {"Collection-Source-Revision": "47316492062", "Content-Type": "application/octet-stream", "MC-ETag": "56467871245", "Collection-Update-Id": "ab9637744956f5fa"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:42.219687Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. }  

Host (what you are running librespot on):

  • OS: macOS 15.5 (24F74)
  • Platform: MacBook Pro

Additional context

You can see the raw data is similar throughout the requests:

  • 1st removed: [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]
  • 1st added: [10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]
  • 2nd removed: [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1] (the same)
  • 2nd added: [10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]

I can see the following:

  • 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40 is common, I'm guessing that the song ID
  • the last byte seems to be a boolean, 1=removed, 0=added

Ultimately, I'm unfamiliar how this works other from my exploration of this codebase.

Originally created by @BenJeau on GitHub (May 27, 2025). Original GitHub issue: https://github.com/librespot-org/librespot/issues/1502 ### Description I'm creating a Spotify client to listen passively on events from Spotify. It works great for most of the events, except the `hm://collection/collection/:username` one, which is used whenever a user adds/remove a song from their liked songs. ### Version Using the latest commit, https://github.com/librespot-org/librespot/commit/8b729540f4ad1e7f8c94ff3bba33095878b24d02 ### How to reproduce 1. Create a new project with the following dependencies + code 2. Execute `cargo run` 3. Login to Spotify in the browser window it opens up 4. Add/Remove a song from your saved library 5. View error in the logs `Cargo.toml` ```toml [dependencies] futures-util = "0.3.31" librespot = { git = "https://github.com/librespot-org/librespot", rev = "8b729540f4ad1e7f8c94ff3bba33095878b24d02" } protobuf = "3.7.2" tokio = { version = "1.45.1", features = ["full"] } tracing = "0.1.41" tracing-subscriber = "0.3.19" ``` `main.rs` - the protobuf message used for `hm://collection/collection/:username` is certainly not the right one, but it would give another error if it reached that part (I'm also unsure which protobuf schema this would relate to): ```rust use futures_util::stream::StreamExt; use librespot::{ core::{Error, Session, SessionConfig, dealer::protocol::Message}, discovery::Credentials, oauth::OAuthClientBuilder, protocol::connect::{ClusterUpdate, MemberType, PutStateReason, PutStateRequest}, }; use protobuf::EnumOrUnknown; const HTML: &str = r#"<!DOCTYPE html> <html> <head> <script> window.close(); </script> </head> </html> "#; #[tokio::main] async fn main() { tracing_subscriber::fmt() .with_max_level(tracing::Level::DEBUG) .init(); let client = OAuthClientBuilder::new( "65b708073fc0480ea92a077233ca87bd", "http://127.0.0.1:8898/login", vec!["streaming"], ) .open_in_browser() .with_custom_message(HTML) .build() .unwrap(); let access_token = client.get_access_token_async().await.unwrap(); let refresh_token = client .refresh_token_async(&access_token.refresh_token) .await .unwrap(); let credentials = Credentials::with_access_token(refresh_token.access_token); let session = Session::new(SessionConfig::default(), None); fn extract_connection_id(msg: Message) -> Result<String, Error> { let connection_id = msg.headers.get("Spotify-Connection-Id").unwrap(); Ok(connection_id.to_owned()) } let mut connection_id_update = session .dealer() .listen_for("hm://pusher/v1/connections/", extract_connection_id) .unwrap(); let mut user_collection_update = session .dealer() .listen_for::<ClusterUpdate>( &format!("hm://collection/collection/{}", session.username()), |message| { println!("user collection update:\n{message:?}"); Message::from_raw(message) }, ) .unwrap(); session.connect(credentials, true).await.unwrap(); session.dealer().start().await.unwrap(); println!("---connected!!!---"); loop { tokio::select! { user_collection_update = user_collection_update.next() => { println!("user collection update:\n{user_collection_update:?}"); } Some(Ok(connection_id)) = connection_id_update.next() => { session.set_connection_id(&connection_id); let request = PutStateRequest { member_type: EnumOrUnknown::new(MemberType::CONNECT_STATE), put_state_reason: EnumOrUnknown::new(PutStateReason::NEW_DEVICE), ..Default::default() }; session.spclient().put_connect_state_request(&request).await.unwrap(); } } } } ``` ### Log My logs are extremely verbose, but I've captured where the parsing issues are logged. I removed/added/removed/added the same song `link=https://open.spotify.com/track/1kuPgsWuwfNVHTPDBGMMj4, uri="spotify:track:1kuPgsWuwfNVHTPDBGMMj4", uid="ea0019b856d6d45d2cc4"` ``` // removed Message { headers: {"MC-ETag": "64941807882", "Collection-Update-Id": "66dce80f5c4d7c20", "Collection-Source-Revision": "55320748569", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" } Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) } 2025-05-27T03:04:23.652329Z WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. } // liked Message { headers: {"Collection-Update-Id": "9ee4a66d1fa26263", "MC-ETag": "46685679324", "Collection-Source-Revision": "64941807882", "Content-Type": "application/octet-stream"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" } Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) } 2025-05-27T03:04:31.498298Z WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. } // removed Message { headers: {"MC-ETag": "47316492062", "Collection-Source-Revision": "46685679324", "Collection-Update-Id": "ee516cc0498e02dd", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" } Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) } 2025-05-27T03:04:40.601061Z WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. } // liked Message { headers: {"Collection-Source-Revision": "47316492062", "Content-Type": "application/octet-stream", "MC-ETag": "56467871245", "Collection-Update-Id": "ab9637744956f5fa"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" } Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) } 2025-05-27T03:04:42.219687Z WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. } ``` ### Host (what you are running `librespot` on): - OS: macOS 15.5 (24F74) - Platform: MacBook Pro ### Additional context You can see the raw data is similar throughout the requests: * 1st removed: `[10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]` * 1st added: `[10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]` * 2nd removed: `[10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]` (the same) * 2nd added: `[10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]` I can see the following: * `8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40` is common, I'm guessing that the song ID * the last byte seems to be a boolean, 1=removed, 0=added Ultimately, I'm unfamiliar how this works other from my exploration of this codebase.
kerem 2026-02-27 19:31:55 +03:00
  • closed this issue
  • added the
    bug
    label
Author
Owner

@BenJeau commented on GitHub (May 28, 2025):

Continued my exploration, but no definitive way to parse it yet.


I think the protobuf file for this type of message would look something like

syntax = "proto3";

package spotify;

message Track {
    int32 status = 1;                   // Always 0 (maybe play state?)
    bytes track_id_binary = 2;          // 16-byte binary Spotify track ID
    optional int32 context_field = 3;   // Possible context/playlist ID (4 bytes)
}

message SpotifyLikeAction {
    repeated Track tracks = 1;      // Can contain multiple tracks
    uint64 timestamp = 5;           // Timestamp (0 for unlike, current time for like in seconds)
    bool action_type = 6;           // false = save, true = unsave
}

For [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1], the parsed message is now SpotifyLikeAction { tracks: [Track { status: 0, track_id_binary: [43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30], context_field: None }], timestamp: 0, action_type: false }

Steps to convert those binary track_id bytes to the Spotify ID:

  1. Convert bytes to u128 using big endian
  2. Convert the number to base62 using the 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ alphabet
  3. You get the ID 1kuPgsWuwfNVHTPDBGMMj4 which is the correct Spotify ID!

Other things I found out/confirmed:

  1. I saw in a few messages that a single message can contain multiple tracks (why the repeated Track tracks).
  2. The last byte corresponds to save/unsaved. 1=unsaved, 0=saved
  3. Anything between the last 48 byte and the 40 byte before it represents the timestamp in seconds
    • For unsaves, it is always 0
    • For saves, it is a timestamp in second - I'm unsure what the type of this field is though
    • I was unable to get this field and the action_type field to work with protobuf, but was able to with a custom decoder.
<!-- gh-comment-id:2914724531 --> @BenJeau commented on GitHub (May 28, 2025): Continued my exploration, but no definitive way to parse it yet. --- I think the protobuf file for this type of message would look something like ```proto syntax = "proto3"; package spotify; message Track { int32 status = 1; // Always 0 (maybe play state?) bytes track_id_binary = 2; // 16-byte binary Spotify track ID optional int32 context_field = 3; // Possible context/playlist ID (4 bytes) } message SpotifyLikeAction { repeated Track tracks = 1; // Can contain multiple tracks uint64 timestamp = 5; // Timestamp (0 for unlike, current time for like in seconds) bool action_type = 6; // false = save, true = unsave } ``` For `[10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]`, the parsed message is now ` SpotifyLikeAction { tracks: [Track { status: 0, track_id_binary: [43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30], context_field: None }], timestamp: 0, action_type: false }` Steps to convert those binary track_id bytes to the Spotify ID: 1. Convert bytes to u128 using big endian 2. Convert the number to base62 using the `0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ` alphabet 3. You get the ID `1kuPgsWuwfNVHTPDBGMMj4` which is the correct Spotify ID! Other things I found out/confirmed: 1. I saw in a few messages that a single message can contain multiple tracks (why the `repeated Track tracks`). 2. The last byte corresponds to save/unsaved. 1=unsaved, 0=saved 3. Anything between the last `48` byte and the `40` byte before it represents the timestamp in seconds * For unsaves, it is always 0 * For saves, it is a timestamp in second - I'm unsure what the type of this field is though * I was unable to get this field and the action_type field to work with protobuf, but was able to with a custom decoder.
Author
Owner

@photovoltex commented on GitHub (May 28, 2025):

Interesting discoveries! But what exactly is your issue you encounter? Is it something rudimentary in the implementation or rather a missing protobuf definition?

From your log there is a WireError and a baes64 error/warning, but I can't tell which of those you refer to.

You also use ClusterUpdate as parsing for the collection event, which is usually used for the connect device communication if I'm remember right. So maybe that's the issue, which wouldn't be directly a library issue, right?

If so I would convert this issue into a discussion, so we can still help you but without declaring non existing proto files as bug^^;

<!-- gh-comment-id:2916377006 --> @photovoltex commented on GitHub (May 28, 2025): Interesting discoveries! But what exactly is your issue you encounter? Is it something rudimentary in the implementation or rather a missing protobuf definition? From your log there is a `WireError` and a `baes64` error/warning, but I can't tell which of those you refer to. You also use `ClusterUpdate` as parsing for the collection event, which is usually used for the connect device communication if I'm remember right. So maybe that's the issue, which wouldn't be directly a library issue, right? If so I would convert this issue into a discussion, so we can still help you but without declaring non existing proto files as bug^^;
Author
Owner

@BenJeau commented on GitHub (May 29, 2025):

Sorry for opening an issue if this isn't considered an issue. Yes it is about the parsing of the data from the hm://collection/collection/:usernameevents, I saw it as an issue as the other events didn't have any issue with parsing. I do not know exactly what is the issue, I have a hunch is more complex than just a missing protobuf definition, but I think it is missing a protobuf definition and I could be wrong.

I'm aware that ClusterUpdate isn't the right message format for this, but decided to just put anything there as the errors without it weren't descriptive. In the code itself, the WireError and base64 error comes from the same place (if my memory serves well), just logged/returned differently.

Feel free to convert this as a discussion, I do not mind :)

On another topic, if you know how to "acquire" or extract the protobuf definitions from a Spotify binary, let me know! Either here or at my email

<!-- gh-comment-id:2917898320 --> @BenJeau commented on GitHub (May 29, 2025): Sorry for opening an issue if this isn't considered an issue. Yes it is about the parsing of the data from the `hm://collection/collection/:username`events, I saw it as an issue as the other events didn't have any issue with parsing. I do not know exactly what is the issue, I have a hunch is more complex than just a missing protobuf definition, but I think it is missing a protobuf definition and I could be wrong. I'm aware that `ClusterUpdate` isn't the right message format for this, but decided to just put anything there as the errors without it weren't descriptive. In the code itself, the `WireError` and `base64` error comes from the same place (if my memory serves well), just logged/returned differently. Feel free to convert this as a discussion, I do not mind :) On another topic, if you know how to "acquire" or extract the protobuf definitions from a Spotify binary, let me know! Either here or at my email
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/librespot#681
No description provided.