starred/librespot-android
1
0
Fork 0
mirror of https://github.com/devgianlu/librespot-android.git synced 2026-04-25 21:35:49 +03:00
Code Issues 5 Projects Releases Packages Wiki Activity

[GH-ISSUE #5] Implement Android native decoders #3

New issue
Closed
opened 2026-02-28 14:23:57 +03:00 by kerem · 7 comments
kerem commented 2026-02-28 14:23:57 +03:00
Owner
Copy link

Originally created by @devgianlu on GitHub (Apr 25, 2021).
Original GitHub issue: https://github.com/devgianlu/librespot-android/issues/5

Originally created by @devgianlu on GitHub (Apr 25, 2021). Original GitHub issue: https://github.com/devgianlu/librespot-android/issues/5
kerem 2026-02-28 14:23:57 +03:00
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@devgianlu commented on GitHub (Apr 29, 2021):

This might actually be need as the Tremolo decoder crashed a couple of times (might be something in my implementation...)

<!-- gh-comment-id:829231063 --> @devgianlu commented on GitHub (Apr 29, 2021): This might actually be need as the Tremolo decoder crashed a couple of times (might be something in my implementation...)
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@devgianlu commented on GitHub (May 2, 2021):

I have tried using the MediaCodec class to make a decoder, but did not make much progress. Also, it doesn't seem able to recognize the input format details (sample rate, channels, ...) which is a bit unfortunate.

<!-- gh-comment-id:830827387 --> @devgianlu commented on GitHub (May 2, 2021): I have tried using the `MediaCodec` class to make a decoder, but did not make much progress. Also, it doesn't seem able to recognize the input format details (sample rate, channels, ...) which is a bit unfortunate.
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@funtax commented on GitHub (May 2, 2021):

It's a long time ago when I tried to decode the Vorbis-files on Android using MediaCodec and MediaExtractor.
These require API 16+ and maybe this or the general hassle + CPU-requirements caused that I implemented the custom decoder via Tremolo (and first via Tremor).

I also remember this post: https://stackoverflow.com/questions/18221064/best-way-to-stream-ogg-vorbis-on-android
I remember you need the MediaExtractor and the MediaCodec together, but pretty sure you already know this :)
Maybe I can find something useful in my personal commit history..

<!-- gh-comment-id:830870829 --> @funtax commented on GitHub (May 2, 2021): It's a long time ago when I tried to decode the Vorbis-files on Android using MediaCodec and MediaExtractor. These require API 16+ and maybe this or the general hassle + CPU-requirements caused that I implemented the custom decoder via Tremolo (and first via Tremor). I also remember this post: https://stackoverflow.com/questions/18221064/best-way-to-stream-ogg-vorbis-on-android I remember you need the MediaExtractor and the MediaCodec together, but pretty sure you already know this :) Maybe I can find something useful in my personal commit history..
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@mitschwimmer commented on GitHub (May 2, 2021):

Yeah, that's unfortunate. Let me add to the things you probably already know, what I found on the solution Martin mentioned already. Let MediaExtractor analyse incoming data and use it to configure MediaCodec. That introduces a small challenge as one needs to implement a custom MediaDataSource that can act on InputStream as MediaExtractor expects files/URL's (MediaDataSource Example).
I have no experience whether this actually works, but it seems plausible.

<!-- gh-comment-id:830873267 --> @mitschwimmer commented on GitHub (May 2, 2021): Yeah, that's unfortunate. Let me add to the things you probably already know, what I found on the solution Martin mentioned already. Let MediaExtractor analyse incoming data and use it to configure MediaCodec. That introduces a small challenge as one needs to implement a custom [MediaDataSource](https://developer.android.com/reference/android/media/MediaDataSource) that can act on InputStream as MediaExtractor expects files/URL's ([MediaDataSource Example](https://stackoverflow.com/a/46141617)). I have no experience whether this actually works, but it seems plausible.
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@devgianlu commented on GitHub (May 2, 2021):

I hadn't figure out the MediaExtractor existed. Will give it another shot tomorrow, thank you for the insights!

<!-- gh-comment-id:830873520 --> @devgianlu commented on GitHub (May 2, 2021): I hadn't figure out the MediaExtractor existed. Will give it another shot tomorrow, thank you for the insights!
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@funtax commented on GitHub (May 2, 2021):

Yes, thanks @mitschwimmer, that triggered my brain. The extractor is based on complete files and I think this might have been the reason for me to give up with this and move on with Tremor.
Crossing fingers i can find some of my old commits 😃

<!-- gh-comment-id:830873827 --> @funtax commented on GitHub (May 2, 2021): Yes, thanks @mitschwimmer, that triggered my brain. The extractor is based on complete files and I think this might have been the reason for me to give up with this and move on with Tremor. Crossing fingers i can find some of my old commits 😃
kerem commented 2026-02-28 14:23:58 +03:00
Author
Owner
Copy link

@funtax commented on GitHub (May 2, 2021):

Found this in my commit-history from 21th November 2016 (on 22th the message says "Starting with Tremor to support InputStreams"):

import android.media.MediaCodec;
import android.media.MediaExtractor;
import android.media.MediaFormat;
import android.util.Log;

import java.io.IOException;
import java.nio.ByteBuffer;

import abc.util.CommonUtils;

import static java.lang.Thread.sleep;

public class AndroidAudioExtractor {

	private final Callback callback;

	private long lastPresentationTimeUs;
	private boolean seeked = false;
	private long startMs;
	private long diff = 0;
	private long lastSeekedTo = 0;
	private long lastCorrectPresentationTimeUs = 0;
	long lastOffset = 0;

	static MediaCodec decoder;
	static MediaExtractor extractor;

	private Thread extractorThread;

	public AndroidAudioExtractor(Callback callback) {
		this.callback = callback;
	}

	public void pause() throws IOException {
		if (extractorThread != null) {
			extractorThread.interrupt();
			try {
				extractorThread.join();
			} catch (InterruptedException e) {
				CommonUtils.logException(e);
			}
			extractorThread = null;
		}
		callback.pause();
	}

	public void play() throws IOException {
		if (extractorThread != null) {
			throw new RuntimeException("Play has already been called!");
		}
		if (decoder == null) {
			throw new RuntimeException("Decoder not yet initialized!");
		}

		callback.start();
		extractorThread = new Thread(new Runnable() {
			public void run() {
				android.os.Process
						.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);

				ByteBuffer[] inputBuffers = decoder.getInputBuffers();
				ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
				MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
				boolean isEOS = false;

				lastOffset = extractor.getSampleTime() / 1000;
				startMs = System.currentTimeMillis();
				long startNs = System.nanoTime();
				long startSampleTimeUs = extractor.getSampleTime();
				while (!Thread.interrupted()) {
					try {
						if (!isEOS) {
							int inIndex = decoder
									.dequeueInputBuffer(1000); // 10000
							if (inIndex >= 0) {
								ByteBuffer buffer = inputBuffers[inIndex];
								int sampleSize = extractor
										.readSampleData(buffer, 0);
								if (sampleSize < 0) {
									decoder.queueInputBuffer(
											inIndex,
											0,
											0,
											0,
											MediaCodec.BUFFER_FLAG_END_OF_STREAM);
									isEOS = true;
								} else {
									decoder.queueInputBuffer(
											inIndex,
											0,
											sampleSize,
											extractor.getSampleTime(),
											0);
									extractor.advance();
								}
							}
						}

						int outIndex = decoder.dequeueOutputBuffer(
								info, 1000); // 10000

						if (info.presentationTimeUs < lastPresentationTimeUs) {      // correct timing playback issue for some videos
							startMs = System.currentTimeMillis();
							lastCorrectPresentationTimeUs = lastPresentationTimeUs;
						}

						switch (outIndex) {
							case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
								outputBuffers = decoder.getOutputBuffers();
								break;
							case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
								break;
							case MediaCodec.INFO_TRY_AGAIN_LATER:
								break;
							default:
								ByteBuffer buffer = outputBuffers[outIndex];
								int bytes = info.size;

								if (bytes > 0) {
									byte[] chunk = new byte[bytes * 4]; // TODO
									// make
									// this
									// dynamic
									// and
									// clever
									buffer.get(chunk, 0, bytes);

									callback.onDataAvailable(chunk, bytes);
								}

								buffer.clear();

								//CommonUtils.log(Log.DEBUG, "Original Presentation time: " + info.presentationTimeUs / 1000 + ", Diff PT: " + (info.presentationTimeUs / 1000 - lastOffset) + " : System Time: " + (System.currentTimeMillis() - startMs));

								lastPresentationTimeUs = info.presentationTimeUs;

								if (seeked && Math.abs(info.presentationTimeUs / 1000 - lastOffset) < 100)
									seeked = false;

								/* TODO activate sleep for realtime-decoding
								while (!seeked && (info.presentationTimeUs / 1000 - lastOffset) > System.currentTimeMillis() - startMs) {
									sleep(5);
								}*/

								if(callback != null) {
									callback.positionChanged(info.presentationTimeUs / 1000);
								}

								if(decoder != null) {
									decoder.releaseOutputBuffer(outIndex, false);
								}
								break;
						}

						if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
							callback.endOfMedia();
							break;
						}
					} catch (Exception e) {
						CommonUtils.log(Log.DEBUG,
								"Thread interrupted", e);
						Thread.currentThread().interrupt(); // important!
					}
				}

				CommonUtils.log(Log.INFO,
						"Interrupted playback!");
			}
		});
		extractorThread.start();
	}


	/*private static native int resample(int inputSampleRate,
									   int inputBufferSize, byte[] buffer);*/

	public boolean setSource(String uri) throws IOException {

		// @see
		// "https://vec.io/posts/android-hardware-decoding-with-mediacodec"

		try {
			extractor = new MediaExtractor();
			extractor.setDataSource(uri/*, headers*/);
		} catch (Exception e) {
			// TODO handle error
			CommonUtils.log(Log.DEBUG,
					"setSource() extractor.setDataSource EXCEPTION", e);
			extractor = null;
			return false;
		}

		CommonUtils.log(Log.DEBUG,
				"setSource() extractor.setDataSource SUCCESS");

		for (int i = 0; i < extractor.getTrackCount(); i++) {
			MediaFormat format = extractor.getTrackFormat(i);
			String mime = format.getString(MediaFormat.KEY_MIME);
			if (mime.startsWith("audio/")) {
				extractor.selectTrack(i);
				Log.d("TAG", "format : " + format);

				callback.durationChanged(format.getLong(MediaFormat.KEY_DURATION) / 1000);

				// TODO right now, the srcBps is always 16bit as per Android-implementation
				int srcBPS = 16;
				int srcSamplingRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
				int srcChannels = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);

				//seek(0);

				// the actual decoder
				decoder = MediaCodec.createDecoderByType(mime);
				decoder.configure(format, null /* surface */, null /* crypto */, 0 /* flags */);
				decoder.start(); // TODO maybe start later?

				return true;
			}
		}

		return false;
	}

	public void stop() throws IOException {
		pause();
		if (decoder != null) {
			decoder.stop();
			decoder.release();
			decoder = null;
		}
		if (extractor != null) {
			extractor.release();
			extractor = null;
		}

		callback.stop();
	}

	public void seek(int position) throws IOException {
		/*
		pause();
		extractor.seekTo(position * 1000,
				MediaExtractor.SEEK_TO_CLOSEST_SYNC);
		callback.positionChanged(extractor.getSampleTime() / 1000);
		play();*/


		seeked = true;
		CommonUtils.log(Log.DEBUG, "SeekTo Requested to : " + position);
		CommonUtils.log(Log.DEBUG, "SampleTime Before SeekTo : " + extractor.getSampleTime() / 1000);
		extractor.seekTo(position * 1000, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
		CommonUtils.log(Log.DEBUG, "SampleTime After SeekTo : " + extractor.getSampleTime() / 1000);

		lastOffset = extractor.getSampleTime() / 1000;
		startMs = System.currentTimeMillis();
		diff = (lastOffset - lastPresentationTimeUs / 1000);

		CommonUtils.log(Log.DEBUG, "SeekTo with diff : " + diff);

	}

	public interface Callback {
		void positionChanged(long position);

		void durationChanged(long duration);

		void onDataAvailable(byte[] chunk, int bytes);

		void endOfMedia();

		void pause() throws IOException;

		void stop() throws IOException;

		void start() throws IOException;
	}

}
<!-- gh-comment-id:830875996 --> @funtax commented on GitHub (May 2, 2021): Found this in my commit-history from 21th November 2016 (on 22th the message says "Starting with Tremor to support InputStreams"): ```java import android.media.MediaCodec; import android.media.MediaExtractor; import android.media.MediaFormat; import android.util.Log; import java.io.IOException; import java.nio.ByteBuffer; import abc.util.CommonUtils; import static java.lang.Thread.sleep; public class AndroidAudioExtractor { private final Callback callback; private long lastPresentationTimeUs; private boolean seeked = false; private long startMs; private long diff = 0; private long lastSeekedTo = 0; private long lastCorrectPresentationTimeUs = 0; long lastOffset = 0; static MediaCodec decoder; static MediaExtractor extractor; private Thread extractorThread; public AndroidAudioExtractor(Callback callback) { this.callback = callback; } public void pause() throws IOException { if (extractorThread != null) { extractorThread.interrupt(); try { extractorThread.join(); } catch (InterruptedException e) { CommonUtils.logException(e); } extractorThread = null; } callback.pause(); } public void play() throws IOException { if (extractorThread != null) { throw new RuntimeException("Play has already been called!"); } if (decoder == null) { throw new RuntimeException("Decoder not yet initialized!"); } callback.start(); extractorThread = new Thread(new Runnable() { public void run() { android.os.Process .setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO); ByteBuffer[] inputBuffers = decoder.getInputBuffers(); ByteBuffer[] outputBuffers = decoder.getOutputBuffers(); MediaCodec.BufferInfo info = new MediaCodec.BufferInfo(); boolean isEOS = false; lastOffset = extractor.getSampleTime() / 1000; startMs = System.currentTimeMillis(); long startNs = System.nanoTime(); long startSampleTimeUs = extractor.getSampleTime(); while (!Thread.interrupted()) { try { if (!isEOS) { int inIndex = decoder .dequeueInputBuffer(1000); // 10000 if (inIndex >= 0) { ByteBuffer buffer = inputBuffers[inIndex]; int sampleSize = extractor .readSampleData(buffer, 0); if (sampleSize < 0) { decoder.queueInputBuffer( inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); isEOS = true; } else { decoder.queueInputBuffer( inIndex, 0, sampleSize, extractor.getSampleTime(), 0); extractor.advance(); } } } int outIndex = decoder.dequeueOutputBuffer( info, 1000); // 10000 if (info.presentationTimeUs < lastPresentationTimeUs) { // correct timing playback issue for some videos startMs = System.currentTimeMillis(); lastCorrectPresentationTimeUs = lastPresentationTimeUs; } switch (outIndex) { case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: outputBuffers = decoder.getOutputBuffers(); break; case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: break; case MediaCodec.INFO_TRY_AGAIN_LATER: break; default: ByteBuffer buffer = outputBuffers[outIndex]; int bytes = info.size; if (bytes > 0) { byte[] chunk = new byte[bytes * 4]; // TODO // make // this // dynamic // and // clever buffer.get(chunk, 0, bytes); callback.onDataAvailable(chunk, bytes); } buffer.clear(); //CommonUtils.log(Log.DEBUG, "Original Presentation time: " + info.presentationTimeUs / 1000 + ", Diff PT: " + (info.presentationTimeUs / 1000 - lastOffset) + " : System Time: " + (System.currentTimeMillis() - startMs)); lastPresentationTimeUs = info.presentationTimeUs; if (seeked && Math.abs(info.presentationTimeUs / 1000 - lastOffset) < 100) seeked = false; /* TODO activate sleep for realtime-decoding while (!seeked && (info.presentationTimeUs / 1000 - lastOffset) > System.currentTimeMillis() - startMs) { sleep(5); }*/ if(callback != null) { callback.positionChanged(info.presentationTimeUs / 1000); } if(decoder != null) { decoder.releaseOutputBuffer(outIndex, false); } break; } if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { callback.endOfMedia(); break; } } catch (Exception e) { CommonUtils.log(Log.DEBUG, "Thread interrupted", e); Thread.currentThread().interrupt(); // important! } } CommonUtils.log(Log.INFO, "Interrupted playback!"); } }); extractorThread.start(); } /*private static native int resample(int inputSampleRate, int inputBufferSize, byte[] buffer);*/ public boolean setSource(String uri) throws IOException { // @see // "https://vec.io/posts/android-hardware-decoding-with-mediacodec" try { extractor = new MediaExtractor(); extractor.setDataSource(uri/*, headers*/); } catch (Exception e) { // TODO handle error CommonUtils.log(Log.DEBUG, "setSource() extractor.setDataSource EXCEPTION", e); extractor = null; return false; } CommonUtils.log(Log.DEBUG, "setSource() extractor.setDataSource SUCCESS"); for (int i = 0; i < extractor.getTrackCount(); i++) { MediaFormat format = extractor.getTrackFormat(i); String mime = format.getString(MediaFormat.KEY_MIME); if (mime.startsWith("audio/")) { extractor.selectTrack(i); Log.d("TAG", "format : " + format); callback.durationChanged(format.getLong(MediaFormat.KEY_DURATION) / 1000); // TODO right now, the srcBps is always 16bit as per Android-implementation int srcBPS = 16; int srcSamplingRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE); int srcChannels = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); //seek(0); // the actual decoder decoder = MediaCodec.createDecoderByType(mime); decoder.configure(format, null /* surface */, null /* crypto */, 0 /* flags */); decoder.start(); // TODO maybe start later? return true; } } return false; } public void stop() throws IOException { pause(); if (decoder != null) { decoder.stop(); decoder.release(); decoder = null; } if (extractor != null) { extractor.release(); extractor = null; } callback.stop(); } public void seek(int position) throws IOException { /* pause(); extractor.seekTo(position * 1000, MediaExtractor.SEEK_TO_CLOSEST_SYNC); callback.positionChanged(extractor.getSampleTime() / 1000); play();*/ seeked = true; CommonUtils.log(Log.DEBUG, "SeekTo Requested to : " + position); CommonUtils.log(Log.DEBUG, "SampleTime Before SeekTo : " + extractor.getSampleTime() / 1000); extractor.seekTo(position * 1000, MediaExtractor.SEEK_TO_CLOSEST_SYNC); CommonUtils.log(Log.DEBUG, "SampleTime After SeekTo : " + extractor.getSampleTime() / 1000); lastOffset = extractor.getSampleTime() / 1000; startMs = System.currentTimeMillis(); diff = (lastOffset - lastPresentationTimeUs / 1000); CommonUtils.log(Log.DEBUG, "SeekTo with diff : " + diff); } public interface Callback { void positionChanged(long position); void durationChanged(long duration); void onDataAvailable(byte[] chunk, int bytes); void endOfMedia(); void pause() throws IOException; void stop() throws IOException; void start() throws IOException; } } ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels   
bug

   
enhancement

   
enhancement

   
enhancement

   
pull-request

Mirrored from GitHub Pull Request

No labels
bug
enhancement
enhancement
enhancement
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/librespot-android#3
No description provided.