Multimedia on android
4 likes•3,041 views
Multimedia playback on Android can occur from the file system, through streaming from a network, or for real-time conversations. The MediaPlayer API provides high-level playback but no access to decoded data, while MediaCodec allows low-level access to encoded and decoded data. MediaExtractor and MediaMuxer facilitate extraction and muxing of media streams. Common streaming protocols include RTSP/RTP/RTCP for low-latency real-time conversations and HTTP Live Streaming (HLS) or MPEG Dynamic Adaptive Streaming over HTTP (DASH) for adaptive streaming over HTTP. ExoPlayer is an open source media player that supports features like DASH and DRM not available on the Android MediaPlayer.
Multimedia on android
- 2. Types of Multimedia Playback Playback from File system - File formats, codecs Streaming from network - Protocols, formats, codecs Real time Conversational - Latency, Protocols, formats, codecs
- 3. Playback from File System High level APIs MediaPlayer VideoView No access to the decoded data - post processing (Audio mixing, Image processing, cropping) etc
- 4. Low Level Media APIs - MediaCodec Introduced in Android 4.1 (API Level 16) Access to low level media codecs for encoding/decoding of media Have access to compressed data as well as decompressed data Compressed data can be used for recording Decompressed data can be used for post processing Need to understand codecs and file formats
- 5. Data Flow Java Application Application Framework Java Runtime JNI Native Libs Kernel Device Drivers Hardware
- 6. Workflow 1. createEncoderByType()/createDecoderByType() a. mime type "video/3gpp" or "audio/vorbis" 2. configure() - properties eg encryption 3. getInputBuffers() and getOutputBuffers() - get buffers 4. dequeueInputBuffer() - index position of buffer to fill 5. queueInputBuffer() - release buffer 4. dequeueOutputBuffer() - index position of buffer data 5. releaseOutputBuffer() - release buffer
- 7. Low Level Media APIs - MediaExtractor Android 4.2 - API Level 17 MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a data source.
- 8. Low Level Media APIs - MediaMuxer Android 4.3 - API Level 18 MediaMuxer facilitates muxing elementary streams. Currently supports mp4 or webm file as the output At most one audio and/or one video elementary stream. MediaMuxer does not support muxing B-frames.
- 10. Case Study 1080p 60 fps 250 ms Latency Playback + Recording
- 11. Streaming RTSP/RTCP/RTP HTTP Live Streaming (HLS) MPEG - Dynamic Adaptive Streaming over HTTP (DASH)
- 12. RTSP/RTCP/RTP RTSP - TCP - connection setup and teardown RTCP - UDP - Connection feedback RTP - UDP - Compressed Audio/Visual Data
- 13. RTSP/RTCP/RTP Low latency, suitable for realtime, conversational applications As it uses UDP, data losses happen causing corrupted playback Uses custom ports, blocked by firewalls Custom servers required - Wowza, Real, Darwin
- 14. HLS Implemented by Apple, non-standard, defacto standard HTTP based, uses TCP, no losses, adaptive Near realtime, not suitable for realtime/conversational applications Suitable for Live broadcasts (one way), VoD Uses manifest with links to the media segments, .m3u8 extension Commodity HTTP infrastructure, caching, CDN (Scale) Facebook live, Periscope etc
- 15. M3u8 File Structure #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000 gear1/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=311111 gear2/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=484444 gear3/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=737777 gear4/prog_index.m3u8
- 17. MPEG - DASH Similar to HLS, Adobe HDS, Microsoft Smooth Streaming but more flexible International standard, XML based manifest with .mpd extension Codec agnostic, HTML5 support Supports multiple use cases like Ad insertion More efficient than HLS Used by Youtube, Netflix etc
- 19. Limitation of MediaPlayer Support for RTSP/RTP/RTCP is good but support for HLS is flaky Playback issues No current playback time No support for discontinuity No support for ID3 Support across different versions inconsistent, new versions not necessarily better No support for DASH, DRM etc
- 20. Exoplayer The Android framework provides MediaPlayer as a quick solution for playing media with minimal code MediaCodec and MediaExtractor classes are provided for building custom media players. The open source project, ExoPlayer, is a solution between these two options, providing a pre-built player that you can extend. ExoPlayer supports features not currently provided by MediaPlayer, including Dynamic adaptive streaming over HTTP (DASH), HLS, SmoothStreaming, DRM - Common Encryption, Widevine, Playready. It's designed to be easy to customize and extend, allowing many components to be replaced with custom implementations. ExoPlayer is a library that you include in your application, it can be easily updated along with your app. https://developer.android.com/guide/topics/media/exoplayer.html