OpenMAX AL, OpenSL ES - PDF Free Download

Copyright Khronos Group, 2011 - Page 1 OpenMAX AL, OpenSL ES Native Multimedia in Android Erik Noreke Chair of OpenMAX AL and OpenSL ES Working Groups

Copyright Khronos Group, 2011 - Page 2 Why Create Khronos Multimedia APIs? Khronos APIs Conformance Tests Portable IP Protection Profiles & Components Predictable set of functionality Open Source Framework Free to change Every implementation unique GPL license Plugins Functionality dependent on implementation

Copyright Khronos Group, 2011 - Page 4 The Need for Application Access Hardware and low level access is powerful but complex - More power than most application developers require Most application developers just want to playback and record media - Specifying where the content comes from and is rendered to - Manipulate a few playback controls - Have simple configurability

Copyright Khronos Group, 2011 - Page 5 The Need for Application Access That are what OpenMAX AL and OpenSL ES provide - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in With Cross-platform portability - No need to rewrite an application for every platform

Copyright Khronos Group, 2011 - Page 6 OpenMAX AL Focused on Streaming Media Media playback and recording - Playback and recording of everything from video files to radio to camera input Full range of video effects and controls including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends

Copyright Khronos Group, 2011 - Page 7 OpenMAX AL Focused on Streaming Media OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks Designed as a complete set of multimedia functionality - Unlike any other streaming media API

Copyright Khronos Group, 2011 - Page 8 OpenSL ES Powerful Mobile Audio Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience Create theater quality audio experience - In a mobile device!

Copyright Khronos Group, 2011 - Page 9 OpenSL ES Powerful Mobile Audio Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES Designed for implementation by either a hardware or software solution - Take full advantage of system capabilities

Copyright Khronos Group, 2011 - Page 10 Object Oriented Media Access Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output Objects have control interfaces - Play, Record, Seek, Rate Media Input Audio Mix Camera (AL) Display Window (AL) URI DSrc Player Object DSnk URI Memory Memory Content pipe Content pipe

OpenMAX AL & OpenSL ES Copyright Khronos Group, 2011 - Page 11

Copyright Khronos Group, 2011 - Page 12 OpenSL ES and OpenMAX AL Working groups collaborated to define the common API functionality Independent, Compatible, Consistent and Distinct Audio Effects Advanced MIDI 3D Audio Buffer queues Audio Playback Audio Recording Basic MIDI Video playback Video recording Radio and RDS Camera Image capture & display

Copyright Khronos Group, 2011 - Page 13 Why Profiles? Functionality divided into basic groups - Most common groupings Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need Applications can query available profiles - To adapt to the current device - Facilitates porting of applications

Copyright Khronos Group, 2011 - Page 14 Why Profiles? Profiles can be combined - To best suit any product Optional features - Some features are optional in all profiles - APIs are consistent when feature is available

Copyright Khronos Group, 2011 - Page 15 OpenSL ES and OpenMAX AL Independent - But designed to work together - No dependency between the APIs either one can exist by itself Compatible - Working groups collaborated to make sure the APIs work together well

Copyright Khronos Group, 2011 - Page 16 OpenSL ES and OpenMAX AL Consistent - Identical API architecture - Identical APIs for same functionality Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality

Copyright Khronos Group, 2011 - Page 17 Complete Multimedia Use-Case Audio stream to DSP 3D Audio Rendering Multimedia Processing Application on CPU Video stream to GPU EGLStream OpenGL ES GL_TEXURE_EXTERN AL_OES texture OpenGL ES EGLSurface Much more flexibility than just show a video

Copyright Khronos Group, 2011 - Page 18 Android NDK for Multimedia NDK Native Development Kit Introduced in Android 2.3 Gingerbread Advanced Audio Mobile 3D

Copyright Khronos Group, 2011 - Page 19 Why the NDK? Better control of fuctionality and timing Access to hardware acceleration Multimedia is the primary reason for the NDK Increases application complexity Requires compilation for each platform family - ARMv5TE - ARMv7-A - x86 instructions

Copyright Khronos Group, 2011 - Page 20 Focus is on Standardized APIs Earlier internal APIs not stable enough Avoid fragmentation Well tested and documented Focus on functionality - Not on implementation Access to hardware acceleration

Copyright Khronos Group, 2011 - Page 21 Native Multimedia APIs in Android 2.3 OpenGL ES 2.0 EGL 1.4 OpenSL ES 1.0.1 - Partial implementation - Almost complete Phone + Music profiles - Buffer queues from 1.1 as Android extension - Google version compliant as far as implemented - Other vendors offer fully compliant solution Mobile 3D Graphics Context Management Advanced Audio

Copyright Khronos Group, 2011 - Page 22 Ice Cream Sandwich Android 4.0 Same OS for both phones and tablets Better multitasking Better support for hardware accelerated graphics DRM Partial OpenMAX AL 1.0.1 implementation - Streaming use-case only - Does not interoperate with OpenSL ES Streaming Media

Copyright Khronos Group, 2011 - Page 23 Android & OpenMAX IL Android uses OpenMAX IL in their Multimedia framework - Codecs only - No graph building Main Multimedia framework is still Stagefright - Introduced in Éclair Allows for both software implemented and hardware accelerated codecs Not part of NDK - Not intended to be accessible from applications

Copyright Khronos Group, 2011 - Page 27 Where is Android Headed? More Devices - TVs? - Netbooks? - Chrome OS? What does the future hold? - Compliant OpenSL ES? - Better OpenMAX AL support? - OpenMAX AL EGL Stream Extension? - Support for OpenCL? - Content Pipes?