Completing the Multimedia Architecture

Copyright Khronos Group, 2011 - Page 1 Completing the Multimedia Architecture Erik Noreke Chair of OpenSL ES Working Group Chair of OpenMAX AL Working Group

Copyright Khronos Group, 2011 - Page 2 Today s Consumer Requirements Rich media applications and UI - Consumer decisions are driven by the multimedia experience - Drives smart phone market Enhanced user experience - Games - Theater-like audio for video playback - Streaming media services - Mobile TV Availability of third party applications - Consumers expected to purchase $35 billion of mobile applications by 2014

Copyright Khronos Group, 2011 - Page 3 Application Developer Requirements Use case driven design - For ease of use Comprehensive feature set - Including advanced audio and streaming media Consistent APIs - Same APIs for both hardware and software implementations Full, predictable functionality - No need to guess available or what is going to work Portable - No need to port application for different platforms

Copyright Khronos Group, 2011 - Page 4 Why Create Khronos Multimedia APIs? Khronos APIs Conformance Tests Portable IP Protection Profiles & Components Predictable set of functionality Open Source Framework Free to change Every implementation unique GPL license Plugins Functionality dependent on implementation

Integration Layer Copyright Khronos Group, 2011 - Page 5

Copyright Khronos Group, 2011 - Page 6 OpenMAX IL Portable Media Integration Low-level interface for audio, video, and imaging codecs Developed for embedded and mobile devices Enables interfacing with multimedia components in a unified manner Allows components to be any combination of hardware or software completely transparently to the media chain The goal of OpenMAX IL is to abstract multimedia components - Combat the problem of integration portability - Consistent functionality

Copyright Khronos Group, 2011 - Page 7 OpenMAX IL Component Networks Enables arbitrary multimedia pipelines by plugging blocks together - Componentized architecture abstracts multimedia functionality block interfaces Wide variety of building blocks for imaging, video and audio functions - Encode, decode, apply an effect, capture, render, split, mix, etc Enables blocks from different sources to work together - Blocks can be implemented in software or hardware Portable & Re-usable streaming media building blocks

OpenMAX IL Example Graph Standardized component interfaces enable flexible media graphs Includes multi-stream synchronization Allows for custom plug-ins OpenMAX Component Interfaces Enables components to be flexibly connected in any graph topology E.g. send dance beats to 3D application for visualization AAC Audio Audio Decoder Plug-in Audio Renderer Speakers *.mp4 / *.3gp File Reader Clock for AV Sync MPEG4/ H.264 Video Video Decoder Time Data Decompressed Video Video Scheduler Video Renderer Display Example: MPEG-4 video synchronized with AAC audio decode Copyright Khronos Group, 2011 - Page 8

Copyright Khronos Group, 2011 - Page 10 The Need for Application Access Hardware and low level access is powerful but complex - More power than most application developers require Most application developers just want to playback and record media - Specifying where the content comes from - Specifying where the content should be rendered to - Manipulate a few playback controls - Have simple configurability That are what OpenMAX AL and OpenSL ES provide - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in With Cross-platform portability - No need to rewrite an application for every platform

Copyright Khronos Group, 2011 - Page 11 OpenSL ES and OpenMAX AL Working groups collaborated to define the common API functionality Independent, Compatible, Consistent and Distinct Audio Effects Advanced MIDI 3D Audio Buffer queues Audio Playback Audio Recording Basic MIDI Video playback Video recording Radio and RDS Camera Image capture & display

Copyright Khronos Group, 2011 - Page 12 Why Profiles? Functionality divided into basic groups - Most common groupings Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need Applications can query available profiles - To adapt to the current device - Facilitates porting of applications Profiles can be combined - To best suit any product Optional features - Some features are optional in all profiles - APIs are consistent when feature is available

Copyright Khronos Group, 2011 - Page 13 Object Oriented Media Access Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output Objects have control interfaces - Play, Record, Seek, Rate Media Input Audio Mix Camera (AL) Display Window (AL) URI DSrc Player Object DSnk URI Memory Memory Content pipe Content pipe

Copyright Khronos Group, 2011 - Page 14 OpenSL ES and OpenMAX AL Independent - But designed to work together - No dependency between the APIs either one can exist by itself Compatible - Working groups collaborated to make sure the APIs work together well Consistent - Identical API architecture - Identical APIs for same functionality Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality

Application Layer Copyright Khronos Group, 2011 - Page 15

Copyright Khronos Group, 2011 - Page 16 OpenMAX AL Focused on Streaming Media Media playback and recording - Playback and recording of everything from video files to radio to camera input Full range of video effects and controls including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends Complete metadata support - Read, Edit, and Set the metadata for most types of content OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks Designed as a complete set of multimedia functionality - Unlike any other streaming media API

Copyright Khronos Group, 2011 - Page 17 OpenMAX AL Features Extensive camera controls - Flash modes, Metering modes - White balance controls, Focusing controls - Exposure compensation, ISO Sensitivity - Shutter speed and Aperture - Zoom (digital and optical) Analog radio controls - Tuning, RDS Audio routing - Application-selectable audio inputs and outputs, based on location, connectivity, etc. - I/O device capability querying Metadata extraction and insertion - Search/extract and insert/overwrite metadata in a variety of file formats

Copyright Khronos Group, 2011 - Page 19 OpenSL ES Powerful Mobile Audio Create theater quality audio experience - In a mobile device! Profiles reduce application customization - Applications can query available profiles - Develop to a specific profile or profile combination Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES Designed for implementation by either a hardware or software solution - Unlike any other advanced audio API

Copyright Khronos Group, 2011 - Page 20 Designed with game developers in mind Audio Playback and Recording - Playback of everything from music files to MIDI ringtones to UI sounds Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience Advanced MIDI - Use the output of the MIDI engine as a 3D sound source, making the ringtone appear to come from a different direction than the music 3D Audio makes OpenSL ES the natural choice for any audio application - Better gaming experience; 3D-audio for conferencing calls; more vivid music experience 3D Position Doppler MIDI Messages Rate Preset Reverb Stereo Widening Environmental Reverb Buffer Queues Pitch EQ Metadata Extraction Virtualization LED and Vibrator

Copyright Khronos Group, 2011 - Page 22 OpenSL ES / OpenMAX AL Implemented on top of OpenMAX IL OR URI DataSource Media Player Output Mix File Reader Audio Decoder Audio Effect Audio Mixer Audio Sink Clock

Copyright Khronos Group, 2011 - Page 24 Latest Updates OpenMAX AL 1.1 & OpenSL ES 1.1 Released January 2011 - Better support for portability between varying operating systems - Additional functionality - Improved buffer queues OpenMAX AL Digital TV Extension Released January 2011 - Technology-neutral access to Digital TV EGLStream Extension Released December 2011 - Connecting the multimedia APIs to the graphics APIs

Copyright Khronos Group, 2011 - Page 25 The Complete Multimedia Use-Case OpenMAX AL Media Player OpenSL ES 3D Audio Effects Native Audio EGLStream OpenGL ES GL_TEXURE_EXTERNAL_OES texture OpenGL ES EGL Surface Native Display