Streaming Media. Advanced Audio. Erik Noreke Standardization Consultant Chair, OpenSL ES. Copyright Khronos Group, Page 1

Transcription

1 Streaming Media Advanced Audio Erik Noreke Standardization Consultant Chair, OpenSL ES Copyright Khronos Group, Page 1

2 Today s Consumer Requirements Rich media applications and UI - Consumer decisions are driven by multimedia experince - Drives smart phone market Enhanced user experience - Games - Theater like audio for video - Streaming media services - Mobile TV Availability of third party applications - Consumers purchased $4.5 billion of mobile applications in Number growing exponentially Copyright Khronos Group, Page 2

3 Application Developer Requirements Use case driven design - For ease of use Comprehensive feature set - Including advanced audio and streaming media Consistent APIs - Same APIs for both hardware and software implementations Full, predictable functionality - No need to guess available or what is going to work Portable - No need to port application for different platforms Copyright Khronos Group, Page 3

4 Why Create A Khronos Media Framework? Khronos APIs Conformance Tests Portable IP Protection Profiles Predictable set of functionality Open Source Framework Free to change Every implementation unique GPL license Plugins Functionality dependent on implementation

5 Mobile OS Fragmentation Every handset is unique from the programmers perspective - Differences in OS functions, Java implementations and media functionality Symbian^3, PocketPC / Windows Mobile / WinCE Linux variants Android, Limo, WebOS Java MIDP-1, MIDP-2, JSR fragmentation RTOS Nucleus, Synergy Brew, WIPI, iphone OS We need cross platform standard APIs to defragment access to multimedia acceleration if the market is to be fully realized Severe platform fragmentation ISVs need to port to and support 100s (even 1000s) of source variants of each title Copyright Khronos Group, Page 5

6 & Application Access Layer Copyright Khronos Group, Page 6

7 The Need for Application Access Hardware and low level access is powerful but complex - More power than most application developers require Most application developers just want to playback and record media - Specifying where the content comes from - Specifying where the content should be rendered to - Manipulate a few playback controls - Have simple configurability That s what OpenMAX AL and OpenSL ES provides - A simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in With Cross-platform portability - No need to rewrite an application for every platform Copyright Khronos Group, Page 7

8 OpenSL ES and OpenMAX AL Framework Enhanced Audio Multimedia 3D Audio Advanced MIDI Audio Playback Video playback Recording Audio Effects Equalizer Image Rendering Camera Image capture Stereo widening Basic MIDI Radio and RDS Copyright Khronos Group, Page 8

9 Why Profiles? Functionality divided into basic groups - Most common groupings Profiles reduce fragmentation - Not all devices support all functionality - Only support profiles you need Application can query available profiles - Adapt to current system - Facilitates porting of applications Profiles can be combined - To best suit any product Optional features - Some features optional in all profiles - APIs are consistent when hardware is available Copyright Khronos Group, Page 9

10 Object Oriented Media Access Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output Objects have control interfaces - Play, Record, Seek, Rate Media Input Audio Mix Camera (AL) Display Window (AL) URI DSrc Player Object DSnk URI Memory Memory Content pipe Content pipe Copyright Khronos Group, Page 10

11 OpenSL ES and OpenMAX AL Independent - No dependency between the APIs either one can exist by itself - But designed to work together Compatible - Working groups collaborated to make sure the APIs work together well Consistent - Identical API architecture - Identical APIs for same functionality Distinct - OpenMAX AL represents basic multimedia functionality (audio, video and image) - OpenSL ES represents advanced audio functionality Copyright Khronos Group, Page 11

12 Application Layer Copyright Khronos Group, Page 12

13 OpenMAX AL Focused on Streaming Media Media playback and recording - Playback and recording of everything from video files to radio to camera input Full range of video effects and controls including playback rate, post processing and image manipulation - Apply advanced effects to captured images and video before sending to friends Complete metadata support - Read, Edit and Set the metadata for most types of content OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks Designed as a complete set of multimedia functionality - Unlike any other streaming media API Copyright Khronos Group, Page 13

14 OpenMAX AL Features Extensive camera controls - Flash modes, Metering modes - White balance controls, Focusing controls - Exposure compensation, ISO Sensitivity - Shutter speed & Aperture - Zoom (digital and optical) Analog radio controls - Tuning, RDS Audio routing - Application-selectable audio inputs and outputs, based on location, connectivity, etc. - I/O device capability querying Metadata extraction and insertion - Search/extract and insert/overwrite metadata in a variety of file formats Copyright Khronos Group, Page 14

15 OpenMAX AL Profiles Two profiles: - Media Player media playback-only devices - Media Player/Recorder full-featured media devices Optional features - Some features optional in all profiles hardware or use case constraints - APIs are consistent when hardware is available - Vibra, LED, Analog Radio and MIDI are optional features of OpenMAX AL Media Player Audio playback Video playback Image rendering Media Player/Recorder Camera controls Audio recording Video recording Image capture Copyright Khronos Group, Page 15

16 Embedded Audio Copyright Khronos Group, Page 16

17 OpenSL ES Powerful Mobile Audio Create theater quality audio experience - In a mobile device! Profiles reduce application customization - Applications can query available profiles - Develop to a specific profile or profile combination Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES Designed for implementation by either a hardware or software solution - Unlike any other advanced audio API Copyright Khronos Group, Page 17

18 Designed with audio application developers in mind Audio Playback and Recording - Playback of everything from music files to MIDI ring-tones to UI sounds Full range of effects and controls including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience Advanced MIDI - Use the output of the MIDI engine as a 3D sound source, making the ring tone appear coming from another direction than the music 3D Audio makes OpenSL ES the natural choice for any audio applicaiton - Better gaming experience; 3D-audio for conferencing calls; more vivid music experience 3D position Doppler MIDI messages Rate Preset Reverb Stereo widening Environmental Reverb Buffer Queues Pitch EQ Metadata extraction Virtualization LED & Vibra Copyright Khronos Group, Page 18

19 OpenSL ES Profiles Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Copyright Khronos Group, Page 19

20 & Practical Applications Copyright Khronos Group, Page 20

21 Audio Playback Example Create Engine object - To drive this session Create Audio Output Mix object - Method on Engine interface - Mix object drives audio output devices Create Media Player object - Method on Engine interface - Input is URI pointing to a local mp3 file - Output drives audio output mix Register event callback - Method on Media Player interface Set PlayState to Playing - Method on Media Player interface Wait for end of file event - Via registered callback Play Event Callback Media Player Object Application PlayItf Engine Object Output Mix Object EngineItf

22 Example: Playback of two audio files URI Memory Buffer Queue URI Memory Buffer Queue Audio Player SLPlayItf Audio Player SLPlayItf SL3DLocationItf SLVolumeItf Output Mix SLVolumeItf Copyright Khronos Group, Page 22

23 OpenSL ES and OpenMAX AL Framework Enhanced Audio Multimedia Advanced MIDI 3D Audio Audio Decoders Readers Audio Playback Video playback Writers Recording Video Scheduler Stereo Widening Clock Reverb Audio Effects Equalizer Audio post-processors Image Rendering Video Renderer Camera Image Image Decoders capture Audio Renderer Audio Encoders Stereo widening Equalizer Basic MIDI Image Encoders Radio and RDS Video Encoders Video Decoders Camera Copyright Khronos Group, Page 23

24 Integration Layer Copyright Khronos Group, Page 24

25 OpenMAX IL Component Networks Enables arbitrary multimedia pipelines by plugging blocks together - Componentized architecture abstracts multimedia functionality block interfaces Wide variety of building blocks for imaging, video and audio functions - Encode, decode, apply an effect, capture, render, split, mix, etc Enables blocks from different sources to work together - Blocks can be implemented in software or hardware Portable & Re-usable streaming media building blocks Copyright Khronos Group, Page 25

26 OpenMAX IL Component A component is a building block encapsulating one function - Parser, codec etc. Plumbing - Each component port is the entry point or exit point for a stream of data - Ports are connected between components Knobs - A component parameter is a value that is set prior to component execution - A component config is a value that may be set during component execution - Examples: rate, volume, resolution, scaling, bit rate, error correction, brightness, etc Control a standard interface common to all components to control: - The connection of ports and how data flows in and out of ports - Query/set configs and parameters - State management Copyright Khronos Group, Page 26

27 OpenMAX IL 1.1 OpenMAX IL 1.1 defines a set of standard components - To make most streaming applications easier to construct and more portable Standardized interfaces and controls - Readers/writers: 3gp, asf, image, video, audio - Audio decoders/encoders: AAC, AMR, MP3, WMA, Real Video - Audio post-processor: stereo widening, equalizer, reverb - Video decoders/encoders: MPEG4, H.264, etc - Image decoders/encoders: JPEG - Input devices: camera, audio input - Output devices: audio renderer, video renderer - Synchronization: clock component, video scheduler Copyright Khronos Group, Page 27

28 OpenMAX IL Example Graph Standardized component interfaces enable flexible media graphs Includes multi-stream synchronization Allows for custom plug-ins OpenMAX Component Interfaces Enables components to be flexibly connected in any graph topology E.g. send dance beats to 3D application for visualization AAC Audio Audio Decoder Plug-in Audio Renderer Speakers *.mp4 / *.3gp File Reader Clock for AV Sync MPEG4/ H.264 Video Video Decoder Time Data Decompressed Video Video Scheduler Video Renderer Display Example: MPEG-4 video synchronized with AAC audio decode Copyright Khronos Group, Page 28

29 OpenSL ES / OpenMAX AL Example Use Case with OpenMAX IL OR URI DataSource Media Player Output Mix File Reader Audio Decoder Audio Effect Audio Mixer Audio Sink Clock Copyright Khronos Group, Page 29

30 Khronos Audio and Streaming Media Khronos provides a full multimedia framework solution through OpenMAX and OpenSL ES - Components and applications are easily ported between platforms The complete package provides innumerable market opportunities for silicon, OS vendors, IP vendors and application developers - With both hardware and software solutions possible Meets the consumer demand for rich multimedia - Everything from interactive games to streaming content Copyright Khronos Group, Page 30

31 Thank You Copyright Khronos Group, Page 31

32 Copyright Khronos Group, Page 32

33 Sneak Peak Shhh! Copyright Khronos Group, Page 33

34 OpenMAX IL 1.2 Sneak Peek OpenMAX IL 1.1 can encounter interoperability issues when building complex multi-vendor graphs - Is primarily used in simpler use cases, for example codec plug-in, or closed frameworks OpenMAX IL 1.2 will address the needs of the multi-vendor market - Bug fixes & enhancements - Focus is on ability to build multi-vendor frameworks - Improved component interoperability - Easier replacement of components from different vendors - Better component control - Expected to be completed second half of 2011 Copyright Khronos Group, Page 34

35 OpenMAX AL & OpenSL ES Looking Forward 1.0 Conformance tests - Bug fixes and improved use case tests API Reference cards Q Easy access to information for developers OpenMAX AL 1.1 & OpenSL ES 1.1 Due out soon - Better support for portability between varying operating systems - Additional functionality - Improved buffer queues - Full set of conformance tests Updating example code - Ensuring example code is correct and functional OpenMAX AL Digital TV Extension - Technology-neutral access to Digital TV Copyright Khronos Group, Page 35

36 OpenMAX and OpenSL ES developer resources Growing developer community Khronos Message Boards - Monitored by the Khronos working groups Open source implementations - OpenMAX IL - OpenMAX AL Examples and tutorials - Provided by the working groups Reference cards - In the works available 2011 Copyright Khronos Group, Page 36

37 Market Overview OpenMAX IL - Implemented in a number of Smartphone operating systems - As well as in numerous feature phones - Open source implementation available Bellagio OpenMAX AL - Targeting the growing smartphone market - Native API for multimedia in Symbian^3 - Partial open source implementation available OpenSL ES - Targeting the growing smartphone market - Growing number of adopters - Native audio API in Android 2.3 Copyright Khronos Group, Page 37

38 OpenMAX & OpenSL ES Multimedia frameworks designed for the mobile arena - Designed by the mobile industry - Used by the mobile industry - Enable a complete mobile multimedia experience Copyright Khronos Group, Page 38

39 Thank You Copyright Khronos Group, Page 39