1HLO7UHYHWW 19,',$.KURQRV *URXS 2SHQ*/(6 Copyright Khronos Group, 2007 - Page 1
Handsets are becoming personal computing platform - not just phones - A real computer in your hand mobility, connectedness and numerous sensors Sophisticated media processing will be central to this handheld revolution - Graphics and media will become as pervasive as it is on the PC Games are leading-edge handheld media applications - As on the PC 100Ms / year 100Ks / year 10Ms / year Media Platforms 80 s Workstations 90 s PCs 00 s Handhelds Copyright Khronos Group, 2006 - Page 2
Accelerated 3D games are first wave of handheld media revolution - Significantly increased revenue per title for 3D games over 2D games is common BUT there are still significant difficulties and obstacles - Ecosystem friction between developers, carriers and handset OEMs - The industry needs to enable, encourage and unleash developer innovation We We will will explore this this handheld media revolution --What What are are the the issues issues holding holding it it back? back? --What What we we can can do do to to make make it it happen happen faster? faster? --What What opportunities it it will will create? create? Spider-Man 2: The Hero Returns Sony Pictures Spider-Man 2 3D: NY Subway Sony Pictures Spider-Man 2 Activision Copyright Khronos Group, 2006 - Page 3 2D Software 3D Accelerated 3D
Every handset is unique from the programmers perspective - Differences in operating system functions, Java implementations and media functionality Severe platform fragmentation exists today - ISVs need to port to and support 100s (even 1000s) of source variants of each title Symbian 7, 8, 9, UIQ, S60 PocketPC / Windows Mobile / WinCE Many Linux variants (and growing) Java MIDP-1, MIDP-2, JSR fragmentation Numerous RTOS with no consistently defined OS abstraction or media portability Nucleus, Synergy etc. Copyright Khronos Group, 2006 - Page 4
Java doesn t provide optimal media performance - Native acceleration can provide X5 higher performance for 3D graphics x2-20 Performance Accelerated Java Higher quality games x3-5 Performance Accelerated Native Seamlessly accelerated mixed-media Java Compelling native mixed-media applications Accelerated UI Un-accelerated Java Measured by Kishonti Java (fps) OpenGL ES (fps) Native Advantage Simple Games Nokia N93 18.6 77.8 x4.2 Sony Ericsson M600 6.0 30.6 X5.1 GLBenchmark direct port of identical Java benchmark Copyright Khronos Group, 2006 - Page 5
1DWLYH0HGLD$3,6HW Fragmentation Lack of Performance Need Consistent APIs on Every Handset Reliably available functionality C Native Programming Environment Familiarity for developers Eases ports from PCs, handhelds and consoles Native Media Acceleration APIs A cross-platform set of native APIs for media application development Copyright Khronos Group, 2006 - Page 6
$3,V Khronos APIs typically define interface between silicon and software - Enabling both communities - everyone wins ISVs see reduced fragmentation across multiple platforms - More software can reach market faster at a better level of functionality and quality Hardware vendors can accelerate many applications - Adding value to their platform +DUGZDUH3URYLGHU +DUGZDUH3URYLGHU +DUGZDUH3URYLGHU +DUGZDUH3URYLGHU +DUGZDUH3URYLGHU 6RIWZDUH'HYHORSHU 6RIWZDUH'HYHORSHU 6RIWZDUH'HYHORSHU 6RIWZDUH'HYHORSHU 6RIWZDUH'HYHORSHU Copyright Khronos Group, 2006 - Page 7
.KURQRV 2SHQ0HGLD Open Membership Any company is welcome Funded by membership dues - $6K / year Open Standards Publicly available on web-site Royalty-free Open Standards for Media Authoring and Acceleration Khronos typically typically develops Foundation-Level APIs APIs High-performance Close-to-the-metal access access to to hardware acceleration Good Good foundation for for higher-level engines engines and and middleware Copyright Khronos Group, 2006 - Page 8
Copyright Khronos Group, 2006 - Page 9 Over 100 companies creating media authoring and acceleration standards
.KURQRV Cross-platform graphics authoring/acceleration ecosystem Safety Critical 2D/3D Cross platform desktop 3D JO); Effects Framework 3D Authoring Dynamic Media Authoring Dynamic Media Authoring Standards 2D/3D DirectX-like set of native APIs Includes mixed media acceleration and OS portability APIs Copyright Khronos Group, 2006 - Page 10 Vector 2D Streaming Media Enhanced Audio Embedded Media Acceleration APIs Composition Working Group Hardware acceleration for window systems
.KURQRV $3,V Market adoption in media-accelerated handsets We Are Here - OpenGL ES 1.1 is widespread - OpenVG is in rapid adoption - OpenMAX is being implemented - OpenSL ES is being designed 100% 3D Graphics Vector 2D Streaming Media Enhanced Audio Mid-2004 OpenGL ES 1.1 Spec release Mid-2005 OpenVG 1.0 Spec release Beginning-2006 OpenMax IL 1.0 Spec release Mid 2007 OpenSL ES 1.0 Spec release End-2007 Copyright Khronos Group, 2006 - Page 11
Faster Performance at Higher Quality Hardware delivers at least 10 times the performance of software even on low-cost systems with low-end CPUs 18 16 14 12 10 8 6 4 2 0 ARM9 32:(5()),&,(1&< 32-Bit RISC at 400MHz DSP at 175MHz APA 512 MiMagic 6 Less Power Hardware accelerators exploit parallelism in a media pipeline to give a x10 increase in power efficiency over software Better User Experience Smaller screens need more advanced graphics processing per pixel Copyright Khronos Group, 2006 - Page 12
1DWLYH0HGLD$3,V 900 800 700 600 MM phones Non 3D MM phones Native API based phones Millions of units Jon Peddie Research Handheld Multimedia Devices report 500 400 300 200 100 0 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Copyright Khronos Group, 2006 - Page 13
2SHQ*/(6² ' JSR 239 Defining official Java Bindings to OpenGL ES Java Applications C/C++ Applications Usable directly by applications Usable by higherlevel graphics libraries Close to the hardware API provides portability AND flexibility Hardware OpenGL ES Engines Scenegraph APIs M3G (JSR 184) Games Engines Middleware Libraries Software OpenGL ES Engines Brings advanced 2D/3D graphics to a wide range of OS platforms Copyright Khronos Group, 2006 - Page 14
6KDGHUV ' OpenGL ES 2.0 for new generation programmable 3D hardware - Vertex and pixel shaders programmed through GLSL ES shading language OpenGL ES 1.1 for fixed function 3D graphics hardware - OpenGL ES 1.1 drivers can be shipped on new generation programmable hardware 3D Tomorrow - Shader Programmability Doom 3 s Zombies Unreal s Rocks 3D Today Fixed Functionality Halo s Ice Far Cry s Water Copyright Khronos Group, 2006 - Page 15
2SHQ9*² Vector graphics - basis of popular formats such as Flash and SVG - User interfaces, screen savers, 2D Games, mapping and GPS, E-book readers Scalable Bezier curves - Scaled and positioned at full quality - not polygon based But 2D vector graphics historically run un-accelerated! - Not effective on low-powered handset CPUs OpenVG ACCELERATES Flash and SVG for the first time - Better quality, more interactivity - at less power Applications and UI SVG/Flash/ Font Packages etc.. 2D Hardware Acceleration Copyright Khronos Group, 2006 - Page 16
2SHQ9* X20 OpenVG 1.0 beginning to be widely deployed - A range of software and hardware implementations are coming to market Vector Graphics Performance X1 Software Implementations X3-7 Layered Implementations over OpenGL ES hardware Dedicated OpenVG Cores low power Copyright Khronos Group, 2006 - Page 17
2SHQ0$;² Componentized architecture abstracts multimedia functionality blocks - Such as video codecs - blocks can be implemented in software or hardware Media blocks cover multiple media domains with extensive functionality - Domains: audio, video, image, of some combination thereof - Functions: encode, decode, apply an effect, capture, render, split, mix, etc Build sophisticated multimedia pipelines by plugging blocks together - Enables blocks from different vendors to work together Portable & Re-usable streaming media building blocks Copyright Khronos Group, 2006 - Page 18
2SHQ0$; Standardized component interfaces enable flexible media graphs Includes multi-stream synchronization Allows for custom plug-ins OpenMAX Component Interfaces Enables components to be flexibly connected in any graph topology E.g. send dance beats to application for visualization *.mp4 *.mp4 / / *.3gp File *.3gp File Reader Reader AAC Audio Clock for Clock for AV AV Sync Sync Audio Decoder Audio Decoder Time Data Plug-in Plug-in Audio Renderer Audio Renderer Speakers MPEG4/ H.264 Video Video Decoder Video Decoder Decompressed Video Video Scheduler Video Scheduler Video Renderer Video Renderer Display Example: MPEG-4 video synchronized with AAC audio decode Copyright Khronos Group, 2006 - Page 19
2SHQ6/(6² Audio API space is fragmented for applications - Many proprietary audio APIs of varying functionality native and Java - Even playing a simple sound on different platform requires different code No standard API to access audio hardware acceleration - Audio silicon developers have to support many high-level APIs directly on silicon Applications OpenSL ES is a royalty-free, open standard Native interface to audio acceleration General audio functionality for applications, high-level libraries or middleware Middleware, JSR 135, JSR 234 High-level Audio Libraries Media Silicon (Audio HW, CPUs, DSPs) Copyright Khronos Group, 2006 - Page 20
2SHQ0$; $/ (Enhanced audio API) Audio Effects Advanced MIDI 3D Audio Buffer queues Audio Playback Audio Recording MIDI Video playback Video recording Analog Radio & RDS Camera Image capture & display (Multimedia API) Overlap streaming media functionality uses consistent APIs Copyright Khronos Group, 2006 - Page 21
DirectX! Except its an open standard, cross-platform, royalty-free and streamlined for handheld devices is a set of C-native APIs for handheld games and media applications 2D, 3D, video and audio media types are all seamlessly accelerated by minimizes source changes when porting games and applications from phone to phone Copyright Khronos Group, 2006 - Page 22
1DWLYH)UDJPHQWDWLRQ Software Platform (e.g. Brew, Symbian UIQ, S60, WIPI) Native Applications JAR File Java JVM Java Applications Carrier OTA Provisioning No Native OTA Deployment User Interface Different OS APIs on every handset 3D Game Engines Flash/SVG Players TV/Video/Audio Players Proprietary media APIs, single-threaded, not tested for mixed-media operation Bindings Handheld devices increasingly need advanced graphics and media acceleration for user interfaces, 3D gaming, Flash, TV Media Accelerator Silicon GPU / DSPs / CPUs Kernel Operating System (e.g. Rex, Symbian OS, Windows Mobile, Linux, Nucleus, Synergy) CPU Severe API fragmentation to access OS resources and media acceleration = HUNDREDS of source variants per game Copyright Khronos Group, 2006 - Page 23
7KH2SHQ.2'( Software Platform (e.g. Brew, Symbian UIQ, S60, WIPI) System Abstraction (OpenKODE Core) User Interface Native Applications 3D Game Engines Flash/SVG Players TV/Video/Audio Players Media Accelerator Silicon GPU / DSPs / CPUs Kernel Operating System (e.g. Rex, Symbian OS, Windows Mobile, Linux, Nucleus, Synergy) CPU AL 2D/3D Vector 2D Streaming Media Enhanced Audio Trans-API communication (EGL) Java JVM Java Applications Bindings Media APIs = Selected Khronos media APIs for state-of-the-art media acceleration + Trans-API data and event coordination WITH full trans-api Conformance Tests (EGL) + System Abstraction APIs for portable access to operating system resources, input devices and displays (OpenKODE Core) = A coherent, crossplatform API set for NATIVE portable media applications Copyright Khronos Group, 2006 - Page 24
2SHQ.2'( Trans-API Conformance Tests A conformant OpenKODE must pass all individual conformance tests PLUS trans-api tests to ensure APIs can communicate and operate concurrently 3D Interactive Audio EGL Trans-API Communication Vector 2D Streaming Media Selected Media APIs PLUS any EGL needed for those APIs to communicate OpenKODE Core Events, IO, files, memory, processes, strings, math, locale, time, networking, windowing, crypto, debug Operating System Abstraction APIs to reduce source fragmentation OpenKODE provides foundation-level acceleration for for advanced user user interfaces and and media media applications that that mix mix multiple multiple media media types types Copyright Khronos Group, 2006 - Page 25
-DYD JAR File Java OTA Provisioning Native Installer Native Applications Software Platform Java JVM Java Applications JSR 184 / 239 Other JSRs Existing provisioning, billing and security systems can be used to OTA provision native OpenKODE applications. Native applications will need certification as Brew does today Media Accelerator Silicon Kernel Operating System CPU Copyright Khronos Group, 2006 - Page 26
2SHQ.2'( OpenKODE 1.0 Provisional released at 3GSM 2007 - Twelve months from kick-off meeting to specification on web-site - Encouraging developer feedback before spec finalization mid-2007 Full conformance tests will be released in 2Q07 - Written by Futuremark tests OpenKODE Core AND trans-api operation Call for participation and contributions Scope Agreed June F2F OpenKODE 1.0 Provisional Release OpenKODE 1.0 Final Release Target 1Q06 2Q06 3Q06 4Q06 1Q07 2Q07 3Q07 Kickoff F2F meeting Strawman specs complete November F2F Conformance Tests to be Released If you are a developer download the spec and give us your feedback! Spec at www.khronos.org/openkode/ Feedback forums at www.khronos.org/message_boards/ Copyright Khronos Group, 2006 - Page 27
2SHQ.2'( OpenKODE 1.0 allows media API selection - To enable widespread early adoption Simple build out of existing OpenGL ES adoption - Expecting rapid industry momentum Can be deployed on almost any OS - Including RTOS for media on feature phones Full DirectX functional equivalence. Robust trans- API functionality and testing OpenKODE Core API has no silicon dependency and takes <50KB. OpenKODE provides enhanced 2D/3D mixed mode operation OpenGL ES is everywhere today 2H07 1H07 Now 5726 Copyright Khronos Group, 2006 - Page 28
OpenKODE 1.0 announced in Feb07 Momentum is already building quickly Futuremark developing Conformance Tests for OpenKODE. Graphic Remedy developing OpenKODE profiling tools OpenKODE Core can drop-in alongside OpenKODE media stacks. Typically middleware solutions provide native OTA provisioning over Java networks sometimes with portable binary capability Announced OpenKODE media stacks with OpenGL ES and increasingly OpenVG Tools OpenKODE Core and Middleware Silicon Acceleration Copyright Khronos Group, 2006 - Page 29
Breakthrough mobile games and applications will not treat mobile devices as small consoles or PCs They will take advantage of the unique capabilities of mobile devices: Use of handset sensors - Cameras for video and images, sound Mobility - With GPS location awareness Connectedness - Unique information flow and social interactions An advanced media stack enables innovative applications that mix many kinds of media acceleration A GPS phone processes OpenMAX video to recognize buildings and landmarks Copyright Khronos Group, 2006 - Page 30
2SHQ.2'( Advanced UI OpenVG - OpenGL ES for special effects Accelerated Java Applications - OpenGL ES 184/239, OpenVG - 287 Augmented Reality OpenMAX AL - Video processed & rotoscoped with OpenGL ES 3D Advanced Navigation - OpenVG - 3D terrain/satellite data using OpenGL ES Video Telephony OpenMAX AL - OpenGL ES lip-synch 3D avatar PVR Application OpenMAX AL - OpenVG-based UI and subtitles Music Visualizer - OpenSL ES - OpenGL ES 3D synchronized to music beat 3D Game OpenGL ES - OpenSL ES positional audio & OpenVG HUD/menus 3D TV Channel Selector OpenMAX AL - OpenGL ES 3D multi-channel visualization OpenKODE creates the opportunity to create new applications and user interfaces that use multiple media types Copyright Khronos Group, 2006 - Page 31
$3, 2SHQ.2'( Applications and Media Engines OpenKODE used both by applications - and the window system to render UI elements Portable applications use OpenKODE Core calls to synchronize updates with the Windowing System OpenKODE Core Media APIs Provided by OS vendor Khronos working to enable it be built using open standard APIs Advanced Window System Window System uses the Screen Composition API to bring application and UI elements together on the screen Screen Composition API GPU Multi-tasking GPU accelerates multiple applications, window system UI / transitions and screen composition Enabling advanced composited window systems to be build entirely from open standards Copyright Khronos Group, 2006 - Page 32
2SHQ.2'( Carriers Open standards to simply specify a native, cross-platform media API stack Compelling media applications increase revenue through existing provisioning, billing, DRM certification infrastructure Native OTA provisioning uses existing certification, security, billing, lifecycle management infrastructure Proven media stack architecture based on multi-vendor openstandards - reduces risk Cross-platform native APIs reduce source fragmentation and increase performance Handset OEMs Native, cross-platform media API stack Content Providers Copyright Khronos Group, 2006 - Page 33
We now have a forward-looking, multi-threaded media stack architecture - Open and royalty-free to encourage industry innovation Foundation for new-generation interfaces and mixed-media applications - Driving new markets and revenue opportunities OpenKODE is a open-standard, multi-vendor native media-stack - Helping to solve industry fragmentation and performance issues If you are an handset OEM.. Embed an OpenKODE 1.0 media stack! If you are a carrier.. Specify OpenKODE 1.0 for your native applications! If you are a developer.. Demand OpenKODE 1.0 for development! OpenKODE is is a significant opportunity for for the the handheld industry to to help help evolve handheld devices into into the the most most pervasive media-capable computing devices the the world has has yet yet seen seen Copyright Khronos Group, 2006 - Page 34
If you are a developer download the spec and give us your feedback! Specification at www.khronos.org/openkode/ Feedback forums at www.khronos.org/message_boards/ These slides and Khronos membership details at www.khronos.org Copyright Khronos Group, 2006 - Page 35