3RUWDEOH1DWLYH0RELOH 0HGLD$SSOLFDWLRQV 1HLO7UHYHWW 9LFH3UHVLGHQW(PEHGGHG&RQWHQW19,',$ 3UHVLGHQW.KURQRV*URXS Copyright Khronos Group, 2007 - Page 1
3HUYDVLYH0RELOH0HGLD&RPSXWLQJ Handsets are becoming personal computing platform - not just phones - A real computer in your hand mobility, connectedness and numerous sensors Sophisticated media processing will be central to this handheld revolution - Graphics and media will become as pervasive as it is on the PC Games are leading-edge handheld media applications - As on the PC 100Ms / year 100Ks / year 10Ms / year Media Platforms 80 s Workstations 90 s PCs 00 s Handhelds Copyright Khronos Group, 2007 - Page 2
0RELOH0HGLD5HYROXWLRQDQG,VVXHV Accelerated 3D games are at the forefront of handheld media revolution - Significantly increased revenue per title for 3D games over 2D games is possible BUT there are still significant difficulties and obstacles - The market for mobile games is not growing as quickly as expected - The industry needs to enable, encourage and unleash developer innovation We We will will explore this this handheld media revolution --What What are are the the issues issues holding holding it it back? back? --What What we we can can do do to to make make it it happen happen faster? faster? --What What opportunities it it will will create? create? Spider-Man 2: The Hero Returns Sony Pictures Spider-Man 2 3D: NY Subway Sony Pictures Spider-Man 2 Activision Copyright Khronos Group, 2007 - Page 3 2D Software 3D Accelerated 3D
0RELOH0HGLD,VVXH)UDJPHQWDWLRQ Every handset is unique from the programmers perspective - Differences in operating system functions, Java implementations and media functionality Severe platform fragmentation exists today - ISVs need to port to and support 100s (even 1000s) of source variants of each title Symbian 7, 8, 9, UIQ, S60 PocketPC / Windows Mobile / WinCE Many Linux variants (and growing) Java MIDP-1, MIDP-2, JSR fragmentation Numerous RTOS with no consistently defined OS abstraction or media portability Nucleus, Synergy etc. Copyright Khronos Group, 2007 - Page 4
0RELOH0HGLD,VVXH/RZ3HUIRUPDQFH Java is most widely used programming language for mobile applications - Well developed over-the-air deployment infrastructure BUT - Java doesn t provide optimal media performance - Native acceleration can provide X5 higher performance for 3D graphics x2-20 Performance Accelerated Java Higher quality games x3-5 Performance Accelerated Native Seamlessly accelerated mixed-media Java Compelling native mixed-media applications Accelerated UI Un-accelerated Java Measured by Kishonti Java (fps) OpenGL ES (fps) Native Advantage Simple Games Nokia N93 18.6 77.8 x4.2 Sony Ericsson M600 6.0 30.6 X5.1 Copyright Khronos Group, 2007 - Page 5 GLBenchmark direct port of identical Java benchmark
$Q2EYLRXV6ROXWLRQ Lack of media Performance Functionality Fragmentation C Native Programming Environment Familiar from PCs and consoles Consistent APIs on every Handset Reliable functionality Native Media Acceleration APIs Open standard, cross-platform, native APIs for media application development Khronos develops Foundation-Level API API Standards Close-to-the-silicon access access to to media media acceleration to to create create a a robust robust foundation for for media media middleware and and applications Copyright Khronos Group, 2007 - Page 6
.KURQRV 2SHQ0HGLD6WDQGDUGV Open International Membership Any company is welcome Funded by membership dues - $6K / year Open Standards Publicly available on web-site To be used by any company in any products Open Standards for Media Authoring and Acceleration Creating Market Opportunities Bringing together industry s leading media technology experts Investing hundreds of man years to create state-of-the-art media API standards Making those standards available to the industry ROYALTY-FREE Copyright Khronos Group, 2007 - Page 7
Copyright Khronos Group, 2007 - Page 8 Over 100 companies creating media authoring and acceleration standards
.KURQRV6WDQGDUGV(FRV\VWHP Cross-platform graphics authoring/acceleration ecosystem Safety Critical 2D/3D Comprehensive Mobile Media Acceleration Solution Cross platform desktop 3D JO); Effects Framework 3D Authoring Dynamic Media Authoring Dynamic Media Authoring Standards 2D/3D DirectX-like set of native APIs Includes mixed media acceleration and OS portability APIs Copyright Khronos Group, 2007 - Page 9 Vector 2D Streaming Media Enhanced Audio Embedded Media Acceleration APIs Composition Working Group Hardware acceleration for window systems
$GYDQWDJHVRI0HGLD$FFHOHUDWLRQ Faster Performance at Higher Quality Hardware delivers smoother interaction with much better looking graphics 18 16 14 12 10 8 6 4 2 0 ARM9 32:(5()),&,(1&< 32-Bit RISC at 400MHz DSP at 175MHz APA 512 MiMagic 6 Software 3D Accelerated 3D Less Power Hardware accelerators exploit media pipeline parallelism and caching for a x10 increase in power efficiency over software Better User Experience Smaller screens need more advanced graphics processing per pixel Copyright Khronos Group, 2007 - Page 10
3HQHWUDWLRQRI1DWLYH0HGLD$3,V 900 800 700 600 MM phones Non 3D MM phones Native API based phones Millions of units Jon Peddie Research Handheld Multimedia Devices report 500 400 300 200 100 0 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Copyright Khronos Group, 2007 - Page 11
2SHQ*/(6² &HQWUDOWR0RELOH' JSR 239 Defining official Java Bindings to OpenGL ES Java Applications C/C++ Applications Usable directly by applications Usable by higherlevel graphics libraries Scenegraph APIs M3G (JSR 184) Games Engines Middleware Libraries Foundation Close to the silicon API provides portability AND flexibility Brings advanced 2D/3D graphics to a wide range of OS platforms Copyright Khronos Group, 2007 - Page 12
2SHQ*/(6$SSOLFDWLRQVDQG'HYLFHV OpenGL ES 1.1 has become the most widely deployed 3D API - Used in diverse applications and devices close to 100M devices already shipped Advanced User Interfaces - iphone uses OpenGL ES 3D Navigation - Using data such as Google Earth Mobile Games - $7.2Bn market in 2010 (Informa) Nokia N93, iphone, Sony PS3 - All use OpenGL ES 1.1 Copyright Khronos Group, 2007 - Page 13
6KDGHUVDQG1H[W*HQHUDWLRQ' OpenGL ES 1.1 for fixed function 3D graphics hardware - OpenGL ES 1.1 drivers can be shipped on new generation programmable hardware OpenGL ES 2.0 for new generation programmable 3D hardware - Vertex and pixel shaders programmed through GLSL ES shading language 3D Tomorrow - Shader Programmability Doom 3 s Zombies Unreal s Rocks 3D Today Fixed Functionality Halo s Ice Far Cry s Water Copyright Khronos Group, 2007 - Page 14
2SHQ9* $FFHOHUDWHG9HFWRU*UDSKLFV Vector graphics - basis of popular formats such as Flash and SVG - Bezier curves - scaled and positioned at full quality - not polygon based But 2D vector graphics historically run un-accelerated! - Not effective on low-powered handset CPUs OpenVG ACCELERATES Flash and SVG for the first time - Better quality, more interactivity - at less power OpenVG is an OpenGL-style, low-level API - Can be used by players such as Flash and SVG or programmed directly Applications and UI SVG/Flash/ Font Packages etc.. 2D Hardware Acceleration Copyright Khronos Group, 2007 - Page 15
9HFWRU*UDSKLFV9DOXH Users - High-quality, graphics-rich applications and content - Wallpapers, Cartoons, 2D Games, Maps / Navigation, E-book readers, Greeting Cards Handset OEMs - Reduces cost of supporting different screen - assets scale at full quality unlike bitmaps - Alternative to 3D graphics at less cost and power Carriers - Animated and interactive advertisements - Attractive, high-quality vector graphics content creates download demand Copyright Khronos Group, 2007 - Page 16
5DLVLQJ'DQG'9LVXDO4XDOLW\ State-of-the-art APIs enable compelling consumer displays Advanced functionality, fast interactivity and extremely high quality High-quality 2D graphics using OpenVG High-quality 3D displays using OpenGL ES Older generation APIs Provide rudimentary graphics functionality and quality Copyright Khronos Group, 2007 - Page 17
2SHQ0$;² 6WUHDPLQJ0HGLD Componentized architecture abstracts multimedia functionality blocks - Such as video codecs - blocks can be implemented in software or hardware Media blocks cover multiple media domains with extensive functionality - Domains: audio, video, image, of some combination thereof - Functions: encode, decode, apply an effect, capture, render, split, mix, etc Build sophisticated multimedia pipelines by plugging blocks together - Enables blocks from different vendors to work together Portable & Re-usable streaming media building blocks Copyright Khronos Group, 2007 - Page 18
2SHQ0$;([DPSOH*UDSK Standardized component interfaces enable flexible media graphs Includes multi-stream synchronization Allows for custom plug-ins OpenMAX Component Interfaces Enables components to be flexibly connected in any graph topology E.g. send dance beats to application for visualization *.mp4 *.mp4 / / *.3gp File *.3gp File Reader Reader AAC Audio Clock for Clock for AV AV Sync Sync Audio Decoder Audio Decoder Time Data Plug-in Plug-in Audio Renderer Audio Renderer Speakers MPEG4/ H.264 Video Video Decoder Video Decoder Decompressed Video Video Scheduler Video Scheduler Video Renderer Video Renderer Display Example: MPEG-4 video synchronized with AAC audio decode Copyright Khronos Group, 2007 - Page 19
&DPHUD3URFHVVLQJLQ2SHQ0$; Panoramas with single click & hold and pan Strobe capture Copyright Khronos Group, 2007 - Page 20
2SHQ6/(6² )RXQGDWLRQ/HYHO$XGLR Audio API space is fragmented for applications - Many proprietary audio APIs of varying functionality native and Java - Even playing a simple sound on different platform requires different code No standard API to access audio hardware acceleration - Audio silicon developers have to support many high-level APIs directly on silicon Applications OpenSL ES is a royalty-free, open standard Native interface to audio acceleration General audio functionality for applications, high-level libraries or middleware Middleware, JSR 135, JSR 234 High-level Audio Libraries Media Silicon (Audio HW, CPUs, DSPs) Copyright Khronos Group, 2007 - Page 21
Its like DirectX for mobile phones! Except its an open standard, cross-platform, royalty-free and streamlined for handheld devices is a set of C-native APIs for handheld games and media applications 2D, 3D, video and audio media types are all seamlessly accelerated by minimizes source changes when porting games and applications from phone to phone Copyright Khronos Group, 2007 - Page 22
1DWLYH)UDJPHQWDWLRQ Software Platform (e.g. Brew, Symbian UIQ, S60, WIPI) Native Applications JAR File Java JVM Java Applications Carrier OTA Provisioning No Native OTA Deployment User Interface Different OS APIs on every handset 3D Game Engines Flash/SVG Players TV/Video/Audio Players Proprietary media APIs, single-threaded, not tested for mixed-media operation Bindings Handheld devices increasingly need advanced graphics and media acceleration for user interfaces, 3D gaming, Flash, TV Media Accelerator Silicon GPU / DSPs / CPUs Kernel Operating System (e.g. Rex, Symbian OS, Windows Mobile, Linux, Nucleus, Synergy) CPU Severe API fragmentation to access OS resources and media acceleration = HUNDREDS of source variants per game Copyright Khronos Group, 2007 - Page 23
7KH2SHQ.2'(6ROXWLRQ Software Platform (e.g. Brew, Symbian UIQ, S60, WIPI) System Abstraction (OpenKODE Core) User Interface Native Applications 3D Game Engines Flash/SVG Players TV/Video/Audio Players Media Accelerator Silicon GPU / DSPs / CPUs Kernel Operating System (e.g. Rex, Symbian OS, Windows Mobile, Linux, Nucleus, Synergy) CPU AL 2D/3D Vector 2D Streaming Media Enhanced Audio Trans-API communication (EGL) Java JVM Java Applications Bindings Media APIs = Selected Khronos media APIs for state-of-the-art media acceleration + Trans-API data and event coordination WITH full trans-api Conformance Tests (EGL) + System Abstraction APIs for portable access to operating system resources, input devices and displays (OpenKODE Core) = A coherent, crossplatform API set for NATIVE portable media applications Copyright Khronos Group, 2007 - Page 24
2SHQ.2'(&RPSRQHQWV Trans-API Conformance Tests A conformant OpenKODE must pass all individual conformance tests PLUS trans-api tests to ensure APIs can communicate and operate concurrently 3D Interactive Audio EGL Trans-API Communication Vector 2D Streaming Media Selected Media APIs PLUS any EGL needed for those APIs to communicate OpenKODE Core Events, IO, files, memory, processes, strings, math, locale, time, networking, windowing, crypto, debug Operating System Abstraction APIs to reduce source fragmentation OpenKODE provides foundation-level acceleration for for advanced user user interfaces and and media media applications that that mix mix multiple multiple media media types types Copyright Khronos Group, 2007 - Page 25
/HYHUDJLQJDQG$FFHOHUDWLQJ-DYD JAR File Java OTA Provisioning Native Installer Native Applications Software Platform Java JVM Java Applications JSR 184 / 239 Other JSRs Existing provisioning, billing and security systems can be used to OTA provision native OpenKODE applications. Native applications will need certification as Brew does today Media Accelerator Silicon Kernel Operating System CPU Copyright Khronos Group, 2007 - Page 26
2SHQ.2'(5ROO2XW 2XW OpenKODE 1.0 allows media API selection - To enable widespread early adoption Simple build out of existing OpenGL ES adoption - Expecting rapid industry momentum Can be deployed on almost any OS - Including RTOS for media on feature phones Full DirectX functional equivalence. Robust trans- API functionality and testing OpenKODE Core API has no silicon dependency and takes <50KB. OpenKODE provides enhanced 2D/3D mixed mode operation OpenGL ES is everywhere today 2H07 1H07 Now Copyright Khronos Group, 2007 - Page 27 5726
2SHQ.2'(1DWLYH(FRV\VWHP Carriers Open standards to simply specify a native, cross-platform media API stack Compelling media applications increase revenue through existing provisioning, billing, DRM certification infrastructure Native OTA provisioning uses existing certification, security, billing, lifecycle management infrastructure Proven media stack architecture based on multi-vendor openstandards - reduces risk Cross-platform native APIs reduce source fragmentation and increase performance Handset OEMs Native, cross-platform media API stack Content Providers Copyright Khronos Group, 2007 - Page 28
,QQRYDWLYH0L[HG0HGLD$SSOLFDWLRQV Breakthrough mobile applications will leverage the unique capabilities of mobile devices: Use of handset sensors - Cameras for video and images, sound Mobility - With GPS location awareness Connectedness - Unique information flow and social interactions Pictures We started at the Eiffel Tower Video Family fun at the fountain OpenKODE enables innovative applications that MIX diverse media types - Augmented Reality - 3D over live video - TV channel selector in 3D - 3D avatars linked to speech - Graphics visualizations linked to music Handheld GeoBlog Concept Navigation using 2D maps, satellite, terrain and 3D city data. GPS phone captures and uploads images, audio and video. Location tagging inserts media correctly into the map. Blog can be viewed on handheld or PC. Opens innovation in Blogs, guides, journalism etc etc etc.. Copyright Khronos Group, 2007 - Page 29
+RZ'RHV.KURQRV0DNH0RQH\" It doesn t! Khronos is purely a non-profit organization - Funded by member dues to cover costs Our members make money by selling PRODUCTS enabled by standards - NOT trying to charge for the standard itself $ An open, royalty free API standard creates much larger market opportunities Selling an API would generate relatively small amounts of revenue Our members cooperate to create standards and compete in the marketplace with products that use Khronos standards Copyright Khronos Group, 2007 - Page 30
:K\8VH.KURQRV6WDQGDUGV" Foundation Level APIs - Close to the silicon fundamental functionality needed on every platform Designed by industry experts - The industry leaders in media silicon, platform and software are all Khronos members Reduces development and deployment costs - Widespread industry adoption ensures competitive silicon and software supply chain Open Standards not controlled by any single company - Any company can join Khronos to have a voice in how standards evolve Royalty-free - Khronos is committed to generating market opportunities for its members and the industry Copyright Khronos Group, 2007 - Page 31
7KDQN<RXIRU<RXU7LPH7RGD\ All these slides and Khronos membership details at www.khronos.org Also at the Chinese version of the web-site at www.khronos.cn Please consider joining Khronos to help us build market opportunities You will be very welcome in the Khronos family! Copyright Khronos Group, 2007 - Page 32