Copyright Khronos Group, Page 1. Khronos Overview. Taiwan, February 2012

Transcription

1 Copyright Khronos Group, Page 1 Khronos Overview Taiwan, February 2012

2 Copyright Khronos Group, Page 2 Khronos - Connecting Software to Silicon Creating open, royalty-free API standards - Focus on graphics, dynamic media, compute and sensor hardware Low-level - just above raw silicon - Foundation functionality needed on every platform Safe forum for industry cooperation - By the industry for the industry - Open to any company to join - IP framework to protect members and industry APIs enable software developers to turn silicon functionality into rich end user experiences

3 Copyright Khronos Group, Page 3 Khronos API Ecosystem - Trends High-end API technology is created on high-end platforms Breakthrough apps embrace mobility s strengths not just treat phones as small PCs - will need complex, interoperating APIs e.g. Augmented Reality Mobile is the new platform for apps innovation. Mobile APIs are needed to unlock hardware potential while conserving battery life As platforms diversify mobile, TV, embedded HTML5 will become increasingly important as a universal app platform

4 Copyright Khronos Group, Page 4 Apple Over 100 members any company worldwide is welcome to join Board of Promoters

5 Khronos Ecosystem of Standards Advanced Audio Camera, Images and Streaming Media Computer Vision 3D Digital Asset Exchange format Hardware accelerated vision functions Surface Management Parallel Computing Embedded and Mobile 3D Vector 2D Unified Sensor and Input Processing Web Compute Cross platform desktop 3D Plugin-free 3D Web Content Khronos creates royalty-free specifications to meet real market needs and helps drive industry adoption across multiple platforms Copyright Khronos Group, Page 5

6 Copyright Khronos Group, Page 6 Khronos Working Group Process Academic Members - Participation in WGs Contributor Members - Participation and vote in WGs Working Groups (WGs) One working group per API Promoter Members - Participation and vote in WGs - Board seat for strategy, budget and spec ratification $$$ $ Conformance Tests and Adopters Program Ratified Specifications SDKs, Sample, Ref Cards and Man Pages Educator Guidelines and Courseware Feedback Members Wider Industry $ Adopters Build conformant implementation and products Developers Develop applications using the APIs Educators Collaboration for Khronos API Education

7 Copyright Khronos Group, Page 7 Milestones This Year at Khronos OpenMAX AL 1.1 Spec Released OpenSL ES 1.1 Spec Released WebCL Working Group Announced OpenGL 4.2 Spec Released Computer Vision Working Group Announced EGLStreams Spec Release Mar11 Nov11 Jan11 WebGL 1.0 Spec Released Aug11 OpenCL 1.2 Spec Release Dec11 StreamInput Working Group Announced COLLADA 1.4 Conformance Tests Released

8 OpenGL for Each GPU Generation Tessellation and Compute 4.X Shape realism Geometry Shaders 3.X Vertex and Fragment Shaders 2.X Fixed Function 1.X Surface realism Copyright Khronos Group, Page 8

9 Copyright Khronos Group, Page 9 Accelerating OpenGL Innovation Bringing state-of-theart functionality to cross-platform graphics OpenGL 4.1 OpenGL 3.3/4.0 OpenGL 3.2 OpenGL 3.1 OpenGL 2.0 OpenGL 2.1 OpenGL DirectX 9.0c DirectX 10.0 DirectX 10.1 DirectX 11

10 Copyright Khronos Group, Page 10 OpenGL ES Adoption OpenGL for embedded and mobile devices - Eliminates redundant and legacy features - Adds extensions to make it mobile-friendly The dominant 3D API for mobile devices - Widely adopted for STB, DTV, automotive, - Hundreds and hundreds of millions shipped OpenGL ES Fully programmable pipeline Runs high-end content and engines - UE3, Unity, Unigine, Rage

11 Copyright Khronos Group, Page 11 Processor Parallelism CPUs Multiple cores driving performance increases Emerging Intersection GPUs Increasingly general purpose data-parallel computing Multiprocessor programming e.g. OpenMP Heterogeneous Computing Graphics APIs and Shading Languages OpenCL is a programming framework for heterogeneous compute resources

12 Copyright Khronos 2009 Group, Page 12 OpenCL Heterogeneous Computing Framework for programming diverse parallel computing resources in a system Platform Layer API - Query, select and initialize compute devices Kernel Language Specification - Subset of ISO C99 with language extensions Runtime API - Execute compute kernels gather results OpenCL has Embedded profile - No need for a separate ES spec

13 Copyright Khronos Group, Page 13 OpenCL 1.2 Announced in December Significant updates - Khronos being responsive to developer requests - Updated OpenCL 1.2 conformance tests available Backward compatible upgrade to OpenCL OpenCL 1.2 will run any OpenCL 1.0 and OpenCL 1.1 programs - OpenCL 1.2 platform can contain 1.0, 1.1 and 1.2 devices - Maintains embedded profile for mobile and embedded devices Significant functionality added in response to developer demands - Partition Devices into sub devices - Separate compilation and linking of programs - Enhanced Image Support - Custom devices and built-in kernels - DX9 and DX11 Surface Sharing - Installable Client Drivers (optional)

14 Copyright Khronos Group, Page 14 Video and Audio Acceleration OpenSL ES 1.1 and OpenMAX AL 1.1 launched in January Additional functionality including improved buffer queuing - OpenMAX AL Digital TV Extension - technology-neutral access to Digital TV OpenMAX AL is designed as application API - OpenMAX IL is for system integration more flexibility, less portability OpenMAX AL enables key image, camera and video use cases - Allows optimal hardware acceleration with application portability Advanced image capture and photography HD content playback with robust DRM HD video teleconferencing Augmented Reality

15 Copyright Khronos Group, Page 15 OpenSL ES Advanced Audio OpenSL ES does for audio what OpenGL ES does for graphics - Advanced audio functionality from simple playback to 3D audio Object-based native audio API for simplicity and high performance - Reduces development time Same API regardless of underlying implementation - Software or hardware accelerated Cross OS portability - Preserves application investment

16 Copyright Khronos Group, Page 16 OpenSL ES and OpenMAX AL Advanced Audio 3D Audio Video playback Multimedia Audio Effects Advanced MIDI Audio Playback Audio Recording Video recording Radio and RDS Camera Buffer queues Basic MIDI Image capture & display Common Object-oriented API Framework

17 Copyright Khronos Group, Page 17

18 Copyright Khronos Group, Page 18 Visual-based Augmented Reality Camera video stream sent to the compositor Camera images used to track the camera s location and orientation 3D augmentations composited with video stream Camera Tracking Camera-to-scene transform locks the 3D rendering to the real world 3D Augmentation Rendering

19 Copyright Khronos Group, Page 19 EGLStream Video/Graphics Interop OpenMAX AL Media Player is the EGLStream Producer and controls production of frames. EGLStreams enables and hides details of video frame transport. Enables multiple buffering modes for different uses cases eg: FIFO and explicit latch/release OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream Consumer and converts video format into RGB OpenGL ES texture Camera File URL Etc. OpenMAX AL MEDIA PLAYER Object EGLStream? OpenGL ES GL_TEXTURE_EXTERNAL

20 Copyright Khronos Group, Page 20 StreamInput Connects Sensors to Apps Apps request semantic sensor information StreamInput defines list of possible semantic requests Am I in an elevator? Give me gestures and face position Standardized Node Intercommunication Universal Timestamps Input Device Input Device Input Device Advanced Sensors Everywhere Standard cameras, depth cameras motion and position, touch, microphones wireless controllers SHOULD NOT FORCE APPLICATIONS TO ACCESS INDIVIDUAL SENSORS Filter Node Filter Node Filter Node Sensor graph created to provide sensor information StreamInput defines graph creation API and node interconnects Low-level sensor processing encapsulated in nodes unleashes fusion innovation Apps gain magical situational awareness App Apps Need Sophisticated Access to Sensor Data Without coding to specific systems or sensor hardware

21 Copyright Khronos Group, Page 21 Current StreamInput Participants Aiming for specification release in 2012

22 Copyright Khronos Group, Page 22 Khronos Computer Vision Working Group Vision Hardware Acceleration Layer - Enable hardware vendors to implement accelerated imaging and vision algorithms CV HAL* can be used by high-level libraries or applications directly - Primary focus on enabling mobile and embedded systems OpenCV is widely used open source library for vision projects - Future version will leverage CV HAL - * CV Hal is placeholder name OpenCV open source library CV HAL Application Other higher-level CV libraries Open source sample implementation? Hardware vendor implementations

23 Copyright Khronos Group, Page 23 Possible Implementation of Vision Stack Camera input from OpenMAX AL or other camera subsystems Implement StreamInput vision sensor modules with OpenCV Accelerate OpenCV library with CV HAL Functions Use OpenCL to implement CV HAL with parallel execution CV HAL Semantics and fusion of camera and positional sensors High-level computer vision library Accelerated computer vision algorithms Data and event interop with CL / GL / ES for display and compute processing Parallel computation

24 Copyright Khronos Group, Page 24 Augmented Reality Functionality Audio Rendering Positional Sensors Positional and GPS Sensor Data Computer Vision and Tracking CV HAL Synchronization and sensor fusion Position and Tracking Semantics Application on CPU Camera Video TAP to CPU Camera Processing Video stream to GPU EGLStream 3D Rendering and Video Composition Control Camera, Preprocess and generate video streams Much more flexibility than just overlay augmentations over background

25 Copyright Khronos Group, Page 25 Android Native API Adoption OpenGL ES OpenSL ES OpenMAX AL EGL OpenCL StreamInput CV HAL OpenGL ES 2.0 Shipping - Android 2.2 OpenSL ES 1.0 Shipping Android 2.3 OpenMAX AL 1.0 Shipping - Android 4.0 EGL 1.4 Shipping under SDK -> NDK Not yet adopted Working group will ship spec in 2012 Working group will ship spec in 2012

26 Copyright Khronos Group, Page 26 Native APIs on Android Khronos developing open cross vendor standards for key use cases - Khronos APIs have strong momentum in the silicon community Drive adoption - add these APIs to Android NDK - Do not break/change existing Google APIs - Evangelize Google to possibly adopt into standard platform Extended APIs can be used by: - Bundled apps, Market apps with API selection - Multiple APKs behind single multi-apk SKU Evangelize Google and ISVs to adopt enabling APIs Standard Android SDK and NDK Extended NDK for Key use cases E.g. video, camera, imaging, sensor and composition APIs

27 Copyright Khronos Group, Page 27 HTML5 Cross OS App Platform Increasing diversity of devices creates a demand for a true cross OS programming platform BUT need more than more HTML Traditional Web-content Rich Experiential Processing Multi-core CPUs Rich 2D and 3D GPU GPU Computing Multiple HD cameras Image and vision processing Video encode/decode Audio encode/decode Inertial and positional sensors How can the Browser rapidly assimilate such diverse functionality?

28 Copyright Khronos Group, Page 28 Leveraging Native API Investment into HTML5 HTML5 evolving into cross-platform programming platform - Gradually exposing complete system capabilities Opportunity to synergize Web and native APIs development - Leverage native API investments, reduce developer learning cycles Khronos and W3C creating close liaison WebMAX? Camera control and video processing WebAudio Advanced JavaScript Audio WebVL? Vision Processing Device and Sensor APIs Device Orientation Working Groups JavaScript CV HAL Native Native APIs shipping or working group underway JavaScript API shipping or working group underway Possible future JavaScript APIs

29 WebGL Deployment WebGL 1.0 Released at GDC March Mozilla, Apple, Google and Opera working closely with GPU vendors Typed array 1.0 spec ratified by Khronos in May - Supporting bulk data transfer between threads (workers) - Many use cases - background mesh loading, generation, deformation, physics release of WebGL spec and conformance suite imminent - 100% robust stance on security - Fixing bugs in conformance suite - Implementations will report getcontext("webgl") (not experimental) WebGL is not enabled by default in Safari Copyright Khronos Group, Page 29

30 Copyright Khronos Group, Page 30 Why Khronos for WebGL? Unique forum where browser and GPU vendors can cooperate Opened process to enable cooperation with web community GPU Silicon Vendors Browser Vendors Web Community

31 Copyright Khronos Group, Page 31 WebCL Parallel Computing for the Web Khronos launching new WebCL initiative - First announced in March API definition already underway JavaScript binding to OpenCL - Security is top priority Many use cases - Physics engines to complement WebGL - Image and video editing in browser Stay close to the OpenCL standard - Maximum flexibility - Foundation for higher-level middleware

32 Native APIs Copyright Khronos Group, Page 32 Khronos Ecosystem of Standards Web APIs Plugin-free 3D Web Content Web Compute Embedded and Mobile 3D Cross platform desktop 3D Heterogeneous Parallel Computing Graphics Compute Sensors Imaging and vision Unified Sensor and Input Processing CV HAL Hardware accelerated vision functions Camera, Imaging and Streaming Media!!

33 Copyright Khronos Group, Page 33 In Summary APIs are key to enable compelling applications on advanced hardware APIs developed on high-end hardware are now enabling mobile devices APIs no longer exist alone they are interoperating to form platform ecosystems for advanced content Significant cooperation happening between native and Web APIs to bring advanced visual computing to HTML5 Khronos is driving open standards for hardware acceleration Participate, change the industry AND get the inside edge for your products! Connecting Software to Silicon