SC24/WG9 Liaison Meeting

Transcription

1 Copyright Khronos Group, Page 1 SC24/WG9 Liaison Meeting Seoul, November 2011 Neil Trevett Vice President Mobile Content, NVIDIA President, The Khronos Group

2 Copyright Khronos Group, Page 2 Thank you for the invitation to be here! Khronos warmly welcomes the opportunity to discuss liaison opportunities regarding Augmented Reality with SC24/WG9 Topics in this presentation: - Brief introduction to Khronos - Khronos Native API standards relevant to Augmented Reality - Industry Adoption of Khronos APIs - examples in Android and HTML5 - Existing liaisons and possible future liaisons

3 Copyright Khronos Group, Page 3 Khronos - Connecting Software to Silicon Creating open, royalty-free acceleration API standards - Focus on graphics, dynamic media, compute and sensor hardware Low-level - just above raw silicon - Foundation functionality needed on every platform Safe forum for industry cooperation - By the industry for the industry - Open to any company to join - IP framework to protect members and industry Khronos APIs designed to enable healthy implementation-level innovation and competition in the open market APIs enable software developers to turn silicon functionality into rich end user experiences

4 Copyright Khronos Group, Page 4 Apple Over 100 members any company worldwide is welcome to join Board of Promoters

5 Authoring and accessibility Application Acceleration Copyright Khronos Group, Page 5 Khronos Family of Standards StreamInput 3D Digital Asset Exchange format Plugin-free 3D Web Content Web Compute Unified Sensor and Input Processing Cross platform desktop 3D Parallel Computing Embedded and Mobile 3D Vector 2D A coordinated ecosystem of compute, graphics and media standards and APIs Streaming Media Advanced Audio Surface Management Khronos creates royalty-free specifications to meet real market needs and helps drive industry adoption across multiple platforms

6 Copyright Khronos Group, Page 7 Responding to Industry Trends High-end API technology is created on high-end platforms Breakthrough apps embrace mobility s strengths not just treat phones as small PCs - will need complex, interoperating APIs e.g. Augmented Reality Mobile is the new platform for apps innovation. Mobile APIs are needed to unlock hardware potential while conserving battery life As platforms diversify mobile, TV, embedded HTML5 will become increasingly important as a universal app platform

7 AR Key Use Case driving Khronos APIs Many APIs need to work closely together - great goal! Commercial uses of AR-related technologies continuing to evolve but - the technologies needed here will enable amazing apps!! Camera video stream sent to the compositor Camera images used to track the camera s location and orientation 3D augmentations composited with video stream Camera Tracking Camera-to-scene transform locks the 3D rendering to the real world 3D Augmentation Rendering Copyright Khronos Group, Page 8

8 Copyright Khronos Group, Page 9 OpenGL ES 3D Graphics OpenGL for embedded and mobile devices - Eliminates redundant and legacy features - Adds extensions to make it mobile-friendly The dominant 3D API for mobile devices - Widely adopted for STB, DTV, automotive, - Hundreds and hundreds of millions shipped Runs high-end content and engines - UE3, Unity, Unigine, Rage

9 Copyright Khronos 2009 Group, Page 10 OpenCL Heterogeneous Computing Framework for programming diverse parallel computing resources in a system Platform Layer API - Query, select and initialize compute devices Runtime API - Execute compute kernels gather results Kernel Language Specification - Subset of ISO C99 with language extensions OpenCL has Embedded profile - No need for a separate ES spec

10 Copyright Khronos Group, Page 11 OpenMAX AL - Object Oriented Media Object oriented processing of camera, images and video with AV sync - Connect to variety of input and output objects to PLAY and RECORD media Object control interfaces - Sources: Mix control, Seek, Rate, Metadata Extraction, Camera Controls - Sinks: Encode control, Tuning, MIDI, Metadata Insertion Video and image stream routing to other APIs - To CPU and GPU subsystems Sources Analog Radio Data Tap to CPU Sinks EGLStream to ES Camera Audio Mix Audio Input URI DSrc OpenMAX AL Media Object DSnk Display Window URI Memory Memory

11 Copyright Khronos Group, Page 12 Camera Controls for AR AR needs extensive camera controls - OpenMAX AL extensions currently in design Query camera information - Focal length (fx, fy), principal point (cx, cy), skew (s), image resolution (h, w) - Spatial information of how cameras and sensors are placed on device - Calibration and lens distortion ROI extraction - From wide angle and fish-eye lenses Extensive exposure parameters in single or burst mode - Shutter speed, aperture, ISO, white balance, frame rate, focus modes, resolution - Synchronization with other system sensors Data output format control - Grayscale, RGB(A), YUV - Access to the raw data e.g. Bayer pattern

12 Copyright Khronos Group, Page 13 EGLStream Video/Graphics Interop OpenMAX AL Media Player is the EGLStream Producer and controls production of frames. EGLStreams enables and hides details of video frame transport. Enables multiple buffering modes for different uses cases eg: FIFO and explicit latch/release OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream Consumer and converts video format into RGB OpenGL ES texture Camera File URL Etc. OpenMAX AL MEDIA PLAYER Object? EGLStream OpenGL ES GL_TEXTURE_EXTERNAL

13 Copyright Khronos Group, Page 14 OpenSL ES Advanced Audio OpenSL ES does for audio what OpenGL ES does for graphics - Advanced audio functionality from simple playback to 3D spatialized audio Object-based native audio API for simplicity and high performance - Reduces development time Same API regardless of underlying implementation - Software or hardware accelerated Cross OS portability - Preserves application investment

14 Copyright Khronos Group, Page 15 StreamInput Connects Sensors to Apps Apps request semantic sensor information StreamInput defines list of possible semantic requests Am I in an elevator? Give me gestures and face position Standardized Node Intercommunication Universal Timestamps Input Device Input Device Input Device Advanced Sensors Everywhere Standard cameras, depth cameras motion and position, touch, microphones wireless controllers SHOULD NOT FORCE APPLICATIONS TO ACCESS INDIVIDUAL SENSORS Filter Node Filter Node Filter Node Sensor graph created to provide sensor information StreamInput defines graph creation API and node interconnects Low-level sensor processing encapsulated in nodes unleashes fusion innovation Apps gain magical situational awareness App Apps Need Sophisticated Access to Sensor Data Without coding to specific systems or sensor hardware

15 Copyright Khronos Group, Page 16 Current StreamInput Participants Aiming for production implementations in September 2012

16 Copyright Khronos Group, Page 17 OpenCV as Potential Khronos Standard OpenCV is widely use open source project for COMPUTER VISION Khronos Hardware Abstraction Layer - Would enable hardware vendors to provide accelerated imaging and vision modules Being sponsored by NVIDIA, Itseez and Willow Garage - Decision to initiate in few weeks time High-level CV algorithms library Application Hardware Abstraction Layer Open source sample implementation Hardware vendor implementations Can optionally use OpenCL to implement

17 Copyright Khronos Group, Page 18 Separation of Vision Stack Layers StreamInput Semantics and fusion of camera and positional sensors Computer Vision and tracking Parallel computation

18 Copyright Khronos Group, Page 19 Augmented Reality Functionality StreamInput Audio Rendering Positional Sensors Positional and GPS Sensor Data Computer Vision and Tracking Synchronization and sensor fusion OpenCV Position and Tracking Semantics Application on CPU Camera Camera Processing Video TAP to CPU Video stream to GPU EGLStream 3D Rendering and Video Composition Control Camera, Preprocess and generate video streams Much more flexibility than just overlay augmentations over background

19 Android Native API Adoption OpenGL ES OpenSL ES OpenMAX AL EGL OpenGL ES 2.0 Shipping - Android 2.2 OpenSL ES 1.0 Shipping Android 2.3 OpenMAX AL 1.0 Shipping - Android 4.0 EGL 1.4 Shipping under SDK Not yet OpenCL adopted? Working group will StreamInput StreamInput ship spec in 2012 OpenCV Khronos voting to establish WG Copyright Khronos Group, Page 21

20 Copyright Khronos Group, Page 22 Leveraging Native API Investment into HTML5 HTML5 evolving into cross-platform programming platform - Gradually exposing complete system capabilities Opportunity to synergize Web and native APIs development - Leverage native API investments, reduce developer learning cycles Khronos and W3C creating close liaison WebMAX? Camera control and video processing WebAudio Advanced JavaScript Audio HTML and Browser Composition Device and Sensor APIs Device Orientation Working Groups JavaScript StreamInput Native Native APIs shipping or working group underway JavaScript API shipping or working group underway Possible future JavaScript APIs

21 Copyright Khronos Group, Page 24 WebGL and HTML Interaction 3D is not trapped in a rectangular window - 3D can overlay and underlay HTML content - Easy to make 2D HTML HUDs or 3D user interfaces Strong ties with other advanced HTML5 - WebGL can use HTML5 <video> or canvas as a texture Can use 3D for core Web UI as well as content - Advanced transforms and special effects Render HTML DOM sub-tree as texture - Mozilla and Google prototyping as extension - Support user interaction when in 3D

22 WebGL Deployment WebGL 1.0 Released at GDC March Mozilla, Apple, Google and Opera working closely with GPU vendors Typed array 1.0 spec ratified by Khronos in May - Supporting bulk data transfer between threads (workers) - Many use cases - background mesh loading, generation, deformation, physics release of WebGL spec and conformance suite imminent - 100% robust stance on security - Fixing bugs in conformance suite - Implementations will report getcontext("webgl") (not experimental) WebGL is not enabled by default in Safari Copyright Khronos Group, Page 25

23 Copyright Khronos Group, Page 26 WebCL Parallel Computing for the Web Khronos launching new WebCL initiative - First announced in March API definition already underway JavaScript binding to OpenCL - Security is top priority Many use cases - Physics engines to complement WebGL - Image and video editing in browser Stay close to the OpenCL standard - Maximum flexibility - Foundation for higher-level middleware

24 Copyright Khronos Group, Page 27 Augmented Reality in the Web Semantic, synchronized sensor fusion Positional Sensors? StreamInput Positional and GPS Sensor Data Computer Vision and Tracking Synchronization and sensor fusion Position and Tracking Semantics WebAudio? Audio Rendering Application on CPU Camera? Sufficiently sophisticated camera control Camera Processing Video TAP to CPU Control Camera, Preprocess and generate video streams Video stream to GPU EGLStream 3D Rendering and Video Composition? Need to explore whether HTML composition can handle all AR use cases

25 WebGL and Declarative 3D Very synergistic - WebGL can accelerate Declarative 3D Declarative 3D seems to be a major opportunity for X3D community - By re-using existing HTML5 machinery rather than duplicate have a chance to be very widely adopted - Name Declarative 3D = X3D 4.0? Cameras and sensors will need to integrated into HTML5 - Other applications will need access to sensors and cameras - Use Canvas as interop hub Scenegraph Immediate Canvas Cameras Canvas DOM and events Sensors WebGL JavaScript 2D 3D Don t Duplicate any of this Copyright Khronos Group, Page 28

26 Need for 3D Transmission Standard? Approaching chaos in data encodings used in WebGL browsers and apps - We need to enable native decompression in browsers rather than JavaScript Possible key requirements - Full-scene - Geometry, textures, materials, animations, physics etc. - Compression of textures and geometry - Streaming support with LOD flexibility Some formats in use but no widespread consensus - COLLADA, KML, MPEG-4, VRML, JSON, X3D binary, PowerVR POD, GZIP etc. etc. Is the industry ready to work on this? Is the need real? Audio Video Images 3D MP3 H.264 PNG/JPEG? All above defined by MPEG should 3D be an MPEG standard too? Must be royalty free! Leverage MPEG-4 Part 16 AFX, Part 25 a lot of investment but why no momentum? Copyright Khronos Group, Page 30

27 Copyright Khronos Group, Page 31 Relevant Current Khronos Liaisons W3C - HTML5/JavaScript bindings: WebGL, WebCL, sensors, video and camera - Browser Acceleration: CSS shaders, composition, audio, SVG, Canvas Web3D - Liaison agreement regarding X3D and COLLADA - Declarative 3D community group at W3C - 3D scene graph in the DOM - WebGL ISO/IEC JTC1 SC4/TC184-3D Visualization Specifications - Copyright License for COLLADA to ISO Executed OMA and OGC - Augmented reality cooperation - complementary activities

28 Copyright Khronos Group, Page 32 Summary Khronos is working hard to create hardware acceleration standards that are relevant to Augmented Reality and promoting their industry adoption - Khronos is focused on one very small piece of the total AR picture how applications on various platforms access acceleration and sensor hardware Khronos welcomes liaison opportunities - To assist in leveraging Khronos standards in larger ecosystems - To cooperate to ensure Khronos APIs interoperate with other standards initiatives - To help identify and help solve issues and gaps in the AR ecosystem Possible areas for cooperation - ISO Standardization of OpenGL ES and other Khronos APIs? - Transmission standard at ISO? - Cooperative camera and sensor integration into Web for AR? - QUESTION cooperate over X3D or Declarative 3D or both?