AR Standards Update Austin, March PDF Free Download

AR Standards Update Austin, March 2012 Neil Trevett President, The Khronos Group Vice President Mobile Content, NVIDIA Copyright Khronos Group, 2012 - Page 1

Topics Very brief overview of Khronos Update on vision, sensor and AR-related Khronos standards Update on Web-oriented standards Discussion on COLLADA, WebGL and need for 3D Transmission Standard Copyright Khronos Group, 2012 - Page 2

Khronos Connects Software to Silicon Khronos creates OPEN, ROYALTY- FREE STANDARD APIs for advanced hardware acceleration Focus on graphics, video, audio, compute, visual and sensor processing Khronos is OPEN for any company to join and participate APIs enable software developers to turn silicon functionality into rich end user experiences Acceleration APIs BY the Industry FOR the Industry Copyright Khronos Group, 2012 - Page 3

Khronos History Over 100 Members 15 Active Standards Plugin-free 3D Web Content Web Compute Cross platform desktop 3D Parallel Computing Unified Sensor and Input Processing Camera, Images and Streaming Media 3D Digital Asset Exchange format Advanced Audio OpenVL Hardware accelerated vision functions Embedded and Mobile 3D Surface Management Vector 2D 2000 2003 2004 2005 2006 2008 2009 2011 Copyright Khronos Group, 2012 - Page 4

Apple Over 100 members any company worldwide is welcome to join Board of Promoters Copyright Khronos Group, 2012 - Page 5

Why are Khronos Standards Key? Khronos APIs define and expand core mobile device functionality - Low-level Foundation functionality needed on every platform Khronos standards have strong industry momentum - Shipping on 100s of million devices on multiple OS They are FREE - No royalties Tianyu ZTE HTC Khronos standards drive mobile market growth by expanding device capabilities Copyright Khronos Group, 2012 - Page 6

Khronos API Ecosystem Evolution DESKTOP MOBILE INTEROP, VISION AND SENSORS WEB New API technology first evolves on high-end platforms Mobile is the new platform for apps innovation. Mobile APIs unlock hardware and conserve battery life Apps embrace mobility s unique strengths and need complex, interoperating APIs with rich sensory inputs e.g. Augmented Reality Diverse platforms mobile, TV, embedded means HTML5 will become increasingly important as a universal app platform Copyright Khronos Group, 2012 - Page 7

Sensor Fusion with App Portability Apps request semantic sensor information StreamInput defines list of possible semantic requests Am I in an elevator? Give me gestures and face position Advanced Sensors Everywhere Standard cameras, depth cameras motion and position, touch, microphones wireless controllers Processing graph provides sensor data stream Apps gain magical situational awareness Utilizes optimized, sensor smart middleware Apps Need Sophisticated Access to Sensor Data Without coding to specific systems or sensor hardware Copyright Khronos Group, 2012 - Page 8

Current StreamInput Participants Aiming for specification release in 2012 Copyright Khronos Group, 2012 - Page 9

StreamInput Directions Main focus on handling camera, depth cameras and motion sensors - Extensible to any sensor type Middleware constructs a graph to process sensors - Building on proven SDKs Key design concerns - Cross sensor synchronization; Power efficiency; - Dynamic sensor configuration; Security GPS Detects when indoors Gyro Accelerometer Compass Use accelerometer to know when to turn on gyro for power efficiency Pedometer Measurement RFID Fix WiFi Fix Position Stream Application developer doesn t need to know what sensors are used or to be a sensor fusion expert -> technology and device PORTABILITY Application Provide position and orientation at best rate available Copyright Khronos Group, 2012 - Page 10

OpenVL Vision Hardware Acceleration Layer - Enable hardware vendors to implement accelerated imaging and vision algorithms OpenVL can be used by high-level libraries or applications directly - Primary focus on enabling mobile and embedded systems OpenCV is widely used open source library for vision projects - Future version will leverage OpenVL OpenCV open source library OpenVL Application Other higher-level CV libraries Working group just starting Get Involved Now! Open source sample implementation? Hardware vendor implementations Copyright Khronos Group, 2012 - Page 11

Possible Implementation of Vision Stack Camera input from OpenMAX AL or other camera subsystems Implement StreamInput vision sensor modules with OpenCV Accelerate OpenCV library with OpenVL Functions Use OpenCL to implement OpenVL with parallel execution OpenVL Semantics and fusion of camera and positional sensors High-level computer vision library Accelerated computer vision algorithms Data and event interop with CL / GL / ES for display and compute processing Parallel computation Copyright Khronos Group, 2012 - Page 12

Augmented Reality Functionality Audio Rendering Positional Sensors Positional and GPS Sensor Data Computer Vision and Tracking Synchronization and sensor fusion OpenVL Position and Tracking Semantics Application on CPU Camera Camera Processing Video TAP to CPU Video stream to GPU EGLStream 3D Rendering and Video Composition Control Camera, Preprocess and generate video streams Copyright Khronos Group, 2012 - Page 13

HTML5 Cross OS App Platform Increasing diversity of devices creates a demand for a true cross OS programming platform BUT need more than more HTML Traditional Web-content Rich Experiential Processing Multi-core CPUs Rich 2D and 3D GPU GPU Computing Multiple HD cameras Image and vision processing Video encode/decode Audio encode/decode Inertial and positional sensors How can the Browser rapidly assimilate such diverse functionality? Copyright Khronos Group, 2012 - Page 14

Leveraging Native API Investment into HTML5 HTML5 evolving into cross-platform programming platform - Gradually exposing complete system capabilities Opportunity to synergize Web and native APIs development - Leverage native API investments, reduce developer learning cycles Khronos and W3C creating close liaison WebMAX? Camera control and video processing WebAudio Advanced JavaScript Audio Canvas WebVL? Vision Processing Device and Sensor APIs Device Orientation Working Groups JavaScript OpenVL Native Native APIs shipping or working group underway JavaScript API shipping or working group underway Possible future JavaScript APIs Copyright Khronos Group, 2012 - Page 15

Good COLLADA Momentum Apple - Natively supports COLLADA in MacOS Lion and ibooks Google - Uses COLLADA in Google Earth and SketchUp Adobe - Imports COLLADA directly in Creative Suite AutomationML - And the CAD industry are investing in COLLADA 1.5 ISO - COLLADA is in process to become a ISO standard for the CAD industry - Key for long-term archival use cases Extensive COLLADA conformance tests freely available from Khronos: - http://www.khronos.org/conformance/implementers/collada/ Copyright Khronos Group, 2012 - Page 16

But! Import/Export Problems! Import/export capability is not reliable - Particularly with 3ds Max, Maya and Blender Significantly diminishes COLLADA s value in enabling authoring pipelines - Many folks using FBX instead OpenCOLLADA is an open source project that has the potential provide widely available reliable import/export - BUT the open source project momentum has stalled The Khronos COLLADA working group has open discussions over the last month or two and has some suggestions But we need your feedback! Copyright Khronos Group, 2012 - Page 17

Proposed COLLADA Actions for Discussion 1. We need much closer Consortium/Community cooperation - Create discussion email list open to all - no Khronos membership needed 2. We need OpenCOLLADA to provide reliable import/export - Khronos fund fixing all major known bugs fix - Find a moderator(s) from the community to keep submissions on track 3. We need an easier way to measure and track conformance - Proposal Khronos will fund a test and easy reporting framework for the conformance tests anyone can test import/export - What gets measured improves! 4. Most content creators do not use all of COLLADA - Agree on a common-used subset for easier and more reliable conformance - Ton Roosendaal suggests Basic model badge + Basic character badge Copyright Khronos Group, 2012 - Page 18

WebGL, COLLADA and 3D Transmission Strong potential COLLADA synergy for WebGL authoring - COLLADA import into common WebGL frameworks such as three.js? COLLADA is NOT an efficient transmission format - Lots of discussion around use of JSON for transmitting WebGL content - BUT - still need binary blobs for textures, geometry, animation, physics Key 3D Binary Data Blob Requirements Discussion - Full-scene: Geometry, textures, materials, animations, physics etc. - Domain specific compression with streaming and LOD flexibility - Separation of data from scripting ignite app innovation - ROYALTY FREE Audio Video Images 3D MP3 H.264 PNG/JPEG?? Copyright Khronos Group, 2012 - Page 19

Khronos ASTC Texture Compression Lack of consistent, cross-platform, texture compression format with 1-2bit per pixel compression and alpha has been major hurdle for content development and easy deployment Adaptive Scalable Texture Compression - 1-8 bit per pixel compression WITH ALPHA - Great quality Competitive refinement at working group - NVIDIA and ARM (with input from AMD) separately evolved ZIL and ASTC proposals ASTC from ARM selected Developing as OpenGL ES Extension - HDR version comes later Copyright Khronos Group, 2012 - Page 20

Requested Feedback Any APIs missing for real-time AR acceleration? Right direction for COLLADA momentum? How can Khronos help with transmission format? Copyright Khronos Group, 2012 - Page 22