Bringing the Power of the GPU to the Web

Transcription

1 Copyright Khronos Group Page 1 Bringing the Power of the GPU to the Web Neil Trevett Vice President NVIDIA President Khronos

2 Mobile is the New Epicenter of Innovation Copyright Khronos Group Page 2

3 Copyright Khronos Group Page 3 Khronos Standards Visual Computing - Object and Terrain Visualization - Advanced scene construction 3D Asset Handling - Advanced Authoring pipelines - 3D Asset Transmission Format with streaming and compression Camera Control API Sensor Processing - Mobile Vision Acceleration - On-device Sensor Fusion Over 100 companies defining royalty-free APIs to connect software to silicon Acceleration in the Browser - WebGL for 3D in browsers - WebCL Heterogeneous Computing for the web

4 Copyright Khronos Group Page 4 Mobile Web is a Real Time Application 1024x K Pixels 132 DPI 320x K Pixels 163 DPI Apple iphone Apple ipad 2048x K Pixels 326 DPI + = Apple ipad Mini Buttery smooth touch interaction needs continuous 60Hz updates In 5 years the number of pixels to process on mobile screens has gone up by factor of TWENTY Need GPU Acceleration for everything Web!

5 Copyright Khronos Group Page 5 How are GPUs Accessible to the Web? Hardware composition - Within the browser stack under the hood Vector Acceleration for SVG - Using NVIDIA OpenGL extensions 3D Developer Functionality - OpenGL ES functionality through JavaScript Compute Acceleration - Offloading compute intensive code to GPU Compression and streaming of 3D assets - For network transmission Camera, vision and sensor processing - Future JavaScript bindings to native APIs?

6 Copyright Khronos Group Page 6 Mobile OS Adoption of Khronos APIs OpenGL ES 2.0 Shipping - Android 2.2 OpenSL ES 1.0 (subset) Shipping Android 2.3 OpenMAX AL 1.0 (subset) Shipping - Android 4.0 EGL 1.4 Shipping under SDK -> NDK Opera and Firefox WebGL now Chrome soon OpenGL 3.2 on MacOS OpenCL 1.2 on MacOS OpenGL ES 3.0 on ios Can enable on MacOS Safari ios5 enables WebGL for iads

7 Copyright Khronos Group Page 7 WebGL 3D on the Web No Plug-in! Leveraging HTML 5 and <canvas> element - WebGL defines JavaScript binding to OpenGL ES Enables a 3D context for the canvas Low-level foundational Web API for accessing the GPU - Flexibility and direct GPU access - Enables higher-level frameworks and middleware Availability of OpenGL and OpenGL ES on almost every web-capable device JavaScript binding to OpenGL ES 2.0 Increasing JavaScript performance. HTML 5 Canvas Tag

8 Copyright Khronos Group Page 8 WebGL Implementation Anatomy Content downloaded from the Web. Middleware can make WebGL accessible to non-expert 3D programmers Content JavaScript, HTML, CSS,... JavaScript Middleware Much WebGL content uses three.js library: Browser provides WebGL functionality alongside other HTML5 technologies - no plug-in required JavaScript HTML5 CSS OS Provided Drivers. WebGL on Windows can use Direct3D - for example Angle open source project creates OpenGL ES 2.0 over DX9 OpenGL ES 2.0 OpenGL DX9/Angle

9 WebGL Availability in Browsers - Microsoft where you have IE11, you have WebGL turned on by default and working all the time - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on ios for iads - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL Copyright Khronos Group Page 9

10 Copyright Khronos Group Page 10 Cross-OS Portability HTML/CSS HTML/CSS HTML/CSS HTML5 provides cross platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems SDK Dalvik (Java) Objective C C# Preferred development environments not designed for portability C/C++ DirectX Native code is portablebut apps must cope with different available APIs and libraries

11 Copyright Khronos Group Page 11 WebGL First Wave Application Categories Maps and Navigation Modeling Tools and Repositories Games 3D Printing Visualization Music Videos and Promotion Education Photo Editors Music Visualizers Vision/Video Processing

12 Copyright Khronos Group Page 12 Google Maps All rendering (2D and 3D) in Google Maps uses WebGL

13 Copyright Khronos Group Page 13 Microsoft PhotoSynth2 Demonstrated at Build :50

14 WebGL on Logan Android Tablet Copyright Khronos Group Page 14

15 WebGL on Logan Android Tablet Copyright Khronos Group Page 15

16 Copyright Khronos Group Page 16 OpenGL 3D API Family Tree WebGL 2.0 is in development now - will bring OpenGL ES 3.0 functionality to the Web Fixed function 3D Pipeline OpenGL ES 1.1 Content Programmable vertex and fragment shaders OpenGL ES 2.0 Content ES3 is backward compatible so new features can be added incrementally OpenGL ES 3.0 Content Mobile 3D WebGL 1.0 WebGL 2.0 OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 OpenGL ES 1.0 ES-Next OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 3.0 OpenGL 3.2 OpenGL 4.0 OpenGL 4.1 OpenGL 4.4 GL-Next Desktop 3D OpenGL 4.4 is a superset of DX

17 Copyright Khronos Group Page 17 OpenGL ES 3.0 Highlights Better looking, faster performing games and apps at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability Backward compatible with OpenGL ES OpenGL ES 2.0 apps continue to run unmodified Standardized Texture Compression - #1 developer request!

18 Copyright Khronos Group Page 18 Why Khronos for WebGL? Hardware API standards must take into account silicon design cycles - Multi-year pipeline of APIs that affect chips that take $100Ms to execute - Deep insights into silicon and driver architectures - Rigorous conformance tests and infrastructure Khronos is committed to being a good citizen in the larger Web community - Opened Khronos WebGL processes to enable cooperation with web community Khronos is the industry forum to drive hardware consensus and cooperation - Help create foundational support for higher-level Web standards that access hardware capabilities

19 Copyright Khronos Group Page 19 Leveraging Proven Native APIs into HTML5 Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve HTML Canvas WebVX? Vision Processing WebStream? Sensor Fusion WebCAM(!) Camera control and video processing JavaScript Path Rendering Camera Control Native Native APIs shipping or Khronos working group JavaScript API shipping, acceleration being developed or work underway Possible future JavaScript APIs or acceleration

20 Copyright Khronos Group Page 20 OpenCL as Parallel Compute Foundation C++ AMP Shevlin Park Uses Clang and LLVM OpenCL HLM C++ syntax/compiler extensions WebCL JavaScript binding to OpenCL for initiation of OpenCL C kernels Aparapi Java language extensions for parallelism River Trail Language extensions to JavaScript PyOpenCL Python wrapper around OpenCL Harlan High level language for GPU programming Compiler directives for Fortran C and C++ OpenCL provides vendor optimized, cross-platform, cross-vendor access to heterogeneous compute resources

21 Copyright Khronos Group Page 21 WebCL WebCL is a JavaScript binding to OpenCL APIs - Enables initiation of Kernels written in OpenCL C within the browser - Requires a conformant underlying OpenCL on the host system Leverage heterogeneous computing resources - 3D asset codecs, video codecs and processing, imaging and vision processing - Physics for WebGL games, Online data visualization, Augmented Reality WebCL 1.0 based on OpenCL 1.1 Embedded Profile: - Implementations may utilize OpenCL 1.1 or 1.2 WebCL API is designed for complete security - Restriction of some OpenCL native functionality - WebCL kernel validation similar to WebGL

22 Copyright Khronos Group Page 22 WebCL 1.0 Kernels HTML data interoperability - <canvas>, <image>, ImageData sources bindable to WebCLBuffer & WebCLImage - <video> tag can be bound to a WebCLImage Interoperability between WebCL and WebGL - Through GL_SHARING extension WebCL may support the following extensions - KHR_FP16 16-bit float support in kernels - KHR_FP64 64-bit float support in kernels No 3D image support in WebCL May change in future WebCL versions

23 WebCL 1.0 Security Leverages OpenCL 1.2 robustness/security extensions - Context Termination - to prevent DOS from long running kernels - Memory Initialization - so no leakage from out of bounds memory access Kernels passed through open source WebCL Kernel Validator Initializes local and private memory if underlying OpenCL implementation does not implement memory initialization extension - Keeps track of memory allocations and traces valid ranges for reads and writes API/Language Restrictions and definition of undefined OpenCL behavior - Kernels do not support structures as arguments - Kernels name must be less than 256 characters - Mapping of CL memory objects into host memory space is not supported - Binary kernels are not supported - Some OpenCL Extension may not be supported or require translation - Certain OpenCL parameters may not directly carry over to WebCL Copyright Khronos Group Page 23

24 Copyright Khronos Group Page 24 WebCL 1.0 Current Status WebCL 1.0 API definition is being publicly developed - Working Public Draft first released April 2012: WebCL distribution lists - public_webcl@khronos.org, webcl@khronos.org WebCL1.0 specification finalization expected in 1H With conformance tests and utilities - Samsung contributed tests, working group reviewed WebCL Conformance Framework and Test Suite (WiP) - Full API coverage and Input/output validation - Available on GitHub:

25 OpenCL to WebCL Translator Utility OpenCL to WebCL Kernel Translator - Input: An OpenCL kernel - Output: WebCL kernel, and a log file, that details the translation process - Tracked by a meta bug on Khronos public Bugzilla - Host API translation (WiP) - Input: an OpenCL host API calls - Output: WebCL host API calls to be wrapped in JS - Provides verbose translation log file, detailing the process and any constraints - Tracked by a meta bug on Khronos public Bugzilla: Copyright Khronos Group Page 25

26 Copyright Khronos Group Page 26 WebCL Prototype Implementations Nokia - Firefox build with integrated WebCL - Firefox extension, open sourced May 2011 (Mozilla Public License 2.0) - Samsung - uses WebKit, open sourced June 2011 (BSD) - Motorola Mobility - uses Node.js, open sourced April 2012 (BSD) - Based on Apple QJulia Based on Iñigo Quilez, Shader Toy Based on Iñigo Quilez, Shader Toy

27 Copyright Khronos Group Page 27 WebCL Parallel Computing for Web Acceleration

28 Khronos APIs for Augmented Reality W3C Augmented Web Community Group discussing many of these issues for the Web: e.g. leveraging WebRTC in the short term Audio Rendering MEMS Sensors Sensor Fusion Application on CPUs, GPUs and DSPs Precision timestamps on all sensor samples Vision Processing Advanced Camera Control and stream generation Camera Control API EGLStream - stream data between APIs 3D Rendering and Video Composition On GPU AR needs not just advanced sensor processing, vision acceleration, computation and rendering - but also for all these subsystems to work efficiently together Copyright Khronos Group Page 28

29 3D Needs a Transmission Format! Compression and streaming of 3D assets becoming essential - Mobile and connected devices need access to increasingly large asset databases 3D is the last media type to define a compressed format - 3D is more complex diverse asset types and use cases Needs to be royalty-free - Avoid an internet video codec war scenario Eventually enable hardware implementations of successful codecs - High-performance and low power but pragmatic adoption strategy is key Audio Video Images 3D MP3 H.264 JPEG?! An effective and widely adopted codec ignites previously unimagined opportunities for a media type Copyright Khronos Group Page 29

30 Copyright Khronos Group Page 30 gltf OpenGL Transmission Format Binary file format for efficient transmission for 3D assets - Reduce network bandwidth and minimize client processing overhead Run-time neutral - DO NOT IMPLY OR MANDATE ANY RUN-TIME BEHAVIOR - Can be used by any app or run-time usually WebGL accelerated Scalable to handle compression and streaming - Though baseline format does not include compression Direct load efficiency for WebGL - Little or NO processing to drop gltf data into WebGL client Carry conditioned data from any authoring format - Prototyping and optimizing efficient handling of COLLADA assets Authoring Playback A standards-based content pipeline for rich native and Web 3D applications

31 Copyright Khronos Group Page 31 COLLADA and gltf Open Source Ecosystem OpenCOLLADA Importer/Exporter and COLLADA Conformance Tests On GitHUB Tool Interop COLLADA2GLTF Translator Other authoring formats Web-based Tools Pervasive WebGL deployment Three.js gltf Importer. Rest3D initiative

32 Copyright Khronos Group Page 32 WebGL as Test-bed for 3D Asset Compression Integrating and benchmarking 3D geometry compression formats with gltf - Baseline is GZIP Scalable Complexity 3D Mesh Compression codec MPEG-SC3DMC - Royalty-free graphics compression technology from MPEG (MIT License) - Open3DGC is efficient JavaScript and C/C++ implementation - Convertor using Open3DGC to compress 3D Meshes, Skinning, Animations - WebGL-loader is Google lightweight compression for WebGL content OpenCTM uses LZMA compression

33 Size (MBytes) Copyright Khronos Group Page 33 Initial Compression Results Compression Efficiency - Gzip (default level=6) - OpenCTM (default settings) - Open3DGC and Webgl-loader - Positions on 14 bits - Normals and texcoords on 10 bits Gzip OpenCTM Webgl-loader + Gzip Open3DGC-ASCII + Gzip Open3DGC-Binary 0 CAD (3748 models) 3D Scanned (78 models) MPEG dataset (1211 models) Open3DGC is 5x-9x more efficient than Gzip 1.3x-2.4x more efficient than OpenCTM and 1.2x-1.5x more efficient than webgl-loader

34 Copyright Khronos Group Page 34 3DGC Decode Times Javascript Decoding Speed - Desktop machine - Windows 64-bit, 8GB RAM, Chrome - AMD Phenom II X4 B95 3.0GHz - Smart phone - Samsung Galaxy S4 - Android Chrome Number of triangles Desktop decoding time (ms) Smart phone decoding time (ms) Hand 100K Dilo 54K Octopus 34K Decoding speed will become even more critical with dense 3D meshes generated by 3D digitization technologies (e.g. 3D scanners) 3D Codec can be accelerated by WebCL Kernels or (eventually) hardware

35 Quality Copyright Khronos Group Page 35 Texture Compression is Key Texture compression saves precious resources - Network bandwidth, device memory space AND device memory bandwidth Developers need the same texture compression EVERYWHERE - Otherwise portable apps such as WebGL need multiple copies of same texture NOT Royalty-free. Platform Fragmentation DXTC/S3TC Windows PVRTC ios Royalty-free BUT only optional in ES. Only 4bpp 3 channel No alpha support ETC1 Mandated in Android Froyo (400M devices) ETC2 / EAC MANDATED in OpenGL ES 3.0 OpenGL 4.3 Royalty-free Backward compatible with ETC1 ETC2: 4bpp 3 channel EAC: 4 (8) bpp 1(2) channel COMBINED: RGBA 8bpp 4 channel Does not have 1-2 bit compression WITH ALPHA ASTC OpenGL ES 3.0 and OpenGL 4.3 extensions -> Core once proven Royalty-free Best quality. Independent control of bit-rate and # channels 1 to 4 channel 1-8bpp in fine steps Pervasive Deployment >

36 ASTC Universal Texture Standard Adaptive Scalable Texture Compression (ASTC) - Quality significantly exceeds S3TC or PVRTC at same bit rate Industry-leading orthogonal compression rate and format flexibility - 1 to 4 color components: R / RG / RGB / RGBA - Choice of bit rate: from 8bpp to <1bpp in fine steps ASTC is royalty-free and so is available to be universally adopted - Shipping as OpenGL/OpenGL ES extension today for industry feedback Original 24bpp ASTC Compression 8bpp 3.56bpp 2bpp Copyright Khronos Group Page 36

37 Copyright Khronos Group Page 37 Path Rendering Acceleration Offload the CPU so the application can run as fast as possible - Make maximum use of the GPU for best performance and power CPU creates paths CPU creates paths CPU creates paths CPU renders paths CPU tessellates paths into polygons CPU Use standard 3D commands to process polygons Define new OpenGL path commands to process paths directly GPU - Software Scanline renderers can be high quality and portable - CPU has to process complete pipeline stealing cycles from the application - Software rendering limits performance - Tessellation loads the CPU stealing cycles from the application so perf sometimes slower than software alone - Tessellation consumes a lot of data and memory bandwidth = power - Quality can be compromised due to tessellation accuracy - Maximum CPU offload - Compact data format sent to GPU renderer - GPU provides excellent performance and power - GPU can increase quality and functionality

38 Copyright Khronos Group Page 38 NV_path_rendering OpenGL Extension Brings Path processing directly to OpenGL - No tessellation necessary Goals - Functionally complete for key standards: SVG, Canvas, PostScript etc. - Much faster often 4x to 100x faster than CPUs - Enhanced quality can avoid approximations needed by CPU renderers - Lower power by leveraging dedicated hardware - New functionality e.g. mix 2D paths with 3D and programmable shading

39 Copyright Khronos Group Page 39 Stencil then Cover Approach Create a path object and pass directly to the GPU - Cubic & quadratic Bezier segments, line segments, partial elliptical arcs GPU Stencils the path object into the stencil buffer - GPU provides massively parallel stenciling of filled or stroked paths - Calculate winding rule or containment at every sub-pixel sample in parallel Cover the path object and stencil test against its coverage - Test against path coverage determined in the 1st step and shade the path Uses GPU MSAA anti-aliasing - 8 or 16 samples/pixel gives good quality Step 1 Stencil Step 2: Cover repeat

40 Enhanced Quality on GPU color bleeding conflation artifacts on CPU regular grid jitter pattern on CPU - sub-optimal Antialiasing GPU Offers Jittered Sampling for Free on GPU for better Antialiasing Conflation free on GPU Eliminate Conflation Artifacts Multiple color AND stencil samples per pixel weird big holes Skia feathers? Cairo NV_path_rendering Stroking approximations avoided by GPU GPUs great at texturing: Mip-mapping Anisotropic filtering Wrap modes GPU Qt Moiré artifacts Similar for Qt & Skia Proper gradient filtering on GPU Cairo Copyright Khronos Group Page 40

41 Comparing Performance Copyright Khronos Group Page 41

42 Copyright Khronos Group Page 42 New GPU Functionality Projective Transformation Fast Arbitrary Path Clipping light source position for BUMP Mapping linear RGB transition between saturated red and saturated blue has dark purple region Programmable Shading Paint in GLSL for filter and blending acceleration srgb perceptually smooth transition from saturated red to saturated blue Fully srgb Correct Rendering Mixing depth tested Text, 3D, and Paths

43 Mixing 2D and 3D Copyright Khronos Group Page 43

44 Copyright Khronos Group Page 44 Standardization and Adoption Pipeline NVIDIA is proposing nvpr to OpenGL working group at Khronos to create open, royalty-free cross platform foundation for vector graphics acceleration Initial functionality proposal. Prove concepts. Solicit industry feedback Pervasive multi-vendor availability. Widespread application usage inspires silicon optimizations Desktop and mobile displays typically >300 DPI Vendor Extension to OpenGL OpenGL Extension or Core OpenGL vector acceleration adopted into OpenGL and OpenGL ES Vector acceleration pervasive on desktop and mobile nvpr is here! Mobile silicon is CUDA/OpenCL capable

45 Path Rendering Acceleration on Android Tablet Copyright Khronos Group Page 45

46 Summary Open standards such as WebGL and WebCL are enabling web applications to reach the power of the GPU through JavaScript GPU acceleration will soon become vital for Web applications wanting to leverage advanced use of camera and sensors Direct acceleration of path primitives directly on GPUs will drive browser smooth touch performance for new classes of applications and devices Work starting on 3D asset streaming and compression standards to enable 3D as a social media type on the web The Web and hardware community have significant opportunity to leverage each others efforts for the benefit of the industry Khronos is committed to enable the hardware community to be a good citizen in creating the next generation of accelerated web standards ntrevett@nvidia.com Copyright Khronos Group Page 46