The Mobile Future of extended Reality (XR) Hugo Swart Senior Director, Product Management Qualcomm Technologies, Inc.

Size: px

Start display at page:

Download "The Mobile Future of extended Reality (XR) Hugo Swart Senior Director, Product Management Qualcomm Technologies, Inc."

Dorcas Burns
5 years ago
Views:

1 The Mobile Future of extended Reality (XR) Hugo Swart Senior Director, Product Management Qualcomm Technologies, Inc. June 29, 2017

2 XR is the future Industrial & manufacturing Engineering Healthcare M Education RMilitary VR XR AR Retail Marketing & advertising Emergency response Entertainment 2

3 Will the smartphone become an XR headset? 3

XR is here today, but it is still in its infancy Analogy to smartphones: XR evolution will take years opportunity will be immense Technology Phase: Infancy Market: Mostly early adopter Prosumers

4 XR is here today, but it is still in its infancy Analogy to smartphones: XR evolution will take years opportunity will be immense Technology Phase: Infancy Market: Mostly early adopter Prosumers Technology Phase: Rapid evolution Market: Surging consumer adoption XR is here today XR by ~2020 Technology Phase: Maturity Market: Worldwide, ubiquitous use XR will follow a similar ~30 year cycle of sleeker designs, with tremendously increasing functionality 4

5 5

6 6

Power and thermal efficiency is essential for XR The XR headset needs to be appropriate to wear and use all day The challenge of XR workloads Very compute intensive Complex concurrencies Always-on

7 Power and thermal efficiency is essential for XR The XR headset needs to be appropriate to wear and use all day The challenge of XR workloads Very compute intensive Complex concurrencies Always-on Real-time Constrained mobile, wearable environment Must be thermally efficient for sleek, ultra-light designs Requires long battery life for all-day use Able to be quickly recharged at least 1,000 times 7

Immersive virtual reality has extreme requirements Achieving full immersion at low power to enable a comfortable, sleek form factor Extreme pixel quantity and quality Screen is very close to the eyes

audio Realistic 3D, positional, surround audio that is accurate to the real world Crystal clear voice clear voice that is enhanced with noise cancellation technology Minimal latency Minimized system

8 Immersive virtual reality has extreme requirements Achieving full immersion at low power to enable a comfortable, sleek form factor Extreme pixel quantity and quality Screen is very close to the eyes High resolution audio Up to human hearing capabilities Spherical view Look anywhere with a full 360 spherical view Stereoscopic display Humans see in 3D Visual quality Immersion Sound quality 3D audio Realistic 3D, positional, surround audio that is accurate to the real world Crystal clear voice clear voice that is enhanced with noise cancellation technology Minimal latency Minimized system latency to remove perceptible lag Intuitive interactions Precise motion tracking Accurate on-device motion tracking Natural user interfaces Seamlessly interact with VR using natural movements, free from wires 8

Immersive visuals with Snapdragon 835 25% faster

9 Immersive visuals with Snapdragon % faster graphics rendering Up to 60x more display colors 9

A heterogeneous computing approach for immersive visuals Efficient processing by running the appropriate task on the appropriate engine Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540

10 A heterogeneous computing approach for immersive visuals Efficient processing by running the appropriate task on the appropriate engine Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540 Graphics Processing Unit (GPU) Immersive visuals GPU Efficient rendering of advanced 3D visuals for DX12, OpenGL ES & Vulkan applications DPU 10-bit display, Q-Sync, and wide color gamut VPU 4K HEVC 10-bit playback, foveated video support Wi-Fi Hexagon DSP HVX Qualcomm Aqstic Qualcomm All-Ways Aware IZat Location Display Processing Unit (DPU) Video Processing Unit (VPU) Qualcomm Spectra 180 Kryo 280 CPU Qualcomm Haven * Not to scale Qualcomm Adreno, Qualcomm Hexagon, Qualcomm All-Ways Aware, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Kryo, Qualcomm IZat, and Qualcomm Haven are products of Qualcomm Technologies, Inc. 10

11 Snapdragon 835 immersive audio experiences 11

A heterogeneous computing approach for immersive audio Efficient processing by running the appropriate task on the appropriate engine Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540

12 A heterogeneous computing approach for immersive audio Efficient processing by running the appropriate task on the appropriate engine Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540 Graphics Processing Unit (GPU) Heightened sounds Support for object and scene-based audio Hi-Fi grade DSD format and SNR & THD+N Crystal Clear voice Wi-Fi Hexagon DSP HVX Qualcomm Aqstic Qualcomm All-Ways Aware Display Processing Unit (DPU) Video Processing Unit (VPU) Qualcomm Spectra 180 Kryo 280 CPU IZat Location Qualcomm Haven * Not to scale Qualcomm Adreno, Qualcomm Hexagon, Qualcomm All-Ways Aware, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Kryo, Qualcomm IZat, and Qualcomm Haven are products of Qualcomm Technologies, Inc. 12

13 Computer vision for XR 6 DoF head tracking Eye tracking Hand tracking Object detection and recognition 13

A heterogeneous computing approach for immersive interactions Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540 Graphics Processing Unit (GPU) Immersive Interactions Wi-Fi Display

14 A heterogeneous computing approach for immersive interactions Qualcomm Snapdragon 835 Snapdragon X16 LTE modem Adreno 540 Graphics Processing Unit (GPU) Immersive Interactions Wi-Fi Display Processing Unit (DPU) Video Processing Unit (VPU) VIO for precise, low latency 6-DOF Computer vision for gesture recognition and eye tracking Machine learning for eye movement prediction and gesture recognition Hexagon DSP is a crucial differentiator for real-time and low power experiences Hexagon DSP HVX Qualcomm Aqstic Qualcomm All-Ways Aware IZat Location Qualcomm Spectra 180 Kryo 280 CPU Qualcomm Haven * Not to scale Qualcomm Adreno, Qualcomm Hexagon, Qualcomm All-Ways Aware, Qualcomm Spectra, Qualcomm Aqstic, Qualcomm Kryo, Qualcomm IZat, and Qualcomm Haven are products of Qualcomm Technologies, Inc. 14

15 AR has additional requirements beyond immersion Providing an always-on experience that intelligently enhances our lives Immersive The visuals, sounds, and interactions are so realistic that they are true to life Cognitive It understands the real world, learns personal preferences, and provides security & privacy Connected An always-on, low power wearable with fast wireless cloud connectivity anywhere 15

Significant progress in VR and AR this year Improving

Snapdragon VR SDK HMD Accelerator Program Ecosystem

AR Easy developer access to Snapdragon accelerated VR

Commercialize VR HMDs quickly with fewer resource

16 Significant progress in VR and AR this year Improving experiences with Snapdragon 835 Snapdragon 835 Snapdragon VR SDK HMD Accelerator Program Ecosystem support Purpose built silicon for superior mobile VR & AR Easy developer access to Snapdragon accelerated VR libraries that simplify application development Commercialize VR HMDs quickly with fewer resource restraints Collaboration with multiple content, technology, and platform companies 16

17 Snapdragon 835 processor is taking us closer to the vision Designed to meet the VR processing demands within the thermal and power constraints Extreme 25% Faster pixel Graphics quantity and Rendering quality Screen is very close to the eyes HDR10/4K UHD Premium Spherical view Look anywhere with Foveated a full graphics 360 spherical and video view QSYNC/Jitter Stereoscopic free display Humans see in 3D Immersive Visual Visuals quality Immersion Heightened Sound Sounds quality Support High resolution for object audio and scene-based audio Up to human hearing capabilities Audiophile-grade 3D audio DSD audio Realistic 3D, positional, surround audio that is accurate to the real world Crystal clear voice clear voice that is enhanced with noise cancellation technology 15ms motion to Minimal photon latency Minimized system latency to remove perceptible lag Intuitive interactions Precise Sensor fusion-based motion tracking 6-DOF motion tracking Accurate on-device motion tracking Natural Acceleration user interfaces of eye Seamlessly and hand interact tracking with VR using natural movements, free from wires 17

Qualcomm Snapdragon VR SDK Access to advanced VR features to optimize applications and simplify development DSP sensor fusion Access to the latest and predictive head pose Asynchronous time warp Warp

lens characteristics APIs optimized for VR Benefits Stereoscopic rendering Generate left and right eye view Single buffer rendering Render directly to the display buffer for immediate display scan

18 Qualcomm Snapdragon VR SDK Access to advanced VR features to optimize applications and simplify development DSP sensor fusion Access to the latest and predictive head pose Asynchronous time warp Warp image based on the latest head pose just prior to scan out Chromatic aberration correction Correct color distortion based on lens characteristics Lens distortion correction Barrel warp image based on lens characteristics APIs optimized for VR Benefits Stereoscopic rendering Generate left and right eye view Single buffer rendering Render directly to the display buffer for immediate display scan out VR layering Generate UI menus and text so that they render correctly in a virtual world Power & thermal management Qualcomm Symphony System Manager provides CPU, GPU, and DSP power, thermal, and performance management Simplified development Optimized VR performance Power and thermal efficiency Qualcomm Symphony System Manager and Qualcomm Snapdragon are products of Qualcomm Technologies, Inc. 18

19 HMD Accelerator Program Pillars Commercialize VR HMDs quickly with fewer resource restraints Product Reference Design ODM Partners Component/ Peripherals/ Technology Partners Performance and Quality Metrics/Testing Marketing Support 19

A comprehensive, robust VR reference design Reference design based on

Sensor Fusion, ATW, Single-Buffer Rendering, Unity plug-in, etc.

Files (schematics, BOM, layout files, thermal design guidelines) VR HMD

20 A comprehensive, robust VR reference design Reference design based on the Snapdragon 835 SoC Snapdragon 835 VR SW Services and SDK (6DoF, Sensor Fusion, ATW, Single-Buffer Rendering, Unity plug-in, etc.) Key components selection (camera, sensors, display) Product/HW Design Files (schematics, BOM, layout files, thermal design guidelines) VR HMD Sample Units (availability in July) MSM8998 is a product of Qualcomm Technologies, Inc. 20

21 Accelerating the development of standalone HMDs HMD Accelerator Program cutting edge components Sensors Camera Controllers 21

22 Actively working with XR device manufacturers XR products based on Snapdragon Mobile VR Platform VR HMD VR HMD VR HMD Tango and Daydream Daydream Daydream Tango Baofeng Matrix Coocaa Wondergate G1 iqiyi Adventure ASUS Zenfone AR Google Pixel Google Pixel XL Lenovo Phab 2 Pro VR HMD VR HMD AR Glasses Daydream Daydream Gear VR Gear VR Pico Neo WhaleyVR ODG R8/R9 ZTE Axon Moto Z Samsung Galaxy S8 Samsung Galaxy S8 Edge 20+ Devices launched In 20+ development 22

23 Google Daydream and Qualcomm collaboration Jointly fostering the ecosystem of standalone mobile VR Google IO accountment: 835 standalone reference design 23

24 Jointly fostering the ecosystem of standalone mobile VR 24

25 Developer Network Snapdragon Mobile VR Development Kit v Snapdragon VR SDK v Snapdragon VR HMD developer.qualcomm.com

26 Thank you Follow us on: For more information, visit us at: & Nothing in these materials is an offer to sell any of the components or devices referenced herein Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm is a trademark of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to Qualcomm may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Preparing for Mass Market Virtual Reality: A Mobile Perspective. Qualcomm Technologies, Inc. September 16, 2017

Preparing for Mass Market Virtual Reality: A Mobile Perspective Qualcomm Technologies, Inc. September 16, 2017 Immersive Always-connected VR is meant to be Mobile 2 Automotive video streaming Crowded event