3D Time-of-Flight Image Sensor Solutions for Mobile Devices

3D Time-of-Flight Image Sensor Solutions for Mobile Devices SEMICON Europa 2015 Imaging Conference Bernd Buxbaum 2015 pmdtechnologies gmbh c o n f i d e n t i a l

Content Introduction Motivation for 3D in mobile 3D-Imaging techniques: A comparison between the approaches CamBoard pico flexx - a new reference design for mobile Roamdap and Applications 2015 pmdtechnologies gmbh 2

Content Introduction Motivation for 3D in mobile 3D-Imaging techniques: A comparison between the approaches CamBoard pico flexx - a new reference design for mobile Roadmap and Applications 2015 pmdtechnologies gmbh 3

Introduction - Who are we? pmd was founded 2002 as a fabless IC company pmd is the leading 3D ToF chip technology provider pmd owns 150+ patent applications safeguarding the technology and applications pmd owns the most mature freely available 3D-ToF technology worldwide: 10,000 customers use pmd based products since 2005 1 Million pmd sensors were sold up to date in Robotics and Industrial applications 30 Million+ USD revenue with pmd based products & services (FYF 2015) Consumer Design Wins will open up the mass market from 2016 Automotive will follow the consumer ramp from 2018 2015 pmdtechnologies gmbh 4

Introduction - Where are we? Headquarters Siegen, Germany Branch Dresden, Germany Branch San José, California Global Footprint through pmd s corporate structure N S Hamburg Berlin Cologne Siegen Dresden Frankfurt Munich www.pmdtec.com 2015 pmdtechnologies gmbh 5

Introduction - What is pmd offering? 3D Time-of-Flight pixel design Offering 3D imagers directly or via partners 3D Camera System layout support, reference designs 2015 pmdtechnologies gmbh 6

Introduction - What is the product? Time-of-Flight (ToF) single-chip FPA CMOS technology SoC integration Design-in support Chip Sales channel ToF pixel design Reference designs System & calibration support Post-Processing support Joint Development Joint Marketing Joint Application Support 2015 pmdtechnologies gmbh 7

Introduction - What is the product? Highest Level of Integration Best Performance and Robustness Highest Flexibility and Functionality Integrated A/D converters for full digital readout Integrated CSI-2 interface Integrated controller and logic for Illumination control Pixel matrix modulation Autonomous imaging phase sequences Optimized technology for highest photo sensitivity Patented Suppression of Background Illumination (SBI) Fast data readout for lowest latency (~4ms) Support of high frame rates (>100fps) Intelligent power management (clock/power gating) Flexible SW configuration Frame rate adaptation Configurable exposure time Amount of phase measurements Up to 100MHz modulation frequency Dedicated variants for different pixel resolution (19k, 38k, 100k) Configurable region of interest (ROI) Smallest form factor Reliable indoor & outdoor operation Dynamically configurable during operation 2015 pmdtechnologies gmbh 8

Motivation for 3D in mobile Giving mobile devices a human-scale understanding of space and motion ToF part: Lens with ToF imager behind VCSEL illumination 12mm x 12mm x 7mm Needed real estate : 3cm² Up to 4m range with depth error of <1% Superior background light performance 80 FOV, 19kpixel resolution 1x VCSEL illumination Module size: 3cm² 5fps <300mW power consumption camera frontend example 12 x 12 x 7mm 2015 pmdtechnologies gmbh 10

Motivation for 3D in mobile Hand tracking in Virtual Reality and 3D environmental awareness Two operation modes with one ToF camera 1. Hand tracking: <1m, 45fps, 300mW Use your hands in the virtual world 2. Environmental Awareness <4m, 5fps, 300mW Augment real objects into the virtual world Avoid collisions while moving Camera with wide opening angle should be used for best user experience 2015 pmdtechnologies gmbh 11

Motivation for 3D in mobile 2015 pmdtechnologies gmbh 12

3D-Imaging techniques: Passive Stereo Stereo Vision Depth through disparity measurement: Two standard 2D cameras are used capturing the scenery from different angles to calculate the distance Benefits Standard imagers can be used Thin module height Cheap solutions possible (but becomes higher cost when performance shall be improved: e.g. global shutter, IR illumination) No conceptual limit on frame rates Weaknesses Physical baseline required trade off between accuracy at long range and close range operation Big x-dimension of module because of baseline Precise mechanical alignment and re-calibration needed High computational load through complex, computationally intensive algorithms Limitation in bad and changing light conditions, shadows and intensity differences 2015 pmdtechnologies gmbh 14

3D-Imaging techniques: Active Stereo Structured Light A known pattern is projected onto the scenery and the depth is achieved by inspecting the pattern distortion Benefits No conceptual limit on frame rates No motion blur (in case static pattern is used) Robust against multi-path interference Weaknesses Complex camera through active illumination Precise and stable mechanical alignment between lens and pattern projector needed High calibration efforts; risk of de-calibration Strong interpolation of depth data. No relation between high 2D imager resolution and real 3D point cloud Reflected pattern is sensitive to optical interference or structures and textures in the environment 2015 pmdtechnologies gmbh 15

3D-Imaging techniques: Time-of-Flight (ToF) Time-of-Flight The phase shift between the modulated illumination and the reflection is measured and translated to distance Benefits Direct measurement of depth and amplitude in every pixel Small form factor and monocular system architecture No mechanical baseline: no special requirements on mechanical stability, no mechanical alignment and angle correction required No risk of de-calibration due to drops, vibrations or thermal bending Easy once-in-a-lifetime calibration Operation in any light condition Weaknesses Complex camera through active illumination z-height vs. resolution Multi-path interference 2015 pmdtechnologies gmbh 16

3D-Imaging techniques: Comparison Stereo Vision Structured Light Time-of-Flight Depth accuracy Low Medium High Range vs. accuracy Limited Limited Scalable Power Consumption (camera only) Low Scalable Scalable Software Complexity, Computational Load High High Low Low-Light Performance Weak Good Good Bright-Light Performance Good Weak Good Changing Light Conditions Weak Weak Good Night Vision No Yes Yes z-height vs. resolution Low Low Medium Module size (x-dimension), PCB Design limitations High High Low Manufacturing & Calibration efforts High High Low Material Cost Low Medium Medium 2015 pmdtechnologies gmbh 17

3D-Imaging techniques: Comparison Time-of-Flight is... highly scalable to dedicated application specs very lean in terms of computational load... a very robust depth sensing technology easy to calibrate (one-time and fast) very mature (since 2005) 2015 pmdtechnologies gmbh 18

Sequence Controller depth map calculation 3D-Imaging techniques: Time-of-Flight (ToF) 1 Active illumination with modulated IR-light 2 transmitted reflected Phase Shift Distance measurement in the time-domain 3 Annotated depth map including amplitude image as a quality-indicator Depth Camera LED or VCSEL Modulation I²C Pixel Matrix amplitude 3D scene Lens A/D Converter 3D image raw data depth 3D-ToF FPA MIPI CSI-2 Host Controller 2015 pmdtechnologies gmbh 19

CamBoard pico flexx 2015 pmdtechnologies gmbh 21

CamBoard pico flexx 2015 pmdtechnologies gmbh 22

CamBoard pico flexx Dual Mode operation with one perception shot, thus without loosing frame rate Both faster short-range as well as slower far-range depth maps in parallel The depth maps delivers same data quality in near as well as in far range This unique feature is enabled due to the principle of ToF and a flexible mode of operation, changeable frame by frame I2C on-chip programmable registers are used to reconfigure the chip on the fly 2015 pmdtechnologies gmbh 23

Roadmap and Applications Sales name Resolution Package IRS1020C CIF: 352 x 288 pixel (100k imager) bare die IRS1010C QVGA: 160 x 120 pixel (19k imager) bare die ROI is flexible: Other pixel resolution below 100k can be defined by SW Engineering Sample are available SOP: Q4 2015 2015 pmdtechnologies gmbh 25

Roadmap and Applications 3D Room- & Object Scanning Virtual- & Augmented Reality Gesture Control & Virtual MMI enhanced 2D camera features e.g. autofocus Computational Photography Secure Face Recognition All-in-One Laptop Tablet Smartphone VR Sunlight robustness Smallest form factor Scalability to application requirements patented SBI (suppression of background illumination) USP single-chip no mechanical baseline USP meeting power consumption and self-heating requirements of mobile devices 2015 pmdtechnologies gmbh 26

Thank you for your attention! 2015 pmdtechnologies gmbh 27