IMAGE AND VISION PROCESSING ON TEGRA K1. Elif Albuz

Transcription

1 IMAGE AND VISION PROCESSING ON TEGRA K1 Elif Albuz

2 IMAGE AND VISION USE CASES Driven by using camera as a sensor Computational Photography and Videography Face, Body and Gesture Tracking 3D Scene/Object Reconstruction Augmented Reality

3 Processing Demands MOBILE VISION COMPUTING Photography Input = 2D Camera Processors = ISP + CPU Product = Static Images Mobile Vision Computing Input = MEMS + Depth Camera Processors = ISP + CPU + GPU Result = Data for advanced user interface and environment modeling Computational Photography Input = MEMS + 2D Camera Processors = ISP + CPU + GPU Result = Enhance Images and Videos Time

4 VISION COMPUTING APP CATEGORIES Vision Computing 3D Reconstruction (constructs 3D geometry) Tracking (constructs positions and motions) Facial Modeling Object Reconstruction Face and gesture tracking Environmental Feature Tracking Body Modeling Scene Reconstruction Body Tracking Indoor/Outdoor Positional Tracking 3D Grid with SFM Ped/car detection & tracking

5 AUGMENTED REALITY HYPER-REALISM Ray-tracing and light-field calculations running today on CUDA laptop PC 50+ Watts Ongoing research to use depth cameras to reconstruct global illumination model in real-time Need on mobile devices at 100x less power = 0.5W High-Quality Reflections, Refractions, and Caustics in Augmented Reality and their Contribution to Visual Coherence P. Kán, H. Kaufmann, Institute of Software Technology and Interactive Systems, Vienna University of Technology, Vienna, Austria

6 SIMULTANEOUS LOCALIZATION AND MAPPING

7 WHAT IS NEEDED? Accelerated image & vision processing Tegra K1: CPU, GPU, ISP Camera flexibility and handling of new sensor types Android HAL V3, V4L Low latency routing of image streams to GPU EGLStreams Integrated handling of various sensors and camera Global Time Stamps Effective image & vision programming frameworks VisionWorks

8 TEGRA K1 PLATFORM Desktop GPGPU features and tools on mobile GPGPU Compute CUDA Tools and Libraries Advanced Graphics

9 THREE GENERATIONS OF REFINEMENT Tesla Fermi Kepler

10 Audio Processor ARM7 28 nm HPM 23x23mm, 0.7mm pitch HS-FCBGA Quad Cortex-A15 4x Cores (1+ GHz) NEON SIMD 2 MB L2 (Shared) ARM Trust Zone Shadow LP C-A15 CPU HD Video Processor 1080p24/30 Video Decode 1080p24/30 Video Encode H.264 MPEG4 VC1 MPEG2 VP8 TEGRA K1: A MAJOR LEAP FORWARD FOR MOBILE & EMBEDDED APPLICATIONS SATA2 x1 USB 2.0 x3 PCIe* G2 x4 + x1 Image Processor 25MP Sensor Support ISP 1080p60 Enhanced JPEG Engine Kepler GeForce GPU w/cuda OpenGL-ES nextgen 192 Stream Processors 2D Graphics/Scaling KEPLER GPU, 192 CORES CUDA USB 3.0* x2 CSI x4 + x4 UART x4 I2C x5 NOR Flash SPI x4 SDIO/MMC x4 DDR3 Ctlr 64b 800+ MHz Display x2 Security Engine HDMI edp/lvds DAP x5 (1 2 S/TDM) 12GB/S BANDWIDTH VIDEO IMAGE COMPOSITOR (VIC)

11 Audio Processor ARM7 CPU 28 nm HPM 23x23mm, 0.7mm pitch HS-FCBGA SATA2 x1 USB 2.0 x Quad Cortex-A15 4x Cores (1+ GHz) NEON SIMD 2 MB L2 (Shared) ARM Trust Zone Shadow LP C-A15 CPU Image Processor HD Video Processor 1080p24/30 Video Decode 1080p24/30 Video Encode H.264 MPEG4 VC1 MPEG2 VP8 Kepler GeForce GPU w/cuda Quad-core A15, NEON Unified memory, access from CPU and GPU PCIe* G2 x4 + x1 25MP Sensor Support ISP 1080p60 Enhanced JPEG Engine OpenGL-ES nextgen 192 Stream Processors 2D Graphics/Scaling low-power Shadow core USB 3.0* x2 UART x4 I2C x5 SPI x4 SDIO/MMC x4 Display x2 HDMI edp/lvds CSI x4 + x4 NOR Flash DDR3 Ctlr 64b 800+ MHz Security Engine DAP x5 (1 2 S/TDM)

12 Audio Processor ARM7 IMAGE PROCESSOR 28 nm HPM 23x23mm, 0.7mm pitch HS-FCBGA SATA2 x1 USB 2.0 x3 Quad Cortex-A15 4x Cores (1+ GHz) NEON SIMD 2 MB L2 (Shared) ARM Trust Zone Shadow LP C-A15 CPU x2 Image Processor HD Video Processor 1080p24/30 Video Decode 1080p24/30 Video Encode H.264 MPEG4 VC1 MPEG2 VP8 Kepler GeForce GPU w/cuda 2 ISPs, independently programmable 25Mp camera support (upto 250Mp) PCIe* G2 x4 + x1 25MP Sensor Support ISP 1080p60 Enhanced JPEG Engine OpenGL-ES nextgen 192 Stream Processors 2D Graphics/Scaling 1.2Gp throughput USB 3.0* x2 CSI x4 + x4 UART x4 I2C x5 NOR Flash SPI x4 SDIO/MMC x4 DDR3 Ctlr 64b 800+ MHz Display x2 Security Engine HDMI edp/lvds DAP x5 (1 2 S/TDM) GPGPU interoperability

13 Audio Processor ARM7 HD DEC/ENC PROCESSOR 28 nm HPM 23x23mm, 0.7mm pitch HS-FCBGA Quad Cortex-A15 4x Cores (1+ GHz) NEON SIMD 2 MB L2 (Shared) ARM Trust Zone Shadow LP C-A15 CPU HD Video Processor 1080p24/30 Video Decode 1080p24/30 Video Encode H.264 MPEG4 VC1 MPEG2 VP8 SATA2 x1 USB 2.0 x3 PCIe* G2 x4 + x1 USB 3.0* x2 CSI x4 + x4 Image Processor 25MP Sensor Support ISP 1080p60 Enhanced JPEG Engine UART x4 I2C x5 NOR Flash SPI x4 SDIO/MMC x4 DDR3 Ctlr 64b 800+ MHz Kepler GeForce GPU w/cuda OpenGL-ES nextgen 192 Stream Processors 2D Graphics/Scaling Display x2 Security Engine HDMI edp/lvds DAP x5 (1 2 S/TDM) VIDEO ENCODE/DECODE 2X 1920X1080@30FPS CUVID/CUVENC VIDEO ENCODE/DECODE INTERFACE MOTION ESTIMATION ONLY MODE

14 VIDEO ENCODER Dedicated hardware accelerator Current frame Ref. frame Encoder Compressed video (h.264,..) Motion Vectors/Track info

15 Audio Processor ARM7 GPU 28 nm HPM 23x23mm, 0.7mm pitch HS-FCBGA Quad Cortex-A15 4x Cores (1+ GHz) NEON SIMD 2 MB L2 (Shared) ARM Trust Zone Shadow LP C-A15 CPU HD Video Processor 1080p24/30 Video Decode 1080p24/30 Video Encode H.264 MPEG4 VC1 MPEG2 VP8 KEPLER Architecture 192 CUDA Cores, SM3.2 ISA Compatible to GeForce, Quadro, Tesla SATA2 x1 USB 2.0 x3 PCIe* G2 x4 + x1 Image Processor 25MP Sensor Support ISP 1080p60 Enhanced JPEG Engine Kepler GeForce GPU w/cuda OpenGL-ES nextgen 192 Stream Processors 2D Graphics/Scaling 64kb L1 Cache and Shared Memory 128kb L2 Cache 128 kb Register File USB 3.0* x2 UART x4 I2C x5 SPI x4 SDIO/MMC x4 Display x2 HDMI edp/lvds CSI x4 + x4 NOR Flash DDR3 Ctlr 64b 800+ MHz Security Engine DAP x5 (1 2 S/TDM)

16 WHAT IS NEEDED? Accelerated image & vision processing Tegra K1: CPU, GPU, ISP Camera flexibility and handling of new sensor types Android HAL V3, V4L, Camera API Low latency routing of image streams to GPU EGLStreams Integrated handling of various sensors and camera Global Time Stamps Effective image & vision programming frameworks VisionWorks

17 CAMERA IMAGE PROCESSING Camera ISP (Image Signal Processor) Little or no programmability Data flows thru compact hardware pipe Scan-line-based - no global memory Best perf/watt ~760 math Ops ~42K vals = 670Kb 300MHz ~250Gops

18 VISUAL SENSOR REVOLUTION Single RGB sensors just the start of mobile visual revolution IR sensors LEAP Motion, eye-trackers Active illumination depth sensors TOF and structured light Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras Stereo pair can enable object scaling and enhanced depth extraction Plenoptic Field processing needs FFTs and ray-casting Hybrid visual sensing solutions Different sensors mixed for different distances and lighting conditions Dual Camera LG Electronics Plenoptic Array Pelican imaging Capri Structured Light 3D Camera PrimeSense

19 ANDROID CAMERA HAL V3 Camera HAL v1 focused on simplifying basic camera apps Difficult or impossible to do much else New features require proprietary driver extensions Extensions not portable - restricted growth of third party app ecosystem Camera HAL v3 is a fundamentally different API Flexible primitives for building sophisticated use-cases Interface is clean and easily extensible Apps can have more control, and more responsibility Enables sophisticated camera applications Faster time to market and higher quality

20 KHRONOS CAMERA API Specification available 2015 Also FCAM-Based Will be available for any OS FCAM running on NVIDIA Linux today No global state State travels with image requests Every stage in the pipeline may have different state Enables fast, deterministic state changes Synchronize devices Lens, flash, sound capture, gyro Devices can schedule Actions E.g. to be triggered on exposure change

21 KHRONOS CAMERA API REQUIREMENTS Application control over ISP processing (including 3A) Including multiple, re-entrant ISPs Control multiple sensors with synch and alignment E.g. Stereo pairs, Plenoptic arrays, TOF/structured light depth cameras Enhanced per frame detailed control Format flexibility, Region of Interest (ROI) selection Global timing & synchronization E.g. Between cameras and MEMS sensors Flexible processing/streaming Multiple input and output streams RAW, Bayer or YUV Processing Streaming of rows (not just frames) Enable new camera functionality not available on current platforms and align with future platform directions for easy adoption

22 TEGRA K1 CAMERA DATAFLOW Flexible routing of sensor data to ISPs and unified memory ISP-A Enables sensor processing by any combination of GPUs, CPUs and ISPs Camera One Camera Two Camera Routing Unified Memory CPU CPU CPU CPU Kepler GPGPU ISP-B

23 COMPUTATIONAL PHOTOGRAPHY Camera ISP (Image Signal Processor) Little or no programmability CPU GPU Single processor or Neon SIMD - running fast Makes heavy use of general memory Non-optimal performance and power Programmable and flexible Many way parallelism - run at lower frequency Efficient image caching close to processors BUT cycles frames in and out of memory ~760 math Ops ~42K vals = 670Kb 300MHz ~250Gops

24 WHAT IS NEEDED? Accelerated image & vision processing Tegra K1: CPU, GPU, ISP Camera flexibility and handling of new sensor types Android HAL V3, V4L Low latency routing of image streams to GPU EGLStreams Integrated handling of various sensors and camera Global Time Stamps Effective image & vision programming frameworks VisionWorks

25 EGL 1.5 RELEASED EGL 1.5 brings functionality from multiple extensions into core Increased reliability and portability EGLImages Sharing textures and renderbuffers Context Robustness Defending against malicious code EGLSync objects Improved OpenGL /OpenCL interop Platform extensions Standardized interactions for multiple OS e.g. Android and 64-bit platforms srgb colorspace rendering API Interop EGL provides efficient transfer of data and events between Khronos APIs Applications OS and Display Platforms Application Portability EGL abstracts graphics context management, surface and buffer binding and rendering synchronization

26 WHAT IS NEEDED? Accelerated image & vision processing Tegra K1: CPU, GPU, ISP Camera flexibility and handling of new sensor types Android HAL V3, V4L Low latency routing of image streams to GPU EGLStreams Integrated handling of various sensors and camera Global Time Stamps Effective image & vision programming frameworks VisionWorks

27 HOW MANY SENSORS ARE IN A SMARTPHONE? Light Proximity 2 cameras 3 microphones Touch Position GPS WiFi (fingerprint) Cellular (tri-lateration) NFC, Bluetooth (beacons) Accelerometer Magnetometer Gyroscope Pressure Temperature Humidity 19 27

28 STREAMINPUT SENSOR ABSTRACTION API Apps request semantic sensor information StreamInput defines possible requests, e.g. Read Physical or Virtual Sensors e.g. Game Quaternion Context detection e.g. Am I in an elevator? Apps Need Sophisticated Access to Sensor Data Without coding to specific sensor hardware Sensor Discoverability Sensor Code Portability Advanced Sensors Everywhere Multi-axis motion/position, quaternions, context-awareness, gestures, activity monitoring, health and environmental sensors StreamInput processing graph provides optimized sensor data stream High-value, smart sensor fusion middleware can connect to apps in a portable way Apps can gain magical situational awareness

29 WHAT IS NEEDED? Accelerated image & vision processing Tegra K1: CPU, GPU, ISP Camera flexibility and handling of new sensor types Android HAL V3, V4L Low latency routing of image streams to GPU EGLStreams Integrated handling of various sensors and camera Global Time Stamps Effective image & vision programming frameworks VisionWorks

30 MOBILE & EMBEDDED DEVELOPERS NEED HELP! Control, coordinate and synchronize a diverse array of mobile sensors Handle a diverse selection of emerging depth camera technologies Write maintainable code for a heterogeneous mix of CPUs, GPUs and DSPs Write code that is deployable across multiple devices, platforms and OS Leverage dedicated vision hardware for minimized power Create fluid 60Hz experiences on batterypowered mobile devices

31 DESKTOP TO MOBILE DEVELOPMENT Image & Vision Processing code development starts at desktop PC Tegra K1 enables easy migration through libraries and tools available across platforms

32 TEGRA K1 CUDA DEVELOPMENT CUDA-Aware Editor Automated CPU to GPU code refactoring Semantic highlighting of CUDA code Integrated code samples & docs Nsight Debugger Simultaneously debug of CPU and GPU Inspect variables across CUDA threads Use breakpoints & single-step debugging Nsight Profiler Quickly identifies performance issues Integrated expert system Source line correlation Cross platform development Native memcheck, GDB, nvprof

33 CUDA LIBRARIES VisionWorks NPP OpenCV CUFFT CUBLAS CUDA Math Lib

34 NPP LIBRARY (CUDA) Data exchange & initialization Set, Convert, CopyConstBorder, Copy, Transpose, SwapChannels Arithmetic & Logical Ops Add, Sub, Mul, Div, AbsDiff Threshold & Compare Threshold, Compare Color Conversion JPEG RGB To YCbCr (& vice versa), ColorTwist, LUT_Linear DCTQuantInv/Fwd, QuantizationTable Functions FilterBox, Row, Column, Max, Min, Median, Dilate, Erode, SumWindowColumn/Row Geometry Transforms Mirror, WarpAffine / Back/ Quad, WarpPerspective / Back / Quad, Resize Statistics Mean, StdDev, NormDiff, MinMax, Histogram, SqrIntegral, RectStdDev Computer Vision ApplyHaarClassifier, GraphCuts

35 OPENCV LIBRARY Initially developed by Intel for single-core x86 CPUs Version >900 functions (x the datatypes) OpenCV4Tegra - Accelerated CUDA+NEON+GLSL+TBB multithreading OpenCV Image processing General Image Processing Segmentation Machine Learning, Detection Image Pyramids Transforms Fitting Video, Stereo, and 3D Camera Calibration Features Depth Maps Optical Flow Inpainting Tracking

36 OPENCV-GPU VALUE ADD ON LOGAN Speedup 7 6 Jetson Kepler GPU /Quadcore A15 Public OCV for Mobile/Embedded core filter imgproc objdetect Average speedup for different function categories with Logan GPU compared to public source code.

37 VISIONWORKS MOTIVATION + = Advanced Silicon VisionWorks Simplify vision programming Fully optimized and accelerated Modular and Extensible Widespread vision processing in embedded, mobile and automotive devices and applications

38 VISIONWORKS Power Efficient Computer Vision Powered with CUDA Supported on Tegra K1 Linux and Android ACCELERATING Advanced Driver Assistance Computational Photography Augmented Reality Robotics Deep Learning and more Version 0.10 is available for registered partners! ADAS Advanced Driver Assistance Systems Augmented Reality Computational Photography Robotics

39 VISIONWORKS SOFTWARE STACK Application Code Applications use combination of direct primitives, the OpenVX framework and supplied pipelines NVIDIA provides sample pipelines for common use cases Object Detection Sample Pipelines SfM 3rd Party Pipelines NVIDIA supplied vision primitives using CUDA and Tegra processing resources Classifier VisionWorks Primitives Corner Detection 3 rd Party Primitives Customers and developers can create their own primitives e.g. using CUDA CUDA Framework OpenVX enables power efficient and flexible chaining of primitives Tegra K1

40 OPENVX POWER EFFICIENT VISION ACCELERATION Khronos, open, cross-vendor vision API Focus on mobile and embedded systems Application Foundational API for vision acceleration Useful for middleware or by applications Enables diverse efficient implementations OpenCV OpenCV open source library VisionWorks Sample Pipelines Complementary to OpenCV Which is great for prototyping Open source sample implementation Hardware vendor implementations

41 OPENVX GRAPHS THE KEY TO EFFICIENCY Directed graphs for processing power and efficiency Each Node can be implemented in software or accelerated hardware Nodes may be fused to eliminate memory transfers Processing can be tiled to keep data entirely in local memory/cache EGLStreams route data from camera and to application Can extend with VisionWorks nodes using CUDA Native Camera Control OpenVX Node VisionWorks Node OpenVX Node VisionWorks Node Application Example OpenVX Graph

42 OPENVX AND OPENCV ARE COMPLEMENTARY Governance Community driven open source with no formal specification Defined and implemented by Khronos Portability APIs can vary depending on processor Tegra K1 mobile platforms Scope Efficiency Very wide 1000s of imaging and vision functions Multiple camera APIs/interfaces Memory-based architecture Each operation reads and writes memory Tight focus on hardware accelerated functions for mobile vision Use external camera API Graph-based execution Optimizable computation, data transfer Use Case Rapid experimentation Production development & deployment

43 OPENVX 1.0 FUNCTION OVERVIEW Core data structures Images and Image Pyramids Processing Graphs, Kernels, Parameters Image Processing Arithmetic, Logical, and statistical operations Multichannel Color and BitDepth Extraction and Conversion 2D Filtering and Morphological operations Image Resizing and Warping Core Computer Vision Pyramid computation Integral Image computation Feature Extraction and Tracking Histogram Computation and Equalization Canny Edge Detection Harris and FAST Corner detection Sparse Optical Flow Widely used extensions adopted into future versions of the core OpenVX Specification Evolution OpenVX 1.0 defines framework for creating, managing and executing graphs Focused set of widely used functions that are readily accelerated Implementers can add functions as extensions

44 VISIONWORKS PRIMITIVES JAN 2014 Sobel Convolve Bilateral Filter Integral Image Integral Histogram Corner Harris Corner FAST Image Pyramid Optical Flow PyrLK Optical Flow Farneback Warp Perspective Hough Lines Fast NLM Denoising Stereo Block Matching IME (Iterative Motion Estimation) HOG (Histogram of Oriented Gradients) Soft Cascade Detector Object Tracker TLD Object Tracker SLAM Path Estimator MedianFlow Estimator

45 OPENVX AND CUDA ARE COMPLEMENTARY Use Case Architecture Target Hardware Precision Ease of Use GPGPU Programming Language-based Exposed architected memory model programmer manages memory Full IEEE floating point mandated General-purpose math and other libraries Domain targeted Vision processing Library-based - no separate compiler required Abstracted node and memory model - diverse implementations can be optimized for power and performance Minimal floating point requirements optimized for vision operators Fully implemented vision operators and framework out of the box Use CUDA to build new VisionWorks OpenVX Nodes

46 VISIONWORKS SAMPLE PIPELINES (V0.10) Structure From Motion/SLAM Pedestrian Detection Vehicle detection Object tracking Dense optical flow Active Shape Model Denoising

47 VISIONWORKS LOOKING FORWARD Enable multi-camera applications 3D sensors Conformance with OpenVX once specification finalized

48 TEGRA K1 DEVELOPMENT PLATFORMS Coming to Android K1 & Other Linux Devices soon.. JETSON X3 (TK1 PRO) gige, usb3.0, HDMI, CANBUS running Vibrante Linux AUTOMOTIVE GRADE

49 QUESTIONS?

50 BACKUP