April 4-7, 2016 Silicon Valley VISIONWORKS A CUDA ACCELERATED COMPUTER VISION LIBRARY S6783. Elif Albuz, April 4, 2016

Transcription

1 April 4-7, 2016 Silicon Valley VISIONWORKS A CUDA ACCELERATED COMPUTER VISION LIBRARY S6783 Elif Albuz, April 4, 2016

2 Motivation Introduction to VisionWorks AGENDA VisionWorks Software Stack VisionWorks Programming Model Conclusion Demo 2

3 COMPUTER VISION Intelligent Video Analytics Autonomous Driving Robotics Drones Augmented Reality 3

4 COMPUTER VISION 4

5 COMPUTER VISION APP DEVELOPMENT Product Port to target & optimize Reference Implementation Concept 5

6 VISIONWORKS MOTIVATION Deliver high performance, robust computer vision primitives Depth Map Ease development of computer vision applications on Tegra platforms Optical Flow Accelerate prototype to product cycle Corner detection 6

7 VISIONWORKS AT A GLANCE CUDA accelerated library (OpenVX primitives + NVIDIA extensions + Plus Algorithms) Flexible framework for seamlessly adding user-defined primitives. Interoperability with OpenCV Thread-safe API Documentation, tutorials, sample software pipelines that teach use of primitives and framework 7

8 VISIONWORKS SUPPORTED PLATFORMS Automotive Embedded Desktop Drive PX JETSON TX1 Ubuntu Linux 14.04, Windows 8 JETSON TK1 Pro Drive PX2 JETSON TK1 8

9 VISIONWORKS TOOLKIT SOFTWARE STACK VisionWorks-Plus VisionWorks SfM... VisionWorks Object Tracker Source Samples VisionWorks Source Samples Feature Tracking, Hough Transform, Stereo Depth Extraction, Camera Hist Equalization.. NVXIO Multimedia Abstraction VisionWorks Core Library VisionWorks CUDA API NVIDIA VisionWorks Framework & Primitive Extensions OpenVXTM Framework & Primitives Khronos CUDA Acceleration Framework NVIDIA 9

10 VISIONWORKS PRIMITIVES All OpenVX Primitives NVIDIA Extensions IMAGE ARITHMETIC Absolute Difference Accumulate Image Accumulate Squared Accumulate Weighted Add/ Subtract/ Multiply + Channel Combine Channel Extract Color Convert + CopyImage Convert Depth Magnitude MultiplyByScalar Not / Or / And / Xor Phase Table Lookup Threshold FLOW & DEPTH Median Flow Optical Flow (LK) + Semi-Global Matching Stereo Block Matching IME Create Motion Field IME Refine Motion Field IME Partition Motion Field GEOMETRIC TRANSFORMS Affine Warp + Warp Perspective + Flip Image Remap Scale Image + FILTERS BoxFilter Convolution Dilation Filter Erosion Filter Gaussian Filter Gaussian Pyramid Laplacian3x3 Median Filter Scharr3x3 Sobel 3x3 FEATURES Canny Edge Detector FAST Corners + FAST Track Harris Corners + Harris Track Hough Circles Hough Lines ANALYSIS Histogram Histogram Equalization Integral Image Mean Std Deviation Min Max Locations + type/mode extension by NVIDIA NVIDIA extension primitives 10

11 VISIONWORKS PRIMITIVES All OpenVX Primitives VisionWorks primitives are CUDA optimized (except MedianFlow & FindHomography extensions) 85% of VisionWorks OpenVX API is also accelerated with NEON. Table of NEON optimized primitives are listed in VisionWorks Toolkit Ref. (Go to "VisionWorks API" -> "NVIDIA Extensions API" -> "Vision Primitives API ) Primitive acceleration with VisionWorks NVIDIA Extensions Up to 92x speedup compared to OpenCV CPU kernels on Drive PX (Ave 8x) Up to 13x speedup compared to OpenCV CUDA kernels on Drive PX (Ave 2x) (Measured on Drive PX, OS= V4L' Linux Kernel=' tegra-g06aec38' CPU Rate='1632 MHz' GPU Rate='844 MHz' EMC Rate='1600 MHz ) 11

12 VISIONWORKS SAMPLE APPLICATIONS Feature Tracker Stereo Depth Extraction OpenCV-NPP- OpenVX Interop Hough Lines & Circles + Video stabilization + Iterative Motion Estimation/Flow and other platform specific samples (available only on certain platforms) Camera Capture, OpenGL interop, Video playback 12

13 VISIONWORKS SAMPLE APPLICATIONS NVXIO MULTIMEDIA ABSTRACTION Camera input Interop/EGLStre ams Interop/EGLStre ams CSI ISP & Camera Processing GFX Render Vision processing Video/image file input Image/Video Decode CUDA Image/Video Encode... Streamed video/image input GPU CPU COMPLEX (Multi-core ARM v8) NVXIO SECURITY ENGINE VIDEO ENCODER VIDEO DECODER AUDIO ENGINE (APE) 2D ENGINE (VIC) SAFETY ENGINE (SCE) SAFETY MANAGER (HSM) BOOT PROC (BPMP) CAN PROC (SPE) IMAGE PROC (ISP) I/O 13

14 VISIONWORKS PLUS ALGORITHMS Structure From Motion Object Tracker 14

15 Programming with VisionWorks Library 15

16 VISIONWORKS PROGRAMMING MODEL VisionWorks OpenVX Immediate Mode VisionWorks OpenVX Graph Mode VisionWorks CUDA API Standard specified heterogeneous compute API with individual function calls Heterogeneous compute API with graph optimizations Extensible with user defined nodes Direct CUDA API for advanced CUDA developers 16

17 VISIONWORKS OPENVX IMMEDIATE MODE VIDEO STABILIZATION SAMPLE OpenVX Immediate mode API enables developers to easily port their applications. OpenVX API Immediate mode calls are prefixed with vxu Ported Video Stabilization algorithm in OpenCV to VisionWorks Immediate Mode. OpenCV image Source Feature detection Cv::Mat to Vx_image Color Conversion Optical Flow Processs pts & Find Homography Warp Perspective Stabilized frames Image Pyramid 17

18 VISIONWORKS OPENVX IMMEDIATE MODE VIDEO STABILIZATION SAMPLE Performance boost: Video stabilization application is accelerated by 2.6x (including the overhead for Mat to vx_image conversions) Cv::Mat to Vx_image OpenCV image Source 0.6x Color Conversion Feature detection 1.4x 1.7x Image Pyramid 4.9x 2.3x 4.6x Processs pts Optical Warp & Find Flow Perspective Homography Stabilized frames 18

19 VISIONWORKS OPENVX GRAPH MODE VIDEO STABILIZATION SAMPLE OpenVX API graph mode calls are prefixed with vx OpenVX Graph enables advanced optimizations Buffer reuse, kernel fusion Efficient use of streaming and CUDA textures Automatic scheduling across processing units based on various factors (safety, perf,..) Tiling and pipelining vision functions at sub-frame level Feature detection Image Source Color Conversion Optical Flow Processs pts & Find Homography Warp Perspective Stabilized frames Image Pyramid 19

20 VISIONWORKS OPENVX GRAPH MODE VIDEO STABILIZATION SAMPLE Performance boost: Video stabilization application is further accelerated compared to immediate mode. Feature detection Image Source Color Conversion Optical Flow Processs pts & Find Homography Warp Perspective Stabilized frames Image Pyramid 20

21 VISIONWORKS CUDA API FEATURE TRACKING SAMPLE VisionWorks CUDA API enables developer with low-level access. Developer manages Data allocations and transfer Scheduling and pipelining Camera/image/video Input data YUV frame Gray frame Rendering/Output nvxcucolor Convert nvxcuchannel Extract nvxcugaussian Pyramid nvxcuoptica lflowpyrlk nvxcuharris Track RGB frame (CUDA buffer) Array of keypoints 21

22 VISIONWORKS API SELECTION VisionWorks OpenVX Immediate Mode VisionWorks OpenVX Graph Mode VisionWorks CUDA API Quick port from other libraries To be able to reassign CPU and GPU tasks based on perf. Let the graph manager to hide overheads, optimize and manage data To be able to reassign CPU and GPU tasks based on perf. Low level CUDA API access for advanced CUDA developers 22

23 DEBUGGING WITH VISIONWORKS Enable VisionWorks debug markers with export NVX_PROF=nvtx 23

24 VISIONWORKS DOCUMENTATION Installed location: /usr/share/visionworks/docs 24

25 VISIONWORKS FACTS First Khronos OpenVX 1.0 compliant library (Jan 2015) VisionWorks enables key demos (CES 16 and more at GTC) 27K downloads (embedded) since release in Nov, Installed by default on all automotive platforms Weekly VisionWorks downloads for various platforms 25

26 CONCLUSION VisionWorks Toolkit delivers multiple levels of API OpenVX Immediate Mode, OpenVX Graph Mode, VisionWorks CUDA API Heterogeneous API enables switching from GPU to CPU this is very powerful, reducing productization time Delivers high performance Offers significant speedup over CUDA optimized OpenCV functions Adopts native media APIs on Tegra platforms and delivers ready to use code samples S6739-VisionWorks Toolkit Programming Tutorial Room LL20A L6129-VisionWorks Toolkit LAB Session Room 210C H Designing Computer Vision Applications with VisionWorks Pod B 26

27 RESOURCES & USEFUL LINKS VisionWorks Webinars

28 FULLY CONVOLUTIONAL NETWORK [1] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition [2] Efficient Convolutional Patch Networks for Scene Understanding CVPR Workshop on Scene Understanding (CVPR-WS). [3] M. Cordts, M. Omran, S. Ramos, T. Scharwächter, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The Cityscapes Dataset," in CVPR Workshop on The Future of Datasets in Vision, VISIONWORKS WITH DEEP LEARNING DEMO 28

29 FULLY CONVOLUTIONAL NETWORK [1] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition [2] Efficient Convolutional Patch Networks for Scene Understanding CVPR Workshop on Scene Understanding (CVPR-WS) DEEP LEARNING & VISION DEMO 29

30 Introduction VisionWorks API OpenVX Sample Overview 30

31 VISIONWORKS Sample Applications Feature tracking with compressed images Histogram Eq w/camera input... Hough Lines with decoded video Source Samples with multimedia I/0 NVXIO (Multimedia Abstraction) Platform Software Stack (Multimedia, Interop, GL, UI, System) 31

32 PLATFORMS & MULTIMEDIA API Platform Camera Decode Interop Render Encode Android Vibrante Android Camera HAL v3.0 NvMedia capture Android API NvMedia +Gst NvMedia h264 ES CUDA-OpenGL interop? EGLStreams OpenGLES 3.0 OpenGLES (GLFW) Linux4Tegra Gst-capture Gst+OpenMAX EGLStreams OpenGLES Gst+OpenMAX (?) Gst Ubuntu Linux V4L through OpenCV4Tegra Gst+VDPAU CUDA-OpenGL Interop OpenGL Gst Windows x64 V4W/OpenCV NVCUVID (Gst?) CUDA-OpenGL Interop OpenGL Ffmeg/OpenCV Gst - Gstreamer 32

33 Multi-quote slide sample. Source: Either a name or publication text here, OR, a company logo to the right Multi-quote slide sample. Source: Either a name or publication text here, OR, a company logo to the right Multi-quote slide sample. Source: Either a name or publication text here, OR, a company logo to the right 33