EGLSTREAMS: INTEROPERABILITY FOR CAMERA, CUDA AND OPENGL. Debalina Bhattacharjee Sharan Ashwathnarayan

Transcription

1 EGLSTREAMS: INTEROPERABILITY FOR CAMERA, CUDA AND OPENGL Debalina Bhattacharjee Sharan Ashwathnarayan

2 Tegra SOC and typical use-cases Why Interops EGLStream and Its Key Features Agenda Examples on EGLStream Connect EGLStream to NvMedia and CUDA Perform CUDA processing on Camera inputs Connect EGLStream to NvMedia and OpenGL Display with OpenGL. Future Scope 2

3 Download: TRY IT OUT!!! scp r nvidia_1b@ :gtcegl12oct /home/nvidia Open the pdf: Go to /home/nvidia/gtcegl12oct/gtcegl12oct.pdf Run: cd /home/nvidia/gtcegl12oct/ export DISPLAY=:0 chmod +x./x11/egl_stream_demo IF Monitor has A on it:./x11/egl_stream_demo A IF Monitor has B on it:./x11/egl_stream_demo B 3

4 TEGRA SOC Tegra SOC engines Armv8 CPU Geforce GPU ISP GEFORCE GPU CPU COMPLEX Video Encode Video Decode SECURITY ENGINE VIDEO ENCODER VIDEO DECODER AUDIO ENGINE (APE) 2D ENGINE (VIC) And More.. SAFETY ENGINE (SCE) SAFETY MANAGER (HSM) BOOT PROC (BPMP) CAN PROC (SPE) IMAGE PROC (ISP) I/O 4

5 WHY IS INTEROP NEEDED? Different API and libraries for different engines Processor API GPU CUDA OpenGL ISP Argus NvMedia 5

6 INTEROP Same physical memory shared between different API and no copy involved No Interop memcpy Avoid Memcpy = Perf Gain With Interop buffer buffer Memory ISP NvMedia GPU buffer Single Buffer = Less Memory FootPrint ISP NvMedia Memory GPU MAP Synchronize 6

7 AUTOMOTIVE USE CASES Typical use cases utilize all resources on SOC ISP Capture GPU compute GPU Display Video Decode GPU compute GPU Display ISP Capture GPU compute Video Encode 7

8 INTEROP BETWEEN APIS Engines Need To Talk With Each Other NvMedia (ISP) CUDA (GPU)? OpenGL (GPU) Argus (ISP) EGLDisplay (Display) 8

9 EGLSTREAM Unified interface to communicate between multiple APIs NvMedia (ISP) CUDA (GPU) EGLStream OpenGL (GPU) Argus (ISP) EGLDisplay (Display) 9

10 EGLSTREAM HOW IT WORKS Transfer a sequence of image frames from one API to another. Producer-Consumer Model Produces Frames Enables and hides details of buffer transport Accepts frame Producer Buffer EGLStream Buffer Consumer 10

11 Usage Semantics: EGLSTREAMS B PresentFrame() Acquire() Producer ReturnFrame() EGL stream Release() Consumer B Buffer/Allocation ownership 11

12 NVMEDIA_CUDA API SEQUENCE Producer NvMediaProducerConnect Consumer connects first CUDAConsumerConnect Consumer NVMEDIA BLIT() NvMediaProducerPostImage Consumer waits for a frame to presented Implicit Synchronization CUDAConsumerAcquireFrame A returned frame is safe to be presented again Release the frame CUDA Kernel Run CUDA kernel on acquired frame NvMedia ProducerGetImage Released frame is Returned to producer CUDAConsumerReleaseFrame Implicit Synchronization NvMediaProducerDisconnect CUDAEGLStreamConsumerDisconnect 12

13 CUDA_EGLSTREAM INTEROP Key Advantages Performance improvement no memcpy needed with igpu Less Memory footprint single buffer is shared with mapping Ease of use - Support for implicit synchronization Cross Process support - Producer/Consumer can be in different processes (IPC) No need of individual interop unified interface Portable across cameras - Support for both Interleaved and Multi-planar format Supported Platforms 13

14 EGLSTREAMS Support for Discrete GPU on DrivePX 2 Transfer buffers from camera to igpu or dgpu efficiently Support on x86/x86_64 Linux CUDA 9.0 support Support added for easier development Additional YUV multiplanar color formats 14

15 DEMO APPLICATION Built on Vibrante Uses NvMedia for Producer CUDA used for compute processing OpenGL used for Display One EGLStream per Camera 15

16 IMPORTANT Don t pull out the camera! If you are confused about which file to edit, call TA. Call for TA s help if something is wrong. Refer to README file for details. (Especially for killing the app) 16

17 DEMO APPLICATION WorkFlow: NVMEDIA CUDA Nvmedia APIs capture Image & Presents it to CUDA CUDA acquires the image, runs fisheye correction & YUV to RGB conversion Hands over the image to googlenet inference Engine. Inference on the image done & result reported. CUDA releases the acquired Image. NvMedia accepts the returned image. Cycle continues. 17

18 NVMEDIA_CUDA API SEQUENCE Producer img_producer.c cuda_consumer.c Consumer Line: 309 NvMediaEglStreamProducerCreate cueglstreamconsumerconnect Line: 256 Line: 152 NvMediaEglStreamProducerPostImage cueglstreamconsumeracquireframe Line: 133 infersingleframe Line: 163 ine: 178 NvMediaEglStreamProducerGetImage cueglstreamconsumerreleaseframe Line: 180 NvMediaEglStreamProducerDestroy cueglstreamconsumerdisconnect ine: 352 Line:

19 Build: NVMEDIA CUDA cd /home/nvidia/gtcegl12oct/ make Run: export DISPLAY=:0 IF Monitor has A on it:./x11/egl_stream_demo A IF Monitor has B on it:./x11/egl_stream_demo B 19

20 OTHER CONSUMERS 20

21 DEMO APPLICATION WorkFlow: NVMEDIA GL Nvmedia APIs capture Image & Presents it to CUDA GL acquires the image & renders it to the DISPLAY. GL releases the acquired Image. NvMedia accepts the returned image. Cycle continues. 21

22 APPLICATION Main/Camera Producer Thread 1. Initialize Camera resources, create an EglStream 2. Launch OpenGL Consumer thread and pass the EglStream to it 3. Connect NvMediaProducer to EGL stream 4. Loop 1. Post frame on NvMediaProducer GL Consumer Thread 1. Initialize helper GL resources 2. Create GLConsumer 3. Connect GLConsumer to the EglStream 4. Loop 1. Acquire frame from the EGLStream. Wait if frame is not available 2. Render the acquired frame 3. Release the acquired frame 22

23 REPLACE CUDA CONSUMER WITH GL CONSUMER cd /home/nvidia/gtcegl12oct/ Open interop.c with an Editor Comment out cuda_consumer.h & uncomment gl_consumer.h Search and Replace CudaConsumer with GlConsumer 23

24 Build: NVMEDIA - GL cd /home/nvidia/gtcegl12oct/ make Run: export DISPLAY=:0 IF Monitor has A on it:./x11/egl_stream_demo A IF Monitor has B on it:./x11/egl_stream_demo B 24

25 More complex pipelines THINGS TO TRY Integrated GPU Camera Producer EGL Stream CUDA Consumer CUDA Processing CUDA Producer discrete GPU EGL Stream OpenGL Consumer EGL Stream CUDA Producer CUDA inference CUDA Consumer 25

26 REFERENCE EGL_KHR_stream: EGL_KHR_stream_consumer_gltexture: CUDA: CUDA EGL.html#group CUDA EGL NvMedia: 26

27 THANK YOU