Kari Pulli Intel. Radhakrishna Giduthuri, AMD. Frank Brill NVIDIA. OpenVX Webinar. June 16, 2016

Transcription

1 Kari Pulli Intel Radhakrishna Giduthuri, AMD Frank Brill NVIDIA OpenVX Webinar June 16, 2016

2 Copyright Khronos Group Page 1 Vision Acceleration Kari Pulli Sr. Principal Engineer Intel

3 Copyright Khronos Group Page 2 Khronos Open Standards Software Silicon Khronos is an Industry Consortium of over 100 companies creating royalty-free, open standard APIs to enable software to access hardware acceleration for graphics, parallel compute and vision

4 Copyright Khronos Group Page 3 Vision Processing Power Efficiency Vision processing just on CPU is too expensive - Especially on battery-powered devices Advanced Sensors GPUs are more power-efficient - They were architected for efficient pixel handling Traditional cameras have dedicated hardware - ISP = Image Signal Processor on all SOCs today Wearables But how to program specialized processors? Performance and Functional Portability Power Efficiency X100 X10 X1 Dedicated Hardware Vision DSPs Vision Processing Efficiency GPU Compute Multi-core CPU Computation Flexibility

5 OpenVX Low-Power Vision Acceleration Higher-level abstraction API - Targeted at real-time mobile and embedded platforms Performance portability across diverse architectures - Multi-core CPUs, GPUs, DSPs, ISPs, Dedicated hardware, Extends portable vision acceleration to very low-power domains - Doesn t require high-power CPU/GPU Complex Power Efficiency X100 X10 X1 Dedicated Hardware Vision DSPs Vision Processing Efficiency GPU Compute Multi-core CPU Computation Flexibility Vision Engine Accelerator Middleware Accelerator Application Accelerator Copyright Khronos Group Page 4

6 Copyright Khronos Group Page 5 OpenVX Graphs OpenVX developers express a graph of image operations ( Nodes ) - Nodes can be on any hardware or processor coded in any language - For example, on GPU, nodes may implemented in OpenCL Minimizes host interaction during frame-rate graph execution - Host processor can setup graph which can then execute almost autonomously Camera Input OpenVX Nodes Pyr t Rendering Output RGB Frame Color Conversion YUV Frame Channel Extract Gray Frame Image Pyramid Optical Flow Array of Keypoints Harris Track OpenVX Graph Ftr t-1 Array of Features

7 Copyright Khronos Group Page 6 OpenVX Framework Efficiency Graph Scheduling Memory Management Kernel Merging Data Tiling Split the graph execution across the whole system: CPU / GPU / dedicated HW Reuse pre-allocated memory for multiple intermediate data Replace a subgraph with a single faster node Execute a subgraph at tile granularity instead of image granularity Faster execution or lower power consumption Less allocation overhead, more memory for other applications Better memory locality, less kernel launch overhead Better use of data cache and local memory

8 Copyright Khronos Group Page 7 OpenVX and OpenCV are Complementary Implementation Conformance Consistency Scope Efficiency Typical Use Case Community-driven open source library Extensive OpenCV Test Suite but no formal Adopters program Available functions can vary depending on implementation / platform Very wide 1000s of imaging and vision functions Memory-based architecture Each operation reads and writes to memory Rapid experimentation and prototyping - especially on desktop Open standard API designed to be implemented by hardware vendors Implementations must pass defined conformance test suite to use trademark All core functions must be available in all conformant implementations Tight focus on core hardware accelerated functions for mobile vision but extensible Graph-based execution Optimizable computation and data transfer Production development & deployment on wide range of mobile and embedded devices

9 Copyright Khronos Group Page 8 OpenVX 1.0 Shipping, OpenVX 1.1 Released! Multiple OpenVX 1.0 Implementations shipping spec in October Open source sample implementation and conformance tests available OpenVX 1.1 Specification released in May Expands node functionality AND enhances graph framework -Sample source and conformance tests will be updated to OpenVX 1.1 soon OpenVX is EXTENSIBLE -Implementers can add their own nodes at any time to meet customer and market needs = provided results for conformance tests

10 Copyright Khronos Group Page 1 OpenVX Technical Overview Khronos Webinar khronos.org/openvx Radhakrishna Giduthuri AMD

11 Copyright Khronos Group Page 2 OpenVX Components Context (vx_context) Data Objects vx_image, vx_pyramid, vx_array, vx_lut, vx_remap, vx_scalar, vx_threshold, vx_distribution, vx_matrix, vx_convolution, vx_delay, vx_object_array Graphs (vx_graph) Nodes (vx_node) Kernel instances, parameters, completion callback functions Kernels (vx_kernel) Built-in vision functions, Vendor extensions, User-defined Virtual Data vx_image, vx_pyramid, vx_array, vx_object_array Miscellaneous Directives, Hints, Logging, Performance Measurements Extensions Tiling, XML Schema

12 Copyright Khronos Group Page 3 Context Context - OpenVX world: need to be created first - All objects belong to a context #include <VX/vx.h>... vx_context context = vxcreatecontext(); * See VX/vx_api.h for framework API function definitions.

13 Error Management Methods return a status vx_status returned: VX_SUCCESS when no error if( vxprocessgraph( graph )!= VX_SUCCESS) { /* Error */ } Explicit status check Object creation: use vxgetstatus to check the object vx_context context = vxcreatecontext(); if( vxgetstatus( (vx_reference)context )!= VX_SUCCESS ) { /* Error */ } More info from the log callback void logcallback( vx_context c, vx_reference r, vx_status s, const vx_char string[] ) { /* Do something */ }... vxregisterlogcallback( context, logcallback, vx_false_e );... vxaddlogentry( reference, VX_INVALID_VALUE, specified value is out of range ); * See VX/vx_types.h for type definitions and error codes. Copyright Khronos Group Page 4

14 Data objects The application gets only references to objects, not the objects -References should be released by the application when not needed -Ref-counted object destroyed by OpenVX when not referenced any more vx_image img = vxcreateimage( context, 640, 400, VX_DF_IMAGE_RGB ); // Use the image vxreleaseimage( &img ); Object-Oriented Behavior -strongly typed (good for safety-critical applications) -OpenVX are really pointers to structs - any object may be down-cast to a vx_reference, e.g., for passing to vxgetstatus() Opaque -Access to content explicit and temporary (map/unmap or copy) - No permanent pointer to internal data -Needed to handle complex memory hierarchies - DSP local memory - GPU dedicated memory Copyright Khronos Group Page 5

15 Copyright Khronos Group Page 6 Enumerated Data Types C data type vx_uint8 (basic data type) vx_int16 vx_uint16 vx_int32 vx_float32 vx_enum Enumeration VX_TYPE_UINT8 VX_TYPE_INT16 VX_TYPE_UINT16 VX_TYPE_INT32 VX_TYPE_FLOAT32 VX_TYPE_ENUM vx_rectangle_t (struct) vx_keypoint_t vx_image (opaque object) VX_TYPE_RECTANGLE VX_TYPE_KEYPOINT VX_TYPE_IMAGE

16 Copyright Khronos Group Page 7 Data Object Creation vx_image img = vxcreateimage( ctx, 640, 400, VX_DF_IMAGE_UYVY ); // supports 13 standard formats vx_pyramid pyr = vxcreatepyramid( ctx, levels, VX_SCALE_PYRAMID_HALF, 640, 400, VX_DF_IMAGE_U8 ); vx_array arr = vxcreatearray( ctx, VX_TYPE_KEYPOINT, capacity ); // array of vx_keypoint_t[] vx_lut lut = vxcreatelut( ctx, VX_TYPE_UINT8, 256 ); // 8-bit look-up table vx_remap remap = vxcreateremap( ctx, src_width, src_height, dst_width, dst_height ); vx_matrix mat = vxcreatematrix( ctx, VX_TYPE_FLOAT32, columns, rows ); vx_distribution dist = vxcreatedistribution( ctx, num_bins, offset, range ); vx_float32 scalar_initial_value = 1.25f; vx_scalar scalar = vxcreatescalar( ctx, VX_TYPE_FLOAT32, &scalar_initial_value ); vx_delay delay = vxcreatedelay( ctx, (vx_reference)pyr, num_slots ); // pyr is an exemplar vx_object_array obj_arr = vxcreateobjectarray( ctx, (vx_reference)pyr, count );

17 OpenVX Graph vx_context context = vxcreatecontext(); vx_image input = vxcreateimage( context, 640, 480, VX_DF_IMAGE_U8 ); vx_image output = vxcreateimage( context, 640, 480, VX_DF_IMAGE_U8 ); vx_graph graph = vxcreategraph( context ); vx_image intermediate = vxcreatevirtualimage( graph, 640, 480, VX_DF_IMAGE_U8 ); vx_node F1 = vxf1node( graph, input, intermediate ); vx_node F2 = vxf2node( graph, intermediate, output ); vxverifygraph( graph ); while(...) { // write to input image vxprocessgraph( graph ); // read from output image } context graph * Use #include <VX/vx.h> for OpenVX header files intermediate input F1 F2 output

18 OpenVX 1.1 Built-in Vision Functions Kernels Pixel-wise Functions Add, Subtract, Multiply, AbsDiff, And, Or, Xor, Not, Magnitude, Phase, Threshold, TableLookup, ColorDepth, ChannelExtract, ChannelCombine, ColorConvert, AccumulateImage, AccumulateSquaredImage, AccumulateWeightedImage, Reduction Functions Histogram, MeanStdDev, MinMaxLoc Filtering Functions Box3x3, Convolve, Dilate3x3, Erode3x3, Gaussian3x3, Median3x3, Sobel3x3, GaussianPyramid, NonLinearFilter, LaplacianPyramid, LaplacianReconstruct Geometric Functions Remap, ScaleImage, WarpAffine, WarpPerspective, HalfScaleGaussian Complex Functions CannyEdgeDetector, EqualizeHist, FastCorners, HarrisCorners, IntegralImage, OpticalFlowPyrLK * See VX/vx_nodes.h for functions to create kernel instances (nodes) in a graph. Copyright Khronos Group Page 9

19 Copyright Khronos Group Page 10 Rectangle typedef struct _vx_rectangle_t { vx_uint32 start_x; /*!< \brief The Start X coordinate. */ vx_uint32 start_y; /*!< \brief The Start Y coordinate. */ vx_uint32 end_x; /*!< \brief The End X coordinate. */ vx_uint32 end_y; /*!< \brief The End Y coordinate. */ } vx_rectangle_t; Type enumeration: VX_TYPE_RECTANGLE Image start : inside rectangle end : outside

20 Copyright Khronos Group Page 11 Keypoints typedef struct _vx_keypoint_t { vx_int32 x; // keypoint x-coordinate vx_int32 y; // keypoint y-coordinate vx_float32 strength; // strength of keypoint vx_float32 scale; vx_float32 orientation; vx_int32 tracking_status; // zero indicates lost point. Initialized to 1 by detectors vx_float32 error; } vx_keypoint_t; Type enumeration: VX_TYPE_KEYPOINT key-point Image

21 Copyright Khronos Group Page 12 Array Data Object vx_array vxcreatearray ( vx_context context, vx_enum item_type, // VX_TYPE_KEYPOINT, VX_TYPE_UINT32,... vx_size capacity ); capacity-1 num_items vx_array array = vxcreatearray( context, VX_TYPE_RECTANGLE, 64 ); // remove all items from array and add 8 items vxtruncatearray( array, 0 ); vxaddarrayitems( array, 8, &rect[0], sizeof(vx_rectangle_t) ); // get number items in the array by querying array attribute vxqueryarray(array, VX_ARRAY_NUMITEMS, &num_items, sizeof(num_items));

22 Copyright Khronos Group Page 13 Array Data Access Copy using application controlled address and memory layout - vxcopyarrayrange: copy (Read or Write) vxqueryarray( arr, VX_ARRAY_NUMITEMS, &num_items, sizeof(num_items) ); vxcopyarrayrange( arr, 0, num_items, sizeof(my_array[0]), &my_array[0], VX_READ_ONLY, VX_MEMORY_TYPE_HOST ); Access limited in time - vxmaparrayrange: get access (Read, Write, Read & Write) - vxunmaparrayrange: release the access vx_map_id map_id; void * ptr; vxqueryarray( arr, VX_ARRAY_NUMITEMS, &num_items, sizeof(num_items) ); vxmaparrayrange( arr, 0, num_items, &map_id, &stride, &ptr, VX_READ_AND_WRITE, VX_MEMORY_TYPE_HOST, 0 ); // Access data in ptr vxunmaparrayrange( arr, map_id );

23 Copyright Khronos Group Page 14 Image Access (1/2) : Overview Copy using application controlled address and memory layout - vxcopyimagepatch: copy (Read or Write) vx_imagepatch_addressing_t addr = { /* to fill stride_x & stride_y */ }; vx_rectangle_t rect = { 0u, 0u, width, height }; vxcopyimagepatch( img, &rect, plane, &addr, my_array, VX_WRITE_ONLY, VX_MEMORY_TYPE_HOST, VX_NOGAP_X ); Access limited in time - vxmapimagepatch: get access (Read, Write, Read & Write) - vxunmapimagepatch: release the access vx_map_id map_id; void * ptr; vx_imagepatch_addressing_t addr; vx_rectangle_t rect = { 0u, 0u, width, height }; vxmapimagepatch( img, &rect, plane, &map_id, &addr, &ptr, VX_READ_AND_WRITE, VX_MEMORY_TYPE_HOST, VX_NOGAP_X ); // Access data in ptr vxunmapimagepatch( img, map_id );

24 Copyright Khronos Group Page 15 Image Access (2/2) : Memory Layout typedef struct _vx_imagepatch_addressing_t { vx_uint32 dim_x; vx_uint32 dim_y; vx_int32 stride_x; vx_int32 stride_y; vx_uint32 scale_x; vx_uint32 scale_y; vx_uint32 step_x; vx_uint32 step_y; } vx_imagepatch_addressing_t; Num of (logical) pixels in a row Num of (logical) pixels in a column Num of bytes between the beginning of 2 successive pixels Num of bytes between the beginning of 2 successive lines Sub-sampling : 1 physical pixel every step logical pixel scale = VX_SCALE_UNITY / step stride_y stride_x Patch

25 Feature Tracking Example vx_context vx_delay of pyramids Optical Flow (LK) vx_node Compute Pyramid Input Image from CAMERA vx_image Convert to Grayscale vx_pyramid at t=n-1 vx_pyramid at t=n keypoint array (x, y, ) at t=n-1 vx_delay of keypoints Copy of Data from Previous Iteration keypoint array (x, y, ) at t=n vx_array keep old copy Computed Data Harris Detector vx_node vx_image vx_graph vx_graph Copyright Khronos Group Page 16

26 Copyright Khronos Group Page 17 Pyramid Data Object vx_pyramid vxcreatepyramid ( vx_context context, vx_size levels, vx_float32 scale, // VX_SCALE_PYRAMID_HALF or VX_SCALE_PYRAMID_ORB vx_uint32 width, vx_uint32 height, vx_df_image format // VX_DF_IMAGE_U8 ); Level 3 Level 0 (base) Level 2 Level 1 Example: vx_pyramid pyramid = vxcreatepyramid(context, ); // get image at pyramid level 2 vx_image img2 = vxgetpyramidlevel( pyramid, 2 ); vxreleaseimage( &img2 ); vxreleasepyramid( &pyramid );

27 Copyright Khronos Group Page 18 Delay Data Object vx_delay vxcreatedelay ( vx_context context, vx_reference exemplar, vx_size count ); Example: vx_pyramid exemplar = vxcreatepyramid(context, ); vx_delay pyr_delay = vxcreatedelay(context, (vx_reference)exemplar, 2); vxreleasepyramid(&exemplar); vx_pyramid pyr_0 = (vx_pyramid)vxgetreferencefromdelay(pyr_delay, 0); vx_pyramid pyr_1 = (vx_pyramid)vxgetreferencefromdelay(pyr_delay, -1); vxagedelay(pyr_delay);

28 Data Objects vx_context Input Image from CAMERA vx_delay of pyramids vx_image vx_pyramid at t=n-1 keypoint array (x, y, ) at t=n-1 vx_delay of keypoints vx_pyramid at t=n keypoint array (x, y, ) at t=n vx_array Copy of Data from Previous Iteration keep old copy Computed Data Copyright Khronos Group Page 19

29 Harris Graph vx_context vx_delay of pyramids Gaussian Pyramid vx_node Input Image from CAMERA vx_pyramid at t=n-1 vx_pyramid at t=n vx_image (RGB) vx_image (virtual U008) vx_node Color Convert keypoint array (x, y, ) at t=n-1 vx_delay of keypoints Copy of Data from Previous Iteration keep old copy Computed Data keypoint array (x, y, ) at t=n vx_array additional parameters Harris Corners vx_node Channel Extract vx_node vx_image (virtual IYUV) vx_graph Copyright Khronos Group Page 20

30 Copyright Khronos Group Page 21 Vision Functions in a Graph RGB -> YUV vxcolorconvertnode( graph, input_rgb_image, harris_yuv_image ); YUV -> Y VX_DF_IMAGE_RGB VX_DF_IMAGE_YUV vxchannelextractnode( graph, harris_yuv_image, VX_CHANNEL_Y, harris_gray_image ); Harris corner - strength_thresh : f - min_distance : 5.0f - sensitivity : 0.04f - gradient_size : 3 - block_size : 3 VX_DF_IMAGE_YUV VX_DF_IMAGE_U8 vxharriscornersnode( graph, harris_gray_image, strength_thresh, min_distance, sensitivity, gradient_size, block_size, keypoint_array_output, NULL );

31 Optical Flow Graph Input Image from CAMERA vx_context vx_delay of pyramids additional parameters Optical FlowPyrLK vx_node Gaussian Pyramid vx_node vx_image (RGB) vx_image (virtual U008) vx_node Color Convert vx_pyramid at t=n-1 vx_pyramid at t=n keypoint array (x, y, ) at t=n-1 vx_delay of keypoints Copy of Data from Previous Iteration keypoint array (x, y, ) at t=n vx_array keep old copy Computed Data Channel Extract vx_node vx_image (virtual IYUV) vx_graph Copyright Khronos Group Page 22

32 Copyright Khronos Group Page 23 Execute a Graph in Loop to Process Input Before executing Harris & Optical Flow Graphs - vxverifygraph API should return VX_SUCCESS (outside the loop) Inside the loop -- process each image from input video sequence - write pixels from input video into input RGB image - Execute Graphs using vxprocessgraph API - Execute Harris Graph for the 1st image from video sequence - Execute Optical Flow Graph from 2nd image onwards - Read previous and current keypoints and draw each item - Use vxgetreferencefromdelay API to get previous and current keypoint arrays - Flip the previous and current pyramid and keypoints in delay objects - Use vxagedelay API - This will automatically trigger flipping of previous and current pyramids in all the graphs After the processing loop - Query VX_GRAPH_ATTRIBUTE_PERFORMANCE for performance measurements - Release all objects -- make sure to release context at the end

33 Copyright Khronos Group Page 24 Summary OpenVX is a low-level programming framework domain to enable software developers to efficiently access computer vision hardware acceleration with both functional and performance portability. OpenVX contains: - a library of predefined and customizable vision functions - a graph-based execution model to combine function enabling both task and data-independent execution, and; - a set of memory objects that abstract the physical memory. OpenVX is defined as a C API - object-oriented design - synchronous and asynchronous execution model - extend functionality using enums and callbacks Useful Links: and github.com/rgiduthuri/openvx_tutorial