How to Accelerate OpenCV Applications with the Zynq-7000 All Programmable SoC using Vivado HLS Video Libraries August 28, 2013

Size: px

Start display at page:

Download "How to Accelerate OpenCV Applications with the Zynq-7000 All Programmable SoC using Vivado HLS Video Libraries August 28, 2013"

Abraham Gallagher
5 years ago
Views:

1 How to Accelerate OpenCV Applications with the Zynq-7000 All Programmable SoC using Vivado HLS Video Libraries August 28, 2013

2 OpenCV Overview Open Source Computer Vision (OpenCV) is widely used to develop Computer Vision applications Library of optimized video functions Optimized for desktop processors and GPUs Tens of thousands users Runs out of the box on ARM processors in Zynq However HD processing with OpenCV is often limited by external memory Memory bandwidth is a bottleneck for performance Memory accesses limit power efficiency Zynq All-programmable SOCs are a great way of implementing embedded computer vision applications High performance and Low Power Page 2

3 Real-Time Computer Vision Applications Computer Vision Applications Real-time Analytics Function Advanced Drivers Assist for Safety Lane or Pedestrian detection Surveillance for Security Friend vs Foe recognition Machine Vision for Quality High velocity object detection Medical Imaging For non invasive surgery Tumor detection Page 3

4 Real-time Video Analytics Processing Pixel based Image Processing and Feature Extraction Frame based Feature processing and decision making 4Kx2K Pixel based Image processing and Feature extraction 1080p F1 F2 F3 720p 480p 100s Ops/pixel 8MPx100 Ops/ frame = 100s Gops 10000s Ops/feature 1000s of features/sec = Mops Page 4

5 Heterogeneous Implementation of Real-time Video Analytics Pixel based Image Processing and Feature Extraction Frame based Feature processing and decision making 4Kx2K Pixel based Image processing and Feature extraction Hardware Domain (FPGA) 1080p 720p 480p F1 Software F2 Domain (ARM) F3 100s Ops/pixel 8MPx100 Ops/ frame = 100s Gops 10000s Ops/feature 1000s of features/sec = Mops Page 5

4Kx2K Pixel based Image processing and Feature extraction 1080p F1 F2 F3 720p 480p 100s

6 Xilinx Real-time Image Analytics Implementation: Zynq All Programmable SoC Pixel based Image Processing and Feature Extraction Frame based Feature processing and decision making 4Kx2K Pixel based Image processing and Feature extraction 1080p F1 F2 F3 720p 480p 100s Ops/pixel 8MPx100 Ops/ frame = 100s Gops 10000s Ops/feature 1000s of features/sec = Mops Page 6

7 Vivado: Productivity gains for OpenCV functions C simulation of HD video algorithm ~1 fps RTL simulation of HD video 1 frame per hour Real-time FPGA implementation up to 60fps Page 7

8 Accelerating OpenCV Applications Driver Assist Broadcast Monitor HD Surveillance Video Conferencing Studio Cinema Camera Frame-level processing Library for PS Pixel processing interfaces and basic functions for analytics Vivado HLS Cinema Projection Digital Signage Office-class MFP Consumer Displays Machine Vision Medical Displays Page 8

9 Zynq Video TRD architecture DDR3 External Memory DDR3 Processing System DDR Memory Controller SD Card Hardened Peripherals Dual Core Cortex-A9 S_AXI_HP 64 bit S_AXI_GP 32b bit AXI4 Stream AXI Interconnect IP Core HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Video access to external memory using 64-bit High Performance ports Control register access using 32-bit General Purpose ports Video streams implemented using AXI4-Stream Page 9

10 IP Centric Design flow Accelerated IP Generation and Integration C based IP Creation User Preferred System Integration Environment C, C++ or SystemC C Libraries System Generator for DSP Floating point mathh Fixed point Video VHDL or Verilog plus SW Drivers Vivado IP Integrator IP Subsystem Xilinx IP 3 rd Party IP User IP Vivado RTL Integration Page 10

11 Page 11

12 Synthesizable Block Synthesized Block Using OpenCV in FPGA designs Pure OpenCV Application Integrated OpenCV Application OpenCV Reference Accelerated OpenCV Application Image File Read (OpenCV) Image File Read (OpenCV) Live Video Input OpenCV2AXIvideo Live Video Input AXIvideo2Mat AXIvideo2Mat OpenCV function chain OpenCV function chain HLS video library function chain HLS video library function chain Mat2AXIvideo Mat2AXIvideo Image File Write (OpenCV) Live Video Output AXIvideo2OpenCV Live Video Output Image File Write (OpenCV) Page 12

13 Pure OpenCV Application Image File Read (OpenCV) DDR3 External Memory DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Image File Write (OpenCV) AXI Interconnect HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 13

14 Pure OpenCV Application Image File Read (OpenCV) DDR3 External Memory 1 DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Image File Write (OpenCV) AXI Interconnect HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 14

15 Pure OpenCV Application Image File Read (OpenCV) DDR3 External Memory DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Image File Write (OpenCV) AXI Interconnect HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 15

Pure OpenCV Application Image File Read (OpenCV) DDR3 External Memory DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory

16 Pure OpenCV Application Image File Read (OpenCV) DDR3 External Memory DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Image File Write (OpenCV) AXI Interconnect HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 16

Integrated OpenCV Application Live Video Input DDR3

SD Card Processing System Hardened Peripherals DDR

Output AXI Interconnect HDMI Video Input AXI VDMA

17 Integrated OpenCV Application Live Video Input DDR3 External Memory DDR3 OpenCV function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Live Video Output AXI Interconnect HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 17

OpenCV Reference / Software Execution Image File Read (OpenCV) DDR3

video library function chain SD Card Processing System Hardened

AXI Interconnect AXIvideo2OpenCV Image File Write (OpenCV) HDMI

18 OpenCV Reference / Software Execution Image File Read (OpenCV) DDR3 External Memory DDR3 OpenCV2AXIvideo AXIvideo2Mat HLS video library function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Mat2AXIvideo AXI Interconnect AXIvideo2OpenCV Image File Write (OpenCV) HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 18

19 OpenCV Reference / In system Test Image File Read (OpenCV) DDR3 External Memory 1 2 DDR3 OpenCV2AXIvideo AXIvideo2Mat HLS video library function chain SD Card Processing System Hardened Peripherals DDR Memory Controller Dual Core Cortex-A9 Mat2AXIvideo AXI Interconnect AXIvideo2OpenCV Image File Write (OpenCV) HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 19

DDR3 DDR Memory Controller Dual Core Cortex-A9 Mat2AXIvideo AXI Interconnect Live

20 Accelerated OpenCV Application Live Video Input AXIvideo2Mat HLS video library function chain SD Card DDR3 External Memory 1 2 Processing System Hardened Peripherals DDR3 DDR Memory Controller Dual Core Cortex-A9 Mat2AXIvideo AXI Interconnect Live Video Output HDMI Video Input AXI VDMA HLS-generated pipeline Xylon Display Controller HDMI Page 20

21 OpenCV design flow OpenCV Block A 1) Develop OpenCV application on Desktop 2) Run OpenCV application on ARM cores without modification OpenCV Block B 3) Abstract FPGA portion using I/O functions 4) Replace OpenCV function calls with synthesizable code OpenCV Block C 5) Run HLS to generate FPGA accelerator 6) Replace call to synthesizable code with call to FPGA accelerator OpenCV Block D Page 21

22 Partitioned OpenCV Application OpenCV Block A opencv2axivideo AXIvideo2HLS OpenCV Block B Synchronization HLS Block B OpenCV Block C OpenCV Block D HLS Block C HLS2AXIvideo AXIvideo2opencv Synthesizable Page 22

23 OpenCV Design Tradeoffs OpenCV-based image processing is built around memory frame buffers Poor access locality -> small caches perform poorly Complex architectures for performance -> higher power Likely good enough for many applications Low resolution or framerate Processing of features or regions of interest in a larger image Streaming architectures give high performance and low power Chaining image processing functions reduces external memory accesses Video-optimized line buffers and window buffers simpler than processor caches Can be implemented with streaming optimizations in HLS Requires conversion of code to be synthesizable

24 HLS Video Libraries OpenCV functions are not directly synthesizable with HLS Dynamic memory allocation Floating point Assumes images are modified in external memory The HLS video library is intended to replace many basic OpenCV functions Similar interfaces and algorithms to OpenCV Focus on image processing functions implemented in FPGA fabric Includes FPGA-specific optimizations Fixed point operations instead of floating point On-chip Linebuffers and window buffers Not necessarily bit-accurate Page 24

Xilinx HLS Video Library 20132 Video Data Modeling Linebuffer class Window class AXI4-Stream IO Functions AXIvideo2Mat Mat2AXIvideo OpenCV Interface Functions cvmat2axivideo AXIvideo2cvMat

25 Xilinx HLS Video Library Video Data Modeling Linebuffer class Window class AXI4-Stream IO Functions AXIvideo2Mat Mat2AXIvideo OpenCV Interface Functions cvmat2axivideo AXIvideo2cvMat cvmat2hlsmat hlsmat2cvmat IplImage2AXIvideo AXIvideo2IplImage IplImage2hlsMat hlsmat2iplimage CvMat2AXIvideo AXIvideo2CvMat CvMat2hlsMat hlsmat2cvmat Video Functions AbsDiff Duplicate MaxS Remap AddS EqualizeHist Mean Resize AddWeighted Erode Merge Scale And FASTX Min Set Avg Filter2D MinMaxLoc Sobel AvgSdv GaussianBlur MinS Split Cmp Harris Mul SubRS CmpS HoughLines2 Not SubS CornerHarris Integral PaintMask Sum CvtColor InitUndistortRectifyMap Range Threshold Dilate Max Reduce Zero For function signatures and descriptions, see the HLS user guide UG 902 Page 25

26 Video Library Functions C++ code contained in hls namespace #include hls_videoh Similar interface, equivalent behavior with OpenCV, eg OpenCV library: HLS video library: cvscale(src, dst, scale, shift); hls::scale<>(src, dst, scale, shift); Some constructor arguments have corresponding or replacement template parameters, eg OpenCV library: HLS video library: cv::mat mat(rows, cols, CV_8UC3); hls::mat<rows, COLS, HLS_8UC3> mat(rows, cols); ROWS and COLS specify the maximum size of an image processed Page 26

27 Video Library Core Structures OpenCV cv::point_<t>, CvPoint cv::size_<t>, CvSize cv::rect_<t>, CvRect HLS Video Library hls::point_<t>, hls::point hls::size_<t>, hls::size hls::rect_<t>, hls::rect cv::scalar_<t>, CvScalar hls::scalar<n, T> cv::mat, IplImage, CvMat hls::mat<rows, COLS, T> cv::mat mat(rows, cols, CV_8UC3); IplImage* img = cvcreateimage(cvsize(cols,rows), IPL_DEPTH_8U, 3); hls::mat<rows, COLS, HLS_8UC3> mat (rows, cols); hls::mat<rows, COLS, HLS_8UC3> img, (rows, cols); hls::mat<rows, COLS, HLS_8UC3> img; hls::window<rows, COLS, T> hls::linebuffer<rows, COLS, T> Page 27

28 Limitations Must replace OpenCV calls with video library functions Frame buffer access not supported through pointers use VDMA and AXI Stream adapter functions Random access not supported data read more than once must be duplicated see hls::duplicate() In-place update not supported eg cvrectangle (img, point1, point2) Read operation OpenCV pix = cv_matat<t>(i,j) pix = cvget2d(cv_img,i,j) HLS Video Library hls_img >> pix Write operation Page 28 cv_matat<t>(i,j) = pix cvset2d(cv_img,i,j,pix) hls_img << pix

29 OpenCV Code One image input, one image output Processed by chain of functions sequentially IplImage* src=cvloadimage("test_1080pbmp"); IplImage* dst=cvcreateimage(cvgetsize(src), src->depth, src->nchannels); Image Read (OpenCV) cvsobel(src, dst, 1, 0); cvsubs(dst, cvscalar(100,100,100), src); cvscale(src, dst, 2, 0); cverode(dst, src); cvdilate(src, dst); OpenCV function chain cvsaveimage("result_1080pbmp", dst); cvreleaseimage(&src); cvreleaseimage(&dst); Image Write (OpenCV) test_opencvcpp Page 29

30 Integrated OpenCV Application System provides pointer to frame buffers Synthesizable code can also be run on ARM void img_process(znq_s32 *rgb_data_in, ZNQ_S32 *rgb_data_out, int height, int width, int stride, int flag_opencv) { // constructing OpenCV interface IplImage* src_dma = cvcreateimageheader(cvsize(width, height), IPL_DEPTH_8U, 4); IplImage* dst_dma = cvcreateimageheader(cvsize(width, height), IPL_DEPTH_8U, 4); src_dma->imagedata = (char*)rgb_data_in; dst_dma->imagedata = (char*)rgb_data_out; src_dma->widthstep = 4 * stride; dst_dma->widthstep = 4 * stride; if (flag_opencv) { opencv_image_filter(src_dma, dst_dma); } else { sw_image_filter(src_dma, dst_dma); } Live Video Input OpenCV function chain Live Video Output cvreleaseimageheader(&src_dma); cvreleaseimageheader(&dst_dma); } img_filtersc Page 30

31 Accelerated with Vivado HLS video library Top level function extracted for HW acceleration #include hls_videoh // header file of HLS video library #include hls_opencvh // header file of OpenCV I/O // typedef video library core structures typedef hls::stream<ap_axiu<32,1,1,1> > typedef hls::scalar<3, uchar> typedef hls::mat<1080,1920,hls_8uc3> AXI_STREAM; RGB_PIXEL; RGB_IMAGE; Image Read (OpenCV) void image_filter(axi_stream& src_axi, AXI_STREAM& dst_axi, int rows, int cols); #include toph IplImage* src=cvloadimage("test_1080pbmp"); IplImage* dst=cvcreateimage(cvgetsize(src), src->depth, src->nchannels); AXI_STREAM src_axi, dst_axi; IplImage2AXIvideo(src, src_axi); image_filter(src_axi, dst_axi, src->height, src->width); toph OpenCV2AXIvideo AXIvideo2Mat HLS video library function chain Mat2AXIvideo AXIvideo2OpenCV AXIvideo2IplImage(dst_axi, dst); cvsaveimage("result_1080pbmp", dst); cvreleaseimage(&src); cvreleaseimage(&dst); Page 31 testcpp Image Write (OpenCV)

32 Accelerated with Vivado HLS video library HW Synthesizable Block for FPGA acceleration Consist of video library function and interfaces Replace OpenCV function with similar function in hls namespace void image_filter(axi_stream& input, AXI_STREAM& output, int rows, int cols) { //Create AXI streaming interfaces for the core #pragma HLS RESOURCE variable=input core=axis metadata="-bus_bundle INPUT_STREAM" #pragma HLS RESOURCE variable=output core=axis metadata="-bus_bundle OUTPUT_STREAM" #pragma HLS RESOURCE variable=rows core=axi_slave metadata="-bus_bundle CONTROL_BUS" #pragma HLS RESOURCE variable=cols core=axi_slave metadata="-bus_bundle CONTROL_BUS" #pragma HLS RESOURCE variable=return core=axi_slave metadata="-bus_bundle CONTROL_BUS" #pragma HLS INTERFACE ap_stable port=rows #pragma HLS INTERFACE ap_stable port=cols RGB_IMAGE img_0(rows, cols), img_1(rows, cols), img_2(rows, cols); RGB_IMAGE img_3(rows, cols), img_4(rows, cols), img_5(rows, cols); RGB_PIXEL pix(50, 50, 50); #pragma HLS dataflow hls::axivideo2mat(input, img_0); hls::sobel<1,0,3>(img_0, img_1); hls::subs(img_1, pix, img_2); hls::scale(img_2, img_3, 2, 0); hls::erode(img_3, img_4); hls::dilate(img_4, img_5); hls::mat2axivideo(img_5, output); } topcpp Image Read (OpenCV) OpenCV2AXIvideo AXIvideo2Mat HLS video library function chain Mat2AXIvideo AXIvideo2OpenCV Image Write (OpenCV) Page 32

$<0x1>; }; Live Video Input Call from userspace after mmap() Ximage_filter xsfilter; int fd_uio = 0; AXIvideo2Mat if ((fd_uio = open("/dev/uio0", O_RDWR)) < 0) { printf("uio: Cannot open device$

33 Using Linux Userspace API Modify device tree to include register map { compatible = "xlnx,generic-hls"; reg = <0x400d0000 0xffff>; interrupts = <0x0 0x37 0x4>; interrupt-parent = <0x1>; }; Live Video Input Call from userspace after mmap() Ximage_filter xsfilter; int fd_uio = 0; AXIvideo2Mat if ((fd_uio = open("/dev/uio0", O_RDWR)) < 0) { printf("uio: Cannot open device node\n"); } xsfiltercontrol_bus_baseaddress = (u32)mmap(null, XSOBEL_FILTER_CONTROL_BUS_SIZE, PROT_READ PROT_WRITE, MAP_SHARED, fd_uio, 0); xsfilterisready = XIL_COMPONENT_IS_READY; HLS video library function chain Mat2AXIvideo Live Video Output // init the configuration for image filter XImage_filter_SetRows(&xsfilter, sobel_configurationheight); XImage_filter_SetCols(&xsfilter, sobel_configurationwidth); XImage_filter_EnableAutoRestart(&xsfilter); XImage_filter_Start(&xsfilter); Page 33

34 HLS Directives for Video Processing Assign input to be an AXI4 stream named INPUT_STREAM #pragma HLS RESOURCE variable=input core=axis metadata="-bus_bundle INPUT_STREAM" Assign control interface to an AXI4-Lite interface #pragma HLS RESOURCE variable=return core=axi_slave metadata="-bus_bundle CONTROL_BUS" Assign rows to be accessible through the AXI4-Lite interface #pragma HLS RESOURCE variable=rows core=axi_slave metadata="-bus_bundle CONTROL_BUS" Declare that rows will not be changed during the execution of the function #pragma HLS INTERFACE ap_stable port=rows Enable streaming dataflow optimizations #pragma HLS dataflow Page 34

35 A more complex OpenCV example: fast-corners This code is not streaming and must be rewritten Random access and in-place operation on dst void opencv_image_filter(iplimage* img, IplImage* dst ) { IplImage* gray = cvcreateimage(cvsize(img->width,img->height), 8, 1 ); cvcvtcolor( img, gray, CV_BGR2GRAY ); std::vector<cv::keypoint> keypoints; cv::mat gray_mat(gray,0); cv::fast(gray_mat, keypoints, 20,true ); int rect=2; cvcopy(img,dst); for (int i=0; i<keypointssize(); i++) { cvrectangle(dst, cvpoint(keypoints[i]ptx,keypoints[i]pty), cvpoint(keypoints[i]ptx+rect,keypoints[i]pty+rect), cvscalar(255,0,0),1); } cvreleaseimage( &gray ); } opencv_topcpp Page 35

36 A more complex OpenCV example: fast-corners This code is streaming Note that function correspondence is not 1:1! void opencv_image_filter(iplimage* src, IplImage* dst) { IplImage* gray = cvcreateimage( cvgetsize(src), 8, 1 ); IplImage* mask = cvcreateimage( cvgetsize(src), 8, 1 ); IplImage* dmask = cvcreateimage( cvgetsize(src), 8, 1 ); std::vector<cv::keypoint> keypoints; cv::mat gray_mat(gray,0); cvcvtcolor(src, gray, CV_BGR2GRAY ); cv::fast(gray_mat, keypoints, 20, true); GenMask(mask, keypoints); cvdilate(mask,dmask); cvcopy(src,dst); PrintMask(dst,dmask,cvScalar(255,0,0)); hls::fastx hls::paintmask cvreleaseimage( &mask ); cvreleaseimage( &dmask ); cvreleaseimage( &gray ); } opencv_topcpp Page 36

37 A more complex OpenCV example: fast-corners Synthesizable code Note #pragma HLS stream hls::mat<max_height,max_width,hls_8uc3> _src(rows,cols); hls::mat<max_height,max_width,hls_8uc3> _dst(rows,cols); hls::axivideo2mat(input, _src); hls::mat<max_height,max_width,hls_8uc3> src0(rows,cols); hls::mat<max_height,max_width,hls_8uc3> src1(rows,cols); #pragma HLS stream depth=20000 variable=src1data_stream hls::mat<max_height,max_width,hls_8uc1> mask(rows,cols); hls::mat<max_height,max_width,hls_8uc1> dmask(rows,cols); hls::scalar<3,unsigned char> color(255,0,0); hls::duplicate(_src,src0,src1); hls::mat<max_height,max_width,hls_8uc1> gray(rows,cols); hls::cvtcolor<hls_bgr2gray>(src0,gray); hls::fastx(gray,mask,20,true); hls::dilate(mask,dmask); hls::paintmask(src1,dmask,_dst,color); hls::mat2axivideo(_dst, output); topcpp Page 37

38 Streams and Reconvergent paths hls::mat conceptually represents a whole image, but is implemented as a stream of pixels template<int ROWS, int COLS, int T> class Mat { public: HLS_SIZE_T rows, cols; hls::stream<hls_tname(t)> data_stream[hls_mat_cn(t)]; }; hls_video_coreh Fast-corners contains a reconvergent path The stream of pixels for src1 must include enough buffering to match the delay through FASTX and Dilate (approximately 10 video lines * 1920 pixels) CvtColor FASTX Dilate PaintMask src1 #pragma HLS stream depth=20000 variable=src1data_stream Page 38

39 Performance Analysis AXI Performance Monitor collects statistics on memory bandwidth see /mnt/axi_perfmonlog Video + fast corners 1920*1080*60*32 = ~4 Gb/s per stream HP0: Read 401 Gb/s, Write 401 Gb/s, Total 803 Gb/s HP2: Read 401 Gb/s, Write 401 Gb/s, Total 803 Gb/s Page 39

40 Power Analysis Voltage and Current can be read from the digital power regulators on the ZC702 board Custom, realtime HD video processing in 2-3 Watts total system power FASTX is less than 200 mw incremental power Page Active Idle Idle + Video Fast Corners + video DDR PL IO PL core PS IO PS core

41 HLS and Zynq accelerates OpenCV apps OpenCV functions enable fast prototyping of Computer Vision algorithms Computer Vision applications are inherently heterogenous and require a mix HW and SW implementation Vivado HLS video library accelerates mapping of opencv functions to FPGA programmable fabric Zynq offers power-optimized integrated solution with high performance programmable logic and embedded ARM Page 41

42 Additional OpenCV Collateral at Xilinxcom Download XAPP1167 from Xilinxcom QuickTake: Leveraging OpenCV and High-Level Synthesis with Vivado Page 42

Multimedia Retrieval Exercise Course 2 Basic of Image Processing by OpenCV

Multimedia Retrieval Exercise Course 2 Basic of Image Processing by OpenCV Kimiaki Shirahama, D.E. Research Group for Pattern Recognition Institute for Vision and Graphics University of Siegen, Germany