Multi-video processing applications on FPGA. Lamjed Touil*, Abdessalem Ben Abdelali and Abdellatif Mtibaa

Transcription

1 Int. J. Advanced Media and Communication, Vol. 5, No. 4, Multi-video processing applications on FPGA Lamjed Touil*, Abdessalem Ben Abdelali and Abdellatif Mtibaa Laboratory of Electronics and Microelectronics, University of Monastir, Tunisia *Corresponding author Abstract: With the increasing needs of processing power in video and image processing for advanced media and communication applications, it is mandatory to go further than the software implementation to provide generic, real time, low cost and high performance hardware platforms. In this paper, we present a re-configurable, hardware platform for video and image processing. The proposed system uses the benefits of field programmable gate array (FPGA) to attain this objective. In this context, a prototype system is developed based on the Xilinx Virtex-5 FPGA with the integration of embedded processor, embedded memory, multi-port memory controller (MPMC), standard interfaces, and different other resources. Our system includes different functional modules: video cut detection, video zoom-in and zoom-out. This provides the flexibility of using this system as a general video processing platform according to different application requirements. The final system utilises ~16% of logic resource and 20% of on chip memory. Keywords: real time video processing; multi-video processing; FPGA; field programmable gate array; SOPC; system on programmable chip; MPMC; multi-port memory controller. Reference to this paper should be made as follows: Touil, L., Abdelali, A.B. and Mtibaa, A. (2015) Multi-video processing applications on FPGA, Int. J. Advanced Media and Communication, Vol. 5, No. 4, pp Biographical notes: Lamjed Touil received his degree in Electrical Engineering from ENIM (National Engineering School of Monastir) in 2007 and his DEA in Electronics and Microelectronics degree from FSM (Science Faculty of Monastir) in In 2003, he joined the High Institute of Technology of Sousse (ISET), Tunisia, as Technologist. He received his PhD from ENIM (National Engineering School of Monastir). He is a Member of the Monastir Laboratory of Electronics and microelectronics. His current research interests include FPGA and hardware implementation of video processing applications. Abdessalem Ben Abdelali received his degree in Electrical Engineering and his DEA in Industrial Informatics from the National School of Engineering of Sfax (ENIS), Tunisia, respectively, in 2001 and He received his PhD from ENIS and Burgundy University (BU), France, in Since 2008, he has been working as an Assistant Professor in digital embedded electronic at the high Institute of computer skills and Mathematics of Monastir (ISIMM). Copyright 2015 Inderscience Enterprises Ltd.

2 246 L. Touil et al. His current research interests include reconfigurable architectures and hardware implementation of image and video processing applications. Abdellatif Mtibaa is currently a Professor in Micro-Electronics and Hardware Design at the Department of Electronics of the National School of Engineering of Monastir and Head of Circuits Systems Reconfigurable-ENIM-Group at Electronic and Microelectronic Laboratory. He got a Diploma in Electrical Engineering in 1985 and received his PhD in Electrical Engineering in His current research interests include System on Programmable Chip, high-level synthesis, rapid prototyping and reconfigurable architecture for real-time multimedia applications. 1 Introduction Actually, vision is the most advanced human sense and images plays the most important role in human perception. In fact, video data are becoming very important in many application domains, such as interactive-tv, video-on-demand and multi-media processing tools. Digital Image Processing is one of the emerging frontiers of advanced research and deals with the process of digital camera-based image capturing, conditioning and measuring. So, that important information and features may be extracted from the acquired images. Actually, video processing techniques are used in television sets, VCRs, DVDs, video codecs, video players and other devices. Over the past decade, we have a great technology evolution in this area. The switching of video technology from standard definition (SD) to high definition (HD), which has a maximum resolution of pixels and requires a six time increase in data processing ( literature/wp-video0306.pdf). The new format ultra high definition (UHD) has a resolution of pixels, four times as wide and four times as high then the existing HD. The video surveillance is also changing from the classic CIF format to the D1 standard ( In this context, the utility of video analysis and indexing extends to a wide range of advanced applications (Iakovidis et al., 2003). But content-based video indexing and retrieval exhibits many challenging problems. Methods developed for processing of video for content search can be roughly categorised into temporal segmentation, spatial segmentation and spatio-temporal video segmentation. At this time, descriptive video is done off-line in the industry. Practical real-time video description for broadcast television does not exist yet. In Haynes et al. (2002), a reconfigurable architecture for video image processing was proposed. This architecture can be used to validate a real-time video indexing applications. An application of content-based video indexing to computer-assisted descriptive video was proposed in Gagnon et al. (2006). Another architecture was proposed by Kim et al. (2012). In Li et al. (2009), a general-purpose, multi-task, field programmable gate array (FPGA)-based reconfigurable platform for video and image processing was proposed. The developed system is based on the Xilinx Virtex-II FPGA. It includes several functional modules. Most of the existing designs are focused on implementing specific algorithms for domain-specific applications. There is a need for general purpose hardware platform to support and facilitate complex video and image processing.

3 Multi-video processing applications on FPGA 247 This paper proposes parallel hardware architecture for analysing video and the final system is implemented on an FPGA. This paper shows the usefulness of the reconfigurable technology for parallel applications and especially for testing and supporting new services for new generation of television. Among these services, we include the analysis of the video and the appropriate techniques to provide results of analysis. In this paper, we also present an architecture model to combine some description and analysis tools with several video display techniques. After that we have designed our system on a hardware platform based on Virtex-5 FPGA. In fact, actually FPGAs deliver ASIC-like density and performance, while their flexibility and operational characteristics offer distinct advantages over their ASIC counterparts. As innovative architectures with embedded processors, memory blocks and DSP emerge, more designers are turning to FPGAs for new system-on-chip (SoC) designs. An FPGA offers a compromise between the flexibility of general purpose processors and the hardware-based speed of ASICs. Performance gains are obtained by bypassing the fetch-decode-execute overhead of general purpose processors and by exploiting the inherent parallelism of digital hardware. Therefore, we have selected the FPGA solution to validate our video architecture. Figure 1 presents the basic principle of the proposed architecture. Figure 1 Diagram for insertion of video analysis and display techniques (see online version for colours)

4 248 L. Touil et al. The rest of this paper is organised as follows. In Section 2, we present the dataflow diagram of the proposed model for possible interaction between different video analysis tools and between display techniques. In Section 3, we present the FPGA-based video processing platform. In Section 4, we describe the specification of the implemented applications. In Section 5, we present the experimental results under different video processing applications. Finally, a conclusion and a brief discussion on future research direction are discussed in Section 6. 2 Dataflow diagram of the proposed model An interactive television service is mainly consists of two phases: Analysis phase of AV content and generation display techniques. In this context, we propose a schematic functional architecture model for video processing and insertion of several interactive display techniques. The proposed architecture includes the stage of video analysis and generation of the results with tools and techniques necessary for visual integration and exploitation of these results. Figure 1 provides the schematic functional architecture model of the proposed system. The architecture is composed of two essential parts: The first part deals with the analysis of the video and the second deals with video display techniques. The combination of several analysis techniques at different levels or the coexistence of different low-level treatment are among the most important characteristics of current systems analysis of AV content. Figure 1 gives a block diagram of the dataflow between the various modules, and a generic structure of a system analysis of AV content. It expresses the possible interactions between different tools of content analysis of different levels of abstraction and for different modalities of a video. Each content calls for the development of a specific module with input coming from another module and output possibly feeding a third one. On the basis of the proposed system level architecture and design strategies, a prototype system is developed based on the Xilinx Virtex-5 FPGA. The proposed architecture consists of two blocks: Block (A) and Block (B). Block (A) describes the general video content analysis part. It provides the video analysis into three levels: low-level, mid-level and a high-level treatment. In the low-level treatment, we quote several blocks such as converting to grey scale, converting to HMMD, the histogram calculation, colour and texture descriptors and others. In the intermediate treatment other calculation blocks such as the calculation of the key frame, the similarity between shots and others can be applied. After that other treatment blocks are applied in the high level treatment. Block (B) provides several useful techniques to provide results of video analysis. In fact, the video content analysis results that are generated are then applied to block display validation techniques. The choice of one of the techniques is determined by the user. Several display technologies and exploitation of results are possible. Among the techniques we quote: PIP or Picture In Picture : the ability to watch two pictures (main and sub) simultaneously. The main picture can be displayed in full screen, and the sub picture can be displayed in an inset window.

5 Multi-video processing applications on FPGA 249 POP : Stands for picture-outside-picture POP allows the user to divide the screen into two same-size pictures. Video Mosaic : the ability to display multiple video sources in small size and at the same time on the same TV screen. Zoom-In is an option to increase a part of the image according to the wishes of viewers. Zoom-out is the ability to reduce the size of the image displayed. In this paper, we propose a preliminary version of hardware architecture to support several video processing applications. In this context, we have implemented three different processing functions: video cut detection, video zoom-in and video zoom-out. 3 FPGA-based video processing platform In our current design, we use a Virtex-5 FPGA, the Virtex-5 family (LX110) as the prototype platform. It has ~110,592 logic Cells, 64 DSPE Slices, 4,608 kb of block RAM, 800 IO pins, 1120 Kbits distributed RAM Mbits ( documentation/user_guides/ug200.pdf; C1/VDEC1-rm.pdf). The Virtex-5 family provides a good platform to meet different design requirements (Sangiovanni-Vincentelli et al., 2004). The DDR SDRAM DIMM can support up to 2 Gbytes of RAM. The XUPV5-LX110T Development System board has useful interface ports, RS-232 port, and others. It also has various expansion connectors to expand the usability of this board to meet the requirements of different video and image processing applications. In our system, a video analogue to digital conversion (ADC) board is used to capture the phase alternate line (PAL) signal and digitise it into CCIR 601/656 format. The ADV7183 video decoder uses the I2C bus for configuration information ( xilinx.com/support/documentation/ip_documentation/mpmc.pdf). The particular format used by the ADV7183 is designed to be compatible with the ITU-601/656 video standard and is actually a modified YUV format that uses the YCBCR colour space. The purpose of using this colour space is to separate the data that arrives over the pixel bus in YUV format, so we need to convert the image data into RGB to allow the display on the VGA output. The YUV format consists of three basic components: Y represents the luminance or brightness of a pixel, U represents the colour difference between blue and yellow, and V represents the colour difference between red and cyan. R = Y V, G = Y 0.344U 0.714, B = Y U. To correctly output the external display, the monitor s HSYNC, VSYNC, and BLANK signals must be generated with correct timings for 60 Hz refresh rate. The format for ITU656 pixel stream is shown in Figure 2.

6 250 L. Touil et al. Figure 2 ITU-R BT 656 YUV 4 : 2 : 2 protocol format The start of active video (SAV) and end of active video (EAV) timing signals delimited the beginning and end of a line of pixels, and are to be used. Our major purpose of this system is to prepare hardware platform to provide a general solution for video and image processing, and demonstrate its effectiveness through various application scenarios. From Figure 3, one can see the FPGA implementation of the functional modules. In this section, we use platform-based design methodologies, which use an existing base of components and architectures to reduce design time. Platform-based design (Boussaid et al., 2007) uses IP blocks to build system architectures. It subdivides HW/SW partitioning into behaviour modelling, architecture modelling, and mapping. In this way it supports IP reuse, and derivative design. Video signal is in a clock domain of 27 MHz that includes the Video decoder, and decoder logic. The custom hardware runs at 100 MHz and this domain crossing is handled by the asynchronous double line buffer. The logic used for the decoder was principally carried over from the video Capture module and integrated with the IP_EHD module. The components that were included for decoding were line decoder, 4 : 2 : 2 to 4 : 4 :4, Ycrcb to RGB, Timing generation, De-entrelacement, These modules performed the task of converting the data from composite format to RGB and also kept track of the VSynch and HSynch to determine new frame and new line. Figure 3 Data processing flow of the proposed system

7 Multi-video processing applications on FPGA 251 In this design, we have used the Multi-Port Memory Controller (MPMC) (v6.03.a) (Gribbon and Bailey, 2004) to satisfy the real-time video constraints. The MPMC is relatively new in the Xilinx IP library starting in version It allows multiple buses to be connected to the same piece of memory through different ports, and the ports are able to access the memory in parallel of each other. We used seven ports from eight ports supported by MPMC. The DDR memory runs at 133 MHz, therefore the MPMC handles all clock domain crossing issues between system components (PLB Bus, custom hardware) which all run at 100 MHz. The VGA output is running at 27 MHz. For compatibility reasons, we used the microprocessor soft MicroBlaze. The I2C bus is used to the video decoder configuration. The architecture in Figure 3 provides different functional modules for video processing. Using the MPMC was crucial to the success of the design since we found bottlenecks in transferring frames to and from memory which was solved by opening more ports and using additional buses. In the design, we used PLB bus which is obtained from the Xilinx IP catalogue under the name of plb_v64. The PLB bus provides a connection between an optional numbers of PLB masters and slaves. It became the key transportation of data from one module to another. To eliminate bottleneck to the Memory, we used three PLB buses in our system: Used for the microblaze to communicate with the DDR and IIC. Used between zoom-in-ip, zoom-out-ip, picture in picture and memory. Used by the TFT to get data from the DDR. The bus has not enough bandwidth to handle two transactions. In practice, word cannot be written on every clock cycle of the bus, because the buffers are not always ready. It takes several cycles to setup a burst write/read. As a solution, we created two separate blocks for reading and processing writing. And each block has its own PLB bus connecting to the memory. This gives the system more bandwidth, it also made the design easier to understand and debug. The XPS_TFT controller is an IP core from the EDK library. It connects to the PLB_v4.6 bus and act as a PLB master. The XPS_TFT is used to read the video pixel data from the PLB attached video memory. The XPS_TFT address starts from 0x The PIP block determines the display area of the second video relative to the primary video. It also allows the image blinding. The functional box can be extended in different application scenarios each video layer can be independently displayed at running time. 4 Specification of the treatment examples In the first, we present the block histogram for video cut detection after that we present the zoom-in application. Finally, we present the zoom-out application. 4.1 Block histogram for video cut detection Image information is usually used for the content-based image retrieval. It is mainly used to segment a video by scene. Figure 4 shows a video structure that consists of frames, shots and episodes.

8 252 L. Touil et al. The standard unit of video is a frame. The positions where scene changes occur are called a cut. The shots are separated by cuts. Small video units consist of consecutive shots called episodes. Dividing videos into shots is called video segmentation. Figure 4 Video structure In this paper, we have used a modified method based on local histogram implemented in Jahne (2002). The Process of cut detection implementation is described in Figure 5. It includes the flowing steps: Step 1: the colour in RGB space is converted to YCbCr colour space. Step 2: Dividing the image into 16 blocks. Step 3: calculating the histogram of the current image. Step 4: calculating the histogram difference between the image i and image i + 1. Step 5: If the result exceeds threshold, then we signal a cut video. Figure 5 Block diagram of the proposed system

9 Multi-video processing applications on FPGA 253 The grey scale can be represented only with Y component. We have tested the quantisation effect on cut detection. We have used uniform quantisation into eight levels (bins) (Table 1). We define eight ranges of discrete values. These ranges are represented by one centre value for each one. 0 1 First range (0). 2 4 Second range (1). 5 8 Third range (2) Fourth range (3) Fifth range (4) Sixth range (5) Seventh range (6) Eighth range (7). The process of cut detection based on local histogram is described in Figure 5. It includes four important steps: division of the image into 16 blocks histograms calculation sum of histograms differences of consecutive frames comparison to a predetermined threshold value. The principle of the local video histogram calculation is shown in Figure 6. Figure 6 Histogram calculation We calculate the distance between two images i and i +1 by measuring the Euclidean distance histograms using the following formula: 16 FDM( ii, + 1) = H H +, c n= 1 icn,, i 1, cn, where c presents the number of grey levels. An image histogram shows the distribution of pixel intensities within an image. The architecture of determination of the video cut detection is shown in Figure 7.

10 254 L. Touil et al. Figure 7 The architecture of the histogram calculation unit To evaluate the performances of the shot boundary detection techniques, many evaluation measures can be used (Wang et al., 2013). The most adapted are the following: Precision, Recall and False positives (FP). Recall = Correct / (Correct + False), Precision = Correct / (Correct + Missed), FP = False / reported, where correct, False, and Missed are, respectively, video cuts correctly detected, false detections and not detected cuts. To evaluate the video cut detection technique, different video genres was used: news, football, films, documentary, etc. The obtained results for the different evaluation tests are given in Table 1. The curve represented in Figure 8 demonstrates the capability of the tested technique in term of video cut detection. Table 1 Results obtained with histograms of blocks to eight grey scale levels Video genre Precision (%) Recall (%) FP (%) News Football Documentary Films Publicity Cartoons According to the obtained results, the application of the local histograms technique for video cut detection can assure a good detection rate. This is can be very interesting for real-time constrained application because the use of only eight grey level can considerably decrease the technique complexity and enable its real-time implementation.

11 Multi-video processing applications on FPGA 255 Table 1 shows the results obtained by applying the technique of local histograms with eight quantisation levels. Compared with the results found with high levels of grey sampling, the eight grey levels also give acceptable results, and because real-time constraint, we used the eight levels of grey sampling. Figure 8 Curve distances (sequence (film)) (see online version for colours) 4.2 Image zoom-in description Digital Video image scaling is a full-fledged technology, and can be found everywhere in our daily life and an important variety of algorithms for scaling was proposed in the last decade (Smeaton, 2006). Bilinear Interpolation uses the four nearest pixel values which are located in diagonal direction from that specific pixel to find the appropriate colour intensity value of a desired pixel. In this work, we implement the zoom-in function. Figure 9 illustrates the method that is used to generate new pixels and new lines of the image using the bilinear interpolation method (Wang et al., 2013). Figure 9 Design of the zoom-in function for video processing

12 256 L. Touil et al. First, new pixels between line n and line n + 1 are generated with a combination factor of ½ then, new pixels between the two vertical pixel lines recreated. In our design, two video frame buffers are used: one is used to store the luminance signals and the other one is used to store the chroma signals. An overview of the zoom-in architecture is illustrated in Figure 10. Figure 10 Architecture for zoom-in video processing A FIFO is located at the entrance to facilitate the synchronisation of the treatment. Each two pixels of the same lines contribute to the creation of a new pixel and even each two lines contribute to create a new line. The system is controlled by a finite state machine. The same principle is applied to the chrominance data. 4.3 Image zoom-out description Replication, bilinear and bicubic are the most popular choices of zooming techniques and they are routinely implemented in digital image processing software. For reason of real-time processing, the bilinear interpolation method is used to calculate the new pixels. The incoming images are first passed through a low pass filter to leave only the odd lines. The zoom-out image is the quarter of the original image. The zoom-out function is illustrated by Figure 11. Figure 11 Architecture for zoom-out video processing 5 Experimental results In this section, we demonstrate the effectiveness of the hardware system for different video processing applications. To verify the timing, the system is simulated by Modelsim, and synthesised by Xilinx Integrated Software Environment (ISE). The input video can be either obtained through a camera system or other video devices. Figure 12 shows the results of using the camera system to capture image data. Figure 12(a) shows the original real video processing image and Figure 12(b) illustrates the effects of the zoom-in function. Figure 12(c) shows the result of the greyscale with cut detection algorithm. Figure 12(d) demonstrates all the functional modules

13 Multi-video processing applications on FPGA 257 in the same window, including, the video cut detection zoom-out and the zoom-in function. Figure 12 System performance based on the camera input data: (a) original frame; (b) zoom-in function; (c) grey scale with cut detection result and (d) display of different functions with the same screen (see online version for colours) (a) (b) (c) (d) The design of the application was based on the reuse of a set of IP and the creation of others. The tools used in the implementation of our application are: EDK Embedded Development Kit and ISE. The VHDL language is used to describe the different modules of the proposed system. Table 2 summarises the major resource utilisation characteristics of the final system, from which one see the final system utilises ~16% of logic resource and 20% of memory on chip. Table 2 Resource utilisation of the entire system Hardware resources Available Used Utilisation (%) Number of logic slices 17,280 5, Number of bonded IOBs Number of CLB 69,120 11, Number of logic cells 110,592 14, Number of DSP-48E slices Number of DCM IO banks Block RAM Conclusions and future work In this paper, we have proposed hardware architecture to support treatment analysis of the real-time video processing. It can also support some tools and techniques that are dedicated to provide the results of visual content analysis. The proposed architecture presents a prototype system for the multi-task video and the image processing. System level hardware architecture and detail design strategies are also presented. The final system was implemented using the Xilinx Virtex-5 development system. This system provides a scalable and real-time reconfigurable platform to meet the requirements for many video processing applications. Furthermore, the reconfigurable and extendable characteristics of this system allow it to be easily modified to embed into different video and image processing scenarios experimental results.

14 258 L. Touil et al. In this paper, we have presented a solution based on reconfigurable technology for the validation of processing applications for real-time video. The proposed method presents several advantages: it is generic (the number of treatment may be increased) it uses a relatively advanced technology it addresses both the implementation of applications and video analysis techniques display video it presents a prototyping environment for testing new services dedicated to modern digital television. In the future work, it would be interesting to integrate more complicated video processing modules (Benabdelali and Mtibaa, 2005) into this platform. The EHD descriptor (Touil et al., 2012) can be also implemented in the same architecture. References Benabdelali, A. and Mtibaa, A. (2005) Toward hardware implementation of the compact color descriptor for real time video indexing, Advances in Engineering Software, Vol. 36, No. 7, pp Boussaid, L., Mtibaa, A., Abid, M. and Paindavoine, M. (2007) A real time shot cut detector: Hardware implementation, Computer Standards & Interfaces, Vol. 29, No. 3, pp Gagnon, L., Foucher, S., Lali berte, F., Lalonde, M. and Beaulieu, M. (2006) Toward an application of content-based video indexing to computer-assisted descriptive video, Proc. Canadian Conf. on Computer and Robot Vision, Quebec, Quebec City, Canada. Gribbon, K.T. and Bailey, D.G. (2004) A novel approach to real-time bilinear interpolation, Proceedings of the Second IEEE International Workshop on Electronic Design, Test and Applications (DELTA 2004) / IEEE. Haynes, S.D., Epsom, H.G., Cooper, R.J. and McAlpine, P.L. (2002) UltraSONIC: a reconfigurable architecture for video image processing, Field-Programmable Logic and Applications, LNCS 2438, Springer, pp Iakovidis, D.K., Maroulis, D.E., Karkanis, S.A. and Flaounas, I.N. (2003) Color texture recognition in video sequences using wavelet covariance features and support vector machines, Proceedings of the 29th Euromicro Conference, Antalya, Turkey, September, pp Jahne, B. (2002) Digital Image Processing, 5th ed., Springer, Berlin, pp Kim, D., Jung, J., Nguyen, T.T., Kim, D., Kim, M., Kwon, K H. and Jeon, J.W. (2012) An FPGA based parallel hardware architecture for real-time eye detection, Journal of Semiconductor Technology and Science, Vol.12, No. 2, pp Li, J., He, H., Man, H. and Desai, S. (2009) A general-purpose FPGA-based reconfigurable platform for video and image processing, Advances in Neural Networks ISNN 2009 Lecture Notes in Computer Science, Vol. 5553, pp Sangiovanni-Vincentelli, A., Carloni, L., Bernardinis, F.D. and Sgroi, M. (2004) Benefits and challenges for platform-based design, Proceedings of Design Automation Conference, San Diego, CA, USA, pp Smeaton, A. (2006) Paul Over: Shot Boundary Detection Task Overview, TRECVID-2006.

15 Multi-video processing applications on FPGA 259 Touil, L., Benabdelali, A. and Mtibaa, A. (2012) Hardware acceleration of a real time video processing, Electrotechnical Conference (MELECON), th IEEE Mediterranean, pp Wang, F., Wu, B., Zhang, H. and Liu, H. (2013) Image zoom method based on bandelet transform modified bilinear interpolation, International Conference on Computational and Information Sciences (ICCIS), Chengdu, China, pp Websites Diligent Video Decoder Board (VDEC1) Reference Manual, Revision: 4 December, 2005, Embedded Processor Block Reference Guide, UG200 (v1.8) 24 February, 2010, LogiCORE IP Multi-Port Memory Controller (MPMC) (v6.03.a), DS643, 1 March, 2011, Video and Image Processing Design using FPGAs, Altera Corporation, altera.com/literature/wp-video0306.pdf XUPV5-LX110T MIG Design Creation Using ISE 12.2, MIG 3.5 and ChipScope TM, August,