Real-Time Image and Video Processing: Method and Architecture

Real-Time Image and Video Processing: Method and Architecture 1 Moiching Mayum Pilot Khan, 2 G.Shashibhushan 1,2 Dept. of ECE, Sir.MVIT, Bengaluru Abstract Real time image and video processing is a very demanding task as it needs to perform high computations for a big amount of data represented by the image, and the complex operations, which may need to be performed on the image. are knowing an explosive growth in manufacturing and usage. Designing such systems is becoming critical task, as implementation technology is oriented toward complex functionalities with an increasing time to market pressure. To resolve these issues, we propose, in this paper, the combination of appropriate methodology and architecture to ensure reuse and control over time and cost. We explore the principle of PBD (Platform Based Design) as a design methodology that favors system reuse and allows the exploration of the trade-offs between different design requirements. We have used the PBD to design a real time video acquisition and display module. The proposed module is implemented on Virtex-5 FPGA. Keywords- FPGA; platform based design; embedded systems; real time; image and video processing. I. INTRODUCTION Image processing is becoming widely used in many domains, including industrial, medical imaging manufacturing, and security systems. Real-time image and video processing is a very consuming task in terms of memory and timing that is difficult to realize on a serial processor. This is due to many factors such as the large data set represented by the image, and the complex operations, which may need to be performed on the image. At real-time video rates of 25 frames per second a single operation performed on every pixel of a 768 by 576 colour image (PAL frame) equates to 33 million operations per second. The large quantity and flow of information submitted by the video stream, and the constraints imposed by real-time applications require the solution purely hardware. The use of reconfigurable architectures based FPGA circuits is of great interest since they have the advantage of greater flexibility, with integrated advanced resources and the possibility of parallelization important treatment due to their structure. Embedded systems for real time image and video processing applications are used in many domains such as robotics, mobile phones and others. The growing advances in chip technology have allowed the integration of a wide range of functionalities on the same chip. As a result, embedded SOC designers are facing increasing design requirements like performance, power consumption, flexibility and cost that have to be satisfied. These conflicting requirements have increased hugely that development time and cost become uncontrolled. For the majority of embedded systems, design space complexity and cost have pushed industries to search for more disciplined design process and methodology in order to optimize over cost and time to market. Hence, there is a growing need for flexible solutions which ensures correctness when being implemented and re-use after final implementation with an optimal cost and effort. For IC and SOC manufacturers, the main goal of embedded system design is how to balance production costs with development time while respecting consideration for performance and wanted functionality. As the hardware components have a big impact on production cost, adapting the hardware architecture to the defined product functionalities in order to choose a minimal cost solution and defining a common reusable hardware components and software IP to be shared across several applications may present a suitable solution for cost minimization. Although hardware components present reliable solution for high computational tasks, they are error-prone and characterized by costly and time consuming development cycles. As a result, a trend toward combining hardware and software solutions is becoming more demanded. With the increasing computational power that software processors have known, software solutions are becoming suitable solution as they offer a more flexible implementation compared to hardware solutions which are more constrained and needs a long verification process. Hence, hardware solutions are kept for constrained applications which need huge computation while other applications can be implemented on software for more flexibility. For all the concerns listed above, we conclude the need for a design methodology that grants the following issues: Defining the design process at different abstraction levels. Favors reuse of software libraries and hardware components in future designs. 25

Defines a bounded space of exploration with the respect for time and resource constraints. Manages co-design issues. Allows for early error detections. The platform based design is a design methodology that has been developed to manage design problems. In this approach, a hardware platform is shared among many specific domain applications with tools and methods for design. It is considered as a relevant solution for constraints listed above as it presents a design methodology that can be applied to all abstraction levels during the design process and which include different software resources, intellectual properties (IP) and others. In this paper we exploit the PBD to propose a general reconfigurable platform for real time image and video processing. The rest of the paper is divided as follows: the second section defines the platform based design and its aspects. The third section discusses existing works for real time image and video processing. The forth part explains the proposed real time image and video processing module with application of the platform based design methodology. The fifth part concludes the current work. II. PLATFORM BASED DESIGN The principle of embedded systems is to implement defined functions with the fulfillment of critical constraints like power consumption, performance and weight. The diversity of embedded systems and the wide range of their applications makes the system design a complicated task. With the inability of traditional design methods to cover the systems complexity, the definition of new design methodology is a need. As a result, we have witnessed the birth of system level design [17-18], which is a design methodology that emphasis on features like separation of concerns [9], platform architecture and allows designers to work at a high levels of abstraction. In the literature, there are different system level design methodologies [1-3] that differ according to system requirements, abstraction level, model refinements and other design issues. Bottom-up and top-down are two different system design views used in the design process. In the bottom-up view, there is an abstraction over the architecture, which is designed to support a set of applications. The designer is provided only with relevant features, the details being abstracted. In the topdown view, an instance of the top platform is mapped into an instance of the lower platform. The designer starts from application specification and then, many decisions are made to get an optimal solution that minimizes over architectural costs. The platform-based design [5-9] is a design methodology that was proposed to decrease time to market and enhance product reuse. This methodology does not use nor the bottom-up neither the top-down view; it is defined as a "meeting in the middle" process where refinements of specifications meet with abstractions of potential implementations. The top-down and the bottom-up view converge to form the final system (figure 1). Fig. 1. System platform design flow[6] The PBD methodology allows designing the system at high levels of abstraction without making distinction between hardware and software [8]. As a result, the designer can share designs with a minimal effort. After defining system specifications and requirements, the designer defines the parts that will be implemented as hardware components, the parts that will be implemented as software running on component and the parts realized with reconfigurable hardware. This adds a new dimension to the design space as it deals with codesign issues. This choice is also determined by the target platform. The definition of a platform varies according to the domain of use. For the integrated circuit domain, platform is an integrated circuit where components are programmed through metal customization, electrical modification or software to running applications. For software platforms are a fixed micro-architecture that are flexible enough to be used by a set of applications. From an architectural design context, a platform can be defined as an abstraction over lower layers refinement and higher layers abstractions. A layer is set of components at a defined abstraction level that supports a design stage and allows estimation for performance. Abstraction is an effective solution in system design methodologies as it allows to master complexity and reuse. This abstraction must have no distinction between hardware and software and thus enhances the separation between function, architecture, communication and computation [9]. Working at low levels of abstraction or levels near the implementation is a limiting factor in front of design reuse and early fault detection, that is why it ismore challenging to work at high levels of abstraction. 26

In the platform based design, different abstraction layers have to be defined explicitly (figure 2). The definition of the join point where system definition and implementation meet is very critical in design process. In this particular point meets two distinct platforms: the architecture platform and the API platform. Fig. 2. Interactions between abstraction layers An architecture platform is defined as a specific family of micro architectures, which are based on a library of components (hardware macros, busses, inputs, outputs and others). Choosing the architecture platform is determined by many factors like size of application domain, performance, CPU speed, memory systems and others. To allow applications reuse, software applications has to use high-level interface for the hardware platform, the application program interface. An Application Program Interface (API) platform is an abstraction via a software layer to the architecture platform. This software layer covers essential parts in the architecture platform like programmable cores and input/ output subsystems. Designing an embedded system using high abstraction level with layer separation is a key to control system complexity and enhance reuse of components. In the next paragraphs, we will present some platforms for real time image video processing and then we will present our proposed system for video acquisition and display, which is designed using the PBD approach. III. PLATFORMS FOR VIDEO PROCESSING In the literature, many platforms for image and video processing were proposed. In [10], the authors propose a design and implementation for a general Image and Video Processing Platform (IVPP) using the Synphony C Compiler based on High-level synthesis. The proposed platform allows the plug in of extra modules without modification to the input (capturing video data) and the output (displaying obtained result). The extra blocks are written in the high level C language and then converted into hardware blocks to be incorporated into the IVPP. The proposed hardware/software co-design platform was implemented on a Xilinx Virtex-5 FPGA. Three applications, a canny edge detector, object tracking and motion detector blocks have been designed and tested. The experimental results show low resource consumption for every implemented application. In [11], an architecture based on hard-core and multiple softcore processors has been proposed. The system is dedicated for a faster execution of parallel applications on FPGA. Four systems with diverse shared memory configurations have been implemented for test. The proposed system gives the wanted acceleration only in the case of parallelizable algorithms. In [14], authors propose a methodology to implement real time DSP applications on FPGA by using the Xilinx System Generator (XSG). They proposed an architecture for Edge Detection using Sobel-Filter. The obtained result is satisfactory for both edge detection and resource utilization. Table1 summarizes different works with used platforms. TABLE I. EXISTING WORKS Work Image and video processing platfom using FPGA. Multi core system for faster execution of parallel algorithms. Edge detection using DSP and FPGA. Used platform Virtex-5 OpenSPARC Xilinx Virtex-5 FX30T FPGA Spartan 3A DSP 3400 (3SD3400A- 4FGG676C) Virtex 5 (xc5vlx50-1ff676). IV. PROPOSED ARCHITECTURE FOR REAL TIME IMAGE AND VIDEO PROCESSING In this work, we have used the platform-based design approach to define a platform for real time image and video processing applications. The proposed platform was chosen to enable customization and reuse of software libraries and hardware components. It is defined using various abstraction layers. In the next paragraphs, we will discuss the choice of the hardware platform, the applications implemented as software and the degree of reuse allowed by this platform. A. Hardware platform Field Programmable Gate Arrays (FPGA) are prefabricated silicon based devices that can be electrically programmed [16]. As a reconfigurable architecture, the FPGA allows significant changes to be made to the software running on the platform as well as to the hardware and how it operates. It also allows the exploitation of parallelism, hardware reuse, and reduces system design cycle and cost. In this paper, we have chosen the Virtex-5 LX110 FPGA as architecture platform (figure3) as it presents a good platform to meet different design requirements. It allows IP reuse and customization and gives the possibility to implement both hardware and software solutions. As a case study, we proposed the implementation of a real time video acquisition and display module. 27

To capture the Phase Alternate Line (PAL) signal and digitize it into CCIR 601/656 format, we have used a video Analogue to Digital Conversion (ADC) board. The ADV7183 video decoder uses the I2C bus for configuration (http://www.digilentinc.com). video Analogue to Digital Conversion (ADC) board. The ADV7183 video decoder uses the I2C bus for configuration (http://www.digilentinc.com). Fig. 3. Virtex 5 board Figure 4 shows a simplified block diagram of the virtex 5. The MicroBlaze is a soft core processor that allows the integration of customized user Intellectual Property (IP) cores which can be defined as software algorithms or structural hardware. The Fast Simplex Link is a dedicated MicroBlaze bus system to connect user IP. The PowerPC is a hard core processor that is used for high performance needs. Custom function blocks are dedicated to add user modules in the limit of available resources. Thus, the overall architecture can be enriched by the add of hardware components or software applications. As a case study, we have implemented a real time video acquisition and display system. After defining system specifications of the video acquisition and display system, the following question has to be answered: which modules will be implemented with reconfigurable technology. The hardware modules allow getting advantage form parallelism and high computation offered by hardware while software modules allow more flexibility of implementation. Fig. 4. Simplified block architecture of the Virtex 5 FPGA To capture the Phase Alternate Line (PAL) signal and digitize it into CCIR 601/656 format, we have used a Fig. 5. Hardware platform of video acquisition and display architecture Video decoding operates in a clock domain of 27MHz while the custom hardware runs at 100 MHz.to handle this domain crossing, we used the asynchronous double line buffer. The components included for decoding are "line field decoder", "4:2:2 to 4:4:4 format conversion", "YCrCb to RGB color space conversion", "Timing generation", "De- interlacement" and a TFT controller. These modules are used to convert the data from composite format to RGB. To allow multiple buses to have parallel access to the memory through different ports, we have used a Multi-Port Memory Controller (MPMC). The MPMC is relatively new in the Xilinx IP library, starting in version 10.1. It has 8 ports connected to the memory and thus allows parallel access. We used 4 ports from the 8 ones supported by the MPMC (v6.03.a): two ports for the MicroBlaze, one for the line field decoder and one for 4:2:2 to 4:4:4 block. B. Software platform To implement the application with its various modules we used the Embedded Development Kit EDK, which is the Xilinx suite tools to design embedded programmable systems and processor sub-system component for larger designs. The kit contains tools and IP required designing systems with embedded hard processor cores and/or soft processor cores. It enables the integration of hardware and software components of an embedded system [19] Fig. 5 shows component diagram of the real time video acquisition and display module. The input video source can be a camera system or other video devices. The video analog to digital conversion (ADC) board is used to capture the Phase Alternate Line (PAL) signal and digitize it into CCIR 601/656 format. Commanding this board is assured by the MicroBlaze through a configuration file. 28

C. Results Table 2 summarizes resource utilization characteristics for the implemented system. TABLE II. USED RESSOURCES Used logical resources Number of logic slices 7896 / 69120 Number of DSP-48E Slices 15/ 64 BRAMs number 1285 / 4608 IOBs number 84/ 800 Number of Logic Cells 16058 / 110592 IO Banks 7/23 Number of DCM 3/12 As the results shows, the number of used resources is very low, it is about 11 % form total resources for logic slices, 14% for logic cells which are very low percentages compared with the complexity of the application. Also, the adoption of the PBD approach, gave us the ability to make the conversion between the platform and the specifications of the application. V. CONCLUSION In this work, we have defined and explored the platform based design as a challenging solution for embedded systems design. Then, we have used this methodology to design a real time video processing system. The proposed platform is defined with different abstraction layers in order to enhance reuse. We have used both hardware and software resources to take advantage from acceleration provided by hardware and flexibility of the software. The final system is implemented on a Virtex5 FPGA with the implementation of video acquisition and display module as a use case. The platform can be enhanced by adding others hardware or software modules. REFERENCES [1] AnanduBasu, Saddek Bensalem, Marius Bozga, Jacques Combaz, Mohamad Jaber, Thanh-Hung Nguyen, and Joseph Sifakis, Rigorous Component-Based System Design Using the BIP Framework, IEEE software,vol. 28, No. 3 May/June 2011. [2] Jeff C. Jensen, Danica H. Chang, Edward A. Lee, A Model-Based Design Methodology for Cyber-Physical Systems, International conference on Wireless Communications and Mobile Computing Conference (IWCMC), 2011. [3] Abdoulaye Gamatié, Sébastien Le Beux, Eric Piel,Rab Ben Atitallah, Lamih Anne Etien, Philippe Marquet, Jean-Luc Dekeyser, A Model- Driven Design Framework for Massively Parallel Embedded Systems, ACM Transactions on Embedded Computing Systems, Vol. 10, No. 4, Article 39, November 2011 [4] FrankVahid and Tony. Givargis, Platform tuning for embedded systems design, IEEE Transactions on Computer, 34 (2), 112-114., 2001. [5] Ivan Augé, FrédéricPétrot, François Donnet, and Pascal Gomez, Platform-Based Design From Parallel C Specifications, IEEE transactions on computer-aided design of integrated circuits and systems, VOL. 24, NO. 12, December, 2005. [6] A. S. Vincentelli and G. Martin, Platform based design and software design methodology for embedded systems,ieee Design and Test of computers, 18(6), 23-33, 2001. [7] A.L. Sangiovanni-Vincentelli, L.P. Carloni, F. De Bernardinis, and M. Sgroi,Benefits and Challenges of Platform-Based Design,The Proceedings of the Design Automation Conference (DAC), 2004. [8] L.P. Carloni, F. De Bernardinis, C. Pinello, A.L. Sangiovanni-Vincentelli, and M. Sgroi, Platform- Based Design for Embedded Systems,In R. Zurawski (Ed.), "The Embedded Systems Handbook", CRC Press, 2005. [9] KurtKeutzer,Sharad Malik, A. Richard Newton, Jan M. Rabaey, and A. Sangiovanni-Vincentelli, System-Level Design: Orthogonalization of Concernsand Platform-Based Design, IEEE transactions on computer-aided design of integrated circuits and systems, VOL. 19, NO. 12, 2000. [10] C. Desmouliers et al., HW/SW Co-design Platform for Image and video Processing Applications on Virtex-5 FPGA Using PICO, IEEE International Conference onelectro/information Technology (EIT), 2010. [11] L.Pantaleone and E. Todorovich, Accelerating embedded software processing in an FPGA with PowerPC and Microblaze,National Congress of Computer Science / Information Systems, Argantina, 2013. [12] Jie Li, Haibo He, Hong Man, Sachi Desai, A General-Purpose FPGA-Based Reconfigurable Platform for Video and Image Processing, Advances in Neural Networks ISNN, Springer, 2009. [13] Wajdi Elhamzi Wajdi Elhamzi, Taoufik Saidani, Yahia Said, Mohamed Atri,FPGA Based Real Time Wavelet Video Coding, International Review on Computers and Software (I.RE.CO.S.), 2013. [14] Yahia Said, Taoufik Saidani, Fethi Smach and Mohamed Atri, Real Time Hardware Cosimulation of Edge Detection for Video Processing System, IEEE 16th Mediterranean Electrotechnical Conference (MELECON), 2012. 29

[15] Aleksander Malinowski, Hao Yu, Comparison of Embedded System Design for Industrial Applications, IEEE transactions on industrial informatics, Vol. 7, No. 2, 2011. [16] Ian Kuon, Russell Tessier, Jonathan Rose, FPGA Architecture: Survey and Challenges, Electronic Design Automation, Vol. 2, No. 2, pp. 135 253, 2007. [17] nikolaos s. voros, konstantinos masselos, System Level Design of Reconfigurable Systems-on- Chip, pp 87-105, springer,2005. [18] Andy D. Pimentel Cagkan Erbas, and Simon Polstra, A Systematic Approach to Exploring Embedded System Architectures at Multiple Abstraction Levels, IEEE transactions on computers, vol. 55, no. 2, february 2006. [19] http://www.xilinx.com/support/ documentation/sw_manuals/xilinx13_1/e dk_ctt.pdf 30