Design of a Network Camera with an FPGA

Transcription

1 Design of a Network Camera with an FPGA Tiago Filipe Abreu Moura Guedes INESC-ID, Instituto Superior Técnico guedes210@netcabo.pt Abstract This paper describes the development and the implementation of a network camera using a digital camera connected to an FPGA based board. The FPGA supports the implementation of a Web Server running on an embedded processor to serve images captured from the digital camera. The access to the FPGA is done through an Ethernet port. Image processing functionalities have also been included in the design in order to show the functionality and the potential of the developed system to process images before making them available through the Web server. 1. Introduction A network camera is a stand-alone camera that uses a web-browser to view live, full motion video from a computer network, including over the Internet. The camera digitizes the images, processes them and then sends them (on demand or not) over the network. With this solution, the images from the camera can be accessed from any point of the network. The access to the images can be restricted using a password, encryption, etc. Besides using a network camera, users are now asking for high definition cameras for their surveillance solutions that may be defined on demand. To support the changing functionality and the performance requirements of a high definition camera, the system must be programmable and must be capable to support the high performance demands of the applications like compression, encryption, image and video processing, etc. The hardware platforms based on reconfigurable hardware are flexible enough to support the integration of a changing functionality and at the same time guarantee the implementation of systems with high performance. Therefore, they are a privileged target to implement a network camera. The capability to develop complex hardware/ software embedded systems in a FPGA has significantly increased with the advance of the programmable logic technology [1]. Embedded systems with network services and digital signal processing capacities can now be efficiently implemented in a single FPGA. In this project, the design and the implementation of a network camera using an FPGA is described. The embedded system includes an embedded processor running an HTML/1.1 server, an image processing function and the TCP/IP protocols. The server receives the images from the digital camera, processes them and sends them to the network. Section 2 describes the network camera system. Section 3 introduces the hardware system architecture. Section 4 describes the software system architecture. Section 5 presents some results from the system. Finally, section 6 concludes the paper and proposes some future work. 2. Network Camera System The complete system is represented in figure 1. Web S er v er Network E t h er n et I m a g e P r o c es s i n g Interface Fig. 1. System of the network camera connected to the communication network

2 The images received from the camera are processed and then sent to the Web server. The Web server saves the images in the system file and then sends them on request to the users through the communication network. The processing algorithm can be changed depending on the type of application. For example, in a surveillance system used only to detect movement, only the changing images would be stored in the Web server. The digital camera used in the project is from Omnivision (OV9650FSL) [2]. Figure 2 shows an image of the camera. Fig. 2. Omnivision Sensor OV9650FSL The camera includes a sensor and an on-board lens and is targeted to mobile applications where the power consumption and the reduced dimensions are considered important. The camera supports up to pixels in format SXGA/VGA and includes control functions such as color saturation and Gamma control. The interface uses the protocol Serial Camera Control Bus (SCCB) similar to I2C, which allows the programming of all its functionalities. 3. Hardware System Architecture The hardware system consists of one PowerPC processor, SDRAM and SRAM memory, an Ethernet module, an interface with the camera, a VGA interface and a UART port for debug purposes. The camera interface, the UART and the VGD module are connected to a single bus that is connected to the system bus through a bridge (see figure 3). Processor bridge Camera Interface Ethernet UART Memory VGA The system was developed using the ML403 board [3], [4] from Xilinx [5], which contains all the necessary physical components to design the system, namely: 64 MB DDR SDRAM and 8Mb ZBT SRAM Debug support: RS-232 port Ethernet 10/100/1000, RJ-45 port VGA port Expansion connectors with 64 bits FPGA VIRTEX-4 FX (XC4VFX12-FF668-10C) with an embedded PowerPC 405 [6], [7], [8] The PPC405 is an hardwired core. For uniformity reasons, it is possible to create an interface (wrapper) with the embedded processor so that it is seen as any another IP Core. The UART implementing the RS-232 protocol is assured by the OPB UARTLite core. It was configured with a Baud rate of bps, 8 bits per character and without parity. The memory interfaces are implemented using the IP Cores PLB DDR and PLB External Memory Controller (EMC), respectively. Both were configured with a data width of 32 bits - the data width of the bus. The access to the Ethernet was implemented with the PLB Media Access Controller (PLB EMAC) core [9]. The core was configured for 100Mbps with FIFOs to transmit/receive packages up to bits. To debug the processor, the JTAGPPC CNTLR core was used to interface with the dedicated hardware for JTAG. Finally, two shared buses are used to allow the interconnection between the elements. These shared buses are implemented with specific IP Cores: - Bus PLB - Processor Local Bus, indicated to connect the processor and external memories as well as the Ethernet Core - Bus OPB - Peripheral On-Chip Bus, used to connect to other types of Cores, for example user cores, UART, etc. The buses have been configured with a width of 32 bits. The PLB bus has 2 masters and 5 slaves. The OPB bus has 1 master and 3 slaves. The interruptions used with the RS232 and the Ethernet peripherals are controlled using another core the OPB Interrupt Controller (OBP INTC). A few extra cores have been used for the generation of some clock signals of the system, for the initial reset control of the system, and to process the vectors of data to be stored in the RAM memories. Finally, the camera interface has been developed as explained in the following section. Fig. 3. Architecture of the system

3 3.1 Camera Interface The original interface module with the camera was developed previously by the SIPS Group [10] of INESC-ID. The module includes an interface to the sensor and an interface to the VGA port (see figure 4). The module implemented in a FPGA shows the real time image in the monitor after configuration of the camera through the 2 wire protocol SCCB and receives data through dedicated output signals of the camera. Fig. 4. Diagram of the original camera interface module The registers are configured with the codification YUV (8 bits - 4:2:2) and a resolution of For testing purposes, only the component of luminance Y from the YUV codification was considered with a resolution of The core stores the frames in two RAMs of bits each. The original design of the camera interface module had to be modified in order to be used in the Web server. Therefore, a new interface was created. The new interface includes a FIFO between the OPB bus and the new module (see figure 5). OPB FIFO S IP S m o dule SIPS module Fig. 5. New camera interface module The new core includes a FIFO with 512 positions and width of 32 bits (equal to the width of the OPB bus). It also contains a counter, from 0 to 8192, to store the number of elements of a frame already read. If it receives a word from the camera and if the FIFO is not full the new word will be stored in the FIFO and an enable signal will be activated. The data is temporarily stored in one block RAM of the main Core of SIPS, where the counter of the FIFO is the reading address and the enable is the read Enable. The format of each FIFO element is illustrated in Figure 6. Fig. 6. FIFO Data Format of the IP FIFO Bit 21 (enable) is only written for debug purposes, not having any application later in the Web server. The fact that there exist some bits not used is because the FIFO width is 32 bits - the same bit width of the bus. 4. Software System Architecture The software implementation includes the following functionalities: Operative System to allow simultaneous connections through threads; Web Server Application; Not used enable luminance value pixel position File System to store HTML files, among others; Program to capture and to process image frames; Due to some design considerations, namely organization and simplicity, the code and the file system were stored in distinct RAMs: - DDR SDRAM - code of the Application - SRAM MFS file system 4.1 Operating System The operating system adopted was the simple yet very useful operative system Xilkernel from Xilinx. The Xilkernel is an embedded kernel that runs in embedded processors like MicroBlaze and PowerPC [11] 405 with an API POSIX and can be used for high level services (as networking, video, etc.) using IPC services (semaphores and shared memory, for example). The Xilkernel supports pseudo-parallelism and includes mechanisms for coordination - IPC. The kernel was configured to use the UART as the stdin and stdout. Other initial configurations have been necessary to define threads of execution as well as the size of the stack for each thread to be launched. The file system MFS (referred previously) is a part of the entity XMK and is stored on volatile memory. 4.2 Web Server The Web server is based on services of the Xilkernel network services, access to the file system and IPC mechanisms, like semaphores. The Web server initiates the LwIP library, so that the sockets communication becomes available. After that he creates the listening socket at port 80. Whenever there is a connection, the application will

4 look for to answer to the request (of type GET) launching a new thread. The thread will look for the respective file (index.html for default) in its file system, in case of an HTML file request, or will execute an image processing function, in case of an image processing request (to be explained below). If the server is not able to satisfy the request it will send the page of error 404 to the user or the message HTTP 404 of code Not Found in case the file cannot be sent. Fig7. Main page (index.html) of the Web Server Figure 7 illustrates the main page (index.html) of the Web Server. This page presents the image of the camera and the user interface mechanisms with the system, namely: Image mode: real time image or detector of presence; Color: color alternation and luminosity. Fig. 8. Page HTML 404.html Figure 8 illustrates the page 404.html. This page is presented to the user when the Web server cannot satisfy the request of the user. The user interface currently implemented is very simple as its main objective has been to demonstrate the functionality and potential of the system. Other functionalities can be easily added, such as authentication of user and detection of contours, among many others. After receiving the page index.html, the client will visualize the image of the camera that is refreshed automatically with a period of 1 s. The automatic refresh is implemented with the TAG of HTML: <META HTTP-EQUIV= Refresh CONTENT= 1; URL=index.html > This form of automatic update was chosen due to its simplicity and to functionality reasons, including the fact that it doesn t overload the server. 4.3 Capturing and processing the images The image capture is obtained through the sequential reading of 8192 FIFO elements corresponding to the number of elements of a frame. The access to the FIFO is executed through software, using the XIo_In32 (primitive address of the FIFO). When the index.html file is requested, the application will read the FIFO to obtain the information to construct a frame. After capturing the frame, an image will be constructed in the file system. Since there is only the luminance information, the presentation of the image resumes to the tonalities of a color. The format of the image file to include in the Web page is BMP, for its relative simplicity. The file system also contains files with heading and RGB tables for some colors. Available RGB tables are for gray tones, red, blue and green. Figure 9 illustrates an image with tones of gray received from the camera. Fig. 9. Example of an image in gray tones The image can be processed somehow before being sent to the Web server. In this paper, we have implemented a detector of presence algorithm to show the potentiality of the system in terms of image processing. A typical scenario using this mode is the monitoring of static cameras where image alteration will be sent and shown only when there is an important alteration. The algorithm determines if the image captured in real time contains significant differences relatively to the initial captured image. This comparison is not trivial due to the impulsive noise that must be filtered reducing false alarms in a monitoring scenario. The filtering algorithm implemented is known as average filter and has 4 steps: differentiation and binarization; calculation of average X; calculation of the average Y and final classification. The first step consists of the creation of a vector (number of elements of a frame), where each element contains the result of pixel comparison: Image without disturbance with the image received in real time 0; different - 1. Since the pixel information is a luminance scale (0-255), there is a tolerance in the difference between pixels. The

5 implementation of the filter of average in two components - X and Y - corresponds to the implementation of two FIR filters represented respectively by a horizontal mask and a vertical mask. For a simplistic approach, the operation of average corresponds, for each element, to the addition of the neighbour values (in the horizontal line for average X and in the vertical line for average Y) and of the value of the proper element - the number of parcels corresponds to the number of elements of the mask. The final step - classification - consists of the classification of the element as being event or not event - for example 255 and 0 respectively. The criterion has the following formula: Classification i = SENSIBILITY * Element i / MASK 2 (1) where SENSIBILITY is a real value between 0 and 1 (notice that sensitivity increases with the reduction of this value) and MASK is a positive odd value - the horizontal and vertical mask are equal. After the last step, an image with the vector containing the result of the classification is created. In the implemented algorithm, the classification of an element as an event is equal to the value of the luminance of the element in real time, and the classification of not event, is equal to 0. The initial parameters of detection are the following: Average Masks 11 Sensibility 0,7 Luminance Tolerance 6 The illustrative images of this presentation mode are shown below. Figure 10 shows an image that corresponds to the image without disturbance. Figure 11 shows the processed image in detector mode without disturbances and Figure 12 shows the image with disturbances. Fig 10. Background image (without perturbations) As we can see, the image is all black when there are not perturbations. Otherwise, the image will show the perturbation (an open hand in the tests). 4. Results The hardware system has been configured and the application has been programmed using the Embedded Development Kit (EDK) [12] from Xilinx. This framework includes simulation and synthesis tools. The interaction with the main elements (Ethernet port for example) is organized in logical blocks called Intellectual Property (IP) Cores. It is possible to create new IP Cores with a simple user interface, which makes flexible the creation of parameterized Cores. The EDK includes the gcc C compiler and a development environment - Platform Studio Development Software Kit (SDK) [13]. The EDK also offers the possibility of inclusion of an embedded kernel. The EDK includes the entity Xilinx MicroKernel (XMK) which includes: C libraries Software Xilkernel embedded kernel Standalone Board Support Package (BSP) hardware interface base system Memory File System (MFS) file system created in RAM memory LibXil Drivers - drivers for peripherals support LwIP Library which integrates the network functionalities through Ethernet The hardware implementation of the system uses the resources indicated in table 1. HW Object Utilization Slice 4441 (81%) Flip-flop 4616 (42%) LUT 4605 (42%) Input/output 184 BRAM 33 (91%) DCM 2 (50%) Table 1. Hardware resources utilization Fig. 11. Detector Mode (without perturbations) Fig. 12. Detector Mode (with perturbations) The PPC405 processor works at 300 MHz with a global system clock of 100 MHz. Although the global system is limited to 100 MHz, the high frequency of the processor contributes for an increment of the processing performance. Table 2 indicates the occupation of the physical memories. As referred, the SRAM memory contains the file system and the DDR memory, being the main memory, contains the code of the application.

6 File System MFS (SRAM Memory) Maximum Available (KB) 1024 Block Size (bytes) 532 Blocks used/total 43/700 File System Size (bytes) Main Memory (DDR SDRAM) Maximum Available (MB) 64 Used (bytes) Table 2. Memory Occupation SRAM memory is used exclusively for the file system and the occupation size is somewhat irrelevant since the space defined for the file system is enough to store the files required. DDR memory contains the code of the application of the Web server. This occupation is not defined directly in the project. The occupation of the memory is divided in segments (for example stack and heap) and these segments are automatically defined by the EDK according to the implementation parameters of the operative system and the LwIP library. [5] Xilinx, Informação complementar (Xilinx) [6] Xilinx, Virtex-4 User Guide, [7] Xilinx, Virtex-4 Configuration Guide, [8] Xilinx, Virtex-4 Packaging and Pinout Specification, [9] Xilinx, Virtex-4 Embedded Tri-Mode Ethernet MAC User Guide, [10] SIPS INESC-ID, Educational Kit, [11] Xilinx, Getting Started with the PowerPC and MicroBlaze Development Kit - Virtex-4 FX12 Edition, [12] Xilinx, EDK Docs, tform_studio.htm [13] Xilinx, EDK System Simulation, Conclusions The project of a Web server of images from a digital camera running on the ML403 board containing a FPGA Virtex-4 FX with an embedded processor PowerPC 405 demonstrates the potential of the configurable hardware to implement complex embedded systems. This project was developed in two main parts: the hardware design and the software design. The project achieved the proposed objectives, namely to demonstrate the functionality and the potential of a Web server of images from a digital camera, and its implementation in an FPGA. A detection mode was included to emphasize the capability of the system to support the execution of image processing functions. As future work, the system can be improved in order to decrease the use of resources and increase its performance. Also extra functionalities in the scope of image processing (e.g., Sobel) or in the scope of the Web server (e.g., user authentication) should be considered to increase the applicability of the system. References [1] Andraka, FPGA Basics [2] Omnivision, [3] Xilinx, ML40x Getting Started Tutorial, [4] Xilinx, Xilinx ML403 Evaluation Platform Demos and Reference Designs, ce_designs.htm