LOW-COST SCALABLE HOME VIDEO SURVEILLANCE SYSTEM

Transcription

1 Image Processing & Communication, vol. 19, no. 2-3, pp DOI: /ipc LOW-COST SCALABLE HOME VIDEO SURVEILLANCE SYSTEM JAROMIR PRZYBYLO 1 JOANNA GRABSKA-CHRZASTOWSKA 1 PRZEMYSLAW KOROHODA 2 1 AGH University of Science and Technology, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, Department of Automatics and Bioengineering, Al. Mickiewicza 30, Krakow, {przybylo,asior}@agh.edu.pl, 2 AGH University of Science and Technology, Faculty of Computing Science, Electronics and Telecom-munications, Department of Electronics, Al. Mickiewicza 30, Krakow korohoda@uci.agh.edu.pl Abstract. Automated and intelligent video processing and analysis systems are becoming increasingly popular in video surveillance. Such systems must meet a number of requirements, such as threat detection and real-time video recording. Furthermore, they cannot be expensive and must not consume too much energy because they have to operate continuously. The work presented here focuses on building a home video surveillance system matching the household budget and possibly making use of hardware available in the house. Also, it must provide basic functionality (such as video recording and detecting threats) all the time, and allow for a more in-depth analysis when more computing power be available. 1 Introduction Automated and intelligent video surveillance systems play important role in current homeland and facilities security applications [4, 6]. In order to be useful and reliable such systems have to meet several requirements: must be accurate and be able to process a received video stream in real-time. However, due to the computational complexity of image processing algorithms, real-time assumption often is associated with low-level image processing tasks such as background generation and motion detection. Therefore, most commercially available systems have very limited functionality, such as recording video stream (DVR - digital video recorders) and motion detection. In addition, they have the ability to work in a network, which allows /ftp notification of events and view monitored area from a web browser or an application for mobile devices. There are many solutions to the problem of efficiency. Most of them are based on the use of more powerful hardware or hardware acceleration to speed up computations and increase FPS (frame per second) rate. Numerous improvements to selected algorithms, such as the background modelling [11], have been proposed to reduce processing time. Most articles [2, 9] report implementations capable of processing consumer

2 52 J. Przybylo, J. Grabska-Chrzastowska, P. Korohoda video formats in real-time when accelerated with FPGAs. Also, there are solutions leveraging GPU devices widely used in personal computers [7, 8]. Typically, using specialized hardware leads to considerable increase in the cost of such system. On the other hand, in a typical household, there are many different devices, such as PCs, mobile phones, tablets and smart TVs. Not all of them are used all the time and may have left the available computational power. Moreover, they are often connected to each other via a home network. Here, we present a different approach to building a robust home video surveillance system that is designed to make use of various devices available at home to create a distributed image processing system. Since, one cannot assume continuous availability of all devices, the system must be scalable. Basic functionality, such as recording and threat detection, must be ensured all the time. On the other hand, the real-time requirement is not essential for more advanced functionality such as scene understanding, which of course is an important part of the system, but usually can be performed later when the computing power become available. 2 Architecture of video surveillance systems Typical architecture of video surveillance systems [3] is presented in Fig. 1. The first step is acquisition of video stream. Typical systems are available with 4, 8 or 16 cameras and are equipped with analog video input interface (i.e. BNC, 1.0 Vp-p, 75ohm). More modern recorders can work with digital signal sources like IP cameras. They have higher resolution but usually provide highly compressed video stream (which may influence the effectiveness of detection and recognition algorithms). The aim of the detection module is to find changes in a scene that may indicate occurrence of situations that re- Fig. 1: Top level architecture of video surveillance systems quire attention of the operator (for example reporting of violation of the protected zone, motion detection, etc.). Typical tasks that are performed on this level are: image preprocessing (for example noise removal, image enhancement), background estimation and foreground object segmentation or motion detection and video recording. Standard functionality of DVRs consists of detection and executive level - such systems usually record frames when motion is detected on the preselected zone of the scene, send alerts to the operator ( , ftp), and provide easy way of offline event browsing. More sophisticated systems can provide also functionality of basic scene analysis and recognition (i.e. people detection and counting), however they are usually very

3 Image Processing & Communication, vol. 19, no. 2-3, pp expensive and not targeted to homeland use. They perform image analysis such as connected component analysis, feature and object detection, an object tracking and counting. Results of analysis (objects features, trajectory, shape, color...) are then presented to recognition module which is responsible of recognition and situation understanding. All levels use scene information provided by the operator (i.e. scene configuration, region of interests) or created during training phase (classifiers like people Hog detector [5]). 3 Analysis of tasks performed on different levels of the video surveillance system Assuming typical home video surveillance system equipped with 4 analog cameras (VGA resolution: ) and having the following functionality: event detection (motion, moving objects), configurable for per individual camera, object trajectory recording, object recognition, recording (motion frames, alarm/event frames) and event reporting. The selected image processing and analysis algorithms (as described in more detail in [10]) can be used to achieve such functionality (Fig. 2): image preprocessing (de-interlace, noise removal), background estimation and foreground object segmentation, binary image postprocessing, connected component analysis, object tracking, object properties extraction (i.e. size, shape, color, etc.) and object recognition. Since, the deinterlace operator is usually implemented on the hardware framegrabber, the first step of an image preprocessing consists of: median filtering (which helps eliminate noise) and color space conversion (RGB to grayscale). The input to this module is RGB image and the output is grayscale image. Then, successive grayscale image frames are supplied to the background estimation module, based on Mixture of Gaussians algorithm [11]. The result of the background estimation is a foreground binary image. Although, background estimation based on MoG is very robust, still many false foreground pixels exist. Therefore, foreground post-processing is applied to video binary stream (removing small noise pixels and fill small holes inside detected binary objects). This information is then used by a tracking algorithm which is responsible for establishing correspondence among detected objects between subsequent frames. Before that step, the connected component analysis has to be performed. It converts the pixels of the binary mask into labeled components that are part of detected objects. Information about tracked object are then passed to classification algorithm which finds matches between each of them and objects in the database. In other words - it performs recognition. This level of analysis often requires additional information extracted from RGB image (i.e. object properties such as color) and provided by analysis level. The output of the classification algorithm is an index of the recognized object in the database. The amount of data sent between each of described steps are summarized in Tab. 1. It can be noticed that the more top-level part of the system it is more difficult to define exact data transfer requirements. Outputs of individual algorithms are less defined and change according to various types of methods

4 54 J. Przybylo, J. Grabska-Chrzastowska, P. Korohoda Fig. 2: Tasks performed on different levels of a video surveillance system used. On the other hand, bottom-level parts consists of more time consuming algorithms and work on larger portion of image data. 4 Proposed scalable architecture of the surveillance system Given the analysis in the previous chapter, the following system architecture is proposed (Fig. 3). System consists of the following parts: source and preprocessing module, analysis module, recognition module. The primary role of the source module is image acquisition. Since there may be many different video sources, ranging from analog cameras connected via framegrabbers to IP-cameras, it is essential to ensure continuous and uninterrupted data acquisition. In addition, this module is responsible for the basic functionality of video recording (similarly to typical DVR). Depending on the available computing power (hardware configuration) the image preprocessing (RGB2GRAY conversion) and a background generation algorithm can also be implemented in this node. The analysis module is responsible for foreground ob- Fig. 3: Architecture of the proposed scalable video surveillance system ject detection (that comes with: binary image postprocessing and connected component analysis), object tracking and extraction of object properties. It represents higher level of a system functionality such as: determining the speed of moving objects, displaying their path, detection of appearance and disappearance of objects, counting, violation of the protected zone, etc. The last module implements the functionality of a recognition and an understanding of the scene content (i.e. people and car detection, abandoned object detection, vandalism, loitering, burglary, fight). This level makes use of object classification and scene understanding algorithms. The key issue that has to be addressed, is how the data transfer between the modules is handled and module s connection topology. Because source module works on large amount of data and provides RGB (or binary) video stream for other modules, the data transmission has to be deterministic with a minimum latency and jitter guaranteed. On the other hand information from analysis module

5 Image Processing & Communication, vol. 19, no. 2-3, pp Tab. 1: Data size transferred among different parts of the video surveillance system (one camera) Processing step Input format and data size Output format and data size Image acquisition and preprocessing RGB, UINT24: = bytes/frame GRAY, UINT8: Background estimation (MoG) Binary image postprocessing Connected component analysis Object tracking Object s parameters Object recognition GRAY, UINT8: BW, LOGICAL: (or bytes/frame*) BW, LOGICAL: (or bytes/frame*) LABEL, UINT16: = bytes/frame (or bytes/frame**) RGB, UINT24: = bytes/frame LABEL, UINT16: = bytes/frame (or bytes/frame**) PROPERTIES (depending on the classification algorithm) BW, LOGICAL: (or bytes/frame*) BW, LOGICAL: (or bytes/frame*) LABEL, UINT16: = bytes/frame (or bytes/frame**) CENTROID, UINT16: NOB*) 2 = bytes/frame PROPERTIES (depending on the classification algorithm) OBJECTS CLASS *) binary image can be compressed - one byte can describe 8 pixels **) assuming 8-pixel object connectivity, the maximum number of objects on the image frame is round(( ) / 9) = one-pixel objects (which yields 2 bytes per object/pixel), also labelled image can be compressed objects 2 bytes **) NOB, maximum number of tracked objects = 34134

6 56 J. Przybylo, J. Grabska-Chrzastowska, P. Korohoda (such as object detection) can be event driven and does not require full bandwidth all the time. The connection topology is usually determined by a network topology available in the home network - central router to which all devices are connected (either through LAN or wireless). Given the above analysis, we propose that the source and the processing node should work on single PC computer equipped with: multicore processor, high-capacity hard disk and connected through LAN to the home router. Multicore processors allow running multiple tasks maintaining for each of them required computational power. Therefore, image acquisition and video recording can work independently of the background generation. Moreover, RGB data stream can be transferred through shared memory or sockets which provides high bandwidth capacity. Also, recorded video stream can be available to other modules using file-sharing. Both, analysis and recognition modules can be implemented on other devices available at home. Required input data (binary video stream, objects properties, and feedback) can be transferred through network using typical protocols such as UDP or RSTP. Finally, results of recognition and understanding can be reported to an operator through , FTP or SMS. Additionally, all of the modules can record their data on the local or network hard drive for offline browsing. 5 Implementation and results Currently, source and preprocessing modules have been implemented and tested. Also, speed of selected algorithms has been verified on BeagleBoard platform. The source node does image acquisition and video recording. The preprocessing node does: image preprocessing (RGB to GRAY conversion), background estimation and foreground object segmentation - MoG algorithm (parameters: history length = 30, nr. of mixtures = 3, threshold = 16). Both nodes run in separate threads and communicate with each other using simple handshaking protocol. The RGB video stream is transferred from the source to the processing node through shared-memory. Application has been implemented in C++ and compiled as 32-bit windows executable. The following libraries have been used: videoinput library [1] for image acquisition and OpenCV v2.4 library for image processing. Performance tests were executed on two hardware configurations: Time [ms] HW1: Industrial PC, Intel(R) Atom(TM) D525, 1.80GHz, 2GB of RAM, Advantech DVP-7030E PCI Video Capture Card, 4 analog cameras, HW2: Lenovo R400 laptop, Intel(R) Core(TM)2 Duo, 2.10GHz, 3GB of RAM, 4 Logitech 9000Pro USB cameras. HW1 HW2 160x x x Number of cameras Fig. 4: Comparison of average acquisition time for different hardware configurations In order to assess the performance of the proposed architecture, computing time have been collected over 100 frames for both - acquisition and processing nodes. Tests were performed for each configuration for three different resolutions ( , , 640 x480) and for varying number of cameras. Results are presented on Figs. 4 and 5. It may be noted that acquisition time is almost constant, which ensures the correct video recording.

7 Image Processing & Communication, vol. 19, no. 2-3, pp HW1 HW2 160x x x x x x480 Time [ms] FPS Number of cameras Number of cameras Fig. 5: Comparison of average processing time for different hardware configurations The processing time depends on video resolution and number of cameras. The important factor, which should not be omitted is a variability of measured computing time. It has been observed, that processing time on HW1 is more stable than on the HW2, see Figs. 6 and 7. In order to increase the readability of the chart, the processing time has been converted to FPS (frame-per-second) units. FPS x x x Number of cameras Fig. 6: Variability of processing time (FPS) on hardware configurations HW1 Communication among processing nodes has not been implemented yet. Therefore, we cannot assess the performance of whole system. However, in order to check performance of developed software on mobile architectures, several tests has been performed on BeagleBoard xm platform. Beagleboard is equipped with AM37x 1GHz ARM Cortex-A8 compatible processor, similar to the hardware used in a typical household devices (mobile phone, smart TV). Fig. 7: Variability of processing time (FPS) on hardware configurations HW2 Tab. 2: Average acquisition and processing time on BeagleBoard platform (one camera) Image resolution FPS RGB 1.3 FPS RGB 6.5 FPS RGB 10 FPS On this platform, same algorithm as in acquisition and processing nodes has been used - image acquisition from Logitech USB camera, RGB to GRAY conversion, MoG background estimation and foreground object segmentation. Algorithm, has been implemented in MATLAB as Simulink model and then automatically converted to C code using MATLAB Coder and uploaded to Beagleboard platform. Results are presented in Tab. 2. It may be noted that Arm-Cortex based platform can handle only lower resolution video with required speed. However, due to different implementation of MoG algorithm (not optimized to Linux platform) and the method of measuring the speed of algorithm (Hardware-in-the-loop simulation) direct comparison to HW1/HW2 configuration is not possible. On the other hand, this experiment allows for an initial assessment of the performance of mobile platforms. 6 Conclusions and future work The proposed architecture allows scalability and flexibility of a home video surveillance system. Various home

8 58 J. Przybylo, J. Grabska-Chrzastowska, P. Korohoda hardware with a different level of computing power available can be used as its components. Basic functionality, such as recording and threat detection are ensured all the time. The more advanced functionality, such as scene understanding, can be performed when computing power become available. The experimental results show that the presented video processing scheme is suitable for use in video surveillance systems. It has to be noted that the background generation performed on the preprocessing node is time consuming and requires hardware with sufficient computational power. However, such hardware should not be expensive and its power requirements must be low (as it has to work 24 hours per day). This suggests the use of specialized architectures such as FPGA (which is unfortunately expensive) or GPU (high power requirements). Currently, only the basic functionality of the system has been implemented and tested, that is: video acquisition, recording and foreground object segmentation. Further research will focus on the development of other modules, which will perform video analysis and recognition. In addition, future work will focus on ensuring effective communication between the system modules. Acknowledgment This work is supported by AGH University Science and Technology, grant nr (the first and second author) and grant nr (the third author). References [1], videoinput. (2015). school/spring05/videoinput/ (last access: March 2015) [2] Bouwmans, T., Baf, F.E., Vachon, V. (2008). Background Modeling using Mixture of Gaussians for Foreground Detection - A Survey, Recent Patents on Computer Science, 1, [3] Bublinski, Z., et al. (2011). System inteligentnego monitoringu przestrzeni i obiektów szczególnego znaczenia SIMPOZ, PAR Pomiary Automatyka Robotyka, R.15(12), [4] Chmiel, W., et al. (2013). Realization of scenarios for video surveillance, Image Processing & Communications, 17(4), [5] Dalal, N., Triggs, B. (2005). Histograms of oriented gradients for human detection, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1, [6] Devasena, C.L., et al. (2011) Video Surveillance Systems - A Survey, IJCSI International Journal of Computer Science Issues, 8(4), [7] Genovese, M., Napoli, E. (2013). ASIC and FPGA Implementation of the Gaussian Mixture Model Algorithm for real-time segmentation of High Definition video, IEEE Transactions On Very Large Scale Integration (VLSI) Systems [8] Jablonski, M., Przybylo, J. (2014). Evaluation Of Mog Video Segmentation On Gpu-Based Hpc System, Computing and Informatics (to appear) [9] Kryjak, T., Komorkiewicz, M., Gorgon, M. (2011). Implementation of a background generation algorithm with moving object detection and shadow suppressing in Spartan 6 series FPGA devices, Automatyka, AGH UWND, 15(3), [10] Przybylo, J. (2013). Object detection and tracking for low-cost video surveillance system, Image Processing & Communications, 18(2-3), [11] Stauffer, C., Grimson, W. (1999). Adaptive background mixture models for real-time tracking, IEEE Computer Society Conference on Computer Vision and Pattern Recognition,