A Multisensor Surveillance System for Automated Border Control (egate)

Transcription

1 Workshop on Activity Monitoring by Multiple Distributed Sensing (AMMDS) in conjunction with th IEEE International Conference on Advanced Video and Signal Based Surveillance A Multisensor Surveillance System for Automated Border Control (egate) David Schreiber, Andreas Kriechbaum, Michael Rauter AIT Austrian Institute of Technology Video- and Security Technology, Donau-City-Straße 1, 1220 Vienna, Austria david.schreiber@ait.ac.at Abstract This paper presents 1 a multisensor surveillance system used inside an Automated Border Control (ABC) system (more specifically, an egate). The system consists of two parts: counting the number of persons inside the egate (person separation), ensuring that no more than one passenger is present; left luggage detection, ensuring that the passenger did not leave any item inside the egate. These tasks are performed using a top-view mounted sensor inside the egate, consisting of a trinocular camera configuration comprised of a monochrome stereo setup which delivers depth information, and a color camera mounted in-between, capturing color information. In contrast to already existing ABC solutions, which mostly use electronic sensors, e.g. simple beam technology, for person separation and left item detection, we introduce vision based technologies to elevate the security of such systems to a higher level, also increasing the usability for the border guard. The system achieves real time and had been demonstrated and evaluated at the Vienna International Airport. 1. Introduction Increasing worldwide travel capacities at airports poses new challenges in the area of border and security control. The proposed Automated Border Control system shall accelerate the border control process by increasing the passenger throughput while maintaining the highest level of security. The developed user-friendly, throughput optimized, automated border crossing prototype operating at the highest security level is currently demonstrated and evaluated at the Vienna International Airport (Terminal 2, non-schengen Arrival). The developed egate is a specific type of an ABC system, characterized by a one-step process and two doors. The developed system architecture is an interconnected 1 Co-funded by the Austrian Security Research Programme KIRAS ( an initiative of the Federal Ministry for Transport, Innovation and Technology, Austria (bmvit, application chain with the following components (see Fig. 1): Transit area with two doors representing the physical barrier of the border; Passport reader for document authentication; Video verification (1:1 face comparison in a live stream) for identity authentication; Video surveillance for reliably counting the number of persons present inside the egate. Furthermore, the left luggage detection ensures that the person exiting the egate did not leave behind any of its personal belongings. Person counting in front of the egate, for optimizing the number of egate operating at any given moment; Connection to the national security database and the certificate server, which are managed by the authority; Monitoring interface for trained border guards to visualize the process and traveler-specific data. The process of using the egate in cooperation with the necessary technologies is depicted in Fig. 2. The traveler places his epassport on the passport reader, which then authenticates the epassport, including electronic and optical security checks. If reading has succeeded, the first egate s door opens automatically, and the passenger goes through the egate. During his walk, recorded live images of his face are captured, and compared against the picture stored in the chip of his epassport. In addition, a security check is performed against the Schengen Information System (SIS). At the same time, a surveillance system (top view sensor) ensures that only a single person is present inside the egate (person separation). Once the identity of the (single) passenger has been authenticated, the second door opens automatically, and the passenger steps out of the egate. The opening of the second doors activates the left luggage detection module. In case that any item was left behind in the egate, the second door opens again, enabling the passenger to return and pickup his luggage. There are already existing ABC solutions that are still highly diverse: ABC systems currently on the market differ in a multitude of aspects such as type of biometrics, topology (e.g. with/without kiosk), number of doors, overall system and process layout, guidance of users, etc. Considerable efforts have already been made to establish best practice guidelines and recommendations by FRONTEX and a limited number of member European states [3]. Most current systems use electronic sensors, e.g. simple beam technology, for person separation and left /13/$ IEEE 432

2 item detection. In this paper, we introduce vision based technologies to elevate the security of such systems to a higher level, also increasing the usability for the border guard. respect to embedded real-time systems in software. A detailed performance analysis of the algorithm was given in [7] for optimized reference implementations on various commercial of the shelf (COTS) platforms, e.g. a PC, a DSP and a GPU. The above sensor is mounted in the egate in a top-view position, 279 cm from the ground, yielding 0.97 cm per pixel visual resolution on the ground floor plane, and 2 cm depth resolution. In the given resolution, the sensor delivers 15 fps (The bottleneck is the USB connection of the sensor to the PC via a single host controller.) The rest of this paper is organized as follows. Section 2 presents the person separation method, while section 3, the left luggage detection algorithm. Each section includes literature survey, an outline of the method and experimental results. Section 4 contains our conclusions. 2. Person separation (CPU) Fig. 1. The developed system architecture, with the following components: Transit area, Passport reader, Video verification, Video surveillance, Connection to the national security database and the certificate server, Monitoring interface. Fig. 2. The process of using the egate in cooperation with the necessary technologies. In this paper, we focus on the surveillance system inside the egate, performing person separation and left luggage detection. We use an in-house developed sensor to extract color as well as depth information, employing canonical stereo setup (monochrome cameras mounted in parallel), with a baseline of 0.2m. A third color camera is mounted in-between, used for capturing color information. The board-level industrial cameras have a USB 2 interface and the resolution of the sensor is 752x480, resampled to 608x328 (WVGA), with 8 bits. This trinocular camera setup is calibrated offline. The stereo matching process outputs depth data alongside with a rectified color image, congruent to the depth image. Depth information is computed via Census-based stereo matching algorithm, described in [7], which is an explicit adaption and optimization of the well-known Census transform in Human counting is a crucial part of the egate system. When multiple passengers try to pass through the egate simultaneously, using a single passport, an alert must be forwarded to a border guard. To achieve usability in the real-world, false alarm rate should be kept low. In particular, the system should not confuse luggage or other objects for a person. Mounting the visual sensor in a top-view configuration, the problem of humans partially occluding each other is drastically decreased. Systems using traditional color cameras suffer from further difficulties, such as sensitivity to illumination changes and shadows or reflections in the scene. Using depth information decreases or even solves entirely these types of problems. Previous literature dealing with the topic of human detection, tracking and counting using depth data employ varying setups with both oblique and top view sensors. We first briefly outline methods employing depth information from oblique view setups. In [11], foreground blobs are detected by background subtraction. The location of humans in the scene is estimated from depth information of each foreground blob. Head detection is done without using stereo information, by applying a partial-ellipse fitting algorithm. In [9], the image is segmented into different disparity layers which are used in a contour fitting step. Finally, a tracking step is performed. In [6], stereo-based human head detection is performed using scale-adaptive filters, and mean-shift algorithm is used to localize heads in the likelihood map. In [16], a method for human detection is presented which performs a scalespace search with a combination of histogram of oriented depths features and color data. In [20], Histogram of Depth Difference is proposed as a new feature descriptor. A Support Vector Machine (SVM) classifier is trained with this descriptor. In [22], the Simplified Local Ternary 433

3 Pattern feature descriptor is presented. The detection window is partitioned into non-overlapping blocks. A histogram is build from these blocks. Other approaches employ top view sensor mounting. In [1], a 3D volume of interest, in head height, is investigated to locate heads. Resampling of the depth image from a vertical projection into an occupancy map is done to remove perspective distortions. A Gaussian mixture model is applied to the occupancy map, to locate the humans. In [21], the depth map is segmented into several height intervals, generating binary image for each depth interval. Morphological operations, such as openings with circular structuring elements, are used to locate the human heads. Finally, the heads are tracked. A Kalman filter is employed for predicting the head positions. In [10], an adaptive background model is used to extract foreground regions. On these foreground regions, a spherical crust template is matched. They determine the number of heads and their position and height in a foreground blob. A blob separation step splits the foreground blobs which contain more than one head. A tracker based on a set of Kalman filters is then being fed with the detections Our method Our system receives depth information provided by the top view mounted sensor. Knowledge of the sensor s height relative to the floor enables us to determine the height of an object in 3D world coordinates. The depth image contains invalid pixels (no valid depth information available), with invalid depth values set to depth equal zero. The depth image contains depth values measured relative to the sensor. These are inverted as to be relative to the egate s floor, and depth values which exceed the sensor s height relative to the floor are clipped. Thus, instead of searching for minima in the depth image, we search for maxima, followed by a non-maxima suppression algorithm. Our optimized non-maxima suppression algorithm is an extension of a recent work reported in [12], where the algorithm locates 1-D peaks along the images scan-line and compares each of these peaks against its 2-D neighborhood in a spiral scan order. We have extended the first solution offered by [12], namely by, first, using a rectangular object window rather than a squared one; second, we enable a variable scale object window (due to perspective effects); finally, we detect plateaus in addition to regular peaks. We use the following method for human detection, tracking and counting (for more details, see [14]). In the first step, we search the depth map for local maxima. The maxima are used as seed points for a head localization algorithm. The centers of the head candidates are found in this step using a gradient climbing algorithm. They are handed over to the head detector. The head detector is based on an SVM classifier, trained on histograms of depth difference features. In a final step, a tracking algorithm aggregates the detections over time and computes trajectories for detected heads. Association costs for 3 different tracking strategies are compared, and the best association configuration is picked. Euclidean distance is used as distance metric. The different strategies are: (1) nearest neighbor association, (2) association of detections with minimum number of candidate tracks first, (3) association of tracks with minimum number of candidate detections first. Note that our algorithm does not depend on background modeling for foreground/background segmentation. Therefore, it does not suffer from typical shortcomings related to background-based methods, e.g. a person that stands still for a while and eventually being integrated into the background. Furthermore, incorrect foreground segmentation can be the consequence of an abrupt short-term illumination/lighting change. In Fig. 3, the resulting human trajectories of our algorithm are shown. As can be seen, even persons walking close to each other (piggy-backing) are detected and tracked. Each track gets a unique identifier assigned. Furthermore, the calculated height of the persons in centimeters is shown. In the upper left corner, the current person count is presented Implementation details and results We used C++ to implement our algorithm. The Intel Integrated Performance Primitives library was used for fast image operations. For classification, the LIBSVM [2] was used, employing radial basis functions kernel. The optimized parameters for the kernel were found via a grid search. The test system consisted of an Intel Xeon CPU with 4 physical and 4 virtual 2.93 GHz, 12 GB RAM running on Windows7 64-bit machine. The runtime performance of our algorithm is shown in table 1, averaged over 1000 frames. The most time consuming algorithmic part is the head hypothesis generation (1.77msec). This is hardly surprising, since it involves both the maxima search as well as the gradient climbing algorithm. Especially the maxima search involves scanning the whole image, which is computationally intensive. Computation time of the head classification is moderate. This is due to the fact that the head hypothesis generation reduces the number of head candidates drastically. Hence, the remaining processing time for this step is about 1msec. The depth values conversion takes up 0 5msec. As one can see from Table 1, the time consumption of the tracking step is negligible. We have evaluated our detector on a test data set of 5184 positive and 7344 negative samples. A true positive detection rate of 0.93 and a true negative detection rate of 0.99 were achieved on the test data set (see [14]). 434

4 Fig. 3: Human trajectories and count computed by our system Table 1: Average runtime of the person separation algorithm. Algorithmic step Computation time Conversion of depth values 0.52 milliseconds Generation of head hypothesis 1.77 milliseconds Head classification 1.13 milliseconds Association/tracking 0.01 milliseconds Total 3.43 milliseconds 3. Left luggage detection (CPU-GPU) Most of the proposed techniques for abandoned object detection rely on tracking informationn to detect drop-off events, while fusing information from multiple cameras [18]. Only few techniques are concerned with abandoned object detection based on a single visual camera. [13] proposed a single camera, non-tracking-based system, which makes use of two backgrounds for detection of stationary objects. The two backgrounds are constructed by sampling the input video at different frame rates (one for short-term and another for long-term events). The ObjectVideo surveillance system [19] keeps track of background regions which are stored right before they are covered by an abandoned object. In case the same object is removed, the stored region can be matched with the current frame to determine that the object was removed. In addition, the system relies on analyzing the edge energy associated with the boundaries of the foreground region for both the current frame and the background model. In [18], a framework to detect abandoned and removed objects is presented, using a mixture of three Gaussians for each pixel in the image. It is assumed that the 1st Gaussian distribution shows the persistent pixels and represents the background image, the repetitive variations and the relative stationary regions are updatedd to the 2 nd Gaussian distribution, while the 3rd Gaussian represents the pixels with quick changes. Accordingly, if the weight of the 2nd Gaussian for a pixel is larger than the threshold, the pixel belongs to the static region. We are not aware of any previous work which employs depth information in the context of abandoned object detection, and none which combines intensity and depth information for the same purpose. In fact, the restricted field of view of the egate s ground floor (180x82 cm), and the height of the top-view mounted sensor relative to the ground (279 cm) make it an ideal scenario for using accurate depth information. Moreover, we combine color and depth cues in order to be able to detect left objects based on either color (e.g. passport), or texture (e.g. gray laptop having similar color to the floor), or based on both cues. Our left luggage detection module is based on the fusion of two separate background subtraction modalities, namely color and depth. Both modalities employ the same GPU-based non-parametric background subtraction algorithm reported in [15], which is extended in the present paper to distinguish between moving and static objects GPU-based background subtraction Most background modeling methods are pixel-based. However, the intensity values of a pixel cannot be modeled properly by a single unimodal distribution. To handle lighting changes, repetitive background motion, and introducing or removing objects from the scene, the Mixture of Gaussians (MoG) approach was introduced in [17], where each background pixel is modeled by a distribution composed of a fixed number of Gaussians. MoG has been widely incorporated into various algorithms, including color, gradient and dense depth data [5]. However, as the distribution of pixel values over time cannot be modeled accurately by a mixture of Gaussians, a non-parametric kernel density estimation method for pixel-wise background modeling was proposed [3]. However, the method is very memory and time consuming. To overcome computational limitation of this technique, [8] proposed a compressed non-parametric representation using a codebook model. Samples at each pixel are clustered into a set of code-words based on a color distortion metric together with brightness bounds. [15] presented a background modeling algorithm for a practical surveillance system that utilizes a compressed non-parametric representation, significantly simplifying the work of [8]. The algorithm is simple, efficient, yet robust, and was successfully ported to GPU, using CUDA, achieving ultra real-time. Moreover, the time it takes until a stopped object is absorbed into the background (integration time) is controllable by the user, and is stable regardless of the complexity of the scene. The distribution of a pixel s history is captured by a set of codes, where a maximal number of 6 codes is sufficient to model outdoor 435

5 videos. Codes lying in the tail of the distribution are classified as foreground pixels Our method Our left luggage detection module is based on the fusion of color- and depth-based background subtraction algorithms, both employing the same compressed nonparametric background subtraction method [15], extended in the present work to distinguish between moving and static objects. The two backgrounds are parameterized similarly as described in [15], except for the sensitivity d of the depth-based background which is set to equal 1 pixel (2 cm of depth). The depth image contains depth values measured relative to the sensor. These are inverted as to be relative to the egate s ground floor, and depth values which exceed the sensor s height (279 cm) relative to the floor are clipped. The intensity and depth images are cropped to 62x192 sub-images, which contain only the floor s area, with image resolution of 0.97 cm per pixel. The depth image contains invalid pixels with undefined depth values, set to depth equal zero. The temporal consistency of the invalid pixels depends on the scene. The gray carpet covering the egate s floor contains low texture and is susceptible to shadows and light reflections, caused by the motion of passengers. Hence, invalid pixels associated with the floor are occasionally set to non-zero values. To handle invalid pixel fluctuations, we use a fuzzy definition of invalidness, maintaining an adaptive count on each pixel s history. A pixel is considered invalid if in the current frame it has zero value, or if it has reappeared as invalid often enough recently (above a threshold). Each of the two backgrounds, color- and depth-based, produces its own foreground-background segmentation. Next, the algorithm distinguishes between moving and static objects for each one of the two segmentations. Again, we maintain an adaptive count on each pixel s history. A foreground pixel is considered as static provided that it persistently reappeared as foreground often enough (above a threshold). The static foreground image is then filtered against small noisy areas using a median filter. Finally, the two static foreground images obtained for color and depth cues are fused (via a union operation). Next, blob extraction (connected component labeling) is performed on the fused static foreground image, and too small foreground regions are rejected. To increase robustness even further, we synchronize the left luggage detection module with the egate s door signals. Thus, background updating is blocked during the period where the passenger is inside the egate, and detection of left items is activated for a short period of time after the person has left the egate Implementation details and results Due the constrained nature of the application restricted field of view and sensor s height, controlled internal illumination; the fusion of color and depth information; the synchronization of the left item detection module with the egate process we achieve a very robust performance. In fact, we consider any region whose area is as small as 9 pixels as a valid left luggage, and are able to detect virtually all possible left items, without being troubled by false alarms. We have tested left luggage detection by deliberately dropping off items in the egate, using 57 items of different size, shape, texture and color laptop bags and trolleys, passports, small personal belongings, dressing, drinking and reading items. The only item which our system was not able to detect was a transparent small drinking bottle, without a cork, and at standing position. The same bottle was detected when lying on the floor (due to its etiquette), or when full. Some detected items can be seen in Fig. 4. Fig. 4: examples of left luggage detection. Color image with detected bounding box (left), depth image (middle) and resulting fused static object foreground (right). From top to bottom: glove, empty bottle, passport, magazine, standing bottle, hat, laptop bag, trolley. False alarms rate was tested further, with persons passing through the egate and not leaving any items. A sequence of 231 egate processes has yielded zero false alarms rate. We note that due to data protection issues, we were not allowed to record video sequences while the egate is being continuously tested with real passengers. However, we hope to obtain more testing data shortly. 436

6 The test system is specified in Section 2.2. The left luggage module average performance, on optimized C++ and on hybrid CPU-GPU implementations, is 4.5msec and 2.3msec, respectively. For the hybrid version, the two background subtraction algorithms consume 0.3sec, using the same GPU implementation as in [15]. The postprocessing step (invalid pixel estimation, distinguishing between moving and static foreground objects, connected component labeling) is implemented in (non-optimized) C++ and consumes 2msec. 4. Conclusions This paper presented a hybrid CPU-GPU multisensor surveillance system used inside an egate, performing person separation, and left luggage detection. The system is based on a top-view mounted sensor inside the egate, fusing color and depth information. In contrast to previous published work, our depth-based person separation method does not make use of background subtraction algorithm, thus avoiding persons blending in to the background and other segmentation errors. To the best of our knowledge, our left luggage detection method is the first to use depth information, and to fuse color and depth information. Due the constrained nature of the application, we achieve a very robust performance. For the person separation task, a true positive detection rate of 0.93 and a true negative detection rate of 0.99 were achieved, while for the left luggage task, we consider any region whose area is as small as 9 pixels as a valid left luggage, and are able to detect virtually all possible left items, without being troubled by false alarms. The person separation module runs on average at 286 fps (C++), while the left luggage detection runs on average at 435 fps (CPU- GPU). References [1] D. Beymer. Person counting using stereo. In Proceedings of the Workshop on Human Motion, pages , [2] C.C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1 27:27, [3] A. Elgammal, D. Harwood and L.S. Davis. Nonparametric model for background subtraction, ECCV, 2:751 67, [4] FRONTEX: Best Practice Guidelines on the Design, Deployment and Operation of Automated Border Crossing Systems; Warsaw, March 2011 Release 1.1. [5] M. Harville. A framework for high-level feedback to adaptive, per pixel, mixture-of-gaussian background models, ECCV, vol.3, pp , [6] X. Huang, L. Li, and T. Sim. Stereo-based human head detection from crowd scenes. ICIP, pages vol.2, [7] M. Humenberger, C. Zinner, M. Weber, W. Kubinger and M. Vincze, A fast stereo matching algorithm suitable for embedded real-time systems, CVIU, Vol. 114, Issue 11, November 2010, Pages [8] K. Kim, T.H. Chalidabhongse, D. Harwood and L. Davis, Real-time foreground background segmentation using codebook model, Journal of Real-Time Imaging, Vol. 11(3), pp:172, [9] R. Luo and Y. Guo. Real-time stereo tracking of multiple moving heads. ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pages 55 59, [10] T. van Oosterhout, S. Bakkes, and B. J. A. Kröse. Head detection in stereo data for people counting and segmentation. 6th International Conference on Computer Vision Theory and Applications, pages , [11] S. Park and J. Aggarwal. Head segmentation and head orientation in 3d space for pose estimation of multiple people. 4th IEEE Southwest Symposium on Image Analysis and Interpretation, pages , [12] T. Pham. Non-maximum suppression using fewer than two comparisons per pixel. In Advanced Concepts for Intelligent Vision Systems, volume 6474, pages , [13] F. Porikli, Detection of Temporarily Static Regions by Processing Video at Different Frame Rates, AVSS, [14] Authors. Reliable Human Detection and Tracking in Top- View Depth Images, submitted to The 3rd International Workshop on Human Activity Understanding from 3D Data [15] D. Schreiber and M. Rauter, GPU-based non-parametric background subtraction for a practical surveillance system. In ECV Workshop, CVPR, pages , [16] L. Spinello and K. Arras. People detection in rgb-d data. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages , [17] C. Stauffer and W.E.L. Grimson, "Adaptive Background Mixture Models for Real-Time Tracking," CVPR, vol. 2, pp.2246, [18] Y.L. Tian, R. S. Feris, H. Liu, A. Hampapur and M.T. Sun, Robust Detection of Abandoned and Removed Objects in Complex Surveillance Videos, IEEE Transactions on Systems, Man and Cybernetics, part C: Applications and Reviews, Vol. 41, No. 5, [19] P. L. Venetianer, Z. Zhang, W. Yin and A. J. Liptop, Stationary Target Detection Using the ObjectVideo Surveillance System, AVSS, [20] S. Wu, S. Yu, and W. Chen. An attempt to pedestrian detection in depth images. IVS, pages , [21] T. Yahiaoui, C. Meurie, L. Khoudour, and F. Cabestaing. A people counting system based on dense and close stereovision. 3rd international conference on Image and Signal Processing, pages 59 66, [22] S. Yu, S.Wu, and L.Wang. Sltp: A fast descriptor for people detection in depth images. In AVSS, pages 43 47,