Backpack: Detection of People Carrying Objects Using Silhouettes

Backpack: Detection of People Carrying Objects Using Silhouettes Ismail Haritaoglu, Ross Cutler, David Harwood and Larry S. Davis Computer Vision Laboratory University of Maryland, College Park, MD 2742 hismail,rgc,harwood,lsd @umiacs.umd.edu Abstract We described a video-rate surveillance algorithm to detect and track people from a stationary camera, and to determine if they are carrying objects or moving unencumbered. The contribution of the paper is the shape analysis algorithm that both determines if a person is carrying an object and segments the object from the person so that it can be tracked, e.g., during an exchange of objects between two people. As the object is segmented, an appearance model of the object is constructed. The method combines periodic motion estimation with static symmetry analysis of the silhouettes of a person in each frame of the sequence. Experimental results demonstrate robustness and real-time performance of the proposed algorithm. 1 Introduction Visual surveillance is both a challenging scientific problem and an important application in computer vision. With increasing processor power, more attention has been given to developing real-time smart surveillance systems. Surveillance cameras are already installed in many locations such as highways, streets, stores, ATM machines, homes and offices. The ability to detect and track people is a key element of such systems. The problem is more challenging when we want to monitor interactions between people and objects, and detect unusual events such as depositing an object (unattended baggage in airports), exchanging bags, or removing an object (theft). It requires an ability to detect people carrying objects, to segment the objects from people, and to construct appearance model for the objects so they can be identified subsequently. Backpack combines two basic observations to analyze people carrying objects: Human body shape is symmetric, and people exhibit periodic motion while they are moving unencumbered. During tracking, the periodic motion of a person and his parts is estimated, and the regions on the silhouette which systematically violate the symmetry-! "# %$& ' (*)!#,+.- / $&12+43&56# 6 6 $&17 8 $-91 +.#,+.17( " (&: 6 $&1;-< (+4 6,+4 8 $ / ( = constraints are determined. Those results are combined to determine if a person is carrying an object and to segment the object from the silhouette. We construct an appearance model for each carried object, so that when people exchange objects, we can detect who carries which object via an analysis of the segmentation. Backpack employs a global shape constraint derived from the requirement that the human body shape is symmetric around its body axis. Backpack uses that constraint to segment outlier regions from their silhouette. The expected shape model of a person is compared with the current person silhouette to determine the outlier regions (nonsymmetric region). One can observe that because of the motion of people s arms, legs, hands, outliers are periodically detected in the vicinity of those body parts. However, outliers are detected continuously in the vicinity of a sufficiently large carried object because of continued symmetry constraint violations. Therefore, Backpack use periodicity analysis to classify whether outlier regions belong to an object or a body part. Backpack has been designed to work under the control of >? [5, 7]. >@? is a real time visual surveillance system for detecting and tracking people and their body parts, and monitoring their activities in an outdoor environment. It operates on monocular grayscale video imagery, or on video imagery from an infrared camera. >A? detects objects through a background subtraction process. Before Backpack, >? simply classified those objects as people, vehi-

cles, or other on the basis of static size and shape properties, and dynamic analysis of shape periodicities. If an object was classified as a person, then >A? segmented the shape into body parts (head, torso, feet and hands), built appearance models for the entire person and the parts, and tracked the person and his parts. Backpack allows >A? to analyze people carrying objects. We are interested in interactions between people and objects; e.g., people exchanging objects, leaving objects in the scene, taking objects from the scene. Backpack forms a basis for developing algorithms to reason about activities involving people and objects. The reminder of this paper is organized as follows. After a brief literature review in Section 1.1, Section 2 describes the silhouette model used in this work. Section 3 focuses on detection of shape periodicity based an moving objects. The symmetry-based region segmentation and dynamic appearance of non-symmetric regions are explained in Section 4. There are many real-time systems proposed in the past few years to detect and track people. Each system uses a particular sensor type (single or multiple camera, color or grayscale, moving or stationary), and have different functionalities (track single person, multiple people, handle occlusion, body part detection and tracking). None of the previous real-time surveillance system attempts to determine whether a person is carrying an object or not. Pfinder [11] is a real-time system for tracking a person which uses a multi-class statistical model of color and shape to segment a person from a background scene. It finds and tracks people s head and hands under a wide range of viewing condition. CMU s system [9] extracts moving targets from a real-time video stream, classifies them into pre-defined categories and tracks them. SRI s system [8] and the other extension of >@? [6] uses real-time stereo to detect and track multiple people. MIT s system [4] uses real-time color based detection and motion tracking algorithms to classify detected objects and to learn common patterns of activity. KidRooms [1] is a color-based multi camera tracking system based on closed-world regions, which allows people to interact with each other. Lehigh/Colombia s system [2] uses an omnidirectional camera to detect and track multiple blobs in real time. Hebrew University s system [1] has the ability to detect moving object from a moving camera in real-time. 2 Silhouette Model Backpack generates a set of shape and appearance features for each person s silhouette. Backpack determines a major axis of a person s silhouette by applying a principal %3 +.+ (#- 1 -<3& #+* - - 1 %$ / $ &+ % # 1 + #,+.1 / ( Backpack 6# 6 $&1 $ / # +.-,&$+4 6 81 &$1 ) # # -! / 1. $&+ +.-( $&1 " /. $# $%& $& #3 %&# +4 8,- $ 8+ - $+ ' / <43& )(, $&+ $&1*# # 6+ & # 6*),+4 8 $ 3 - + & 6,+ - % 8! + "8+ - (&+.& %$&1 ( " # 6*),+4 8 $ 3 - + & & -! $ $ %$& # 6-4 $ / $1 &## %$& / ( = component analysis to the silhouette pixels. The best fitting axis constrained to pass through the median silhouette coordinate is computed by minimizing the sum of absolute perpendicular distances to that axis. The direction of the major axis is given by an eigenvector associated with the largest eigenvalue of its covariance matrix (Figure 2(b)). The shape of a 2D binary silhouette is represented by its projection histogram. We compute the 1D vertical and horizontal projection histograms of the silhouettes in each frame. Vertical and horizontal projection histograms are computed by projecting the binary foreground region onto axes perpendicular to and along the major axis, respectively (Figure 2(f) (g)). Projection histograms are normalized by rescaling them to a fixed length, and aligning the median coordinate at the center. 3 Shape Periodicity Analysis People exhibit periodic motion while they are moving. We previously introduced a robust, image-correlation based technique to compute the periodicity of a moving object [3]. It was used during tracking to determine whether the region is a vehicle or a person. The method computes image intensity self-similarities as the object appearance evolves over time. A computationally inexpensive version of [3], which also requires less memory, is employed by Backpack to determine shape periodicity. Periodic motion is deter-

? 4 M 4 { 6 $ 3& # 6*) + $@3 -' + & 6 - ' -4 $'3&$ - 1 %$& &%:# %$ 3 $&& -* % 8 + " 8+;$1@ 8 1' (&+ $ %$& - + *),+4 8 $ 3 - + & & - -<3 $ mined by self-similarity of silhouettes over time using the silhouette s projection histograms. The vertical and horizontal projection histograms of a walking person are shown in Figure 4. Note that the person completes one walking cycle in 17 frames. Shape periodicity is obtained by using similarities of the last projections (typically ). The projections are aligned and normalized during tracking. A similarity plot! between the projection histogram, "$#&%, at time and "$#(' at time is computed as follows: )* +-,/.1 2 354 2 687:9 ; 6=<6=>@? " # <1AB354DC % " # < ' (1) where E and F are the lower and upper bounds of projection histograms. In order to account for tracking error, the minimum similarity is computed by translating "=#&% over a small search window G. A row-based auto correlation method is applied to to detect periodicity. Note that the similarity plot has a very distinctive pattern when the shape is changing periodically. In Figure 3, the similarity plots for a walking person and a moving car are shown. The similarity values in the plot have been scaled to a grayscale intensity range where darker regions indicate higher similarity. Periodic motions will have dark lines or curves parallel to the diagonal which represents self-similarity of each projection histogram to itself. For each row of, a period value H < is determined where the absolute auto-correlation of that row has a peak. Among all peaks, H <, the most frequently occurring peak is selected as the fundamental frequency, H, of the motion. A confidence value is computed based on the number of rows in that have similar frequency to H. In Figure 4, similarity plots obtained by using vertical and horizontal projection histograms of a walking person and their determined fundamental frequencies are shown. A detailed description of periodic motion detection algorithms can be found in [3]. I J $ - "& +4 ; 6 8 $ - &$+ #+ $ - 1 %$ BackpackK 4 Symmetry Analysis Silhouettes of human are typically close to symmetric about the body axis while standing, walking or running. Let L&M be the symmetry axis constructed for a given silhouette [7]. Each pixel is classified as symmetric or nonsymmetric using the following simple procedure: Let "=N and "$O be a pair of pixels on the silhouette boundary such that the line segment from "=N to "$O is perpendicular to LPM and intersects with LQM at " (shown in Figure 5). Let M G < N and G < O be the length of line segment ["=NR" ] and length of line M segment [" M &"$O ], respectively. A pixel S lying on the line segment [" N " O ] is classified as follows: STVU Non-Symmetric Symmetric if G otherwise MXWZY\[P]_^ G < N G <a`cbed O where G is the length of the line segment from pixel S to". M and d is a constant. Figure 6 shows example of symmetrybased segmentation results for people with and without an object by showing their detected head location, computed hypothetical symmetry axis, and non-symmetric region segmentation. f gihch Bjlk nm co pj _jrqtsulmvm awqk =xyw_jls >? constructs a dynamic template -called a temporal textural template [5] while it is tracking and segmenting individual people. A similar appearance model is generated and updated for non-symmetric regions over time in Backpack. The temporal texture template for an object z is defined by: { # S=l:} (2) (S= be~ #P S=y #P S= ~ #P (3) S= b Here, } S= refers to the intensity of pixel(x) which is classified as non-symmetric pixel, and all coordinates are represented relative to the median coordinate of the person. The

{ W { 6 #&8 - " +4 " (&#- 1 - $&+ +4 8 $ -< +.- # 8i +*3 $&1n +*3& + &$ (4) #,+ ( " -<3 %$ +*3# % 1 +.,+.#1 3&1 8 #+ $ +.1 3" +43& +4 8& - "& +4 " # -! $&1i $&& $ $ - " +4 8 6 8 $;- &$+ #+ $ H is the fundamental frequency of shape periodicity for the entire body, and is a constant. f g wqj ẍ w cwqk=w!u 6 c $ 3& 2+. 6& +. +* $ +. +.- ## # - 1! %$ +4 6:# %$ +*3 +*3& + $ (*),+ 3 -<3& $1 %$ +.&$-* + " $&$+.- +*3 +.& 8#+ $&+4 ( 1 " / & 1 $&1$& $ - " +4 8 8 $&- / < -<3 $ weights ~ # are the number of times that a pixel in is classified as a non-symmetric pixel during tracking. The initial weights ~ # (S= of are zero and are incremented each time that the corresponding location (relative to the median template coordinate) is detected as a foreground pixel in the input image. Note that a temporal textural templates has two components that can be used for subsequent identification: a textural component which represents the appereance of the object (Figure 7d,f); and a shape component (~ # ) which represents weighted shape information (Figure 7c,e). Examples of temporal textural templates for the entire body and for non-symmetric regions of a person while they are walking with and without an object are shown in Figure 7. Backpack segments the shape component of a temporal textural template to determine the regions where periodic motion analysis should be applied. Periodic motion analysis is applied to a non-symmetric pixel S if ~ # (S= H b where Non-symmetric pixels are group together into regions, and the shape periodicity of each non-symmetric region is computed individually. The horizontal projection histogram segment bounded by a non-symmetric region is used to compute the shape periodicity of the corresponding nonsymmetric region. A non-symmetric region which does not exhibit significant periodicity is classified as an object carried by a person, while a non-symmetric region which has significant periodicity is classified as a body part. In Figure 8, the final classification results are shown for a walking person who is not carrying an object, and a person who is carrying an object. In the first example, a person is walking with 1Hz frequency (15 frames per half period with 1% confidence value); the similarity plot of the vertical projection histogram for the entire body is shown in Figure 8(a)(right). Note that the legs and arms of the person violate the symmetry constraint periodically during walking. The pixels around the legs and arms are detected as non-symmetric pixels and grouped into two non-symmetric regions (region 1 around legs, and region 2 around arms). Then, the similarity plots for region 1 and region 2 are obtained as shown in Figure 8(a). Note that the shape periodicity algorithm is applied only to the horizontal projection histogram segments bounded by regions 1 and 2. Periodicity is detected for region 1 at 1.1Hz and for region 2 at 1.3 Hz, which are very similar to the shape periodicity of the entire body. Therefore those regions are classified as body parts (shown in green). In the second example, a person is walking and carrying a bag with.85hz frequency (17.9 frame per half

6 %$ (*),+ 1 +.+ $; -< +.-(&#- 1 $ $ $ - "& +4 8 $- &$+ #+4 8 $ &$1-<3& # 1- + ";$&& " -* 8- &%:# %$& \ +*3 + $&1 +*3 ( / $ - $ 1 $& ##-< ; %$ 6,&$+ # $&1 - &$&," %$ 1 ROC graph.8 Detection(true positive).6.4.2.2.4.6.8 1 False positive a '# # 6# (4),+ 1+.,+4 8 $ 6 (4) #,+ 1 +.#,+4 8 $ -< +.- 3 # -4 $ - &# %$ 1 $&+ 1 #,+4 8 $period with 98% confidence value); its similarity plot from the vertical projection histogram of the entire body is shown in the Figure 8(b). The legs of the person and the bag violate the symmetry constraint during walking, and the regions around the legs (region 1) and the backpack (region2) are grouped into non-symmetric regions. Shape periodicity is detected for region 1 at.84hz with high confidence and for region 2 at 2.5Hz with low confidence. The periodicity of region 1 is very similar to the periodicity of the entire body, and it is classified as a body part. However, region 2 does not have a significant fundamental frequency similar to the entire body, so it is classified as a carried object. The symmetry and shape periodicity analysis used in Backpack are view-based techniques; the results depend on the direction of motion of the person, and location of the object on the silhouettes. Figure 9 shows detection results where a person is carrying an object in his hand while moving in different directions. We ran a series of experiments using 1 sequences where a person is moving in different directions (people carry an object in 62 sequences, and not carry in 38 sequences). We estimated the Receiver Operating Curve (ROC) which plots the probability of detection along z -axis and the probability of false detection along S -axis (Figure 1). An ideal recognition algorithm would produce results near the top left of the graph (low false alarm and high detection probability). For different periodicity confidence thresholds, we computed the number of instances that are correctly classified as person-with-object (true positive), and the number of instances that are misclassified as people-with object (false positive). For the optimal choice of thresholds, Backpack successfully determined whether a person is carrying an object in 91/1 sequences. It generally failed on sequence where there is not a large enough non-symmetric region (5/1) or insufficient shape changes (4/1) (causing low periodicity confidence value) e.g., when a person is moving towards the camera. In those cases, Backpack uses a non-global 2D-intensity based periodicity analysis [3] to compute periodicity to decrease the false positive rate (yielding a 95/1 success rate). Backpack uses appearances (shape, intensity, and position) information embedded into its temporal textural templates to track carried objects that they have been detected and their temporal textural templates generated.

35 3 25 Trajectoriy of People Person Person 1 X position 2 15 1 5 Carry Object 1.5 1.5 -.5 2 25 3 35 4 45 5 Time (in Frames) People Carrying Object 2 Person Person 1-1 6 (4),+ 1+ #,+4 8 $ 6 -< 8+ - 3 8 # -4 $ -! "# %$& 1 $&+ + " - (*)!#,+ $&1 &# %$ 1 $&+ 1 #,+4 8 $&- -1.5 2 25 3 35 4 45 5 Time (in Frames) 5 - #3&$ $ (*)!#,+ # &$+ References Detection- Tracking Shape Analysis Symmetric segmentation Similarity Plot Computation Periodicity Computation Temporal Textural Template 11.92 ms 2.63 ms.27 ms 1.5 ms.32 ms 1.17 ms (8 # # &+4 8 $+4 % ; 6 & 5 Conclusion and Discussion We have describe a silhouette-based method to determine if a person is carrying and object, and to segment the object from the silhouette. We construct an appearance model for each carried object, so that when people exchange objects, we can detect who carries which object via an analysis of the segmentation. Backpack has been implemented in C++ and runs under the Windows NT operating system. Currently, for 32x24 resolution gray scale images, Backpack runs at 3 Hz on Pentium II 4 MHz PC. Table 1 gives benchmarks for different component of Backpack. There are several directions that we are pursuing to improve the performance of Backpack and extend its capabilities. We are studying how to improve detection results using local shape and appearance information, such as, Is there an object near a person s hand that looks like a briefcase?. We are also interested in interactions between people and objects; e.g., people exchanging objects, leaving objects in the scene, taking objects from the scene. The description of people; their positions, and motions-developed by Backpack is designed to support such activities. Backpack forms a basis for developing algorithms to reason about activities involving people and object, such as depositing/removing objects, and exchanging objects (Figure 12). [1] A. Bobick, J. D. S., Intille, F. Baird, L. Cambell, Y. Irinov, C. Pinhanez, and A. Wilson. Kidsroom: Action recognition in an interactive story environment. Technical Report 398, M.I.T Perceptual Computing, 1996. [2] T. Boult, Frame-rate omnidirectional surveillance and tracking of camouflaged and occlude targets. In Second Workshop of Visual Surveillance at CVPR, pages 48-58, 1999 [3] R.Cutler and L. Davis. Real-Time Periodic Motion Detection, Analysis, and Applications. In Computer Vision and Pattern Recognition Conference, (2) pages 326-331, 1999. [4] E. Grimson and C. Stauffer and R. Romano and L. Lee. Using adaptive tracking to classfy and monitoring activities in a site In Computer Vision and Pattern Recognition Conference, pages 22-29, 1998. [5] I. Haritaoglu, D. Harwood, and L. Davis. W4: Who, when, where, what: A real time system for detecting and tracking people. In Third Face and Gesture Recognition Conference, pages:222-227, 1998 [6] I. Haritaoglu, D. Harwood, and L. Davis. W4S: A real time system for detecting and tracking people in 2.5 D In Eurepean Conference on Computer Vision, pages: 877-892, 1998 [7] I. Haritaoglu, D. Harwood, and L. Davis. Hydra: Multiple People Detection and Tracking Using Silhouettes In Second Workshop of Visual Surveillance at CVPR, pages 6-13, 1999 [8] K. Konolige. Small vision systems: hardware and implementation. In International Symposium of Robotics Research, pages 111-116, 1997. [9] A. Lipton, H. Fujiyoshi, and R. Patil. Moving target detection and classification from real-time video In Proceedings of IEEE Workshop on Application of Computer Vision, 1998. [1] A. Peleg, et. al. Multi sensor representation of extended scenes using multi-view geometry In DARPA Image Understanding Workshop, 1998. [11] C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland. Pfinder: Real-time tracking of the human body. IEEE Transaction on Pattern Analysis and Machine Intelligence, 19(7), pges 78-785,1997