Introduction to behavior-recognition and object tracking

Transcription

1 Introduction to behavior-recognition and object tracking Xuan Mo ipal Group Meeting April 22, 2011

2 Outline Motivation of Behavior-recognition Four general groups of behaviors Core technologies Future direction 4/22/2011 ipal Group Meeting 2

3 Motivation of Behavior-recognition Visual surveillance is necessary to deter and respond to accidents, crime, suspicious activities, terrorism and vandalism. Threat detection is currently performed by assigning analysts to simultaneously watch the same video stream The human resources are costly. Manual analysis of video is labor intensive, fatiguing and prone to errors. Software-aided real-time video analytics would considerably alleviate the human constraints. The idea of creating a virtual analyst or software tools for video analytics has become of great importance to the research community. 4/22/2011 ipal Group Meeting 3

4 Four general groups of behaviors 1 1 Single person or no interaction. 2 Multiple-person interactions. 3 Person-vehicle interactions. 4 Person-facility/location interactions. 1 Candamo et. al., IEEE Trans. Intelligent transportation system, /22/2011 ipal Group Meeting 4

5 Single person or no interaction Consists of behaviors that can be defined only by considering person(s), which are not interacting with any other person or vehicle. For example, loitering, people (crowd) counting, crowd flow (behavior) analysis, and person talking on a cell phone. Figure: Single-person or no interaction behavior 4/22/2011 ipal Group Meeting 5

6 Single person: Loitering An individual in an area for a period of time > time threshold. It s common practice of drug dealers,beggars and muggers. transit system application mostly consist of tracking indoor video. Outdoor loitering detection is scarce. [Bird et. al.,2005] uses a refined Gaussian mixture background subtraction algorithm to detect motion blobs in a calibrated scene. 4/22/2011 ipal Group Meeting 6

7 Single person: Crowd counting For example, estimation of crowds in underground transit systems can be used to give passengers a good estimate of the waiting time in a queue. Crowd counting is highly sensitive to training data. Overhead-view camera is more popular than side-view camera, which will be used when overhead-view camera is not available. Methods of crowd segmentation include shape indexing, face detection and skin color and motion. Most of these methods heavily rely on image quality and frame rate for accurate results. Shape indexing and skin colors are considered robust to poor video quality, whereas motion and face detection are most dependent on video quality. Occlusion is another problem. 4/22/2011 ipal Group Meeting 7

8 Single person: Crowd behavior The flow of large human crowds is a useful cue to deal with accidents or preventively to timely control situations that potentially could lead to graver incidents. Recent crowd behavior analysis methods include tracking of moving objects, motion models using optical flow, and crowd-density measurement using background reference images. Common abnormal crowd characteristics that have been researched are fallen person, blocked exit, and escape panic. Behavior classification is often based on the vector fields generated by crowd motion instead of individual person tracking. 4/22/2011 ipal Group Meeting 8

9 Single person: Human Pose Estimation Human pose estimation refers to the pose of the entire human body, and not a pose related to a single body part, such as a head pose, Two main approaches: 1 Calculates ratios between the height and the width of the bounding box around a detected human. 2 Track specific joints and body parts, both because: they are useful for indicating the human pose when accurately modeled, they can be used to recover the pose even after occlusion and other common tracking failures. 4/22/2011 ipal Group Meeting 9

10 Multiple-person interactions Consist of behaviors that involve persons interacting with each other. For example, following, tailgating, meeting, gathering, moving as a group, dispersing, shaking hands, kissing, exchanging objects, and kicking. Figure: Multiple-person interaction behavior. Pedestrians on a crosswalk Nearest neighbor classifier based on trajectory information to detect human interactions such as walking together, approaching... Bayesian networks; and moment-invariant feature descriptions to detect events, including sitting down, standing up... Performance relies on the ability to accurately segment and separate multiple human motions. 4/22/2011 ipal Group Meeting 10

11 Person-vehicle interactions Consist of behaviors that are defined through interactions with persons and vehicles. For example, driving, getting in (out), loading(unloading), opening (closing) trunk, crawling under car, breaking window, dropping off, and picking up. Figure: PersonCvehicle interaction. Person being run over by a vehicle 4/22/2011 ipal Group Meeting 11

12 Person-facility/location interactions Behaviors defined through interactions with persons and facilities/locations. For example, entering (exiting), standing, waiting at checkpoint, evading checkpoint, passing through gate, object left behind, and vandalism. Figure: Person - facility/location interaction. Object left behind in a train station 4/22/2011 ipal Group Meeting 12

13 Core technologies 2 Object representation. Feature selection for tracking. Object detection Object tracking 2 Yilmaz et. al., ACM Computing Surveys, /22/2011 ipal Group Meeting 13

14 Object representation = Shape + Appearance Figure: Object representations. (a) Centroid, (b) multiple points, (c) rectangular patch, (d) elliptical patch, (e) part-based multiple patches, (f) object skeleton, (g) complete object contour, (h) control points on object contour, (i) object silhouette. 4/22/2011 ipal Group Meeting 14

15 Appearance representations Templates Formed using simple geometric shapes or silhouettes. Suitable for tracking objects whose poses do not vary considerably during the course of tracking. Self-adapation of templates durch the tracking is possibe. Figure: Template representation 4/22/2011 ipal Group Meeting 15

16 Appearance representations Probability densities of object appearance, can either be parametric (Gaussian and mixture of Gaussians) or nonparametric (histograms) Characterize an image region by its statistics. If the statistics differ from background, they should enable tracking. Figure: Nonparametric: histogram representation 4/22/2011 ipal Group Meeting 16

17 n-d Gaussian distribution Figure: n-d Gaussian distribution 4/22/2011 ipal Group Meeting 17

18 Gaussian Mixture Models (GMM) Figure: Gaussian Mixture Models 4/22/2011 ipal Group Meeting 18

19 How to choose the object representation Point representations appropriate for tracking objects, which appear very small in an image (e.g. track distant birds) For the objects whose shapes can be approximated by rectangles or ellipses, primitive geometric shape representations are more appropriate (e.g. face) For tracking objects with complex shapes, for example, humans, a contour or a silhouette-based representation is appropriate (surveillance applications) 4/22/2011 ipal Group Meeting 19

20 Feature selection for tracking In general, the most desirable property of a visual feature is its uniqueness so that the objects can be easily distinguished in the feature space Color: RGB, L*u*v*, L*a*b*, HSV, etc. There is no last word on which color space is more effective; a variety of color spaces have been used Edges: less sensitive to illumination changes compared to color features. Algorithms that track the object boundary usually use edges as features. Because of its simplicity and accuracy, the most popular edge detection approach is the Canny Edge detector Texture: measure of the intensity variation of a surface which quantifies properties such as smoothness and regularity 4/22/2011 ipal Group Meeting 20

21 Object detection Object detection mechanism: required by every tracking method either at the beginning or when an object first appears in the video Point detectors: find interest points in images which have an expressive texture in their respective localities Harris detector: evaluate for each pixel in a small neighborhood: Figure: Harris detector formulation and example If R > T hreshold interest point 4/22/2011 ipal Group Meeting 21

22 Background subtraction Detecting the foreground objects as the difference between the current frame and an image of the scenes static background Running Gaussian average frame i background i > T h modeling the color of each pixel, I (x, y), of a stationary background with a single 3D (Y, U, and V color space) Gaussian, I(x, y) N((x, y), σ(x, y)). F µ > T h deviate from background are labeled as the foreground Mixture of Gaussians to model the pixel color is better 4/22/2011 ipal Group Meeting 22

23 Other detection approaches Segmentation: partition the image into perceptually similar regions. 1 Mean-Shift Clustering 2 Graph-Cuts 3 Active Contours Supervised learning: require a large collection of samples from each object class. 1 Adaptive Boosting 2 Support Vector Machines 4/22/2011 ipal Group Meeting 23

24 Mean-shift 4/22/2011 ipal Group Meeting 24

25 Mean-shift 4/22/2011 ipal Group Meeting 25

26 Object Tracking Aim of Object Tracking: generate the trajectory of an object over time by locating its position in every frame of the video. (a) Point Tracking. Objects detected in consecutive frames are represented by points, and a point matching is done. This approach requires an external mechanism to detect the objects in every frame. (b) Kernel Tracking. Kernel = object shape and appearance. E.g. kernel = a rectangular template or an elliptical shape with an associated histogram. Objects are tracked by computing the motion (parametric transformation such as translation, rotation, and affine) of the kernel in consecutive frames. (c)+(d) Silhouette Tracking. Such methods use the information encoded inside the object region (appearance density and shape models). Given the object models, silhouettes are tracked by either shape matching. 4/22/2011 ipal Group Meeting 26

27 Point Tracking Constraints (a)proximity assumes the location of the object would not change notably from one frame to other (b)maximum velocity defines an upper bound on the object velocity and limits the possible correspondences to the circular neighborhood around the object (c)small velocity change assumes the direction and speed of the object does not change drastically (d)common motion constrains the velocity of objects in a small neighborhood to be similar. This constraint is suitable for objects represented by multiple points. (e)rigidity assumes that objects in the 3D world are rigid, therefore, the distance between any two points on the actual object will remain unchanged 4/22/2011 ipal Group Meeting 27

28 Point Tracking Categories Deterministic Methods for Correspondence: Use a combination of motion-based constraints - common constraint, proximity constraint optimize the cost of neighboring frames. Statistical Methods for Correspondence 1 Kalman Filter: is used to estimate the state of a linear system where the state is assumed to be distributed by a Gaussian 2 Particle Filter: is a hypothesis tracker, that approximates the filtered posterior distribution by a set of weighted particles. It weights particles based on a likelihood score according to the motion model. 4/22/2011 ipal Group Meeting 28

29 Kernel Tracking Template Matching: brute force method for tracking single objects. 1 Define a search area 2 Place the template defined from the previous frame at each position of the search area and compute a similarity measure between the template and the candidate 3 Select the best candidate with the maximal similarity measure The similarity measure can be a direct template comparison or statistical measures between two probability densities Figure: Direct comparison 4/22/2011 ipal Group Meeting 29

30 Mean-Shift Object Tracking 4/22/2011 ipal Group Meeting 30

31 Mean-Shift Object Tracking 4/22/2011 ipal Group Meeting 31

32 Silhouette Tracking Objects may have complex shapes, for example, hands, head, and shoulders that cannot be well described by simple geometric shapes. Silhouette-based methods provide an accurate shape description for these objects. Shape Matching: similar to template matching,compute the similarity of the object. Contour Tracking 1 Tracking Using State Space Models: Kalman Filter Particle Filter Hidden Markov model 2 Tracking by Direct Minimization of Contour Energy Functional 4/22/2011 ipal Group Meeting 32

33 Taxonomy of tracking methods 4/22/2011 ipal Group Meeting 33

34 Future direction for object tracking Defects: Too many assumptions used to make the tracking problem tractable, for example, smoothness of motion, minimal amount of occlusion, illumination constancy, high contrast with respect to background, etc., are violated in many realistic scenarios. Future work 1 Develop algorithms for tracking objects in unconstrained videos. 2 Employ audio in addition to video for object tracking. 3 Integrate contextual information. For example, in a vehicle tracking application, the location of vehicles should be constrained to paths on the ground as opposed to vertical walls or the sky. 4 Online selection of discriminative features and learning object models online 5 More efficient solutions for inference before DBNs are more commonly used 4/22/2011 ipal Group Meeting 34

35 Future direction for Human behavior recognition Defects: Some algorithms requires a moderately high frame rate and significant processing power. There is no common set of data set and metrics! Most papers, regardless of the review process, chose to not completely disclose the data set description of their work. Future work 1 Resolve Occlusion. 2 Multiple Camera Tracking. 3 Standard baseline algorithms are required for comparison purposes. 4/22/2011 ipal Group Meeting 35