Computer Vision Object and People Tracking

Transcription

1 Computer Vision Object and People Prof. Didier Stricker Doz. Dr. Gabriele Bleser Kaiserlautern University DFKI Deutsches Forschungszentrum für Künstliche Intelligenz

2 AG Augmented Vision Uni-KL und DFKI Research Computer vision: 3D reconstruction and object recognition/tracking Sensor fusion: (body) motion tracking and activity recognition Visualization and rendering: realistic rendering, visualization and interaction Application domains Virtual Engineering Ambient Assisted Living Safety and Security

3 Computer Vision: Object and People Lecture: Monday 08:00 09:30 Room: SWS: 2V+1Ü Credit point: 4 LP Language: English

4 Object and People : lecture Topics Introduction Basics on image processing Edge, corner and blob detection Feature descriptors and matching Background modelling and subtraction Feature tracking and optical flow Recursive estimation Bayesian tracking (Extended) Kalman filter Particle filter Advanced tracking and applications

5 Object and People : exercises 4 exercise sheets and tutorial sessions planned throughout semester dates and material will be published on the website Theoretical questions + implementation tasks (Matlab) Answers to questions and implementations MUST be handed in (submission after approx. 1 week) Groups of up to 4 students are allowed Tutorial sessions shortly after submission deadline: discussion of questions, solution of implementation tasks

6 Contact Prof. Didier Stricker Dr. Gabriele Bleser: -> Teaching

7 Other courses taught by AV SS 2014: Seminar 3D Computer Vision & Augmented Reality, 4 CP Project 3D Computer Vision & Augmented Reality, 8 CP WS 2014/15 Lecture 3D Computer Vision, 4 CP, 2+1 Seminar Computer Vision: Object and People, 4 CP Project Computer Vision: Object and People, 8 CP Build upon OPT We also offer student jobs in various areas (CV, sensor fusion, HCI, ). Check the website, or just ask us! 3DCV topics: complementary to OPT Camera model and camera calibration Fitting and parameter estimation (deterministic) Two and multiple view reconstruction Structure from motion Dense reconstruction

8 What is Visual? Video tracking is the process of locating a moving object (or multiple objects) over time using a camera. The objective of video tracking is to associate target objects in consecutive video frames. -- Models of the objects are supposed to be available in advance, from which the relevant image features can be automatically selected.

9 What is tracking? detection recognition tracking Blair Bush Chirac

10 What is tracking? Definition: using image measurements and a predictive dynamic model to consistently estimate the state(s) Xt of one or more object(s) over the discrete time steps corresponding to video frames. X t-1 X t X t+1 t-1 t t+1 10

11 What is tracking? Why not just do detection? Estimate the state X at each time step - inefficient - data association problem

12 What is tracking? It s better to do tracking Maintain an estimate of X over time, predict the future location + efficient, restricts search space + smoothes noisy measurements - requires knowledge about object behavior 12

13 assumptions Smooth camera No instant transitions between viewpoints Any camera pose/parameter changes are gradual Object motion can be modeled Linear models Non-linear models Likelihood of object presence at a location in the image can be modeled Typically uses local image information 13

14 Approaches to tracking Sequential (recursive, online) + Inexpensive real-time - no future information - cannot revisit past errors Batch Processing (offline) - Expensive not real-time + considers all information + can correct past errors t=1,,t t-1 t t+1

15 Approaches to tracking Parallel trackers several single-object trackers computationally less expensive how to handle interaction, cross-overs? Joint state single multi-object representation computationally expensive principled interaction models x t x t x t x t x t x t x t x t x t x t x t 15

16 Approaches to tracking Non-probabilistic + quick convergence + efficient - stuck in local minima - does not model multiple objects Probabilistic + flexible, principled + multi-modal - slower - interpretation 16

17 applications 17

18 Abstraction: the 3 main aspects of object tracking Models Object Vision From: Giorgio Panin Book: Model-based Visual

19 Prior Models

20 Object model Object model Models Object Giorgio Panin, TUM-Informatik VI

21 Object model ground shape Shape Object model Models 2D model Object 3D (polyhedral) 3D mesh (approximate round surfaces)

22 Object model appearance Shape Object model Appearance Models Reference images Object Surface colors Texture map

23 Object model degrees of freedom Shape Object model Appearance Degrees of freedom Models 2D transformation Object 3D (rigid) 3D (deformable)

24 Object model temporal dynamics Shape Appearance Degrees of freedom Object model Models t0 t1 t3 t2 Random motion Temporal dynamics Object t0 tk ti Smooth trajectory (free) t0 t1 t2 Smooth motion (constrained)

25 Sensor models Sensors Models Object

26 Sensor models Sensors External (camera positions) Internal (projection model) Models Object

27 Context information Context Models Object Background Environment constraints Light sources Target interactions

28 Model adaptation from a prototype If we know the class that the model belongs to, we may adapt a prototype of this class to the specific object to be tracked. Example 1: head shape adapts the prototype using feature lines Example 2: pin-hole camera model Estimate focal length and resolution ry f rx

29 Visual data processing

30 Example: Model features for tracking Shape blobs Intensity gradients Contour lines Color statistics Texture template Motion (optical flow) Local keypoints

31 Visual modalities and features Natural features Color Motion Edges Keypoints Texture Modalities Object Vision

32 Data association Associate data to target/background Multi-target: solve ambiguities Missing detection x x x x Vision Object x x x False alarms Predicted pose and covariances x Measurements Data association

33 Multi-modal data fusion Object Vision Combine multiple information sources, at different levels Multi-modal data fusion Improve robustness, adaptivity

34 Multi-sensor data fusion + Object Vision Combine multi-camera information Multi-sensor data fusion Improve robustness, adaptivity

35 Likelihood models Pixel-level measurement Predicted shadow Object Segmented image Vision Measurement likelihood = how good is our prediction Error image

36 Likelihood models Pixel level Feature level Predicted features Object Vision Likelihood Matched data

37 Likelihood models Pixel level Feature level Object level Object Predicted state Estimated state Likelihood Vision

38 The tracking pipeline

39 Image acquisition Object Image acquisition

40 Initial target detection Object Time 0: Detection

41 State prediction Object Last estimates New pose? Time t: Prediction

42 Data processing (measurement) Object Measurement

43 Data association and fusion Object Data association

44 State update Object Correction

45 The tracking pipeline 1. Acquire New image 2. Predict object position 3. Measure Feature points Last estimates New pose? RESULT: Estimated pose 5. Correct the prediction 4. Match model features

46 The tracking pipeline 2. Prediction 3. Measurement 1. Image acquisition 4. Model matching 5. Correction

47 A general tracking system Main components of a tracking system Detection and loss detection: (re-)initialize the system : measurement processing and state update Models: all useful prior information about objects, sensors, and environment t It Visual Processing Measurement Prediction Output Update It Object Detection Initial estimate Loss Detection Shape Appearance Degrees of freedom Models Dynamics Sensors Context

48 Detection vs. tracking Detection = find the object without any prior prediction, only with the model Only static data association and measurement fusion can be done The search is global (over the full state-space) coarse and slow Features must be invariant to viewpoint change = update the object state(s) using the previous state and the dynamical model Dynamic data association/fusion can be done Search is local (around the predicted features/states) fast and accurate Features do not have to be invariant (the viewpoint is not much different)

49 Resume: tracking on a slide Object Sensors Context Models Image acquisition Modalities Object Detection / Prediction Likelihood Vision Measurement Data association Data fusion State update

50 What is the state of the art? Major sources of difficulties

51 What is the state of the art? Despite being classic computer vision problem, tracking is largely unsolved Some limited successes No general-purpose tracker No standard data corpus for comparison No standard evaluation methodology Challenging problems remain

52 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Ruei-Sung Lin, David Ross, Jongwoo Lim, Ming-Hsuan Yang, Adaptive discriminative generative model and its Applications Neural Information Processing Systems Conference (NIPS), 2004 Slide: Kevin Smith

53 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Amit Adam, Ehud Rivlin and Ilan Shimshoni, Robust Fragments-based using the Integral Histogram (pdf). IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2006

54 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Michael Isard and Andrew Blake CONDENSATION -- conditional density propagation for visual tracking International Journal of Computer Vision (IJCV), 29, 1, 5--28, (1998)

55 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Ruei-Sung Lin, David Ross, Jongwoo Lim, Ming-Hsuan Yang, Adaptive discriminative generative model and its 55 Applications Neural Information Processing Systems Conference (NIPS), 2004

56 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Michael Isard and Andrew Blake CONDENSATION -- conditional density propagation for visual tracking International Journal of Computer Vision (IJCV), 29, 1, 5--28, (1998)

57 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Saad Ali and Mubarak Shah, Floor Fields for in High Density Crowd Scenes, European Conference on Computer Vision (ECCV), 2008.

58 Obstacles to tracking appearance change occlusion distraction illumination change difficult motion multiple objects scale change Shawn Lankton, James Malcolm, Arie Nakhmani, and Allen Tannenbaum. Through Changes in Scale Proceedings of International Conference on Image Processing (ICIP), 2008.

59 Application examples

60 applications is an essential step in many computer vision based applications Detection + Feature Extraction Activity Recognition + Event Recognition Behavior Analysis + Social Models Prithwijit Guha, Amitabha Mukerjee and K. S. Venkatesh, Spatio-temporal Discovery: Appearance + Behavior = Agent, Computer Vision, Graphics and Image Processing , 2007

61 applications Surveillance 61 K. Smith, P. Quelhas, and D. Gatica-Perez, Detecting Abandoned Luggage Items in a Public Space, Performance Evaluation of and Surveillance (PETS) Workshop at CVPR, New York, NY, June

62 applications Biological Research Goal: develop a method to trap Salmonella bacteria P. Horvath, Q. Buhkari, 3D of point-like objects in 2D image sequences, LMC, ETHZ

63 Thank You!