Model-based Visual Tracking:

Transcription

1 Technische Universität München Model-based Visual Tracking: the OpenTL framework Giorgio Panin Technische Universität München Institut für Informatik Lehrstuhl für Echtzeitsysteme und Robotik (Prof. Alois Knoll) Dr. Ing. Giorgio Panin

2 Technische Universität München Contents Object tracking: theory Building applications with OpenTL Dr. Ing. Giorgio Panin

3 Object tracking

4 Goal: Multi-Target / -Sensor / -Modal localization O 1 O 2 C 1 C i W

5 Model-based tracking Model Visual processing Localization (tracking) Video surveillance Control/navigation Face tracking...

6 Ground shape O 2 O 1 W

7 Pose parameters single-body transforms Base Euclidean Similarity Affine Homography 2D 3D Distances Angles Parallel lines Straight lines Invariant Angles Parallel lines Straight lines properties Parallel lines Straight lines Straight lines

8 Pose parameters articulated body T W,l0 W ρ T l 0,l 1 T l 1,l 2 x W = TW, l x 3 l3 x T l 2,l 3

9 Pose parameters deformable shapes x T W, 0) 0 ( x Link 0 y T W, p) 0 ( Link 0 y z W y z y T ( W, 0 0 ) x T W, p) 0 ( x W

10 Active Shape Model (2D) face tracking Learning deformation modes from examples (PCA)

11 Object appearance Key-frames Texture map Active Appearance Model

12 Object dynamics 2nd order, Auto-Regressive model Damped-spring motion Process noise xt x = F1 ( xt 1 x) + F2 ( xt 2 x) + W0 w t x Motion type: specified by three main parameters Average state: x x Oscillation frequency: f Damping rate: β

13 Object dynamics - examples Brownian Motion ( f, β undefined) F 1 2 = 1 F = 0 W 0 = 0.1 White Noise Acceleration f = 0 β = F = 2 F = 1 W = 0.1 Periodic, undamped : f = 0.1 F 1 = 1.99 β = 0 F 2 = 1 W 0 = 0.1 Periodic, damped : f = 0.1 F 1 = 1.98 β = 0.05 F 2 = 0.99 W 0 = 0.88 Aperiodic (critically damped): f = 0 β = 0.5 F 1 = 1.9 F 2 = 0.9 W 0 = 2.12

14 Object dynamics multi-dim. = = I W I I I F = = I W I I F t=50 t=1000 Unconstrained Constrained ( ) t t t W F I F w s s s + + = 1 t=10000

15 Object model Shape Appearance Pose t t 2 0 t 1 Dynamics t 0 t 1 t 2 t 3 t 0 t i t k

16 Camera model Intrinsic parameters Pin-hole model Radial distortion x optical axis z c C x c c y c y f retinal plane

17 Camera model Extrinsic parameters c c 1 2 x w T c, w 1 w T c, w 2

18 Camera model Object-to-image projection (and back-projection) y 2 y 1 y x C T c, w Depth map W T w, o O

19 Visual modalities Shape moments Intensity gradients Contour lines Color statistics Texture template Optical flow Local keypoints (and others: Background subtraction / CCD / Harris keypoints / Histogram of oriented gradients / SIFT)

20 The tracking pipeline Data acquisition Preprocessing Matching Targets update Output Targets prediction Off-line features sampling Data fusion On-line features update Back-projection s t s t s t Prediction Rendered view Sampling model features New features Update model features s t Data acquisition Image features Pre-processing Re-projection Matching

21 Abstraction: visual modality processing Rule: any modality class must implement Model free pre-processing Off-line and on-line features sampling and back-projection Multi-level data association (Pixel-, Feature-, State-space) Residuals, covariances and Jacobians computation Additional classes: Multi-modal, multi-sensor data fusion (cascade, parallel) Likelihood computation

22 Features sampling GPU assisted

23 Multi-level visual processing h(s) z e = z-h Pixel-level measurement h(s) z Feature-level measurement z = s* (LSE estimate) h = s - (predicted pose) Object-level measurement

24 Measurement: pixel- vs. feature-level Analogy with fluid mechanics Eulerian = pixel-level Lagrangian = feature-level v v Dense optical flow (HS) Sparse optical flow (LK)

25 Feature-level: re-projection vs. tracking Model feature re-projection Feature tracking (= flow )

26 Feature-level: validation gates (local search) x 2 x 1 x 3 Prior density (state-space) ( s, P Innovation densities (measurement space) ) ( yi, Si ) y, S 1 1 yi, S i y 3, S 2 2 y, S 3

27 Example: color histograms Model Color segmentation Pixel-level matching Feature-level matching = histogram distance Object-level matching = mean-shift optimization

28 Example: intensity edges Pixel-level Draw the silhouette Match silhouette to the Distance Transform sample contour points Feature-level Re-project and search in the image

29 Building in OpenTL Multi-camera, multi-level data fusion View 1 Color segmentation Pix Weighted Average Pix Blobs Feat s - Motion Pix Joint likelihood P( Z s ) = View 2 Edges Feat = P( Z s ) P( blobs Z obj MLE s ) Joint MLE Obj Keypoints Feat

30 Building processing trees in OpenTL Generalization of the tree Pix I 1 Mod 1 Static fusion Pix Mod 5 Feat s - Mod 2 Pix Dynamic fusion P( Z s ) Feat Mod 3 I 2 Static fusion Obj Mod 4 Feat

31 Data fusion multi-modal Background Color model Pixel-wise (AND) Benefits: Combine independent information sources Increase robustness (tracking fails if ALL modalities fail) Drawbacks: Need to define a proper fusion scheme, and parameters Higher computational effort slower frame rate

32 Data fusion multi-camera Complimentary setup Redundant setup

33 Data fusion multi-camera Complimentary setup: Indoor people tracking Redundant setup: 3D hand tracking

34 Multi-target occlusion handling Pixel-level Data (multi-class segmentation) Feature-level

35 Bayesian Tracking: prediction - correction Gaussian filters (Extended) Kalman filter Information filter Unscented Kalman/Information filter Monte-Carlo filters S-I-R particle filter MCMC particle filter

36 Representing the target distribution Prior density P( st zt 1,..., z0) Posterior density P( st zt, zt 1,..., z0) True density ML / MAP Kalman Unscented Kalman Mix. Of Gaussians Kernel Particle Histogram (Eulerian)

37 Flow diagram of OpenTL-based applications Track Maintainance t I t Local processing Meas t Obj t Bayesian tracking Obj t Post-processing I t Track Initiation t Detection/ Recognition + Obj t 1 Obj t 1 Models Shape Appearance Degrees of freedom Dynamics Sensors Environment

38 Object detection (examples) General-purpose Monte-Carlo sampling in state-space People detection based on foreground blobs clustering Invariant keypoints matching (for textured objects) Marker detection based on intensity edges Hand detection based on color and edge lines Face detection based on a trained classifier (with Haar features)

39 Resume: model-based object tracking Objects Sensors Environment Models Features Sampling Occlusion Handling Pre-processing Data association Visual processing Detection/ Recognition Object Tracking Tracking Target Prediction Target Update Measurement Data fusion

40 Building applications with OpenTL

41

42 Features of OpenTL Modularized, object-oriented software architecture Common abstractions for layers Real-time performance Different Bayesian filters A large variety of visual modalities, with multiple processing levels Robust improvements (multi-hypotheses, data fusion, ) Generic sensor abstraction Support multi-camera, multi-target and multi-modal applications Support for GPU acceleration

43 Application: planar object tracking Single-target, single-camera, color-based s Color histograms Z feat Likelihood P( Z feat s ) Particle Filter s Meas. processing Tracking Modeling

44

45

46 Application: planar object tracking Fusion color+background (pixel-level), and blob detection Color statistics Background subtraction Z pix s - Z pix Image fusion Z pix Blobs Z feat Likelihood P( Z feat s ) Particle Filter s

47 People tracking with distributed cameras Distributed, multi-camera setup CoTeSys ITrackU People detection and tracking in real-time 3D localization Goal: Human-Robot Interaction (coffee-break scenario)

48 Hierarchical grid-based localization

49 Stereo tracking of a quadcopter

50

51 Application: face tracking Cascade of two pipelines, with 3D pose upgrade s ( i) 2D s 2D CCD Z feat Object-level upgrade Zobj Kalman Filter s 2D s ( i) 3D s 3D (0) s 2 D s 3D 2D 3D pose upgrade Template matching Z feat Object-level upgrade Zobj Kalman Filter s 3D 2D tracking 3D model 3D tracking

52

53 Application: 3D hand tracking

54

55 Technische Universität München Our Tracking Projects Pedestrian Tracking Vivid cell tracking Human Robot Interaction VR TV Studio Automation Quadcopter tracking ITrackU (CCRL) Dr. Ing. Giorgio Panin

56 Technische Universität München Our Team graduate and post-doc coworkers student coworkers ITrackU (Panin) manifold research projects funded by DFG, EU, industry partners focus on robust, real-world applications Dr. Ing. Giorgio Panin