Motion and Target Tracking (Overview) Suya You. Integrated Media Systems Center Computer Science Department University of Southern California

Transcription

1 Motion and Target Tracking (Overview) Suya You Integrated Media Systems Center Computer Science Department University of Southern California 1

2 Applications - Video Surveillance Commercial - Personals/Publics - Environment/Wildlife animal monitoring - Traffic measurement Law enforcement - National security Military & defense 2

3 Sensor and Technology Cheaper, cheaper - Very prevalent in commercial/military establishments High performance - Millions pixels - Full range - Networked (wired/wireless) - On-board processors 3

4 Machine Vision Covers many of challenging issues Sensor & data acquisition - Multiple & distributed sensor network Scene analysis & understanding - Detection - Tracking - Recognition Data representation & comprehension - Object and environment modeling - Simulation and Visualization 4

5 Research & Systems Ground Small/modest-scale environment - Infrastructure, Military base - Intelligent traffic monitoring Airborne Large-scale environment - National Infrastructure, Battlefield Space Global/outspace - Battlefield, Environment monitoring, Mars 5

6 Example: Intelligent Traffic Monitoring Concept of Operations Sensor network Information processing Information access Distributed sensor network - Rectilinear CCD, omnidirectional, IR cameras - Location sensor - GPS - Fixed, active, and mobile - Networked wired and wireless Dynamic event detection & analysis - Target detection/tracking/recognition - Incident detection/classification/reporting 3D environment - 3D scene model (city 3D digital map) - Target 3D geo-localization - Immersive 3D visualization Real-time information access - Control center and drivers 6

7 Vision Processing Issues Camera modeling and calibration - Perspective, panoramic cameras - Allows automatic and on-site Dynamic image analysis - Dynamic target detection/tracking - Vehicle and people - Target recognition - Classification approximately - Active vision - Fixed and mobile platforms 3D processing - 3D scene modeling: - City model (building and road) - Target 3D geo-localization - Tracking and positioning in 3D world - Visualization - Immersive 3D (base station) - Abstract and full data (Web, drivers) 7

8 Challenges Camera modeling and calibration - Basic techniques are pretty as is - Main challenges are automatic and on-site calibrations - Model based approach given 3D model - Self-calibration vision approach included in the tracking module Dynamic image analysis - Outdoor imaging environment lighting, weather - Dynamic background modeling approach - Visual modeling finding imaging invariant (lighting, geometry) - Target detection/tracking long sequence, drifts, self-motion - Model based approach 3D scene - Distributed vision approach multi-view/camera geometry - Hybrid approach Active sensor (GPS/INS) aided vision - Active sensors aid video system - Reduces frame-frame vision processing - Video processing aids sensor performance - Allows estimate of camera attitude - Improves speed and accuracy 3D scene modeling - Urban site model (building and road) city scale, accuracy to level of street block, less manual interaction - Stereo approach still plays a main role - LiDAR is pretty new and promising approach - Ground based laser range finder 8

9 Challenges (con.) Heavy computation load is a main barrier - High resolution sensor better for image analysis (e.g. detection ) - Fast processing - can loose lots of vision processing jobs (e.g. tracking) - Multiple camera arrays huge data needs to be fused and computed - Users want the results what they are seeing Real-time vision computation - Developing fast algorithms e.g. Pyramid technique is a good example - Aided by other sensors e.g. Inertial sensor, GPS... - Hardware - General computer - Special CPU features (low-level programming) - Processor clusters (parallelization programming) - Special processor/board - DSP technique - FPGA technique (cheaper, flexible) - GPU power (CG language) - Smart camera (on-board processors) 9

10 Research Components (image related) Dynamic Global Image Construction and Registration Construct video Mosaic and register mission-collected video frames to previously prepared reference imagery in order to geolocate both moving and stationary targets in real time Multiple Target Surveillance Simultaneously track multiple moving targets in a sensor s field of regard Fixed sensors and active moving platforms (Satellite, UAV, robot) Activity Monitoring The monitoring of several areas of the battle space for distinctive motion activities such as a soldier incursion and vehicle movement 10

11 Motion Estimate A Pyramid-Based Approach Achieved through successive refinement within a multi-resolution pyramid structure - 2D motion flow estimation - Fit motion model (linear/nonlinear) - Warp to align Highly efficient can handle very large camera motions of the field of view, and provide very precise alignment 11

12 Multi-resolution Approach It s simple, but still very useful Target detection Motion tracking Navigation Compression It can handle large motion and be helpful for vision acceleration, but construction of itself needs extra computation Pyramid Vision Processor/Board Single-chip Simultaneous input/processing of up to 2 channels Real-time (30fps), low-latency processing (1-2 frame delay) 12

13 Robust Image Motion Estimation - Hybrid point and region - selecting good points and regions as tracking features - Multi-stage tracking strategy - multiresolution - A closed-loop cooperative manner integrating the feature detection, tracking, and verification Region/Point Detect & Select Multiscale Region Optical Flow Affine Region Warp and SSD Evaluation Linear Point Motion Refinement by Search Affine Region Warp and SSD Evaluation Iteration Control 13

14 Robust Image Motion Estimation (con.) Image i Source Region Affine model defines warp of source region to a confidence frame Affine Warp Image i+1 Target Region R t R t0 Confidence Frame R c SSD Normalized SSD measures the difference between warped source and target regions, thereby measuring the quality of tracking δ=(0,1] δ 1 = 1 + ε Rt ( x, t) Rc ( x, t) ε= 2 max{ R( x, t), R( x, t) t c 2 2 } 14

15 Performance Evaluation (a) detected tracking features (b) estimated motion field Synthetic image sequence (Yosemite-Fly-Through) Technique Average Angle Error Standard Deviation Horn and Schunck Lucas and Kanade Anandan Fleet and Jepson Closed-loop approach

16 Some Applications -Tracking for ground and Aerial image - Movie special effects including X-Men 2, Daredevil, and Dr. Seuss The Cat in the Hat. - Hardware implementation is under way (Olympus): PCMCIA size card 16

17 Video Stabilization/Mosaic Inter-frame image motion estimation (Parameters) Motion compensation and registration (Model) Image alignments and mosaicking (Composition) 17

18 Global Motion Compensation Image stabilization Registering the two images and computing the geometric transformation that warps the source image such that it aligns with the reference image cancel the motion of observer Registration model translation, affine, and perspective u( x, y) = v v( x, y) v 3 + v x+ v + v x+ v Model fitting An over-constrained SVD solution - Motion vector field (every pixel) - Feature based approach - Coarse-fine approach y+ ux 1 y+ u y u xy 2 + u xy 2 18

19 Video Stabilization/Mosaic Frame-to-Mosaic alignment Mosaic reference (first, middle, defined ) Warping each frame to reference Hierarchical alignment Temporal filtering (for mosaic) Intensity blending Weighted average blending function 19

20 Target Detection & Tracking Goal Moving target detection/tracking Vehicle and people Landmark recognition Interested buildings and reference features Platform Stationary sensors Ground cameras (perspective, panoramic cameras) Moving sensors Satellite, UAV, robot carried Image, GPS, and INS data are available 20

21 Stationary & Moving Platforms Stationary cameras Background is static - assumption Foreground is moving BG/FG classification Background matching Matching image Identification Tracking Moving cameras Background is moving camera motion Foreground is moving BG/FG classification Motion compensation Background matching Matching image Identification Tracking Challenges: Background modeling and maintaining Motion compensation (image stabilization) 21

22 Target Detection/Tracking (stationary sensor) Video image Preprocessing Video image Motion compensation Preprocessing Background model Background matching Background model Background matching Detection Tracking Detection Tracking 22

23 Background Modeling It s a challenging problem Appearance changes Time, lighting, weather Waking/sleeping objects BG objects moving, FG object still Color/contrast aperture Subsumed BG/FB, Homogeneous region Waving trees Vacillating BK Apparent Motion Camera motion 23

24 Constant Intensity Model Pre-defined constant BK Blue screen - movie special effect Everything is predefined no need to be estimated on-line Some preprocessing may be necessary log filtering Adjacent Frame Difference (AFD) approach Constant BK, but unknown BK is modeled as intensity constant Need parameter estimate/update on-line Mean Estimate Approach N 1 1 Linear model, i.e. m ( x, y) = mold ( x, y) + I( x, y) N N Mean-Covariance Approach Both m,σ need to be estimated Optimal estimators (Kalman filter) Block Correlation Matching Approach Block-wise median template Correlation matching 24

25 Statistical Feature Model Complex background Feature based approach - matching feature is a 4D Spatio-Temporal vector, i.e. m = BK is modeled as a certain statistical distribution in the 4D vector space 1 1 m ( m m )( m m ) N N = mean m, σ = i N i N 1 i Background update temporal blending i mean i mean T [ I, I x, I y, I t ] Single Gaussian Estimate approach m B new = (1 α ) B + α m old N 1 1 ( x, y) = mold ( x, y) m( x, y) N N new + σ new N N ( x, y) = σ old + ( m mmean)( m mmean) 2 N + 1 ( N + 1) T Mixture of Gaussian Estimate Approach BK is modeled as multiple Gaussian distributions Multiple frequency Gaussian channels Markov Model, EM (Expectation-Maximization) approaches 25

26 26 Motion Estimate Techniques Instead of using intensity constant constraint, BK is modeled as constant motion/optical flow field Matching feature is a 3D-vector, i.e. Background update is an optical estimation problem Extend to Multi-resolution detection and update Motion Field Model = t y x I v I u I ],, [ t y x I I I m = = t y t x y y x y x x I I I I I I I I I I v u 1 2 2

27 Prediction Model Statistical Prediction Techniques The BK pixels are predicted what are expected in next input frame B ( x, y) = i= 1 Linear estimation problem - LS, Wiener filtering E t i ( x, y) More complex prediction model is possible Motion/optical filed prediction model Non-linear prediction model t a 2 2 [ e ] = E[ I ] a E[ I I ] t t i I + i= 1 i t t i 27

28 Statistical Recognition Model Estimate as a Recognition Problem Training motionless background frames Feature extraction statistical image feature Eigenbackground PCA (Principal Component Analysis) Matching PCA projection Image space PCA Space Image space Live video projection Background training Foreground Background 28

29 More Techniques - The problem of above approaches is to separate three detection/tracking processes into independent phases - Low level pixel-wise detection/segmentations - Middle level labeling pixels as grouped targets - High level temporal-tracking, Spatio-recognition - An Integrated Approach integrating the pixel classification, region detection, and interframe tracking in closed-loop manner Pixel-wise processing (Segmentation) Region-wise processing (Clustering) Frame-wise processing (Matching) Linear prediction K-means clustering 29

30 More Techniques (con.) - Illumination invariant I = x, y ( x, y x, y x, y α ρ ϕ L ) γ Surface lighting model log( I x y ) = log( α ) + γ log( L x, y ) + γ log( ρ x, y ) + γ log( ϕ x,, y I ( x, y ) = L 1 ( 2 x, 3 y ) + M 14 ( 2 x 43, y ) illuminati on - dependent illuminati on - invariant ) L ˆ ( x, y ) = Filter ( I ( x, y )) Mˆ ( x, y ) = I ( x, y ) Lˆ ( x, y ) or mˆ ( x, y ) = exp( I ( x, y ) Lˆ ( x, y )) Illumination invariant Strong surface shading: effective Strong illumination gradients: less effective Low intensity: none or worst 30

31 Target Detection/Tracking (moving sensor) Moving cameras - Background is moving camera motion - Foreground is moving Motion compensation - Registering images and computing geometric transformation that compensates the source image such that it aligns with reference images Background model Detection Video image Motion compensation Preprocessing Background matching Tracking 31

32 Motion Compensation Parametric model translation, affine, and perspective x = v y v Model fitting v1x + v + v x+ v y + u1x y + u y u 2 + u 2 xy xy An over-constrained optimal estimate problem - It s hard BK contains moving objects - Motion vector field vs. Feature based approaches - Iterative vs. Non-iterative approaches 32

33 Motion Vector Field Estimation Parametric model Affine transformation x y Optical flow tracking and warping = Frame i-1 Source point R t0 v v 0 2,, v v 1 3 Affine Warp x y + v v 4 5 Affine model defines warp of source frame to a reference frame R c Frame i Target point R t Multi-resolution Iterative refinement SSD Normalized SSD measures the difference between warped source and target 33

34 Dynamic Object Tracking: Results Hand-held camera: Multiple objects Tracked object visualized in 3D Hand-held camera: Integration of mosaic, image stabilization, and object tracking 34

35 Dynamic Object Tracking: Results UAV sensor: Integration of mosaic, image stabilization, and object tracking 35

36 Feature Matching Parametric model Affine transformation x y = v v 0 2,, v v 1 3 x y + v v 4 5 Feature tracking and warping Affine (T1) Frame i-1 Frame i Feature selection (N) & SSD Affine (T2) Affine Warp Selection optimal T Affine (TM) Multi-resolution Iterative refinement 36

37 Others Perceptual Grouping Methodology - Tensor Voting - A simulation of perceptual organization infer what we perceive from noise/missing data - A Computational Framework for Segmentation and Grouping (formalized by USC Prof. Gérard Medioni) - Tensor Voting - Description data is represented as tensors to generate descriptions in terms of surface, regions, curves, and labeled junctions, from sparse, noisy, binary data in 2D/3D - Voting how the tensors communicate and propagate information between neighbors - Has been apply to many vision problems, including - Segmentation/detection - Motion tracking, Trajectory extraction - Stereo vision - Epipolar geometry estimation 37

38 Others (con.) Multi-view Cameras - Continuous cross-view tracking - Stationary platform Stationary platform - Stationary platform Moving platform - Moving platform Moving platform - Requires continuous and complete tracking trajectories - Requires trajectories and view points registrations 38

39 Others (con.) Omnidirectional Image - Wide (360 degree) horizontal FOV - Less partial occlusions - Less motion ambiguities (pure translation and rotation) - Limited resolution used for close range objects 39

40 Benefits Using Panoramic Imaging Wide FOV ensures: - A sufficient number of features for tracking - Less partial occlusion Accurate estimates for large motion - Provides sufficient information for distinguishing motion ambiguities (pure translation and rotation) 40

41 Others (con.) Integration of Imagery and Range Data - Wide coverage - Rapidness and robustness - Direct recover of 3D models and geolocations Camera parameters Live images Image warping Residual estimate Reference images DEM Space Filtering Detected targets LiDAR has accuracy typically as ~ m ground-spacing and centimeters height 41