Segmentation, Classification &Tracking of Humans for Smart Airbag Applications

Transcription

1 Segmentation, Classification &Tracking of Humans for Smart Airbag Applications Dr. Michael E. Farmer Dept. of Computer Science, Engineering Science, and Physics University of Michigan-Flint Importance of Topic Between 1986 and 2001 airbags caused the following deaths, mostly during low speed crashes: 19 infants in RFIS 85 children 7 adults due to proximity at time of deployment Unfortunately, as recently as 1998: over 50% of a NHTSA survey respondents had placed their children in the front seat of their vehicle over the last 30 days Deaths may continue until a system to automatically disable the airbag is developed. 1

2 Automotive Occupant Classification &Tracking Problem National Highway Transportation Safety Administration (NHTSA) mandate (2004) Requires two suppression conditions: Suppress the airbag if child or infant Detect change within 10 seconds Suppress the airbag if occupant is within Automatic Suppression Zone (ASZ) Disable bag within 20 msecs Definition of Automatic Suppression Zone (ASZ) Oriented along Instrument Panel Distance from Airbag where risk of injury in minimized ASZ ASZ 2

3 NHTSA Approved Methods for Performing Airbag Suppression 1-year old infant 3 to 6-year old child Adult Protected by Classification Suppress if Present Suppress if Present OR Protected by Tracking Suppress if in ASZ Suppress if in ASZ If decide to protect children by classification then get 4-class problem (infant, child, adult, empty) If decide to protect children by tracking then get 2-class problem (infant versus all else) 4-class Classification Problem Infant Child Adult Empty 3

4 2-class Classification Problem Vs. Adults Infants Difficulties with Occupant Classification Large Intra-class Variability Infant Class Adult Class 4

5 Difficulties with Occupant Classification II Low Inter-class Variability for 4-Class Classification Problem 6 Year-old on Booster 5th % Adult Female Note that the 6 YO is actually taller in these images when seat is in forward position for child and rear-most position for the adult. Difficulties with Occupant Classification - System experiences extreme variations in illumination 5

6 Difficulties with Occupant Tracking Extensive occupant deformation during movement Occupant occlusion Variability in size of occupant Summary of Difficulties with Occupant Airbag Suppression Problem Large intra-class variability of the various occupant types Low inter-class variability for 4-class classification problem Camouflaged classes (e.g., blanketed RFIS) Large variation in illumination Severe automotive environmental conditions Low cost Extremely high reliability and performance 6

7 Training images: Test images: Summary of Image Data Used Occupant Type Classification Number of Images RFIS+FFIS Infant 2657 Child Child 620 Adult Adult 983 Empty Seat Empty 72 Total number of images: 4332 Occupant Type Classification Number of Images Infant (RFIS+FFIS) Infant 1807 Child Child 236 Adult Adult 210 Empty Seat Empty 8 Total number of images: 2261 System Algorithm Architecture Standard components of pattern recognition system: 1. data acquisition and pre-processing 2. data representation 3. decision making Every 3 seconds Classifier Segmenter Feature Extraction Occupant Classifier Input Image Every 1/40 second Tracker Segmenter Occupant Model Head/Torso Tracker & Predictor 7

8 Approach for Classifier Segmentation Raw Image Classifier Segmenter Feature Extraction Occupant Classifier Pal and Pal state: hundreds of segmentation techniques in the literature, but there is no single method which can be considered good for all images Pal and Pal also state: semantics and prior information about the type of images are critical to the solution of the segmentation problem In light of these, we will utilize all the information we have regarding the interior of the vehicle The approach we will take is one of background removal. Segmentation Processing For Background Removal tested 2 methods De-correlation Processing Eigen-images Results: De-correlation processing outperforms Eigenimages Also developed 2 post-processing methods Hole filling using binary morphology Use closing: ( A B ˆ ) ΘB Further background reduction with Watershed 8

9 Eigen-image Processing T L = ΦCΦ Φ Mb Used sequence of 722 images to compute covariance matrix p new T Mb = Φ ( I new µ new ) I background = Φ p Mb new + µ new I I > threshold new background De-correlation Processing [ ] grad T g( x, y) g( x, y) AB g( i, j) = x, grad C = y 2 g( x, y) g( x, y) A B 2 9

10 Eigen-image Processing Results Input Image Transformed Image Difference Image Histogram of Difference Image Thresholded Difference Image Segmentation Processing Results Input Image Reference Image De-correlation Image Thresholded De-correl. Image Postprocessing Input Image Hole Filling Watershed Image 10

11 Approach for Feature Extraction Raw Image Classifier Segmenter Feature Extraction Occupant Classifier Devijer and Kittler define Feature Extraction as: extracting from the raw data the information which is most relevant for classification. feature extraction is probably the single most important factor in achieving high recognition performance Feature Extraction Taxonomy Content Retrieval Methods Shape Description Texture Description Color Description Spatial Location Boundary Methods Region-based Methods Structural Transform Domain Spatial (Geometric) Domain Transform Domain Spatial (Geometric) Domain 11

12 Feature Extraction - Child on seat has same boundary as empty seat so cannot use boundary methods Recall we segment the occupant & the seat - Child and infant have common color/grayscale & texture distribution so cannot use color/grayscale or texture - We will use region methods to characterize the occupant Legendre Moments of Edge Image for Feature Extraction Segmented Images Edge Images 12

13 Legendre Moment Reconstruction Infant Image Adult Image 25 Order (351 features) 35 Order (666 features) 45 Order (1081 features) Will use the highest order to capture the most internal features Results for Feature Extraction Tested the following different moment features: Geometric, Legendre, and Zernike Recall we do not want moment invariants, since size of occupant depends on spatial location Tested moments based on separation of features using Mann- Whitney distribution test 13

14 Feature Selection Taxonomy Feature Selection Methods Complete Random Heuristic Exhaustive Non- Exhaustive Type I Type II (SA, GA, RMHC) All Breadth First Branch & Bound Best First Beam Search Forward Selection Backward Selection Will test our own Mann-Whitney as well as Mutual Information for Best First Instance Based (RELIEF) Mann-Whitney Feature Selection 1. Mann-Whitney sorts features based on separability First compute the mean rank sums n µ = σ A AB A n = ( N + 1), and 2 A n ( N + 1) B. 12 z = n and n are the number of samples from classes A and A B B and S is the actual sum of the ranks A 2. Correlation post processing tests and removes correlated features. Cov ( ) ( A, B) ) ρ A, B = Use Spearman-R coefficient 2 2 σ ( A) σ ( B) Cov(A,B) is covariance of the ranks ( S ) A µ ± 0. 5 σ A AB 14

15 Discrimination Ability of Features Correlated Features Removal 15

16 Results for Feature Selection Feature Selection Method Classification Accuracy # of Features Retained Two-class Four-class Two-class Four-class Using all Features Random Mutation Hill Climbing ( α = 0. 2 ) Random Mutation Hill Climbing ( α = 0. 8 ) Forward Sequential Search * 9 13* Mann-Whitney w/ Correl. (all 1081) Mann-Whitney w/ Correl. (retain top 200) Mann-Whitney w/ Correl. (retain top 120) Mutual Information w/ Correl. (all 1081) Mutual Information w/ Correl. (retain top 200) Mutual Information w/ Correl. (retain top 120) * FSS results after 20 days execution, rather than after convergence Classifier tested on extended drive sequence FSS-+l-R selected features (41% correct) 16

17 Classifier tested on extended drive sequence Mann-Whitney selected features (64% correct) Results for Feature Selection - Another consideration is processing time for massive scale feature selection problems such as ours where we started with 1081 features. Method Mann-Whitney w/ Correlation Time < 10 minutes Random Mutation Hill 3.25 hours Climbing Forward Sequential Search ~ 1 day per output feature (2- class) ~ 2 day per output feature (4- class) 17

18 Approach for Occupant Classifier Raw Image Classifier Segmenter Feature Extraction Occupant Classifier Prior Information COMPLETE INCOMPLETE Bayes Decision Theory Supervised Learning Unsupervised Learning Parametric Approach Nonparametric Approach Parametric Approach Non-parametric Approach "Optimal" Rules Plug-in Rules Density Estimation Geometric Rules (K-NN, Tree, ) (MLP,rule-based) Mixture Resolving Cluster Analysis (Hard, Fuzzy) Note: we will also attempt to exploit history in the classification since we know the occupant cannot change type while driving. Occupant Classifier Tested the following classifiers: Bayes k-nn Modified k-nn SVM Performed 2 types of testing 50/50 Cross validation Independent test data set 18

19 Developed modified k-nn Algorithm for Classification Traditional k-nn Average Distance k-nn Results for Occupant Classifier Two-class Four-class Classifier Method 50/50 Independent 50/50 Independent Bayes 99.7 % 98.2 % 98.4 % 88.4 % k-nearest neighbor (k=9) Modified k-nearest neighbor (k=9) Support Vector Machine 99.0 % 95.1 % 97.7 % 88.2 % 99.5 % 96.7 % 98.5 % 88.2 % 98.9 % 96.8 % 93.6 % 87.8 % Performed 2 types of testing: 50/50 Cross validation Independent test data set 19

20 Contextual Processing for Classifier Errors in classification output have two sources: 1) Random errors: Due to the inherent overlap in the class decision boundaries 2) Deterministic errors: Due to the movements and behaviors of the occupant Contextual processing provides a historical integration of data to reduce both types of error Appears as adult Appears as infant Contextual Processing Details Uses Dempster-Shafer Evidential Reasoning Explicitly allows the use of ignorance in cases where the occupant type suddenly changes. Based on set theory probability mass is assigned at a subset level: Θ 2 = { { infant}{, child},{ adult},{ empty}, { infant, child}{, infant, adult}{, infant, empty}, { child, adult}{, child, empty}{, adult, empty}, { infant, child, adult}{, child, adult, empty}{, infant, adult, empty}, { infant, child, empty}{, child, adult, empty}, { infant, child, adult, empty}} 20

21 Typical Image Sequence for Contextual Processing Results for Contextual Processing Incoming Confidences After Dempster-Shafer P(class) Image Frame Number Class P(class) Incoming Classifications Image Frame Number Classifications After Dempster-Shafer Pattern Class Image Frame Number Image Frame Number 21

22 Confidences after Dempster-Shafer History Processing Effects of history on the extended video sequence (80.0% correct) 22

23 Dynamics of ATD During Pre-crash Braking Approach for Tracker Segmentation Raw Image Tracker Segmenter Occupant Model Head/Torso Tracker & Predictor Segmentation for image sequences has two key differences: 1) object motion can be used as an additional cue for segmentation 2) image sequence segmentation must be accomplished at a considerably faster (full real-time) to support tracking. Two of the more common approaches: 1) template tracking (solid 2-D,line, and point-set templates) 2) optical flow We will define a 3rd method based on mutual information in an image stream 23

24 Results for Tracker Segmentation Template Matching Optical Flow Results for Tracker Segmentation II Template Matching Optical Flow Note while the optical vectors are erroneous, we are still able to use the output for segmentation, while the template matching fails. 24

25 Occupant Shape Modeling Raw Image Tracker Segmenter Occupant Model Head/Torso Tracker & Predictor Bounding Ellipse of Head and Torso Subject Y-axis Centroid Major Axis θ X-axis Minor Axis Occupant Motion Modeling Raw Image Tracker Segmenter Occupant Model Head/Torso Tracker & Predictor Human Activity Recognition Template Matching State-space Points Meshes Posture Sequences Dynamics Sequences Points Lines Blobs 25

26 Occupant Motion Modeling Model human dynamics by three distinct types of motion Commonly use HMMs to model the transitions between states We use an alternative method called Interacting Multiple Models (IMMs) which allows mixing of the separate states. Stationary Human Crash Approach for Occupant Tracking Raw Image Tracker Segmenter Occupant Model Head/Torso Tracker & Predictor Recall the two representations required for tracking an object (shape and motion) Implies there should also be two trackers running: Motion tracker Will use Interacting Multiple Model (IMM) Kalman filter Shape tracker We have defined a new method called Shape from Deformation 26

27 Performance of Tracker for Pre-crash Braking Intrusion Time Results for Robotic Tester HTT Response Delay Count Response Delay (msec) Maximum 5.3 Minimum -21 Average Standard Deviation

28 Performance of Tracker for Human Occupant Shape from Deformation Concept All SFM algorithms rely on point correspondences between image frames. 3-D structure is computed from the relative motions of these points Must define image points that are suitable for tracking. Recent research on using image deformations Infer the direction of motion of an object Still relies on point tracking body. Point tracking is not suitable for the human body It is not rigid and hard to define good points It is also not randomly deformable so may be able to use deformations to infer 3-D shape or orientation 28

29 Future Research - Classification Feature Selection We will Test Mann-Whitney with correlation postprocessing on other massive-scale feature selection problems Wrapper-based segmentation algorithm Apply the wrapper method on other databases Attempt to extend paradigm to general content-based image retrieval application and image data mining Contextual Processing Explore use of Evidential Reasoning for classifier combining Explore integrating track information with classsifier results 29