Multiple target detection in video using quadratic multi-frame correlation filtering

Transcription

1 Multiple target detection in video using quadratic multi-frame correlation filtering Ryan Kerekes Oak Ridge National Laboratory B. V. K. Vijaya Kumar Carnegie Mellon University March 17,

2 Outline PART I Correlation filtering overview Database PART II Multi-frame correlation theory Multi-frame simulation results 2

3 Why use correlation filters? Shift invariance: useful when we don t know: Target locations in the scene Number of targets in the scene Distortion tolerance: CF s can be trained to tolerate some distortion and reject false targets Graceful degradation: noise, occlusion Training pattern Scaled versions False patterns CF 3

4 Correlation peak metric Often we will measure peak sharpness Peak-to-sidelobe ratio (PSR) 4

5 Correlation peak metric Often we will measure peak sharpness Peak-to-sidelobe ratio (PSR) metric invariance to overall image brightness PSR =

6 Correlation filter usage Typically computed in the frequency domain 2 FFTs complexity reduction PSR computed at every point in frequency domain 4 more FFTs FT of PSR window FT of filter FFT * IFFT 4 FFTs input correlation array 6 PSR array

7 Linear filter vs. quadratic filter Consider thresholding the output results in a discriminant at each shift class 1 class 2 pixel 3 pixel 3 example: elliptical boundary pixel 2 pixel 2 threshold threshold linear filter quadratic filter 7

8 Turntable database Captured rotational imagery of three targets Depression angles used: 17, 19, 21, 23 Green background used to aid in segmentation camera 211cm 17 target green background turntable 8

9 Filter training Each filter trained on scaled and out-of-planerotated images of true- and false-class targets True Missile Scale (PHTW) 16 False (see n) Leopar d Azimuth False (unsee n) Abrams 9

10 Filter comparisons Showed that RQQCF filters outperform others each filter bank trained to divide up 360 azimuth range different number of filters/bank, equal computation LEGEND filter design EOTSDF UOTSDF EMACH CPCF/ UPCF RQQCF/ SSQSDF Eigen-filter Unconstrained OTSDF Extended MACH Polynomial CFs Quadratic filters 10

11 PART II Multi-frame correlation filtering MFCF 11

12 CF in video sequences Three types of approaches to the problem TYPE 1: SINGLE-FRAME Frame 1 Test images CF Correlation Detections planes THRESHOLD Frame 2 Frame 3 CF CF THRESHOLD THRESHOLD 12

13 CF in video sequences Three types of approaches to the problem TYPE 2: DETECTION-FIRST (CLASSICAL TRACKING) Frame 1 Test images CF Correlation Detections planes THRESHOLD Frame 2 Frame 3 CF CF THRESHOLD THRESHOLD Relate information 13

14 CF in video sequences Three types of approaches to the problem TYPE 2: DETECTION-FIRST (CLASSICAL TRACKING) Frame 1 Test images Information loss due to thresholding CF step Frame 2 CF Correlation Detections planes THRESHOLD THRESHOLD Frame 3 CF Data association problem due to false alarms THRESHOLD Relate information 14

15 CF s in video sequences Three types of approaches to the problem TYPE 3: RELATION-THEN-DETECTION (our work) Frame 1 Test images CF Correlation planes THRESHOLD Detections Frame 2 Frame 3 CF CF THRESHOLD THRESHOLD Relate information 15

16 Efficient output combination We want to combine outputs while preserving information Avoid preliminary thresholding step Avoid assumptions on number of targets Avoid a large computational increase We have developed a probabilistic approach We can derive defensible theory We can confidently utilize all available information Pr ( target frame1, frame 2, ) 16

17 Typical detection schematic 17

18 Probability mapping We map each correlation value to probability of target presence using all available information high likelihood of target (PSR 10) reasonable likelihood of target (PSR 4) mapping PSR array probability array 18

19 PSR score distributions Must assume distributions for target scores and non-target scores for each filter (we used Gaussians) Example from a FLIR sequence target non-target (clutter) 19

20 Target motion model Must assume transition probability function between adjacent frames can take velocity into account Function of displacement (position-independent) (we used centered, rotationally symmetric functions) Gaussian exponential uniform 20

21 Posterior probability array We can derive a pixel-wise mapping function for the posterior probability array information from Frame 3 enhanced output Frame 3 output map knowledge from Frames 1 and 2 Frame 3 prior probs. Frame 3 posterior probs. 21

22 Mapping function Various properties of the mapping function Already normalized frame index filter communication Uses PSR scores from all filters sub-class (filter) index Uses prior arrays from all filters i-th frame posterior probability array i-th frame prior probability array score pdf of filter l on target k 22

23 Mapping function Example of a mapping in a two-class problem PSR value target probability 23

24 Prior probability array We can derive the prior probability array for the next frame: based on convolution predictor for Frame 4 motion model prior formula knowledge from Frames 1, 2, and 3 Frame 4 prior probs. Frame 3 posterior probs. 24

25 Prior array formula Various properties of the prior array formula probability of target emergence ε(x) non-zero probability region, zero probability region i-th frame posterior array i-th frame prior array Motion model function Emergence/disappearance probability functions 25 Subclass transition probs.

26 Prior array formula Various properties of the prior array formula probability of target disappearance non-zero probability region δ(x) zero probability region, i-th frame posterior array i-th frame prior array Motion model function Emergence/disappearance probability functions 26 Subclass transition probs.

27 Prior array formula Various properties of the prior array formula only one convolution per filter * i-th frame posterior array i-th frame prior array Motion model function, Emergence/disappearance probability functions 27 Subclass transition probs.

28 Prior array formula uses posterior arrays from all Various filters filter properties of the prior array formula communication subclass transition probability, i-th frame posterior array i-th frame prior array Motion model function Emergence/disappearance probability functions 28 Subclass transition probs.

29 Multi-frame schematic Frame i priors Frame i output map Frame i Motion model Frame i posterior probabilities Frame i+1 priors prior formula THRESHOLD (to next iteration) detections 29

30 Multi-frame schematic Frame i priors Table lookup Frame i output map Frame i 2 FFTs Motion model Frame i posterior probabilities Frame i+1 priors big formula THRESHOLD (to next iteration) detections 30

31 Synthetic sequence demo Sequence Original output Prior probability array Posterior probability array PSR values log-likelihood values 31

32 Synthetic sequence demo MFCF can handle multiple targets Example: sequence with 3 true-class targets, 3 false-class MFCF single-frame MFCF detections 32

33 Synthetic sequence demo False alarms reduced in target and non-target 90% detection rate single-frame MFCF target paths 33

34 FLIR sequence results Two FLIR sequences tested Two types of noise each MFCF offers greater performance improvement in white noise than in compression noise Sequenc e L2117 MFCF AWGN, SNR=20dB H.264 compression noise 34 (65:1)

35 Noise vs. clutter in MFCF MFCF favors filters with better clutter rejection Example: sequence Synthetic3, noisy (SNR = 20dB) Varied number of eigenvalues (N e ) retained in filter design single-frame worse than single-frame multi-frame 35

36 Summary of main findings MFCF algorithm: MFCF is best-suited for handling temporally uncorrelated noise Clutter rejection should be handled by the filter(s) Filter communication can help reject false targets MFCF degrades gracefully under parameter changes 36

37 Future work Multi-view correlation filtering (multiple cameras) View 1 View 2 scen e 37