Camera Motion Identification in the Rough Indexing Paradigm

Transcription

1 Camera Motion Identification in the Rough Indexing aradigm etra KRÄMER and Jenny BENOIS-INEAU LaBRI University Bordeaux I, France Camera Motion Identification in the Rough Indexing aradigm p./

2 Introduction Task: Given the shot boundary reference Identify the shots in which a certain camera motion (pan, tilt, zoom) is present Rough Indexing aradigm: Work on a lower spatial and temporal resolution i.e. -Frames Aim: Reuse motion low-level descriptors from the compressed stream Main challenge in TRECVID 00: Jitter camera motion due to hand-carried cameras Camera Motion Identification in the Rough Indexing aradigm p./

3 Overview -Frames Global Motion Estimation Significance Value Computation Motion Segmentation Thresholding Classification Motion feature j related to frames, m related to segments of homogeneous motion Camera Motion Identification in the Rough Indexing aradigm p./

4 Overview -Frames Global Motion Estimation Significance Value Computation Motion Segmentation Thresholding Classification Motion feature j related to frames, m related to segments of homogeneous motion Camera Motion Identification in the Rough Indexing aradigm p./

5 Global Motion Estimation Robust global motion estimator for -Frames [DB0]: Estimation of the affine D motion model: ( ) ( ) ( ) ( ) dx i a a a x i = + dy i a a a 6 y i Based on the weighted least squares method: ˆθ = (H T W H) H T W Z ˆθ Z H W = (a, a, a, a, a, a 6 ) T MEG motion compensation vectors macroblock centers weights defined by the derivative of the Tukey function Camera Motion Identification in the Rough Indexing aradigm p./

6 Global Motion Estimation The derivative of the Tukey function: ψ(r, λ r ) = { The weights are [OB9]: r(r λ r) if r < λ r 0 otherwise λ r r i = z i ẑ i z i threshold residuals w i = ψ(r i) r i i-th MEG motion vector Sfrag replacements -8 ẑ i estimation of z i ψ Camera Motion Identification in the Rough Indexing aradigm p./

7 Global Motion Estimation 0 Motion Compensation Vectors (9087) 0 Estimated Vectors (9087) a) c) b) a) -Frame motion vectors b) Estimated vectors c) Macroblocks: Outliers Dominant estimation support D (w i > 0) Camera Motion Identification in the Rough Indexing aradigm p.6/

8 Global Motion Estimation roblem: The global motion parameters are noisy due to jitter motions. The global motion parameters have different meanings. Solution: Significance test of the motion parameters: Thresholding of likelihood values Camera Motion Identification in the Rough Indexing aradigm p.7/

9 Significance Value Computation Based on [BGG99]: Change to another basis of elementary motion-subfields: φ = (pan, tilt, zoom, rot, hyp, hyp) with zoom = (a + a 6 ) rot = (a a ) hyp = (a a 6 ) hyp = (a + a ) Consider two hypotheses H 0 and H H 0 : the considered component of φ is significant with ˆφ 0 as the corresponding motion model H : the considered component of φ is not significant (= 0) with ˆφ as the corresponding motion model Camera Motion Identification in the Rough Indexing aradigm p.8/

10 Significance Value Computation Likelihood function associated to each hypothesis: Assumption: f( ˆφ l ) = ( ( π det(σ i D l ) exp ) ) (rt i Σ l r i ) = exp ( D ), l = 0, (πσ x,l σ y,l ) D Σ σ x, σ y D Σ l = ( σ x,l 0 0 σ y,l covariance matrix ) variances for x and y dominant estimation support Camera Motion Identification in the Rough Indexing aradigm p.9/

11 Significance Value Computation The significance value s is: s = ( f( ln ) ) f( ˆφ = D (ln(σ x,0 σ y,0 ) ln(σ x, σ y, )) 0 ) = D ( ln(σ0) ln(σ) ) assuming that σ x = σ y Aim: Use s to test the significance Idea: If a motion feature (pan, zoom, tilt) is present in a shot, its corresponding motion parameter is significant during a sufficient number of frames. Camera Motion Identification in the Rough Indexing aradigm p.0/

12 Significance Value Computation roblem: Solution: The significance values can be noisy due to jitter motions. The motion models ˆθ can be inaccurate. Smooth the significance value along the time and take decision on the temporal mean value. > Segment shots into subshots of homogeneous motion Introduce confidence measures in order to reject frames with an inaccurate motion model. Camera Motion Identification in the Rough Indexing aradigm p./

13 Significance Value Computation Two reasons for inaccurate motion models: Failure of the MEG encoder > Confidence measure c D D Failure of the global motion estimation algorithm > Confidence measure c σ σ 0 Reject of the frame if: c D < λ D c σ > λ σ λ D λ σ threshold threshold Camera Motion Identification in the Rough Indexing aradigm p./

14 Motion Segmentation Hinkley test to detect changes on the temporal mean value s(t): Downward jump: Upward jump: U k = k t=0 ( s t s + δ ) min (k 0) M k = max 0 i k U i; detection if M k U k > λ H V k = k t=0 ( s t s δ ) min (k 0) N k = min V i; detection if V k N k > λ H 0 i k s δ min λ H temporal mean value minimal jump magnitude predefined threshold Camera Motion Identification in the Rough Indexing aradigm p./

15 Motion Segmentation rinciple of the Hinkely test: s and s Down M k U k Up V k N k Camera Motion Identification in the Rough Indexing aradigm p./

16 Thresholding Selection of the hypothesis: s(t) = T t 0 t=t t=t 0 s(t) And relative thresholding to determine the dominant motion: ζ(t) = { s(t) if s(t) < α min{ s pan, s tilt, s zoom, s rot, s hyp, s hyp } 0 otherwise T t 0 λ s α H 0 < > H λ s segment of homogeneous motion threshold constant Camera Motion Identification in the Rough Indexing aradigm p./

17 Classification The following classification scheme is applied to the thresholded mean significance values ζ = ( ζ pan, ζ tilt, ζ zoom, ζ rot, ζ hyp, ζ hyp ): ζ motion feature (0, 0, 0, 0, 0, 0) static camera/ no significant motion ( ζ pan, 0, 0, 0, 0, 0) pan (0, ζ tilt, 0, 0, 0, 0) tilt ( ζ pan, ζ tilt, ζ zoom, 0, 0, 0) zoom others complex camera motion Camera Motion Identification in the Rough Indexing aradigm p.6/

18 Classification ostprocessing: Join neighbored segments with the same motion feature Reject segments with a duration shorter than t min frames t min t Sfrag replacements If a motion feature is still present: The shot is identified to contain the motion feature. Camera Motion Identification in the Rough Indexing aradigm p.7/

19 Results Results for the shot shot06_6 : pan static tilt zoom eplacements Sfrag replacements a) λ s frame number b) λ s 0 reject a a a a a a6 reject reject pan static tilt zoom frame number pan tilt zoom rot hyp hyp reject eplacements c) reject pan static tilt zoom frame number pan tilt zoom rot hyp hyp λ s reject a) Global motion parameters ˆθ b) Significance values s c) Online mean values s Camera Motion Identification in the Rough Indexing aradigm p.8/

20 Results recision and recall for all submissions: recall U yu D R RS Labs RI HU VISION 0LF Marburg A_CAM RI > LaBRI recision 0.9 Recall precision Camera Motion Identification in the Rough Indexing aradigm p.9/

21 Conclusion and erspectives Conclusion: roposition of a method based on global motion estimation and significance test. The proposed method can handle moving objects and jitter motions. No decoding of the compressed stream. erformance - times faster than real time. Since no ground truth available, difficulties to determine the best parameter set. Future work: Focus mainly on the correction of motion models if the encoder block-matching algorithm fails. Camera Motion Identification in the Rough Indexing aradigm p.0/

22 References [BGG99]. Bouthemy, M. Gelgon, and F. Ganansia. A unified approach to shot change detection and camera motion characterization. IEEE Trans. on Circuits and Systems for Video Technology, 9(7):00 0, October 999. [DB0] [OB9] M. Durik and J. Benois-ineau. Robust motion characterisation for video indexing based on MEG optical flow. In International Workshop on Content-Based Multimedia Indexing, CBMI 0, pages 7 6, 00. J.M. Odobez and. Bouthemy. Robust multiresolution estimation of parametric motion models. Journal of Visual Communication and Image Representation, 6():8 6, 99. Camera Motion Identification in the Rough Indexing aradigm p./