Multiview 3D tracking with an Incrementally. Constructed 3D model

Transcription

1 Multiview 3D tracking with an Incrementally Tomáš Svoboda with Karel Zimmermann and Petr Doubek Czech Technical University Prague, Center for Machine Perception Last update: September 22, 2006; Working document Talk Outline Motivation Incremental model construction Illumination compensation Discussion, problems Constructed 3D model

2 Motivation a typical office scene Detect and track persons in the scene 2/20 video Note: limited visibility, different patterns of activity...

3 Motivation MultiCam approach 3/20 Simple FireWire cameras are cheap. Full calibration is easy. video,

4 Model based approach model known in advance 4/20 Segmented data Structure of the model known Size parameters and kinematics estimated Useful if complete visibility video 1 1 Input data kindly provided by Lars Mündermann, Stanford BioMotion lab

5 Connected component tracking static cameras assumed 5/20 motion segmentation biggest connected component is detected head is supposed to be on top segmentation required problems with crowd video

6 Head tracking based on segmentation static cameras assumed 6/20 motion segmentation, object boundary ellipsoid is tracked by SMC (particle filtering) + surprisingly stable and efficient segmentation required no head orientation video

7 Our Programme 7/20 significant occlusions various activities head orientation important learn models from the data

8 Interleaved modeling and tracking - Principle 8/

9 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/

10 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ reconstruction 1 3 Tracking starts in 1,2 4

11 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ Tracking in 1,2 3 4

12 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ Tracking in 1,2,3 4

13 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ Tracking in 1,2,3 4

14 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ reconstruction 2 3 Alignment by tracking 4

15 Interleaved modeling and tracking - Principle Stereo cams 1 2 8/ complete model 3 Tracking in 1,2,3,4 4

16 Radim Stereo - oriented points Šára fish-scales [ICCV98, ECCV02] 9/20 video

17 3D Tracking - Problem definition Motion model between two consecutive time instances 10/20 x = Rx + d (I + [u] )x + d where [u] = d = d 1 d 2 d 3 0 u 3 u 2 u 3 0 u 1 u 2 u 1 0 is a translation. is an infinitesimal rotation, There are six parameters [d1, } d2, {{ d3 }, u } 1, u {{ 2, u } 3 ] T to be optimized. d u

18 3D Tracking - Criteria function Criteria function F : R6 R is the point-wise colour dissimilarity between model M projection and observation J: [ F (d, u) = M (x) J ( f (x + [u] x + d) )] 2 dx. 11/20 x M

19 3D Tracking - Approximation Criteria function. We want closed-form solution for d, u F (d, u) = [ M (x) J ( f (x + [u] x + d) )] 2 dx. 12/20 x M

20 3D Tracking - Approximation Criteria function. We want closed-form solution for d, u F (d, u) = [ M (x) J ( f (x + [u] x + d) )] 2 dx. 12/20 x M the unknowns d, u are factored out from J(f (x + [u] x + d) ) by the first order Taylor approximation J ( f(x + [u] x + d) ) J ( f(x) ) + J T ( f(x) ) f (x) }{{} g T ([u] x + d) we obtain F (d, u) = x M [ M (x) J ( f(x) ) g T ([u] x + d)] 2dx.

21 3D Tracking - Solution The local extreme conditions 13/20 F (d, u) d = 0, F (d, u) u = 0 yield 6 6 linear system with following solution [ u d ] = A 1 b, where A = b = [ ] (g x)(g x) T (g x)g T g(g x) T gg T [M(x) ( )] [ ] (g x) J f(x), g,

22 3D tracking - iterative method 14/20 [ u d ] = A 1 b,

23 3D tracking - iterative method 14/20

24 3D tracking - iterative method 14/20

25 3D tracking - iterative method 14/20

26 Compensation of uneven illumination idea Model intensity depends on albedo and light sources. 15/20 Model points X are clustered into n groups G1,..., Gn according to their normals. Illumination of i-th cluster is compensated by a intensity correction factor e i. Simultaneously, each point is projected into m different cameras, where k-th camera is considered to have a different color properties compensated by color correction matrix H k.

27 Compensation of uneven illumination example 16/20 left light right light Left: The image with projected model. Colors correspond to the computed illuminance e i of each particular cluster. Right: Values of e 6 during the the 360-turn.

28 Interleaved modeling and tracking - Results 17/20 video video video

29 3D tracking algorithm that: Summary 18/20 needs no model at the beginning (can track object) builds model incrementally from observations video any stable tracks a complete turn-around (no problem with out-of-plane rotation) Ongoing work Towards simple articulated structures. (Known) problems depends on stereo gradient based method, local extrema

30 References 19/20

31 End 20/20

32

33

34

35

36

37 1 Stereo cams

38 1 Stereo cams reconstruction 1 3 Tracking starts in 1,2 4

39 1 Stereo cams Tracking in 1,2 3 4

40 1 Stereo cams Tracking in 1,2,3 4

41 1 Stereo cams Tracking in 1,2,3 4

42 1 Stereo cams reconstruction 2 3 Alignment by tracking 4

43 1 Stereo cams complete model 3 Tracking in 1,2,3,4 4

44

45

46

47

48

49

50

51

52

53

54 left light right light

55