A global issue in Computer Vision and Cognitive&Physiology Vision (

Transcription

1

2 Motion

3 A global issue in Computer Vision and Cognitive&Physiology Vision (

4 Are we working on real videos in which spatial redundancy is pre-processed? Much more on Image Sequence or stacks : Def: An Image sequence is a series of N images, or frames, acquired at discrete intervals of time tk=t0+k t, where t is constant and k=0, 1,, N-1. Note : we need a frame grabber able to store frames at high rates and high volumes ( Frame rate : t =1/24s or field rate : t =1/30s, time lapse t =1 image every hour). FYI : retinal persistence 1/12s -> video illusion?

5

6 Importance of visual motion In human or computer vision, apparent motion of objects on 2D image plane is an important visual clue to understand the 3D motion AND structure in a scene Shape from Motion topic : Or Shape segmentation

7

8

9

10 Visual Motion Importance

11 What can we infer from 2D motion? Change detection Litterature : Background Subtraction Foreground Detection

12 Optical-flow based analysis Litterature : Optical Flow Estimation

13 Tracking Litterature : Tracking of : Shape, Spots and Objects

14 Model-based tracking Litterature : Tracking Shape, Spots and Objects

15 And so more... Video Mosaicing / Stitching Litterature : Registration Alignment MATCHING!! Video Compression

16 What kind of information to extract? Low Conceptual Level High Conceptual Level 3D RECONSTRUCTION

17 The two Universal Vision issues 1. MATCHING (or registering, tracking etc.) : Which elements of the frame t to match with those of frame t'? Two theoretical frameworks: Differential approaches (PDE or so) : in output, we get dense measures, i.e.. computed for every pixel of each frame. We use temporal derivatives of the signal because of the t'-t << epsilon hypothesis. Matching or tracking approaches (Kalman or so) : in output, we get sparse measures, i.e. computed on a subset of image feature points (SIFT, Harris etc.)

18 The two Universal Vision issues 2. SEGMENTATION (motion-based) : FishTank.avi What are the various semantic objects in the scene inferred from intensity, colour AND motion (easiest so?) Regions of different moving Matching? objects : Segmentation Problem Problem

19 Now it's time for :-) bio-medical imaging

20 C.elegans developing embryo (3D) Waterston Lab -The George Washington University. Washington DC (USA) Microscope: Zeiss LSM 510 Meta Objective lens: Plan-Apochromat 63X/1.4 (oil) Pixel size (microns): 0.09 x 0.09 x 1.0 Time step (min): 1 or 1.5 From ISBI cell-tracking challenge: Sea urchin embryogenesis Nicolas Minc Lab -Institut Jacques Monod Université Paris Diderot (France), SPC

21 In bio-medical imaging, basically, video sequence analysis = registration or tracking issues

22

23

24

25

26

27 Take Home Messages Applications (Computer Vision vs. Bio-Medical Imaging) are different but techniques are similar Differences : the constraints and a priori knowledge (like anatomy) You really learn and understand by teaching or implementing :-) And then a theoretical example...

28 Motion

29 Objectives : Understand the concept of motion field and optical flow; Knowledge of the brightness constancy equation; Being able to implement methodologies/algorithm for optical flow computation;

30 RealTITracker A toolbox for real-time 2D/3D optical flow based medical image registration

31 SIFT (Scale Invariant Feature Transform) Detection And Caracterisation

32 Matching by RANSAC

33 The matching issue: block matching strategy Greedy search? Left Right

34 Block-Matching principle

35 Algo CORR_MATCHING INPUT : Pair of images Il and Ir Let pl and pr be the pixels in images Il and Ir respectively. 2W+1 the size in pixels of the correlation window/matrix. R(pl) the search region in image Il corresponding to pl. Let (u,v) be a function of two pixel values. For every pixel pl=[i,j]t in image Il : For every displacement d=[d1,d2]t R(pl) compute : W c( d ) W ( I (i k, j 1), I (i k d, j 1 d l r 1 2 )) k W k W The flow/displacement of pl is the vector d d1, d 2 T maximizing c(d) over R(pl) : d =Arg max { c( d ) }, y) d x ( I j) 2 y, d i (x 2 I 2 ) y, x ( I1 ) j, y) d y, i x ( I x ( j) 2 y, d i, j I1 i 2 (x I 2 y), x ( I i, j ) 1 j y, i I 1( x d R OUTPUT : set of displacements for every pixel in Il. C ( x, y, d ) 2 i, j

36 MPEG Compression

37

38 Due to the t'-t << epsilon hypothesis : -> in video analysis, the matching issue becomes the apparent motion estimation of radiometric patterns in the image, what is called the optical flow.

39 Primate biological vision?

40 If an algorithm is available to compute this motion in digital images, what should we observe/compute if we observe a pure translation moving object : Object and camera are moving closer Object and camera are moving away Parallel motion of objects and camera

41 Problem definition: optical flow, how to compute it? x I ( x, y ) t t How to estimate pixel motion from image H to image I? Solve pixel correspondence problem given a pixel in H, look for nearby pixels of the same color in I Key assumptions color constancy: a point in H looks the same in I For grayscale images, this is brightness constancy small motion: points do not move very far

42 Problem definition: optical flow : limits et constraints (brightness constancy) E(x,y) Brightness constancy : 0 E ( x u, y v) H ( x, y ) Small displacements : u= x and v= y < 1 pixel suppose we take the Taylor series expansion of E E E E ( x+δx, y+ Δy )=E ( x, y )+ Δx+ Δy x y E E E ( x+δx, y+ Δy) E ( x, y )+ Δx+ Δy x y

43 Combining these two equations 0 E ( x x, y y ) H ( x, y ) E 0 E ( x, y ) E x x E y y H ( x, y ) avec E x x 0 ( E ( x, y ) H ( x, y )) E x x E y y 0 E E x x E y y 0 E E x y E x 0 E t t y t

44 In the limit as u and v go to zero, this becomes exact : dx Et E dt dy 0 dt And so the following fundamental equation : de ( x, y, t ) 0 dt

45 As x(t) and y(t) de ( x(t ), y (t ), t ) E dx E dy E 0 dt x dt y dt t It expresses the common sense : ( x, y ) I x t t Energy conservation principle Related to Optimal Transport issues (Field Medal Villani)

46 Where (Frame spatial gradient) (optical flow) and (derivative across frames)

47 Brightness Constancy Equation E(x,y,t) being the image brightness and v the motion field, we write : E T v Et 0 where Et is the temporal partial derivative. As t'-t<<epsilon, we can compute/measure E and Et (image processing), hence v is not far. So what? How estimate the motion field v from this equation?

48 The aperture problem The Image Brightness Constancy Assumption only provides the OF component in the direction of the spatial image gradient

49 Intuitively, what does this constraint mean? E T E v Et vn E The component of the flow in the gradient direction is determined The component of the flow parallel to an edge is unknown vy E E E x x -Et/ E vx

50 Mean Optical Flow The optical flow is a motion field satisfying the brightness constancy equation:

51 Motion field Estimation From an image sequence Two kinds of algorithmic approaches Dense matching techniques: differential techniques (like PDE) : -> optical flow methods Sparse matching techniques : -> tracking methods Genuine optical flow Can do without brightness constancy by using geometric constraints

52 Differential Technique: Optical Flow direct computation by MSE regression For every pixel pi inside a small patch Q of size NxN (e.g. 5x5) we can write : T ( E ( p i )) v( p i )+E t ( p i )=0 and i, v( p i )=v Q T hence p i Q, ( E ( pi ) ) v Q + E t ( pi )=0 (v ) E ( p ) pi Q i T v Et ( pi ) 2

53 (v ) E ( p ) pi Q Av b E x ( p1 ) E (p ) x 2 i T v Et ( pi ) E y ( p1 ) E y ( p2 ) E x ( p25 ) E y ( p25 ) u v 2 Et ( p1 ) E (p ) t 2 Et ( p25 )

54 Whose solution is v A A Q v T 1 T A b

55 Algorithm CONSTANT_FLOW INPUT : A temporal sequence of n images E1, E2,, En. Let Q be a squared region of size NxN pixels (e.g. 5x5) Filter every image of the sequence with a Gaussian filter of standard deviation S (e.g. S = 1,5 pixels) along each spatial dimension. Filter every image of the sequence with a Gaussian filter of standard deviation S (e.g. S = 1,5 pixels) along each spatial dimension. Filter every image of the sequence with a Gaussian filter of standard deviation S (e.g. t = 1,5 frames) along the temporal dimension. If 2k+1 is the size of the temporal filter, you need not to process the k first and last images. For each pixel p of each image in the sequence : Compute matrix A and vector b Compute optical flow v(p) = (ATA)+ATb OUTPUT : the optical flow of the entire image sequence Key concept of Lucas-Kanade method Implemented in opencv for instance (Try it with python and cv2 module)

56 «Example of echocardiographic sequence (two-chamber view is shown), with poorly visualized cardiac wall in the anterior segments. Green dotted line denotes the manually delineated ground truth. Magenta solid line denotes optical flow tracking. Misinterpretation of the anterior wall (arrow) may lead to considerable inaccuracies in quantification.» From Ultrasound in Med. & Biol., Vol. 37, No. 4, pp , 2011

57 Once computed: motion-based segmentation

58 Motion model vx(x,y) a v y ( x, y ) a x0 a xx x a xy y y0 a yx x a yy y Initial images Segmentation by classification on velocity parameters

59 Motion

60 Intra-subject, inter-subject, atlas-based operation etc.

61 Multi-modal registration Only rigid-transformation like registration (motion compensation)

62 Non-rigid/Elastic/Deformable registration Related to optical-flow based techniques

63

64

65 Key applications : - Change Detection - Image Fusion - and many more... From a survey available on course website Image registration methods: a survey, IVC 2003

66 Motion

67 Objectives (limited time resources :-)): Have in mind a scope of techniques with references and current BME applications: Kalman filtering: one spot tracking of one element with incertitude Particle filtering: mutliple spots tracking (multiple targets) Active Contours: shape deformation tracking

68 C.elegans developing embryo (3D) Waterston Lab -The George Washington University. Washington DC (USA) Microscope: Zeiss LSM 510 Meta Objective lens: Plan-Apochromat 63X/1.4 (oil) Pixel size (microns): 0.09 x 0.09 x 1.0 Time step (min): 1 or 1.5 From ISBI cell-tracking challenge: Sea urchin embryogenesis Nicolas Minc Lab -Institut Jacques Monod Université Paris Diderot (France), SPC

69 Shape tracking : Snake or Active Contours Reading : (dufourthesis.pdf in French now at Pasteur Institute, promoting Icy software)

70 Shape tracking : Level Set Methods Definition : Tracking of several object contours whose number varies and slightly moving over a long image sequence. EDP framework Segmentation technique for still images then naturally extended to image sequence via the iterative implementation: initialisation and optimisation

71 Bibliography Performance of Optical Flow Techniques, J. Barron et al., IJCV, vol. 12, pp. 4377, 1994 Determining Optical Flow, Horn et Schunck, AI, vol.17, pp , 1981 Use of Optimal Estimation Theory, in Particular the Kalman Filtering, in Data Analysis and Signal Processing, W. Cooper, Review of Scientific Instrumentation, vol. 57, pp , 1986 Three-dimensional computer vision : A geometric Viewpoint, O.D. Faugeras, MIT Press, Cambridge (MA), 1993 Introductory techniques for 3D computer vision, E. Trucco et Alessandro Verri, Prentice Hall, 1998 Conclusion : explore the software Institut Pasteur or the Fiji environment (before imagej). Can also test with python and cv2 module