Find the Correspondences

Transcription

1 Advanced Topics in BioImage Analysis - 3. Find the Correspondences from Registration to Motion Estimation CellNetworks Math-Clinic Qi Gao

2 Math-Clinic core facility Provide services on bioinformatics & bioimage analysis: 1-to-1 consultancies research collaboration courses and workshops internship, MSc/BSc thesis Room 001, BioQuant (INF 267) +49 (0)

3 with slides and figures from Jean-Yves Tinevez Perrine Paul-Gilloteaux

4 Correspondence problems identify corresponding structures in two or more images appear in the context of biomedical image registration motion analysis (optical flow, particle image velocimetry, ) and many more

5 The goal of this tutorial not to understand the algorithms and work on the math but to get the basic concepts and principles of registration and optical flow know what problems in bioimage analysis need correspondence analysis have some ideas how to set the parameters when using a plugin and identify and solve correspondence problems in your research

6 Stitching: Building large images. Application context: You want at the same time: Large field of view and High resolution. High resolution objectives have small field of view. combine several views (tiles) by moving the sample. stitching, mosaic, tiles, etc. [JT] Kurt Anderson.

7 Drift correction in time-lapse movies. Focus drift Stage drift Additional challenge: The two images we align are not rigorously part of the same scene (the sample change over time). [JT] MicroscopyU website

8 Correction of chromatic/lens aberrations. Imperfect (real) lenses image different colors differently. Sometimes as bad as move in Z or even deform the field. Before the actual experiment, image multi-color beads for each fluorescence channel then compute a correction that would bring their image together. [JT] Kozubek & Matula, Journal of Microscopy, 2000

9 Correlative microscopies. 1. Same sample imaged by (very) different microscopes. What can be done automatically is very limited. 2. [JT] FEI.

10 New dimensions in microscopy. Examples in light-sheet fluorescence microscopy. [JT] Rotation of the sample: New dimension: θ. Several illumination directions: New dimension: illumination index.

11 Volume reconstruction heavy distortion might occur complex patterns of the deformation fields Zuo et al, Nature 2015

12 Cell nuclei deformation compensation time-lapse microscopy images of live cells motion of subcellular particles can help to understand the dynamics of cellular processes cell nuclei generally undergo significant complex deformations (position & form) registration can compensate the movement and deformation of cell nuclei, and help to find real motion of particles ch1 ch1 ch2 ch2 data by D.L. Spector, CSHL

13 Motion analysis optical flow fields between frames

14 Particle Image Velocimetry instantaneous velocity measurements and related properties in fluids. An experimental fluorescent bead image for traction force microscopy (TFM) The bead displacement field displayed as color coded vector plot

15 Registration. Image registration is the process of aligning two or more images of the same scene. This process involves designating one image as the reference (also called the reference image), and applying geometric transformations to the other images so that they align with the reference. x MATLAB documentation. x1 Scene, with global coordinate system. y1 y2 x2 Image 1, with its own image coordinate system. Image 2. [JT] y Goal: find the optimal alignment of one with respect to the other.

16 Linear registration - transformation classes 1. Translation. (Displacement) 2. Rigid transformation. (Isometry) 3. Scaling. (Similarity) 4. Affine transformation. [JT]

17 Linear registration - transformation classes x1 1. Translation. (Displacement) y1 2. Rigid transformation. (Isometry) 1 x 3. Scaling. (Similarity) x1 4. Affine transformation. y1 y [JT]

18 Linear registration - transformation classes x1 1. Translation. (Displacement) y1 2. Rigid transformation. (Isometry) 2 x x1 3. Scaling. (Similarity) 4. Affine transformation. y1 [JT] y = translation + rotation

19 Linear registration - transformation classes x1 1. Translation. (Displacement) y1 2. Rigid transformation. (Isometry) 2 x 3. Scaling. (Similarity) x1 y1 4. Affine transformation. [JT] y = translation + rotation + change of scale (isotropic)

20 Linear registration - transformation classes x1 1. Translation. (Displacement) y1 2. Rigid transformation. (Isometry) 2 x 3. Scaling. (Similarity) x1 4. Affine transformation. y1 [JT] y Anisotropic scaling and possibly shearing

21 A few properties of transformations. contains contains contains Translation. (Displacement) Rigid transformation. (Isometry) Scaling. (Similarity) Affine transformation. Preserve below + global orientation. Preserve below + distances. Preserve below + relative angle. Preserve parallelism. [JT]

22 Linear registration - transformation representation 1. Translation. (Displacement) 2. Rigid transformation. (Isometry) Relatively simple mathematically. In 3D, can be expressed as a matrix of 4x4 numbers. 3. Scaling. (Similarity) 4. Affine transformation. [JT]

23 How does it work for an image? Image 1 coordinate system. Forward transformation T Global coordinate system. Global coordinate system. Backward transformation T -1 Image 1 coordinate system. Scene x Image 1 x1 y y1 [JT] T -1 fetch me this pixel.

24 Trivia. A. Multiple tiles stitching (microscope stage movement). 1. Translation. (Displacement) B. Correlative light-electron microscopy. 2. Rigid transformation. (Isometry) C. Alignment of SPIM images. 3. Scaling. (Similarity)? D. 3D alignment of microscopy sections. 4. Affine transformation. E. Drift correction in time-lapse movies. [JT] 5. Many others, more complex. F. Correction of chromatic aberrations / lens aberrations.

25 Trivia. 1. Disclaimer: Practicalities might overrule common sense. Translation. (Displacement) A. B. Multiple tiles stitching (microscope stage movement). Correlative light-electron microscopy. 2. Rigid transformation. (Isometry) C. Alignment of SPIM images. 3. Scaling. (Similarity) D. 3D alignment of microscopy sections. 4. Affine transformation. E. Drift correction in time-lapse movies. [JT] 5. Many others, more complex. F. Correction of chromatic aberrations / lens aberrations.

26 Registration of 2d images: linear vs non-linear linear: global parameters translation, rotation, scale, shearing x1 x non-linear (non-rigid) dense deformation field one deformation vector for each pixel elastic transformations no appearing or vanishing information during registration arbitrary deformation y1 y y2 x2 by arrows by color by mesh

27 Registration techniques. Intensity based methods. Align using a common volume. Directly look for correlation in the intensity. Many different implementations, typically more complex than what our eyes do when we do puzzles (phase correlation, etc.). Requires an interpolation scheme ( give me pixel at x=9.65, y=-0.12 ). Requires a metric: how to quantify the intensity distance between two images. Then you use an optimizer to minimize this distance. Read access intensive: the optimizer needs to access a lot of pixels a considerable amount of time. Often sensitive to noise (random intensity fluctuations artificially increases metrics like sum of squares). [JT]

28 Registration techniques. Feature based methods. Saalfeld et al., Nature Methods, 2012 Find and match a discrete number of points. Points can be found automatically inside the image (corners, salient points, etc.). SIFT, SURF, Harris Points can be fiducial markers in the sample (fluorescent beads). Find some features for them, that give them identity (same identity over several modalities). Minimize the total distance between the two set of points, paired by identity. Works well as semi-automatic registration: user manually determine the points and their identity. [JT]

29 Some Fiji plugins. Name. Type. Application. StackReg and TurboReg. Intensity-based. Alignment of stacks (3D or 2D+T). Descriptor-based registration (2d/3d). Feature-based. Alignment of stacks (3D+C or 3D+T). bunwarpj Both. Pairwise alignment, often use as a library by other plugins. Correct 3D drift. Intensity-based. As the name implies. Grid/Collection Stitching Plugin. Intensity-based. Stitching. Image Stabilizer. Intensity-based. Alignment of stacks (3D or 2D+T). Align Image by line ROI. Feature-based with 2 manual landmarks. Elastic Alignment and Montage. Feature-based. Stitching. Pairwise alignment, rigid only. Landmark Correspondences. Feature-based. Points must be provided by another plugin. Pairwise alignment Linear Stack Alignment with SIFT. Feature-based. Alignment of stacks (3D or 2D+T). Manual drift correction plugin. Register Virtual Stack Slices. Feature-based. Points are taken from the ROI manager. Alignment of stacks (3D or 2D+T). Alignment of stacks (3D or 2D+T). Stack does not have to be in memory. Multiview-Reconstruction. Feature based. Multi-view alignment for big images. [JT]

30 bunwarpj speed vs accuracy coarse to fine strategy; useful for large image deformation precision model parameters; how important a factor is. stop criterion output info

31 Motion estimation optical flow the motion field between frames (arbitrary motion) dense: one motion vector (horizontal and vertical components) for each pixel

32 Why optical flow rather than tracking? density and the lack of prominent features prevent the individual extraction of objects of interest undergoing complex motion arbitrary motion possible Focal Adhesion and Actin Dynamics in Migrating Cells 10 min: vinculin-dsred (epi) 80 s: actin-gfp (TIRF)

33 Why optical flow rather than tracking? density and the lack of prominent features prevent the individual extraction of objects of interest undergoing complex motion arbitrary motion possible Cell nuclei deformation compensation ch1 ch2

34 Motion field = optical flow field? motion field the projection of scene motion onto the image plane. optical flow 2D velocity field describing the apparent motion in the images.

35 Motion field = optical flow field? barber pole illusion motion field optical flow

36 Local methods assumption image measurements (e.g. brightness) in a small region remain the same although their location may change basic approach: block matching find the most similar patch inefficient; non-unique solution; poor quality time t time t+1

37 Local methods assumption image measurements (e.g. brightness) in a small region remain the same although their location may change. Lukas-Kanade method new assumption: neighboring points have similar motions flow is locally constant similar to blocking matching; but direct optimization efficient only for small displacements; yields non-dense flow fields a early model (1982); not performing well

38 Global methods explicitly integrate the constraint (e.g., brightness constancy) and the regularization of optical flow vectors into one single objective function all optical flow vectors are computed simultaneously based on the whole images E(u, v) = Ω D(u, v) }{{} +α S(u, v) }{{} data term smoothness term dx dy. data term D(u, v) penalises deviations from constancy assumptions smoothness term S(u, v) penalises dev. from smoothness of the solution regularisation parameter α > 0 determines the degree of smoothness

39 Recommendation of plugins No! optical flow is generally a complex problem varying brightness, noise, occlusions, (dis)appearing objects underlying physical constraints lead to different models particular properties of your images optimization of the model objective come to our Math-Clinic with your problem!

40 Regularization weight increasing weight α [PG]

41 Different models [PG]