Geometric Transformations and Image Warping Chapter 2.6.5

Transcription

1 Geometric Transformations and Image Warping Chapter Ross Whitaker (modified by Guido Gerig) SCI Institute, School of Computing University of Utah Univ of Utah, CS

2 Geometric Transformations Greyscale transformations -> operate on range/output Geometric transformations -> operate on image domain Coordinate transformations Moving image content from one place to another Two parts: 1. Define transformation 2. Resample greyscale image in new coordinates Univ of Utah, CS

3 Geom Trans: Distortion From Optics Barrel Distortion Pincushion Distortion Univ of Utah, CS

4 Computer Vision Radial distortion example

5 Geom Trans: Distortion From Optics Univ of Utah, CS

6 Geom. Trans.: Brain Template/Atlas Univ of Utah, CS

7 Geom. Trans.: Mosaicing of Series of Images Univ of Utah, CS

8 Domain Mappings Formulation New image from old one Coordinate transformation Two parts vector valued g is the same (intensity) image as f, but sampled on these new coordinates Univ of Utah, CS

9 Domain Mappings Formulation (E. H. W. Meijering) g is the same (intensity) image as f, but sampled on these new coordinates Univ of Utah, CS

10 Domain Mappings Formulation Vector notation is convenient. Bar used some times, depends on context. T may or may not have an inverse. If not, it means that information was lost. Univ of Utah, CS

11 Domain Mappings f g f g Univ of Utah, CS

12 No Inverse? f g Not one to one f g Not onto - doesn t cover f Univ of Utah, CS

13 Example Univ of Utah, CS

14 Transformation Examples Linear Rotation, Translation, Scaling? 14

15 10/5/04 University of Wisconsin, CS559 Fall D Rotation Rotate counter-clockwise about the origin by an angle θ + = 0 0 cos sin sin cos y x y x θ θ θ θ x y x y θ

16 Rotating About An Arbitrary Point What happens when you apply a rotation transformation to an object that is not at the origin? y x? 10/5/04 University of Wisconsin, CS559 Fall 2004

17 Rotating About An Arbitrary Point What happens when you apply a rotation transformation to an object that is not at the origin? It translates as well y x x 10/5/04 University of Wisconsin, CS559 Fall 2004

18 Now: First Rotate, then Translate Rotation followed by translation is not the same as translation followed by rotation: T(R(object)) R(T(object)) θ x θ x 10/5/04 University of Wisconsin, CS559 Fall 2004

19 Series of Transformations 2D Object: Translate, scale, rotate, translate again Problem: Rotation, scaling, shearing are multiplicative transforms, but translation is additive. Guido Gerig COMP 256 / Fall

20 Excellent Materials for self study 10/5/04 University of Wisconsin, CS559 Fall 2004

21 10/5/04 University of Wisconsin, CS559 Fall 2004 Homogeneous Coordinates Use three numbers to represent a point (x,y)=(wx,wy,w) for any constant w 0 Typically, (x,y) becomes (x,y,1) To go backwards, divide by w Translation can now be done with matrix multiplication! = y x b a a b a a y x y yy yx x xy xx

22 10/5/04 University of Wisconsin, CS559 Fall 2004 Basic Transformations Translation: Rotation: Scaling: y x b b y x s s cos sin 0 sin cos θ θ θ θ

23 Transformation Examples Linear Homogeneous coordinates Univ of Utah, CS

24 Special Cases of Linear Translation Rotation Rigid = rotation + translation Scaling Include forward and backward rotation for arbitary axis Skew p, q < 1 : expand Reflection Univ of Utah, CS

25 Linear Transformations Univ of Utah, CS

26 Cascading of Transformations Demo: Univ of Utah, CS

27 Homogeneous Coordinates: A general view Acknowledgement: Greg Welch, Gary Bishop, Siggraph 2001 Course Notes (Tracking). Guido Gerig COMP 256 / Fall

28 Series of Transformations 2D Object: Translate, scale, rotate, translate again Problem: Rotation, scaling, shearing are multiplicative transforms, but translation is additive. Guido Gerig COMP 256 / Fall

29 Solution: Homogeneous Coordinates In 2D: add a third coordinate, w Point [x,y] T expanded to [x,y,w] T Scaling: force w to 1 by [x,y,w] T /w [x/w,y/w,1] T Guido Gerig COMP 256 / Fall

30 Resulting Transformations new: before: Guido Gerig COMP 256 / Fall

31 Linear Transformations Also called affine 6 parameters Rigid -> 3 parameters Invertability Invert matrix What does it mean if A is not invertible? Univ of Utah, CS

32 Affine: General Linear Transformation 6 parameters for Trans (2), Scal (2), Rot (1), Shear X and Shear Y 7 Parameters????? Univ of Utah, CS

33 Affine: General Linear Transformation 6 parameters for Trans (2), Scal (2), Rot (1), Shear X and Shear Y 7 Parameters????? 1) Rot 90deg Shear X Rot -90deg 2) Shear Y Shear Y can be formulated as Shear X applied to rotated image -> There is only one Shear parameter Univ of Utah, CS

34 Implementation Two major procedures: 1. Definition or estimation of transformation type and parameters 2. Application of transformation: Actual transformation of image Univ of Utah, CS

35 Implementation Two Approaches to Apply Transf. 1. Pixel filling forward mapping g to f T() takes you from coords in g() to coords in f() Need random access to pixels in f() Sample grid for g(), interpolate f() as needed f g Interpolate from nearby grid points Univ of Utah, CS

36 Interpolation: Bilinear Successive application of linear interpolation along each axis Source: WIkipedia Univ of Utah, CS

37 Bilinear Interpolation Not linear in x, y Univ of Utah, CS

38 Bilinear Interpolation Convenient form Normalize to unit grid [0,1]x[0,1] Univ of Utah, CS

39 Implementation Two Approaches 2. Splatting backward mapping f to g T -1 () takes you from coords in f() to coords in g() You have f() on grid, but you need g() on grid Push grid samples onto g() grid and do interpolation from unorganized data (kernel) f g? Nearby points are not organized scattered Univ of Utah, CS

40 Scattered Data Interpolation With Kernels Shepard s method Define kernel Falls off with distance, radially symmetric Kernel examples g Required grid coordinates in g Grid coordinates in f Transformed coord. from f Univ of Utah, CS ?

41 Shepard s Method Implementation If points are dense enough Truncate kernel For each point in f() Form a small box around it in g() beyond which truncate Put weights and data onto grid in g() Data and weights accumulated here Divide total data by total weights: B/A g Univ of Utah, CS

42 ESTIMATION OF TRANSFORMATIONS Univ of Utah, CS

43 Determine Transformations All polynomials of (x,y) Any vector valued function with 2 inputs How to construct transformations? Define form or class of a transformation Choose parameters within that class Rigid - 3 parameters (T, R) Affine - 6 parameters Univ of Utah, CS

44 Correspondences Also called landmarks or fiducials Univ of Utah, CS

45 Question: How many landmarks for affine T? Estimation of 6 parameters 3 corresponding point pairs with (x,y) coordinates Univ of Utah, CS

46 Transformations/Control Points Strategy 1. Define a functional representation for T with k parameters (β) 2. Define (pick) N correspondences 3. Find β so that 4. If overconstrained (K < 2N) then solve Univ of Utah, CS

47 Example Affine Transformation: 3 Corresponding Landmarks Univ of Utah, CS

48 Example: Quadratic Transformation Denote Correspondences must match Note: these equations are linear in the unkowns Univ of Utah, CS

49 Write As Linear System A matrix that depends on the (unprimed) correspondences and the transformation x unknown parameters of the transformation b the primed correspondences Univ of Utah, CS

50 Linear Algebra Background Simple case: A is square (M=N) and invertable (det[a] not zero) (Numerics: Don t find A inverse. Use Gaussian elimination or some kind of decomposition of A.) Univ of Utah, CS

51 Linear Systems Other Cases (M<N) or (M = N and the equations are degenerate or singular): System is underconstrained lots of solutions Approach Impose some extra criterion on the solution Find the one solution that optimizes that criterion Regularizing the problem Univ of Utah, CS

52 Linear Systems Other Cases M > N (e.g. more points than parameters): System is overconstrained No solution Approach Find solution that is best compromise Minimize squared error (least squares) Univ of Utah, CS

53 Solving Least Squares Systems Pseudoinverse (normal equations) Issue: often not well conditioned (nearly singular) Alternative: singular value decomposition SVD Univ of Utah, CS

54 Singular Value Decomposition Invert matrix A with SVD Univ of Utah, CS

55 SVD for Singular Systems If a system is singular, some of the w s will be zero Properties: Underconstrained: solution with shortest overall length Overconstrained: least squares solution Univ of Utah, CS

56 SPECIFYING WARPS VIA SPARSE SET OF LANDMARKS Univ of Utah, CS

57 Specifying Warps Another Strategy Let the # DOFs in the warp equal the # of control points (x1/2) Interpolate with some grid-based interpolation E.g. binlinear, splines Univ of Utah, CS

58 Landmarks Not On Grid Landmark positions driven by application Interpolate transformation at unorganized correspondences Scattered data interpolation How do we do scattered data interpolation? Idea: use kernels! Radial basis functions Radially symmetric functions of distance to landmark Univ of Utah, CS

59 Concept Univ of Utah, CS

60 Concept Warping a Neuro-Anatomy Atlas on 3D MRI Data with Radial Basis Functions H.E. Bennink, J.M. Korbeeck, B.J. Janssen, B.M. ter Haar Romeny Univ of Utah, CS

61 Concept Univ of Utah, CS

62 RBFs Formulation Represent T as weighted sum of basis functions T Sum of radial basis functions Basis functions centered at positions of data Need interpolation for vector-valued function, T: Univ of Utah, CS

63 Choices for ϕ Gaussian: g(t) = exp(-0.5(t 2 /σ 2 ) Multiquadratics: g(t) = 1/Sqrt(t 2 +c 2 ), where c is least distance to surrounding points Univ of Utah, CS

64 Solve For k s With Landmarks as Constraints Find the k s so that T(x) fits at data points Univ of Utah, CS

65 Issue: RBFs Do Not Easily Model Linear Trends f(x) f 3 f 2 f 1 x x 1 x 2 x 3 Univ of Utah, CS

66 RBFs Formulation w/linear Represent T as weighted sum of basis functions and linear part T Term Linear part of transformation Sum of radial basis functions Basis functions centered at positions of data Need interpolation for vector-valued function, T: Univ of Utah, CS

67 RBFs Linear System Univ of Utah, CS

68 RBFs Solution Strategy Find the k s and p s so that T() fits at data points The k s can have no linear trend (force it into the p s) Constraints -> linear system Corresponde nces must match Keep linear part separate from deformation Univ of Utah, CS

69 RBF Warp Example Univ of Utah, CS

70 What Kernel Should We Use Gaussian Variance is free parameter controls smoothness of warp σ = 2.5 σ = 2.0 σ = 1.5 From: Arad et al Univ of Utah, CS

71 RBFs Aligning Faces Mona Lisa Target Venus Source Venus Warped Univ of Utah, CS

72 Symmetry? Image-based Talking Heads using Radial Basis Functions James D. Edge and Steve Maddock Univ of Utah, CS

73 Application Modeling of lip motion in speech with few landmarks. Synthesis via motion of landmarks. Univ of Utah, CS

74 RBFs Special Case: Thin Plate Splines A special class of kernels Minimizes the distortion function (bending energy) No scale parameter. Gives smoothest results Bookstein, 1989 Univ of Utah, CS

75 Application: Image Morphing Combine shape and intensity with time parameter t Just blending with amounts t produces fade Use control points with interpolation in t Use c 1, c(t) landmarks to define T 1, and c 2,c(t) landmarks to define T 2 Univ of Utah, CS

76 Image Morphing Create from blend of two warped images I 1 I t I 2 T 1 T 2 Univ of Utah, CS

77 Image Morphing Univ of Utah, CS

78 Application: Image Templates/Atlases Build image templates that capture statistics of class of images Accounts for shape and intensity Mean and variability Purpose Establish common coordinate system (for comparisons) Understand how a particular case compares to the general population Univ of Utah, CS

79 Templates Formulation N landmarks over M different subjects/samples Images Correspondences Mean of correspondences (template) Transformations from mean to subjects Templated image Univ of Utah, CS

80 Cars Univ of Utah, CS

81 Car Landmarks and Warp Univ of Utah, CS

82 Car Landmarks and Warp Univ of Utah, CS

83 Car Mean Univ of Utah, CS

84 Cars Univ of Utah, CS

85 Cats Univ of Utah, CS

86 Brains Univ of Utah, CS

87 Brain Template Univ of Utah, CS

88 APPLICATIONS Univ of Utah, CS

89 Warping Application: Lens Distortion Radial transformation lenses are generally circularly symmetric Optical center is known Model of transformation: Univ of Utah, CS

90 Correspondences Take picture of known grid crossings Measure set of landmark pairs Estimate transformation, correct images Univ of Utah, CS

91 Image Mosaicing Piecing together images to create a larger mosaic Doing it the old fashioned way Paper pictures and tape Things don t line up Translation is not enough Need some kind of warp Constraints Warping/matching two regions of two different images only works when Univ of Utah, CS

92 Applications Univ of Utah, CS

93 Microscopy (Morane Eye Inst, UofU, T. Tasdizen et al.) Univ of Utah, CS

94 Univ of Utah, CS

95 Special Cases Nothing new in the scene is uncovered in one view vs another No ray from the camera gets behind another 1) Pure rotations arbitrary scene 2) Arbitrary views of planar surfaces Univ of Utah, CS

96 3D Perspective and Projection Camera model y x z f Image coordinates Univ of Utah, CS

97 Perspective Projection Properties Lines to lines (linear) Conic sections to conic sections Convex shapes to convex shapes Foreshortening Univ of Utah, CS

98 Image Homologies Images taken under cases 1,2 are perspectively equivalent to within a linear transformation Projective relationships equivalence is Univ of Utah, CS

99 Transforming Images To Make Mosaics Linear transformation with matrix P Perspective equivalence Multiply by denominator and reorganize terms Linear system, solve for P Univ of Utah, CS

100 Image Mosaicing Univ of Utah, CS

101 4 Correspondences Univ of Utah, CS

102 5 Correspondences Univ of Utah, CS

103 6 Correspondences Univ of Utah, CS

104 Mosaicing Issues Need a canvas (adjust coordinates/origin) Blending at edges of images (avoid sharp transitions) Adjusting brightnesses Cascading transformations Univ of Utah, CS