Coordinate Transformations, Tracking, Camera and Tool Calibration

Transcription

1 Coordinate Transformations, Tracking, Camera and Tool Calibration Tassilo Klein, Hauke Heibel Computer Aided Medical Procedures (CAMP), Technische Universität München, Germany

2 Motivation 3D Transformations and Registration The Island 2005 by Michael Bay

3 Outline Transformations and Projective Space Transformations in 3D Tracking Tracking of Medical Instruments Calibration of Medical Instruments Camera Calibration

4 Hierarchy of Transformations Projective linear group Affine group (last row (0,0,1)) Euclidean group (upper left 2x2 orthogonal) Oriented Euclidean group (upper left 2x2 det 1) Alternative characterization of transformations in terms of elements or quantities that are preserved or invariant is possible. e.g. Euclidean transformations leave distances unchanged similarity affinity projectivity

5 Class I: Isometries or Euclidean Transformation proper rotation improper rotation (reflection, inversion) 3DOF (1 rotation, 2 translation) special cases: pure rotation, pure translation Invariants: length, angle, area (iso=same, metric=measure)

6 Class II: Similarities (isometry + isotropic scaling) 4DOF (1 scale, 1 rotation, 2 translation) also know as equi-form (shape preserving) metric structure = structure up to similarity (in literature) Invariants: ratios of length, angle, ratios of areas, parallel lines

7 Class III: Affine Transformations 6DOF (2 scale, 2 rotation, 2 translation) non-isotropic scaling! (2DOF: scale ratio and orientation) Invariants: parallel lines, ratios of parallel lengths, ratios of areas

8 Class IV: Projective Transformations x' H P x = A v T t x v 8DOF (2 scale, 2 rotation, 2 translation, 2 line at infinity) Invariants: cross-ratio of four points on a line (ratio of ratios)

9 Summary of Transformations in 2D Projective 8dof Affine 6dof Similarity 4dof Concurrency, colinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallelism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l Ratios of lengths, angles. The circular points I,J Euclidean 3dof lengths, areas.

10 3D Transformations

11 3D Transformations Basic Coordinate Transformations y y y x x x translation scaling rotation

12 Class I: Isometries or Euclidean Transformations proper rotation improper rotation (reflection, inversion) Invariants: length, angle, area, volume (iso=same, metric=measure)

13 Class I: Isometries or Euclidean Transformations All combinations are possible! Which one do we choose? 6DOF (3 rotation, 3 translation) Invariants: length, angle, area, volume (iso=same, metric=measure)

14 Class II: Similarities (isometry + isotropic scaling) 7DOF (1 isotropic scaling, 3 rotation, 3 translation) also know as equi-form (shape preserving) metric structure = structure up to similarity (in literature) Invariants: ratios of length, angle, ratios of areas, ratios of volumes, parallel lines

15 Class III: Affine Transformations 12DOF (3 rotation, 3 translation, 3 an-isotropic scaling, 3 rotation (coord. frame transformation) Invariants: parallel lines, ratios of parallel lengths, ratios of areas, ratios of volumes

16 Class IV: Projective Transformations 15DOF (3 scale, 3 rotation, 3 translation, 3 shearing, 3 plane at infinity ) Invariants: cross-ratio of four points on a line (ratio of ratios) π

17 References Transformations, Coordinate Systems, and Computer Vision Lecture: 3D Computer Vision by Prof. Navab (and colleagues) Gary Bishop, Greg Welch, and B. Danette Allen; Course 11 Tracking: Beyond 15 Minutes of Thought; 2001; Semple, JG and Kneebone, GT; Algebraic Projective Geometry; Oxford University Press; 1998 Olivier Faugeras; Three-Dimensional Computer Vision (Artificial Intelligence); 1993 Richard Hartley, Andrew Zisserman; Multiple View Geometry in Computer Vision; 2003 Emanuele Trucco, Alessandro Verri; Introductory Techniques for 3-D Computer Vision; 1998

18 Tracking 18

19 Example :: Computer Aided Orthopedic Surgery Vector Vision System by BrainLAB AG 3D Ultrasound Mosaicing - Wachinger et al. 19

20 subject for tracking in IGS surgical Instruments laparoscopic tools drills biopsy needles catheter imaging devices mobile c-arm US / laparoscopic US intraoperative probes endoscopes

21 tracking outline motion tracking tracking systems mechanical tracking optical tracking electromagnetic tracking hybrid tracking applications in computer aided surgery

22 tracking systems :: requirements for IGS accuracy (mm / sub mm) precision (mm / sub mm) robustness reliability availability usability (ease of use) sterilizable decreasing precision decreasing accuracy

23 tracking systems :: overview optical tracking fiducial based vs. natural landmarks active vs. passive markers electromagnetic tracking mechanic link (e.g. via robot API) inertial tracking acoustic tracking GPS

24 tracking systems :: optical tracking technology: camera takes images, features are extracted and compared to a model of a visible object

25 optical tracking :: fiducials vs. natural landmarks artificial marker (= fiducials) model of fiducials is known in general optimized fiducials for certain feature extraction algorithms improves the accuracy of a measurement fast feature extraction algorithms possible natural landmarks (= markerless tracking) large area application have to rely on natural landmarks model of the landmarks has to be generated

26 optical tracking :: active vs. passive markers active markers Passive markers needs a source of power bright, large range of view Polaris with active tools by NDI allows for a high update rate can be identified by triggering identification cost either accuracy, flexibility or CPU time without metal cheap, easy to make without cable Polaris with passive tools by NDI

27 optical tracking :: summary high accuracy (< 1 mm) (depending on the baseline and the distance of target to cameras) Sufficient update rate (60Hz) robust line of sight problem medical relevance: high (fiducial based), markerless: low

28 optical tracking :: examples NDI Optotrak: three IR line camera, active markers update rate Hz, accuracy < 1mm A.R.T: two IR cameras, passive markers, update rate 60Hz, accuracy < 1mm Non commercial: one camera (e.g. webcam), passive markers (paper prints), update rate 30Hz, accuracy ~1cm

29 tracking systems :: mechanical tracking technology: Object to track is attached to mechanical link, angle in junctions are measured by sensors biological counterpart: haptics + sense of preprioception (if you hold something with your hand you know where it is!)

30 mechanical tracking :: summary high update rate high accuracy (< 1 mm) (the more junctions and the longer the links, the worse the measurement) tracking of a robot for free by robot API practical problem of mechanical link: range of use, weight, link might be in the way medical relevance: low, only in robotically assisted surgery

31 tracking systems :: electromagnetic tracking technology: sender emits a magnetic field, sensor measures its field vector and calculates its position and orientation from it basic idea like a compass Aurora by NDI

32 electromagnetic tracking :: examples Ascension Microbird 6DOF NDI Aurora 5 DOF

33 electromagnetic tracking :: summary high update rate (>100Hz) no line of sight problem small area of high accuracy: 20 cm - 60 cm from transmitter environment (especially ferromagnetic material) distorts the magnetic field and thus the measurement strongly limited amount of tracking probes possible medical relevance: high (only way to track flexible instruments without fluoroscopy imaging)

34 tracking systems :: summary

35 hybrid tracking system what is hybrid: something that has two different types of components performing essentially the same function (merriam-webster online) hybrid tracking: combination of two tracking system to overcome their individual shortcomings increase accuracy and precision increase robustness and availability

36 hybrid tracking :: optical & electromagnetic optical tracking line of sight electromagnetic tracking magnetic field distortion low accuracy small tracking volume small numbers of sensors hybrid system high accuracy as long as line of sight available dynamic magnetic field distortion correction higher tracking volume and flexibility

37 Example :: Calibration Free Navigation Orthopedics Klinikum Innenstadt (Dr. Euler & Dr. Heining) using the Medivision Navigation System

38 Calibration Free Navigation Tracking Intraoperative imaging Visualization Orthopedics Klinikum Innenstadt (Dr. Euler & Dr. Heining) using the Medivision Navigation System

39 Calibration Free Navigation tracking of c-arm & tools In the same coordinate system

40 Calibration Free Navigation

41 Calibration Free Navigation Tc-arm toolt c-arm Ttool

42 Hot Spot Calibration Aim: estimation of the fixed position (3DOF) of the tip in the coordinate system of the tracking target (TCS) Method: rotate the tool (here drill) around a fix point in the world coordinate system (WCS) TCS Tool Tip WCS

43 Hot Spot Calibration positions of the tracking targets in WCS Graphic by Martin Bauer positions of the tool tip in WCS Unknown: Position of the tip in WCS (3 DOF) Position of the tip in TCS (3 DOF) Constraint: Position in both CS is constant for all recordings

44 Hot Spot Calibration relation between the point in WCS and TCS where R and t are the 6 DOF rotation and translation of the tracking target in the WCS construction of a linear equation system solving linear equation system e.g. by applying SVD

45 References Hot Spot Calibration Tuceryan, M., Greer, D. S., Whitaker, R. T., Breen, D. E., Rose, E., Ahlers, K. H., and Crampton, C.; Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System; IEEE Transactions on Visualization and Computer Graphics; 1(3); pp.: ; 1995 Fuhrmann, A. and Splechtna, R. and Prikryl, J.; Comprehensive Calibration and Registration Procedures for Augmented Reality; In Proceedings of the joint 5th Immersive Projection Technology and 7th Eurographics Virtual Environments Workshop (EGVE); pp.: ; 2001

46 Camera Models 46

47 Pinhole Camera Model A pinhole camera is a very simple camera with no lens and a single very small aperture. mapping between 3D world and 2D image central projection models are described in matrices with particular properties

48 X Y Z 1 x y z : fx fy Z = f 0 f X Y Z 1 = x = PX Central Projection Using Homogeneous Coordinates Projecting a point from 3D to 2D: determine intersection of image plane with the ray connecting the projection center with the 3D point => a homogeneous 3-vector describing a point can be thought of as: direction vector of ray projecting a 3D point onto a 2D image plane. P R 3 4

49 Central Projection Using Homogeneous Coordinates

50 Central Projection

51 Principal Point Offset principal point (perpendicular intersection point of principal axis and image plane) where are the coordinates of the principal point

52 Principal Point Offset where is called camera calibration matrix

53 Camera Rotation and Translation inhomogeneous coordinates where represents the point in world coordinates represents the same point in camera coordinates represents the coordinates of the camera origin in the world coordinate frame

54 Camera Rotation and Translation homogeneous coordinates projection to image plane from camera coordinates projection to image plane from world coordinates

55 Extrinsic and Intrinsic Parameters where projection matrix of a general pinhole camera with 9 DOF intrinsic camera parameters with 3 DOF extrinsic camera parameters with each 3 DOF (camera orientation and position in world coordinates)

56 Extrinsic and Intrinsic Parameters where P = KR I C [ ] [ ] = K[ R RC ] = K R t projection matrix of a general pinhole camera with 9 DOF t = R C extrinsic camera parameters with each 3 DOF (camera orientation and position in world coordinates) intrinsic camera parameters with 3 DOF

57 CCD Cameras :: Non-Square Pixel number of pixels per unit distance 4 DOF 10 DOF

58 Skew Parameter skew parameter skewing of the pixel elements in the CCD array - the x- and y-axes are not perpendicular. 5 DOF finite projective camera with 11 DOF

59 Finite Projective Camera :: Summary where camera matrix P is identical with the set of homogeneous 3x4 matrices for which the left hand 3x3 submatrix is non singular {finite cameras}={p det M 0} ={P rank(m)=3} If rank(p)=3, but rank(m)<3, then camera at infinity if rank(p)<3 the matrix mapping will be a line or a point and not a plane (not a 2D image)

60 Camera Anatomy camera center column vectors principal plane axis plane principal point principal ray

61 Camera Center P has a 1D null-space we will proof that the 4-vector C is the camera center points on a line through A and C since all 3D points on the line are mapped on the same 2D image point, and thus the line is a ray through the camera center Finite cameras: [ ] [ ] P = KR I C = M I C Camera at infinity: C = d, where Md = 0 0

62 Column Vectors column vectors are the image points, which project the axis directions (X,Y,Z) and the origin example for the image of the y-axis (vanishing pt) is the image of the world origin

63 Row Vectors represent geometrically particular world planes row vectors column vectors

64 Row Vectors of the Projection Matrix P 1 is defined by the camera center and the line x=0 on the image P 2 is defined by the camera center and the line y=0 on the image Example P 2 respectively for P 1

65 Principal Plane plane through camera center and parallel to the image plane if X is on the principle plane points X are imaged on the line at infinity especially

66 Camera Calibration

67 Resectioning numerical methods for estimating the camera projection matrix from corresponding world and image entities simplest entities are the correspondence between a 3D point X and its image x P will be determined for sufficient many corresponding points (or other entities such as lines)

68 Basic Equations find P Homogenous vectors, in general: x i PX i [ x i ] PX i = x i PX i = 0 three equations, however, only two of them are linearly independent x i

69 Basic Equations minimal solution P has 11 dof, 2 independent eq./points 5½ correspondences needed (say 6) Over-determined solution n 6 points minimize subject to constraint

70 DLT algorithm Objective Given n 6 3D to 2D point correspondences {X i x i }, determine the Projection matrix P such that x i =PX i Algorithm (i) For each correspondence X i x i compute A i. Usually only two first rows needed. (ii) Assemble n 2x12 matrices A i into a single 2nx12 matrix A (iii) Obtain SVD of A (i.e. of V (iv) Determine P from p ). Solution for p is last column

71 Geometric Error

72 Gold Standard Algorithm for the estimation of P form world to image correspondences (no error in world points) Objective Given n 6 3D to 2D point correspondences {X i x i }, determine the Maximum Likelihood Estimation of P Algorithm (i) Linear solution: (a) Normalization: (b) DLT (ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: ~ ~ ~ e.g. Levenberg-Marquart optimizer (iii) Denormalization:

73 Calibration Example (i) Canny edge detection (ii) (Straight) line fitting to the detected edges (iii) Intersecting the lines to obtain the images corners (iv) Matching image corners and 3D target checkerboard corners (counting if entire target is visible in image) (v) Correspondences: typical precision <1/10 pixel (HZ rule of thumb: 5n constraints for n unknowns, i.e. at least 28 points)

74 Radial Distortion short and long focal length

75 Distorted and Undistorted Image without distortion: straight lines remain straight lines

76 Correction of Radial Distortion ˆ x = x c + L(r)(x x c ) ˆ y = y c + L(r)(y y c ) r 2 = (x x c ) 2 + (y y c ) 2 (x, y) : measured image coordinates ( ˆ x, ˆ y ) : corrected coordinates (x c, y c ) : center of radial distortion Distortion Correction Function (polynomial): L(r) =1+ κ 1 r + κ 2 r 2 + κ 3 r 3... with distortion coefficients {κ 1,κ 2,κ 3,..., x c, y c } Distortion coefficients may computed in a multitude of ways, e.g. inclusion in iterative camera parameter estimation procedures minimizing the geometric error. Cost function based on the deviation from linear mapping. 76

77 Literature on Camera Models & Calibration Chapter 6 in R. Hartley and A. Zisserman, Multiple View Geometry, 2 nd edition, Cambridge University Press, Chapter 3 in O. Faugeras, Three-dimensional Computer Vision, MIT Press, Chapter 2 in E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision, Prentice Hall, H. Gernsheim, The Origins of Photography, Thames and Hudson, A. Shashua. Geometry and Photometry in 3D Visual Recognition, Ph.D. Thesis, MIT, Nov AITR-1401.

78 Thanks for your attention! Any questions? 78