Intelligent Robotics

Transcription

1 Intelligent Robotics Intelligent Robotics lectures/2013ws/vorlesung/ir Jianwei Zhang / Eugen Richter University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Technical Aspects of Multimodal Systems Winter semester 2013/2014 Jianwei Zhang / Eugen Richter 1

2 Outline Intelligent Robotics 1. Sensor fundamentals 2. Rotation and motion 3. Force and pressure 4. Frame transformations 5. Distance measurement 6. Scan data processing 7. Recursive state estimation 8. Vision systems 9. Fuzzy logic Jianwei Zhang / Eugen Richter 2

3 8Visionsystems University of Hamburg Outline Intelligent Robotics 1. Sensor fundamentals 2. Rotation and motion 3. Force and pressure 4. Frame transformations 5. Distance measurement 6. Scan data processing 7. Recursive state estimation 8. Vision systems Transformations Camera calibration Applications 9. Fuzzy logic Jianwei Zhang / Eugen Richter 406

4 8Visionsystems University of Hamburg Vision systems in robotics Intelligent Robotics I Linear camera (e.g. barcode scanner) I Analog CCD camera (black/white) I Analog CCD color camera (1 chip or 3 chips) I High-Dynamic-Range CMOS camera I Digital camera (USB or Firewire) I Camera + Structured light (Infrared, Laser, RGB) I Stereo systems I Omnidirectional vision systems (catadioptrical systems) I dioptrics! lenses I catoptrics! mirrors Jianwei Zhang / Eugen Richter 407

5 8Visionsystems University of Hamburg Vision systems and manipulation Intelligent Robotics Jianwei Zhang / Eugen Richter 408

6 8Visionsystems University of Hamburg Vision systems in industrial applications Intelligent Robotics I Object grasping tasks I Objects with predetermined positioning (e.g. production line) I Randomly positioned objects (e.g. bin-picking ) I Object handling tasks I Cutting, tying, wrapping, sealing, etc. I Inspection during assembly I Mounting tasks I Welding, screwing, attaching, gluing, etc. Jianwei Zhang / Eugen Richter 409

7 8Visionsystems University of Hamburg Vision systems in cognitive robotics Intelligent Robotics I Perception of objects I Static: Recognition, searching, indexing,... I Dynamic: Tracking, manipulation,... I Perception of humans I Face recognition I Gaze tracking I Gesture recognition I... Jianwei Zhang / Eugen Richter 410

8 8Visionsystems University of Hamburg Vision systems in cognitive robotics (cont.) Intelligent Robotics Modeling of the world: I Object recognition and localization (e.g. landmarks) I 3D-reconstruction I Allocation of the environment I Position and orientation of the robot I relative I absolute I... in relation to various coordinate systems Jianwei Zhang / Eugen Richter 411

9 8Visionsystems University of Hamburg Vision systems in cognitive robotics (cont.) Intelligent Robotics Movements led by the vision system: I Visual-Servoing I Coarse and fine positioning I Tracking of movable objects I Swinging, juggling, balancing,... I Collision avoidance I Based on the principle of optical flow I 3D-based distance measurement I Coordination with other robots and/or humans I I Intention recognition Motion estimation Jianwei Zhang / Eugen Richter 412

10 8Visionsystems University of Hamburg Coordinate frames of a manipulation system Intelligent Robotics Jianwei Zhang / Eugen Richter 413

11 8.1 Vision systems - Transformations Intelligent Robotics Robot - Table - Camera Jianwei Zhang / Eugen Richter 414

12 8.1 Vision systems - Transformations Intelligent Robotics Transformations Jianwei Zhang / Eugen Richter 415

13 8.1 Vision systems - Transformations Intelligent Robotics Transformations (cont.) Points inside one coordinate frame can be transferred to another coordinate frame using transformations: Z: Transformation from world coordinates to manipulator base coordinates T 6 : Complete kinematic transformation (6DOF manipulator) from the base of the manipulator to the end of the manipulator E: Transformation from the end of the manipulator to the gripper B: Transformation from world coordinates to object coordinates G: Specification of the grip position and grip orientation using object coordinates Jianwei Zhang / Eugen Richter 416

14 8.1 Vision systems - Transformations Intelligent Robotics Transformations (cont.) If the robot grasps an object, the world coordinates of the grasp point can be determined in two ways and results in: ZT 6 E = BG Jianwei Zhang / Eugen Richter 417

15 8.1 Vision systems - Transformations Intelligent Robotics Transformations (cont.) If the manipulator needs to be moved to the grasp position, the following equation needs to be solved: T 6 = Z 1 BGE 1 In order to determine the position of the object after the completion of the grasp operation, one calculates: B = ZT 6 EG 1 Jianwei Zhang / Eugen Richter 418

16 8.1 Vision systems - Transformations Intelligent Robotics Transformations (cont.) A camera included in the system yields two additional transformations: C: Transformation of camera coordinates into world coordinates (Determination of the transformation o -line through camera calibration) I: Transformation of grasp point coordinates into the camera coordinate system (Grasp point is determined using image processing techniques) Jianwei Zhang / Eugen Richter 419

17 8.1 Vision systems - Transformations Intelligent Robotics Transformations (cont.) The transformation P from the grasp point to the world coordinate system is: P = IC The camera-world transformation is determined through the following equation: C = I 1 P Jianwei Zhang / Eugen Richter 420

18 8.2 Vision systems - Camera calibration Intelligent Robotics Camera calibration Camera calibration in the context of three-dimensional image processing is the determination of the intrinsic and/or extrinsic camera parameters Intrinsic parameters: Internal geometrical structure and optical features of the camera Extrinsic parameters: Three-dimensional position and orientation of the camera s coordinate system in relation to a world coordinate system Jianwei Zhang / Eugen Richter 421

19 8.2 Vision systems - Camera calibration Intelligent Robotics Camera calibration (cont.) What information does one get? In order to reconstruct 3D-objects from two or more images, it is necessary to know the relation between the coordinate systems of the 2D-image and the 3D-object The relation between 2D and 3D can be described using two transformations: Jianwei Zhang / Eugen Richter 422

20 8.2 Vision systems - Camera calibration Intelligent Robotics Camera calibration (cont.) 1. Perspective projection of a 3D-point onto a 2D-image point I Using the estimation of a 3D-object point and the corresponding error covariance matrix, the 2D-projection can be predicted 2. Backprojection of a 2D-point onto a 3D-beam I If a 2D-image point is given, there is a ray in 3D space on which the corresponding 3D object point lies I If there are two or more views of the 3D-point, its coordinate can be determined using triangulation Jianwei Zhang / Eugen Richter 423

21 8.2 Vision systems - Camera calibration Intelligent Robotics Camera calibration (cont.) I The first transformation is useful in order to reduce the search space during feature comparison or hypothesis verification in scene analysis I The second transformation is helpful for deriving 3D-information based on features in 2D-images I These transformations can be used in various application fields: I I I I I I Automatic assembly 3D-metrology Robot calibration Tracking Trajectory analysis Automatic vehicle guidance Jianwei Zhang / Eugen Richter 424

22 8.2 Vision systems - Camera calibration Intelligent Robotics Calibration techniques I Camera calibration can be done on-line or o -line I Using a calibration object: I Identification of the camera parameters I Direct creation of coordinate transformation between camera coordinates and world coordinates I Using self-calibration approaches I Using machine learning methods Jianwei Zhang / Eugen Richter 425

23 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Model of a camera without distortion Pinhole camera with and without radial lens distortion Jianwei Zhang / Eugen Richter 426

24 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Model of a camera without distortion (cont.) I (x w, y w, z w ): 3D world coordinate system with the origin O w I (x, y, z): 3D coordinate system of the camera with the origin O (optical center) I (X, Y ): 2D image coordinate system with the origin O 1 I f : Focal length of the camera Jianwei Zhang / Eugen Richter 427

25 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Transformation from world to camera coordinates I Let P(x w, y w, z w ) be a point in the world coordinate system I Its projection onto the image plane can be determined as follows: 2 3 x 2 3 x w 4y5 = R 4y w 5 + t z z w r 1 r 2 r 3 having R = 4r 4 r 5 r 6 5 and t = 4 r 7 r 8 r 9 t x t y t z 5 I The parameters R and t are the extrinsic parameters Jianwei Zhang / Eugen Richter 428

26 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Projection of camera coordinates onto image coordinates I Point P is projected onto the corresponding (ideal) image coordinate (u, v) I Perspective projection with focal length f : u = f x z v = f y z I The image coordinate (X, Y ) is calculated from (u, v) as follows: X = s u u Y = s v v I The scaling factors s u and s v are used to convert the image coordinates from meters to pixels I s u, s v and f are the intrinsic camera parameters Jianwei Zhang / Eugen Richter 429

27 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Projection of world coordinates on image coordinates I Since only two independent intrinsic parameters exist, one defines: f x fs u and f y fs v I These equations yield the distortion-free camera model: X f = f x r 1 x w + r 2 y w + r 3 z w + t x r 7 x w + r 8 y w + r 9 z w + t z Y f = f y r 4 x w + r 5 y w + r 6 z w + t y r 7 x w + r 8 y w + r 9 z w + t z Jianwei Zhang / Eugen Richter 430

28 8.2.1 Vision systems - Camera calibration - Pinhole camera model Intelligent Robotics Pixel coordinates I The coordinates (C x, C y ) of the image center are subtracted from the image coordinates (X f, Y f ) determined during perspective projection I Due to the above, one has: X = X f Y = Y f C x C y I The uncertainty regarding the image center may reach pixels Jianwei Zhang / Eugen Richter 431

29 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Main calibration parameters The pinhole camera model contains the following calibration parameters: I The three independent extrinsic parameters of R I The three independent extrinsic parameters of t I The intrinsic parameters f x, f y, C x and C y Jianwei Zhang / Eugen Richter 432

30 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Calibration points Calibration requires a set of m object points, which 1. have known world coordinates {x w,i, y w,i, z w,i }, i = 1,...,m with su ciently accurate precision 2. lie within the camera s field of view These calibration points are detected in the camera image with their respective camera coordinates {X i, Y i } Jianwei Zhang / Eugen Richter 433

31 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Calibration I The main problem during camera calibration is the identification of the unknown parameters of the camera model I The determination of these parameters for the distortion-free camera model yields the position of the camera in world coordinates I The most basic strategy for camera calibration determines the associated coe cients using linear-least-squares-identification of the perspective transformation matrix Jianwei Zhang / Eugen Richter 434

32 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Distortion-free camera model The distortion-free camera model X = f x r 1 x w + r 2 y w + r 3 z w + t x r 7 x w + r 8 y w + r 9 z w + t z, can be rearranged to Y = f y r 4 x w + r 5 y w + r 6 z w + t y r 7 x w + r 8 y w + r 9 z w + t z X = a 11x w + a 12 y w + a 13 z w + a 14 a 31 x w + a 32 y w + a 33 z w + a 34 Y = a 21x w + a 22 y w + a 23 z w + a 24 a 31 x w + a 32 y w + a 33 z w + a 34 Jianwei Zhang / Eugen Richter 435

33 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Perspective transformation matrix I We can assign a 34 = 1, since scaling the coe cients a 11,...,a 34 does not change the values of X and Y I The coe cients a 11,...,a 34 correspond with the so called perspective transformation matrix I The previous two equations can be summarized in the following identification model: 2 3 apple a 11 xw y w z w Xx w Xy w Xz w = x w y w z w 1 Yx w Yy w Yz w. a 33 apple X Y Jianwei Zhang / Eugen Richter 436

34 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Least squares approach I The eleven unknown coe cients a 11,...,a 33 are determined using the least squares method I At least six calibration points are necessary I Each pair of data points {(x w,i, y w,i, z w,i ), (X i, Y i )} yields two algebraic equations with the wanted coe cients I It can be shown, that the calibration points may not be coplanar I If this is not the case, the first matrix in the identification model is singular, since the columns 3 and 4 as well as 7 and 8 are linearly dependent Jianwei Zhang / Eugen Richter 437

35 8.2.2 Vision systems - Camera calibration - Basic concept of camera calibration Intelligent Robotics Problems I The presented solution is not globally optimal, since lens distortion has not been considered yet I It is not possible to determine the rotation matrix R and the translation vector t explicitly I This means that the presented calibration does not allow the use of a camera which is mounted to a moving robot arm I The creation of a precise 3D calibration setup is more complex than that of a 2D calibration object Jianwei Zhang / Eugen Richter 438

36 8.2.3 Vision systems - Camera calibration - Stereo vision Intelligent Robotics Stereo vision I Nevertheless, the previously presented calibration method allows a fast, although unprecise measurement of points with a stereo camera setup I For this purpose, two cameras A and B are calibrated and yield the calibraton vectors a A and a B I Then, the coordinates {x w, y w, z w } of each point which is captured by both cameras can be calculated I Each unknown point has the corresponding image coordinates {X A, Y A } and {X B, Y B } Jianwei Zhang / Eugen Richter 439

37 8.2.3 Vision systems - Camera calibration - Stereo vision Intelligent Robotics Stereo vision (cont.) Using the equation apple a11 a 31 X a 12 a 32 X a 13 a 33 X a 21 a 31 Y a 22 a 32 Y a 23 a 33 Y 2 3 x w apple 4y w 5 X a14 = Y a z 24 w for each camera, an over-determined equation system is formed, which allows the determination of the 3D-coordinate of a point from the image coordinates Jianwei Zhang / Eugen Richter 440

38 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Camera model with lens distortion I Real cameras and lenses produce a variety of imaging errors and do not satisfy constraints of the pinhole camera model I The main error sources are: 1. Spatial resolution quite low, since the resolution of the cameras is still low as well (current DV cameras: 320x200, fps; 800x600, fps; fps) 2. Most (cheap) lenses are asymmetrical and generate distortions 3. Assembly of the camera in a precise way is not possible (center of the CCD chips does not lie on the optical axis; chip is not parallel to the lens, etc.) 4. Timing errors between camera hardware and grabber hardware Jianwei Zhang / Eugen Richter 441

39 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Distortion I Distortion by the lens system results in a changed position of the image pixels on the image plane I The pinhole camera model is no longer su I It is replaced by the following model: u 0 = u + D u (u, v) v 0 = v + D v (u, v) cient where u and v are the non-observable, distortion-free image coordinates, and u 0 and v 0 the corresponding distorted coordinates Jianwei Zhang / Eugen Richter 442

40 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Distortion (cont.) Jianwei Zhang / Eugen Richter 443

41 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Distortion (cont.) Jianwei Zhang / Eugen Richter 444

42 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Types of distortion I There are two types of distortions: I I radial tangential I Radial distortion causes an o set of the ideal position inwards (barrel distortion) or outwards (pincushion distortion) I Cause: Flawed radial bend of the lens Jianwei Zhang / Eugen Richter 445

43 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Radial distortion Straight lines! no distortion Jianwei Zhang / Eugen Richter 446

44 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Tangential distortion Straight lines! no distortion Jianwei Zhang / Eugen Richter 447

45 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Modeling of the lens distortion I According to Weng et. al. (1992), three kinds of distortion are distinguished: 1. Radial distortion 2. Decentering distortion 3. Thin prism distortion I Decentering distortion and thin prism distortion are both radial and tangential I In the case of decentering distortion, optical centers of the lenses are not colinear Jianwei Zhang / Eugen Richter 448

46 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Model: Radial distortion Radial distortion D ur = ku(u 2 + v 2 )+O[(u, v) 5 ] D vr = kv(u 2 + v 2 )+O[(u, v) 5 ] Jianwei Zhang / Eugen Richter 449

47 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Simplified model Since radial lens distortion is the dominating e ect, the following equation system can be used as simplified camera model: Simplified camera model with distortion: u 0 = u(1 + k 0 r 02 ) v 0 = v(1 + k 0 r 02 ) with r 02 = u 2 + v 2 Jianwei Zhang / Eugen Richter 450

48 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Radial distortion coe cient Since u and v are unknown, they are replaced by the measurable image coordinates X and Y and one has r 02 =(X/s u ) 2 +(Y /s v ) 2 Defining k k 0 s 2 v,theradial distortion coe cient, one has µ f y f x = s v s u and r 2 µ 2 X 2 + Y 2 Jianwei Zhang / Eugen Richter 451

49 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Model for small radial distortions With the previously mentioned modifications, one gets the following camera model for small radial distortions X(1 + kr 2 ) = f x r 1 x w + r 2 y w + r 3 z w + t x r 7 x w + r 8 y w + r 9 z w + t z, Y (1 + kr 2 ) = f y r 4 x w + r 5 y w + r 6 z w + t y r 7 x w + r 8 y w + r 9 z w + t z Jianwei Zhang / Eugen Richter 452

50 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Variation AAusefultrickfortheleast squares method is the usage of the following variation of the previous model X 1 + kr 2 = f x r 1 x w + r 2 y w + r 3 z w + t x r 7 x w + r 8 y w + r 9 z w + t z, Y 1 + kr 2 = f y r 4 x w + r 5 y w + r 6 z w + t y r 7 x w + r 8 y w + r 9 z w + t z which applies under the assumption, that kr 2 << 1 Jianwei Zhang / Eugen Richter 453

51 8.2.4 Vision systems - Camera calibration - Camera model with lens distortion Intelligent Robotics Radial alignment constraint If radial distortion is the only distortion that occurs, one gets the radial alignment constraint (RAC) respectively: with X d = f x X and Y d = f y Y X Y = µ 1 r 1x w + r 2 y w + t x r 4 x w + r 5 y w + t y X d : Y d = x : y Jianwei Zhang / Eugen Richter 454

52 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Tsai s RAC-based camera calibration I Assumption: C x, C y and µ are known I Extrinsic parameters R and t and the intrinsic parameters f x, f y and k are to be determined I For calibration, a number of coplanar calibration points is used I Calibration consists of two steps 1. Determination of the rotation matrix R and the components t x and t y of the translation vector 2. Estimation of the other parameters based on the results of the first step Jianwei Zhang / Eugen Richter 455

53 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 1. Calculation of the image coordinates (X i, Y i ) Let N be the number of image points, then, for i = 1, 2,...,N one has X i = X f,i Y i = Y f,i C x C y where X f,i and Y f,i are the pixel values in the computer Jianwei Zhang / Eugen Richter 456

54 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) 2. Determination of intermediate parameters {v 1, v 2, v 3, v 4, v 5 } I Since RAC is independent from k and f! R, t x and t y can be calculated I For that we define {v 1, v 2, v 3, v 4, v 5 } {r 1 t 1 y, r 2 t 1 y, t x t 1 y, r 4 t 1 y, r 5 t 1 y } Jianwei Zhang / Eugen Richter 457

55 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) I If one divides both sides of the RAC-equation by t y for the i-th calibration point and rearranges the resulting expression, one has 2 3 xw,i Y i y w,i Y i Y i x w,i µx i v 2 y w,i µx i 6v 3 7 4v 4 5 = µx i v 5 where x w,i and y w,i are the x- andy-coordinates of the i-th calibration point v 1 Jianwei Zhang / Eugen Richter 458

56 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) I The minimum number of necessary non-colinear calibration points is N = 5 I In practice, an appropriate choice would be N > 5 I Note: If t y = 0, the above equation can also be formulated as a function of t x I If one determines t x = t y = 0, the chosen camera setup needs to be changed appropriately Jianwei Zhang / Eugen Richter 459

57 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) 3. Calculation of R, t x and t y apple v1 v I Define C 2 v 4 v 5 I If no line or column equals zero, one has: t 2 y = S r p S 2 r 4(v 1 v 5 v 4 v 2 ) 2 2(v 1 v 5 v 4 v 2 ) with S r v 2 1 +v 2 2 +v 2 4 +v 2 5 I Otherwise one has: t 2 y =(v 2 i + v 2 j ) 1 where v i and v j are the elements from C, which are non-zero Jianwei Zhang / Eugen Richter 460

58 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) I Physically, the algebraic signs of x and X as well as y and Y should be equal I This property is used to determine the algebraic sign of t y I Assuming t y > 0 following components can be calculated r 1 = v 1 t y r 2 = v 2 t y r 4 = v 4 t y r 5 = v 5 t y t x = v 3 t y Jianwei Zhang / Eugen Richter 461

59 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) I Using an arbitrary calibration point, the following coordinates can be determined: x = r 1 x w + r 2 y w + t x y = r 4 x w + r 5 y w + t y I If sign(x) =sign(x) and sign(y) =sign(y ) apply, then the assumption sign(t y )=1istrueandwekeepr 1, r 2, r 4, r 5 and t x I Otherwise, we set sign(t y )= 1 and change the algebraic signs of r 1, r 2, r 4, r 5 and t y accordingly Jianwei Zhang / Eugen Richter 462

60 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) I There are two possible solutions for the rotation matrix R, if a 2 2-submatrix is known I These solutions are due to f x having a positive and negative sign I R can be calculated as follows r 3 = ±(1 r 2 1 r 2 2 ) 1/2 r 6 = ±sign(r 1 r 4 + r 2 r 5 )(1 r 2 4 r 2 5 ) 1/2 [r 7 r 8 r 9 ] T =[r 1 r 2 r 3 ] T [r 4 r 5 r 6 ] T I One of the two solution leads to a positive f x in Step 2 of the calibration method Jianwei Zhang / Eugen Richter 463

61 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 1 (cont.) Note: I The resulting matrix R might not be orthonormal I Thus, orthonormalisation steps, which are not explained in more detail here, are additionally needed Jianwei Zhang / Eugen Richter 464

62 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 2 Determination of the parameters t z, k, f x and f y I If R, t x and t y are known, the remaining parameters for the i-th calibration point can be determined using the following equation: having Xi x i x i r 2 i 2 4 t z f x kf x 3 5 = X i w i x i r 1 x w,i + r 2 y w,i + t x w i r 7 x w,i + r 8 y w,i Jianwei Zhang / Eugen Richter 465

63 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics Camera calibration according to Tsai: Step 2 (cont.) I Whenever more than three calibration points are used, an over-determined equation system is the result I The solution using the least-squares-procedure yields the parameters k, t z and f x I Using f x the other parameters can be calculated: f y = f x µ k =(kf x )f x 1 Jianwei Zhang / Eugen Richter 466

64 8.2.5 Vision systems - Camera calibration - Camera calibration according to Tsai Intelligent Robotics 3D calibration setup typical 3D calibration setup Jianwei Zhang / Eugen Richter 467

65 8.2.6 Vision systems - Camera calibration - Fast RAC-based calibration Intelligent Robotics Fast RAC-based calibration I If only calibration points on the x- andy-axis of the world coordinate system are used in the first step of the Tsai algorithm, the RAC-equation is simplified I Typically: The middle line and middle column of a calibration plate define the x w -andthey w -axis I In the Tsai-algorithm, the linear-least-squares-procedure is applied to five variables in step one, and to three in step two I Using above simplification, the linear-least-squares-procedure needs to be applied three times for two variables I Since there s a closed solution for this, the calculation time needed for calibration is reduced significantly Jianwei Zhang / Eugen Richter 468

66 8.2.6 Vision systems - Camera calibration - Fast RAC-based calibration Intelligent Robotics Fast RAC-based calibration (cont.) I Requirement for the fast version of the Tsai-algorithm is that µ, C x and C y are known in advance I As with Tsai s calibration, there are two necessary steps: 1. Usage of calibration points on the x w -andy w -axis and a simplified RAC-equation, to determine R, t x and t y 2. Determination of the other parameters using all visible calibration points Jianwei Zhang / Eugen Richter 469

67 8.2.6 Vision systems - Camera calibration - Fast RAC-based calibration Intelligent Robotics Fast RAC-based calibration (cont.) Calibration points for the first phase of the fast RAC-based calibration Jianwei Zhang / Eugen Richter 470

68 8.2.6 Vision systems - Camera calibration - Fast RAC-based calibration Intelligent Robotics Fast RAC-based calibration (cont.) Typical calibration plate Jianwei Zhang / Eugen Richter 471

69 University of Hamburg Vision systems - Camera calibration - Fast RAC-based calibration Intelligent Robotics Fast RAC-based calibration (cont.) Using a calibrated camera, the image can be rectified Jianwei Zhang / Eugen Richter 472

70 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of a pointing direction: Motivation Motivation: I The recognition of hand gestures can be used in the field of human-computer-interaction I Applications in the field of virtual reality, multimedia, robot instruction or teleoperation Solutions: I Sensors on the hand (e.g. data glove) I Stereo vision (calibrated/uncalibrated) Jianwei Zhang / Eugen Richter 473

71 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of a pointing direction: Stereo vision Basic stereo setup with parallel optical axes Jianwei Zhang / Eugen Richter 474

72 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of a pointing direction: Epipolar lines The point corresponding to a point from Image 1 can be found on the corresponding epipolar line in Image 2 Jianwei Zhang / Eugen Richter 475

73 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of a pointing direction: Epipolar lines In the case of parallel optical axes, the epipolar lines are horizontal lines Jianwei Zhang / Eugen Richter 476

74 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Uncalibrated stereo vision I Cipolla et. al. (1994) present an uncalibrated stereo system for the recognition of pointing gestures I Assumption: Pinpoint camera model with view on a plane I The relation between plane coordinate system (X, Y ) and image coordinate system (u, v) is: su X 4sv5 = T 4Y 5 s 1 whereas T 3 3 is a homogeneous matrix having t 33 = 1 Jianwei Zhang / Eugen Richter 477

75 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Uncalibrated stereo vision (cont.) I In order to determine T, at least four points need to be observed I One defines the borders of the working plane as (0, 0), (0, 1), (1, 0) and (1, 1) I For both cameras, the transformations T and T 0 are determined Jianwei Zhang / Eugen Richter 478

76 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of the pointing spot Notation: l w : Longitudinal axis of the pointer in the world l i : Projection of l w onto the image plane l gp : Projection of l w on plane G Procedure: I Using the image of the second camera, one gets a projection l 0 gp, whose intersection point is the pointing spot l gp I l i is the image of l gp, in other words l i = Tl gp I As a consequence l gp = T 1 l i and l 0 gp = T 0 1 l i Jianwei Zhang / Eugen Richter 479

77 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of the pointing spot (cont.) Jianwei Zhang / Eugen Richter 480

78 8.3.1 Vision systems - Applications - Determination of a pointing direction Intelligent Robotics Determination of the pointing spot (cont.) Jianwei Zhang / Eugen Richter 481

79 8.3.2 Vision systems - Applications - Hand camera calibration Intelligent Robotics Hand camera calibration Camera-, gripper- and world-coordinate-system Jianwei Zhang / Eugen Richter 482

80 8.3.2 Vision systems - Applications - Hand camera calibration Intelligent Robotics Hand camera calibration (cont.) Task: Determination of the fixed spatial relation between camera- (C) and gripper-coordinate system (G) represented by the homogeneous transformation C H G Idea: Direct determination of C H G through model based localisation of visible gripper features Jianwei Zhang / Eugen Richter 483

81 8.3.2 Vision systems - Applications - Hand camera calibration Intelligent Robotics Hand camera calibration (cont.) Solution: I Positioning of the gripper on a planar calibration object with several measuring points I Result: Gripper- and world-coordinate system coincide I Plane coincidence allows composition of the problem C H G = C H W W H G Jianwei Zhang / Eugen Richter 484

82 8.3.2 Vision systems - Applications - Hand camera calibration Intelligent Robotics Hand camera calibration (cont.) Approach: 1. Determination of intrinsic and extrinsic camera parameters using the calibration object ) C H W 2. Determination of the parameters of a 2D-transformation W H G using visible gripper features Jianwei Zhang / Eugen Richter 485

83 8.3.2 Vision systems - Applications - Hand camera calibration Intelligent Robotics Hand camera calibration (cont.) Advantages of "self-visibility": I "Self-visibility" allows calibration of the configuration without test movements of the manipulator as opposed to classical procedures I Two dot-shaped gripper features are su cient for the determination of C H G, solution in closed form I Online-surveillance of the relative position between gripper and object I Higher accuracy of o kinematic errors ine calibration due to exclusion of I Higher level of robustness due to the possibility of online-calibration Jianwei Zhang / Eugen Richter 486

84 8.3.3 Vision systems - Applications - Visually controlled grasping Intelligent Robotics Visually controlled grasping Task: Two-dimensional fine positioning of a robot hand or a gripper in relation to the object that is to be grasped Procedure: 1. O ine-specification of the target position (e.g. object features from stereo image processing) 2. Online transformation of the current di erence in relation to the target position (e.g. with a hand camera) Jianwei Zhang / Eugen Richter 487

85 8.3.3 Vision systems - Applications - Visually controlled grasping Intelligent Robotics Visually controlled grasping (cont.) Jianwei Zhang / Eugen Richter 488

86 University of Hamburg Vision systems - Applications - Visually controlled grasping Intelligent Robotics Visually controlled grasping (cont.) Jianwei Zhang / Eugen Richter 489