ROBOTICS 01PEEQW. Basilio Bona DAUIN Politecnico di Torino

Transcription

1 ROBOTICS 01PEEQW DAUIN Politecnico di Torino

2 Mobile & Service Robotics Sensors for Robotics 4

3 Vision Vision is the most important sense in humans and is becoming important also in robotics not expensive rich of information Vision includes three steps Data recording and transformation in the retina Data transmission through the optical nerves Data elaboration by the brain 3

4 Vision sensors CCD (Coupled Charge Device, light-sensitive, discharging capacitors) CMOS (Complementary Metal Oxide Semiconductor technology) 4

5 CCD sensors A CCDsensor consists in a range of capacitors, each accumulating a charge proportional to the quantity of light that is hitting it. The charge in each capacitor is then turned into a numerical value by the camera inner system to produce a picture. Advantages/Features of CCD Sensors Conversion takes place in the chip without distortion CCDs have very high uniformity Good for HD quality images (not videos) These sensors are more sensitive Produce Better Images in Low Light CCD sensors produce cleaner and less grainy Images Low-noise images CCD sensors has been produced for a longer period of time Disadvantages of CCD Sensors CCD sensors consume much more power CCDs are interlaced Inferior HDvideos Less pixel rates CCDs are expensive as they require special manufacturing 5

6 CMOS sensors CMOS sensors can directly make the charge conversion on the generation photosite thanks to their pixel amplifier. Severaltransistors at each pixel amplify and move the charge using more traditional wires. Charge isthen turned into a numericalcal value which correspond to the image. This characteristic give them the ability to avoid several transfers and to increase the processing speed CMOS do not require any specialmanufacturing. Mostof the digital cameras these days use CMOSSensor as itreduces cost Advantages/ Features Of CMOS Sensor CMOS consumes less power (100 times less than CCD) CMOS sensors are cheaper Each pixel can be individually addressed. High reading rate They produce better HD videos Disadvantages Of CMOS Sensor CMOS sensors are also more susceptible sometimes images are grainy CMOS sensors need more light for better image Despite some disadvantages CMOSsensor is widely used in Mobile Phones, Tablets, PDAs and on most of the digital cameras. CCD cameras produce better images but CMOS sensors are catching up fast with its low power consumption. 6

7 Artificial vision issues Projection from a 3D world on a 2D plane: perspective projection (transformation matrices) Discretization effects due to pixels (CCD or CMOS) Misalignment errors (hardware) Parallel lines Converging lines Pixel discretization 7

8 Camera models Pinhole camera (aka perspective camera) 8

9 Pinhole camera image planes A hole diameter point images the image is reversed B point images Decreasing the image plane distance or the hole diameter makes the point images sharper Increasing the hole diameter makes the point images brighter Infinite depth-of-field Infinite depth-of-focus 9

10 Camera models Thin lens camera: the lenshas a thickness d that is negligible compared to the radii of curvatureof the lens surfaces R 1, R 2 Rays are refracted as they go through the lens (refraction index n) Thin lens equation ( n 1) ; R i > 0 if convex f R R

11 Thin lens camera Thin lens camera is reversible Rays parallel to the optical axis pass through the focus and viceversa Rays through the lens center are not refracted There are two symmetrical foci True lens shows aberration phenomena lens center optical axis f f 11

12 Aberration Spherical 12

13 Image formation Thin lens approximation Pinhole camera Principal image plane π Reversed image plane Optical axis π F π 3D object Focal Plane 13

14 Image formation and equations Lens equation ( p f) f = pq = f( p + q) + = f ( q f) p q f real object field of view angle focal plane image plane p object distance f f focal distance q image distance q pf = if p f then ( p f) q f the image plane is approx in the focal plane 14

15 Image formation P P i p x p= p y p z x i p = i y i k c i c C c f P i x i p x π F π P p z 15

16 Transformations Coordinate transformation between the world frame and the camera frame Projection of 3D point coordinates onto 2D image plane coordinates Coordinate transformation between possible choices of image coordinate frame 16

17 Transformations c c R t 0 0 = 1 T 0 T R 0 World frame c 0 R c Camera frame i T pix T c i = f pix R i R π Rescaling Optical correction Image plane 17

18 Reference frames f World frame R 0 p 0 i c p c P i p i i i k c C c R i Optical axis P p c R c Focal plane j c π F u R pix O i p pix v π j i Image plane translation translation+scale R R R c i pix 18

19 Vector notation in 3D R R R p = x y x p = x y x p = x y x c c c c c c c c c c T T T in 2D R R i i i i pix p = x y T p = u v pix T in pixel units 19

20 Camera projections Perspective projection Orthographic projection p x p = x = f if p const x =αp z i x p f x i x i p z z large compared to the distance from the camera small compared to the distance from the camera The pixel height of similar subject is different if the distance from the camera varies a lot. On the left the persons have different pixel height while on the right they have approximately similar heights, since their distance from the camera is high and does not vary much 20

21 Projections 21

22 Perspective projection i c p z P i k c C c x i C i All points give the same image P p x P P i π π π F f f p x p = p = f p f x x i x z z i Usually the negative sign is avoided considering the reversed image plane 22

23 Perspective projection P p x p = p c y p z p i px f p x z i p f y P p = f = y c p i z perspective projection pz f p z f f 0 0 i p = p p p p = = 0 f 0 p Pp z i c z i c c c arbitrary positive constant λ f 0 0 f i p = 0 f 0 p = = 0 f p Pp i c c c c 23

24 Perspective projection Homogeneous coordinates pɶ = p p p 1 c x y z T pɶ = x y 1 i i i T Homogeneous perspective/projection matrix f x i p = p i z i p y p = z i p x f x i p y i i c pɶ = z i z i = = = c c c 0 0 p Πpɶ ΠTpɶ z p p y f λp = ΠTpɶ i i c c 0 0 f x i y i 24

25 Perspective projection this is the ideal case Canonical projection matrix ΠT i c c 0 f c c f R t = T 25

26 Perspective projection PP is studied by projective geometry PP preserves linearity; lines in 3D correspond to lines in 2D and viceversa PP does not preserve parallelism The intersection points in 2D of parallel lines in 3D define vanishing points (points infinitely far away) 26

27 Camera parameters Intrinsic parameters: the parameters that link the pixel coordinates of an image point to the corresponding (metric) coordinates in the camera reference frames Extrinsic parameters: the parameter that define the location and orientation of the camera reference frame with respect to a known world reference frame T W c W W R t c c = 1 6 parameters 0 T Camera calibration: the procedure to estimate these parameters 27

28 Camera intrinsic parameters s y R pix ncolumns s x c pix u i pix pix sx, sy in m s s x y aspect ratio mrows v R i i i uv, in pixel units j pix j i ( u= 6, v= 8) c pix o = x oy camera center in pixel units 28

29 Camera intrinsic parameters Focal length f Transformation between pixel coordinates and camera coordinates Geometric distortion introduced by the optical lens systems 29

30 Camera intrinsic parameters Transformation between pixel coordinates and camera coordinates (scaling + translation) in homogeneous coordinates = x i+ x = y i + y u sx o v sy o x i sx 0 so x px x pi y i = 0 sy so y py y pi pɶ i 30

31 Lens distorsion Types Pincushion distortion Radial distortion Barrel distortion Non radial distortion (tangential) Radial distortion is modelled by a function D(r) that affects each point vin the projected plane relative to the principal point p, where D(r) is normally a non-linear scalar function and pis close to the midpoint of the projected image. Barrel projections are characterized by a positive gradient of the distortion function, whereas pincushion by a negative gradient v = Dv ( pv ) + p d 31

32 Lens distortion Radial distortion is approximated by i i id id ( kr kr kr ) x = x ( kr kr kr ) y= y where xid, yid are the coordinates of the distorted image points r = x + y id id is the square of the distance from the camera image center k1, k2, k3 are intrinsic parameters 32

33 Other image sensor errors Errors are due to the imperfect orthogonalityof pixel elements in CCD or CMOS sensors 33

34 Optical sensors Optical distance sensors Depth from focus Stereo vision ToF cameras Motion and optical flow 34

35 Depth from focus The method consists in measuring the distance of objects in a scene evaluating from two or more images the focal length adjustment necessary to bring objects in focus Short distance focused Medium distance focused Far distance focused 35

36 Depth from focus 36

37 Depth from focus Near focusing Far focusing 37

38 Depth from focus = + f f D e D L D (, x y, z) image plane ( x, y) i i focal plane δ e bx ( ) L( d+ e D )1 1 1 = 2 f ( d+ e) sx ( ) blur radius shape 38

39 Stereo Cameras 39

40 Stereo disparity ( x, y, z) left lens f z x right lens image plane ( x, y) l l ( x, y) r r b baseline (known) 40

41 Stereo disparity 41

42 Stereo disparity Idealized camera geometry for stereo vision x /2 x l x+ b r x b/2 =, = f z f z x x l r b = f z ( x + x) /2 l r x= b x x l r ( y + y) /2 l r y= b y y l r f z= b x x l r Disparity between two images Depth computation 42

43 Stereo vision Distance is inversely proportional to disparity closer objects can be measured more accurately Disparity is proportional to baseline For a given disparity error, the accuracy of the depth estimate increases with increasing baseline b However, as bis increased, some objects may appear in one camera, but not in the other A point visible from both cameras produces a conjugate pair Conjugate pairs lie on epipolar line(parallel to the x-axis for the arrangement in the figure above) 43

44 Stereo vision 44

45 Stereo points correspondence These two points are corresponding: how do you find them in the two images? Left image Right image Right Left Disparity 45

46 Epipolar lines P corresponding points stay on the epipolar lines π τ τ 1 2 q 1 q 2 l 1 l 2 C 1 epipolar lines e 1 Rt, e 2 these two points are known and fixed (they are called epipoles) C 2 46

47 Stereo vision Depth calculation The key problem in stereo vision is how to optimally solve the correspondence problem Corresponding points lie on the epipolar lines Gray-Level Matching Match gray-level features on corresponding epipolar lines Zero-crossing of Laplacian of Gaussians is a widely used approach for identifying the same feature in the left and right images Brightness = image irradiance or intensity I(x,y) is computed and used as shown below 47

48 Depth images 48

49 Time of Flight (ToF) Cameras Range imaging systems based on the known speed of light They measure the time-of-flight, i.e., the time from the emission to the return of the signal The measurement is performed for each point of the image (different from Lidars) The distance resolution is 1 cm (larger than Lidars ) The simplest version of a time-of-flight camera useslight pulses 49

50 Time of Flight (ToF) Cameras Modulated light is emitted by a transmitter. This light is reflected by the object to be detected. The returning light is sampled by an on-chip photosensitive TOF CCD array. The receiver compares the phase difference between the emitted and the received light and computes the time difference of the "Time-of-Flight" individually per pixel. This value multiplied by the speed of light (ca. 300'000km/sec) and divided by 2 corresponds directly linearly to the distance. 50

51 Time of Flight (ToF) Cameras The single pixelconsists of a photo diode that converts the incoming light into a current In analog solutions fast switches are connected to the photo diode, which sends the current to one of two memory elements (capacitors) that act as summation elements In digital solutions a time counter, running at several gigahertz, is connected to each pixel and stops counting when light is sensed 51

52 Time of Flight (ToF) Cameras Pros Simplicity In contrast tostereo vision ortriangulation systems, the whole system is very compact: the illumination is placed just next to the lens, whereas the other systems need a certain minimum base line. In contrast tolaser scanning systems, no mechanical moving parts are needed Efficient distance algorithm It is very easy to extract the distance information out of the output signals of the TOF sensor, therefore this task uses only a small amount of processing power, again in contrast to stereo vision, where complex correlation algorithms have to be implemented. After the distance data has been extracted, object detection, for example, is also easy to carry out because the algorithms are not disturbed by patterns on the object Speed Time-of-flight cameras are able to measure the distances within a complete scene with one shot. As the cameras reach up to 160 frames per second, they are ideally suited to be used in real-time applications 52

53 Time of Flight (ToF) Cameras Cons Background light When using CMOS or other integrating detectors or sensors that use visible or near visible light (400nm -700nm), although most of the background light coming from artificial lighting or the sun is suppressed, the pixel still has to provide a highdynamic range. The background light also generates electrons, which have to be stored. For example, the illumination units in many of today's TOF cameras can provide an illumination level of about 1 watt. The Sun has an illumination power of about 50 watts per square meter after the optical band-pass filter. Therefore, if the illuminated scene has a size of 1 square meter, the light from the sun is 50 times stronger than the modulated signal. For non-integrating TOF sensors that do not integrate light over time and are using near-infrared detectors (InGaAs) to capture the short laser pulse, direct viewing of the sun is a non-issue. Such TOF sensors are used in space applications and in consideration for automotive applications 53

54 Time of Flight (ToF) Cameras Cons Interference In certain types of TOF devices, if several time-of-flight cameras are running at the same time, the TOF cameras may disturb each other's measurements. Multiple reflections In contrast to laser scanning systems, where only a single point is illuminated at once, the time-of-flight cameras illuminate a whole scene. On a phase difference device, due to multiple reflections, the light may reach the objects along several paths and therefore, the measured distance may be greater than the true distance. Direct TOF imagers are vulnerable if the light is reflecting from a specular surface. There are published papers that outline the strengths and weaknesses of the various TOF devices and approaches 54

55 Optical flow Optical flow is the pattern of apparent motion of objects, surfaces, and edges in successive scenes caused by the relative motion between the camera and the scene Optical flow techniques motion detection object segmentation time-to-collision motion compensated encoding stereo disparity measurement 55

56 Optical flow 56

57 Optical flow The optical flow methods try to calculate the motion between two image frames which are taken at times t and t + δtat every voxelposition. These methods are called differential since they are based on local Taylor series approximations of the image signal, i.e., they use partial derivatives with respect to the spatial and temporal coordinates I(, x yt,) = I( x+ δx, y+ δyt, + δt) I I I = I(, x yt,) + δx+ δy+ δt+... x y t I I I δx+ δy+ δt= x y t 0 Assuming the movement to be small Avoxel(volume+pixel) is a volume element representing a value on aregular gridin3d space 57

58 Optical flow I I I V + V + = x y x y t T I V= I t 0 This problem is known as the aperture problemof the optical flow algorithms There is only one equation in two unknownsand therefore cannot be solved To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow. 58

59 Optical flow Lucas Kanade Optical Flow Method A two-frame differential methods for motion estimation The additional constraints needed for the estimation of the flow are introduced in this method by assuming that the flow V, V is constant x y in a small window of size with m> 1 which is centered at pixel(, x y) 2 Numbering the pixels as 1,,n= m a set of equations can be found I V + I V = I x1 x y1 y t I I I 1 x1 y1 t1 I V + I V = I I I V I x2 x y2 y t 2 x2 y2 x t2 = V y I V + I V = I I I I xn x yn y t xn yn t n n T 1 T Ax= b x= ( AA) Ab 59