All human beings desire to know. [...] sight, more than any other senses, gives us knowledge of things and clarifies many differences among them.

Transcription

1 All human beings desire to know. [...] sight, more than any other senses, gives us knowledge of things and clarifies many differences among them. - Aristotle

2 University of Texas at Arlington Introduction to Vision Sensing CSE Autonomous Robots

3 How to study vision? The eye Vision is our most powerful sense! It consists of discovering from images what is present in the scene and where it is. For Aristotle (4th century B.C.) vision was possible because the observed object altered the medium (i.e., the air) between the objet itself and the eye. In the Middle Ages some popular theories suggested that the viewer's eyes sent out emissions to the object, and that those emissions enabled vision to occur. 1 (from left) Plato and Aristotle (by Raphael) Today, from empirical observations, we know more about the eye! & fovea 7

4 Rods and Cones The retina measures ~ 5x5 cm and contains 108 sampling elements (rods and cones) The eye's spatial resolution is about 0.01deg over 150deg field-of-view (not evenly spaced). Intensity resolution is about 11bits/element, spectral resolution is about 2 bits/elem. Temporal resolution is about 100 ms (10 Hz). What is the Bytes/sec rate that arrives to the brain? 108*11*10/8*2 = 2.75 GBytes/s Large portion of the brain dedicated to processing the signals from our eyes!

5 Why not to copy the biology? Very difficult to copy the eye and brain... human vision involves almost 60 bilion neurons! Computers available cannot perform like the human brain. It is more important to understand the underlying principle rather than the particular implementation or replica.... like in flight! VS

6 What is Computer Vision? In Computer Vision a camera (or several cameras) is connected to a computer GOAL: to interpret images of an observed scene automatically and obtain some useful information about the scene (e.g., 3-D, distance, shape, etc.) This information is necessary to perform actions (navigation, manipulation and recognition) Note that Computer Vision is not : Image processing: Process an image to produce a new image which is more desirable (e.g., image enhancement, image restoration, compression, etc.) Pattern Recognition: Classify patterns into a finite set of prototypes.

7 Applications of Computer Vision (1) Automation of industrial processes - Object recognition - Robot Hand-Eye Coordination - Robot Navigation - Visual inspection, quality control Courtesy of VisionX Inc. Courtesy of Stevens Univ. (Multi-Scale Rob. Lab)

8 Applications of Computer Vision (2) Surveillance and tracking - Traffic, aircraft, motion capture, pedestrian detection, etc...

9 Applications of Computer Vision (3) Augmented reality and visualization - 3D model building from image sequences - Important for archeology and for enhanced web-browsing experience Courtesy of Noah Snavely (Cornell University)

10 Applications of Computer Vision (4) Medical Applications Photogrammetry and Forensic Applications

11 Applications of Computer Vision (5) Wearable Devices: Localization and Scene recognition (courtesy of Active Vision Lab, Oxford University)

12 Applications of Computer Vision (6) 3D Reconstruction: (courtesy of Center for Machine Perception, Prague)...

13 The camera Cameras are the common ingredient to all the previous applications! CCD measures about 1x1 cm and contains 5x105 elements (pixels) Intensity resolution is about 8 bits/pixel. Most CV application use monochrome images. Temporal resolution is about 40 ms (25 Hz). The information rate given by a camera is about 12 MBytes/sec.

14 Vision as... information processing Vision involves a huge amount of data reduction: Images (12 Mbytes/s) Salient features (5 Kbytes/s) Representation and Actions (1-100 bits/s) Vision is about resolving inherent ambiguities of the imaging process (e.g., depth) - More images are needed to infer appropriate constraints; - Make assumptions about the observed scene. One cannot understand what seeing is, and how it works, unless one understands the underlying information processing tasks being involved [David Marr] Computer Vision is about inverting the vision process...from images to 3-D!

15 Camera models We first need to understand how an image is formed... that is, we need to construct a camera model Why knowing the camera model is important? (e.g., predict object appearance)

16 Orthographic Projection In the orthographic camera model, all the rays are assumed to arrive parallel to the image plane. z World object Image plane x Some images we take with CCD cameras do, indeed, look as if they were formed by orthographic projection...some other do not!

17 Perspective Projection (1) We need a more general projection model, that can explain better the observed scene. The pin-hole camera model is based on the so-called perspective projection Brunelleschi, Camera Obscura, XVth century World object Real image Pin-hole (or Optical Centre) 2-D pin-hole model principal point [meters]

18 Perspective Projection (2) principal point By analysing similar triangles, we find that: In the three-dimensional case, we also have Vector notation: 3-D point (with coordinates X,Y and Z) expressed in the {C} camera frame

19 Perspective Projection (3) principal point An alternative way to obtain the perspective projection model is the following: (i) By expressing (i) in vector form we finally obtain the pin-hole model : What is the unit of measurement for x? Note that distant points to the optical axis appear to converge to the principal point x=[0,0]' 2-D image point projection

20 Example: Perspective Projection A 3-D point is distant 2m from the optical centre, with an elevation angle of 45 deg. and zero azimut angle (i.e., about Xc). The focal length is f=20mm. Compute the projection x of the point in the image plane. (pair!) x y

21 Vanishing Points (1) Parallel lines in the image meet at the horizon line.... vanishing points and horizon lines were used to introduce a new realism into art!

22 Vanishing Points (2) A vanishing point is a point in the image where parallel lines appear to meet What happens to the projection of B? Each set of parallel lines will have a different vanishing point in the image

23 Intrinsic Pin-hole Camera Parameters A full camera model must take into account also for other intrinsic camera parameters: u - geometry of the CCD array - its position wrt the optical axis v We define the pixel coordinates coordinates What is the measuring unit of and in addition to the image plane homogeneous coordinates representation

24 Intrinsic Camera Calibration Matrix Recall the pin-hole model we obtained before: (check this!) and, without loss of generality s=1, so we obtain the image plane coordinates: (ii) equiv. to: Using (ii) into the expression for the pixel coordinates (prev. slide) : pixels and finally... Ideal Projection Matrix Intrinsic Camera Calibration Matrix

25 Looking for a General Pinhole Model So far we examined a simple pinhole camera model... why simple? Image Coordinates Pixel Coordinates GOAL: Extend the simple model to the case of a 3-D point expressed in the world reference frame.?

26 Rigid Transformations Our goal is to find a representation for a rigid transformation between {C} and {W} Extrinsic Camera Parameters

27 Homogeneous Transformations (1) In the figure, a rotation and a translation separate {C} from {W}. GOAL: On the basis of simple geometry If we consider the 2-D case we have: Which, rewriting the points in homogeneous coordinates, it leads to an expression linear in point coordinates Homogeneous coordinate transformation

28 Homogeneous Transformations (3) The rigid transformation between two frames can be expressed in homogeneous form as: where Suppose we want to compute the inverse homogeneous transform from {W} to {C}, i.e.: Thus showing that

29 Our Goal: Generalized Pinhole Model Rewrite the pinhole camera model as: such that The generic rigid motion transformation is given by: Intrinsic parameters Extrinsic parameters Generalized Pinhole Camera Model

30 Stereo Vision It is impossible to infer the depth of an object from only one image. Stereo Vision refers to the ability to: infer info about the 3-D structure (and distance) of a scene from two (or more) images taken from different viewpoints. depth map left image right image The difference (disparity) in perception between left and right eye allows to have a 3-D perception (example: use the thumb!)

31 Stereo Correspondence & Reconstruction From a computational point of view, stereo-vision must solve : 1) a correspondence problem left image right image 2) a reconstruction problem 3-D scene left camera right camera

32 A simple stereo system Triangulation allows to obtain the spatial coordinates of ( {W}={L} ) from,, and. Solution for and solving for left camera and comes from geometry:, we have: right camera disparity and solving for, we have: Check on some basic cases:

33 Simple Stereo Model: Properties Parallax: - distant objects appear to have low disparity Baseline: left camera right camera - longer baseline helps computing depth of distant objects. FPGA: - Modern models come directly with a FPGA (Field-Programmable Gate Array) that can compute the 3-D in real-time!