Computer Vision. Marc Pollefeys Vittorio Ferrari Luc Van Gool

Transcription

1 Computer Vision Marc Pollefeys Vittorio Ferrari Luc Van Gool

2 Lecturers Marc Pollefeys Vittorio Ferrari Luc Van Gool Teaching assistants Olivier Saurer Petri Tanskanen

3 Administrivia Classes: Wed, 13-16, CHN C 14 Exercises: Thu, 13-15, RZ F 1 Instructors: Assistants: Marc Pollefeys Vittorio Ferrari Luc Van Gool Olivier Saurer Petri Transkanen Prerequisite: Visual Computing (or equivalent) (if missing background, rather choose: Image Analysis and Computer Vision, Székely, Ferrari, Van Gool, Thu 13-16) 3

4 Material Slides Webpage (with slides, assignments.) Reference book: Computer Vision: Algorithms and Applications Rick Szeliski 4

5 What is computer vision? Done?

6 What is computer vision? Automatic understanding of images and video Computing properties of the 3D world from visual data (measurement) Algorithms and representations to allow a machine to recognize objects, people, scenes, and activities. (perception and interpretation) Lots of slides borrowed from Svetlana Lazebnik, Trevor Darrell

7 Vision for measurement Real-time stereo Structure from motion Multi-view stereo for community photo collections NASA Mars Rover Pollefeys et al. Goesele et al. Slide credit: L. Lazebnik

8 Vision for perception, interpretation The Wicked Twister deck ride Lake Erie sky tree tree bench water Ferris wheel amusement park tree Cedar Point tree ride carousel 1 E Objects Activities Scenes Locations Text / writing Faces Gestures Motions Emotions ride people waiting in line people sitting on ride umbrellas pedestrians maxair

9 Related disciplines Graphics Image processing Photogrammetry Artificial intelligence Computer vision Algorithms Robotics Machine learning Cognitive science

10 Vision and graphics Images Vision Model Graphics Inverse problems: analysis and synthesis.

11 Why vision? As image sources multiply, so do applications Relieve humans of boring, easy tasks Enhance human abilities: humancomputer interaction, visualization Perception for robotics / autonomous agents Organize and give access to visual content

12 Why vision? Images and video are everywhere! Personal photo albums Movies, news, sports Surveillance and security Medical and scientific images Slide credit; L. Lazebnik

13 Why is vision hard? 13 slide credit: Michael Black

14 Challenges: viewpoint variation Michelangelo slide credit: Fei-Fei, Fergus & Torralba

15 Challenges: illumination image credit: J. Koenderink

16 Challenges: scale slide credit: Fei-Fei, Fergus & Torralba

17 Challenges: deformation Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba

18 Challenges: occlusion Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba

19 Challenges: background clutter slide credit: Svetlana Lazebnik

20 Challenges: Motion slide credit: Svetlana Lazebnik

21 Challenges: object intra-class variation slide credit: Fei-Fei, Fergus & Torralba

22 Challenges: local ambiguity slide credit: Fei-Fei, Fergus & Torralba

23 Challenges or opportunities? Images are confusing, but they also reveal the structure of the world through numerous cues Our job is to interpret the cues! Image source: J. Koenderink

24 Depth cues: Linear perspective slide credit: Svetlana Lazebnik

25 Depth cues: Aerial perspective slide credit: Svetlana Lazebnik

26 Depth ordering cues: Occlusion Source: J. Koenderink

27 Shape cues: Texture gradient slide credit: Svetlana Lazebnik

28 Shape and lighting cues: Shading Source: J. Koenderink

29 Position and lighting cues: Cast shadows Source: J. Koenderink

30 Grouping cues: Similarity (color, texture, proximity) slide credit: Svetlana Lazebnik

31 Grouping cues: Common fate Image credit: Arthus-Bertrand (via F. Durand)

32 Bottom line Perception is an inherently ambiguous problem Many different 3D scenes could have given rise to a particular D picture Possible solutions Bring in more constraints (more images) Use prior knowledge about the structure of the world Need a combination of different methods

33 Origins of computer vision L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.

34 Again, what is computer vision? Mathematics of geometry of image formation? Statistics of the natural world? Models for neuroscience? Engineering methods for matching images? Science Fiction?

35 Vision Demo? Terminator we re not quite there yet.

36 Every picture tells a story Goal of computer vision is to write computer programs that can interpret images

37 Can computers match (or beat) human vision? Yes and no (but mostly no!) humans are much better at hard things computers can be better at easy things

38 Human perception has its shortcomings Sinha and Poggio, Nature, 1996

39 Copyright A.Kitaoka 003

40 Current state of the art The next slides show some examples of what current vision systems can do

41 Earth viewers (3D modeling) Image from Microsoft s Virtual Earth (see also: Google Earth)

42 Photosynth Based on Photo Tourism technology developed by Noah Snavely, Steve Seitz, and Rick Szeliski

43 Photo Tourism overview Scene reconstruction Input photographs Relative camera positions and orientations Point cloud Sparse correspondence Photo Explorer System for interactive browsing and exploring large collections of photos of a scene. Computes viewpoint of each photo as well as a sparse 3d model of the scene.

44 Photo Tourism overview

45 Urban 3D Reconstruction 45 (Gallup, Frahm, Pollefeys DAGM010)

46 Optical character recognition (OCR) Technology to convert scanned docs to text If you have a scanner, it probably came with OCR software Digit recognition, AT&T labs License plate readers

47 Face detection Many new digital cameras now detect faces Canon, Sony, Fuji,

48 Smile detection? Sony Cyber-shot T70 Digital Still Camera

49 E.g. Picasa Face recognition 49

50 Object recognition (in supermarkets) LaneHawk by EvolutionRobotics A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk,you are assured to get paid for it

51 Face recognition Who is she?

52 Vision-based biometrics How the Afghan Girl was Identified by Her Iris Patterns Read the story

53 Login without a password Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely

54 Object recognition (in mobile phones) This is becoming real: Point & Find, Nokia Kooaba (ETH start-up) SnapTell

55 Special effects: shape capture The Matrix movies, ESC Entertainment, XYZRGB, NRC

56 Special effects: shape capture 56 (Beeler et al. Siggraph 010)

57 Special effects: motion capture Pirates of the Carribean, Industrial Light and Magic

58 Special effects: match moving Compute camera motion from point motion 58

59 Sports Sportvision first down line Nice explanation on

60 60 LiberoVision

61 Smart cars Mobileye Vision systems currently in high-end BMW, GM, Volvo models By 010: 70% of car manufacturers. Video demo Slide content courtesy of Amnon Shashua

62 Driver Assistance Daimler stereo system

63 Vision-based interaction (and games) Digimask: put your face on a 3D avatar. Nintendo Wii has camera-based IR tracking built in. See Lee s work at CMU on clever tricks on using it to create a multi-touch display! Game turns moviegoers into Human Joysticks, CNET Camera tracking a crowd, based on this work.

64 Vision-based interaction (and games) MS project natal/kinect

65 Vision in space NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 007. Vision systems (JPL) used for several tasks Panorama stitching 3D terrain modeling Obstacle detection, position tracking For more, read Computer Vision on Mars by Matthies et al.

66 Robotics NASA s Mars Spirit Rover

67 Micro Aerial Vehicles

68 Medical imaging 3D imaging MRI, CT Image guided surgery Grimson et al., MIT

69 Current state of the art You just saw examples of current systems. Many of these are less than 5 years old This is a very active research area, and rapidly changing Many new apps in the next 5 years To learn more about vision applications and companies David Lowe maintains an excellent overview of vision companies

70 Schedule (tentative) 70 # date topic 1 Sep. Introduction and geometry Sep.9 Camera models and calibration 3 Oct.6 Invariant features 4 Oct.13 Multiple-view geometry 5 Oct.0 Model fitting (RANSAC, EM, ) 6 Oct.7 Stereo Matching 7 Nov.3 Segmentation 8 Nov.10 D Shape Matching 9 Nov.17 Shape from Silhouettes 10 Nov.4 Optical Flow 11 Dec.1 Structure from motion 1 Dec.8 Tracking (Kalman, particle filter) 13 Dec.15 Object recognition 14 Dec. Object category recognition

71 Projective Geometry points, lines, planes conics and quadrics transformations camera model Read tutorial chapter and Szeliski s book pp.9-41

72 Homogeneous coordinates Homogeneous representation of D points and lines ax + by + c = 0 ( a,b,c) T ( x,y, 1) = 0 The point x lies on the line l if and only if l T x = 0 Note that scale is unimportant for incidence relation T T ( a,b,c) ~ k( a,b,c), k 0 T T ( x, y,1) ~ k( x, y,1), k 0 equivalence class of vectors, any vector is representative Set of all equivalence classes in R 3 (0,0,0) T forms P ( ) T Homogeneous coordinates x1, x, x Inhomogeneous coordinates ( x, y) T 3 but only DOF

73 Points from lines and viceversa Intersections of lines The intersection of two lines l and l' is x = l l' Line joining two points The line through two points x and x' is l = x x' Example Note: x x' = x [ ] x' x ( 0,1,-1) y 1 y = 1 with [ x] 0 = - z y z 0 - x - y x 0 x = 1 x ( 1,0,-1) y 1

74 Ideal points and the line at infinity Intersections of parallel lines l = ( ) ( ) T a, b, c T and l'= a, b, c' l l' = ( b, a,0) T Example ( b,-a) ( ) a,b tangent vector normal direction x = 1 x = Ideal points ( x, x,0 ) T 1 Line at infinity ( ) T l = 0,0,1 P = R l Note that in P there is no distinction between ideal points and others

75 3D points and planes Homogeneous representation of 3D points and planes π1x1 + π X + π3x 3 + π4 X 4 = 0 The point X lies on the plane π if and only if π T X = 0 The plane π goes through the point X if and only if π T X = 0

76 Planes from points 0 π X X X 3 1 = T T T π = 0 X π = 0 X π = 0, X π 3 1 T T T and from Solve (solve as right nullspace of ) π T T T 3 1 X X X

77 Solve π π π T 1 T T 3 X = Points from planes T T T X from π1 X = 0, πx = 0 and π3 X = 0 0 X = M x M = [ X X ] π T M = 0 X (solve as right nullspace of T ) Representing a plane by its span 1 X3 π π π T 1 T 3

78 Lines (4dof) Representing a line by its span W A B T = T Dual representation W * P Q T = T λa+ μb λp+ μq * T W W = WW * T = 0 Example: X-axis 0 W = W * = (Alternative: Plücker representation, details see e.g. H&Z)

79 Points, lines and planes W M = T M π = 0 X X W W * M = T M X = 0 π * W π

80 81 9//010

81 Conics Curve described by nd -degree equation in the plane ax + bxy + cy + dx + ey + f = 0 or homogenized x x 1, x3 y x ax1 + bx1x + cx + dx1x3 + exx3 + fx or in matrix form 5DOF: x T C x = 0 with { } a : b : c : d : e : x 3 a C = b / d / f b / c e / 3 = 0 d / e / f

82 Five points define a conic For each point the conic passes through = f ey dx cy y bx ax i i i i i i or ( ) 0 =,1.,,,, c i i i i i i y x y y x x ( ) T f e d c b a,,,,, = c = c y x y y x x y x y y x x y x y y x x y x y y x x y x y y x x stacking constraints yields

83 Tangent lines to conics The line l tangent to C at point x on C is given by l=cx x l C

84 Dual conics A line tangent to the conic C satisfies l T * C l = 0 In general (C full rank): * -1 C = C Dual conics = line conics = conic envelopes

85 Degenerate conics A conic is degenerate if matrix C is not of full rank e.g. two lines (rank ) C = lm T + ml e.g. repeated line (rank 1) T m l T C = ll l Degenerate line conics: points (rank ), double point (rank1) Note that for degenerate conics * * ( C ) C

86 Quadrics and dual quadrics X T QX = 0 9 d.o.f. (Q : 4x4 symmetric matrix) in general 9 points define quadric det Q=0 degenerate quadric tangent plane π = QX Q = π T * Q π = 0 Q * = Q relation to quadric (non-degenerate) -1

87 D projective transformations Definition: A projectivity is an invertible mapping h from P to itself such that three points x 1,x,x 3 lie on the same line if and only if h(x 1 ),h(x ),h(x 3 ) do. Theorem: A mapping h:p P is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P reprented by a vector x it is true that h(x)=hx Definition: Projective transformation x' 1 h = x' h x' 3 h h h h 1 3 h h h x x x 1 3 or x' = 8DOF H x projectivity=collineation=projective transformation=homography

88 Transformation of D points, lines and conics For a point transformation x' = H x Transformation for lines l'= -T H Transformation for conics l C ' = H -T CH -1 Transformation for dual conics C' * = * HC H T

89 Fixed points and lines H e = λ e (eigenvectors H =fixed points) (λ 1 =λ pointwise fixed line) H -T l l = λ (eigenvectors H - T =fixed lines)

90 Hierarchy of D transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof h11 h1 h31 a11 a1 0 sr sr 0 r11 r h h h a a r r sr sr h13 h 3 h 33 tx t y 1 tx t y 1 tx t y 1 transformed squares invariants Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l Ratios of lengths, angles. The circular points I,J lengths, areas.

91 The line at infinity 0 T T A 0 l = H A l = 0 = l T T t 1 A 1 The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity Note: not fixed pointwise

92 Affine properties from images projection rectification A PA l l l H H = [ ] 0, l l l l l T =

93 Affine rectification v 1 v l l 1 l 3 l l 4 v = l3 l4 v 1 = l1 l l = v v 1

94 The circular points = 0 1 I i = 0 1 J i I cos sin sin cos I I = = = = i se i t s s t s s i y x S θ θ θ θ θ H The circular points I, J are fixed points under the projective transformation H iff H is a similarity

95 The circular points circular points l x1 + x + dx1x3 + exx3 + fx3 = 0 x + 1 x = 0 x 3 = 0 T I = 1, i,0 Algebraically, encodes orthogonal directions I = ( ) ( ) T 1,0,0 T + i 0,1, 0 J = ( ) ( 1,-i,0) T

96 Conic dual to the circular points C C * T = IJ + * = H C JI T * T S H S * C = J l I * C The dual conic is fixed conic under the projective transformation H iff H is a similarity Note: * C has 4DOF l is the nullvector

97 Angles ( )( ) = cos m m l l m l m l θ ( ) T 3 1,, = l l l l ( ) T 3 1,, = m m m m Euclidean: Projective: ( )( ) m m l l m l cos * * * = C C C T T T θ m = 0 l * C T (orthogonal)

98 Transformation of 3D points, planes and quadrics For a point transformation X' = Transformation for lines π' = H X -T H π Transformation for quadrics Q' = H -T QH -1 (cfr. D equivalent) ( x' = H x) -T ( ) l'= H ( -T CH -1 ) C ' = H l Transformation for dual quadrics Q' * * = HQ H T * T ( ) C' * = HC H

99 Hierarchy of 3D transformations Projective 15dof A T v t v Intersection and tangency Affine 1dof A T 0 t 1 Parallellism of planes, Volume ratios, centroids, The plane at infinity π Similarity 7dof s R T 0 t 1 Angles, ratios of length The absolute conic Ω Euclidean 6dof R 0 T t 1 Volume

100 The plane at infinity 0 T T A 0 0 π = H A π = = π T T -t A The plane at infinity π is a fixed plane under a projective transformation H iff H is an affinity π = ( 0,0,0,1) T ( ) T 1. canonical position. contains directions D = X1, X, X 3,0 3. two planes are parallel line of intersection in π 4. line // line (or plane) point of intersection in π

101 The absolute conic The absolute conic Ω is a (point) conic on π. In a metric frame: X or conic for directions: (with no real points) 1 + X X 4 + X The absolute conic Ω is a fixed conic under the projective transformation H iff H is a similarity 3 = ( X, X, X ) I( X, X X ) T , 3 1. Ω is only fixed as a set. Circle intersect Ω in two circular points 3. Spheres intersect π in Ω

102 The absolute dual quadric = Ω I * T The absolute dual quadric Ω * is a fixed conic under the projective transformation H iff H is a similarity 1. 8 dof. plane at infinity π is the nullvector of Ω 3. Angles: ( )( ) * 1 * 1 * 1 π Ω π π Ω π π Ω π = cos T T T θ

103 10 Next week: cameras

104 Computer Vision Lab OPEN NIGHT Where? ETF C109 When? Today from 5 pm /home/nrazavi/desktop/screenshot-3.png