Computer Vision I - Introduction

Size: px

Start display at page:

Download "Computer Vision I - Introduction"

Miles Cross
6 years ago
Views:

1 Computer Vision I - Introduction Carsten Rother 21/10/2014 Computer Vision I:Introduction

2 Computer Vision I: Introduction 21/10/ Admin Stuff Language: German/English; Slides: English (all the terminology and books are in English) Lecturer: Carsten Rother and Holger Heidrich Exercises: Dmitri Schlesinger and Holger Heidrich Staff carsten.rother@tu-dresden.de Announcements: online (to be set up) Course Books: Image Processing/Geometry: Computer Vision: Algorithms and Applications by Rick Szeliski; Springer An earlier version of the book is online: Geometry: Multiple View Geometry; Hartley and Zisserman; Cambridge Press Second edition. Parts of book are online: Also pointers to conference and journal articles

3 Course Overview (total 14 lectures) Computer Vision I: Introduction 21/10/ VL1 (17.10): Introduction Ex1: Intro to OpenCV VL2 (24.10): Image Processing Part 1 Ex2: Intro to exercise: Image Processing VL3 (7.11): Image Processing Part 2 Ex3: homework VL4 (14.11): Fourier Analysis Part 1 Ex4: homework VL5 (21.11): Fourier Analysis Part 2 Ex5: homework VL6 (28.11): Projective Geometry Ex6: Intro to exercise: Fourier Analysis VL7 (5.12): Image Formation Process Ex7: homework

4 Course Overview (total 14 lectures) Computer Vision I: Introduction 21/10/ VL8 (12.12): 2-view Geometry Part 1 Ex8: homework VL9 (19.12): 2-view Geometry Part 2 Ex9: homework VL10 (9.1): Multi-View Geometry Part 1 Ex10: Intro to exercise: Panoramic Stitching / Geometry VL11 (16.1): Multi-View Geometry Part 2 Ex11: homework VL12 (23.1): Tracking Part 1 Ex12: homework VL13 (30.1): Tracking Part 2 Ex13: homework VL14 (6.2): Wrap-Up: 100 things we have learned Ex14: homework

5 Computer Vision I: Introduction 21/10/ Exams and Exercises Exam: in person Exercises/homework: There are 3 blocks Each block has several exercises with different points The exercises have to be handed in until end of semester (ideally after each block) Last possible date to hand in is end of semester (end of January) Collaboration: You are encouraged to discuss the topics You are not allowed to copy any code for the homework from other people

6 Computer Vision I: Introduction 21/10/ CVLD Lectures WS 14/15 Computer Vision 1 (2+2) Machine Learning 1 (2+2) Intelligent Systems (Vordipolm) (2+2) SS 15 Computer Vision 2 (2+2) Machine Learning 2 (2+2) Image processing (2+2) For doing a Master/PhD in the CVLD one should do the computer vision or machine learning track Computer graphics (Prof. Gumhold) (Introduction, I, II) 3D Scanning with structured light; Illumination models; Geometry

7 Computer Vision I: Introduction 21/10/ Before we start some Advertisement CVLD Overview Interactive Image and Data manipulation Applied Optimization, Models, and Learning 3D Scene Understanding Inverse rendering from moving images Benchmarking and Label collection BioImaging

8 Future in Computer Vision Computer Vision I: Introduction 21/10/ A project work in the CVLD is a good stepping stone if you: want to do a PhD in computer vision, graphics, machine learning want to become a researcher or software developer in one of the big research labs (Microsoft Research, Google, Adobe, TechniColor, etc) If you are interested in doing a start-up Other computer vision related industry

9 Introduction to Computer Vision Computer Vision I: Introduction 21/10/ What is computer Vision? (Potential) Definition: Developing computational models and algorithms to interpret digital images and visual data in order to understand the visual world we live in.

10 Introduction to Computer Vision Computer Vision I: Introduction 21/10/ What is computer Vision? (Potential) Definition: Developing computational models and algorithms to interpret digital images and visual data in order to understand the visual world we live in.

11 What does it mean to understand? Computer Vision I: Introduction 21/10/ Physics-based vision: Geometry Segmentation Camera parameters Emitted light (sun) Surface properties: Reflectance, material (Potential) Definition: Developing computational models and algorithms to interpret digital images and visual data in order to understand the visual world we live in. Semantic-based vision: Objects: class, pose Scene: outdoor, Attributes/Properties: - old-fashioned train - A-on-top-of-B

12 Image-formation model Very many sources of variability Image [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

13 Image-formation model Scene type Scene geometry Street scene [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

14 Image-formation model Scene type Scene geometry Object classes Sky Building 3 Road Street scene Sidewalk Bicycle Tree 3 Car 5 Person 4 Bench Bollard [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

15 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Sky Building 3 Road Street scene Sidewalk Bicycle Tree 3 Car 5 Person 4 Bench Bollard [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

16 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Street scene [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

17 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

18 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

19 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

20 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

21 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows Motion blur Camera effects [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

22 Image-formation model Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows Motion blur Camera effects [Slide Credits: John Winn, ICML 2008] Computer Vision I: Introduction 21/10/

23 The Scene Parsing challenge --- a grand challenge of computer vision Computer Vision I: Introduction 21/10/ (Probabilistic) Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} Single image Many applications do not have to extract the full probabilistic script but only a subset, e.g. does the image contain a car? many examples to come later

24 Why is scene parsing hard? Computer Vision I: Introduction 21/10/ Computer Graphics 3D Rich Representation, Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} 2D pixel representation Computer Vision Computer Vision can be seen as inverse graphics

25 Example of a recent work Computer Vision I: Introduction 21/10/ Scene graph Input [Gupta, Efros, Herbert, ECCV 10]

26 Why is scene parsing hard? [Xiao et al. NIPS 2012] [Sussman, Lamport, Guzman 1966] [Slide credits Andrew Blake] Computer Vision I: Introduction 21/10/

27 Introduction to Computer Vision Computer Vision I: Introduction 21/10/ What is computer Vision? (Potential) Definition: Developing computational models and algorithms to interpret digital images and visual data in order to understand the visual world we live in.

28 How can we interpret visual data? Computer Vision I: Introduction 21/10/ D pixel representation Computer Graphics 3D Rich Representation, Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} Computer Vision What general (prior) knowledge of the world (not necessarily visual) can be exploit? What properties / cues from the image can be used? Both aspects are quite well understood (a lot is based on physics) but how to use them is efficiently is open challenged (see later)

29 How can we interpret visual data? Computer Vision I: Introduction 21/10/ D pixel representation Computer Graphics 3D Rich Representation, Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} Computer Vision What general (prior) knowledge of the world (not necessarily visual) can be exploit? What properties / cues from the image can be used? Both aspects are quite well understood (a lot is based on physics) but how to use them is efficiently is open challenged (see later)

30 Prior knowledge (examples) Computer Vision I: Introduction 21/10/ Hard prior knowledge Trains do not fly in the air Objects are connected in 3D Soft prior knowledge: The camera is more likely 1.70m above ground and not 0.1m. Self-similarity: all black pixels belong to the same object

31 Prior knowledge harder to describe Computer Vision I: Introduction 21/10/ Describe Image Texture Real Image zoom Not a real Image zoom Microscopic Images. What is the true shape of these objects

32 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ Which patch is brighter: A or B? [Edward Adelson]

33 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ Which patch is brighter: A or B? [Edward Adelson]

computer sees A B Ambient Light An unlikely 3D representation

likely 3D representation Direct Light True colours In 3D world

34 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ D 3D 3D 2D Image - local What the computer sees A B Ambient Light An unlikely 3D representation (hard to see for a human) A B True colors in 3D world The most likely 3D representation Direct Light True colours In 3D world This is what humans see implicitly. Ideally the computer sees the sane. A B

35 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ Light 2D Image 3D representation Humans see an image not as a set of 2D pixels. They understand an image as a projection of the 3D world we live in. Humans have the prior knowledge about the world encoded, such as: Light cast shadows Objects do not fly in the air A car is likely to move but a table is unlikely to move We have to teach the computer this prior knowledge to understand 2D images as picture of the 3D world

36 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ Which monster is bigger?

The importance of Prior knowledge Computer Vision I: Introduction 21/10/2014

37 The importance of Prior knowledge Computer Vision I: Introduction 21/10/ In the 2D Image Which monster is bigger? 1meter 2meter In the 3D world (true)

38 Two Explanations: a) People are different height and room right shape b) People are same height but room weirdly shaped Computer Vision I: Introduction 21/10/

39 Human Vision can be fooled Computer Vision I: Introduction 21/10/

40 Male or Female Computer Vision I: Introduction 21/10/

41 How can we interpret visual data? Computer Vision I: Introduction 21/10/ D pixel representation Computer Graphics 3D Rich Representation, Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} Computer Vision What general (prior) knowledge of the world (not necessarily visual) can be exploit? What properties / cues from the image can be used? Both aspects are quite well understood (a lot is based on physics) but how to use them is efficiently is open challenged (see later)

42 Cue: Appearance (Colour, Texture) for object recognition Computer Vision I: Introduction 21/10/ To what object does the patch belong to?

43 Cue: Outlines (shape) for object recognition Computer Vision I: Introduction 21/10/

44 Guess the Object Computer Vision I: Introduction 21/10/ Colour Texture Shape [from JohnWinn ICML 2008]

45 Guess the ob ject Computer Vision I: Introduction 21/10/ ? Colour Texture Shape [from JohnWinn ICML 2008]

46 Cue: Context for object recognition Computer Vision I: Introduction 21/10/

47 Cue: Context for object recognition Computer Vision I: Introduction 21/10/

48 Cue: stereo vision (2 frames) for geometry estimation Computer Vision I: Introduction 21/10/ Ground truth Algorithmic output

49 Cue: Multiple Frames for geometry estimation Computer Vision I: Introduction 21/10/

50 Cue: Convergence for geometry estimation Computer Vision I: Introduction 21/10/ vp Lines with same vanishing point may also be parallel in 3D

51 Cue: Shading & shadows for geometry and Light estimation Computer Vision I: Introduction 21/10/

52 Texture gradient for geometry estimation Computer Vision I: Introduction 21/10/

53 The Scene Parsing challenge --- a grand challenge of computer vision Computer Vision I: Introduction 21/10/ (Probabilistic) Script = {Camera, Light, Geometry, Material, Objects, Scene, Attributes, Others} Single image Many applications do not have to extract the full probabilistic script but only a subset, e.g. does the image contain a car? many examples to come later

many application scenarios are in reach Computer Vision I:

- Modern sensing technology - Moving images - User involvement 2)

(online games, mechanical turk) - Powerful graphics engines 3) For

54 many application scenarios are in reach Computer Vision I: Introduction 21/10/ To simplify the problem: 1) Richer Input: - Modern sensing technology - Moving images - User involvement 2) Rich Data to learn from: - use the web - crowdsourcing to get labels (online games, mechanical turk) - Powerful graphics engines 3) For many practical applications: We do not have to infer the full probabilistic script

55 Kinect has simplified (revolutionized) computer vision Computer Vision I: Introduction 21/10/ [Izadi et al. 11]

56 Animate the world Computer Vision I: Introduction 21/10/ [Chen et al. UIST 12]

57 New hardware design Computer Vision I: Introduction 21/10/

58 Kinect Body Pose estimation and tracking Computer Vision I: Introduction 21/10/

59 Kinect Body Pose estimation and tracking Computer Vision I: Introduction 21/10/

60 behind the scene Computer Vision I: Introduction 21/10/ Graphics simulation Synthetic (graphics) Real (hand-labelled)

21/10/2014 61 StartUp 2012: Try Fashion online

61 Body tracking and Gesture Recognition has many applications Computer Vision I: Introduction 21/10/ StartUp 2012: Try Fashion online Very large impact in many field: Gaming, Robotics, HCI, Medicine,

62 Real-time pedestrian detection Computer Vision I: Introduction 21/10/

63 Real-time Face recognition Computer Vision I: Introduction 21/10/ e.g. Canon powershot

64 General Object recognition & segmentation Computer Vision I: Introduction 21/10/ Good results [TextonBoost; Shotton et al, 06]

65 General Object recognition & segmentation Computer Vision I: Introduction 21/10/ Failure cases [TextonBoost; Shotton et al, 06]

66 Start-Up Company: Like.com Computer Vision I: Introduction 21/10/

67 Interactive Image manipulation Computer Vision I: Introduction 21/10/ [Agrawal et al 04]

68 Interactive Image manipulation Computer Vision I: Introduction 21/10/

69 Image de-convolution Computer Vision I: Introduction 21/10/ Input Output Output kernel [Schmidt, Rother, Nowozin, Jancsary, Roth 2013] Best Student Paper award

70 Image de-convolution (other domains) Computer Vision I: Introduction 21/10/ input output

71 Video Editing Computer Vision I: Introduction 21/10/ [Rav-Acha et al. 08]

72 Automatic Video Summary (StartUp: Magisto) Computer Vision I: Introduction 21/10/

73 Automatic Photo Summary - Commercial Computer Vision I: Introduction 21/10/ AutoCollage Microsoft Research [Rother et al. Siggraph 2006]

74 Movie Industry Computer Vision I: Introduction 21/10/ Pirates of the Caribbean, Industrial Light and Magic

75 Robotics Computer Vision I: Introduction 21/10/ Robocup Nasa Mars exploration

76 Introduction to Computer Vision Computer Vision I: Introduction 21/10/ What is computer Vision? (Potential) Definition: Developing computational models and algorithms to interpret digital images and visual data in order to understand the visual world we live in.

77 Interactive Segmentation Computer Vision I: Introduction 21/10/

Model versus Algorithm Computer Vision I: Introduction 21/10/2014 78 Example: Interactive Segmentation Goal z = R, G, B n x = 0,1 n (user-specified pixels are

78 Model versus Algorithm Computer Vision I: Introduction 21/10/ Example: Interactive Segmentation Goal z = R, G, B n x = 0,1 n (user-specified pixels are not optimized for) Given z; derive binary x: Model: Energy function E x (implicitly models a statistical model P(x z) ) Algorithm to minimization: x = argmin x E(x)

79 Model for a starfish Computer Vision I: Introduction 21/10/ Goal: formulate E(x) such that E x = 0.01 E x = 0.05 E x = 0.05 E x = 0.1 Optimal solution x = argmin x E(x)

80 How does the energy looks like? Computer Vision I: Introduction 21/10/ Energy function (sum of terms θ): E(x) = θ i x i + i Unary terms i,j θ ij (x i, x j ) Pairwise terms

81 How does the energy looks like? Computer Vision I: Introduction 21/10/ x j θ ij (x i, x j ) pairwise terms x i θ i (x i ) unary terms Visualization: Undirected graphical models

82 Green Unary term Green Computer Vision I: Introduction 21/10/ Red User labelled pixels Red Gaussian Mixture Model Fit

83 Unary term New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only Computer Vision I: Introduction 21/10/

Pairwise term Computer Vision I: Introduction 21/10/2014 84 θ ij x i, x j = x i x j Ising Prior When is θ ij (x i, x j ) small, i.e. likely configuration?

84 Pairwise term Computer Vision I: Introduction 21/10/ θ ij x i, x j = x i x j Ising Prior When is θ ij (x i, x j ) small, i.e. likely configuration? Most likely Most likely Intermediate likely most unlikely This models the assumption that the object is spatially coherent Next step could be: model shapes of starfishes

85 Energy minimization (optimization) Computer Vision I: Introduction 21/10/ ω = 0 ω = 10 E(x) = i ω i,j θ i x i + x i x j ω = 40 ω = 200

The key Questions What type of modelling language should be chosen: undirected or directed discrete Graphical models, Continuous-Domain models This is

vision: What is the structure Geometry, Image Processing and Tracking How do the terms look like Can we learn the Model from Data: Learn structure Learn

86 The key Questions What type of modelling language should be chosen: undirected or directed discrete Graphical models, Continuous-Domain models This is the focus of the course (SS 15): Computer Vision 2, and Machine Learning 2 How does the exact model look This like: lecture is more physics-based vision: What is the structure Geometry, Image Processing and Tracking How do the terms look like Can we learn the Model from Data: Learn structure Learn potential functions How do we optimize the model (perform inference): fast, approximate Exactly solvable? NP-hard? Computer Vision I: Introduction 21/10/

87 Another Example: Model versus Algorithm Computer Vision I: Introduction 21/10/ Input: Image sequence [Data courtesy from Oliver Woodford] Output: New view Model: Minimize a binary 4-connected pair-wise graph (choose a colour-mode at each pixel) [Fitzgibbon et al. 03]

Another Example: Model versus Algorithm Computer Vision I: Introduction 21/10/2014 88 Ground Truth Graph Cut with truncation [Rother et al.

88 Another Example: Model versus Algorithm Computer Vision I: Introduction 21/10/ Ground Truth Graph Cut with truncation [Rother et al. 05] (approximate solution) Belief Propagation (approximate solution) ICM, Simulated Annealing (approximate solution) QPBOP [Boros et al. 06; Rother et al. 07] (exact solution) Why is the result not perfect? Model or Optimization

89 Why is computer vision interesting (to you)? Computer Vision I: Introduction 21/10/ It is a challenging problem that is far from being solved It combines insights and tools from many fields and disciplines: Mathematics and statistics Cognition and perception Engineering (signal processing) And of course, computer science

90 Why is computer vision interesting (to you)? Computer Vision I: Introduction 21/10/ Allows you to apply theoretical skills... that you may otherwise only use rarely. Quite rewarding: Often visually intuitive and encouraging results. It is a growing field: Cameras are becoming more and more popular There are a lot of companies (big, small, startups) working in vision Conferences are growing rapidly.

91 Relationship to other fields Computer Vision I: Introduction 21/10/ [Wikipedia]

92 Relationship to other fields my personal view Computer Vision I: Introduction 21/10/ Human-Computer Interaction Biology Robotics Medicine (many more) Computer Vision Applications AI

93 Reading for next class Computer Vision I: Introduction 21/10/ This lecture: Chapter 1 (in particular: 1.1) Next lecture: Chapter 3 (in particular: 3.2, 3.3) - Basics of Digital Image Processing Chapter 4.2 and Edge and Line detection

Computer Vision I - Algorithms and Applications: Introduction

Computer Vision I - Algorithms and Applications: Introduction Carsten Rother 22/10/2013 Computer Vision I:Introduction Admin Stuff Computer Vision I: Introduction 22/10/2013 2 Language: German/English;