Introduction. Prof. Kyoung Mu Lee SoEECS, Seoul National University

Transcription

1 Introduction 1 Introduction to Computer Vision Introduction Prof. Kyoung Mu Lee SoEECS, Seoul National University Goal and Objectives Introduction 2 To introduce the fundamental problems of computer vision. To introduce the main concepts and techniques used to solve those. To enable participants to implement solutions for reasonably complex problems. To enable the student to make sense of the literature of computer vision.

2 Course Information Introduction 3 Classes: Mon & Wed, 10:30-11:45, Rm: Instructor: Prof. Kyoung Mu Lee kyoungmu@snu.ac.kr Tel: , Room Prerequisites: Linear algebra Probability and Statistics Image Processing (optional) A good programming skill on C, C++, MATLAB Textbook: Computer Vision: A modern approach by Forsyth & Ponce Webpage: TA: Min Su Cho, , Rm# , chominsu@gmail.com Course Information Introduction 4 Reference Books: Computer Vision, by L. G. Shapiro and G. C. Stockman, Prentice Hall, The Computer Image, by A. Watt, Addison-Wesley, Image Processing, Analysis, and Machine Vision, M. Sonka, V. Hlavac and R. Boyle, Brooks/Cole Publishing, Introductory Techniques for 3-D Computer Vision, E. Trucco and A. Verri, Prentice-Hall,1998. Machine Vision, R. Jain, R. Kasturi, and B. G. Schunck, MacGraw-Hill, A Guided Tour of Computer Vision, by V. S. Nalwa, Addison-Wesley, 1993.

3 Grading Introduction 5 Program Assignments & Quiz 50% Final Project 30% Presentation 10% Participation 10% All the Assignments and Project must be done individually What is Computer Vision? Introduction 6 Forsyth and Ponce Extracting descriptions of the world from pictures or sequences of pictures Ballard and Brown The construction of explicit, meaningful description of physical objects from images Trucco and Verri Computing properties of the 3D world from one or more digital images

4 Why study Computer Vision? Introduction 7 An image is worth 1000 words Images and movies are everywhere Fast-growing collection of useful applications building representations of the 3D world from pictures automated surveillance (who s doing what) movie post-processing face detection & recognition Intelligent robots Various deep and attractive scientific mysteries how does object recognition work? Many biological systems rely on vision Greater understanding of human vision Camera and computer are cheap What Computer Vision Focuses? Introduction 8 What information should be extracted? How can it be extracted? How should it be modeled or represented? How can it be computed to achieve the goal?

5 Applications Introduction 9 Industrial inspection, quality control Surveillance and security Assisted living Human computer interfaces Intelligent robot Medical image analysis Virtual/Augmented reality Properties of Vision Introduction 10 One can see the future Cricketers avoid being hit in the head qthere s a reflex --- when the right eye sees something going left, and the left eye sees something going right, move your head fast. Gannets pull their wings back at the last moment qgannets are diving birds; they must steer with their wings, but wings break unless pulled back at the moment of contact. qarea of target over rate of change of area gives time to contact.

6 Properties of Vision Introduction 11 3D representations are easily constructed There are many different cues. Useful qto humans (avoid bumping into things; planning a grasp; etc.) qin computer vision (build models for movies). Cues include qmultiple views (motion, stereopsis) qtexture qshading qfocus/defocus qsilhouette qcalled Shape form X problem Properties of Vision Introduction 12 People draw distinctions between what is seen Object recognition This could mean is this a fish or a bicycle? It could mean is this George Washington? It could mean is this poisonous or not? It could mean is this slippery or not? It could mean will this support my weight? Great mystery qhow to build programs that can draw useful distinctions based on image properties.

7 Perception Introduction 13 Vision is inferential: light & shadow ( ) Perception Introduction 14 Vision is inferential: Prior knowledge

8 Perception Introduction 15 Vision is inferential: expectation Perception Depth by motion parallax Introduction 16 Ambiguity by shadow IllusionWorks ( ball.h tml )

9 Main topics Introduction 17 Camera & Light Geometry, Radiometry, Color Digital images Filters, edges, texture, optical flow Shape (and motion) recovery What is the 3D shape of what I see? Multi-view geometry Stereo, motion, photometric stereo, Segmentation What belongs together? Clustering, model fitting, probablistic Tracking Where does something go? Linear dynamics, non-linear dynamics Recognition What is it that I see? templates, relations between templates Part I: The Physics of Imaging Introduction 18 How images are formed Cameras qwhat a camera does qhow to tell where the camera was Light qhow to measure light qwhat light does at surfaces qhow the brightness values we see in cameras are determined Color qthe underlying mechanisms of color qhow to describe it and measure it

10 Part II: Early Vision in One Image Introduction 19 Representing small patches of image For three reasons qwe wish to establish correspondence between (say) points in different images, so we need to describe the neighborhood of the points qsharp changes are important in practice --- known as edges qrepresenting texture by giving some statistics of the different kinds of small patch present in the texture. Tigers have lots of bars, few spots Leopards are the other way Representing an image patch Introduction 20 Filter outputs essentially form a dot-product between a pattern and an image, while shifting the pattern across the image strong response -> image locally looks like the pattern e.g. derivatives measured by filtering with a kernel that looks like a big derivative (bright bar next to dark bar)

11 Introduction 21 Convolve this image To get this With this kernel Texture Introduction 22 Many objects are distinguished by their texture Tigers, cheetahs, grass, trees We represent texture with statistics of filter outputs For tigers, bar filters at a coarse scale respond strongly For cheetahs, spots at the same scale For grass, long narrow bars For the leaves of trees, extended spots Objects with different textures can be segmented The variation in textures is a cue to shape

12 Introduction 23 Introduction 24

13 Part III: Early Vision in Multiple Images Introduction 25 The geometry of multiple views Where could it appear in camera 2 (3, etc.) given it was here in 1 (1 and 2, etc.)? Stereopsis What we know about the world from having 2 eyes Structure from motion What we know about the world from having many eyes qor, more commonly, our eyes moving. Shape from X Introduction 26 Many different approaches/cues Shape from stereo Shape from shading Shape from texture Shape from focus/defocus Shape from motion Shape from silhouette Etc..

14 Stereo Introduction 27 ( Real-Time Stereo Introduction 28 (Bumblebee, Point Grey Research Inc.) (STH-MDCS2, Videre Design) (Digiclops, Point Grey Research Inc.)

15 Real-Time Stereo Introduction 29 ( Structure from Motion Introduction 30

16 Photometric stereo + structured light Introduction 31 IBM s pieta project 3D Laser Scanning Introduction 32 The Digital Michelangelo Project ( ) 2 BILLION polygons, accuracy to.29mm

17 Part IV: Mid-Level Vision Introduction 33 Finding coherent structure so as to break the image or movie into big units Segmentation: qbreaking images and videos into useful pieces q E.g. finding video sequences that correspond to one shot qe.g. finding image components that are coherent in internal appearance Tracking: qkeeping track of a moving object through a long sequence of views Segmentation Introduction 34 Which image components belong together? Belong together = lie on the same object Cues similar colour similar texture not separated by contour form a suggestive shape when assembled

18 Introduction 35 Introduction 36 LOCUS (John Winn et. al. 2006)

19 Introduction 37 Introduction 38

20 Tracking Introduction 39 Use a model to predict next position and refine using next image Model: simple dynamic models (second order dynamics) kinematic models etc. Face tracking and eye tracking now work rather well Optical Flow Introduction 40 Where do pixels move? Translation Rotation Scaling

21 Tracking Introduction 41 Isard&Blake ECCV 96 (Condensation) More tracking examples Introduction 42

22 Part V: High Level Vision (Geometry) Introduction 43 The relations between object geometry and image geometry Model based vision qfind the position and orientation of known objects Smooth surfaces and outlines qhow the outline of a curved object is formed, and what it looks like Aspect graphs qhow the outline of a curved object moves around as you view it from different directions Range data Part VI: High Level Vision (Probabilistic) Introduction 44 Using classifiers and probability to recognize objects Templates and classifiers qhow to find objects that look the same from view to view with a classifier Relations qbreak up objects into big, simple parts, find the parts with a classifier, and then reason about the relationships between the parts to find the object. Geometric templates from spatial relations qextend this trick so that templates are formed from relations between much smaller parts

23 Part VII: Some Applications in Detail Introduction 45 Finding images in large collections searching for pictures browsing collections of pictures Image based rendering often very difficult to produce models that look like real objects qsurface weathering, etc., create details that are hard to model qsolution: make new pictures from old What are the problems in recognition? Introduction 46 Which bits of image should be recognized together? Segmentation. How can objects be recognized without focusing on detail? Abstraction. How can objects with many free parameters be recognized? No popular name, but it s a crucial problem anyhow. How do we structure very large model bases? again, no popular name; abstraction and learning come into this

24 Appearance-based recognition Introduction 47 Parametric eigenspace (Nayar et al. 96) Doesn t work in cluttered scenes Feature-based recognition Introduction 48 SIFT-based Matching, Lowe (UBC) SARG ( Stochastic Attributed Relational Graph ) Representation (SNU)

25 Recognition & segmentation Co-Recognition (SNU) Matching templates Introduction 50

26 Relations between templates Introduction 51 e.g. find faces by finding eyes, nose, mouth finding assembly of the three that has the right relations om/facereco.html (Xie and Comaniciu 03) 3D Object Recognition Introduction 52 (

27 Readings for Next Class Introduction 53 Ch 1.1 Cameras and Projection Ch 4. light