Web site. Introduction to Computer Vision. Computer Vision. Text Book. Computer Vision. Relation to other fields

Transcription

1 Introduction to Computer Vision CS / ECE 181B Tuesday, March 30, 2004 Web site Prof. B. S. Manjunath ECE/CS Department Last Year (Prof. Turk) Has some useful information. Course introduction and overview [thanks to Professor Matthew Turk, CS department for the slides from S2003] 2 Text Book NO Text Book Follow the classroom discussions Handouts and Lecture slides Reference Books (not required) Forsyth and Ponce, Computer Vision: A Modern Approach (was the text used last year) Nalwa, A Guided Tour of Computer Vision ; a good introductory book for casual reading Jain and Kasturi, Machine Vision ; I used it in Horn, Robot Vision. A classic, but outdated. Computer Vision What is computer vision? Making computers see Nice sunset! 3 Extracting descriptions of the world from pictures or sequences of pictures 4 Computer Vision Also known as image understanding, machine vision, computational vision Relation to other fields Image processing AI, robotics CV is about interpreting the content of images* Field is years old Child of AI Vision is easy, right? Just open your eyes! No, it s a hard problem! Much of your very complex brain is devoted to doing vision It involves cognition, navigation, manipulation and learning Not just simple match a feature vector to a database tasks 5 Geometry Probability and statistics Computer graphics Optics Perception Pattern recognition Machine learning 6

2 Computer Graphics Computer Vision Projection, shading, lighting models Output: Output: Image Synthetic Camera Model Model Real Scene 7 Cameras Images 8 Aims of Computer Vision Automate visual perception Construct scene descriptions from images Make useful decisions about real physical objects and scenes based on sensed images Produce symbolic (perhaps task-dependent) descriptions from images Produce from images of the external world a useful description that is not cluttered with irrelevant information Support tasks that require visual information Some applications of computer vision Photogrammetry, GIS Commercial, military, government Robotics Industrial, military, medical, space, entertainment Inspection, measurement Medical imaging Automatic detection, outlining, measurement Graphics and animation, special effects Surveillance and security Multimedia database indexing and retrieval, compression Human-computer interaction (VBI) 9 10 What Does Computer Vision Do? 3D models m of objects Object recognition Navigation Event/action recognition 11 Vision Transforms From This A D 1D E 0C 0A 0A 0A B A 08 0B 0A 0D 0B 0B 0C F 0B E 0C A 0A 0B 0F 0A 0C B 07 0B 05 0B E 0C A A A C 0B 0A A 0C 0A 0A 08 0A 0A A B 14 0F 0F 0D 0A 0E 0A 0C 0C 0E 0A 0C 0B 09 0A 09 0A 0A 09 0B 0B 05 0C 0C 0A B 10 0A 0D 0D 0B 0D 0C 0B 0B 0C 0D 0B 0B 0A 0A 0A 0B 0C C 15 0D E C 0D 0C 0C 0A 0B 0B 09 0C 0F D 07 0B D D 0A F 0E B 0C 0F 0F 0E 0C 10 0D E 3F A 09 0A C B 10 0F 0C D 0F 0D 0D 0B 25 7A 7F 79 6D 80 6E 54 0C 0D 09 0A F 0D 12 0E 10 0E 0F F E 8C 73 7A 5C 1E 05 0A 0F 0E 0C F 15 0D D B A 12 0A 0B E 1A F 7D C 7F 6F 5F 0B D 0C F 16 0F D E 21 1F 1C 1D 2D 1A 2D 7C 7A 95 6B C 0A C C A 0D 1A 1A B 3E C 83 5E 7B 94 8A 5A 3D C F 0C A F 1F F 93 B4 AE D F F B F 3D 3E 42 3B 45 2E F 96 6B 24 0F 22 4B C3 A4 3F 4F 0C F C D D A 27 3B 33 A8 A B A1 75 4B AC A1 B5 79 0C 0B 13 0F 0B B 1D 1C 1C 1C 1B 1B 1E A 24 9F AD AC AA B1 9C 8D 5F 3E 98 B7 B7 A A 0D D E 36 5B 29 2C F AF BC AF AB 9E A F AE AD A E 0A 0C B B 1A 1A 2B 1B 2A C 1B 26 4C 40 BA BB B5 AE A 8A 9A B9 BB AD 9C 8A B 0D 0F 0B A 18 1C 1E D 3F 4E B 1B AF AB B1 AC A AA 9F AD 7F 0C 0B 0E 0B 0C 0C A 21 1E 2E 1B E B AA AC B7 AF A6 9A 93 8F 85 7F A0 A4 C2 9F 99 4E A 0D 0C 0A 0C C 1F 2B 1C 8B 9B 42 9B A7 A1 B4 B0 AA A0 9D E A A 0A 0D 0D 09 0D 0C A 1C A C C 90 A1 39 A0 97 B8 AA B2 A5 A6 A D 08 0D C 0B 0E 0D 0D 0A 04 1E 29 1F E 41 4A 2C A A5 89 9E A3 B0 B7 AF AB AB A C C 0D 0B 01 1F A 2C 36 4D B AA 7E AD B3 AA B2 A8 B E 9E 8E A 0D 0D 0D 0F 0C E C 2A A 39 4F B A5 86 AA B2 B3 AE A0 A3 9C B 25 2D 07 0E C 0A 0F 0D 09 0C F 2A C 93 A3 AC 60 BA BD B4 AE A8 A F 52 4F 3F 09 0D 0D 09 0E 0E 0B 12 0B 0B E 2C 29 2A 3B 30 4E 3C E AE 9F A4 B1 4E AA AA A0 A4 9C 94 A2 AB A E 0E 09 0B 0D 10 0C 0C E D 5A B 8C A3 AC A5 3E A1 AF A8 82 A4 AC A B 0B 0B 0E 0F A E 3C 3F 3E C 5C A A2 A2 A6 8E 4E AC A6 A2 89 7E 5B 11 0E D 0C 0D 0C 00 4B C 2E 2D B 0D C A8 B5 AA B3 AE A0 9C 8C 62 0A D E 0D C 24 2E B B 2B A1 AB AC B2 A6 A6 A F 10 1C F 0C 0F A C F 33 1F B 4D A3 A5 99 8D 7A 4E 0E 1B F 0F B 01 1D 1F 2B F 40 2F 2D 2A 25 2B 2C E 55 5E 62 6D 6D 6E 68 5E 43 0D A F 2A B 45 2E 3A D 2F 1F 1E 1B C 3F 3C D 0B 0E 11 1E 23 1B D 10 0F 12 0F A 1B 35 4A 1D 20 2C 2F 1F 1F 3B 34 1A 2A E 0C 0C 06 0C B B 0E 10 0D 0D 0F D 1E C E 0E 1A A 18 2C 2E 19 0F 0D 10 0E 0E 14 0D B 1E 17 1C 1F 1F 1F 1C C 2F B D D A 1D 1F 1D 1B 1E 1B B 22 1A 1E 1B C 0D 11 0E 12 0D C 2E E 20 1F 1F 1D 1B 1C 29 CS/ECE b B 1E 15 1A A E 1D B 1B A 4C 33 1C E A E

3 To this Objects Cat, chair, window, star, bush, water, a shoe, my mother Properties Big, bright, yellow, fast, graspable, moving Relations In front, behind, on top, next to, larger, closer, identical Shapes Round, rectangular, star-shaped, symmetric Textures Rough, smooth, irregular Movement Turning, looming, rolling 13 A harbor with many dozens of boats; water is calm and glassy; masts are all vertical; mountains in background, blue sky with a touch of clouds 14 J Hallway straight ahead Why is Vision Difficult? Angry Surprised Happy Disgusted Consider the input... Not this 17 18

4 But this A D 1D E 0C 0A 0A 0A B A 08 0B 0A 0D 0B 0B 0C F 0B E 0C A 0A 0B 0F 0A 0C B 07 0B 05 0B E 0C A A A C 0B 0A A 0C 0A 0A 08 0A 0A A B 14 0F 0F 0D 0A 0E 0A 0C 0C 0E 0A 0C 0B 09 0A 09 0A 0A 09 0B 0B 05 0C 0C 0A B 10 0A 0D 0D 0B 0D 0C 0B 0B 0C 0D 0B 0B 0A 0A 0A 0B 0C C 15 0D E C 0D 0C 0C 0A 0B 0B 09 0C 0F D 07 0B D D 0A F 0E B 0C 0F 0F 0E 0C 10 0D E 3F A 09 0A C B 10 0F 0C D 0F 0D 0D 0B 25 7A 7F 79 6D 80 6E 54 0C 0D 09 0A F 0D 12 0E 10 0E 0F F E 8C 73 7A 5C 1E 05 0A 0F 0E 0C F 15 0D D B A 12 0A 0B E 1A F 7D C 7F 6F 5F 0B D 0C F 16 0F D E 21 1F 1C 1D 2D 1A 2D 7C 7A 95 6B C 0A C C A 0D 1A 1A B 3E C 83 5E 7B 94 8A 5A 3D C F 0C A F 1F F 93 B4 AE D F F B F 3D 3E 42 3B 45 2E F 96 6B 24 0F 22 4B C3 A4 3F 4F 0C F C D D A 27 3B 33 A8 A B A1 75 4B AC A1 B5 79 0C 0B 13 0F 0B B 1D 1C 1C 1C 1B 1B 1E A 24 9F AD AC AA B1 9C 8D 5F 3E 98 B7 B7 A A 0D D E 36 5B 29 2C F AF BC AF AB 9E A F AE AD A E 0A 0C B B 1A 1A 2B 1B 2A C 1B 26 4C 40 BA BB B5 AE A 8A 9A B9 BB AD 9C 8A B 0D 0F 0B A 18 1C 1E D 3F 4E B 1B AF AB B1 AC A AA 9F AD 7F 0C 0B 0E 0B 0C 0C A 21 1E 2E 1B E B AA AC B7 AF A6 9A 93 8F 85 7F A0 A4 C2 9F 99 4E A 0D 0C 0A 0C C 1F 2B 1C 8B 9B 42 9B A7 A1 B4 B0 AA A0 9D E A A 0A 0D 0D 09 0D 0C A 1C A C C 90 A1 39 A0 97 B8 AA B2 A5 A6 A D 08 0D C 0B 0E 0D 0D 0A 04 1E 29 1F E 41 4A 2C A A5 89 9E A3 B0 B7 AF AB AB A C C 0D 0B 01 1F A 2C 36 4D B AA 7E AD B3 AA B2 A8 B E 9E 8E A 0D 0D 0D 0F 0C E C 2A A 39 4F B A5 86 AA B2 B3 AE A0 A3 9C B 25 2D 07 0E C 0A 0F 0D 09 0C F 2A C 93 A3 AC 60 BA BD B4 AE A8 A F 52 4F 3F 09 0D 0D 09 0E 0E 0B 12 0B 0B E 2C 29 2A 3B 30 4E 3C E AE 9F A4 B1 4E AA AA A0 A4 9C 94 A2 AB A E 0E 09 0B 0D 10 0C 0C E D 5A B 8C A3 AC A5 3E A1 AF A8 82 A4 AC A B 0B 0B 0E 0F A E 3C 3F 3E C 5C A A2 A2 A6 8E 4E AC A6 A2 89 7E 5B 11 0E D 0C 0D 0C 00 4B C 2E 2D B 0D C A8 B5 AA B3 AE A0 9C 8C 62 0A D E 0D C 24 2E B B 2B A1 AB AC B2 A6 A6 A F 10 1C F 0C 0F A C F 33 1F B 4D A3 A5 99 8D 7A 4E 0E 1B F 0F B 01 1D 1F 2B F 40 2F 2D 2A 25 2B 2C E 55 5E 62 6D 6D 6E 68 5E 43 0D A F 2A B 45 2E 3A D 2F 1F 1E 1B C 3F 3C D 0B 0E 11 1E 23 1B D 10 0F 12 0F A 1B 35 4A 1D 20 2C 2F 1F 1F 3B 34 1A 2A E 0C 0C 06 0C B B 0E 10 0D 0D 0F D 1E C E 0E 1A A 18 2C 2E 19 0F 0D 10 0E 0E 14 0D B 1E 17 1C 1F 1F 1F 1C C 2F B D D A 1D 1F 1D 1B 1E 1B B 22 1A 1E 1B C 0D 11 0E 12 0D C 2E E 20 1F 1F 1D 1B 1C B 1E 15 1A A E 1D B 1B A 4C 33 1C E A E E 2D 2D 23 1D CS/ECE 2A 5F b A 1B B F B E 1B C 5E F F 21 1B F A 1D B0 2C C 22 1D 1A 10 1A 1D 1A C 21 1B Some Possible Outputs depth object pose object or (facing away, action recognition understanding segmentation facing forward) 20 What Your Brain Does Almost certain to be Bill Clinton Gray hair Dark circular overlay Right ear Neck White shirt Armani suit Clinton greeting Lewinsky Person with glasses in crowd Pony tail Dark brown hair Right eye (open) Left eye (open) Nose Cheek Monica s mouth (smiling) Person Lapel Illuminated contour from above CNN caption Necklace (Washington 1995?) Clinton occluding Monica Lewinsky Woman s dress suit Monica 21 Variation in Appearance 22 What makes computer vision hard? Underspecified problem! 3D world projected onto 2D sensor(s) Environment Lighting, background, movement, camera Varying appearance of objects Calibration, FOV, camera control, image quality Computational complexity (speed of processing) Etc., etc. Progress in Computer Vision First generation: Military/Early Research Few systems, each custom-built, cost $Ms Users have PhDs 1 hour per frame Second generation: Industrial/Medical Numerous systems, of each, cost $10Ks Users have college degree RT with special hardware Third generation: Consumer (00) systems, cost $100s Users have little or no training RT in software 23 24

5 Examples Computer Vision Areas of study MIT Media Lab CMU NavLab Cognex Camera (sensor) calibration and image formation Binary image analysis Edge detection and low-level filtering Color representation and segmentation Texture description and segmentation Depth/shape from X Stereopsis Optical flow Motion computation Object matching, detection, and recognition 3D sensing and shape description Object and scene tracking 25 Camera calibration Edge detection Original planar pattern Corrected for radial lens distortion Two calibrated cameras 27 Stereopsis Multi-Camera Stereo Images 3D reconstruction 29 30

6 Motion computation Object Detection and Recognition Model, detect, and recognize objects in images Optical flow Main issues Object representation, feature extraction, matching, learning from data Cars Faces 31 Texture analysis 32 Some Computer Vision Examples Academic and industrial research Companies and products 33 Early CV research 34 Multimedia database applications 35 36

7 Labeling images Vision for HCI 37 Vision and computer graphics 38 Vision for Biometrics Viisage Eyematic Interfaces Example application: Face recognition Examples of Commercial Vision Systems If we could reliably detect and recognize faces in images Digital libraries Find me the picture of my brother and Ted Show all the scenes with more than four people Surveillance Alert the operator when a known suspect enters the area HCI Automatically log in as I walk up to the computer Refuse to open my files when someone else is at my machine Military Friend or foe? Personal/entertainment robots Follow the master, look the customer in the eyes Toshiba Hand Motion Processor Point Gray Research Eyematic Interfaces 41 42

8 So why study computer vision? p Fusion of several old and new research fields Engineering reasons Images are become more and more ubiquitous Plenty of useful applications Great mixture of applied mathematics, signal processing, computer science, etc. Solving interesting, underconstrained ( ill-posed ) problems Etc. Psychology Neurology Physiology Signal/ image analysis Computer Vision Artificial intelligence Science reasons How does the brain do it? Understanding biological vision Deep computational issues Etc. Robotics Pattern Recognition Graphics ComputerVision &Image Analysis 43 Psychology, Neurology, and Physiology Human vision not perfect.. Understand human (biological) perception Check out the illusions and ambiguities.. 44 YF Wang Study the structure of the eye, and the neural-networks for visual information processing Design computational algorithms to evaluate biological visual perception functions

9 49 50 This Quarter Vision is unlike any other course you may have taken in engineering/science. There are MANY questions, very few answers! Remember this as we go through this subject. We are so familiar with seeing, that it takes a leap of imagination to realize that there are problems to be solved ---- Richard Gregory Vision is a bag of tricks --- Ramachandran 51 (tentative) Grading Project Homeworks & Programming Assignments (25%) Topic: Face recognition Midterm (15%) More details soon. Project on Face Recognition (20%) Individual or groups of no more than 4 students. 52 Final Examination (40%) 53 54

10 What will we study? Part I: The physics of imaging Camera models and calibration Radiometry Color Part II: Early vision Filtering and edge detection Stereo, optical flow Part III: Mid-level vision Segmentation Classification Tracking Part IV: High-level vision and applications Model-based vision Various applications of computer vision 55 Course web site this is last year s web site that you might find very useful. Visit it often! 56