Introduction to Computer Vision MARCH 2018

Transcription

1 Introduction to Computer Vision RODNEY DOCKTER, PH.D. MARCH

2 Rodney Dockter (me) Ph.D. in Mechanical Engineering from the University of Minnesota Worked in Dr. Tim Kowalewski s lab Medical robotics and machine learning Now working at Danfoss Power Solutions developing autonomous systems in agriculture and heavy machinery 2

3 Contact Information Office: ME Office Hours: W, F 1:15pm 2:00 pm 3

4 Book References (optional) David A. Forsyth and Jean Ponce: Computer vision: a modern approach. Prentice Hall, Peter Corke: Robotics, Vision and Control: Fundamental Algorithms In MATLAB. Springer, 2017 R.C. Gonzalez and R.E. Woods: Digital Image Processing, Prentice-Hall, 2002 Richard Hartley and Andrew Zisserman: Multiple View Geometry in Computer Vision. Cambridge 2004 Christopher M. Bishop: Pattern Recognition and Machine Learning. Springer,

5 Class Organization All computer vision from here on out! ~10 lectures, 3 assignments, 30% of total grade Prerequisites : Linear algebra Statistics Vector Calculus Coding! Course does not presume prior computer vision experience Emphasis on coding! Matlab will be required for all homework assignments 5

6 Class Organization Cont. Collaboration Policy: - You are encouraged to discuss assignments with your peers. However, all written work and coding must be done individually. Individuals found submitting duplicate or substantially similar materials due to inappropriate collaboration may get an F in this class and may receive other serious consequences. - In the real world no one will write your code for you! Libraries and Code Found Online: - None of the Matlab libraries for computer vision may be used in assignments. You will writing your own computer vision code from the ground up. - Students found plagiarizing code found on the internet will fail the course (we know how to use google too) 6

7 Course Syllabus (Vision Portion) March 30 F 1 - Introduction to computer vision and image processing April 04 W 2 - Image Formation Camera Fundamentals April 06 F 3 - Digital Image Representation (Quiz #2 review) April 11 W 4 - Spatial Domain + Computer Vision in Matlab April 13 F QUIZ #2 No Lecture April 18 W 5 - Image Histograms April 20 F 6 - Edge Detection April 25 W 7 - Edge Detection Cont. Canny and Sobel April 27 F 8 - Interest Point Detection May 02 W 9 - Line Detection. Hough Transform. May 04 F 10 - The future. Deep Learning. 7

8 Outline What is computer vision? What is image processing? Basics of Computer Vision Processing Interesting examples of computer vision in the wild? Brief introduction to Matlab for computer vision use. 8

9 Part 1 - What is computer vision? Allowing robots and machines to see the world around them. 9

10 What is computer vision? Once robots can see the world, they can interact with it. 10

11 What is computer vision? Machines can learn to perform tasks just like humans. 11

12 What is computer vision? Deals with the formation, analyses and interpretation of images Integral to robotics and Artificial Intelligence (AI) Interdisciplinary subject area: Robotics Autonomous Vehicles Medical Applications Security Practical and Useful Challenging and continuously evolving 12

13 Difficulties in Computer Vision Images are ambiguous: dependent on perspective Images are affected by many factors: Sensor model Illumination (lighting) Shape of object(s) Color of object (s) Texture of object (s) No universal solution to vision and sensing Many theories and potential algorithms Gold standard: Human Vision! 13

14 Human Vision: 14

15 Difficulties in Computer Vision It is a many-to-mapping: Different objects with different material and geometric properties, possibly under different lighting conditions could lead to identical images. The same object, viewed from different perspectives or with different lighting can result in vastly different images (under constrained mapping) Information is lost in the transformation from the 3D world to a 2D image. 15

16 Difficulties in Computer Vision It is computationally intensive. Usually requires high end processors or graphics cards to achieve real time performance. We still do not fully understand the recognition problem. 16

17 Nomenclature Many names for mostly similar things: Computer Vision General All encompassing term Computational Vision Modeling of biological vision Image Processing Generally refers to static images (frame to frame). Building block for computer vision. Machine Vision Industrial or factory applications for inspection and measurement. 17

18 Related Fields Pattern Recognition Machine Learning Robotics (obviously) Medical Imaging Computer Graphics Human Robot Interaction 18

19 Why study computer vision Images and videos are everywhere Mobile phones, cheap cameras, real time streaming New applications all the time Face recognition 3D representations from pictures Automatic surveillance Driverless vehicles Shipping and warehouse management Various deep scientific mysteries Human vision system etc 19

20 Brief History of Computer Vision B.C. (Before Computers) Philosophy Optics Physics Neurophysiology Early Cameras (Late 1800s) Earliest way to represent the real world in a consistent manner Precursor to video and digital representations of the world around us 20

21 Brief History of Computer Vision Early Computer Vision Systems (1960 s) Minsky at MIT Attempted to solve vision in a summer (lol) Empirical approaches Ad hoc, bag of tricks Image processing plus+ Solutions tailored to each problem. Simplified worlds Blocks worlds then generalize Known perspectives, known illumination 21

22 Brief History of Computer Vision 1980 s improved computational power + robotics innovations 1990 s personal computing brings computer vision possibilities to researchers everywhere s OpenCV, ROS, and Matlab provide generalized computer vision computing abilities to researchers 2010 s Computer vision comes to the masses: Microsoft Kinect Leap Motion Google Tango Augmented Reality iphone Face ID 22

23 Progress in Computer Vision First Generation: Military/ Early Research Few systems, custom built. Cost: $1 Million+ Users have PhDs Slow (1 hour per frame) Second Generation: Industrial / Medical Numerous systems (1000+). Cost: $10,000+ Users have bachelor degrees Specialized hardware Third Generation: Consumer systems worldwide. Cost: $100 Users have little or no training Raspberry + webcam = $50 = limitless potential 23

24 The Future Software: 1 solution for all vision problems Provide unexperienced users with an easy way to detect/track objects. Does it exist? Convolutional Neural Networks? Tensorflow? Hardware: Cheap cameras (cell phone) Depth cameras (Kinect, stereo cameras) Cheap, powerful, mobile processing (getting there) 24

25 Part 2 Computer Vision Processing Processing Levels: Low Level Mid Level High Level Image Formulation 25

26 Three Processing Levels Low Level Low Level Processing: Standard procedures to improve image quality and format No intelligence 26

27 Three Processing Levels - Mid Level Intermediate Processing: Extract features and characterize components Some intelligence 27

28 Three Processing Levels - Mid Level Representing small patches in an image Want to establish correspondence between points or regions in different images, so we need to mathematically describe the neighborhood around a point Sharp changes are important features! (Known as edges ) Representing texture by giving some statistics of the different types of patches present in a region i.e. Tigers have lots of stripes, few spots Leapords have lots of spots, no stripes How can we mathematically describe this? 28

29 Three Processing Levels - Mid Level Filter Outputs Filters form a dot-product between a pattern and an image, while shifting the pattern across the image Strong Response -> image region matches a pattern e.g. derivatives measured by filtering with a kernel that looks like a big derivative (bright line next to a dark line would yield a large derivative) 29

30 Three Processing Levels - Mid Level Many objects are distinguished by their texture Tigers, cheetahs, grass, trees We represent texture with statistics from filter outputs For tigers, a stripes filter at a coarse scale responds strongly For cheetahs, a spot filter with a coarse scale responds strongly For grass, long narrow lines For the leaves of a trees, extended spots Objects with different textures can be segmented The variation in textures in a cue to a shape 30

31 Three Processing Levels - Mid Level Geometry of multiple views (images) Where might an object appear in camera #2 (or #3, ) given its position in camera 1 Stereopsis What we know about the world from having 2 eyes (cameras) Depth and 3D information Structure from motion What we know about the world from having many eyes (a single eye, moving quickly) 31

32 Three Processing Levels - Mid Level Finding coherent structures in an image to break it into smaller units Segmentation: Breaking images and videos into useful pieces e.g. finding image components that are coherent in appearance e.g. separating image foreground from background Tracking: Keeping track of a moving object through a sequence of views 32

33 Three Processing Levels High Level High-Level Processing: Recognizing patterns, comparing to known models High intelligence (ie machine learning) 33

34 Three Processing Levels - High Level Relationship between object geometry and image geometry Model Based Vision Find the position and orientation of known objects (e.g. squares, circles) Smooth surfaces and outlines How the outline of a curved object is formed and what it looks like (contours) Aspect graphs: How the outline of a curved object changes as you view it from different directions Range data 34

35 Three Processing Levels - High Level Using classifiers and probability to match and recognize objects Templates and classifiers How to find objects that look the same from view to view with a classifier Relations Break up objects into big, simple parts, find the parts with a classifier, then identify relationships between parts to find the object Geometric templates from spatial relations Extend this trick so that templates are formed from relations between smaller parts Bordering on machine learning 35

36 Image Formulation 36

37 Image Formulation 37

38 Simplistic View 38

39 The Physics of Imaging How images are formed: Cameras How a camera creates an image How to tell where the camera was Light How to measure light What light does at surfaces How the brightness values we see in cameras are determined Color The underlying mechanisms of color How to describe and measure color 39

40 The Physics of Imaging 40

41 The Physics of Imaging More on this later 41

42 Part 4 Computer Vision in the wild There are countless products that have an element of computer vision, and the number increases daily. Consumer, defense, and research. 42

43 Computer Vision in the wild NASA Rover PAR systems: Vision Guided Robots 43

44 Computer Vision in the wild Face Recognition For Security, User Interface Apple Face ID 44

45 Computer Vision in the wild Challenges in Face Recognition: -Illumination -Pose Variations -Facial Expressions -Facial Similarity 45

46 Computer Vision in the wild 3D Sensors, Microsoft, Intel, etc Time of Flight Depth Gesture Recognition Leap Motion, Realsense 46

47 Computer Vision in the wild Optical Character Recognition (Hand Written Digits) MNIST Data Set 47

48 Computer Vision in the wild Traffic Monitoring Augmented Driver Assist Systems 48

49 Computer Vision in the wild Autonomous Vehicles 49

50 Computer Vision in the wild Robotics Navigation 50

51 Computer Vision in the wild Brain Imaging MRI Vital Images Inc. 51

52 Computer Vision in the wild Types of images: Infra-red Ultra-violet Radio-waves Visible-light Radar Tomography Sonar Microscopy Magnetic Resonance 52

53 Computer Vision in the wild Thermal Imaging: Applications in defense, agriculture Local company: Fluke Thermography 53

54 Computer Vision in the wild Human Activity Recognition 54

55 Computer Vision in the wild Video Surveillance and Tracking 55

56 Can computers match human perception? Not yet Computer vision is still no match for human perception But computers are catching up in certain areas Classifying the ImageNet Database: 1.2 million images, 1000 categories Human: ~5.1% Error Rate Inception-v3 Convolutional Neural Network: 3.46% Error Rate 56

57 Conclusion Computer Vision is a challenging and exciting field Applied to many real world situations Tremendous progress in the last two decades There is still work to be done 57