Computer Vision. Lecture # 1 Introduction & Fundamentals

Transcription

1 Computer Vision Lecture # 1 Introduction & Fundamentals

2 Introduction Area of research: Analysis of medical images/signals using Image/signal processing and Machine Learning Techniques Current Research Areas: Biomedical Image/Signal Analysis (Retina, Cardiac, Dental, EEG, Breath Sounds etc) Biometrics (FP, Dental, Retina, Dorsal hand veins etc) * *

3 Text Book & References: David A. Forsyth and Jean Ponce, Computer Vision A Modern Approach, 22 Ed (available from local market) Class slides & selected research papers to be distributed by the instructor Mubarak Shah, Fundamentals of Computer Vision, 1997 (soft copy available online) Linda Shapiro and George Stockman, Computer Vision, 2 (soft copy available online) Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, 3rd Edition, 29 (available from local market)

4 Course Material Course Information Lectures slides, assignments (computer/written), solutions to problems, projects, and announcements will be uploaded on course web page.

5 Course Contents Introduction Camera Segmentation Image Registration Geometry and Transformations Camera Model and Parameters Camera Calibrations Multiview Geometry and Stereopsis K means algorithm Mean Shift Algorithm Background subtraction Line fitting by RANSAC Graph cut Graph Theory Dynamic Programming Coherent Tensors Hyperspectral Images MAC filters Template Matching Hausdroff Distance Texture Analysis Tracking Classificat ion Gabor Filters Wavelets Oriented pyramids (Gaussian and Laplacians) Spot and Bar filters Law Texture energy Synthesis Local Binary Patterns KLT Optical flow Motion vectors Kalman Filters MeanShift Markov Models for Compute Vision Deep Learning

6 Prerequisites Linear algebra, basic calculus, and probability Experience with image processing or Matlab will help but is not necessary

7 CODE OF ETHICS All students must come to class on time (Attendance will be taken in first 5 to 1 mins) Students should remain attentive during class and avoid use of Mobile phone, Laptops or any gadgets Obedience to all laws, discipline code, rules and community norms Respect peers, faculty and staff through actions and speech Student should not be sleeping during class Bring writing material and books Class participation is encouraged

8 Policies No extensions in assignment deadlines. Quizzes will be unannounced. Exams will be closed book. Never cheat. Better fail NOW or else will fail somewhere LATER in life Plagiarism will also have strict penalties. Courtesy Dr. Khawar Adapted from What is Plagiarism PowerPoint

9 Grading Policy Sessional Exams: 25% Quizzes (4-6): 8% Computer and numerical assignments: 7% Paper + Presentation 1% Project 1% Final Exam: 4%

10 What is computer vision? Automatic understanding of images and video Computing properties of the 3D world from visual data (measurement) Algorithms and representations to allow a machine to recognize objects, people, scenes, and activities. (perception and interpretation)

11 Computer Vision goal is to emulate human vision (which is limited to the visual band of electromagnetic (EM) spectrum), including learning and being able to make inferences and take actions based on visual inputs

12 Why Computer Vision? An image is worth 1 words Many biological systems rely on vision The world is 3D and dynamic Cameras and computers are cheap

13 Real World Overview Image Formation and Camera Geometry Modeling and Calibration Image rectification Recognition Recognize objects using probabilistic techniques Processing on Single Image Linear Filters Edge detection Texture Interpretation Interpret objects using geometric information Multiple Images Multi-view geometry Stereo imaging Structure from motion Segmentation Impose some order on group of pixels to separate them from each other or infer shape information Action

14 What is Computer Vision? given an image or more, extract properties of the 3D world: - Traffic scene - Number of vehicles - Type of vehicles - Location of closest obstacle - Assessment of congestion - Location of the scene captured -

15 Vision for perception, interpretation The Wicked Twister ride Lake Erie sky water Ferris wheel amusement park Cedar Point tree ride 12 E Objects Activities Scenes Locations Text / writing Faces Gestures Motions Emotions ride tree people waiting in line people sitting on ride deck tree bench tree carousel umbrellas pedestrians maxair

16 Related disciplines Graphics Image processing Artificial intelligence Computer vision Algorithms Machine learning Cognitive science

17 Computer Vision and Nearby Fields Derogatory summary of computer vision: Machine learning applied to visual data. J

18 Computer Vision and Nearby Fields Derogatory summary of computer vision: Machine learning applied to visual data. Question answering Real world Digital world Images, videos, sensor data Model of the world Images, videos, interaction Computer Vision Computer Graphics J

19 Why vision? Images and video are everywhere! Personal photo albums Movies, news, sports Surveillance and security Medical and scientific images Slide credit; L. Lazebnik

20 Optical character recognition (OCR) Technology to convert scanned docs to text If you have a scanner, it probably came with OCR software Digit recognition, AT&T labs License plate readers J

21 Examples: HCI Try to make human computer interfaces more natural Gesture recognition Facial Expression Recognition Lip reading 22

22 Examples: Sign Language/Gesture Recognition British Sign Language Alphabet 23

23 Examples: Robotics 24

24 Safety and Security Autonomous robots Driver assistance Monitoring pools (Poseidon) Pedestrian detection [MERL, Viola et al.] Surveillance

25 Face detection Almost all digital cameras detect faces Snapchat face filters

26

27

28

29 Smile detection Sony Cyber-shot T7 Digital Still Camera J

30 Object recognition (in supermarkets)

31 How does it work? Think-Pair-Share

32 AND+IN/Kessel)

33 Vision-based biometrics How the Afghan Girl was Identified by Her Iris Patterns Read the story (Wikipedia) J

34 Login without a password

35 Login without a password

36 Object recognition (in mobile phones) e.g., Google Lens

37 3D from images Building Rome in a Day: Agarwal et al. 29

38 Human shape capture

39 Human shape capture

40 Human shape capture

41 Human shape capture

42 Special effects: shape capture Star Wars: Rogue One Peter Cushing / Admiral Tarkin

43 Special effects: shape capture

44 Special effects: motion capture

45 Interactive Games: Kinect Object Recognition: Mario: 3D: Robot: J

46 Sports Sportvision first down line Nice explanation on J

47 Medical imaging 3D imaging MRI, CT Image guided surgery Grimson et al., MIT J

48 AutoCars - Uber bought CMU s lab

49

50

51 Industrial robots Vision-guided robots position nut runners on wheels J

52 Vision in space NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 27. Vision systems (JPL) used for several tasks Panorama stitching 3D terrain modeling Obstacle detection, position tracking For more, read Computer Vision on Mars by Matthies et al. J

53 Mobile robots NASA s Mars Spirit Rover Saxena et al. 28 STAIR at Stanford J

54 Augmented Reality and Virtual Reality MS HoloLens, Oculus, Magic Leap, ARCore / ARKit

55 Problem Domain Application Input Pattern Output Class Document Image Analysis Document Classification Document Classification Optical Character Recognition Document Image Characters/words Internet search Text Document Semantic categories Junk mail filtering Junk/Non-Junk Multimedia retrieval Internet search Video clip Video genres Speech Recognition Natural Language Processing Telephone directory assistance Speech waveform Spoken words Information extraction Sentence Parts of Speech Biometric Recognition Personal identification Face, finger print, Iris Authorized users for access control Medical Military Summary of Applications Computer aided diagnosis Automatic target recognition Microscopic Image Infrared image Industrial automation Fruit sorting Images taken on conveyor belt Healthy/cancerous cell Target type Grade of quality Bioinformatics Sequence analysis DNA sequence Known types of genes 56

56 Jitendra Malik, UC Berkeley Three R s of Computer Vision [Further progress in] the classic problems of computational vision: reconstruction recognition (re)organization [requires us to study the interaction among these processes].

57 Recognition, Reconstruction & Reorganization Recognition Reconstruction Reorganization

58 The Three R s of Vision Recognition Reconstruction Reorganization Each of the 6 directed arcs in this diagram is a useful direction of information flow

59 The Three R s of Vision Recognition Superpixel assemblies as candidates Reconstruction Reorganization

60 PASCAL Visual Object Challenge (Everingham et al)

61 How about the other direction Recognition Reconstruction Reorganization

62 Recognition Helps Reorganization

63 We train classifiers to predict top-down the pixels belonging to the object Score Original detection Search nearby Segment Score Regress boxes Score

64 Actions and Attributes from Wholes and Parts G. Gkioxari, R. Girshick & J. Malik

65 The Three R s of Vision Recognition Reconstruction Reorganization We have explored category-specific 3D reconstruction.

66 Category Specific Object Reconstruction Kar, Tulisiani, Carreira & Malik

67 Basis Shape Models

68 Results

69 The Three R s of Vision Recognition Reconstruction Reorganization These ideas apply equally well in a video setting

70 Images Video Image classification Object detection Action classification Action detection Is there a dog in the image? Is there a dog and where is it in the image? Is there a person diving in the video? Is there a person diving and where is it in the video?

71 Assignment 1: Image Filtering and Hybrid Images Implement image filtering to separate high and low frequencies. Combine high frequencies and low frequencies from different images to create a scale-dependent image. J

72 Assignment 2: Local Feature Matching Implement interest point detector, SIFT-like local feature descriptor, and simple matching algorithm. J

73 Assignment 3: Scene Recognition with Bag of Words Quantize local features into a vocabulary, describe images as histograms of visual words, train classifiers to recognize scenes based on these histograms. J

74 Assignment 4: Convolutional Neural Nets Asg 3 again, but state of the art. J

75 Journals Computer Vision Publications IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) Internal Journal of Computer Vision (IJCV) IEEE Trans. on Image Processing

76 Computer Vision Publications Conferences International Conference on Computer Vision (ICCV), once every two years IEEE Conf. of Computer Vision and Pattern Recognition (CVPR), once a year Europe Conference on Computer Vision (ECCV), once every two years

77 Today s Class PART II Transformation Matrix

78 y x y x y x z y x z y x z y x Translation: (2D) (3D) Images courtesy of Dr Imtiaz A Taj ) z z z', y y y', x x x' ( Basic Transformations

79 Cartesian Coordinate System Homogeneous Coordinate System Z Y X W k kz ky kx W h (Euclidean Geometry) (Projective Geometry) h4 h3 h4 h2 h4 h W W W W W W W W W W

80 Scaling: ) z S z' y, S y' x, S x' ( z y x Basic Transformations y x s s y x y x z y x s s s z y x z y x (2D) (3D)

81 Rotation (2D): - around origin Basic Transformations y x Cos Sin Sin Cos y x p) T p(r T r - r - around an arbitrary point (not origin) r

82 Rotation (3D): Basic Transformations (Cont.) z R x R y R 1 z y x 1 1 Cos Sin Sin Cos 1 z y x 1 z y x 1 Cos Sin Sin Cos 1 1 z y x 1 z y x 1 Cos Sin 1 Sin Cos 1 z y x around x- axis around y- axis around z- axis

83 3D Rotation of Points Rotation around the coordinate axes, counter-clockwise: 1 cos sin sin cos ) ( cos sin 1 sin cos ) ( cos sin sin cos 1 ) ( z y x R R R p p y z Slide Credit: Saverese

84 HomeTask- 1 (Ungraded) Download and install the latest release of OpenCV. Build and run your first opencv program. Related Tutorials: - Installing OpenCV 3 on Ubuntu: - Using OpenCV 3 with Eclipse:

85 Material in these slides has been taken from, the following resources Acknowledgements Some Slide material has been taken from Dr. Mehmood and Dr. Imtiaz Ali Taj Computer Vision Lectures CSCI 143: Introduction to Computer Vision by James Tompkin Statistical Pattern Recognition: A Review A.K Jain et al., PAMI (22) 2 Pattern Recognition and Analysis Course A.K. Jain, MSU Pattern Classification by Duda et al., John Wiley & Sons. Digital Image Processing, Rafael C. Gonzalez & Richard E. Woods, Addison-Wesley, 22 Machine Vision: Automated Visual Inspection and Robot Vision, David Vernon, Prentice Hall, Advances in Human Computer Interaction, Shane Pinder, InTech, Austria, October 28 Computer Vision A modern Approach by Frosyth 13