Computer and Machine Vision Lecture Week 11 Part-2 Segmentation, Camera Calibration and Feature Alignment March 28, 2014 Sam Siewert
Outline of Week 11 Exam #1 Results Overview and Solutions Wrap up of SIFT and David Lowe s Paper Primitive Operations to Accelerate SIFT Uses of SIFT and Comparison to Hough and RANSAC Introduction to Segmentation, Camera Calibration and Feature Alignment Sam Siewert 2
Recall Image Pyramid from Chapter 3 in CV Page 133 A fundamental for scale invariance (SIFT search for keypoint candidates in different scales) Resolution Decimation Reduction in Resolution, or downscaling Resolution Interpolation Increased Resolution, or upscaling (Can t create data where there was none in original, but we can interpolate) L=0, e.g. 9x9 L=1, 5x5 L=2, 3x3 OpenCV Image Pyramids Low Pass Filtering (Gaussian kernel convolution) followed by pixel decimation (removal of odd or even numbered rows and columns) Rows 0 8, Col 0 8 Drop rows 1,3,5,7 to go to L=1 5x5 Drop rows 1,3 to go to L=2 3x3 Sam Siewert 3
Feature Vector Concept General Concept - Summarize Data with Fewer Data points E.g. an Intensity Histogram, Feature Vector for a Color Image If Histogram(Image_A) = Histogram(Image_B), Image_A = Image_B more likely Much like a Fingerprint for Human Identification (not definitive, but narrows search) The Intensity Histogram is Not a Robust Feature Vector (Lighting Sensitivity) Increasing Dimensions, Normalization, Scale, Orientation and Quality Improves R G B R G B Sam Siewert 4
Image Gradient Concept Sam Siewert 5
Example Feature Vectors Any N dimensional vector that represents and object A form of compression for the reference database and faster compare Vector and Distance Metric Definitions are Key, But Once Defined, Simple Distance Measures Can Be Applied Corners of Objects Radial Distance Profile (Centroidal) Various Histograms Derived from Scans with Binning (Blobs) Sam Siewert 6
Feature Vectors Unique Patterns That Define Objects of Interest How to Generalize Keypoints of Feature Vectors is the Challenge Bio-inspired (Human Observation) is A Promising Approach, but Search is Costly! Operation in Real-Time? PCA Principal Component Analysis http://en.wikipedia.org/wiki/file:sift_keypoints_filtering.jpg http://en.wikipedia.org/wiki/file:gaussianscatterpca.png Sam Siewert 7
N Dimensional Distance Harder to Visualize than 3D Vector Differences, but Same Concept as Euclidian Quick Review of Euclidian One Dimension: abs(p 1 p 2 ), or Sqrt((p 1 p 2 ) 2 ) Two Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) 2 ) Three Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) 2 + (p 3 - q 3 ) 2 ) N Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) 2 + + (p n - q n ) 2 ) Sam Siewert 8
Other Distance Measures Defines the Difference Between {A} and {B} in General with a Metric (Number) Euclidian Distance is One Such Metric Manhattan Distance is Another (Sum of Orthogonal Line Segments on a Grid Between 2 Points) Hamming Distance is Another - # of Positions at Which 2 Equal Length Strings Have Different Symbols E.g. 1111_0000 and 0000_1111 have maximum distance or toned and roses has distance of 3 in ASCII Sam Siewert 9
Final Words on SIFT Re-Read David Lowe s Paper 1. Scale Space Extrema Detection - Find Best Candidate Image Locations to Form Keypoint feature vectors (scale invariant through search in Gaussian pyramid) 2. Keypoint Localization - Filter Out Keypoints (to remove minimum contrast and noisy edge curvature feature vectors) 3. Keypoint Orientation So Descriptors are relative to orientation of the keypoint (to provide orientation invariance) and for magnitude to select proper level in Gaussian image 4. Keypoint Descriptor Representation in database is a feature vector with magnitude, orientation for nxn sub-regions with vectors of m dimensions (e.g. Figure 7 2x2 with 8x8 feature vector) Notes from the Paper on the overall Algorithm follow Sam Siewert 10
SIFT Interesting Aspects Feature Selection with Mathematical Significance Calculations Scale, Orientation and Lighting Invariance Goals Keypoints are Complex Feature Vectors to Use for Match Bio-Inspired (see page 14, local image descriptor) Edelman, Intrator and Poggio on Neurons for 3D recognition (gradient at orientation with spatial frequency key trigger in visual cortex) Sam Siewert 11
Image Correspondence and Registration Historically Features as Simple as Intensity and Color Vectors in an n x n ROI Have Been Used to Register Images Simpler Feature Vectors Work for Left- Eye, Right-Eye Images (Simple Baseline Separation of Known Distance) Still Complicated by Camera Calibration Sam Siewert 12
Hough and RANSAC Recall Hough Idea is to Fit Threshold Points to Lines, Circles/Elipses, Shape (RANSAC fits to Lines) Feature Matching as Well SIFT Results in Many More Keypoint feature vectors compared to Hough Advantage for Correspondence and Recognition (Like more Detailed Fingerprint) Likewise, Sometimes a Disadvantage Large Search for Match Detection Sam Siewert 13
Alternatives in Machine Vision Shape and Pattern Recognition Alternatives to Hough and SIFT Sam Siewert 14
Centroidal Profile Find Centroid Based on Threshold and Pixel COM (As Discussed Before) Define Shape In Terms of Radius as a Function of Angle Around Centroid (Shape Signature) From Computer and Machine Vision, E.R. Davies, page 271. Sam Siewert 15
Centroidal Profile Problems Notched or Partially Occluded Objects Computationally Complex Must Have Accurate Centroid First Suggested for Inspection Applications (Within Shape Tolerances?) From Computer and Machine Vision, E.R. Davies, page 272. Sam Siewert 16
Segmentation, Camera Calibration and Feature Alignment Sam Siewert 17
Segmentation (CV Chapter 5) Foreground and Background E.g. Background Elimination (Simplify to 2 regions over time) Simple Threshold (Simplify to 2 regions by intensity) Edge Detection to Define Contour Boundaries (Gradient separation) Clustering Methods Use Distance Measures to Assign Pixels to a Center Value (could be random color or intensity) E.g. OpenCV K-means Clustering Divide Image of N pixels into K clusters with Each Belonging to Cluster with Nearest Mean http://en.wikipedia.org/wiki/k-means_clustering Watershed Gradient Lines (Like Contour Map) Define Regions, Fill Between them Sam Siewert 18
Segmentation Use Divides Image Logically Rather than Arbitrary Selection or ROI (Region of Interest) as a Grid Does Localized Feature Vectors in Each ROI Classification and Object Recognition Step Defines Features Rather than Looking for Feature Matches (E.g. Lines in Image like Hough) Sam Siewert 19
Cameras and Calibration Basics for 2D and Considerations for 3D Mapping and Ranging Sam Siewert 20
Camera Calibration Intrinsics Features of the Camera Itself Optical Characteristics (Fish Eye wider in center, Hourglass thinner in center) Detector Characteristics Pixels with Value at Zero Illumination (Christmas Light) or Pixels with Low Value, Pixel Type (Bayer Filter, or Other) Extrinsics Camera Mount, Thermal Issues, Vibration, Orientation and Viewpoint http://en.wikipedia.org/wiki/bayer_filter Sam Siewert 21
CV Ch. 6 & Learning OpenCV Ch. 11 Read to Understand Basic Concepts Coordinate Transformation Between Pixel and Real World Coordinates E.g. Tilt, Pan, Zoom of Camera and Location in X, Y, Z of target of Interest in Scene Why is this an issue and why is calibration required? Camera optical distortions Lighting conditions Thermal and Mechanical Perturbations (change in camera mount over time) Detector Manufacturing Imperfection (Same Camera Model, Different Image from Same Scene) Radial Distortions Practical Methods Provided in Learning OpenCV and Here Sam Siewert 22
Camera Mount Basic Concepts Single Camera Tilt/Pan Object Tracking (Common Configuraiton) 2 Axes of Rotation (Others are Fixed) +/- 45 deg Tilt Rotation Servo (typical, could be up to 90 deg) +/- 45 deg Pan Rotation Servo (typical, could be up to 180 deg) Side or Rear Mounting of Tilt Servo Ideally No Vibration or Rotation, Just Commanded Tilt/Pan Sam Siewert 23
Dual Camera Tilt/Pan Tracking Baseline with 2 Fixed Cameras (for Stereo Ranging) Pan Servo Pans the Entire Baseline Tilt Servo Tilts Pan Servo Side Mounting Plate Ideally Baseline Separation is Constant (no bending, twisting of the beam) Sam Siewert 24
Pixel Coordinates Define Image Coordinates to Track Object Centroid Tilt/Pan With Servos to Keep Target Centroid in FOV Center E.g. ½ Resolution 320x240 AR=4:3 Image from Analog Camera X N T S C 2 4 0 P i x e l R o w F o r m a t Frame Origin Pixel Address 0, 0 Reference Target Pixel Address 160, 120 Y NTSC 320 Pixel Column Format Sam Siewert 25
Stereo Ranging with Common Tilt/Pan Fixed Camera Baseline Sam Siewert 26
Pixel to Servo Calibration at Distance Tilt Shifted Frame Tilt One Servo Increment Pan Shifted Frame Pan One Servo Increment N T S C 2 4 0 P i x e l R o w F o r m a t Pan Shifted Target Pixel Address 158, 120 Reference Target Pixel Address 160, 120 Tilt Shifted Target Pixel Address 160, 122 NTSC 320 Pixel Column Format Pan 1 Servo Increment and Find Centroid X Pixel Change Tilt 1 Servo Increment and Find Centroid Y Pixel Change Sam Siewert 27
Characterize Camera FOV at Distances FOV Width/Height Linear as a Function of Distance Pixels/Inch Not Necessarily Linear as Function of Distance Use to Calibrate Servo Step Size (Gain) for Target Centering For Servo Step Sizes of 1 Increment in Tilt/Pan, Camera Will Track Slowly For Larger Servo Step Sizes May Over-shoot Determine Dead-bands (Servo Limits for Pixel Change Accuracy) Sam Siewert 28
Calibrate with Reference Image at Known Distance Generalize and Expand Auto-Calibration Sequence as Described for N x M known Targets Most Common is the Chess/Checker Board OpenCV Support for Calibration Using Chess Board Sam Siewert 29
Finding Centroid In Image Target Known by Color, Shape, or Brightness Raster to Find Target Edges and Max Width and Height Scanline for Symmetric targets Mark Centroid in Image for Easy Debug Noise in Image Will Cause Centroid Error Use Averaging Pan Right to Move Target Left in Image Tilt Up to Move Target Down in Image Sam Siewert 30
Tips for Practical Real-time Calibration Consider Controlled Lighting or For Mobile Robots, On-Board Lighting with LEDs (Control your Own Lighting) Use Frame Grabber ADC Sensitivity Settings to Control Brightness Consider Automatic Calibration Sequences Place Reference Targets to Set Pixel Step Size as a Function of Servo Step Size Use Stereo Range Estimation to Determine Distance to Target and Set Tracking Gains for Current Distance When Target is Lost, Go Into Search Mode Start from Max Tilt/Pan and Raster to Min Tilt/Pan to Find Target For Search Modes Use Coarser Step Size Consider Small Servo Step Size, But High Frame Rate and Servo Command Rate Be Careful of Processor Overload with Image Processing at 30 FPS Sam Siewert 31
Feature Alignment Mechanical vs. Digital Image Processing Sam Siewert 32
Mosaics by Scan Mirror and Gimbal Tilt, Pan, Rotate Gimbal to Stabilize Field of View Electro-mechanically (Works, but Expensive) Steadicam (Automatic Gimbal to Hold FoV) CCTV Security Cameras Scan Mirror in Optical Path to Stabilize Images (Spitzer Space Telescope) Scan Mirror Moves in Counter-veiling Direction of Telescope Rotation (Ideally Single Axis) Works Because Perturbations in Deep Space are Minimal http://en.wikipedia.org/wiki/steadicam Sam Siewert 33
Digital Stabilization Let Camera Vibrate, Tilt/Pan/Rotate on Non- Rigid Mounts Digitally Stabilize Find Successive Corresponding Feature Keypoints in Images Over time Re-align Center of Each Image Crop Outermost Pixels Common Use of SIFT and Other Feature Registration Methods (Hough, RANSAC, etc.) Sam Siewert 34
Mosaics ios and Android Panorama Feature Successive Images are Taken Correspondence Found at Keypoints Stitch Images Together and Crop Overlaps Alternative is Tilt/Pan with Gimbal and Scan Mirror Tilt or Pan at Continuous Known Rate Rotate Scan Mirror in Optical Path in Opposite Direction to Stare at Target FoV At Known Radial Distance Snap Back to Stabilize Next Image Same Concept as Staring at a Fixed Object from a Moving Vehicle Sam Siewert 35
Stereo Correspondence Use Similar Triangles To Estimate Distance Right-Eye, Left-Eye Images with Known Intrinsics and Extrinsics Keypoints Found in Both are Common Target Centroid Planar? Out of Plane Causes Error Vanishing Points Need Numerous Common Targets to Form Passive Depth Map Human eye concept from http://en.wikipedia.org/wiki/binocular_disparity The dl, dr, f must be known from intrinsic calibration The b is known extrinsic, and solve for d Sam Siewert 36