VOLUME COMPUTER AND ROBOT VISION Robert M. Haralick University of Washington Linda G. Shapiro University of Washington T V ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California New York Don Mills, Ontario Wokingham, England Amsterdam Bonn
CONTENTS K 12.1 12.2 12.3 12.4 12.5 12.6 12.7 12.8 12.9 12.10 ^ Illumination Introduction Radiometry 12.2.1 Bidirectional Reflectance Function 12.2.2 Photometry 12.2.3 Torrance-Sparrow Model 12.2.4 Lens Collection 12.2.5 Image Intensity Photometrie Stereo Shape from Shading 12.4.1 Shape from Focus Polarization 12.5.1 Representation of Light Using the Coherency Matrix 12.5.2 Representation of Light Intensity Fresnel Equation Reflection of Polarized Light A New Bidirectional Reflectance Function Image Intensity Related Work 1 1 2 7 11 12 13 15 15 17 21 21 26 29 30 32 34 35 35 Perspective Projective Geometry 43 13.1 Introduction 13.2 One-Dimensional Perspective Projection 44 43 m
IV 13.3 The Perspective Projection in 3D 47 13.3.1 Smaller Appearance of Farther Objects 47 13.3.2 Lines to Lines 51 13.3.3 The Perspective Projections of Convex Polyhedra are Convex 52 13.3.4 Vanishing Point 53 13.3.5 Vanishing Line 57 13.3.6 3D Lines-2D Perspective Projection Lines 59 13.4 2D to 3D Inference Using Perspective Projection 61 13.4.1 Inverse Perspective Projection 61 13.4.2 Line Segment with Known Direction Cosines and Known Length 61 13.4.3 Collinear Points with Known Interpoint Distances 66 13.4.4 N Parallel Lines 68 13.4.5 N Lines Intersecting at a Point with Known Angles 69 13.4.6 N Intersecting Lines in a Known Plane 72 13.4.7 Three Lines in a Plane with One Perpendicular to the Other Two 73 13.4.8 Point with Given Distance to a Known Point 74 13.4.9 Point in a Known Plane 76 13.4.10 Line in a Known Plane 77 13.4.11 Angle 78 13.4.12 Parallelogram 80 13.4.13 Triangle with One Vertex Known 81 13.4.14 Triangle with Orientation of One Leg Known 83 13.4.15 Triangle 85 13.4.16 Determining the Principal Point by Using Parallel Lines 91 13.5 Circles 93 13.6 Range from Structured Light 100 13.7 Cross-Ratio 101 13.7.1 Cross-Ratio Definitions and Invariance 102 13.7.2 Only One Cross-Ratio 103 13.7.3 Cross-Ratio in Three Dimensions 103 13.7.4 Using Cross-Ratios Hl Analytic Photogrammetry 125 14.1 Introduction 125 14.2 Nonlinear Least-Squares Solutions 129 14.3 The Exterior Orientation Problem 131 14.3.1 Standard Solution 133 14.3.2 Auxiliary Solution 136 14.3.3 Quaternion Representation 140
v 14.4 Relative Orientation 144 14.4.1 Standard Solution 144 14.4.2 Quaternion Solution 147 14.5 Interior Orientation 151 14.6 Stereo 156 14.7 2D-2D Absolute Orientation 160 14.8 3D-3D Absolute Orientation 163 14.9 Robust M-Estimation 167 14.9.1 Modified Residual Method 169 14.9.2 Modified Weights Method 170 14.9.3 Experimental Results 171 14.10 Error Propagation 174 14.10.1 Implicit Form 176 14.10.2 Implicit Form: General Case 177 14.11 Summary 178 Motion and Surface Structure from Time Varying 187 Image Sequences 15.1 Introduction 187 15.2 The Fundamental Optic Flow Equation 187 15.2.1 Translational Motion 188 15.2.2 Focus of Expansion and Contraction 191 15.2.3 The Moving Line Segment 192 15.2.4 Optic Flow Acceleration Invariant 193 15.3 Rigid Body Motion 194 15.4 Linear Algorithms for Motion and Surface Structure from Optic Flow 201 15.4.1 The Planar Patch Case 201 15.4.2 General Case: Optic Flow-Motion Equation 204 15.4.3 A Linear Algorithm for Solving Optic Flow-Motion Equations 205 15.4.4 Mode of Motion, Direction of Translation, and Surface Structure 207 15.4.5 Linear Optic Flow-Motion Algorithm and Simulation Results 208 15.5 The Two View-Linear Motion Algorithm 212 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review 212 15.5.2 General Curved Patch Motion Recovery from Two Perspective Views: A Simplified Linear Algorithm 215 15.5.3 Determining Translational Orientation 218 15.5.4 Determining Mode of Motion and Relative Depths 219
vi 15.5.5 A Simplified Two View-Motion Linear Algorithm 219 15.5.6 Discussion and Summary 221 15.6 Linear Algorithm for Motion and Structure from Three Orthographie Views 221 15.6.1 Problem Formulation 221 15.6.2 Determining r 3 3,s 3 3,... 223 15.6.3 Solving a Unique Orthonormal Matrix R 226 15.6.4 Linear Algorithm to Uniquely Solve R, S, a s 226 15.6.5 Summary 227 15.7 Developing a Highly Robust Estimator for General Regression 227 15.7.1 Inability of the Classical Robust M-Estimator to Render High Robustness 227 15.7.2 Partially Modeling Log Likelihood Function by Using Heuristics 229 15.7.3 Discussion 232 15.7.4 MF-Estimator 233 15.8 Optic Flow-Instantaneous Rigid-Motion Segmentation and Estimation 236 15.8.1 Single Rigid Motion 236 15.8.2 Multiple Rigid Motions 237 15.9 Experimental Protocol 238 15.9.1 Simplest Location Estimation 238 15.9.2 Optic Flow-Rigid-Motion Segmentation and Estimation 239 15.10 Motion and Surface Structure from Line Correspondences 245 15.10.1 Problem Formulation 246 15.10.2 Solving Rotation Matrices R',R" and TranslationsT'.T" 248 15.10.3 Solving Three Dimensional Line Structure 251 15.11 Multiple Rigid Motions from Two Perspective Views 251 15.11.1 Problem Statement 252 15.11.2 Simulated Experiments 254 15.12 Rigid Motion from Three Orthographie Views 255 15.12.1 Problem Formulation and Algorithm 260 15.12.2 Simulated Experiments 267 15.12.3 Further Research on the MF-Estimator 268 15.13 Literature Review 268 15.13.1 Inferring Motion and Surface Structure 271 15.13.2 Computing Optic Flow or Image Point Correspondences 274 f Image Matching 16.1 Introduction 289 16.1.1 Image Matching and Object Reconstruction 289 289
Vll 16.1.2 The Principle of Image Matching 291 16.1.3 Image Matching Procedures 293 16.2 Intensity-Based Matching of One-Dimensional Signals 298 16.2.1 The Principle of Differential Matching 299 16.2.2 Estimating an Unknown Shift 301 16.2.3 Estimating Unknown Shift and Scale 305 16.2.4 Compensation for Brightness and Contrast 307 16.2.5 Estimating Smooth Deformations 309 16.2.6 Iterations and Resampling 313 16.2.7 Matching of Two Observed Profiles 315 16.2.8 Relations to Cross-Correlation Techniques 316 16.3 Intensity-Based Matching of Two-Dimensional Signals 320 16.3.1 The Principle and the Relation to Optical Flow 321 16.3.2 Estimating Constant-Shift Parameters 322 16.3.3 Estimating Linear Transformations 325 16.3.4 Invariant Points 330 16.4 An Interest Operator 332 16.4.1 Introduction 332 16.4.2 Estimating Corner Points 334 16.4.3 Evaluation and Classification of Selected Windows 338 16.4.4 Selection of Optimal Windows 342 16.4.5 Uniqueness of Selected Points 345 16.5 Robust Estimation for Feature-Based Matching 348 16.5.1 The Principle of Feature-Based Matching 348 16.5.2 The Similarity Measure 349 16.5.3 Heuristics for Selecting Candidate Pairs 351 16.5.4 Robust Estimation for Determining the Spatial Mapping Function 352 16.5.5 Evaluating the Final Result 356 16.6 Structure from Stereo by Using Correspondence 357 16.6.1 Epipolar Geometry 357 16.6.2 Generation of Normal Images 359 16.6.3 Specializing the Image-Matching Procedures 361 16.6.4 Precision of Three-Dimensional Points from Image Points 362 The Consistent-Labeling Problem 379 17.1 Introduction 379 17.2 Examples of Consistent-Labeling Problems 380 17.2.1 The TV-Queens Problem 380 17.2.2 The Latin-Square Puzzle 380 17.2.3 The Edge-Orientation Problem 381 17.2.4 The Subgraph-Isomorphism Problem 382 17.2.5 The Relational-Homomorphism Problem 382
viii 17.3 Search Procedures for Consistent Labeling 385 17.3.1 The Backtracking Tree Search 385 17.3.2 Backtracking with Forward Checking 387 17.3.3 Backtracking with Discrete Relaxation 392 17.3.4 Ordering the Units 394 17.3.5 Complexity 395 17.3.6 The Inexact Consistent-Labeling Problem 395 17.4 Continuous Relaxation 399 17.5 Vision Applications 403 17.5.1 Image Matching Using Continuous Relaxation 407 17.6 Characterizing Binary-Relation Homomorphisms 408 17.6.1 The Winnowing Process 409 17.6.2 Binary-Relation Homomorphism Characterization 417 17.6.3 Depth-First Search for Binary-Relation Homomorphisms 420 Object Models and Matching 427 18.1 Introduction 427 18.2 Two-Dimensional Object Representation 427 18.2.1 Global Feature Representation 427 18.2.2 Local Feature Representation 432 18.2.3 Boundary Representation 433 18.2.4 Skeleton Representation 437 18.2.5 Two-Dimensipnal Part Representation 439 18.3 Three-Dimensional Object Representations 440 18.3.1 Local Features Representation 440 18.3.2 Wire Frame Representation 440 18.3.3 Surface-Edge-Vertex Representation 441 18.3.4 Sticks, Plates, and Blobs 445 18.3.5 Generalized Cylinder Representation 447 18.3.6 Superquadric Representation 449 18.3.7 Octree Representation 452 18.3.8 The Extended Gaussian Image 453 18.3.9 View-Class Representation 455 18.4 General Frameworks for Matching 458 18.4.1 Relational-Distance Approach to Matching 458 18.4.2 Ordered Structural Matching 465 18.4.3 Hypothesizing and Testing with Viewpoint Consistency Constraint 465 18.4.4 View-Class Matching 466 18.4.5 Affine-Invariant Matching 468 18.5 Model Database Organization 479
IX Knowledge-Based Vision 493 19.1 19.2 19.3 19.4 19.5 19.6 19.7 Introduc ;tion Rnowlec Ige Representations 19.2.1 Feature Vectors 19.2.2 Relational Structures 19.2.3 Hierarchical Structures 19.2.4 Rules 19.2.5 Frames and Schemes Control Strategies 19.3.1 Hierarchical Control 19.3.2 Heterarchical Control Information Integration 19.4.1 Bayesian Approach 19.4.2 Dempster-Shafer Theory A Probabilistic Basis for Evidential Reasoning 19.5.1 19.5.2 19.5.3 19.5.4 Exampli 19.6.1 19.6.2 19.6.3 19.6.4 19.6.5 19.6.6 19.6.7 19.6.8 19.6.9 Summai Legal Court Paradigm Degree of Belief as Chance Probability of Being Inferred Belief Calculus Examples z Systems VISIONS ACRONYM SPAM MOSAIC Mulgaonkar's Hypothesis-Based Reasoning SCERPO Ikeuchi's Model-Based Bin-Picking System Jain's Evidence-Based Recognition System ABLS 493 493 493 494 495 497 499 500 500 502 503 504 507 513 514 516 516 517 521 521 522 524 525 526 527 528 529 530 531 Accuracy 535 20.1 Introduction 535 20.2 Mensuration Quantizing Error 535 20.3 Automated Position Inspection: False Alarm and Misdetection Rates 544 20.3.1 Analysis 546 20.3.2 Discussion 551 20.4 Experimental Protocol 553 20.5 Determining the Repeatability of Vision Sensor Measuring 554
X 20.5.1 The Model 554 20.5.2 Derivation 555 20.6 Determining the Positional Accuracy of Vision Sensors 557 20.6.1 The Model 558 20.6.2 Derivation 558 20.7 Performance Assessment of Near-Perfect Machines 560 20.7.1 The Derivation 561 20.7.2 Balancing the Acceptance Test 566 20.7.3 Lot Assessment 567 20.8 Summary 568 Glossary of Computer Vision Terms 571 21.1 21.2 21.3 21.4 21.5 21.6 21.7 21.8 21.9 21.10 21.11 21.12 21.13 21.14 21.15 21.16 21.17 21.18 21 The Image Photometry and Illumination Photogrammetry Image Operators Point Operators Spatial Operators Morphological Operators Hough Transform Digital Geometry 2D Shape Description Curve and Image Data Structures Texture Segmentation Matching Localization General Image Processing Vision Pattern Recognition Index of Terms 571 574 576 578 578 580 583 584 585 586 589 592 593 593 595 596 597 602 609 Index 621