Computer Science Workbench

Computer Science Workbench Editor: Tosiyasu L. Kunii Springer Japan KK

Computer Science Workbench N. Magnenat Thalmann, D. Thalmann: Image Synthesis. Theory and Practice. XV, 400 pp., 223 figs., including 80 in color. 1987 B.A. Barsky: Computer Graphics and Geometrie Modeling Using Beta-splines. IX, 156 pp., 85 figs., including 31 in color. 1987 H. Kitagawa, T.L. Kunii: The Unnormalized Relational Data Model. For Office Form Processor Design. XIII, 164 pp., 78 figs. 1989 N. Magnenat Thalmann, D. Thalmann: Computer Animation. Theory and Practice. Second Revised Edition. XIII, 245 pp., 156 figs., including 73 in color. 1990 N. Magnenat Thalmann, D. Thalmann: Synthetic Actors in Computer-Generated 3D Films. X, 129 pp., 133 figs., including 83 in color. 1990 K. Fujimura: Motion Planning in Dynamic Environments. XIII, 178 pp., 85 figs. 1991 M. Suk, S.M. Bhandarkar: Three-Dimensional Object Recognition from Range Images. XXII, 308 pp., 107 figs. 1992 H. Ishikawa: Object-Oriented Database System. Design and Implementation for Advanced Applications. XVIII, 166 pp., 38 figs. 1993 S.Z. Li: Markov Random Field Modeling in Computer Vision. XVI, 260 pp., 71 figs. 1995

S.Z. Li Markov Random Field Modeling in Computer Vision With 72 Figures, Springer

S.Z. Li School of Electrical and Electronic Engineering Nanyang Technological University Nanyang Avenue, Singapore 2263 ISBN 978-4-431-66935-7 Printed on acid-free paper Library of Congress Cataloging-in-Publication Data Li, S. Z., 1958- Markov random field modeling in computer vision I S. Z. Li. p. cm. - (Computer science workbench) Includes bibliographical references and index. ISBN 978-4-431-66935-7 ISBN 978-4-431-66933-3 (ebook) DOI 10.1007/978-4-431-66933-3 1. Computer vision-mathematical models. 2. Markov random fields. I. Title. 11. Series. TA1634.L5 1995 006.3'7'0151954-dc20 95-23304 Springer Japan 1995 Originally published by Springer-Verlag Tokyo Berlin Heidelberg New York in 1995 Softcover reprint of the hardcover 1 st edition 1995 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in other ways, and storage in data banks. The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeseuing: Camera-ready by author using Springer LaTeX macropackage

In Memory 0/ My Mother

Series Preface Computer Science Workbench is a monograph series which will provide you with an in-depth working knowledge of current developments in computer technology. Every volume in this series will deal with a topic of importance in computer science and elaborate on how you yourself can build systems related to the main theme. You will be able to develop a variety of systems, including computer software tools, computer graphics, computer animation, database management systems, and computer-aided design and manufacturing systems. Computer Science Workbench represents an important new contribution in the field of practical computer technology. Tosiyasu L. Kunii

Foreword by Anil K. J ain The objective of mathematical modeling in image processing and computer vision is to capture the intrinsic character of the image in a few parameters so as to understand the nature of the phenomena generating the image. Models are also useful to specify natural constraints and general assumptions about the physical world; such constraints and assumptions are necessary to solve the "inverse" problem of three-dimensional scene interpretation, given two-dimensional image(s) of the scene. The introduction of stochastic or random field models has led to the development of algorithms for image restoration, segmentation, text ure modeling, classification, and sensor fusion. In particular, Gibbs and Markov random fields for modeling spatial context and stochastic inter action among observable quantities have been quite useful in many practical problems, including medical image analysis and interpretation of remotely sensed images. As a result, Markov random field models have generated a substantial amount of excitement in image processing, computer vision, applied statistics, and neural network research communities. This monograph presents an exposition of Markov random fields (MRFS) that is likely to be extensively used by researchers in many scientific disciplines. In particular, those investigating the applicability of MRFs to process their data or images are bound to find its contents very useful. The main focus of the monograph, however, is on the application of Markov random fields to computer vision problems such as image restoration and edge detection in the low-ievel domain, and object matching and recognition in the high-level domain. Using a variety of examples, the author illustrates how to convert a specific vision problem involving uncertainties and constraints into essentially an optimization problem under the MRF setting. In doing so, the author intro duces the reader to the various special classes of MRFs, including MRFs on the regular lattice (e.g., auto models and multi-ievellogistic models) that are used for low-ievel modeling, and MRFs on relational graphs that are used for high-level modeling. The author devotes considerable attention to the problems of parameter estimation and function optimization, both of which are crucial in the MRF paradigm. Specific attention is given to the estimation of MRF parameters in the context of object recognition, and to the issue of algorithm selection for

x Foreword MRF-based function optimization. Another contribution ofthe book is a study on discontinuities, an important issue in the application of MRFs to image analysis. The extensive list of references, high-level descriptions of algorithms and computational issues associated with various optimization algorithms are some of the nice features of this book. On the whole, the contents of this monograph nicely complement the material in Kindermann and Snell's book "Markov Random Fields and Their Applications," and Chellappa and Jain's edited volume entitled "Markov Random Fields: Theory and Ap'plications." In my opinion, the main contribution of this book is the manner in which significant MRF-related concepts are lucidly illustrated via examples from computer vision. Anil K. Jain East Lansing, Michigan

Preface Since its beginning, computer vision research has been evolving from heuristic design of algorithms to systematic investigation of approaches. In this process, researchers have realized: (1) The solution to a vision problem should be sought based on optimization principles, albeit explicitly or implicitly, and (2) contextual constraints are ultimately necessary for the understanding of visual information in images. Two questions follow: how to define an optimality criterion under contextual constraints and how to find its optimal solution. Markov random field (MRF) theory, a branch of probability theory, provides a foundation for the characterization of contextual constraints and the derivation of the prob ability distribution of interacting features. In conjunction with methods from decision and estimation theory, the MRF theory provides a systematic approach for deriving optimality criteria such as those based on the maximum aposteriori (MAP) concept. This MAP-MRF framework enables us to systematically develop algorithms for a variety of vision problems using rational principles rather than ad hoc heuristics. For these reasons, there has seen increasing interest in modeling computer vision problems using MRFs in recent years. This book provides a coherent reference to theories, methodologies and recent developments in solving computer vision problems based on MRFs, statistics and optimization. It treats various problems in low- and high-level computational vision in a systematic and unified way within the MAP-MRF framework. The main issues of concern are: how to use MRFs to encode contextual constraints that are indispensable to image understanding, how to derive the objective function, typically the posterior distribution, for the optimal solution to a problem, and how to design computational algorithms for finding the optimal solution. As the first thorough reference to the subject, the book has four essential parts for solving computational vision problems using MRFs: (1) introduction to fundamental theories, (2) formulations of various vision models in the MAP MRF framework, (3) parameter estimation, and (4) optimization methods. Chapter 1 introduces the notion of visual labeling and describes the important results in MRF theory pertinent to applications in vision modeling. A vision problem is posed in terms of Bayes labeling of an MRF. Its optimal

XII Preface solution is then defined as the MAP configuration of the MRF. The role of optimization in computer vision is discussed. These form the basis on which MAP-MRF models are formulated. Chapter 2 formulates MRF models for low-ievel vision problems, such as image restoration, reconstruction, edge detection, texture, and optical flow. The systematic MAP-MRF approach for deriving the posterior distribution is illustrated step by step. Chapter 3 addresses the issue of discontinuities in low-ievel vision. An important necessary condition is derived for any MRF prior potential function to be adaptive to discontinuities to avoid oversmoothing. This gives rise to the definition of a dass of adaptive interaction functions and thereby a dass of MRF models capable of dealing with discontinuities. Chapter 4 provides a comparative study on discontinuity adaptive MRF priors and robust M-estimators based on the results obtained in Chapter 3. To tackle the problems associated with M-estimators, a method is presented to stabilize M-estimators with respect to its initialization and convergence. Chapter 5 presents MRF models for object recognition and pose determination in high-level vision. Relational measurements are incorporated into the energy function as high-level constraints. The concept of line process is extended to the separation of overlapping objects and the elimination of outlier features. Chapter 6 describes various methods for both supervised and unsupervised parameter estimation, induding coding method, pseudo-likelihood, least squares method and expectation maximization. A simultaneous image labeling and parameter estimation paradigm is also presented, which enhances the low-ievel models in Chapter 2. Chapter 7 presents a theory of parameter estimation for optimizationbased object recognition. Two levels of criteria are proposed for the estimation: correctness and optimality. Optimal parameters are learned from examples using supervised learning methods. The theory is applied to parameter learning for the MRF recognition. Chapters 8 and 9 present local and global methods, respectively, for energy optimization in finding MAP-MRF solutions. These indude various algorithms for continuous, discrete, unconstrained and constrained minimization and strategies for approximating global solutions. The final version of this manuscript benefited from comments on earlier vers ions by a number of people. I am very grateful to Anil K. Jain and Kanti V. Mardia for their valuable suggestions. I would like to thank Kap Luk Chan, Lihui Chen, Yi-Ping Hung, Eric Sung, Han Wang, Ming Xie and Dekun Yang. Their corrections have had a very positive effect on the book. I am particularly indebted to Yunjun Zhang, Weiyun Yau and Yihong Huang for their proofreading of the whole manuscript. Finally, I owe a deep debt of gratitude to my wife, Qun Pan, for her understanding, patience, and support. S.Z. Li

Contents Foreword by Anil K. J ain Preface 1 Introduction 1.1 Visual Labeling........ 1.1.1 Sites and Labels... 1.1.2 The Labeling Problem 1.1.3 Labeling Problems in Vision. 1.1.4 Labeling with Contextual Constraints 1.2 Markov Random Fields and Gibbs Distributions 1.2.1 Neighborhood System and Cliques 1.2.2 Markov Random Fields.. 1.2.3 Gibbs Random Fields.... 1.2.4 Markov-Gibbs Equivalence.... 1.2.5 Normalized and Canonical Forms. 1.3 Useful MRF Models.... 1.3.1 Auto-Models.... 1.3.2 Multi-Level Logistic Model 1.3.3 The Smoothness Prior.. 1.3.4 Hierarchical GRF Model. 1.4 Optimization-Based Vision... 1.4.1 Research Issues.... 1.4.2 Role of Energy Functions 1.4.3 Formulation of Objective Functions. 1.4.4 Optimality Criteria. 1.5 Bayes Labeling of MRFs.. 1.5.1 Bayes Estimation.. 1.5.2 MAP-MRF Labeling 1.5.3 Regularization... 1.5.4 Summary of MAP-MRF Approach IX XI 1 3 3 4 6 7 8 8 11 12 14 16 17 17 19 21 23 24 25 26 27 29 30 31 32 33 34

XN Contents 2 Low Level MRF Models 2.1 Observation Models 2.2 Image Restoration and Reconstruction 2.2.1 MRF Priors for Image Surfaces 2.2.2 Piecewise Constant Restoration. 2.2.3 Piecewise Continuous Restoration 2.2.4 Surface Reconstruction.... 2.3 Edge Detection.... 2.3.1 Edge Labeling using Line Process. 2.3.2 Forbidden Edge Patterns. 2.4 Texture Synthesis and Analysis 2.4.1 MRF Texture Modeling 2.4.2 Texture Segmentation 2.5 Optical Flow....... 2.5.1 Variational Approach. 2.5.2 Flow Discontinuities 3 Discontinuities in MRFs 3.1 Smoothness, Regularization and Discontinuities 3.1.1 Regularization and Discontinuities 3.1.2 Other Regularization Models.. 3.2 The Discontinuity Adaptive MRF Model 3.2.1 Defining the DA Model... 3.2.2 Relations with Previous Models 3.2.3 Discrete Data and 2D Cases. 3.2.4 Solution Stability.... 3.3 Computation of DA Solutions... 3.3.1 Solving the Euler Equation 3.3.2 Experimental Results 3.4 Conclusion.... 37 38 39 39 42 43 46 48 49 51 52 53 56 59 59 61 63 64 65 69 69 70 74 76 77 78 78 80 81 4 Discontinuity-Adaptivity Model and Robust Estimation 83 4.1 The DA Prior and Robust Statistics 84 4.1.1 Robust M Estimator..... 85 4.1.2 4.1.3 4.1.4 4.1.5 Problems with M Estimator. Redefinition of M Estimator. AM Estimator.... Convex DA and M-Estimation Models 4.2 Experimental Comparison..... 4.2.1 Location Estimation... 4.2.2 Rotation Angle Estimation. 87 88 89 90 92 92 96

Contents 5 High Level MRF Models 5.1 Matching under Relational Constraints 5.1.1 Relational Structure Representation 5.1.2 Work in Relational Matching.. 5.2 MRF-Based Matching.... 5.2.1 Posterior Probability and Energy 5.2.2 Matching to Multiple Objects 5.2.3 Experiments 5.2.4 Extensions.... 5.3 Pose Computation....... 5.3.1 Pose Clustering and Estimation. 5.3.2 Simultaneous Matching and Pose 5.3.3 Discussion.... 6 MRF Parameter Estimation 6.1 Supervised Estimation with Labeled Data 6.1.1 Maximum Likelihood. 6.1.2 Pseudo-Likelihood.... 6.1.3 Coding Method.... 6.1.4 Mean Field Approximations 6.1.5 Least Squares Fit.... 6.2 Unsupervised Estimation with Unlabeled Data 6.2.1 Simultaneous Restoration and Estimation 6.2.2 Simultaneous Segmentation and Estimation 6.2.3 Expectation-Maximization. 6.2.4 Cross Validation.... 6.3 Further Issues.... 6.3.1 Estimating the Number of MRFs 6.3.2 Reduction of Nonzero Parameters. 7 Parameter Estimation in Optimal Object Recognition 7.1 Motivation.... 7.2 Theory of Parameter Estimation for Recognition 7.2.1 Optimization-Based Object Recognition 7.2.2 Criteria for Parameter Estimation... 7.2.3 Linear Classification Function.... 7.2.4 A Non-parametric Learning Algorithm. 7.2.5 Reducing Search Space.... 7.3 Application in MRF Object Recognition 7.3.1 Posterior Energy.... 7.3.2 Energy in Linear Form...... 7.3.3 How Minimal Configuration Changes 7.3.4 Parametrie Estimation under Gaussian Noise 7.4 Experiments.... 7.4.1 Recognition of Line Patterns xv 101 101 102 106 108 109 111 113 122 124 124 127 130 131 133 133 135 136 137 139 143 144 145 150 152 153 153 155 157 157 159 160 161 164 167 169 170 170 171 172 173 176 176

XVI Contents 7.4.2 Recognition of Curved Objects 7.5 Conclusion.... 8 Minimization - Local Methods 8.1 Classical Minimization with Continuous Labels 8.2 Minimization with Discrete Labels 8.2.1 Iterated Conditional Modes 8.2.2 Relaxation Labeling... 8.2.3 Highest Confidence First. 8.2.4 Dynamic Programming 8.3 Constrained Minimization. 8.3.1 Penalty Functions. 8.3.2 Lagrange Multipliers 8.3.3 Hopfield Method.. 8.3.4 RL using Lagrange-Hopfield Method 9 Minimization - Global Methods 9.1 Simulated Annealing.... 9.1.1 Random Sampling Algorithms. 9.1.2 Annealing.... 9.2 Mean Field Annealing.... 9.3 Graduated Non-Convexity.... 9.3.1 Annealing Labeling for MAP-MRF Matching 9.4 Genetic Algorithms... 9.5 Experimental Comparison.... 9.6 Accelerating Computation.... 9.6.1 Multi-resolution Methods 9.6.2 Use of Heuristics 9.7 Model Debugging.... References List of Notation Index 181 182 185 187 189 189 190 195 197 199 200 201 202 204 207 208 209 210 211 214 219 220 222 228 228 229 230 231 259 261