COMPUTER VISION FOR VISUAL EFFECTS Modern blockbuster movies seamlessly introduce impossible characters and action into real-world settings using digital visual effects. These effects are made possible by research from the field of computer vision, the study of how to automatically understand images. Computer Vision for Visual Effects will educate students, engineers, and researchers about the fundamental computer vision principles and state-of-the-art algorithms used to create cutting-edge visual effects for movies and television. The author describes classical computer vision algorithms used on a regular basis in Hollywood (such as blue screen matting, structure from motion, optical flow, and feature tracking) and exciting recent developments that form the basis for future effects (such as natural image matting, multi-image compositing, image retargeting, and view synthesis). He also discusses the technologies behind motion capture and three-dimensional data acquisition. More than 200 original images demonstrating principles, algorithms, and results, along with in-depth interviews with Hollywood visual effects artists, tie the mathematical concepts to real-world filmmaking. is an Associate Professor in the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute. His current research interests include computer vision problems related to modeling 3D environments with visual and range imagery, calibration and tracking problems in large camera networks, and machine learning problems for radiotherapy applications. Radke is affiliated with the NSF Engineering Research Center for Subsurface Sensing and Imaging Systems; the DHS Center of Excellence on Explosives Detection, Mitigation and Response (ALERT); and Rensselaer s Experimental Media and Performing Arts Center. He received an NSF CAREER award in March 2003 and was a member of the 2007 DARPA Computer Science Study Group. Dr. Radke is a senior member of the IEEE and an associate editor of IEEE Transactions on Image Processing.
Computer Vision for Visual Effects RICHARD J. RADKE Rensselaer Polytechnic Institute
cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Mexico City Cambridge University Press 32 Avenue of the Americas, New York, NY 10013-2473, USA Information on this title: /9780521766876 2013 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2013 Printed in China by Everbest A catalog record for this publication is available from the British Library. Library of Congress Cataloging in Publication Data Radke, Richard J., 1974 Computer vision for visual effects /. pages cm Includes bibliographical references and index. ISBN 978-0-521-76687-6 1. Cinematography Special effects Data processing. 2. Computer vision. I. Title. TR858.R33 2013 621.39 93 dc23 2012017763 ISBN 978-0-521-76687-6 Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate.
You re here because we want the best and you are it. So, who is ready to make some science? Cave Johnson
Contents 1 Introduction... 1 1.1 Computer Vision for Visual Effects 2 1.2 This Book s Organization 4 1.3 Background and Prerequisites 6 1.4 Acknowledgments 7 2 Image Matting... 9 2.1 Matting Terminology 10 2.2 Blue-Screen, Green-Screen, and Difference Matting 13 2.3 Bayesian Matting 16 2.4 Closed-Form Matting 20 2.5 Markov Random Fields for Matting 29 2.6 Random-Walk Methods 30 2.7 Poisson Matting 35 2.8 Hard-Segmentation-Based Matting 36 2.9 Video Matting 40 2.10 Matting Extensions 42 2.11 Industry Perspectives 45 2.12 Notes and Extensions 50 2.13 Homework Problems 51 3 Image Compositing and Editing... 55 3.1 Compositing Hard-Edged Pieces 56 3.2 Poisson Image Editing 62 3.3 Graph-Cut Compositing 69 3.4 Image Inpainting 73 3.5 Image Retargeting and Recompositing 80 3.6 Video Recompositing, Inpainting, and Retargeting 92 3.7 Industry Perspectives 94 3.8 Notes and Extensions 100 3.9 Homework Problems 102 4 Features and Matching... 107 4.1 Feature Detectors 108 4.2 Feature Descriptors 127 4.3 Evaluating Detectors and Descriptors 136 vii
viii Contents 4.4 Color Detectors and Descriptors 138 4.5 Artificial Markers 139 4.6 Industry Perspectives 140 4.7 Notes and Extensions 143 4.8 Homework Problems 145 5 Dense Correspondence and Its Applications... 148 5.1 Affine and Projective Transformations 150 5.2 Scattered Data Interpolation 152 5.3 Optical Flow 157 5.4 Epipolar Geometry 168 5.5 Stereo Correspondence 175 5.6 Video Matching 184 5.7 Morphing 187 5.8 View Synthesis 191 5.9 Industry Perspectives 195 5.10 Notes and Extensions 200 5.11 Homework Problems 203 6 Matchmoving... 207 6.1 Feature Tracking for Matchmoving 208 6.2 Camera Parameters and Image Formation 211 6.3 Single-Camera Calibration 216 6.4 Stereo Rig Calibration 221 6.5 Image Sequence Calibration 225 6.6 Extensions of Matchmoving 241 6.7 Industry Perspectives 244 6.8 Notes and Extensions 248 6.9 Homework Problems 250 7 Motion Capture... 255 7.1 The Motion Capture Environment 257 7.2 Marker Acquisition and Cleanup 260 7.3 Forward Kinematics and Pose Parameterization 263 7.4 Inverse Kinematics 266 7.5 Motion Editing 273 7.6 Facial Motion Capture 279 7.7 Markerless Motion Capture 281 7.8 Industry Perspectives 290 7.9 Notes and Extensions 294 7.10 Homework Problems 297 8 Three-Dimensional Data Acquisition... 300 8.1 Light Detection and Ranging (LiDAR) 301 8.2 Structured Light Scanning 307 8.3 Multi-View Stereo 320 8.4 Registering 3D Datasets 329 8.5 Industry Perspectives 341
Contents ix 8.6 Notes and Extensions 346 8.7 Homework Problems 349 A Optimization Algorithms for Computer Vision... 353 A.1 Dynamic Programming 353 A.2 Belief Propagation 355 A.3 Graph Cuts and α-expansion 357 A.4 Newton Methods for Nonlinear Sum-of-Squares Optimization 360 B Figure Acknowledgments... 364 Bibliography... 367 Index... 393