COMPUTER VISION FOR VISUAL EFFECTS

Transcription

1 COMPUTER VISION FOR VISUAL EFFECTS Modern blockbuster movies seamlessly introduce impossible characters and action into real-world settings using digital visual effects. These effects are made possible by research from the field of computer vision, the study of how to automatically understand images. Computer Vision for Visual Effects will educate students, engineers, and researchers about the fundamental computer vision principles and state-of-the-art algorithms used to create cutting-edge visual effects for movies and television. The author describes classical computer vision algorithms used on a regular basis in Hollywood (such as blue screen matting, structure from motion, optical flow, and feature tracking) and exciting recent developments that form the basis for future effects (such as natural image matting, multi-image compositing, image retargeting, and view synthesis). He also discusses the technologies behind motion capture and three-dimensional data acquisition. More than 200 original images demonstrating principles, algorithms, and results, along with in-depth interviews with Hollywood visual effects artists, tie the mathematical concepts to real-world filmmaking. is an Associate Professor in the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute. His current research interests include computer vision problems related to modeling 3D environments with visual and range imagery, calibration and tracking problems in large camera networks, and machine learning problems for radiotherapy applications. Radke is affiliated with the NSF Engineering Research Center for Subsurface Sensing and Imaging Systems; the DHS Center of Excellence on Explosives Detection, Mitigation and Response (ALERT); and Rensselaer s Experimental Media and Performing Arts Center. He received an NSF CAREER award in March 2003 and was a member of the 2007 DARPA Computer Science Study Group. Dr. Radke is a senior member of the IEEE and an associate editor of IEEE Transactions on Image Processing.

2

3 Computer Vision for Visual Effects RICHARD J. RADKE Rensselaer Polytechnic Institute

4 cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Mexico City Cambridge University Press 32 Avenue of the Americas, New York, NY , USA Information on this title: / This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2013 Printed in China by Everbest A catalog record for this publication is available from the British Library. Library of Congress Cataloging in Publication Data Radke, Richard J., 1974 Computer vision for visual effects /. pages cm Includes bibliographical references and index. ISBN Cinematography Special effects Data processing. 2. Computer vision. I. Title. TR858.R dc ISBN Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate.

5 You re here because we want the best and you are it. So, who is ready to make some science? Cave Johnson

6

7 Contents 1 Introduction Computer Vision for Visual Effects This Book s Organization Background and Prerequisites Acknowledgments 7 2 Image Matting Matting Terminology Blue-Screen, Green-Screen, and Difference Matting Bayesian Matting Closed-Form Matting Markov Random Fields for Matting Random-Walk Methods Poisson Matting Hard-Segmentation-Based Matting Video Matting Matting Extensions Industry Perspectives Notes and Extensions Homework Problems 51 3 Image Compositing and Editing Compositing Hard-Edged Pieces Poisson Image Editing Graph-Cut Compositing Image Inpainting Image Retargeting and Recompositing Video Recompositing, Inpainting, and Retargeting Industry Perspectives Notes and Extensions Homework Problems Features and Matching Feature Detectors Feature Descriptors Evaluating Detectors and Descriptors 136 vii

8 viii Contents 4.4 Color Detectors and Descriptors Artificial Markers Industry Perspectives Notes and Extensions Homework Problems Dense Correspondence and Its Applications Affine and Projective Transformations Scattered Data Interpolation Optical Flow Epipolar Geometry Stereo Correspondence Video Matching Morphing View Synthesis Industry Perspectives Notes and Extensions Homework Problems Matchmoving Feature Tracking for Matchmoving Camera Parameters and Image Formation Single-Camera Calibration Stereo Rig Calibration Image Sequence Calibration Extensions of Matchmoving Industry Perspectives Notes and Extensions Homework Problems Motion Capture The Motion Capture Environment Marker Acquisition and Cleanup Forward Kinematics and Pose Parameterization Inverse Kinematics Motion Editing Facial Motion Capture Markerless Motion Capture Industry Perspectives Notes and Extensions Homework Problems Three-Dimensional Data Acquisition Light Detection and Ranging (LiDAR) Structured Light Scanning Multi-View Stereo Registering 3D Datasets Industry Perspectives 341

9 Contents ix 8.6 Notes and Extensions Homework Problems 349 A Optimization Algorithms for Computer Vision A.1 Dynamic Programming 353 A.2 Belief Propagation 355 A.3 Graph Cuts and α-expansion 357 A.4 Newton Methods for Nonlinear Sum-of-Squares Optimization 360 B Figure Acknowledgments Bibliography Index