The VES Handbook of Visual Effects

' The VES Handbook of Visual Effects Industry Standard VFX Practices and Procedures Edited By Jeffrey A. Okun Susan Zwerman Co-Editors: Scott Squires Toni Pace Carstensen Kevin Rafferty AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Focal Press is an imprint of Elsevier

CONTENTS V CONTENTS About the VES Foreword xxiii xxv Chapter 1 Introduction 1 Visual Effects and Special Effects 1 Visual Effects 2 Special Effects 2 Why Use Visual Effects? 2 The Creation of Visual Effects 3 A Bit of Visual Effects History 4 Optical Printers 7 Electronics for Camera Control 8 The Digital Age 10 Unintended Consequences: Where Does Creativity End? 13 Chapter 2 Pre-Production/Preparation 17 Overview 17 Breaking Down a Script Budgeting 18 Ballpark Budget 19 More Detailed Budgets 20 Bidding 20 Plate Photography 21 Temp Screenings 22 Reviewing Bids 22 Contracts 23 Rebidding during Shooting 23 Rebidding in Post 23 Monitoring the Budget and Schedule 23 Keeping the Budget Down 23 Working with the Director and Producer 25 Demo Reel 25 The Meeting 26 Moving Forward 26 Production Departments 28 Production Design 28 Camera 29 Special Effects 30 Stunts 30

VI CONTENTS Wardrobe 31 Makeup 32 Production 32 Visual Effects 33 Editorial 34 Locations 35 Production Meeting 37 Designing Visual Effects Shots 37 Guidelines for Directors 38 Storyboards 38 Previs 39 Objective of the Shot 40 Concept Art 40 Continuity 40 Photorealism 41 Original Concepts 41 Budget 41 Reality and Magic 42 Camera Angles 42 Framing 42 Scale 43 Detail 43 Speed 43 Scaled Images 44 Depth of Field 44 Sequence of Shots 44 Camera Motion 44 Less Is More 45 Action Pacing 45 CG Characters 45 Creatures and Character Design 46 Powers of 10 Shots 46 Visual Effects Techniques 47 Technique Considerations 47 Additional Suggestions for DeterminingTechniques 49 What is Previs? 53 Development of PrevisTechniques 55 History and Background 55 The Applications of Previs: Who Benefits From it and How? 57 Postvis 62

CONTENTS VII Cautions and Suggestions for Good Practice 66 The Perils of Previs! 66 Passing the Work On 70 The Role of the VFX Supervisor in Previs 71 The Future of Previs: AdvancedTechniques 72 Environment Input 72 Character Input 72 Camera Input 73 GamingTechniques 74 On-Set Previs 74 Lighting Previs 74 3D Stereo Previs 75 Virtual Production 75 Camera Angle Projection 76 Drawing What the Lens Sees 76 Chapter 3 Acquisition/Shooting 79 Working on Set 79 CommonTypes of Special Effects 85 What Are Special Effects? 85 A Brief History of Special Effects 85 The Special Effects Supervisor 85 Working with the Visual Effects 86 Visual Effects in Service to SFX 86 Special Effects Design and Planning 86 Storyboards and Previs 86 The Elements: Rain, Wind, and Snow and Ice 87 Smoke, Fire, and Pyrotechnics 88 Mechanical Effects 90 Flying Wire Rigs and Stunts 91 Safety 92 Front and Rear Projection Systems for Visual Effects 93 Rear Projection 94 Front Projection (Blue or Green Screens and Picture Imagery) 94 Rear Projection Equipment 95 Front Projection Equipment 96 Large-Area Emissive Displays (LCD, Plasma, and Jumbotron Screens) 96 Greenscreen and Bluescreen Photography 96 Function of the Backing Green, Blue, or Red 97

viii CONTENTS Fabric and Paint 97 Backing Uniformity and Screen Correction 98 How to Expose a Greenscreen Shot and Why 103 Setting Screen Brightness 105 Floor Shots, Virtual Sets 110 Foreground Lighting 111 Controlling Spill Light 116 Lighting Virtual Sets 117 Tracking Marks on the Screen 118 On-Set Preview 118 Camera for Bluescreen or Greenscreen Photography 119 Negative Scanning and Digital Conversion 122 The Processed Foreground 124 Underwater Photography 124 Working with the Cinematographer 125 The Alpha Channel 125 Compositing Software 125 On-Set Data Acquisition 127 Camera Report 128 Tracking Markers 131 Props for the Actors 133 Cyberscanning 135 Digital Photos 136 Lidar/Laser Scanning 137 Lens Distortion Charts 137 HDRI and Chrome Balls 139 Lidar Scanning and Acquisition 140 On-Set 3D Scanning Systems 145 On-Set Data Acquisition 145 3D Scanning Systems 146 Prepping the Actors for Scanning 147 Scanning Props or Cars 148 Review AIIThat Has Been Scanned 148 3D Scanning Post-Production 149 Lighting Data 150 Gathering Lighting Data 150 Beware of False Savings! 151 Goals 151 Using Conventional Still Cameras 155 Shooting Considerations 156

CONTENTS IX Clean Plates 158 Shooting the Clean Plate 159 Locked-Off Camera 160 Moving Camera 160 Other Issues 161 Postprocess 162 Alternates without Clean Plates 162 Other Uses for Clean Plates 163 Monster Sticks 163 On-Set Animation Capture: Witness Cam 169 Wireless Nonvideo Motion Capture 169 Factors Affecting Witness Cameras 170 Dealing with the Data in Post-Production 171 Real-Time Matchmoving and Cam era-tracking Data 172 Triangulation as a Method of Recording Camera Data 177 Camera/Subject Positional Information 178 Basics:TheTool Kit 178 Basics: Nodal Point 180 Photographic Reference 195 Shooting Video as a Reference 197 Rules, Setup, andtesting 199 Digital Cinematography 204 The Viewing System 218 The Recording System 219 Filming Live-Action Plates to be Used in VFX 222 Camera Position (Station Point) 223 Angle of View 223 Lighting Considerations 223 Camera Tilt 224 Background Quality 224 Moving Plates 224 Scouting the Camera Positions 225 A Case Study 225 Camera Cars 226 Camera Car Safety Issues 226 Purpose-Built Crane Cars 227 Vibration and Camera Stabilization 227 Road Speed 228 Precautions 228 Panoramic Rigs 229

X CONTENTS On the Water 229 Air to Air 230 Cable Systems 231 Shooting Elements for Compositing 232 What Is an Element? 232 Stock Footage 233 Types of Elements 233 Generic versus Shot-Specific Elements 234 Determining Element Needs 234 Cheating 235 Backgrounds 236 Black Backgrounds 237 Line-Up 237 Camera Format Considerations 239 Assorted Methods for Shooting Elements 241 High-Speed Photography and Filming Elements 246 High-Speed Photography 246 Cameras 246 Technicians 249 Director of Photography 249 Lighting 250 Application 250 Locking Down the Camera 252 Video Assist 253 Post 253 Supervising Motion Control 253 What Is Motion Control? 254 Performance Choreography 255 Multiple-Pass Photography 256 Scaling 256 Import and Export of Camera Move Data 257 The Data 258 Types of Motion Control Systems 259 Motion Control Software 264 Camera Types 264 Sync and Phase 264 Dealing with Production 265 Acquisition of Motion/Still PhotographicTextures for Mapping onto CG 266

CONTENTS XI Panoramic Backgrounds 267 Tiled Stills 268 MotionTiling and Synchronous Plates 269 Practical Considerations 271 Stills fortextures and Lighting 272 Stop-Motion 273 Evolution of Stop-Motion Photography 274 TheTime Required to Shoot in Stop-Motion 275 Preparation before Shooting 276 Setting Up a Shooting Space for Stop-Motion 277 Use of Motion Control in Stop-Motion 279 Useful Caveats 280 Evolution of a Shot 282 Use of Stop-Motion in Visual Effects 283 What are Miniatures? 284 What Are Miniatures and Why AreThey Used? 284 The Advantages and Disadvantages of Miniature Effects 285 Execution of Planning 287 Design and Integration within the Film 288 Photographic Continuity 288 Miniature Categories 288 Scale Determinations 289 Forced Perspective Miniatures 292 In-Camera Compositing of Miniatures with Full-Scale Live-Action Actors 292 Nodal Pans andtilts 296 Forced Perspective 296 Mixed Scales 296 The Fabrication of Miniature Effects 297 Scale and Purpose Requirements of a Miniature 297 Construction Materials 299 The Paint Process 303 Previsualization and Miniatures 304 The Basic StandardTools 304 The Incorporation of Mechanical, Practical, and Pyrotechnic Effects with Miniatures 304 Water Effects 306 Fire, Explosives, and Collapsing 307 Shooting Locations and Conditions 308

XII CONTENTS Photography of Miniature Effects: Motion Control 308 What Is Motion Control and What Are Its Benefits to Miniature Photography? 308 Execution andtechnique Using Previs 311 Photography 312 Photography of Miniature Effects: High-Speed Photography 314 Depth of Field 316 Pyrotechnics 316 Smoke, Fire, and Water 317 The Use of Miniatures in the Digital World 319 Special Effects for Miniatures 322 Shrinking Reality 322 Scale Considerations 324 Camera 324 Water 326 Rain 327 Fire 328 Smoke 330 Explosions 332 Breakaways 333 Chapter 4 Performance and Motion Capture 335 What is Motion Capture? 335 OtherTypes of Motion Capture 336 Resistance-BasedTechnologies 336 Electromechanical-BasedTechnologies 336 Computer Vision 337 Time-of-FlightTechnologies 338 Is Motion Capture Right for a Project? 338 Preparing for Motion Capture 341 What Can Be Captured 341 What Cannot Be Captured 342 Technology Considerations 346 Marker Placement 350 Marker Placement on Faces 351 Marker Masks 353 Hardware 353 The Strobe 354 Markers 355 Lenses 356

CONTENTS XIII Filter 358 Charge-Coupled Device 359 Onboard Processor 361 Inputs/Outputs 362 Setup 362 Balance 362 Software 363 Acquisition 363 Post-Processing 365 Reconstruction 366 Labeling 366 Rigid Bodies 367 Cleaning 368 Solving Motion Capture 368 Facial Capture 370 Capture Techniques 371 Markerless Motion Capture 375 Stabilization 376 The Face Model 377 Facial Rigging 378 Facial Action Coding System 381 Real-Time Motion Capture 384 Limitations 384 Line of Sight 384 Markers 385 Solving 385 Visualization 385 Alternate Technologies 385 Chapter 5 Stereoscopic 3D 387 How 3D Works 387 Accommodation and Convergence 387 Interaxial Separation 388 Toe-in versus Horizontal ImageTranslation 390 Parallax or Depth Budget 391 Positive and Negative Parallax 392 Floating Windows 393 Fix It in Post 396 Stereoscopic Design 396 The Emerging Grammar of 3D 397

XJV CONTENTS Creative Use of Depth 397 Previsualization 400 Avoiding Painful 3D 400 The Aesthetic of Scale 401 Cutting for 3D 403 Designing for Multiple Release Formats 404 Immersion-Based versus Convergence-Based Stereo 404 Virtual 3D Photography 406 Pros and Cons of Virtual 3D Photography 406 Multiple Camera Rigs 407 The 3D Camera Rig 409 Implementing Convergence 409 Manipulating the Screen Surround 410 Special Cases for Virtual 3D Photography 413 2D to 3D Conversion 414 Depth Creation Preparation 414 Visual Analysis of 2D Depth Cues 415 Main Artistic Stages of 2D-to-3D Conversion 416 Major 2D-to-3D Conversion Workflows 421 Special Cases 422 Transparencies 422 Reflections 422 Re-Projection Mapping Method Workflow 422 Pixel Displacement or Pixel Shifting 425 Minor 2D-to-3D Conversion Workflows 426 Automatic Conversion 426 Temporal Offset 426 DynamicTemporal Offset 427 Is "Real" Always Right? 427 3D Stereoscopic Visual Effects 427 Prepping for thethird Dimension 429 Shooting thethird Dimension 430 Visual Effects in thethird Dimension 431 Photographed Elements 432 Accuracy and Attention to Detail 433 Artistic Skill Level 433 Data Management 433 3D Stereo Digital Intermediate Workflow 434 Stereoscopic 3D Process Milestones 435 Viewing 3D Dailies 436

Projection Screens for 3D Stereoscopic Viewing 438 3D Editorial Processes 439 3D Stereoscopic Conforming 441 Data Workflow 444 2D versus 3D Grading 445 3D Stereo RealD Mastering Considerations 446 Geometry and Correction of Undesirable Binocular Disparity 446 3D Stereo Deliverables 447 Stereoscopic Window 448 The Stereoscopic Window 448 Placement of the Window in Relation to the 3D Scene 451 Window Violations 453 Window Placement Logic 455 How to Create a Stereoscopic Window 457 Producing Movies inthree Dimensions 458 Development Getting the Greenlight 458 Production-What to Look Out For 461 Chapter 6 Post-Production/Image Manipulation 465 Resolution and Image Format Considerations 465 Formats 465 Transport 467 Resolution 467 Image Compression/File Formats for Post-Production 469 Image Encoding 470 Still Image Compression 471 File Formats 478 4k+ SystemsTheory Basics for Motion Picture Imaging 482 Part 1: Resolution and Sharpness 483 Part 2: Into the Digital Realm 490 Part 3: Does 4k Look BetterThan 2k? 498 Part 4: Visual Perception Limitations for Large Screens 502 Film Scanning and Recording 506 Scanning 506 Recording 509 Color Management 512 TheThree Guidelines 512 Digital Color Image Encodings and Digital Cameras 512 Color Management at the Desktop 519 Bringing Color Management to Film Workflows 525

XVi CONTENTS Digital Intermediate 530 VFX Editorial 532 Editing within a Shot:The Art of Pre-Compositing 533 How It Came to Be 535 Modern Day Tracking and Disseminating of Information 536 As the Shot Changes 542 Wrapping It Up 543 Communication with Artists 545 Starting 545 Working withteams 546 Reference and Perspective 547 Shot Production 548 Communicating with Artists in Other Departments 548 Completion 550 The History of Compositing 550 The History of OpticalTechnique 550 Traveling Matte Technique and the Digital Age 551 Historical Notes on Blue Screen 553 Film versus Digital 560 Compositing of Live-Action Elements 562 Modern Digital Compositing 562 Capturing the Image to Composite 562 After the Shoot 564 Extractions and the Magic Bullet 564 Starting the Composite 567 Compositing Screen Elements in Stereoscopic 3D 569 Rotoscoping 569 Digital Painting and Plate Reconstruction 571 SceneTracking 573 Matte Paintings/Creative Environments 574 Matte Paintings: Art of the Digital Realm 574 What Is a Matte Painting? 575 Matte Painting Pioneers and History 575 Visualizing the Matte Painting Shot in Pre-Production 578 On-Set Supervision for Matte Painting Shots 578 Basic Skills andtricks of thetrade 581 Miniatures and Computer-Generated Sets 584 Finding the Best Frame 586 Re-Projected Photo Survey 588 The Need for Creative Compositing 589

CONTENTS XVII Chapter 7 Digital Element Creation 591 Digital Modeling 591 Overview:The Importance of Modeling 591 Types of Modeling 591 Model Data Types 593 Development of Models 595 Modeling for a Production Pipeline 599 Engineering Aspects for Polygons 600 Engineering Aspects for NURBS 601 Rigging and Animation Rigging 601 Rigging: What Is It? 601 Animation Rigging 602 Deformation Rigging 605 Texturing and Surfacing 608 The Importance oftexture Painting 608 Hard Surface Models 608 Creature Models 610 Types of Geometry:Their Problems and Benefits 610 Prepping the Model to Be Painted 612 Texture Creation 614 Various Other Map-Driven Effects 616 Texture Painting in Production 617 Model Editing 618 Digital Hair/Fur 619 Hair Generation Process 621 General Issues and Solutions 625 Digital Feathers 632 Morphology of Real Feathers 633 Modeling Digital Feathers 633 Similarities between Hair and Feathers 634 Differences between Hair and Feathers 635 Dynamics and Simulation 637 How Is a Simulation Created? 638 When Is Simulation Appropriate? 639 Tricks and Cheats 640 Important Considerations 640 Planning and Preparation 641 Software Solutions: A Broad Overview of Current Options 641 Particles 642 What Are Particle Systems? 642

XVIII CONTENTS The Next Step 642 The Birth of Particles 644 Creating Effects 644 Rigid-Body Dynamics 646 How Rigid-Body Dynamics Are Created 647 Potential Problems 647 Other Issues 648 Tricks for Getting It Done 649 Digital Lighting 649 Light in Reality and in Computer Graphics 649 Case Study of Reality Compared with Simple CG Simulation 651 Visual Sophistication through Texture Mapping 653 Physically Derived Shading Models 653 Beneath the Surface 655 Goals of Lighting in Visual Effects 656 Work Flow for Successful Creative Digital Lighting 658 TheTechnologies of Lights in Computer Graphics 660 Direct Lighting: Source to Surface to Camera 660 Negative Lights 661 Reflections 662 Photographed Reflections 663 Shadows 664 Image-Based Lighting 665 Rendering Occlusion 665 Ambient Occlusion 666 Reflection Occlusion 666 Average Light Direction Vectors: Bent Normals 667 Creating Light Sources from Environment Maps 667 Physically Based Rendering 668 Volumetric Lighting Effects 670 Shader Basics 672 What Are Shaders? 672 Shading Models 673 Bump and Displacement 673 Map-Based Shaders 675 Procedural Shaders 678 Shader Design 679 Antialiasing Considerations 680 3D Compositing 681 Color Representation 681

CONTENTS XIX Bit Depth and Dynamic Range 682 Mattes 684 Compositing CG 685 2.5D Compositing 689 Crowd Generation and SimulationTechniques 690 Live-Action Replication 690 Sprites 691 Computer-Generated Crowds 691 Modeling for Replication 691 Variation 691 Mesh Density 692 Animation Cycles for Replication 692 Motion Capture 693 Keyframe Animation 693 Dynamic Motion Synthesis 693 Behaviors and Crowd Control 693 CG Prosthetics and Actor Enhancements 694 On-SetTracking and Capture Considerations 695 Eye Enhancements 695 3DTechniques 695 2DTechniques 696 2.5DTechniques 696 Silhouette Changes 697 Re-Projection 697 3D Products, Systems, and Software 697 Digital Element Creation Process 697 3D Graphics Software 699 3DTracking 702 Special Effects 703 Rendering 704 Texturing 705 Chapter 8 Interactive Games 707 Films versus Games 707 Basic Differences 707 Most Important Aspect 708 Narrative Storytelling 709 Differences in Limitations 710 Cost of Iterations and Changes during Production 711 Category Types 712

XX CONTENTS FormatTypes 712 Transmedia Production DesignTechniques 714 Games and Platforms 717 GameTypes 718 Game Platforms 720 What Are Game Engines and How DoThey Function? 722 Working with a Game Engine 723 What Are the Production Steps in Creating a Game? 726 Marketing to an Audience 727 Visuals 728 Phases of Development 731 Game Cinematic Sequences 734 Noninteractive Scenes 734 Game Engine-Rendered NIS Scenes 735 The Design and Production Process 735 Chapter 9 Complete Animation 737 What is an Animation Project? 737 Full Animation versus Visual Effects 737 Difference Between Visual Effects and Animation 740 Production Pipelines 740 Production 741 A Survey and History of AnimationTechniques 747 Traditional Animation 748 Stop-Motion 751 Computer GraphicTechnology 754 Considerations for a Full CG-Animated Feature Pipeline 756 CG Feature Animation Pipeline 758 Managing an Animated Film 762 Film Management and Personal Style 762 Building BrainTrusts 763 Building the Core Creative Team 765 Writing and Visual Development 766 Working with a Studio 767 Facilities and Environment 769 Managing the Event 774 The Production Process: An Animator's Perspective 775 Working on CG-Animated Content in Live-Action Features 775 Planning the Process 775 Production 777 Character and Environment Interaction 780

CONTENTS XXI Chapter 10 Other Workflow Considerations 783 Virtual StudioTechnology 783 Analysis of a Production Workflow 784 From Workflow to Pipeline 786 Service Bureau versus In-House Requirements 792 Design of a Production Workflow 794 From Analysis to Design 795 Deploying a Production Workflow 801 From Design to Implementation 801 Infrastructure 807 Tracking Assets 809 What IsTask and Asset Tracking? 809 Commercial Task and Asset Tracking Systems 813 BuildingTask and Asset Tracking Systems 814 Scene Assembly 817 3D Scene Assembly 817 2D Scene Assembly (Compositing) 821 Working across Multiple Facilities 823 Images 824 Models 825 Texturing 825 Animation 825 Compositing 826 R&D 826 Acknowledgments 827 Appendix A: Charts and Formulas 829 Appendix B: Credits/Titles to Be Submitted in Accordance with VES Guidelines 835 Appendix C: Glossary 839 Index 895