Motion Capture CS 448D: Character Animation Prof. Vladlen Koltun Stanford University
History: Rotoscope Trace animated character over recorded actor, frame-by-frame Invented by Max Fleicher in 1915 and used for Betty Boop Used by Walt Disney for human characters in Snow White and the Seven Dwarfs in 1937
History: Mike the Talking Head (1988) Live digital puppetry performance at SIGGRAPH 1988 Promotional event for Silicon Graphics Character driven in real time by human actor Images from Robertson, Mike, the talking head, Computer Graphics World, 1988
History: Jurassic Park (1993) Dinosaurs animated using stop-motion armatures equipped with sensors that measured joint angles. Computer graphics models driven with keyframes obtained from armatures. More details in Dinosaur input device, CHI 95.
History: Late 90s Motion capture takes off in Hollywood: Titanic, The Mummy,...
Uses of Motion Capture Film, broadcast Games, virtual environments Live performance Biomechanics, kinesiology, sports science, medicine Robot control Human-Computer Interaction Miscellaneous (Andrew Ng s helicopter tracking)
Types of Motion Capture Electromagnetic Electromechanical Inertial Optical Marker-based (Passive, Active) Optical Markerless
Electromagnetic motion capture Transmitters establish magnetic fields within space Sensors determine position and orientation Data at each sensor, has to be transmitted over cables or wirelessly Pros: 6DOF, no occlusion, unique sensor identification Cons: Limited range, noise, interference from metal objects in the environment, extremely costly sensors No need for post-processing if environment is carefully controlled More popular in the past, especially for performance animation Companies: Ascension Image from Bodenheimer et al., The process of motion capture: dealing with the data, 1997
Electromechanical motion capture Exo-skeleton worn over subject Rigid rods connected by potentiometers Pros: Real-time recording, high accuracy, affordable, self-contained, no occlusion, unique sensor identification Cons: Restrictive, restricted, no global position Originally used with stop-motion animation Companies: Animazoo Image from Animazoo
Inertial motion capture Inertial gyroscopes embedded in suit Pros: Real-time recording, high accuracy, affordable, self-contained, portable, no occlusion, unique sensor identification large number of simultaneous subjects Cons: Restricted, no global position Hybrid systems possible Companies: Animazoo, Xsens Image from Xsens
Optical Most popular method currently Surrounding stationary cameras see markers Sensing at cameras Cameras have to be calibrated Pros: Highly adaptable Cons: Limited resolution (capture volume, dynamic range, number of subjects), controlled lighting, occlusion, correspondence problems, etc. Partial solution: Active markers Companies: Vicon, MotionAnalysis (passive), PhaseSpace (active) Image from House of Moves
Optical Markerless No markers, just calibrated cameras A detailed scan of the character Research prototypes Pros: Tracks minute surface deformations Cons: Currently not ready for reliable deployment. Sometimes fails to reconstruct character. Hard to recover from failure. Images from de Aguilar et al., Performance Capture from Sparse Multi-view Video, 2008
Motion Capture Pipeline The rest of the lecture is on optical motion capture with passive markers, the most widespread variety at present Process: 1. Camera calibration 2. Subject calibration 3. Capture 4. 2D marker identification 5. 3D position reconstruction 6. Skeletal motion reconstruction
Camera calibration Record a number of markers in 3D Detect corresponding positions in camera s image plane Solve linear system. Compute least-squares solution to deal with noise and discretization Handled by software provided by motion capture equipment manufacturer
Subject calibration Give identifiers to markers Fit skeleton to markers: Express joint positions in terms of marker positions Bone lengths Joint limits Useful trick: Place two markers on opposite sides of joint, express joint as average Otherwise express joint in terms of at least three markers, to compensate for lack of orientation Try to place markers so they move rigidly with joints
2D marker identification Handled by software provided by motion capture equipment manufacturer. Hugely problematic: Markers disappear, appear, merge, swap. Need to examine data frame-by-frame. The most time-consuming stage.
3D position reconstruction In every frame, identify all cameras that see a particular marker Construct corresponding lines in 3D Find least squares solution Cameras must be synchronized. Can use clap-board. Must have enough cameras at enough angles. At least 8 for full-body motion, sometimes over 30 if multiple subjects. Occlusion still a problem in many settings.
Skeletal motion reconstruction Place joints based on 3D marker positions and subject calibration Markers slip and slide. Other errors. Leads to inaccurate joint positions that do not fit bone lengths. End effectors often end up in undesirable configurations. (Feet penetrate floor or skate.) Need to correct using inverse kinematics.
Motion Capture Debate: Pro Captures natural movement in minute detail. Biomechanic subtleties and dynamics. Human actors can be directed at a high level. Rapid iterations. Detailed emotional states and subtle movement styles. Recent uses: Star Wars prequels, Lord of the Rings, Polar Express, Beowulf, King Kong
Motion Capture in King Kong (2005) video Video from www.kongisking.net
Motion Capture Debate: Con Mapping motion to character with different biomechanics and dynamics forfeits the realism advantage. Characters with different proportions move differently. Sometimes character movement combines aspects that cannot coexist in reality. (Shrek, Godzilla.) Animated characters are perceived differently. Exaggerated motions can be more appropriate. (Discussed in Lasseter paper next week.) Animators still spend significant work modifying mocap data, adding subtle details and expressivity. Not used: Pixar movies, Shrek, Kung Fu Panda, etc.
Motion Capture Debate: Con This is only my personal opinion, but motion capture is not animation. To me animation is an art. It's putting a little bit of yourself into the character. With motion capture, you're capturing a live action performance. And, I think, the difference in the quality of motion between a live action performance and good key frame is huge. Motion capture has a very realistic look. For example, if someone is clapping their hands, motion capture will give you a very accurate recreation of someone clapping. But, when you actually look at it, I think it would feel kind of stiff and lifeless, even though it's an exact replica of how a human claps. I think that to really sell the idea of someone clapping, you want to exaggerate it a little bit. You want to dilate the motion, so you really feel the contact of the hands slapping together. Glenn McQueen, Pixar (responsible for animation of Woody in Tory Story and Boo in Monsters, Inc. )
Motion Capture Debate: Pro Lower budget productions Interactive applications
Research theme Motion capture specifies the entire configuration (all joint angles) at every frame. No key frames, no information on what is important and what is not. A flat description of motion in complete detail. Very unwieldy if want to change something! Techniques needed for editing and combining motion capture data in various ways. Also for mapping data onto different characters. Major theme in research papers we will read starting next week.