CS448A: Experiments in Motion Capture. CS448A: Experiments in Motion Capture

Transcription

1 CS448A: Experiments in Motion Capture Christoph Bregler, Gene Alexander, CS448A: Experiments in Motion Capture Project Course Lectures on all key-algorithms 3-4 Homeworks (implement key-algorithms) Lot s of programming and experimenting! Final Group Project In-Class presentation of Paper from reading list Page 1 1

2 CS448A: Experiments in Motion Capture Lambsoft Motion Capture - Pipeline Tracking Model Fitting/Acquisition Kinematics Dynamics Learning Applications Page 2 2

3 isual Tracking isual Tracking: Unsolved for general settings isual Tracking Standard Techniques: Template Matching Edges / Shape / Color Background Subtraction Optical Flow New Challenges: Complex ariation Self Occlusion Noise ( Folds, Low Contrast) Page 3 3

4 Example Optical Flow local ambiguities Page 4 4

5 Kinematic Model ts Optical Flow/Feature tracking: no constraints Layered Motion: rigid constraints Articulated: kinematic chain constraints Nonrigid: implicit / learned constraints Tracking = ed Optimization - E() Page 5 5

6 Tracking = ed Optimization - E() Marker-Tracking Dense Flow ts = Subspaces - E() Analytically derived: Affine, Twist/Exponential Map Learned: Linear/non-linear Sub-Spaces Page 6 6

7 & Tracking = ed Optimization - - E() ed Function Minimization - t 1 t 2! " #$ % t... n = E() Page 7 7

8 1 1 ed Function Minimization - +,-. / +,-. 0 t 1 +,1. / +,1. 0 t 2 = E() ' (*) ( θ )... +,2!. / +#,2$. 0 t n 2D Translation: Lucas-Kanade 2D - +,-. / +,-. 0 t 1 +,1. / +,1. 0 t 2 = E() 3 ( dx, dy dx, dy ,2!. / +#,2$. 0 t n dx, dy Page 8 8

9 : 2D Affine: Bergen et al, Shi-Tomasi 6D t 1 4 5:7 8 45:7 9 t 2 4 5;!7 8 4#5;$7 9 t... n = E() < = a1, a2 x i + i a3, a4 yi dx dy 2D Affine: Bergen et al, Shi-Tomasi 6D - Page 9 9

10 D T U Z \ F Z ^ ] K-DOF 3D chain: Yamamoto, Bregler-Malik K-DOF - >?@A B >?@A C t 1 >?DA B >?DA C t 2 = E() G*H ( θ )... >?E!A B >#?E$A C t n Twist and Product of Exp. Maps orthographic projection ξl PRQ S M NONON N!M$NON X ξ X ξ[ α[ X ξ_ α _ Y I J δ δ t αw αk Μ ξ α Image World Page 10 10

11 } ~ Analytically derived ts - `acbed f gihkjelnm t 1 opcqer s oipkqernt t 2 uvkwyx z t... uivkw{xn n = E() ξ α... α Example Track - Page 11 11

12 Multiple iews Example Track -2- Page 12 12

13 Tracking + Acquistion of Kinematics - Acquire Kinematic Chains. Kathy Pullen Non-Rigid ed Spaces E(S) S = (p,,p ) 1 n Page 13 13

14 Non-Rigid ed Spaces Linear Subspaces: Small Basis Set Principal Components Analysis Nonlinear Manifolds: Mixture Models Nonrigid Examples Page 14 14

15 Nonrigid 3D Acquisition Human Dynamics Forces applied on masses Exploit models from Biomechanics Motor control / High-level processes Learning Statistical Models of Dynamics Motion-Phonemes : Movemes Laban Movement Analysis Page 15 15

16 Probabilistic Models / Bayesian Techniques Carl Lewis Sprint Movemes High-Level Categories Composition Dynamical Models Kinematic Model ts Image Sensors Probabilistic Models / Bayesian Techniques Bayesian Belief Networks Multiple Hypothesis Tracking Particle Filters Regularization Hidden Markov Models Markov Random Fields Kalman Filters Page 16 16

17 HM 3 Hybrid Models Language Dynamics Human Movements Complex Dance Motion Kinematics Features Standard ideo Simple (Animal) Motion High-Speed Multi-Cameras Active Sensing Hybrid Measurements Performance Capture based Animation Page 17 17

18 Rotoscope / Mocap: History Disney: Step-mother of Cinderella Rotoscope / Mocap: History Disney: Eleanor Audley Page 18 18

19 Rotoscope / Mocap: History Rebecca Allen / Twyla Tharp: The Catherine Wheel Paul Kaiser / Merce Cunningham: ƒ Biped Performance Capture based Animation Lambsoft Page 19 19

20 Performance Capture based Animation Lambsoft Performance Capture based Animation -> Popovic Files Page 20 20

21 CS448A: Homeworks 3 Homeworks on Tracking: Lucas-Kanade, Extensions, Kalman 1 Homework on Camera Calibration [ 1 Homework on Model Acquisition/Fitting ] CS448A: Paper Presentation Presentation for 20 min (depending on class size) Choose from Reading List or find it yourself Page 21 21

22 CS448A: Project Start thinking about it now In-class brainstorming session Proposal with 2 Milestones Group work CS448A: Project Ideas Full-Body Tracker: - Offline / Real-time - Resolution: 3D Blob, Kinematic Model Hand Tracker: - 2D vs 3D / how many cameras - Features: Color, Silhouette, Flow, Regions, Edges - Explicit kinematic model vs learned PCA model Face Tracker: - Markers / no Markers - same issues as above Page 22 22

23 CS448A: Project Ideas Generic 3D Model Acquisition based on Marker tracking Kinematic model fine-tuning Pan-Tilt tracker - Integrate with 3D blob tracker or other - Angle, Zoom, Focus,. Gesture Recognition - HCI applications -> iroom Application of Mocap to Animation -Full Body / Face CS448A: How to get in to: cs448a-staff@cs - Your name, , website - Probability of taking this class - Your background (Classes, childhood, etc ) - What you would like to do for a project website: graphics/courses/cs448a Page 23 23