Motion Capturing for Virtual Humans. Sophie Jörg, Frank Hülsken & Christian Eckes

Transcription

1 Motion Capturing for Virtual Humans Sophie Jörg, Frank Hülsken & Christian Eckes

2 The Virtual Human Project Aim: Development of virtual characters as autonomous dialogue partners Runtime: Funded by the German Federal Ministry of Education and Research with 7 million (Dr. Reuse) Cooperation of seven partners throughout Germany

3 2nd Application Scenario: Interactive TV Game Show Dialog between 5 Virtual & Human Agents Virtual Experts Virtual Moderator 2 Human Candidates

4 Topic: World Cup 2006 in Germany Z WEIUND A CHTZIG M ILLIONEN B UNDESTRAINER Twenty-eight million national coaches Round 1: Interruption of Video Sequences Prediction of the Final Result Round 2: Line-up of the German National Team

5 The Consortium Seven Partners throughout Germany Consortium Manager Fraunhofer Institute for Media Communication, Sankt Augustin German Research Center for Artificial Intelligence, Saarbrücken Fraunhofer Institute for Computer Graphics, Darmstadt OTLO VR Systems, Rostock Charamel, Cologne rmh, new media gmbh, Cologne Computer Graphics Center, Darmstadt

6 Main Tasks of the IMK Modelling Full body and cloths modelling 3D reconstruction of faces, creation of blend shapes Animation and Control Motion capturing and control of body movements Facial tracking and face editor Interaction Robust speech recognition Gesture recognition

7 Modelling of Body and Cloths Modelling based on photos from a real person Geometry (10 K polygons) Texture

8 3D-Scanning of the Head Structured light approach Shape Cam from Eyetronics ShapeSnatcher and in-house software MEdit to optain patches Two shots with a flash projecting a fine grid 3D geometry with a polarisation filter diffuse texture

9 Reconstruction of the Head Iterated closest point (ICP) algorithm to align patches Volumetric range image processing (VRIP) algorithm to obtain a continuous mesh heavily reduced head (3K polygons) High-quality head (70K polygons)

10 Motion Capture at Metricminds Vicon8 RT system 18 Vicon-M cameras (120 fps) 7x4x3m capture volume CyberGloves from IMK

11 Motion Capture at the IMK - Tasks Create animations for virtual characters Realistic human animations for body and hands More than 90 gestures per character reduce the time of producing for a single gesture mocap-session: visual feedback in realtime postproduction: automating processes Gestures accompanying speech a small capture area is possible slow capture frequency is possible

12 Basic Ideas icone as capture area (6m ) capture with ART cameras (60 fps) usage of 6-DoF bodies usage of cybergloves

13 Construction of the Bodies and the Suit Fabrication of 16 6-DoF bodies four to five sensors per body bodies have to be different from each other Suit flexible fixation on body usage of 12 bodies to capture one person

14 Measurement of the Body Posture 9 A.R.T. Cameras AR-Track GmbH, Munich, Germany integrated pattern recognition unit 60 fps positioned to cover the whole capture area three syncgroups 5 ART-1 cameras ontop of icone display wall 4 ART-2 cameras in the rear

15 Evaluation of the Camera Data Software Dtrack calibration of bodies calibration of cameras calculates positions and rotations

16 Measurement of the Finger Posture Two cybergloves (Immersion, USA) Calibration in Motionbuilder

17 Animation of the Character MoDTrack: IMK PlugIn to stream DTrack data to Motionbuilder Calibration of the actor (TStance) Preparation of the character Usage of the actor as input for the character Visualisation in real-time

18 Overview of the Motion Capture System Measuring of the body posture: cameras and sensors Evaluation of the images from the cameras: dtrack virtual character performer Measuring of the finger posture: cybergloves Animation of the character: Motionbuilder

19 During the Session Four persons: performer, Motionbuilder-Operator, dtrack-operator, director Saved data: Motionbuilder, dtrack, camera 137 takes, 4h45 of capturing, 3h of preparation and calibration

20 Postproduction Usage of batch processes and scripts: plot animations from one character to another use dtrack data correct or improve postures set offsets for the fingers fix the feet two days of manual postproduction

21 Advantages and Disadvantages Advantages real-time visualisation adaptable suit bodies are recognised after they have been lost optical Motion Capture: high precision, nearly no jitter low costs for an optical system body and finger movements captured simulanously Disadvantages bodies can be lost low frequency (60 fps), no fast moves possible small capture area Gloves still wired

22 Current Work in Character Animation at IMK Evaluation of the system Improvement of the calibration methods before capture fast recalibration during capture Automatisation of the postproduction Use system for Facial MoCap and animation Use MoCap to evaluate and bootstrap markerless gesture recognition

23 Markerless MoCap and Gesture recognition Task Recognize user gestures Approach Use of 3D-Camera Swissranger by CSEM Zürich able to measure depth by in-pixel phase measure (Time-Of-Flight) 160x124 16Bit depth ~ 30Hz Subtasks Recogize pointing gestures in real-time Recognize hand pose

24 Exploitation IMK has developed two products: MaDTrack : Plugin for A.R.T.'s Dtrack and Autodesk MAYA MoDTrack : Plugin for A.R.T.'s Dtrack and Autodesk MOTIONBUILDER Distributed through our license partner A.R.T. FH Erfurt (Prof. Grimm) has become beta tester Interests from local animation groups to perform MoCap KHM, Animation department, Prof. Lilas, Cologne FhG FIT for research project

25 Thank you for your attention! More information: Contact: Tina Walber, Tobias Kilian, Florian Hühnermann, Simon Schüller, Raimund Cürten, Sophie Jörg, Frank Hülsken, Konstantin Biatov, Christian Eckes, Joachim Köhler and Martin Reiser and former members of the team Jörg Unterberg, Kai Hüttemann, Marion Langer, Martin Stöcker, Pia Breuer, Saskia Groenewegen and Johannes Strassner