Three-dimensional Proxies for Hand-drawn Characters

Transcription

1 Three-dimensional Proxies for Hand-drawn Characters 1 Eakta Jain 1 Yaser Sheikh 2 Moshe Mahler 1,2 Jessica Hodgins 1 Carnegie Mellon University 2 Disney Research Pittsburgh 1

2 Hand-drawn animation 3D CG animation 2

3 Hand-drawn animation 3D CG animation 2

4 Differences between hand animation and computer animation Hand animator s workdesk 3

5 Differences between hand animation and computer animation Hand animator s workdesk 3D animation software 3

6 Input 4

7 Input 4

8 5

9 5

10 Hand-drawn character 3D proxy 6

11 Geometry Motion Hand-drawn character 3D proxy 6

12 Challenge: Inferring third dimension Possible 3D points 2D points Camera center Image plane 7

13 Challenge: Composite motion ambiguity 8

14 Challenge: Composite motion ambiguity 8

15 Challenge: Composite motion ambiguity Character turns, camera in place 8

16 Challenge: Composite motion ambiguity Character turns, camera in place Camera turns, character in place 8

17 Challenge: Artistic license Frame #1 Frame #40 9

18 Challenge: Artistic license Frame #1 Frame #40 9

19 Challenge: Artistic license Frame #1 Frame #40 9

20 Challenge: Artistic license Frame #1 Frame #40 Overlaid 9

21 Challenge: Artistic license Frame #1 Frame #40 Overlaid 9

22 Three-dimensional proxies with different levels of detail Single points (3D markers) 3D Polygonal shapes 10 Joint hierarchy based skeleton

23 Three-dimensional proxies with different levels of detail Single points (3D markers) 3D Polygonal shapes 10 Joint hierarchy based skeleton

24 Past work (e)(b) Warped model Final in frame 3-D model (c)(f)edges model art (e) Warped model et al. Li et al.(f) Final fram Petrovic et al.(d) Edges indavis Correa et al. ating one texture mapped frame. Figure 2: The process of creating one texture mapped frame. (2003) (2000) (1998) (2003) (d)hand-drawn Edges in artart (a) Figure 14: Motion of walking cartoon character retargeted to 3 model Figure Tracking Baloo s dance and retargeting a flower. used later as the input to the warp. Sectio edges can be usedexisting later asproduction the input topipeline the warp. Section Our method fitssafely into the for11: cel ani-of4.4 edges can beto safely c Disney shows[6, how to specify theand edges 2-D drawing that correspond mation 18]. Steps (a) (f) on arethe stages in the current digital shows how to specify the edges on the 2-D drawing that corres to the edges foundwith on the model. stage between them. We production process, the 3-D ink-and-paint to the edges found on the 3-D model. are offering, as an alternative to the constant colors of the ink-andpaint stage, a process that applies complex textures to the drawings. 4.1 Silhouette Detection 4.1 Silhouette Detection The problem of applying textures to hand-drawn artwork poses a challenge: the line must be interpreted somevisible kind ofsilhouette shape. Figure a 15:scheme Motion of jumping character retargeted to 3Dsilho mode In this section we art describe a scheme for as finding In this section we describe for finding visible 12: acquire Broom Retargeting Sequence. (The source is not shown Given set of black paper, the computer must and aborder edges lines in a on 3-Dwhite model represented by afigure polygon mesh. and edges inin a 3-D model represented by a polygon m here It is the famous sequence from the border Sorcerer s Apprentice 9, we show some of the key poses. Disney s at These least a features primitiveare model 3-D formstoconveyed by the handart. likelyfortothe correspond features in Fantasia) the These features are likely to correspond to features in the Figure 14 shows some final rendered frames in the retarget This information necessary if we are to 3-D effects forthe sequence. Again, please consult for more information. drawn line art; is such correspondences areprovide the primary input to drawn line art; such correspondences are the thevideo primary input In the next example, we capture a cartoon character jumping in single line running through the character, that represents the overthewarp texture such as self-occlusion and foreshortening. (See, for we describe in Section 5. We also allow thealluser to specify warp we describe in Section We for also allow theanduser way that is 5. impossible an average human, retargetto ontosp t force and direction of each drawing. Generally, before drawing same 3D model. Again, contour capture is used to estimate the m example, the difference in occlusion betweenonfigures and 2ebut oranimator the2b full character, an in the line-of-action helpdrawing them directly on the 3-D model, model markers by drawing them directly the 3-D model, it draws model markerstoby tion parameters. Figure 15 shows some frames from the retarget determine the position of the character. By simply changing the thewould foreshortening shown to in have Figure Note all thatmodel withline-of-action the constant 11 sequence. be cumbersome to7.) specify marker curves would be cumbersome to have to specify all model marker c making it more curved, sloped or arched in a difthe video shows all results for the different target media. Our e ferenteffects direction the essence of the drawing can be changed. colors of thethus, traditional ink-and-paintconstruct stage, these 3-D areentire this way. we automatically model markers for all this way. Thus, we automatically markers f periments in cartoonconstruct capture providemodel an opportunity to test the ca We experiment with using the line-of-action as the source in order Johnston (2002) Bregler et al. (2002)

25 User Input Virtual markers Limb bounding boxes Color coded body parts 12

26 User Input Virtual markers Limb bounding boxes Color coded body parts Motion capture segment with similar depth information, time-warped via Dynamic Time Warping 12

27 Camera Estimation 3D points Motion capture poses Estimated camera Image plane Hand drawings

28 3D points Back-projected rays 2D points z-depth Image plane Estimated camera 14

29 arg min(e a (x)+e m (x)+e s (x)) x 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

30 arg min(e a (x)+e m (x)+e s (x)) x input-matching 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

31 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

32 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

33 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

34 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

35 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

36 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

37 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

38 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

39 arg min(e a (x)+e m (x)+e s (x)) x input-matching depth prior smoothing term Linear system: least squares solution 3D points 2D points Back-projected rays z-depth Estimated camera Image plane 14

40 Depth compositing Rendered image 15

41 Depth compositing Rendered image Depth map for rendered image 15

42 Depth compositing Rendered image Depth map for rendered image Depth map for hand drawing 15

43 Depth compositing Rendered image Depth map for rendered image Depth map for hand drawing Composited frame 15

44 16

45 16

46 17

47 17

48 18

49 18

50 19

51 20

52 21

53 22

54 22

55 23

56 23

57 24

58 24

59 25

60 25

61 Summary 3D Polygonal shapes 26

62 Summary 3D Polygonal shapes 26

63 Summary 3D Polygonal shapes 3D Joint hierarchy skeleton 26

64 Summary 3D Polygonal shapes 3D Joint hierarchy skeleton 26 Hand animator modifies physical simulation?

65 Summary 3D Polygonal shapes 3D Joint hierarchy skeleton 26 Hand animator modifies physical simulation? Learn cartoon physics?

66 Extra Slides 27

67 Camera Estimation Camera rotation and translation ρ(i) =(θ x (i),θ y (i),θ z (i),t x (i),t y (i),t z (i)) T ρ (i) = arg min(w 1 e g + w 2 e l + w 3 e o + w 4 e s ) ρ 28

68 Camera Estimation Camera rotation and translation ρ(i) =(θ x (i),θ y (i),θ z (i),t x (i),t y (i),t z (i)) T ρ (i) = arg min(w 1 e g + w 2 e l + w 3 e o + w 4 e s ) ρ Geometric projection error e g = K/2 t= K/2 Hand drawings x i+t x proj i+t where x proj i+t = M i Xi+t Motion capture poses 28

69 arg min x (λ a e a (x)+λ m e m (x)+λ s e s (x)) input-matching motion prior smoothing term e a = x ij x proj ij x proj ij = M i X w ij x ij M i X w ij =0 CM i M = X w ij Y w ij Z w ij 1 m T 1 m T 2 =0 m T 3 29

70 arg min x (λ a e a (x)+λ m e m (x)+λ s e s (x)) input-matching motion prior smoothing term e a = x ij x proj ij e m = m T 3 X w ij mt 3 X ij x proj ij = M i X w ij m T 3 X w ij = mt 3 X ij x ij M i X w ij =0 CM i M = X w ij Y w ij Z w ij 1 m T 1 m T 2 =0 m T 3 29

71 arg min x (λ a e a (x)+λ m e m (x)+λ s e s (x)) input-matching motion prior smoothing term e a = x ij x proj ij e m = m T 3 X w ij mt 3 X ij x proj ij = M i X w ij m T 3 X w ij = mt 3 X ij x ij M i X w ij =0 CM i M = X w ij Y w ij Z w ij 1 m T 1 m T 2 m T 3 =0 e s = X w ij Xw (i+1)j I I X w ij X w (i+1)j = 0 29

72 arg min x (λ a e a (x)+λ m e m (x)+λ s e s (x)) input-matching motion prior smoothing term e a = x ij x proj ij e m = m T 3 X w ij mt 3 X ij x proj ij = M i X w ij m T 3 X w ij = mt 3 X ij x ij M i X w ij =0 CM i M = X w ij Y w ij Z w ij 1 m T 1 m T 2 m T 3 =0 e s = X w ij Xw (i+1)j I I X w ij X w (i+1)j WA i X w i = b i = 0 29

73 30

74 30