From Art to Engine with Model I/O

Transcription

1 Session Graphics and Games #WWDC17 From Art to Engine with Model I/O 610 Nick Porcino, Game Technologies Engineer Nicholas Blasingame, Game Technologies Engineer 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

2 Model I/O Apple's toolkit for building pipelines Import and export 3D assets Geometry, materials, lighting, cameras, voxels Data format conversions Processing tools

3 Model I/O What's new? NEW Improved Importers Skinned Character Animation Blend Shapes Transform Stacks

4 Model I/O Intuitive Asset Traversal Format independent graph MDLAsset Logical Consistent Home Camera Sun Light Car Root Chassis Body Painted metal Suspension... Wheel Wheel Wheel Wheel Rubber Rubber Rubber Rubber

5 Art Assets Models, materials, animations Textures Scenes composed of many files

6 From Art to Engine Art asset is like source code Compiled for an engine

7 From Art to Engine UI based tools Easy the first few times Overwhelming during revision

8 How can the work be scaled?

9 Introducing the Pipeline Export the art Artwork Asset Engine-Ready Data Exporter

10 Introducing the Pipeline Transform the asset Artwork Asset Engine-Ready Data Tool

11 Introducing the Pipeline Load engine-ready data Artwork Asset Engine-Ready Data Engine

12 Introducing the Pipeline Export the art Artwork Asset Engine-Ready Data Exporter

13 Introducing the Pipeline Exporter Maya Asset Exporter Complex hierarchies of files Export script in the sample Art Asset

14 Introducing the Pipeline Exporter Maya Asset Exporter Complex hierarchies of files Export script in the sample Art USD

15 Universal Scene Description

16 Universal Scene Description Composition Shot Layer Car Wheel Master_1 Master_2 Tire Wall Pit Building Pit Building Race Track Race Track

17 Universal Scene Description Classes, Variations, and Overrides shadingvariant modelingvariant

18 Universal Scene Description Powerful Text Format, Fast Binary over "World" { over "anim" { over "chars" { def "Car" ( add references ) { color3f displaycolor = (0.9, 0, 0) } } } }

19 Introducing the Pipeline Scriptable command line tool Artwork Asset Engine-Ready Data Tool

20 Introducing the Pipeline Scriptable command line tool Asset Engine-Ready Data

21 Introducing the Pipeline Scriptable command line tool Repeatable Asset Engine-Ready Data

22 Introducing the Pipeline Scriptable command line tool Repeatable Consistent Asset Engine-Ready Data

23 Introducing the Pipeline Scriptable command line tool Repeatable Consistent Scriptable Asset Engine-Ready Data

24 Introducing the Pipeline Scriptable command line tool Repeatable Consistent Scriptable Scalable Asset Engine-Ready Data

25 Introducing the Pipeline Scriptable command line tool Repeatable Consistent Scriptable Scalable Asset Engine-Ready Data Composable

26 Introducing the Pipeline The sample Demonstrates principles Simplified data Uncompressed Good jumping-off point Asset Engine-Ready Data

27 Introducing the Pipeline Game engine Artwork Asset Engine-Ready Data Engine

28 Game Engine Simple renderer Single-pass forward renderer Physically-based shader Mesh instancing Skinned and animated meshes Multiple materials

29 Render Loop Set Transform Buffer Set Skinning Data Set Vertex Buffer C Set Pipeline State D Set Material Uniforms Set Fragment Textures Draw Indexed Primitive

30 The Pipeline Artwork Asset Engine-Ready Data Baker

31 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

32 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

33 Geometry + Transform A simple scene graph Transform Hierarchy Tree of nodes A Meshes and transform objects B C D

34 Geometry + Transform Transform hierarchy Moving the parent will move the children A B C D

35 Geometry + Transform Compactly encode transform hierarchy A B C D

36 Geometry + Transform Array of local transforms A B A C D Local Transforms

37 Geometry + Transform Array of local transforms A B A B C D Local Transforms

38 Geometry + Transform Array of local transforms A B A B C C D Local Transforms

39 Geometry + Transform Array of local transforms A B A B C C D D Local Transforms

40 Geometry + Transform Assign indices A B A 0 B 1 C C 2 D D 3 Local Transforms

41 Geometry + Transform Array of parent indices A B A 0 B 1 - C C 2 D D 3 Local Transforms Parent Indices

42 Geometry + Transform Array of parent indices A B A 0 B 1-0 C C 2 D D 3 Local Transforms Parent Indices

43 Geometry + Transform Array of parent indices A B A 0 B 1-0 C C 2 1 D D 3 1 Local Transforms Parent Indices

44 Geometry + Transform Array of parent indices A B A 0 B 1-0 C C 2 1 D D 3 1 Local Transforms Parent Indices

45 Geometry + Transform Array of mesh indices A B A 0 - B 1 0 C C D D 3 1 Local Transforms Parent Indices Mesh Indices

46 Geometry + Transform Array of mesh indices A B A 0 - B 1 0 C C D D Local Transforms Parent Indices Mesh Indices

47 Geometry + Transform Mesh data Descriptor Vertex Buffers Index Buffer Descriptor Vertex Buffers Index Buffer

48 Geometry + Transform Vertex buffers //for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { vertexdescriptors.append(mesh.vertexdescriptor) for vertexbuffer in mesh.vertexbuffers { let vertexbufferdata = Data(bytes: vertexbuffer.map().bytes, count: vertexbuffer.length)... } for submesh in mesh.submeshes! { if let indexbuffer = (submesh as? MDLSubmesh)?.indexBuffer { let indexbufferdata = Data(bytes: indexbuffer.map().bytes, count: indexbuffer.length)... } } }

49 Geometry + Transform Vertex buffers //for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { vertexdescriptors.append(mesh.vertexdescriptor) for vertexbuffer in mesh.vertexbuffers { let vertexbufferdata = Data(bytes: vertexbuffer.map().bytes, count: vertexbuffer.length)... } for submesh in mesh.submeshes! { if let indexbuffer = (submesh as? MDLSubmesh)?.indexBuffer { let indexbufferdata = Data(bytes: indexbuffer.map().bytes, count: indexbuffer.length)... } } }

50 Geometry + Transform Vertex buffers //for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { vertexdescriptors.append(mesh.vertexdescriptor) for vertexbuffer in mesh.vertexbuffers { let vertexbufferdata = Data(bytes: vertexbuffer.map().bytes, count: vertexbuffer.length)... } for submesh in mesh.submeshes! { if let indexbuffer = (submesh as? MDLSubmesh)?.indexBuffer { let indexbufferdata = Data(bytes: indexbuffer.map().bytes, count: indexbuffer.length)... } } }

51 Geometry + Transform Index buffer //for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { vertexdescriptors.append(mesh.vertexdescriptor) for vertexbuffer in mesh.vertexbuffers { let vertexbufferdata = Data(bytes: vertexbuffer.map().bytes, count: vertexbuffer.length)... } for submesh in mesh.submeshes! { if let indexbuffer = (submesh as? MDLSubmesh)?.indexBuffer { let indexbufferdata = Data(bytes: indexbuffer.map().bytes, count: indexbuffer.length)... } } }

52 Geometry + Transform Local transform var localtransforms: [matrix_float4x4] = [] // for every mdlobject in MDLAsset: if let transform = mdlobject.transform { localtransforms.append(transform.matrix) }

53 Geometry + Transform Local transform var localtransforms: [matrix_float4x4] = [] // for every mdlobject in MDLAsset: if let transform = mdlobject.transform { localtransforms.append(transform.matrix) }

54 Geometry + Transform Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Transform Data Local Transforms

55 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

56 Texture Paths + Materials Materials stored on MDLSubmesh Fetch properties referenced by shader Record texture paths and values MDLSubmesh MDLMaterial MDLMesh AO Base Color Metallic

57 Texture Paths + Materials // for every submesh: if let material = submesh.material { } for property in material.properties(with:<mdlmaterialsemantic>) { if property.type ==.string property.type ==.URL { // texture } else if property.type == <MDLMaterialPropertyType> { // uniform value } }

58 Texture Paths + Materials // for every submesh: if let material = submesh.material { } for property in material.properties(with:<mdlmaterialsemantic>) { if property.type ==.string property.type ==.URL { // texture } else if property.type == <MDLMaterialPropertyType> { // uniform value } }

59 Texture Paths + Materials Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Transform Data Local Transforms

60 Texture Paths + Materials Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Transform Data Local Transforms Material Data Material Uniforms Texture Paths

61 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

62 Instancing A mesh can be used multiple times A B C D

63 Instancing A mesh can be used multiple times A B C D E

64 Instancing A mesh can be used multiple times Why store it multiple times? A B C D E

65 Instancing Masters Masters MDLAsset has a masters array A B C D E

66 Instancing Model I/O support Masters MDLObject instance pointers refer to masters A B C D E

67 Instancing Masters A 0 - A AB 1 0 B C C D D E E Local Transforms Parent Indices Mesh Indices

68 Instancing Masters A 0 - A AB 1 0 B C C D D E E Local Transforms Parent Indices Mesh Indices

69 Instancing Masters A 0 - A AB 1 0 B C C D D E E Local Transforms Parent Indices Mesh Indices

70 Instancing Masters A 0 - A AB 1 0 B C C D D E E Local Transforms Parent Indices Mesh Indices Instance Count

71 Instancing Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Transform Data Local Transforms Material Data Material Uniforms Texture Paths

72 Instancing Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Instance Count Transform Data Local Transforms Material Data Material Uniforms Texture Paths

73 Demo Geometry, Materials, and Instancing Nicholas Blasingame, Game Technologies Engineer

74 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

75 Transform Animation Transforms that vary over time A B C D

76 Transform Animation A B C D

77 Transform Animation Sample the animations A t0, t1 tn B B C D D Animated Local Transforms

78 Transform Animation var localtransforms: [matrix_float4x4] = [] // for every mdlobject in MDLAsset: if let transform = mdlobject.transform { localtransforms.append(transform.matrix) }

79 Transform Animation var localtransforms: [matrix_float4x4] = [] // for every mdlobject in MDLAsset: if let transform = mdlobject.transform { localtransforms.append(transform.matrix) } if (transform.keytimes.count > 1) { let sampledxm = sampletimes.map{ transform.localtransform!(attime: $0) } animatedlocaltransforms.append(sampledxm)... }

80 Transform Animation var localtransforms: [matrix_float4x4] = [] // for every mdlobject in MDLAsset: if let transform = mdlobject.transform { localtransforms.append(transform.matrix) } if (transform.keytimes.count > 1) { let sampledxm = sampletimes.map{ transform.localtransform!(attime: $0) } animatedlocaltransforms.append(sampledxm)... }

81 Transform Animation Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Instance Count Transform Data Local Transforms Material Data Material Uniforms Texture Paths

82 Transform Animation Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Instance Count Transform Data Local Transforms Animated Local Transforms Material Data Material Uniforms Texture Paths

83 Baking Operations 1. Geometry + Transforms 2. Texture Paths + Materials 3. Instancing Data 4. Transform Animation 5. Skinning + Character Animation

84 Skinned Character Animation

85 Skinned Character Animation

86 Skinned Character Animation Mesh has geometry

87 Skinned Character Animation Embedded skeleton

88 Skinned Character Animation Vertex weighting to joints

89 Skinned Character Animation Vertex weighting to joints

90 Skinned Character Animation Vertex blending in shader {... float4 position = vertex.pos; packed_uchar4 jidx = vertex.jointindices; packed_float4 w = vertex.jointweights; float4 skinnedposition = w[0] * (palette[jidx[0]] * modelposition) + w[1] * (palette[jidx[1]] * modelposition) + w[2] * (palette[jidx[2]] * modelposition) + w[3] * (palette[jidx[3]] * modelposition); }...

91 Skinned Character Animation Input vertex attributes {... float4 position = vertex.pos; packed_uchar4 jidx = vertex.jointindices; packed_float4 w = vertex.jointweights; float4 skinnedposition = w[0] * (palette[jidx[0]] * modelposition) + w[1] * (palette[jidx[1]] * modelposition) + w[2] * (palette[jidx[2]] * modelposition) + w[3] * (palette[jidx[3]] * modelposition); }...

92 Skinned Character Animation Per vertex joint indices {... float4 position = vertex.pos; packed_uchar4 jidx = vertex.jointindices; packed_float4 w = vertex.jointweights; float4 skinnedposition = w[0] * (palette[jidx[0]] * modelposition) + w[1] * (palette[jidx[1]] * modelposition) + w[2] * (palette[jidx[2]] * modelposition) + w[3] * (palette[jidx[3]] * modelposition); }...

93 Skinned Character Animation Indexed bones for weight each vertex {... float4 position = vertex.pos; packed_uchar4 jidx = vertex.jointindices; packed_float4 w = vertex.jointweights; float4 skinnedposition = w[0] * (palette[jidx[0]] * modelposition) + w[1] * (palette[jidx[1]] * modelposition) + w[2] * (palette[jidx[2]] * modelposition) + w[3] * (palette[jidx[3]] * modelposition); }...

94 Skinned Character Animation A B C D E Root Skeleton Body Wheel 1 Wheel 2

95 Skinned Character Animation E Root Skeleton Body Wheel 1 Wheel 2

96 Skinned Character Animation E Root Skeleton Body Wheel 1 Wheel 2

97 Skinned Character Animation E Root 0 Skeleton Body Wheel Wheel 2 3

98 Skinned Character Animation E Root 0 - Skeleton Body Wheel Wheel Joint Parent Indices

99 Skinned Character Animation Bind Poses + Joint Indices E Root 0 - Body B1-1 Skeleton Wheel B2-1 Wheel B3-1 Joint Parent Indices Bound Joint Indices Inverse Bind Pose

100 Skinned Character Animation Bind Poses + Joint Indices E Root 0 - Body B1-1 Skeleton Wheel B2-1 Wheel B3-1 Joint Parent Indices Bound Joint Indices Inverse Bind Pose

101 Skinned Character Animation Bind Poses + Joint Indices E Root 0 - Body B1-1 Skeleton Wheel B2-1 Wheel B3-1 Joint Parent Indices Bound Joint Indices Inverse Bind Pose

102 Skinned Character Animation Time sample animation E Root 0 - Root t0 tn Body B1-1 Body t0 tn Skeleton Wheel B2-1 W1 t0 tn Wheel B3-1 W2 t0 tn Joint Parent Indices Bound Joint Indices Inverse Bind Pose Animation Clip

103 Skinned Character Animation // for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { if let skin = object.componentconforming(to:mdlskindeformercomponent.self) as? MDLSkinDeformerComponent { let inversebindtransforms = skin.jointbindtransforms().map{simd_inverse($0)} }...

104 Skinned Character Animation // for every mdlobject in MDLAsset: if let mesh = mdlobject as? MDLMesh { if let skin = object.componentconforming(to:mdlskindeformercomponent.self) as? MDLSkinDeformerComponent { let inversebindtransforms = skin.jointbindtransforms().map{simd_inverse($0)} }...

105 Skinned Character Animation Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Instance Count Transform Data Local Transforms Animated Local Transforms Material Data Material Uniforms Texture Paths

106 Skinned Character Animation Mesh Data Descriptors Vertex + Index Buffers Scene Composition Data Parent Indices, Mesh Indices Instance Count Transform Data Local Transforms Animated Local Transforms Animation Clips Material Data Skinning Data Material Uniforms Texture Paths Inverse Bind Transforms Joint to Palette Mapping Skeleton Parent Indices

107 Demo Instancing and Characters

108 Recap Artwork Asset Engine Ready Data USD

109 Enhancements Light Mapping

110 Enhancements Light Mapping UV Unwrapping

111 Enhancements Light Mapping UV Unwrapping Ambient Occlusion

112 Enhancements Image-based lighting

113 More Information

114 Related Information Managing 3D Assets with Model I/O WWDC 2015 Introducing Metal 2 WWDC 2017 What s New in SceneKit WWDC 2017

115