What s New in ARKit 2

Transcription

1 Session #WWDC18 What s New in ARKit Arsalan Malik, ARKit Engineer Reinhard Klapfer, ARKit Engineer 2018 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

2 ARKit

3

4

5

6

7

8

9 ARKit Recap

10 Tracking

11 Tracking Scene Understanding

12 Tracking Scene Understanding Rendering

13

14 What s New in ARKit 2 Agenda NEW

15 What s New in ARKit 2 Agenda NEW Saving and Loading Maps

16 What s New in ARKit 2 Agenda NEW Saving and Loading Maps Environment Texturing

17 What s New in ARKit 2 Agenda NEW Saving and Loading Maps Environment Texturing Image Tracking

18 What s New in ARKit 2 Agenda NEW Saving and Loading Maps Environment Texturing Image Tracking Object Detection

19 What s New in ARKit 2 Agenda NEW Saving and Loading Maps Environment Texturing Image Tracking Object Detection Face Tracking Enhancements

20 Saving and Loading Maps

21 World Tracking Recap

22 World Tracking Recap Position and orientation

23 World Tracking Recap Position and orientation Physical scale

24 World Tracking Recap Position and orientation Physical scale 3D feature points

25 World Tracking Recap Position and orientation Physical scale 3D feature points Relocalization

26 Mapping

27 Mapping

28 ARWorldMap NEW

29 ARWorldMap NEW Mapping of physical 3D space

30 ARWorldMap NEW Mapping of physical 3D space Mutable list of named anchors

31 ARWorldMap NEW Mapping of physical 3D space Mutable list of named anchors Raw feature points and extent

32 ARWorldMap NEW Mapping of physical 3D space Mutable list of named anchors Raw feature points and extent Serialization

33 Persistence

34 Persistence

35 Persistence

36 Persistence

37 Multi-User Experiences World Map

38 Multi-User Experiences World Map World Map

39 // Saving and Loading World Maps // Retrieve world map from session object session.getcurrentworldmap { worldmap, error in guard let worldmap = worldmap else { showalert(error) return } }

40 // Saving and Loading World Maps // Retrieve world map from session object session.getcurrentworldmap { worldmap, error in guard let worldmap = worldmap else { showalert(error) return } }

41 // Saving and Loading World Maps // Retrieve world map from session object session.getcurrentworldmap { worldmap, error in guard let worldmap = worldmap else { showalert(error) return } } // Load world map and run the configuration let configuration = ARWorldTrackingConfiguration() configuration.initialworldmap = worldmap session.run(configuration)

42 Acquiring Good World Maps

43 Acquiring Good World Maps Important for relocalization

44 Acquiring Good World Maps Important for relocalization Multiple points of view

45 Acquiring Good World Maps Important for relocalization Multiple points of view Static, well-textured environment

46 Acquiring Good World Maps Important for relocalization Multiple points of view Static, well-textured environment Dense feature points on the map

47 Acquiring Good World Maps Important for relocalization Multiple points of view Static, well-textured environment Dense feature points on the map open class ARFrame : NSObject, NSCopying { open var worldmappingstatus: ARFrame.WorldMappingStatus }

48 World Mapping Status

49 World Mapping Status Not Extending Mapped Limited Available Not Available Limited Extending Mapped

50 Using WorldMappingStatus public enum WorldMappingStatus : Int { case notavailable case limited case extending case mapped } Share Map

51 Using WorldMappingStatus public enum WorldMappingStatus : Int { case notavailable case limited case extending case mapped } Share Map

52 Using WorldMappingStatus public enum WorldMappingStatus : Int { case notavailable case limited case extending case mapped } Share Map

53 Using WorldMappingStatus public enum WorldMappingStatus : Int { case notavailable case limited case extending case mapped } Share Map

54 Demo Saving and Loading Map Arsalan Malik, ARKit Engineer

55 World Tracking Enhancements NEW

56 World Tracking Enhancements NEW Saving and loading maps

57 World Tracking Enhancements NEW Saving and loading maps Faster initialization and plane detection

58 World Tracking Enhancements NEW Saving and loading maps Faster initialization and plane detection Robust tracking and plane detection

59 World Tracking Enhancements NEW Saving and loading maps Faster initialization and plane detection Robust tracking and plane detection More accurate extent and boundary

60 World Tracking Enhancements NEW Saving and loading maps Faster initialization and plane detection Robust tracking and plane detection More accurate extent and boundary Continuous autofocus

61 World Tracking Enhancements NEW Saving and loading maps Faster initialization and plane detection Robust tracking and plane detection More accurate extent and boundary Continuous autofocus New 4:3 video formats

62 Environment Texturing NEW

63 Realistic Rendering

64 Realistic Rendering Position and orientation

65 Realistic Rendering Position and orientation Scale

66 Realistic Rendering Position and orientation Scale Lighting

67 Realistic Rendering Position and orientation Scale Lighting Shadow

68 Realistic Rendering Position and orientation Scale Lighting Shadow Reflection of textures

69

70 Environment Texturing NEW

71 Environment Texturing NEW Gathers scene texture

72 Environment Texturing NEW Gathers scene texture Cube map representation

73 Environment Texturing NEW Gathers scene texture Cube map representation Used as reflection probe

74 Environment Texturing NEW Gathers scene texture Cube map representation Used as reflection probe Reflective objects

75 Environment Texturing

76 Environment Texturing

77 Environment Texturing

78 Environment Texturing

79 // Environment Texturing // Create world tracking configuration let configuration = ARWorldTrackingConfiguration() // Enable automatic environment texturing configuration.environmenttexturing =.automatic // Run the configuration session.run(configuration)

80 // Environment Texturing // Create world tracking configuration let configuration = ARWorldTrackingConfiguration() // Enable automatic environment texturing configuration.environmenttexturing =.automatic // Run the configuration session.run(configuration)

81 // Environment Texturing // Create world tracking configuration let configuration = ARWorldTrackingConfiguration() // Enable automatic environment texturing configuration.environmenttexturing =.automatic // Run the configuration session.run(configuration)

82 // Environment Texturing // Create world tracking configuration let configuration = ARWorldTrackingConfiguration() // Enable automatic environment texturing configuration.environmenttexturing =.automatic // Run the configuration session.run(configuration)

83 AREnvironmentProbeAnchor NEW

84 AREnvironmentProbeAnchor NEW Cube map (MTLTexture)

85 AREnvironmentProbeAnchor NEW Cube map (MTLTexture) Physical extent

86 AREnvironmentProbeAnchor NEW Cube map (MTLTexture) Physical extent Same lifecycle as other anchors

87 AREnvironmentProbeAnchor NEW Cube map (MTLTexture) Physical extent Same lifecycle as other anchors Integrated in ARSCNView

88 AREnvironmentProbeAnchor NEW Cube map (MTLTexture) Physical extent Same lifecycle as other anchors Integrated in ARSCNView Manual placement // Enable manual placement configuration.environmenttexturing =.manual

89 Demo Environment Texturing Arsalan Malik, ARKit Engineer

90 Image Tracking Reinhard Klapfer, ARKit Engineer

91 Image Detection Recap

92 Image Detection Recap Recognition of known static 2D images

93 Image Detection Recap Recognition of known static 2D images Position and orientation

94 Image Detection Recap Recognition of known static 2D images Position and orientation Integrated into world tracking

95 Image Detection Recap Recognition of known static 2D images Position and orientation Integrated into world tracking Supported by Xcode asset catalog

96 Image Tracking NEW

97 Image Tracking NEW Images don t need to be static

98 Image Tracking NEW Images don t need to be static

99 Image Tracking NEW Images don t need to be static Position and orientation for every frame

100 Image Tracking NEW Images don t need to be static Position and orientation for every frame

101 Image Tracking NEW Images don t need to be static Position and orientation for every frame Track multiple images simultaneously

102 Image Tracking NEW Images don t need to be static Position and orientation for every frame Track multiple images simultaneously ARImageTrackingConfiguration

103 Image Tracking ARReferenceImage ARReferenceImage ARReferenceImage File Asset Catalog

104 Image Tracking.detectionImages ARReferenceImage ARReferenceImage ARReferenceImage ARWorldTrackingConfiguration File Asset Catalog

105 Image Tracking.detectionImages ARReferenceImage ARReferenceImage ARReferenceImage ARWorldTrackingConfiguration ARImageTrackingConfiguration.trackingImages File Asset Catalog

106 Image Tracking.detectionImages.run(configuration) ARWorldTrackingConfiguration ARReferenceImage ARReferenceImage ARReferenceImage ARSession ARImageTrackingConfiguration.trackingImages File Asset Catalog

107 Image Tracking.detectionImages.run(configuration) ARWorldTrackingConfiguration ARReferenceImage ARReferenceImage ARReferenceImage ARSession ARImageTrackingConfiguration.trackingImages.currentFrame File Asset Catalog ARFrame ARFrame ARFrame

108 Image Tracking.detectionImages.run(configuration) ARWorldTrackingConfiguration ARReferenceImage ARReferenceImage ARReferenceImage ARSession ARImageTrackingConfiguration.trackingImages.currentFrame File Asset Catalog ARFrame ARFrame ARFrame ARImageAnchor

109 ARImageAnchor open class ARImageAnchor : ARAnchor, ARTrackable { public var istracked: Bool { get } open var transform: simd_float4x4 { get } open var referenceimage: ARReferenceImage { get } }

110 Using Good Images

111 Using Good Images

112 Using Good Images Distinct features Well textured Good contrast

113 Using Good Images Distinct features Well textured Good contrast

114 Using Good Images Distinct features Repetitive structures Well textured Uniform color regions Good contrast Narrow histogram

115 Asset Catalog in Xcode

116 Asset Catalog in Xcode

117 Configurations for Image Tracking

118 Configurations for Image Tracking ARWorldTrackingConfiguration

119 Configurations for Image Tracking ARWorldTrackingConfiguration Image anchors in world reference frame

120 Configurations for Image Tracking ARWorldTrackingConfiguration Image anchors in world reference frame Detected images can be tracked NEW

121 Configurations for Image Tracking ARWorldTrackingConfiguration Image anchors in world reference frame Detected images can be tracked NEW ARImageTrackingConfiguration NEW

122 Configurations for Image Tracking ARWorldTrackingConfiguration Image anchors in world reference frame Detected images can be tracked NEW ARImageTrackingConfiguration NEW Independent from world tracking

123 Configurations for Image Tracking ARWorldTrackingConfiguration Image anchors in world reference frame Detected images can be tracked NEW ARImageTrackingConfiguration NEW Independent from world tracking Position and orientation for every frame

124 // Image Tracking // Create an image tracking configuration let configuration = ARImageTrackingConfiguration() // Set of images to be tracked configuration.trackingimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

125 // Image Tracking // Create an image tracking configuration let configuration = ARImageTrackingConfiguration() // Set of images to be tracked configuration.trackingimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

126 // Image Tracking // Create an image tracking configuration let configuration = ARImageTrackingConfiguration() // Set of images to be tracked configuration.trackingimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

127 // Image Tracking // Create an image tracking configuration let configuration = ARImageTrackingConfiguration() // Set of images to be tracked configuration.trackingimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

128 // Image Tracking // Create an image tracking configuration let configuration = ARImageTrackingConfiguration() // Set of images to be tracked configuration.trackingimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

129 // Image Tracking // Create a world tracking configuration let configuration = ARWorldTrackingConfiguration() // Set of images to be detected configuration.detectionimages = [catphoto, dogphoto, birdphoto] // Optionally specify the maximum number of images to track in parallel NEW configuration.maximumnumberoftrackedimages = 2 // Run the session session.run(configuration)

130 Demo Augmented reality photo frame Reinhard Klapfer, ARKit Engineer

131 Object Detection NEW

132 Object Detection

133 Object Detection Detection of a known static 3D object

134 Object Detection Detection of a known static 3D object Objects need to be scanned first

135 Object Detection Detection of a known static 3D object Objects need to be scanned first Well-textured, rigid, non-reflective

136 Object Detection Detection of a known static 3D object Objects need to be scanned first Well-textured, rigid, non-reflective Position and orientation

137 Object Detection Detection of a known static 3D object Objects need to be scanned first Well-textured, rigid, non-reflective Position and orientation Integrated into world tracking

138 Object Detection ARReferenceObject ARReferenceObject ARReferenceObject File Asset Catalog

139 Object Detection.detectionObjects ARReferenceObject ARReferenceObject ARReferenceObject ARWorldTrackingConfiguration File Asset Catalog

140 Object Detection.detectionObjects.run(configuration) ARReferenceObject ARReferenceObject ARReferenceObject ARWorldTrackingConfiguration ARSession File Asset Catalog

141 Object Detection.detectionObjects.run(configuration) ARReferenceObject ARReferenceObject ARReferenceObject ARWorldTrackingConfiguration ARSession.currentFrame File Asset Catalog ARFrame ARFrame ARFrame

142 Object Detection.detectionObjects.run(configuration) ARReferenceObject ARReferenceObject ARReferenceObject ARWorldTrackingConfiguration ARSession.currentFrame File Asset Catalog ARFrame ARFrame ARFrame ARObjectAnchor

143 ARObjectAnchor open class ARObjectAnchor : ARAnchor { open var transform: simd_float4x4 { get } open var referenceobject: ARReferenceObject { get } }

144 // Object Detection // Create a world tracking configuration let configuration = ARWorldTrackingConfiguration() // Set of objects to be detected configuration.detectionobjects = [ancientbust, claypot] // Run the session session.run(configuration)

145 // Object Detection // Create a world tracking configuration let configuration = ARWorldTrackingConfiguration() // Set of objects to be detected configuration.detectionobjects = [ancientbust, claypot] // Run the session session.run(configuration)

146 // Object Detection // Create a world tracking configuration let configuration = ARWorldTrackingConfiguration() // Set of objects to be detected configuration.detectionobjects = [ancientbust, claypot] // Run the session session.run(configuration)

147 // Object Detection // Create a world tracking configuration let configuration = ARWorldTrackingConfiguration() // Set of objects to be detected configuration.detectionobjects = [ancientbust, claypot] // Run the session session.run(configuration)

148

149

150 Object Scanning

151 Object Scanning Accumulated scene information

152 Object Scanning Accumulated scene information Transform, extent, center

153 Object Scanning Accumulated scene information Transform, extent, center Supported by Xcode asset catalog

154 Object Scanning Accumulated scene information Transform, extent, center Supported by Xcode asset catalog ARObjectScanningConfiguration

155 Object Scanning Accumulated scene information Transform, extent, center Supported by Xcode asset catalog ARObjectScanningConfiguration Sample code available Scanning and detecting 3D objects

156

157

158

159

160

161

162

163

164 ARReferenceObject open class ARReferenceObject : NSObject, NSCopying, NSSecureCoding { open var name: String? open var center: simd_float3 { get } open var extent: simd_float3 { get } open var rawfeaturepoints: ARPointCloud { get } }

165 Face Tracking

166 Face Tracking Recap

167 Face Tracking Recap Robust face detection and tracking

168 Face Tracking Recap Robust face detection and tracking Position and orientation for every frame

169 Face Tracking Recap Robust face detection and tracking Position and orientation for every frame Fitted triangle mesh/arfacegeometry

170 Face Tracking Recap Robust face detection and tracking Position and orientation for every frame Fitted triangle mesh/arfacegeometry ARFaceAnchor

171 Directional Light Estimation

172 Directional Light Estimation Uses face as light probe

173 Directional Light Estimation Uses face as light probe Light intensity, direction, color temperature

174 Directional Light Estimation Uses face as light probe Light intensity, direction, color temperature Spherical harmonics coefficients

175 Blendshapes Recap

176 Blendshapes Recap Expressions tracked in real time

177 Blendshapes Recap Expressions tracked in real time Recognizes 50+ specific features

178 Blendshapes Recap Expressions tracked in real time Recognizes 50+ specific features Animation of virtual character

179 Blendshapes Recap Expressions tracked in real time Recognizes 50+ specific features Animation of virtual character

180 Animoji

181 Animoji

182 Face Tracking Enhancements NEW

183 Face Tracking Enhancements NEW Gaze tracking

184 Face Tracking Enhancements NEW Gaze tracking open class ARFaceAnchor : ARTrackable { open var lefteyetransform: simd_float4x4 { get } open var righteyetransform: simd_float4x4 { get } open var lookatpoint: simd_float3 { get } }

185 Face Tracking Enhancements NEW Tongue support

186 Face Tracking Enhancements NEW Tongue support

187 Face Tracking Enhancements NEW Tongue support extension ARFaceAnchor.BlendShapeLocation { public static let tongueout: ARFaceAnchor.BlendShapeLocation }

188 Summary

189 ARKit 2 Summary

190 ARKit 2 Summary Saving and loading maps

191 ARKit 2 Summary Saving and loading maps World tracking enhancements

192 ARKit 2 Summary Saving and loading maps World tracking enhancements Environment texturing

193 ARKit 2 Summary Saving and loading maps World tracking enhancements Environment texturing Image tracking

194 ARKit 2 Summary Saving and loading maps World tracking enhancements Environment texturing Image tracking Object detection

195 ARKit 2 Summary Saving and loading maps World tracking enhancements Environment texturing Image tracking Object detection Gaze and tongue

196 Building Blocks of ARKit AROrientationTrackingConfiguration ARWorldTrackingConfiguration ARFaceTrackingConfiguration NEW ARImageTrackingConfiguration ARObjectScanningConfiguration NEW ARSession

197 Building Blocks of ARKit AROrientationTrackingConfiguration ARFrame ARWorldTrackingConfiguration ARCamera ARFaceTrackingConfiguration ARLightEstimate NEW ARImageTrackingConfiguration ARReferenceImage NEW NEW ARObjectScanningConfiguration ARReferenceObject NEW ARSession ARWorldMap

198 Building Blocks of ARKit AROrientationTrackingConfiguration ARFrame ARAnchor ARWorldTrackingConfiguration ARCamera ARPlaneAnchor ARFaceTrackingConfiguration ARLightEstimate ARFaceAnchor NEW ARImageTrackingConfiguration ARReferenceImage ARImageAnchor NEW NEW NEW ARObjectScanningConfiguration ARReferenceObject ARObjectAnchor NEW NEW ARSession ARWorldMap AREnvironmentProbeAnchor

199 Building Blocks of ARKit AROrientationTrackingConfiguration ARFrame ARAnchor ARWorldTrackingConfiguration ARCamera ARPlaneAnchor ARFaceTrackingConfiguration ARLightEstimate ARFaceAnchor NEW ARImageTrackingConfiguration ARReferenceImage ARImageAnchor NEW NEW NEW ARObjectScanningConfiguration ARReferenceObject ARObjectAnchor NEW NEW ARSession ARWorldMap AREnvironmentProbeAnchor

200 More Information ARKit Lab Technology Lab 5 Tuesday 12:00PM Integrating Apps and Content with AR Quick Look Hall 2 Tuesday 4:00PM AR Get Together Technology Lab 10 Wednesday 6:15PM

201