Creating Photo and Video Effects Using Depth

Transcription

1 Session #WWDC18 Creating Photo and Video Effects Using Depth 503 Emmanuel Piuze-Phaneuf, Core Image Ron Sokolovsky, Video Engineering 2018 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

2 Portrait

3 Portrait Depth

4 NEW Portrait Depth Matte

5

6

7 Portrait Segmentation API

8 Portrait Matte

9 Portrait Matte Segmentation Background Foreground

10 Portrait Matte Segmentation Fuzzy mask 1.0 Background Foreground

11 Portrait Matte Segmentation Fuzzy mask 1.0 Fine details Background Foreground

12 Portrait Matte Segmentation Fuzzy mask 1.0 Fine details Background Many applications Foreground

13

14 ios 12

15 ios 12 Front and rear

16 ios 12 Front and rear Portrait

17 ios 12 Front and rear Portrait People

18 ios 12 Front and rear Portrait People Linear

19 ios 12 Front and rear Portrait People Linear No guarantee

20 Loading ImageIO let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex( imagesource, // CGImageSource 0, // Int kcgimageauxiliarydatatypeportraiteffectsmatte) // New ImageIO flag

21 Loading ImageIO let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex( imagesource, // CGImageSource 0, // Int kcgimageauxiliarydatatypeportraiteffectsmatte) // New ImageIO flag

22 Loading ImageIO let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex( imagesource, // CGImageSource 0, // Int kcgimageauxiliarydatatypeportraiteffectsmatte) // New ImageIO flag { kcgimageauxiliarydatainfodata: CFDataRef, // Image data kcgimageauxiliarydatainfodatadescription: CFDictionary, // Pixel buffer attributes kcgimageauxiliarydatainfometadata: CGImageMetadataRef // Image metadata }

23 Loading AVPortraitEffectsMatte let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex(...) let mattedata = AVPortraitEffectsMatte(fromDictionaryRepresentation: auxiliarydata)

24 Loading AVPortraitEffectsMatte let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex(...) let mattedata = AVPortraitEffectsMatte(fromDictionaryRepresentation: auxiliarydata)

25 Loading AVPortraitEffectsMatte let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex(...) let mattedata = AVPortraitEffectsMatte(fromDictionaryRepresentation: auxiliarydata)

26 Loading AVPortraitEffectsMatte let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex(...) let mattedata = AVPortraitEffectsMatte(fromDictionaryRepresentation: auxiliarydata) let buf = mattedata.mattingimage; // CVPixelBuffer let format = mattedata.pixelformattype; // OSType kcvpixelformattype_onecomponent8

27 Loading AVPortraitEffectsMatte let auxiliarydata = CGImageSourceCopyAuxiliaryDataInfoAtIndex(...) let mattedata = AVPortraitEffectsMatte(fromDictionaryRepresentation: auxiliarydata) let buf = mattedata.mattingimage; // CVPixelBuffer let format = mattedata.pixelformattype; // OSType kcvpixelformattype_onecomponent8

28 Capture AVCapturePhoto AVCaptureInput AVCaptureDevice AVCaptureSession

29 Capture AVCapturePhoto AVCaptureInput Check support AVCaptureDevice AVCaptureSession AVCapturePhotoOutput.isDepthDataDeliverySupported.isPortraitEffectsMatteDeliverySupported

30 Capture AVCapturePhoto AVCaptureInput Check support AVCaptureDevice Enable with Depth AVCaptureSession AVCapturePhotoOutput.isDepthDataDeliverySupported.isPortraitEffectsMatteDeliverySupported AVCapturePhotoSettings.isPortraitEffectsMatteDeliveryEnabled=true.isDepthDataDeliveryEnabled=true

31 Capture AVCapturePhoto AVCaptureInput Check support AVCaptureDevice Enable with Depth AVCaptureSession Get the matte AVCapturePhotoOutput.isDepthDataDeliverySupported.isPortraitEffectsMatteDeliverySupported AVCapturePhotoSettings didfinishprocessingphoto:.isportraiteffectsmattedeliveryenabled=true.isdepthdatadeliveryenabled=true AVCapturePhoto.portraitEffectsMatte

32 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts)

33 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts)

34 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts)

35 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts) let opts = [ CIImageOption.portraitEffectsMatteImage : matte ] [ctx writeheifrepresentationofimage(of:image to:url format:.rgba8 colorspace:cs options:opts error:&error]

36 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts) let opts = [ CIImageOption.portraitEffectsMatteImage : matte ] [ctx writeheifrepresentationofimage(of:image to:url format:.rgba8 colorspace:cs options:opts error:&error]

37 Loading and Saving Core Image let opts = [ CIImageOption.auxiliaryPortraitEffectsMatte: true ] let matte = CIImage(imageWithContentsOfURL:url options:opts) let opts = [ CIImageOption.portraitEffectsMatteImage : matte ] [ctx writeheifrepresentationofimage(of:image to:url format:.rgba8 colorspace:cs options:opts error:&error]

38 RGB Depth Matte RFC 4032 x x x 1512 FFC TrueDepth 3088 x x x 1160

39 RGB Depth Matte RFC 4032 x x x 1512 FFC TrueDepth 3088 x x x 1160

40 Demo Jupyter Notebook

41 Related Session Core Image: Performance, Prototyping, and Python Executive Ballroom Thursday 5:00PM

42 Real-Time Video Effects with TrueDepth Ron Sokolovsky, Video Engineering

43

44

45 TrueDepth stream Point clouds Backdrop

46 TrueDepth stream Point clouds Backdrop

47 Depth Map Minimum to maximum depth in frame Black pixels no depth

48

49 Video TrueDepth stream

50 The dark side TrueDepth stream

51 The dark side TrueDepth stream

52 How do you add a TrueDepth stream?

53 // Discover the TrueDepth camera let videodevicediscoverysession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtintruedepthcamera], mediatype:.video, position:.front) // Initialize capture device input let videodeviceinput = try AVCaptureDeviceInput(device: videodevicediscoverysession.devices.first!) // Add a depth data output to the session let depthdataoutput = AVCaptureDepthDataOutput() session.addoutput(depthdataoutput) depthdataoutput.setdelegate(self, callbackqueue: dataoutputqueue)

54 // Discover the TrueDepth camera let videodevicediscoverysession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtintruedepthcamera], mediatype:.video, position:.front) // Initialize capture device input let videodeviceinput = try AVCaptureDeviceInput(device: videodevicediscoverysession.devices.first!) // Add a depth data output to the session let depthdataoutput = AVCaptureDepthDataOutput() session.addoutput(depthdataoutput) depthdataoutput.setdelegate(self, callbackqueue: dataoutputqueue)

55 // Discover the TrueDepth camera let videodevicediscoverysession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtintruedepthcamera], mediatype:.video, position:.front) // Initialize capture device input let videodeviceinput = try AVCaptureDeviceInput(device: videodevicediscoverysession.devices.first!) // Add a depth data output to the session let depthdataoutput = AVCaptureDepthDataOutput() session.addoutput(depthdataoutput) depthdataoutput.setdelegate(self, callbackqueue: dataoutputqueue)

56 // Discover the TrueDepth camera let videodevicediscoverysession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtintruedepthcamera], mediatype:.video, position:.front) // Initialize capture device input let videodeviceinput = try AVCaptureDeviceInput(device: videodevicediscoverysession.devices.first!) // Add a depth data output to the session let depthdataoutput = AVCaptureDepthDataOutput() session.addoutput(depthdataoutput) depthdataoutput.setdelegate(self, callbackqueue: dataoutputqueue)

57 // Discover the TrueDepth camera let videodevicediscoverysession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtintruedepthcamera], mediatype:.video, position:.front) // Initialize capture device input let videodeviceinput = try AVCaptureDeviceInput(device: videodevicediscoverysession.devices.first!) // Add a depth data output to the session let depthdataoutput = AVCaptureDepthDataOutput() session.addoutput(depthdataoutput) depthdataoutput.setdelegate(self, callbackqueue: dataoutputqueue)

58 Disparity or Depth? Disparity usually yields better results Depth values are intuitive, but the error goes with Z 2 let disparitymap = depthmap.applyingfilter( CIDepthToDisparity, parameters: nil)

59 Resolution sessionpreset Aspect Ratio RGB TrueDepth.photo 4:3 1440x x480.hd1920x :9 1920x x360.hd1280x720 16:9 1280x x360.vga640x480 4:3 640x x480

60 Resolution sessionpreset Aspect Ratio RGB TrueDepth.photo 4:3 1440x x480.hd1920x :9 1920x x360.hd1280x720 16:9 1280x x360.vga640x480 4:3 640x x480

61 Test System Pressure High resolutions + fast frame rate + processing + duration AVCaptureDevice.SystemPressureState.Level Nominal Fair Serious Critical Shutdown

62 Monitor System Pressure and Adapt Prevent AVCaptureDevice shutdown Reduce frame rate when level is increasing: Nominal (30 fps) Serious (15 fps) Nominal (30 fps) Fair (24 fps) Serious (20 fps) Critical (15 fps) capturedevice.lockforconfiguration() capturedevice.activevideominframeduration = CMTimeMake(1, 15) // 15 fps capturedevice.unlockforconfiguration()

63 Holey depth map, Batman!

64 Filtered Depth Map Smoothed, spatially and temporally Holes filled, using RGB For photography/segmentation applications ios 12: spatial resolution of filtering improved from ios 11 depthdataoutput.isfilteringenabled = true // default

65

66

67 Using Raw Depth Data Use for point clouds, or real world measurements No depth value == 0 Watch out for averaging/downsampling depthdataoutput.isfilteringenabled = false

68 Depth Map Holes The TrueDepth camera detects most objects up to about five meters Low reflectivity materials absorb light

69

70

71

72

73 Depth Map Holes Reflectivity Specular materials reflect light only from certain directions

74

75

76

77

78 Depth Map Holes Reflectivity Specular Outdoor conditions are more challenging

79 Outdoor

80 Outdoor

81 Depth Map Holes Reflectivity Specular Outdoor Shadow from parallax between projector and camera

82 Shadows

83 Shadows

84 Shadows

85 Demo TrueDepthStreamer

86 TrueDepth stream Point clouds Backdrop

87 TrueDepth stream Point clouds Backdrop

88 Depth Map Z = func(u, V)

89 Depth Map Z = func(u, V) Point Cloud (X, Y, Z)

90 Depth Map Z = func(u, V) Point Cloud (X, Y, Z)

91 From Depth Map to Point Cloud Intrinsics Matrix X = (U - X0) Z fx Y = (V - Y0) Z fy f s x x f y y

92 From Depth Map to Point Cloud Intrinsics Matrix X = (U - X0) Z fx Y = (V - Y0) Z fy f s x x f y y matrix_float3x3 intrinsicsdepthdata.cameracalibrationdata.intrinsicmatrix;

93 RGB-D Infrared Camera Front RGB Camera

94 RGB-D Infrared Camera Front RGB Camera

95 Metal Graphic Shaders

96 Vertex Shader Point location Z = func(u, V) Transform to (X, Y, Z) Reproject with view matrix

97 Vertex Shader Point location Z = func(u, V) Transform to (X, Y, Z) Reproject with view matrix

98 Fragment Shader Point color Get vertex info Discard holes (Z == 0) Apply color from RGB frame at vertex coordinates

99 Fragment Shader Point color Get vertex info Discard holes (Z == 0) Apply color from RGB frame at vertex coordinates

100 TrueDepth stream Point clouds Backdrop

101 TrueDepth stream Point clouds Backdrop

102 Demo Backdrop

103 Per-Frame, Real-Time Processing Detect a face Create binary foreground mask, smooth, and upscale Upscale foreground to background and blend

104 Per-Frame, Real-Time Processing Detect a face Create binary foreground mask, smooth, and upscale Upscale foreground to background and blend Resize background to video resolution once: Not upscaling foreground Blending at lower resolution

105 Getting Face Metadata Until a face is detected, use default face depth of 0.5 meters Transform center of face to depth map coordinates, and get its depth metadataoutput.metadataobjecttypes = [AVMetadataObject.ObjectType.face]

106 Binary Foreground Mask Threshold = face depth meters Binary mask: Foreground = 1 Background = 0

107 Binary Mask

108 Binary Mask

109 Gaussian Blur Background Foreground Mask Blurred DepthMask GaussianBlur

110 Blur Radius

111 Blur Radius

112 Gamma Adjust Background Foreground Mask Blurred Gamma=2 DepthMask GaussianBlur Gamma = 2

113 Gamma Adjust Background Foreground 1 Gamma= Mask Blurred DepthMask GaussianBlur Gamma = 2 Gamma = 0.5 Gamma=0.5

114 Adjusting Gamma

115 Adjusting Gamma

116 Binary Mask Alpha matte let matte = depthmask.applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).applyingFilter( CIBicubicScaleTransform, parameters: inputscale : RGBwidth/depthWidth]) Mask

117 Binary Mask Alpha matte let matte = depthmask.applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).applyingFilter( CIBicubicScaleTransform, parameters: inputscale : RGBwidth/depthWidth]) Mask Blurred

118 Binary Mask Alpha matte let matte = depthmask.applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).applyingFilter( CIBicubicScaleTransform, parameters: inputscale : RGBwidth/depthWidth]) Mask Blurred Gamma

119 Binary Mask Alpha matte let matte = depthmask.applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).applyingFilter( CIBicubicScaleTransform, parameters: inputscale : RGBwidth/depthWidth]) Mask Blurred Gamma Upscale to RGB

120 Clamp Before Filtering Avoiding edge softening let matte = depthmask.clampedtoextent().applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).applyingFilter( CIBicubicScaleTransform, parameters: [ inputscale : RGBwidth/depthWidth])

121 Crop After Filtering Keeping the original extent let matte = depthmask.clampedtoextent().applyingfilter("cigaussianblur", parameters: ["inputradius": 20]).applyingFilter("CIGammaAdjust", parameters: ["inputpower": 2]).cropped(to: depthmask.extent).applyingfilter( CIBicubicScaleTransform, parameters: [ inputscale : RGBwidth/depthWidth])

122 Blending Foreground with Background let output = rgbimage.applyingfilter( CIBlendWithMask", parameters: ["inputmaskimage": matte, inputbackgroundimage : background])

123 Blending Foreground with Background let output = rgbimage.applyingfilter( CIBlendWithMask", parameters: ["inputmaskimage": matte, inputbackgroundimage : background])

124 TrueDepth Stream 640x480 depth map registered to video at 30 fps

125 TrueDepth Stream 640x480 depth map registered to video at 30 fps

126 TrueDepth Stream 640x480 depth map registered to video at 30 fps

127 More Information AVCapture Lab Technology Lab 4 Thursday 3:00PM

128