Deep Learning in Image and Face Intrinsics

Transcription

1 Deep Learning in Image and Face Intrinsics Theo Gevers Professor in Computer Vision: University of Amsterdam Co-founder: Sightcorp 3DUniversum Scanm WIC, Eindhoven, February 24th, 2018

2 Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics

3 Intrinsic images: Old School Approach

4 Reflection Model C m b n, s e f d m n, s, v C s e c f C d e n s v f C c surface albedo illumination object surface normal illumination direction viewer s direction sensor sensitivity specular scene & viewpoint invariant scene dependent object shape variant scene dependent viewpoint variant camera dependent surface dependent

5 For giving the red, green and blue sensor response under white light. Consider normalized color: Then: Also holds for g and b. In the sequel: Consider the body reflection term: rgb Photometric Invariance: Proof d f, em C C b WB s n B G R B b B G R G g B G R R r,, d f d f d f, em d f, em r B G R b R b n s n s WB WB WB WB B G R C,, d f k C C

6 Colour Invariance / rgb space Original RGB image 3D plot of RGB image rgb image 3D plot of rgb image

7 Colour Invariance - Summary shadows shading highlights ill. intensity ill. Colour I R,G,B r,g,b c1,c2,c Hue l1,l2,l m1m2m no invariance + invariance

8 Summary: Intrinsic Images Old School Reflection models help to understand the image formation process. Color invariance at the pixel. White light source or successful color constancy. Be aware of instabilities d f c e, m d f e, m C C s C b, v n s n s

9 Intrinsic images: New School Approach AT THE PIXEL

10 ConvNet: Encoder-Decoder Style Model Decoders Encoder S I R

11 Intrinsic images: New School End-to-end solution but no insight in why and how it works. Still outperformed by old school methods in terms of accuracy. Simple loss functions based on reconstruction loss ignoring the image formation model.

12 ConvNet: Encoder-Decoder Style

13 ShapeNet Rendered Dataset

14 Qualitative Results: MIT Dataset

15 IntrinsicNet: Conclusion Advantage + Integrated image formation model + Can capture most of the color information & shading + Colors are more vivid and eliminates most of the color artifacts + Fast and more memory efficient To be improved - Edge sharpness Next Retinex revisited

16 Intrinsic images: Old School Approach RETINEX DERIVATIVES

17 C C f And narrow-band filters: Then: Assuming body reflection Therefore, we have: Von Kries Model B G R C d f e m C C b,, for, s n B G R C e, m C C C b,, for n s R R b e, m R n s B G b e, m G n s B B b e, m B n s

18 Solving the Inverse Problem I Image e ˆ d 2 Laplacian I 2 eˆ 2 t Thresholding T d 2 Lightness l x, y Find lightness lx,y from tx,y: Poisson s Equation x l t 2 lx, y tx, y 2 y We have to find gx,y which satisfies x, y tu, vgx u y v l, x, y tx, y gx y l, dudv

19 Intrinsic Images Original Re-integrated Colour Channels Edge Maps of Channels Shadow Edges Removed

20 Intrinsic images: New School Approach Retinex-inspired ConvNets

21 RetiNet: Retinex-Inspired ConvNet RGB Differentiation R RGB Integration x, y gx y t, RGB d 2 eˆ 2 S RGB t x, y gx, y d R S

22 ShapeNet Rendered Dataset

23 Qualitative Results: MIT Dataset

24 Summary Intrinsic image decomposition End-to-end solution. Full testing on intrinsic image decomposition using synthetic and real data. Full model including light source and highlights. Raw data preprocessing and augmentation.

25 Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics

26 Basic Emotions

27 Face Analysis by Deep Learning Frame Happy

28 Sightcorp

29 FaceWarehouse: a 3D Facial Expression Database for Visual Computing Chen Cao Yanlin Weng Shun Zhou Yiying Tong Kun Zhou State Key Lab of CAD&CG, Zhejiang University Michigan State University

30 Face Analysis: The Face Model

31 Face Analysis: The Camera Model

32 Face Analysis

33 Co nv 16 Co nv 32 Co nv 64 Co nv 128 Co nv 256 Co nv 256 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 Decoder- Shading Decoder -Specular Decoder- Albedo Encoder CNN Model: Supervised

34 Illumination Changes IMAGE ALBEDO SHADING SPECULARITY

35 Illumination, Geometry, Albedo Changes IMAGE ALBEDO SHADING SPECULARITY

36 CNN Model: Multi-Task Unsupervised Encoder Decoder rendering

37 Consumer Applications

38 Question: Is your presidency fake?

39 A.I. Face Technology Powered by:

40 Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics

41 The True Depth camera of the iphone X: Face ID Apple s new TrueDepth camera for the iphone X sure works by using a projector to cast 30,000 dots on your face, which it then reads with an infrared camera like the Microsoft Kinect.

42 Consumer Applications: AR and VR Hololens Lenovo phab 2 pro

43 Retail & Fashion: 3D scans allow proper sizing for online shopping no more wasting time and product returns! Scan Faces and Bodies using Mobile Devices Virtual fitting, recommendation and tailoring 3DU has developed deep learning based automatic recognition algorithms which automatically detects 50K facial points eyes, temples, nose bridge, pupils, mouth corners etc. and body parts arms, waist, shoulders, legs etc..

44 Real Estate & Home Deco : Digital visits and try furniture at finger tips - refurnish your house from your couch Scan Rooms - Houses Virtual Tours, Measurements and Refurnishing

45 Scanm B.V.: Real Estate: Low-Cost Solution! Scan Rooms - Houses Equipment 3D Scanning for Everyone Lenovo + Rotator

46 Deep Learning in Computer Vision Object classification, detection and segmentation Optical flow Color constancy Intrinsic image decomposition 3D and slam Human behavior analysis Etc