Deep Learning in Image and Face Intrinsics Theo Gevers Professor in Computer Vision: University of Amsterdam Co-founder: Sightcorp 3DUniversum Scanm WIC, Eindhoven, February 24th, 2018
Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics
Intrinsic images: Old School Approach
Reflection Model C m b n, s e f d m n, s, v C s e c f C d e n s v f C c surface albedo illumination object surface normal illumination direction viewer s direction sensor sensitivity specular scene & viewpoint invariant scene dependent object shape variant scene dependent viewpoint variant camera dependent surface dependent
For giving the red, green and blue sensor response under white light. Consider normalized color: Then: Also holds for g and b. In the sequel: Consider the body reflection term: rgb Photometric Invariance: Proof d f, em C C b WB s n B G R B b B G R G g B G R R r,, d f d f d f, em d f, em r B G R b R b n s n s WB WB WB WB B G R C,, d f k C C
Colour Invariance / rgb space Original RGB image 3D plot of RGB image rgb image 3D plot of rgb image
Colour Invariance - Summary shadows shading highlights ill. intensity ill. Colour I - - - - - R,G,B - - - - - r,g,b + + - + - c1,c2,c3 + + - + - Hue + + + + - l1,l2,l3 + + + + - m1m2m3 + + - + + - no invariance + invariance
Summary: Intrinsic Images Old School Reflection models help to understand the image formation process. Color invariance at the pixel. White light source or successful color constancy. Be aware of instabilities d f c e, m d f e, m C C s C b, v n s n s
Intrinsic images: New School Approach AT THE PIXEL
ConvNet: Encoder-Decoder Style Model Decoders Encoder S I R
Intrinsic images: New School End-to-end solution but no insight in why and how it works. Still outperformed by old school methods in terms of accuracy. Simple loss functions based on reconstruction loss ignoring the image formation model.
ConvNet: Encoder-Decoder Style
ShapeNet Rendered Dataset
Qualitative Results: MIT Dataset
IntrinsicNet: Conclusion Advantage + Integrated image formation model + Can capture most of the color information & shading + Colors are more vivid and eliminates most of the color artifacts + Fast and more memory efficient To be improved - Edge sharpness Next Retinex revisited
Intrinsic images: Old School Approach RETINEX DERIVATIVES
C C f And narrow-band filters: Then: Assuming body reflection Therefore, we have: Von Kries Model B G R C d f e m C C b,, for, s n B G R C e, m C C C b,, for n s R R b e, m R n s B G b e, m G n s B B b e, m B n s
Solving the Inverse Problem I Image e ˆ d 2 Laplacian I 2 eˆ 2 t Thresholding T d 2 Lightness l x, y Find lightness lx,y from tx,y: Poisson s Equation x 2 2 2 l t 2 lx, y tx, y 2 y We have to find gx,y which satisfies x, y tu, vgx u y v l, x, y tx, y gx y l, dudv
Intrinsic Images Original Re-integrated Colour Channels Edge Maps of Channels Shadow Edges Removed
Intrinsic images: New School Approach Retinex-inspired ConvNets
RetiNet: Retinex-Inspired ConvNet RGB Differentiation R RGB Integration x, y gx y t, RGB d 2 eˆ 2 S RGB t x, y gx, y d R S
ShapeNet Rendered Dataset
Qualitative Results: MIT Dataset
Summary Intrinsic image decomposition End-to-end solution. Full testing on intrinsic image decomposition using synthetic and real data. Full model including light source and highlights. Raw data preprocessing and augmentation.
Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics
Basic Emotions
Face Analysis by Deep Learning Frame Happy WWW.SIGHTCORP.COM
Sightcorp WWW.SIGHTCORP.COM
FaceWarehouse: a 3D Facial Expression Database for Visual Computing Chen Cao Yanlin Weng Shun Zhou Yiying Tong Kun Zhou State Key Lab of CAD&CG, Zhejiang University Michigan State University
Face Analysis: The Face Model
Face Analysis: The Camera Model
Face Analysis
Co nv 16 Co nv 32 Co nv 64 Co nv 128 Co nv 256 Co nv 256 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 DeCo nv 256 DeConv 256 DeConv 128 DeConv 64 DeConv 32 DeConv 16 Decoder- Shading Decoder -Specular Decoder- Albedo Encoder CNN Model: Supervised
Illumination Changes IMAGE ALBEDO SHADING SPECULARITY
Illumination, Geometry, Albedo Changes IMAGE ALBEDO SHADING SPECULARITY
CNN Model: Multi-Task Unsupervised Encoder Decoder rendering
Consumer Applications
Question: Is your presidency fake?
A.I. Face Technology Powered by:
Amsterdam A.I. - Computer Vision Group: from the lab to the real world 1. Image Intrinsics 2. Face Intrinsics 3. 3D Reconstruction Intrinsics
The True Depth camera of the iphone X: Face ID Apple s new TrueDepth camera for the iphone X sure works by using a projector to cast 30,000 dots on your face, which it then reads with an infrared camera like the Microsoft Kinect.
Consumer Applications: AR and VR Hololens Lenovo phab 2 pro
Retail & Fashion: 3D scans allow proper sizing for online shopping no more wasting time and product returns! Scan Faces and Bodies using Mobile Devices Virtual fitting, recommendation and tailoring 3DU has developed deep learning based automatic recognition algorithms which automatically detects 50K facial points eyes, temples, nose bridge, pupils, mouth corners etc. and body parts arms, waist, shoulders, legs etc..
Real Estate & Home Deco : Digital visits and try furniture at finger tips - refurnish your house from your couch Scan Rooms - Houses Virtual Tours, Measurements and Refurnishing
Scanm B.V.: Real Estate: Low-Cost Solution! Scan Rooms - Houses Equipment 3D Scanning for Everyone Lenovo + Rotator
Deep Learning in Computer Vision Object classification, detection and segmentation Optical flow Color constancy Intrinsic image decomposition 3D and slam Human behavior analysis Etc