Lecture 9: Other Applications of CNNs

Transcription

1 Applications of Convolutional Neual Netwoks CSED703R: Deep Leaning fo Visual Recognition (2016S) Lectue 9: Othe Applications of CNNs Bohyung Han Compute Vision Lab. Face ecognition and veification Peson eidentification Text egion detection Neual atistic style Object geneation Visual analogy Autonomous diving Many othes 2 Definition Face Veification Given two faces, detemine whethe they ae same peson o not. Binay decision by one-to-one matching Related poblem Face detection: finding faces Face ecognition: multi-class classification poblem Standad pipeline Siamese Netwok Deep Disciminative Metic Leaning (DDML) Leaning a distance metic! " $ %,$ ' = ) $ % ) $ ' Repesentation leaning Two banches shae weights. Objective + %',! " $ %, $ ' > 1 + %' = 1: same ID + %' = 1: diffeent ID Face Detection Face Alignment Featue Extaction Binay Classification, > [Hu2014] J. Hu, J. Lu, Y.-P. Tan: Disciminative Deep Metic Leaning fo Face Veification in the Wild. CVPR 2014

2 DeepID I CNN Achitectue Deep hidden IDentity featues (DeepID) 97.45% veification accuacy: as good as human pefomance (97.53%) CNN achitectue Type equation hee. Multiple scales > ' = max 0, B C D % E D %,' + C % E %,' + G ' % [Sun2014a] Y. Sun, X. Wang, X. Tang: Deep Leaning Face Repesentation fom Pedicting 10,000 Classes. CVPR Pepocessing Face detection Featue Extaction Landmak detection: two eyes, nose tip and two mouth cones Global face alignment: by tansfomation using two eye centes and mid-point of two mouth cones Featue extaction by CNN Extacted fom 10 sub-egions, thee scales, and {RGB, gay} 60 featues in total ConvNets 60 CNNs coesponding to 60 desied featues 160 dimensional vecto fo each featue Hoizontal flipping Length of DeepID: 160x2x60=19,200 8 Joint Bayesian Veification Algoithm $ = H + I, whee H is face identity and J is inta-class vaiation H ~ L 0, M H and I ~ L 0, M I Compute N $ O,$ P = log S $ O, $ P T U S $ O,$ P T V, which has a closed-fom solution. Neual netwoks highly-coelated subfeatue (640D) 60 goups

3 9 11 Compaison between Two Veifies Joint Bayesian is bette than neual netwok. Test accuacy (%) Joint identification-veification Numbe of classes fo taining DeepID II Face identification: inceases the inte-pesonal vaiations by dawing DeepID2 featues extacted fom diffeent identities apat Face veification: educes the inta-pesonal vaiations by pulling DeepID2 featues extacted fom the same identity togethe Featue extaction ) = Conv C; Z [ [Sun2014b] Y. Sun, Y. Chen, X. Wang, X. Tang: Deep Leaning Face Repesentation by Joint Identification- Veification. NIPS Results Compaison of state-of-the-at face veification methods on LFW Method Accuacy (%) No. of points No. of outside images images Featue dimension Joint Bayesian [8] (o) 5 99, ConvNet-RBM [31] (o) 3 87,628 N/A CMD+SLBP [17] (u) 3 N/A 2302 Fishe vecto faces [29] (u) 9 N/A Tom-vs-Pete classifies [2] (o+) 95 20, High-dim LBP [9] (o) 27 99, TL Joint Bayesian [6] (o+u) 27 99, DeepFace [32] (o+u) ,400, ,000, DeepID on CelebFaces (o) 5 87, DeepID on CelebFaces (o) 5 202, DeepID on CelebFaces+ & TL (o+u) 5 202, : esticted taining potocol, whee 6000 face pais given by LFW ae used fo 10-fold coss-validation u: unesticted taining potocol, whee moe taining pais can be geneated fom LFW using identity o: using outside taining data, howeve, without using taining data fom LFW o+: using both outside data and LFW data in the esticted potocol fo taining o+u: using both outside data and LFW data in the unesticted potocol fo taining TL: Joint Bayesian tansfe leaning fom CelebFaces+ to LFW 12 Human-level pefomance: Two loss functions Identification loss: coss-entopy Ident ), ^; Z %_ Veification loss Veif ) %,) ', > %' ; Z gh = Z gh = j Taining CNN b = B `% log ` % = log ` d %cd 1 2 ) % ) ' 1 2 max 0, j ) % ) ' ) = Conv C; Z [ Z %_ : paametes of softmax laye if > %' = 1 if > %' = 1

4 13 Goal Taining CNN Loss ) %, ) ',^, > %' ; Z [, Z %_, Z gh = Ident ), ^; Z %_ + oveif ) %,) ', > %' ; Z gh Taining Z [ while Z %_ and Z gh should be tained but ae not used in testing Magin j: a special case Cannot be updated by SGD since this will collapse to zeo Fixed and updated evey L(= 200,000) taining pai so that it minimizes veification eo of the pevious L taining pais while not convege do t t +1 sample two taining samples (x i,l i ) and (x j,l j ) fom χ f i = Conv(x i, θ c ) and f j = Conv(x j, θ c ) θ id = Ident(fi,li,θid) + Ident(fj,lj,θid) θid θid θ ve = λ Veif(fi,fj,yij,θve), whee y θve ij =1if l i = l j, and y ij = 1 othewise. f i = Ident(fi,li,θid) fi f j = Ident(fj,lj,θid) fj θ c = f i Conv(xi,θc) θc + λ Veif(fi,fj,yij,θve) fi + λ Veif(fi,fj,yij,θve) fj + f j Conv(xj,θc) θc update θ id = θ id η(t) θ id, θ ve = θ ve η(t) θ ve, and θ c = θ c η(t) θ c. end while output θ c method accuacy (%) High-dim LBP [4] ± 1.13 TL Joint Bayesian [2] ± 1.08 DeepFace [21] ± 0.25 DeepID [20] ± 0.26 GaussianFace [13] ± 0.66 DeepID ± 0.13 Human-level pefomance: Results on LFW 14 Featue extaction Veification Algoithm Detect 21 facial landmaks by SDM algoithm and align faces globally Cop 400 face patches with vaiations in positions, scales, colo channels, and hoizontal flipping ConvNet 200 CNNs: geneate 400 DeepID2 featue vectos with hoizontal flipping Featue vecto: 160D Featue dimensionality eduction Select 25 patches in a geedy manne PCA fom 25x160D to 180D Main goal Selected 25 face patches Neual Atistic Style Joint Bayesian fo veification Synthesizing two images epesenting both content and style Exploiting a petainedcnn fo image classification CNN VGG 19 laye net without fully connected layes No fine-tuning Aveage pooling: impoves gadient flow and get moe appealing esults 15 [Gatys15] L. A. Gatys, A. S. Ecke, M. Bethge: A Neual Algoithm of Atistic Style. axiv: ,

5 Method Optimization Notations Input: content Input: style Final Output output + Loss in featue map Loss in featue map coelation CNN p and q : oiginal content image and its featue map in the +-th laye $ and s : geneated image and its featue map in the +-th laye t and u : oiginal style image and its featue map in the +-th laye s,q, u R x y { y, whee L is the numbe of featue maps and is the size of featue map = width height ~ %' Å %' Loss s 'Ä u 'Ä = Ä s %Ä R x y x y : coelation of featue maps in the +-th laye = Ä u %Ä R x y x y : coelation of featue maps in the +-th laye Ç ÉÑÉÖÜ p, t, $ = áç àñâéäâé p, $ + ãç åéçüä t, $ Ç àñâéäâé p, $, + = 1 2 B s %' %,' q %' è E Ç åéçüä t, $ = 1 2 B 4L B ~ %' %,' Å %' Eo back-popagation Optimization Content: select a paticula laye such as conv4_2 Update ule: Ç àñâéäâé p, $, + = 1 2 B s %' ëç àñâéäâé ës %' q %' Style: use conv*_1 with equal weights (E = 0.2) Ç åéçüä t, $ = 1 2 BE ê Update ule: ëç åéçüä ës %' = ëç åéçüä ëê è ëê ës %' = í s %' = ï %,' q %' if s %' 0 if s %' 0 < 0 1 ê = 4L B ~ %' 1 2 E s '% ~ %' %,' Å %' Å %' if s %' 0 if s %' 0 < 0 Geneated Images Style1: The Stay Night Souce Style2: The Sceam [Gatys15] L. A. Gatys, A. S. Ecke, M. Bethge: A Neual Algoithm of Atistic Style. axiv: ,

6 Moe Examples Balance between Content and Style [Gatys15] L. A. Gatys, A. S. Ecke, M. Bethge: A Neual Algoithm of Atistic Style. axiv: , 2015 [Gatys15] L. A. Gatys, A. S. Ecke, M. Bethge: A Neual Algoithm of Atistic Style. axiv: , Multiple Styles Disciminative vs. Geneative CNN Disciminative CNN CNN Style1: The Stay Night Geneative CNN Object class Viewpoint Style Souce Style2: The Sceam [Gatys15] L. A. Gatys, A. S. Ecke, M. Bethge: A Neual Algoithm of Atistic Style. axiv: , Object class Viewpoint Style

7 Goal Geneate an object based on high-level inputs such as Class Oientation with espect to camea Additional paametes Rotation, tanslation, zoom Stetching hoizontally o vetically Hue, satuation, bightness Knowledge tansfe Geneative CNN leans the manifold of chais. Intepolation between viewpoints and diffeent objects Data Using 3D chai model dataset [Auby14] Oiginal dataset: 1393 chai models, 62 viewpoints, 31 azimuth angles, 2 elevation angles Sanitized vesion: 809 models, tight copping, esizing to 128x128 Notations ñ = ó D, ò D,Z D, ó, ò, Z,, ó x, ò x, Z x ó: class label ò: viewpoint Z: additional paametes ö = $ D, õ D, $, õ,, $ x, õ x $: taget RGB output image õ: segmentation mask 25 [Dosovitskiy15] A. Dosovitskiy, J. T. Spingenbeg, T. Box: Leaning to Geneate Chais with Convolutional Neual Netwoks. CVPR 2015 Netwok Achitectue 26 [Auby14] M. Auby, D. Matuana, A. Efos, and J. Sivic, Seeing 3D Chais: Exempla Pat-based 2D-3D Alignment using a Lage Dataset of CAD Models. CVPR 2014 Opeations h ù Unpooling: 2x2 Fixed location unpooling Deconvolution: 5x5 32M paametes altogethe û = ù h ReLU 27 28

8 Taining Netwok Capacity Objective function Minimizing the Euclidean eo in 2D of x Reconstuction of the segmented-out chai image Segmentation mask min B o ù h ó %, ò %, Z % $ % õ % + ùåä h ó %, ò %, Z % õ % %cd Visualization of uconv-3 laye filtes in 128x128 netwok RGB steam Tanslation Rotation Zoom Stetch Satuation Segmentation steam Bightness Colo [Saxe14] A. M. Saxe, J. L. McClelland, and S. Ganguli, Leaning a Nonlinea Embedding by Peseving Class Neighbouhood. ICLR Mophing Diffeent Chais Autonomous Diving Two pevious appoaches Mediated peception: pasing the entie scene to make a diving decision (e.g., Mobileye, Google) Behavio eflex: diectly mapping an input image to a diving decision by an egesso (ALVINN, LeCun et al.) Mediated Peception Input Image Diving Contol Behavio Reflex Diect Peception (ous) 31 Viewpoints in taining set [Chen15] C. Chen, A. Seff, A. Konhause, J. Xiao: DeepDiving: Leaning Affodance fo Diect Peception in Autonomous Diving. ICCV

9 Diect peception Deep Diving Estimating the affodance fo divinginstead of visually pasing the entie scene o blindly mapping an image to steeing angles Mapping an input image to a small numbe of key peception indicatos that diectly elate to the affodance of a oad/taffic state fo diving Poviding a set of compact yet complete desciptions of the scene to enable a simple contolle to dive autonomously Appoach Built upon deep convolutional neual netwok Tained and tested on TORCS (The Open Racing Ca Simulato) Automatically lean image featues fo estimating affodance elated to autonomous diving Much simple stuctue than the typical mediated peception appoach Moe intepetable than the typical behavio eflex appoach System achitectue RC Envionment Image peed ite Read haed Memo Diving Contols Platfom Read Image peed C Focusing on highway diving with multiple lanes Diving Contolle Thee configuations: a oad of one lane, two lanes, o thee lanes Read ite angle toma ing... dist... Contolle utput (a) one-lane (b) two-lane, left (c) two-lane, ight (d) thee-lane (e) inne lane mak. (f) bounday lane mak. Convolutional Neual Netwok Affodance Indicato Pediction of affodance indicato CNN angle tomaking_ll dist_ll tomaking_l dist_l always in lane system on making system always: 1) angle: angle between the ca s heading and the tangent of the oad in lane system, when diving in the lane: 2) tomaking LL: distance to the left lane making of the left lane 3) tomaking ML: distance to the left lane making of the cuent lane 4) tomaking MR: distance to the ight lane making of the cuent lane 5) tomaking RR: distance to the ight lane making of the ight lane 6) dist LL: distance to the peceding ca in the left lane 7) dist MM: distance to the peceding ca in the cuent lane 8) dist RR: distance to the peceding ca in the ight lane on making system, when diving on the lane making: 9) tomaking L: distance to the left lane making 10) tomaking M: distance to the cental lane making 11) tomaking R: distance to the ight lane making 12) dist L: distance to the peceding ca in the left lane 13) dist R: distance to the peceding ca in the ight lane (a) angle (b) in lane: tomaking (c) in lane: dist (d) on mak.: tomaking (e) on making: dist (f) ovelapping aea 35 36

10 Contolle Logic while (in autonomous diving mode) ConvNet outputs affodance indicatos check availability of both the left and ight lanes if (appoaching the peceding ca in the same lane) if (left lane exists and available and lane changing allowable) left lane changing decision made else if (ight lane exists and available and lane changing allowable) ight lane changing decision made else slow down decision made if (nomal diving) cente line= cente line of cuent lane else if (left/ight lane changing) cente line= cente line of objective lane compute steeing command compute desied speed compute acceleation/bake command based on desied speed Contolle Logic Steeing command: desied diection Desied speed steecmd = C angle dist_cente oad_width angle Ø, Ø dist_cente: distance to the cente line of the lane If the ca is tuning, a desied speed dop is computed accoding to the past few steeing angles. If thee is a peceding ca in close ange and a slow down decision is made, the desied speed is also detemined by the distance to the peceding ca. ^ = Ö 1 exp Æ Ö dist ^! dist ^ : distance to the peceding ca Implementation Achitectue and taining details Extension of AlexNet Input size: convolutional layes and 4 fully connected layes (4096, 4096, 256, 13 nodes) Mini-batch size: 64 Initial leaning ate: ,815 images fo taining: no copping and flipping Euclidean loss of nomalized output to [0.1, 0.9] Numbe of iteations: 140,000 Data geneation Using 7 tacks and 22 taffic cas in TORCS Geneating vaious taffic pattens of the taffic cas Manually diving the host ca on each tack multiple times Ceating exteme diving conditions (e.g. off the oad, collide with othe cas) Moe effective than automatic data collection using an AI ca 40 Testing Implementation Infomation fom TORCS: font facing image and speed of the host ca Contol fequency: 10 Hz, which is sufficient fo diving below 80 km/h Assumption The host ca is faste than the taffic when ovetaking cas in its left/ight lane. This assumption is equie because the system cannot see things behind.

11 Visualization of Leaned Models Deep Diving Demo Response map of KITTI-based ConvNet model 41 Response map of TORCS-based ConvNet model 42 C. Chen, A. Seff, A. Konhause,J. Xiao: DeepDiving: Leaning Affodance fo Diect Peception in Autonomous Diving. ICCV