Recovering Joint and Individual Components in Facial Data

Transcription

1 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Recovering Join and Individual Componens in Facial Daa Chrisos Sagonas, Evangelos Ververas, Yannis Panagakis, and Sefanos Zafeiriou, Member, IEEE Absrac A se of images depicing faces wih differen expressions or in various ages consiss of componens ha are shared across all images (i.e., join componens) imparing o he depiced objec he properies of human faces as well as individual componens ha are relaed o differen expressions or age groups. Discovering he common (join) and individual componens in facial images is crucial for applicaions such as facial expression ransfer and age progression. he problem is raher challenging when dealing wih images capured in unconsrained condiions in he presence of sparse non-gaussian errors of large magniude (i.e., sparse gross errors or ouliers) and conain missing daa. In his paper, we invesigae he use of a mehod recenly inroduced in saisics, he so-called Join and Individual Variance Explained (JIVE) mehod, for he robus recovery of join and individual componens in visual facial daa consising of an arbirary number of views. Since he JIVE is no robus o sparse gross errors, we propose alernaives, which are (1) robus o sparse gross, non-gaussian noise, (2) able o auomaically find he individual componens rank, and (3) can handle missing daa. We demonsrae he effeciveness of he proposed mehods o several compuer vision applicaions, namely facial expression synhesis and 2D and 3D face age progression in-he-wild. Index erms Low-Rank, Sparsiy, Facial Expression Synhesis, Face Age Progression, Join and Individual Componens. 1 INRODUCION Facial images convey rich informaion, which can be perceived as a superposiion of componens associaed wih aribues, such as facial ideniy, expression, age ec. For insance, a se of images depicing expressive faces consiss of componens ha are shared across all images (i.e., join componens) imparing o he depiced objec he properies of human faces. Besides join componens, an expressive face consiss of individual componens ha are relaed o differen expressions. Such individual componens can be expression-specific deformaion of a face, i.e., deformaions around lips and eyes in case of smiles. Similarly, a se of images depicing faces in differen ages can be seen as a superposiion of join componens ha are invarian o he age and age-specific componens ha are individual o each age group (e.g., wrinkles). Consequenly, being able o exrac such join and individual componens from facial images is crucial for applicaions such as facial expression synhesis and age progression 1], 2], 3], 4], 5], 6], among oher visual daa analysis asks. Exracing he join componens among daa has creaed a wealh of research in saisics, signal processing, and compuer vision. wo mahemaically similar bu concepually differen models underlie he bulk of he mehodologies. In paricular, he Canonical Correlaion Analysis (CCA) 7] and is varians e.g., 8], 9], have been proposed for exracing linear correlaed componens among wo or more ses of variables. Similarly, iner-baery facor analysis 10] and is exensions e.g., 11], deermines he common facors among wo ses of variables. he main limiaion of he aforemenioned mehods is ha hey only C. Sagonas is wih Onfido, London WC2E 9LG, UK ( ch.sagonas@gmail.com). his work was compleed while C. Sagonas was a Imperial College London, UK. V. Ververas, Y. Panagakis, and S. Zafeiriou are wih he Deparmen of Compuing, Imperial College London, SW7 2RH, London, UK. Y. Panagakis is also wih he Deparmen of Compuer Science, Middlesex Universiy London, UK. recover he mos correlaed linear subspace of he daa, ignoring he individual componens among he differen views or daases. he above menioned limiaion is alleviaed by recen mehods such as he Join and Individual Variaion Explained (JIVE) 12], he Common Orhogonal Basis Exracion (COBE) 13], and he Robus Correlaed and Individual Componen Analysis (RCICA) 14], which are briefly described in Secion 2. Besides he rich srucure in facial visual daa, images are subjec o various ypes of errors, disorions, and noise. Common dense disorions such as ambien noise or quanizaion noise are of small magniude and i is naural o assume ha hey follow a Gaussian disribuion of small variance. Mehods such as he CCA and is varians, JIVE, and COBE are sable in he presence of Gaussian noise. Apar from hese small bu dense noises, here are gross errors ha are sparsely suppored bu of large or even unbounded magniude, such as he sal-and-pepper noise in imaging devices, occlusions in facial images, regisraion errors, or errors due incorrec localizaion and racking. hese errors rarely follow a Gaussian disribuion and due o heir sparse naure (i.e., he number of errors is bounded below some consan) are collecively referred o as sparse gross errors or noise. Excep for he mos recen RCICA, he COBE and JIVE rely on leas squares error minimizaion and hus hey are prone o gross errors and ouliers 15]. ha is, he esimaed componens can be arbirarily away from he rue ones. herefore, he problem of join and individual componens recovery is raher challenging when dealing wih facial images and in general visual daa capured under unconsrained (i.e., in-hewild ) condiions. In his paper, we invesigae he problem of recovering he join and individual componens from facial (and in general visual) daa consising of an arbirary number of views, capured in-hewild. Such daa are herefore conaminaed by sparse, gross, non- Gaussian noise and possibly conain missing values. o his end,

2 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS we propose robus alernaives o he JIVE (coined collecively as Robus-JIVE, ), where he componens are esimaed by employing he l 1 -norm. he l 1 -norm is suiable for robus esimaion in he presence of sparse gross errors 15]. he conribuions of he paper are summarized as follows: We propose a novel, general framework, he in Secion 3, for he robus recovering of join and individual componens from muli-view daa in he presence of sparse gross errors and possibly missing values. he proposed decomposes he daa ino hree erms: a low-rank marix ha capures he join variaion across views, lowrank marices accouning for srucured variaion individual o each view, and a sparse marix collecing he sparse gross errors 1. In paricular, he consiss of 4 differen models, namely l 1 -, NN-l 1 -, S, and - M. In he l 1 -, he rank of boh join and individual componens are user-defined, while in he NN-l 1 - he rank of each one of he individual componens is auomaically esimaed via nuclear norm minimizaion. As opposed o he previous wo models, he S direcly exracs he orhonormal bases of join and individual componens and improves heir scalabiliy. Finally, he -M exends he S in order o handle missing values. Based on he recovered join and individual componens from raining daa, wo suiable opimizaion problems ha exrac he corresponding modes of variaion (i.e., join and individual componens) of unseen es samples, are proposed in Secion 4. o ackle he proposed opimizaion problems, algorihms based on he Alernaing-Direcions Mehod of Mulipliers (ADMM) 17] are developed in Secions 3, 4, 5, and 6. We demonsrae he applicabiliy of he proposed mehods in hree challenging compuer vision asks, namely facial expression synhesis, face age progression in 2D images and 3D daa capured in-he-wild. Experimenal resuls corroborae he effeciveness of he proposed approach in Secion 7. Noaion: hroughou he paper, scalars are denoed by lowercase leers, vecors (marices) are denoed by lower-case (uppercase) boldface leers i.e., x, (X). I denoes he ideniy marix. he j-h column of X is denoed by x j. Several norms and merics will be used. he l 1 - and he l 2 -norms of x are defined as x 1 = i x i and x 2 = i x2 i, respecively. denoes he absolue value operaor. he marix l 1 norm is defined as X 1 = i j x ij, he Frobenius norm is defined as X F = i j x2 ij, and he nuclear norm of X (i.e., he sum of singular values of a marix) is denoed by X. he vecor (marix) l 0 -(quasi) norm reurns he oal number of nonzero elemens in a vecor (marix). he rank funcion is denoed by rank( ). 1. A preliminary version of he presen work has been proposed in 16], where he he main model and is algorihmic framework has been inroduced. In his paper, we furher invesigae and propose a unified model ha direcly exracs he orhonormal bases of join and individual componens and improves he scalabiliy of he main model. Besides ha we propose an exension of for handling missing daa. Moreover, new qualiaive and quaniaive experimenal resuls are included in his paper. he minimizaion of boh he rank funcion and he l 0 - norm are NP-hard problems 18], 19]. Consequenly, he rank funcion and he l 0 -norm are ypically replaced by heir convex surrogaes 20], 21]. Operaors: he soluion of he several problems appeared in he paper relies on differen (proximal) operaors which are defined nex. Le for any marix X = UΣV be he Singular Value Decomposiion (SVD). Shrinkage operaor 22]: S τ σ] = sgn(σ) max( σ τ, 0). Singular Value hresholding (SV) operaor 23]: D τ = US τ V. Rank-r SVD operaor: Q r X] = U(:, 1 : r)σ(1 : r, 1 : r)v(:, 1 : r) ]. Procruses operaor: P D] = GR (given he rank-r SVD of a marix D = GPR ). 2 BACKGROUND o make he paper self-conained, his secion includes a brief review of he JIVE 12], COBE 13], and RCICA 14]. 2.1 Join and Individual Variaion Explained (JIVE) he JIVE recovers he join and individual componens among M 2 daases {X R d J, i = 1, 2,..., M}, where J is he number of samples of each daase. In paricular, each marix is decomposed ino wo erms: a low-rank marix J R d J capuring join srucure among daase and a low-rank marix A R d J capuring individual srucure of each daase. ha is, X = J + A, i = 1, 2,..., M. Le X and J be M i=1 d J marices consruced by concaenaion of he corresponding marices, i.e., X = X (1), X (2),..., X (M) ], J = J (1), J (2),..., J (M) ], JIVE solves he rankconsrained leas-squares problem 12]: min J,{A } M i=1 1 2 ] X J 2 A (1),, A (M). s.. rank(j) = r, {rank(a ) = r, JA = 0} M i=1. (1) Problem (1) imposes rank consrains on he join and individual componens and requires he rows of J and {A } M i=1 o be orhogonal. he inuiion behind he orhogonaliy consrain sems for he fac ha sample paerns responsible for join srucure beween daa ypes are unrelaed o sample paerns responsible for individual srucure 12]. By adoping he leas squares error, he JIVE assumes Gaussian disribuions wih small variance 15]. Such an assumpion rarely holds in real world daa, where gross, non-gaussian corrupions are in abundance. Consequenly, he componens obained by employing he JIVE in he analysis of grossly corruped daa may be arbirarily away from he rue ones, hus degeneraing heir performance. 2.2 Common Orhogonal Basis Exracion (COBE) A closely relaed mehod o he JIVE is he COBE which exracs he common and individual componens of M daases of he same dimensions by solving a se of leas-squares minimizaion problems 13]. More specifically, each daase X R J d is facorized as Ξ Λ, where a column of Ξ signifies a laen variable o be found and Λ signifies a marix of weighs. Ξ is assumed o be decomposable in blocks as Ξ Ξ] where F

3 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Ξ R n m, Ξ R n (d m) and m min{d, i = 1,, M}. In oher words, Ξ is assumed o be common in all facorizaions and hence i presens join srucure, while Ξ is assumed o represen individual srucure. Similarly, Λ splis ino Λ and Λ. he opimizaion problem of he COBE akes he following form: min Ξ, Ξ M X Ξ Λ Ξ Λ 2 i=1 F. s.. Ξ Ξ = I, { Ξ Ξ = I, Ξ Ξ = 0} M i=1. Similarly o he JIVE, he uilizaion of he leas square error renders he COBE non-robus agains sparse, non-gaussian errors. 2.3 Robus Correlaed and Individual Componen Analysis (RCICA) he goal of he RCICA 14] is o exrac boh he correlaed and he individual componens beween wo known, high-dimensional daases or views, namely {X R d J } 2 i=1, in he presence of sparse noise (or errors). o his end, he RCICA seeks a decomposiion of each daa marix {X } ino hree erms: X = C + A + E, i = 1, 2. C R d J and A R d J are low-rank marices, wih rank(c ) k c and rank(a ) k and muually independen columns, capuring he correlaed and individual componens, respecively and E R d J is a sparse marix accouning for he sparse noise. o exrac he correlaed componens C R d J, he cos funcion of he Canonical Correlaion Analysis (CCA) 7] is adoped. ha is, by furher decomposing he marix {C } 2 i=1 as: C = U V X, he maximally correlaed componens are derived by minimizing he CCA cos, namely λ c 2 V (1) X (1) V (2) X (2) 2 F. Here, U are orhonormal bases, ransforming he correlaed componens back o he observaion space X. Since he column space of he individual componens A is desired o be orhogonal o he one of he correlaed componens, we have o enforce {Q U } 2 i=1 = 0, where Q are column orhonormal bases spanning he column space of he individual componens A. ha is, A = Q H. Consequenly, a naural esimaor accouning for he upperbounded rank of he correlaed and independen componens and he sparsiy of {E } 2 i=1 is o minimize he objecive funcion of CCA, i.e., 1 2 V(1) X (1) V (2) X (2) 2 F as well as he rank of {C = U V X, A = Q H } 2 i=1 and he number of non-zero enries of {E } 2 i=1 measured by he l 0- (quasi) norm, e.g., 22]. o avoid he NP-hardness of rank and l 0 -norm minimizaion, he nuclear- and he l 1 -norms are ypically adoped as surrogaes o rank and l 0 -norm, respecively 20], 21]. By employing he uniary invariance of he nuclear norm i.e., Q V = V, he opimizaion problem of RCICA (2) is formulaed as he following consrained non-linear one: 2 ] min V + λ H + λ 1 E 1 V i=1 + λ c 2 V(1) X (1) V (2) X (2) 2 F, s.. X = U V X + Q H + E, (ii) V X X V = I, (iii) U U = I, Q Q = I, (iv) Q U = 0, i = 1, 2, where he posiive parameers λ c, λ (1), λ (2), λ (1) 1 and λ (2) 1 conrol he correlaion, rank and sparsiy of he derived spaces and V = {U, V, Q, H, E } 2 i=1 collecs he opimizaion variables. Consrains (ii) in (3) have been adoped from he CCA 7], while he consrains (iii) and (iv) ensure ha boh he recovered correlaed and individual componens are linearly independen. Alhough he RCICA is robus o sparse, non-gaussian error, is exension o more han wo daases is no rivial due o he orhogonaliy among he correlaed and individual componens and column orhonormaliy of he basis marices U and Q, i = 1, 2,... M, wih M being he number of differen views. his makes he resuling opimizaion problem highly-nonlinear and hence difficul o solve. 3 ROBUS JIVE Consider daa consising of M views, namely {X R d J } M i=1, wih x j R d, j = 1,..., J being a vecorized (visual) daa sample, possibly conaminaed by gross, sparse errors. he goal of he is o robusly recover he join componens which are shared across all views as well as he componens which are deemed individual for each view. ha is: ] X = J + A (1),, A (M) + E, (4) where X = X ] (1),, X (M) R q J, J = ] J (1),, J (M) R q J, {A R d J } M i=1, q = d (1) + + d (M), are low-rank marices capuring he join and individual variaions, respecively and E R q J denoes he error marix accouning for he gross, bu sparse non-gaussian noise. In order o ensure he idenifiabiliy of (4), he join and common componens should be muually incoheren, i.e., {JA = 0} M i=1. Assuming ha he number of errors is bounded below some consan, he number of errors in he esimaed componens is similarly bounded and hence, a naural esimaor accouning for he sparsiy of he error marix E is o minimize he number of he non-zero enries of E measured by he l 0 -quasi norm 22]. However, as in case of he RCICA, o make he problem compuaionally racable he l 0 -norm is replaced by is convex surrogae, namely he l 1 -norm. herefore, he join and individual componens as well as he sparse error are recovered by solving he following consrained, non-linear opimizaion problem: min A ] X J (1),, A (M) 1. J,{A } M i=1 (5) s.. rank(j) = r, {rank(a ) = r, JA = 0} M i=1 (3)

4 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Algorihm 1: ADMM solver for (7) (l 1 -). Inpu : Daa {X R d J } M i=1. Rank of join componen r. Ranks of individual componens {r } M i=1. Parameer ρ. Oupu : Join componen J, individual componens {A } M i=1 Iniialize: Se J 0, {A 0 }M i=1, E 0, L 0 o zero marices, ] = 0, µ 0 > 0, X = X (1),, X (M). 1 while no converged do 2 M = X A (1),, A (M) 3 J +1 = Q r M], U, Σ, V] = svd(m); 4 P = I V(:, 1 : r)v(:, 1 : r) ; 5 for i = 1 : M do 6 A ( +1 = Q r 7 end 8 E = 9 L +1 = S 1 µ X J +1 L +µ (X J +1 X J +1 E 10 µ +1 = min(ρ µ, 10 7 ); = + 1; 11 end ] E + µ 1 L ; + µ 1 L ) ] P ; ] ] A (1) +1,, A(M) +1 µ 1 L ; ] ) A (1) +1,, A(M) +1 E+1 ; Algorihm 2: ADMM solver of (9) (NN-l 1 -). Inpu : Daa {X R d J } M i=1. Rank of join componen r. Parameer ρ. Oupu : Join componen J, individual componens {A } M i=1 Iniialize: Se J 0, {A 0, R 0, Y 0 }M i=1, E 0, F 0 o zero marices, = 0, µ 0 > 0, ] X = X (1),, X (M). 1 while no converged do 2 J +1 = Q r X A (1) 3 for i = 1 : M do ( 4 A +1 = X J 5 R +1 = D 1/µ A 6 Y +1 = Y 7 end 8 E +1 = S λ µ X J +1 ],, A (M) E + F µ ]; +1 E + F +R µ + Y )P µ ; 2 +1 Y µ ]; + µ (R +1 A +1 ); 9 F +1 = F + µ (X J +1 E +1); = + 1; 10 end ] A (1) +1,, A(M) +1 + F µ ]; ] A (1) +1,, A(M) +1 Clearly, (5) is a robus exension o JIVE 12], and requires an esimaion for he rank of boh join and individual componens. However, in pracice hose (M + 1) values are unknown and difficul o esimae since an exensive uning procedure is required. o alleviae his issue, we propose a varian of (5), which is able o deermine he opimal ranks of individual componens direcly. By assuming ha he acual ranks of individual componens are upper bounded, i.e., {rank(a ) K } M i=1, problem (5) is relaxed o he following one: min λ A ] X J (1),, A (M) 1 + J,{A } M i=1 M (6) A, s.. rank(j) = r, {JA = 0} M i=1, i=1 where he rank funcion is replaced by is convex envelope, namely he nuclear norm and λ > 0 is a regularizer. 3.1 Opimizaion Algorihms In his secion, algorihms for solving (5) and (6) are developed. o solve (5), he Alernaing-Direcion Mehod of Mulipliers (ADMM) 17] is employed. o his end, problem (5) is reformulaed o he following separable one: min E 1, J,{A } M i=1,e ] s.. X = J + A (1),, A (M) + E, rank(j) = r, {rank(a ) = r, JA = 0} M i=1, where E is an auxiliary variable. o solve (7), he corresponding augmened Lagrangian funcion is given by: L(J, {A } M i=1, E, L) = E 1 1 2µ L 2 F + µ ] 2 X J L 2 (8) A (1),, A (M) E + µ, F (7) where L is he Lagrange mulipliers marix relaed o he equaliy consrain in (7), and µ is a posiive parameer. Subsequenly, by employing he ADMM, (8) is minimized wih respec o each variable in an alernaing fashion and finally he Lagrange mulipliers L are updaed. he ADMM solver of (7) is oulined in Algorihm 1. Algorihm 1 erminaes when X J +1 A (1) +1,, A(M) +1 ] E +1 2 F / X 2 F is less han a predefined hreshold ɛ or he number of ieraions reach a maximum value. o solve problem (6) via he ADMM, we firsly reformulae i as: M min R + λ E 1, J,{A,R } M i=1,e i=1 ] (9) s.. X = J + A (1),, A (M) + E, rank(j) = r, {R = A, JA = 0} M i=1 where {R R d J } M i=1, {R = A } M i=1 are auxiliary variables and he corresponding consrains, respecively. he ADMM solver of (9) is wrapped up in Algorihm 2 where F, {Y } M i=1 are he Lagrange mulipliers relaed o he equaliy consrains in (9), and µ is a posiive parameer. A convergence crierion similar o Algorihm 1 is employed. he augmened Lagrangian funcion of (9) as well as he derivaion of he proposed Algorihm can be found in he supplemenary maerial. 4 -BASED RECONSRUCION Having recovered he individual and common componens of he M views or differen daases during raining, we can exploi hem in order o exrac he join and individual modes of variaions of a es sample. For insance, he componens recovered by applying he on a se of facial images of M differen expressions can be uilized in order o reconsruc M expressive images

5 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Algorihm 3: ADMM-based solver of (10). Inpu : Inpu sample. Orhonormal bases B R d W J λ, ρ. Oupu : Clean reconsruced image y. Iniialize: Se {v (n) 0, c (n) 0 }2 n=1, {h (n) vecors, = 0, µ 0 > 0. 1 while no converged do 2 for n=1:2 do 3 v (n) +1 = S µ 1 c (n) 4 end, D R d W A. Parameers h(n) µ ]; µ 1 ; + h (4) µ 1 ; ( )+v (1) +1 +h(1) µ 1 ; 3 e + h(3) µ 1 ; +1 + h(4) µ 1 ; ( )+v (2) +1 +h(2) µ 1 ; = D c (2) e + h (3) 6 2 = y D c (2) 7 c (1) +1 = B 8 1 = B c (1) = y B c (1) 10 c (2) +1 = D 0 }4 n=1, y 0, and e 0 o zero 11 y +1 = max ( B c (1) +1 + D c (2) +1 h(4) /µ, 0 ) ;] 12 e +1 = S λ B c (1) +1 D c (2) +1 + h(3) µ 1 ; µ 13 h (1) +1 = h(1) + µ (v (1) +1 c(1) 14 h (2) +1 = h(2) + µ (v (2) +1 c(2) +1 ); +1 ); 15 h (3) +1 = h(3) + µ ( B c (1) +1 D c (2) +1 e+1); 16 h (4) +1 = h(4) + µ (y B c (1) +1 D c (2) +1 ); 17 µ +1 = min(µ ρ, 10 7 ); 18 end {y } M i=1 of an inpu face. he key moivaion here is ha he expression-relaed paerns of he image in he expression lie in a linear subspace spanned by D R d W A, where D has been obained by applying he SVD ono he exraced A componens. herefore, he expression-relaed (individual) par of he es image in expression can be represened as a linear combinaion of he orhonormal bases D, i.e., y individual D c (2) wih c (2) R W A 1 being a sparse coefficien vecor. Similarly, he join par y join of he orhonormal bases B is expressed as a linear combinaion R d W J, exraced from join B c (1), he corresponding join componen J i.e., y c (1) R W J 1. hus, he expressive image y of he unseen inpu face is reconsruced by solving he following consrained opimizaion problem: min {c (n),v (n) } 2 n=1,y 0 2 v (n) 1 + λ e 1, n=1 s.. {v (n) = c (n) } 2 n=1 = B c (1) + D c (2) + e, y = B c (1) + D c (2) (10) where λ is a posiive parameer ha balances he norms, v (1) and v (2) are auxiliary variables which are employed in order o make he problem separable, y corresponds o he non-negaive clean reconsrucion, and e is an error erm accouning for he gross, non-gaussian sparse noise. Equaion (10) resembles he dense error correcion model proposed in 24], which is suiable for guaraneed recovery of sparse represenaions from highdimensional measuremens, such as images of high resoluion (e.g., pixels in his paper) in he presence of noise. he ADMM solver of (10) is oulined in Algorihm 3. Algorihm 3 erminaes when B c (1) +1 D c (2) +1 e / 2 2 is less han a predefined hreshold ɛ or he number of ieraions reached. he augmened Lagrangian funcion of (10) can be found in he supplemenary maerial. 5 SCALABLE he compuaional complexiy of he vanilla JIVE as well as he l 1 - and NN-l 1 - a each ieraion is O(max(q 2 J, qj 2 )) + M i=1 O(max(d2 J, d J 2 )) = O(max(q 2 J, qj 2 )) due o SVD. Clearly, his is compuaionally prohibiive when dimensionaliy of images {d } M i=1 becomes very large, e.g., in our case. o alleviae he aforemenioned compuaional complexiy issue and a he same ime learn he orhonormal bases ha are used for reconsrucion, we propose o facorize marices J, {A } M i=1 as producs of orhonormal bases marices B R (d(1) + d (M) ) W J, B B = I, {D R d W A D D = I} M i=1 and low-rank coefficiens marices G R WJ J, {C R W A J } M i=1 such ha J = BG and {A = D C } M i=1. I can be easily shown ha he consrains are now wrien as {JA } M i=1 = = 0 GC and rank(j) = rank(bg) = rank(g) = r. In addiion, due o he uniary invariance propery of he nuclear norm we have A = D C = C. herefore, by incorporaing he facorizaions of join and individual componens, he opimizaion problem (9) now reformulaes as follows: M min + λ E 1, B,G,{D,C, } M i=1,e i=1 ( s.. X = BG + D (1) C (1)), (D (M) C (M)) ] rank(g) = r, B B = I, + E, { = C, GC = 0, D D = I} M i=1, (11) where { R W A J } M i=1 and { = C } M i=1, are auxiliary variables and he corresponding consrains, respecively. he ADMM solver of he proposed S mehod is oulined in Algorihm 4, where Γ and {Z } M i=1 are he Lagrangian mulipliers relaed o he equaliy consrains of (11) (he Lagrange funcion corresponding o problem (11) can be found in he supplemenary maerial). he compuaional complexiy of Algorihm 4 is dominaed by he cos of he SVD involved in he compuaion of SV and Procruses operaors in Seps 4 and 5, respecively. herefore, he compuaional complexiy of each ieraion is O(max(WJ 2J, W JJ 2 )) and O(max(q 2 W J, qwj 2 )), respecively. Given ha W J q = d (1) + d (M) (in his paper q = and W J 600), which implies W J J +qw J qj, he proposed scalable version of, i.e., he S, has a significanly reduced compuaional cos compared o ha of JIVE and. Regarding he convergence of he presened Algorihms 2, 1, 4, here is currenly no heoreical proof known for he ADMM in problems wih more han wo blocks of variables. However, ADMM has been applied successfully in non-linear opimizaion problems in pracice 14], 25], 26], 27], 28]. In addiion, he horough experimenal evaluaion of he proposed mehods presened in Secion 7, indicaes ha he obained soluions are good for he daa upon which was esed.

6 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Algorihm 4: ADMM solver of (11) (Scalable NN-l 1 -, S). Inpu : Daa {X R d J } M i=1. Rank of join componen r. Number of bases o be exraced from he Join and Individual componens W J and W A, respecively. Parameer ρ. Oupu : Orhonormal Join and Individual bases marices B, {D } M i=1. Coefficien marices G, {C } M i=1. ] X (1),, X (M). Iniialize: Se G 0, B 0, { 0, D 0, C 0, Z 0 }M i=1, E 0, Γ 0 o zero marices, = 0, µ 0 > 0, X = 1 while no converged do 2 M = B ( X 3 G +1 = Q r ( M]; 4 B +1 = P X ( D (1) C (1) ), ( D (M) ( ) ( D (1) C (1), 5 M = X B +1G +1 E + µ 1 Γ ; 6 for n=1:m do 7 D +1 = P ] ; C M C 8 +1 = D µ 1 C +1 µ 1 Z ( 9 Z +1 = Z +1 + µ ( +1 ] = 0.5 ; ) ; +1 C +1 C (M) D (M) D ) ] E + µ 1 Γ ); U, Σ, V] = svd(m); C (M) ) ] ) E + µ 1 Γ G +1 ; +1 M + + µ 1 Z ] ) (I VV ) ; 10 end ( ) ( ) ] 11 E = S λ X B µ +1G +1 D (1) +1 C(1) +1, D (M) +1 C(M) +1 + µ 1 Γ ]; ( ) ( ) ] ) 12 Γ +1 = Γ + µ (X B +1G +1 D (1) +1 C(1) +1, D (M) +1 C(M) +1 E +1 ; 13 µ +1 = min(ρ µ, 10 7 ); = + 1; 14 end 6 WIH MISSING VALUES AND APPLICA- ION O FACE AGING USING 3D MORPHABLE MOD- ELS 3D Morphable Models (3DMMs) are saisical deformable models of he 3D shape and appearance of he human face 29]. ypically, a 3DMM consiss of PCA models for shape and appearance, as well as a camera projecion model. More specifically, he shape model describes facial meshes ha consis of L verexes and is buil by applying dense regisraion on a se of raining meshes followed by PCA 29]. An insance of he shape model can be expressed as he linear combinaion of a mean shape s and he subspace U s wih parameers p as s = s+u s p. Similarly, he exure model is a linear PCA model ha describes he exure associaed wih he shape model and can be consruced from capured 3D exure as in 29], or from single 2D images as in 30]. Moreover, he camera model maps a 3D mesh on he image plane, uilizing an orhographic or a perspecive ransformaion W (p, c), where c are he camera parameers. Fiing a 3DMM ino a new image is an ieraive process, where he model parameers (regarding shape, exure, and camera) are updaed a each ieraion. ypically, he fiing procedure is formulaed as a Gauss-Newon opimizaion problem, where he main ask is he minimizaion of he error beween he inpu and he reconsruced image 30]. he exracion of 3D exure from single images commences wih fiing a 3DMM on hem. hen a UV exure map is calculaed by projecing he reconsruced 3D shape on he image plane and subsequenly sampling he image a he locaions of he shape s verexes. However, exracing he 3D exure from a 2D image in his way leads o incomplee 3D exure represenaions, mainly due o he presence of self-occlusions, especially when he person depiced in he image is no in a fronal pose. herefore, daa colleced wih he aforemenioned echnique include missing values. In order o specify he locaion (i.e., image coordinaes) of he missing values in a UV exure image, a self-occlusion mask for each image is calculaed by casing a ray from he camera o each verex of he reconsruced shape. Each elemen of he exraced mask denoes wheher a value of he UV exure map is missing or no (please see he firs column of Figure 11 for a visualizaion of he exraced UV space). Even hough can robusly recover join and individual componens in he presence of sparse non-gaussian errors of large magniude, i is no able o handle daa wih missing values. o overcome his limiaion of he, we propose he -Missing (-M). Consider M daases of differen ages {X R d J } M i=1, wih x j R d, being a vecorized form of he j-h gross corruped and incomplee UV exure, j = 1,..., J, ha displays a face wihin he i-h age group, i = 1,... M. he goal of he -M is no only o recover he join and individual componens bu also o perform compleion on he UV exures wih missing values. o his end, problem (11) is reformulaed o he following one: M min + λ W E 1, B,G,{D,C, } M i=1,e i=1 ( s.. X = BG + D (1) C (1)), (D (M) C (M)) ] rank(g) = r, B B = I, + E, { = C, GC = 0, D D = I} M i=1, (12) where denoes he Hadamard (elemen-wise) produc and W = W ] (1),, W (M) R q J, W = w 1, w 2,, w J ] {0, 1}q J, wih w j being a vecorized form of he self-occlusion mask ha corresponds o he j-h UV

7 JOURNAL OF LAEX CLASS FILES, VOL. 14, NO. 8, AUGUS exure of he i-h daase. he Algorihm for solving he proposed -M problem is similar o he S one and has he same complexiy and convergence crierion. he only difference is in he updaing sep of he error marix E. More specifically, he following addiional sep is performed afer execuing he sep 13 of Algorihm 4: E = W E + W X B+1 G+1 (1) (1) (M ) (M ) (D+1 C+1 ), (D+1 C+1 ) ] + µ 1 Γ ]. Similarly, he presened -based reconsrucion mehod can be also exended o handle missing values in a es image. o his end, given a es sample wih missing values (e.g., facial UV exure) and he vecorized form of he corresponding occlusion mask w, problem (10) is exended o he following one: 2 X min2 {c(n),v(n) }n=1,y 0 n=1 (n) {v s.. v(n) 1 + λ kw ek1. (13) = c(n) }2n=1 = B c(1) + D c(2) + e, y = B c(1) + D c(2) An ADMM-based solver similar o he Algorihm 3 is employed in order o solve problem (13). More specifically, he updae sep of he error vecor performed in sep 12 of he = w e + w h Algorihm 3 is followed by e+1 i (3) 1 (1) (2) B c+1 D c+1 + h µ. 7 ABLE 1: Parameers used in he conduced experimens. Secion r λ 1 max(q,j) WJ Sparse = Error + E XPERIMENAL E VALUAION he performance of he proposed mehod is assessed on synheic daa corruped by boh Gaussian and sparse, nongaussian noise (Secion 7.1), as well as on daa capured under consrained and in-he-wild condiions wih applicaions o (a) facial expression synhesis, (b) 2D and (c) 3D face age progression. 7.1 mehods accuraely recovered boh he join and individual componens. I is worh menioning ha he NN-`1 - successfully recovered all componens by uilizing only he rue rank of he join componen. In he oher hand, all he oher mehods require knowledge regarding he rue rank for boh join and individual componens. Furhermore, he S achieved same resuls o he NN-`1 - by reducing he compuaion ime more han five imes. Based on he performance of S on he synheic daa, we decided o uilize i in he experimens described bellow and refer i as hereafer. Furhermore, we esed he on synheic daa conaminaed by Gaussian error. he can implicily handle daa conaminaed by Gaussian noise by vanishing he error erm. ha is we se he regularizer λ in problems (7), (9), (11) λ i.e. E = 0. In such case, he Frobenius norm corresponding o he equaliy consrains X = J + A(1),, A(M ) ] + E, X = BG + (D(1) C(1) ),, A(M ) ] + E appearing in he corresponding augmened Lagrangian funcions are deemed as he appropriae regularizer for handling Gaussian noise. he RRE of all compared mehods are repored in able 2. As i can be seen, he proposed mehods accuraely recovered boh he join and individual componens. WA λ Synheic In his secion, he abiliy of o robusly recover he common and individual componens of synheic daa corruped by sparse non-gaussian noise, is esed. o his end, ses of marices {X = J + A + E Rd J }2i=1 of varying dimensions were generaed. In more deail, a rank-r join compo(1) (2) nen J R(q=d +d ) J was creaed from a random marix X = X(1), X(2) ] Rq J. Nex, he orhogonal o J rank(1) (2) r(1), r(2) common componens A and A were compued (1) (2) by A, A ] = (X J )(I VV ), where V was formed from he firs r columns of he row space of X. E is a sparse error marix wih 20% non-zero enries being sampled independenly from N (0, 1). he Relaive Reconsrucion Error (RRE) of he join and individual componens achieved by boh `1 - and NuclearNorm regularized (NN-`1 -) for a varying number of dimensions, join and individual ranks, are repored in able 2. he corresponding RRE obained by JIVE 12], 31], COBE 13], and RCICA 14] are also presened. As i can be seen, he proposed (a) (b) (c) Fig. 1: Procedure followed o generae daa conaminaed by sparse, non-gaussian noise. he efficiency of he JIVE and mehods was qualiaively evaluaed on real daa conaminaed by sparse, non-gaussian noise. In order o generae he corruped daa, we firsly superimposed he painings of Figure 1(a) wih he paining appeared in Figure 1(b) and subsequenly a sparse error marix was added. In each image he error marix has 20% non-zero enries being sampled independenly from N (0, 1). hen, he concaenaion of he generaed painings (Figure 1(c)) was given as inpu o he JIVE and. he join and individual componens as well as he corresponding error marices obained from he compared mehods are depiced in Figure 22. As i can be observed, he Inpu JIVE Fig. 2: Join, individual componens, and error marices produced by he compared JIVE and mehods. accuraely recovered boh he join and individual componens. On he oher hand, he join componens exraced from JIVE are no accurae, while he corresponding individual ones are conaminaed by he sparse error. his is due o he fac ha he JIVE is no robus o sparse, non-gaussian noise. 2. Addiional resuls can be found in he supplemenary maerial.

8 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS ABLE 2: Quaniaive recovering resuls produced by JIVE 12], COBE 13], RCICA 14], l 1 - (7), and NN-l 1 - (9) under Gaussian and gross non-gaussian noise. Each compared mehod was applied on he same daa generaed by uilizing each se of parameers. he average recovery accuracy and compuaion ime (in CPU seconds) were compued by repeaing he experimen 10 imes. ( d (1), d (2), J, r, r (1), r (2)) (500, 500, 500, 5, 10, 10) (1000, 1000, 1000, 10, 20, 20) (2000, 2000, 2000, 20, 40, 40) Mehod { J 2 / } J J 2 2 { A 2 / } A A 2 2 ime (in CPU seconds) F F i=1 F F i=1 non-gaussian Gaussian non-gaussian Gaussian non-gaussian Gaussian COBE JIVE e e RCICA e e l e e e e NN-l e e e e S e e e e COBE JIVE e e RCICA e e l e e e e NN-l e e e e S e e e e COBE JIVE e e RCICA e e l e e e e NN-l e e e e S e e e e Facial Expression Synhesis In his secion, we invesigae he abiliy of he o synhesize a se of differen expressions of a given facial image. Consider M daases, where each one conains images of differen subjecs ha depic a specific expression. In order o effecively recover he join and common componens, he faces of each daase should be pu in correspondence. herefore, heir N = 68 facial landmark poins are localized using he deecor 32], 33] and subsequenly employed o compue a mean reference shape. hen, he faces of each daase are warped ino a corresponding reference shape by using he piecewise affine warp funcion W( ) 34]. Afer applying he on he warped daases, he recovered componens can be used o synhesize M differen expressions of an unseen subjec. o do ha, he new (unseen) facial image is warped o he reference frame ha corresponds o he expression ha we wan o synhesize and subsequenly is given as inpu o solve (10). he performance of in FES ask is assessed by conducing inner- and cross-daabases experimens on MPIE 35], CK+ 36], and in-he-wild facial images colleced from he inerne (IW). he synhesized expressions obained by are compared o hose obained by he sae-of-he-ar BKRRR 37] mehod. In paricular, he BKRRR is a regression-based mehod ha learns a mapping from he Neural expression o he arge ones. hen, given he Neural face of an unseen subjec, new expressions are synhesized by employing he corresponding learned regression funcions. he performance of he compared mehods is measured by compuing he correlaion beween he vecorized forms of rue images and he reconsruced ones Conrolled Condiions In he firs experimen, 534 fronal images of MPIE daabase ha depic 89 subjecs under six expressions (i.e., Neural, Scream, Squin, Surprise, Smile, Disgus ) were employed o rain boh and BKRRR. hen, all expressions of 58 unseen subjecs from he same daabase were synhesized by using heir images corresponding o Neural expressions. In Figure 3(a), he average correlaions obained by he compared mehods for he differen expressions are visualized. As i can be seen he proposed mehod achieves he same accuracy o BKRRR wihou learning any kind of mappings beween he differen expressions of he same subjec. Specifically, he exracs only he individual componens of each expression and he common one. Furhermore, he performance of boh mehods is compared by performing a cross-daabase experimen on he CK+ daabase. More specifically, we employed he Neural, Smile, and Surprise images of MPIE for raining purposes while images of 69 subjecs (hree images per subjec) of CK+ were used as es ones. In Figure 3(b) we can see ha ouperforms BKRRR by a large margin. his is due o he fac ha he BKRRR performs he regression based on how close is he unseen Neural face o he raining ones. herefore, in cases where he unseen subjecs (e.g., subjecs of CK+) presen enough differences compared o he raining ones (e.g., subjecs of MPIE), he synhesized expressions are characerized as non-accurae. he synhesized expressions of subjecs 014 and 015 from MPIE produced by he BKRRR and are visualized in Figure 4. Clearly, he proposed mehod produces expressed images of higher qualiy compared o he BKRRR. Finally, he recovering accuracy of JIVE and in FES was also qualiaively assessed. o his end, he images used in he previous experimens were conaminaed by sparse error were subsequenly provided o he compared mehods. Figure 5 2 displays he obained componens and he corresponding error marices. Clearly, he proposed mehod successfully recovered all he componens. I is worh menioning ha he removed he added sparse noise as well as he occlusions produced by eyeglasses and painings (please see red doed boxes). Insead, he JIVE was no able o remove neiher he added noise nor he occlusions In-he-Wild Condiions As an addiional experimen, we colleced from he inerne 180 images depicing 60 subjecs wih Surprise, Smile, and Neural expressions (hree images for each subjec). hen, all he expressions were generaed by employing he Neural images and he BKRRR and mehods rained on MPIE. Figure 3(c) depics he obained correlaions for each subjec. Clearly, he ouperforms he BKRRR. Compared o he previous experimens, here is a drop in he performance for boh mehods.

9 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS (a) (b) (c) Fig. 3: Mean average correlaion achieved by JIVE and BKRRR mehods on (a) MPIE, (b) CK+, and (c) IW daabases. Inpu BKRRR G Inpu BKRRR (a) is no he case of BKRRR, which requires he correspondence of expressions across he raining subjecs. Collecing in-he-wild images of same subjecs under differen expressions is a very edious ask. In order o improve he performance of, we augmened he raining se wih anoher 1200 images from he WWB daabase 38] (400 images for each expression). As i can be observed in Figure 3(c), he in-he-wild rain se improved he accuracy of in boh CK+ and IW daases. Figure 6 depics examples synhesized in-he-wild expressions produced by he. he images from he Inpu column were given as inpu o he and subsequenly, he synhesized expressions were warped and fused wih he acual images 39]. Clearly, he produced expressions are characerized by high qualiy of boh expression and ideniy informaion. I is worh menioning ha synhesizes almos perfecly he inpu images wihou using any kind of informaion abou he depiced subjec. G (b) Fig. 4: Synhesized expressions of MPIE s subjec (a) 014 and (b) 015 produced by he BKRRR and mehods. JIVE JIVE JIVE Fig. 5: Join, individual componens and error marices produced by he compared JIVE and mehods. his is aribued o he fac ha he mehods were rained by employing only images capured under conrolled condiions. hus, synhesizing expressions of in-he-wild images is a very difficul ask. In order o alleviae his problem we can augmen he raining se wih in-he-wild images. Alhough he can be rained on in-he-wild images of differen subjecs, his 7.3 Face Age Progression In-he-Wild D age progression of an unseen subjec Face age progression consiss in synhesizing plausible faces of subjecs a differen ages. I is considered as a very challenging ask due o he fac ha he face is a highly deformable objec and is appearance drasically changes under differen illuminaion condiions, expressions, and poses. Various daabases ha conain faces a differen ages have been colleced in he las couple of years 40], 41]. Alhough hese daabases conain huge number of images, hey have some limiaions including limied images for each subjec ha cover a narrow range of ages and noisy age labels, since mos of hem have been colleced by employing auomaic procedures (crawlers). A new daabase ha overcomes he aforemenioned problems was recenly proposed in 42]. he AgeDB was manually colleced and annoaed. I consiss of images ha depic 568 subjecs from 0 o 101 years old. Annoaions in erms of age and ideniy of he depiced subjecs are provided. On average, here are 29 images ha span 50.3 years for each subjec. In order o rain he, he AgeDB was divided ino M = 10 age groups: 0-3, 4-7, 8-15, 16-20, 21-30, 31-40, 41-50, 51-60, 61-70, and hen, following he same procedure as in he FES ask, was employed o exrac he join and common componens from he warped images. he performance of in face age progression inhe-wild is qualiaively assessed conducing experimens on images from he FG-NE daabase 43]. o his end, we compare he performance of wih he Illuminaion Aware Age Progression (IAAP) mehod 1], Coupled Dicionary Learning (CDL) mehod 2], Deep Ageing wih Resriced Bolzmann

10 JOURNAL OF LAEX CLASS FILES, VOL. 14, NO. 8, AUGUS 2015 Inpu Inpu G G 10 Inpu DARB G Inpu IAAP CG EAP DARB G Fig. 7: Progressed faces produced by he compared mehods on he FG-NE daabase. Fig. 6: Synhesized in-he-wild expressions produced by he mehod. Machines (DARB) mehod 3], Craniofacial Growh (CG) 4] model, Exemplar-based Age Progression (EAP) 5] mehod, Face ransformer (F Demo) 44], and Recurren Face Aging (RFA) mehod 6]. In Figures 7, 8, progressed images produced by he compared mehods are depiced. Noe ha all he progressed faces have been warped back and fused wih he acual ones. Figure 92 depics faces synhesized by he DARB, IAAP, and mehods. By observing he resuls, i can be clearly seen ha he ideniy informaion is no preserved in he case of DARB. In paricular, he progressed faces of all subjecs for a specific age group are very similar among hem. I looks like all of hem were creaed by ransferring he skin colour from he inpu image o he same mean appearance. On he oher hand, he ideniy informaion in he faces produced by he proposed mehod remains. Finally, progressed examples faces in all of he agegroups produced by he are visualized in Figure D age progression of an unseen subjec Here he abiliy of he proposed -M mehod o perform 3D face age progression is demonsraed. Similarly o he 2D face age progression experimens presened previously, he AgeDB daabase was divided ino M = 6 age groups ( 21-30, 31-40, 41-50, 51-60, 61-70, 71+ ) and used o rain he M. In order o acquire he 3D raining daa for his ask he 3DMM-IW 30] was employed. he opimal shape and camera parameers were exraced by fiing he model o each one of he images of all age groups. In order o recover 3D shapes of high qualiy, we used he age and gender specific version of he LSFM shape model inroduced in 45] in order o describe ideniy and he blendshapes of 46] in order o describe facial expressions. Having recovered he 3D shape of each face, we compued he self-occlusion mask by using ray-racing (see firs row of Figure 11). hen he compleed join and individual componens of he grossly corruped and incomplee UV exures were obained by employing he -M. he join componens obained by applying a varian of JIVE wih missing values, i.e., JIVE-M and he -M on UV exures are displayed in Figure 112. By observing he resuls, we can clearly see ha he successfully removed he occlusions produced by eyeglasses and fingers in all images. his is aribued o he fac ha he marix `1 -norm loss adoped in, which effecively handles sparse noise of possibly large magniude. Similarly, in he 2D face aging experimens we can apply he -M o he recovered UV maps o learn componens ha can be used o age he UV exure of a es subjec. Since, he 3D shapes are produced by he LSFM model hey neiher have missing values nor are conaminaed by noise. herefore, o find aging componens for he 3D shape we used he sandard JIVE. In he es phase, he 3D shape of he es face is obained by using he 3DMM-IW algorihm 30]. hen he UV exure and he corresponding self-occlusion mask are compued by employing he recovered 3D shape. he progression of he exure of he es subjec in an age group is obained by solving he problem (13) (for he shape we use he problem in (10)). Progressed unseen subjecs in all age groups, projeced back in he image plane, are visualized in Figure 122. Having calculaed a progressed 3D exure image and a 3D shape, he resuling is projeced back in

11 JOURNAL OF LAEX CLASS FILES, VOL. 14, NO. 8, AUGUS 2015 Inpu F Demo CDL RFA Inpu Inpu CDL RFA Fig. 8: Progressed faces produced by he compared mehods on he FG-NE daabase. Inpu Warped Fig. 10: Progressed faces produced by he proposed mehod. DARB IAAP Inpu Warped DARB IAAP Fig. 9: Comparisons beween he IAAP, DARB, and mehods. he image plane using he camera parameers iniially acquired by fiing he 3DMM-IW in he es image. Figure 132 presens addiional resuls ha demonsrae he abiliy of he -M o perform no only age progression bu also compleion. For visualizaion purposes he compleed and age-progressed 3D faces produced by he -M were mapped on he progressed 3D shape. For each subjec, he original and wo side poses are depiced. he exraced by he 3DMM-IW 3D face Inpu JIVE-M -M Fig. 11: Inpu images and corresponding join componens produced by he compared JIVE-M and -M mehods. As i can be observed he proposed mehod is able o remove occlusions produced by fingers and glasses. model of he inpu image is displayed in he firs row. By observing he resuls i becomes apparen ha due o self-occlusions, he insance of he 3D model wih pose differen o he inpu one, conains huge areas of missing values (black color). his is no he case for he progressed and compleed resuls produced by he -M (second row). As i can be seen, he compleion of he regions wih missing daa is accurae and proves he significan represenaional power of he bases exraced by -M.

12 JOURNAL OF LAEX CLASS FILES, VOL. 14, NO. 8, AUGUS predefined value i.e., {5 years, 10 years, 20 years, 30 years} ( Proocol 5 years ) ( Proocol 10 years ) ( Proocol 20 years ) ( Proocol 30 years ) Fig. 14: ROC curves of on he proposed proocols. Original images corresponds o he resuls obained by employing he acual images. 18 Inpu Fig. 12: Progressed faces produced by he proposed -M, projeced back in he image plane using mean 3D face shapes and camera parameers acquired by fiing he 3DMM-IW M M Fig. 13: Progressed and compleed 3D exure images, produced by he proposed -M mehod, mapped on mean 3D face shapes. he 3D face models are visualized in he original and wo side poses, in order o make he differences beween he compleed and missing daa visible. In order o assess he performance of, he following procedure was performed. For each fold of a specific proocol he raining images were spli ino M = 10 age-groups and subsequenly he was applied on heir warped version in order o exrac he join and individual componens. All images of each raining pair were hen progressed ino M = 10 age groups resuling ino 10 new pairs. he progressed images of six subjecs are depiced in Figure 10. As we waned o represen each pair by using a single feaure, gradien orienaions were exraced from he corresponding images and subsequenly he mean value of heir cosine difference was employed as he pair s feaure. M differen Suppor Vecor Machines (SVM) were rained by uilizing he exraced feaures. Finally, he scores produced by all he SVMs were laely fused via an SVM. In Figure 14, Receiver Operaing Characerisic (ROC) curves compued based on he 10 folds of each one of he proposed proocols are depiced. he corresponding mean classificaion accuracy and Area Under Curve (AUC) are repored in able 3. In order o assess he effec of progression, he resuls obained ABLE 3: Mean AUC and Accuracy on he proposed proocols. Original Images Age-invarian face verificaion in-he-wild he performance of he is also quaniaively assessed by conducing age-invarian face verificaion experimens. Following he successfully used verificaion proocol of he LFW daabase 47], we propose four new age-invarian face verificaion proocols based on he AgeDB daabase. Each one of he proocols was creaed by spliing he AgeDB daabase ino 10 folds, wih each fold consising of 300 inra-class and 300 iner-class pairs. he essenial difference beween hese proocols is ha in each proocol he age difference of each pair s faces is equal o a AUC Accuracy AUC Accuracy 5 years years years years by uilizing only he original images are also provided. Some ineresing observaions are drawn from he resuls. Firsly, he improvemen in accuracy validaes ha he ideniy informaion of he face remains afer he -based progression. Furhermore, he improvemen in accuracy is higher when he age difference of images of each pair is large enough. For insance, he improvemen in accuracy in Proocol 30 years is higher han he corresponding in Proocol 5 years. Finally, he produced resuls jusify ha he problem of age-invarian face verificaion becomes more difficul when he age 5 difference is very large (e.g., 30 years).

13 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS Fig. 15: ROC curve of he and IAAP on FG-NE daabase. he performance of in age-invarian face verificaion is also compared agains he IAAP 1] by conducing experimens on he FG-NE daabase. he experimenal proocol employed is he following. By selecing images where he depiced subjecs are older han he age of 18 years, we creaed a subse of he FG-NE daabase consiss of 518 images. hen, based on he seleced images we creaed 1250 inra-class pairs, i.e., he images of each pair depic he same subjec under differen ages, and anoher 1250 iner-class pairs. he experimen proocol was finally creaed by dividing he pairs on 5 folds wih each fold conaining 250 inra-class and 250 iner-class pairs. All images were hen progressed by employing he and IAAP mehods. A similar o he previous experimen procedure was followed in order o perform he age-invarian verificaion. he produced ROC curves are displayed in Figure 15. As i can be observed, he proposed mehod ouperforms he IAAP by a large margin, indicaing ha he produces progressed images of high qualiy wihou removing he ideniy informaion. 8 CONCLUSIONS A general framework for robus recovering of join and individual variance among several daases possibly conaminaed by gross, non-gaussian errors and missing values has been proposed in his paper. Four differen models namely l 1 -, NN-l 1 -, S, and -M have been proposed. Furhermore, based on he recovered componens from he raining daa, wo novel opimizaion problems ha exrac he join and individual componens of an unseen es sample, are inroduced. he effeciveness of he was firs esed by conducing experimens on synheic daa. Moreover, exensive experimens were conduced on facial expression synhesis and 2D and 3D face age progression by uilizing five daases capured under boh conrolled and in-he-wild condiions. he experimenal resuls validae he effeciveness of he proposed framework over he saeof-he-ar. ACKNOWLEDGMENS his work was parially funded by EPSRC Projec EP/N007743/1 (FACER2VM). Finally, we would like o hank Alina Leidinger for some insighs on an earlier version of his manuscrip. REFERENCES 1] I. Kemelmacher-Shlizerman, S. Suwajanakorn, and S. M. Seiz, Illuminaion-aware age progression, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), ] X. Shu, J. ang, H. Lai, L. Liu, and S. Yan, Personalized age progression wih aging dicionary, in Proceedings of IEEE Inl Conference on Compuer Vision (ICCV), 2015, pp ] C. N. Duong, K. Luu, K. G. Quach, and. D. Bui, Longiudinal face modeling via emporal deep resriced bolzmann machines, Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), ] N. Ramanahan and R. Chellappa, Modeling age progression in young faces, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), vol. 1, 2006, pp ] C.-. Shen, W.-H. Lu, S.-W. Shih, and H.-Y. M. Liao, Exemplar-based age progression predicion in children faces, in IEEE Inl Symposium on Mulimedia (ISM), 2011, pp ] W. Wang, Z. Cui, Y. Yan, J. Feng, S. Yan, X. Shu, and N. Sebe, Recurren face aging, Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), ] H. Hoelling, Relaions beween wo ses of variaes, Biomerika, vol. 28, no. 3/4, pp , ] A. Klami and S. Kaski, Probabilisic approach o deecing dependencies beween daa ses, Neurocompuing, vol. 72, no. 1, pp , ] M. A. Nicolaou, V. Pavlovic, and M. Panic, Dynamic probabilisic cca for analysis of affecive behavior and fusion of coninuous annoaions, IEEE ransacions on Paern Analysis and Machine Inelligence (PAMI), vol. 36, no. 7, pp , ] L. R. ucker, An iner-baery mehod of facor analysis, Psychomerika, vol. 23, no. 2, pp , ] A. Klami, S. Viranen, E. Leppäaho, and S. Kaski, Group facor analysis, IEEE ransacions on Neural Neworks and Learning Sysems (NNLS), vol. 26, no. 9, pp , ] E. F. Lock, K. A. Hoadley, J. S. Marron, and A. B. Nobel, Join and individual variaion explained (jive) for inegraed analysis of muliple daa ypes, he annals of applied saisics, vol. 7, no. 1, p. 523, ] G. Zhou, A. Cichocki, Y. Zhang, and D. P. Mandic, Group componen analysis for muliblock daa: Common and individual feaure exracion, IEEE ransacions on Neural Neworks and Learning Sysems (NNLS), vol. 11, pp , ] Y. Panagakis, M. Nicolaou, S. Zafeiriou, and M. Panic, Robus correlaed and individual componen analysis, IEEE ransacions on Paern Analysis and Machine Inelligence (PAMI), vol. 38, no. 8, pp , ] P. J. Huber, Robus saisics. Springer, ] C. Sagonas, Y. Panagakis, S. Zafeiriou, and M. Panic, Robus join and individual variance explained, Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), ] D. P. Bersekas, Consrained opimizaion and Lagrange muliplier mehods. Academic press, ] L. Vandenberghe and S. Boyd, Semidefinie programming, SIAM review, vol. 38, no. 1, pp , ] B. K. Naarajan, Sparse approximae soluions o linear sysems, SIAM journal on compuing, vol. 24, no. 2, pp , ] M. Fazel, Marix rank minimizaion wih applicaions, Ph.D. disseraion, Sanford Universiy, ] D. L. Donoho, For mos large underdeermined sysems of linear equaions he minimal 1-norm soluion is also he sparses soluion, Communicaions on pure and applied mahemaics, vol. 59, no. 6, pp , ] E. J. Candès, X. Li, Y. Ma, and J. Wrigh, Robus principal componen analysis? Journal of he ACM (JACM), vol. 58, no. 3, p. 11, ] J.-F. Cai, E. J. Candès, and Z. Shen, A singular value hresholding algorihm for marix compleion, SIAM Journal on Opimizaion, vol. 20, no. 4, pp , ] J. Wrigh and Y. Ma, Dense error correcion via l 1 -minimizaion, IEEE ransacions on Informaion heory, vol. 56, no. 7, pp , ] C. Georgakis, Y. Panagakis, and M. Panic, Dynamic behavior analysis via srucured rank minimizaion, Inl Journal of Compuer Vision (IJCV), pp. 1 25, ] C. Sagonas, Y. Panagakis, S. Zafeiriou, and M. Panic, Robus saisical fronalizaion of human and animal faces, Inl Journal of Compuer Vision (IJCV), pp. 1 22, ], RAPS: Robus and efficien auomaic consrucion of personspecific deformable models, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), 2014, pp ] Y. Peng, A. Ganesh, J. Wrigh, W. Xu, and Y. Ma, RASL: Robus alignmen by sparse and low-rank decomposiion for linearly correlaed images, IEEE ransacions on Paern Analysis and Machine Inelligence (PAMI), vol. 34, no. 11, pp , ] V. Blanz and. Veer, A morphable model for he synhesis of 3d faces, in Proceedings of he 26h annual conference on Compuer graphics and ineracive echniques. ACM Press/Addison-Wesley Publishing Co., 1999, pp

14 JOURNAL OF L A E X CLASS FILES, VOL. 14, NO. 8, AUGUS ] J. Booh, E. Anonakos, S. Ploumpis, G. rigeorgis, Y. Panagakis, and S. Zafeiriou, 3d face morphable models in-he-wild, Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion (CVPR), ] C. Sagonas, Y. Panagakis, S. Arunkumar, N. Raha, and S. Zafeiriou, Back o he fuure: A fully auomaic mehod for robus age progression, in Proceedings of IEEE Inl Conference on Paern Recogniion (ICPR), 2016, pp ] J. Alabor-i-Medina, E. Anonakos, J. Booh, P. Snape, and S. Zafeiriou, Menpo: A comprehensive plaform for parameric image alignmen and visual deformable models, in Proceedings of he ACM Inl Conference on Mulimedia, Open Source Sofware Compeiion, Orlando, FL, USA, November 2014, pp ] C. Sagonas, E. Anonakos, G. zimiropoulos, S. Zafeiriou, and M. Panic, 300 faces in-he-wild challenge: daabase and resuls, Image and Vision Compuing (IMAVIS), vol. 47, pp. 3 18, ] I. Mahews and S. Baker, Acive appearance models revisied, Inl Journal of Compuer Vision (IJCV), vol. 60, no. 2, pp , ] R. Gross, I. Mahews, J. Cohn,. Kanade, and S. Baker, Muli-pie, Image and Vision Compuing (IMAVIS), vol ] P. Lucey, J. F. Cohn,. Kanade, J. Saragih, Z. Ambadar, and I. Mahews, he exended cohn-kanade daase (ck+): A complee daase for acion uni and emoion-specified expression, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion, Workshops (CVPR-W), 2010, pp ] D. Huang and F. De la orre, Bilinear kernel reduced rank regression for facial expression synhesis, in Proceedings of European Conference on Compuer Vision (ECCV). Springer, 2010, pp ] A. Mollahosseini, B. Hassani, M. J. Salvador, H. Abdollahi, D. Chan, and M. H. Mahoor, Facial expression recogniion from world wild web, Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion, Workshops (CVPR-W), ] P. Pérez, M. Gangne, and A. Blake, Poisson image ediing, in ACM ransacions on Graphics (OG), vol. 22, no. 3, 2003, pp ] B.-C. Chen, C.-S. Chen, and W. H. Hsu, Cross-age reference coding for age-invarian face recogniion and rerieval, in Proceedings of European Conference on Compuer Vision (ECCV). Springer, 2014, pp ] R. Rohe, R. imofe, and L. Van Gool, Dex: Deep expecaion of apparen age from a single image, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion, Workshops (CVPR-W), 2015, pp ] S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kosia, and S. Zafeiriou, Agedb: he firs manually colleced, in-he-wild age daabase, in Proceedings of IEEE Inl Conference on Compuer Vision & Paern Recogniion, Workshops (CVPR-W), ] A. Laniis, C. J. aylor, and. F. Cooes, oward auomaic simulaion of aging effecs on face images, IEEE ransacions on Paern Analysis and Machine Inelligence (PAMI), vol. 24, no. 4, pp , ] Face ransformer, hp://morph.cs.sandrews.ac.uk/ransformer/. 45] J. Booh, A. Roussos, A. Ponniah, D. Dunaway, and S. Zafeiriou, Large scale 3d morphable models, Inl Journal of Compuer Vision (IJCV), pp. 1 22, ] C. Cao, Y. Weng, S. Zhou, Y. ong, and K. Zhou, Facewarehouse: A 3d facial expression daabase for visual compuing, IEEE ransacions on Visualizaion and Compuer Graphics, vol. 20, no. 3, pp , ] G. B. Huang and E. Learned-Miller, Labeled faces in he wild: Updaes and new reporing procedures, Dep. Compu. Sci., Univ. Massachuses Amhers, Amhers, MA, USA, ech. Rep, pp , Chrisos Sagonas received he BSc and MSc degrees in Compuer Science from Arisole Universiy of hessaloniki, Greece in 2009 and 2011, respecively. In 2012 he joined he Inelligen Behaviour Undersanding Group (IBUG) a Compuing Deparmen, Imperial College London, where he received he PhD degree in During his PhD, Chrisos worked mainly on machine-learning and compuer vision models for auomaed facial landmark localizaion under oally unconsrained condiions. His curren research ineress include machine learning and compuer vision wih applicaions o human face analysis. Evangelos Ververas graduaed in Sepember 2016 from he Deparmen of Elecrical and Compuer Engineering in Arisole Universiy of hessaloniki, in Greece. He joined he Inelligen Behavior Undersanding Group (IBUG) a he Deparmen of Compuing, Imperial College London, in Ocober 2016 and he is currenly working as a PhD Suden/eaching Assisan under he supervision of Dr. Sefanos Zafeiriou. His research focuses on machine learning and compuer vision models for 3D reconsrucion and analysis of human faces. Yannis Panagakis is a Lecurer (Assisan Professor equivalen) in Compuer Science a Middlesex Universiy London and a Research Fellow a he Deparmen of Compuing, Imperial College London. His research ineress lie in machine learning and is inerface wih signal processing, high-dimensional saisics, and compuaional opimizaion. Specifically, Yannis is working on models and algorihms for robus and efficien learning from high-dimensional daa and signals represening audio, visual, affecive, and social informaion. He has been awarded he presigious Marie-Curie Fellowship, among various scholarships and awards for his sudies and research. Yannis currenly serves as an Associae Edior of he Image and Vision Compuing Journal. He co-organized he BMVC 2017 and several workshops and special sessions in op venues such as ICCV. He received his PhD and MSc degrees from he Deparmen of Informaics, Arisole Universiy of hessaloniki and his BSc degree in Informaics and elecommunicaion from he Universiy of Ahens, Greece. Sefanos P. Zafeiriou (M09) is currenly a Reader in Machine Learning and Compuer Vision wih he Deparmen of Compuing, Imperial College London, London, U.K, and a Disinguishing Research Fellow wih Universiy of Oulu under Finish Disinguishing Professor Programme. He was a recipien of he Presigious Junior Research Fellowships from Imperial College London in 2011 o sar his own independen research group. He was he recipien of he Presiden s Medal for Excellence in Research Supervision for He has received various awards during his docoral and pos-docoral sudies. He currenly serves as an Associae Edior of he IEEE ransacions on Affecive Compuing and Compuer Vision and Image Undersanding journal. In he pas he held ediorship posiions in IEEE ransacions on Cyberneics he Image and Vision Compuing Journal. He has been a Gues Edior of over six journal special issues and co-organised over 13 workshops/special sessions on specialised compuer vision opics in op venues, such as CVPR/FG/ICCV/ECCV (including hree very successfully challenges run in ICCV13, ICCV15 and CVPR 17 on facial landmark localisaion/racking). He has co-auhored over 55 journal papers mainly on novel saisical machine learning mehodologies applied o compuer vision problems, such as 2-D/3-D face analysis, deformable objec fiing and racking, shape from shading, and human behaviour analysis, published in he mos presigious journals in his field of research, such as he IEEE -PAMI, he Inernaional Journal of Compuer Vision, he IEEE -IP, he IEEE -NNLS, he IEEE -VCG, and he IEEE -IFS, and many papers in op conferences, such as CVPR, ICCV, ECCV, ICML. His sudens are frequen recipiens of very presigious and highly compeiive fellowships, such as he Google Fellowship x2, he Inel Fellowship, and he Qualcomm Fellowship x3. He has more han 4500 ciaions o his work, h-index 36. He is he General Chair of BMVC 2017.