POSITION-PATCH BASED FACE HALLUCINATION VIA LOCALITY-CONSTRAINED REPRESENTATION. Junjun Jiang, Ruimin Hu, Zhen Han, Tao Lu, and Kebin Huang

Transcription

1 IEEE International Conference on ultiedia and Expo POSITION-PATCH BASED FACE HALLUCINATION VIA LOCALITY-CONSTRAINED REPRESENTATION Junjun Jiang, Ruiin Hu, Zhen Han, Tao Lu, and Kebin Huang National Engineering Research Center for ultiedia Software, School of Coputer, Wuhan University, Wuhan, 437, China {jiangjunjun, hr964, hanzhen, lut, ABSTRACT Instead of using probabilistic graph based or anifold learning based odels, soe approaches based on position-patch have been proposed for face hallucination recently. In order to obtain the optial weights for face hallucination, they represent iage patches through those patches at the sae position of training face iages by eploying least square estiation or convex optiization. However, they can hope neither to provide unbiased solutions nor to satisfy locality conditions, thus the obtained patch representation is not the best. In this paper, a sipler but ore effective representation schee Locality-constrained Representation (LcR) has been developed, copared with the Least Square Representation (LSR) and Sparse Representation (SR). It iposes a locality constraint onto the least square inversion proble to reach sparsity and locality siultaneously. Experiental results deonstrate the superiority of the proposed ethod over soe state-of-the-art face hallucination approaches. Index Ters face hallucination, super-resolution, Locality-constrained Representation, position-patch. INTRODUCTION Fig.. Flow diagra of the position-patch based face hallucination fraework. To obtain the optial weight vector w= [ w, w,, w ] is the key issue. Learning-based face iage super-resolution, referred as face hallucination, has attracted nuerous heed of researchers []. The availability of high-resolution (HR) face iage, which basically eans that it is reproduced with a high level of detail, allows any applications, such as robust face recognition, 3D facial odeling and Intelligent Video Surveillance. Eploying a training set of face iages, the face hallucination technology could generate a HR face iage fro a low-resolution (LR) one. By building a co-occurrence odel, the issing details of the input LR face iage could be ade up globally (global face space paraeter estiation, e.g., [3],[]), locally (local iage priitive intensity restoration, e.g., [],[5],[6],[7], [9],[]), or both (e.g., [],[8]). Since the local patch based ethods can greatly iprove the quality of hallucinated faces, they gain uch ore concern in the super-resolution research counity. Inspired by the results of face analysis that huan face is a class of highly structured object and consequently position plays an increasingly iportant role in its reconstruction, soe position-patch based face hallucination ethods have been proposed. The key issue of those ethods is how to represent the iage patch and obtain the optial weight vector (Fig.). For exaple, a et al. [6-7] eploy patches at sae position of all training face iages to hallucinate the HR iage patch through solving a least square proble. Due to reasonable utilization of position inforation, it is ore efficient and the perforance is excellent, copared with soe anifold learning based ethods [4]. However, when the nuber of the training saples is uch larger than the diension of the patch, the solution of least square estiation is not unique [8]. In order to solve the biased solutions, Jung et al. [9] take a convex constrained optiization to replace the least square estiation to obtain ore suitable solution. However, it overephasizes the sparsity and neglects the locality condition, which is ore iportant than sparsity in revealing the non-linear anifold structure of face iage patch space [-3]. Thus, the obtained patch representation is still not the best. Since the local iage patches share the greatest siilarity, it would be ore accurate to represent one iage patch using a few neighbor patches, leading to a ore effective representation of iage patches. Contribution. In this paper, we introduce a novel patch representation ethod for face iage hallucination, called Locality-constrained Representation (LcR) which iposes a locality constraint onto the least square inversion proble. Instead of representing one input iage patch collaboratively [6-7] or sparsely [8-9], we utilize the locality constraint to project each iage patch into its several neighborhoods, thus achieving sparsity and locality / $6. IEEE DOI.9/ICE..5

2 siultaneously. There are three characteristics of the proposed ethod: () By introducing a locality constraint, it akes the solution of the least square proble fixed, eanwhile it captures fundaental siilarities between neighbor patches; () Copared with traditional locality preserving ethods that use a fixed nuber of neighbors for reconstruction, it chooses the ost relevant patches adaptively to avoid over- or under-fitting, generating sharper contours and richer details; (3) While sparse representation ethods ust solve the -nor constrained iniization proble, it derives an analytical solution to the constrained least squares proble, significantly reducing the coputational coplexity. The rest of this paper is organized as follows. Section briefly reviews LSR, SR and then presents our LcR ethod. Section 3 describes the proposed face hallucination ethod based on LcR. Experiental results and analyses are provided in Section 4, and we conclude this paper in Section 5.. LOCALITY-CONSTRAINED REPRESENTA- TION This section reviews two existing patch representation schees, LSR and SR, and introduces our proposed LcR ethod. Let Y denote the training face iages, =,...,, where is the saple nuber. Each face iage is divided into N sall overlapping patch sets{ Y ( i, j) i U, j V, N = UV, U represents the patch nuber in every colun, V represents the patch nuber in every row, and the ter (, i j) indicates the position inforation (as illustrated in Fig.). For the patch located at position (, i j ), it can be represented by training saples located at the sae position with a weight vector, wi (, j) = [ w( i, j), w( i, j),..., w ( i, j)]. For one input face iage denoted in patches as { X( i, j) i U, j V, different representation schees convert each patch into a -diensional optial weight vector to generate the final patch representation. ο (,) (,) i U j ( i, j) (, ij ) (, ij) ( i+, j) ( i+, j) ( UV, ) u Fig.. Dividing a face into N=UV patches. The ter (i, j) indicates the coordinate of one patch in the patch coordinate systeο -uv. patch_size and overlap denote the side pixels of one square patch and the overlap pixels between patches respectively... Least Square Representation LSR [6-7] uses patches fro all training saples at the V ( UV, ) v sae position (, i j ) to represent one patch X(, i j: ) Xij (, ) = w(, ijy ) (, ij) + e () = where e is the reconstruction error. The optial weight can be solved by the following constrained least square fitting proble: * (, ) = argin (, ) (, ) (, ) wi (, j) = st.. w (, ) i j = = w i j X i j w i j Y i j * where w (, i j) = [ w(, i j), w(, i j),..., w (, i j)] is the optial - diensional weight vector for X(, i j. ).. Sparse Representation In practice, the solution wi (, j ) to equation () ay not unique and a possible way is to ipose several regularization ters onto it. Jung et al. [9] introduce the Sparse Representation theory and use a sall subset of patches to represent X(, i j ) instead of perforing collaboratively over the whole training saples. It converts equation () to a standard sparse representation proble: in wi (, j ), () (3) = st.. X(, i j) w (, i j) Y (, i j) ε where denotes the -nor. This sparsity constraint can not only ensure that the under-deterined equation have an exact solution but also allow the learned representation for each patch X(, i jto ) capture salient properties, thus achieves uch less reconstruction error..3. Locality-constrained Representation The work in [9] ephasizes that a strong sparsity of the weight vector wi (, j ) is iportant in representing the input patch but it doesn t investigate uch the role of a locality constraint, which is ore essential than sparsity in revealing the true geoetry of a nonlinear anifold [-3]. In other words, SR ight represent one input patch by training data fro distinct patches. Following the intuition that Local Coordinate Coding (LCC) [-3] could explicitly encourage the representation to be local, we incorporate a locality constraint instead of the sparsity constraint into the objective function. Additionally, we adopt shrinkage easures as in ridge regression on the weight vector wi (, j ). Based on the above discussions, our objective function is forulated as follows: in dij (, ) wij (, ), (4) = st.. X(, i j) w (, i j) Y (, i j) ε where denotes a point wise vector product. And dij (, ) is a -diensional vector that penalizes the distance between X(, i jand ) each training patch at the sae position. It is siply deterined by the squared Euclidean distance: 3

3 d(, i j) = X(, i j) Y (, i j), (5) Equivalently, equation (4) can be written as: X( i, j) w ( i, j) Y ( i, j) + = w (, i j) = argin wi (, j) τ w( i, j) d( i, j) = The above objective function consists of two parts: The first ter easures the reconstruction error and the second ter preserves locality in representation. τ is a regularization paraeter balancing the contribution of the reconstruction error and locality of the solution. When τ =, LcR reduces to LSR. In our ethod, the locality constraint has a twofold role. On one hand, it akes the solution fixed; On the other hand, it introduces a locally constrained sparse representation to each patch, yet this sparsity is uch weaker than that in the sense of -nor. This will be discussed in Section Optiization Following [3], the solution of a regularized least square in equation (6) can be derived analytically by: wi (, j) = ( Gi (, j) + τ D)\ (7) where D is a diagonal atrix: (6) D (, ) = d i j, (8) G is local covariance atrix for X(, i j ) as: Define C as: T Gi (, j) = CC. (9) C = ( X( i, j) ones(, ) Y( i, j)) () where Yij (, ) is a atrix with its coluns being training patches Y (, i j ), ones(, ) is a row vector of ones, and the superscript T eans transpose. The final optial weight is obtained by rescaling it so that w(, i j) =. = 3. FACE HALLUCINATION VIA LOCAL- ITY-CONSTRAINED REPRESENTATION The training set is coposed of HR and LR face iage pairs. HR face iages are denoted as { Y H = and their LR counterparts are denoted as{ Y L =. The priary task is to reconstruct the HR face iage X H fro the observed LR face iage X L. At the beginning, we divide the training face iages and the input LR face iage into patches using the sae dividing schee. In order to avoid changing the size of the iage because of cutting or filling, we take a "fallback" dividing strategy as illustrated in Fig.3. Therefore, the patch nuber in every colun, U, and the patch nuber in every row, V, can be obtained: irow - overlap U = ceil patch_size - overlap () Algorith Face hallucination via LcR.. Input: Training set { Y H = and { Y L =, a LR iage X L, patch_size, overlap and regularization paraeterτ.. Copute U and V: U = ceil((irow - overlap)/(patch_size - overlap)) V = ceil((icol - overlap)/(patch_size - overlap)) Divide each of the training iages and the input LR iage into N sall patches according to the sae location of face, { YH (, i j) i U, j V =, { YL (, i j) i U, j V = and { X L ( i, j) i U, j V, respectively. 4. For i = to U do 5. For j = to V do Calculate the distance between the input LR iage patch XL (, i j ) and all the training iage patches { YL ( i, j ) = at position (, i j ): d(, i j) = XL(, i j) YL (, i j), Copute the optial weight vector w * (, i j) for the input LR iage patch XL (, i j ) with the LR training iage patches { Y ( i, j ): XL(, i j) w(, i j) YL (, i j) + = w (, i j) = argin wij (, ) τ w( i, j) d( i, j) = Construct the HR patch by * H(, ) (, ) (, ) = H X i j = w i j Y i j. 6. End for 7. End for 8. Integrate all the reconstructed HR patches above according to the original position. The final HR iage X H can be generated by averaging pixel values in the overlapping regions. 9. Output: HR hallucinated face iage X H. icol - overlap V = ceil () patch_size - overlap where irow and icol denote the rows and coluns of a face iage, respectively, ceil(x) is the function that rounds the eleents of x to the nearest integers towards infinity. For each input LR patch iage, by iposing locality constraint onto the objective function, the representation can achieve not only locality but also sparsity. The entire face hallucination process is suarized in Algorith. Fig.3. Illustration of the fallback dividing strategy. (a) When the current patch slid off the right edge, let it fallback subject to the right edge; (b) when the current patch slid off the botto edge, let it fallback subject to the botto edge. L 4

4 4.. Database 4. EXPERIENT RESULTS In the experients, we use the frontal and pre-aligned iages of FEI Face Database [4]. The database contains 4 iages fro subjects ( en and woen) and each subject has two frontal iages, one with a neutral expression and the other with a siling facial expression. Huan faces in the database are ainly fro 9 and 4 years old with distinct appearances, hairstyles and adornents (Fig.4). All the iages are cropped to pixels, and we choose 36 iages (8 subjects) as the training set, leaving the rest 4 iages ( subjects) for testing. Therefore, all the test iages are not in the training set. The LR iages are fored by soothing (an averaging filter of size 4 4) and down-sapling (by a factor of 4, thus the size of LR face iages are 3 5 pixels) fro corresponding HR iages. (a) (b) (c) (d) (e) (f) (g) Fig.5. Coparison of results based on different ethods. (a) Input LR faces. (b) Wang et al. [3]. (c) Chang et al. [4] (d) a et al. [7] (e) Jung et al. [9] (f) Our ethod. (g) Original HR faces. (Note that the effect is ore pronounced if the figure of the electronic version is zooed.). Fig.4. Soe training faces in FEI Face Database. 4.. Paraeter Settings We copare the proposed ethod with Wang s eigentransforation ethod [3] and soe patch based ethods, such as Chang s Neighbor Ebedding ethod [4], a s LSR ethod [7] and Jung s SR ethod [9], under FEI Face Database. In order to obtain the best perforance, we adjust the paraeters for each coparative ethod. For Wang s eigentransforation ethod, we let the variance accuulation contribution rate of PCA be 98.5% (around bases). For the sake of fair copetition, we set the HR patch size to in Chang s Neighbor Ebedding ethod, LSR, SR and our LcR ethod and the overlap between neighbor patches is 4 pixels while corresponding LR iage patch size is set to 3 3 with overlap of pixel. In the experients, we find that the ore overlap the better the final result is. Obviously, the ore overlap the longer the running tie is. Therefore, we ust balance perforance and efficiency. The nuber of neighbors in Chang s Neighbor Ebedding is set to 5. For Jung s SR ethod, we set error tolerance to. and use a prial-dual algorith [6] for sparse representation proble as in their ethod. Our algorith has only one free paraeterτ. In our experience, τ is set to.4 and we will deonstrate the perforance versus the value of τ in Section Results The experiental results are shown in Fig.5. Due to space liitation, we don t provide all the results here. We can observe that () Patch based ethods outperfor global faces reconstruction ethods. In Wang s eigentransforation ethod, it hardly aintains global soothness but also leads to ghost artifacts around contour. Copared with that, patch based ethods show their superiority and further enhance the edges and textures. This is ainly because Wang s ethod is based on a statistical ode, which could not reveal the distribution of data when the training saples are not sufficient (e.g., 36 training saples in our experients while the feature diension is 3 5 = 75); () Position-patch based ethods are better than Chang s Neighbor Ebedding based patch ethod [4]. Fro Fig.5 (c), we can see that the hallucinated face iages by Chang s ethod have blurring effects, priarily due to over or under-fitting. In addition, by using the position inforation of huan faces in face reconstruction, position-patch based ethods capture the latent seantic inforation (e.g., patch position), while Chang s ethod considers only the aspect of reconstruction and ignores the seantic features of faces; (3) A further exaination shows that our ethod successfully captures ore high frequency coponents and fewer artifacts than a and Jung s position-patch based ethods (see the outh of -th row, the edge of 3-th row and the face contours of 5-th and 6-th rows in Fig.5). The excellent visual quality of our ethod owes to the LcR strategy; (4) The PSNR It is publicly available on 5

5 and SSI [5] by different ethods are shown in box- -plots in Fig.6. Again, we can see that our ethod gives higher PSNR and SSI values than those of other four ethods. Specifically, the average PSNR and SSI iproveents of LcR based face hallucination ethod over the second best ethod (i.e., Jung s SR based ethod [9]) are.65 db and.97, respectively. PSNR(dB) SSI connecting the optial weight vectors wi (, j) of each patch of all 4 test iages, for different representation ethods (Fig.8). The plots show that reconstruction weight vector under LSR is not sparse for it treats all the saples indifferently. Our LcR obtains a sparse result to soe extent when copared with SR. And it allows us to ensure that LcR can truly reveal the iage space, which is ebedded in a nonlinear anifold [4]. At the sae tie, we also find an encouraging phenoenon. The weights obtained by LSR and SR are equably distributed on both sides of zero, while LcR gains a non-negative reconstruction weights (ost values are higher than zero as shown in the botto left of Fig.8). This phenoenon can be easily explained if we use several neighbor patches of one iage patch to represent it, each iage patch should have a positive contribution. [3] [4] [7] [9] Our ethod.65 [3] [4] [7] [9] Our ethod Fig.6. PSNR (left) and SSI (right) copetitions of different ethods. The average PSNR and SSI of different ethods: [3] (PSNR = 7.6, SSI =.7684), [4] (PSNR = 3.6, SSI =.8944), [7] (PSNR = 3.9, SSI =.93), [9] (PSNR = 3., SSI =.948) and our ethod. (PSNR = 3.76, SSI =.945) LSR.. -. LSR 4.4. The Perforance of Different τ Additionally, we test the perforance of our ethod by choosing different regularization paraeter τ, which controls the weights of locality in the objective function. As shown in Fig.7, we can find that when τ =, which can be regarded as the sae case of LSR, the perforance of LcR is not the best. With the increase of locality, uch benefit on perforance can be gained. This iplies that the locality constrain is essential for patch reconstruction. However, we also should see that the value of τ could not be set too high because the reconstruction error in the objective function can not be ignored also. Therefore, selecting a proper regularization paraeter τ, LcR will gain good results. Average PSNR(dB) for all test iages the value of τ Average SSI for all test iages the value of τ Fig.7. The average PSNR and SSI values versus differentτ The Sparsity of LcR To get a better understanding of our approach, we plot the connected optial weight vector, which is fored by The higher the SSI value, the better is the face hallucination quality. The axiu value of SSI is, which eans a perfect reconstruction x 6 SR x 6 LcR x x 6 SR x 6 LcR x 6 Fig.8. The sparsity of the connected optial weight vector for different representation ethods (Top row: LSR, iddle row: SR, Botto row: LcR). The left figures are the plots of the connected optial weight vector and the right figures are the plots of sorted connected optial weight vector The locality of LcR In this subsection, we will check the locality of LSR, SR and LcR, respectively. At the beginning, we define an evaluation etric of locality, which called K-ean distance (K-D). Denote X(, i j ) as the inference patch and k { Y ( i, j) k C( K) as the K ost significant patches in training saples at the sae position (, i j ) and CK ( ) are their saple indexes. The larger the entries in the weight vector wi (, j ), the ore iportant the corresponding patch. 6

6 K-D is defined as: K-D( i,j)= (, ) k Xij Y ( ij, ) k C( K) K (3) Fro the definition of K-D, we learn that it easures the locality of a representation approach by coputing the distance between the inference patch and K largest weight patches in the representation of the inference patch. Therefore, a sall K-D indicates better preservation of locality. If the value of K-D is high, it will choose very distinct patches as the base, which is not consistence with the coon sense. The evaluation of locality of LSR, SR and LcR is given in Fig.9. Fro this coparison, we can clearly see that LcR can better capture the locality property than LSR and SR. LSR(5-D) LSR(-D) LSR(5-D) on the sae database have deonstrated the superiority of the proposed ethod over soe state-of-the-art ethods. Nonetheless, we set the regularization paraeter τ experientally, the ipact of noise and the regularization paraeter τ on the proposed LcR ethod needs further work. 6. ACKNOWLEDGENTS: The research was supported by the ajor national science and technology special projects (ZX34-3-3), the National Basic Research Progra of China (973 Progra) (9CB396), the National Natural Science Foundation of China (6773, 683, 6976, 678, 6384), the National Science Foundation of Hubei Province of China (CDB53), the Fundaental Research Funds for the Central Universities (9, 3), the science and technology Progra of Wuhan (73366) REFERENCES SR(5-D) LcR(5-D) SR(-D) SR(5-D) LcR(-D) LcR(5-D) Fig.9. Histogras of the K-ean distance for different ethods when K = 5,, 5, respectively. (Top row: LSR, iddle row: SR, Botto row: LcR). The red arrows indicate the edian values. 5. CONCLUSION AND FUTURE WORK In this paper, we have proposed a novel patch representation algorith, called Locality-constrained Representation (LcR), for face hallucination. It iposes a locality constraint rather than sparsity constraint onto least square inversion proble, aiing at obtaining the optial representation of one iage patch. In this ethod, each patch can be represented by a sall nuber of bases, which are adaptively selected fro its neighborhoods, thus achieving sparsity and locality siultaneously. Experiental results [] K. Jia and S. Gong, Generalized Face Super-Resolution, IEEE Trans. Iage Process., vol.7, no.6, pp , 8. [] C. Liu, H. Shu, and W. Freean, Face hallucination: Theory and practice, Int. J. Coput. Vis., vol.7, no., pp.5 34, 7. [3] X. Wang and X. Tang, Hallucinating face by eigen-transforation, IEEE Trans. Systes, an, and Cybernetics. Part C, vol.35, no.3, pp , 5. [4] H. Chang, D. Yeung, and Y. Xiong, Super-resolution through neighbor ebedding, in Proc. IEEE Conf. Coputer Vision and Pattern Recognition, pp. 75 8, 4. [5] B. Li, H. Chang, S. Shan and X. Chen, Aligning Coupled anifolds for Face Hallucination, IEEE Signal Process. Lett., vol.6, no., pp , 9. [6] X. a, J. Zhang, and C. Qi, Position-based face hallucination ethod, in Proc. IEEE Conf. ultiedia and Expo., pp.9 93, 9. [7] X. a, J. Zhang, and C. Qi, Hallucinating face by position-patch, Pattern Recognition, vol.43, no.6, pp ,. [8] J. Yang, H. Tang, Y. a, and T. Huang, Face hallucination via sparse coding, in Proc. IEEE Conf. Iage Process., pp.64 67, 8. [9] C. Jung, L. Jiao, B. Liu, and. Gong, Position-Patch Based Face Hallucination Using Convex Optiization, IEEE Signal Process. Lett., vol.8, no.6, pp ,. [] X. a, H. Huang, S. Wang, and C. Qi, A siple approach to ultiview face hallucination, IEEE Signal Process. Lett., vol.7, no.6, pp ,. [] J. Park and S. Lee, An exaple-based face hallucination ethod for single-frae, low-resolution facial iages, IEEE Trans. Iage Process., vol.7, no., pp.86 86, 8. [] K. Yu, T. Zhang, and Y. Gong, Nonlinear learning using local coordinate coding, in Proc. of Advances in Neural Inforation Processing Systes, pp.3-3, 9. [3] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, Locality-constrained Linear Coding for Iage Classification, in Proc. IEEE Conf. Coputer Vision and Pattern Recognition, pp ,. [4] C. Thoaz, G. Giraldi, A new ranking ethod for principal coponents analysis and its application to face iage analysis, Iage and Vision Coputing, vol.8, no.6, pp.9 93,. [5] Z. Wang, A. Bovik, H. Sheikh, and E. Sioncelli, Iage quality assessent: Fro error visibility to structural siilarity, IEEE Trans. Iage Process., vol.3, no.4, pp.6 6, 4. [6] E. Candes and J. Robergt, -agic: Recovery of Sparse Signals via Convex Prograing 5 [Online]. Available: 7