Hallucinating multiple occluded face images of different resolutions

Transcription

1 Pattern Recognton Letters 27 (2006) Hallucnatng multple occluded face mages of dfferent resolutons Ku Ja *, Shaogang Gong Department of Computer Scence, Queen Mary Unversty of London, Mle End Road, London E1 4NS, UK Avalable onlne 19 Aprl 2006 Abstract Learnng-based super-resoluton has recently been proposed for enhancng human face mages, known as face hallucnaton. In ths paper, we propose a novel algorthm to super-resolve face mages gven multple partally occluded nputs at dfferent lower resolutons. By ntegratng herarchcal patch-wse algnment and nter-frame constrants nto a Bayesan framework, we can probablstcally algn multple nput mages at dfferent resolutons and recursvely nfer the hgh-resoluton face mage. We address the problem of fusng partal magery nformaton through multple frames and dscuss the new algorthm s effectveness when encounterng occluded low-resoluton face mages. We show promsng results compared to those of exstng face hallucnaton methods from both smulated facal database and lve vdeo sequences. Ó 2006 Elsever B.V. All rghts reserved. Keywords: Super-resoluton (Hallucnaton); Bayesan framework; Image algnment 1. Introducton Super-resoluton s a technque to generate a hgher resoluton mage gven a sngle or a set of multple lowresoluton nput mages. The computaton requres the recoverng of lost hgh-frequency nformaton occurrng durng the mage formaton process. Super-resoluton can be performed usng two dfferent approaches: reconstructon-based (Elad and Feuer, 1997; Iran and Peleg, 1991; Schulz and Stevenson, 1996; Harde et al., 1997), and learnng-based (Freeman and Pasztor, 1999; Baker and Kanade, 2000a; Capel and Zsserman, 2001; Lu et al., 2001; Dedeoglu et al., 2004; Sun et al., 2003). The reconstructon-based approach nherts lmtatons when the magnfcaton factor ncreases. In ths paper, we focus on learnng-based superresoluton, when appled to the human face, also commonly known as hallucnaton (Baker and Kanade, 2000b). Capel and Zsserman (2001) used egenfaces from a tranng face database as model pror to constran and * Correspondng author. Fax: E-mal addresses: chrsja@dcs.qmul.ac.uk (K. Ja), sgg@dcs.qmul. ac.uk (S. Gong). super-resolve low-resoluton face mages. To further mprove the performance, they dvded the human face nto sx unrelated parts and appled PCA on them separately. Combned wth a MAP estmator, they can recover the result from a hgh-resoluton egenface space. A smlar method was proposed by Baker and Kanade (2000a). Rather than usng the whole or parts of a face, they establshed the pror based on a set of tranng face mages pxel by pxel usng Gaussan, Laplacan and feature pyramds. Freeman and Pasztor (1999) took a dfferent approach for learnng-based super-resoluton. Specfcally, they tred to recover the lost hgh-frequency nformaton from low-level mage prmtves, whch were learnt from several general tranng mages. They broke the mages and scenes nto a Markov network, and learned the parameters of the network from the tranng data. To fnd the best scene explanaton gven new mage data, they appled belef propagaton n the Markov network. A very smlar mage hallucnaton approach was also ntroduced n (Sun et al., 2003). They used the prmal sketch as the pror to recover the smoothed hgh-frequency nformaton. Lu et al. (2001) combned the PCA model-based approach and Freeman s mage prmtve technque. They developed a mxture model combng a global parametrc model called global /$ - see front matter Ó 2006 Elsever B.V. All rghts reserved. do: /j.patrec

2 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) face mage carryng common facal propertes, and a local nonparametrc model called local feature mage recordng local ndvdualtes. The hgh-resoluton face mage was naturally a composton of both. More recently learnng-based technques have also been extended to vdeo. In (Bshop et al., 2003), a drect applcaton of (Freeman and Pasztor, 1999) to vdeo sequences was attempted, but severe vdeo artfacts were found. As a remedy, an ad hoc soluton was proposed. It conssted of re-usng hgh-resoluton solutons for achevng more coherent vdeos. In (Dedeoglu et al., 2004), the authors extended the work of (Baker and Kanade, 2000a) to super-resolve a sngle human face vdeo, usng dfferent vdeos of the face of the same person as tranng data. By explotng Bayesan framework and spatal-temporal constrants, they reported an extremely hgh face vdeo magnfcaton factor. However, all exstng technques have not addressed the problem of varable resolutons of partally occluded nputs often encountered n vdeo. In ths paper, we extend the work n (Ja and Gong, 2005) and propose a novel algorthm to super-resolve face mages gven multple partally occluded nputs at dfferent lower resolutons. By ntegratng herarchcal patch-wse algnment and nter-frame constrants nto a Bayesan framework, we can probablstcally algn multple nputs at dfferent resolutons and recursvely nfer the hgh-resoluton face mage. We address the problem of fusng partal magery nformaton through multple frames and dscuss the new algorthm s effectveness when encounterng occluded low-resoluton face mages. We show promsng results compared to those of exstng face hallucnaton methods from both smulated facal database and lve vdeo sequences. The paper s organzed as follows. In Secton 2, we defne the problem of hallucnatng multple partally occluded face mages at dfferent lower resolutons, and present a novel algorthm to probablstcally both algn and nfer hgh-resoluton face mages from a Bayesan framework perspectve. Secton 3 extends the new algorthm to cope wth occluded face mages n super-resoluton. Expermental results are presented n Secton 4 before conclusons are drawn n Secton 5. multple partally occluded nputs of dfferent resolutons. As shown n Fg. 1, low-resoluton face mages are automatcally detected as patches wthn rectangular regons n a vdeo. We are nterested n developng an algorthm that take nto account these multple nputs of dfferent resolutons n order to yeld an optmal hgher resoluton mage. Furthermore, we also wsh to perform super-resoluton when some or all of the low-resoluton nputs are occluded n the face detecton process. An example of a hallucnated complete face mage from multple occluded nputs s also shown n Fg. 2. The underlyng challenges we am to address are therefore three-fold. Frstly, how to algn multple nputs at dfferent lower resolutons. Secondly, how to cross-refer and recover mssng pxel nformaton n dfferent mage frames due to occluson. And fnally, a unfed algorthm to perform algnng, nferrng mssng nformaton and superresolvng a hgh-resoluton face mage gven multple sources A Bayesan formulaton In ths secton, we formulate our problem of hallucnatng multple nputs of dfferent resolutons by means of a Bayesan framework. Assumng H s the hgh-resoluton mage needs to be constructed, L 1,L 2,...,L S are the lowresoluton nputs wth dfferent resolutons. The task comes as fndng the Maxmum A Posteror (MAP) estmaton of H gven L 1,L 2,...,L S. Let us frst consder the problem of only two low-resoluton nputs L 1, L 2 (see Fg. 2), whch can be formulated as H MAP ¼ arg max log PðHjL 1 ; L 2 Þ ð1þ H Furthermore, we defne T as an unknown ntermedate template and I as the algnng parameter between low-resoluton nputs L 1 and L 2 (how to determne parameter I and generate template T wll be presented n Secton 2.3). We can margnalze P(HjL 1,L 2 ) over these unknown parameters as PðHjL 1 ; L 2 Þ¼ X X PðH; T ; I j jl 1 ; L 2 Þ j 2. Hallucnatng multple mages of dfferent resolutons 2.1. Problem defnton In a survellance vdeo, a sequence or some snapshots of a human face can be captured, where ther resolutons are often too small and vary sgnfcantly over tme. The mages can also be partally occluded. Such condtons make the mages less useful for automatc verfcaton or dentfcaton. Exstng technques have not consdered hallucnatng a hgh-resoluton face mage under these condtons. In ths paper, we defne the problem of face hallucnaton n vdeo as how to super-resolve a face mage wth Fg. 1. A realstc face detecton envronment.

3 1770 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) Fg. 2. An llustraton of our hallucnaton process gven multple occluded nput mages: L 1 and L 2 are occluded low-resoluton nputs, T and bt are ntermedate templates, H s the fnal hallucnaton result of a hgher resoluton: (a) s the herarchcal mage algnng process, and (b) s the process of patch learnng and nter-frame constrant for estmatng optmal ntermedate template bt. where and j are possble choces for T and I respectvely. By applyng the Bayes rule twce, the above becomes X X PðHjI j ; T ; L 1 ; L 2 ÞPðI j ; T jl 1 ; L 2 Þ j ¼ X X PðHjT ; I j ; L 1 ; L 2 ÞPðT ji j ; L 1 ; L 2 ÞPðI j jl 1 ; L 2 Þ j Assumng algnng parameter I wll peak at the true value I w, gves PðIjL 1 ; L 2 Þ¼dðI I H Þ By substtutng (3) nto (2) and usng the Bayesan rule, we have X PðHjI H ; T ; L 1 ; L 2 ÞPðT ji H ; L 1 ; L 2 Þ ¼ X PðL 1 ; L 2 jh; I H ; T ÞPðHjI H ; T Þ PðT PðL 1 ; L 2 ji H ji H ; L 1 ; L 2 Þ ; T Þ Assumng H exsts and based on the basc mage observaton model, the low-resoluton nputs can be ndependently sub-sampled from H, then we have P(L 1,L 2 jh, I w,t )= P(L 1 jh)p(l 2 jh). By settng the denomnator as a constant C, P(HjL 1,L 2 ) can be rewrtten as C X PðL 1 jhþpðl 2 jhþpðhji H ; T ÞPðT ji H ; L 1 ; L 2 Þ ð5þ Although there could be many optons for the ntermedate template T, the one whch s optmal maxmzes the probablty P(T ji w,l 1,L 2 ). We defne t as bt, and compute template bt by means of herarchcal low-level vson, smlar to that of (Baker and Kanade, 2000a; Dedeoglu et al., 2004) (more detals n Secton 2.3). Then wth (1) and (5), we maxmze the followng cost functon for H MAP : log PðL 1 jhþþlog PðL 2 jhþþlog PðHjI H ; bt Þ þ log PðbT ji H ; L 1 ; L 2 Þ ð2þ ð3þ ð4þ ð6þ Ths resultng cost functon s easly generalzed from 2 to S nputs as follows: log PðL 1 jhþþlog PðL 2 jhþþþlog PðL S jhþ þ log PðHjI H 1 ; I H 2 ;...; I H S 1 ; bt Þ þ log PðbT ji H 1 ; I H 2 ;...; I H S 1 ; L 1; L 2 ;...; L S Þ where I H 1 ; I H 2 ;...; I H S 1 are algnng parameters for S 1 low-resoluton nputs wth respect to the largest resoluton. The ndvdual components n (7) can be nterpreted as follows. The frst S terms requre the nferred hgh-resoluton result H to satsfy the basc mage observaton model wth respect to each of nput L. The penultmate term assures bt to serve as a pror n ths Bayesan framework. Fnally, the last term provdes an entrance to compute bt Fndng the ntermedate template The basc dea for fndng the ntermedate template comes from (Dedeoglu et al., 2004). As n (6), to fnd the best bt, we need to maxmze the probablty PðbT ji H ; L 1 ; L 2 Þ. By the Bayes rule we have PðbT ji H ; L 1 ; L 2 Þ/PðL 1 ; L 2 jbt ; I H ÞPðbT Þ Assumng L 1 s the low-resoluton nput that algnng s based on, I w defnes the herarchcal patch-wse correspondence between L 1 and L 2, we factorze the low-resoluton nputs nto ndependent patches. The above lkelhood can be derved as! X M q¼1 PðL 1 p ; L2 q j bt p ; I H ÞPðbT Þ where L 1 p, L2 q refer to the local patches n L 1 and L 2, N and M are ther patch numbers respectvely. Regardng each patch p for L 1, there s only one matchng q from 1 to M. Assumng I w s known, we have the fnal lkelhood functon as ð7þ

4 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) PðL 1 p ; L2 p j bt p ÞPðbT Þ¼ YN PðL 1 p j bt p ÞPðL 2 p j bt p ÞPðbT Þ where the L 2 p stands for the herarchcally correspondng patch n L 2 wth regard to L 1 p. In other words, a patch n L 2 corresponds to multple patches n L 1. Ths expresson can also be generalzed from two to S low-resoluton nputs, and the lkelhood expresson becomes ð8þ PðL 1 p j bt p ÞPðL 2 p j bt p ÞPðL S p j bt p ÞPðbT Þ The frst S N terms n (9) gve the basc dea for how to generate the ntermedate template from the herarchcal patch matchng perspectve. But ther constrants are stll too weak consderng each low-resoluton patch could be generated from many hgh-resoluton database patches. One remedy to ths problem s to pool contextual nformaton among patches. To ths end, we used parent vector (DeBonet and Vola, 1998) as a local feature structure to strengthen these constrants. The pror PðbT Þ fnally provdes a spatal dependency constrant to refne the generated template Algnng multple low-resoluton nputs For determnng the algnng parameter I w, let us frst consder the two nputs L 1 and L 2 case agan. Assumng L 1 s the low-resoluton nput that algnng s based on, we sub-sample L 1 to the resoluton of L 2, and t becomes L 1. To compute the algnng parameter I w, we need to maxmze the lkelhood functon PðIjL 1 ; L 2 Þ, whch s consstent wth the lkelhood functon P(IjL 1,L 2 ) n (2) and (3). Assume patches n L 1 are mutually ndependent, by applyng Bayes rule we attan PðIjL 1 ; L 2 Þ¼PðL 1 ; L 2 jiþpðiþ ¼ Y PðL 1 ; L2 jiþpðiþ ð10þ where L 1 and L 2 have smlar meanngs as L 1 p and L2 p n (8). Gven any algnng parameter estmaton, we defne the above probablty densty functon as PðL 1 ; L2 jiþ /exp kf F L 2 k 2 L 1 where F L 1 and F L 2 are local patch feature vectors to be defned n Secton The value I w that maxmzes the cost functon (10) gves the optmal algnng parameter. Smlarly we can generalze the two nput case to that of S nputs Template pror and local feature structure The Markov Random Feld (MRF) model assgns a probablty to each template patch confguraton T, and accordng to the Hammersley Clfford theorem, P(T) s a product Q T m;t n /ðt m ; T n Þ of comparablty functon /(T m,t n ) over all pars of neghborng patches. The detals as how to compute P(T) can be found n (Dedeoglu et al., 2004). ð9þ Suppose L s p s an mage patch n low-resoluton nput L s, and T p s a random patch from the tranng database whch has already been sub-sampled to the resoluton of L s p. For each of these patches, we adopt the parent vector (DeBonet and Vola, 1998) as ther feature vectors, whch stacks together local ntensty, gradent and Laplacan mage values at multple scales. To each of the term PðL s p jt pþ, we defne the probablty densty functon as PðL s p jt pþ/exp kf L s p F T p k 2 where F L s p and F T p are the feature vectors for L s p and T p. Smlarly to (7), the pdf of the frst S N terms n (9) s generalzed as PðL 1 p j bt p ÞPðL 2 p j bt p ÞPðL S p j bt p Þ / YN exp XS s¼1 kf L s p F T p k 2! The fnal ntermedate template bt s estmated as arg max T PðL 1 p ; L2 p ;...; LS p jt pþ Y m;n 2.4. Inferrng the hgh-resoluton mage /ðt m ; T n Þ ð11þ ð12þ After obtanng the ntermedate template bt, we can use the frst S + 1 terms of (7) as objectve functon to nfer the fnal result H. Suppose the acquston of L 1,L 2,...,L S should observe the mage observaton model by blurrng and sub-samplng the hgh-resoluton H, we approxmate the process as L s ¼ A s H þ g Ls where s =1,...,S, A s s a sub-samplng model, and g Ls s Gaussan nose. Assumng any L s s pxel-wse ndependent, then we have PðL s jhþ ¼ Y! 1 pffffffffff exp ðl sðuþ ða s HÞðuÞÞ 2 ð13þ u r Ls 2p 2r 2 L s The fnal nference of H should be coherent wth the ntermedate template bt wth a probablty of PðHjI H ; bt Þ. We express the relatonshp as H ¼ bt þ g H Assumng nose g H s pxel-wse ndependent and Gaussan, we have PðHjbT ; I H Þ¼ Y! 1 pffffffffff exp ðhðvþ bt ðvþþ 2 ð14þ v r H 2p 2r 2 H Substtute (13) and (14) nto the above objectve functon, we can fnally nfer hgh-resoluton H by mnmzng the followng quadratc expresson:

5 1772 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) r 2 H r 2 L 1 kl 1 A 1 Hk 2 þ r2 H r 2 L 2 kl 2 A 2 Hk 2 þ þ r2 H r 2 L S kl S A S Hk 2 þkbt Hk 2 3. Hallucnatng multple occluded face mages ð15þ Another sgnfcant advantage of the Bayesan framework presented above s ts ablty n recoverng mssng data from occluded low-resoluton face mages. Ths capablty s mportant because occluson s very common when capturng faces n vdeo, as llustrated n Fg. 1. Gven occluded low-resoluton nputs L 1,L 2,...,L S, the task here s to super-resolve the hgh-resoluton H, even n the extreme case that none of the nputs captures a complete face. Wthn our Bayesan framework, we need frst to estmate the algnng parameter I w, and then compute the ntermedate template bt. Gven bt, we can nfer the fnal hgh-resoluton H by mnmzng (15). Assumng L 1 and L 2 are two partally occluded mages, even though not all patches from both mages are present (.e. partally mssng) for algnment, a I w can stll be estmated by maxmzng (10). Gven I w, we can smplfy (12) as Y arg max PðL 1 p T jt p Þ Y PðL 2 p j jt p Þ p p j Y PðL 1 p k jt p ÞPðL 2 p k jt p Þ Y /ðt m ; T n Þ ð16þ p k m;n from whch bt can be generated, where p stands for the patches n L 1 wthout correspondng patches n L 2, p j stands for the patches n L 2 wthout correspondng patches n L 1, and p k are those patches that are common n both L 2 and L 2. The remanng process follows detals n the above secton. Fg. 2 llustrates the entre process for hallucnatng occluded face mages. 4. Expermental results 4.1. Setup Our face mages come from a subset of AR, FERET and Yale databases. The database conssts of 845 mages of 169 dfferent ndvduals (60 women and 109 men), n whch each person has fve dfferent face mages. Orgnally face mages from these databases have dfferent szes, and also the area of the mage occuped by the face vares consderably. To buld up a standard tranng patch database, we need to algn these face mages manually. Ths algnment was performed by hand markng the locaton of three ponts: the centers of the eyeballs and the lower tp of the nose. These three ponts defne an affne warp, whch was used to warp the mages nto a canoncal form. The canoncal mage has pxels wth the rght eye at (25, 31), the left eye at (25,16), and the lower tp of the nose at (34,24). In our current experments, nstead of testng our algorthm on automatcally detected face mages n lve vdeo, we generated the test mages as follows. We frst blurred any gven hgh-resoluton mage from ths database wth dfferent flters to ntroduce dfferent Pont Spread Functons (PSF) accordngly, and then sub-sampled the blurred mages to low-resolutons. We then added random translatonal moton to ntroduce a measurable degree of random msalgnment resultng from most automatc face detecton processes on lve vdeo feed. For selectng hgh-resoluton face mages to generate testng data, we used leave-oneout methodology: For any seres of generated testng low-resoluton face mages, we removed ther correspondng hgh-resoluton source from the database, and the remanng hgh-resoluton mages serve as the learnng database. The removed hgh-resoluton mages later served as the ground truth mages n the experments on quantfyng model error as shown n Fg Comparson of sngle face mages wthout mssng parts One advantage of our framework s ts ablty to deal wth face hallucnaton wth multple nputs at dfferent resolutons. To evaluate ts effectveness, for any gven mage from the database, we generated three low-resoluton mages at the szes of 14 11, 9 7 and 7 5 usng the above method. Gven these three testng face mages, we took the largest one as the low-resoluton nput that algnment s to be based upon, and estmated the algnng parameters for the 9 7 and 7 5 mages. Then we generated the ntermedate template based on (12). The hgh-resoluton result was constructed by solvng the quadratc cost functon (15). Column (b) of Fg. 3 shows some example hgh-resoluton results. To compare these results wth hallucnaton usng a sngle face mage smlar to those n (Baker and Kanade, 2000a; Dedeoglu et al., 2004), we performed experments by takng only smulated mages as low-resoluton nput; example results are shown n column (c) of Fg. 3. Comparng (b) and (c) n Fg. 3 suggests that we mproved the hallucnaton results. However, the mprovement s not dramatc because the largest low-resoluton nputs already contan most of the nformaton that could also be contrbuted from the other low-resoluton nputs. In other words, the nformaton from the other low-resoluton nputs s mostly redundant Comparson of occluded face mages Sgnfcantly, the advantage of our multple nput based approach over exstng hallucnaton methods becomes dramatc when low-resoluton nput mages are partally occluded wth mssng parts. Such nput mages are common when detectng and trackng face mages of movng targets n lve vdeo. In other words, f many of the low-resoluton nputs at dfferent resolutons mss pxels due to

6 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) Fg. 3. Comparng face hallucnaton usng sngle and multple nputs wthout occluson or mssng parts: (a) multple low-resoluton nputs wth frame resoluton of 14 11, 9 7 and 7 5, (b) results from our approach, (c) results usng sngle mage face hallucnaton and (d) ground truth mages wth resoluton of occluson (or pool lghtng and vewpont), t becomes essental to algn them before super-resolvng a hgh-resoluton mage takes place. Dfferent from early expermental settngs, gven any mage n ths experment, we frst randomly removed part of t to smulate the face mage beng partally occluded, and then generated the frst occluded test mage wth the frame resoluton of By the same token we could yeld another test mage wth frame resoluton of 7 5. Some examples are shown n column (a) of Fg. 4. Based on the deduced objectve functon (16) n Secton 3, combned wth Eqs. (10) and (15) n Secton 2, we can probablstcally nfer a hgh-resoluton reconstructon makng use of all the nformaton from the two occluded test nputs. Wth exstng learnng-based super-resoluton technques, nether of two partally occluded low-resoluton nput mages can provde suffcent nformaton for recoverng a complete face mage at a hgher resoluton. Fg. 4 shows example results usng sngle-mage face hallucnaton technque smlar n (Baker and Kanade, 2000a; Dedeoglu et al., 2004) gven partally occluded low-resoluton nput mages of (b) and 7 5 (c) respectvely. As expected, only part of a face was recovered at the hgher resoluton of Furthermore, we show results n column (d) based on fusng the partally hallucnated face mages from columns (b) and (c) of Fg. 4. It shows clearly that moton and llumnaton varatons between dfferent occluded nput mages at dfferent lower resolutons make smple fusng a poor soluton. On the other hand, our results shown n (e) mprove sgnfcantly those of ether (b) or (c) at the resoluton of It s also worth pontng out that gven that our nputs were partally occluded wth sgnfcant mssng parts at the resolutons of 7 5 and 14 11, our magnfcaton factor s effectvely over 8 8 whch goes beyond the exstng 4 4 lmt (to obtan a desred hgh-resoluton result) for the current hallucnaton technques. To quantfy the performance of dfferent technques, we measured the average root Sum of Squared Error (SSE) per pxel w.r.t. the orgnal hgh-resoluton mage ground truth, as shown n Fg. 5. Consstent to Fg. 4, the average root SSE/pxel from our results (represented by the sold lne) are the smallest compared to both those usng the occluded nputs (represented by the dotted lne) and those usng the occluded 7 5 nputs (represented by the dashed lne). Fg. 5 also suggests that the results based on fusng the partally hallucnated parts by pxel averagng (represented by the dash-dot lne) are much worse than our results. To explan ths, we should notce that, although the partal face mages n columns (b) and (c) of Fg. 4 (correspondng to the dotted and dashed lnes n Fg. 5) are

7 1774 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) Fg. 4. Hallucnaton wth occluded faces: (a) occluded face mages wth resoluton of and 7 5, (b) results usng the occluded nput only, (c) results usng the occluded 7 5 nput only, (d) results based on fusng the partal face mages n column (b) and (c) by pxel averagng at overlapped parts, (e) our hallucnaton results. Ground truth mages are the same as n column (d) of Fg. 3. Average root SSE/pxel w.r.t. ground truth Our results Fusng results Results usng occluded 14x11 Results usng occluded 7x5 already ndependently algned nto general face frames wth reference to the tranng database, they are essentally pxel-wse uncorrelated. The occluded low-resoluton nputs were respectvely super-resolved nto partal hghresoluton face mages wthout consderng the moton and llumnaton varatons between them. Indeed t s these varatons at low-resoluton that make algnng and fusng at hgh-resoluton fal. Only by utlzng a herarchcal and recursve formulaton of an ntermedate template as proposed n our approach, we are able to algn and super-resolve across occluded nputs of dfferent resolutons Index of testng mages Fg. 5. Average root Sum of Squared Error (SSE) per pxel w.r.t. ground truth of hallucnaton results. The sold lne represents error from our hallucnaton results, the dotted lne represents error from partally hallucnated parts usng occluded nputs, the dashed lne represents error from partally hallucnated parts usng occluded 7 5 nputs, and the dash-dot lne shows error from fusng the partally hallucnated results usng occluded and 7 5 nput mages respectvely by pxel averagng at overlapped parts Experments on lve vdeo data For testng the robustness of our approach when appled on real data, we captured vdeo sequences from a corrdor survellance camera. Snce t s unrealstc to assume that partally occluded face mages could be accurately detected, especally n case of low-resoluton, n ths experment we selected frames where complete faces at dfferent lower resolutons exsted, and manually cropped out these face mages. After that we randomly removed partal faces

8 K. Ja, S. Gong / Pattern Recognton Letters 27 (2006) Fg. 6. Example of hallucnatng multple occluded face mages usng lve vdeo data: (a) low-resoluton facal mages n lve vdeo were located and segmented frst and (b) partal faces were randomly removed to smulate the occluson condtons. and used the left parts as testng nputs. The hgh-resoluton tranng mages are from a subset of AR face database. Fg. 6 demonstrates the llustraton example. In the lve sequence experments, we have no ground truth mages whch could be collected, to verfy the lkeness of hallucnated results. But the resultng mages do demonstrate the robustness of our approach. As suggested n Fg. 6, the qualty of hallucnated face s as good as, f not better than, those n smulated database experments. 5. Concluson In summary, by ntroducng an ntermedate template recursvely estmated nto a Bayesan framework, we present a novel model to super-resolve face mages wth multple occluded nputs at dfferent lower resolutons. The model n essence performs herarchcal patch-wse algnment and global Bayesan nference. Beyond the classc face hallucnaton algorthms, we both consder the spatal constrants and explot the nter-frame constrants across multple face mages of dfferent resolutons. As a consequence, the new algorthm s more effectve for dealng wth occluded low-resoluton face mages. We showed sgnfcantly mproved results over exstng face hallucnaton methods. In ths work, we manually conducted experments on lve vdeo sequences, n whch the pose and llumnaton varatons were solved by manual algnment and normalzaton. In the future we wll extend our work on hallucnatng automatcally detected low-resoluton face vdeos. References Baker, S., Kanade, T., 2000a. Lmts on super-resoluton and how to break them. In: Proc. IEEE Internat. Conf. on Computer Vson and Pattern Recognton. South Carolna, vol. 2, pp Baker, S., Kanade, T., 2000b. Hallucnatng faces. In: Proc. IEEE Automatc Face and Gesture Recognton. Grenoble, France, pp Bshop, C.M., Blake, A., Marth, B., Super-resoluton enhancement of vdeo. In: Proc Artfcal Intellgence and Statstcs. Socety for Artfcal Intellgence and Statstcs. Key West, Florda, USA. Capel, D.P., Zsserman, A., Super-resoluton from multple vews usng learnt mage models. In: Proc. IEEE Intl. Conf. on Computer Vson and Pattern Recognton. Kaua, HI, USA, vol. 2, pp DeBonet, J.S., Vola, P.A., A non-parametrc mult-scale statstcal model for natural mages. In: Advances n Neural Informaton Processng Systems (NIPS). Denver, USA, vol. 10, pp Dedeoglu, G., Kanade, T., August, J., Hgh-zoom vdeo hallucnaton by explotng spato-temporal regulartes. In: Proc. IEEE Intl. Conf. on Computer Vson and Pattern Recognton. Washngton, DC, USA, vol. 2, pp Elad, M., Feuer, A., Restoraton of a sngle superresoluton mage from several blurred, nosy, and undersampled measured mages. IEEE Trans. on Image Processng 6 (12), Freeman, W., Pasztor, E., Learnng low-level vson. In: 7th Internat. Conf. on Computer Vson. Kerkyra, Greece, pp Harde, R.C., Barnard, K.J., Armstrong, E.E., Jont MAP regstraton and hgh-resoluton mage estmaton usng a sequence of undersampled mages. IEEE Trans. Image Process. 6, Iran, M., Peleg, S., Improvng resoluton by mage regstraton. CVGIP: Graphcal Models Image Proc. 53, Ja, K., Gong, S., CCTV face hallucnaton under occluson wth moton blur. In: Proc. IEE Internat. Symposum on Imagng for Crme Detecton and Preventon. London, UK, pp Lu, C., Shum, H., Zhang, C., A two-step approach to hallucnatng faces: Global parametrc model and local nonparametrc model. In: Proc. IEEE Intl. Conf. on Computer Vson and Pattern Recognton. Kaua, HI, USA, pp Schulz, R.R., Stevenson, R.L., Extracton of hgh-resoluton frames from vdeo sequences. IEEE Trans. Image Process. 5, Sun, J., Zhang, N., Tao, H., Shum, H., Image hallucnaton wth prmal sketch prors. In: Proc. IEEE Intl. Conf. on Computer Vson and Pattern Recognton. Madson, USA, vol. 2, pp