RECOGNITION AND AGE PREDICTION WITH DIGITAL IMAGES OF MISSING CHILDREN

Transcription

1 RECOGNIION AND AGE PREDICION WIH DIGIAL IMAGES OF MISSING CHILDREN A Wrtng Project Presented to he Faculty of the Department of Computer Scence San Jose State Unversty In Partal Fulfllment of the Requrements for the Degree Master of Scence by Wallun Chan December 005

2 005 Wallun Chan ALL RIGHS RESERVED

3 APPROVED FOR HE DEPARMEN OF COMPUER SCIENCE Dr. Chrs Pollett Dr. Mark Stamp Dr. Robert Chun 3

4 Abstract Prncpal Components Analyss s a dmensonalty reducton technque that determnes egenvectors (prncpal components) and correspondng egenvalues from the covarance matrx of a data set. he egenvectors that do not contrbute much to scatter are truncated wthout excessve data loss. he remanng egenvectors represent a new coordnate system n lower dmensonal space, allowng a more compact and effcent representaton of the orgnal data. In ths project, we age-progress dgtal face mages of chldren by applyng PCA to tranng mages that maps young to aged faces. Once we compute the prncpal components from the tranng mages, we project an nput mage onto the prncpal components to obtan a weght vector. he coeffcents of the weght vector represent proportons of each correspondng prncpal component needed to approxmately reconstruct the orgnal nput mage by a weghted summaton. We want the weght vector to be n close proxmty to a cluster of projected tranng mages. As a result, the reconstructed mage should capture the aged features by resemblng a weghted average result. In ths report, we explan the mathematcal dervaton of PCA, and the egenface approach that apples PCA to mage data. We then cover the desgn and development of our age progresson program that apples PCA to grayscale mages of whole faces. Next, we dscuss the extenson of the program to support color mages, and a feature-based approach to age progresson. Fnally, we show results of tests performed to evaluate the performance of the age progresson program usng varous tranng mage sets of young and aged face mages. 4

5 able of Contents Page. Introducton 8. Background and heory.. Prncpal Components Analyss (PCA).. PCA heory 3.3. Solvng for Egenvalues and Egenvectors 6.4. PCA Algorthm and Egenfaces 8 3. Intal Desgn and Implementaton 3.. Desgn and Evaluaton 3.. Egenface Program 3... Image Recognton 3... Program Functonalty Program Desgn Reconstructon and Recognton Results Grayscale Age Progresson Image Retreval and Preprocessng Age Progresson est Results Feature Matchng Program Shape Context and Bpartte Matchng Program Desgn Feature Matchng est Results Dscusson of Intal Results Fnal Desgn and Implementaton Feature-Based Age Progresson Feature-Base Overvew Feature Detecton and Extracton Feature Blendng and Reconstructon Program Functonalty Program Desgn 58 5

6 4.5. Feature-Based est Results Colorzaton Applcaton and Results Concluson 67 References 70 Web References 7 6

7 Lst of Fgures and ables Fgure : Prncpal components of -D data ponts 5 Fgure : Egenface program nterface 5 Fgure 3: Egenface program class desgn 7 Fgure 4: ranng mages of two face classes 8 Fgure 5: Example egenfaces (left four) and mean mage 8 Fgure 6: Input (top) and reconstructed (bottom) mages 9 Fgure 7: Egenvector truncaton results 30 Fgure 8: Image retreval, preprocessng, and age progresson 3 Fgure 9: Young and correspondng aged mage par 34 Fgure 0: Concatenated tranng mage 34 Fgure : Concatenated nput mage 34 Fgure : Reconstructed nput mage 34 Fgure 3: Concatenated tranng mage 34 Fgure 4: Grayscale reconstructons of aged female and male 35 Fgure 5: Edge detected mage 36 Fgure 6: Log-polar bns and flattened hstogram 37 Fgure 7: Feature Matchng Program Interface 40 Fgure 8: Feature matchng program class desgn 4 Fgure 9: Shape matchng results 44 Fgure 0: Successful feature matchng 45 Fgure : Unsuccessful feature matchng 45 Fgure : Feature-based age progresson process 50 Fgure 3: Parameterzed face dmensons 5 Fgure 4: Feature detected faces 5 Fgure 5: RGB mage matrx format 5 Fgure 6: Reconstructon and blendng process 54 Fgure 7: Aged and contoured features and blended mage 55 Fgure 8: Feature-based age progresson program nterface 56 Fgure 9: Feature-based program class desgn 59 Fgure 30: ranng mages for clusterng test 6 Fgure 3: Reconstructed results baby, toddler, and adolescent 63 Fgure 3: Reconstructed results wth extended tranng mage set 64 Fgure 33: Example colorzaton results 65 Fgure 34: Colorzed face wth brght regons 65 Page able : Egenface classfcaton rules 3 able : Recognton results for 00 test mages 30 7

8 Chapter Introducton In ths project, we use the egenface technque as descrbed n the classc paper by urk and Pentland [] to produce age-progressed face mages of mssng chldren. he egenface approach s based on usng PCA on mage data. Prncpal Components Analyss (PCA or Karhunen-Loéve transformaton) s a technque that has been used extensvely for pattern recognton applcatons such as face detecton and recognton. PCA determnes a lnear transformaton of the coordnate system of mult-varate data such that the axes of the new coordnate system are n the drectons of maxmum pont scatter. he data wth respect to each new axs are then completely de-correlated. PCA has the advantage of allowng the reducton of hgh-dmensonal data onto lower dmensonal space by combnng features n a least squares fashon. In other words, data ponts may be represented wth fewer varables n ths new coordnate system. Mathematcally, we wsh to determne the egenvectors of the covarance matrx of a sample data set that represents a new coordnate system n a lower dmensonal space. he covarance matrx conssts of covarances between all possble pars of scalar elements n a vector, where covarance measures how closely two varables change n relaton to each other. herefore, the covarance matrx ndcates the amount of correlaton of data between axes. PCA mnmzes 8

9 the amount of correlaton between axes, and hence the amount of redundancy. If we assume a matrx that s real and symmetrc, a set of orthonormal egenvectors can always be found. he egenvectors are then sorted by ther respectve egenvalues. he egenvectors wth the largest egenvectors are retaned, such that the lesser egenvectors that do not result n much scatter, may be truncated wthout sacrfcng excessve accuracy of data reconstructon. hus hgh-dmensonalty data s reduced sgnfcantly n dmensons, allowng effcent algorthms that use only the requred egenvectors for the largest few correspondng egenvalues. PCA has the property such that projectng data onto the egenvectors results n a set of weghts that represent lnear proportons of the prncpal components. he weghts can also be used to reconstruct a data pont. PCA represents a sample data pont wth a proper lnear combnaton of the egenvectors (prncpal components) that mnmzes the error between the reconstructed result and all sample data ponts. he goal of ths project s to collect an adequately large tranng mage set, compute the prncpal components, and reconstruct an nput mage by determnng the proper lnear combnaton of the prncpal components. In dong so, we attempt to capture the essence of agng for each reconstructed mage by proportonally combnng the varatons across the set of tranng mages represented by each prncpal component. hs project conssts of several smaller projects that focus on research, demonstraton, and evaluaton of several key concepts and technques used for 9

10 mage reconstructon and age progresson. Frst, we explore the feasblty of usng PCA by mplementng a program for mage reconstructon and recognton of grayscale mages. Second, we mplement an mage-retreval and preprocessng tool to obtan and prepare a suffcent set of tranng and nput mages, and use the egenface program to reconstruct aged grayscale mages of mssng chldren. hrd, we begn explorng feature-based age progresson by lookng at shape contexts to solve the key problem of locatng features on a face mage (e.g. eyes, nose, and mouth). Fnally, we take use these concepts and technques to mplement a program that takes color face mages as nputs and locates major features (e.g. eyes, nose, mouth, and face) to be PCA traned. Pror to ths, the mages are preprocessed n terms of sze, removal of n-plane rotaton, and croppng to elmnate background and maxmze face area. For each selected feature, each (RGB) color channel s extracted to produce three separate grayscale mages that are concatenated to form an overall grayscale mage. hs s done for young and aged sets of mages, results of whch are also concatenated to form fnal tranng mages for each feature. hese tranng mages provde a mappng between young and correspondng aged features. An nput face mage s preprocessed n a smlar manner as mentoned above except that the selected features are concatenated wth themselves. Each selected feature from the nput mage s projected onto a correspondng set of prncpal components to produce a set of weghts, whch are then used to reconstruct a predcted aged feature. he aged features are then blended back onto the aged faces at ther respectve locatons. PCA s an unsupervsed learnng technque 0

11 that reles entrely on the tranng data, and therefore does not take advantage of specfc target data. Selectng, removng, and tranng on specfc features allows more control over how PCA performs mage reconstructon. In addton, we rely on the preprocessng and normalzaton of tranng and nput data to mnmze any varants that may negatvely affect the output results.

12 Chapter Background and heory. Prncpal Component Analyss (PCA) In ths secton, we gve a bref descrpton and analyss of PCA. We provde more mathematcal detal for PCA n the comng sectons as t apples to mages. Gven N ponts n d-dmensonal space, the egenvectors of the covarance matrx for the data set form the new axes of the d-dmensonal data ponts. Mult-varate data can be expressed more effcently n terms of the top d orthogonal egenvectors. hs s done by projectng the data ponts onto a vector passng through the mean of the data ponts n the drecton of the egenvectors wth the largest egenvalues. he egenvectors are sorted such that the ones wth the largest egenvalues domnate n scatter n ther respectve drectons. Gven a d x d covarance matrx, there are d egenvectors that may be computed. PCA reduces ths dmensonalty by usng a mappng process (Hotellng transform) that attempts to project vectors n d-dmensonal space to vectors n d dmensonal space n a least squares fashon for d < d. Specfcally, nstead of a d x d transformaton matrx, we form a d x d transformaton matrx from the top d egenvectors that performs the mappng process. However, snce dmensonalty reducton s a lossy process, reconstructon of each sample data pont s no longer exact. o mnmze the error between each reconstructed data pont n d dmensonal space and orgnal sample data pont, we solve for the top

13 d egenvectors from the covarance matrx of the sample data set. In applcatons wth hgh-dmensonal data (e.g. dgtal mage data), the frst d egenvectors have sgnfcantly greater egenvalues than the other d - d egenvectors. PCA determnes and uses the egenvectors to represent patterns across all sample data ponts. Each egenvector represents a prncpal component such that any sample data pont may be reconstructed wth a weghted lnear combnaton of the prncpal components. Each prncpal component contrbutes more or less to each orgnal data pont. o reconstruct each tranng data pont from the egenvectors, the proper proportons or weghts must be determned. hs s done by projectng each sample data pont onto the subset of prncpal components to form a weght vector.. PCA heory he followng mathematcal dervaton of PCA s taken from Pattern Classfcaton [3]. Gven N d-dmensonal sample data vectors x, K, x, we want n to fnd a d-dmensonal vector x 0 that best represents x, K, x n a least squares n sense. Specfcally, m = N N x = mnmzes the error functon E N ( x ) = 0 x0 x. However, the mean m does not show the varablty of the = data set. o do so, we project the data set onto a d -dmensonal representaton, where d d. Startng wth a smpler and more ntutve example, we have e as a 3

14 4 unt vector passng through m, and project the data onto a lne gven by ce m x + =. Gven n c c c,, K =, we redefne the error functon to be ( ) ( ) ( ) ( ) ( ) ( ),, = + = + = + = = = = = = = = = e where m x m x c e c m x m x c e e c m x c e x c e m e c E x E N N N N N N N N () Solvng for the partal dervatve of the error functon, we have 0 = + = m e x e c c E and obtan ( ) m x e c = () Substtutng equaton () nto (), and notng that the covarance matrx s gven by ( )( ) = N m x m x N, we obtan ( ) ( ) [ ] ( )( ) = = = = = = = = + = + = + = + = N N N N N N N N m x Se e m x e m x m x e m x m x e m x c c e c E, where the scatter matrx S s the covarance matrx tmes N -. By maxmzng Se e the error functon ( ) e c E, s mnmzed. We use the Lagrange multpler method gven by ( ) ( ) ( ) x g x f x L + = λ λ, subject to ( ) 0 = x g and dervatve 0 = + = x g x f x L λ. We have ( ) ( ) ( ) e g e f e L λ λ + =, where

15 f () e = e Se, and snce e = e e =, the constrant functon becomes ( e) = e e = 0. We compute the partal dervatve of L ( e,λ ) yeldng g ( e, λ ) f () e ( g() e ) L e = e + λ e = Se λe = 0 Se = λe (3) Equatons () and (3) ndcate that the egenvector e wth the largest egenvalue λ of the scatter matrx ponts n the drecton such that the projecton of the data onto = s maxmzed. Fgure and E( x) = E( c, e) = c e ( x m) x m + ce N = from equaton () ndcate that the subtracton of m centers the entre pont dstrbuton about m, and c extends e such that c e approaches x m. We Fgure : Prncpal components of -D data ponts also note that e n Fgure s clearly the domnant prncpal component. herefore, the projecton of data onto e s well approxmated wthout excessve loss of accuracy. We can extend x = m + ce and equaton () n one dmenson to obtan a least squares projecton n d dmensons wth the followng error functon n whch c j are coeffcents that represent lnear proportons of e j 5

16 D N D' () e = m + E ' c e x (4) = j = j j (prncpal components) to approxmate the orgnal sample data pont. In general, PCA provdes a mappng between N data vectors ( x,, ) N K of d dmensons to ( x,, x ) N ' K of d dmensons where d d. he top d egenvectors of the ' D' covarance matrx represent the orthogonal axes of the new coordnate system. x D.3 Solvng for Egenvalues and Egenvectors We frst assume a symmetrc real matrx S such that S = S where ( s j = s j ). We use the Jacob method wth the followng dervaton from Numercal Recpes n C [8] that apples a sequence of planar rotatonal transformaton pq to S untl pq S' = S s dagonal. Gven the planar rotatonal matrx pq pq = p q L p a M b L L q b M a L, subscrpts p and q refer to the rows and columns of scalars a = cos φ and b = sn φ. All dagonal elements are unty except for scalars a, and all off-dagonals are zero except for scalars b. Multplyng out the dagonalzaton of S' = pq S pq, the followng equatons are obtaned: 6

17 s' s' s' s' s' rp rq pp qq pq = a s = a s = a = b = s pq rp rq s s b s + b s pp pp rq + b + a rp s s qq qq ab s + ab s ( r p, r q) ( 5a) ( r p, r q) ( 5b) pq ( 5c) ( 5d ) ( a b ) + ab( s s ) ( 5e) pp pq qq We see that only rows p and q, and columns p and q are changed for dea s to zero-out the non-dagonal elements of S ' by applyng the planar S '. he rotaton matrx pq wth θ teratvely. Snce the subscrpt pq s off-dagonal, we a b sqq s pp set equaton (5e) to zero and defne θ = and ab s have pq b t. We then a θ a a t t t = a + θ a a t t a + tθ = 0 = 0 (6) Equaton (6) may be solved wth the quadratc formula for t. Wth equaton (6) and settng s ' pq = 0, we substtute (5e) nto (5c), separate s pp from s qq, and obtan ( a) s' pp = s pp t s pq 7 Further substtutons wth (5a 5e) yeld s' s' s' qq rp rq = s = s = s qq rp rq + t s b b pq b φ where τ = = tan. + a ( 7b) ( srq + τ srp ) ( 7c) ( s + τ s ) ( 7d ) rp rq herefore, for each off-dagonal element of S ', the Jacob method nvolves usng equatons (7a 7d) teratvely to compute S S 7

18 S N 3 3 K K S = S' = Z S Z. Once N S ' becomes dagonal, the egenvalues are located n the dagonals of S ', and the egenvectors are located n the columns of Z. Numercal Recpes n C [8] gves 3 an algorthm that executes n ( n ) O..4 PCA Algorthm and Egenfaces We begn wth a dgtal mage as an N N array of 8-bt grayscale pxel values from 0 to 55. o prepare a set of M mages for PCA analyss, we frst stretch out each N N mage array so that t s N, and concatenate M mages to form mage matrx Γ = { Γ, Γ,, } K Γ M of sze N M. We then calculate mean mage Ψ = M M = Γ and obtan mean adjusted mage matrx { Φ, Φ, } A = K, Φ M wth Φ = Γ Ψ. he correspondng egenvalues λ KM and egenvectors u K are determned from the covarance matrx M M C = Φ Φ = A A. Recall from Secton.. that we wsh to fnd the top d M = (M n ths case) egenvectors of the covarance matrx that best represents the data set wth respect to the new coordnate system by solvng equaton (3). It M = M follows that Se λ e e Se = λe e λ = u ( Γ Ψ)( Γ Ψ) u =, such that the th largest egenvalue λ and correspondng egenvector u are subject to 8

19 M maxmzng λ = ( Φ ) u M =, and u j u = j = δ j = so that egenvectors 0 otherwse u K must be orthonormal. M We note that snce A A s N N, computng for all N egenvalues and correspondng egenvectors s nfeasble for very large mages. An optmzed method gven by urk and Pentland [] n computng the prncpal components frst solves for the egenvalues and egenvectors of number of egenvectors s reduced from A A. By dong so, the total N to M. Mathematcally, we solve for the egenvectors v of A A such that A Av = λ v, and multply both sdes by A yeldng A A Av = λ Av. It s apparent that A v are egenvectors of A A. o obtan the prncpal components u, we form the proper lnear combnatons of the mean adjusted mages Φ by M u = vjφ j = Av, =, K, M j = (8) Snce M << N for typcal mage szes, only a relatvely small number M < M of egenvectors are needed. herefore computaton tme to solve for the egenvalues and correspondng egenvectors s sgnfcantly reduced usng ths method. Usng proper termnology, we refer to the set of egenvectors computed from the covarance matrx of a set of mages as egenfaces, as they resemble ghostly mages of faces. Fnally, to project an nput mage onto the set of egenvectors to obtan ts correspondng weght vector we compute Ω ω K u ( Ψ) ( Γ Ψ) = { ω }=,, M ' Γ nput ω (9) = u nput 9

20 for =, K, M '. o reconstruct an mage from Ω, we have ( u ) Ω = ( u ) u ( Γ Ψ) nput Γnput = u Ω + Ψ (0) where ( u ) = ( u ) = u and u u = I for orthonormal vectors M '. We note that u K these equatons are rreversble snce the prncpal components u represent a lossy reducton n dmensonalty such that some amount nformaton s lost, but mnmzed wth respect to equaton (4). 0

21 Chapter 3 Intal Desgn and Implementaton 3. Desgn and Evaluaton In ths chapter and sectons that follow, we descrbe, demonstrate, and evaluate key concepts and technques that are used to support our man focus of face reconstructon and age progresson. 3. Egenface Program he purpose of the egenface program s to demonstrate the feasblty of the egenface approach for face recognton and reconstructon. he egenface method apples PCA to reduce a set of tranng mages by representng them n terms of a more compact set of orthogonal vectors (prncpal components). An nput mage s then projected onto the set of prncpal components to obtan a weght vector that represents the proportons of each egenface to reconstruct the nput mage. 3.. Image Recognton o extend the applcaton of egenfaces for face recognton, we use the face class and face space Eucldean dstance metrcs defned by class k k face Φ face ε = Ω Ω and ε = Φ respectvely. he Eucldean dstance between two vectors x = { x,, } and y { y,, } K x N = s determned by K y N

22 d ( x y) = ( ) N, x y. A face class s defned as a group of mages (varyng n = orentaton, lghtng, and expresson, etc.) of the same person (or object). he weght vector Ω k s the average of the weght vectors of mages n face class k. For the face space metrc, we project an nput mage onto face space consstng of M egenfaces, obtan ts characterstc weght vector wth equaton (9), and compute Φ face M ' = u Ω = ω u. = o perform face recognton, we frst gather a set of M tranng mages { Γ, Γ, } Γ =, K Γ M for X face classes, and assgn one or more mages to each person such that each mage of the person vares n facal expresson, orentaton, and lghtng, etc. L = A A s computed where A = { Φ, Φ, K, Φ M }, M Φ = Γ Ψ, and Ψ = Γ M =. he egenvalues λ KM and correspondng egenvectors v K M are computed from L, M v K sorted n the order of descendng egenvalues, and the top M egenvalues and correspondng egenvectors are selected such that M M. Next, the egenfaces (prncpal components) are computed from equaton (8). hen for each tranng mage Γ tranng, we compute the weght vector from equaton (9). Ω k s obtaned by averagng the weght vectors of tranng mages n face class k. Ω K X represents the knowledge base that the PCA classfer uses to dstngush between the X face classes. Fnally,

23 we specfy face class and face space threshold values θ class and θ face, respectvely, such that the classfcaton rules n able apply. able : Egenface classfcaton rules a) Image s a face and s recognzed ( class k ε < θ class and ε face < θ face ) b) Image s a face and s not recognzed ( ε class k > θ class and ε face < θ face ) c) Image s not a face and s recognzed ( ε class k < θ class and ε face > θ face ) d) Image s not a face and s not recognzed ( ε class k > θ class and ε face > θ face ) For nput mage Γ nput, we subtract Ψ from t to form mean adjusted nput mage Φ nput, and project Γ nput onto face space to obtan weght vector Ω. We then compute Φ face and the face class class k ε and face space ε face metrcs, and classfy the nput mage wth respect to the rules n able. Note that ε class k represents the smallest Eucldean dstance between k =, K, X. Ω and Ω k for face classes he θ class and θ face threshold values are carefully chosen by tral and error, snce they are crtcal to the recognton performance of the PCA classfer. In addton, one cannot be specfed ndependently of the other n the case that nput mages are not n the same classfcaton as the tranng mages. For example, f the PCA classfer s traned wth face mages and nput mages are not faces, then specfyng a hgh θ face value causes the classfer to err for nput mages that are known not to be face mages. On the other hand, specfyng a low θ face value 3

24 causes the classfer to be overly conservatve and reject mages that are known to be faces, regardless of whether a match s found for a face class. However, f we assume a pror that all mages are face mages, then θ face may be dsregarded by settng t to a suffcently large value. We note that the optmal values for θ class and θ face to maxmze the percentage of recognzed faces s dependent on the set of nput mages. he classfcaton rules n able essentally dvde the nput space nto four classfcaton groups. herefore, the addton of new nput mages can change the values for θ class and θ face and shft the optmal dvson lnes. 3.. Program Functonalty he program functonalty follows the algorthm and detals as outlned n the prevous secton. Fgure shows the user-nterface for the egenface program. Pror to runnng the program, a set of tranng mages s loaded and processed by ntally assgnng to each person (face class) a drectory (name of whch s also the name of the face class) contanng one or more mages of the person n dfferent orentatons, lghtng, or facal expressons, etc. he user selects the root drectory path where the face class drectores are stored, and the program loads nto memory the names of the face classes and the paths of all the assocated tranng mages. PCA tranng s then performed by computng the egenfaces (prncpal components) of all the tranng mages n the selected root drectory, and determnng the average weght vector of each face class. After the tranng process, the user can match a sngle face mage to ts 4

25 Fgure : Egenface program nterface correspondng face class, or classfy a group of nput mages and determne the maxmum face class and face space values. hese values are then used to maxmze the percentage of recognzed faces from the set of nput mages. he egenface program mplementaton also supports mage reconstructon. Both mage classfcaton and reconstructon frst projects the nput mage onto the set of egenfaces to obtan ts characterstc weght vector. For mage classfcaton, the nput weght vector s used to compute the face class and face space metrcs class k by ε = Ω Ω and ε k face = Φ Φ face, respectvely. he classfcaton rules of able are then used to classfy the nput mage accordngly. For mage reconstructon, the nput weght vector s used to compute equatons (9) and (0). 5

26 3..3 Program Desgn he egenface program s mplemented n C++ and MFC (Mcrosoft Foundaton Classes) for user-nterface development n the Vsual Studo envronment. he class dagram of the egenface program s shown n Fgure 3. he CImageVewer and CEgenDlg classes are derved from the MFC class CDalog, whch s used to dsplay a dalog box on the screen. he CImageVewer class mplements functonalty to dsplay grayscale mages by usng the MFC class CImage, whch provdes magng support between the user and the wndows devce context of a dalog box. he CEgenDlg class mplements functonalty for all of the GUI controls. CStrngoPtrMap s an MFC class that maps an MFC CStrng object to a ponter of type CObject. CFaceClassData nherts from CObject and maps face class names to CImageData objects, whch encapsulate mage pxel data stored n CMatrx objects. he CImageUtl class provdes support for the readng, wrtng, and scalng of grayscale pgm mages. he pgm mage format represents pxel values n 8 or 6 bt P or P5 ASCII format. he P verson represents each pxel value by an ASCII number strng (e.g. 55 for whte). he P5 verson represents each pxel value by an ASCII character n bnary format (e.g. character ÿ for whte). he CWnUtl class supports the retreval of path names of fles or drectores stored n a gven drectory, dsplay of a drectory search dalog box, and varous path name manpulaton functons. he CWnUtl class encapsulates code that s taken from the Wndows Programmng book []. CEgenSolver encapsulates functonalty to solve and sort egenvalues and egenvectors of a symmetrc real 6

27 matrx. he mplementaton s taken from Chapter of Numercal Recpes n C [8]. he matrx class provdes support for most matrx operatons and s taken from the Matrx CL webste [9]. CMatrx extends the matrx class wth addtonal functons specfc to the computaton of egenfaces. Fgure 3: Egenface program class desgn 7

28 3..4 Reconstructon and Recognton Results o obtan mage data for the egenface program, we retreve 400 pgm mages from the Cambrdge Unversty Engneerng Department Database of Faces [5]. hese mages consst of 40 face classes wth 0 mages for a gven person n each face class that vares n lghtng, facal expressons, as well as nplane and out-of-plane face rotatons, wth no background detals. o tran the egenface program, we create 40 face classes each assgned 5 mages of the same person for a total of 00 tranng mages. Fgure 4 shows several tranng mages. After tranng, we obtan egenfaces and mean mages as shown n Fgure 5. o test the program, we reconstruct the remanng 00 non-tranng (a) (b) (c) (d) (e) (f) (g) (h) () (j) Fgure 4: ranng mages of two face classes Fgure 5: Example egenfaces (left four) and mean mage 8

29 mages from the same face classes as the tranng mages, and observe that about half are reconstructed such that they resemble human faces wth reasonable clarty. For ths test, we don t separate the nput mages by gender snce they are from the same face classes as the tranng mages. We also retan all egenvectors to maxmze reconstructon accuracy. Fgure 6 shows several reconstructed mages. We note that the lower mages n Fgures (6b) and (6d) look very smlar to the mages n Fgures (4b) and (4), respectvely. Patterns observed from the test results demonstrate that f the nput mage looks smlar to a subset of the tranng mages, the reconstructed mage also looks smlar to ths subset. Conversely, f an nput mage does not resemble a subset of tranng mages, the qualty of the reconstructed mage suffers. (a) (b) (c) (d) Fgure 6: Input (top) and reconstructed (bottom) mages o obtan recognton performance statstcs, we classfy the same 00 non-tranng mages as n the reconstructon case. We obtan results n able by settng θ class = 360 and θ face = 680, just slghtly larger than the maxmum ε class and ε face values. hs causes the classfer to regard all mages as faces 9

30 and maxmzes the percentage of correctly classfed faces at the expense of ncreasng the percentage of ncorrectly classfed faces. able : Recognton results for 00 test mages a) Correctly classfed faces 9% b) Incorrectly classfed faces 8% c) Unrecognzed faces (no face class matches) 0% d) Not faces 0% e) Mnmum ε class 5.56 f) Maxmum ε class g) Mnmum ε face h) Maxmum ε face Fgure 7 shows the reconstructed results of a tranng mage usng 00, 75, 50, 5, and 0 percent of top egenvectors retaned, respectvely. As the percentage of retaned egenvectors approaches zero, the reconstructed mage approaches the mean mage, as predcted by equaton (0). Even wth only half of the egenvectors, the reconstructed mage s stll dscernble. Fgure 7: Egenvector truncaton results 3.3 Grayscale Age Progresson In ths secton, we descrbe the mplementaton of an mage-retreval and preprocessng tool that downloads applcable mages from a mssng chldren's web server, preprocesses the mages n terms of sze, orentaton, format, and 30

31 creates a drectory structure to organze and store the mages. Specfcally, ths tool retreves young and correspondng aged mage pars of a gven person. he (young, aged) mage pars are then concatenated and nput nto the egenface program to produce age-progressed grayscale mages Image Retreval and Preprocessng he mage retreval and preprocessng tool conssts of three command lne Python scrpts (webcrawler.py, face_ops.py, and mage_ops.py) and a C helper program (trackface.exe) that calls a face detecton lbrary from [6]. Fgure 8 llustrates a hgh-level overvew of the mage retreval, preprocessng, and age progresson process for grayscale mages. he functon of the webcrawler.py scrpt s to communcate wth a specfc mssng chldren s web server and download all mssng chldren's mages by state (e.g. Calforna). Each mssng chld s represented by one young mage and a correspondng dgtally aged mage. hs scrpt nteracts wth a Java servlet object by sendng the approprate parameters. For example, the url usmapsearch&mssstate=ca&searchlang=en_us returns a web-page that contans mages of all mssng chldren n the state of Calforna. he webcrawler.py scrpt parses ths web-page for all the relevant mage tags and uses the url s to download the mages. Each (young, aged) mage par s stored 3

32 n a drectory assgned to a gven mssng person. All mage drectores are then stored n a specfed root drectory. Fgure 8: Image retreval, preprocessng, and age progresson process he face_ops.py scrpt traverses the drectory structure created by webcrawler.py and preprocesses all mages stored n t. Frst, the scrpt converts each mage to grayscale pgm mage fle format. It then nvokes a helper program trackface.exe to read the pgm mage. he helper program uses a neural network based face detecton lbrary [6] to locate the boundary of a face and center ponts of the eyes. It then saves the dmensons to a fle that face_ops.py parses. he face_ops.py scrpt reads the dmensons and then crops a face n each mage to maxmze the face area (.e. reduce pxels covered by har and 3

33 background), and corrects any n-plane rotaton so that faces are vertcal. he n-plane rotaton angle s determned between a lne through the centers of the eyes and the horzontal axs, and s appled to the center of an mage. Each rotated and cropped (young, aged) face par s reszed, concatenated, and stored n a specfc drectory structure. he mage_ops.py scrpt traverses the drectory structure created by face_ops.py, and creates a new drectory structure that contans concatenated nput or tranng grayscale pgm mages as llustrated n Fgures and 3, respectvely. We rely on the Python programmng envronment and magng lbrary to provde the necessary mage processng tools and operatons as needed Age Progresson est Results We use the egenface program to tran on a number of concatenated mages created by face_ops.py. hen we nput a concatenated mage of the same young face on both sdes of the mage nto the egenface program. We wsh to obtan a reconstructed mage wth one sde of the mage agedprogressed and extracted. We frst download a number of mages from by usng the webcrawler.py python scrpt. An example par of young and aged mages s shown n Fgure 9. he Python scrpts face_ops.py and mage_ops.py are then nvoked to produce a set of preprocessed tranng mages as shown n Fgure 0. Gven a concatenated nput mage as shown n Fgure, the egenface program produces a 33

34 reconstructon of the nput mage that s shown n Fgure. he correspondng tranng mage s shown n Fgure 3. Note that the nput mage n Fgure s Fgure 9: Young and correspondng aged mage par the same left face of Fgure 3, and therefore the resultant mage should be well reconstructed. In general, however, we want to reconstruct nput mages that are not from the tranng mage set. Fgure 0: Concatenated tranng mage Fgure : Concatenated nput mage Fgure : Reconstructed nput mage Fgure 3: Concatenated tranng mage 34

35 We retreve and preprocess 300 female (young, aged) face pars (00 by 00 pxels) wth the Python scrpts webcrawler.py and face_ops.py. We assgn 00 of them as concatenated tranng mages as n Fgure 0, and the rest as young concatenated nput mages as n Fgure. For ths test we do not separate the tranng mages accordng to any crtera, and essentally gather them n a sngle group. We use the egenface program to reconstruct the 00 nput mages wthout truncatng any egenvectors, and vsually determne that 60 out of the 00 resultant mages are reconstructed successfully, and 5 of the 60 successfully reconstructed mages are adequately age-progressed. Dong the same for 50 male tranng mages and 50 male nput mages, we observe 30 out of the 50 resultant mages are successfully reconstructed, and 0 of the 30 successfully reconstructed mages are adequately age-progressed. Fgure 4 shows successful grayscale reconstructons of aged female and male grayscale mages for ths test. Of the successfully reconstructed mages that are not well age-progressed, the egenface program s not able to dscernbly age the mages. Fgure 4: Grayscale reconstructons of aged (rght sde) female and male 35

36 3.4 Feature Matchng Program In ths secton, we dscuss the feasblty of shape contexts as descrbed n the paper by Belonge and Malk [] to provde a robust means of quantfyng and representng the shape of an object. hs representaton s then used n conjuncton wth other technques to locate features on face mages. he dea s to manually select features on a template mage, perform edge detecton and sample ponts from the edges, and then fnd ponts on a test mage that best correspond to sampled ponts of features on the template mage Shape Context and Bpartte Matchng Shape context s characterzed by the spatal relatonshp between a gven pont and all other ponts n a shape. Specfcally, t s defned by a set of ponts sampled from the nternal or external edges of a shape. he edges are determned by usng an edge detecton algorthm that searches for regons of changng mage ntensty and localzes the edges between these regons. hs applcaton uses the Boe-Cox edge detector from Practcal Algorthms for Image Analyss [0]. Fgure 5 shows an example of an edge-detected mage. Fgure 5: Edge detected mage 36

37 Snce face mages are nherently random, t s generally dffcult to regster ponts on the edges of an edge-detected face for exact spatal correspondence and orderng of ponts between two mages. herefore, we use a random samplng of ponts for ths applcaton. For a gven sampled pont, a shape context descrptor s defned by determnng the set of vectors from the pont to all other sampled ponts on the shape. Specfcally, the shape context for a pont s a log-polar hstogram that sorts all vectors for a gven pont by relatve dstance and angular orentaton. herefore, for N sampled ponts, each pont has N - vectors to all other ponts. And the correspondng log-polar hstogram that has x radal and y angular separatons has x * y bns. he log-polar plot can be vsualzed as a seres of concentrc crcles enclosng a number of wedge bns. he wedge bns are unform n angular spacng, but vary logarthmcally n the radal drecton. In other words, n Cartesan space, the nner crcles are closer together than the outer crcles. But n log space, the crcles are spaced unformly n the radal drecton. Fgure 6 below llustrates ths. he second pcture shows Fgure 6: Log-polar bns and correspondng flattened hstogram [] 37

38 a flattened shape context hstogram wth the relatve darkness of the bns ndcatng pont densty. he advantage of usng shape contexts s that several geometrc nvarances are nherent wth ths method. Specfcally, nvarance to translaton s bult-n snce vectors are between all ponts on a gven shape. Invarance to scalng s acheved by normalzng all vectors wth respect to the mean vector for all N * (N - ) pars of ponts. Rotatonal nvarance s acheved by determnng the average vector from a pont to all other ponts and usng ths vector as the axs to measure the angle from for each pont. Qualtatvely, the shape context for each pont gves a precse descrpton of the relatve poston of a pont to all other ponts. hus, the shape context can be used as an effectve measurement of shape smlarty for a correspondng pont on another shape to be compared. hs mples that there must be a pontto-pont correspondence between ponts on two shapes. o obtan pont-to-pont correspondence, the log-polar hstogram representng the shape context for each pont on a shape, may be used to calculate the measurement cost between ponts on two shapes. Specfcally, the χ (ch-squared) dstance metrc s used. Gven two shapes each wth N sampled ponts, a cost matrx of sze N N s set up where each matrx element s the χ cost between ponts on the two shapes. he cost equaton s defned by C j = K [ h ( k) h j ( k) ] h ( k) h ( k) k= + j for K bns, where represents the cost between ponts p on one shape and p j on the other. he term h ( k) represents the k th bn of the normalzed hstograms at p and p j. In C j 38

39 essence, we are comparng the smlarty between hstograms at two ponts. he denomnator for rato for a partcular k. C j may go to zero, n whch case we just assgn a zero to the herefore, the matrx represents the cost between all possble pars of ponts, and s an nstance of the weghted bpartte matchng problem. hs problem may be cast n the more famlar form of the machne schedulng problem, where the goal s to assgn one task to every machne so that every task wll be tended to. he cost elements of the matrx measure the effectveness of a machne to perform a specfc task, whle the objectve value ams to maxmze the value for all machne and task pars. In modelng the bpartte matchng problem wth respect to shape contexts, we wsh to mnmze the total cost for all pont pars. he bpartte matchng problem may be solved n O(n 3 ) by usng the Hungaran algorthm. hs applcaton uses a more effcent algorthm gven by Jonker and Volgenant [4] n solvng the lnear assgnment problem. By solvng the bpartte matchng problem, we have the optmal correspondence between sampled ponts from features on the template and test mages. We then determne the centrod of each group of ponts, whch approxmately locates correspondng features on the test mage. For each group of corresponded ponts on the test mage, we remove outlers greater than a 39

40 number of standard devatons from the mean radal dstance between the group center and each pont Program Functonalty As detaled above, the applcaton frst performs edge detecton to retreve the edge data of mages. hese edges are then sampled for both test and target pont-sets for the two mages to be mapped. Next, pont-to-pont correspondence s performed between the two pont-sets to obtan a mnmum overall cost. he Fgure 7: Feature Matchng Program Interface 40

41 user-nterface shown n Fgure 7 conssts of four (control, template, test, and feature-locate) vews. he control vew allows the user to load, dsplay, edge detect, and sample ponts from mages n the template and test vews. After determnng the optmal mappng of ponts, the user selects features n the template vew to be corresponded to features n the test vew, whch are then dsplayed n the feature-locate vew. he program takes all the user-selected ponts from the template vew, maps t to the test mage, and removes outlers that are not wthn a specfed number of standard devatons from the group center. Next, the convex hull of each group of matched and culled ponts are computed, and dsplayed along wth the group centers n the feature-locate vew. he user can fne-tune overall performance by modfyng sgma and threshold values for edge detecton senstvty, angular and radal bn count for pont matchng performance, and sample pont count from the edge data Program Desgn he feature matchng program s mplemented n C++ and MFC and supports the dsplay of pgm mages only. A class dagram of the featurematchng program s shown n Fgure 8. he CControlVew class mplements functonalty for all of the GUI controls, and s derved from MFC class CFormVew, whch s a vew wth embedded GUI controls. he scroll-enabled CemplateVew, CestVew, and CFeatureVew classes provde support for the dsplay of grayscale template, test, and feature-matched mages, respectvely. he CPGMUtl class mplements functonalty to read, wrte, and supermpose 4

42 Fgure 8: Feature matchng program class desgn 4

43 dfferent mages (e.g. sampled ponts and edges overlad on top of a face mage). he CPontUtl class mplements functons for random pont samplng from edges, and determnng the centrod and removng outlers from a group of ponts. CShapeContextUtl mplements the shape context descrptor as detaled n Secton he mplementaton takes two sets of ponts and rearranges one of them such that each par of ponts s (least-cost) optmal n accordance wth a cost matrx determned by the shape context descrptor. CShapeContextUtl reles on the CBpartteMatcher class to solve the pont correspondence problem represented by a cost matrx. he CBpartteSolver class encapsulates functonalty and source code taken from CConvexHull mplements Graham s scan algorthm and encapsulates source code taken from Computatonal Geometry [6]. he CEdgeDetector class encapsulates edge detecton functonalty and source code taken from Practcal Algorthms for Image Analyss [0]. he CWnUtl and CMatrx classes are descrbed n Secton Feature Matchng est Results We test the program wth three sets of data consstng of shape and face mages, and obtan test results by expermentng wth varous values of detector senstvty, angular and radal bn counts, and number of sampled ponts. We select values for detector senstvty, bn count, and sampled pont count that avod excessve detected edges, decrease n pont matchng performance, and ncrease n computaton tme, respectvely. Excessve detected edges mply a 43

44 hgher probablty that ponts may be sampled from non-feature edges, whch bases the cost matrx towards non-features. A large bn count causes the cost matrx to approach a zero matrx and degenerate nto random pont matchng, assumng a random samplng of ponts from the edges. Increasng the number of sampled ponts also ncreases the sze of the cost matrx, whch s solved by the lnear assgnment algorthm [4] n O(n 3 ). o test the program wth shapes, we use pgm mages of alphanumerc characters for a total of 36 pars of test mages (A Z and 0 9). Wth ths test set, the program successfully locates features on 3 of the 36 character mages. Fgure 9 shows several feature-located mages for varous shapes. Next, to test Fgure 9: Shape matchng results (selected boxes top, matched features bottom) 44

45 the program for face mages of the same person, we use 50 (young, aged) mage pars retreved and preprocessed by the Python scrpts webcrawler.py and face_ops.py descrbed n Secton he program successfully locates userselected features (e.g. eyes, nose, and mouth) for 30 of the 50 (young, aged) pars. Fgure 0 shows a successfully feature-located test mage. We then test 50 pars of face mages of dfferent persons and successfully match features for 5 face pars. Fgure llustrates an unsuccessful attempt at feature locaton for two dfferent faces. Fgure 0: Successful feature matchng of (young, aged) par of same person Fgure : Unsuccessful feature matchng of dfferent faces 45

46 3.5 Dscusson of Intal Results he test results from Secton 3..3 suggest that an nput mage must be smlar to a subset of the tranng mages to be accurately reconstructed. More specfcally, we want the projected nput mage onto the transformed coordnate system to be n close proxmty to a cluster of tranng mages wth specfc aged features. However, PCA s an unsupervsed technque such that the outputs are dependent solely on the tranng data wth no drect control over how the clusters are formed. herefore, the optmal tranng mage set s one that contans a suffcent number of faces of dfferent people, as well as adequate varaton n expresson, orentaton, lghtng, and other parameters for a gven person, etc. hs ncreases the probablty that the projected nput mage s postoned near a cluster of tranng mages that captures the desred age trats, and wth the reconstructed mage resemblng the nput mage, snce PCA mnmzes the error between the two. Image dmensonalty s another factor that must be consdered. By ncreasng the dmensons of the tranng mages, we ncrease the number of parameters that may be used to dscrmnate the dfferent clusters. As a result, we ncrease the dmensonalty and descrptveness of the set of prncpal components. Also, the dmensonalty reducton process s more effectve snce for hgh-dmensonal data, the number of egenvectors that may be truncated far exceeds the dmensons needed to accurately represent a data pont n the new 46

47 coordnate system. We conclude that mage reconstructon for smaller mages s more senstve to dfferences between nput and tranng mage data. o mtgate the lmtatons of nadequate range and sze of the tranng data n terms of the number of avalable mages, Secton 3.4 begns explorng feature-based reconstructon and age progresson by evaluatng a soluton for locatng features on a face mage. he results of the feature matchng program ndcate that usng shape contexts and solvng the bpartte matchng problem, provde a robust soluton for locatng features for smlar shapes wth an exact or smlar orderng and confguraton of vertces. However, ths soluton does not work well for mages that do not have a well regstered set of vertces. Gven that face mages are nherently random, t s dffcult to obtan a precse regstraton of ponts between mages. We look to a more robust method of locatng features by usng the face detecton lbrary from [6], and to explore the feasblty of usng color mages to ncorporate RGB data and ncreasng reconstructon performance. 47

48 Chapter 4 Fnal Desgn and Implementaton 4. Feature-Based Age Progresson In ths chapter and followng sectons, we dscuss the extenson of the egenface program to support feature-based face reconstructon and age progresson for color mages. Snce PCA s an unsupervsed tranng technque, there s no drect control over the tranng process. By extractng and analyzng ndvdual mage features, ths gves some ndrect control over how ndvdual features are traned and reconstructed. In addton, there s less varaton for ndvdual features as compared to an entre face, and as a result the probablty of accurately reconstructng a feature ncreases. Next, we conclude from the results of Secton that a more relable method of locatng features on a face mage s needed. herefore, we use a neural network based face detecton lbrary [6] that s descrbed n Secton Also, snce ths applcaton supports color mages, and egenfaces are derved from grayscale mages, we use an mage matrx encodng to ncorporate the (red, green, blue) values of a 4-bt color mage n a format that supports matrx operatons. 4. Feature-Based Overvew In feature-based age progresson, the process frst searches for key features on a face mage (.e. eyes, nose, mouth, and face). hen, features are extracted and encoded n a specfc mage matrx format as n Fgure 5. hs s 48

49 done for each young and correspondng aged mage, results of whch are concatenated to form a tranng mage that separates the RGB ntensty values and creates a mappng between each (young, aged) mage par. Next, egenfeatures (prncpal components) are computed from the sets of tranng mages for each feature on a face mage. o age progress an nput mage, the features are extracted, projected onto ther correspondng egenfeatures, and reconstructed to obtan age-progressed features (Sectons..3 and 3..). hese age-progressed features are then blended back to respectve locatons to form an overall age-progressed face. Fgure llustrates ths feature-based agng process. 4.. Feature Detecton and Extracton o locate features on a face mage, we use the neural network based face detecton lbrary from [6]. However, ths face detecton lbrary only locates the center of the eyes and face crop boundary of a head mage. herefore, we use the poston of the eyes and face to locate and bound the rest of the features (.e. nose and mouth). More specfcally, we use the dstance between the left and rght eye as a base metrc from whch to estmate the locatons and szes of the other features. Fgure 3 llustrates a smple dmenson scheme to parameterze all major features on a face. 49

50 Fgure : Feature-based age progresson process 50

51 Fgure 3: Parameterzed face dmensons (not to scale) hs dmenson scheme, obtaned by tral and error, only gves rough estmates of the dmensons, assumes that certan face feature ratos do not vary sgnfcantly between face mages, and works best for frontal faces (no out-ofplane rotatons). o refne the locatons and szes of the face feature boundares, we use a smple boundng box algorthm. he algorthm frst re-centers a rough boundng box by performng edge detecton on the bounded feature, and usng the edge map wthn the rough boundary box to approxmate the new center. It then determnes the extremtes of the edge map wthn the re-centered box and uses them to set the borders of the new boundng box. Fgure 4 shows several feature-detected face mages. 5

52 Fgure 4: Feature detected faces As Fgure 5 llustrates, the RGB matrx format s a concatenaton of the red, green, and blue color channels for a young face mage that s concatenated wth the same results of an aged face mage. herefore, t s essentally sx Fgure 5: RGB mage matrx format mages concatenated nto a larger one. Snce PCA calculates the egenvalues and egenvectors of a real matrx for ths applcaton, ths s a smple method of ncorporatng the RGB values nto a format that supports matrx operatons. It also allows an aged-progressed mage to be easly extracted by truncatng the 5

53 proper half a reconstructed mage, and then multplexng the RGB values nto a color mage. Subsequent to feature-detecton, n-plane face / head rotaton s removed. he correctve angle θ s measured between a lne passng through the center of the eyes and the horzontal axs, and the correspondng rotaton s appled at the center of the face mage. We obtan the new centers of the eyes by applyng the D rotaton matrx cosθ snθ snθ cosθ. By applyng the rotaton angle θ, all features are then centered about the md vertcal axs. he dmenson scheme n Fgure 3 s then used n conjuncton wth edge detecton to determne the locaton and best ft boundng box of each feature. Next, snce PCA derves prncpal components from data wth the same dmensonalty, correspondng features extracted from each face mage are reszed to the common dmensons to ensure vald matrx computatons. 4.. Feature Blendng and Reconstructon Subsequent to the computaton of the prncpal components (egenfeatures) of each feature, we extract and age-progress features from an nput mage by projectng them onto ther correspondng egenfeatures by equaton (9). Age progresson s performed by reconstructng the features from the weght vectors by equaton (0). An overall age-progressed face s then formed from the consttuents as llustrated n Fgure 6. o blend n the ndvdual aged features, edge detecton s frst appled to determne the 53

54 Fgure 6: Reconstructon and blendng process extremtes of each feature. he convex hull s then computed from the edges to obtan a rough contour. By dong so, each feature can more readly blend nto the face wthout leavng behnd demarcaton lnes of the boundng boxes around each feature. We then substtute the hulled features onto the aged face n ther respectve locatons, essentally overwrtng the features on the aged face. he fnal post-processng step nvolves applyng a low-pass Gaussan flter to remove 54

55 hgh-frequency effects, resultng n the slght blurrng of the mage to further reduce the lnes of demarcaton left from the blendng process. hs flter convolves a square kernel matrx wth a Gaussan profle to each pxel of the nput matrx, essentally applyng a neghborhood averagng effect that emphaszes the center kernel values. he Gaussan profle provdes a more effectve flter as compared wth a unform flter by removng nose whle preservng detal. Fgure 7 shows features that are solated to hghlght the approxmated contour, and the correspondng aged and blended mage. Fgure 7: Input mage, aged and contoured features, and blended mage 4.3 Program Functonalty he feature-based age progresson program s mplemented n the Mcrosoft Vsual Studo envronment usng C++ and MFC, and ncorporates most of the functonalty llustrated n Fgure 8 except for the retreval of mages, thereby elmnatng the need for multple modules. Fgure 8 shows a screenshot of the user-nterface. he program s nterface conssts of four (control, tranng, 55

56 nput, and output) vews. he left control vew allows a user to load, tran, and reconstruct color mages, manually modfy the n-plane rotaton angle of an nput Fgure 8: Feature-based age progresson program nterface or tranng mage, dsplay the edge map of an mage, and select a featuredetected tranng mage to dsplay. he top-rght tranng vew dsplays featuredetected tranng mages, and allows the user to modfy the feature detecton results by manually draggng and selectng new feature-boundng boxes n the 56

57 vew area. he lower-mddle nput vew dsplays the mage to be ageprogressed, and also allows user to drag and select boundng boxes around features to be age-progressed. And the lower-rght output vew dsplays the reconstructed and age-progressed mage. he user may also fne-tune parameters for edge detecton and low-pass flter senstvty, and percentage of egenvectors to truncate. Modfyng edge detecton senstvty affects the locaton of the boundng box of each detected feature. Increasng flter senstvty causes the output mage to be more blurred. And truncatng egenvectors decreases executon tme at the expense of reconstructon qualty. o run the program, we use the Python scrpt webcrawler.py descrbed n Secton 3.3. to retreve the needed tranng (young, aged) mage pars. Each par s stored n a separate drectory, wth all drectores stored n a root drectory. he user selects the root drectory and the program loads all the tranng (young, aged) mage pars nto memory. PCA tranng s then performed by frst nvokng the face detecton lbrary to locate the features (.e. eyes, nose, mouth, and face) for all tranng mages. he extracted features from the young and aged mages are formed nto tranng mages n the format llustrated n Fgure 5, and the prncpal components are computed for each respectve feature. Next, the user selects an nput mage to load nto memory to be reconstructed and ageprogressed as llustrated n Fgure. 57

58 4.4 Program Desgn A class dagram of the feature-based age progresson program s shown n Fgure 9. he CControlVew class mplements functonalty for all of the GUI controls. he CImageVew class supports the dsplay of 4-bt RGB mages. CSelectVew extends CImageVew by mplementng functonalty that allows user to drag selecton boxes n the vew area. COutputVew extends CImageVew by mplementng a pop-up context menu n the vew area. he CInputVew and CranngVew classes provde support for both selecton boxes and pop-up menus. he CImageMatrx class mplements functonalty for varous mage processng operatons such as rotaton, scalng, low-pass flterng, and converson from a 4-bt RGB mage to the mage format as llustrated n Fgure 5. It encapsulates MFC class CImage to provde low-level magng support. he source code for mage rotaton, scalng, and low-pass flterng are taken from Practcal Algorthms for Image Analyss [0]. he CFaceMap class maps the path name of each tranng mage wth a CranngData object. Each CranngData object contans two CImageData objects, one to hold data for a young mage, and the other for an aged mage. he CFaceParams class stores varous face parameters such as boundng box dmensons of detected features and n-plane face rotaton angle. he CFeatureLocate class mplements functonalty that locates features on a face mage (eyes, nose, mouth, and face), and determnes the best-ft boundng box around each feature by usng edge detecton provded by CEdgeDetector. It nvokes the face detecton lbrary [6] to fnd the center postons of the eyes and boundares of the face, and uses the 58

59 Fgure 9: Feature-based program class desgn 59

60 dmenson scheme n Fgure 3 to determne the locatons of the other features. he dmensons of the face features are then mapped n CFaceMap and stored n CImageData objects for each tranng mage. he CEgenUtl class mplements functonalty that traverses a CFaceMap object, uses the n-plane rotaton angle and boundng box dmensons stored n each CImageData object, and extracts the pxel values n boundng boxes from each tranng mage. It then forms a set of tranng mages for each feature, and uses each set to compute the prncpal components for a gven feature. CEgenUtl uses CEgenSolver to compute the egenvalues and egenvectors from the covarance matrx of each feature. he CReconstruct class mplements functonalty to extract features from an nput face mage by usng the CFeatureLocate class, performs feature age-progresson by projectng each nput feature onto a correspondng set of prncpal components, and reconstructs an overall aged face usng the mage processng functonalty provded by CImageMatrx. he CPolygonester class mplements functonalty to test whether a pont s n a gven polygon. hs class s used to extract pxels from a convex hull regon of a face mage as llustrated n Fgure 7. CMatrx and CWnUtl are dscussed n Secton And CConvexHull and CEdgeDetector are dscussed n Secton Feature-Based est Results We use the same test mages as n Secton 3.3., but of dfferent dmensons. hs ncludes 300 female (young, aged) face pars that are 40 by 300 pxels of varous age ranges. We assgn 00 of them as concatenated 60

61 tranng mages as n Fgure 0, and the rest as young concatenated nput mages as n Fgure. We use the feature-based agng program to reconstruct the 00 nput mages wthout truncatng any egenvectors, and vsually determne that 85 out of the 00 resultant mages are reconstructed successfully, and 30 of the 85 successfully reconstructed mages are adequately age-progressed. Dong the same for 50 male tranng mages and 50 male nput mages also 40 by 300 pxels, we observe 35 out of the 50 resultant mages are successfully reconstructed, and 5 of the 35 successfully reconstructed mages are adequately age-progressed. here s a marked mprovement n the number of successfully reconstructed and age-progressed mages as compared to the results from Secton Specfcally, we get a 00% and 50% mprovement for female and male age progresson results, respectvely. hs s attrbuted to the use of color mages and ncrease n mage dmensons, as dscussed n Secton 3.5. We attempt to mprove on the feature-based age progresson results by ncludng more (young, aged) tranng mage pars for a gven person. Due the lmted number of (young, aged) pars per person that can be retreved from we obtan mages from a famly album to obtan create more tranng mages for a gven person. he dea s to gather a number of mages to represent four dstnct age groups: baby, toddler, adolescent, and adult. We then produce a set of tranng mages concatenatng all possble pars between mages n the baby toddler, and adolescent group wth mages n the 6

62 adult group. Specfcally, we use the baby, toddler, adolescent, and adult mages as shown n Fgure 30 to produce a total of ( ) x 6 = 0 concatenated tranng mages. Fgure 30: ranng mages for clusterng test (permsson from Chrs Pollett) he dea behnd ths test, as dscussed n Secton 3.5, s to have the projected nput mage be n close proxmty to a cluster of projected tranng mages wth the desred age trats. We nput a baby, toddler, and adolescent mage and obtan correspondng reconstructed mages as shown n Fgure 3, wth feature extracton dsabled due to poor extracton results. he results demonstrate that the close clusterng of tranng mages causes the reconstructed 6

63 result to resemble a weghted averagng of all the adult tranng mages. he averagng depends on the nput mage and where t les n reference to the tranng mages. We note that the reconstructed results of (5a) and (5c) are qute smlar, whch mples that the projecton of the baby and adolescent nput mages onto the set of prncpal components has a smaller Eucldean dstance value as compared to the projecton of the toddler nput mage. herefore, the PCA classfer regards the baby and adolescent nput faces as beng more smlar. (a) (b) (c) Fgure 3: Reconstructed results (bottom) of baby, toddler, and adolescent We then add 00 (young, aged) mage pars nto the tranng mage set retreved by the Python scrpt webcrawler.py to observe how clusterng performs wth mages from dfferent people. We add a suffcent number of addtonal tranng mages and hope that excessve skewng from the ntended cluster of tranng mages s prevented by an averagng effect. Fgure 3 shows the 63

64 correspondng reconstructed results of the same nput mages usng the extended tranng mage set wth feature extracton dsabled. We see from Fgure 3 that the addton of tranng mages of dfferent people dmnshes the qualty of the reconstructon such that the mages no longer resemble the adult mages n Fgure 30. hs s because as more tranng mages are added, a reconstructed mage s formed wth a greater number of prncpal components due to the ncrease n dmensonalty of the transformed coordnate system needed to represent the extra data. As a result, the reconstructed result also captures unwanted features from the added tranng mages. Fgure 3: Reconstructed results wth extended tranng set 4.6 Colorzaton Applcaton and Results o perform further testng of the feature-based agng program, we create tranng mage pars to consst of a gray and color mage of the same person, as opposed to a young and aged mage par. By dong so, we attempt to convert the applcaton from one that ages a color face mage, to one that colorzes a grayscale face mage by changng the tranng mage data. Fgure 33 shows two grayscale mages and correspondng colorzed mages. 64

65 Fgure 33: Example colorzaton results For the colorzaton test, we use webcrawler.py to retreve 300 young face mages, and copy and convert 00 of them to grayscale mages. Each grayscale and correspondng color mage s concatenated to form a tranng mage. Colorzaton s then attempted on the remanng 00 grayscale mages. Results show that 90 of the 00 mages are successfully reconstructed to reasonably resemble the nput mages n color. Vsual nspecton of the colorzaton results shows that the colorzed mages demonstrate consstent usage of color for skn tone and features. However, some of the results tend to exhbt regons of extreme whte blendng nto areas of color as shown n Fgure 34. Also, as Fgure 34: Colorzed face wth brght regons 65