A Survey on Matching Sketches to Facial Photographs

Transcription

1 A Survey on Matching Sketches to Facial Photographs Archana Uphade 1, Prof. J. V. Shinde 2 1 M.E Student, Kalyani Charitable Trust s Late G.N. Sapkal College of Engineering 2 Assistant Professor, Kalyani Charitable Trust s Late G.N. Sapkal College of Engineering Savitribai Phule Pune University Abstract Recently face recognition is attracting more attention in many computer applications for automatically identifying or verifying a person. There has been number of ways to match facial features to photographs. In criminal investigation, hand drawn sketches are used to match possible suspect with mug-shot gallery set. Various representation methods has been developed and used for face identification. These representation methods are used to solve the problem of matching facial sketches to photographs. 1. Introduction Face recognition is one of the most studied research area in computer vision. Also it has wide potential applications in commercial area like face identification, face verification, criminal investigation etc. As use of such application increases emerging technologies also developed to matching sketches to facial photographs. 2. Representation The face images in each modality can be represented by various representation approaches. Common options for representations includes component-based, analytic, patch-based and holistic. 2.1 Component-based representation This method detects facial parts like eyes and mouth, and represents appearance of each component individually [Liu et al.2012; Han et al.2013]. This allows the matching to be measured separately of each component [Liu et.al.2012]; after detecting component correctly matching is provided. 2.2 Analytic representations [Nejati and Sim 2011; Yuen and Man 2007; Pramanink and Bhattachajee 2012] detect facial components and viewed points, 1983

2 allowing face to be modeled geometrically, 2.4 Patch-based holistic representations it experiment using point distribution This encode the appearance of each image in models [Nejati and Sim 2011; Yuen and Man 2007]. This representation has the advantage over other methods is that if a model can be fit to a face in each modality, and then analytic/geometric representation is relatively invariant to modality. However this representation is not robust to errors in face model fitting and may require some form of manual intervention to avoid this [Yuen and Man 2007]. Moreover geometry patches with a feature vector per patch [Wang and Tang 2009; Liu at al.2005; Galoogahi and Sim 2012b; Kiani Galoogahi and Sim 2012; Bhatt et al.2012]. Subsequent large feature vector [Klare and Jain 2010b], or learning a mapping/classifier per patch [Zhang et al.2011]. The later strategy can provide some robustness if the true mapping is not constant over the whole face, but does require a patch fusion scheme. is not robust to facial expression [Pramanink and Bhattachajee 2012], and does not exploit texture information by default. 3. Matching facial sketches to images The problem of matching facial sketches to photos is commonly known as sketch-based 2.3 Holistic representations This method represents the whole image in each modality with a single vector [Yi et al.2007; Tang and Wang 2002; Lin and Tang 2006].Compared to component-based face recognition (SBFR). It involves a gallery dataset of images and a large dataset of facial sketches. An important application of SBFR is in law enforcement agencies to identify suspects by retrieving their photos and analytic approaches, this has the automatically from existing police advantage of encoding all available databases. In most cases actual photograph appearance information. However, it is sensitive to alignment and expression/pose of the suspect are not available, in these circumstances drawing a sketch following variation, and may provide a highdimensional the description provided by eyewitness is feature vector that risks over- fitting [Tan et al.2006]. method used police to assist artist to draw sketches. But a professional artist required large training in both drawing and sculpting, due to budgetary reason many law enforcement 1984

3 agencies used composite software to Existing facial sketch datasets synthesize a sketch. Dataset Pair of Hand s sketch/p viewed/for drawn/comp hoto ensic osite 3.1 Categorization of facial sketches CUFS[W 606 Viewed Hand-drawn Facial sketches can be created either by an ang and artist or by software, and are referred to as hand-drawn sketch and composite sketch Tang 2009] respectively. Depending on the artist CUFSF[Z 1,194 Viewed Hand-drawn ang et.al. observes the actual face of the suspect or not 2011; before sketching, they can be categorized as viewed and forensic (unviewed). -Forensic hand drawn sketches: These are produced by a forensic artist based on the description provided by the eye witness ([Klum et al.2013]) -Forensic composite sketches: They are Wang and Tang 2009] IIIT-D viewed sketch[b hatt 238 Viewed Hand-drawn synthesized by computer software by et.al.2012 selecting the various facial components ] IIIT-D 140 Semi- Hand-drawn based on the description provided by a semiforensic Forensic witness. It is observed that 80% of law enforcement agencies use this form of software to synthesize sketches. The most widely used software for generating facial composite sketches are Identikit [Wright et sketch[b hatt et.al 2012] IIIT-D 190 Forensic Hand-drawn al.2007], Photo-Fit [G.Wells and Hasel 2007], FACES [Biometrix 2011], Mac-a- Mug [G.Wells and hasel 2007], and EvoFIT [Frowd et al.2004]. -Viewed hand drawn sketches: These types of sketches are drawn by artist while hatt et.al 2012] 4. Viewed sketch face recognition Viewed sketch recognition is the most studied problem; it has an importance forensic and sketch[b Composite looking at a corresponding photo. 1985

4 towards result because it ultimately face image at different scales using a improving forensic sketch accuracy. Viewed sketch-based face recognition is classified into synthesis, projection and feature-based methods. technique Markov Random Fields (MRF). In later work, a multi-scale MRF learns local patches and scales jointly instead of independently which used in [Liu et al. 2005]. The scale of learned local face 4.1 Synthesis-based approaches This synthesis based approach synthesizes a photo from corresponding sketch or viceversa. structures is based on the size of overlapped patches. By converting sketches or facial photos into same modality, modality-gap is An eigensketch transformation reduced. algorithm is used to convert a photo into sketch which is proposed by [Tang and Wang 2002].Classification can be provided by the obtained eigenskecth features. The 4.2 Projection based approaches A key strategy in projection based approach is to find a lower-dimensional sub-space so eigensketch transformation algorithm that two modalities are directly comparable. reduced the discrepancies between photo and sketch. Lin and Tang [Lin and Tang 2006] proposed a linear transformation which can be used Liu et al. [Liu et al.2005] proposed a Local between different modalities (sketch/photo), Linear Embedding (LLE) method to convert called common discriminant feature photos into sketches based on the image patches. Those sketches are geometry preserving synthetic sketches. For each image patch to be converted, it finds the extraction (CDFE). In this method, images from two modalities are projected into a common feature space so that matching can be efficiently performed. nearest neighbors in the training set, and uses their corresponding sketch patches to synthesize the sketch patch. In [Liu et al. 2005] a kernel-based nonlinear discriminant analysis (KNDA) classifier is used by Liu et al. for sketch photo Tang and Wang [Wang and Tang 2009] further improved [Liu et al. 2005] by developing an approach to synthesize local recognition. The main contribution is to use the nonlinear kernel trick to map input data into an implicit feature space. 1986

5 4.3 Feature based approaches Datasets Feature-based approach focuses on Metho Dat Feature Trai Accu designing a feature descriptor for each image that is invariant to the modality, while d Publica tions aset n: Test racy being variant to the identity of the person rather than matching photos into sketches, or both into a common subspace. The most 1. Synth esis [Tang and Wang CU HK Eigensketch feature 88:1 00 about 60% widely used image feature descriptor are based 2002] Scale invariant feature transform (SIFT), [Liu et CUF - 306: Gabor transform, Histogram of Averaged Oriented Gradients (HAOG) and Local Binary Pattern (LBP). Once sketch and al 2005] [Wang and Tang S CUF S Multisca le MRF : 300 % 96.3 % photo images are encoded using these 2009] descriptors, they may be matched directly. [Liu et CUF E-HMM - 100% al.2007] S Klare et al. [Klare and Jain 2010b] proposed the first direct sketch/photo matching method based on invariant SIFT-features [Lowe 2004]. SIFT features provide a compact vector representation of an image patch based on the magnitude, orientation and spatial distribution of the image gradients [Klare and Jain 2010b]. SIFT feature vectors are first sampled uniformly from the face images and concatenated together separately for sketch and photo images. Then by using Euclidean distance computed between concatenated SIFT feature vectors of sketch and photo images for NN matching. 2. Projec tion 3. Featu re based [Zhong et al.2007] CUF S E-HMM % [Klare CUF SIFT 100: 96:47 and Jain S 300 % 2010b] [Sharm CU 88: a and HK 00 % Jacobs 2011] [Choi et CUF Gabor 0: al.2012] S, and 00 % CUF CCS- SF POP [Klare CUF SIFT 100: and Jain S 300 % 2010b] [Khan CUF Self/Sim 161:

6 et S iarity 150 % al.2012] [Bhatt CUF et.al S 2010] [Galoog CUF ahi and S Sim 2012] [Kiani CUF Galoog SF ahi and Sim 2012] [Bhatt IIITet.al D 2010] EUCLB P 78: % HAOG 306: % Gabor 500: Shape 694 % EUCLB 58: P 73 % which is computed on uniform patches across the entire face images. Later translate it into a facial Sketch/mugshot into 154 uniform patches, SIFT [Lowe 2004] and multi-scale local pattern (MLBP) [Ojala et al. 2002] invariant features are extracted from each patch. With this feature encoding, as improved version of the common representation from [Klare and Jain 2010b] is applied, later RS-LDA [Wang and Tang 2006b] is applied to generate a discriminative subspace for NN matching with cosine distance. The score generated by each feature and patch are used for final recognition. 5. Composite sketch based face recognition These studies have focused on face recognition using composite sketches which are synthesized by composite software. Yuen et al. [Yuen and Man 2007] uses the both local and global features to represent sketches. This method required user input in the form of relevance feedback in the recognition phase. The second two focus on holistic [Klare and Jain 2013] representation. The holistic method [Klare and Jain 2013] uses similarities between local features, In component based method [Han et al. 2013], they uses component based approach, in which they calculate similarity score in between different facial component. By using Active Shape Model (ASM) facial landmarks in composite sketches and photos are automatically detected. To extract component features, Multiscale Local Binary Pattern (MLBP) are applied and similarity score is calculated. To obtain the overall sketch-photo similarity, similarity scores of each facial component are normalized and fused together. 1988

7 Datasets Publicati ons [Yuen and Man 2007] [Han et.al. 2013] 6. Conclusion Composi te-mug shot pairs Datase t Availabil ity Unknown 123 AR Databa se publically available From the survey we observed, as compared to the photo-based face recognition there is limited amount of the work is based on the sketch recognition; lots of the published work is based on the hand drawn sketches. For hand drawn sketches a professional artist is required so due to budgetary reason many law-enforcement agencies now using composite software to synthesize a sketch. But hand drawn sketches depict each facial component correctly with shading and composite software lack in textual description. 7.References [1] Hu Han, Brendan F. Klare, Kathryn Bonnen, and Anil K. Jain. Matching Composite Sketches to Face Photos: A Component-Based Approach IEEE Transactions on Information Forensics And Security, Vol. 8, No. 1, January [2] X. Tang and X.Wang, Face sketch recognition, IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 1, pp , Jan [3] D. Lin and X. Tang, Inter-modality face recognition, in Proc. Eur. Conf. Computer Vision, 2006, pp [4] X. Wang and X. Tang, Face photosketch synthesis and recognition, IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 11, pp , Nov [5] B. Klare, Z. Li, and A. Jain, Matching forensic sketches to mug shot photos, IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 3, pp , Mar [6] P. C. Yuen and C. H. Man, Human face image searching system using sketches, IEEE Trans. Syst., Man, Cybern. A, Syst. Humans, vol. 37, no. 4, pp , Jul [7] D. Lin and X. Tang, Recognize high resolution faces: From macrocosm to microcosm, in Proc. IEEE Computer Vision and Pattern Recognition, 2006, pp [8] H. Han, S. Shan, X. Chen, S. Lao, and W. Gao, Separability oriented preprocessing for illumination-invariant face recognition, in Proc. Eur. Conf. Computer Vision, 2012, pp [9] B. Klare and Anil K. Jain. 2010a. Heterogenous Face Recognition: Matching NIR to Visible Light Images. In International conference on Pattern Recognition (ICPR)

8 [10] Brendan Klare and Anil K. Jain. 2010b. Sketch-to-photo matching : a feature based approach. In Biometric Technology for Human Identification VII.SPIE [11] Brendan F. Klare and Anil K. Jain Heterogenous Face Recognition Using Kernel Prototype Similarities. TPAMI(2013), [12] P. C. Yuen and C. H. Man Human Face Image Searching System Using Sketches. IEEE Transaction on Systems, Man and Cybernetics, Part A: System and Humans (TSMC) (2007),