AUTOMATIC fingerprint recognition systems (AFRS) are

Transcription

1 1808 IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 12, DECEMBER 2017 DeepPore: Fingerprint Pore Extraction Using Deep Convolutional Neural Networks Han-Ul Jang, Student Member, IEEE, Dongkyu Kim, Student Member, IEEE, Seung-Min Mun, Sunghee Choi, and Heung-Kyu Lee Abstract As technological developments have enabled highquality fingerprint scanning, sweat pores, one of the Level 3 features of fingerprints, have been successfully used in automatic fingerprint recognition systems (AFRS). Since the pore extraction process is a critical step for AFRS, high accuracy is required. However, it is difficult to extract the pore correctly because the pore shape depends on the person, region, and pore type. To solve the problem, we have presented a pore extraction method using deep convolutional neural networks and pore intensity refinement. The deep networks are used to detect pores in detail using a large area of a fingerprint image. We then refine the pore information by finding local maxima to identify pores with different intensities in the fingerprint image. The experimental results show that our pore extraction method performs better than the state-of-the-art methods. Index Terms Biometrics, convolutional neural network (CNN), fingerprint, pore extraction. I. INTRODUCTION AUTOMATIC fingerprint recognition systems (AFRS) are currently one of the most widely used biometric systems. Due to the development of the fingerprint scanner, more detailed Level 3 features are observed compared to Level 1 (pattern) and Level 2 (minutiae points), which are features used in low-resolution fingerprint images [1]. Level 3 features can be extracted from high-resolution fingerprint images of dots per inch (DPI), including sweat pores, incipient ridges, and edge contours [2]. The coordinates of pores are highly discriminative information [3] and the fingerprint recognition technology using Level 3 features shows high accuracy. The pore information is also used for partial matching [4] and fingerprint liveness detection [5] with Level 2 minutiae features. Pore extraction is an essential process for the variety of biometric systems based on pores. Thus, it is important to develop a pore extraction technique that has high accuracy. Manuscript received July 3, 2017; revised September 13, 2017; accepted September 26, Date of publication October 9, 2017; date of current version October 25, This work was supported by the National Research Foundation of Korea Grant funded by the Korea government (MSIT) (NRF- 2016R1A2B ). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Sanghoon Lee. (Corresponding author: Heung-Kyu Lee.) H.-U. Jang, D. Kim, S.-M. Mun, and H.-K. Lee are with the Multimedia Computing Laboratory, KAIST, Daejeon 34141, South Korea ( hanulj@kaist.ac.kr; d.kim@kaist.ac.kr; smmun@mmc.kaist.ac.kr; heunglee@ kaist.ac.kr). S. Choi is with the Geometric Computing Laboratory, KAIST, Daejeon 34141, South Korea ( sunghee@kaist.edu). Color versions of one or more of the figures in this letter are available online at Digital Object Identifier /LSP The existing pore extraction techniques are feature-based approaches [1], [6], [7] or a learning-based approach [8]. The feature-based approaches use isotropic or anisotropic models to detect pores. Qijun et al. proposed an adaptive pore detection technique using an anisotropic model [7]. Although, it is more accurate than the previous isotropic model based methods [1], [6], there are many limitations to adapt various pore shapes of various people. Labati et al. suggested the learning-based technique using convolutional neural networks (CNN) [8]. Since the technique involves subsampling, the pore intensity map is blurred that makes it difficult to accurately detect the pore location. Also, using the CNN structure without nonlinearity may limit the detection of various types of pores. The learning-based method uses Otsu s algorithm [9], which is binary thresholding (BT), for postprocessing to refine a pore intensity map. However, it is difficult to accurately detect pores with BT over the entire pore intensity map because the pore intensity is different for each pore. As a result, the accuracy of the learning-based method is similar to the feature-based technique using anisotropic models. Adaptive pore extraction is a very difficult problem because the shape of the pores differs from person to person and there are differences in the shape of the pores in the fingerprint of the same person. To overcome this issue on pore extraction, we propose a Deep CNN based Pore extraction (DeepPore). DeepPore consists of pore intensity extraction using CNN, and postprocessing using pore intensity refinement (PIR). The main contributions of our approach are the deep CNN architecture for detailed pore extraction and the refinement for pore peak finding, and more specific descriptions are as follows. 1) We suggest deep CNN for detailed pore extraction. Since deep CNN with many layers has a large receptive field [10], it can use a lot of information, such as the pore surrounding shape and the ridge shape of the pore belongs to, in the fingerprint image to detect pores. 2) We propose a postprocessing technique that detects pore peaks adaptively in the pore intensity map. In order to detect pore peaks with different intensities for each fingerprint region, we use a refinement method to find local maxima instead of the BT [9]. II. THE PROPOSED METHOD DeepPore consists of pore intensity extraction using CNN and postprocessing using PIR as presented in Fig. 1. In this section, we describe the CNN architecture for pore intensity extraction IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See standards/publications/rights/index.html for more information.

2 JANG et al.: DEEPPORE: FINGERPRINT PORE EXTRACTION USING DEEP CONVOLUTIONAL NEURAL NETWORKS 1809 Fig. 1. Overview of DeepPore. Fig. 2. CNN architecture for pore detection. Each convolutional layer contains 64 filters and some sample feature maps are drawn for visualization. The color of the feature map is inverted so that it can be visually confirmed. and then explain how to train the CNN. Finally, we introduce PIR, the postprocessing process. A. Proposed Network We use a deep convolutional network inspired by CNN-visual geometry group (VGG) [10] and train the CNN of DeepPore (CNN DP ) in a supervised learning manner. The goal of training to estimate a pore intensity map where the pores are enhanced and other patterns are reduced. We design the ground truth of a pore intensity map using soft labels. That is, the closer the distance to the pore, the higher the value of the label. Let l i,j and d i,j np denote the label value at a pixel coordinate (i, j), and the Euclidean distance between (i, j) and its nearest pore from (i, j) in the fingerprint image. If d i,j np is less than d τ, l i,j =1 d i,j np/d τ where d τ is the pore distance threshold. Otherwise (i, j) is a nonpore pixel, so l i,j =0. The CNN architecture of DeepPore is illustrated in Fig. 2. We perform the pore extraction process using d layers. Each layer has the same type: 64 filters with a size of 3 3. Therefore, 64 feature maps are generated for each layer. The layers 1 through d 1 perform convolution, batch normalization, and rectified linear unit (ReLU). In the last layer d, only the convolution process proceeds. Thus, the number of learnable layers is d and the number of nonlinearity functions is d 1. Weusethe fingerprint image as the input of CNN DP, and the output is the pore intensity map corresponding to the fingerprint image. The pore intensity map has the same size as the fingerprint image. We pad zeros before the convolution process in each layer to produce output equal in size to the input. Subsampling, which is commonly used for feature compression in object recognition, is not used because it may cause distortion in accurate pore position detection. In this letter, we use deep CNN because deep networks have larger receptive field sizes. For CNN with depth D and the size of the filters is 3 3, the receptive field size has (2D +1) (2D +1). In the task of pore extraction, the shape of the pores depends on the shape of the ridge to which they belong, their types (open or closed pore), and so on. We, therefore, use deep networks with large receptive field sizes to analyze a wider range Fig. 3. Visualization of (a) an example of pore intensity map and (b) an exampleof pore map. Asubblock of size is set for better visualization. of fingerprint information from the image and more precisely detect the pores. B. Training We now describe the process of training CNN DP.Letx and y denote the fingerprint image and the label of its pore intensity map, respectively. Given a training dataset {x i, y i } N i=1, our goal is to train model f to predict ŷ=f(x), where ŷ is the estimate of the pore intensity map and N is the batch size. The loss function of CNN DP is defined as follows: L DP = 1 2N N y i ŷ i 2. (1) i=1 C. Pore Intensity Refinement (PIR) The pore pixels appear as peaks in the pore intensity map. The intensity of each pore peak varies depending on the thickness of the ridge to which it belongs and its type (open or closed pore). Therefore, it is difficult to accurately detect the pores with BT over the entire pore intensity map. As shown in Fig. 3(a), the pore pixels are local maxima in the pore intensity map. We estimate the pore map by finding local maxima (pore peaks) [11]. Algorithm 1 presents our postprocessing method for pore map estimation. In summary, we estimate local maxima of the pore intensity map, which are equal to or greater than P τ, as pore pixels. The

3 1810 IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 12, DECEMBER 2017 Algorithm 1: Finding local maxima of a pore intensity map. M I is the pore intensity map of size R C M P is the pore map of size R C, ( (i, j) M i,j P 0) W i,j is the window of size s s and the center coordinate is (i, j) M I (W i,j ) is the matrix of M I belonging to W i,j window P τ is the pore detection threshold for i =1to R do for j =1to C do if M i,j I >= P τ then if M i,j I == max(m I (W i,j )) then M i,j P 1 M I (W i,j ) 0 end if end if end for end for pore map is generated by marking the coordinates corresponding to the estimated pore pixel to one and the other coordinates to zero. The example of the pore map is illustrated in Fig. 3(b). III. EXPERIMENTS A. Implementation Details We used a high-resolution-fingerprint (HRF) database [7] with 30 labeled images for pore extraction. The HRF database has pores coordinates, and each image has pixels and 1200 DPI resolution. For training CNN DP steadily, we partitioned each fingerprint image into overlapped subblocks and performed data augmentation to obtain sufficient training datasets. The size of the subblock was 80 80, which can accommodate the receptive field size of CNN DP with up to 10 learnable layers. A step size of 10 was used to generate overlapped subblocks, and a total of 89,250 subblocks were used in the experiment. The data augmentation included three types of flips and three types of counterclockwise rotation (90, 180, and 270 ). For testing, we used full-sized fingerprint images. The number of learnable layers d was 10, and ReLU was used as the activation function for training the model. CNN DP was trained using the Caffe [12] framework, which provides very fast CPU and GPU implementations. The base learning rate was set to We used batches of size 5 and trained all experiments for 100 epochs. To optimize CNN DP learning, we used the adaptive moment estimation (ADAM) to compute adaptive learning rates [13]. This is because, the ADAM has a better loss reduction than the stochastic gradient descent used for general purposes. We also used adjustable gradient clipping because the vanishing/exploding gradients problem can occur if the adaptive learning rate computed by ADAM is too [ high ][14]. In the learning process, the gradient is clipped to θ γ, θ γ, where γ is the current learning rate and θ =50is the gradient clipping parameter. Pore distance threshold d τ was set to 5. The size of the sliding window W for PIR was set to 11 11, which can accommodate TABLE I AVERAGE PERFORMANCE METRICS IN PERCENTAGE AND STANDARD DEVIATION (IN PARENTHESIS) DeepPore Labati et al. [8] DAPM [7] Adapt. DoG [7] R T (4.63) (7.81) 84.8 (4.5) 80.8 (6.5) R F 8.64 (4.15) (6.20) 17.6 (6.3) 22.2 (9.0) the range of pore adjacent pixels in the pore intensity map. Pore detection threshold P τ was set to d τ, the size of W, and P τ were experimentally set to values having the best performance. B. Evaluation Procedure We tested DeepPore using a five-fold validation strategy [15]. Three folds were used for training, one fold for validation, and one fold for testing. The validation fold is used to find the model with the lowest loss and the testing fold is used for evaluation. In each evaluation, the training, validation, and testing folds were mutually exclusive. Average performance is calculated by averaging the results of five evaluations. The true detection rate (R T ) and the false detection rate (R F ) are used to measure the accuracy of the pore extraction. R T is the ratio of the number of detected true pores to the number of all true pores, and R F is the number of falsely detected pores to the total number of detected pores. A good pore extraction algorithm should have a high R T and a low R F. We consider that the true pore is correctly detected if the Euclidean distance between the coordinate of the true pore and the coordinate of the detected pore is less than d p pixels. d p is set to RW /2, where RW is the average ridge width in the images of the test dataset. This is the same test environment as the state-of-the-art method [8]. IV. RESULTS A. Quantitative Evaluation We compare the state-of-the-art methods and DeepPore performance, and the results are described in Table I. DeepPore outperforms the existing feature-based methods such as adaptive DoG and dynamic anisotropic pore model (DAPM) [7] as well as the learning-based method [8]. DeepPore achieves higher R T,lowerR F, and lower standard deviation of R F compared to the existing techniques. Fig. 5 shows the performance of the networks with various depths. We measured the performance of each network by varying the pore detection threshold P τ. The networks with ten layers show the best performance when R F is less than 0.1. As the depth of network increases, the performance improves and the size of the performance improvement decreases. The performance of networks with ten layers is very slightly improved than that of networks with eight layers. It is expected that there will be no significant performance improvement even if the networks with a depth of eleven or more are used. In Table II, we compare performance against changes in networks and postprocessing. The networks and the postprocessing technique used by Labati s method [8], which is the

4 JANG et al.: DEEPPORE: FINGERPRINT PORE EXTRACTION USING DEEP CONVOLUTIONAL NEURAL NETWORKS 1811 Fig. 4. (a) Original image. The pore detection results of (b) DeepPore (c) and Labati s method [8]. The green, yellow, and red dots in these images represent the true positive, false negative, and false positive, respectively. TABLE III AVERAGE TIME BY TEST PHASE PER IMAGE Technique Generating pore intensity map (T G ) Postprocessing (T P ) Testing per image (T G + T P ) DeepPore 20 ms 30 ms 50 ms Labati et al. [8] 2 ms 30 ms 32 ms detection results of DeepPore and the existing method. In order to visualize the experimental results, true positive, false negative, and false positive are represented by green, yellow, and red, respectively. These results appear to suggest that the performance of DeepPore is superior to the comparison method. Fig. 5. Average performance metrics for networks with varying depths. TABLE II AVERAGE PERFORMANCE METRICS ON CHANGES IN NETWORKS AND POSTPROCESSING IN PERCENTAGE AND STANDARD DEVIATION (IN PARENTHESIS) CNN Labati with PIR CNN DP with BT R T (6.43) (8.57) R F (6.79) (3.61) state-of-the-art technique, are set as comparative experiments. The combination of Labati s networks and PIR is superior to that of Labati s method, but it is not as good as DeepPore. The combination of CNN DP and BT also gives the same result. These results show that both CNN DP and PIR contribute to the performance improvement. B. Qualitative Evaluation We evaluate the qualitative performance in comparison with the state-of-the-art method [8]. Fig. 4 visually shows the pore C. Computational Time Analysis In real applications, a short amount of time is required for pore extraction. We analyze the testing time of DeepPore and compare with the existing method [8]. We executed the tests using a quad-core machine with 4.0 GHz CPU, RAM 32 GB, and Nvidia GTX GB GPU. The operating system was Ubuntu LTS 64-bit and all methods were implemented using MATLAB. Table III demonstrates the average times for generating pore intensity map (T G ), the average times for postprocessing (T P ), and the average testing times that totalizes T G and T P. Since DeepPore has more layers than the comparison method [8], it needs more time for pore intensity map generation. However, the total testing time of DeepPore is as fast as 50 ms, so the pore extraction is possible in real time. V. CONCLUSION Adaptive pore extraction is a difficult problem because the pore information depends on the person, region, and pore type. To solve the problem, we have presented a pore extraction method using deep CNN and pore intensity refinement. We have demonstrated that our DeepPore outperforms the state-ofthe-art methods by a large margin on the benchmark database. In the future, we will research various biometric systems based on pores.

5 1812 IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 12, DECEMBER 2017 REFERENCES [1] A. K. Jain, Y. Chen, and M. Demirkus, Pores and ridges: High-resolution fingerprint matching using level 3 features, IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 1, pp , Jan [2] Q. Zhao et al., High resolution partial fingerprint alignment using pore valley descriptors, Pattern Recognit., vol. 43, no. 3, pp , [3] D. R. Ashbaugh, Quantitative-Qualitative Friction Ridge Analysis: An Introduction to Basic and Advanced Ridgeology. Boca Raton, FL, USA: CRC press, [4] D. Zhang, F. Liu, Q. Zhao, G. Lu, and N. Luo, Selecting a reference high resolution for fingerprint recognition using minutiae and pores, IEEE Trans. Instrum. Meas., vol. 60, no. 3, pp , Mar [5] P. Johnson and S. Schuckers, Fingerprint pore characteristics for liveness detection, in Proc. Int. Conf. Biometrics Special Interest Group, 2014, pp [6] N. R. Parsons et al., Rotationally invariant statistics for examining the evidence from the pores in fingerprints, Law, Probab. Risk, vol. 7, no. 1, pp. 1 14, [7] Q. Zhao et al., Adaptive fingerprint pore modeling and extraction, Pattern Recognit., vol. 43, no. 8, pp , [8] R. D. Labati et al., A novel pore extraction method for heterogeneous fingerprint images using convolutional neural networks, Pattern Recognit. Lett., [9] N. Otsu, A threshold selection method from gray-level histograms, IEEE Trans. Syst., Man, Cybern., vol. TSMC-9, no. 1, pp , Jan [10] K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arxiv: , [11] T. H. Cormen, Introduction to Algorithms. MIT press, [12] Y. Jiaet al., Caffe: Convolutional architecture for fast feature embedding, in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp [13] D. Kingma and J. Ba, Adam: A method for stochastic optimization, arxiv: , [14] R. Pascanu, T. Mikolov, and Y. Bengio, On the difficulty of training recurrent neural networks, in Proc. Int. Conf. Mach. Learn., 2013, pp [15] R. Kohavi et al., A study of cross-validation and bootstrap for accuracy estimation and model selection, in Proc. Int. Joint Conf. Artif., Stanford, CA, USA, vol. 14, no , pp