Selecting Discriminative Binary Patterns for a Local Feature

Transcription

1 BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 15, No 3 Sofia 015 rint ISSN: ; Online ISSN: DOI: /cait Selecting Discriminative Binary atterns for a Local Feature Yingying Li 1,, Jieqing Tan 1,3, Jinqin Zhong 1 1 School of Comuter and Information, Hefei University of Technology, Hefei Anhui, China School of Electronics and Information, Anhui Jianzhu University, Hefei Anhui, China 3 School of Mathematics, Hefei University of Technology, Hefei Anhui, China s: jieqingtan@hfut.edu.cn yyhgd@ahjzu.edu.cn jinqinzhong@163.com Abstract: The local descritors based on a binary attern feature have state-of-theart distinctiveness. However, their high dimensionality resists them from matching faster and being used in a low-end device. In this aer we roose an efficient and feasible learning method to select discriminative binary atterns for constructing a comact local descritor. In the selection, a searching tree with Branch&Bound is used instead of the exhaustive enumeration, in order to avoid tremendous comutation in training. New local descritors are constructed based on the selected atterns. The efficiency of selecting binary atterns has been confirmed by the evaluation of these new local descritors erformance in exeriments of image matching and object recognition. Keywords: Selecting atterns, searching tree, local descritor, matching, binary attern. 1. Introduction Local feature is widely used in many comuter vision tasks, such as image retrieval [1, ], object detection and recognition [3, 4], scene categories [4, 5] and action classification [6, 7]. The local feature is the first stage to get mid-level visual reresentation or high-level semantic descrition in most of the vision alications [1-7]. lentiful alications drive the researches to roose a variety of local feature descritors. 104

2 Among these resented descritors, CS-LB [8], MRRID [9], OC-LB [4] have taken the limelight for their high distinctiveness, robustness to illumination and simle comutation. These three local descritors are all based on the histogram of a binary attern feature on the given satial ooling. The local descritors CS-LB and MRRID use CS-LB histogram [8], and the local descritor OC-LB uses OC-LB histogram [4]. These three descritors have more discrimination than SIFT [10], however the higher dimensionality imedes them from matching faster and being used in a low-end device. Fig. 1. CS-LB transformed image: (a) original image, (b) CS-LB transformed image with 16 atterns, (c) CS-LB transformed image with 8 selected atterns In our work we find that some atterns contribute more distinctiveness to the image descrition, while some atterns contribute less. For examle, Fig. 1 shows two CS-LB transformed images with 16 atterns and 8 atterns having similar distinctiveness in aearance, though (c) has lost the information of 8 atterns. Accordingly, different atterns may contribute differential discrimination to the local descritor. We aim to select discriminative atterns to construct a more comact local descritor and discard less informative atterns in order to reduce the dimension of the local descritors with low loss of distinctiveness. Many learning methods on local descritors target at achieving a comact higher level feature, such as middle feature, image feature, leading to a different learned result in different vision scenes and tasks. Few learning is used to construct a local descritor roviding a low level feature. However, recently learning for constructing a local descritor has received increased interest by the researchers. K e and S u k t h a n k a r [11] use rincial Comonents Analysis (CA) to the normalized gradient atch to gain a more comact descritor. C a i, M i k o l a j c z y k and M a t a s [1] roose Linear Discriminant rojections (LD) for reducing the dimensionality of local descritors. A large data set from multiimage 3D reconstruction is introduced by B r o w n, H u a and W i n d e r [13], which makes ossible the learning result to be suitable for universal alication. LDB [14], RFD [15] and discriminative Learning of local image descritors [13] are all trained on this large data set. The authors of [13] make use of Linear Discriminant Analysis (LDA) and owell minimization to solve the roblem of the arameters for gaining a more discriminative descritor. LDB, RFD both use greedy learning techniques to select a salient bit for constructing a comact and distinctive binary descritor. LDB uses a similar AdaBoost-based Bit Selection and RFD emloys iterative otimization. 105

3 This aer s contributions lie in: 1) we roose an efficient learning method to select discriminative binary atterns in a non-greed way; ) we design a Loss function as a selecting criterion to evaluate the discrimination of the selected atterns in constructing a local descritor. The efficiency of atterns selection is confirmed by evaluating the erformance of the generated local descritors based on the selected atterns in exeriments of image matching and objects recognition.. The method roosed The aim is to select a generic atterns subset for constructing a comact local descritor. There are three roblems to solve: the first one is to generalize the uniform flow for constructing a local descritor based on a binary attern; the second is to define a selecting criterion to decide which is the discriminative attern for the local descritor; the third is how to learn feasibly on a large data set to avoid a local otimum..1. Uniform flow for constructing a local descritor In order to make atterns selection indeendent on descritor construction, we generalize the uniform construction flow from CS-LB, OC-LB and MRRID, as shown in Fig.. Firstly, detect the region. These three descritors can emloy the same region detector. In this aer Hessian-Affine detector is used. Secondly, artition the region into several sub-regions. CS-LB and OC-LB use SIFT-like grid, MRRID uses intensity order ooling artition. Thirdly, comute the attern occurrence histogram on each sub-region. CS-LB and MRRID both use the centersymmetric local binary attern, while OC-LB uses orthogonal combination of a local binary attern. At the fourth and fifth stes, these three descritors do the same thing: concatenating the histograms of each sub-region together and generating the final local descritor... Selecting criterion 106 Fig.. The uniform flowchart for constructing a local descritor As well known, the recall versus 1-recision curve [16] is widely acceted as the evaluation criterion by many local descritors, which assesses the erformance of the local descritor under different thresholds. In order to eliminate the influence of the different thresholds on the learning result, we evaluate the local descritor using

4 its ROC area of the recall versus 1-recision curve between 80% and 95% recision, as shown in Fig. 3. Fig. 3. The recall versus 1-recision curves for two descritors. S is the ROC area between 80% and 95% recision of the descritor below. S is the ROC area of the descritor above Based on ROC area, a Loss function is designed to evaluate the discrimination loss of the descritor based on the selected atterns subset, comared with the original descritor. It is defined as (1) S (1) Loss( ) = 1 = 1 S where F( x, recision F( x, recision i 1 i 1 i 1 i 1 i, recision, recision i, recision, recision i i , recision, recision x (1 ) 1 r r1 r3 r x (1 3) F( x, 1, r1,, r, 3, r3, 4, r ) 4 = 1 r ( x (1 ) ) x (1 3) 1 r3 r r4 r 3 x (1 ) + 1 r3 + ( x (1 3) ), x [1,1 3], ) dx, ) dx where S is the ROC area of the original local descritor between 80% and 95% recision (the area below the uer curve), and S is the ROC area of the local descritor based on the selected attern subset (the area below the lower curve); recision i and recall i (i = 0, 1,, n) are multile recisions and recalls of the local descritors generated on the candidate atterns set under different thresholds, i recision and recall i (i = 0, 1, n) are multile recisions and recalls on the attern subset and n is the number of thresholds we used (n = 8 in our selecting rocess). The call and recision are comuted in () by atches matching in a training set using the corresonding local descritor, #correct matches # false matches () recall = ; 1 recision =, # corresondences #all matches where #corresondences is the ground truth number of matches in the training set; #correct matches is the number of correctly returned matches; #false matches reresents the number of falsely returned matches; the total of 107

5 #false matches and #correct matches is #all matches. The ROC area is comuted using the cubic Hermite interolation of these recisions and recalls in the interval [0.8, 0.95] that 1-recision belongs to. The dimensionality of the local descritor is determined by the number of atterns used in its construction. The most discriminative atterns are selected to construct a local descritor with low dimension at minimum loss of distinctiveness. A Loss function is used as our selecting criterion. The smaller the Loss function is, the better the corresonding attern subset is in selecting..3. Selecting with a searching tree The source of our training data comes from ublicly available data sets (Liberty, Yosemite, and Notre Dame [13]). Each data set contains over 400K atches, samled densely from 3D reconstruction scenes. In each data set, the ground truth data indicating the match and mismatch has been given. In rearing the training set, we randomly chose 50K airs of matching atches and 00K airs of nonmatching atches from Liberty and Notre Dame dataset as the training set. We use Yosemite as a testing data set. 108 Fig. 4. Selecting with a searching tree In order to gain a stable and generic result, we use a non-greedy learning to select the otimum. However, the evaluation of the selecting criterion for all candidate subsets in exhaustive enumeration is a huge amount of work. For each candidate attern subset, we must construct their corresonding descritors for all the atches in the training data set and comare their Euclidean distance in airs to decide the match or the mismatch, finally comute its loss function as (1) by the statistics of the match and the mismatch and the given ground truth data. To avoid exhaustive enumeration and local otimum, our selecting emloys the selecting tree with Branch&Bound [17], as shown in Fig. 4. In this searching tree, the root reresents the candidate attern set before selecting and each node is the subset of atterns corresonding to a local descritor based on this subset. The middle nodes reresent the intermediate subset in selecting rocess and the leaves contain the final selected result. The node excet the root is achieved by removing a attern from its father node. The atterns number in the node is reduced successively with the tree level down. The deth of

6 the tree relates to the attern number to be selected, which is determined by the requirement, such as memory store limitation. The rightmost successor has one branch, which is achieved by a greedy way. This means that each node in the rightmost branch (see Fig. 4) has the least loss among the subsets enumeration that are achieved by removing a attern from its father node resectively. We construct the asymmetric tree structure with sarser branches in the right, than in the left, as in the method given in [17]. Based on this tree structure, we try to osition a good attern subset node into the right sarse art of the tree. We need to order the nodes among their brothers according to their discrimination loss. In order to avoid a mass of heavy comutation on the Loss function evaluation for each node, we utilize a simle rediction mechanism. We redict the Loss function of the subset using the distribution roortion of the corresonding discarded attern from its father node, since the extensive exeriments in [18, 19] have demonstrated that the attern with a small roortion is inadequate to rovide reliable and discriminative information. In this rediction mechanism, the nodes in the right art of the searching tree may likely rovide a better selected result than in the left with high robability. Our searching ath is a re-ordered traversal from right to left, shown as a dashed line in Fig. 4. When the node is visited, we comute its Loss function and comare it with the bound: when the Loss function of the current node is lower than the bound, we continue to visit the next node in the order of the re-order traversal; otherwise cut off its subtrees. An initial bound is set as the Loss function of the rightmost leaf. In the traversal, the bound would be udated by the lower Loss function of the visited leaf. The bound reresents the currently least discrimination loss for the visited leaves, and its corresonding leaf is saved as the currently best attern subset for the given attern number to be selected. The bound is also used to cut off the subtrees of the node whose loss function is higher than the bound. We seed u the searching rocess by cutting off the bad subset. In order to avoid a discarding ossible otimum in cutting off the subtrees, the monotonicity condition should be ensured: the loss function of the subset should not be lower than that of the set which contains this subset. In general, without disturbing of noise or blurring, the attern feature always obeys the monotonicity condition. In our training set, most atches are high-quality, the atches disturbed by the blurring and noises are only minority. So the losses of the nodes satisfy the monotonicity in our selecting. Suose that n is the attern number to be selected and N is the total number of the candidate atterns. The above selecting is efficient when n is not less than N/. When n is less than N/, we deal with selecting in dichotomization: Ste 1. Select the N/ most discriminative atterns from N candidate atterns with the above selecting method. Ste. If n is not less than N/4, then select n atterns from the N/ most discriminative atterns selected at Ste 1, else set the selecting n atterns from the N/ most discriminative atterns as a new selecting roblem and iterate the whole rocess. 109

7 . 4. Selecting result Our learning is conducted three times: one is selecting from 16 atterns of CS-LB based on a Cartesian grid (4 4); the second is selecting from 3 atterns of OC-LB on a Cartesian grid (4 4); the third is selecting from 16 atterns of CS-LB on intensity order ooling (4 artitions). From the first and the second learning based on the same artition, we gain two grous of selected subsets with different sizes. We comare the discrimination of these attern subsets with different attern number in serving to construct a local descritor by evaluating their corresonding generated local descritors on a test data set Yosemite, as shown in Fig. 5. The evaluation exeriment demonstrates that CS-LB rovides better distinctive atterns than OC-LB in constructing a local descritor. Considering the balance of erformance and dimension, CS-LB atterns serve better in constructing a local descritor than OC-LB. Fig. 5. The call at 95% recision for different descritors on the selected attern subsets with different attern numbers. Train: Liberty and Notre Dame (50k), Test: Yosemite (100k) For the first and the third selecting from the same attern set on different artition, we gain the same subset. This demonstrates that our selected result is indeendent on the artition in local descritor construction. 3. Exeriments In order to demonstrate that the selected result is discriminative and suitable for constructing a owerful local descritor with lower dimension, we use the selected CS-LB atterns and MRRID-like construction to generate the new local descritors 4_MRRID and 8_MRRID. 4_MRRID uses 4 selected binary atterns; 8_MRRID uses 8 selected binary atterns; 4_MRRID and 8_MRRID are based on 4 artition sub-regions and 4 suort regions. They both have lower dimension than their original descritors. Their dimensions are denoted in the legend of Fig. 6, which is labelled behind the notation _. We comare the erformance of these new local descritors with the state-of-the-art local descritors in exeriments of image matching and image recognition. 110

8 3. 1. Matching on the Oxford data set The new generated descritors based on the selected atterns are evaluated on a standard Oxford data set (htt:// which contains images in different transformations: viewoint change ( graf, wall ), scale change and image rotation ( boat ), image blur ( bike, tree ), illumination change ( leuven ), and JEG comression ( ubc ). The evaluation criterion uses the call at 95% recision, where the call and the recision are defined as. (). Fig. 6. Exerimental results under various image transformations in the Oxford data set for Hessian- Affine Region We comare the erformance of the roosed descritors with SIFT, original CS-LB, original MRRID, original OC-LB with Hessian-Affine regions, as shown in Fig. 6. 8_MRRID has similar erformance to original MRRID, while it only has half of the dimensions of the latter. It surasses SIFT, original CS-LB and original OC-LB with the same dimension or lower dimension. In some cases like leuven, bike, ubc, 4_MRRID has also shown retty better distinctiveness than SIFT, original CS-LB and original OC-LB, though its dimension is much lower. This is mainly because of two reasons: 1) the generated local descritor uses better construction than SIFT, CS-LB and OC-LB; ) the selected attern reserves most information of the original atterns. 3.. Object recognition on the Kentucky data set There are images in the Kentucky data set on the website htt://vis.uky.edu/~stewe/ukbench/data/, in grous of four that belong to the same object. One image of every grou is queried as the test, and the measure is based on whether the other three images of its grou are returned in the to three with the largest similarities. The recognition accuracy is defined as the ratio of the number of images returned in the to three correctly to the number of the total returned 111

9 images [10]. Suose that I 1 and I are two images in the data set, and {f 1 1, f 1,, f 1 m }, {f 1, f,,f n } are two sets of local descritors of the same tye extracted from I 1 and I. The similarity of the two images is defined as (3): 1 g( f, f ) i, j i j similarity=, m n (3) where if dist( fi, f j ) T g( fi, f j ) =, 0 otherwise and T is a threshold which is tuned to give the best result for each evaluated descritor. The object recognition result is shown in Table 1. 8_MRRID is aroaching the best erformance of MRRID with half of its dimensions. This means that binary attern selecting is efficient and the selected atterns are discriminative for constructing a local descritor. 8_MRRID surasses SIFT, OC-LB and OC-LB. 4_MRRID has comarable erformance of SIFT, OC-LB and OC-LB with the least dimensions among them. Table 1. Object recognition results on the Kentucky data set SIFT MRRID CS-LB OC-LB 8-mrrid 4-mrrid Descritor (18) (56) (56) (51) (18) (64) Recognition accuracy 48.% 57.5% 49.1% 48.8% 57.0% 49.0% 4. Conclusion In the aer we roose a feasible learning method to select a discriminative binary attern in order to construct a comact local descritor. We design a Loss function, using the area of the call versus 1-recision curves as the selecting criterion to gain a stable and distinctive result. In selecting, a simle rediction mechanism and Branch&Bound technology are emloyed to accommodate the tremendous comutation in training on a large data set. The exeriments in image matching and object recognition demonstrate that the selected attern is distinctive in serving for constructing a local descritor; the generated local descritors are comact and have comarable erformance to the original local descritors, being with lower dimensionality than their cometitors. Acknowledgements: This work was suorted by the NSFC-Guangdong Joint Foundation Key roject under Grant No U , the National Natural Science Foundation of China under Grant No and No and the National Science-technology Suort lan rojects under Grant No 01BAJ08B01. 11

10 References 1. Jégou, H., F. erronnin, M. Douze, J. Sa nchez,. érez. Aggregating Local Image Descritors into Comact Codes. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 34, 01, No 9, Y a n g, Y., S. N e w s a m. Geograhic Image Retrieval Using Local Invariant Features. IEEE Transactions on Geoscience and Remote Sensing, Vol. 51, 013, No, Seidenari, L., G. Serra, A. Bagdanov, S. Lorenzo, S. Giusee, D. B. A n d r e w. Local yramidal Descritors for Image Recognition. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 36, 014, No 5, Z h u, C., C. B i c h o t, L. C h e n. Image Region Descrition Using Orthogonal Combination of Local Binary atterns Enhanced with Color Information. attern Recognition, Vol. 46, 013, No 7, D o e r s c h, C., S. S i n g h, A. G u t a. What Makes aris Look like aris?. ACM Transactions on Grahics, Vol. 31, 01, No 4, K a â n i c h e, M., F. B r é m o n d. Recognizing Gestures by Learning Local Motion Signatures of HOG Descritors. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 34, 01, No 11, T h i, T. H., L. C h e n g, J. Z h a n g, L. W a n g, S. S a t o h. Structured Learning of Local Features for Human Action Classification and Localization. Image and Vision Comuting, Vol. 30, 01, No 1, Heikkilä, M., M. ietikäinen, C. Schmid. Descrition of Interest Regions with Local Binary atterns. attern Recognition, Vol. 4, 009, No 3, F a n, B., F. W u, Z. H u. Rotationally Invariant Descritors Using Intensity Order ooling. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 34, 01, No 10, L o w e, D. G. Distinctive Image Features from Scale-Invariant Keyoints. International Journal of Comuter Vision, Vol. 60, 004, No, K e, Y., R. S u k t h a n k a r. CA-SIFT: A More Distinctive Reresentation for Local Image Descritors, In: IEEE Conference on Comuter Vision and attern Recognition, Washington, DC, USA, 004, Cai, H., K. Mikolajczyk, J. Matas. Learning Linear Discriminant rojections for Dimensionality Reduction of Image Descritors. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 33, 011, No, B r o w n, M., G. H u a, S. W i n d e r. Discriminative Learning of Local Image Descritors. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 33, 011, No 1, X i n, Y., C. T i m. Local Difference Binary for Ultrafast and Distinctive Feature Descrition. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 36, 014, No 1, Fan, B., Q. Kong, T. Trzcinski, W. Zhiheng. Recetive Fields Selection for Binary Feature Descrition. IEEE Transactions on Image rocessing, Vol. 3, 014, No 6, Mikolajczyk, K., C. Schmid. A erformance Evaluation of Local Descritors. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 7, 005, No 10, Somol,.,. udil, J. Kittler. Fast Branch & Bound Algorithms for Otimal Feature Selection. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 6, 004, No 7, O j a l a, T., M. i e t i k ä i n e n, T. M ä e n ä ä. Multi Resolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary atterns. IEEE Transactions on attern Analysis and Machine Intelligence, Vol. 4, 00, No 7, Bongjin, J., K.Taewan, K. Daijin. A Comact Local Binary attern Using Maximization of Mutual Information for Face Analysis. attern Recognition, Vol. 44, 011,