METHODOLOGY AND EXTENSIONS OF LOCAL BINARY PATTERN: A SURVEY

METHODOLOGY AND EXTENSIONS OF LOCAL BINARY PATTERN: A SURVEY CHANDRAJA D Department of Electrical Engineering (UG Student), BITS Pilani Hyderabad E-mail: chandraja.dharmana@gmail.com Abstract- The Local Binary Pattern is an image operator based on gray level differences between the center and the neighborhood of a pixel. It has been extensively used in texture classification, pattern recognition and facial image analysis. Owing to its simplicity and efficiency, several extensions have been developed. In this paper, LBP has been surveyed and implemented. It states in detail how the LBP operator works for texture as well as face description. While discussing each extension, modification in the basic algorithm, impact on the LBP image, major application and significant improvement over the original LBP have been addressed. The emphasis has been primarily laid on face description using LBP during implementation. Keywords- Local Binary Pattern, Feature Histogram, face description. I. INTRODUCTION The Local Binary Pattern is a feature descriptor which summarizes the small-scale appearance of an image by converting it into an array of labels. The labeling of pixels is based on difference in intensities between the central pixel value and the pixels in its corresponding neighborhood. Although LBP was originally developed as a texture descriptor [1]-[5] it has been exploited in facial image analysis and face recognition [6]-[11]. Facial expression recognition with boosted LBP features was proposed in 2007[12]. The 2D face description has been extended to temporal domain with the help of volume local binary patterns (VLBP)[13], combining motion and appearance. Recognition systems have been made more robust by taking into account varying factors like illumination, expression, occlusion and even gender. LBP has found application in medical image analysis [14],[15],object detection[16],etc The popularity of Local Binary Pattern can be attributed to mainly two characteristics-computational simplicity and discriminative power. In addition, the operator does not vary with monotonic gray scale transformations as the difference between the intensities of the pixels is considered. This in turn makes the operator robust against changes like illumination variations. To expand the scope of application, numerous extensions have been proposed and implemented. The central idea however remains the same, a binary code based on gray level differences between the center and neighborhood of a pixel. The paper broadly focuses on two aspects. It describes sequentially how the simple LBP feature vector is built for texture as well as face description. Secondly, it presents a survey of LBP and several of its extensions and variants. The need for the extension, how the modified operator encodes the missing information in original LBP to produce better classification results, the change in the basic algorithm, major application are the chief points that have been dealt with. As a typical application of LBP, the discussed extensions along with basic LBP have been implemented as face descriptors. The variations in the LBP image and feature histogram with the alteration have been observed. However, scope of discussion has been limited to spatial domain. The paper is organized as follows. Section 2 reviews LBP. Section 3 presents the implementation and results. Finally, Section 4 concludes the paper. II. LOCAL BINARY PATTERN 2.1. LBP as a texture descriptor The original LBP as introduced by Ojala et al. [1] is rooted in 2D texture analysis[2][3]. Texture is distinguished by two properties which supplement each other, pattern and contrast. The operator was developed to be used jointly along with simple local contrast measure and performed well in unsupervised texture segmentation.[5] In the original LBP, the labels for the image pixels are obtained by thresholding 3X3 neighborhood of each pixel as shown in Fig 1.Each bit is made zero or one based on the difference in intensities between the corresponding pixel and the central pixel. The string of bits obtained is followed in clockwise or counterclockwise direction to get an 8 digit binary number. The binary number is converted into its decimal equivalent to obtain LBP label for the center pixel. Since the 3x3 neighborhood consists of 8 pixels excluding the center, a total of 2 8 =256 different labels are possible. The histogram of all the labels was used as texture descriptor. LBP(x, y ) = s(l l )2 (1) 1,if k 0 s(k)= 0, if k<0 l : The 8-neighbor pixels values l : Center pixel s value 17

Figure 1: Thresholding the 3 x 3 neighborhood of each pixel with the center value and considering the result as a binary 2.2. LBP for face description The LBP based approach for texture analysis when applied to face description leads to loss of spatial information. The texture information has to be coded such that the physical locations are also retained. In order to achieve this, several regional or local descriptors of the face are built and combined into a global descriptor. In this paper, the methodology proposed by Ahonen et al.[6] has been incorporated. The image is divided into regions or blocks. Regional descriptors (block histograms) are obtained from each block independently with the help of LBP texture descriptors. The regional descriptors are then concatenated to form the feature vector. The above approach describes an image in three different levels. At the pixel level, LBP code is calculated for each pixel. Histogram calculated for each individual block encodes information at the regional level. The feature vector obtained by concatenation of all block histograms gives the final global description of an image. Rotation of the input image has two effects: the center pixel of each local neighborhood is rotated into another location. The corresponding local neighborhood is rotated along with the center pixel. Secondly, the orientation of the sampling points within each circular neighborhood changes.[17]to make LBP rotation-invariant, the effects of both translation to a different location and rotation about their origin have to be eliminated. Translation can be normalized by obtaining the histogram of LBP codes. Rotation invariant mapping technique is widely employed to normalize rotation. As per the mapping, each binary number is circularly rotated right until the minimum decimal equivalent of the binary pattern is obtained. LBP, = min ROR (LBP,, i) (7) where ROR(x, i) denotes the circular bitwise right rotation of bit sequence x by i steps. For instance, 8- bit LBP codes 10110100b, 01101001b, 10100101b, 01011010b all map to minimum code 00101101b. This method leads to loss of discriminative information because 8 different patterns are represented by a common LBP code. In case of a neighborhood with 8 sampling points, the number of distinct labels is reduced from 256 to 36. 2.3. Generic LBP The original LBP is calculated in fixed 3x3 neighborhoods with 8 pixels. In 2002, Ojala et al. [17] presented the extended generic form of LBP in which size of the neighborhood as well number of sampling points both could be varied. The principal definition was broadened to arbitrary circular neighborhoods. In a monochrome image I(x,y), consider an arbitrary pixel (x, y) with gray level g c =I(x, y).the gray level of a sampling point in circular neighborhood with (x, y) as the center pixel is given as g p = I (x p, y p ) p = 0,..., P 1 (2) x p = x +R cos(2πp/p) (3) y p = y R sin(2πp/p) (4) where R is the radius of the circular neighborhood and p is the number of sampling points. The LBP P,R operator is defined as LBP, = s(g g )2 (5) s(x) = 1, if x 0 0, otherwise Where (x c,y c ) denotes the center pixel of a particular neighborhood and g C =I (x c, y c ) corresponding gray level. P sampling points in a neighborhood give a P- bit binary number. Therefore, a given LBP code can have one of 2 P distinct values. This 2 P bin discrete distribution of LBP codes is used to approximately describe texture as T (LBP, x, y ) (6) 2.4. Rotation Invariance Figure 2: Effect of rotation in a neighborhood [17] 2.5. Uniform Patterns It was observed that the occurrence of some binary patterns was more dominant than others. This led to mapping of LBP labels as uniform or nonuniform. If the number of bitwise transitions from 0 to 1 or vice-versa in a circular fashion is atmost 2, the binary pattern is considered uniform. Otherwise, it is classified as non-uniform.[17].while computing LBP histogram, each uniform pattern is assigned a distinct label and all the non-uniform patterns are assigned to a single common label. Uniform LBP is indicated by LBP, where superscript u2 refers to uniform patterns (bitwise transitions U 2). LBP, = s(g g ) if U(LBP, ) 2 P(P 1) + 2 otherwise (8) In a neighborhood with P sampling points, the number of different labels is given by p (p-1) + 3. [26]Uniform patterns thus, help in reducing the length of feature vector. In addition, uniform patterns are robust and stable exhibiting better performance. Rotation invariance with uniform patterns 18

The LBP binary pattern can be made rotationally invariant followed by mapping of uniform patterns [17].LBP, is defined as LBP, = s(g g ) if U(LBP, ) 2 (9) P + 1 otherwise The mapping from LBP, to LBP, (superscript riu2 means rotation invariant uniform patterns with U 2) which has P+2 distinct output values can be implemented with the help of a lookup table. 2.6. Thresholding On the downside, LBP feature is noisesensitive and not very effective in the flat areas of an image. To diminish the effects of noise and constant gray level, the thresholding section of the operator was improvised by replacing[16] the term s(g p g c ) in Eq.6 with the term s(g p g c + a).thresholding makes the LBP image robust to noise as well as flat image areas. If the value of a is large, substantial changes in pixel values are allowed. This would eliminate noise but certain features might also be compromised. Since the discriminative power of LBP is to be retained, small value of a is optimal. 2.7. Center Symmetric LBP Center-Symmetric Local Binary Patterns (CS-LBP)[19] offered higher stability in areas of constant gray level (flat image regions) unlike original LBP. They were developed primarily for interest region description. In CS-LBP, gray level operator is computationally simpler and outperforms it. In addition, CS-LBP is tolerant to changes in illumination. 2.8. Multi-Scale LBP A major drawback of original LBP is its local fixed 3x3 neighborhood and subsequently, small spatial support area. Image structure is captured at a particular resolution [25]. Large scale features might not be detected which may be dominant features of certain textures. MS-LBP provides a larger spatial support area [20]. The multi-resolution representation based LBP considers LBP images of different scales. The radius of the neighborhood R as well as the number of sampling points P can be varied for the N LBP operators. Each pixel in an image gets N distinct LBP codes. The texture descriptor is given by the marginal distribution of these codes. The high dimensionality coupled with the small sample size limits the maximum number of scales to 3. When applied to a facial image, histograms are computed for each scale in a particular regional block. Concatenation of the N different histograms gives the regional descriptor [21]. Concatenation of the regional descriptors into a single vector gives the global descriptor. The performance of final histogram is compromised because of its high dimensionality and presence of redundant information. The central idea behind developing MSLBP was to encode structures differing in scales for better classification accuracy. From the results obtained in [20], the accuracy attained with LBP 8,1 was 92.7%. The MS- LBP with three scales LBP, +, +, offered better performance with an accuracy of 96.3%.However, increase in accuracy does not compensate for the additional strain while computing. Figure 3: CS-LBP calculated gray-level difference of symmetrically opposite pixels [19] differences between pairs of pixels symmetrically opposite with respect to the center are calculated. CS-LBP operator is defined as CS LBP, = s(g g )2 (10) s(x) = 1, if x 0 0, otherwise As evident from the Fig. 3, in case of a neighborhood of 8 pixels, the number of comparisons has reduced producing a 4 bit binary number. The numbers of output labels have reduced from 256 to 2 4 =16. CS-LBP, thus helps in dimensionality reduction by producing shorter histograms.[19]cs- LBP descriptor unites the merits of SIFT descriptor and LBP. In comparison to SIFT descriptor, the 2.9. Opponent Color LBP Although the primary LBP operator was developed exclusively for monochrome images, the concept has been extended for color images as well. A joint color-texture operator was needed to incorporate both gray scale and color texture features [22] and OCLBP was defined. In OCLBP, the operator is applied on each individual color channel followed by pairs of color channels. Of the six pairs of color-channels, only three are taken into account. R-G, R-B and G-B for instance. Opposing pairs such as G-B and B-G are highly redundant, so any one of them can be considered in the computation. While computing the pattern for a pair, the center pixel is taken from a channel and the corresponding neighborhood pixels from the other. For each block, a total of six distinct histograms are computed and concatenated. The final feature histogram obtained by concatenating the block histograms is six times longer than the basic LBP version. 19

Figure 4: Opponent color LBP for a red center. The three planes illustrate color channels.[22] OCLBP has achieved commendable results when compared to existing color-texture operators. Experimentally, it has been concluded that all joint color texture descriptors and all methods of combining color and texture on a higher level are outperformed by either color or gray-scale texture alone [22]. Segregating color and texture and handling them individually is still a popular approach in recent studies. complementary sections: CLBP Sign (CLBP_S) encodes the difference in signs and CLBP Magnitude (CLBP_M) the difference in magnitudes. Sign component preserves most of the information in local structure which is why simple LBP has fared well in representing local features of an image. If utilized well, significant discriminative information missing in original LBP code can be represented by the magnitude component. The center pixel s intensity value could provide useful information. It has been observed that the inclusion of all the three components has produced better texture classification results than the basic design. 2.10. Transition Coded LBP The LBP method thresholds gray values of the neighborhood pixels against the center [23].LBP label, thus gives an idea about difference in intensity values between the center and the neighboring pixels. Figure 5: Neighbor pixel comparisons in clockwise direction for all pixels except the central.[23] However if two adjacent pixels have the same intensity value, LBP codes may or may not differ. In a particular neighborhood, Transition coded-local binary pattern (tlbp) compares neighbor pixels in a clockwise fashion for all pixels except the center. tlbp establishes the relationship between neighbor pixels. tlbp, = s(g g ) + s g g 2 (11) tlbp was employed in gender recognition, face detection and car detection experiments as part of extended MB-LBP set offering better accuracy and speed. 2.11. Completed LBP The CLBP represents a local region by its center pixel and local difference sign-magnitude transform (LDSMT)[24].The center pixel is globally thresholded and labeled by a binary code. This binary map forms CLBP Center (CLBP_C).The LDSMT segregates the image local structure into two Figure 6[24]: (a) 3x3 sample block, (b) the local differences, (c) the sign and (d) magnitude components The difference between center pixel g c and its neighbors g p (p=0,1, p- 1 ) is represented by d p. d p = g p - g c d p can be further decomposed into two components: d = s m and s = sign(d ) (12) m = d s = 1, if d 0 1, if d < 0, where s p is the sign of d p and m p is the magnitude of d p. CLBP_M is defined along similar lines as CLBP_S CLBP, = t m, c 2 (13) t(x, c) = 1 x c 0 x < c The value of the threshold c is set as the mean value of m from the complete image. To make CLBP_C compatible with CLBP_M and CLBP_S, the operator is defined as CLBP_C, = t(g, c ) (14) t is defined in eq (13). c is the mean gray level of the complete image. III. IMPLEMENTATION AND RESULTS LBP and discussed extensions have been implemented as face descriptors. Regional descriptors are built first and then combined to give a global descriptor. The platform utilized is MATLAB 7.12.0(R2011a).The images used in Fig 9, Fig 10,Fig 20

12, Fig 13 and Fig 15 have been taken from the Extended Yale Face Database B [27],[28]. The images have been then resized to 256x256 pixels. Except for simple LBP which is computed on fixed 3x3 neighborhood, the remaining can operate on variable circular neighborhoods. The image boundary is padded with zeros for LBP calculation of border pixels. All the other patterns been implemented as per the Ahonen et al. methodology of block division[6]as shown in Fig. 7 and Fig. 8.The entire image is divided into blocks and LBP is calculated block-wise. The LBP for border pixels of a particular block is computed from adjoining blocks. For example, the image is split into blocks of size 64x64 pixels. Firstly, the boundary of input image is padded with zeros of width r (where r is the radius of the circular neighborhood). Size of block from which LBP is calculated is (64+2r)x(64+2r). Histograms computed from each block are optimally normalized and then concatenated. For variable circular neighborhoods, pixel values have been bilinearly interpolated if the sampling point is not in the center of the pixel. For a generic neighborhood, predefined values of radius and number of sampling points have been set as 1 and 4 respectively. Thresholding has been implemented only in the case of generic LBP and CS-LBP. Implementation has been designed for both monochrome as well as color images but accepts input images of size 256x256 exclusively. Mapping type of the LBP codes could be rotation invariant, uniform patterns, or both rotation invariant with uniform patterns. The images shown in Fig. 12, Fig. 14 and Fig. 15 are obtained when number of sampling points is 8 and radius of the circular neighborhood is 3. Evidently, larger size of the block lowers the computation time. The length of the feature histogram is diminished. The effect of varying radius and number of sampling points in case of generic LBP has been shown in Table 1. Increasing the number of sampling points increases the number of distinct labels. More patterns and consequently more information can be encoded. On the downside, huge length of the feature vector consumes fair amount of computation time. As the value of the radius increases, the LBP image becomes smoother. The larger details are captured better. Parallely, the number of sampling points should also be increased. Otherwise, operator s discriminative power is compromised. Although mapping increases the time of processing because the data needs to be referred, it systematically reduces the size of the feature vector while conserving discriminative information. 21

Table 1: Effect of varying radius and number of sampling points in generic LBP Figure 12: LBP images with(a) Test Image[27],[28], (b) Generic LBP, (c) Center-Symmetric LBP, (d) transition coded LBP, (e) Completed LBP components :CLBP_S (left) CLBP_C (center) and CLBP_M (right) Figure 13: Multi-Scale LBP (a)test image[27][28], (b) P:8 R:2, (c) P:4 R:3(d) P:8 R:4 22

Figure 14: Opponent Color LBP (a)test image[30], (b)center: Red Nhood: Red, (c)center: Green Nhood: Green, (d)center: Blue Nhood: Blue, (e)center: Green Nhood: Red, (f)center: Blue Nhood: Red, (g)center: Blue Nhood: Green Figure 15: (a) Test Image [27],[28], (b) a=0, (c) a=3, (d) a=5 where a is the parameter in thresholding section in generic LBP CONCLUSION Due to its merits, Local Binary Pattern has developed into a popular and powerful operator for image description. Depending on the application, numerous variations and extensions have been proposed. MSLBP and CLBP outperform LBP in terms of accuracy. Inclusion of a parameter in the thresholding section makes the operator robust. OCLBP is a joint color-texture descriptor and so on. In this paper, LBP and some of its extensions in spatial domain have been surveyed and implemented as face descriptors. Different mapping of the LBP codes like rotation invariance, uniform patterns have been incorporated into the discussed extensions and their effect on the LBP image and the feature histogram was studied. To find a suitable method for a given application, various factors come into picture like computational efficiency, robustness to noise and other variations, discriminative power, etc. Therefore, the LBP operators presented in this paper provide a good starting point to determine an appropriate extension for a particular application. ACKNOWLEDGEMENT I would like to express my gratitude to Central Electronics Engineering Research Institute (CEERI), Chennai during the course of which I could work and gather enough material to write this paper. I would also like to thank Practice School Division- BITS Pilani for making it possible through Practice School-1 programme. REFERENCES [1] Ojala, Timo, Matti Pietikäinen, and David Harwood. "A comparative study of texture measures with classification based on featured distributions." Pattern recognition,vol 29, no. 1,pp. 51-59,1996. [2] DC. He and L. Wang, "Texture Unit, Texture Spectrum, And Texture Analysis", Geoscience and Remote Sensing, IEEE Transactions on, vol. 28, pp. 509 512,1990. [3] L. Wang and DC. He, "Texture Classification Using Texture Spectrum", Pattern Recognition, Vol. 23, No. 8, pp. 905 910,1990. [4] Mäenpää, Topi, and Matti Pietikäinen. "Texture analysis with local binary patterns." Handbook of Pattern Recognition and Computer Vision 3,pp. 197-216,2005. [5] Ojala, Timo, and Matti Pietikäinen. "Unsupervised texture segmentation using feature distributions." Pattern Recognition,vol 32, no. 3,pp. 477-486,1999. [6] Ahonen, Timo, Abdenour Hadid, and Matti Pietikainen. "Face description with local binary patterns: Application to face recognition." Pattern Analysis and Machine Intelligence, IEEE Transactions,vol 28, no. 12,pp. 2037-2041,2006. [7] Ahonen, Timo, Abdenour Hadid, and Matti Pietikäinen. "Face recognition with local binary patterns." In Computer vision-eccv 2004, pp. 469-481. Springer Berlin Heidelberg, 2004. [8] Wright, J.,Yang, A.Y., Ganesh, A., Sastry, S.S., and Yi Ma "Robust Face Recognition via Sparse Representation", Pattern Analysis and Machine Intelligence, IEEE Transactions, Vol 31, no. 2,pp. 210 227, Feb. 2009. [9] Zhang, Wenchao, Shiguang Shan, Wen Gao, Xilin Chen, and Hongming Zhang. "Local gabor binary pattern histogram sequence (lgbphs): A novel non-statistical model for face representation and recognition." In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, vol. 1, pp. 786-791, 2005. [10] Shan, Caifeng, Shaogang Gong, and Peter W. McOwan. "Robust facial expression recognition using local binary 23

patterns." In Image Processing, 2005. ICIP 2005. IEEE International Conference on, vol. 2, pp. II-370. IEEE, 2005. [11] Shan, Caifeng, Shaogang Gong, and Peter W. McOwan. "Facial expression recognition based on local binary patterns: A comprehensive study." Image and Vision Computing,vol 27, no. 6,pp. 803-816,2009. [12] Li, Stan Z., Senior RuFeng Chu, ShengCai Liao, and Lun Zhang. "Illumination invariant face recognition using nearinfrared images." Pattern Analysis and Machine Intelligence, IEEE Transactions,vol 29, no. 4,pp. 627-639,2007 [13] Guoying Zhao; Pietikainen, M. "Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions", Pattern Analysis and Machine Intelligence, IEEE Transactions, Vol 29, no. 6, pp. 915-928,June 2007 [14] Nanni, Loris, Alessandra Lumini, and Sheryl Brahnam. "Local binary patterns variants as texture descriptors for medical image analysis." Artificial intelligence in medicine, vol 49, no. 2, pp. 117-125,2010 [15] Sørensen, Lauge, Saher B. Shaker, and Marleen De Bruijne. "Quantitative analysis of pulmonary emphysema using local binary patterns." Medical Imaging, IEEE Transactions,vol 29, no. 2,pp. 559-569,2010 [16] Heikkilä, Marko, and Matti Pietikäinen. "A texture-based method for modeling the background and detecting moving objects." Pattern Analysis and Machine Intelligence, IEEE Transactions,vol 28, no. 4 pp. 657-662,2006 [17] Ojala, T., Pietikäinen, M., Mäenpää, T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intelligence, IEEE Transactions,vol 24, no. 7,pp. 971-987,2002 [18] Ahonen, Timo, Jiří Matas, Chu He, and Matti Pietikäinen. "Rotation invariant image description with local binary pattern histogram fourier features." In Image Analysis, pp. 61-70. Springer Berlin Heidelberg, 2009. [19] Heikkilä, Marko, Matti Pietikäinen, and Cordelia Schmid. "Description of interest regions with local binary patterns." Pattern recognition,vol 42, no. 3, pp.425-436,2009 [20] Mäenpää, Topi. The Local binary pattern approach to texture analysis: Extensions and applications.phd thesis, Acta Universitatis Ouluensis C 187, University of Oulu,2003. [21] Chan, Chi-Ho, Josef Kittler, and Kieron Messer. Multi-scale local binary pattern histograms for face recognition. Springer Berlin Heidelberg, 2007. [22] Mäenpää, Topi, and Matti Pietikäinen. "Classification with color and texture: jointly or separately?" Pattern recognition Vol 37, no. 8,pp. 1629-1640,2004. [23] Trefný, Jirí, and Jirí Matas. "Extended set of local binary patterns for rapid object detection." Proceedings of the Computer Vision Winter Workshop. Vol. 2010,2010. [24] Guo, Zhenhua, Lei Zhang, and David Zhang. "A completed [25] modeling of local binary pattern operator for texture classification." Image Processing, IEEE Transactions on Vol 19, no. 6, pp. 1657-1663,2010. [26] Pietikäinen, M., Hadid, A., Zhao, G., Ahonen, T., Computer Vision Using Local Binary Patterns,Springer-Verlag London, 2011 [27] Lindahl, Tobias. "Study of local binary patterns.",2007. [28] Lee, Kuang-Chih, Jeffrey Ho, and David J. Kriegman. "Acquiring linear subspaces for face recognition under variable lighting." Pattern Analysis and Machine Intelligence, IEEE Transactions on 27, no. 5, pp. 684-698,2005 [29] Georghiades, Athinodoros S., Peter N. Belhumeur, and David J. Kriegman. "From few to many: Illumination cone models for face recognition under variable lighting and pose." Pattern Analysis and Machine Intelligence, IEEE Transactions on 23, no. 6, pp. 643-660,2001 [30] Test image for the Table 1, pixabay.com, Available from: https://pixabay.com/en/child-boy-kid-people-young-hat- 83814/ [31] Opponent Color LBP (a)test image, pixabay.com, Available from: https://pixabay.com/en/snow-colors-wintergirl-merry-713463/ [32] Huang, Di, Caifeng Shan, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. "Local binary patterns and its application to facial image analysis: a survey." Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions,Vol 41, no. 6, pp.765-781,2011. [33] Marcel, Sébastien, Yann Rodriguez, and Guillaume Heusch, On the recent use of local binary patterns for face authentication., No. LIDIAP-REPORT-2006-037. IDIAP, 2006. 24