Performance of Histogram-Based Skin Colour Segmentation for Arms Detection in Human Motion Analysis Application

Transcription

1 World Aademy of Siene, Engineering and Tehnology Performane of Histogram-Based Skin Colour Segmentation for Arms Detetion in Human Motion Analysis Appliation Rosalyn R. Porle, Ali Chekima, Farrah Wong, and G. Sainarayanan Abstrat Arms detetion is one of the fundamental problems in human motion analysis appliation. The arms are onsidered as the most hallenging body part to be deteted sine its pose and speed varies in image sequenes. Moreover, the arms are usually oluded with other body parts suh as the head and torso. In this paper, histogram-based skin olour segmentation is proposed to detet the arms in image sequenes. Six different olour spaes namely RGB, rgb, HSI,, SCT and CIELAB are evaluated to determine the best olour spae for this segmentation proedure. The evaluation is divided into three ategories, whih are single olour omponent, olour without luminane and olour with luminane. The performane is measured using True Positive (TP) and True Negative (TN) on 50 images with manual ground truth. The best olour is seleted based on the highest TN value followed by the highest TP value. Keywords image olour analysis, image motion analysis, skin, wavelet transform. I. INTRODUCTION ETECTING arms for human motion analysis appliation D is a hallenging task as the arms tends to move farther, faster and more often than the other body parts. Silhouette and edge features may not be suffiient and effetive for arms detetion espeially when the arms were oluded in the head or torso regions. Colour is a useful feature for arms detetion. It provides omputationally effetive yet, robust information against rotation, saling and partial olusions [1]. Performing skin olour segmentation for arms detetion in aount of all variation suh as rae fators, illumination and blurriness due to fast movement is diffiult and time onsuming. Several issues have to be onsidered. This inludes the hoies of olour spae to be implemented and how the skin olour R. R. Porle is with Computer Engineering Program, Shool of Engineering and Information Tehnology, Universiti Malaysia Sabah, Kota Kinabalu, Sabah, Malaysia (phone: ext 3083; fax: ; rosalynrporle@yahoo.om.my). A. Chekima, is with Computer Engineering Program, Shool of Engineering and Information Tehnology, Universiti Malaysia Sabah, Kota Kinabalu, Sabah, Malaysia ( hekima@ums.edu.my). F. Wong is with the Eletrial and Eletronis Engineering Program, Shool of Engineering and Information Tehnology, Universiti Malaysia Sabah, Kota Kinabalu, Sabah, Malaysia ( farrah@ums.edu.my). G. Sainarayanan is with the ICT Aademy of Tamil Nadu, ELCOT Complex, -7 Developed Plots, Industrial Estate, Perungudi, , Chennai, India ( sainarayanan@itat.org). distribution should be modelled. In human motion analysis, the skin olour template an be onstruted speifi to the human subjet onsidered in the system []. Several researhers in human motion analysis field apply olour for performing arms or legs detetion. In [3], the normalized RGB is used to segment the lower arms and upper-leg regions. A skin olour model is onstruted from the pixels of skin olour luster obtained from the k-means algorithm within the deteted fae region. The skin region is segmented using the learned olour histogram. In [], the mean and standard deviation of RGB omponent in skin region are evaluated to segment the skin and non-skin regions. Thresholding method is used for the segmentation. In [4], the bivariate Gaussian (Hue and Saturation) is onstruted from skin samples taken from natural images. A refined skin model is reated based on a new bivariate that is learned from the inlier in region H. In this paper, histogram-based skin olour segmentation is proposed for arms detetion. A skin olour model is onstruted from the skin olour pixels that are extrated within the fae region. Histogram analysis is then performed to the skin olour model to obtain two threshold values. These threshold values are used to segment the image. Lastly, post proessing is performed to eliminate other regions in the image exept the arm regions. To selet the best olour spae for the histogram-based skin olour segmentation, six olour spaes are evaluated. These olour spaes are RGB, rgb, HSI,, SCT and CIELAB. The performane evaluation is performed in three ategories, whih are single olour omponent, olour without luminane and olour with luminane. True Positive (TP) and True Negative (TN) are used as measurement metris. The olour that provides the highest TN followed by TP value is seleted as the best olour for the proposed segmentation method. This paper is organized into six setions. The first setion is the introdution. The seond setion desribes the olour spaes that are used in this paper. The histogram-based skin olour segmentation proedure is explained in the third setion. Later, in the fourth setion, the post proessing proedure is desribed. Experimental results and disussion are inluded in setion five. Finally, overall researh is onluded in setion six. 169

2 World Aademy of Siene, Engineering and Tehnology II. COLOUR SPACES FOR SKIN COLOUR SEGMENTATION The seletion of olour spae is ruial as it influenes the performane of the segmentation methodology. There are many olour spaes available. In this paper, instead of randomly hoosing a olour spae, six olour spaes; RGB, rgb, HSI,, SCT and CIELAB are examined. The desription of eah the olour spae is presented as the following: A. RGB RGB olour spae is onsidered as a basi olour spae. It orresponds to the three primary olours whih are the red, green and blue omponents. It is the most ommonly used olour spae for digital image storing and representation. B. rgb The rgb olour spae is the normalized RGB, whih is defined as ( R + G B) ( R + G B) r rgb = R / + (1) g rgb = G / + () b rgb = B /( R + G + B). (3) The b rgb omponent is usually omitted in the skin segmentation as it does not hold any signifiant information. One of the advantages of this olour spae is that it is invariant to lighting variation [5]. C. HSI HSI is a olour spae that desribes olours as pereived by human beings [6]. HSI represents the Hue, Saturation and Intensity respetively. Hue defines the dominant olour suh as red, green, purple and yellow of an area. Meanwhile, the saturation measures the olourfulness or whiteness in the pereived olour. Lastly, the intensity provides a measure of the brightness of the olour. The HSI omponents an be expressed by 1 H HSI = os 1 [( R G ) + ( R B) ] ( R G ) + ( R B)( G B) [ 3( min ( R, G, B) ) ( R + G B) ] ( R + G B) / 3 S HSI = 1 / + (5) I HSI = + (6) D. (Tint, Saturation and Luminane) is a pereptual olour that is similar to HSI olour spae [7]. T desribes the olour, S is the saturation and L is the luminane (for Gamma orreted RGB values). This is given as (4) T 1 1 r tan π g 1 1 r = tan π g 0; g = ; ; 4 ( r g ) g g < 0 > 0 S = 9 / 5 + (8) L = 0.99 R G B (9) where r = r rgb 1 / 3 (10) g g 1 / 3 (11) = rgb E. SCT The Spherial Coordinate Transform (SCT) olour omponents are defined in (1) to (14). L SCT is the intensity; A SCT and B SCT represent olours independently from intensity. If a partiular olour is plotted as a point in RGB spae, then the norm of the vetor from the origin to the point is L SCT, the angle between the vetor and the blue axis is A SCT, and the angle between the red axis and the projetion of the vetor onto the RG plane is B SCT [8]. L SCT + A / B = R + G B (1) SCT = os ( B L SCT ) (13) SCT = tan 1 ( G /( L SCT ( sin ASCT ))) (14) F. CIELAB The last olour spae is CIELAB, whih was adopted as an international standard in the 1970s, by CIE (International Commission on Illumination). The first olour omponent, L CIELAB, is the lightness of olour. The A CIELAB omponent represents its position between red/magenta and green. Meanwhile the B CIELAB omponent represents its position between yellow and blue. 3 ( Y / Yn) 1 / 16 ( Y / Yn ) ; Y ; [ f ( X / Xn ) ( Y Yn) ] [ f ( Y / Yn) ( Z Zn )] 116 / Yn > L = (15) CIELAB otherwise A CIELAB = 500 / (16) B CIELAB = 00 / (17) where X = Y Z R G B (7) (18) 170

3 World Aademy of Siene, Engineering and Tehnology and 1 / 3 t ; t > f () t = (19) t + ( 16 /116 ) ; otherwise III. HISTOGRAM-BASED SKIN COLOUR SEGMENTATION The flowhart of the histogram-based skin olour segmentation is illustrated in Fig. 1. Firstly, the olour spae of input image is onverted. Approximation image, whih is extrated using Haar wavelet transform at level one [9], is used as input image. The wavelet transformation shrinks the images and at the same time smoothes the texture of the image. This an redue the effet of lightness variations in the images. The input images are in RGB olour format. The desription of the desired olour was given in previous setion. Then, the onverted olour image is separated into three olour image omponents. For example the rgb olour image is separated into r rgb, g rgb and b rgb olour image omponent respetively. The skin olour sample that is speifi to the user in the image an be obtained from the head region. The loation of the head region is estimated using pose estimation tehnique. One the loation is determined, the olour information within the head region is extrated. The olour information within the head region ontains skin and non-skin olours. The non-skin olour is usually originated from the hair, eyes and the lips regions of the user. Extration of the non-skin olour is time onsuming. Moreover, it is diffiult to distinguish the non-skin olour regions when the head is relatively small in the image. Due to these reasons, the non-skin olour in the skin olour sample an be ignored. The extrated skin olour sample is analyzed using histogram distribution. From this histogram, the skin olour range for the user has to be determined. It is noted that eah of the olour image omponents provides different range of histogram distribution. This is due to the small olour range in the skin olour sample. Therefore, eah bin in the histogram is set to represent only one olour value. The ourrene in eah bin is referred as frequeny, f. Two threshold values are used to determine the skin olour range. The low threshold, T l, is the minimum olour value in the skin olour range. Meanwhile, the high threshold, T h, is the maximum olour value in the skin olour range. Minimum frequeny, f min, is used to alulate these threshold values. Let f max be the maximum frequeny and P f is the frequeny perentage. The range of P f is set between 0 and 1. The minimum frequeny is alulated as f = f P (0) min max f The skin olour range is assumed to be in the range where the frequeny is more than the minimum frequeny. If the histogram bin is denoted as H b, the low threshold value is alulated as End Fig. 1 Flowhart of the histogram-based skin olour segmentation l [ H ( f f )] T = min > (1) h b min Meanwhile, the high threshold value is alulated as [ H ( f f )] T = max > () b Start Input Image Colour Spae Conversion Skin Colour Sample Histogram Analysis Skin Colour Segmentation Arms Detetion min Head Region Loation After the threshold values are obtained, the olour image omponents an be segmented. The olour pixel in the olour image omponent is denoted as I. The olour pixel is deteted as skin olour if its value lies between the low threshold and the high threshold values. If the olour pixel value is outside of this range, it is deteted as non-skin olour. The olour pixel value is then set to 1 if it a skin olour. Otherwise, if it is a non-skin olour, the value is set to 0. This an be expressed as 1 if Tl I Th I = (3) 0 else After the olour image omponents are segmented, logial operator is used to ombine these images. Two logial operators are suggested for the ombination. The first logial operator is OR operator. For OR operator, the olour pixel of the ombined image is assigned to 1 if any of the olour pixels, at the same loation in the olour image omponents are deteted as skin olour. Otherwise, the olour pixel of the ombined image is assigned to 0 if none of the olour pixels, at the same loation in the olour image omponents are deteted as non-skin olour. The seond logial operator is 171

4 World Aademy of Siene, Engineering and Tehnology AND operator. For AND operator, the olour pixel of the ombined image is assigned to 1 if all the olour pixels, at the same loation in the olour image omponents are deteted as skin olour. Otherwise, the olour pixel of the ombined image is assigned to 0 if any of the olour pixels, at the same loation in the olour image omponents are deteted as nonskin olour. The best logial operator for the olour image omponents ombination will be determined in setion IV. IV. EXPERIMENTAL RESULTS AND DISCUSSION Six olour spaes are evaluated in this paper. They are RGB, rgb, HSI,, SCT and CIELAB. The performanes of these olour spaes are measured using True Positive (TP) and True Negative (TN). In the skin olour segmentation ontext, True Positive is defined as the number of orretly deteted skin olour pixels divided by the total number of skin olour pixels. High TP value means that the olour has high skin olour detetion performane. On the other hand, low TP value means that the olour has low performane in skin olour detetion. The True Negative is defined as the number of orretly deteted non-skin olour pixels divided by the total number of non-skin olour pixels. High TN value means that the olour has high performane in eliminating non-skin olour pixels. On the other hand, low TN value means that the olour has poor performane in eliminating non-skin olour pixels. The performane measurement is performed in three ategories. The first ategory is to evaluate the performanes of eah olour omponent. There are 18 olour omponents to be analyzed. The aim of this evaluation is to determine whether a single olour omponent an be used to segment the arms in the image. Another purpose is that the possibility of a single olour omponent to be ombined with other olour omponent of different olour spae. This ombination may be performed if the three ategories of the performane measurement did not produe satisfatory or higher performane. The seond ategory is to evaluate the olour spae without the luminane or intensity olour omponent. Many researhers suggested that omitting the luminane/intensity omponent from the olour spae will improve the segmentation performane [10]. This possibility is also investigated in this paper. In this ase, ten olour ombinations are evaluated. Five olour ombinations used OR operator and the other five olour ombinations used AND operator. The last ategory is to evaluate the olour spae with luminane or intensity olour omponent. For this ategory, 1 olour omponents ombinations are evaluated. The experimental database onsists of 150 images. These images were extrated from 30 reorded videos in indoor environment with different user for eah reorded video. For the evaluation purpose, most of the parameters are fixed. It is assumed that only P f in (0) is adjustable. Ten different ratio values were seleted (i.e. 0.1, 0., 0.3,, 1) and from eah ratio value, TP and TN are then alulated. Ground truth for test images were onstruted manually. From these three ategories, the best olour omponent or ombination that produes the highest TN is seleted as the best olour for the segmentation proedure. The results of the evaluation are presented as the following: A. Single Colour Component The first experiment evaluates the performane of 18 single olour omponents. Table I shows the performane result of 18 olour omponents. The best TN for eah olour omponent is presented in the third olumn. The orresponding TP and P f values are presented in the fourth and fifth olumn respetively. Three olour omponents have higher TN value, whih are above 80 perent. They are H HSI, T and A CIELAB. By referring to the desription of these olour omponents, they represent the olours (hue, tint and red/magenta and green) in the olour spae respetively. The A CIELAB olour omponent partiularly has the highest TN value ompared to the other olour omponents. This is beause A CIELAB only detet a speified olour range (red/magenta and green). This may not overs the olour range available in the bakground. Other olour omponents suh as R, G, B, r rgb and g rgb provide moderate performane. In this ase TN value is around 70 to 80 perent. Meanwhile, the olour omponents suh as b rgb, I HSI, L, L SCT, A SCT and B SCT have performane that is around 60 to 70 perent of TN value. The remaining olour omponents, whih are S HSI, S, L CIELAB and B CIELAB, are the olour omponents that provide lowest TN performane. B. Colour without Luminane Component The seond experiment evaluates the performane of ten olour ombination without luminane or intensity omponent. In more detail; the rgb without b rgb omponent, HSI without I HSI omponent, without L omponent, No. TABLE I THE PERFORMANCE OF SINGLE COLOUR COMPONENT Colour Component True Negative True Positive 1 R G B r rgb g rgb b rgb H HSI S HSI I HSI T S L L SCT A SCT B SCT L CIELAB A CIELAB B CIELAB Pf 17

5 World Aademy of Siene, Engineering and Tehnology SCT without L SCT omponent and CIELAB without L CIELAB omponent. Five olour ombination used OR operator and the other five olour ombination used AND operator. The result of olour ombination without luminane omponent using OR and AND operators is shown in Table II. For OR operator, TN value for all ombination is around 40 to 60 perent. Meanwhile, their respetive TP values are around 50 to 70 perent. CIELAB reorded the highest performane with TN value of perent for the OR operator. The seond and third highest performanes for the OR operator are HSI and SCT, with TN value of and perent respetively. For the AND operator, the TN value for all olour ombination is more than 80 perent exept SCT olour with perent. Their orresponding TP values are around 50 perent. CIELAB has the highest TN with perent for the AND operator. This is followed by rgb, HSI and also when applying the OR operator. By omparing between the OR and AND operators, it is onluded that the AND operator performs better than the OR operator. Among all olour ombination, CIELAB with AND operator ahieved the highest performane. C. Colour Spae The third experiment evaluates the performane of 1 olour spaes. Six olour ombination used OR operator and the other six olour ombination used AND operator. The result of olour spae performane using OR and AND operators is shown in Table III. For OR operator, RGB olour spae has the highest performane with TN value of perent. This is followed by HSI, SCT, rgb, CIELAB and olour spaes. All olour spaes exept rgb have TP performanes around 50 perent. The rgb olour spae has the highest TP, whih is around perent. For the AND operator, the best olour spae for this ombination is CIELAB with TN and TP values of and perent respetively. The other olour spaes have TN values around 70 perent. By omparing these two operators, the AND operator gives better performane than OR operator. Among the olour spaes, CIELAB ahieved highest performane in this ategory. D. Overall result The single olour omponent, olour spae without luminane and olour spae performanes are ompared to TABLE II THE PERFORMANCE OF COLOUR COMBINATION WITHOUT LUMINANCE COMPONENT Colour Operator True True No. Pf Combination Negative Positive 1 rgb OR HSI OR OR SCT OR CIELAB OR rgb AND HSI AND AND SCT AND CIELAB AND TABLE III THE PERFORMANCE OF COLOUR COMBINATION WITH LUMINANCE COMPONENT Colour Operator True True No. Pf Component Negative Positive 1 RGB OR rgb OR HSI OR OR SCT OR CIELAB OR RGB AND rgb AND HSI AND AND SCT AND CIELAB AND determine the best result for the histogram based skin olour segmentation. In all three ategories CIELAB olour spae has better performanes than the other olour spaes. The CIELAB without luminane omponent using AND operator has the highest performane in eliminating non-skin pixels, whih reahed about perent. This attribute is desirable in human motion analysis appliation as the olusion and motion blur usually ours during the video apture. The performane in deteting the skin pixel is moderate, whih is around perent. This is expeted as the skin olour information in human s head region may not over all skin olour range in the arms. The illumination hanges and motion blur are also some of the fators that the olour in the arm region is slightly differ from the skin olour in the head region. In motion analysis appliation, the deformable arm regions an be solved by estimating or modelling the arms using robust tehnique. V. CONCLUSION Arms detetion for human motion analysis appliation an be performed by evaluating the skin olour information in human s head region. The designed histogram-based skin olour segmentation omputed the maximum olour frequeny and two threshold values were derived based on the evaluated perentage of the maximum olour frequeny. From the six olour spaes proposed for this algorithm, the CIELAB without luminane omponent using AND operator has ahieved the highest TN value, thus, this olour is seleted as the best olour to be used in the segmentation proedure. REFERENCES [1] P. Kakumanu, S. Makrogiannis and N. Bourbakis, A survey of skinolor modelling and detetion methods, Pattern Reognition, vol. 40, no. 3, pp , Marh, 007. [] J. Mulligan, Upper Body Pose Estimation from Stereo and Hand-Fae Traking, in Pro. nd Canadian Conf. on Computer and Robot Vision, 005, pp [3] G. Hua, M. H. Yang and Y. Wu, Learning to estimate human pose with data driven belief propagation, in Pro. of the IEEE Computer Soiety Conf. on Computer Vision and Pattern Reognition, 005, Vol., pp [4] A. S. Miilotta and R. Bowden, View-based loation and traking of body parts for visual interation, in Pro. of British Mahine Vision Conferene. 004, vol., pp

6 World Aademy of Siene, Engineering and Tehnology [5] S. Mallat, Wavelets for a vision, Pro. of the IEEE, 1996, vol. 84 (4), pp [6] T. Aharya, and A. Ray, Image Proessing: Priniples and Appliation. John Willey & Sons, 005. [7] V. Vezhnevets, V. Sazonov and A. Andreeva, A Survey on Pixel- Based Skin Color Detetion Tehniques, in Pro. of GRAPHICON, Mosow, Russia, 003, pp [8] S. E. Umbaugh, Computer imaging: digital image analysis and proessing. CRC Press, 005. [9] J. -C. Terrillon, M. N. Shirazi, H. Fukamahi and S. Akamatsu, Comparative performane of different skin hrominane models andhrominane spaes for the automati detetion of human faes in olor images, in Pro. of 4th IEEE International Conferene on Automati Fae and Gesture Reognition, Grenoble, Frane, 000, pp [10] S. Jayaram, S. Shmugge, M. C. Shin and L. V. Tsap, Effet of olorspae transformation, the illuminane omponent, and olor modeling on skin detetion, in Pro. of the 004 IEEE Computer Soiety Conferene on Computer Vision and Pattern Reognition, 004, vol., pp