Face and Facial Feature Tracking for Natural Human-Computer Interface

Transcription

1 Fae and Faial Feature Traking for Natural Human-Computer Interfae Vladimir Vezhnevets Graphis & Media Laboratory, Dept. of Applied Mathematis and Computer Siene of Mosow State University Mosow, Russia Abstrat A method for fae and faial features traking in a low-resolution web amera video stream is desribed. The detetion results are: fae bounding ellipse, eyebrow line, nostrils and mouth position. The method works at reasonable framerates on a low-quality web amera on a PIII system. The implementation of the method an be used as input module for a natural human-omputer interfae system to provide amera-based mouse ontrol. The tehniques used are: fae traking using olo image edge maps analysis, variant of Hough transform, template mathing, olor-based image segmentation. Keywords: Fae detetion, Fae traking, Faial Features traking. 1. INTRODUCTION Human fae detetion and traking is a neessary step in many fae analysis tasks - like fae reognition, natural HCI systems, model-based video oding, and ontent-aware video ompression. Although these problems are fairly easy for a human vision system, the mahine vision labels them as hard. This paper desribes a method for fae and faial features traking, whih is designed as an input module for natural Human-Computer Interfae system. This task originated from a neessity of reation of means for ontrolling omputer for the hildren unable to use onventional HCI means (for example, suffering from erebral pals. This have set the onditions and limitations for the software it should funtion on an inexpensive home ompute in parallel with the regular mouse devie and should be able to work with image aquired by a heap web amera. Despite latest advanes in the field of fae and faial feature traking (see for example [1-3], most proposed methods still give aeptable results only in limited set of onditions. The reason for this is mostly the high variability of the input data. Unfortunately, many issues in the problem of robust fae and faial features traking an be treated as unsolved in general ase. The author hopes, that methods and algorithms, desribed in this papers will ontribute to the solution of the task of automati analysis of human fae. 2. FACE TRACKING BASED ON COLOR INFORMATION Color is a distintive feature of human fae, making olor information useful for loalization of a human fae on stati and video images. Color information allows fast proessing, whih is important for a traking system that needs to run at reasonable framerates (at least 10 fps. Nevertheless olor alone does not provide enough reliable information to detet and trak a fae, due to noisiness of olor information and possible presene skin-olored non-fae objets in the images. Some additional methods are needed to analyze the results of olor segmentation. Often, onneted omponents analysis or integral projets of skin likelihood images are used for deteting and traking of fae andidates [1-4]. These simple methods do not take advantage of expeted shape and size of the traked fae and are prone to errors in ase of non-ideal olor segmentation results. The algorithm desribed here uses skinolored pixel grouping method desribed in [5] to detet fae andidate position in proessed frame. This method shows robust result even when a person is positioned in front of the skinolored bakground. This method of fae loalization was adopted for fae-traking purposes. The algorithm for fae traking ontains those steps: 1. Initialization of the fae ellipse; 2. Automati training of the skin olor filter; 3. Traking, using the reated olor filter and initialized fae position; During the initialization step the user sets the fae ellipse position and size. Using this information, the image is divided into two areas fae and non-fae and is used to train olor-based skin detetor. The Maximum Likelihood Bayes lassifier in normalized r-g hrominane olorspae is used for skin olor modeling. The r-g olorspae was hosen beause of its fast and simple onversion from the RGB spae, whih is signifiant for a realtime fae traking appliation, and due to mentioned results of skin olor modeling with Bayes ML lassifier [6] (the hanging of olorspaes CIELab, YCrCb, HSV, r-g made very little impat on skin detetion orretness.

2 Figure 1: Image used for fae traker initialization The training is performed by gathering two statistis p (( g skin and p (( g skin, alulated from skin and non-skin histograms built from fae and non-fae areas of the image. The skin likelihood image is onstruted by alulating this measure for eah pixel of the input frame: Figure 3: Ellipti model updating Updated ellipse position and orientation is evaluated using the alulated region statistis: µ x, µ y - mass enter oordinates; 02, µ 20, µ 11 µ - seond order entral moments; Skin( g = p(( g skin p(( g skin Whih onforms to Bayes maximum likelihood riteria (assuming a priori skin and non-skin probabilities are equal. The resulting skin likelihood image (see Fig. 2 is used for fae region traking. During the moments alulation, the pixels inside the minor ellipse are weighted twie against the ones in the bigger one. The pixels in the possible nek region (see Fig. 3 are weighted by 0.5 of their real likelihood, to lessen the nek influene on the fae ellipse detetion. New ellipse enter and axis are set at: ( µ y - new ellipse µ µ 11 2 enter point, ( µ, µ µ + ( µ µ unnormalized ellipse major axis vetor. Ellipse size is not updated in the urrent version, for the user is rarely moving from or towards the amera. Usually two or three steps of model position/orientation update proedure are suffiient for aurate fae region loalization (see Fig. 4. Figure 2: Example skin likelihood images (note the skin-olored bakground Ellipti model, used for fae region detetion, is initialized near the expeted fae position in the proessed frame and then is adapted step-by-step to fit the image data. The step of the adaptation proess onsists of the onsidering the skin pixels lying inside the ellipse of a slightly larger size, and alulating the mass enter and seond order moments of the pixels formation inside this larger ellipse (see Fig. 3. Figure 4: Traked faes 3. FACIAL FEATURES TRACKING The faial features detetion and traking methods an be roughly divided into two groups: modeling features appearane with some pattern reognition method [3, 4, 8, 12], and usage of empirial rules, derived from observations of exhibited feature appearane properties [2, 9, 10, 11]. The algorithm desribed in this paper uses the latter idea. The features to trak were hosen from the observations on the reliability of eah feature traking and also on the usefulness of the features for fae orientation determination.

3 The minimal set of features inludes: eyebrows, nostrils and mouth position. enter is also restrited to min and max values, derived from the fae ellipse size (see Fig. 7. The features detetion is performed after the fae region is determined. After the fae ellipse is known, the fae image is rotated to ahieve an upright fae position, to ease the proess of faial features detetion. Several andidate positions for eah faial feature are deteted (similar to [3], then the sets of possible faial features onfigurations are tested (having in mind biometri rules of human fae struture. Eah feature s detetion is based on analysis of several attributes, unique for onrete feature. 3.1 Traking eyebrows Figure 6: Eyebrow lines parameterization Figure 7: Eyebrow lines parameters margins Usually faial feature traking systems fous on traking eye positions [2, 3, 4, 10, 11]. But if person wears eyeglasses, eye traking beomes a problem, mostly beause of eyeglasses highlights, so we have hosen eyebrows as a more frequently visible feature. The detetion of eyebrow line inside the fae ellipse is based on the following assumptions (mind that we use a low-resolution image: 1. The eyebrows usually appear as areas signifiantly darke than the forehead; 2. The forehead usually an area of smooth texture and slowly varying brightness and olor; Keeping in mind these two statements the eyebrows are found on the edge image (onstruted by Prewitt edge detetion operato using a variant of Hough transform, whih tries to find lines with lear areas right above them (see Fig. 5. The aumulator array of dimensions ρ ρmin / ρ θ θ / θ x means losest integer max min, where x max by, is used for best eyebrow line seletion. It is filled by the following algorithm: For eah pixel inside the upper ellipse part: 1. Transform it s oordinate to ellipse-aligned system; 2. If the pixel brightness is less than a threshold, take next pixel (goto A; 3. For eah of the θ in range ( π / 6, π / 6 θ alulate ρ: ρ x os( θ + y sin( θ, with step = ; 4. If ρ is in the defined margins, inrement aumulator ell, orresponding to x means losest ρ / ρ, θ /, where ( θ integer x, and taking into aount the disrete nature of aumulator array and rounding inauray, also inrement ρ / 0.5, / and ( ρ + θ θ ( ρ / ρ 0.5, θ / θ are not same with ( / ρ, θ / θ, in ase when these ells ρ ; 5. Derement aumulator ells, orresponding to ( ρ ρ / ρ, θ / ( 2 ρ / ρ, θ / ( θ ( θ and ρ to penalize the lines, whih have the bright edge pixels right above them. The neighboring ells are also deremented, if the rounding is not exat (like in D ; Figure 5: Eyebrow detetion The eyebrow lines are parameterized in the ellipse-based oordinate system - the enter is positioned in the ellipse enter and the oordinate axes are aligned with ellipse axes (see Fig. 6. The possible orientation of eyebrow line is restrited to -30 to 30 degrees from the vertial fae ellipse axis (the restrition is rather vague to make orretion in ase of poor fae ellipse orientation detetion. The distane of the eyebrow line from the ellipse This eyebrow line detetion method shows robust results, making eyebrows a stable feature to trak (see Fig. 8.

4 template is applied to the lip region, to find most probable lip position. The template s size is hosen proportional to the fae ellipse size. The templates goodness of fit is measured by: 3.2 Traking lips Figure 8: Eyebrow detetion E( x, y In( x = α Lip( β Lip( ; Out ( x where In( x - is the inner area of the template positioned at ( x and Out( x - is the border and part of template outer area (marked with light gray in the figure, and Lip is the lip funtion value in the image loation ( x,. Some methods ([3], [12] use brightness information, or pretrained olor prediate [2] for lip detetion. We have tried to use method of lip detetion, whih does not need prior alibration, but differentiates the lips from the fae, using olor information. The lip traking method is based on assumption that lips usually have harateristi olo differing from generi fae olor (similar method used in [7] and [11]; Do detet probable lip regions several steps are taken: Fae image is transformed to speial one-hannel image funtion: Lip( RGB = ( u / v (1 Skin( g 0.3 where ( u, v CIELuv olorspae, ( g - are the hrominane oordinates in Skin is the skin likelihood for the urrent olo and 0.07 and 0.3 are the empirially seleted weights; Taking into aount the noisiness of web-amera image and the low resolution of the piture, some filtering of the lip funtion image for noise elimination is performed: Figure 10: Lip template with marked In and Out regions To find most probable lips enter the positions, whih give not less than 90 % of the maximum goodness of fit are averaged. The results of lips detetion are showed in the figure. Of ourse, the assumption about the vivid lips olo unfortunately, is not always true. The ases, when this assumption fails will be addressed in the future researh. o max filter with 3x3 window; o median filter with 3x3 window; o thresholding with 0,4 threshold value; o morphologial opening with 5x5 round mask; Figure 11: Deteted lips 3.3 Traking nostrils Figure 9: Lip funtion images To find the region, orresponding to lips (it an be seen, that some spurious high lip funtion values exist - see Fig. 9, an ellipti If a amera is positioned properly, nostrils are very learly visible at most head orientation angles, giving a stable and relatively easy-deteted feature. They represent two high ontrast regions, exhibiting ertain brightness patterns (dark round spot inside a relatively bright bakground and high brightness gradient values (see Fig. 12.

5 4. CONCLUSION AND FUTURE WORK Figure 12: Nostril area on gradient and graysale images The nostrils detetion is performed by sanning the part of fae area with templates, whih try to find regions that exhibit high brightness gradient values and low brightness in the middle, while showing high brightness values at the region borders. Median filter is applied to the graysale image, prior to nostrils detetion. The most likely positions form a set of nostril andidates, whih are then tested relatively to fae position, deteted eyebrows and eah other to find the pair of the most probable nostrils positions. Figure 13: Nostril template with marked In and Out regions; The nostril template goodness of fit is measured by the formula: E( x, y = α γ In( x I In( x I( β I( + Out ( x where In and Out areas of the nostril template are shown in the figure, and I and I are the image brightness and absolute value of the brightness gradient vetor respetively. The examples of the deteted nostrils are shown in the figure: Figure 14: Deteted nostrils A fae and faial features traking method is desribed, whih works at reasonable framerates on a low-quality web amera on a PIII system. The most reliable features, as the experiene shows, are the eyebrow line and the nostrils positions. The implementation of the method an be used as a module for a natural human-omputer interfae system, for the information provided by the algorithm (fae orientation and faial features positions on eah frame an be used for amera-based mouse ontrol. The urrently used methods, although showing aeptable performane in most ases, leave a plenty of room for improvement: first of all, this onerns the limitations and assumptions made in eah ase. The inreasing of robustness and auray as well as traking other faial features is another issue. Also, as the fae is known as a highly variable struture, whih shows very different appearane from image to image, more intelligent seletion of feature andidates, analyzing and prediting the features positions in the whole may be helpful. Eah feature an be assigned a onfidene measure based upon pre-defined information (lighting onditions, presene of eyeglasses and faial hair and traking error statistis, gathered during the system funtioning. The system may use different algorithms of feature andidates detetion and verifiation, depending of these onfidene values. 5. REFERENCES [1] K. Shwerdt and J. Crowley, Robust Fae Traking using Color, Fourth IEEE International Conferene on Automati Fae and Gesture Reognition, Marh, 2000, Grenoble, Frane [2] Paul Smith, Mubarak Shah, and Niels da Vitoria Lobo, Monitoring Head/Eye Motion for Driver Alertness with One Camera, Fifteenth IEEE International Conferene on Pattern Reognition, September 3-8, 2000.Barelona, Spain [3] Alper Yilmaz, Mubarak A. Shah Automati Feature Detetion and Pose Reovery for Faes, ACCV2002: The 5th Asian Conferene on Computer Vision, January 2002, Melbourne, Australia [4] S. Spors, R.Rabenstein A Real-Time Fae Traker For Color Video IEEE Int.Conf. on Aoustis, Speeh & Signal Proessing (ICASSP, Utah, USA, May 2001 [5] V. Vezhnevets "Method For Loalization Of Human Faes In Color-Based Fae Detetors And Trakers" The Third International Conferene on Digital Information Proessing And Control In Extreme Situations, May 28-30, 2002, Minsk, Belarus. [6] B. D. Zarit, B. J. Supe and F. K. H. Quek, Comparison of five olor models in skin pixel lassifiation. In Proeedings of the International Workshop on Reognition, Analysis, and Traking of Faes and Gestures in Real-Time Systems, pages 58-63, Kerkyra, Greee, September [7] S. Soldatov Lip reading: Lip ontours detetion, Intelletual Information Proessing Conferene, June, 2002, Alushta, Russia (in russian

6 [8] A. Colmeranez, B. Frey, Th. S. Huang Detetion and Traking of Faes and Faial Features, ICIP 1999, pp , [9] V. Baki and G. Stokman, Menu Seletion by Faial Aspet, Proeedings of Vision Interfae 99, Trois Rivieres, Quebe, CAN (19-21 May 99. [10] K. Toyama, "'Look, Ma -- No Hands!' Hands-Free Cursor Control with Real-Time 3D Fae Traking", In Pro. Workshop on Pereptual User Interfaes (PUI'98, San Franiso, November 1998 [11] R.-L. Hsu, M. Abdel-Mottaleb, and A. K. Jain, ``Fae detetion in olor images,'' IEEE Trans. Pattern Analysis and Mahine Intelligene, vol. 24, no. 5, pp , May 2002 [12] M. Gargesha and S. Panhanathan, "A Hybrid Tehnique for Faial Feature Point Detetion", Fifth IEEE Southwest Symposium on Image Analysis and Interpretation, 7-9 April 2002, Santa Fe, New Mexio About the author Vladimir Vezhnevets is the PhD student of department of Applied Mathematis and Computer Siene of Mosow State University.