3D Shadows: Computer Vision for an Unencumbered Interface

Transcription

1 3D Shadows: Computer Vision for an Unenumbered Interfae Robert Virtue University of British Columbia 356 Main Mall Vanouver, BC Canada ABSTRACT In this paper we desribe a real-time system for deteting pointing gestures using a single amera. Three dimensional information regarding position and orientation obtained through traking the users hand and the shadow ast by the hand is used in a trial appliation simply to provide an unenumbered two dimensional point-and-lik interfae. Similar funtionality has been obtained through the use of multiple ameras, expliit modeling, pattern mathing and through the use of mirrors. In ontrast, our system allows the use arbitrary pointing devies (not ust the hand) to attain interfae-quality pointing using low end onsumer hardware. Provided there is a dominant light soure, the amera may be flexibly positioned in suh a way that hand and shadow images will be aptured. Per-pixel Kalman filtering enhanes foreground extration of hand and shadow ontours from noisy data. Further proessing extrats the intersetion of the pointing ray with an arbitrarily oriented monitor sreen. Keywords Bakground subtration, Kalman filter, gesture, 3D mouse, omputer vision 1. INTRODUCTION The ability to reognize and quantify human ation through nonintrusive means suh as omputer vision is key to allowing improved and intelligent human-mahine interation. Visual traking and motion analysis play an important role in extrating information from video sequenes for appliations suh as surveillane, video oding, traffi ontrol and human-omputer interfaing. The sizeable body of work whih exists in hand traking and gesture reognition is rapidly growing as inreasingly apable tehnial resoures (eg PCs) are beoming inreasingly affordable. While hand gestures have the potential to be a more natural way for humans to interat with omputers, the high variability and artiulation of human hand movement has made this a diffiult task. Preise measurement and good interpretation of human postures in three dimensions is omputationally expensive in general and often requires two or more ameras. In the design of a natural feeling interfae however, exat knowledge and quantifiation of posture may be unneessary provided the interfae behaves niely and in a onsistent manner. Human pointing is a good example of an impreise physial gesture generated from a (possibly) preise mental targeting. For preise pointing some visual feedbak is required. We are able extrat appropriate information from a single amera view so that the intent of a pointing gesture is well represented without having to resort to expliit modeling of the hand and without depending on the premise that the pointing will be done by a hand. The salient vetor/point onept expressed by a pointing gesture may be enated with some hand-held obet, suh as a pen, or what have you. Knowledge of the tip of the pointing devie used is extrated from the video stream, but the three dimension information is only used in this initial appliation to establish appropriate frame transforms to find the intersetion of the pointing ray with a plane, that being a omputer monitor. This approah is partiularly well-suited to providing a minimal hardware, unenumbered interfae in suh realms as publi kiosks. The tehniques used may be further developed to exploit low ost, low quality CMOS ameras to provide preise desktop gesture interfaes.. RELATED WORK Previous work in the visual analysis of gestures and whole-body movement may be ategorized aording to many riteria. These inlude the modeling used (eg kinemati, statistial, template based, stik figures, blobs), sensor modality (infra-red, visible light, range), traking spae dimensionality (D or 3D), number of sensors, sensor mobility and sensor plaement. Expliit shape models are widely used in 3D traking, as kinemati onstraints and known target types failitate the quantifiation and analysis of three dimensional movement. Examples of this may be found in the referenes. The use of expliit shape models in D traking is made diffiult through olusion and arbitrary movements and orientations. A well known example of this approah is Pfinder [6] whih uses statistial region based models to identify and trak people. D approahes without expliit models have been

2 applied to hand pose reognition for sign language reognition (although template mathing is the normal subsequent ation) and gesture based dialogue management. 3D approahes without expliit models are less frequently enountered. Similar work inludes [4] whih uses a mirror to provide the required seond view for 3D information and [] whih uses a planar view for diret translation to monitor spae. 3. IMPLEMENTATION 3.1 Algorithm A onvenient loation for the amera is perpendiular to the line of sight between user and monitor suh that the monitor is inluded in the aptured video frame. A region of interest (ROI) is stipulated by mouse on the video preview frame. Video data from outside this region will not be proessed. Within this region, two sub-regions may be reated to allow different proessing within eah sub-region. This is desirable as different thresholds may be applied for hand and shadow. Additional landmarking is done by hand as well. The four orners of the monitor are found in the video frame. Two points orresponding to the shadow of the monitor edges as proeted onto the desktop are loated. This is shown in figure 1. with the monitor moved for larity. The use of these landmarks will be desribed shortly. A number of empty frames are gathered from the ROI and statistially profiled on a per-pixel basis. Due to the poor quality of the amera used, red and green omponents are used to produe a modified luma metri. The main program loop then onsists of: - Reeive a new frame - Per-pixel Kalman filtre - Denoising and foreground extration - Contour traing - Tip extration - Pointing vetor extration - Transformation and intersetion - Cursor filter A detailed explanation follows. 3. Sensor and Data Consumer grade DV (Digital Video) ameras are not designed for omputer vision appliations. Considerable noise levels, automati fous, exposure and white leveling, along with other onvenient on-board proessing make them a poor platform for rigorous vision appliations. They are, however, relatively heap and available. A reasonable understanding of the harateristis of the data reeived from the amera should be undertaken in order to devise a robust detetion algorithm. By the time a video frame is reeived by the PC as a sequene of RGB (Red, Green Blue) bytes, the sensor data has undergone onsiderable transformation. The first layer is that the RGB triplets are atually R G B where linear RGB values have been transformed to non-linear R G B through gamma orretion speifi to the monitor type. Gamma γ differs between for instane Apple monitors and Windows monitors whih is why some video equipment is OS speifi. The relationship is: and similarly for G and B. 1 γ R = R ' Sensor data for a single pixel with a stati bakground was profiled, as shown in figure. Figure 1. Region of Interest and Landmarking

3 3.3 Bakground Subtration Bakground haraterization typially involves Gaussian models. Most often applied on a per-pixel basis, an entire frame of bakground might be haraterized by a Mixture of Gaussians (MoG). The MoG algorithm models the bakground with independent distributions whih are updated online. With the pixels reent history modeled by K Gaussian distributions, the probability of a pixel x = (x R,x G,x B ) still being in the bakground is where K = 1 P( x) = wη ( x, µ, Σ ) 1 1 η x µ Σ = xµ Σ xµ T 1 (,, ) exp( ( ) ( )) d 1 ( π) Σ Figure. R G B as Normal Random Variables The orrelation matrix for this data showed that this data set annot be desribed as independent random variables. It may not be erroneous to treat the noise as suh, given that during the 13+ seonds required to gather the data set the exhibited orrelation may be due to a hange in the atual pixel value. Video systems exploit the poor olour auity of vision. As long as the omponent representing lightness (luma) is preserved, olour omponents (hroma) may be redued by subsampling (filtering). These omponents are denoted Y C B C R. The amera used was onsumer 480i9.97 DV5 whih (along with professional 480i9.97 DV5 and 576i5 DV5) uses 4:1:1 hroma subsampling. This means that C B and C R omponents are eah subsampled by a fator 4 horizontally and are osited with every fourth luma sample. The 1 R G B bytes this represents are redued to 6, effeting :1 ompression. This means that the olour information is only gathered every fourth pixel, implying a olour pixel position unertainty of about pixels! So the amera gathers data in the non-linear Y C B C R spae, whih is interpolated and transformed to R G B data by the time the operating system gives us aess to it. This suggests that working with Y data is desireable. The proper transform from R G B to Y C B C R is given by: Y = 0.99 R G B C B = (B Y ) C R = (R Y ) Given the signifiant noise exhibited by B during pixel haraterization along with the small weighting in the Y transform, the primary bakground riteria hosen was Y as given by: Y = R G This maintains the relative weighting of R and G while disarding B data. w is the weight estimate for distribution K, Σ is the ovariane matrix and η is a multivariate Gaussian. This approah is partiularly well suited to bi-state bakgrounds (eg traffi lights) and has the harateristi of gradually absorbing bakground hanges into the bakground, whih is advantageous when obets are moved and the bakground must be adusted but a liability when the desired foreground (eg a person) is stationary long enough to beome onsidered bakground. Data gathered from a noisy pixel (shown in figure 3) suggested that a MoG approah might be helpful. This was tested and found to provide small return for the omputational expense as ompared to straight noise redution. Figure 3. Indiation for MoG Approah Another statistial bakground disturbane tehnique is to separate hromaity from brightness by gathering R G B mean and variane data from N empty frames. Brightness distortion β is given by: β = { R', G', B'} { R', G', B'} x µ ( ) µ ( )

4 And hrominane distortion CD is given by: CD = x βµ ( ) { R', G', B'} By applying various thresholds, new pixels are lassified as foreground, bakground, shadow or highlight. Another interesting tehnique is adaptive estimation using Kalman Filtering, as pioneered by C. Riddler [X]. Greysale images used filtering tehniques to provide differentiation between bakground and foreground. The approah used here was to sample N bakground frames and determine mean and variane for R, G, and Y on a per-pixel basis: N i, µ =, { R', G'},( i, ) ( x, y) pixel _ oordinates i, N N i, N = ( µ ), { R', G'} i, i, N N Contour Traing I don t know the formal title of the method used, but it has doubtless been applied elsewhere. To trae the hand ontour, we run down the left edge of the ROI until we hit a foreground pixel. A set of eight vetors desribing diretions to the 8-onneted pixels is used. It is initially assumed that the diretion ust followed was (1,0), the doublet representing x,y displaement. Using a lokwise rotation around the 8-onneted pixels, only five pixels need be examined. Again, assuming the ontour transition was (1,0), the pixel ust left is in the urrent (-1,0) diretion. Current (-1,1) and (0,1) pixels were heked by the previous position as (0,1), (1,1). So, beginning at (1,1) we rotate lokwise looking for foreground. When one is found, we also sniff out the pixel value for the next rotation to avoid following a filament. This funtion is alled reursively, generalized to base the searh from the diretion ust followed. Termination ours on self-intersetion and returning to the left edge of the ROI. Contouring is shown in figure 4. µ and With Y " Y " being alulated one at frame N. Inoming R G values were Kalman filtered on a per-pixel basis using + K = k1 n k1 + n + v y = Kz + (1 K) y k k k 1 = (1 K) + k k1 n Where z k is the urrent measurement, k is the updated noise variane, v is the measurement variane, K is the gain and y k is the modified estimate. The filtered R G values were onverted to Y and segmented aording to thresholds established for eah sub-roi. 3.4 Morphology Spekle noise was filtered with and smoothed by x = ( x x ) ( x x ) i, i 1, i+ 1, i, 1 i, + 1 x = ( x = + ) > i, i n, m n { 1,0,1} m { 1,,1} These operations lean up the bakground, smoothed foreground ontours and filled some spaes. Importantly, it allowed the ontour traing to perform well. Figure 4. Contour and Curvature 3.6 Tips and Lines One the ontour is established for both hand and shadow, eah ontour is examined for loal urvature maxima. Denoting a ontour pixel as C i the pixel at loal maximum urvature χ is For the segment χ i = N ( C C i s n i n) ( C C i s n i n) 0 n + > One χi was found, it was lassified as fingertip or roth by examining the foreground/bakground state of the pixel loated midway between C i-s and C i+s. These pixels are shown in red (foreground) and green (bakground) in figure 4. Alternatives exist, suh as seond and third Gaussian filtering of the ontour whih produes zero-rossing upon urvature infletions, or the use of urvature sale spae. Both of these approahes were used, but the dot produt approah desribed above was more flexible. N i i

5 4. POINTING The diretion of the shadow pointing is determined first. Information available to do this inludes the shadow entroid, weighted values of ontour proximal to fingertips and the entire ontour dataset. The entroid approah is shown in figure 5, weighted ontour segments in figure 6 and regression analysis of the entire ontour in figure 7. Figure 5. Using Centroids Figure 6. Weighted Contour Segments Figure 7. Regressed Shadow Contour The regression of the entire ontour provided the most stability in the alulated diretion, whih had the subetive effet of inreased auray. As the interfae is meant to feel natural, the subetive effet is in fat the ranking riteria. Stability in the other approahes is inreased through weighting the per-pixel Kalman filter to suh an extent that reation speed lags and auray (ie the possibility to make small ursor movements) is diminished. Note in figure 7 that the hand has a yellow segment follow by a red segment along the index finger. The primary weighting on hand pointing diretion is the index finger and stability is inreased in this way: the first 60 pixels from a 6 pixel offset bak from the tip are seleted for linear regression. Pixels further bak from the tip are then added to the regression provided their loation does not deviate from the initial regression line by some threshold distane. Pixels inluded in this extension are shown in red. Also notie in figure 7 that the shadow line intersets our landmark line, whih represents the monitor plane proetion onto the desk. If the end points of this line are given as P 3 and P 4 and two points obtained from the shadow regression (and are loated on the extreme left and extreme right of the frame) are given by P 1 and P, then the point of intersetion P = (x,y) is given by: x = y x + y x x + y x x x x y x y y x

6 y 1 y = ( x x1) + y1 x1 Where x i = P ix P x y i = P iy - P y The salar frax= P P3 establishes the proportional position P4 P3 for loating the vertial line shown in yellow in figure 7, whih onnets the points P T and P B where P T is loated on the landmark line at the top of the monitor and P B is loated on the landmark line on the bottom of the monitor. That is, if the endpoints of the top landmark line are P TL and P TR (and similarly for the monitor bottom landmark line P BL and P BR ) P = P P P P 3 T ( TR TL) P4 P3 P = P P P P 3 B ( BR BL) P4 P3 The diretional ray from the hand is now interseted with the line established by P T P B. The frational distane fray of this intersetion along with frax define measured sreen oordinates for this frame. This predited point is also filtered by Kalman filters in both dimensions. As the was no available data for establishing parameters for the filters some extreme values were initially hosen to provide stability. As of this writing insuffiient experimentation has been done to arrive at satisfatory values whih provide stability, adequate response time and fine ontrol. Initial forays are very promising, but this essential last polishing is inomplete and this was refleted in the user assessments. A mouse button down state is indiated by the lassi pistol pose whih is to say a perpendiular thumb. This is deteted in both the hand silhouette and the shadow silhouette. The deteted state is indiated by the vertial line at the tip of the index finger as shown in figure 8. As yet unimplemented by planned is a right mouse button down state. This will be indiated by extending the pinkie finger. Figure 8. Left Mouse Button Down State 5. USER ASSESSMENT Six subets provided feedbak on the system. The approah was liked in general. Filter response needs adustment for better response. The thumb gesture an ause vertial deviation of the pointer when the thumb is extended vertially, but is not an issue when the thumb is extended horizontally. Pointing preision was on the order of ten pixels with the aveat of a long aquisition time. 6. ACKNOWLEDGMENTS Our thanks to ACM SIGCHI for allowing us to modify templates they had developed. 7. REFERENCES [1] K Oka, Y Sato, H Koike. Real-time Traking of Multiple Fingertips and Gesture Reognition for Augmented Desk Interfae Systems. Pro. IEEE Int'l Conf. Automati Fae and Gesture Reognition (FG 00), pp , May 00. [] R. O'Hagan and A. Zelinsky. Finger Trak A Robust and Real-Time Gesture Interfae Advaned Topis in Artifiial Intelligene, Tenth Australian Joint Conferene on Artifiial Intelligene (AI'97) Proeedings, Perth, Australia, pp Deember [3] A Wu, M Shah, N da Vitoria Lobo. A Virtual 3D Blakboard: 3D Finger Traking Using a Single Camera FG'000 Fourth IEEE International Conferene on

7 Automati Fae and Gesture Reognition Marh 6-30, 000. Grenoble, Frane. [4] S Li, W Hsu, H K Pung. Twins: A Pratial Vision-based 3D Mouse Real-Time Imaging 4, , 1998 [5] J Ku, T Huang. Virtual gun: a vision based human omputer interfae using the human hand Pro. IAPR Workshop on Mahine Vision Appliation Tokyo, pp , 1994 [6] J Rehg. DigitEyes: Vision-based Human Hand Traking for Human-Computer Interation. Pro. Of the 1994 IEEE Workshop on Motion of Non-rigid and Artiulated Obets, pp [7] C R Wren, A Azarbayeani, A Pentland, Pfinder: Real-Time Traking of the Human Body, Pro. Of the nd International Conferene on Automati Fae and Gesture Reognition, 1996 [8] A Al-Mazeed, M Nixon, S Gunn, Fusing Complementary Operators to Enhane Foreground/Bakground Segmentation, Proeedings of British Mahine Vision Conferene 003, pages pp , Norwih. [9] C Ridder, O Munkelt, H Kirhner. Adaptive Bakground Estimation and Foreground Detetion using Kalman- Filtering, Proeedings of International Conferene on reent Advanes in Mehatronis, ICRAM'95, UNESCO Chair on Mehatronis, , [10] M Piovoso, P Laplante. Kalman filter reipes for real-time image proessing, Real-Time Imaging, 9 (003) [11] H Zhou, T Huang. A Bayesian Framework for Real-Time 3D Hand Traking in High Clutter Bakground. Pro. 10th Intl. Conf. on Human-Computer Interation, 003, June -7, Crete, Greee [1] D Gavrila. The Visual Analysis of Human Movement: A Survey. Computer Vision and Image Understanding, Vol 7 (1) pp 8-98, Jan 1999 [13] N Shimada, K Kimura, Y Shirai. Real-time 3-D Hand Posture Estimation based on -D Appearane Retrieval Using Monoular Camera, IEEE ICCV Workshop on Reognition, Analysis, and Traking of Faes and Gestures in Real-Time Systems (RATFG-RTS'01) July 13 - August 13, 001 Vanouver, Canada [14] H Fei, I Reid. Probablisti Traking and Reognition of Non- Rigid Hand Motion IEEE International Workshop on Analysis and Modeling of Faes and Gestures Otober 17-17, 003 Nie, Frane [15] C Poynton. Chroma subsampling notation. Available at [16] C Stauffer, W Grimson. Learning Patterns of Ativity Using Real-Time Traking, IEEE Transations on Pattern Analysis and Mahine Intelligene, Vol (8) Aug 000 [17] R Bowden, T Mithell, M Sarhadi. Reonstruting 3D Pose and Motion from a Single Camera View. Proeedings of the British Mahine Vision Conferene 1998, BMVC 1998, Southampton, UK, 1998 [18] I Reid, A North. 3D Traetories from a Single Viewpoint using Shadows Proeedings of the British Mahine Vision Conferene 1998, BMVC 1998, Southampton, UK, 1998 [19] J MaCormik, M Isard. Partitioned sampling, artiulated obets, and interfae-quality hand traking. European Conferene on Computer Vision, Dublin, 000. [0] C von Hardenberg, F Berard. Bare-Hand Human-Computer Interation, Proeedings of the 001 workshop on Pereptive user interfaes, 001 Orlando, Florida USA [1] Lien, C. and Huang, C. Model-Based Artiulated Hand Motion Traking For Gesture Reognition, Image and Vision Computing, vol. 16, no., , February [] Quek, F., Mysliwie, T. and Zhao, M. Finger mouse: A freehand pointing interfae, International Workshop on Automati Fae- and Gesture-Reognition, Zürih, [3] Segen, J. GestureVR: Vision-Based 3D Hand Interfae for Spatial Interation, ACM Multimedia Conferene, Bristol, [4] D Hekenberg, B Lovell. MIME: A Gesture-Driven Computer Interfae Proeedings Visual Communiations and Image Proessing, SPIE, V 4067, 0-3 June, 000, pages 61-68, Perth, Australia.