Human Tracking and Following Using Sensor Fusion Approach for Mobile Assistive Companion Robot

Transcription

1 Human Trackng and Followng Usng Sensor Fuson Approach for Moble Assstve Companon Robot Ren C. Luo, Na-Wen Chang, Shh-Ch Ln and Shh-Chang Wu Intellgent Robotcs and Automaton Laboratory Department of Electrcal Engneerng atonal Tawan Unversty o., Sec. 4, Roosevelt Road, Tape, Tawan Abstract The ablty to track and follow target person n ntellgent servce moble robot s ndspensable. A robust method for trackng and followng a target person wth a small sze moble robot by ntegratng sngle vson sensor and laser range fnder s proposed. Instead of stereo-vson, we acqure the dstance between moble robot and target person by sngle camera. The laser range fnder and vson sensor have ther respectve drawbacks. To compensate the drawbacks of each sensor we present the complementary data fuson approach Covarance Intersecton, t wll complement the uncertanty of each sensor measure and enhance the relablty of human s poston nformaton. The Vrtual Sprng Model s the control rule of moble robot that can smoothly trackng target person. Expermental results valdate the robust performance of the method. Index Terms Sensor Fuson, Robot Trackng, Intellgent Robot, Vrtual Sprng Model I. INTRODUCTION Moble robots n recent years have been gradually ntegrated nto our daly lves. Human Robot Interacton (HRI) becomes one of mportant topcs. In tradton, HRI systems such as voce command and other Human-Computer Interface nteract wth user passvely. Now we try to make moble robot nteract wth users actvely. For actve nteracton, automatcally detectng and followng users are essental and fundamental functons. Robot wth such functons can offer better servce to users. The challenges nclude detecton of the target person, calculatng the poston and orentaton of the one and real-tme and robust methods. For target person detecton, there have several methods such as dstrbuton based on skn color [], moton-based [], and machne learnng [] those have been developed. Upward methods are usng sngle camera to process. All the above methods are short of the dstance nformaton. In order to acqure the dstance nformaton, some papers propose some methods to detect people by usng laser range fnder (LRF) wth hgh precson, wdely range detecton and hgh relablty. Therefore, many human detecton approaches are usng LRF to detect human s leg or torso [4]. Although the laser range fnder has good accuracy n dstance measurement but the effects of dentfcaton human have not been very well /9/$5. 9 IEEE So some research use two or more complementary characterstcs such as combnng vson sensor, laser range fnder or ultrasonc sensors to enhance the accuracy of human detecton [5][6][7]. Some of human detecton methods use Partcle Flter. Isard and Blake [8] ntroduce the Partcle Flter for vsual trackng. Even though the Partcle flter can handle nonlnear and non-gaussan systems, t s hghly-dependent to samplng and have to desgn a good proposal dstrbuton whch s usually dffcult to desgn. In order to cope wth ths problem, Merwe, Doucet, Fretas et al [9] developed the unscented Partcle Flter that uses the unscented Kalman flter to produce proposal dstrbuton. In current lterature, moble robots whch have human trackng functon are almost as hgh as human bengs. But the feld of how to use small sze moble robot to track human, there s no dscusson wth detals. Sngle vson-sensor methods only detect human s orentaton but not dstance nformaton except stereo-vsual sensor. In ths paper, we ntroduce to detect and estmate the human s dstance and orentaton by usng sngle camera. The laser range fnder also can detect the dstance and orentaton of human s legs. Fnally t ntegrates the nformaton by the vson sensor and laser range sensor to do sensor fuson Covarance Intersecton that wll enhance the relablty of human s poston nformaton. The paper s organzed as follows: Secton II descrbes the system structure of the moble robot. Secton III presents the detecton method of vson sensor and laser range fnder. Secton IV presents the method of sensor fuson that ntegrates the nformaton of human s poston from each sensor. Secton V presents the method of robot followng. Fnally the expermental results and concluson show n Secton VI. II. SYSTEM ARCHITECTURE A. Robot Hardware Archtecture The robot we present s a frendly robot assstant LuoGude. The robot s specally desgned by our Laboratory and used to assst to serve for the Laboratory. It ncludes a embedded PC (.5GHz, GB), and s equpped wth a laser range fnder (URG), color camera, mcrophone, Wreless LAN, and has a storage box can help us to delvery the books or notebooks. The hardware structure s shown n Fg.. 5

2 Fg. The heght relatons of robot and human. From rght to left s a general heght person, a normal sze servces robot and our experment platform. Fg. Hardware structure of frendly robot assstant LuoGude. B. The Human Trackng and Followng Method Framework In ths paper, we propose a method to use two knds of sensor to accomplsh the ablty of human trackng and followng. The human detecton system s a parallel processng, dealng wth mage data and laser data at the same tme. The flow chart s shown n Fg.. Fst, the vsual and laser sensor detect human respectvely and estmate the dstance and orentaton. After estmatng poston of human, t starts to do sensor fuson to reduce the data uncertanty and enhance the relablty of human s poston. Fnally use the poston comes from sensor fuson to conduct moton control to mplement the real-tme human trackng. In order to accurately detect user s poston, a method for trackng and followng people s proposed. The Adaboost algorthm s be recommend to fast detect objects [4], t s a machne learnng approach for vsual object detecton whch s able to processng mages extremely rapdly and measure up hgh detecton rates. For mplementng real-tme target person detecton, we make use of Adaboost algorthm. There are three man contrbutons of fast object detecton. The frst s creatng a new mage representaton called the Integral Image whch extracts features very quckly on our detector. The second s Adaboost learnng algorthm whch selects a small number of mportant features from a larger set and generates extremely effcent classfers. The thrd s usng cascade structure to combne ncreasngly more complex classfers whch can let background regons of mage to be quckly throw away and makes more calculaton on lkely object regon. In other words, AdaBoost s an algorthm for constructng a strong classfer as lnear combnaton of smple weak classfers. The above advantages mentoned make Adaboost has excellent performance for fast object detecton. To make a better detecton from any sde vew of the target person, so we tran our classfer, ncludng the postve sample from front sde, back sde and profle human s upper half-body mage as shown n Fg. 4. That can realze no matter whch sde vew the moble robot detect, t all wll take t as an object. Adaboost produces many weak classfers after tranng, and then cascade those weak classfers to become a strong classfer as shown n Fg. 5. Our tranng data have, pctures of postve sample (human),, pctures of negatve sample (not human), and the strong classfer stage s splt to stages. Fg. 4 Each of tranng samples ncludes front sde, back sde and profle human mages. Fg. 5 The structure of the human detector Fg. The human Detecton flow chart III. THE HUMAN TRACKING ALGORITHM A. Image Processng /9/$5. 9 IEEE When the human upper half-body s detected by the Adaboost algorthm, we deduce the relatonshp between moble robots and human. Suppose people are n front of the 6

3 moble robot n the world coordnate, the relaton of relatve poston appears by pnhole mode. Assume that human bengs stand n front of moble robot. There s no offset angle as shown n Fg. 6. And H m wth H c are respectvely representatve heght of human and camera, n addton they are known. Dm s the dstance of human and robot, θc s camera s angle of elevaton, f s camera focal length, the human s poston n mage plane s (u ', v' ), by the pnhole mode can be obtaned the object project on mage plane produce an angle φ wth center-lne on v drecton. If φ was obtaned, the dstance between moble robot and human Dm can be calculated by (). Dm = Hm Hc tan(θc φ ) ; φ = tan ( v' v ) f () If human bengs stand n front of moble robot but not on the center-lne as shown n Fg. 7, the object project on mage plane produce an angle ϕ wth centre-lne on u drecton ths was the offset angle between robot and human. After applyng () we get the offset angle ϕ. By the relaton of smlar trangles (), t s easy to acqure the length Dh. ϕ = tan ( u ' u ) f f h ; h= = Dm + f Dh + h f + u () () Fg. 7 Human bengs has offset angle to estmate poston schematc B. Laser Processng The laser scanner on robot platform s Hokuyo URG-4LX, and t s fxed on the robot approxmately cm above the ground. Therefore t can only perceve the legs of people. Ths Sensor, whch determnes the dstance by measurng the tme of flght (TOF) of the emtted pulse, commonly appled n automaton feld []. The Hokuyo URG-4LX sensor scans at Hz, the sensor can rotate by a sgnfcant amount between the start and end of a gven regon thus t becomes necessary to nterpolate the servo s poston between readngs usng ths velocty [], [], []. Generally, a person can be located by two closely postoned segments (as shown n Fg. 8). Each leg can be perceved an arc-lke shape, and the dameter of the arc s close to 7mm~8mm. The human detectng s based on couple arclke segments congregatng n a laser scan, and set the center error δ = 4mm. If the ntersecton ponts of any perpendcular bsector of the segment are n the crcle wth the radus δ, the segment s a canddate leg of person. When there are many leg-lke dsturbers (such as table or char legs), the par-legs s dffcult to dstngush. For our experments, URG-4LX s confgured to work n 4 meter range mode, wth the angular resoluton of.5 (6 /4 step). In order to have a physcal calbraton for the further error propagaton and proper treatment of the spatal uncertanty n the feature extracton, we expermentally determne the man uncertanty components n the range measurements of the URG-4LX scanner. A more detaled characterzaton of the URG-4LX sensor s provded by [4]. Fg. 6 Human bengs stand n front of robot s schematc and they s no offset angle. rˆ = r + δr, δr = (,σ r ) Fnally the lght of the above mentoned methods can be defned relatons of dstance and angle between robot platform and human. So n the secton IV, we ntroduce sensor fuson of CCD and Laser Range Fnder (LRF) whch estmate the human s poston /9/$5. 9 IEEE where r s an actual range and σ r (4) s a devaton. The uncertanty n the angle of the measurements s caused by the fnte resoluton of the scanner mrror encoder and the fnte measurng wdth of the laser. However, n most of the laser scanners ths uncertanty s very small, and t s often neglected [5]. In order to adjust the Hokuyo URG-4LX s measurng naccuracy, we amed wth 5cm, cm, 5cm, cm, 5cm, cm correspondng bas measurng for tmes. Computng the mean bas, varance and dstrbuton of 7

4 each smple dstance, to accomplsh reduce the space uncertanty and naccuracy. Detal results were shown n Fg. 9. together to yeld an output µ o, where µ v and µl are estmaton from vson and laser range fnder system model, and Pvv, Pll and Po are ther covarance. The Covarance Intersecton Algorthm (CI) [6] s used to fuse these. The ntersecton s characterzed by the convex combnaton of the covarance, and the Covarance Intersecton algorthm s: (5) Po = ωpvv + ( ω ) Pll Fg. 8. Laser processng perceves the legs of people. The center of couple legs (red). 7 6 mllmeter(mm) (6) where ω [, ], and ω modfes the relatve weghts assgned to Pvv and Pll. Dfferent choces of ω can be used to optmze the covarance estmate wth respect to dfferent performance crtera such as mnmzng the determnant of Po. The fact that ths update s conservatve for every ω and Bas data Regresson curve Varance data 8 Po µ o = ωpvv µ v + ( ω ) Pll µ l 5 t can be shown usng a proof by whch demonstratng the matrx (7) Po E[ µ~o µ~ot ] Measurement Dstance (mm) 5 µ~l are the errors n µv and µl. IV. COMPLEMENTARY FUSION ALGORITHM In general, for the feature extracton from human wth sngle camera, we can obtan relable orentaton nformaton from vson system, but may get rough depth nformaton. On the other hand, LRF model process s n contrast wth vson process, t s easy to measure the accurate dstance of an extracted object, but t s hard detect and recognze the feature n the envronment. In these proposed approach, we combne the vson feature wth range model feature to reduce uncertanty of detectng and recognzng the poston of the human. Sensor fuson mproves the measurement performance when dfferent sensor modaltes are measurng the same target. Fg. shows the complementary correlaton. The ellpses represent the measurng uncertantes. In other words, the Covarance Intersecton method provdes an estmate and a covarance matrx whose covarance ellpsod encloses the ntersecton regon. The estmate s consstent rrespectve of any value of Pvl. If leg features are observed n a laser scan (LRF features), these features have to be assocated to the already known ones n the state vector. A common way to do ths s to calculate the Mahalanobs Dstance, and then the Covarance Intersecton Algorthm (CI) s used to fuse these to estmate an accurate human poston. The detaled algorthm processng as follows: Complementary Fuson Algorthm Data: all LRF features ( µl, Pll ) n a local scan x µ l = l and correspondng covarance matrx Pll yl Vson Process Estmate Parameter of vson observaton - (a) Result: To estmate of the trackng human poston nformaton, and update them when they are dentcal. Intalze: object detecton = False For all LRF features from each scan do Fuson Result µ v = v and correspondng covarance matrx Pvv y v x LRF Process (b) Compute Mahalanobs Dstance between vson observaton and all LRF features n a laser scan Fg.. (a) Shows the complementary correlaton, and (b) shows the fused correlaton. DM ( µl, µ v ) = ( µl µ v )T S ( µl µv ) Two peces of nformaton, µ v and µl, are to be fused /9/$5. 9 IEEE Pvl between the two pror estmates: ~ = P (ωp µ~ + ( ω ) P µ~ ), where µ~ and The error µ o o v v l l v Fg. 9. (Left) To be amed wth 5cm, cm, 5cm, cm, 5cm, cm correspondng measurable dstrbuton. (Rght) To be amed wth 5cm, cm, 5cm, cm, 5cm, cm correspondng bas (red) and ts regresson curve (blue) and correspondng varance data (green). Human leg detect by LRF Covarance of vson process Covarance of LRF process s postve semdefnte for any cross covarance 8

5 (Mahalanobs dstance be defned as dssmlarty measure between µ l and v r w / v v = w / ω l µv of the same dstrbuton wth the covarance matrx S) If the Mahalanobs Dstance s wthn a threshold T, assume t s the same trackng object then Fuse LRF feature and vson feature wth CI Po = [ ωpvv + ( ω ) Pll () ] µo = ωpo Pvv µv + ( ω ) Po Pll µl object detecton = True End End If object dcton =true then Trgger trackng moton control End Fg. The structure of the human detector VI. EXPERIMENTAL RESULTS AND CONCLUSION V. ROBOT FOLLOWING CONTROL METHOD After the complementary fuson algorthm can accurately estmate human poston. We propose a vrtual sprng model to control the robot. Ths proposed method s derved from the assumpton that the target people and the moble robot are connected by a vrtual sprng. Fg. shows the vrtual sprng model. The vector of the moble robot s represented by follow. (8) X = [ xr yr θ r ]T The nput ( d, φ ) of the vrtual sprng model are obtaned form complementary fuson algorthm, where d represents the dstance between robot and target human and φ the angle from The proposed data fuson usng covarance ntersecton approach s mplemented n our robot. Fg. shows usng sngle camera to detect and estmate the human s orentaton and dstance. The laser range fnder also can detect the orentaton and dstance of human s legs, shown n Fg.. Specally, when there are many leg-lke dsturbers (such as table or char legs), the par-legs s dffcult to dstngush. So after above processng, we ntegrate the human nformaton by the vson sensor and laser range sensor to do sensor fuson process covarance ntersecton that wll enhance the relablty of human s poston nformaton. the drecton of the robot to the vrtual sprng. Elastc F s proportonal to d and F s proportonal to φ. We assume that the free length of the vrtual sprng s whle the robot s successfully tracked. F and F are defned as follows: F = kd Fg.. The vson feature of LRF process detects human body. (a) front, (b) flank and (c) back. (9) F = kφ Where, k and k are the expanson and bendng coeffcents of the vrtual sprng respectvely. The dynamc equatons are derved as follows: mv& = F cos φ F sn φ kv () ω& = ( F sn φ + F cos φ k 4ω ) L m s the mass of the robot and s the moment of nerta of robot. k and k 4 are the vscous frcton coeffcents of (a) (b) (c) (d) translaton and rotaton respectvely. Thus, we obtan follows: v& = F F k cos φ sn φ v m m m ω& = ( F sn φ + F cos φ k 4ω ) Fg.. The leg feature of LRF process detects human (a) stand up, (b) walkng, (c) sdeways and (d) two people. () L vr and vl represent the velocty of the of the rght and wheels respectvely. w s the dstance between the rght and left wheels. Fnally the robot moton control system can be obtaned as follows: /9/$5. 9 IEEE Fg. 4 (rght) shows the trackng scenaro that our robot combned ntensty and range data to locate the human by covarance ntersecton. The result of fusng ntensty and range data usng covarance ntersecton s shown n Fg. 4 (left). Fg. 5 shows the dstance varaton and orentaton 9

6 varaton between robot and target human durng the trackng and followng. Frame 68 Dstance (cm) Covarance of LRF Covarance of Vson Covarance of CI Result Mean of LRF Mean of Vson Mean of CI Result Oentaton (degree) 4 Frame Dstance (cm) Covarance of LRF Covarance of Vson Covarance of CI Result Mean of LRF Mean of Vson Mean of CI Result Orentaton (degree) 4 Frame Dstance (cm) Covarance of LRF Covarance of Vson Covarance of CI Result Mean of LRF Mean of Vson Mean of CI Result Orentaton (degree) 5 Fg. 4. The robot trackng human by fusng vson feature (blue) and LRF feature (red), and estmate the fused result (black). LRF Orentaton LRF Dstance Vson Orentaton Vson Dstance Dstance (red) (cm) Orentaton (blue) (degree) [] Prahlad Vadakkepat, Peter Lm, Lyanage C. De Slva, Lu Jng, and L L Lng, Multmodal Approach to Human-Face Detecton and Trackng IEEE transactons on ndustral electroncs, vol. 55, no., March 8. [] A. Utsum and N. Tetsutan, Human trackng usng multplecamerabased head appearance modelng, n Proc. 6th IEEE Int. Conf. Autom. Face Gesture Recog., May 4, pp [] P. Vola and M.J. Jones. Robust real-tme object detecton. In: Proceedngs of the IEEE Workshop on Statstcal and Theores of Computer Vson,. [4] A. Fod, A. Howard, and M. J. Matarc, Laser based people trackng, n Proc. of the IEEE Internatonal Conference on Robotcs & Automaton (ICRA), Washngton DC, Unted States, pp.4-9,. [5] Wen Da, Aysegul Cuhadar, Peter X. Lu, Robot Trackng Usng Vson and Laser Sensors IEEE Conference on Automaton Scence and Engneerng, Washngton DC, August 6, 8. [6] A. Schedg, S. Mueller, C. Martn, and H. M. Gross, Generatng Persons Movement Trajectores on a Moble Robot IEEE Internatonal Symposum on Robot and Human Interactve Communcaton, Hatfeld, UK, September 6-8, 6. [7] X. Ma, C. Hu, X. Da, K. Qan, Sensor Integraton for Person Trackng and Followng wth Moble Robot, Internatonal Conference on Intellgent Robots and Systems, France, August 6, 8. [8] M. Isard, A. Blake, Vsual trackng by stochastc propagaton of condtonal densty, n Proc. 4th European Conf. Computer Vson, Cambrdge, UK, Aprl 996 [9] R. Merwe, A. Doucet, N. Fretas et al, The unscented partcle flter, n Techncal Report CUED/F-INFENG/TR 8, Cambrdge Unversty Engneerng Department, August. [] Terawak, Measurng Dstance Type Obstacle Detecton Sensor PBSJN Instructon Manual, Hokuyo Automatc Co., LTD., Aprl 4, pp. 9. [] R.O.Duda, R.E.Hart. Use of the Hough Transform to Detect Lnes and Curves n Pctures, CACM(5). No., January 97, pp. 5. [] P.V.C. Hough. Method and Means for Recognzng Complex Patterns, US Patent,69,654, December 96. [] T.S. Mchael and T. Qunt. Sphere of nuence graphs n general metrc spaces. Mathematcal and Computer Modellng, 9:45{5, 994. [4] [Hokuyo Automaton, 5] Hokuyo Automaton. Scannng laser range fnder for robotcs [5] M. D. Adams, Sensor Modellng, Desgn and Data Processng for Autonomous Navgaton, Sngapore, World Scentfc, 999. [6] S.J. Juler, J.K Uhlmann, A non-dvergent estmaton algorthm n the presence of unknown correlatons, Amercan Control Conference, vol.4, pp.697, June Frame Fg. 5. The dstance varaton and orentaton varaton n frames. The target starts movng n frame and frame 5 and stop n frame 49 and frame 9. We have proposed a method for real-tme trackng and followng of the human on small sze moble robot. Proved by experments, t can accurately calculate dstance of target human by usng sngle camera. The complementary sensor fuson mproves the weakness of measurement from each sensor and reduces the uncertanty on calculaton of target person s poston so that t comes out a better relablty. Integratng wth the vrtual sprng model, we successfully mplement the functon that makes moble robot real-tme track and follow target person. REFERENCES /9/$5. 9 IEEE 4