Towards A Human Robot Interaction Framework with Marker-less Augmented Reality and Visual SLAM

Transcription

1 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 Towards A Human Robot Interacton Framework wth Marker-less Augmented Realty and Vsual SLAM Eranda Lakshantha and Smon Egerton Faculty of Informaton Technology, Monash Unversty Emal: eranda.lakshantha@monash.edu; smon.egerton@monash.edu Abstract Ths paper presents a novel framework for Human Robot Interacton (HRI) usng marker-less Augmented Realty (AR). Unlke marker-based AR, marker-less AR does not requre the envronment to be nstrumented wth specal markers and so t works favorably for unknown/unprepared envronments. Current state-of-the-art vsual SLAM approaches lke PTAMM (Parallel Trackng and Multple Mappng) acheve ths wth constraned moton models wthn local co-ordnate systems. Our framework relaxes moton model constrants enablng a wder range of camera movements to be robustly tracked and extends PTAMM wth a seres of lnear transformatons. The lnear transformatons enable AR markers to be seamlessly placed and tracked wthn a global co-ordnate system of any sze. Ths allows us to place markers globally and vew them from any drecton and perspectve, even when returnng to the markers from a dfferent drecton or perspectve. We report on the models performance and show how the model can be appled to help humans nteract wth robots. In ths paper we look at how they can assst robot navgaton tasks. operated n any envronment. But most marker-less AR applcatons requre a steady camera Feld-Of-Vew (FOV) whereby the camera orentaton does not vary more than a fxed amount.e.180 degrees. Extendng the trackng robustness for unknown scenes was explored n [1] and [2]. However a contnuous camera trackng s not always possble and, losng the AR poses as the camera changes vewponts s common. Our prme concern s about robustly trackng marker-less AR under such crcumstances and ts applcaton as a contemporary human-robot nterface. To put our queston nto perspectve lets consder a practcal scenaro. Suppose a moble robot mounted wth a camera on top moves towards an AR object. It keeps on movng forward untl t passes the AR object. Once t has travelled a slght dstance away from the AR marker (.e. AR object), t turns back and looks n the drecton t travelled. At that nstance the AR marker must be stll vsble. These AR markers persst n the envronment untl they are removed and so appear and behave as real world objects would do through the camera of the robot. Ths noton of persstence enables many Human Robot Interacton tasks such as navgaton, programmng, mult-robot/mult-user collaboraton and so on. In ths paper we report on HRI navgaton use as t s one of the most unversal robotc operatons. By connectng a seres of persstent AR markers an AR navgaton path can be overlad on to the camera vew and become a powerful nterface n remote robot operatons especally for people who have lttle to no expertse nteractng wth robots, for example navgatng a domestc robot at home or navgatng a servce robot around the factory floors. We develop our work by optmzng PTAMM [1] whch s the current state-of-the-art. It s known to be the most sutable approach for trackng camera pose n prevously unknown envronments. We extend PTAMM to map n a global space of any sze and call the extended model Parallel Trackng And Global Mappng (PTAGM). We show emprcally that persstent AR markers are useful and robust. Later sectons of ths paper shows PTAGMs operaton as a human-robot nterface; thereby showng how to gude a robot towards a desgnated locaton n space usng marker-less AR. Index Terms augmented realty, human-robot nteracton, robotcs, SLAM. I. INTRODUCTION Applcatons of Augmented Realty (a technology that can overlay vrtual graphcs on top of a real-tme vdeo feed), have great potental for brngng new avenues n human-robot nteracton. We present a robust marker-less AR technque n whch we could place an AR object at any pont n space (as seen by the camera) and then assocate a command to be fulflled by the robot at that pont. Ths s qute smlar to the behavor of pontng somewhere n space and then gvng an nstructon lke `move here whch could be often observed n human-human engagements. Whle buldng such a HRI framework, marker-less AR s consdered more sutable as opposed to marker-based AR snce the latter approach requres the envronment to be attached wth physcal markers whch n turn mposes strct lmtatons to the robots envronment. On the other hand marker-less AR does not assume the target envronment to be known n advance, thereby the robots could be readly Manuscrpt receved August 1, 2013; revsed November 6, Engneerng and Technology Publshng do: /joace

2 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 The next secton outlnes the related lterature whereas later sectons descrbe n detal the methodology, results obtaned and ts applcatons. II. RELATED WORK A sold marker-less AR technque always demands a real-tme and an accurate computaton of the camera pose [3]. Mantanng such scene augmentatons have been well explored by [4], [5] and [6]. However, these studes look promsng only f a predefned knowledge of the envronment s avalable; thereby they lack support for unknown or unprepared envronments. On the contrary, our focus s to robustly track marker-less AR under unknown envronments (.e. less constraned and better for robotcs). [7] presented the noton of mantanng a growng map as the robot explores new regons. They managed to buld a map based on the texture of a floor whch was dstrbuted n a unform manner. Despte ther lmtedness for specal textures, t appeared as a possble replacement for [4], [5] and [6]. Ths mples the potental of a `mappng approach, lke Vsual SLAM 1 n order to substantate the trackng qualty of marker-less AR for robotcs. [8] emerged as the frst vsual SLAM approach n the doman of moble robotcs. Ther work, dubbed as MonoSLAM presents a probablstc 3D map whch s dynamcally updated by a EKF (Extended Kalman Flter). The map s bult accordng to a metrc scale, and so t always requres the system to ntalze by lookng at an object of a known sze. Ths s a weakness of MonoSLAM that makes t dscourageable for robotcs applcatons. However t was no longer a problem wth the advent of PTAM (Parallel Trackng And Mappng) [2]. Instead of a metrc scale, PTAM mantaned a map, bult accordng to an arbtrary scale. Along wth ts other advantages lke, no pror assumpton about the envronment, ablty to sustan a growng map, and the presence of an external coordnate frame, PTAM grounds a sold base for our noton of brngng up marker-less AR as a human-robot nterface. However, the expermental results wth PTAM are seen questonable for ts ablty to perform n a persstent manner. Constant jtterng took place whenever the camera was moved far away from ts orgnal locaton. Ths happened as t lost ts pose and we observed t strugglng to re-localze tself wth the best known pose. It totally lost the AR object at a pont where the camera orentaton spanned more than 180 degrees. To some extent, PTAMM dealt wth ths nadequacy by generatng multple local coordnate frames assocated wth multple maps n a one-to-one fashon. These coordnate frames are produced on a domnant plane formed by ts own map ponts. It gave the user freedom to move the camera around and pck multple locatons n the envronment where they act as placeholders for generatng local coordnate frames. The supplementary vdeo fle 2 shows a typcal operaton of PTAMM. However PTAMMs maps operate ndependently and so they do not unformly track a sngle AR marker n a global fashon. Instead PTAMM dsplays the AR object assocated wth the current local map. Furthermore, the ndvdual local maps are seen to be swtchng among each other whle delverng an rrtated mpresson for the user, whch n turn makes t hard for postonng the cameras global locaton wthn the envronment. Ths effectvely lead us to extend PTAMMs ablty to mantan a sngle global pose throughout multple local coordnate frames, whch n turn could act as a robust nterface for human-robot operatons. III. OUR METHODOLOGY IN THE CONTEXT OF PTAMM The method we followed can be summarzed as below. Global camera pose s mantaned by generatng a set of multple local coordnate systems. A seres of lnear transformatons s used to combne these multple local coordnate frames. To place our method nto focus, t s useful comparng our method wth both PTAMM and PTAM. In our knowledge PTAMMs argument of multple maps emerges as a complmentary soluton to address the weaknesses of ts predecessor PTAM. Authors of both PTAMM and PTAM beleved that mantanng a global map would be a resource ntensve and a tme consumng operaton. The reason s that the bundle adjustment tmes lnearly amplfy as the number of map ponts grows up; whch ultmately caused the AR object to be unstable under dfferent vew drectons, often losng ts vsblty when the vew angle changes over a great extent (.e. more than 180 degrees). On the other hand, we operate a sngle map at a gven tme, and preserve a common global pose among subsequent local frames (local maps) by lnear transformaton. We take the cameras ntal locaton as the orgn of the global coordnate frame and express the camera pose wth respect to ths global-frame orgn. The system descrbed here (.e. PTAGM) supports formng new coordnate frames as the camera enters unexplored regons wthn the envronment. Consequently the camera swtches nto a new coordnate frame at the presence of a new regon; and n such a manner the total set of coordnate frames could resemble the entre area of camera moton. Whle swtchng between multple coordnate frames, PTAGM s capable of mantanng a seamless camera pose n relaton to the global frame by takng lnear transformatons nto account. Inter-coordnate transformatons are calculated wth the followng mathematcal formula. x R t x (1) x and x are vector representatons of an dentcal pont n a global frame and a local coordnate frame respectvely. R denotes the amount of rotaton n matrx form whereas t sgnfes the amount of translaton n vector form between 1 Smultaneous Localzaton And Mappng based on computer vson Engneerng and Technology Publshng 251

3 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 the two Cartesan spaces. R t s collectvely known as the transformaton matrx whch transforms a homogeneous pont (here t s x ) from one coordnate frame to another. Based on ths prncple our algorthm sets out a persstent way for trackng marker-less AR under unknown scenes, whle capturng the rotaton and translaton of consecutve local frames n relaton to a fxed global frame. by fndng the coordnate transformaton 3 of C4 wth respect to C1. In other words, the cameras poston and orentaton has to be expressed n terms of the global frame C1. Fgure. 2. Producng another local coordnate frame C4 Fgure 1. Global frame C1 and local frame C2. The camera ntally les wthn C1 Fg. 1 depcts the ntal setup durng our algorthms operaton. The system produces two coordnate frames durng ts ntalzaton namely C1 and C2. The camera ntally remans at the orgn of C1 and so C1 s consdered as the global frame. C2 on the other hand represents the local map (local frame) whch n turn denotes the current regon beng operated. The rotaton and the translaton of C2 n relaton to C1 s gven by R and t and these values are stored nsde the current map. As mentoned above, our algorthm permts to create more local coordnate frames, whenever the camera enters nto new regons. However these local coordnate frames descrbe only the regon of current camera operaton, and computes only a local camera pose n the context of the currently operatng local map. As a result, the camera pose changes frequently each tme t swtches nto a new local coordnate frame - whch s problematc for persstent trackng. Thereby we need a mechansm for mantanng a steady and a contnuous camera pose whle swtchng between dfferent coordnate frames. In dong so, our system employs an nter-medate coordnate frame for each local frame, whereby t acts as a brdge to connect wth the global frame C1. The rotaton and translaton of such an nter-medate frame (n relaton to C1) s gven by R and t. Consder the example llustrated n Fg. 2. Fg. 2 depcts the camera moton through coordnate frames C1 to C3 and the creaton of a new local frame C4. C3 represents an nter-medate coordnate frame whch connects C4 wth C1. Here R and t sgnfes the rotaton and translaton of C3 wth respect to C1. It s mportant to realze that the camera now operates totally on a new regon, dentfed by the local frame C4. Therefore the ultmate result - the cameras global pose could be obtaned In dong so, our method follows a two-step approach. Frst t fnds the transformaton of C4 wth respect to C3. The goal of ths step s to express the camera pose n terms of C3. Then t computes the transformaton of C3 wth respect to C1, hence delverng the camera pose n terms of C1. We hereby outlne the equatons occuped for these transformatons n the order as they operate. Two varables are ntroduced namely R cur and t to ndcate the local cur camera pose whch s assocated wth the local map C4. Assumng the camera s postoned as n Fg. 2, transformaton from C4 to C3 can be gven by, t C R. tcur t (2) 3 R R. R (3) C3 1 cur where R and C3 t are rotaton and translaton (pose) C3 parameters of the camera wth respect to C3. Once we have equatons (2) and (3) n place transformaton from C3 to C1 can be computed as, t C1. C3 R R t t (4) R R (5) 1 C1 C3. Fnally the values of R and C1 t consttute the cameras C1 global pose. Our algorthm teratvely computes the equatons (2), (3), (4) and (5) at frame rate whch results n carryng out a seres of lnear transformatons throughout multple coordnate frames (maps). To vsualze a better pcture of our algorthm we exposed ts steps n Fg. 3. IV. RESULTS & EXPERIMENTS Evaluatons are performed wthn an ndoor space usng a standard web-cam. Frst, we testfy PTAGMs trackng robustness for marker-less AR whle takng PTAM as a benchmark, snce PTAM and our system (.e. PTAGM) shares the common trat of havng a sngle global map. 3 Expressng a coordnate system n terms of another one 2014 Engneerng and Technology Publshng 252

4 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 However, we could not take PTAMM for these comparsons as t operates on a set of multple local maps - nstead of the noton of a sngle global map. Secondly, we nvestgate the applcaton of our system as a human-robot nterface - specally for robot navgaton, n whch we navgate a robot wthn a less constraned envronment. Secton A n the followng descrbes the trackng robustness of PTAGM whereas secton B demonstrates the use of PTAGM n HRI navgaton. Fgure 3. PTAGM algorthm Fgure 4. Persstent trackng of an AR object for wder camera perspectves A. Trackng Relablty for Wde Camera Moton Trackng relablty of both PTAM and PTAGM was put nto test under smlar condtons. After the system ntalzaton, the camera was moved towards the AR object. We kept on movng the camera n a straght lne untl t went pass the AR object. Stll a steady moton was taken forward and fnally turned the camera around wth an angle of 180 degrees. At ths nstance the camera was lookng back at the path t travelled. The argument to 2014 Engneerng and Technology Publshng 253

5 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 valdate at ths pont s that the camera should stll see the AR object persstently anchored at ts orgnal locaton wthout any jtter. The results are showcased for PTAGM n Fg. 4 where each ndvdual mage represents the steps mentoned above. The full demonstraton can be observed n the accompanyng vdeo fle 4. It can be seen that PTAM lost the sphere lke AR object at the end of the sequence whereas PTAGM dsplayed t persstently. Such a qualtatve comparson durng a lve run happens to gve notable results for PTAGM over PTAM. Vdeo fle 5 for PTAMs operaton further justfes ths dstncton. (Note that we have modfed the PTAMs GUI slghtly to brng both systems nto equal grounds). PTAGM generated fve maps durng ths experment wth each map holdng 31, 26, 5, 82, and 94 keyframes respectvely. On the other hand feature ponts are recorded as 2446, 1984, 861, 221, and 540. Therefore the largest map generated here s map 0 wth 2446 feature ponts. Its bundle adjustment tmes are llustrated n Fg. 5. Note that the bundle adjustment tme keeps on rsng as the number of feature ponts grow up. The vertcal dotted lne ndcates the pont where the camera swtched nto a new local coordnate frame. Even though we used a hand-held camera for ths test run, the same results can be magned for a robot movng wth a camera on ts top. B. Applyng PTAGM for Robot Navgaton For ths test scenaro we employed a ground robot and a camera that outlooks the robots target envronment. Our marker-less AR nterface (.e. PTAGM) les n between the human operator and the robot. In ths set up, the AR nterface enables human operator to pont to a locaton n the scene dsplayed by the camera and then navgate the robot towards that locaton. In ths case, the act of `pontng to a locaton s accomplshed by dsplayng an AR object at the locaton beng ponted. The accompanyng vdeo 6 demonstrates how ths functons. observable n typcal human-human nteractons. Nevertheless the robot needs the dstance and rotaton requred to reach the target locaton desgnated by the AR object. The global coordnate frame n our system asssts n computng these parameters n terms of the coordnate frames unts. However the robot only accepts metrc dstances, thereby a correlaton between the PTAGMs global frame and the metrc scale s needed. Ths s acheved by sldng the robot 10cm at the systems outset. The correlaton process operates only once snce we mantan a global pose and the metrc proporton remaned unchanged throughout the entre applcaton runtme. Ths demonstraton s done wth Edde robot platform as llustrated n Fg. 6. Fgure. 6. Robot navgaton wth PTAGM. The top left-hand represents the robots start locaton. Next shows the placement of the AR marker at a pont n front. The bottom-most mage depcts the robots destnaton. V. LIMITATIONS & FUTURE WORK Fgure 5. Bundle adjustment tmes for map 0 Such a `pont and go manner s a natural mode of communcaton for humans as t closely relates to expressons lke `Move here and `Go there, whch s often The system descrbed here comes wth some known ssues and drawbacks. Frst t requres the user nvolvement for ntalzng a new map. Ths does not delver a reasonable authentc value for HRI, especally for non-robotc experts. As a workaround we plan to ncorporate two cameras wth a fxed baselne, nstead of havng a sngle camera. Despte usng a sngle AR marker, the system must functon wth multple AR objects. Ths s mportant snce the user must be able to place several markers that act as a seres of wayponts representng a navgaton path. Nevertheless our system must not preclude HRI, only for navgaton. Instead, t should support other areas of HRI such as mult-robot control, robot gamng, robot programmng, learnng from demonstraton (LfD), etc. Further extensons to the current model are beng 2014 Engneerng and Technology Publshng 254

6 Journal of Automaton and Control Engneerng Vol. 2, No. 3, September 2014 underway for such a HRI task - mult robot control. A group of small robots could be effectvely controlled by placng an AR object through PTAGM. For an example, a sngle AR marker can be shared among multple robots and so a fleet of robots could be flocked at the locaton represented by the AR object. In such a manner, t reduces the operators tme to complete the task, rather than controllng each robot ndvdually. VI. CONCLUSION Marker-less augmented realty has a great capacty to functon as a novel human-robot nterface. A 3D pont n the robots vcnty could be ndcated wth an AR object, whle assgnng a task to be fulflled by the robot at that pont. When combned wth Vsual SLAM, marker-less AR has the ablty to render a vrtual AR object n a readly avalable envronment. Such envronments are consdered more natural for robot operatons snce hghly controlled envronments wth fxed nstallatons are not always guaranteed. But most marker-less AR applcatons fnd t challengng to confer a persstent trackng behavor n such envronments as they fal to render the AR object frmly at ts orgnal locaton - specally when the camera vew changes n a large angle. Ths hnders the usefulness of marker-less AR for HRI. In order to address ths shortcomng, we present a system (.e. PTAGM) based on PTAMM. Rather than beng constraned by steady camera feld-of-vews, our system mantaned a persstent global pose throughout wder and changng camera perspectves, whle extendng PTAMM wth lnear transformatons. Havng our marker-less AR nterface n place, we show how t operates n the context of HRI by navgatng a ground robot va vrtual AR objects. Even though ths study focuses robot navgaton, expandng our approach to other areas of HRI, also seems plausble. REFERENCES [1] R. Castle, G. Klen, and D. W. Murray, Vdeo-rate localzaton n multple maps for wearable augmented realty, n Proc. 12th IEEE Internatonal Symposum on Wearable Computers, 2008, pp [2] G. Klen and D. Murray, Parallel trackng and mappng for small AR workspaces, n Proc. 6th IEEE and ACM Internatonal Symposum on Mxed and Augmented Realty, 2007, pp [3] J. Herlng and W. Broll, Chapter 11 markerless trackng for augmented realty, Handbook of Augmented Realty, B. Furht, Ed. New York, NY: Sprnger New York, 2011, pp [4] T. Lee and T. Höllerer, Multthreaded hybrd feature trackng for markerless augmented realty, IEEE Transactons on Vsualzaton and Computer Graphcs, vol. 15, no. 3, pp , [5] I. Gordon and D. Lowe, What and where: 3D object recognton wth accurate pose, Toward Category-Level Object Recognton - Lecture Notes n Computer Scence, vol. 4170/2006, pp , [6] R. O. Castle, G. Klen, and D. W. Murray, Combnng monoslam wth object recognton for scene augmentaton usng a wearable camera, Image and Vson Computng, vol. 28, no. 11, pp , Nov [7] G. Gn and A. March, Indoor robot navgaton wth sngle camera vson, n Pattern Recognton n Informaton Systems, [8] A. J. Davson, I. D. Red, N. D. Molton, and O. Stasse, MonoSLAM: Real-tme sngle camera SLAM, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 29, no. 6, pp , June Eranda Lakshantha receved hs B.Sc. degree n Informaton Technology from Unversty of Moratuwa, Sr Lanka, n He worked as a software engneer snce 2008 and s currently a Ph.D. Canddate n the Faculty of Informaton Technology at the Monash Unversty. In 2012, he was awarded wth a scholarshp for PhD study at Monash Unversty, Sunway Campus. Hs research nterests are n the areas of Augmented Realty, Human-Robot Interacton, and computer vson wth emphass on extendng AR asssted nterfaces for HRI navgaton and for other ecologes n HRI. Smon Egerton receved the PhD degree n Computer Scence from the Department of Computer Scence, Unversty of Essex and conducts research on ntellgent systems and bo-nspred robot control. He s a researcher who develops machne learnng frameworks and technques to advance the area of cogntve AI, appled to robotcs, smart devces and ther ecologes. He s the Deputy Head of School (Research) and leads the Intellgent Systems Research Laboratory. He s co-founder and drector of the Creatve Scence Foundaton, a charty dedcated to explorng the use of scence fcton as a means to motvate and drect research nto new technologes. He joned academa havng spent a perod of tme n ndustry where he was responsble for desgnng, buldng and mplementng embedded and real-tme systems Engneerng and Technology Publshng 255