COMPARISON BETWEEN LOOK-AND-MOVE AND VISUAL SERVO CONTROL USING SIFT TRANSFORMS IN EYE-IN-HAND MANIPULATOR SYSTEM

Transcription

1 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl COMPARISON BETWEEN LOO-AND-MOVE AND VISUAL SERVO CONTROL USING SIFT TRANSFORMS IN EYE-IN-HAND MANIPULATOR SYSTEM Ilana Ng, lang@gmal.om Mao Antono Meggolao, megg@pu-o.b Raul Queoz Fetosa, aul@ele.pu-o.b Pontfal Cathol Unvesty of Ro de Janeo, Rua Maquês de São Vente 5 Gávea, Ro de Janeo, RJ, Bazl Abstat. The pesent wok has the objetve to develop and mplement vsual ontol tehnques to self-loalze and poston obot manpulatos. It s assumed that a monoula amea s attahed to the obot end-effeto (eye-nhand onfguaton). Two lassal vsual ontol tehnques ae studed: look-and-move and vsual sevo ontol. The man ontbuton of ths wok s the use of the SIFT tansfom n these ontol tehnques to obtan and oelate keyponts between efeene mages and mages aptued n eal tme by the obot amea. The poposed methodology s expementally valdated usng a thee degee-of-feedom manpulato, wth psmat jonts and otatonal jont, espeally desgned and bult fo ths wok. The manpulato desgn s based on the mehanal stutue of an x-y- oodnate table, poweed by DC geamotos wth enodes, ontolled by a ompute though an eleton system, nludng ontol softwae whee all pesented methodologes ae mplemented. A monoula amea fxed to the obot end-effeto s able to aptue mages fom the envonment, used to ontol the elatve poston between the manpulato and a gene objet. Pelmnay tests ae pefomed on smple ula-shaped objets, wthout the need fo SIFT tansfoms. It s shown that t s possble to automatally ontol the elatve poston between the obot endeffeto and the tested ula objets. Next, tests ae pefomed wth a photo of an atual manfold panel typally used n submane nteventons. The loalzaton of the manpulato wth espet to the panel and ts valves s pefomed wth the ad of the SIFT tansfom, used to detemne key-ponts n the amea mages. The expemental esults show that the fequeny of the vsual sevo ontol s vey muh dependent on the mage poessng tme, as opposed to the look-and-move tehnque, whh only needs to aptue a sngle mage. On the othe hand, vsual sevo ontol pesents smalle steady-state postonng eos, beause the aptued mages when the manpulato s lose to ts desed poston ae used n the system feedbak to mpove n eal tme the obot auay. eywods: obot manpulato, eye-n-hand, look-and-move ontol, vsual sevo ontol, SIFT tansfom. INTRODUCTION Reent developments n obots and ompute vson have allowed the wdespead use n seveal aeas. Robot systems ae able to eplae humans n dangeous o epettve tasks, whle ompute vson allows suh obots to eognze objets, shapes and olos, usng ths nfomaton to exeute omplex tasks. One applaton of geat nteest s based on the use of ompute vson to albate and self-loalze a obot manpulato. Ths applaton an be useful n submane nteventons, whee a obot manpulato s mounted on a Remote Opeated Vehle (ROV) to exeute tasks at hgh depths, suh as handlng manfold valves. Suh task s uently pefomed by tele-opeatos. To patally automate ths task, the obot must be able to measue n eal tme ts pose wth espet to the seved equpment and ts omponents (suh as manfold valves). An applaton example s based on the TA-4 obot manpulato, used by Petobas n submane nteventons. Seveal woks wee pesented n the lteatue, ombnng obots wth ompute vson. The most ommon applaton onssts on a obot exeutng a task ommanded by vsual nfomaton. Inoue & Sha (97) bult a obot manpulato wth 7 degees of feedom, wth an eye-n-hand system. The objetve was to ft an objet n a hole of the same fomat. Wth the own softwae, the ontol estmated the dstane between the obot and the objet, and t eahed the objet only usng mage nfomaton. Allota & Colombo (999) desgned a obot eye-n-hand system. Vsual featues wee obtaned though edge-fndng. Usng a D/3D ontol, the system was able to pefom postonng tasks. Houshang (99) desgned a system wth a fxed amea, and developed a system to aptue movng objets. The pesent wok has the objetve to develop and mplement vsual ontol tehnques to self-loalze and poston obot manpulatos. It s assumed that a monoula amea s attahed to the obot end-effeto (eye-n-hand onfguaton). Two lassal vsual ontol tehnques ae studed: look-and-move and vsual sevo ontol. The man dffeene s elated to the adopted feedbak sensos. The fst tehnque uses poston sensos wth the ad of a sngle mage aptued at the begnnng of the obot movement. The seond tehnque, on the othe hand, does not make use of poston sensos, t only eles on seveal mages aptued n eal tme dung the obot movement. Eah of these tehnques an be mplemented aodng to two dffeent hoes fo state vaables: vaables based on pose (postons and oentatons), o vaables based on mage featues. When dealng wth pose vaables, a desed elatve pose between the amea and an objet s hosen; the obot s then ontolled untl suh poston and oentaton

2 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl s aheved. When dealng wth mage featues, the obot only eeves an mage assoated wth the desed poston, whle the ontol moves the manpulato untl ts end-effeto amea aptues an mage as smla as possble to the povded one. In ths wok, the SIFT tansfom (Lowe, 999) s used n these ontol tehnques to obtan and oelate key-ponts between efeene mages and mages aptued n eal tme by the obot amea. The SIFT tansfom s obust to otatons, tanslatons, sale and lghtng hanges, mpovng the ontol system obustness.. ANALYTICAL BACGROUND Next, some tehnques used n ths wok wll be pesented, neessay to undestand the expemental poedue... SIFT (Sale Invaant Featues Tansfom) The man objetve of the SIFT algothm (Lowe, 4) s the nvaant featues extaton fom mages, n ode to fnd mathng ponts between two mages. The featues ae nvaant to mage sale and otaton, and ae shown to povde obust mathng aoss a substantal ange of affne dstoton, hange n 3D vewpont, addton of nose, and hange n llumnaton. The esult of the SIFT method appled to an mage an be seen n Fg., whee the keyponts found by the algothm ae shown n blue, suh as the pont oentaton as defned by the algothm Fgue - The esult of the SIFT method appled n a mage Afte fndng the keyponts of the mage pa, the mathng poess stats. The oelaton between the mages s then found. The man objetve s to fnd the same pont n the dffeent vews of the objet. A mathng esult s shown n Fg.. Fgue Mathng esult between two mages

3 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl.. Contol Ahtetue The two ontol tehnques based on mages, whh wll be studed n ths wok, dffe eah othe wth espet to the system feedbak. Sandeson and Wess (98) ntodued two onepts to lassfy vsual-sevo systems. The look-andmove system uses vsual omputaton to geneate the set-ponts to the jonts fom a sngle mage, wthout vsual feedbak. On the othe hand, vsual-sevo systems do not make use of onventonal poston ontolle/sensos, usng nstead mages to oet fo the jonts eos. Eah of these tehnques an be mplemented aodng to two dffeent hoes fo state vaables: vaables based on pose (postons and oentatons), o vaables based on mage featues. When dealng wth pose vaables, a desed elatve pose between the amea and an objet s hosen; the obot s then ontolled untl suh poston and oentaton s aheved. When dealng wth mage featues, the obot only eeves an mage assoated wth the desed poston, whle the ontol moves the manpulato untl ts end-effeto amea aptues an mage as smla as possble to the povded one. The Jaoban matx s esponsble fo onvetng the desed and atual featues n nputs to the ontolle. nowng that vsual ontol an be lassfed fom the pesene o absene of a onventonal poston ontolle, and by the desed vaable (pose o mage), fou dffeent ontolles an be defned: look-and-move based on pose, lookand-move based on mage, vsual-sevo based on pose and vsual-sevo based on mage. In the look-and-move ontol, the system feedbak s ealzed n the jonts, usng poston sensos n the system feedbak. Vsual-sevo ontol does not use poston ontolles n the jonts. Feedbak s ealzed though mages aptued n eal tme. At eah ontol loop, a new mage fame s aptued and a new dffeene between the eal and desed poston s obtaned. Fgue 3. Look-and-Move ontol based on pose Fgue 4. Vsual Sevo ontol based on pose Fgue 5. Vsual Sevo ontol based on mage

4 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl.3. PID Contol One the desed poston s detemned, a poston ontol s neessay to tansfom the desed poston nto nfomaton to the motos. Many tehnques an be found n the lteatue, but fo ths wok PID ontol was hosen, one of the most ommon ontol tehnques. The PID ontol an be undestood as a ombnaton of thee dffeent tehnques: Popotonal, Integal and Devatve, followng the equaton u = P e + I e dt + t D e () whee epesents the lnk numbe, u s the esultng foe o toque to be appled by eah atuato, e s the eo between the eal and desed poston, P s the popotonal gan, gan. The gans wee albated fom expemental poedues. 3. EXPERIMENTAL SYSTEM The pojet sheme and man steps ae pesented n Fg. 6. s the ntegal gan and I s the devatve D Fgue 6. Shemat of the expemental system The poposed methodology s expementally valdated usng a thee degee-of-feedom manpulato, mplemented usng an automated x-y- oodnate table. A amea s fxed at the end-effeto of the table, extatng an mage fom the taget. Vson softwae omputes the desed oodnates fom the taget and sends them to an eleton system. A moontolle nsde the eleton system estmates the neessay toques to eah the desed poston, and sends t to the oodnate table motos. 3.. Mehanal System The automated oodnate table used n ths wok has psmat jonts and otatonal jont, poweed by DC geamotos wth enodes. A monoula amea fxed to the obot end-effeto s able to aptue mages fom the envonment, used to ontol the elatve poston between the manpulato end-effeto and a gene objet. An mage of the oodnate table s shown n Fg. 7. Fgue 7. x-y- oodnate table

5 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl 3.. Eleton System An eleton system ntefae has been developed to ontol the movement of the oodnate table motos usng a ompute. The eleton system ommunates wth the ompute though a seal pot. It ontans a moontolle esponsble to exeute a PID algothm and detemne the uents to be appled to the motos. The output sgnals ae of the PPM type (Pulse Poston Modulaton). To atvate the motos, speed ontolles Banebots BB-45 ae used, whh an povde A ontnuous and 45A peaks. Fo the ontols based on pose, ths eleton system eeves the nfomaton about a desed poston fom a ompute, ompaes t to the measued postons fom the moto enodes, and then alulates and sends sgnals to geneate toques to the motos aodng to a PID ontol law. Fo the mage-based ontols, the ompute s esponsble fo alulatng the eos between the desed mage and the aptued one, detly sendng the ontol sgnals to the eleton system, whh then ats only onvetng them to the PPM fomat Contol Softwae The Matlab softwae s hosen to mplement the ontol, beause of ts ompehensve lbay on mage poessng, n addton to ts smple-to-use ommunaton wth a seal pot. The man seen pesents a few buttons that allow the use to hoose the desed vaable to be ontolled between pose o mage, and among the fou ontol ombnatons (Look-and-Move based on pose, Look-and-Move based on mage, Vsual-Sevo based on pose and Vsual-Sevo based on mage). Two types of tagets ae used n the expements, one based on a smple ula objet, easly dentfable by the mage poessng softwae, and anothe based on a gene D mage, whh eques the use of SIFT. Both ae desbed next Cula taget To detemne the elatve dstane between a ula-shaped objet and the amea, a few equatons had to be developed based on geomet pnples. Fgue 8 shows a sheme of the expement usng a (ed) ds as a taget. Fgue 8. Expement shemat usng a ed ula taget The shemat pesented n Fg. 8 assumes that the ds axs of symmety s algned wth the y deton, whle x and y epesent the oodnate axes fom the expemental table. The angles s defned n the fgue as the angle between the lne jonng the amea ente and the ds ente and the x axs, whle the α angle s elated to the optal axs of the amea. The dstane d between the amea and the ds ente s also shown. The last paamete a epesents the dstane between the ds ente and the optal axs of the amea. The followng equatons an then be wtten x = d os () y = d sn a snα = d (3) (4)

6 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl ' = α (5) When the le s not enteed n the mage, t s possble to obseve two dffeent values fo and R, the smalle and the lage sem-axes n the mage, espetvely. A dstane a between the mage and the ds ente an be also obseved. Assumng that the amea and the ds entes ae always at the same vetal level, t s possble to affm that the lagest sem-axs wll only hange ts sze when the dstane n y s hanged. In othe wods, when the amea gets lose to the objet, the lage sem-axs wll beome lage n the mage, esultng n d = R a = a R (7) whee s onstant and s the ds atual adus. To fnd the ds otaton wth espet to the amea, t s possble to use the ato between the sem-axes (see Fg. 9): = = R sn sn (8) R (6) Fgue 9. Fontal and uppe vews fom the ds One the equatons above ae detemned, t s possble to fnd the values fo x, y and '. Fo the tehnques based on mage, t s neessay to wte the equatons fo the featue vaables, and 3. Fo ths wok, these vaables a wll be based on a, and R, and ae defned as: =, =, 3 =. Rewtng all the equatons, the R R R values of x, y and ' beome x = (9) y = () 3 ( ) ' = sn sn () It s possble now to wte n matx fom the elatonshp between the poston veto q and the featue veto. Fom by the paamete veto p defned below t s possble to fnd the tansfom mates J p and J qp : = J 3 p d a Sp and x d y = J qp () a ' q p

7 Poeedngs of COBEM 9 th Intenatonal Congess of Mehanal Engneeng Copyght 9 by ABCM Novembe 5-, 9, Gamado, RS, Bazl = 3 ' p J qp J y x (3) whee, = J p se (4) ( ) ( ) = = ' se ' tan sn os os sn d d d d J qp (5) nowng the eal and desed values of, and 3, t s possble to fnd ',, y x usng Eq Gene D Taget Fo the seond pat of the expements, nstead of usng a ed le, a gene D mage s hosen as a taget. The hosen mage eflets the vew fom a manpulato of a manfold ontol panel. Usng the SIFT method, the keyponts n the mages ae obtaned. Fst, SIFT s appled to a efeene mage fom a known poston of the amea, obtanng the oodnates of the keyponts n spae. Wth these efeene oodnates, t s possble to fnd the oodnates fom the same ponts, found by a mathng tehnque, n mages taken fom any othe poston. The expement shemat s shown n Fg., whee x epesents the atual dstane between the keyponts and the ente of the ontol panel and x epesents the dstane n pxel oodnates fom the pojeton to the ente of the mage. Fgue. Expement shemat usng gene D objets as tagets D Objet Cente ABCM Symposum Sees n Mehatons - Vol. 4 - pp Copyght by ABCM

8 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl One the keypont spae and pxel oodnates ae detemned, and knowng that f s the foal length, one an get ( x + x) snα + y osα ( x + x) and onsequently f = y tanα x tanα osα y [ f x x x ] x tanα + y = [ f X ] X x Fo M pas of ponts ( ) f x xx f x x x X = f xm xm xm A x snα x,, and usng the pseudo-nvese fomulaton, the X veto an be detemned by x x f A X = B X = pnv A xm B ( ) B One found the veto X, the desed dstanes ( x y, α ), an be detemned. (8) (6) (7) 4. RESULTS Two dffeent types of tests wee pefomed. The fst tests wee pefomed usng a ed le as taget, and the seond usng the mage of the manfold panel, epesentng a eal applaton. The le tests used mages wth 96 x 7 pxels. The panel test, on the othe hand, used a lowe esoluton beause the SIFT algothm s elatvely slow, onsdeably neasng the poessng tme (5 seonds to poess a 96 x 7 pxel mage n Matlab). As the vsual sevo ontol depends detly on the poessng tme, the mage sze was hanged to 35 x 88 pxels. In look-and-move ontol, howeve, the mage ould be used n hgh esoluton beause ths tehnque only needs to poess a sngle mage. In the fst test, t was desed to poston the amea m n the X axs, m n the Y axs, and wth no otaton. In ths poston, t would be desed to see n the mage = R = 95 pxels, and a = pxels. The amea stats n x = m, y = m and =. The ntal mage fom the amea and the desed mage ae ndated n Fg.. Fgue. Intal mage (left) and fnal desed mage (ght) Table ndates the eahed poston ( x, y, ) fo eah ontol tehnque. Fo look-and-move ontol, the elatve poston ( x el, y el, el ) found by the softwae fom the fst aptued mage s also ndated. Ths elatve poston s not shown fo vsual sevo ontol, beause t s ontnually hanged at eah new mage that s poessed. The last 3 lnes epesent the fnal values of the featues a, and R, whh an be ompaed to the desed values 95,95 and dsussed above.

9 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl Table. Fnal and desed postons fo the fou ontol tehnques Look-and-Move Contol based on pose Look-and-Move Contol based on mage x. m 7.9 m el y 8. m 8.3 m el 3 el Vsual-Sevo Contol based on pose Vsual-Sevo Contol based on mage 48 nteatons nteatons x.7 m 9. m. m 5.4 m y 8. m 8.4 m.3 m.5 m 3 79 pxels 83 pxels pxels 89 pxels R a 8 pxels 83 pxels pxels 95 pxels 5 pxels -37 pxels - pxels -37 pxels Note fom the table that the vsual-sevo ontol based on pose obtaned the best postonng esult. The look-andmove ontol s not vey auate beause t uses a sngle mage taken fom the statng poston of the manpulato. Note also that the ontols based on mage featues obtaned dffeent fnal postons fom the desed ones. Ths happened beause the hosen fnal onfguaton was lose to a sngulaty of the mage Jaoban matx. Theefoe, the fnal mage aptues by the amea was atually vey smla fom the desed one, howeve t was taken fom a dffeent pose. Futhe tests wth desed fnal postons away fom ths sngulaty esulted n bette esults fo the mage-based ontols, smla to the pose-based ones. Fo the panel test, t was desed to poston the amea n x=8m, y=5m, and =3. The ntal and desed vews ae pesented n Fg.. The poston esults ae pesented n the Table. Fgue. Intal mage (left) and fnal desed mage (ght) Table. Fnal and desed postons fo the fou ontol tehnques Look-and-Move Contol based on pose Look-and-Move Contol based on mage x 6.6 m 6.6 m el y 5.9 m 9 m el Vsual-Sevo Contol based on pose Vsual-Sevo Contol based on mage 3 nteatons 5 nteatons el x 6.8 m 6.8 m 6.5 m 7. m y 6.3 m 9.4 m 7 m 6.6 m

10 Copyght Poeedngs of COBEM by ABCM 9 Copyght 9 by ABCM th Intenatonal Congess of Mehanal Engneeng Novembe 5-, 9, Gamado, RS, Bazl Table shows that fo look-and-move ontol the bggest eo soue was due to the lmtatons n the esoluton of the mage. The elatve values x el, y el and el estmated fom a sngle mage pesented sgnfant eos of up to 6%, howeve the poston ontol was able to move the amea to atual postons x, y and wthn % of x el, y el and el. 5. CONCLUSIONS The man objetve of ths wok onssted n ompang the look-and-move and vsual-sevo mage ontol tehnques. Usng the SIFT algothm t was possble to apply the vsual ontol to poston the amea n any pose wth espet to a gene D mage. Even though t s vey used n the lteatue, SIFT s stll a slow algothm when usng t n eal tme. Wth the ed le tests, t was possble to see how vsual-sevo s bette than look-and-move when the poessng tme s not an ssue. Contols based on pose and on mage had vey smla esults, exept fo onfguatons lose to sngulates n the mage Jaoban matx. 6. REFERENCES Allota, B., & Colombo, C. (999). On the use of lnea amea-objet nteaton models n vsualsevong. IEEE Tansatton on Robots and Automaton, pp Houshang, N. (99). Contol of a obot manpulato to gasp a movng taget usng vson. IEEE ntenatonal Confeene on Robotts and Automaton, pp vol.. Inoue, H., & Sha, Y. (97). Gudng a obot by vsual feedbak assemblng tasks. Eletotehnal Laboatoy Chyoda-ku Tokyo, Japan. Lowe, D. (999). Objet Reognton fom Loal SaleInvaant Featues. ICCV, ekya. Sandeson, A., & Wess, L. (98). Image-based vsual sevo ontol usng elatonal gaph eo sgnals. Po. IEEE, pp RESPONSIBILITY NOTICE The autho(s) s (ae) the only esponsble fo the pnted mateal nluded n ths pape.