Cooperative Human and Machine Perception in Teleoperated Assembly

Size: px

Start display at page:

Download "Cooperative Human and Machine Perception in Teleoperated Assembly"

Dylan Conley
6 years ago
Views:

1 Expermental Robotcs VII. The Ffth Internatonal Symposum, Honolulu, HI, December 10-13, 2000, D. Rus and S. Sngh, eds., Lecture notes n control and nformaton scences, Sprnger-Verlag. Cooperatve Human and Machne Percepton n Teleoperated Assembly Thomas Debus and Jeffrey Stoll Boston Unversty Boston. MA tdebus@bu.edu and jstoll@bu.edu Robert D. Howe Harvard Unversty Cambrdge, MA howe@deas.harvard.edu Perre Dupont Boston Unversty Boston. MA perre@bu.edu Abstract: Ths paper presents results on a teleoperator expert assstant a system that n cooperaton wth a human operator estmates propertes of remote envronment objects n order to mprove task performance. Specfcally, an undersea connector-matng task s nvestgated n the laboratory usng a PHANToM master and WAM remote manpulator. Estmates of socket orentaton are obtaned durng task performance and conveyed to the operator through a graphcal dsplay. Task performance, measured by completon tme and peak nserton force, s compared for operators usng combnatons of vdeo mages, the graphcal dsplay and a shared control mode n whch the connector automatcally rotates to the estmated socket orentaton. The graphcal dsplay and automatc orentaton controller reduce task completon tmes and contact forces by over one-thrd for nclned sockets when the vdeo sgnal s nosy, e.g., due to water turbdty. 1 Introducton At present, teleoperaton s the only way that robots can perform sophstcated manpulaton tasks n unstructured envronments. In ths control mode, the human operator performs all requred sensng and plannng, and generates all moton commands based on feedback from the remote envronment. In practcal teleoperaton systems (e.g. undersea operatons [1] and surgery [2]), the sensory feedback s often lmted to vdeo mages wthout force feedback, whch greatly restrcts dexterty and productvty. We have been workng to allevate ths stuaton by usng nformaton from the remote robot arm's sensors to assst n teleoperated manpulaton tasks [3][4][5]. We have derved algorthms that dentfy essental propertes of objects n the remote envronment ncludng geometry, mass, complance, and frcton. 1

2 In ths paper, we focus on a specfc practcal applcaton, undersea connector matng, where a cylndrcal connector s nserted nto a socket [1]. Ths peg-n-hole task s commonly performed n the offshore ol ndustry to provde hydraulc or electrcal power to equpment on the sea floor. There are a number of factors that make ths task troublesome. Frst, the clearance between the connector and receptacle s small, so a few degrees of angular msalgnment can cause the connector to become jammed [6]. Second, sensory feedback s lmted: because of strngent cost and relablty requrements, the master control devce does not provde force feedback (Fgure 1). Vsual nformaton s also restrcted to monocular cameras that may be obstructed by sedment from nearby drllng operatons. Thrd, manpulator arms for ths applcaton are hydraulcally drven, wth lttle complance that would facltate nsertons. All these lmtatons make tmely completon of ths task dffcult; n some cases t can take over an hour to nsert a sngle connector, resultng n sgnfcant costs for the ol platform operator. For a soluton to fnd acceptance wth the robot vendor and offshore operators, t must nvolve a mnmum of modfcaton to the robot, ts controller and ts software, as well as meetng rgorous relablty constrants. Thus, tradtonal robotc solutons to the nserton problem, such as mechancal or programmed remote center of complance, are precluded. The deal soluton s one that can be mplemented on an exstng robot nstallaton. Our proposed soluton s to use the nformaton from the jont angle sensors on the remote manpulator arm to determne the key geometrc parameters of the remote envronment. Specfcally, we have developed an algorthm that automatcally determnes the relatve orentaton of the connector and the socket durng the nserton task. The process begns as the operator brngs the connector nto contact wth the planar surface surroundng the openng of the socket, and sldes the connector over the surface near the openng. The recorded jont sensor values are then combned wth a model of the contact constrant between the cylndrcal connector and the planar surface. Soluton of the constrant equatons yelds a value for the orentaton of the planar surface. Ths new nformaton s presented to the operator as a graphcal dsplay to assst n orentng the connector. To explore the performance lmtatons mposed by the mnmum modfcaton rule, we have also mplemented a shared control mode. In ths approach, the connector s automatcally orented accordng to the estmated socket axs, whle the operator controls translaton to complete the nserton. Ths approach does not nvolve modfcaton of the teleoperator system, but only access to jont encoder values. In ths paper, we present expermental results confrmng the benefts of the approach. 2 Methods The laboratory teleoperator testbed used n these experments conssts of a PHANToM haptc nterface as the master controller and a Barrett Whole Arm Manpulator (WAM) as the remote robot (Fgure 2). To emulate the undersea applcaton, the PHANToM (Model 1.5, Sensable Technologes, Cambrdge, Mass., USA) s used as a passve 6 degree of freedom nput devce, and the motors are not actvated. The WAM (Barrett Technologes, Cambrdge, Mass., USA) s a redundant arm wth 7 degrees of freedom, but only 5 axes are requred for ths nserton task, so the upper arm roll and fnal wrst roll axes are locked. Optcal encoders measure the 2

jont poston on both robots, and veloctes are computed usng fltered backward dfferences. The workspace s roughly 0.2 m n dameter for the master robot and 1.0 m n dameter for the remote robot.

The PHANToM jont data s read by a PC at a rate of 1 khz and wrtten nto memory shared by the PC and the RISC processor. Fgure 1.

3 jont poston on both robots, and veloctes are computed usng fltered backward dfferences. The workspace s roughly 0.2 m n dameter for the master robot and 1.0 m n dameter for the remote robot. The WAM robot s controlled by a dedcated RISC processor (Model DS1103, dspace GmbH, Paderborn, Germany) runnng at a 10 khz servo rate. The PHANToM jont data s read by a PC at a rate of 1 khz and wrtten nto memory shared by the PC and the RISC processor. Fgure 1. Schllng Robotc Systems Ttan II manpulator, one of the leadng commercal robots for undersea applcatons. (a) Remote manpulator arm s hydraulcally powered. (b) The passve master arm provdes no force feedback. Fgure 2. (a) WAM remote robot arm wth connector matng apparatus. (b) PHANToM master arm. Teleoperaton s accomplshed wth a smple proportonal-dervatve controller wth feedforward gravty and motor torque rpple compensaton on the arm and ntegral feedback on the wrst. In ths control method, ncremental Cartesan poston, velocty and orentaton of the master robot are mapped to the remote workspace, converted to remote robot jont postons and velocty usng nverse knematcs, and then to torque commands by the followng control laws. τ ( θ θ ) ( θ θ ) τ τ, arm remote master remote master remote gravty rpple KP KV τ K ( θ θ ) K ( θ θ ) K ( θ θ ) dt, wrst remote master remote master remote master remote P V I ( 1) gravty Here θ, θ, τ, τ and τ rpple are the th components of jont poston, velocty, torque, gravty compensaton and motor torque rpple compensaton. 3

4 2.1 Envronment modelng and operator assstance. To fnd the orentaton of the socket axs, we use technques developed n our prevous work [4],[5]. Ths approach estmates the remote object parameters by combnng constrant equatons descrbng the geometry of the contactng surfaces, closed loop knematc relatons, and knematc data from the remote robot s jont sensors. In the present case, we are concerned wth descrbng the contact pont as the operator sldes the cylndrcal connector over the planar surface surroundng the socket. We assume that the connector s rgdly grpped by the remote robot n a tool T known confguraton. The contact pont p = [ xyz,, ] on the connector s constraned to le along ts outer crcular rm, whch may be wrtten n tool coordnates as x + y r = 0, z l = 0 ( 2) where r and l are the known peg length and radus. The planar surface contact pont world T p = uvw,, may be wrtten n world coordnates as [ ] au + bv + cw + d = 0 ( 3) where a, b, c, and d are the desred parameters descrbng the orentaton of the socket surface; we assume that the socket axs s normal to ths surface, so that estmaton of these parameters yelds the correct connector orentaton for nserton. These descrptons of the contact pont p are related by the knematc transformaton between tool and world frames ( ) T θ p = p ( 4) tool tool world world As the connector sldes over the planar surface, the jont postons θ are tool sampled, from whch the transformaton Tworld ( θ ) s calculated. Smultaneous soluton of the above equatons provdes an estmate of the plane parameters a, b, c, and d and thus the correct connector orentaton. Ths typcally took 15 seconds to accomplsh and 25 ponts were selected for the least squares estmaton process usng master-remote poston error to nfer contact. The output s an estmate of the socket orentaton for the nserton task. We are nvestgatng a number of methods for conveyng ths nformaton to the operator. The frst s a 3D graphcal model showng the manpulator, connector, and estmated plane locaton and orentaton (Fgure 3a), whch s dsplayed on the montor of the PHANToM WAM nterface computer. The model moves n real tme to reflect the WAM s moton, and the operator can select the optmum renderng angle and dstance. The second dsplay shows a par of targetng crcles (Fgure 3b). The larger crcle represents the proxmal end of the connector and the smaller crcle the dstal end, as projected onto the socket plane. The operator rotates the connector untl the crcles are concentrc, whch corresponds to the correct orentaton, then proceeds wth the nserton. 4

5 In addton to these dsplays, we have mplemented a shared control mode. Followng the estmaton procedure, the orentaton of the connector s automatcally drven to the estmated socket angle, and orentaton changes at the PHANToM stylus are gnored. The operator retans control of translatonal moton of the remote robot to perform the connector nserton. Addtonal dsplay and shared control methods under development are descrbed n the Dscusson secton below. WAM Orthogonal vew WAM Top vew Proxmal end of connector Ptch Plane estmate Dstal end of connector Yaw Fgure 3. Graphcal dsplay (a) WAM model and (b) target crcles, whch are peg ends projected on the estmated socket plane. 2.2 Expermental protocol The connector-socket apparatus was smulated by a par of PVC plastc tubes (Fgure 2a). The connector was 50 mm n dameter and 315 mm long. The socket had an nsde dameter of 53 mm, and was mounted perpendcular to a planar surface that could be pvoted to a range of nclnaton angles between 0 and 57 degrees from horzontal. The operators used a montor to vew vsual feedback from the remote robot va a vdeo camera mounted adjacent to the shoulder jont of the WAM (Fgure 4). Socket WAM wrst Connector Fgure 4. Typcal vsual feedback to operators from remote robot, showng the WAM wrst, connector, and socket openng. (a) Clear vdeo sgnal. (b) Nosy vdeo sgnal produced by defocusng vdeo lens. 5

6 Two operators experenced n use of the system performed the nserton task under three dfferent control modes and two vsblty condtons, for a range of socket orentatons. The frst control mode provded only vsual feedback wth no estmaton of the socket orentaton, as n the current undersea connector matng task. The second mode added the two graphcal dsplays of the estmated socket orentaton descrbed above, and the thrd used the shared control mode to automatcally set the connector orentaton n addton to the vdeo and graphcal dsplays. To smulate poor undersea vsblty condtons, n some trals the vdeo sgnal was degraded by defocusng, as shown n Fgure 4(b). Each operator performed the task fve tmes n each combnaton of control mode, vsblty, and socket orentaton. In each tral that nvolved the estmaton algorthm, the operator brought the end of the connector nto contact wth the planar surface surroundng the socket openng. The operator then depressed a swtch that actvated recordng of jont angle data, and sld the connector over the surface. After releasng the swtch, the operator proceeded wth the nserton task. The tme requred for the estmaton process and for the rest of the nserton task was recorded. In addton, the forces produced n the nserton process were approxmated by recordng the poston errors at the remote robot and multplyng by the controller gan. Ths gnores nertal, mpact, and frctonal forces, but because of the clean drve tran of the WAM robot and the quas-statc nature of the nserton task, t provdes a reasonable estmate of the contact forces generated n the task. 3 Experment results In ntal tests, we assessed the estmaton algorthm s accuracy n determnng the orentaton of the planar surface around the socket openng. Fve trals were conducted at fve values of the orentaton, at 0, 11, 30, 41 and 57 degrees. Fgure 5 compares the estmated and actual orentatons. These results show good agreement between the estmated and actual angles: the largest error s 1.3 degrees, and the largest standard devaton of the estmates at each angle s 0.6 degrees. 60 ESTIMATED ANGLE (degrees) ANGLE (degrees) Fgure 5. Estmates of the orentaton of the planar surface around the socket openng. 6

7 Fgure 6 and Fgure 7 show the tme-to-completon results for the connectormatng task for the three control modes and two vsblty condtons at two surface orentatons. For the two modes that used the estmaton algorthm (.e., graphc dsplay and auto-orentaton) separate results are ndcated for the tme to perform the estmate, the tme to perform the nserton, and the sum of these, whch s the total tme to perform the task. Fgure 6. Task completon tme as a functon of control modes and vsblty condtons for a horzontal surface. Symbols ndcate mean; bars ndcate standard devatons. Fgure 7. Task completon tme as a functon of control modes and vsblty condtons for a surface nclned at 57 degrees. 7

8 As the angle of nclnaton was decreased from 57 degrees to horzontal, task completon tme decreased unformly. Ths s due to the assstance that gravty provdes n correctly orentng the relatvely complant WAM wrst for a non-contact surface (Wrst complance s due to tendon drve elastcty as well as delberately reduced feedback gans, the latter n order to avod damage-nducng contact forces). In addton, the steeper surface was less ntutve for the operators, especally durng the estmaton phase These results ndcate that, not surprsngly, the task takes sgnfcantly less tme wth clear vdeo feedback. The more pertnent comparsons are wthn the clear and nosy vdeo feedback condtons. Wth good vsblty, task completon tme s comparable wth and wthout estmaton algorthm. Ths s because executng the sldng moton that produces the data for the estmaton algorthm takes approxmately 40% of the task completon tme. The nserton phase tself was 30-50% faster wth ether graphcal or shared control assstance from the estmated angle compared to vsual feedback only. The effects of the estmaton modes are more evdent n the cases wth poor vsblty. Wth the socket plane nclned at 57 degrees, the entre task, ncludng estmaton and nserton, was 38% faster than when usng vsual nformaton alone. For a horzontal socket plane, estmaton ncreased total task tme, whle the nserton phase was consderably faster. For the non-zero nclnaton cases that used estmaton, the shared control mode was unformly faster than the graphc dsplay mode. Fgure 8 shows estmates of the normalzed peak force levels generated durng the task for a socket plane nclned at 57 degrees. Normalzaton was based on vdeo-only nserton force. As wth the task completon tme measure, the man dfference of nterest s among modes wth poor vsblty; the good vsblty case produces the lowest forces. Both of the estmaton-based modes produced peak forces over a thrd lower than the nosy vdeo only mode. Fgure 8. Normalzed peak force estmates as a functon of control mode for a socket plane nclned at 57 degrees. 8

9 4 Dscusson These results demonstrate that parameter estmaton based on remote robot knematc sensors can produce useful nformaton for teleoperated manpulaton tasks. Accuracy of surface orentaton estmates was wthn 1.6 degrees, wth a standard devaton of less than 0.6 degrees. Operators were able to accomplsh the nserton phase of the task far more quckly wth the estmaton-based modes n all cases. These results ndcate the vablty of the technque and suggest a clear beneft for the ntended applcaton. The approach has a number of features that make t well suted to the undersea applcaton. The estmaton algorthm s based on jont angle sensors only, whch are standard components of the manpulators used n offshore applcatons. The algorthm does not requre relatvely expensve and fragle force sensng (although force nformaton may mprove the estmates and enable dentfcaton of addtonal propertes [3]). Ths means that t s possble to confgure a system that looks over the shoulder of the operator, wthout nterferng wth normal task executon and wthout sgnfcant modfcatons to the exstng manpulator. If a graphcal dsplay s used to communcate wth the operator, the computer that executes the estmaton algorthm smply montors the remote robot s jont sensors. If the estmaton system fals, t has no mpact on the manpulator s capabltes, so operatons can contnue as wth the orgnal manpulator system. Ths s an essental consderaton for the offshore ol ndustry, where manpulator downtme can ncur large costs. Several mportant ssues must be addressed pror to undersea mplementaton. One mportant ssue s the dfference n complance between the WAM and undersea teleoperated robots, whch are hydraulcally powered, resultng n hgh endpont mpedance. Hgh mpedance makes nsertons partcularly dffcult because the connector does not conform to contact forces and torques generated by msalgnments. The relatvely hgh complance of the WAM resulted n much faster nsertons than commonly observed n the undersea applcaton. The low complance of the hydraulc undersea manpulators makes the proposed estmatonbased approach partcularly benefcal, but t also makes t more dffcult for the operator to slde the connector smoothly over the surface surroundng the socket. Further work wll be drected at ncreasng the effectve stffness of the WAM arm to permt better nvestgaton of these competng ssues. Ths applcaton uses only a porton of the technques we have developed for dentfyng the propertes of remote envronments durng teleoperaton [3],[4],[5]. In addton to estmatng a varety of propertes, these technques can segment the data stream from the robot sensors and determne whch contact states are actve at each tme. Ths enables a much greater range of applcatons. For example, the assumptons n the model, such as known length and radus of the connector, and perpendcular orentaton between the socket axs and the surroundng surface, can be relaxed. A smlar approach can be used to estmate other useful geometrc parameters, such as the locaton of the socket openng. Such parameters can be presented to the operator usng a varety of dsplay modaltes. Alternatvely, they can be used to effect sophstcated shared controlled strateges. Ths capablty promses to not only smplfy teleoperaton, but may represent a new level of perceptual capablty for autonomous manpulaton as well. 9

10 Acknowledgements The support of the Natonal Scence Foundaton under grant IIS s gratefully acknowledged. References [1] Dennerlen J.T., Mllman P., and Howe R.D. (1997). Vbrotactle Feedback for Industral Telemanpulators. Sxth Annual Symposum on Haptc Interfaces for Vrtual Envronment and Teleoperator Systems, ASME Internatonal Mechancal Engneerng Congress and Exposton, Dallas, Nov , 1997, DSC-Vol. 61, pp [2] Hll J.W, Jensen J.F. (1999). Telepresence technology n medcne: Prncples and applcatons. Proc. IEEE, 86(3): [3] Dupont P., Schultes T., Mllman P., and Howe R.D. (1999). Automatc Identfcaton of Envronment Haptc Propertes. Presence 8(4): [4] Debus T., Dupont P., and Howe R.D. (1999). Automatc Property Identfcaton va Parameterzed Constrants. Proc IEEE Conference on Robotcs and Automaton, Detrot, MI, May, p [5] Debus T., Dupont P., and Howe R.D. (2000). Automatc Identfcaton of Local Geometrc Propertes Durng Teleoperaton. Proc IEEE Conference on Robotcs and Automaton, San Francsco, CA, Aprl, p [6] Whtney DE (1982). Quas-Statc Assembly of Complantly Supported Rgd Parts, ASME Journal of Dynamc Systems, Measurements, and Control 104:65. 10

SLAM Summer School 2006 Practical 2: SLAM using Monocular Vision

SLAM Summer School 2006 Practical 2: SLAM using Monocular Vision SLAM Summer School 2006 Practcal 2: SLAM usng Monocular Vson Javer Cvera, Unversty of Zaragoza Andrew J. Davson, Imperal College London J.M.M Montel, Unversty of Zaragoza. josemar@unzar.es, jcvera@unzar.es,