Intelligent telerobotic assistance for enhancing manipulation capabilities of persons with disabilities

Transcription

1 Univesity of South Floida Schola Commons Gaduate Theses and Dissetations Gaduate School 4 Intelligent teleobotic assistance fo enhancing manipulation capabilities of pesons with disabilities Wentao, Yu Univesity of South Floida Follow this and additional woks at: Pat of the Ameican Studies Commons Schola Commons Citation Yu, Wentao,, "Intelligent teleobotic assistance fo enhancing manipulation capabilities of pesons with disabilities" (4). Gaduate Theses and Dissetations. This Dissetation is bought to you fo fee and open access by the Gaduate School at Schola Commons. It has been accepted fo inclusion in Gaduate Theses and Dissetations by an authoized administato of Schola Commons. Fo moe infomation, please contact scholacommons@usf.edu.

2 Intelligent Teleobotic Assistance Fo Enhancing Manipulation Capabilities Of Pesons With Disabilities by Wentao Yu A dissetation submitted in patial fulfillment of the equiements fo the degee of Docto of Philosophy Depatment of Mechanical Engineeing College of Engineeing Univesity of South Floida Majo Pofesso: Rajiv V. Dubey, Ph.D. Glen Bestefield, Ph.D. Daniel Hess, Ph.D. Shuh-Jing Ying, Ph.D. Wilfido A. Moeno, Ph.D. A.N.V. Rao, Ph.D. Date of Appoval: August 11, 4 Keywods: Rehabilitation, Hidden Makov Model, Motion Intention Recognition, Vitual Fixtue, Skill Leaning, Theapy Copyight 4, Wentao Yu

3 Acknowledgements Fist, I would like to thank D. Rajiv Dubey fo being a geat pofesso and giving me the oppotunity to pusue eseach wok in an aea that I enjoy; I will neve foget the oppotunity that you gave me. I would also like to thank D. Glen Bestefield, D. Daniel Hess, D. Shu-Jing Ying, D. Moeno and D. Rao fo seving on my advisoy committee. I would like to thank Aaon Gage fo setting up the Ghost SDK fo the PHANToM, and making some softwae fo ou system. I would like to thank Noali Penalete fo making such an inteesting pevious eseach fo the lab. Michael Juczyk povided invaluable assistance in setting up the hadwae. He got involved with the vision system getting the camea to wok. His wok led to the Halcon softwae configuation. I would like to thank Dwayne Polze fo attaching the sensos on the end-effecto. I would like to thank Redwan Alqasemi fo being a good fiend and making caeful coection on my dissetation. Thank you fo you time and had wok. I would also like to acknowledge the financial suppot fo the ehabilitation obotics eseach which I have been involved in the past few yeas. Stephen Sundaao and his Rehabilitation Engineeing and Technology Pogam povided the funding fo eseach wok I did in this lab. Finally, my thanks go to the lab membes that I have met: Ben Fitz, Ed McCaffey, Sashi Konda, Ashwin Upadhyay, Kevin Edwads etc.

4 Table of Contents List of Tables List of Figues Abstact iv v ix Chapte 1: Intoduction Motivation Dissetation Objectives 1.3. Dissetation Outline 3 Chapte : Backgound 5.1. Rehabilitation Robotics 5.. Teleobotics 1.3. Teleopeation Assistance Backgound Regulation of Positions Regulation of Velocities 16 Chapte 3: Teleopeation with Assistance Functions 3.1. Intoduction 3.. Assistance Functions Concept Box and Blocks Task 3.4. Senso Assist Function Desciption Stage One Stage Two Stage Thee Stage Fou Stage Five Stage Six Stage Seven Expeimental Results Telemanipulation System Stuctue Softwae Implementation Results Simulation Mode Real Test Mode 34 i

5 3.6. Summay 36 Chapte 4: Telemanipulation Assistance Based on Motion Intention Recognition Telemanipulation Assistance Classes of Motion in Telemanipulation Hidden Makov Model Based Motion Recognition Data Pepocessing Vecto Quantization HMM Taining Motion Recognition Design of Fixtue Assistance Fixtue Assistance Foce Field Design fo Tagets and Obstacles Expeiments Expeimental Test Bed Expeimental Results Without Assistance Motion Recognition Expeiment Results with Assistance Based on Motion Intention Recognition Summay 66 Chapte 5: Robotic Theapy fo Pesons with Disabilities Using Skill Leaning Motion Theapy Hidden Makov Model Based Skill Leaning Raw-data Convesion Hidden Makov Model Computation Expeiments in Vitual Envionment Tasks and Expeimental Test-Bed Skill Leaning and Tansfeing Motion Theapy Expeiments Motion Pefomance befoe Theapy Taining Motion Pefomance afte Theapy Taining Summay 9 Chapte 6: Conclusions and Recommendations Dissetation Oveview Vitual Fixtue Assistance Based on Motion Intention Robot Theapy and its Effectiveness Geneal Discussion Recommendations 94 Refeences 95 Appendices 14 ii

6 Appendix A: System Testbed and Expeiment Design 15 A.1. Intoduction 15 A.. Hadwae 15 A..1. Robotics Reseach Copoation Manipulato 15 A... PHANTOM Pemium A.3. Softwae 11 A.3.1. R Contolle Pogam 11 A.3.. HALCON Compute Vision Softwae 11 A.3.3. Teleobot Contol Inteface 11 A.3.4. Teleopeation System Achitectue 113 A.4. RRC GUI 113 A.4.1. Safe Opeating Instuctions 115 A Simulation Mode 115 A Robot Mode 117 A.4.. Jog Contol 118 A.4.3. Position Feedback 1 A.4.4. Teach Pendant 1 A.4.5. Pogam Contol 11 A.4.6. Client-Seve Inteface 14 Appendix B: Visual Sevoing fo Gasping 16 B.1. Configuation of Vision System 16 B.. 3D Pose Detemination of Taget with Respect to End-effecto 17 B.3. Visual Sevo Contolle Design 136 B.4. Tele-autonomy Design 138 About the Autho End Page iii

7 List of Tables Table 3.1 Compaison of Aveages fo Box and Blocks Test Using Wokspace Constaint 36 Table 4.1 Pefomance Summay without Assistance 63 Table 4. Motion Recognition Rate 64 Table 4.3 Pefomance Summaies with Assistance 66 Table 5.1 Movement Pefomance Summay 89 Table A.1 Joint Limits fo the RRC Manipulato 17 Table A. Phantom Pemium 1.5 Specifications 19 iv

8 List of Figues Figue.1 RAID Wokstation 6 Figue. Manus Manipulato 6 Figue.3 Rapto Manipulato 8 Figue.4 (a) MIT-MANUS[85], (b) MIME[16] 9 Figue.5 Tele-autonomy is the Combination of Teleopeation and Autonomy 13 Figue.6 Tele-collaboation with Infomation Feedback 13 Figue.7 Human-machine Coopeative Teleopeation Concept [9] 15 Figue.8 Repesentation of Slave Constaint in the Constaint Plane[67] 15 Figue.9 Scaling Facto Function [53] 16 Figue.1 Scaling Facto Vaying fo Appoach [9] 17 Figue.11 Coodinate Fames fo Coss Alignment task [9] 17 Figue.1 Two Types of Refeence Diection Fixtues [55] 19 Figue.13 Vitual Fixtues to Aid Extact / Inset Motion [69] 19 Figue 3.1 Box and Blocks Test Window Inteface 3 Figue 3. Box and Blocks Test, Maste and Slave 3 Figue 3.3 Teleopeation Testbed 4 Figue 3.4 Sensos Mounted on End-Effecto 4 Figue 3.5 The Seven Stages of the Scaling Scheme 5 v

9 Figue 3.6 Image Fame Showing Vecto Detemination 6 Figue 3.7 ScaleFacto Accoding to LRF Data (DME) 8 Figue 3.8 The Telemanipulation System 31 Figue 3.9 Region Gowing Image 3 Figue 3.1 Sobel Edge Detection Image 3 Figue 3.11 Tajectoy Compaison of PhanTom and Slave Manipulato 33 Figue 3.1 Box and Block Time Execution 34 Figue 3.13 Tajectoy of Box and Blocks Task 35 Figue 4.1 Path Following Motion and its Velocities Pofile 4 Figue 4. Aligning with Taget Motion and its Velocities Pofile 4 Figue 4.3 Avoiding Obstacle Motion and its Velocities Pofile 4 Figue 4.4 Opeation Stopping its Velocities Pofile 41 Figue 4.5 Convesion of Continuous Velocity Data to Discete Symbols 43 Figue 4.6 LBG Codebook Taining 46 Figue 4.7 Figue 4.8 LBG Vecto Quantization fo Some Random D Data, as L Equals,4,8,16, states Left-ight Hidden Makov Model, with 3 Obsevable Symbols in Each State 48 Figue 4.9 Fowad Computation Illustation 55 Figue 4.1 Vitual Fixtue Definition 57 Figue 4.11 Stiffness Coefficients of Diffeent Fixtues 59 Figue 4.1 Foce Fields Illustation (a: Attactive foce, b: Repulsive foce) 6 Figue 4.13 Simulation of the Task Execution 61 vi

10 Figue 4.14 Velocity Components without Assistance 6 Figue 4.15 Tajectoies without Assistance 63 Figue 4.16 Velocity Components with Assistance 65 Figue 4.17 Tajectoies with Assistance 66 Figue 5.1 Raw-data Vectos 71 Figue 5. PSD Vectos 71 Figue 5.3 Vecto Quantization When Codebook Length is 4 7 Figue 5.4 Two-state Left-ight Hidden Makov Model 73 Figue 5.5 Hidden Makov Model with the Adjusted Paametes 74 Figue 5.6 Vitual Envionment fo Simulation Testbed 77 Figue 5.7 Fowad Scoes fo all 1 Times of Task Execution 79 Figue 5.8 Actual Moving Distance is 716.8mm, Skill Moving Distance is 495.mm, and Distance Ratio is Figue 5.9 Temo Measuements 83 Figue 5.1 Collisions: 15 Collisions Occued 84 Figue 5.11 Tajectoies Afte Theapy Taining 85 Figue 5.1 Tanslation Temos Afte Theapy 86 Figue 5.13 Collisions Afte Theapy 87 Figue A.1 RRC Manipulato Joints and Limits 15 Figue A. RRC Manipulato 17 Figue A.3 RRC Manipulato with Sensos and End-Effecto 18 Figue A.4 PHANTOM Pemium Figue A.5 Integated Development Envionment of Halcon 111 vii

11 Figue A.6 Telemanipulation Inteface 11 Figue A.7 Teleopeation System Achitectue 113 Figue A.8 RRC Gaphical Use Inteface 114 Figue A.9 RRC GUI Main Window 114 Figue A.1 Contolle Buttons 116 Figue A.11 Desktop Icons on Robot Contolle Compute 117 Figue A.1 Jog Contol Window and Position Feedback Window 119 Figue A.13 Teach Pendant fo RRC Manipulato 11 Figue A.14 MainWindow fo Move Data / Recod 13 Figue A.15 File Management 13 Figue A.16 Execution and Status Windows 14 Figue A.17 Client Management Window on Robot Compute 15 Figue B.1 Configuation of Vision System 16 Figue B. Coodinate System fo Pespective Pojection 18 Figue B.3 Coodinates System Assignment fo Vision System 19 Figue B.4 Pespective Pojection of a Line Segment in Image Plane 13 Figue B.5 Tele-autonomy Illustation 138 viii

12 Intelligent Teleobotic Assistance fo Enhancing Manipulation Capabilities of Pesons with Disabilities Wentao Yu ABSTRACT This dissetation addesses the development of a telemanipulation system using intelligent mapping fom a haptic use inteface to a emote manipulato to assist in maximizing the manipulation capabilities of pesons with disabilities. This mapping, efeed to as assistance function, is detemined on the basis of envionmental model o eal-time sensoy data to guide the motion of a teleobotic manipulato while pefoming a given task. Human input is enhanced athe than supeseded by the compute. This is paticulaly useful when the use has esticted ange of movements due to cetain disabilities such as muscula dystophy, a stoke, o any fom of pathological temo. In telemanipulation system, assistance of vaiable position/velocity mapping o vitual fixtue can impove manipulation capability and dexteity. Conventionally, these assistances ae based on the envionmental infomation, without knowing use s motion intention. In this dissetation, use s motion intention is combined with eal-time envionmental infomation fo applying appopiate assistance. If the cuent task is following a path, a vitual fixtue othogonal to the path is applied. Similaly, if the task is to align the end-effecto with a taget, an attactive foce field is geneated. In ode to successfully ecognize use s motion intention, a Hidden Makov Model (HMM) is developed. ix

13 This dissetation also descibes the HMM based skill leaning and its application in a motion theapy system in which motion along a labyinth is contolled using a haptic inteface. Two pesons with disabilities on uppe limb ae tained using this vitual theapist. The pefomance measues befoe and afte the theapy taining, including the smoothness of the tajectoy, distance atio, time taken, temo and impact foces ae pesented. The esults demonstate that vaious foms of assistance povided educed the execution times and inceased the pefomance of the chosen tasks fo the disabled individuals. In addition, these esults suggest that the intoduction of the haptic endeing capabilities, including the foce feedback, offes special benefit to motion-impaied uses by augmenting thei pefomance on job elated tasks. x

14 Chapte 1: Intoduction 1.1. Motivation Physical disabilities make it difficult o sometimes impossible fo individuals to pefom seveal simple job elated tasks such as pessing a button to opeate a machine, moving light objects etc. While consideing employment, the tue potential of individuals with disabilities can be enhanced by technology to augment human pefomance. New developments in teleobotic systems can allow geate numbe of individuals with disabilities to compensate fo thei lost manipulation skills. In the past two decades, eseaches in ehabilitation obotics have designed and developed a vaiety of passive/active devices to help pesons with limited uppe-limb functions to pefom essential daily manipulation tasks. Since the use is inside the contol loop, most of these eseach o commecial poducts have adopted telemanipulation system, in which the use issues obot motion commands though an inteface [3]. Howeve, pactical esults ae limited, mainly due to the fact that although telemanipulation may elieve the use of the physical buden of manipulative tasks, it intoduces the mental buden of contolling the input device [4]. With typical telemanipulation, the use is in the contol loop, sensing the envionment infomation such as the location and the distance of the taget and poviding the appopiate contol signal to the input device. In liteatue [84], afte taining all opeatos fo a cetain time (nomal subjects), only 6% of them wee skilled enough to complete teleopeation tasks. A geneal method fo intoducing compute 1

15 assistance in task execution without oveiding an opeato s command to the manipulato is used. The appopiate movement fo the task is kept o even enhanced, but the undesiable movements ae educed. This is done using assist functions, which scale the input velocity accoding to the task. This methodology has been peviously employed by the autho in the execution of manual dexteity assessment tasks with fully able individuals [53]. Beside this functional appoach in ehabilitation, obotics applications can also assist clinically in theapy. Much evidence suggests that intensive theapy impoves movement ecovey. But such theapy is expensive, because it equies theapists on a peson-to-peson basis. Recently thee has been inceased inteest in estoing functions though obot-aided theapy. This appoach is to design theapy platfom to substitute some of the theapist s wok. 1.. Dissetation Objective The goal of this dissetation is to design an intelligent teleobotic system that can maximize the manipulation capabilities and educe the mental buden fo pesons with disabilities on the uppe-limb: 1. Develop senso-based assistance functions to incease the limited motion ange and enhance manipulation accuacy.. Implement these assist functions to pefom a common vocational ehabilitation test efeed to as a Box and Blocks. Duing task opeation, adjust the scaling accoding to the available sensoy data.

16 3. Develop an algoithm to ecognize opeato s motion intention by using Hidden Makov Model (HMM). Apply appopiate fixtue assistance based on opeato s motion. If the ecognized motion is following a path, a vitual fixtue othogonal to the path is applied. If the task is to align the end-effecto with a taget, an attactive foce field is geneated. Similaly, if the task is to avoid obstacles, a epulsive foce field is poduced. 4. Develop a obotic theapy system based on skill leaning though Hidden Makov Model. Since HMM is feasible to model a stochastic pocess, such as speech o a cetain assembly skill, it can be used to chaacteize the skill of moving along a labyinth path. The skill of moving along a labyinth is leaned and consideed as a vitual theapist, which eplaces the ole of a physical theapist fo motion theapy. Pefom motion expeiments with two subjects with disabilities. The contibution of this dissetation is that teleobotic system with intelligent opeation can enhance the manipulation capabilities and educe the mental buden, and leaned skill of a specific task can be used as a obotic theapist to do motion theapy Dissetation Outline The histoy and the backgound of ehabilitation obotics and telemanipulation system aeas elated to this wok ae discussed in chapte. The concept of ehabilitation obotics, haptic inteface and teleopeation assistance ae taced though histoy to the pesent state of knowledge in these aeas. Chapte 3 descibes a telemanipulation system to assist pesons with disabilities pefom dexteous manipulation tasks. In this chapte, 3

17 assistance functions ae used fo mapping such that human input is enhanced and Box and Blocks is chosen to test the effectiveness of this senso-based assistance function. The Hidden Makov Model (HMM)-based human motion intention ecognition is developed in chapte 4 and then the implementation of appopiate vitual fixtue assistance is applied to teleopeation. Chapte 5 descibes the Hidden Makov Model based skill leaning and its application in motion theapy system using a haptic inteface. Chapte 6 concludes with a discussion of the expeimental esults, and suggested futue wok. 4

18 Chapte : Backgound.1. Rehabilitation Robotics Physical and cognitive disabilities make it difficult o impossible fo individuals to pefom seveal simple wok and household tasks such as pessing a button to opeate a machine, opening a doo, moving light objects etc. A study by J. Schuyle et al concluded that a slight incease in manipulation ability, mobility and stength esults in substantial incease in the numbe of jobs fo which an individual might be eligible [31]. In many instances, such enhancements may mean the ability to do a task that the peson is othewise unable to pefom. Assistive devices have attempted to fully o patially estoe the lost functions and enable people with disabilities to pefom many Activities of Daily Life (ADL) affecting thei employment and quality of life [1, 7, 3, 4, 17]. The ealiest eseach in this aea (posthetics and obotic ams) began in the late 196s []. The Rancho Golden am, developed at Rancho Los Amigos Hospital in Downey, Califonia in 1969 was the fist successful ehabilitation obot manipulato [3]. It used seven tongue switches in a sequential mode to successfully maneuve the am in space. Johns Hopkins am [1, 5], evolved fom posthetics, could execute tasks in pepogammed and diect modes though a chin manipulandum and othe body-poweed switches. The Heidelbeg Manipulato was the ealiest example of the wokstation-based appoach to the implementation of obotic systems [6, 7]. Spatacus poject poposed that mounting a manipulato am on a wheelchai would incease the effectiveness of 5

19 manipulation ehabilitation [8, 9]. Though all these assistive devices saw limited use by consumes, they established the foundation fo futhe eseach. Figue.1 RAID Wokstation Figue. Manus Manipulato 6

20 Since the 198 s, consideable pogess has been made in the field of ehabilitation obotics technology. One example is the wokstation obotic device. The goal of a wokstation obotic device is to enable the use to pefom tasks typically encounteed in office o at home. These tasks include moving books fom a shelf to a eading boad, opening the book and flipping though its pages, inseting CD-ROMs and floppy diskettes into a compute. The most commonly used obotic wokstation available to uses with disabilities is the RAID (Robot fo Assisting the Integation of the Disabled, Figue.1) wokstation [1]. DEVAR (desktop assistant obot fo vocational suppot in office settings) [16] can be used to handle pape, floppy disks, pick up and use the telephone, and etieve medication. RAA (Robotic Assistive Appliance) offes a human size manipulato at a wokstation with 6 degees of feedom with eithe pogammed o diect contol [17] and is cuently undegoing testing to assess its advantages ove an attendant [18]. The othe kind of device is wheelchai-mounted obot. A powe wheelchai is used as a mobile base whee a mechanical manipulato can be attached. Seveal wheelchai-mounted manipulatos ae available to the consume, but two in paticula, MANUS and the Rapto, ae moe successful. MANUS is the most well known of those successos (Figue.). Rapto manipulato is the fist obot assistive manipulato that has gained FDA appoval fo use in the US [35] (Figue.3). Because of its inceased size, though, the ange of the Rapto is 1 cm compaed to the 8 cm of the Manus. It can also lift up to.5 kg. Anothe poject that has enjoyed elative success is the Handy 1 [7,11], which was pimaily used as a feeding device fo childen with ceebal palsy. Moe ecently, besides impoving eating skills, the aid has been consideed fo othe activities including application of cosmetics leisue activities [6]. 7

21 Figue.3 Rapto Manipulato In addition, in FRIEND Robot am system [15], a multimedia use inteface was included to enlage the functionality of existing technical aids. ISAC incopoated Atificial Intelligence (AI) into its contolle to educe the mental load on the use duing the pefomance of manipulative tasks []. KARES uses a SPACEBALL 3 as an input device to teleopeate the obotic am[1]. In KAREA II, an advanced vesion of KARES has a visual sevo, which allows the obotic am to opeate autonomously though the visual feedback of a binocula camea head [8]. The obot am wokstations o wheelchai-mounted manipulato above compensated fo the activity deficiencies of people with disabilities. But because of the high cost, the poo inteface between a complex electomechanical system and a peson 8

22 with limited capabilities, and social stigma attached with a obot, these assistive devices have had limited success as commecial poducts[1,3,4,7]. Besides assistive obots, anothe type of ehabilitation obotic system is theapy obot. MIT-MANUS (Figue.4 (a)) is the most successful obot-aided theapy platfom to undego intensive clinical testing [85, 86]. This device is a plana, two-evolute-joint, backdiveable obotic device that attaches to the patient s hand and foeam though a bace. The patient can move the obot, o the obot can move the patient, in the hoizontal plane. The patient eceives feedback of the hand tajectoy on the compute sceen. The esults of clinical tials suggested that execise theapy impoved moto ecovey [87-89]. (a) (b) Figue.4 (a) MIT-MANUS[85], (b) MIME[16] 9

23 MIME (Figue.4 (b)) is poweful enough to move a patient s am thoughout the thee-dimensional wokspace against gavity [79]. When the patient moves he/his unimpaied am, a mechanical digitizing stylus senses the movement. The PUMA 56 obot am then moves the patient s impaied am along a mio-symmetic tajectoy. The esult of clinical tests with MIME showed integation of obot-aided theapy into clinical execise pogams would allow epetitive, time-intensive execises to be pefomed without one-to-one attentions fom a theapist [16]. The ARM (Assisted Rehabilitation and Measuement) was designed to guide eaching movements acoss the wokspace, and to measue multi-axis foce geneation and ange of motion of the am [79]. Like MIT-MANUS and MIME, the ARM device can assist o esist movements and can also measue hand movements. The ARM Guide has been used to quantify and undestand abnomal coodination, spastic eflexes, and wokspace deficits afte stoke [9]. The testing esults suggested that the constaint foce and ange of motion measuements duing mechanically guided movement may pove useful fo pecise monitoing of am impaiment and of the effects of teatment techniques tageted at abnomal synegies and wokspace deficits[91, 9]... Teleobotics Due to the unstuctued envionment of ADL and vaieties of the tasks and the pesence of the use, many ehabilitation obots adopt teleobotics systems so that uses can issue commands though a human-machine inteface [8, 11, 15, 8]. Regading teleopeation studies, seveal types of systems and concepts have been defined in the aea of emote manipulation technology [39]. The concept developed by Ray Goetz in the 1

24 195's, in which a peson's sensing and manipulation capability is extended to a emote location, is efeed to as teleopeation. His mechanisms wee mechanical pantogaph devices which allowed adioactive mateials to be handled at a safe distance. Late, electical sevos eplaced mechanical linkages and cameas eplaced diect viewing, so that the opeato could be abitaily fa away. Human opeatos look at video displays, and opeate emotely located slave obot via a hand contolle. Usually the tem teleopeation efes to systems in which the human opeato diectly and continuously contols the emote manipulato. In these systems, the kinematic chain which is manipulated by the opeato and may povide foce feedback is efeed to as the maste, while the emote manipulato is efeed to as the slave. Fom the point of view of autonomy, teleobot is classified into tele-autonomy and tele-collaboation [57]. The fome tem efes to the combination of teleopeation and autonomous obotic contol. In some cases, a unilateal contolle is used. In this case, thee is no infomation feedback fom slave to maste o fom maste to human. The latte means all opeations ae contolled by the human-machine collaboation, usually in the fom of foce eflection. Fo teleopeation itself, it can be classified into unilateal and bilateal teleobotics accoding to the data flow. In the fome case, the slave obot is opeated in fee teleopeation, just like an open-loop system. The only feedback is the task execution video of the slave o even no video if the maste and slave ae in the same oom. This case is illustated in figue.5 (uppe pat). The latte one has foce feedback povided to the teleopeato, thus foming a kinesthetic o tele-pesence system [33, 34, 37, 73]. Figue.6 shows the achitectue of a typical bilateal teleopeation. In this case, stategies in which human decisions ae meged with compute-based assistance 11

25 have been made possible by moe complex foms of automatic contol and senso data fusion. The contol system adds compute-geneated velocity/foce inputs to those fom the maste in the impedance-contolled fomulation to assist contolling the motion of the manipulato, such as moving along a suface without impact and obstacle avoidance. Bilateal impedance contol in teleobotic systems povides good teleopeation since foce eflection is povided to the opeato duing opeation [33, 36, 39]. Dubey et al poposed vaiable impedance paametes to adapt to vaiable cicumstances thus ovecoming the conflict poblem of choosing desied dynamics paametes [34]. This contolle is pimaily used in tasks equiing contact, such as needle inseting into tissue, object suface exploation. Teleopeation system design usually takes opeation accuacy into account, not the convenience and simplification of opeation. With the impovement of the contolle achitectue and assistance attempt, the task pefomance of teleobotic system in ehabilitation engineeing is still not satisfactoy [4, 41, 44]. Fo a simple "go get a cup and put it on a pad" task, it takes the opeato 5 seconds, mostly due to the indexing the maste once the maste eaches its wokspace limit and tuning the gippe to gasp the taget [53]. Futhemoe, the pefomance lagely depends on the opeato's familiaity with the system. In most cases, using a obot as a teleopeated device to complete a task is much hade than using human am and hand. It can soon become vey exhausting, especially if it has to pefom epeated tasks such as feeding, even with some assistance. Many eseaches tied to impove the opeation accuacy, educe execution time and elieve the opeato's mental labo though adding atificial intelligence. Kawamua et al [51] looked at how fa ehabilitation obots had come in possessing abilities that elieve 1

26 the use fom the mental buden of contolling the obot. They had developed modules fo fuzzy commands inteface, object ecognition and task planning. In intelligent teleobot system, vision-based assistance has impoved the opeation of aligning the endeffecto with the taget [45, 5]. Output: velocity/position, and/o foce Input: video of slave with envionment opeato maste slave envionment - slave envionment C(s) Figue.5 Tele-autonomy is the Combination of Teleopeation and Autonomy Output: velocity/position, and/o foce Input: video of slave with envionment human maste slave envionment Figue.6 Tele-collaboation with Infomation Feedback 13

27 The teleobot emphasized in this dissetation, is the open loop telemanipulation with assistance. The challenge is to make it moe functional and moe intelligent. This dissetation is an attempt to addess the issue of combining human flexibility and machine intelligence into an efficient ehabilitation obotic system..3. Teleopeation Assistance Backgound In teleopeation, it is essential to povide as much assistance as possible fo the opeato. Basically, the assistance algoithm is to map the maste commands to the slave in a way that scales up o down depending on the task and envionment infomation. The scaling factos vay accoding to the tasks and envionment. The idea behind the assistance function concept is the genealization of position and velocity mappings between maste and slave manipulatos of a teleopeation system. This concept was conceived as a geneal method fo intoducing compute assistance in task execution without oveiding opeato s commands to the manipulato (Figue.7). The assistance functions can be classified as egulation of position, velocity and contact foces. All of these assistance stategies ae accomplished by modification of system paametes. A simple fom of position assistance is scaling, in which the slave wokspace is enlaged o educed as compaed to the maste wokspace. The velocity assistance is commonly used in appoaching taget and in avoidance of obstacles. In both cases, the velocity scaling vaies accoding to whethe motion in that paticula diection is seving to futhe the desied effect of the motion. 14

28 Figue.7 Human -Machine Coopeative Teleopeation Concept [9].3.1. Regulation of Positions In these functions, the motion of the manipulato is constained to lie along a given line o in a plane. This helps pesons with disabilities opeate moe stably and smoothly. The details of these functions wee pesented in a diffeent wok by the authos [67] (See Figue.8). Figue.8 Repesentation of Slave Constaint Fame in the Constaint Plane [67] 15

29 Figue.9 Scaling Facto Function [53].3.. Regulation of Velocities In this case the mapping between the maste and slave is done based on velocities. The velocity scaling used vaies accoding to whethe the motion in a paticula diection is seving to futhe the desied effect of the motion. In the appoach assistance, the velocity is scaled up if the motion educes the distance between the cuent and goal positions of the manipulato. Othewise, the velocity is scaled down. Fo velocity egulation, the scaling facto s changing is depicted in Figue.9. The scaling facto depends on the subtask being executed and the diection of tavel. The elationship between the maste/slave velocities is: V slave = ScaleFacto V maste. Figue.1 shows a velocity scaling facto vaying based on the distance eading when the end-effecto is appoaching a wall. Using a vision system, Eveett designed a vision-based mapping to align the endeffecto of the slave manipulato with a coss object [9, 45]. The velocities that educe the alignment eo ae scaled up and the ones that incease the alignment eo ae scaled down (Figue.11). 16

30 Figue.1 Scaling Facto Vaying fo Appoach [9] Figue.11 Coodinate Fames fo Coss Alignment Task [9] In tele-collaboation, anothe type of assistance is vitual fixtue. This assistance is functions of spatial paametes, instead of time. But what is vitual fixtue? Vitual fixtues ae defined, accoding to [68], as abstact pecepts ovelaid on top of the 17

31 eflected sensoy feedback fom a emote envionment such that a natual and pedictable elation exists between an opeato s kinesthetic activities and (effeence) the subsequent changes in the sensations pesented (affeence). Intuitively, it is vey easy to undestand this. As a matte of fact, eveyone has expeience of using a eal fixtue, fo example, dawing a staight line using a ule. By pessing you pencil against this "fixtue", we ae able to quickly daw a vey staight line. Now imagine if thee was no ule thee, but thee was a vitual wall you could pess against instead of a ule. Similaly, what if thee wee invisible foces pulling on you pencil, focing it to follow a staight path. These ae vitual fixtues. Vitual fixtues play the same ole in obot motion as they do in ou line dawing motion. As a matte of fact, vitual fixtue is a computed-geneated constaint that displays position o foce limitations to a obot manipulato o opeato. It can be used to constain the manually contolled manipulato s motion on a desied suface o to be pulled into alignment with a task [37, 38, 61, 64]. Usually, two stiffness coefficients ae defined: stiffness along the desied path and stiffness othogonal to the path. The atio between these two stiffness coefficients indicates the softness o hadness. If the atio is close to zeo, it is the hadest fixtue, which means that end-effecto can only move along the path, not deviating at all. If the atio is close to 1, it is the softest fixtue, whee the end-effecto can move feely. So this kind of fixtue is usually used fo path following (Figue.1). Vitual fixtue can also be in the fom of potential foce fields [68, 69]. Potential fields wee used to poduce velocity commands, which, when added to those geneated by the input device, maneuve the manipulato towad the taget o away fom obstacles [69]. Foce field is usually in the magnetic fom. The ole of this type of fixtue is the 18

32 same, guiding the end effecto into a goal o away fom an obstacle. Figue.13 shows that extact and inset fixtues estict the motion of the end-effecto when it is close to the tool gasping position. This behavio is implemented in ode to avoid a collision of the manipulato with the tool, while allowing the opeato to quickly extact/inset the gasping position [69]. Figue.1 Two Types of Refeence Diection Fixtues [55] Figue.13 Vitual Fixtues to Aid Extact / Inset Motion [69] 19

33 Chapte 3: Teleopeation with Assistance Functions 3.1. Intoduction This chapte descibes a telemanipulation system to assist pesons with disabilities pefom dexteous manipulation tasks. This wok is expected to enhance the teleopeation pefomance though the use of scaled mapping fom maste to slave manipulation based upon sensoy data. The concept is that appopiate movement fo the task is kept o even enhanced, but the undesiable movements ae educed. This is done using assist functions, that scale the input velocity accoding to the task. This assistance appoach uses assist functions and available sensoy data to pefom vaiable velocity mapping between the maste and slave, efeed to as the Senso Assist Function(SAF). A common vocational ehabilitation test efeed to as Box and Blocks was chosen to test the effectiveness of this senso-assisted function. A vaiable scaling scheme was developed using available sensoy data. In the simulation mode, a visual envionment was ceated fo the Box and Blocks test. This was used to pedict if a peson with disabilities would be able to pefom a task comfotably. The eal test was pefomed using a maste and slave manipulato system with a camea and lase ange finde. A motion constaint was added to the maste to simulate a use with disabilities. The esults demonstated that the senso assistance not only educed equied input motion, idle time, and execution time, but also inceased manipulation accuacy duing the Box and Blocks test. This wok

34 pompted the need of building a test-bed that uses available sensoy infomation to adjust paametes duing task execution. 3.. Assistance Functions Concept Assistance functions wee developed to assist the opeato by scaling the input velocity accoding to the task. The assistance includes linea assistance, plana assistance, and velocity assistance. The linea assist function constains the input velocity along a line. The input velocity is tansfomed to a task fame and multiplied by a scaling matix, and then tansfomed back to the base fame. A goal line is detemined between two points and defined as the X-axis of the linea task fame. The Z-axis is defined as the pependicula vecto, and the Y-axis is defined by the coss poduct of Z coss X. A tansfomation matix is calculated accoding to the task fame, and is multiplied by the input velocity. V V V slavex slavey slavez = a a a b b b c V V V mastex 1. c 1 mastey (3.1) 31 c 31 mastez whee V slave is the input velocity in the task fame. Then a scaling matix is applied to scale down the velocity in the undesied diections along the task fame Y and Z-axis. V V V scaledx scaledy scaledz = k x k y V V k z V slavex slavey slavez (3.) whee the values of k x, k y, k z depend on a specific task. In the linea assistance case, the values of k y and k z ae vey small. Then, V scaled is tansfomed back to the base fame 1

35 using the tansfomation matix, and that becomes the modified velocity that is sent to the obot contolle. The plana assist function constains the input velocity along a plane. To constuct this task fame, thee points ae used to define a plane. The X-axis is defined as the line between points 1 and. The Z-axis is defined as the nomal to the plane, and the Y-axis is defined as the coss poduct of Z and X. A tansfomation matix is detemined, and the input velocity is conveted to the task fame accoding to equation (3.1), the same as the linea case. Fo the plana assistance, howeve, the value of the scale matix is diffeent. Since the desied motion lies in the X-Y plane, only motion along the Z-axis will be scaled, so k z is vey small. Afte the task fame velocity, V slave, is multiplied by the scale matix, it is conveted back to the base fame and sent to the obot contolle, accoding to equation (3.). The velocity assist function inceases and deceases the velocity accoding to the distance to the goal object o an obstacle. As the distance to the goal is known, a velocity scale facto can be applied to the velocity in ode to incease o decease the input velocity. These assistance stategies ae integated togethe to povide a fom of assistance fo uses with disabilities to pefom the Box and Blocks task in this eseach Box and Blocks Task The Box and Blocks test measues goss manual dexteity and is fequently used in eseach on ehabilitation. This test, epesented in figue 3.1(simulation mode) and figue 3. (eal testing), consists of moving one-inch blocks fom one side to anothe in a two-sided box. A wall divides the two sides. This test the use of lage motions in all

36 diections. The goal is to pick up the block fom one side, and place it in the othe side. In simulation mode (Figue 3.1), foce feedback was added to make use feel esistive foce and collision. In eal test (Figue 3.), a sphee constaint was applied to simulate the wokspace of pesons with disabilities. Since the possible input motion has been deceased, the able-bodied use will bette epesent a peson with disabilities. Assistance function algoithm is based on sensoy data. Figue 3.1 Box and Blocks Test Window Inteface DME Lase Range Finde Hitachi Camea SLAVE RRC Manipulato End-Effeecto BOX WALL BLOCK MASTER PHANToM Input Device Y Z X Figue 3. Box and Blocks Test, Maste and Slave 3

37 3.4. Senso Assist Function In this eseach, a combination of the linea, plana and velocity assistance, efeed to as the Senso Assist Function (SAF), was developed fo the Vocational Rehabilitation test called Box and Blocks. The SAF essentially uses sensoy data to pefom vaiable velocity mapping fom maste to slave (Figue 3.3). Figue 3.3 Teleopeation Test-bed Figue 3.4 Sensos Mounted on End-Effecto 4

38 The sensos include a DME Lase Range Finde (LRF), and a vision system using a Hitachi KP-D5. These sensos ae mounted on the end-effecto accoding to figue 3.4. The vision system is used to locate the goal object and obstacles. The image pocessing softwae, Halcon [77], obtains the cente position of the goal object in the image plane. Once the end-effecto gasps the object, the softwae obtains the edge of the wall, which is used to avoid obstacles. The LRF is used in the velocity assistance in the Z-diection depending on the depth of the obstacles and the object Desciption Thee ae seven stages of assistance shown in figue 3.5. At the stat of the task, the obot is in the home position and thee is no scaling until the object is seen by the vision system. START 1 FINISH Y Z X Figue 3.5 The Seven Stages of the Scaling Scheme The fist stage involves minimizing the distance between the end-effecto and the object in the X-Y plane. The second stage adds z-diection scaling as the manipulato moves down. The thid stage assists the manipulato when the vision system can no 5

39 longe see the goal object. Once the object is obtained, the fouth stage assists the opeato in avoiding the wall obstacle. The fifth stage is activated when the ange data is too close to an object. The sixth stage involves the vision system, and enhances the movement in the hoizontal plane to clea the wall hoizontally. The seventh stage simply fees the use to place the object down on the coect side of the box. Since the cente of the camea is not the end-effecto position, the camea needs to be calibated with the end-effecto. Accoding to figue 3.6, the end-effecto position is pojected on the image fame, and its pixel position is detemined elative to the cente position of the goal object. Goal Object VisionX, VisionY Y X Image Fame Vecto End Effecto Image Pojection EndX, EndY Figue 3.6 Image Fame Showing Vecto Detemination Stage One Fo stage one, the scaling is based upon the position of the object and the pojected end-effecto position. A vecto is ceated between these two points, in the X-Y plane, and the task fame is calculated using this vecto and a Z-axis. The x-diection of the image fame is opposite to the x-diection of the slave fame, so the vecto calculation is as follows: 6

40 ( - VisionX ) x, ( VisionY - ) y Vecto = EndX EndY (3.3) A tansfomation matix is detemined fom the PhanToM fame to the task fame accoding to the task fame calculations in section 3., and the input velocity is scaled accoding to the following equations: V SLAVE = V Tansfom (3.4) INPUT C O VisionScale Scale =.1 (3.5).1 V = Scale (3.6) SCALED V SLAVE V MODIFIED SCALED ( Tansfom C ) T = V (3.7) O whee, fo stage one, VisionScale anges fom 1.5 to 3 maximum. If the dot poduct of V SLAVE and Vecto is negative, then VisionScale is.1. This means that the input velocity is in the opposite diection of the goal object. The modified velocity, V MODIFIED is sent to the low-level contolle Stage Two Stage two stats when the magnitude of the Vecto is less than 75 pixels. This means that the end-effecto is close to the coect x, y position ove the goal object, and the opeato can stat moving down towads the object. Stage one exists to help educe the senso eo by keeping the end-effecto in the X-Y plane fo lage movements while the opeato is appoaching the goal. Stage uses the same task fame as stage 1, but the scale matix eflects inceased velocity in the z-diection. 7

41 VisionScal e Scale=.1 ScaleFacto (3.8) whee VisionScale anges between 1 and 1.5, and if the dot poduct of V SLAVE and Vecto is negative, then VisionScale is.1. ScaleFacto depends on the value of the LRF, shown in figue Velocity ScaleFacto vs. DME Velocity Scale DME Range Value (mm) Figue 3.7 ScaleFacto Accoding to LRF Data (DME) So this scale matix helps to guide the end-effecto down towads the goal object. It inceases the scale in the Z-diection, and allows motion in the hvecto diection to pull the end-effecto to the goal object Stage Thee The thid stage stats when the vision system can no longe see the object. As the end-effecto gets close to the object, it will eventually move out of the image fame because of the location of the camea on the end-effecto. In this stage the task fame will not be calculated since thee is no data fom the vision system. So the following scale matix will be diectly applied to the input velocity. Since the end-effecto is nea the object, thee will be little motion equied in the X and Y diection. 8

42 k Scale = k ScaleFacto (3.9) V = Scale (3.1) MODIFIED V INPUT Using a scale of k =.5 in the X and Y diection allows fo some eo coection, but it scales down lage movements fom the opeato away fom the goal object Stage Fou The fouth stage begins when the end-effecto gabs the object. This stage scales the velocity in ode to avoid the cente wall obstacle. At fist, the velocity is scaled to move the end-effecto in the positive z-diection accoding to AvoidScale. AvoidScale depends on the LRF value, and anges fom 3 to 1. If the input velocity is in the downwad z-diection, then AvoidScale is.1. The y-diection is scaled down because the desied motion fo the task is in the x-diection. The vision system pefoms edge detection and etuns the geatest x-value of that edge in the image fame. The initial scaling equation is: V MODIFIED 1 =.1 V V AvoidScale V X Y Z (3.11) Stage Five A as the end-effecto moves to the left to place the object on the othe side of the box, the LRF is monitoed fo obstacles. If the LRF sees an obstacle, then all velocity inputs ae scaled down, and the upwad z-diection is inceased by AvoidScale, accoding to the following equation: 9

43 V MODIFIED.1 =.1 V V AvoidScale V X Y Z (3.1) As the end-effecto moves to the left, the LRF leads, accoding to figues 3.1, 3. and 3.3. Figue 3.3 shows how the LRF can measue the wall without a collision. Theefoe, the LRF checks the z-diection to make sue the whole end-effecto can clea an obstacle Stage Six Now that the end-effecto has enough height to clea the wall vetically, it must clea the wall hoizontally. So, once the wall comes into the image fame, the scaling is shown by the following equation: V MODIFIED Avoidwall =.1 V V.1 V X Y Z (3.13) whee Avoidwall inceases the negative x-diection, see figues 3.1 and 3.3, to assist in avoiding the seen obstacle. Once the wall obstacle is seen, the z-diection will be scaled down Stage Seven Once the camea can no longe see the wall, the end-effecto has avoided the wall obstacle. The scaling etuns to egula z-diection velocity assistance accoding to the following equation. V MODIFIED 1 = 1 V V ScaleFacto V X Y Z (3.14) 3

44 Once the object is nea the table on the coect side of the box, the opeato is eady to elease the object. Now the task is completed, and the completion time is ecoded. By etuning the end-effecto to home position, the opeato is now eady to pefom anothe Box and Blocks test Expeimental Results Telemanipulation System Stuctue In this system (figue 3.8), the maste obot is a PhanToM with 6 degees-offeedom fom Sensable Technologies. It can povide tactile feedback fo the use. A 7 DOF industial obot RRC K-17a is used as a slave manipulato in this application. A Windows PC is used to contol the PhanToM and compute the mapping fom maste to slave. The slave manipulato contolle uns anothe PC. A thid PC handles the sensoy data. All PCs ae linked togethe though an Ethenet, and sensoy data is sent to the PhanToM PC and the velocity commands ae sent to the manipulato PC. RRC Contol PC Single Boad Compute Phantom Hand Contolle PhanToM PC DME Hitachi KP-D5 RRC Manipulato PC With Fame Gabbe and HALCON Figue 3.8 The Telemanipulation System 31

45 3.5.. Softwae Implementation Two majo pogams have been developed in this chapte. One is the image pocessing, which does the Sobel edge detection and egion gowing to obtain the coodinates of the object in image plane (figue 3.9 and 3.1). This pogam uses API functions povided by HALCON. RGB Image RGB Image Image Afte afte Region Gowing Too Lage Object of Inteest Too Smal Figue 3.9 Region Gowing Image RGB Image RGB Image Sobel Image Sobel Image Majo Hoizontal Lines Lagest Row Value Figue 3.1 Sobel Edge Detection Image 3

46 The othe is the contol pogam un in the maste PC. It was developed using the GHOST SDK fom Sensable Technology [48]. This softwae obtains the accuate position and oientation of the PhanToM. Foce eflection is also available with the softwae. The sample time of the maste PC getting position o velocity data fom the maste device is.s. Once the maste velocity is obtained, it is modified accoding to the SAF. This adjusted velocity command is sent to the slave PC at the same ate as its sample ate Results Simulation Mode 15 1 Box and Blocks Test using Assistance Function Maste Slave All values ae in geneic units z x y 4 Figue 3.11 Tajectoy Compaison of PhanTom and Slave Manipulato 33

47 Box and Blocks Test Time (secs) Block Numbe Figue 3.1 Box and Block Time Execution No Assistance Povided Assistance Povided An able-bodied peson pefomed the Box and Blocks simulation with assistance function to detemine the effect of the assistance. When use s movement is away fom the desied tajectoy, foce eflection will be felt by the use that makes the use move back to the desied tajectoy. Figue 3.11 is the tajectoy compaison of the Phantom and slave manipulato when doing box and blocks test with assistance function. Obviously, though the maste has some andom movements, the slave manipulato moves along a desied tajectoy vey well. A sample of time executions of seven tests is shown in figue 3.1. It is noticed that due to the assistance function, the aveage time was educed consideably (fom 1.33 to 5.66 seconds), and the standad deviation (fom.81 to.5) was smalle as well Real Test Mode An able-bodied peson pefomed the Box and Blocks eal test with and without the SAF to detemine the effect of the assistance with a sphee constaint in his wokspace, which simulated the motion of pesons with disabilities. The height of the wall in the tests is 1 inch. Figue 3.13 shows the tajectoy of the slave manipulato with 34

48 no assistance vesus the slave with assistance when doing eal box-block test. Accoding to this figue, the tajectoy with assistance is a smooth cuve appoaching the object, and then avoiding the wall obstacle. The cuve shows how the use was guided towad the object. The tajectoy with no assistance shows that the use has a andom appoach to the object, while showing many uncetain and unnecessay movements. It also shows the effect of each stage of scaling. Assistance Vs. No Assistance 1-1 Slave No Assistance Slave Assistance Stat Stage 1 Stage Z-Axis Stage 5 Stage 6 Stage 7 Stage 3-6 Stage Finish Object Wall Obstacle 1 Y -Axis X -Axis Figue 3.13 Tajectoy of Box and Blocks Task Fo data analysis, the peson pefomed the test 3 times with assistance and 3 times without assistance. Table 3.1 shows the esults of the tests. It includes the decease of necessay input motion, idle time, and execution time when using the developed compute assistance. Wheneve in simulation o eal test mode, assistance functions not 35

49 only deceased the execution time, but also educed its standad deviation fom 4.51s to.86s. Table 3.1 Compaison of Aveages fo Box and Blocks Test Using Wokspace Constaint Aveage Test Data-All Positions No Assistance SAF Assistance % Decease Total Distance % Numbe of Times Reposition % Time Spent Repositioning % Total Completion Time % 3.6. Summay This wok povides a vitual simulation and senso-assistance appoach fo a complex teleopeation task to be executed by pesons with disabilities. It can be used as a vocational taining platfom and as an evaluation tool afte theapy in ehabilitation engineeing. The assistance will incease the safety and dexteity of these uses who would not be able to pefom the task othewise. In this dissetation, the Box and Blocks test was explained as well as a suitable combination of assistance that vaiably scales the input velocity. Able-bodied pesons initially pefomed the test to show the effect of the assistance concept. A constaint was added to the input to simulate a peson with disabilities by deceasing the possible movements of the able-bodied use, and moe tests wee pefomed. The esults show how the desied motion was kept o sometimes augmented, and how the unwanted motion was educed. Theefoe, when applying this assistance, the pefomance of a peson with disabilities will be dastically enhanced. 36

50 Chapte 4: Telemanipulation Assistance Based on Motion Intention Recognition In telemanipulation systems, assistance though vaiable position/velocity mapping o vitual fixtue can impove manipulation capability and dexteity [37, 45, 53, 61, 64]. Conventionally, such assistance is based on the sensoy data of the envionment and without knowing use s motion intention. In this dissetation, use s motion intention is combined with eal-time envionment infomation fo applying appopiate assistance. If the cuent task is following a path, a vitual fixtue is applied. If the task is aligning the end-effecto with a taget, an attactive foce field is poduced. Similaly, if the task is avoiding obstacles that block the path, a epulsive foce field is geneated. In ode to successfully ecognize use s motion intention, a Hidden Makov Model (HMM)-based algoithm is developed to classify human actions, such as following a path, aligning taget and avoiding obstacles. The algoithm is tested on a simulation platfom. This chapte pesents the teleopeation assistance algoithm development based on opeato's motion intention ecognition though Hidden Makov Model (HMM). The basic theoy and the application of HMM ae also pesented Telemanipulation Assistance The fundamental pupose of a teleobotic system is to extend opeato s sensoymoto facilities and manipulation capabilities in emote envionment [7]. This appoach is guided by the philosophy that the human opeato should emain in diect contol of the 37

51 slave at all times, with human-independent contol paametes alteed accoding to senso infomation. Howeve, manipulation tasks such as assembly ae still difficult fo a teleobotic system. In many cases, the use s physical labo load of completing a task manually is eplaced by mental buden of contolling the emote input device mentally. In the field of ehabilitation obotics, this is the main hindeing fo the wide application of teleobot assistive devices [71]. So assistance fo teleopeation has become essential in ode to educe the opeation fatigue. The fist kind of assistance is the vaiable position and velocity mapping based on sensoy infomation and foce feedback [53]. The othe is vitual fixtue, which has been used as means of poviding diect, physical assistance [37, 61, 64]. Just imagine dawing a staight line without a ule, it is vey difficult. Vitual fixtue plays the same ole as a ule to enhance human s dawing a staight line. Both of these assistances can enhance a human s pefomance accuacy fo complex tasks execution and educe time consumption. But the limitation is that they ae elated to some specific tasks. Ou ecent wok in telemanipulation systems fo ehabilitation engineeing motivated us to enhance manipulation accuacy and educe opeato s fatigue [9, 5, 53]. In ode to povide geneal assistance, specific tasks need to be divided into seveal simple and geneal subtasks. Ou wok ties to combine the envionment infomation with use s motion intention befoe applying appopiate assistance. Human motion intention is classified by movement velocities though Hidden Makov Model: following a path, aligning with a taget, avoiding an obstacle and stopping. Fo each motion, appopiate assistance is povided. Fo example, if the motion is following a path, a vitual fixtue othogonal to the path is applied, just like a ule. If the motion is aligning with a taget, an attactive foce field is applied. 38

52 4.. Classes of Motion in Telemanipulation With typical telemanipulation, the use entes the contol loop, sensing the envionment infomation such as the location and the distance of the taget and poviding the appopiate contol signal though moving the input device. Fo a common task, such as gasping a cup and putting it on a cup pad, the motion pocess can be divided into fou classes: 1. Following the desied tajectoy;. Aligning with the taget; 3. Avoiding an obstacle; and 4. Stopping The "following the desied tajectoy" motion happens when a desied tajectoy is planned. Fo the go gasp task, the desied tajectoy is a staight line if thee is no obstacle blocking the path. We can decompose the velocity vecto v c into two pats, v p, velocity component along the desied path tangent diection and v o, velocity component othogonal to the desied path tangential (Figue 4.1). While uses ae following a path, v p >>v o (Figue 4.1); While aligning the end effecto with the taget, both v p and v o ae elatively small and close to each othe (Figue 4.); while avoiding an obstacle, v p <<v o (Figue 4.3); and when stopping, both v p and v o ae close to zeo (Figue 4.4). But these featues ae not tue fo each sample. We can not classify these fou motions fo each sample value using a simple theshold. So Hidden Makov Model, a technique of stochastic pocess is used. Since these two velocity components ae othogonal, they ae independent. In ode to apply HMM to model these two velocities components, a - dimensional HMM is used. 39

53 Figue 4.1 Path Following Motion and its Velocity Pofile Figue 4. Aligning with Taget Motion and its Velocity Pofile Figue 4.3 Avoiding Obstacle Motion and its Velocity Pofile 4

54 Figue 4.4 Stop Motion and its Velocity Pofile 4.3. Hidden Makov Model based Motion Recognition Data Pepocessing The velocity of the input device is sampled at 1Hz ate. The data is denoted as V, V p ] V =, Vp and Vo ae the sets of velocity sampling values v p and v o. [ O V V p O = [ v = [ v 1, p 1, o, v, v, p, o,... v,... v n, p n, o ] ] (4.1) whee n is the sample numbe. Since V p and V o play the same ole, we just demonstate the data pocessing of one of them, i.e. Vp. Since we use discete HMM, we need to convet this velocity data into a sequence of discete symbols. We follow two steps in this convesion: (1) data pepocessing and () vecto quantization, as illustated in Figue 4.5. The pimay pupose of data pepocessing is to extact meaningful featue vectos fo the vecto quantization. In ou case, the pepocessing poceeds in two steps: (1) spectal convesion, and () powe spectal density (PSD) estimation. Fist, a 16-point width window with 5% ovelap is used to select data: 41

55 v = v, v,... v ] (4.) p [ 1, p, p 16, p Pio to spectal convesion, a hamming window is used to filte each fame, thus minimizing spectal leakage. The Hamming tansfomation T v ( ) maps a k-length (k =16 in this case) eal vecto to a new k-length eal vecto. v h T v ) = [ H v H v... H v ] (k 16) (4.3) = ( = H p 1 1, p, p k k, p H H i π ( i 1) = cos[ ], i k 1 { 1,,..., k } (4.4) Next, FFT (Fast Fouie Tansfom) analysis is applied fo evey Hamming windowed data. The FFT tansfom T h ( ) maps a k-length vecto h = h, h,... h ] to a k- length complex vecto z = z, z,... z ]. [ 1 k F [ F ( h) F ( h)... F ( )], whee 4 [ 1 k h z = TH ( h) = 1 k 1 h (4.5) 1 ( ) = Fp h h q= k {,1,... 1} π ipq / k 1e, p q k Now, let us define the powe spectal density (PSD) estimates fo the hamming-fouie output z. The PSD estimates is given by, P z) = [ P ( z) P ( z)... P ( ) ], whee (4.6) ( 1 k / z 1 P ( z) = z, H ss 1 Pi ( z) = ( zi zk i ), i { 1,,..., k / 1 }, H ss 1 P k / ( z) = fk /, and H = k H ss k ss H k q= 1

56 Figue 4.5 Convesion of Continuous Velocity Data into Discete Symbols 43

57 Due to the symmety stuctue, the length of PSD estimates output is k/ = 8. As illustated above, a 16-point velocity samplings window is mapped to an 8-point PSD vecto. Let us epesent the hamming windowing, Fouie tansfom and powe spectal v density by T ( ). If thee ae m sampling windows, the PSD estimation vectos ( H, F, P) fom a matix as shown below, V m P v { P( z)} T 1 ( H, v { P( z)} = = T( H, M v { P( z)} m T( H, F, P) F, P) M F, P) ( v ( v ( v p,1 p, p, m ) ) ) (4.7) In the same way, the second dimensional data, V o can be conveted into a PSD matix as m VO above Vecto Quantization and m O In the pevious section, we conveted aw velocity data into the featue matix V. Let { v }, t { 1,,... m} V t = denote the set of all featue vectos. In ode to apply discete-output HMMs, we now need to convet the featue vectos V to N discete symbols, whee N is the numbe of output obsevables in ou HMMs. In othe wods, we m V P want to eplace the many v with L pototype vectos Q { xn }, n { 1, N} t N =,..., known as the codebook, such that we minimize the total distotion D ( V, ) T (, Q ) min d ( vt, xn ), whee d ( vt, xn ) = ( x n vt ) ( xn vt ) D V N = (4.8) t ove all featue vectos. We choose the well-known LBG vecto quantization (VQ) algoithm [7] to pefom this quantization. The illustation of LBG algoithm fo 44 Q N

58 diffeent N is shown in Figue 4.7. Fo ou case, N is detemined to be 56. Fo ou data, we set the split offset ε =.1 and the convegence citeion δ VQ = 1.e -15. With these paamete settings, the centoids usually convege within only a few iteations. Thus, the velocity signal is tained and classified into 56 vectos, denoted by VQ codebook Q N. Now, given a sequence of featue (velocity fo ou case) vecto V f, we can convet them into a symbol vecto S { s s..., s } TVQ f =,, 1 f with length f. Let us use ( ) to epesent the convesion fom featue vecto into symbol, then 1 f { T ( v, Q ), T ( v, Q ),..., T ( v, Q )} f S = T ( V, Q ) = (4.9) f VQ N VQ N VQ N VQ N s i { 1,,... N} f f = T ( v, Q ) = index[min d( v, x )], n (4.1) VQ N n We tain the VQ codebook by these vectos and the codebook is poduced by LBG algoithm (see Figue 4.6). The LBG VQ (vecto quantization) technique maps these 8- dimensinal vectos into a finite set of vectos Y = {y i : i = 1,,..., L}, whee L is the length of the codebook(it is detemined to be 56 in ou case). Each vecto y i is called a code vecto o a codewod and the set of all the codewods is efeed to as a codebook. Associated with each codewod, y i, is the neaest neighbo egion called Voonoi egion, and it is defined by [7]: V i k = { x R : x y x y, fo all j i} (4.11) i j The 56 8-dimensional vectos in the codebook ae 56 symbols in the output pobability distibution functions fo discete HMM. Similaly, a codebook fo the velocity component v o vecto and the 56 symbols ae also obtained in the same way. The computation pocedues of the data pepocessing pat ae illustated in Figue 4.5. This method is simila to the continuous-symbol convesion in [6] 45

59 Figue 4.6 LBG Codebook Taining 46

60 Figue 4.7 LBG Vecto Quantization fo Random D Data, as L Equals,4,8,16, HMM Taining 47

61 HMM Taining HMM is usually used in continuous and discete foms. Relatively, discete HMM is easie fo computation. In this dissetation, discete HMM is adopted. A discete HMM can be defined as follows [63]: 1. A set of N states S={S 1, S S N }. A set of M possible obsevations V={v 1, v v M } 3. A state tansition pobability distibution A={a ij }, whee a ij =P [q t1 =S j q t = S i ], 1<=i, j<=n 4. Obsevation pobability distibution in each state j, B={b j (k)} whee b j (k)=p [v k at t q t = S j ], 1<=j<=N, 1<=k<=M 5. Initial State distibution π = {p i }, whee p i = P [q i =S i ] 1<= i <=N 6. Let λ = (A, B,π) be the complete paamete set. Figue 4.8 epesents a 5 state HMM, whee each state emits one of 56 discete symbols in two dimensions. Figue states Left-Right Hidden Makov Model, with 3 Obsevable Symbols in Each State 48

62 In ode to tain an HMM model and use it to do ecognition, the following thee basic poblems fo HMM need to be solved [63]: 1. Given the obsevation sequence O = o 1 o o T, and a model λ = (A, B, π), how to detemine P(O λ), the pobability of the obsevation sequence, given the model? This can be viewed as scoing a model in tems of how well it matches the obsevation.. Given the obsevation sequence O = o 1 o o T, and a model λ = (A, B, π), what is the best coesponding state sequence Q = q 1 q q T, that best explains the obsevation (e.g. the most pobable sequence). 3. How do we set o adjust the paametes of a model λ = (A, B,π) to maximize P(O λ). This is the taining o leaning poblem of adjusting the model's paametes to best fit a set of taining data. In ode to classify fou diffeent motions, we need to design a sepaate HMM fo each motion. The obsevations ae a sequence of coded spectal vectos whee each spectal vecto is mapped to one of seveal code wods which is the closet match. Also the obsevations ae sequences of codes epesenting the motion executed epeatedly by one o moe opeatos. The solution to poblem 3 is to set the paametes of the model fo each motion. The solution to poblem is to segment each of the motion taining sequences into states and theeby gain infomation about how to adjust the numbe of states o the codebook. Once the fou models ae built; we can use the solution to poblem 1 to scoe each motion model s match to a given obsevation sequence and select the best model. The computation of the thee poblems will be explained in this section. 49

63 Poblem 1 is to detemine P(O λ). Examine evey state sequence length T, Q = q 1,q,,q T, how likely this state sequence is and how likely it is to geneate the obsevation sequence. Fist, we assume that individual obsevations ae independent, and then the pobability of obseving O given Q is [63]: T ( t= 1 P O Q, λ ) = P( Ot qt, λ) = bq 1( o1 ) bq ( o ) bq ( ot ) (4.1) T The pobability of a given state sequence is simply: P ( O, Q λ) = P( O Q, λ) P( Q λ) (4.13) So the joint pobability of an obsevation and a state sequence is: P( O λ) = P( O Q, λ) P( Q λ) (4.14) all Q The computation of Eq.(4.14) equies summing ove N T possible sequences. Instead, a fowad-backwad pocedue is used to do this. The detailed algoithms is descibed by L. Rabine[63]. Poblem is to find the state sequence, Q, which is the most pobable given a sequence of obsevations, i.e want to maximize P(Q O,λ), o equivalently maximize P(Q,O λ). The Vitebi algoithm [63] finds this state sequence by defining δ t( q1, q,... q 1 t λ i) = max ( P[ q, q,... q = i, O ]) (4.15) t i.e. the pobability of the best subsequence that accounts fo the fist t obsevations and ends in state S i. The induction δ j) max ( δ ( i) a ) b ( O ) (4.16) t 1 ( = i t ij j t 1 computation is used. Also it is necessay to stoe the state agument i that maximizes this function fo each t and j, this will be kept in the vecto ψ t (j). 5

64 Poblem 3 is about taining. So fa thee is no known way to analytically calculate the paametes of a model that maximizes the pobability of an obsevation. Howeve, the paametes can be locally maximized using an iteative hill-climbing method called Baum-Welch o EM(expectation modification)[63]. Let us explain Baum-Welch method. Define ξ t (i,j) as the pobability of being in state S i at time t and state S j at time t1. This can be calculated as [63] ξ ( i, j) = P( q = S, q 1 S O, λ) (4.17) t t i t = j ξ ( i, t P( qt = Si, qt 1 = S j, O λ) αt ( i) aijbj ( Ot 1) βt 1( j) j) = = (4.18) P( O λ) P( O λ) Let γ t (i) be the pobability of being in state S i at time t given the sequence and the model. N γ ( i) = ξ ( i, j) (4.19) t j= 1 The expected numbe of tansitions fom S i is then t= T 1 tansition fom S i to S j is then ξ t= 1 paametes given the obsevation as [63]: t t T 1 1 γ ( i). Then the expected numbe of t ( i). Now we can estimate new values of the π i = Expected pobability of being in state i at t=1 = γ 1 (i) a ij exp ected numbe of tansitio ns fom state i to state = expected numbe of tansitio ns fom state i T 1 = t = 1 T - 1 ξ t = 1 t γ ( i, j) t ( i) j 51

65 exp ected numbe of times in state jobsevatin g v b j ( k) = expected numbe of times in state j T γ t t= 1 = T ( j), O t= 1 γ t t ( i) = v k It can be poven that the updated model, o say a new model λ is then eithe [63] λ =λ (We ae at local maximum. This is also the stopping citeion fo taining) o λ is bette than λ egading given obsevation, i.e. P ( O λ) > P( O λ) Oveall, the taining step is to obtain a maximum likelihood estimate of an HMM fo an obsevation. The flow of this algoithm can be descibed as follows [63]: Initialize λ= λ =(A, B, π) to andom estimates that satisfy the pobabilistic constaints (see below) Repeat o Set λ: = λ k o Calculate A, B, π based on O and λ and set λ : = A, B, π. Until λ = λ Always maintains pobabilistic constaints: N π i = 1, N i= 1 j= 1 a ij = 1(1 i N ), L k = 1 b j( k) = 1(1 j N) 5

66 In pactice, it is impossible that λ =λ. But they could be vey close. Let λ (k-1) denote the HMM λ afte k-1 iteations of Baum-Welch algoithm, and let λ (k) denote the cuent iteation of Baum-Welch. Then, the taining computation stops if P( O λ [ P( O λ ( k ) ) P( O λ ) P( O λ ( k ) ( k 1) ( k 1) ) )] / < ε HMM (4.) whee ε HMM =.1. In addition, in ode to avoid computation oveflow due to the multiplication of vey small pobability numbes, scaling up fo too small pobability values ae applied if necessay. This scaling up does not affect the taining of the HMM since the only useful infomation is the atio of diffeent pobabilities and not thei eal values. As explained in the pevious section, a model coesponds to a motion. So we need to tain fou sepaate HMMs. Obviously, poblem 3(taining) is the most difficult one of the HMM s thee poblems. Suppose the HMM fo path following is initialized as follows: λ = (A, B,π). π = [.,.,.,.,.].8.1 A = B = 1. / / / / / / / / / / / / / / / 56 L L L L L 1. / / / / / 56 53

67 Fom these, we can see the pobability constaints: the sum of the pobability distibution fom the cuent state to othe states is 1 ; at each state, the sum of the pobability distibution of all possible obsevations is also 1. Using the obsevation sequences of path following, the HMM is tained, that is, the pobability paametes ae adjusted. The tained HMM is expessed by the updated values until convegence occus. π = [.8,.15,.8,.19,.9] A = B.3.13 = L L L L L Motion Recognition Once the fou HMMs ae tained by thei coesponding taining set, they can classify motions. The classification citeion is the fowad scoe of a sequence of obsevations fo a given model. This fowad calculation is the same as the fowad pat of the Fowad-Backwad pocedue used in solving poblem 1. 54

68 α i) = b ( o ), 1 i N (4.1) 1( πi i 1 N 1 t T 1 α t 1( j) = αt ( i 1) aij bj( ot 1), (4.) i= 1 1 j N N 1, o,... o t ) = T ( j) j = 1 P( o λ α (4.3) Figue 4.9 Fowad Computation Illustation Let us illustate this computation by two one-dimensional, two-state, left-ight HMMs as an example. Figue 4.9 shows two HMMs epesenting two classes. The length of the obsevation vecto is 4. Theefoe, at each time t, one of the fou symbols, A, B, C o D will be obseved fo each state. Fom the stuctue of the fist HMM (Figue 4.9 (a)), it s paametes ae:.5.5 π = [.3.7], A =, B =

69 Fo the given obsevation sequence ABA, its fowad scoe is computed as follows: α 1 ( 1) = π 1b1 ( A) =.3. = α ) = π b ( A) =.7.8 α α α α 1 ( =.6.56 ( 1 = ( = 3 ( 1 = 3 ( 1 = 1) = [ α1(1) a11 α1() a ] b ( B) = [ ].4.1 ) = [ α1(1) a1 α1() a] b ( B) = [ ] ) = [ α (1) a11 α () a1] b ( A) = [ ]..1 ) = [ α (1) a α () a ] b ( A) = [ ].8.84 P ( O = ABA λ1 ) = α 3(1) α 3() =.96 This is the pobability of the fist HMM fo the given obsevation sequence ABA. Fo the second HMM (Figue 4.9 (b)), it s paametes ae: π.3.7 = [.5.5], A =, B = The fowad scoe fo the given obsevation sequence ABA is computed in exactly the same way: α 1 ( 1) = π 1b1 ( A) =.5.1 = α ) = π b ( A) = ( =.5. α α (1) [ α1(1) a11 α1() a ] b1 ( B) = [.5.3. ].5 = 7. 5 ( ) = [ α1(1) a1 α1() a ] b( B) = [ ].5 = 4 (1) [ α (1) a11 α () a1] b1 ( A) = [7.5e ].1 =. 4 3 () = [ α (1) a1 α () a] b ( A) = [7.5e ].4 = 4 = e α 3 = 5e α.47 P ( O = ABA λ ) = α 3(1) α3 () =.47 Since P ( O = ABA λ ) =. 47> P ( O = ABA λ1 ) =. 96, it can be concluded that λ is moe likely to geneate the obsevation sequence ABA. In othe wods, if we get the obsevation sequence ABA, the undelying pocess epesented by HMM has 56

70 been ecognized. In ou case, the HMMs have two dimensions and the length of each dimension of obsevation vecto is 56. The successive fou symbols obtained by data pepocessing ae used fo the patial obsevation sequence. It could be, fo example, {, 55, 1, 19}. This vecto is used to compute the fowad likelihood of the fou HMMs as shown in the illustation above. Then fo the given obsevation vecto, we choose the model that has the lagest likelihood as ou ecognized model at time t Design of Fixtue Assistance Once use s motion intentions ae ecognized, appopiate assistance can be designed fo each motion. We define the path cuve as p(s) and denote the taget position by t. When the goal duing task execution is to move to a taget, we assume that the desied tajectoy is a staight line that connects the cuent Catesian position of the endeffecto and the taget. A pefeed efeence diection d can be defined fo each point of the end-effecto x c as: d(x c ) v p v c v o p(s) Figue 4.1 Vitual Fixtue Definition d x x t c ( xc) = (4.4) xt xc 57

71 Whee x t and xc ae the taget position and the cuent position of the end-effecto espectively. We decompose v c, the cuent velocity, into two othogonal components: v v = ( v d d (4.5) p c ) = v ( v d d (4.6) o c c ) whee v p is the velocity component along the path cuve tangent and v o is the velocity component othogonal to the cuve tangent. The desied path following is such that the velocity tangent to the cuve is lage and velocity components in othogonal diection ae elatively small. If the desied tajectoy of a sub-task is a staight line, a vitual fixtue can povide the same assistance as a ule helps in dawing a line Fixtue Assistance Fixtue assistance is always applied fo path following except when the use is tying to align an object o avoid an obstacle. So the stiffness coefficient k d along the cuve tangent is set to be zeo. The stiffness othogonal to the cuve tangent is defined as: k o = k c k c (1 d / ) d d > (4.7) whee k c is the fixtue coefficient (it is detemined to be.5n/mm fo this expeiment), d is the distance between the end-effecto and the cente position of the foce fields, and is the foce fields adius. This means that once the end-effecto goes inside foce field, path following fixtue is emoved (See Figue 4.11 fo fixtue coefficient). 58

72 stiff N/mm Attacttive N/mm Repulsive N/mm distance to foce fields Figue 4.11 Stiffness Coefficients of Diffeent Fixtues Foce Field Design fo Tagets and Obstacles In geneal, aligning the end effecto with a taget and avoiding obstacles ae not easy to execute, especially fo pesons with disabilities on the uppe-limb. Potential fields geneated fom the cente position of the taget o the obstacle can povide some assistance. Based on this concept, foce fields ae designed aound tagets and obstacles. We define the adius of foce field to be. In this dissetation, the foce field is defined using sping foce. Fo appoaching a taget, the foce is defined as: f = k f ( d ) d d > (4.8) whee k f is.1n/mm. 59

73 Fo obstacle avoidance, the foce is defined as: f = k f ( d) d d > (4.9) whee k f is.1n/mm. Once the end-effecto goes within the adius fo aligning with the taget, the attactive foce oiginated fom the object cente position can povide assistance. The foce vectos geneated by position and appoach fixtues ae shown in Figue 4.1. Payandeh et al used such vitual fixtue as a task-dependent telemanipulation aid [5, 14]. Howeve, the oigin of the foce fields needs to be detemined fom the sensoy data. In addition, should be lage than the size of the taget o the obstacle. (a) (b) Figue 4.1 Foce Fields Illustation (a: Attactive foce, b: Repulsive foce) 4.5. Expeiments We have implemented the algoithm descibed above, and conducted expeiments to detemine the system s pefomance without and with the assistance Expeimental Test Bed Ou telemanipulation simulation system is composed of a visualization scene and a haptic device. The visualization component, simulation scene, is ealized though the PhanToM and GHOST [48]. In this expeiment, the task is to move the end-effecto fom 6

74 the oigin (,,) to (-8,5,), efeed to as taget Gasp (this means the end-effecto must eside in the object sphee fo a shot time) and then avoid the obstacle (,45,) and then put the taget at the taget destination (8,5,) and go back to the oigin. The taget gasp and the taget destination ae simulated as 8mm adius sphees. The obstacle and the end-effecto ae simulated as 15mm and 5mm adius sphees; espectively. Use is asked to move the end-effecto as fast and as smoothly as possible (Figue 4.13). In ode to avoid confusion, the opeato is allowed to move on a plana suface and a plana constaint is added to the haptic device. In this expeiment, we ae concened about the staight-line path since it is elatively easy to obtain fom the envionment infomation. This algoithm can be extended to a complex tajectoy application if we can define the tajectoy using visual infomation fo the unstuctued envionment. Taget Gasp Obstacle Taget Destination Endeffecto Oigin Figue 4.13 Simulation of the Task Execution 61

75 4.5.. Expeimental Results Without Assistance Fist, an expet use completed the task seveal times without assistance. Duing the fist seveal tests, the common pefomance of the system is shown in Figue 4.14 and As expected, the fee motion has much difficulty in aligning with the taget and following the path. The velocity components othogonal to the path ae not small compaed to the useful velocity components tangential to the path. Table 4.1 summaizes the esults, including path following eo (mm) and execution time(s). 1 5 v p mm/s v o mm/s Time (ms) Figue 4.14 Velocity Components Without Assistance 6

76 1 8 Task Tajectoy test 1 test 6 Y-axis (mm) X-axis(mm) Figue 4.15 Tajectoies without Assistance Table 4.1 Pefomance Summay without Assistance Subject Path Eo (mm) Execution Time(s) Mean Stdev Mean Stdev

77 Motion Recognition Fo the task used in this dissetation, fou uses in the lab completed the task fo 1 times each. We collected 5 samples of data fo each motion, the fist fo taining and the est of the samples ae fo testing. Fo a total 5 testing samples of fou motions, the system successfully ecognized 43 samples. The accuacy is 86%. Definitely, the size of the taining set influences the ecognition accuacy. Afte we included 5 samples into the taining set, the system ecognized 9 samples fom 1 testing samples. The motion ecognition pefomance is shown in Table 4.. Table 4. Motion Recognition Rate Motion Coect ate Incoect ate to 1 to to 3 to 4 1: Path following 9.5% %.3% 3.% : Taget aligning 89.1% 6.4% %.% 3: Obstacle avoidance 88.3% 7.7%.% ----.% 4: Stopping 98.7%.% 1.3%.% Expeimental Results with Assistance Based on Motion Intention Recognition As mentioned befoe, the esultant assistance is applied to each motion of a task. If the motion at a cetain stage is path following, a had fixtue is applied so that the endeffecto can move along the path. Once the motion has been changed into aligning with a taget motion, had fixtue is eplaced by an attactive foce field. Fo avoiding an obstacle, a epulsive foce field is applied. If the motion is classified as stopping, no assistance is applied. In geneal, the shape of an obstacle is difficult to detemine fom the sensoy infomation of the envionment. So ceating a desied path fo obstacle 64

78 avoidance is not feasible. This epulsive foce field povides assistance fo the opeato to go aound the obstacle. With these assistances, fou uses executed the same tasks fo multiple times. Evey time, the system pefomance was consistent and had vey little vaiation. Two andom tajectoies fom diffeent subjects ae shown below. The fixtue helped significantly fo path following, pimaily due to the fact that the constaints applied to the PhanToM tool tip could foce it to back up once thee was some deviation fom the path. Most of the time, the velocity component was much smalle compaed to the velocity component tangential to the path. The lage othogonal velocity occus when the use is aligning with a taget o avoiding an obstacle. Time (ms) Figue 4.16 Velocity Components with Assistance 65

79 Table 4.3 Pefomance Summaies with Assistance Subject Path Eo (mm) Execution Time(s) Mean Stdev Mean Stdev Task Tajectoy test 1 test 6 Y-axis (mm) X-axis(mm) Figue 4.17 Tajectoies with Assistance 4.6. Summay Hidden Makov Model is effective fo the classification of andom pocesses such as human s motion intention in a teleopeation task. As long as the taining set is sufficiently lage, the motion ecognition accuacy is close to 1%. The selected assistance based on the ecognized motion is appopiate fo each type of motion. The 66

80 expeimental esults without assistance have shown that the opeato always has andom eos that esult in difficulty in following a path and aligning with a taget. The expeimental esults with assistance showed that the undesied andom eos wee emoved o educed. The HMM based assistance is useful fo impoving pefomance accuacy and deceasing execution time. These esults indicate that the appopiate assistance appoach selection based on motion intention is possible. Based on the opeato s motion intention, it is possible to detemine if an object is a taget o an obstacle. In ode to impove the ecognition accuacy, the dimension numbe of the Hidden Makov Model can be expanded. As long as they all ae independent, the added dimensions will only linealy incease the computational equiements. 67

81 Chapte 5: Robotic Theapy fo Pesons with Disabilities Using Skill Leaning This chapte descibes the Hidden Makov Model (HMM)-based skill leaning and its application in a motion theapy system using a haptic inteface. A elatively complex task, moving along a labyinth, is used. A nomal subject executes this task fo a numbe of times and the labyinth skill is leaned by Hidden Makov Model. The leaned skill is consideed as a vitual theapist who can tain pesons with disabilities to complete the task. Two pesons with disabilities on uppe limb (ceebal palsy) wee tained by the vitual theapist. The pefomance befoe and afte theapy taining, including the smoothness of the tajectoy, distance atio, time taken, temo and impact foces ae pesented in this chapte. This labyinth can be used as a theapy platfom fo uppe limb coodination, temo eduction and motion contol impoving Motion Theapy Much evidence suggests that intensive theapy impoves movement ecovey [78, 79]. But such theapy is expensive, because it equies theapists on a peson-to-peson basis. Recently, thee has been an inceased inteest in estoing functions though obotaided theapy. This appoach is to design theapy platfom, such as foce fields and moving constaints, to substitute theapist s wok. In this chapte, the ole of the theapist is eplaced by the leaned skill. When humans execute a task, thei actions eflect the skill associated with that task. When one does a paticula task many times, each time the 68

82 pefomance is diffeent even though it epesents the same skill. Fo example, when one daws 5 cicles of the same adius by hand, each cicle will be diffeent fom the othes although they may look close. But any one of the 5 cicles is the esult fom opeato s cicle-dawing skill. The diffeent looking of these cicles is due to the andom contol commands fom bain and the andom movements of hand. Since Hidden Makov Model is feasible to model a stochastic pocess, such as speech signal, it is possible to chaacteize the skill of the uppe-limb motion fo a specific task. In this dissetation, we have modeled the human movement along a labyinth so that the undelying natue of it is evealed and can be used to tansfe the skill to people with disabilities. It is desied that pesons with disabilities can be tained fo manipulation capabilities, which ae incementally impoved though leaning pactice. Leaning fom obsevation is a paadigm whee one obseves othe pesons pefomance and leans fom it. This is also like physical theapy fo a specific disability. 5.. Hidden Makov Model Based Skill Leaning In this dissetation, we model the motion of moving along a labyinth task skill using HMM. In ode fo the use to visualize the vitual theapist moe effectively, the tajectoy of the movement is chosen as the skill fo leaning. Since we only conside the movement in X-Y plane, position coodinates, Px and Py ae used to epesent the movement. In chapte 4, it has been explained how to convet continuous velocity data into discete symbols. Simila pocedues ae used in this chapte to convet continuous position data into discete symbols. 69

83 5..1. Raw-data Convesion The aw data used by HMM fo motion intention in chapte 4 is the use s velocity. In this chapte, the aw data is the tanslation tajectoy, P x, P y. In ode to use discete HMM, we still need to convet aw data into symbols. The pocedues will be explained in this section. Fist of all, the tanslation tajectoy is sampled by 1Hz ate. Since P x and P y ae independent vectos and pocessed in the same way, we just demonstate the pepocessing pocedues of P x. Fo simplicity, we use an example with less data. Let us assume that the position samples fo a specific task esult in the following 3 vectos. V1 = [ ]; V = [ ]; V3 = [ ]; The vecto length is 16 points. In othe wods, we cut evey 16 points and fom a vecto. These vectos ae so called aw data. Thei wavefoms ae shown in Figue 5.1. They do not have much useful infomation, just like ou voice signal wavefom in time domain. So we need to do some tansfomation. As illustated in chapte 4, each aw data vecto is multiplied by a Hamming window and then tansfomed by 16-point FFT. It is well known that the esult of FFT is a symmetical vecto. So in ode to educe computation complexity, only half of the FFT esult is used in PSD computation. The 3 vectos shown peviously ae tansfomed into the following 3 vectos with 8-point length. 7

84 P1 = 1 4 *[ ]; P = 1 4 *[ ]; P3 = 1 3 *[ ] Figue 5.1 Raw-data Vectos x x x Figue 5. PSD Vectos 71

85 If this task is executed 1 times, we will have 3 PSD vectos. Fo a simple task, we can use all these 3 vectos in the computations. But fo geneal applications in eal life, this numbe could be vey huge. It is impossible to do the computations using all of these vectos. This is why we need to do vecto quantization. As fo as vecto quantization, it is an algoithm to goup vectos into diffeent clustes accoding to the vecto distance citeia. The numbe of the clustes is detemined based on the application and accuacy. Fo some simple applications, usually 3 o 64 will be enough. The set and the numbe of the clustes ae called codebook and codebook length. The clustes ae called codewods of the codebook. The lage is the codebook length, the highe accuacy is the gouping. Fo this simple example, the length of the codebook fo vecto quantization is detemined to be 4. Figue 5.3 shows the illustation of vecto quantization when the codebook length is 4. In othe wods, the vecto quantization is to divide the whole vectos set into 4 clustes accoding to how the vectos ae close to each othe. Thee ae many available vecto quantization algoithms in liteatue. The well known one is LBG [7]. Figue 5.3 Vecto Quantization When Codebook Length is 4 7

86 Once the codebook is obtained, we can use it as a template to convet any vecto into discete symbols. As a matte of fact, the 3 vectos used in vecto quantization can also be epesented by symbols. If we epesent each cluste by a symbol, (fo example, A o 1 epesents cluste 1, B o epesents cluste and so on), we may expess the 3 PSD vectos as ABACDCDBACDCDABACDCAABDACDABDB o This is the esult of data pepocessing. When new vectos come in, they will be compaed with codewods and placed into the coesponding clustes with which the vectos ae closest, thus conveting aw position data into discete symbols. We did this so that we can use discete Hidden Makov Model to do all computation Hidden Makov Model Computation Let us assume that we executed this task 3 times to do skill leaning. We need to detemine which one of the thee task executions epesents ou skill. Fo each task execution, thei aw position data is pepocessed and conveted into 3 discete symbols. Let us assume that the symbols fom the fist task execution ae ABA, the second one CBD, and the thid one BDB. All symbols fom these thee task executions will be used as the taining set. So the taining set fo HMM is ABACBDBDB. In ode to explain the computation clealy, we use a two states left-ight HMM as shown below. Figue 5.4 Two-state Left-ight Hidden Makov Model 73

87 Befoe taining, all paametes of HMM ae initialized by andomly geneated pobability values, as shown below. π.5.75 = [.35.65], A =, B = Using the taining set ABACBDBDB, these HMM paametes ae updated using the same algoithm explained in chapte 4. Afte taining, the HMM paametes ae: π.3.7 = [.5.5], A =, B. 1. = The HMM with the adjusted paametes is shown in Figue 5.5: Figue 5.5 Hidden Makov Model with the Adjusted Paametes Once the HMM is tained by the taining set, they can be used to evaluate any given obsevation sequence. The evaluation citeion is the fowad scoe of a sequence 74

88 of obsevations given in a model. The fowad scoe of a given obsevation sequence is computed as follows: α i) = b ( o ), 1 i N (5.1) 1( πi i 1 N 1 t T 1 α t 1( j) = αt ( i 1) aij bj( ot 1), (5.) i= 1 1 j N N 1, o,... o t ) = T ( j) j = 1 P( o λ α (5.3) whee N is the numbe of states ( in this case), T is the time coesponding to a symbol. Fo the HMM tained by the combination of the thee-time execution data, we might as well evaluate the fowad scoe of each task execution. Fo the fist task execution, the obsevation sequence is ABA. The fowad scoe of this obsevation set is computed as following: α 1 ( 1) = π 1b1 ( A) =.5.1 = α ) = π b ( A) =.5.4 α α 1 ( =.5. (1) [ α1(1) a11 α1() a] b1 ( B) = [.5.3. ].5= 7. 5 ( ) = [ α1(1) a1 α1 () a ] b ( B) = [ ].5 = = e α α 4 3(1) [ α(1) a11 α() a1] b1 ( A) = [7.5e ].1= () = [ α (1) a1 α () a] b ( A) = [7.5e ].4 = = e 5.47 P ( O = ABA λ ) = α 3 (1) α 3() =.47 So the fowad scoe of the obsevation sequence ABA is

89 Fo the second task execution, the obsevation sequence is CBD. The fowad scoe of this obsevation sequence is computed in the same way: α 1 ( 1) = π1b1 ( C) =.5.6 =.3 α ) = π b ( C) = ( = α α (1) [ α1(1) a11 α1() a ] b1 ( B) = [ ].5 = 4. 5 ( ) = [ α1(1) a1 α1() a ] b ( B) = [ ].5 = = e.13 3 α α 3 3(1) [ α(1) a11 α() a1] b1 ( D) = [4.5e.3.13 ].5 = () = [ α(1) a1 α() a] b ( D) = [4.5e ] =. = e 4 P( O = CBD λ ) = α e 3(1) α 3() = Fo the thid task execution, the obsevation sequence is BDB. The same way, its fowad scoe is: P( O = BDB λ ) = α e Since 3(1) α 3() = P( O 4 4 = ABA λ ) =.47 > P( O = BDB λ) = 6.844e > P( O = CBD λ) = 3.375e It can be concluded that the task execution with ABA obsevation epesent the task skill moe closely than the othe two obsevation sequences. In othe wods, the task execution whose obsevation sequence has the highest fowad scoe epesents the task skill Expeiments in Vitual Envionment Tasks and Expeimental Test-Bed To evaluate the validity and effectiveness of the HMM fo skill leaning and its application fo theapy, we designed a haptic inteactive simulation test bed (Figue 5.6). 76

90 It is composed of a visualization scene and a PhanToM Pemium 1.5 [48]. The PhanToM is an impedance haptic device that can povide foce eflection to opeatos if collision happens. The simulation scene is ealized though API functions of GHOST [48]. The end-effecto is simulated as a sphee whose adius is 5mm. The width of the labyinth is 18 mm. In this expeiment, the task is defined to move the end-effecto fom the oigin (,, ) to get out of the labyinth as quickly and smoothly as possible, and with as few collisions as possible. In ode to avoid the depth peception poblem, opeatos ae only allowed to move in the X-Y plane by adding a plana constaint to the haptic device. Badofe et al used this haptic inteface to do motion analysis of uppe-limb fo patients with neuological diseases (ND), but they did not ty to impove the manipulation pefomance [8]. Figue 5.6 Vitual Envionment fo Simulation Test-bed 77

91 5.3.. Skill Leaning and Tansfeing HMM is used to model the tanslation skill of moving along the labyinth. The leaned skill is late used as a vitual theapist in motion taining. This task was executed twelve times to poduce the taining set fo HMM by a nomal subject. The tanslation data of the end-effecto is ecoded and conveted into discete symbols using the pepocessing appoach as illustated in section The discete symbols of these task executions ae used to tain HMM. Once the HMM has been tained, it can be used to evaluate each task execution. The set of symbols that poduce the lagest fowad likelihood P(O M) coespond to the motion that is most likely executed by the nomal subject. In othe wods, it epesents the skill needed by that specific task. We use a 5- state, left-ight, two dimensional HMM fo skill leaning. So the pio matix π is a 1 5 matix, the tansition matix A is a 5 5 matix with each ow epesenting the tansition pobability fom a cetain state to othe states. It is necessay to note that we have two obsevability matices B, each of which is π, A and B matices ae initialized by the unifomly distibuted andom numbe as usual. Stating with these initial paametes, the HMM is tained by the taining test. The fowad algoithm was used to scoe each tajectoy (Figue 5.7). It can be seen that No. 7 is the highest and No. 6 is the lowest in the pobability values. It is impotant to note that the best (highest) o wost (lowest) scoes do not efe to the pefomance, but to the accuacy of epesenting the skill of doing the task. Fo example, if we ae asked to daw many line segments with the same diection and length, it would be likely that we would daw a couple of close to pefect ones and a couple of vey bad ones. But these exteme cases do not epesent ou linedawing skill. The lines that we ae most likely to daw epesent ou line-dawing skill. 78

92 The tajectoy with the highest scoe epesents the tanslation skill of the subject most likely to do this task Pobability Data Index Figue 5.7 Fowad Scoes fo all 1 Times of Task Execution 5.4. Motion Theapy Expeiments Since the skill of this task has been leaned, the tajectoy of the leaned skill is displayed on the sceen acting as a theapist. Duing the theapy taining session, opeatos ty to follow it as accuately as possible (Figue 5.6). Two subjects: one is female, ceebal palsy with ight hemipaesis and spasticity, pesistent low back pain; the othe is male, 19, ceebal palsy, patial paalysis of his uppe and lowe extemities, executed this task seven times each befoe and afte taining. Befoe collecting data, they pacticed this movement fo seveal times until they felt comfotable about it. Thei data, including tanslation, velocity and eaction foces, wee sampled at 1Hz. The evaluation indexes include: 79

93 Distance atio R d. Its value eflects the tajectoy optimization capabilities. The smalle, the bette. The ideal value is a little geate than 1. d actual R d = (5.4) d skill whee d actual is the actual distance taveled, d skill is the distance taveled by the leaned skill. Time taken to complete the task T; Numbe of collisions with walls N c ; 1, Collide with wall C ( n ) = (5.5), elsewhee N c = ( C ( j 1 ) C ( j ) = 1 ) (5.6) j Time duation of the collisions T i. It eflects eaction capabilities. T (5.7) i = t j, C ( j 1) C ( j ) = 1 t j, C ( j 1) C ( j ) = 1 Impact foce of the collisions with walls F i ; F = F F (5.8) i i, x i, y whee F i,x and F i,y ae impact foces when the end-effecto collides with the X- diection wall and Y-diection wall espectively. Temo magnitude M t and fequency F t. Fo motion analysis, opeato s collisions with X-diectional wall and Y-diection wall do not make much diffeence. So only the magnitude of impact foce is analyzed. The diection of impact foce is not meaningful. Temo infomation is extacted by 8

94 applying a high pass filte, which has a cut-off fequency fc = fmax/1 (fmax is maximum temo fequency). Temo magnitude is available in time domain. The temo fequency can be obtained though discete Fouie tansfom (DFT). The collision foces along X- and Y-axes ae combined and the magnitude of the combined foce was analyzed. C(n) indicates the case when collisions occu. Nc is the numbe of collisions occuing duing task execution. Ti is the time duation of each collision. Nc and Ti ae obtained though checking the tansition of C(n) between and Motion Pefomance befoe Theapy Taining Two pesons with disabilities pefomed the task befoe and afte theapy taining. Figues 5.8, 5.9, and 5.1 pesent the pefomance of subject 1 befoe taining. Figue 5.8 shows an actual tajectoy and the skilled tajectoy. Figue 5.9 shows the tanslation temo along X and Y-axes, including temo magnitude and fequency. Figue 5.1 pesents the collision infomation, including the impact foce and the time duation fo each collision occuing. 81

95 1 8 tanslation with temo Actual Movement Skilled Tajectoy 6 4 Y-axis(mm) X-axis(mm) Figue 5.8 Actual Moving Distance is 716.8mm, Skill Moving Distance is 495.mm, and Distance Ratio is

96 3 x-tanslation temo 6 x-tan temo FFT magnitude(mm) numbe of sample y-tanslation temo fequency/fs y-tanslation tmo FFT 6 magnitude(mm) numbe of sample -5 5 fequency/fs Figue 5.9 Temo Measuements. X temo magnitude mean is 8.4mm and STD is 6.9mm. Y temo magnitude mean is 9.3mm and STD is 8.8mm 83

97 Reaction Foce(N) collision indicato time inteval(s) collision detection 1=collision,= no collision The numbe of sample The numbe of collision The numbe of sample Figue 5.1 Collisions: 15 Collisions Occued. The max time duation is 5.87s and the minimum is.14s. The max impact foce is 1.1N and the minimum is.15n 84

98 5.4.. Motion Pefomance afte Theapy Taining Afte theapy taining, the data fo each subject was collected. The analysis fo subject 1 is pesented in Figues 5.11, 5.1, and tanslation with temo Actual Movement Moving Skill 6 4 Y-axis(mm) X-axis(mm) Figue 5.11 Tajectoies afte Theapy Taining Actual Moving Distance is and Distance Ratio is

99 15 x-tanslation temo 1.5 x-tan temo FFT magnitude(mm) numbe of sample y-tanslation temo -5 5 fequency/fs y-tanslation tmo FFT magnitude(mm) numbe of sample -5 5 fequency/fs Figue 5.1 Tanslation Temos Afte Theapy. X-axis Temo Magnitude Mean Is 5.7 mm and STD Is Y-axis Temo Magnitude Mean Is 6.71 and STD Is

100 collision indicato time inteval(s) Reaction Foce(N) collision detection 1=collision,= no collision The numbe of sample x The numbe of collision The numbe of sample Figue 5.13 Collisions Afte Theapy. 6 Collisions Occued. The maximum impact foce is.39n and the minimum is.18n 87

101 The X-Y plane tajectoy pesents movement quality. The smoothness eflects the capability of contolling the end-effecto duing movement. The temo infomation plots pesent temo magnitude, without consideing its diection since magnitude is moe meaningful than diection. The temo fequency was always low, -3 Hz fo the two subjects. Impact foce occus when thee is a collision. Geneally, the impact foce is elated to the smoothness of the tajectoy. The smoothe the tajectoy is, the smalle the temo magnitude is. The time duation of each collision indicates the eaction to collision. Fom figues 5.8 and 5.11, it can be seen that the tajectoy was impoved significantly. Figues 5.1 and 5.13 show the collision infomation befoe and afte the theapy taining, espectively. As we can see, the numbes of collisions, the collision duations and the impact foces wee deceased. Though the temo magnitude was educed consideably, the temo fequency was about the same. This is due to the fact that temo fequency is not obsevable to the use. Befoe and afte theapy taining, seven tials of execution data wee collected fo each subject. The pefomance summay is pesented in Table

102 Table 5.1 Movement Pefomance Summay Subject 1(Femal, ceebal palsy with ight hemipaesis and spasticity (Mean / Std) Subject (male, 19, ceebal palsy, patial paalysis) (Mean / Std) Befoe Taining Afte aining Befoe Taining Afte Taining Length Ratio R Time taken(s) Collision Numbes X temo Mag-(mm) Y-Temo Mag-(mm) Temo feq(max) MaxTime Duation(s) Impact foce(max,n) 1.68/ / /.4 1.1/ / / /.8 1.4/ / / / / / / / /.3 1.1/ / / / Hz/-- 3.4Hz/--.8Hz/--.5Hz/ / / / /.65 1./.53.71/.33.89/.35.65/. 89

103 5.5. Summay In this chapte a HMM based appoach fo labyinth moving skill leaning and tansfeing of the leaned skill to pesons with disabilities is pesented. The two dimensional model is built fo the XY plane tanslation. The leaned skill is not the best o the wost one of the numeous task executions, but the one that the opeato is most likely to do. That is, the most natual one. The leaned skill was used as a vitual theapist fo pesons with disabilities. Pesons with disabilities wee asked to follow the vitual theapist as closely as possible. The diffeence between the subject and the vitual theapist povides visual feedback which helps the eye-hand coodination contol capability. Afte seveal times of theapy taining, opeatos could contol the end-effecto bette, and hence educing collisions and making the tajectoy smoothe. 9

104 Chapte 6: Conclusions and Recommendations 6.1. Dissetation Oveview An intelligent teleopeation system using assistance functions was developed to impove task execution efficiently and to decease the execution time. The appoach was guided by the philosophy that the human opeato should emain in the contol loop of the slave manipulato, thus using human intelligence fo the teleobotics system contol. A common ehabilitation evaluation task, Box and Blocks was tested using teleopeation assistance functions. The esults showed how the desied motion was kept o sometimes augmented and how the unwanted motion was educed. Complex telemanipulation tasks wee decomposed into geneal and elatively simple subtasks: following a path, aligning with a taget, avoiding an obstacle and stopping. Hidden Makov Model was used to classify human motion intention into one of the fou classes. Fo diffeent subtasks, appopiate assistance was applied to enhance the input fom maste device. Anothe ehabilitation-obotics application is motion theapy. Using HMM, a labyinth movement skill was leaned by the obot. The leaned skill then acted as a vitual theapist and two pesons with disabilities on uppe limb wee tained using this appoach. The skill leaning based obot theapy and its effectiveness wee discussed. 6.. Vitual Fixtue Assistance Based on Motion Intention In telemanipulation systems, assistance though vaiable position and velocity mapping o vitual fixtue can impove manipulation capability and dexteity. This 91

105 assistance is useful not only fo path following, but also fo aligning with tagets and avoiding obstacles. Conventionally, such assistance is based on the envionmental infomation and without knowing the use s motion intention. In this dissetation, use s motion intention is combined with eal-time envionmental infomation to apply appopiate assistance. If the cuent task equies following a path, a had vitual fixtue othogonal to the path is applied. Similaly, if the task is to position a taget, an attactive foce field is poduced to povide a guide fo appoaching. Hidden Makov Model is effective fo motion classification. As long as the taining set is sufficiently lage, the motion ecognition accuacy is close to 1%. The assistance is appopiately selected based on the ecognized motion. The expeimental esults without assistance showed that the opeato always had andom eos that esulted in difficulty in following a path and positioning a taget. The expeimental esults with assistance showed that all those undesied andom eos wee emoved o educed. The HMM based assistance is useful fo impoving pefomance accuacy and deceasing execution time. In ode to impove the ecognition accuacy, the Hidden Makov Model can be expanded. As long as they ae independent, the added dimensions linealy incease the computation complexity Robot Theapy and its Effectiveness A HMM based appoach fo labyinth moving skill leaning and tansfeing the leaned skill to pesons with disabilities on thei uppe limb was pesented. The multidimensional model is built fo the leaning X-Y plane tanslation skill. The leaned skill was used as a theapist fo pesons with disabilities. They need to follow the vitual 9

106 theapist as close as possible. The diffeence between the subject and the vitual theapist povides visual feedback that helps the eye-hand coodination contol capability. Duing the taining pocess, the tajectoy smoothness did not impove significantly even though the use had less collisions and shote execution time. This could be due to the fact that opeatos tend to quickly withdawn the ball afte the collision to follow the continuously updated tajectoy. Afte many epetitions of theapy, opeatos wee able to contol the end effecto to avoid collisions and make the tajectoy smooth. They displayed some movements to avoid unnecessay body aangements and postued themselves accodingly. The pupose of theapy is to estoe some of the lost functions of pesons with disabilities. This obot-aided theapy emphasizes the movement contol though eye-hand coodination taining leaned fom nomal subject s pefomance. This compensation allows pesons with disabilities to impove uppe limb coodination; temo eduction and motion contol capabilities Geneal Discussion Oveall, when applying teleopeation assistance, the pefomance of subjects with disabilities can be enhanced. The esults of the vaious expeimental esults wee pomising, and indicated that the poposed assistances techniques have eal potential in speeding up the execution of a vaiety of tasks, impoving opeation accuacy and educing opeato s fatigue. The Hidden Makov Model based skill leaning poposed a new appoach fo motion theapy. While physical theapy diected by a theapist estoes the lost motion though physical execise, obot theapy supevised by a vitual theapist impoves eye-hand coodination by leaning fom a demonstato. 93

107 6.5. Recommendations The assistance algoithms wee tested by using simulation platfoms. It is ecommended to use a obot manipulato to test fo a vaiety of eal ehabilitation tasks. These tests could be implemented on the wokstation-based teleopeation system, which consists of a PhanToM Pemium 1.5 and PUMA o RRC manipulato, both of which will be available in ou laboatoy. Although teleopeation assistance povides vey valuable assistance fo complex task execution, autonomous execution fo some epetitive tasks equiing accuate fine tuning movement is ecommended. Fo the obot system in ou lab, compute vision can be configued to implement visual sevoing fo taget gasping. 94

108 Refeences 1. William S. Hawin, Taiq Rahman, and Richad A. Foulds, A Review of Design Issues in Rehabilitation Robotics With Refeence to Noth Ameican eseach, IEEE Tansactions on Rehabilitation Engineeing, VOL. 3, NO.1, Mach R. Allen, A.kachak, J., and E.L. Bontage, Design and fabication of a pai of Rancho anthopomophic am, Attending Staff Assoc. Rancho Los Amigos Hospital, Inc., and Tech. Rep., Geldeblom, G.J., Cemes G., and Soede,M. Review of Rehabilitation Robotics Application, Intenational Jounal of Human-Fiendly Welfae Robotics System, Vol.4, No Axel Gäse, Chistan Matens, Rehabilitation Robusts Tansfe of Development and Reseach Results to Disabled Uses, Intenational Jounal of Human- Fiendly Welfae Robotics System, Vol.3, No W. Seamone and G.Schmeisse, Ealy clinical evaluation of a obot am/woktable system fo spinal-cod-injued pesons. J.Rehab. Res. Dev., pp , Jan H. Roesle, H.J.Kuppes, and E.Schmalenbach, The medical manipulato and its adapted envionment: A system fo the ehabilitation of seveely handicapped, in IRIA Poc. Int. Conf. Telemanipulatos fo the Physically Handicapped, pp.73-77, John L. Dallaway, Robin D. Jackson, and Oaul H. A. Timmes, Rehabilitation Robotics in Euope, IEEE Tansactions on Rehabilitation Engineeing, VOL. 3, NO.1, Mach J. Guittet, H. H. Kwee, N.Quetin, and J.Yclon, The Spatacus telethesis: Manipulato contol and expeimentation, in IRIA Poc. Int. Conf. Telemanipulatos fo the physically Handicapped, pp H. H. Kwee, Spatacus and Manus: Telethesis developments in Fance and the Nethelands, in Inteactive Robotic Aids-One Option fo Independent living. Wold Rehabilitation Fund, 1986, pp

109 1. H. H. Kwee, J.J. Duimel, J.J. Smits, A.A. Tuinhofde Moed, and J.A. van Woeden, The Manus wheelchai-bone manipulato: System eview and fist esults, in IARP Poc. nd Wkshp. Medical and Healthcae Robotics, pp , M. Topping, Handy 1, a obotic aid to independence fo seveely disabled people, in Poc. 3 d Cambidge Wokshop Rehabilitation Robotics, pp.13-16, J.L. Dallaway and R.D. Jackson, RAID a Vocational obotic wokstation, in ICORR 9-confeence poceedings, Michael Hillman*, Kaen Hagan, Sean Hagan, Jill Jepson and Roge Opwood, A Wheelchai Mounted Assistive Robot, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA, U.S.A. 14. Philippe Hoppenot and Etienne Colle, Location and Contol of a Rehabilitation Mobile Robot by Close Human-machine Coopeation, in IEEE Tansaction of Neual System and Rehabilitation Engineeing, Vol.9, No., June Chistian Matens, Oleg Ivlev and Axel Gäse. Buga, Inteactive Contolled Robotic System FRIEND to Assist Disabled People, in 7 th Intenational Confeence on Rehabilitation Robotics, Fance, May H. F. Machiel Van De Loos, VA/Stanfod Rehabilitation Robotics Reseach and Development Pogam: Lessons Leaned in the Application of Robotics Technology to the Field of Rehabilitation, IEEE Tansactions on Rehabilitation Engineeing, VOL. 3, NO.1, Mach Vijay Kuma, Taiq Rahman and Venkat Kovi, Assistive Devices Fo People With Moto Disabilities, Wiley Encyclopaedia of Electonics Engineeing G.E.Bich, M. Fengle, R.G.gosine, K. Schoede, M. Schoede, M. Schoede and D.L. Johnson, An assessment methodology and its application to a obotic vocation assistive device, in Technology and Disability, 5(): , S. J. Sheedos, B. Taylo, C. B. Cobb and E. E. Dann, Peliminay evaluation of the helping hand electo-mechanical am. Technology and Disability, 5(): 9-3, Kazuhiko Kawamua, Sugato Bagchi, Moenes Iskaous, and Magued Bishay, Intelligent Robotic Systems in Sevice of Disabled, IEEE Tansactions on Rehabilitation Engineeing, VOL. 3, NO.1, Mach Jin-Woo Jung, Won-Kyung, Heyoung Lee and Jong-Sung Kim, A Study on the Enhancement of Manipulation Pefomance of Wheelchai-Mounted Rehabilitation Sevice Robot, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA. 96

110 . Kelly McClenathan and Taiq Rahman, Powe Augmentation in Rehabilitation Robots, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA, U.S.A. 3. Chistine Wight-Ott, The GOBOT: A Tansitional Poweed Mobility Aid Fo Young Childen With Physical Disabilities, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA, U.S.A. 4. N. Didi, M.Mokhtai, A.Roby-Bami, Pepogammed Gestues fo Robotic Manipulatos: An Altenative Speed up Task Execution Using MANUS, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA, U.S.A. 5. Richad M. Mahoney, The Rapto Wheelchai Robot System, in 7 th Intenational Confeence on Rehabilitation obotics, Fance, May Mike Topping, Handy 1, A Robotic Aid to Independence fo Seveely Disabled People, in 7 th Intenational Confeence on Rehabilitation obotics, Fance, May O. Ait Aide, P. Hoppenot, E.Colle, Localization by Camea of a Rehabilitation obot, in 7 th Intenational Confeence on Rehabilitation obotics, Fance, May Zeungnam Bien, Won-Kyung Song, Dae-Jin Kim, Jeong-Su Han, Visionbased Contol with Emegency Stop though EMG of the Wheelchai-based Rehabilitation Robotic Am, KARES II, in 7 th Intenational Confeence on Rehabilitation obotics, Fance, May Steven Edwad Eveett, Human-Machine Coopeative Teleobotics Using Uncetain Senso and Model Data. Ph.D. Dissetation, The Univesity of Tennessee, Knoxville, P.G. Backes, Multi-senso based impedance contol fo task execution, In Poceedings of the 199 IEEE Intenational Confeence on Robotics and Automation, pages , Nice, Fance, May, Schuyle, R. Mahoney. Job Identification and Analysis fo Vocational Robotics Applications. Poceedings RESNA L. Leife, Rehabilitation Robots, Robotics Age, pp 4-15, May/June

111 33. T. F. Chan and R. V. Dubey, "Genealized Bilateal Contolle fo a Teleopeato System with a Six DoF Maste and a Seven DoF Slave," Poceedings of the IEEE Intenational Confeence on Robotics and Automation, San Diego, Califonia, May 8-13, 1994, pp R. V. Dubey, T. F. Chan and S. E. Eveett, Vaiable Damping Impedance Contol of a Bilateal Teleobotic System, IEEE Contol System Magazine, Febuay K. Sato, M. Kimua, and A. Abe, Intelligent Manipulato System with Nonsymmetic and Redundant Maste-Slave, Jounal of Robotic System, 9():81-9, Luc D. Joly and Claude Andiot, "Imposing motion constaints to a foce eflecting teleobot though eal-time simulation of a vitual mechanism," in Poceedings of the 1994 IEEE Intenational Confeence on Robotics and Automation, San Diego, CA, May 1994, pp Kazuhio Kosuge, Koji Takeo, and Toshio Pukuda, "Unified appoach fo teleopeation of vitual and eal envionment manipulation based on efeence dynamics " in Poc. of the 1995 IEEE Intenational Confeence on Robotics and Automation, Nagoya,Japan,May Thomas B. Sheidan. Teleobotics, Automation, and Human Supevisoy Contol. The MIT Pess, London, England, Gunna Bolmsjo, Hakan Neveyd, and Hakan Efting. Robotics in Rehabilitation. IEEE Tansactions on Rehabilitation Engineeing, Vol. 3, No. 1, Mach Noali Penalete. "Development Of A Robotic Haptic Inteface To Pefom Vocational Tasks By People With Disabilities." Ph.D. Dessetation, Depatment of Electical Engineeing, Univesity of South Floida, Decembe Koivo, A.J., Houshangi, N. Real-time vision feedback fo sevoing obotic manipulato with self-tuning contolle, Systems, Man and Cybenetics, IEEE Tansactions on, Volume: 1, Issue: 1, Jan.-Feb. 1991, Pages: Jia Li; Najmi, A.; Gay, R.M.; Image classification by a two-dimensional hidden Makov model. Signal Pocessing, IEEE Tansactions on [see also Acoustics, Speech, and Signal Pocessing, IEEE Tansactions on], Volume: 48, Issue:, Feb. Pages:

112 44. C. Stange, C. Angling, W. Hawin, D. Romilly. Devices fo Assisting Manipulation: A Summay of Use Task Pioities. IEEE Tansactions on Rehabilitation Engineeing. Vol, No 4, Decembe Steven E. Eveett, Rajiv V. Dubey, Y. Isoda, and C. Dumont. Vision-Based End-Effecto Alignment Assistance fo Teleopeation. Poceedings of the IEEE Intenational Confeence on Robotics and Automation, Detoit, MI.. May 1999, pp Jie Yang, Yangshen Xu, Chiou S. Chen, Hidden Makov Model Appoach to Skill leaning and Its Application to Teleobotics, IEEE Tansactions on Robotics and Automation, Volume: 1 No.5,Oct B. Hannafod and P. Lee, Hidden Makov Model Analysis of Foce/Toque Infomation in Telemanipulation. The Intenational Jounal of Robotics Reseach, Oct.1991, Vol. 1, No.5, pp SensAble Technologies Robotics Reseach Copoation, P.O. Box 6, Amelia, OH 451. K-Seies Robot Ams Uses Manual, Publication UM Wentao Yu, Benjamin Fitz, Noali Penalete, Michael Juczyk,Rajiv V. Dubey "Sensos Assisted Telemanipulation fo Maximizing Manipulation Capabilities of Pesons With Disabilities", Haptics Symposium 3, Los Angles, CA, Kazuhiko Kawamua, Sugato Bagchi, Moenes Iskaous, and Magued Bishay, Intelligent Robotic Systems in Sevice of Disabled, IEEE Tansactions on Rehabilitation Engineeing, VOL. 3, NO.1, Mach S. Hayati and S. T. Venkataman, Design and Implementation of a Robot Contol System with Taded and Shaed Contol Capabilities, In Poceedings of the 1989 IEEE Intenational Confeence on Robotics & Automation, pages , Scottsdale, AZ. May N. Penalete, Wentao Yu, R. V. Dubey, W.A. Moeno, Development of an Intelligent Mapping Based Teleobotic Manipulation System To Assist Pesons With Disabilities. In Poceedings of the IEEE Intenational Confeence on Robotics & Automation, Washington, DC U.S.A., May. 54. J.Yang, Y.Xu and C.S. Chen, "Hidden makov Model Appoach to Skill Leaning and its Application to Teleobotics," IEEE Tans. On Robotics and Automation, vol.1, no.5, pp.61-31,

113 55. A. Bettini, S. Lang, A. Okamua and G. Hage, "Vision Assisted Contol fo Manipulation Using Vitual Fixtues," IEEE/RSJ Intenational Confeence on Intelligent Robots and Systems, 1, pp Gegoy D. Hage, A Modula System fo Robust Positioning Using Feedback fom Steeo Vision", IEEE Tansactions on Robotics and Automation, vol.13, No.4, August Young S. Pak, Hyosig Kang, Tomas F. Ewing, Eic L. Fauling, J. Edwad Colgate, Michael A. Peshkin, " Enhanced Teleopeation fo D & D, in the IEEE Intenational Confeence on Robotics and Automation, New Oleans, Jiang Wang, William J. Wilson, " 3D Relative Position And Oientation Estimation Using Kalman Filte Fo Robot Contol", in Poceedings of the 199 IEEE Intenational Confeence on Robotics and Automation, Nice, Fance-May Seth Hutchinson, Gegoy D. Hage and Pete I. Coke, "A Tutoial on Visual Sevo Conto," IEEE Tansactions on Robotics and Automation, vol.1, No.5, Octobe Benad Espiau, Fancois Chaumette, and Patick Rives, "A new Appoach to Visual Sevoing in Robotics", IEEE Tansactions on Robotics and Automation, vol.8, No.3, June P. Maayong, A. Bettini and A. Okamua, Effect of Vitual Fictue Compliance on Human-Machine Coopeative Manipulation, in IEEE/RSJ poceedings, Lausanne, Switzeland, Oct. 6. Jie Yang, Yangshen Xu, Chiou S. Chen, Human Action Leaning via Hidden Makov Model, IEEE Tansactions on Systems,Man, And Cybenetics-Pat A: Systems and Humans, Volume: 7 No.1, Januay L. R. Rabine, A Tutoial on Hidden Makov Model and Selected Applications in Speech Recognition, Poceedings of the IEEE, Volume: 77 Issue:, Feb Z. Stanisic, S. Payandeh, and E. Jackson. Vitual fixtue as an aid fo teleopeation., in 9 th Canadian Aeonautic and Space Inst. Confeence., Chistophe Collewet, FanÇois Chaumette, and Philippe Loisel, Image-based visual sevoing on plana objects of unknown shape, in Poceedings of IEEE Intenational Confeence on Robotics and Automation, Seoul, Koea, May 1. 1

114 66. W. Yu, N. Penalete, R. V. Dubey Telemanipulation Enhancement though Use s Motion Intention Recognition and Vitual Fixtue, submitted to IEEE Intenational Confeence on Intelligent Robots and Systems N. Penalete, W. Yu, R. Dubey. Augmentation of manipulation Capabilities of Pesons with Disabilities Using Scaled Telemanipulation, IEEE/RSJ Intenational Confeence on Intelligent Robots and Systems, Lausanne, Switzeland, Octobe. 68. Rosenbeg, L.B.; Vitual fixtues: Peceptual tools fo teleobotic manipulation, Vitual Reality Annual Intenational Symposium, IEEE, 18- Sep 1993 Page(s): Payandeh, S.; Stanisic, Z. On application of vitual fixtues as an aid fo telemanipulation and taining, Haptic Intefaces fo Vitual Envionment and Teleopeato Systems,. 1th Symposium. 7. Peppe, R.L.and Kaomea, P.K. Reseach Issues in Teleopeato Systems, 8 th annual Human Factos Society Meeting, San Antonio, Tx, Bolmsjo, G.; Neveyd, H.; Efting, H.; Robotics in ehabilitation, Rehabilitation Engineeing, IEEE Tansactions, Volume: 3 Issue: 1, Ma 1995 Page(s): R.M. Gay, Vecto quantization, IEEE ASSP Mag., vol.1,no.. pp.4-9, N. Tuo, O.Khatib,E.Coste-Maniee, Haptically Augmented Teleopeation, Intenational Confeence on Robotics and Automation. Seoul, Koea, May Roge Y. Tasi, Reima K. Lenz, " A new Technique fo Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibation", IEEE Tansactions on Robotics and Automation, vol.5, No.3, June Y.Shiai, H. Inoue, Guiding a obot by visual feedback in assembly tasks", Patte Recognition., vol.5, pp.99-18, G. Puskoius and I. Feldkamp, Calibation of obot vision, in Intenational Confeence on Robotics and Automation. Raleigh, NC The Softwae Solution fo Machine Vision Application. 78. C. Butefisch, H. Hummelsheim, P. Denzle, and K. Mauitz, "Repetitive taining of isolated movement impoves the outcome of moto ehabilitation of the centally paetic hand," Jounal of the Neuological Sciences, vol. 13, pp ,

115 79. Mye Kutz, McGaw-Hill, Biomedical Enginee s Handbook,. 8. Aleš Badofe, Mako Munih, Anton Zupan, Alenka Pimožic, Uppe Limb Motion Analysis Using Haptic Inteface, IEEE/ASME Tansactions on Mechatonics, VOL.6, NO.3, Septembe Allen, P.K; Timcenko, A.; Yoshimi, B.; Michelman. P; Tajectoy filteing and pediction fo automated tacking and gasping of a moving object, IEEE Intenational Confeence on Robotics and Automation, May 1-14, Hage, G.D.; Gunwald, G.; Hizinge, G.; Featue-based visual sevoing and its application to teleobotics, IEEE/RSJ Intenational Confeence, Sept Robotics Reseach R Contolle Manual V1.4, Bian P. DeJong, J. Edwad Colgate, Michael A. Peshkin, Impoving Teleopeation : Reducing Mental Rotations and Tanslations, IEEE Intenational Confeence on Robotics and Automation, Apil 6-May 1, B. Volpe, H. Kebs, N. Hogan, L. Edelstein OTR, C. Diels, and M. Aisen, "A novel appoach to stoke ehabilitation: obot-aided sensoy moto stimulation," Neuology, vol. 54, pp ,. 86. M. L. Aisen, H. I. Kebs, N. Hogan, F. McDowell, and B. Volpe, "The effect of obot assisted theapy and ehabilitative taining on moto ecovey following stoke," Ach. Neuol., vol. 54, pp , H. I. Kebs, N. Hogan, B.T. Volpe, M.L.Aisen, L. Edelstein, and C. Diels, Robot-Aided Neuo-Rehabilitation in Stoke: Thee-Yea Follow-Up, in Intenational Confeence on Rehabilitation Robotics, Stanfod, CA, U.S.A. 88. H. I. Kebs, B.T. Volpe, B.Rohe, M. Feao, S. Fasoli, L. Edelstein, and N. Hogan, "Robot-Aided Neuo-Rehabilitation in Stoke: Inteim Results on the Follow-Up of 76 Patients and on Movement Pefomance Indices," in 7 th Intenational Confeence on Rehabilitation obotics, Fance, May H. I. Kebs, N. Hogan, M. L. Aisen, and B. T. Volpe, "Robot-aided neuoehabilitation," IEEE Tans. Rehab. Eng., vol. 6, pp , D. J. Reinkensmeye, L. E. Kahn, M. Avebuch, A. N. McKenna-Cole, B. D. Schmit, and W. Z. Ryme, "Undestanding and teating am movement impaiment afte chonic bain injuy: Pogess with the ARM Guide," Jounal of Rehabilitation Reseach and Development, vol. 37, pp ,. 1

116 9. Leonad E. Kahn, Michele Avebuch, W. Zev Ryme, David J. Reinkensmeye, Compaison of Robot-Assisted Reaching in Pomoting ecovey Fom Chonic Stoke, in 7 th Intenational Confeence on Rehabilitation Robotics, Fance, May D. J. Reinkensmeye, J. P. A. Dewald, and W. Z. Ryme, "Guidance based quantification of am impaiment following bain injuy: A pilot study," IEEE Tansactions on Rehabilitation Engineeing, vol. 7, pp. 1-11,

117 Appendices 14

118 Appendix A: System Testbed and Expeiment Design This chapte pesents the system test bed at ehabilitation obotics lab which is used by this poject. The hadwae and softwae used in the poject will be intoduced. A.1. Intoduction The peviously outlined concept was implemented on the hadwae and softwae in this laboatoy. This chapte descibes the hadwae used to test the new assistance stategy and the softwae we used in the testbed. A.. Hadwae Duing the couse of this poject, it was necessay to econfigue the peviously constucted teleobotic system used by students at Univesity of Tennessee at Knoxville [9]. The Kaft Maste Hand Contolle has been eplaced by a PHANTOM pemium 1.5. The cuently used hadwae and coesponding schematic ae descibed in this section. A..1. Robotics Reseach Copoation Manipulato The Rehabilitation Robotics and Telemanipulation Laboatoy in the Mechanical Engineeing Depatment at the Univesity of South Floida uses a seven-degee of feedom obot manipulato fom Robotics Reseach Copoation (RRC), model k-17, as the emote manipulato. The manipulato has seven evolute joints boasting a edundant joint fo obstacle avoidance. Joints 1, 3, 5, and 7 ae oll type joints, while, 4, and 6 ae wist type joints. The total length of the am when all the joints ae positioned fowad, such as Figue A.1, 15

119 Appendix A (Continued) eaches.1 metes, about seven feet. Figue A.1 also shows the schematic of the obot manipulato s seven joints including and location of each joint and thei espective tavel limits. The tavel limits ae displayed in table A.1. The motions of the seven evolute joints and an end-effecto ae displayed in figue A.. Figue A.3 shows a pictue of the complete teleobotic system including the actual mounting of the obot manipulato on the hoizontal plane that is not eflected in the pevious figue. Figue A.1 RRC Manipulato Joints and Limits 16

120 Appendix A (Continued) Table A.1 Joint Limits fo the RRC Manipulato Joint Numbe Lowe Limit Uppe Limit Figue A. RRC Manipulato The manipulato uses a PC based contolle. The contolle uses inputs fom the compute s gaphical use inteface (GUI) o the teach pendent as the efeence position fo each of the seven joints. Fom these positions, the invese kinematics is calculated, and seven joint commands ae detemined and sent to the low level contolle. The obot contolle is capable of position, velocity, and toque contol fo the motos fo each of the seven joints to maintain the appopiate joint angles of the manipulato. 17

121 Appendix A (Continued) Figue A.3 RRC Manipulato with Sensos and End-Effecto A... PHANTOM Pemium 1.5 Figue A.4 PHANTOM Pemium

122 Appendix A (Continued) Developed by SensAble Technologies [48], the PHANTOM device epesents a esolution in human compute inteface technology. Pio to its invention, compute uses only had the capability to inteact though the sense of sight, and moe ecently, sound. The sense of touch, the most impotant sense in many tasks, has been conspicuously absent. The PHANTOM device changes all of this. Just as the monito enables uses to see compute-geneated images, and audio speakes allow them to hea synthesized sounds, the PHANTOM device makes it possible fo uses to touch and manipulate vitual objects. The PHANTOM haptic inteface is distinguished fom othe touch intefaces by what it is not. It is not a bulky exoskeleton device, a buzzing tactile stimulato no a vibating joystick. PHANTOM application aeas include medical and sugical simulation, geophysics and nanomanipulation. The device used in this poject is a pemium 1.5, whose spec is as follows: Table A. PHANTOM Pemium 1.5 Specifications Wokspace 7.5 x 1.5 x 15 inches/19.5 x 7 x 37.5 cm Range of motion Lowe am movement pivoting at elbow Nominal position esolution 86 dpi /.3 mm Back dive fiction.15 oz /.4 N Maximum Exetable Foce 1.9 lbf / 8.5 N Continuous Exetable Foce.3 lbf/ 1.4 N Stiffness lbs./in / 3.5 N/mm Inetia <.17 lbm < 75 g Footpint 1 x 13 inches / 5 x 33 cm Foce feedback x, y, z(3dof) Position sensing x, y, z tanslation and otation (6DOF optional) Inteface Via Paallel Pot Suppoted platfoms Intel-based PCs 19

123 Appendix A (Continued) A.3. Softwae Seveal independently unning pogams on vaious computes make up the softwae which acts to simulate telemanipulation and contol this teleobotics system. The code includes that supplied by RRC manipulato manufactue, puchased as geneal pupose softwae, and witten in the lab. A.3.1. R Contolle Pogam The R contolle is developed on the basis of eal-time motion contolle, suppoting vitually any obotic mechanism with minimum softwae changes. It is completely configuable though the use of text configuation files with espect to manipulato and contol hadwae [83]. The R contolle povides a seve-client TCP/IP potocol inteface, which indiectly utilizes the Dynamic Host Configuation Potocol (DHCP) sevice and the Windows Intenet Name Sevice (WINS) fo dynamic mapping of netwok names and addess. A thid paty application can inteface to the R seve and the R eal-time contolle via the R Seve API seve-client potocol. All the motion contolle commands ae suppoted in the R Seve so that the manipulato can be diected fom eithe a client emotely via an Ethenet communication o an intepocess communication potocol. This API decouples the highe-level contol development fom the lowe level motion contolle. A.3.. HALCON Compute Vision Softwae HALCON is commecial softwae fo machine vision application, which has flexible achitectue fo apid development of image analysis and machine vision 11

124 Appendix A (Continued) applications. HALCON povides a libay of moe than 11 image pocessing opeatos with outstanding pefomance fo blob analysis, mophology, patten matching, metology, 3D calibation, and binocula steeo, to name just a few [77]. Fo example, if we need to get image edge, we can choose Sobel, o Canny edge detecto to do that. Also Halcon suppots most of the cuently used fame gabbes. We can just call open_famegabbe and gab_image functions to get eal-time image. Components in Halcon ae independent objects in the C object and VB modules which can be used by uses fo application development. The image acquisition and pocessing pogam can be developed in the integated development envionment (shown in Figue A.5). But usually, in ode to implement some complex computation, the pogam edited in Halcon opeatos is conveted into C o VB in which use s algoithm can be done easily. In this poject, the image pocessing pogam and the data communication ae developed using VC. Figue A.5 Integated Development Envionment of Halcon 111

125 Appendix A (Continued) A.3.3.Teleobot Contol Inteface This is the main contol pogam to implement telemanipulation system. It is a client of the R contolle TCP/IP Seve-Client achitectue via Ethenet communication. It is developed in VC to get the Catesian 3D position and velocity of the maste input device, PHANTOM pemium 1.5. Two diffeent opeation modes ae available: one is the position mapping; the othe is the velocity mapping, woking like a 3D joystick. Also fo visual sevo contolle, this pogam gets 3D pose of the taget and sends the coesponding visual sevoing velocity commands to the R contolle. Figue A.6 Telemanipulation Inteface 11

126 Appendix A (Continued) A.3.4. Teleopeation System Achitectue Figue A.7 Teleopeation System Achitectue A.4. RRC GUI The gaphical use inteface (GUI) is povided by RRC. The RRC GUI includes jog contol, pogam contol, position feedback, client management and file management. This section descibes the featues of the RRC GUI. Figue A.8 illustates the diffeent windows, in a custom aangement. The main window is shown in figue A

127 Appendix A (Continued) Figue A.8 RRC Gaphical Use Inteface Figue A.9 RRC GUI Main Window 114

128 Appendix A (Continued) A.4.1. Safe Opeating Instuctions As with any machine, a list of guidelines and instuctions descibes how to safely opeate the obot and avoid causing injuies to humans, the obot, o the envionment. Upon integating the many components of the obot contolle inteface, a list of instuctions was developed fo the opeation of the RRC manipulato. Not only do these instuctions povide details fo futue uses, it also points out the many featues of the RRC GUI. Thee ae thee diffeent modes in which to opeate the obot: simulation mode, obot mode, and PHANToM client mode, explained in the flowing sections. A Simulation Mode Instuctions wee developed fo safe opeation of the simulation of the teleobotic system. This mode has all the capabilities of the system without sending any commands to the RT Sevo Contolle. The following is a list of step-by-step instuctions to safely opeate the obot in simulation. 1. Flip the powe switch on the back of the contolle box to the "On" position.. Pess the geen contolle on button to tun on the contolle. (Pess contolle off to tun off). See figue A To opeate the obot in simulation, make sue the main.cfg file has the simulation tuned on, do this by the following steps. 4. Open the file to edit: \config\main.cfg (Right click on the icon) 5. On the second line, the simulation statement must ead: Simulation = (On). 6. When the simulation is tuned on, double click on the Rseve.exe icon on the desktop. See figue A

129 Appendix A (Continued) 7. Once the message says Sevo Initialized fo Type upgade, then double click on the RGUI.exe icon on the desktop. See figue A Click the position feedback on the Rcontolle window to see the position of the seven joint angles and the global Catesian coodinates of the obot. 9. To see visual simulation, double click on Solidwoks file on the desktop of the PHANToM compute called: 17iFA.SLDASM Figue A.1 Contolle Buttons 1. Click on RRC Simulation / Feedback Simulation, and then click connect. The obot should follow the same configuation of the obot position feedback window on the contolle compute. 11. Thee ae thee diffeent coodinate systems in which to jog (move) the obot: Joint space, hand space, and linea space. Choose linea fo most applications. 1. The teach pendant allows fo jogging as well. It woks in conjunction with the jog contol buttons on the sceen. 13. To quit, fist close all windows on the contolle compute, and then teminate the R.RTA pocess by clicking on the RT Pocess Manage (See figue A.11) and clicking local. Find the line with R.RTA, and click: Kill Pocess. 116

130 Appendix A (Continued) Figue A.11 Desktop Icons on Robot Contolle Compute A Robot Mode Instuctions wee developed fo safe opeation of the teleobotic system whee all commands ae sent to the RT Sevo Contolle. Some instuctions ae simila, so those steps ae not epeated. The necessay instuctions ae as follows. 1. To opeate the obot, tun the simulation off by changing the main.cfg configuation file. The icon is on the desktop, figue A.11.. Open the file to edit: D:\config\main.cfg. 3. On the second line, the simulation statement must ead: "Simulation = (Off)." 4. Follow the same instuctions fo when the simulation is tuned on. 5. Once the GUI is activated, the Enable Am window will appea. Click the "Enable Am" button, and then the compute will count fo seconds. 6. Upon being awae of the obot and its location, pess the geen machine stat button, see figue A.1. If this is not done befoe the compute counts to seconds, the Machine Stat button will not activate the obot, and step 5 will need to be epeated. This is incopoated as a safety mechanism. The ed e-stop button must be attended wheneve the obot is enabled. 117

131 Appendix A (Continued) 7. Now the obot is enabled, and the homing pocess can begin. 8. The teach pendant will show the seven joints. Move each joint sepaately to accommodate the joint angles fo home position in table A.1. Once a joint has eached its home position, the compute will beep. 9. Once all the seven joints ae in the home position, pess and hold the ed CNL button on the teach pendant until the homing window disappeas. The obot will move a little bit to settle in the appopiate home position. Then stat using the GUI functionality. A.4.. Jog Contol Jog contol allows the use to manipulate the obot incementally. Since the simulation acts as a client to the seve, the jog contol featue also contols the simulation as well. Jog contol, shown in figue A.1, offes thee diffeent types of coodinate fames in which to move the obot, linea space, joint space, and hand space. In linea movement, the use can activate the jog buttons and give commands to move in any axis in the Catesian coodinate system, X, Y, and Z, and also adjust the oientation, oll, pitch and yaw. The GUI takes the commanded position and oientation in Catesian coodinates and calculates the invese kinematics to detemine the low level commands to contol the joint angles. Since thee ae six commands coesponding to the six degees of feedom to define position and oientation, the seventh command is called obit. The obit command changes the joint angles of the manipulato while leaving the position and oientation of the end-effecto unchanged. The speed in which the jogging 118

132 Appendix A (Continued) of the obot in linea space can be adjusted to un fast o slow, while the ecommendation emains to opeate the obot at a safe velocity. Figue A.1 Jog Contol Window and Position Feedback Window Anothe coodinate system is called joint space. Each of the seven jog buttons coesponds with its same numbeed joint. Fo example, when the opeato pesses the 1 button, joint numbe one will change its angle in the positive diection, accoding to the velocity set by the use. Duing the homing opeation, the joint space is used to adjust the joints individually to achieve the home position of the obot. This featue is advantageous, especially when the configuation of the obot needs to be adjusted slightly. 119

133 Appendix A (Continued) The last coodinate system is called hand space. This coodinate system changes with the oientation of the end-effecto. The hand X, Y, and Z-axes ae fixed on each of the thee oientation axes: oll, pitch, and yaw. This is the coodinate system used in teleopeation. A.4.3. Position Feedback Position feedback is offeed as anothe window in the GUI envionment, shown in figue A.1. This window simply displays the cuent position of the obot. The values of each of the seven joints ae displayed, as well as the coesponding position and oientation in the base coodinate fame. The cuent Catesian coodinates ae calculated fom the manipulato s kinematics, accoding to its joint angles. These joint angles ae eceived fom the feedback of the manipulato. Resolve boads eceive the seven joint angles, and send the exact feedback position to be displayed in the feedback window. This infomation is helpful to the use especially when opeating the obot unde simulation. A.4.4. Teach Pendant The teach pendant, figue A.1, is a hand held contol device fo opeating the obot manipulato. The teach pendant is hooked up to the compute and povides eal time contol of the obot unde the jog contol mode. Once Enable jog buttons is activated in the RRC GUI jog contol window, figue A.1, the teach pendant buttons ae activated and coincide with the commands fom the GUI on the compute sceen. The 1

134 Appendix A (Continued) teach pendant allows the use to adjust the speed of the obot and change the coodinate system, as well as move the obot. Since the teach pendant opeates in conjunction with the jog contol buttons, fouteen buttons fo the diect opeation of the obot, depending on the coodinate system, ae pesent on the hand held teach pendant. The advantage of using the teach pendant ove the RRC GUI's jog contol is that the opeato can be away fom the compute obseving the obots movements without being obstucted by the compute monito. Figue A.13 Teach Pendant fo RRC Manipulato A.4.5. Pogam Contol Most obot manipulato contol pogams have the ability to pogam the obot though a gaphical use inteface o a teach pendant, to pefom a seies of movements to pedetemined points. This is automating the obots motions. The GUI fo the RRC manipulato has this function called, Move Data/Recod. The obot can be pogammed, 11

135 Appendix A (Continued) once moving thee, to ecod a point in space. It saves the joint angle configuation coesponding to the appopiate x, y, z, and the otation in x, y, and z. A seies of these ecoded points can be pogammed and executed to pefom a cetain automated task. Fo example, in the case of teleopeation, the teleopeato would like to change the tool on the end of the obot. This would equie the teleopeato to position and align the obot to exchange tools. This pocess is advantageous to have automated befoe the teleopeato begins the tasks, so that in the event of a necessay tool change, the opeato needs only to select which tool is the desied tool fo the next task, and the obot can switch tools at the supevision of the teleopeato, instead of changing tools in teleopeation. The opeato must define the points that detemine the automated path. The objective is to use the teach pendant o the jog contol of the RRC GUI to move the manipulato to the desied points and ecod the points by clicking "Recod" on the move window, see figue A.14. Fom the RRC GUI main window of commands, figue A.9, check the box fo move / data ecod, and a window shown in figue A.14 will appea. Click ceate path, and the pogam equests a path name. Recoded points can now be added the path. Click on the execution and the pogam status check box to eveal the path name and the ecoded points, and to monito the pogess of the path execution, see figue A.16. 1

136 Appendix A (Continued) Figue A.14 MainWindow fo Move Data / Recod Figue A.15 File Management Paths o a goup of paths can be saved using the file management window, figue A.16. Fo example, in figue A.16, the path name is called "mountain." This path can be saved in a file and opened again at anothe time. Though pogam contol, epetitive paths can be automated with a high degee of pecision. 13

137 Appendix A (Continued) Figue A.16 Execution and Status Windows A.4.6. Client Seve Inteface In the RRC GUI, the client management window displays the list of connected clients. Clients can be eithe active o passive. Evey client is passive until made active by clicking the activate button in the client management window, figue A.17. Only one client can be active at once. Once a client is activated, that client can send commands to the RT sevo contolle, and eceive position feedback data. The R seve ignoes the commands fom a passive client. Howeve a passive client can equest feedback data 14

138 Appendix A (Continued) fom the seve, and will eceive the most ecent position feedback data. The active o maste client has contol ove the obot, whethe it is in simulation o obot mode. Figue A.17 Client Management Window on Robot Compute 15