Robot Path and End-Effector Orientation Planning Using Augmented Reality

Size: px

Start display at page:

Download "Robot Path and End-Effector Orientation Planning Using Augmented Reality"

Cori Richards
6 years ago
Views:

Tel: +65-6516-2222; fax: +65-6779-1459; E-mail address: meongsk@nus.edu.

1 Available online at Procedia CIRP 3 (2012 ) th CIRP Conference on Manufacturing Systems 2012 Robot Path and End-Effector Orientation Planning Using Augmented Reality H.C. Fang a, S.K. Ong a, *, A.Y.C. Nee a a Mechanical Engineering Deartment, Faculty of Engineering, National University of Singaore Engineering Drive 1, Singaore , Singaore * Corresonding author. Tel: ; fax: ; address: meongsk@nus.edu.sg Abstract This aer resents an Augmented Reality-based aroach for lanning the ath and the orientation of the end-effector for an industrial robot. The targeted alications are those where the end-effector is constrained to follow a visible ath, the osition and model of which are unknown, at suitable inclination angles with resect to the ath. The roosed aroach enables the users to create a list of control oints interactively on a arameterized curve model, define the orientation of the end-effector associated with each control oint, and generate a ruled surface reresenting the ath to be lanned The Authors. Published by Elsevier B.V. Selection and/or eer-review under resonsibility of of Professor D. Mourtzis and Professor G. Chryssolouris. Keywords: Robot lanning; Orientation lanning; Augmented Reality 1. Introduction With the revalence of robots suorting humans in key activity areas, there has been a rogressive increase in the demand for robotic systems that are safer, easier to install and rerogram, etc., with intuitive Human-Robot Interaction (HRI) interfaces for the SMEs and users who may not be exerienced in the field of robotics [1-2]. Breazeal et al. [2] identified the overlaing sace that can be erceived by both the human user and the robot system as a key requirement for effective HRI. Recent research on HRI suggests that augmented reality (AR) interfaces can enrich the interaction rocess in robot maniulation [3-6]. In addition, the users are able to interact with the satial environment in ath lanning and robot end-effector (EE) orientation lanning [7-10]. Interaction with a virtual robot model instead of a real robot increases the safety of the oerator when he/she is resent within the oerating range of the robot. This aer resents an AR aroach, namely, Robot Programming using Augmented Reality (RPAR-II), for robot ath lanning, robot EE orientation lanning, and ath otimization incororating robot dynamics. The RPAR-II aroach assists the users in the evaluation of the trajectory and rovides cues to tune the controller arameters. The aer is organized as follows. Section 2 resents the related studies. Section 3 gives an overview of the system. Section 4 resents the roosed aroach for interactive robot ath and EE orientation lanning. Section 5 resents the case studies to evaluate the system develoed. Section 6 summarizes the aer. 2. Literature survey Numerous research studies have been reorted on HRI interfaces as more enabling technologies become available. One examle is virtual reality (VR), which is aimed at increasing the intuitiveness for the oerators in an immersive environment [11-12]. A major advantage of using VR is the rovision of various evaluation otions due to its scalable modeling caability. However, majority of the VR systems require extensive modeling of the environment and additional effort to maintain/udate the maing between the virtual environment and its real counterarts. Another examle is robot rogramming by demonstration (PbD), which involves a user erforming a task manually, leaving the The Authors. Published by Elsevier B.V. Selection and/or eer-review under resonsibility of Professor D. Mourtzis and Professor G. Chryssolouris. doi: /j.rocir

192 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) 191 196 robot to observe, follow and learn the demonstrations in real-time.

2 192 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) robot to observe, follow and learn the demonstrations in real-time. This aroach, however, requires the robotic systems to be integrated with suitable sensors and rocessing techniques for knowledge acquisition, human gesture recognition, etc. [13]. AR has been alied in various robotics alications. AR can assist the users in re-oerative lanning as well as real-time tasks imlementation. Marin et al. [3] resented a mixed reality-based visualization interface where virtual cues can be dislayed to the users for overlaed objects isolation and grasing oerations conducted at a remote site. An AR-based cueing method was reorted [4] to assist the users in robot navigation in a tele-oeration task under dislay-control misalignment conditions. These studies show ositive effects of using AR on oerator erformance in ad hoc tele-robotic tasks. AR offers the ossibility to visualize and interact with the robot aths augmented onto the real environment. Present research on robot task lanning using AR mainly focuses on geometric ath lanning considering only robot kinematics. The AR-based simulations resented in these systems are conducted to verify the accessibility of the lanned aths. Chong et al. [7] resented a method to lan a collision-free ath through guiding a virtual robot using a robe attached with a lanar marker. Zaeh and Vogl [8] introduced a laser-rojectionbased aroach where the oerators can manually edit the aths rojected over a real workiece using an interactive stylus. Reinhart et al. [9] adoted a similar aroach [8] in robotic remote laser welding tasks. Several aroaches have been reorted on the lanning of the EE orientation along a ath using AR. Ong et al. [10] resented an aroach which enables the user to modify the orientations to avoid ossible collisions. Reinhart et al. [9] develoed a comuter-based aroach for lanning a smooth ath along a weld seam by otimizing the EE orientations. The generation of an otimized trajectory for a given robotic task considering motion constraints based on the de-couled aroach [14] has been well studied and reorted. Convex otimization techniques, such as interior oint methods [15-16], have been alied to solve de-couled trajectory lanning. These methods solve the otimization issue by reducing it to a sequence of linear equality constrained roblems. In this research, a similar aroach is imlemented to determine the timescale trajectory taking into account joint velocity and joint torque constraints. 3. System overview The setu of the RPAR-II system is shown in Fig 1. It includes a Scorbot-ER VII robot arm mounted with an EE, a comuter, a monitor-based dislay, a camera, and an interaction device attached with a marker-cube. Stereo camera End-effector Deskto dislay Robot arm Workiece Obstacles Interaction device Fig 1. System setu The AR environment consists of the hysical entities in the robot oeration sace and a virtual robot model. The target alications for the roosed system are ickand-lace oerations and ath following oerations. The main modules of the system are as follows. Interaction Module: An ARToolKit-based tracking method is used for virtual robot registration and interaction device tracking. The interaction device is used to guide the virtual robot in task lanning. The robot has five degree-of-freedom (DOF). Therefore, a ose tracked using this interaction device should be maed to a valid robot configuration through inverse kinematics techniques. Path and EE Orientation Planning Module: Generation of a suitable solution sace and a smooth ath within the solution sace, and the determination of the EE orientations along the ath. Path Otimization Module: Techniques to transform a geometric ath into an otimized trajectory subject to robot dynamics. Simulation Module: Interactive simulation of the lanned motion, allowing the user to evaluate visually the quality of the trajectory rior to the translation of the trajectories into robot rograms.. 4. Methodology This section resents a methodology for robot ath and EE orientation lanning using AR. The role of the users in HRI are (1) generating the Collision-Free Volume (CFV); (2) selecting and creating the control oints; (3) modifying the control oints where necessary, and (4) defining and adjusting the EE orientation at each control oints Path lanning Given a ick-and-lace task, instead of searching directly for a collision-free ath, the roosed aroach first creates a series of control oints within the CFV to form a ath. Next, the ath is otimized using convex

H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) 191 196 193 otimization techniques.

3 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) otimization techniques. A bounding cylinder enclosing both the EE and the virtual object is used as a swet model to facilitate the CFV check, which determines whether the swet model is within the CFV Control oint creation The control oints created must be accessible by the EE, and the swet model at each control oint should be within the CFV. The bounding cylinders corresonding to the control oints can be defined by Equation (1), where r and h are the radius and height; o i and z i are the origin and axis of the cylinder corresonding to the ith control oint; N is the number of control oints; and t i is the index of the control oints. 3 o, z,,, ; 1, 2,..., (1) BC BCi i i r h ti i N Control oint modification A ath formed by a list of control oints may not be collision-free even if these oints are within the CFV. Therefore, there may be a need to modify these oints to generate a new ath. A Euclidean distance-based method is roosed to assist the user in selecting a control oint. This method comutes the distance between the robe and each control oint, and associates the value with the corresonding oint. The distances are udated automatically as the robe moves, and the control oint that has the minimum distance to the robe is highlighted as a candidate oint to be selected. Equation (2) gives the definition of a control oint to be selected, where S o0, v i is the Euclidean distance between o 0 (the ti of the robe) and v i (the ith control oint). distance from the start oint to this oint. The time stam for the jth control oint is given by Equation (3), where d i is the Euclidean distance between the jth and (j-1)th oints. The two end-oints are t 0 0 and t. j d i1 i N 1 d i1 i tj, j1,2,..., N 4.2. EE orientation lanning N 1 1 A curve model can be reresented by a sequence of arameterized oints with even interolation intervals. This reresentation is useful for trajectory otimization where the curve has to be redefined in a scalar ath coordinate by evenly discretized oints. The curve reresentation is given by Equation (4), where N s is the number of the arameterized oints; k is the kth oint, and the auxiliary index for this oint is ak k Ns. (3) 3 (4) P Pk k, ak, k ; k 0,1,..., Ns Before roceeding to lan the EE orientation, the Euclidean distance-based method as described in Section 4.1, is used to select the list of control oints among the arameterized oints of the curve. In the modification mode, a oint from the control oint list can be deleted (Fig 2), or a oint be inserted into the existing list (Fig 2), where N Ns in Equation (2). A control oint to be deleted A control oint to be inserted o0 o0 v oi :S, oi min S, i ; i0,1,2,..., N (2) Path generation If the ath generated through interolation is within the CFV, it will be collision-free in the joint sace. Two issues need to be addressed in ath interolation, namely, the number of control oints, and the data sacing between the control oints. In the control oint creation stage, a small number of control oints may generate a ath where one or more segments are likely to be outside the CFV. On the other hand, an excessive number of control oints may roduce an overly fitted ath. As the number of control oints is task-deendent, a user can first select a small number of control oints (e.g., five) to erform the interolation, and insert new oints rogressively where necessary. Considering the Euclidean distance between two consecutive control oints, a normalized measurement can be assigned to each oint as a time stam for ath interolation. It is roortional to the cumulative Fig 2. Control oint selection End-effector orientation at a control oint A data structure associated with each control oint is illustrated in Fig 3. The arameters defined in the data structure are listed in Table 1. α is task-deendent and redefined with resect to the Z-axis of the coordinate system defined at each control oint; k, a, and α can be determined for each oint during the control oint selection stage. The rest of the arameters can be initialized as zero and secified later during the EE orientation lanning stage. A coordinate frame is first defined at the start of the curve. The coordinate frames of other control oints can be defined by alying the transformations reflecting the

4 194 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) changes in the curve direction, which define R. The EE orientations at these oints are used to define e and β. EE orientation EE orientation range Robot base coordinate frame Fig 3. Parameters associated with a control oint Table 1. Data structure of the control oints k a R α e β Index of control oint Auxiliary index of control oint Positional vector of coordinate frame defined on control oint Rotation matrix of coordinate frame defined on control oint EE inclination range Vector defines the orientation of the EE Angle between EE and z-axis of robot base frame End-effector orientation interolation At any arameterized oint on the curve, an EE orientation can be reresented with resect to the robot base frame by a unit vector. The angle between the unit vector and the Z-axis of the robot base frame can be interolated. The interolation arameters are the same as the arameters in Equation (4). The EE orientations along the curve are given by Equation (5). Hence, a ruled surface [18] can be given by Equation (6), where P m is the curve model and it defines the ruled surface directrix; W m defines the direction of the ruling; is a the length of the ruling. 3 W Wm l, am, m ; m 0,1,..., Ns (5) R y Grs Pm Wm; m 0,1,..., Ns (6) 4.3. Trajectory otimization and simulation R R z In this research, a trajectory minimizing the ath duration subject to joint torque and velocity constraints is determined through solving a convex otimization roblem using the log-barrier method [15]. The objective α R x Control oint 3D curve Workiece function for time-otimal trajectory is reresented using the arameters associated with a scalar ath coordinate, and the enalty function for the constraints is given by an averaging sum-log function [16]. The imlementation of trajectory otimization considering joint torque constraints has been described in Ref. [19]. The recursive Newton-Euler algorithm [17] is used to model the dynamic behavior of the robot. In this research, only the dynamics of the ositional DOF of the Scorbot-ER VII robot are considered. An estimation of these arameters is adoted from [20]. The trajectory can be simulated using a virtual robot under a discrete Proortional-Derivative (PD) control scheme. A normalized measurement associated with the simulated torque and velocity, as defined in Equation (7), is comared along the trajectory. sim i and qsim i are the simulated torque and velocity of joint i; i and q i are their uer bounds. mu i i sim i qsim i max, q i From Equation (7), if 0,1 mu i, both the comuted torque and velocity for each joint/link are within their limits. In this case, the link that has the largest torque and velocity among all the links is highlighted to indicate that it is the link most likely to deviate from the lanned ath. If 1 u m i, the comuted torque or velocity for joint i violates their constraints, the simulation will sto with link i highlighted. The user can adjust the control gain relevant to this joint, and execute the simulation again. Some rules can be alied for rough tuning of the control gains based on the cues observed during simulation, i.e., the derivative gains usually have the largest effect on the outut of the control system; the roortional gains will need to be tuned if the simulated trajectory is found to have shifted away from the lanned trajectory. 5. Case studies and discussions 5.1. Case study I (7) This case study demonstrates a ick-and-lace task for transferring an object from a start oint to a goal oint in a full-scale environment. Fig 4 shows the rocess of lanning a collision-free ath for this task. Fig 5 illustrates the rocedure to modify the control oints.

EE orientation lanning along a satial S-shaed curve which

The lanning results show that the robot EE is able to travel

Geometric ath generation; (e) Trajectory simulation:

Iteration: 1, Joint (d) (e) 3 overloaded (f) Iteration: 2

control oint Fig 5. Control oints modification.

orientation lanning along a satial circular curve, where the

gluing, etc. Visible curve/edge Obstacles Tracked curve Fig 7.

Outut curve and selected control oints; CFV generation;

surface generation; (e) Trajectory simulation: 5.3.1.

affecting the overall accuracy in this case study, namely,

robot modeling error, i.e., the kinematics and dynamics modeling error.

0mm, with the camera installed at 1.5m away from the workiece.

adoted in the rocess of acquiring 3D data oints.

5 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) Goal oint 5.3. Case study III Start oint Fig 7 demonstrates the method for EE orientation lanning along a satial S-shaed curve which lies on a curved surface. The lanning results show that the robot EE is able to travel along the ath at orientations within an accetable range with resect to the ath. (d) (e) Iteration: 1, Joint 3 overloaded (f) Iteration: 2 Successful Fig 4. Case study I. Task setu; CFV generation; Creation of control oints; (d) Geometric ath generation; (e) Trajectory simulation: unsuccessful trial; (f) Successful trajectory simulation Iteration: 1, Joint (d) (e) 3 overloaded (f) Iteration: 2 Successful Undesired ath segment Selected control oints New control oint Fig 5. Control oints modification. Initial ath formed; Control oints selection; Path re-generation 5.2. Case study II Fig 6 demonstrates the roosed method for EE orientation lanning along a satial circular curve, where the orientation of the EE needs to avoid the obstacles and the edge along the curve. This case study is designed to emulate robotic arc welding, gluing, etc. Visible curve/edge Obstacles Tracked curve Fig 7. Case study III. Outut curve and selected control oints; CFV generation; Definition of EE orientation at each control oint; (d) Ruled surface generation; (e) Trajectory simulation: unsuccessful trial; (f) Successful trajectory simulation Accuracy evaluation There are two main sources of errors affecting the overall accuracy in this case study, namely, curve tracking error due to the tracking method adoted, and robot modeling error, i.e., the kinematics and dynamics modeling error. Fig 8 deicts the tracking errors in the X-, Y- and Z-directions in this case study. The average tracking error is nearly 11.0mm, with the camera installed at 1.5m away from the workiece. The error is mainly caused by the ARToolKit tracking method adoted in the rocess of acquiring 3D data oints. However, it should be noted that the EE orientation lanning rocess would not introduce additional errors as the control oints are selected from the curve model. To this end, the tracking errors do not severely affect the roosed method for lanning the EE orientation along a visible curve. (d) (e) (f) (g) (h) Fig 6. Case study II. Workiece; circular curve model; control oints selection; (d) modification of control oints; (e) definition of coordinate frame at curve model; (f) definition of EE orientation at a control oint; (g) generation of ruled surface; (h) successful trajectory simulation. Fig 8. Curve tracking error in case study III

196 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) 191 196 The trajectories determined in this case study have been tested on the Scorbot-ER VII robot.

The discreancies shown in Fig 9 are mainly caused by the robot dynamics modeling errors where only a set of estimated robot dynamics arameters are imlemented in the trajectory otimization rocess.

6 196 H.C. Fang et al. / Procedia CIRP 3 ( 2012 ) The trajectories determined in this case study have been tested on the Scorbot-ER VII robot. The lots in Fig 9 are the actual trajectories of the EE in the Cartesian coordinates, as comared with the referenced curve. The discreancies shown in Fig 9 are mainly caused by the robot dynamics modeling errors where only a set of estimated robot dynamics arameters are imlemented in the trajectory otimization rocess. Fig 9. Trajectory imlemented on real robot 6. Conclusions and future work In this research, methodologies integrating AR technology have been roosed to assist users in robot ath and EE orientation lanning incororating robot dynamics. A Euclidean distance-based method has been develoed to facilitate control oints selection and modification. A convex otimization method has been imlemented to obtain an aroximated time-otimal trajectory. Prior to translation into robot controller codes, the lanned trajectory is simulated with a virtual robot and controller arameters can be adjusted gradually during simulation. The three case studies demonstrate the RPAR-II aroach in lanning ickand-lace tasks and ath following tasks. A number of areas can be further exlored to imrove the roosed system. A more accurate and robust tracking method can be develoed to enhance the accuracy of this system. Stereo camera with unaralleled otical axes will imrove the accuracy of the disarity ma, yielding better erformance in stereo deth estimation. Imrovement can be made to develo a more intuitive and non-distracting interface facilitating robot EE orientation definition and modification. The system can be extended to suit other tyes of tasks, such as robotic ainting, assembly, etc., where both ick-andlace oerations and ath following oerations may be required for a single task. (d) References [1] Thrun, S., Toward a Framework for Human-Robot Interaction, Human-Comuter Interaction, 19/1-2:9-24. [2] Breazeal, C., Edsinger, A., Fitzatrick, P., Scassellati, B., Active Vision for Sociable Robots, IEEE Transactions on System, Man, and Cybernetics, Part A: Systems and Humans, 31/5: [3] Marin, R., Sanz, P.J., Nebot, P., Wirz, R., A Multimodal Interface to Control a Robot Arm via the Web: A Case Study on Remote Programming, IEEE Transactions on Industrial Electronics, 52/6: [4] Chintamani, K., Cao, A., Ellis, R.D., Pandya, A.K., Imroved Tele-maniulator Navigation during Dislay-Control Misalignments using Augmented Reality Cues, IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 40/1: [5] Jara, C.A., Candelas, F.A., Gil, P., Fernández, M., Torres, F., An Augmented Reality Interface for Training Robotics through the Web, 40 th International Symosium on Robotics. Barcelona, Sain: March 2009, [6] Bischoff, R., Kazi A., Persectives on Augmented Reality based Human-robot Interaction with Industrial Robots, International Conference on Intelligent Robots and Systems. Sendai, Jaan: 28 Setember - 2 October 2004, [7] Chong, J.W.S., Ong, S.K., Nee, A.Y.C., Youcef-Youmi, K., Robot Programming using Augmented Reality: An Interactive Method for Planning Collision-Free Paths, Robotics and Comuter-Integrated Manufacturing, 25/3: [8] Zaeh, M.F., Vogl, W., Interactive Laser-Projection for Programming Industrial Robots, 5 th International Symosium on Mixed and Augmented Reality, Santa Barbara, CA: Oct 2006, [9] Reinhart, G., Munzert, U., Vogl, W., A Programming System for Robot-based Remote-Laser-Welding with Conventional Otics, CIRP Annals Manufacturing Technology, 57/1: [10] Ong, S.K., Chong, J.W.S., Nee, A.Y.C., A Novel AR-based Robot Programming and Path Planning Methodology, Robotics and Comuter-Integrated Manufacturing, 26/3: [11] He, X., Chen, Y., Hatic-aided robot ath lanning based on virtual tele-oeration, Robotics and Comuter-Integrated Manufacturing, 25/4-5: [12] Paakostas, N., Alexooulos, K., Koanakis, A., Integrating Digital Manufacturing and Simulation Tools in the Assembly Design Process: A Cooerating Robots Cell Case, CIRP Journal of Manufacturing Science and Technology, 4/1: [13] Aleotti, J., Caselli, S., Reggiani, M., Leveraging on a virtual environment for robot rogramming by demonstration, Robotics and Autonomous Systems, 47/2-3: [14] Craig, J.J., Introduction to Robotics, Mechanics and Control, Pearson Education Inc., NY. [15] Boyd, S., Vandenberghe, L., Convex Otimization, University Press, Cambridge. [16] Verscheure, D., Diehl, M., De Schutter, J., Swevers, J., On-line time-otimal ath tracking for robots, International Conference of Robotics and Automation. Kobe, Jaan: May 2009, [17] Roboo: htt:// last accessed on 5 Nov [18] Tsai, M.J., Stone, D.J., Inverse velocity analysis for line guidance five-axis robots, Robotics and Comuter-Integrated Manufacturing, 25/4-5: [19] Fang, H.C., Ong, S.K., Nee, A.Y.C., Interactive trajectory lanning and simulation using Augmented Reality, Robotics and Comuter-Integrated Manufacturing, 28/2: [20] Constantinescu, D., Croft, E.A., Smooth and time-otimal trajectory lanning for industrial maniulators along secified aths, Journal of Robotic Systems, 17/5:

Matlab Virtual Reality Simulations for optimizations and rapid prototyping of flexible lines systems

Matlab Virtual Reality Simulations for otimizations and raid rototying of flexible lines systems VAMVU PETRE, BARBU CAMELIA, POP MARIA Deartment of Automation, Comuters, Electrical Engineering and Energetics