AUTOATIC TRACKING OF OVING OBJECT WITH EYE-IN- HAND ROBOT ANIPULATOR 1 JIN-SIANG SHAW, WEN-LIN CHI Institute of ehatroni Engineering, Taipei Teh, Taipei, Taiwan Email: 1 jshaw@ntut.edu.tw, ton455448@gmail.om Abstrat: In this paper, traking strategies of a moving objet using visual servoing are presented with an ee-in-hand robot arm. First of all, CAshift (Continuousl Adaptive eanshift) algorithm is emploed to trak ontinuousl the moving objet in the image plane. Then b omparing the traking window from CAshift and the retangle generated b the minimum retangle method after bak projetion of the objet model, a new retangle is formed and an be seen as the objet itself. Finall, objet features for traking purpose an be etrated from the new retangle. Furthermore, through appliation of image Jaobian matri, the traking error in the image plane an be transformed to be the displaements of robot s end effetor. Aordingl, robot arm an be moved for traking this moving objet. In addition, trajetor planning of robot arm with blending tehnique is used for smoothl and ontinuousl traking the moving objet. The developed objet traking strategies using ee-in-hand robot arm an be applied to an assembl task in a moving onveor without knowing the onveor speed or having to stop the onveor for assembl. Kewords: Automati Traking, Visual Servoing, Ee-in-Hand I. INTRODUCTION The fous of our work is to ahieve translation and rotation information with moving objet in real time via ee-in-hand robot arm. In this paper, we attahed a amera to a gripper module and fi it on the end effetor of a robot arm. Ee-in-hand arhiteture is used primaril to guide end effetor and ensure that the tool properl engages the intended target. Like the work in [1], the appliation is peg-and-hole alignment. It needed to keep deteting whether the target is on the desired position and to reat quikl for doing the job suessfull. Visual servoing plas an important role in the robot appliations. It mainl applies with moving objets in unknown environment. Image based visual servoing (IBVS) is an important ontrol tehnique used for solving the omple problems of a 6-DOF robot ontrol for objet traking. Using image Jaobian matri, we an transfer the image spae ontrol errors into errors in Cartesian spae, and then with a suitable ontrol algorithm, it is possible to obtain the loal onvergene to a desired set point. Furthermore, using predition approah an make the result more robust []. The performane of traking stati objet is good in IBVS, so we tr to use this method to trak moving objet. In this paper, we set the desired point to be the position between the two finger tips of a robot arm so that we an grasp the target in the future. The image Jaobian matri depends on the amera parameter, visual feature oordinates and depth of target with respet to amera frame. Using transform oordinate frame, details in omputing image Jaobian matri an be found in [3]. In this paper, we tend to trak the target in 3-dimensional spae, hene two feature points are needed. The reason we will eplain setion. In order to aquire feature points of the objet, we need to use robust method of mahine vision to keep the feature points deteted when the target objet is moving. CAshift is an objet traking algorithm proposed b Bradski [4] that used olor histogram as its target model. This algorithm ma not be easil influened b the hanges in the shape of the target. It usd olor probabilit so that it an effiientl solve the problem that the target is moving or partl sheltered. Although it an ahieve the purpose of the target traking fast, CAshift onl relies on bak projetion. Therefore, it would fail in some ases. For eample, the objet s olor is hanged b environment or similar olor objets are deteted and hene influened the result. oreover, it would lose the target. Therefore, [5] and [6] also used other ontrol methods or ombined other algorithms to improve this problem and make it more robust. THE PROPOSED APPROACH A 6-ais robot arm TX6L from Staubli, with a gripper module having a amera on it, is attahed to its end effetor, forming an ee-in-hand robot manipulator [7]. For the Staubli robot manipulator, onl the point-to-point ontrol ommand is available to users and will be used here to trak moving objets on a onveor. Therefore, we first need to detet the objet and etrat features b using digital image proessing tehniques [8]. There are two major methods we used in digital image proessing. The first one is CAshift algorithm and the other is minimum area retangle method..1 CAshift Algorithm CAshift is an objet traking algorithm. When the target is hosen, the trak window would keep following it. CAshift uses olor information to trak the moving target. It is mainl based on bak projetion alulation and mean shift algorithm. CAshift uses olor histogram as its target model. Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 5
Automati Traking of oving Objet With Ee-in-Hand Robot anipulator Etrated image ontains man number of piels and eah piel has assoiates with a set of values of Hue(H), Saturation(S) and Value(V). Through bak projetion, we an generate olor probabilit of this image from the Hue histogram distribution. Then, it would ome out a gra sale image whih means the lighter piel is the most possible with the target model. CAshift uses an elliptial searh window and the alulation of window s loation is a onverging proess. Eah time the new window is omputed aording to the window of the last frame. The loation of new window is determined b the differene of the two windows and it needs to be smaller than the pre-set threshold. Fig. 1 shows the blok diagram of CAshift algorithm. There are some formulas whih show how to alulate the entre s oordinate of the searh window as follows -order moment: I(, ) 1-order moment: 1 I (, ) -order moment: 1 I (, ) 11 I (, ) I (, ) I (, ) where I(,) is eah piel s value in bak projetion. Then the entre s oordinates are given as, (1) 1 1 Using the oordinates of enter point to build an ellipse as the searh window with length L, width W and angle with respet to ais, we obtain a b ( a ) L () a b ( a ) W (3) 1 tan 1 b ( ) (4) a where a, b ( ) 11 Fig.1. Blok Diagram of CAshift algorithm. inimum Area Retangle After using CAshift, we onl aquire trak window whih is a retangle due to using Emgu, whih is a ross platform.net wrapper to the OpenCV image proessing librar. This trak window an t represent the target, but we do know where the target is. The main goal is to find robust feature points in the objet so that visual servoing has enough parameters to work with. In this paper, we use minimum area retangle method whih is one tpe of bounding retangle and it has a funtion to all in Emgu. It an generate the minimum retangle b a set of points so the retangle ma rotate. Therefore, we use minimum area retangle from bak projetion image and generate a retangle to represent target objet beause the retangle has enough information about the rotation and the sie of the objet if the image is lear. However, bak projetion ma have some noises whih is the problem of olor probabilit. We ombine trak window whih is generated b CAshift algorithm and minimum area retangle to generate new retangle whih is more robust and have useful information of the objet. We define the wa to determine a point (,) is true point of the objet or noise in the bak projetion b Eulidean distane to trak window: width height ( g ) ( ) ( ) ( ) g where, are oordinates in piels in image plane; (, ) are enter point of the trak window and g g width, height are the width and height of the trak window. Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 53
Automati Traking of oving Objet With Ee-in-Hand Robot anipulator dp P T (6) dt Using lassial perspetive projetion model, (a) (b) u v (7) () Fig.. generate retangle in Bak Projetion (d) where is the fous distane, (u,v) is the feature point oordinate in the image plane, and (,) is oordinate in spae. In equation (7), the unit of (u,v) is length, but we aquire (u,v) in the unit of piel. Using transformation to piel units, equation (7) beomes: (a) original image with trak window (b) use onl inimum Area Retangle from Bak Projetion of Fig.1(a) () original image with trak window (d) use trak window and inimum Area Retangle from Bak Projetion of Fig.1 () Fig. illustrates results of appling CAshift and minimum area retangle algorithms. It is learl seen that if we onl use CAshift algorithm, the trak window is not good enough to represent the objet. If we onl use minimum area retangle, the retangle would be easil influened b noises. But if we ombine both, the result is good. Finall, we set the two feature points as the middle points in the long side of the retangle..3 Visual Servoing In this paper, image based visual servoing is emploed, whih is based on the error between urrent and desired feature points on the image plane. The goal of the visual servoing proess is to minimie the image ontrol error e( k) f f ( k) (5) d so that the image feature f an reah the desired target f d. In the IBVS struture, there is a feature spae ontrol law whih uses the error as input and generates relativel robot ommand. In the following omputation, we would find out the relationship between the hanging rate of feature point in the image spae and end effetor veloit in the task spae. First, we assume the amera is in the stati environment. When the amera moves in the task spae with a linear veloit T [ T, T, T ] and angular veloit [,, ]. The veloit of a point P relative to the amera frame an be epressed as u / s v / s (8) where s, s are transformation oeffiient of, ais in the image plane respetivel. s, s and are assoiated amera parameters. We an measure them through eperiments. To simplif equation (8), we define / s, / s Equation (8) now beomes u 1 v (9) The relationship between an image point and the veloit of P an be alulated as follows. Let P [,, ] be oordinate relative to amera frame. Aording to equations (6) and (9). The derivative of oordinate P in terms of image feature oordinate u,v is, v T u T (1) v u T Derivative of oordinates of feature parameters in terms of P are u uv u u T T v v v T T u v uv (11) Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 54
Rewrite the above equation (11) in vetor-matri form as T u uv u T v u T v v v uv u Equation (1) an be written as where f J V (13) img img p Automati Traking of oving Objet With Ee-in-Hand Robot anipulator J is the image Jaobian matri, V p (1) veloit of p point with respet to amera frame. In this paper, we need to ontrol robot manipulator in 3 dimensional spae, hene at least feature points are required. In addition, we further assume that T for in-plane objet traking. Consequentl, the final feature points in image plan an be obtained is Transfer the position of P with respet to the tool 6 frame P to the inertial frame P P T P 6 6 we an know the relationship between tool frame and inertial frame of robot manipulator. Finall, send this displaement ommand to robot arm ontroller for traking the target objet. It is noted that the, ais of image plane needs to hek whether the are in the same diretions as, ais of tool frame in the perspetive projetion or not. Fig. 3 shows a blok diagram for image based visual servoing ontrol for. Fig.3. Image Based Visual Servo u v1 u 1 1 T v 1 T u v v u (14) After introdution of mahine vision based digital image proessing and visual servoing approahes, the overall flow hart for objet traking using an ee-inhand robot manipulator is shown in Fig. 4. Beause this image Jaobian matri is not a square matri, we an get the relationship below through pseudo inverse matri where V J f (15) p img J ( J J ) J T 1 T img img img img. Beause veloit of P is relative to amera frame, so we an set amera frame on the tool frame due to tool frame and amera frame moving together. Then, we an see the veloit of P as the displaement ommand of the robot end effetor. In this paper, the displaement ommand of end effetor also needs to transfer the tool frame to the global frame of robot manipulator. Using forward kinematis and the Denavit-Hartenberg onvention, we an know eah of oordinate transformation matri from the previous oordinate sstem to the net oordinate sstem on the eah joint of robot. ultipl all transformation matri, we an know the oordinate transformation from inertial oordinate to end effetor oordinate T A A A A A A 1 3 4 5 6 1 3 4 5 6 Fig.4. Blok diagram of the traking sstem.4 Blending ove In this paper, we use a robot arm from Staubli ompan. There is a movement ontrol ommand alled blending move in the ontroller. Net, we will eplain how this funtion is defined. Assume there is a tpial pik-and-plae task: piking up at ppik point and putting down at the pplae point, as shown in Fig. 5. The robot is initiall at the ppik point. Straight line movements are used for disengagement and approah. However, the main movement is pointto-point movement, as the geometr of this part of the trajator does not need to be auratel ontrolled, beause it is desirable to move robot arm as quikl as possible. In the absene of an speifi movement requirement, the robot will stops at the pdepart and pappro points, as the trajetor is sharpl turning at these points. This unneessaril inreases the duration of the operation and there is no need to pass through these Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 55
Automati Traking of oving Objet With Ee-in-Hand Robot anipulator points preisel. The duration of the movement an be signifiantl redued b blending the trajetor in the viinit of the pdepart and pappro points. In the blending move, there are two parameters whih are alled leave and reah respetivel. Leave is the distane from arrival point to start blending point. Reah is the distane from arrival point to end blending point. One we know these two parameters for blending, we an add blending move to straight line movements. In this wa, the robot manipulator no longer stops at the pdepart and pappro points. This movement is therefore faster and smoother than the point-to-point movement. Fig. 7 shows the proesses of robot arm traking a moving objet: (a) we put the objet on the onveor; (b) we seleted the target and the robot began to trak; (),(d),(e) robot kept following the objet;(f) trak mode was losed at the end of onveor. In Fig.8 we an see that after trak mode started, robot tried to move toward the desired position. At the end, the error of v was lose to piel and onl had error of u in the ais beause the objet is moving with a veloit. oreover, we an find the value of u was hanging in a bounded small area, meaning the robot was able to trak the moving objet without divergene. Trajetor of the robot arm traking a moving objet is shown in Fig. 9. Fig.5. Pik and plae trajetor (with blending) EXPERIENTAL RESULTS 3.1 Eperimental Setting We plae target objets on a ontinuousl operating onveor and make sure the objet movement is in the workspae of the robot arm. Then, we attahed a gripper module with a amera to the end effetor. We assume the traking movement has same height at 39 mm from amera to objet. We also need to set the desired points at oordinate (191,158) and (145,158) whih are the positions of the two fingure tips. In this stud, we trak objet moving in - plane and the robot an also rotate about the ais if the objet is randoml orientated. In addition, we need to measure, mentioned earlier. In this paper, we use bakground subtration method and edge detet method to easil know the piels of the objet. Then, measure the length, width of objet and the distane from amera to objet. Through equation (9), we an aquire the values for,. 3. Results Fig. 6 shows time response of the error of feature points. Before the traking started, we an see the error of u whih is ais of image plane was monotoniall dereasing and the error of v whih is ais of image plane didn t have obviousl hange. When the traking was started, the error of v suddenl dereased and was lose to piel. However, the error of u, tring to onverge, an t go to piel beause the objet was moving. It maintained the error around 4 piels that means robot arm kept traking the objet and didn t lose it. Fig.6. Error of feature points in the image plane Fig.7. Snapshots of robot arm traking a moving objet Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 56
Automati Traking of oving Objet With Ee-in-Hand Robot anipulator CONCLUSIONS Fig.8. Path of feature points in the image plane In this paper, we ombined CAshift algorithm, minimum area retangle method and visual servoing for traking moving objet with an ee-in-hand robot manipulator. In order to make feature points more robust, we generated new retangle to represent the target objet itself from trak window b CAshift algorithm and minimum area retangle in bak projetion image. Although the robot arm ontrol onl use point-to-point movement, robot arm still move smoothl with blending. In future works, the robot gripper is to grasp the moving objet b using some preditive ontrol methods, like Kalman filter or adaptive linear preditor. B emploing these preditive ontrol methods, a moving objet an be suessfull grabbed as soon as possible b the robot arm. REFERENCES Fig.9. Trajetor of robot arm 3.3. Disussion During the eperiment, we found that if we didn t use blending movement, the robot arm might not ath up the moving objet in high speed movement and the error of u, v would be bigger than with blending. Beause the amera etrats the image muh faster than robot movement, we an t keep sending all feature errors to robot ontroller. It would have the problem of staking ommands beause we use point to point movement no matter we use the movement with blending or not. Therefore we use an asnhronous method to handle the two things: one is to keep etrating image. The other is to send robot ommand onl if the last movement is finished over 5%. 1. Shouren Huang, Yuji Yamakawa, Taku Senoo and asatoshi Ishikawa, Realiing Peg-and-Hole Alignment with One Ee-in-Hand High-Speed Camera, 3 IEEE/ASE International Conferene on Advaned Intelligent ehatronis, pp.117-113 Jul 13.. C. Laar and A. Burlau, Visual Servoing of Robot anipulators Using odel-based Preditive Control, 7 th IEEE International Conferene on Industrial Informatis June 9. 3. Chung-Hsien Yeh, Grasping Control of a obile anipulator, Department of Eletrial and Control Engineering in National Chiao Tung Universit, pp1-57 Jul 6. 4. G.R. Bradski, Computer Vision Fae Traking for Use in a Pereptual User Interfae, Intel Tehnolog Journal, nd Quarter, 1998. 5. Saket Joshi, Shounak Gujarathi, and Abhishek irge oving objet traking method using improved CAshift with SURF algorithm, Inernational Journal of Advanes in Siene Engineering and Tehnolog, ISNN:31-99, pp.14 18 April 4. 6. Sheldon Xu and Anthon Chang Robust Objet Traking Using Kalman Filters with Dnami Covariane Cornell Universit 14 7. J. Shaw and Vipul Dube, Design of Servo Atuated Roboti Gripper Using Fore Control for Range of Objets, International Conferene on Advaned Robotis and Intelligent Sstems, Taipei, Taiwan, Aug. 31~Sept., 16. 8. R.C. Gonale and R.E. Woods, Digital Image Proessing, Addison Wesle, 9. Proeedings of 93 rd The IIER International Conferene, Los Angeles, USA, th -1 st Januar 17, ISBN: 978-93-8691-99-8 57