A New Generic Model for Vision Based Tracking in Robotics Systems

Transcription

1 A New Generic Model for Vision Based Tracking in Robotics Systes Yanfei Liu, Ada Hoover, Ian Walker, Ben Judy, Mathew Joseph and Charly Heranson lectrical and Coputer ngineering Departent Cleson University Cleson, SC lyanfei, ahoover, iwalker, athewj, bjjudy, Abstract Visual sensing for robotics has been around for decades, but our understanding of a tiing odel reains crude. By tiing odel, we refer to the delays ( lag and otion lag) between reality (when a part is sensed), through data (the of iage data to deterine part and orientation), through control (the coputation and initiation of robot otion), through arrival (when the robot reaches reality ). In this work we introduce a tiing odel where sensing and control operate asynchronously. We apply this odel to a robotic workcell ubli RX-130 industrial robot anipulator, consisting of a St a network of six caeras for sensing, and an off-the-shelf Adept MV-19 controller. We deonstrate soe experients to show how the odel can be applied. Key words: workcell, tiing odel, visual servoing I. INTRODUCTION Figure 1 shows the classic structure for a visual servoing syste [1]. In this structure, a caera is used in the feedback loop. It provides feedback on the actual of soething being controlled, for exaple a robot. This structure can be applied to a variety of systes, including eye-in-hand systes, part-in-hand systes and obile robot systes. In an eye-in-hand syste [2], [3], [4], the caera is ounted on the end-effector of a robot and the control is adjusted to obtain the desired appearance of an object or feature in the caera. Gangloff [2] developed a visual servoing syste for a 6-DOF anipulator to follow a class of unknown but structured 3-D profiles. Corke [3], [4] presented a visual feedforward controller for an eye-inhand anipulator to fixate on a ping-pong ball thrown across the syste s field of view. In a part-in-hand syste [5], the caera is fixed in a to observe a part which is grasped by a robot. The robot is controlled to ove the part to soe desired. For exaple, Stavnitzky [5] built a syste to let the robot align a etal part with another fixed part. Since the part is always grasped by the anipulator, we can also say that the part is soething being controlled. Or in the other words, we can say that how the object appears in the caera is controlled. In soe obile robot probles [6], the caera is ounted over an environent to sense the actual of the obile robot as feedback to the controller. For exaple, Ki [6] built a obile robot syste to play desired + - e control actual Fig. 1. desired e caera Fig. 2. power aplifiers caera robot Classical visual servoing structure. control actual joint controller robot Vision guided control structure. encoders soccer. The caera is fixed over the field and acts as a feedback sensor. Here, the caera observes soething which is directly controlled. All of these systes, regardless of where the caera is ounted, use the caera in the sae control structure. In each case the syste regulates how the object appears in the caera. In this paper, we consider the proble where the caera is used to provide the desired or reference to the internal robot controller. Figure 2 shows the structure for this syste. There are several types of probles that fit this kind of syste, where the object of interest cannot be controlled directly. For exaple, iagine a robot trying to pick up live chickens, or a robot trying to anipulate parts hanging on a swaying chain conveyor. Siilar probles have been investigated in soe works. Allen [7] deonstrated a PUMA-560 tracking and grasping a oving odel train which oved around a circular railway. Nakai [8] developed a robot syste to play volleyball with huan beings. Fro the above systes, we notice that the otion of the object was liited to a known class of trajectories. In this paper, we seek to extend this to let the robot follow an unstructured (copletely unknown) trajectory. This will be enabled in part by providing a generic tiing odel for this kind of syste. Our tiing odel considers the proble where iage and control happen asynchronously. We are

2 work iage rate (HZ) control rate (HZ) lag (s) otion lag (s) Corke and Good [3], [4] Stavnitzky and Capson [5] Ki et. al. [6] Allen et. al. [7] Nakai et. al. [8] this work TABL I SUMMARY OF RLATD WORK faced with the following three probles: 1) The axiu possible rate for coplex visual sensing and is uch slower than the iniu required rate for echanical control. 2) The tie required for visual introduces a significant lag between when reality is sensed and when the visual understanding of that reality (e.g. iage tracking result) is available. We call this the lag. 3) The slow rate of update for visual feedback results in larger desired otions between updates, producing a lag in when the echanical syste copletes the desired otion. We call this the otion lag. Table I presents a suary of how previous works have featured and addressed these three probles. Fro this table we note that the first two of the three probles have been addressed to soe extent in previous works. However, no work appears to have explicitly considered proble #3. All of these works neglect the otion tie (otion lag) of the robot. One work [7] noted this proble and used an predictor to copensate for it instead of explicitly odeling it. None of these works has considered the generic odeling of this type of syste. Soe works synchronized the iage rate and control frequency for a ore traditional solution. In [2], the frequency of visual sensing, and control were all set to 50 HZ. Basically, the control frequency was synchronized to the iage rate for siplicity. Siulation results of high frequency control, i.e. 500 HZ, were also shown in [2]. Perforance of the high frequency controller was, as expected, better than the low frequency version otivating a ore thorough investigation of a generic tiing odel to solve the proble. Corke [4] and Ki [6] presented tiing diagras to describe the tie delay. Based on the tiing diagras, these works tried to use discrete tie odels to odel the systes. In order to do this, the authors siplify these asynchronous systes to single-rate systes. It is well known that the discrete tie odel can only be applied into single-rate systes or systes where the control rate and the vision sensing rate are very close. However, fro Table I, we notice that ost real systes do not satisfy this condition. Therefore, in this paper we propose a continuous generic tiing odel to describe asynchronous vision-based control systes. The reainder of this paper is organized as follows. In Section II, we describe our generic tiing odel, and then apply this odel to an industrial robot testbed that uses a network of caeras to track objects in its workcell. In Section III, We deonstrate the iportance of the application of our odel by using it to derive a lunge expression that lets the robot intercept an object oving in an unknown trajectory. Finally, we conclude the paper in Section IV. II. MTHODS Figure 3 illustrates our tiing odel. Fro top to botto, each line shows a coponent of the syste in process order (e.g. sensing coes before iage ). The horizontal axis represents tie. We use this odel to quantify the lag and otion lag of the syste. The lag is the tie between reality (e.g. the actual of an object) and when an estiate of that reality is available (e.g. the tracking result fro the iage of reality). Siilarly, the otion lag is the tie between when the control coand is issued and when the echanical syste finishes the otion. The sensing and control processes operate at fixed intervals and, where (sensing is slower than control). The tie required for all iage and tracking operations is designated. This starts when an input buffer is filled with iage data (on a clock or sync signal defined by the sensing line). An input buffer cannot be filled with new iage data until the of the previous iage data in that buffer is copleted. In Figure 3, this is why starts on the next sync after the end of. Figure 3 depicts the case where (the takes longer than the sensing interval) and when there is only one iage buffer. Figure 4 depicts the case where two input buffers are used (coonly called double buffering ). In this case, a second iage buffer is being filled while the iage data in the first buffer is

3 sensing s 1 s 2 iage u 1 u 2 synchronizing tracking controlling q k c... 1 c N finishing otion lag f otion lag reality estiated reached Fig. 5. Our prototype dynaic workcell. sensing Fig. 3. iage Fig. 4. Tiing odel for estiating the lag and latency. s 1 s 2 s 3 s 4 w u 1 u 2 u 3 u 4 buffer 1 buffer 2 Tiing odel using double buffering for iage data. being processed. Double buffering increases the lag (note the extra tie between the end of and the start of ) but increases the throughput (note the greater nuber of iages processed in Figure 4 as copared to Figure 3). In all of these cases, we have assued that the takes longer than the sensing interval ( ). In the case where (the is faster than the sensing rate), double buffering akes it possible to process every iage. In any case, there is always a iniu lag of! ", but depending on the buffering used and the relation of to the lag can be larger. In order to handle all of these cases, we introduce a synchronous tracking process (line 3 in Figure 3) operating at a rate of #. The tracking line takes the ost recent result fro the iage line and updates it for any additional delay (depicted as $ ) using a Kalan filter. In general, we desire #%& so that tracking is updated approxiately as fast as new results becoe available. A secondary benefit of the synchronous tracking line is that it satisfies standard control algorith requireents that assue synchronous input data. Without this line, the results fro iage can arrive asynchronously (as in Figures 4). The fourth and fifth lines in Figure 3 represent the control process and copletion of otion. We consider the case where the distance traveled by an object between tracking updates is larger than a robot could safely or soothly ove during a single iteration of control. The control is therefore broken up into a series of ' subotion coands occurring at a rate of. Additionally, we expect the otion requested by any new iteration of control to take ( tie to coplete. Figure 3 depicts the case where control coands are cuulative (each new control coand is relative to the last coanded goal). In Section II-A we describe our prototype, which uses an off-the-shelf Adept controller that operates in this anner. For this controller, the otion is copleted soe tie ( after the last given control coand. Once values are known for the variables,, #, and (, it is possible to derive various expressions for controlling a robot to solve specific probles, for exaple, to intercept a oving object. In the next section, we describe our prototype workcell and derivation of the tiing variables. In Section III, we derive an expression for ipleenting a lunge of the robot to intercept an object oving with an a priori unknown trajectory. A. Prototype Figure 5 shows a picture of our prototype workcell for this project. We use a St) * ubli RX130 anipulator with its conventional controller, the Adept Corporation odel MV-19. A network of six caeras surrounds the workcell, placed on a cube of aluinu fraing. In [9], we detailed the workcell configuration, calibration, iage differencing and real-tie robot otion planning. In [10], we presented soe tracking experients to show that our syste can track different kinds of objects using continuous visual sensing. Figure 6 shows a tiing diagra describing the flow of inforation through the syste, along with how long each step takes. Soe of these estiates were derived analytically fro knowledge of the hardware, while other

4 Sensing (33s) b1_ b2_ b1_ b2_ b1_ b2_ b1_ b2_ PCI bus transfer (19s) 33s 66s 99s 132s 165s 198s 231s w w w w w 297s w 330s w 363s w Iage (30s) 52s 85s 118s 151s 189s 233s 277s 321s 365s 409s Display (14s) 82s 126s 170s 214s 258s 302s 346s 390s 96s 140s 184s 228s 272s 316s 360s 404s Fig. 6. Inforation flow through syste. estiates were derived fro easureents taken while the syste was operating. We will discuss each part in detail in the following paragraphs. Figure 6 starts with an event that is happening in real tie (e.g. an object oves). The caeras operate at 30 Hz using the standard NTSC (U.S. television) forat, so that for this syste,+.-/- s. The reality that is iaged takes 33 s to transfer fro the caera to the fraegrabber (an NTSC signal uses its full sapling interval for transission). The fraegrabbers have icrocontrollers that can operate as PCI bus asters, initiating transfers fro fraegrabber eory to ain eory. Assuing negligible traffic on the PCI bus, the tie for this transfer can be coputed as the total nuber of iage bytes divided by the bandwidth of the bus: ( bytes) / (4 bytes 0 33 MHz) = 19 s, where the bandwidth is the theoretical axiu provided by the 32-bit 33 MHz PCI standard. The iage portion of our syste creates a two-diensional occupancy ap of the space in a horizontal plane of interest in the workcell, locates the centroid of an object in this space, and displays the result on-screen [11]. Based upon epirical easureents of the run-tie of the iage ethods on the Copaq, we observed the to take approxiately 30 s on average each iteration. This tie can vary by approxiately 132 s depending upon the content of the iages. We discuss the variances of our easureents in ore detail at the end of this section. We also easured that the iage display takes 14 s on average. Therefore, the total tie for this syste is = 63 s. The iages ay wait in buffers before being processed, due to our use of double buffering. The waiting tie can vary depending on the phasing of and as shown in Figure 6. The buffer in ain eory is right after the occupancy ap is available. We noticed that after several iterations, the waiting tie repeats in a pattern. The ain eory waiting tie 54 becoes a constant (39 s) after 4 iterations. The fraegrabber waiting tie 76 is repeating in 3 nubers, 5 s, 16 s, 27 s. So we take the average waiting tie 86 as ( ) / 3 = 16 s. Thus we can get the total average waiting tie,+-:9; =<?>+A@:@ s. Adding up the appropriate ters (B3 ), the lag (C ) for this syste is = 151 s. The process that synchronizes the tracking result uses a Kalan filter to update the actual $ for the given iteration. This is sent through a 10 Mbit ethernet link fro the Copaq to the Adept. Based on epirical easureents, these operations were observed to collectively take less than 1 s. For this syste we set # = 40 s which is near but slightly under the occupancy ap tie plus the iage display tie ( s). At this point the Adept (robot) has a new goal. This goal is directly forwarded to the otor-level controller through the Alter coand [9]. According to the anufacturer (Adept), the axiu issue rate for the Alter coand is 500 Hz (once every 2 s), but through experientation we observed that this rate could not always be aintained. Therefore we set = 4 s, issuing a new Alter every 4 s. The precise details of the otor-level controller are proprietary to Adept Corporation and could not be deterined. Therefore we deterined ( epirically through repeated easureents of the tie it took to coplete an Alter coand. We varied the distance oved, direction oved, and initial condition (starting with the robot in otion at varying velocities, including at rest). Across all cases the tie to coplete an Alter was observed to vary fro 120 to 150 s, with a coonly occurring edian value of 130 s. We therefore set ( to be 130 s.

5 ' In our odel we assue constants for ost of the lag ters. It is iportant to note that all of these ters have variances, soe of the having appreciable size in this context (ore than 1 s). For the sensing and ters, it would be ideal to tiestap each iage upon acquisition and easure the ters precisely. However, in order to solve probles involving estiates into the future (for exaple to plan a catch of an object) it is necessary to have averages. Therefore we only note the variances here, and leave a ore thorough understanding of their effect to future work. x t- l- k x t x t+ i III. XPRIMNTS In order to test our ethods, we experient with the proble of catching a oving object. Figure 7 depicts the scenario. The object is oving at an unknown constant velocity in a straight line. In this exaple, the object is oving in one diension; however, we forulate the solution using vectors to indicate that the solution is also applicable to 2D and 3D probles. Due to the lag, the ost recently easured of the object is where the object was (C;D$ ) s previously. We denote this location as FHGJIK;LMINKPO. The current of the object, i.e. the tie when the robot starts to lunge towards the object, is denoted as F G. The current velocity of the object is denoted as Q G, and is assued to be equal to the last easured velocity Q GJIK;LMINKPO. Therefore, the relationship between F G and F GJIK;LMINKPO is described in the following equation: F G +R F GJINK;LMIKPOS Q GJINK;LTINKPOVU CH $XW (1) It will take soe aount of tie Y s for the robot to reach the point of ipact where it will catch the object. We denote the ipact location as F GTZK\[. Therefore, We can describe the intercept proble as the following: if the robot desires to intercept the object at tie t while following the object, how any control coands (' ) should be issued between tie ] and the tie when the robot intercepts the object, and what is the constant distance ( ) that each single control coand should ove. Suppose that F GJINKP_ is the where the robot was last coanded to ove to, i.e. the where the object is at tie ]! `#, a is the nuber of alters which will be executed, and is the axiu distance a single alter can ove. The solution now involves two equations: F GTZbK\[dc Yfe+g F Ghc YfeVS Q Ghc Yie U j0 U a c YieH <Wk(lW (2) F GTZbK\[nc YieH D F GJINKP_oc Yfe +p a c YieB0 Cobining quations 2 and 3, based on the assuption that the object does not change velocity direction between (3) Fig. 7. (a) initial Fig. 8. ]rqsqtq ]! =#, we obtain: u a c U Yiel+ (v =ww Scenario for intercepting objects. (b) ipact scene xperiental setup for catching the oving object. QxG c Yie z FyG c Yfey S FyGJIKP_ c Yfe Q Ghc { (4) Yfe When we solve for a, there is a constraint: Q Ghc Yfe (5) Then ' is chosen as the axiu eleent of vector a. Therefore, c YieN+ F GTZK\[dc Yfey D F GJINKP_xc Yfe A. xperiental setup and results To verify that our odel is effective, we design an experient to let our industrial anipulator catch a oving object. A sall cylindrical object is dragged by a string tied to a belt oving at a constant velocity. The belt is oved by a DC otor. The object oves along a straight line. Figure 8 shows our experiental setup. The robot follows the object with the end-effector pointing straight down approxiately 300 above the object. When the robot is coanded to intercept the object, the robot will lunge and cover the object on the table with a odified end-effector, a sall plastic bowl. The diaeter of the object is 70, the diaeter for the sall bowl is 90. Therefore, the error should be less than 10 on each side in order to successfully catch the object. (6)

6 #+ ~} #+A/} velocity (/s) stdev (/s) catch percentage velocity (/s) stdev (/s) catch percentage % % % % % % TABL II XPRIMNTAL RSULTS FOR CATCHING TH MOVING OBJCT In order to test the applicability of our tiing odel, we conducted two sets of experients. We set # to two different values, 40 and 80, in these two sets of experients. We varied the voltage of the otor driving the conveyor to let the object ove at three different velocities. For each velocity, we kept the voltage of the otor constant to ake the object ove in a relatively fixed velocity, and ran the experient ten ties. Table II shows the results. The velocity colun is filled with the range of the average velocity in the ten experients. The standard deviation colun is the range of the standard deviation of the velocity of each experient. The results indicate that if the object oves at a relatively fixed velocity, the robot catches the object 100% of the tie independent of the velocity of the object and the update tie (# ). IV. CONCLUSION In this paper, we present a generic tiing odel for a robotic syste using visual sensing, where the caera provides the desired to the robot controller. We deonstrate how to obtain the values of the paraeters in the odel, using our dynaic workcell as an exaple. Finally, we show how this tiing odel can be used to solve probles, using as an exaple the proble of our industrial robot intercepting a oving object. The results of the experients show that our odel is highly effective, and generalizable. V. ACKNOWLDGMNTS The South Carolina Coission on Higher ducation seed funded this project during The Stäubli Corporation partially donated a state-of-the-art RX-130 industrial anipulator. The U.S. Office of Naval Research currently funds the exploration of this technology to naval warehousing through the xpeditionary Logistics progra. We gratefully thank all these organizations. VI. RFRNCS [1] S. Hutchinson, D. Hager and P. Corke, A Tutorial on Visual Servo Control, I Trans. Robotics Autoat., vol. 12, no. 5, pp , Oct [2] J. Gangloff and M. F. de Mathelin, Visual Servoing of a 6-DOF Manipulator for Unknown 3-D Profile Following, I Trans. Robotics Autoat., vol. 18, no. 4, pp , August 2002 [3] P. Corke and M. Good, Dynaic ffects in Visual Closed-Loop Systes, I Trans. Robotics Autoat., vol. 12, no. 5, pp , Oct [4] P. Corke and M. Good, Dynaic ffects in High- Perforance Visual Servoing, in Proc. I Int. Conf. Robotics and Autoation, pp , Nice, France, May 1992 [5] J. Stavnitzky and D. Capson, Multiple Caera Model-Based 3-D Visual Servo, I Trans. Robotics Autoat., vol. 16, no. 6, pp , Dec [6] S. H. Ki, J. S. Choi and B. K. Ki, Visual Servo Control Algorith for Soccer Robots Considering Tie-delay, Intelligent Autoation and Soft Coputing, Vol. 6, no. 1, pp , 2000 [7] P. Allen, A. Ticenko, B. Yoshii and P. Michelan, Autoated Tracking and Grasping of a Moving Object with a Robotics Hand-ye Syste, I Trans. Robotics Autoat., vol. 9, no. 2, pp , April 1993 [8] H. Nakai, Y. Taniguchi, M. Uenohara and T. Yoshii, A Volleyball Playing Robot, in Proc I Int. Conf. Robotics and Autoation, Belgiu, pp , May 1998 [9] Y. Liu, A. Hoover and I. Walker, Sensor Network Based Workcell for Industrial Robots, in I/RSJ International Conference on Intelligent Robots and Systes, pp , Hawaii, Oct [10] Y. Liu, A. Hoover and I. Walker, xperients Using a Sensor Network Based Workcell for Industrial Robots, in Proc. I Int. Conf. Robotics and Autoation, pp , Washington, USA, May 2002 [11] A. Hoover and B. Olsen, A Real-Tie Occupancy Map fro Multiple Video Streas, in Proc I Int. Conf. Robotics and Autoation, pp , Detroit, May 1999