Linderoth, Magnus; Robertsson, Anders; Åström, Karl; Johansson, Rolf

Transcription

1 Vision Base Tracker for Dart-Catching Robot Lineroth, Magnus; Robertsson, Aners; Åström, Karl; Johansson, Rolf 9 Link to publication Citation for publishe version (APA): Lineroth, M., Robertsson, A., Åström, K., & Johansson, R. (9). Vision Base Tracker for Dart-Catching Robot Paper presente at 9th IFAC International Smposium on Robot Control, Gifu, Japan. General rights Copright an moral rights for the publications mae accessible in the public portal are retaine b the authors an/or other copright owners an it is a conition of accessing publications that users recognise an abie b the legal requirements associate with these rights. Users ma ownloa an print one cop of an publication from the public portal for the purpose of private stu or research. You ma not further istribute the material or use it for an profit-making activit or commercial gain You ma freel istribute the URL ientifing the publication in the public portal Take own polic If ou believe that this ocument breaches copright please contact us proviing etails, an we will remove access to the work immeiatel an investigate our claim. L UNDUNI VERS I TY PO Box7 L un +66

2 Vision Base Tracker for Dart Catching Robot Magnus Lineroth*. Aners Robertsson**. Karl Åström***. Rolf Johansson**** *Department of Automatic Control, LTH, Lun Universit, SWEDEN (Tel: ; magnus.lineroth@control.lth.se). **Department of Automatic Control, LTH, Lun Universit, SWEDEN (Tel: ; aners.robertsson@control.lth.se). ***Centre for Mathematical Sciences, LTH, Lun Universit, SWEDEN (Tel: ; kalle@maths.lth.se). ****Department of Automatic Control, LTH, Lun Universit, SWEDEN (Tel: ; rolf.johansson@control.lth.se). Abstract: This paper escribes how high-spee computer vision can be use in a motion control application. The specific application investigate is a art catching robot. Computer vision is use to etect a fling art an a filtering algorithm preicts its future trajector. This will give ata to a robot controller allowing it to catch the art. The performance of the implemente components inicates that the art catching application can be mae to work well. Conclusions are also mae about what features of the sstem are critical for performance. Kewors: Dart catching, computer vision, camera calibration, state estimation.. INTRODUCTION In robot applications it is often esirable to have sstems that respon quickl to input from the robot environment. Using cameras as sensors has several avantages. The can make touch free measurements of ver generic tpes, e.g. position, shape or color. The limitation for what can be one often lies in the image analsis algorithms. Some image properties ma be ver ifficult to extract an the execution time is often significant. This paper escribes how high spee computer vision can be use in a motion control application. The specific problem chosen was to evelop the founation for a art-catching robot. A sketch escribing the problem can be seen in Fig.. A art boar was mounte on a robot an when a art was thrown towar the boar, the robot shoul move the boar so the art alwas hit the bull s ee. The etection of the art x.. Dart.6.5 Robot Fig.. Overview of the experiment setup. z -.5 was performe with cameras that provie ata to the algorithms that estimate the position an future trajector of the art. In turn this information shoul be use to move the art boar to the correct position. An overview of object tracking techniques can be foun in e.g. (Yilmaz et al. 6). The focus of m work has been to use the simplest possible algorithm that oes the job robustl in orer to optimize for spee. The art catching application is closel relate to the table tennis plaing robot, which has been treate in numerous works, e.g. (Matsushima et al. ), which uses an aaptive black-box moel for preiction.. METHODS. Harware an Sstem Architecture Two cameras were locate one on each sie of the art boar. The application require goo real-time camera performance an for this two Basler A6fc with an IEEE9 serial interface (also known as Firewire) were use. The coul suppl color images at resolutions up to pixels with a maximum frame rate of fps at full resolution. A thir slower camera was pointe towar the art boar to evaluate where the art hit. Most off-line calculations were performe in Matlab. Image acquisition, image analsis an state estimation were performe in C for spee an because the API use for image acquisition was mae in C. The trajector generation an robot control was one in Java, because the interface to the robot was mae in Java. A schematic overview of the sstem

3 s Image Acquisition Preiction Trajector Generation Java Firewire Socket architecture can be seen in Fig.. For more information about the robot control sstem, see (Robertz et al. 7).. Positioning Man ifferent was of positioning the cameras were possible. The chosen positions of the high spee cameras were on each sie of the art boar, pointing towar the thrower, illustrate in Fig.. This positioning allowe the cameras to see the art at a big istance if the thrower was straight in front of the boar. The art catcher application ha a higher requirement on the accurac of the estimate art trajector when the art was close to the art boar, which was fulfille b the chosen camera positioning, since the art was close to the cameras an observe from better angles when it was close to the boar.. Calibration Matlab Image Analsis Kalman Filter Robot Control Calibration Image coorinate to D measurement Robot Fig.. A schematic overview of the components in the sstem an how the communicate. z (m) FWB Dart Z X Y x (m) FWA - For camera calibration the Calibration Toolbox for Matlab (Bouguet 8) b Jean-Yves Bouguet was use. It uses series of images of a checkerboar pattern in ifferent EtherCAT Fig.. Top view of the setup illustrating the stereo coverage. The cones show the fiel of view for the respective cameras. NW C positions to extract both intrinsic camera parameters an the relative position of the cameras in a stereo pair. The stereo camera calibration feature in the calibration toolbox b Bouguet was not well aapte to the nees for this project, so a new script for calibration of the extrinsic parameters was evelope. It use a function in the toolbox to extract the D position of a checkerboar gri with respect to a camera with known intrinsic parameters. To measure the relative position of two cameras, a set of snchronize images was capture, showing the checkerboar in the same positions from the ifferent viewpoints of the cameras. Two sets of D points were then obtaine, each belonging to one camera. The relative position of the cameras coul be obtaine b matching these points. For this purpose, a weight function was efine as the root-mean-square value of the istances between the corresponing points. Its minimum was then foun numericall using the Neler-Mea simplex metho (Neler an Mea 965).. Image Analsis In the evelopment of the image analsis algorithms an important consieration was to make it simple an computationall efficient ue to the high real-time requirements on the implementation. Pixel Classification As a first step the color of each pixel was examine in orer to etermine whether it was likel to be part of a art in the image. To o this the RGB (re, green, blue) image was converte to the HSV (hue, saturation, value) color space. To etermine if a pixel was part of a art, the likelihoo criterion (.55 < / 6 <.65) (. < v <.7) h () was foun to work well empiricall for the art epicte in Fig.. The basic iea was to use mostl the hue, since it oes not var much with ifferent lighting conitions. Bouns on the intensit were use to iscar ranges where the hue calculation was not reliable. Using (), a new image coul be generate, where the value of each pixel was if it was likel to be part of a art, otherwise. An example of this can be seen in Fig. 9 (c). Calculating the Number of Dart Pixels in a Rectangle In the process of fining arts in an image, the number of art pixels in a rectangle was calculate a large number of times. To o this efficientl the integral image was first calculate. This was quite a heav computation, but once it was one the number of art pixels in an rectangle coul be compute b onl summing values. Fining Darts The criterion () for etecting art pixels generate quite a large number of false positives, as can be seen in Fig. 9 (c). A large number of these were cause b the Baer pattern color ecoing, which can be seen in Fig. 9 (b). Because of the isplacement of the color filters a large number of colors were generate at sharp eges in the images. These artifacts

4 occurre in lines that were one, or possibl two, pixels wie. Thus the coul be eliminate b onl keeping the pixels that were the centers of -b- pixel boxes where at least 7 out of the 9 pixels satisfie (). This resulte in Fig. 9 (). Still, some outliers remaine. To filter these out, the pixel that was the center of the 5-b-5 pixel box with the largest number of pixels satisfing () was use as the estimate of the position of the art. If several pixels share the same number of neighbors satisfing (), their center of gravit was use. B means of the integral image the arts coul be foun efficientl with a binar search over the image. The basic iea was to iscar a rectangular part of the image if it i not contain enough art pixels to contain a art. Otherwise the rectangle was split in half an each half was recursivel examine for the possibilit of containing a art..5 State Estimation an Preiction A stanar Kalman filter (Kalman 96) was use to estimate the state of the art an to preict its future trajector. The general form of the filter is erive e.g. in (Åström, Wittenmark 997). Moel of Dart Flight Dnamics The art was assume to fl with negligible air friction. Since the image analsis i not have the functionalit to extract the art s orientation, the art was moele as a particle. This was represente b the state-space moel x( kh + h) = Φx( + Γu( + v( ( = C( x( + e( with the state vector ( ) T Φ = h h () x = x z x z an gh / h, Γ =, u( gh where h is the time step, g is the earth gravitation an v an e are iscrete time Gaussian white noise processes. A problem with this simple moel was that the point that conforme well to the moel was the center of mass, but onl the position of the tail was measure, an the point of interest was at the tip, since this etermine where the art hit the boar. For the moel to be goo it was thus require that the throws were frienl, so that the three points traverse point of interest center of mass Fig.. Image of art. measure point approximatel the same trajector in space. A wobbling tail woul cause ba velocit estimates that got amplifie when the trajector was extrapolate to the art boar. The ifferent parts of the art are shown in Fig.. Preiction In each filtering iteration the art trajector was extrapolate to preict where the art woul hit the boar. This was one b simpl running more iterations of the Kalman filter, assuming that no measurements were available. The filtering was continue until ẑ was equal to the z-coorinate of the art boar. The values of xˆ an use as the hit point estimate. ŷ at that instant were Outlier Detection Before a measurement of the art position was use in the Kalman filter, it was checke that it was close to what was expecte base on the state estimate. Otherwise the measurement was iscare. Each measurement,, an each expecte measurement base on the state estimate, ˆ = Cxˆ, were associate with a covariance matrix. Hence the covariance of the ifference = ˆ an a corresponing confience ellipsoi coul be calculate. For the measurement to be accepte, ha to be within this ellipsoi. Managing of Multiple Trajectories Simultaneousl On top of the Kalman filter was a laer that coul keep track of several state vectors, each representing the trajector of one art. The obvious avantage of this was that the trajectories of several arts coul be tracke simultaneousl. A more important avantage ha to o with outlier etection. In each iteration of the Kalman filter, the measurements were iscare if the were not close to where the were expecte to be (cf. Outlier Detection). If the measurement initiating the trajector were a false positive, successive correct measurements woul be iscare, since the were not close to what was expecte after the incorrect measurement. This meant that one incorrect measurement woul block the sstem. The algorithm use to hanle this situation is escribe b the following pseuo coe: for all trajectories if the measurement is not an outlier to this trajector perform iteration of Kalman filter if the trajector has not receive a measurement for a while throw it awa if the measurement i not match an existing trajector create a new trajector.6 Conversion from Image Measurements to D Space Measurements Two approaches were consiere for converting the image coorinates of the arts to coorinates in D space.

5 Approach In the first approach a pair of stereo images acquire simultaneousl coul be use onl if the art was etecte properl in both images. The x,, an z coorinates of the art at that instant were then calculate an use as input to the Kalman filter that estimate the state of the art. When the position of the art was etecte at two time samples, an estimate of both position an velocit coul be estimate an be use as the initial state to the Kalman filter. From the position of the art in a single image it was possible to etermine a line in D-space along which the art must be. In this paper this line will be referre to as the viewline (cf. Fig. 5). If viewlines for two cameras were known for the same art, the position of the art coul be calculate as the point where the viewlines intersecte. In practice, measurement noise cause the lines not to intersect. Instea, the points closest to each other (p a an p b in Fig. 6) were compute an the mipoint of the line connecting them, p = ( p )/ a + pb, was use as an estimate of the art position. A simple wa to etect outliers uring triangulation was to examine the shortest istance = p () a p b between the viewlines. If was above some threshol it was consiere that in at least one of the images something else than the art was foun. Approach The secon approach for converting measurements in the images into input to the Kalman filter was a bit more flexible an i not perform an explicit triangulation. This approach has not et been implemente on the real sstem. A viewline coul be expresse as the intersection of two planes, e.g. the one containing the focal point an l x, an the one containing the focal point an l in Fig. 5. Each of these planes puts a constraint of the form T X = on the art position X. This coul be rewritten to a constraint on the state vector of the Kalman filter: = T X = cx = ( )( z ) x = ( ) ( x z x z ) T where c an x = If is normalize so + + =, can be interprete as the signe length of the orthogonal projection. of the art position onto the irection ( ) T Each image with a measurement of the art position gave two constraints, each specifie b a row vector c in (), one in the x irection an one in the irection of the image. If several images were available, this was hanle simpl b aing rows in the measurement matrix C of the moel (): cx c C = c x c No explicit triangulation was one. However, if the rows in C correspone to measurements in man ifferent irections, an estimate with low variance in all irections coul be obtaine. The Kalman filter was initialize to some reasonable state with a big variance in all irections. Thus, the actual value of the initial state was negligible when measurements ha been one with c in () pointing in man ifferent irections. One avantage of this secon approach was that it easil extene to an number of cameras. Two more rows were simpl ae to C for ever camera. Another avantage was that a measurement coul be use even if there were no vali measurements from an other cameras.. RESULTS () l x image plane. Calibration viewline viewline b viewline a Fig. 6. Illustration of triangulation process. l image point p a p b p Z focal point Fig. 5. Illustration of viewline. An object projecte onto some image point, can be locate anwhere along the corresponing viewline. Y X Performing camera calibration took quite long, up to hours. No explicit error estimate of the extrinsic parameters has been erive. A goo inicator of the accurac, though, is the consistenc of the position estimates mae b the ifferent cameras. Fig. 7 shows a histogram of the istances between two cameras D-coorinate estimates of the same point uring calibration. Fig. 8 shows a histogram of in () for fling arts. Small values of inicate goo accurac in the image analsis an camera calibration.. Image Analsis The image analsis worke well with few false positives an unetecte arts. It was har to give a useful quantitative measure of the error rate, since it varie much with the

6 Histogram of istances beween ientical points in the two cameras Number of occurrences istance (mm) Fig. 7. Distribution of istances between corresponing points in the calibration of the extrinsic parameters (a) Original image. There is a art at (r, c) = (, 89) (b) Close-up showing artifacts of Baer pattern. frequenc triangulation error: (m) Fig. 8. Distribution of the triangulation error,. backgroun an lighting conitions. An example of how the image analsis performe can be seen in Fig (c) Pixels with colors similar to a art. Man false positives () Pixels left after art etection. The computation time for the image analsis of a stereo image pair was approximatel ms. Consiering that the sstem ran with a sampling perio of ms it becomes apparent that the image analsis was the step that consume the most computing power, an a faster computer (or more efficient algorithms) woul allow aitional image analsis steps to increase robustness. The main limitation impose b the image analsis algorithm was that the backgroun must not contain large objects with the same color as the arts. Another limitation was that it coul not hanle the situation with more than one art in the image correctl. Further, etection of the art s orientation woul improve the abilit of the sstem to preict the future movements of the art.. State Estimation The entire process of extracting D-coorinates from the image coorinates, running an iteration of the Kalman filter an preicting where the art woul hit the art boar plane took a fraction of a millisecon an its computation time was hence negligible in comparison to that of the image analsis. Approximatel % of the position measurements were erroneousl classifie as outliers b the state filter. If one, instea of throwing the art, kept it in the han an trie to move it as if it was thrown, it was not trivial to fool the state estimator into believing that it was an actual art throw. Hence it can be conclue that the moeling worke well an most measurements were classifie correctl with respect to being the position of a real art or not.. Composite Sstem Fig. an Fig. show how the hit point estimate evolve as a art approache the art boar. Each plot correspons to one throw. The perimeter an center of the art boar are marke in re. The estimates of where the art woul hit the boar are marke with blue stars an connecte with blue lines to show how the estimate evolve. The last estimate is marke with a green star. Each estimate is surroune b an ellipse at the istance of one stanar eviation of the estimate. The coorinate where the art actuall hit the boar is marke with a magenta star. When the experiments presente in Fig. were performe, the thrower trie to make the art wobble as little as possible. The last estimate of the hit point was then generall within or cm of the actual hit point. When the experiments presente in Fig. were performe, the art wobble more since the thrower i not pa an attention to reucing the wobbling. It was estimate that the art orientation eviate up to from the irection of movement. The error of the last estimate was then up to 5 cm. In all it took -5 ms from the time when an image was capture until the estimation of the hit point was receive an coul start being processe in the Java part of the sstem. The first successful measurements were usuall one when z m (when the art was approximatel m awa from (e) Close-up of art with estimate position marke. Fig. 9. Illustration of image analsis process. the art boar) an the last when z.5 m. When the first hit point estimate was receive in the Java program there was usuall 5- ms left to impact an when the last estimate was receive there was usuall -6

7 (a) (b) estimates of what seeme reasonable. Possibl better tuning coul e.g. filter out the wobbling of the art tail better. With a frame rate of 5 fps 5- position measurements were mae uring one throw. This seeme quite sufficient, but if the wobbling of the art shoul be moele, its namics are probabl so fast that a higher frame rate is neee. To allow this, preiction coul be use to calculate where the art is expecte to be seen an onl capture this part of the image, thus reucing time for ata transfer an image analsis. CONCLUSIONS (c) Fig.. Development of estimate hit points for arts thrown in a frienl non-wobbling wa. (a) (c) Fig.. Development of estimate hit points for arts thrown in a non-frienl wobbling wa. ms left. During one throw 5- position measurements were usuall acquire.. DISCUSSION The two hours it took to calibrate the cameras was an issue, since it mae one hesitate to perform a calibration. Much time was spent on manuall clicking on the corners in ever image. With full automatic corner ientification the calibration time coul probabl come own to -5 minutes. () (b) () Seeing how wobbling of the art egrae the performance, it became obvious that etection of the art orientation was neee to get goo performance. As a first step this coul be use without making a more complex moel of the art namics. The orientation of the art an the position of the tail coul be use to calculate the position of the center of mass. If this was use as input to the Kalman filter escribing a particle, the accurac coul probabl be improve a lot. The sstem tpicall ha to move the boar to its estination in less than ms. For comparison a robot with a maximum acceleration of 6g can move the art boar. m in ms. This istance is probabl longer than is usuall neee, but on the other han the final estination is not known from the beginning, causing the movement time to be longer. The numbers inicate that the sstem ha the performance neee to catch a art, but the margins were not large. REFERENCES Bouguet, J.Y. Calibration Toolbox for Matlab, California Institute of Technolog. Site visite August 8. Kalman, R.E. A new approach to linear filtering an preiction problems, Trans. ASME J. Basic Engineering, 8 (96), pp Matsushima, M., Hashimoto, T. an Miazaki, F. Learning to the Robot Table Tennis Task Ball Control & Rall with a Human, Proc. IEEE Int. Conf. SMC, pp ,. Neler, J.A. an Mea, R. (965). A simplex metho for function minimization, Comput. J., 7, pp. 8. Robertz, S.G., Henriksson, R., Nilsson, K., Blomell, A., Tarasov, I. Using Real-time Java for Robot Control, Journal 7 IEEE Conference on Emerging Technologies & Factor Automation (EFTA 7), pp. 5-56, 7. Yilmaz, A., Jave, O. an Shah, M. Object Tracking: A Surve, ACM Computing surves, Vol. 8, No., pp. - 5, 6. Åström, K.J. an Wittenmark, B. (997). Computer Controlle Sstems, Chapter. Prentice Hall, Upper Sale River, New Jerse The implementation of the Kalman filter was straight forwar, but the choice of covariance matrices has not been thoroughl investigate. The variances were set to crue