A Robust Algorithm for Segmenting Deformable Linear Objects from Video Image Sequences

Transcription

1 Submitted to: The 15th Int. Conf. on Pattern Recognition (ICPR 2000), Barcelona, Spain, Sep 3-8, 2000 A Robust Algorithm for Segmenting Deformable Linear Objects from Video Image Sequences Frank ABEGG, Dirk ENGEL, and Heinz WÖRN Institute for Process Control and Robotics (IPR), Computer Science Department, Universität Karlsruhe (TH), D Karlsruhe, Germany, [Abegg dengel Woern]@ira.uka.de, Abstract A new algorithm for segmenting and tracking deformable (shape-changing) linear (thin and long with negligible diameter) objects in video images is presented. The core of algorithm is to detect the two boundary curves of a deformable linear object within adaptive tracking windows. The boundaries are found by analyzing the gray value gradients within the tracking windows. The algorithm allows moderate changes in the image brightness by introducing tolerance parameters to the algorithm. It also provides a robust tracking of the shape- and width-changing object by using a fuzzy estimation of the object s membership function at the tracked points. The result of the segmentation is a list of base points with equal distance regarding to each other which describes the shape of the object. This list is used to define features for shape change detection. Experiments on robotic applications using the algorithm show promising results. 1 Introduction Although today robots can perform sensor-based operations on rigid objects very well, handling deformable objects with industrial robots is still a field where a lot of research has to be done. A new approach for handling deformable linear objects using the principle of contact state transitions was proposed by [Henrich99]. Investigations in the field of force/torque based robot operations using this approach showed the applicability of the new approach [Remde00]. Since there are operations, of which we believe they can be performed better with machine vision sensors, we have built a basic image processing system for recognizing and tracking deformable linear objects. A description of the general approach combining contact state transitions of deformable linear objects with vision-based robot operations can be found in [Abegg99]. Many methods for realizing different tasks performed by a visually guided robotic system can be found in literature [Hutchinson96]. However, there are only few approaches concerning deformable objects, even though there are many industrial applications [Byun96]. One example is cabling car door frames (Figure 1). Figure 1: Example task for manipulating deformable linear objects: cabling of car door frames. Handling deformable objects with an industrial robot driven by a visual sensor is slightly different from standard image processing tasks. Additionally to the realtime requirements for an industrial task, the following conditions must be met: beneath initially recognizing and classifying the deformable object, its shape also has to be tracked and analyzed over time. the segmentation should be robust given industrial environments as background, where image analysis may be perturbed. an easy integration of the image processing system into a control environment for an industrial robot must be available. Regarding to the entire process of handling and assembly operations of deformable linear objects, we need to distinguish several modules, each responsible for one subtask of the process. We need a module for object recognition and tracking, a module for object analysis, and another module for controlling the robot. In this article, we investigate the module for object recognition and tracking. We formulate a general algorithm for the vision-based recognition and tracking of deformable linear objects (DLO) using a robot handmounted standard video camera. The output of the algorithm leads to the definition of a visual feature, being a robust indicator for change in the shape of the observed object. Work concerning the autom atic segmentation of general deformable objects is still a field of great interest in the image processing community. In the case of deformable objects, two main approaches can be found in the literature: model-based object segmentation and image-based object segmentation. The approaches for model-based segmentation of deformable objects use active contours which are often called snakes. Active contours can be viewed as a kind of deformable template of an object. The model consists of energy-based model equations describing the behavior

2 of non-rigid material by using constraint conditions for internal and external forces [Kass87, Lai95]. In the case of recognizing a deformable linear object without a loop, active contours are not applicable due to the restrictions in their geometric flexibility [McInerney95]. Other methods extending the active contours to topologically adaptable snakes as the one of Lai are difficult to initialize. Image-based approaches use the information provided by the pixels and their connections and can be found in every image processing book such as [Jain95]. They can be classified into methods based on points, methods based on regions, and edge-based methods. Using these methods for recognizing deformable objects in front of a structured or textured background without any further information is hardly possible. This is because it seems to be improbable that a general function exists which uses only the pixel information for the decision whether a pixel belongs to the object of interest or not. Additional knowledge about the object of interest is necessary in order to apply an image-based segmentation on cable-like deformable linear objects. Additional knowledge like the geometric structure and geometric constraints of the object provide additional decision functions. For deformable linear objects, the curve describing the object shape and the width in pixels are very important parameters. The algorithm introduced in this paper extracts geometric and topological parameters from the image and uses them in the process of segmenting a deformable linear object. These parameters limit the list of possible object points and finally lead to a well-defined decision of the object membership function. The basic image analysis requires a gray-valuebased and gradient-based pre-segmentation using only a few automatically computable thresholds. The base points describing the shape of the deformable objects are computed by taking the midpoint of two points lying each on one of the two found contours of the object, similar to the work of [Han94]. Skeletonizing like in [Byun96], we consider to be critical because this method produces artifacts and is very time consuming [Lam92]. In the next section, a brief overview of our approach is given. The subsequent sections consider the questions: How does the algorithm for segmenting and tracking deformable linear objects work, and what are its input and output parameters (Section 3)? What are the results in using this approach (Section 4)? What are the conclusions and how should the work be continued (Section 5)? 2 Task Description and Solution In the following, the observation of a deformable linear object (called workpiece) in a static environment (called obstacle) is considered. The observation is done with a monocular standard video camera mounted on the flange of an industrial robot (called hand camera). Comparisons between the observation with a stationary camera and the observation with a hand camera show that it is more robust to detect state changes with a hand camera, since it can be managed that the whole workpiece always appears in the camera image. Even when assuming a nearly constant illumination, we have less problems with light reflections and shadows since an additional lamp is moved with the robot and can be for example a ring light around the camera. A typical workpieces can be a hose, an electric cable, or a piece of spring steal. The linear workpiece is gripped at one end and the robot gripper may perform arbitrary linear motions. Now, it is supposed that there is an assembly task where the robot has to manipulate the workpiece in order to mount it onto a device. During the assembly task, the workpiece is tracked with the vision system and its shape appearing in the camera image is analyzed in order to adapt the instructions given to the robot (Figure 2). Thus, the sensor driven mounting process can be modeled as a sensor-driven control loop for an industrial robot [Abegg99]. Figure 2: Setup of robot hand-mounted camera (left) and robot and hand camera in front of an obstacle (right) Given this task, the vision system must be able to segment the shape of the workpiece, to track it, and to adapt the recognized shape when the workpiece is deformed by a contact with an obstacle. Since the contact detection is one of the basic requirements of our approach in handling deformable linear objects in assembly processes, the geometric image model of the workpiece shape derived from the vision system must provide the possibility to compute geometric features which give an indication of those shape changes. Furthermore, the recognition process must meet real time requirements when it is applied in an industrial process. In order to solve a mounting task like given above, we propose an image-based algorithm using additional knowledge about the used workpiece. As the algorithm is initialized by the user, the initial shape of the workpiece is detected. A tracking mechanism is able to adapt automatically the geometric image-model of the preceding workpiece shape with respect to the current workpiece shape. Image-geometric features derived from the workpiece model allow a further analysis of the workpiece shape. In the following section, the algorithm is described in detail. 3 Shape Recognition and Shape Tracking The approach of the algorithm for segmenting DLOs of a camera image bases on edge detection. The idea is to follow the shape of the DLO represented by its contours whereby the width of the object may also change.

3 Titel: segments.eps Erstellt von: Applixware 4.42 ( ) Vorschau: Diese EPS-Grafik wurde nicht gespeichert mit einer enthaltenen Vorschau. Kommentar: Diese EPS-Grafik wird an einen PostScript-Drucker gedruckt, aber nicht an andere Druckertypen. 3.1 Recognition Process In order to reduce the cost of the convolution operation carried out by the edge detection operator, a preprocessing of the image is performed. For this preprocessing, the user determines the average gray value of the DLO and its tolerance. For further computations only image subareas within this given gray value interval are regarded. Within this subareas, rectangular regions of interest are processed in order to find corresponding borderline points of the DLO. Dividing up the DLO into several rectangle segments is a kind of pre-segmentation. Only within these segments the edges are computed. This strategy reduces computing costs and improves the liability to disturbance, because only relevant image areas are considered by the algorithm. As the initial rectangle segment is given by the user, further segments along the DLO are computed automatically. For segmenting the DLO within the rectangle segments, more user given information is required. The sharpness threshold of the edges to detect and the object thickness must be adjusted. For further influence on the edge detection, the user has the choice between four implemented edge detection operators: Difference, Roberts, Sobel, or Laplacian-of-Gaussian operator. Nevertheless, the Sobel-Operator has found to be the best suitable operator for our application. Another input parameter is the maximum allowed distance between two consecutive base points. By controlling this parameter, the user can increase the capability of bridging interruptions in the object shape. After setting the parameters, the algorithm selects the base points within the rectangle segments is done. The selection bases on the evaluation of a function using fuzzy logic. There are three criteria to be considered, forming our additional knowledge: the distance between the current point and the last segmented base point c point the minimal distance between the current point and the tangent through the last segmented base point c tangent the thickness of the DLO in current points c thickness All of these three criteria are normalized. An evaluation of one criterion with zero means a worst-suited point, an evaluation with one means the best-suited point. These three criteria are combined to form one total evaluation criterion c total : ctotal = cpoint ctangent cthickness The data structure of the segmented base points contains the u- and v-coordinates of the current point, the object thickness in this point, and the slope of the tangent at this point. If the borders of a rectangle segment are reached, a new segment has to be placed. Such a new rectangle segment is placed next to the last rectangle segment and concentric to the intersection of the tangent at the last base point and the border of the last segment side (Figure 3). Figure 3: The placing of new rectangle segments For a better adaptability to the property of a deformable object and its shape, the position of the rectangle segments is additionally shifted dependent from the slope of the tangent through the last base point (Figure 4). Titel: /home/dirk/projekte/diplomarbeit/tex/shift2.eps Erstellt von: GIMP PostScript file plugin V 1.06 by Peter Kirchgessner Vorschau: Diese EPS-Grafik wurde nicht gespeichert mit einer enthaltenen Vorschau. Kommentar: Diese EPS-Grafik wird an einen PostScript-Drucker gedruckt, aber nicht an andere Druckertypen. Figure 4: Adaptability to the object shape without considering the tangent slope (left) and with considering the tangent slope (right) The previously described algorithm and methods are solutions to segment DLO from a single image. The goal of tracking is to segment the DLO from an image sequence. With the tracking option, the user can define how many of the previously computed base points are used for the invocation of the next segmenting process. Around these tracking points, new rectangle segments are created as long as the searched object is detected again in one of the segments. The criterion of detecting the right DLO is the last segmented object length which corresponds to the number of segmented base points, since we have an equidistant base point list. 3.2 Feature Extraction The base point list as a representation of the workpiece in the image space allows straightforward derivation of characteristic features like the length or the curvature of the workpiece in the image, which provide hints to contact state transitions of the workpiece when observed over time [Abegg99]. Since the data of these straightforward features are superimposed by noise, we developed a more robust feature. A linear approximation of the workpiece shape is produced by computing the average base point coordinates and the average tangent angle of each base point. The resulting line and its intersection with the image borders enclose a trapezoidal surface (Figure 5). The size of this surface with respect to the size of the image surface carries only low noise and produces big peaks or rising

4 curves when the workpiece shape changes. Evaluation results of this normalized feature (we name it F) are given in the next section. Figure 5: Trapezoid given by the linear approximation of the shape a pneumatic wire (line through DLO together with dashed lines) 4 Experimental Results Based on the algorithm introduced in the last section, several experiments were performed in order to detect contact state transitions of the workpiece. At first, some experiments on the shape recognition are presented. Then, other experiments on investigating the tracking of the changing workpiece shape are shown together with the results of the feature evaluation. The manipulation experiments were performed with a Kuka KR15 robot with a pneumatic polyurethane wire with outer diameter of 6 mm and a length of 300 mm as gripped workpiece. The robot controller executes motion commands sent from a Linux-PC with two 350 MHz Pentium II Processors. The PC also includes the image processing software with the presented algorithm and the frame grabber card (Eltec PC-Eye I). As hand camera, a standard video CCD-Camera with a remote sensor head is used (Teli CS 3710 C). The working environment providing the obstacles is either a car door frame which is mounted in a horizontal lying position or an artificial environment built of aluminum sheets mounted in different angles and with some holes of several different diameters (Figure 2). Images of a tracking sequence, where the pneumatic wire is bent by moving it against an obstacle, show that our algorithm tracks well the changing contour of the wire (Figure 6). Another experiment shows the stability of the segmented base points of the pneumatic wire (Figure 7). The average change of the endpoint angle w remains nearly constant when the robot is moved 30 steps without causing a change of the wire shape. The dashed line shows the average length change l recorded during a movement causing a change of the wire shape. It can be seen that the maximum aberration is less than 5 pixels. The diagram shown in Figure 8 was recorded during a point contact establishment operation with the robot and displays the feature of the linear approximation from Section 3.2. The sudden rise shows well the change of the extracted shape of the pneumatic wire. Further experiments confirmed the applicability of this feature for recognizing other state transitions of the wire [Engel99]. Figure 6: A bent pneumatic wire is tracked by the vision system. The circles indicate the computed base points % 0,2 0,15 0,1 0, ,05-0,1-0,15-0,2 t Pixel Figure 7: Diagram showing the stability of the segmented base points of a pneumatic wire Figure 8: Diagram showing the feature F of the linear approximation from Section 3.2 during a point contact establishment operation where n is the number of frames 5 Conclusions and Future Work In this paper, a robust and fast algorithm for segmenting and tracking deformable linear objects with a standard video camera is proposed. The algorithm works well under daylight and artificial lighting conditions. The algorithm output can be used for computing indicators of object shape changes and enables a robot to manipulate deformable linear objects in order to solve an industrial- w l

5 like task. This is proved by experimental results of which some are presented in this paper. Future work will include the improvement of the algorithm and its implementation in stereo matching. Improvements for the user can be done by using yet more automatic threshold computations. For detecting changes in view direction, the feature analysis must be extended to a third cable dimension which can be the average of the width in every computed base point. Furthermore, we will investigate handling uncertainty and error estimation, what includes an extensive stability analysis. Acknowledgements The work is funded by the European Commission in the framework of the BriteEuram-project HANDFLEX (Integrated modular solution for HANDling of FLEXible materials in industrial environments). We would like to thank the DaimlerChrysler AG, Germany, for supplying cables and cable forms. [Lam92] Lam L., et al.: "Thinning Methodologies A Comprehensive Survey". In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 9, pp , September [Lai95] Lai K. F., Chin R. T.: "Deformable Contours: Modeling and Extraction". In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 11, November [McInerney 95] McInerney T., Terzopoulos D.: "Topologically Adaptable Snakes". In: Proc. of the Fifth Int. Conference on Computer Vision (ICCV'95), Cambridge, MA, USA, June [Remde00] Remde A., Henrich D., Wörn H.: "Manipulating deformable linear objects Force based detection of contact state transitions ". Submitted to: 2000 IEEE International Conference on Robotics and Automation (ICRA 2000), San Francisco, CA, USA, April References [Abegg99] Abegg F., Henrich D., Wörn H.: "Manipulating deformable linear objects Vision-based recognition of contact state transitions ". In Proc. Ninth Int. Conf. on Advanced Robotics (ICAR'99), pp , Tokyo, Japan, October 27-29, [Byun96] Byun J.-E., Nagata T.: "Determining the 3- D pose of a flexible object by stereo matching of curvature representations". In: Pattern Recognition: The Journal of the Pattern Recognition Society, vol. 29, no. 8, pp , [Engel99] Engel D.: " Entwicklung und Einsatz eines Algorithmus zur Segmentierung deformierbarer linearer Objekte aus Videobildern einer robotergefuehrten Kamera". Diploma thesis, Institute for Process Control and Robotics (IPR), Universität Karlsruhe (TH), Germany, October [Han94] Han C.-C., Fan K.-C.: "Skeleton Generation of Engineering Drawings via Contour Matching". Pattern Recognition, Vol 27, No. 2. pp , 1994.[Henrich99] Henrich D., Ogasawara T., Wörn H. "Manipulating deformable linear objects Contact states and point contacts". In: 1999 IEEE Int. Symp. on Assembly and Task Planning (ISATP'99), Porto, Portugal, July 21-24, [Hutchinson96] Hutchinson S., Hager G.D., and Corke P.I.: "A tutorial on visual servo control". In: IEEE Trans. on Robotics and Automation, vol. 12, no. 5, October [Jain 95] Jain R., Kasturi R., Schunck B.G.: "Machine Vision", McGraw-Hill, Inc., [Kass87] Kass M., Witkin A., Terzopoulos D.: "Snakes: Active Contour Models", In: Proc. of the First Int. Conference on Computer Vision (ICCV'87), pp , 1987.