VISION-BASED HANDLING WITH A MOBILE ROBOT

Transcription

1 VISION-BASED HANDLING WITH A MOBILE ROBOT STEFAN BLESSING TU München, Institut für Werkzeugmaschinen und Betriebswissenschaften (iwb), München, Germany, bl@iwb.mw.tu-muenchen.de STEFAN LANSER, CHRISTOPH ZIERL TU München, Institut für Informatik, Forschungsgruppe Bildverstehen (FG BV), München, Germany, flanser,zierlg@informatik.tu-muenchen.de ABSTRACT Mobile systems become more and more important in the area of modern manufacturing. In order to handle an object with a manipulator mounted on an autonomous mobile system (AMS) within a changing environment, the object has to be identied and its 3D pose relative to the manipulator has to be determined with sucient accuracy, because in general its exact position is not known a priori. The object recognition unit of the presented system accomplishes this 3D pose estimation task using a single CCD camera mounted in the gripper exchange system of a mobile robot. The reliability of the results is checked by an independent fault-detection unit. A recovery unit handles most of the possible faults autonomously increasing the availability of the system. KEYWORDS: vision-based handling, autonomous mobile system, 3D object recognition, fault-detection and recovery. INTRODUCTION The automation of manipulation tasks in manufacturing environments is often based on industrial robots. To make this automation more protable, a mobile robot can be used to perform manipulations at dierent places, wherever it is needed. Within a joint research project 1 towards the development of autonomous mobile systems located at the TU Munchen the mobile robot MobRob has been developed to full manipulation tasks autonomously, even in a changing environment. The required autonomy increases the demands on the sensors of such systems, because both the position of the autonomous mobile system (AMS) and the pose of the object to be grasped are aected by uncertainty. In order to handle an object with a manipulator mounted on an AMS, the object has to be identied and its 3D pose relative to the manipulator has to be determined with sucient accuracy. The presented vision-based object recognition system uses images taken from the camera mounted in the gripper exchange system of the mobile robot (see Fig. 1). The system architecture is shown in Fig. 2. The vision-based object recognition unit consists of a recognition and a localization module described in the following section. 1 This work was supported by Deutsche Forschungsgemeinschaft within the Sonderforschungsbereich 331, \Informationsverarbeitung in autonomen, mobilen Handhabungssystemen", projects L9 and M2.

2 (a) (b) (c) Figure 1. (a) MobRob (Mobile Robot) at the Institut fur Werkzeugmaschinen und Betriebswissenschaften (iwb) with (b) a CCD camera mounted in the gripper exchange system of the robot. (c) Vision-based grasping of a workpiece. Fault detection Decision Object recognition and localization Order No.: 0815 IWBDECKEL Result x: 320,02 320,1 y: 298,71 98,1 z: 753, 22 88,5? OK not OK Recovery Figure 2. Closed-loop architecture of the presented grasping system. The fault-detection module of the error-handling unit presented in the subsequent section analyzes the grabbed image and the result of the object localization. If there are any faults obstructing the correct handling of the object the recovery module tries to clear the fault autonomously increasing the availabilty of the system. VISION-BASED OBJECT RECOGNITION In this section the object recognition unit of the proposed system is briey described. Based on a single image from a calibrated CCD camera it identies objects and computes their 3D pose relative to the robot manipulator. For a general introduction into this eld of research see e.g. [1] or [9]. Calibration In order to obtain the 3D object pose from the grabbed video image the internal camera parameters (mapping the 3D world into pixels) as well as the external camera parameters (pose of the CCD camera relative to the manipulator) have to be determined with sucient accuracy. Camera Model. The camera model describes the projection of a 3D point P W in the scene into the 2D pixel [c; l] T of the video image of the CCD camera. The proposed

3 Figure 3. The calibration table mounted on the mobile robot seen from dierent viewpoints with known relative movements of the manipulator. (R V ; T V ) (R 1 M ; T 1 M ) (RC ; T C ) (R V ; T V ) (R 2 M ; T 2 M ) (RC ; T C ) (R V ; T V ) (R K M ; T K M ) (R C ; T C ) (a) (b) Figure 4. (a) Estimation of the camera pose (R C ; T C ) relative to the tool center point (hand-eye calibration) based on known relative movements (R k M ; T M k ) of the manipulator. (b) Each triangle of the tesselated Gaussian sphere denes a 2D view of an object. approach uses the model of pinhole camera with radial distortions [11]: It includes a rotational matrix R describing the orientation, a vector T describing the position of the camera in the world (external parameters), and the internal parameters b (eective focal length), (distortion coecient), S x and S y (scaling factors), and [C x ; C y ] T (image center). Internal Camera Parameters. In the rst stage of the calibration process the internal camera parameters b,, S x, S y, and [C x ; C y ] T are computed by simultanously evaluating images of a 2D calibration table with N circular marks taken from K dierent viewpoints, see Fig. 3. This multiview calibration [5] minimizes the distances between the projected 3D midpoints of the marks and the corresponding 2D points in the video images. The 3D pose of the camera R; T is estimated during the minimization process. Thus, only the model of the calibration table itself has to be known a priori. Hand-Eye Calibration. Once the internal camera parameters have been determined the 3D pose of the camera relative to the tool center point is estimated in the second stage of the calibration process (hand-eye calibration). In the case of a camera mounted on the manipulator of a mobile robot the 3D pose of the camera (R; T ) is the composition of the pose of the robot (R V ; T V ), the relative pose of the manipulator (R M ; T M ), and the relative pose of the camera (R C ; T C ), see Fig. 4(a). Performing controlled movements (R k M ; T M) k of the manipulator similar to

4 [12] (R C ; T C ) can be determined by minimizing KX NX e(~x) = k~s k i c i (M i ; ~x; R k M ; T M)k k 2?! min k=1 i=1 with ~s k i the normalized vector of the line of sight through the 2D point ~m k i in the k th video image and c i (M i ; ~x k ; : : :) the 3D midpoint of a mark on the calibration table transformed in the camera coordinate system, for details see [5]. Since the used 2D calibration table is mounted on the mobile robot itself, the manipulator can move to the dierent viewpoints for the multiview calibration automatically. Thus, the calibration can be accomplished in only a few minutes. 3D Pose Estimation Our object recognition system uses a priori known models of 3D objects, which are generated in a oine process, and a single intensity image of the scene. The pose estimation is performed in two steps: First, hypotheses of visible objects and their rough pose are generated by a recognition module. In a second step, these hypotheses are veried and rened by a localization module. For details see [4]. In case of multiple instances of the same object appearing in a scene, this process can be iterated. After each iteration the image features already mapped to previously detected objects are eliminated. Model Generation. Using a tesselated Gaussian sphere each object is represented by a set of up to 320 normalized perspective 2D views, see Fig. 4(b). This model generation process is based on a geometric model of the environment described in [10]. The underlying boundary representation (B-Rep) of a polyhedral 3D object can be derived from a CAD modeler. The comparability between the highly detailed CAD model and the extracted image features (which are limited by the resolution of the CCD camera) is ensured by simulating the image preprocessing on the model features. Object Recognition. The aim of the object recognition module is to identify objects and to determine their rough 3D pose by searching for the appropriate 2D model view matching the image. This is done by establishing correspondences between image lines extracted from the CCD image and model lines from a 2D view of an object. First, a set of associations is built. An association is dened as a quadruple (I j ; M i ; v; c a ) where I j is an image feature, M i is a model feature, v is one of the 320 2D model views of an object, and c a is a condence value of the correspondence between I j and M i. This value can be obtained by traversing aspect-trees [7] or by a simple geometrical comparison of the features incorporating topological constraints. In order to select the \correct" view the associations are used to build hypotheses f(object; A i ; v i ; c i )g. For each 2D view v i all corresponding associations with sucient condence are considered. From this set of associations the subset of associations A i with the highest rating forming a consistent labeling of image features is selected. The condence value c i depends on the condence values of the included associations and the percentage of mapped model features. The result of the described recognition process is a ranked list of possible hypotheses (see Fig. 5(a)) which are veried and rened by the localization module. Object Localization. In the case of a successful verication the localization module computes a modied hypothesis where some correspondences may be changed based on the viewpoint consistency constraint. This renement of correspondences can be

5 (a) (b) (c) Figure 5. (a) Extracted image lines of a toolbox with the object Iwbdeckel. Projection of object Iwbdeckel into the original video image according to the (b) initial and (c) rened pose estimation. Reflections Blurredness Contrast Wrong image part Hidden object Figure 6. Typical problems encountered by vision-based systems in a manufacturing environment. accelerated by computing specic search spaces in the video image [3]. By aligning model and image lines the nal object pose (R; t) with full 6 DOF is computed using a weighted least squares technique similar to [6]. If only coplanar features are visible which are seen from a large distance compared to the size of the object (Fig. 5), the 6 DOF estimation is quite unstable because some of the pose parameters are highly correlated. In this case, a priori knowledge of the orientation of the manipulator with respect to the ground plane of the object might be used to determine two angular degrees of freedom. Naturally, this approach decreases the exibilty of the system. Tilted objects cannot be handled any longer. A more exible solution is the use of a second image of the scene taken from a dierent viewpoint with known relative movement of the manipulator (motion stereo). By simultanously aligning the model to both images the at minimum of the 6 DOF estimation can be avoided. Note, that for well structured objects with some not coplanar model features a 6 DOF estimation based on a single video image yield good results as well. ERROR-HANDLING In a manufacturing environment there are some factors aggravating vision-based pose estimation. The fault-detection module detects these faults also using additional information not available to the object recognition unit. In case of a detected failure the recovery module is activated in order to overcome the problem autonomously. Typical Problems in a Manufacturing Environment Applying vision-based object recognition methods in a manufacturing environment is aected by a wide range of disturbing factors, e.g. reections and shadows due to

6 Image Fault-detection indicators Recovery planning module Recovery operators Cleared Image Faults cause Figure 7. The error-handling unit consist of fault-indicators and recovery operators connected by the recovery planning module. specic illumination conditions and surface characteristics, or objects, from which only a fraction is visible, see Fig. 6. In general, these faults result in additional or missing edges in the image obstructing the interpretation. This may lead to dierent pose hypotheses all compatible with the extracted image features or to a complete failure of the object recognition unit. Most of these spurious hypotheses can be detected by exploiting external information like the expected distance to the object. On the other side, it is very dicult to determine the reason for a failure as listed in Fig. 6. However, in some specic environments, e.g. a toolbox with a homogenous surface, low-level indicators analyzing selected image characteristics can be used to detect these problems. Recovery Strategies Corresponding to the various faults listed in Fig. 6 the system can choose between dierent strategies to clear a detected fault: Considering the next pose hypothesis (object recognition unit) Adapting parameters for the image preprocessing (object recognition unit) Adapting parameters for the image interpretation (object recognition unit) Adapting the aperture or focus of the CCD camera (robot guiding system) Moving the manipulator to a more suitable position (robot guiding system) Reporting to the external error-handling (manufacturing control system) A controlled movement of the manipulator can be used to increase the accuracy of a successful pose estimation as well, see the previous section. Structure of the error-handling unit Most of the faults spoiling the object recognition are a superposition of several faults what makes detection of these faults even more dicult. The denition of some basic faults leads to specialized error-handling modules. These modules consist of an indicator to detect a specic fault and a recovery operator to clear it, see Fig. 7. Based on the results of all indicators analyzing the current image as well as the result and condence of the localization, a decision is made, whether the results are reliable enough to grasp the object. Otherwise the detected fault has to be cleared by the system. In this case, based on the results obtained, the recovery planning module generates a plan to clear the fault. The plan is executed by the recovery operators, which are together with an indicator part of a fault-specic error-handling module.

7 (a) (b) (c) Figure 8. Successful localization of two workpieces of the type Iwbdeckel in a toolbox: (a) the detected image lines, (b) the rst Iwbdeckel found in the image, and (c) the other Iwbdeckel found. (a) (b) (c) Figure 9. Example for a successful error-handling: (a) no reliable pose estimation of the workpiece due to invisible image features, (b) video image after a controlled movement of the camera mounted on the manipulator, and (c) nal pose estimation of the workpiece. ROBUST POSE ESTIMATION WITH ERROR-HANDLING In Fig. 2 the structure of the whole system is shown. The sequence is started by the robot guiding system [8], instructing the object recognition unit to detect and localize an object. After grabbing a video image the object recognition unit generates hypotheses about the 3D pose of the object as described in the second section. At the same time, the fault-detection indicators analyze the image and forward their results to the decision module. Considering the results of the object recognition and the fault-detection indicators, a decision is made, whether the pose estimation is reliable and can be returned to the robot guiding system, see the previous section. Fig. 8 shows an example for a successful 3D pose estimation. If the result of the pose estimation is considered to be unreliable a recovery plan is generated automatically. Depending on the chosen recovery plan, the next hypotheses of the object recognition unit are tested, the object recognition is re-parameterized or a request to move the manipulator or to adapt the camera parameters is sent to the robot guiding system. This sequence is iterated (closed-loop) until a reliable pose estimation for grasping the object is found or no internal error-handling is possible, see Fig. 9. In case of a successful pose estimation the 3D simulation system Usis [2] is activated to perform a collision-free grasp planning for the manipulation process. Finally, this online generated robot program is downloaded and executed, see Fig. 1(c).

8 SUMMARY AND FUTURE WORK An architecture for a grasping system on an autonomous mobile robot was presented consisting of a vision-based object recognition unit and an explicit error-handling unit. Based on a priori known models the object recognition unit identies objects in single video images and determines their 3D pose (all 6 DOF). The aim of the error-handling unit is to detect and to clear possible failures increasing the availability of the whole system. Future research will be focused on improving the fault indicators and the corresponding recovery strategies. For intelligent error-handling a database should be integrated in the error-handling system, storing information from all recognitions and error-handling procedures. With this knowledge-based error detection the faults then will be detected and their reasons concluded more reliable. REFERENCES 1. T. Y. Young (Ed.). Handbook of Pattern Recognition and Image Processing: Computer Vision, volume 2. Academic Press, Inc., D. Kugelmann. Autonomous Robotic Handling Applying Sensor Systems and 3D Simulation. In IEEE International Conference on Robotics and Automation, volume 1, pages 196{201. IEEE Computer Society Press, S. Lanser and T. Lengauer. On the Selection of Candidates for Point and Line Correspondences. In International Symposium on Computer Vision, pages 157{162. IEEE Computer Society Press, S. Lanser, O. Munkelt, and C. Zierl. Robust Video-based Object Recognition using CAD Models. In U. Rembold, R. Dillmann, L.O. Hertzberger, and T. Kanade, editors, Intelligent Autonomous Systems IAS-4, pages 529{536. IOS Press, S. Lanser and Ch. Zierl. Robuste Kalibrierung von CCD-Sensoren fur autonome, mobile Systeme. In R. Dillmann, U. Rembold, and T. Luth, editors, Autonome Mobile Systeme, Informatik aktuell, pages 172{181. Springer-Verlag, D. G. Lowe. Fitting Parameterized Three-Dimensional Models to Images. IEEE Trans. on Pattern Analysis and Machine Intelligence, 13(5):441{450, O. Munkelt. Aspect-Trees: Generation and Interpretation. CVGIP: Image Understanding, 61(3):365{386, May K. Pischeltsrieder. Steuerung autonomer mobiler Roboter in der Produktion. iwb Forschungsberichte. Springer-Verlag, To appear. 9. A. R. Pope. Model-Based Object Recognition. Technical report 94-04, University of British Columbia, January N. O. Stoer and T. Troll. Model Update by Radar- and Video-based Perceptions of Environmental Variations. In International Symposium on Robotics and Manufacturing. ASME Press, New York, To appear. 11. R. Y. Tsai. An Ecient and Accurate Camera Calibration Technique for 3D Machine Vision. In Computer Vision and Pattern Recognition, pages 364{374. IEEE Computer Society Press, C. C. Wang. Extrinsic Calibration of a Vision Sensor Mounted on a Robot. Transactions on Robotics and Automation, 8(2):161{175, April 1992.