Perception and Control for a Humanoid Robot using Vision

Transcription

1 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) 1 Perception and Control for a Humanoid Robot using Vision Geoffrey Taylor and Lindsay Kleeman Department of. Electrical. and Computer Systems Engineering Monash University, Clayton 3800 Victoria, Australia {Geoffrey.Taylor;Lindsay.Kleeman}@eng.monash.edu.au Abstract This paper provides a summary of our research towards developing a humanoid robot capable of performing interactive manipulation tasks in a domestic/office environment. Visual sensing makes important contributions to all areas of task specification, planning and actuation by enabling the robot to interact through gesture recognition, recognize and locate objects and obstacles, and employ visual feedback for robust control of robotic limbs. An overview of the steps involved in the sensing, planning and actuation of a typical grasping task are described and demonstrated experimentally on our upper-torso humanoid platform. Accurate sensing of range data in a domestic/office environment required the development of a novel laser stripe scanner to overcome the limitations of existing sensors. Our scanner uses stereo measurements to eliminate sensor noise, spurious reflections and cross talk from other robots, and can capture registered colour/range measurements of arbitrary objects in ambient indoor light. Task planning is performed by processing the range data to identify and model objects of interest using simple geometric primitives, and calculating a suitable grasp. Finally, we present a position-based visual servoing framework that provides the robot with automatic hand-eye calibration, allowing accurate and robust placement of the limbs. The fusion of these components allows the humanoid robot to locate and grasp a class of a priori unknown objects in its workspace. 1. Introduction Humanoid robotics has enjoyed a great deal of attention from both popular culture and the research community. This popularity may be due to the perception that humanoids are a natural vehicle by which mechatronics will find application outside assembly lines and in the wider community. The office/domestic environment is clearly designed for the anthropomorphic form, and humanoid modes of communication (speech, gestures, etc) facilitate a natural interface for human-robot interaction. Thus, humanoid robots allow humans and machines to work cooperatively without training or modifications to infrastructure. It is only recently that the first steps have been taken Figure 1. Metalman: an experimental uppertorso humanoid robot platform. towards a fully autonomous humanoid robot [1, 2, 3]. Current research projects have focused on specific aspects such as human-computer interaction, mobility, cognition and learnging. In contrast, the objectives of this research are deliberately task-driven: to develop a domestic humanoid robot capable of aiding an elderly or disabled person in performing daily tasks. Such a domestic robot faces a number of distinct challenges. Task specifications are likely to be ad hoc in nature as users find random tasks for the robot to perform, so the robot must not require special operating or initial conditions. Task planning in a cluttered environment may involve obstacle avoidance in addition to manipulating specific targets. Importantly, the robot must be capable of interacting and performing tasks at the same speed as its human companion. While a fully autonomous robot would benefit from a multi-modal sensory framework (touch, vision, smell, etc) we have focused our attention on visual perception as it provides a basis for addressing significant aspects such as recognizing and locating objects, feedback control of the robot limbs and the ability to interact through gesture recognition. This paper summarizes our contributions to the field of visual perception that enable an experimental humanoid robot to perform the basic task of recognizing and grasping an a priori unknown object. Figure 1 shows the experimental humanoid robot plat-

2 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) 2 form used in this research. The arms are approximately anthropomorphic in configuration and scale, and consist of two 6-DOF Puma 260 robots with 1-DOF prosthetic hands. Red LEDs used for tracking are attched to the hands and actuated via HC11 microcontrollers. Vision is provided by a pair of PAL cameras on a Biclops pan/tilt/verge robotic head. The cameras capture stereo images at PAL frame rate and half-pal resolution. A laser stripe generator is mounted above the cameras, consisting of a 5 mw laser diode module and cylindrical len to generate a vertical stripe, and a DC motor with an optical encoder to drive the stripe across a scene. Motor control and encoder measurements are implemented on a PIC microcontroller. The hareware components are coordinated via serial links from a dual 2.2 GHz Intel Xeon PC, which also performs image processing. This paper is structured to follow the data acquisition, processing and actuation steps involved in grasping an unknown object. Section 2 describes the novel laser stripe scanner developed to acquire accurate and robust depth/colour measurements. Section 3 overviews the object recognition and grasp planning algorithms that bridge perception and control. Finally, Section 4 describes a position-based visual servoing framework enabling the robot to accurately and robustly execute a grasp. 2. Robust Laser Stripe Scanning left image plane x L 2b L left camera frame laser stripe x R X f R left camera frame A crucial aspect of the robot s task is the acquisition of dense and reliable colour/range measurements. Passive stereo is usually associated with humanoid sensing, but the accuracy and reliability of current techniques often depend on the content of the scene. Laser stripe range sensing offers a computationally efficient alternative, but also unique challenges when used on a domestic robot; the sensor must operate in normal ambient light and be capable of rejecting sensor noise, spurious reflections and cross talk from other robots. Existing single-camera scanners have limited application in robotics due to an inability to distinguish the laser stripe from these spurious noise mechanisms. A number of robust scanning methods have been proposed in other work to overcome this problem [4, 5, 6, 7], but these suffer from issues including assumed scene structure, inability to capture colour, and lack of error recovery. This led the authors to developed a robust stereo laser stripe scanner which provides dense, accurate colour/range measurements for robotic applications [8]. Figure 2 illustrates the basic operation of our scanner. A vertical laser stripe is projected into the scene, and its position is measured on stereo image planes. Triangulation allows the 3D location of each point on the stripe to be recovered, and a complete depth image is obtained by sweeping the laser across the scene. Noise rejection is achieved by exploiting redundancy in the stereo measurements to disambiguate the light stripe from other features. Correct error modelling also allows the scanner to achieve more optimal reconstructions than exisiting contarget object right image plane Figure 2. Stereoscopic stripe scanner. figurations. Furthermore, a simple framework exists for on-line calibration of the system parameters using measurements of an arbitrary non-planar target. Thus, the sensor may be calibrated at any time during normal operation. After capturing a stereo image of the stripe, each scanline is processed using edge filters to determine a set of candidate stripe locations on the left and right image plane, x Li and x Ri. Most candidates result from noise in the detection process, except for a single pair corresponding to the actual stripe, x L and x R. The valid measurements are identified as the pair which minimizes the following error distance over all left/right candidates: d 2 = (x Li + αx Rj + βy + γf) 2 /(α 2 + 1) where f is the camera focal length, and α, β and γ are related to the laser plane position, determined by online calibration. A further validation step is applied by requiring the minimum error be below a fixed threshold. The above function is related to the distance on the image plane between a measurement pair and a projection of the corresponding optimal reconstruction on the laser plane. The optimal 3D reconstruction is only calculated for valid measurements x L and x R. To compare the performance of our robust scanner with other proposed methods, we implemented our system along with two common techniques on the same experimental platfrom. The first comparative method was a simple single-camera scanner without any optimal or robust properties, and the second was a robust method which requires consensus between two independent single-camera reconstructions for validation. In the latter case, the 3D data from the two independent sensors was averaged to obtain the final reconstruction. Figure 3 shows the test scene designed to assess the robustness of the three scanning methods for rejecting cross talk and reflections. The scene is scanned in normal indoor light and contains typical domestic objects, while a mirror creates a reflected image to simulate the effect of

3 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) 3 Figure 3. Mirror experiment arrangement cross talk and specularities. The resulting colour/range scans from the single scanner, double scanner and our robust stereo method are shown in Figure 4. The inability of the single-camera scanner to distinguish the laser from its reflection is clear from Figure 4(a), while the lack of data points in Figure 4(b) arises from the inability of the double-camera method to resolve ambiguities. Coversely, our method results in dense, robust colour/range data suitable for furher processing by the robot. The reconstruction accuracy of the three methods were compared by examining the depth variance from 200 samples of a single point on the laser stripe. The experiment was repeated for target points at varying distance to reveal the effect of range on reconstruction error, and the results are shown in Figure 5. As expected, a simple single-camera triangulation using either the left or right camera produces a significantly larger error than the robust methods, which use optimal estimation from stereo measurements. The accurate error model used in our method results in the lowest error variance. (a) Single-camera scan (b) Double-camera scan 3. Object Recognition and Grasp Planning Once the measurements have been acquired, the robot must localize objects of interest and plan the manipulation task. The range/colour scan of a typical scene shown in Figure 6(a) will be used as an example in the following discussion. In this case the system fits textured rectangular prisms to the colour/range data, although more recent versions have been expanded to include the recognition of cylinders, spheres and cones. The use of texture allows the robot to distinguish objects by colour in addition to geometry; in Figure 6(a) the robot is given the task of locating and grasping the yellow box. The work presented in this section was published recently in [9]. The first step in scene analysis is to segment and parameterise the raw range data into planar regions. A surface normal is determined for each small patch of (c) Robust stereoscopic scan Figure 4. Experimental results for mirror experiment.

4 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) Robust Double Single Left Single Right Error Variance (mm^2) Encoder Count Figure 5. Measured Depth Variance points using least squares plane fitting [10]. The data is then segmented into planar regions using a two step process: an initial segmentation designed to over-segment the data, followed by region merging. The initial segmentation is similar to the technique presented in [11], based on the familiar sequential binary connectivity algorithm. Each neighbouring pair of range points is tested for coplanarity, and coplanar pairs are assigned an identical initial region label. The initial regions are then selectively merged using an iterative boundary cost minimization algorithm [12, 13]. For each pair of regions sharing a common boundary, the residual error of a plane fitted to both regions is calulated. The regions which result in the minimum combined error are merged, and the process is repeated until the minimum error exceeds a threshold. Merging only a single pair of regions at each iteration ensures a controlled growth of the total error for all regions. Figure 6(b) demonstrates the result of applying the segmentation algorithm to the range data shown in Figure 6(a). After segmentation, planes describing distinct convex objects may be grouped by examining the boundaries between neighbouring planes (covex edge or concave corner). A generic box is modelled as three pairs of parallel planes, with each pair parameterised by a common normal and two perpendicular distances to the origin. The system must be able to view at least two sides of the box to determine all parameters of the model. The visible faces of the box are used to determine the surface normal vectors and up to three distance parameters. The distance parameters for the hidden rear faces of the box are calculated by fitting planes to the edges of the visible faces using a numerical method similar to the Hough transform. The final step in model construction is to extract surface textures from the colour component of the laser scan. The colour/range data is projected onto the plane associated with each visible face, tessellated into triangles and rendered onto a texture map. Figure 6(c) shows the final 3D rendered models of the boxes extracted from the scan in Figure 6(a). (a) Raw range/colour data. (b) Extracted planes labelled with uniform colour. (c) Extracted box models and planned grasp. Figure 6. Experimental results for object detection and grasp planning.

5 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) 5 After the object has been modelled and localized, a grasp planner calculates the pose of the gripper for a stable grasp. We adopt a typical approach to grasp planning: generate a set of candidate grasps using heuristics, then apply appropriate criteria to select the best grasp. Calculation of candidate grasps is partly based on the ideas developed in [14]. The force applied by the fingers should be normal to the gripped surface to minimize the effect of unknown surface friction, and the object should be grasped near the centre of mass to minimize load torque when lifted. These rules are easily applied to a box under the reasonable assumption of a uniformly distributed mass, resulting in two distinct grasps. For each grasp, we calculate the transformation that aligns the fingertips of the robot gripper to the candidate contact points and check that the pose is physically realizable. When both candidate grasps are reachable, we simply choose the preferred grasp as the one that minimizes the angle between the wrist and forearm of the robot. The wireframe model of the robot gripper in Figure 6(c) shows the final planned grasp for the yellow box. 4. Visual Servoing The visual servoing component of the system is based on our work first published in [15]. Visual servoing describes the feedback control of a robot using measurments from a camera, and proposed techniques are commonly classified as image-based, position-based or hybrid depending on how the control error is formulated. Our position-based visual servoing technique overcomes common problems including recovery from loss of feedback, handling large initial errors, and planning motions in Cartesian space. Furthermore, our technique allows visual servoing to be performed without knowledge of the transformation between the camera frame and robot base frame, provided the kinematics of the robot arm are reasonably well known. This is achieved by measuring the pose of the robot gripper directly, and formulating the pose error relative to the gripper. Figure 7 illustrates the basic visual servoing task. Artificial cues in the form of red LEDs are attached to the gripper to simplify image processing and increase tracking robustness. The position of the LEDs in the gripper frame G are manually calibrated and form an internal model which is used to determine its pose. At each iteration of the control loop, the position of the visible LEDs are measured in the camera frame C and processed using a Kalman filter to provide an optimal estimate of the gripper pose. A simple proportional control law is used for visual servoing, based on the translation and rotation required to move the gripper to the target pose T, expressed in the current frame of the gripper G. These differential pose parameters are passed to the Puma controller which calculates the appropriate joint motions. At the commencement of a new servoing task, the initial state of the gripper is determined autonomously. First, right camera stereo baseline target object C camera frame left camera T target gripper pose gripper LEDs G initial gripper pose Figure 7. Visual servoing framework. the cameras scan the workspace to find the LEDs, and the Kalman filter is initialised with the average measured position. The LEDs are then flashed individually to provide unambiguous position measurements, which are processed by the Kalman filter to estimate the initial pose of the gripper. During servoing, loss of visual feedback is minimized by actively tracking the motion of the gripper with the robotic head. However, if tracking is lost, the initialisation procedure provides an automatic recovery mechanism. Figure 8 shows the robot executing the grasping task planned in Figure 6(c). A view through the right camera in Figure 8(b) shows the current and target position of the gripper, and the location of the box (overlaid as wireframe models) during the servoing task. In this experiment, the entire process of data acquisition, modelling and visually servoed grasping required about 90 seconds. 5. Summary and Future Work We have presented a fusion of self-calibrated visual servoing, robust laser stripe scanning and object modelling to enable our experimental humanoid robot to perform simple grasping tasks. The significance of the system is that target objects may be a priori unknown, making the robot suitable for performing ad hoc tasks in an office/domestic environment. Work has already progressed on expanding the variety of objects that can be recognized and grasped by the robot. However, the issue of obstacle avoidance still needs to be addressed by implementing a motion planner to ensure that the arm avoids any collisions while moving towards its target. We also intend to exploit the colour and geometric information in the textured models to track the target while servoing, and thus improve grasping robustness and accuracy. This project is funded by the Strategic Monash University Research Fund for the Humanoid Robotics: Perception, Intelligence and Control project at IRRC.

6 Dept. Elect. And Computer Systems Engineering, Academic Research Forum (Feb 11, 2003) 6 References (a) Initial pose of robot and target object. (b) View through right camera of tracked gripper and box. (c) Successful completion of grasping task. Figure 8. Experimental results for visually servoed grasping. [1] B. Adams et al, Humanoid robots: A new kind of tool, IEEE Intelligent Systems, vol. 15, no. 4, pp , [2] K. Hirai, The honda humanoid robot: Development and future perspectives, Industrial Robot, vol. 26, no. 4, pp , [3] A. Price, R. Jarvis, R.A. Russell, and L. Kleeman, A lightweight plastic robotic humanoid, in Proc IEEE/RSJ International Conference on Intelligent Robots and Systems, 2000, pp [4] J. Haverinen and J. Röning, An obstacle detection system using a light stripe identification based method, in Proc. IEEE International Joint Symposium on Intelligence and Systems, 1998, pp [5] M. Magee, R. Weniger, and E. A. Franke, Location of features of known height in the presence of reflective and refractive noise using a stereoscopic light-striping approach, Optical Engineering, vol. 33, no. 4, pp , April [6] J. Nygards and Å. Wernersson, Specular objects in range cameras: Reducing ambiguities by motion, in Proc. of the IEEE Int. Conf. on Multisensor Fusion and Integration for Intelligent Systems, 1994, pp [7] E. Trucco et al, Calibration, data consistency and model acquisition with a 3-d laser striper, Int. Journal of Computer Integrated Manufacturing, vol. 11, no. 4, pp , [8] G. Taylor and L. Kleeman, Grasping unknown objects with a humanoid robot, in Proc Australasian Conference on Robotics and Automation, [9] G. Taylor, L. Kleeman, and Å. Wernersson, Robust colour and range sensing for robotic applications using a stereoscopic light stripe scanner, in Proc IEEE/RSJ International Conference on Intelligent Robots and Systems, [10] O.D. Faugeras, Three Dimensional Computer Vision: A Geometric Viewpoint, MIT Press, [11] I. Stamos and P.K. Allen, 3-d model construction using range and image data, in Proc. International Conference on Computer Vision and Pattern Recognition, 2000, vol. 1, pp [12] J.R. Goldschneider and A.Q. Li, Variational segmentation by piecewise facet models with application to range imagery, in Proc IEEE International Converence on Image Processing, 2001, vol. 1, pp [13] D. Cobzas and H. Zhang, Planar patch extraction with noisy depth data, in Proc. Third International Conference on 3D Digital Imaging and Modeling, 2001, pp [14] G. Smith, E. Lee, K. Goldberg, K. Böringer, and J. Craig, Computing parallel-jaw grips, in Proc International Conference on Robotics and Automation, 1999, pp [15] G. Taylor and L. Kleeman, Flexible self-calibrated visual servoing for a humanoid robot, in Proc Australasian Conference on Robotics and Automation, 2001.