Towards Safe Interactions with a Sturdy Humanoid Robot

Transcription

1 Towards Safe Interactions with a Sturdy Humanoid Robot Ester Martínez and Angel P. del Pobil Abstract On the way to autonomous robots, interaction plays a main role since it allows to adequately perform their tasks as well as identifying human beings or other robots who indicate the task to be performed or jointly carry out a task. System dependability in a dynamic, unknown environment requires to properly detect all the surrounding elements, especially those in movement, in order to avoid collisions. A variety of sensors have been developed with that aim, but they are not limited in robotic systems which operate in humanpopulated, everyday environments. Here, we propose a sturdy humanoid torso which incorporates a visual system composed of a fisheye camera mounted in the front side of the robot base. We present a robust visual application that allows the system to interact with its environment and all its surrounding elements by properly adapting to it, as shown by the presented experimental results on a laboratory set-up. I. INTRODUCTION On the way to autonomous robots, interaction plays a main role since it allows to adequately perform their tasks as well as identifying human beings or other robots who indicate the task to be performed or jointly carry out a task. So, system dependability is defined in terms of adequately task performing in a dynamic, unknown environment. For that, all the surrounding elements should be properly detected, especially those in movement, in order to avoid collisions. However, devices developed for collision avoidance in industry such as cages, laser fencing or visual acoustic signals, are not suitable because they considerably restrict the system autonomy, flexibility or performance time. Thus, with the aim of avoiding those constraints, a robot-embedded sensor might be suitable for our goal. Among the available ones, intensity cameras are a good alternative since they are an important source of information. Nevertheless, traditional cameras have a narrow field of view by partially covering the environment. That makes necessary to combine some of them to cover the whole workspace and, as a consequence, realtime execution cannot be obtained. As an alternative, fisheye cameras may be used because they provide panoramic vision and one or two of them allow to cover the whole workspace without resulting in a time-consuming application. Here, we propose a humanoid robot torso which incorporates a visual system composed of a fisheye (Foculus FO124TC IEEE1394 camera, with a FUJINON-YV2.2X1.4A- 2 lens which provides 185 field of view), pointing upwards to the ceiling, to guarantee the whole coverage of the system workspace (see Fig. 1). With the aim of designing dependable robotic systems, we present a robust visual application that allows the system to interact with its environment and all its elements by properly adapting to it. This is mainly important when real environments are considered because different uncontrolled factors such as blinking of screens, passage of time or sudden changes in illumination might occur. For that, under the hypothesis that the system is at rest, we present an approach based on motion detection that accurately detects, recognizes (when human- or robot-robot interaction takes place) and tracks moving surrounding elements. Since it is based on panoramic vision, it can efficiently handle interactions with its environment by showing up in any 360 direction around the robot. Basically, the visual application is divided into two steps. In a first step, moving elements are detected by means of a robust, modified background maintenance approach; while, in the second stage, two different actions can take place by depending on the robot goal. When the system is waiting for having a direct interaction with a person or another robotic system, the designed application performs a process for interaction element identification. On the contrary, if it is performing a manipulation task, the visual application simply controls the activity around it in order to make decisions such that no damage is caused. In the following sections we present a deeper view of the system. We begin in Section II with an overall description of the modified background maintenance approach for a robust motion detection. Then, the element identification method for interaction is presented in Section III, whereas Section IV introduces a surrounding activity surveillance for safety issues. Finally, some experimental results are shown in Section V and discussed in Section VI. The authors are with the Robotic Intelligence Lab, Universitat Jaume- I, Castellón, Spain. Angel P. del Pobil is on leave in Department of Interaction Science, Sungkyunkwan University, Seoul, South Korea. {emartine, pobil}@icc.uji.es Fig. 1. camera Experimental Setup: external view of the humanoid robot and the

2 II. OUR BACKGROUND MAINTENANCE APPROACH FOR MOTION DETECTION An adaptive background maintenance approach combined with a global illumination change detection method is used to proper detect any moving object in a robot workspace and its surrounding area under the hypothesis that the system is at rest. Thus, first of all, it is necessary to describe the kind of environments robotic system has to work with, in order to know the issues to be solved. They should be real scenarios where conditions may considerably vary. Therefore, if the vision system is analysing an outdoor environment, it must be capable of dealing with shadows, gradual changes in illumination due to day time, sudden changes in illumination if any artificial lightning is switched on/off at any time, reflections on mirrors, the waving of trees and/or plants and occlusions. On the contrary, if system is working in indoor environments, source of condition changes, but not its effect. That is why so many conditions as possible should be taken into account. A deep research has been carried out to adapt detection approaches to those factors, although one of the most common is background maintenance approach [1] [2] [3] [4]. In that approach, a moving object is identified by using a background model obtained after observing several seconds the scene. Mainly, the background model consists of an estimation of a Gaussian distribution, or a mixture of them as it was presented in [5] [6] [7] [8], such that pixels are classified as foreground when they do not fit the estimated background model. Nevertheless, this approach presents some drawbacks to be overcome: everything seen when the initial background model is being built is considered background no sudden change in illumination occurs during the whole experiment only non-stationary regions are highlighted, so an object which appeared in the scene and then stopped moving for a period of time can be modelled as a part of the background no difference between a foreground object that becomes motionless and a background object that moves, and then becomes motionless a foreground object s pixel characteristics may be subsumed by the modeled background What we propose is to take advantage of difference techniques to overcome background modelling problems. Thus, in the same way to techniques belonging to this approach, the developed method involves two different stages: 1) A training period such that a scene is observed during several seconds in order to build a starting statistical background model. In our case, it corresponds to a Gaussian distribution. Moreover, another technique, based on frame differences, is used to control the activity within the robot workspace during this stage. In that way, the initial background model is obtained without any restrictions of bootstrapping, unlike previous approaches. 2) A segmentation period where, once an initial background model is built, both an image processing and a background model updating phase start. In addition, when a global change in illumination occurs, it is detected at frame level and the background model is properly adapted. Thus, when a human or another moving object like a robot enters a room where the robot is, it is detected by means of the background model at pixel level. It is possible because each pixel belonging to the moving object has an intensity value which does not fit into the background model. Then, the obtained binary image is improved based on the proper combination of subtraction techniques. Moreover, two consecutive morphological operations are applied to erase isolated points or lines caused by the dynamic factors abovementioned. The next step is properly updating the statistical model with the values of the pixels classified as background in order to adapt it to some small changes that do not represent targets. At the same time, a process for sudden illumination change detection is performed at frame level. This step is necessary because the model is based on intensity values and a change in illumination causes a variation of them. So, a new adaptive background model is built when an event of this type occurs. Note that it is assumed that a significant illumination change has taken place when there is a change in more pixels than two thirds of the image size and the difference between the number of foreground pixels in the current frame with respect the previous one is less than a threshold. The last constraint is necessary in order to avoid false alarms due to target proximity to the camera. III. ELEMENT IDENTIFICATION FOR ROBOT INTERACTION Once all moving elements in the scene are detected, an element identification method is required when system is waiting for an interaction. So, on the one hand, when targets are human beings, the Viola-Jones classifier can be used. However, it is important to take into account that face detection is only applied over the detected blobs and not over the whole image. In that way, time-consumption is considerably reduced. In addition, an important distinction is achieved, namely between people who the system is interested in interacting with and other people or moving objects around the system. On the other hand, when interaction is with another robot, a different identification technique has to be used. In our case, colour patterns have been used as it is depicted in Fig. 2. Nevertheless, each detected blob has different orientation by depending on its position inside the scene. So, several rotations would be necessary in order to correctly identify/match the images of the same object in two different frames. Thus, it is necessary to obtain the panoramic image of the potential targets before running the corresponding identification algorithm in order to be successful in our goal.

3 Fig. 2. Colour patterns used for robot recognition in robot-robot interaction mode This transformation (graphically described in Fig. III) is as follows: φ = c/r med R med = (R min + R max )/2.0 (1) c 1 = c 0 + (R min + r)sinφ r 1 = r 0 + (R min + r)cosφ where (c 0,r 0 ) correspond to the image centre coordinates in terms of column and row respectively, while (c 1,r 1 ) are the coordinates in the new perspective image; φ is the angle between Y-axis and the ray from the image centre to the considered pixel; and, R min and R max are, respectively, the minimum and maximum radius of a torus which encloses the area to be transformed. These radii are obtained from the four corners of the minimum rectangle that encloses the blob to be transformed. Fig. 3. Representative perspective image from a fisheye image Note that this transformation is done only for each region detected as object of interest since transformation of the whole image could become very high time-consuming. IV. SURROUNDING ACTIVITY MONITORING On the contrary, the visual application may simply control the activity around the robotic system in order to make correct decisions such that no damage is caused. In this case, after detecting all surrounding moving elements, the robotic system should estimate collision risk. For that, the system needs to track and estimate the 3D pose of each detected blob. Firstly, with the aim of guaranteeing that each target element is described by means of one blob, a merge algorithm, based on neighbourhood and feature similarity, is applied. Then, minimum rounded rectangles are generated. After that, to perform the corresponding tracking, a pattern is built from each of them. In this case, a pattern is intended as the data structure such that allows the system to track the moving objects by means of matching an object in two consecutive frames even when it suffers a partial or whole occlusion. Thus, in our case, a target pattern is composed of two different items: a representative image of the target, obtained as explained in the previous section by reducing timeconsumption a feature array whose elements contain information about brightness and blob width and height, among other things, used to properly match the images of the same object in two consecutive frames Therefore, representative images are compared with the extracted from the previous frame. In this way, a pixelsimilarity likehood between representative images is obtained. Furthermore, a feature-similarity likehood is generated from feature array comparison. Both likehoods are properly combined to properly match the two images from the same object in consecutive frames. With regard to 3D pose estimation, since the visual system is composed of one fisheye camera, another approach to obtain that information is required. Here, we have used the stereo head camera embedded in the humanoid torso. So, the direction of moving element with respect to the robotic system may be estimated from the fisheye camera. From that information and the next movement of the robotic system, we determine the target with the high likelihood to crash with the system. Then, after properly positioning the stereo head, depth estimation takes place. Finally, we determine whether the next movement may be safely carried out or the system has to wait for a while until the moving element is out of its workspace. V. EXPERIMENTAL RESULTS As mentioned above, our experimental set-up is composed of a humanoid robot which incorporates a visual system composed of a Foculus camera with a fisheye lens mounted in the front side of the robot base such that it is pointing upwards to the ceiling in order to cover the whole robot system workspace (see Fig. 1). Images are acquired in 24- bit RGB colour model with a 640x480 resolution. Two different kinds of experiments were carried out. First, the performance was evaluated when the system is waiting for an interaction. Then, environment monitoring was analised. Firstly, an interaction task was studied. For that, visual system was located at different places in our laboratory room and some individuals were moving around it in order to be detected and identified. Figure 4 shows some of the obtained results such that, in a first step, individuals were successfully detected anywhere they were located with respect to the

4 robotic system (up to a distance of 10 metres) from the captured image. Then, their corresponding perspective image was obtained in order to be finally identified by means of Viola-Jones classifier. Take into account that the classifier input is the whole perspective image of the individual and not the top part of it. The reason lies in there is no warranty the body proportion is visible at any time. So, people might be identified even when partial occlusions of the body occur. Anyway, computation time for face detection was considerably reduced from ms -required when the whole image was considered (640x480 resolution)- to ms when the generated perspective image of the detected blobs were analysed. Capture Image Segmentation Result Panoramic Image Fig. 5. Some experimental results for detection evaluation such that different factors, which might affect segmentation process (i.e. clothes colour, individuals location or illumination), are studied Fig. 4. Experimental samples of people detection and identification. The designed visual application detects different individuals moving around the system from fisheye images. For each one, a perspective representation is obtained and provided to Viola-Jones classifier for their identification by considerably reducing its time-consumption Secondly, the system dependability was evaluated when environment surveillance is considered. Again, two different kind of experiments were carried out. As it can be observed in Fig. 5, visual application was tested from a detection point of view when some individuals were moving around the system. It was only considered detection aspect because in that way it could be studied system restrictions when different scenarios and conditions such as clothes colour, individuals location or illumination, are considered. Segmentation results show the proper detection of all the individuals whenever they were and whichever their features were. Moreover, designed application was assessed for safety issues in the sense of avoiding collisions. It is a keypoint when robot systems have considerable dimensions and caused damage could be considerable. So, from a fisheye image, the visual application detects all moving surrounding elements and decides which one has the highest risk to be crashed with the next movement to achieve its task. Once their direction is estimated, the stereo head is located in order to take a stereo pair that allows to estimate their depth, as it is depicted in Fig. 6. In that way, system determines if next movement may be safely carried out or, on the contrary, it has to wait until its movement path is free of collisions. VI. CONCLUSIONS In this paper, a robust visual application for motion detection has been presented. For that, fisheye cameras are used since they provide panoramic vision and a reduced number of them allows to covering the whole workspace. So, firstly, a new background maintenance technique has been implemented. The proposed approach combines moving object detection with global illumination change identification. So, basically, it is composed of two different stages. The first one is to build a background model but considering the possibility that foreground objects may be present. So, contraints such as waiting a period of time to build the initial background, clothes colour or illumination conditions do not exist since a segmentation process is carried out with a mixture of two well-known difference techniques. In the second stage, a background subtraction takes place. After that, the obtained raw pixel classification is improved by means of the same difference mixture. Moreover, a sudden illumination change detection is incorporated in both stages to properly adapt the system to it. In a similar way, it is also capable of detecting people when they are not in movement. In addition, we have presented a target identification technique for human- or robot-robot interaction. Its main advantage is that, instead of searching in the whole image, its input is only composed of the detected moving elements what considerably reduces time consumption. Moreover, an environment surveillance has been also introduced. Mainly, its goal is guaranteeing safety to the surrounding elements when tasks to be performed can cause any damage. Finally, experimental results have highlighted its performance, by recognising all-embracing activity in real, unknown environments.

5 Fisheye Image Pair Stereo Image Depth Map Estimation Fig. 6. Depth estimation process during environment monitoring. Firstly, a fisheye image is captured in order to detect moving elements around the system. Then, among potential targets, the system chooses the one with the highest risk to be crashed with the next movement in order to estimate the likehood of collision. Two stereo images of that target are taken by means of the stereo head of the humanoid torso and, from them, a depth estimation is obtained. VII. ACKNOWLEDGMENTS Authors want to thank the support received from WCU (World Class University) program through the National Research Foundation of Korea funded by the Ministry of Education, Science and Technology (Grant No. R ), by the European Commission s Seventh Framework Program FP7/ under grant agreement (EYESHOTS project), by Ministerio de Ciencia e Innovación (DPI ), by Generalitat Valenciana (PROMETEO/2009/052) and by Fundació Caixa Castelló- Bancaixa (P1-1B ). REFERENCES [1] K. Toyama, J. Krum, B. Brumitt, and B. Meyers, Wallflower: Principles and practice of background maintenance, in Seventh IEEE Int. Conf. on Computer Vision, vol. 1, Kerkyra, Greece, 1999, pp [2] I. Haritaoglu, D. Harwood, and L. S. Davis, W4: Real-time surveillance of people and their activities, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp , August [3] H. Liu, W. Pi, and H. Zha, Motion detection for multiple moving targets by using an omnidirectional camera, in IEEE Int. Conf. on Robotics, Intelligent Systems and Signal Processing, vol. 1, Changsha, China, October 2003, pp [4] Z. Zhu, D. Karuppiah, E. Riseman, and A. Hanson, Keeping smart, omnidirectional eyes on you. adaptive panoramic stereovision for human tracking and localization with cooperative robots, IEEE Robotics and Automation Magazine, pp , December 2004, special Issue on Panoramic Robotics. [5] C. Stauffer and W. Grimson, Learning patterns of activity using real-time tracking, IEEE Trans. on Pattern Analysis and Machine Intelligence PAMI, vol. 27, no. 5, pp , Aug [6] P. KaewTraKulPong and R. Bowden, An improved adaptive background mixture model for real-time tracking with shadow detection, in 2nd European Workshop on Advanced Video Based Surveillance Systems (AVBS01), [7] D.-S. Lee, Effective gaussian mixture learning for video background subtraction, IEEE Trans. on Pattern Analisys and Machine Intelligence, vol. 27, no. 5, pp , Aug [8] L. Niu and N. Jiang, A moving objects detection algorithm based on improved background subtraction, in Eighth Int. Conf. on Intelligent Systems Design and Applications, vol. 3, Kachsiung, Taiwan, Nov. 2008, pp [9] Y. Benezeth, P. Jodoin, B. Emile, H. Laurent, and C. Rosenberger, Review and evaluation of commonly-implemented background subtraction algorithms, in 19th International Conference on Pattern Recognition (ICPR), Tampa, Florida, December 2008, pp. 1 4.