Bridge inspection robot system with novel image processing

Transcription

1 Bridge Maintenance, Safety, Management, Health Monitoring and Informatics Koh & Frangopol (eds) 2008 Taylor & Francis Group, London, ISBN Bridge inspection robot system with novel image processing Je-Keun Oh, An-Yong Lee, Se Min Oh, Youngjin Choi, Byung-Ju Yi & Hai Won Yang Department of Electronic, Electrical, Control and Instrumentation Engineering, Hanyang University, Ansan-Si, Gyeonggi-Do, Korea Jeong Ho Lee & Young Shik Moon Department of Conputer Science and Engineering, Hanyang University, Ansan-Si, Gyeonggi-Do, Korea ABSTRACT: This paper proposes the design and control methods for a bridge inspection robot system with the novel image processing system. The bridge inspection robot system has been developed with the aim of checking the safety status of a real bridge and gathering accurate data such as crack width and length. The developed robot system is composed of the moving mechanism mounted on the specially designed car for bridge inspection and the novel image processing system. Especially, this paper emphasizes the system integration method to design and control the entire robot system. Also, the users can see the operating conditions of long-distance robot with 3-dimensional graphical user interface. Moreover, the users can feel the operating status of long-distance robot through haptic device. 1 INTRODUCTION Although the robot technologies have advanced in a variety of the industrial fields, the robot application technologies for the safety diagnosis and maintenance of real bridge have lagged behind. Currently, bridge inspection and maintenance have been manually conducted by the educated inspection workers in poor surroundings (FHWA 1991, US FHA 1992). In fact, the inspection workers check the safety status beneath the bridge only by counting the number of cracks or measuring the maximum width of a crack line. As a result, the accuracy of safety report has been weakened. Also, the quality of inspection work varies greatly according to the diligence of inspection workers. Also, since the bridge inspection is performed outdoors, especially beneath the bridge, there is always concern with the safety of inspection workers. Figure 1 shows inspection workers standing on temporary scaffolding established in order to inspect the safety status (Tsutomu & Atsushi 1997, Paxton-Mitchell Snooper). Figure 1. Inspection workers standing on temporary scaffolding. The absence of the safety device and standard can cause industrial disaster in performing the bridge inspection. So, the improvements of working environment have become the first consideration in bridge inspection. Hence, there is a necessity to develop the bridge inspection robot system by using the unmanned robotic technologies. An inspection robot is the most useful when it can carry sensors into inaccessible or hazardous areas, thereby making the task safer for 1207

2 the inspection workers (R.T.Pack 1997). For example, the robot system for the underwater inspection of the bridge piers has already been developed in (De Vault 200). In this paper, we suggest a bridge inspection robot system which can move to desired location for the precise inspection of a specific location with various sensors. Also, the developed robot system is equipped with the tele-inspection system to control the robot at a distance. This paper is organized as follows: first, the total bridge inspection robot system and mechanism are explained in section 2. In section 3, a tele-inspection system is proposed for the longdistance operation and inspection. In section 4, we explain the simple method about the position control. And in section 5, The method for novel image processing is explained. Finally, we draw total experiment and conclusions in section 6. 2 TOTAL SYSTEM FOR BRIDGE INSPECTION The total system for the bridge inspection consists of a specially designed car, the guide rail and the inspection robot as shown in Figure 2. The guide rail is located on the end-effector of multilinkage system unfolded from a specially designed car. Also, the inspection robot system is equipped on the end-effector of multi-linkage system of special car. It was designed to be able to move on a guide rail for long-distance tele-inspection and to precisely inspect the wide area beneath the bridge. Now, we explain each component of the bridge inspection robot system in detail. The bridge inspection robot platform consists of three parts, the base platform, up and down platform, and the camera mounting mechanism, which is designed to find the cracks beneath the bridge. Figure 2. Total system with bridge (cross section view). 2.1 Mechanism to move up and down We have designed two mechanisms of the sliding and scissors types. The scissors type as shown in Figure 3 is able to have the larger workspace than the sliding type. Also, the motion of the former is smoother than the latter, but it has the weak point that the mechanism of scissors type is very complex. On the other hand, the sliding type in Figure 4 is more stable and substantial than the scissors type in view of the operation. Figure 3. Scissors type. Figure 4. Sliding type. 1208

3 2.2 Base platform mechanism The base platform is designed with four motors for the back and forth movement of the robot on the guide rail as shown in Figure 5. Figure 5. Base platform. Figure 6. Camera mounting mechanism. 2.3 Camera mounting mechanism The environment under a real bridge is not constant, for example, the wind can cause the swing motion of guide rail. In this case, the rail is shaken and bent by the external environmental disturbances. In order to deal with these kinds of unknown disturbances, we attached to the driving motor and gyro sensor to maintain the horizontality as shown in Figure Mechanism integration Each module explained before is assembled as shown in Figures. 7 and 8. These modular mechanisms have the merit of the easy maintenance and management about the total system because the developed one has the decoupled modular structure. Also, these modular mechanisms make the control and kinematics easy. Figure 7. sliding type robot. Figure 8. scissors type bridge inspection robot and guide rail. 3 TELE-INSPECTION ALGORITHM 3.1 Composition Phantom Omni Haptic Device This device can give 'force reflection' to operator about the robot motion status. The device consists of 3 translations and 3 rotations, and force control by user is possible(sensable Co.). Now we are to implement the X-directional and the Z-directional force reflection by using the feedback signals as suggested in Figures 8 and 9. Also, the force reflection means the implementation of force given to the haptic system by the feedback signal. Then, the user can feel the force as strong as the quantity given to inspection robot. 1209

4 Figure 9. Haptic System. Figure 10. User Interface User Interface Currently, the developed UI has the structures in Figure 10. The developed UI consists of the status window of the haptic signals and the OpenGL window for the 3D visualization. Also, we added 3D robot simulator to the UI based on the geometrical modeling of the real robot as shown in Figure 10. We developed the LAN interface board with 10Mbps. The DSP 2812(manufactured by TI Co.) is able to communicate with PC in 150MHz as shown in Figure System integration Tele-Inspection System The tele-inspection algorithm is implemented in real time by using the RTX kernel taking charge of UDP communication. The RTX operating system by using RTX.dll is shown in Figure 11. Figure 11. RTX Operation System. Figure 12. Tele-inspection system. The win32 process receives the haptic signals and it transfers the haptic signals to the RTX process by using the shared memory as suggested in Figure 12. Also, the RTX process communicates with the robot through the UDP and sends the received feedback signals to the shared memory. The feedback signal is used for the force reflection and the 3D visualization Total Control System of Bridge Inspection Robot The PC in the tele-inspection station and the DSP2812 of the robot communicate with each other through the wireless UDP as shown in Figure 13. In other words, the PC in the teleinspection station transfers the haptic signal to the robot through the wireless UDP, also, the robot returns the feedback signal to the tele-inspection station after processing the haptic signal from the tele-inspection station. Finally, the tele-inspection station realizes the force reflection through the haptic device and the 3D visualization. 1210

5 Figure 13. Tele-Inspection System. Figure 14. Communication Delay Time. Actually, the communication delay can cause the serious problem in the case of the real-time force reflection system. In our case, the average of the communication delay time between the DSP2812 on the robot and the tele-inspection station is about 17[ms] as we can see in Figure 14. This value does not cause the serious problem in our system. 4 POSITION CONTROL This section suggests the position control method for the tele-inspection robot system. The total control structure consists of the laser sensor to measure the distance between beneath the bridge and a camera mounted on the end-effector of the bridge inspection robot system, the DC motor, DC motor drive, and LAN communication as shown in Figure 15. In order to read the analog value of the laser sensor, the precise 16-bit ADC and its interface board are developed. DSP PID Control Motor & Driver Motor Encoder LAN Board ADC (AD7890) Laser sensor Figure 15. Position Control Structure. As shown in Figure 15, the well-known PID (Proportional, Integral, and Derivative) controllers (Kuo & Golnaraghi 2002, Ogata 2002) are used for the X-directional and Z-directional position control as following form: de u = K e + K edt + K p I D dt (1) where u is the control input, e is the error, K p is the proportional gain to be multiplied with K is the differ- D the error. K is the integral gain to be multiplied with the integral error and I ential gain to be multiplied with the derivative error. 4.1 Position control of Z-directional motion with laser sensor The Z-directional motion control has two modes such as the automatic mode implemented in normal inspection and the manual mode for the precise inspection about a specific location. 4.2 Position control of X-directional motion with encoder The DSP2812 equipped with LAN module receives the encoder data and efficiently controls the DC motor with the PID control algorithm for the robot motion to a specific location by the user. Also, the user can see the situation of the robot with the 3D visualized UI. 1211

6 4.3 Experimental results In order to confirm the control performance, first, the position control, velocity control, and tracking control are implemented for the bridge inspection robot system. A shown in Figure 16, the experimental result of position control shows good performance for the desired 67 degree position. In this case, the maximum steady state error is about 0.5 degree. This corresponds to about 0.75% position error to the target. Figure 16. The result about desired 67 degrees. Figure 17. Tracking performance. Finally, the tracking performance is important to confirm the stability of the entire system. So, we repeatedly accelerate and decelerate the desired motion with RPM as much as between 1500RPM and 540RPM. The experimental result shows the good tracking performance as shown in Figure 17. Through these experiments, we can guarantee the stability of total control system when the robot is operated beneath the bridge for long distance tele-inspection 5 MACHINE VISION SYSTEM 5.1 Purpose of the system The purpose of the machine vision system is to detect cracks of bridge surface automatically from image inputs. The crack information can be used to maintain the bridge and to decide an appropriate rehabilitation method to fix the bridge with cracks. This information is very important for managing the bridge. Since most existing bridge inspection systems require many operators to inspect the defects of bridges by human eyes, the safety of inspectors in dangerous circumstances of the workplace is not ensured. Besides, the results of inspection can be degraded in terms of accuracy, effectiveness and practicality. Therefore, it is necessary to develop an automatic machine vision system, in order to improve worker s safety, effectiveness, accuracy as well as cost reduction. This system is composed of CCD cameras, a DVR board and a computer. The specifications for the vision system should be determined, considering weight, electric power, communication scheme and cable width. Conforming to the system specifications, the hardware system is designed. Related algorithms for processing the images captured from cameras have been designed and implemented. 5.2 Crack detection and tracing Existing systems for crack detection simply display the detected cracks. However, for more effective bridge inspection, we also need some information about the crack assessment such as length or width of the cracks. In the crack detection, there are many problems such as irregularities in crack shape and size, various soiling and painted surfaces, and irregularly illuminated conditions. These may cause serious problems in automatic crack detection. In order to solve these problems, we propose the following method for automatic crack detection. Our method consists of two steps: crack detection and crack tracing. For the crack detection, we perform three steps of pre-processing and extract the candidate cracks. Firstly, we subtract the smoothed image from the original image. The smoothed image is obtained by using a me- 1212

7 dian filter (Fujita 2006). Smoothing with the median filter is used to remove thin line structures such as cracks. Therefore, in the subtracted image, cracks are prevented without variations. Secondly, we remove some artifacts using a filter for removing isolated points. Thirdly, we apply some morphological processing such as dilation and thinning to guarantee the connection between cracks, where the number of iterations is determined by the distribution of candidate cracks. After this process, we obtain the real cracks from the original image. Figure 18. Result of crack detection. For the crack tracing, we divide the image with detected cracks into several regions and select a seed point in each region. For each seed point, we examine the intensities of 8-neighbor pixels to determine the next pixel with the minimal intensity. To avoid local minima, the range of direction is restricted. We measure the width and the length of each crack. Figure 18 shows the result of crack detection. 5.3 Making the results into database The results of crack detection should be stored into database, in order to keep all the information necessary to maintain the bridge. The results of inspection are converted into an interchangeable file so that the results can be used in the Bridge Management System(BMS). The interchangeable file is stored in dxf format that is compatible with CAD files. Moreover, we should also be able to display the whole image of the results so that the defects in a wide area beneath the bridge can be examined at a glance The structure of the dxf file format is carefully investigated to parse the syntax of each component, in order to write the information of detected cracks into a dxf file. Figure 19 shows the result of created dxf file. The detected cracks and the result of image stitching are shown in Figure 19. Figure 19. Result of crack detection. 6 TOTAL EXPERIMENT AND CONCLUSION We have tested an inspection robot system beneath the miniature bridge model as shown in Fig- 1213

8 ure 20. Both the automatic mode and manual mode were implemented successfully. Two snapshots (3 seconds, 20 seconds) captured in the operation were suggested in Figure 20 according to the time progress. The robot could move on the rail about the X-axis according to the command by haptic device and it could maintain the distance of 983[mm] constantly between beneath the bridge model and the end-effecter of the inspection robot about the Z-axis automatically as shown in Figure 20. As a result, the inspection robot system could gather the visual information about the cracks. Also, you can see the experimental video clips through our ftp server (Humanoid Lab 2006). (a) After 3 seconds (b) After 20 seconds Figure 20. Driving Test. This inspection robot system has been developed for real application. In this paper, the mechanism design and control structure used for the bridge inspection were proposed. Actually, the system integration is important for the real robotic application and commercialization. And the robustness of tele-inspection will be later the main issue in the bridge inspection because the outdoor environments are seriously bad compared to the indoor. Therefore, we have suggested the concept and software structure of tele-inspection by the long distance operator. Moreover, the proposed machine vision system can not only detect cracks in real time, but also has some utility functions for supervised manipulation. It is possible to detect the crack of over 0.2mm width with this machine vision. ACKNOWLEDGMENT This work was supported by MOCT (Ministry of Construction and Transportation) & KICTTEP, Rep. of Korea. REFERENCES Federal Highway Administration (FHWA) Bridge Inspection's Training Manual, July. US Federal Highway Administration Bridge Maintenance Training Manual, FHWA-HI , Prepared by Wilbur Smith Associates. S. Tsutomu, & S. Atsushi Summary Report of Research and Study on Robot Systems for Maintenance of Highways and Bridges Robot. no JARA Tokyo. Japan : Product Catalog, Paxton-Mitchell Snooper Series 140, R. T. Pack, J. L. Christopher Jr & K. Kawamura A Rubbertuator-based structure-climbing inspection robot, IEEE International Conference on Robotics and Automation. J.E. De Vault Robot system underwater inspection of bridge piers. IEEE Instrumentation and Measurement Magazine 33: Phantom Omni Technical Specification, B. C. Kuo & F. Golnaraghi Automatic Control; Systems, 8th Edition, WILEY K. Ogata Modern Control Engineering 4th ed, Prentice Hall. Fujita, Y A method for crack detection on a concrete structure. International Conference on Pattern Recognition 2006 : Humanoid Lab.1996: ftp://humanoid.hanyang.ac.kr 1214