Automated Sewer Inspection Using Image Processing and a Neural Classifier

Transcription

1 Automated Sewer Inspection Using Image Processing and a Neural Classifier Olga Duran, Kaspar Althoefer, Lakmal D. Seneviratne Department of Mechanical Engineering King's College London Strand, London WCR LS, UK olga.duran@kcl.ac.uk Abstract: - The focus of the research presented here is on the automated assessment of sewer pipe conditions using a laserbased sensor. The proposed method involves image and data processing algorithms categorising signals acquired from the internal pipe surface. Fault identification is carried out using a neural network. Experimental results are presented. I. INTRODUCTION The degradation of sewer networks, mainly due to its age, is becoming a serious issue in most industrialised countries. According to a study by a UK-based company, there are approximately 5,000 collapses and 00,000 blockages per year in the United Kingdom and around 0% of sewer networks are damaged [1]-[]. Considering that only 0.1 % of the sewer system is replaced every year, these figures are expected to increase by 3% every year. Early detection of pipe faults may prevent severe failures that can involve environmental damage and high economic costs. Closed circuit television (CCTV) surveys are widely used to assess pipe condition when direct human access is not possible. A camera fitted on a mobile and remotely controlled platform travels through the pipe recording images onto a videotape. The videotapes are assessed off-line by an engineer. One of the major drawbacks of this technique is the exorbitant amount of information generated, that can reach about 3 hours of video for the inspection of 1Km of line [3]. Moreover, the time required by the engineer to assess the pipe condition depends on the number of defects. Owing to the variability and the time consumption incurred by the humanbased assessment process, automation of this part of the inspection task becomes an important target. Some relevant research has been carried out in the field of automated sewer inspection. For instance, the Civil and Environmental Engineering group, at Concordia University (Canada), is developing a model which will help to automate the process of identifying and classifying several surface defects from digitised video images of underground water and waste pipes, using image analysis, pattern recognition and neural networks [4]-[5]. The Department of Systems Design Engineering, at the University of Waterloo (Canada), have developed a neuro-fuzzy algorithm to recognise and classify cracks in underground pipes [6]. Another example, is PIRAT (Pipe Inspection Real-Time Assessment Technique), an experimental system developed by CSIRO (Commonwealth Scientific and Industrial Research Organisation) and Melbourne Water, Australia. The PIRAT platform measures and analyses the internal geometry of the sewer. Automatic recognition, rating and classification of pipe defects are carried out by means of artificial intelligence software tools. Images are segmented into potentially defective regions and characteristic geometric features. Region classification is performed by a feed-forward, off-line trained neural network classifier [7]. In this paper, the defect detection method, based on camera images, tries to cope with some CCTV drawbacks, mainly related to illumination. Another drawback of current CCTVbased inspection systems is that only non-flooded pipes can be inspected. Considering that typically, the lowest water level is around 5% in a combined sewer during dry weather, other sensors are needed to inspect the flooded part of the pipe. The investigation reported in this paper is part of a research programme that aims to develop automated sewer inspection methods. The objective of this project is to study and develop multi-sensor systems and to create intelligent sensor fusion and sensor data processing algorithms. A sonar has been found most suitable in order to inspect the flooded areas [8]-[9], while an optical-based sensor is appropriate for inspecting the area above the water level. It is envisaged that a laser-profiler-ccd camera system scans the inner wall of the drained part of a sewer. This paper presents research progress on the inspection and automated condition assessment of the non-flooded part of the pipe. The method described makes use of image processing techniques and intelligent classification algorithms, providing an identification of defective pipe segments. The different

2 processing stages such as data acquisition, image processing and classification are described in the following sections. In Section II, the experimental set-up together with the transducer that is used to acquire the images is described. Image segmentation into regions of interest and feature extraction will be shown in Section III. The classification step is shown in Section IV. Finally, conclusions are given in Section V. II. EXPERIMENTAL SET-UP (a) (b) In this work, a laser-based profiler, which can easily be incorporated into existing camera-based systems, is used to inspect the areas above the water level. The transducer consists of an optical pattern generator that projects rings of laser light onto the pipe wall, illuminating an entire circular pipe section. The optical ring generator is made of an assembly of a laser generator and diffractive diffuser optics. The experimental rig used consists of a 1mW-semiconductor laser diode of 635nm wavelength and a diffractive diffuser for the generation of ring patterns. The projected circular patterns of light are read by a calibrated CCD camera, and stored in a computer via a frame grabber (fig. 1). Tests were conducted on PVC pipe sections with inner diameters of 60 and 300 mm. Holes and cracks of different sizes were pierced into the walls of the pipe segments in order to simulate defects. (c) Figure : Images of a pipe with a mm hole. The location of the hole is marked with a small circle. a) Image acquired using the laser-based transducer, where the crack can be easily identified. b) Image acquired with a conventional CCD camera and lighting system. Note that the white spot next to the circle is created by a light reflection; it is not a defect, but could be wrongly identified as one. c) Image acquired using the laser-based transducer, corresponding to the location of the non-defective point caused by light reflection as referred to in b). This image shows that the proposed method correctly presents this point as non-defective. Figure 1: Experimental set-up, where a camera and a laser projector are placed next to each other. The camera images are read into a PC by means of a frame-grabber. An important advantage of this method is that the processed and normalised signal can be fused together with sonar data from below the sewer liquid in order to create a full picture of the internal walls of the pipe. Furthermore, as the geometric characteristics of the cone of light are fixed for a given diffuser optic, the location of the surface under investigation relative to the platform is known. Consequently discontinuities and their location can be determined. This kind of sensor has been successfully used to measure pipe deflection or change of shape comparing consecutive images using the triangulation method. Besides that, local discontinuities, such as cracks or holes can be detected by analysing the light intensity of the projected rings in the same images. At points where discontinuities occur, the laser light is scattered resulting in changes of intensity levels in the acquired camera image (fig. ). Analysing these intensity levels, defects can be detected. III. FEATURE EXTRACTION A set of image processing stages is used to identify the ring profile and to extract it from the image background. Once a ring profile is properly extracted, a feature extraction algorithm scans the profile only, instead of searching the

3 entire and potentially very large pipe wall image for defects. This process is repeated for each of the pipe profiles. This locally approach has been shown to considerably reduce the amount of data to be processed, and consequently the computation time. Prior to applying the conic-fitting algorithm, image quality needs to be enhanced. Histogram equalisation and a median filter are applied, so that noise is attenuated and edges are preserved. Finally, a canny edge detector has been found most efficient to pre-segment the image by finding the edges of the ring of light [10]. 4 B. Intensity transform b = 1 ac (6) In this stage, the extraction of intensity information along the ellipse is carried out. In order to achieve this, a signal representing a partial histogram of the image is created using local averages of intensity along the ring profile. The intensity transform algorithm computes local averages of intensity along a defined number of segments of the ellipse. The local average intensity is computed using the equation (7): A. Ellipse extraction conic fitting The method has been used in this work to efficiently fit the image to a conic. It is based on the ellipse fitting method proposed by Fitzgibbon [11], improved by a method suggested by Halýr [1]. The approach is a non-iterative algorithm based on a least squares minimisation of the algebraic distance between the data points and the ellipse. A quadratic constraint guarantees an ellipse-specific solution even for scattered or noisy data, or for a partial occlusion condition, which can occur when the sewer is partially flooded. Also to improve the behaviour of the algorithm in the presence of noise, a weighted function is applied to the points before starting the computation. µ x µ Where x is the grey level of a certain image point, and µ the frequency of that grey level. (7) This method is based on the representation of a conic by an implicit second order polynomial: ( a, x) = a x= a x + b x y+ c y + d x+ e y+ f = 0 F (1) Where, a [ a b c d e f ] T = () x [ x x. y y x y 1] T = (3) ( a, x) F (4) Figure 3: Feature extraction algorithm; a sliding window is applied along the ellipse. The peak indicates the location of a potential discontinuity. In the current experiment, a pipe with a diameter of 60 mm is used. A mm hole has been pierced at 1 degrees from the horizontal axis. The partial histograms have been computed with steps of one degree along the ellipse segments. Equation (4) is the algebraic distance of a point (x, y) to the conic. The fitting consists of minimising the squares of the distances, in equation (5), subject to the equality in constraint (6), which incorporates both scaling and ellipse-specific conditions. ( a, x) = 0 F (5) These averages are computed over a sliding window covering areas along individual ellipse segments. Figure 3 shows the results of the feature extraction process displayed as a partial image histogram. Figure 4 shows the open-up representation of the inner surface of a pipe. This is created with successive histograms computed from the pipe surface images when a platform holding the transducer is travelling along the pipe.

4 1) Emphasising intensity variations: The differential of the signal is computed in order to emphasise intensity variations. This operation effectively attenuates small changes in intensity, that are primarily due to minor differences in illumination across the section under study. Figure 4: Open-up representation of the inner surface of the pipe. The intensity peak shows the location of a potential defect. IV. CLASSIFICATION USING A NEURAL NETWORK ) Signature generation: Although the data resulting from the previous operation could be used to feed the NN, one more step is taken in order to further enhance the data supplied to the network. The data received form each pipe section is made up of individual samples, which are sorted by signal amplitude in descending order, using the quicksort algorithm. This means that any defective section, having one or more intensity peaks, will be converted into a monotonically decreasing sequence. A normalisation step is now applied, by subtracting the values representing each section to their respective maximum value. The resulting signal is a monotonically increasing sequence starting from zero, where non-defective sections will appear as fairly flat signal. On the other hand, signals representing a defective section will present a sharp slope reaching then a fairly flat region at a higher amplitude (see figure 5). We will call these sorted signals signatures. The aim of this research is to use a neural network for the identification of defective and non-defective pipe sections and for the classification of different defects in the surface of the pipe. A binary approach is taken in this work, where pipe sections are classified as defective or non-defective. In further work, defects will be classified into different categories such as longitudinal cracks, radial cracks, holes and obstacles and a fault severity rating will be given as well. The input to the NN is a pre-processed signal that highlights the characteristics of defective versus non-defective pipe sections. The network employed here is a multi-layer perceptron (MLP). The training algorithm used in this work is based on backpropagation, which is widely used for solving classification problems [13]-[14]. A. Pre-processing A first pre-processing stage is used to normalise and reduce the amount of data supplied to the network. In order to identify potential defects in the surface of the pipe, the target data has to be prepared in order to show great variations in intensity levels. It has to be considered that the main difficulty is to discard the points where the intensity changes are caused by variations in the lighting conditions. Before feeding the neural network, the raw data is pre-processed and normalised as follows: Amplitude [intensity] Defective sections Non-defective sections Sorted values Figure 5: Normalised signature of defective and non-defective sections: Defective sections with faults at different spatial locations along the profile with different widths have similar signatures. In previous work, for the classification of holes, a different approach has been used for the signature creation. In that case a Fourier transform (FFT) of the signal was used as a signature [15]. Figure 6 shows Fourier transforms of defective and non-defective pipe profiles. The peaks that can be observed in the mid-frequency range of the Fourier domain correspond to intensity variations due to physical defects. Although both the FFT and the quicksort have a computational complexity of O(n log(n)), where n is the

5 number of samples, the algorithm proposed here for signature creation has been found computationally more efficient than the FFT-based algorithm, reducing the computation time by 50%. This is believed to be due to the large number of complex products that have to be computed for the Fourier Transform. Besides that, in the FFT-based algorithm used, a low-pass moving average filter is applied to remove highfrequency noise. This step leads to an increase in computation time investigation may show that it is possible to achieve comparable results with a single hidden layer [13]. The best performance was achieved with a multi-layer perceptron with four layers: The first layer with 50 input nodes, the second with 100 hidden nodes, the third with 0 hidden nodes and finally one output node in the fourth layer. Sigmoidal activation functions were used for the hidden nodes. The data used during algorithm was a stream of normalised signatures, as shown in section 4.1, containing both faulty and non-faulty pipe sections. The training behaviour shows that this algorithm is able to classify the training data bringing the error down to 1E-8 after 1518 epochs (fig. 7). Amplitude Defective sections Non defective sections Frequency 10-6 Figure 6: FFT of defective and non-defective sections Figure 5 shows normalised signatures of defective and nondefective pipe profiles. The peaks that can be observed in the first data points correspond to intensity variations due to physical defects. A neural network can be trained with this normalised data and it will recognise all defects, regardless of their position in the spatial domain and its width along a certain profile (since the differential has been applied previously). In other words, by using these normalising steps, the NN does not need to be trained with a stream of sections that contain defects of all sizes and at all possible points in the pipe section. B. Experimental results Experiments with different configurations of NN architectures, different numbers of hidden layers, nodes and training parameters were carried out. A scaled conjugate gradient (SCG) algorithm has been found more time- and performance-efficient than conventional backpropagation algorithms [16]. The selection of training parameters such as number of nodes, learning rate or activation function, is done in a trial and error procedure. Networks with two hidden layers were found to respond slightly better at the generalisation stage, although training took longer. Considering that a trial and error procedure was used, further Epochs Figure 7: Training error with a two-hidden-layer Network Epochs Figure 8: Training error with a single-hidden-layer Network

6 The best performance, using a single hidden-layer and the same training data as above, was achieved with 100 hidden nodes. After 944 epochs, the error was also down to 1E-8 (fig. 8). The training time is shorter and the number of epochs is reduced using a single- instead of a two-hidden-layer network. The ability of the trained networks to cope with unseen data is also tested. As an example, Table 1 and summarise the output results for the five pipe profiles shown in figure 4, which were not seen during the training. Table 1 contains the results when using two hidden layers while in table, the results using a single layer are shown. The target column specifies the value 1 or 0 corresponding to non-defective and defective sections respectively. The output column represents the output node. The error is the absolute difference between the desired target and the output values. These results show that both networks are able to identify irregularities in the data (related to defective sections) even though the data was not seen by the network beforehand. TABLE 1: NN RESPONSE TO UNSEEN DATA USING TWO HIDDEN LAYERS Pipe Section Target Result Error Non-defective Non-defective Defective Non-defective Non-defective TABLE : NN RESPONSE TO UNSEEN DATA USING A SINGLE HIDDEN LAYER Pipe Section Target Result Error Non-defective Non-defective Defective Non-defective Non-defective V. CONCLUSIONS In this paper, a method for the detection of pipe defects using NNs is presented. The proposed method uses a new laser-based inspection system. The methods is based on the projection and analysis of a laser-generated pattern onto the pipe walls. Defects in the inner surface of pipes are located by analysing the light intensity of the projected rings. Imageprocessing techniques combined with an MLP trained using backpropagation provide the classification between defective and non-defective pipe sections. Experiments have been conducted in a realistic environment and results have been presented. Future work will focus on the classification of defects such as longitudinal or radial cracks and holes. The long-term objective of this research is to assess partially flooded pipes using a multi-sensor system. The transducer described in this paper inspects the non-flooded surface, while an ultrasonic sensor will inspect the flooded parts. In order to get a complete image representing the condition of the entire surface of the pipe, sensor fusion can be carried out at the pixel, signal or feature level. VI. REFERENCES [1] Roth, H, Schilling, K. Navigation and Control for Pipe Inspection and Repair Robots : Proc of IFAC World Congress, [] WEF Manuals & Reports FD-6, ASCE Manuals & reports on engineering, Existing sewer evaluation and rehabilitation, No. 6, [3] Pace, NG. Ultrasonic surveying of fully charged sewage pipes, Electronics and Communications Engineering Journal, pp 87-9, [4] Moselhi, O. Shehab-Eldeen, T. Automated detection of surface defects in water and sewer pipes. Automation in Construction 8, pp , [5] Moselhi, O. Shehab-Eldeen, T. Classification of defects in sewer pipes using neural networks. Journal of infrastructure systems 6, 3, pp , 000. [6] Sinha, K.S, Karray, F. Fieguth, P.W. Underground Pipe Cracks Classification Using Image Analysis and Neuro-Fuzzy Algorithm : Proceedings of the International Symposium on Intelligent Control/ Intelligent Systems and Semiotic, Cambridge, MA, pp , 1999 [7] Campbell, G. Rogers, K. Gilbert, J. Pirat a system for quantitative sewer assessment, International No Dig 95, Dresden, Germany, [8] Bin Hussen, M. P. Althoefer, K. A. Seneviratne, L. D. Automated Inspection of Sewers Using Ultrasonic Sensors, Proc. EUREL European Advanced Robotics Systems Masterclass and Conf., Vol.1, Salford, UK, 000. [9] Bin Hussen, M. P. Althoefer, K. A. Seneviratne, L. D. "Automatic sewer defect detection and classification using a neural network", Neural Network World, Vol. 10, No. 4, pp , 000. [10] Gonzalez R.C. Digital Image Processing. Addison-Wesley, MA, [11] Fitzgibbon, A. Pilu, M. Fisher R. Direct Least Square Fitting of Ellipses. Pattern analysis and machine intelligence, Vol. 1 Issue 5, pp , [1] Halir, R. Flusser J. Numerically stable direct least squares fitting of ellipses. The Sixth International Conference in Central Europe on Computer Graphics and Visualization, Plzeò, pp , [13] Hush, D.R. Horne, B.G. Progress in supervised Neural Networks. IEEE Signal Processing Magazine, pp 8-39, [14] Haykin, S. Neural Networks. A Comprehensive Foundation. Macmillan College Publishing Company, NY, [15] Duran, O. Althoefer, K. Seneviratne, L.D. Automated Laser-based Profiler for Sewer Inspection. Proceedings of the 7th International Conference on Intelligent Autonomous Systems (IAS-7), Marina del Rey, California, USA, 00 (Accepted for publication). [16] Moller, M. F. "A scaled conjugate gradient algorithm for fast supervised learning," Neural Networks, vol. 6, pp , 1993.