Unsupervised Video Analysis for Counting of Wood in River during Floods

Unsupervised Video Analysis for Counting of Wood in River during Floods Imtiaz Ali and Laure Tougne Université delyon,cnrs Université Lyon 2, LIRIS, UMR5205, F-69676, France Abstract. This paper presents a framework for counting the fallen trees, bushes and debris passing in the river by monocular vision. Automatic segmentation and recognition of wood in the river is relatively new field of research. Unsupervised segmentation of the wooden objects moving in the river has been developed. A novel method is developed for the separation of wood from water waves. The counting of number of fallen trees in the river is realized by tracking them in the consecutive continuous frames. The algorithm is tested on multiple videos of floods and the results are evaluated both qualitatively and quantitatively. 1 Introduction Automatic video surveillance addresses the challenge to perform real-time analysis and constant monitoring of activity [1]. This automation helps in the improvement of the safety in our surroundings. The remote surveillance of unattended environments is often done in places like airports, highways, railway infrastructures, parking lots and on the roads. In most of the cases the surveillance systems detect the potentially threatening incidents. The monitoring of rivers using cameras is done from many years. During the floods, there are large numbers of fallen trees, debris, branches and roots of trees carried by water. These fallen trees and bushes block the flow of water in mountains. Moreover they threat the bridges and dams as these fallen trees are accumulated over the period of time during floods. The monitoring systems installed over the rivers are usually manually supervised. Automatic detection of these trees will help to take preventive measures during floods. The statistics of the fallen trees carried every year with floods will help in finding the maximum number of wood passing in the river every year and the time of the year during which one could expect the flooding. The number of fallen trees and wood in the river requires image segmentation and motion tracking of the fallen trees inside the water. The detection of wood inside the river is the example detection of object motion within moving background. The videos we study in this paper are from the camera installed on the river Ain (France). Figure 2 gives some examples of extracted images from such videos in the first row. The complex natural environments often have many constraints. Such constraints can be classified in two groups, the constraints for detection and recog- G. Bebis et al. (Eds.): ISVC 2009, Part II, LNCS 5876, pp. 578 587, 2009. c Springer-Verlag Berlin Heidelberg 2009

Unsupervised Video Analysis for Counting of Wood in River during Floods 579 nition, and the constraints of tracking of moving wood in the river. The detection and recognition of wood depend on the luminosity difference between wood and water. The flow of water in rivers contains turbulences and waves that are more prominent in case of floods. In addition to that the cloud movement in sky that causes changes in the brightness over the surface of river. The difference of the luminosity of the waves and the wood is not very important. Moreover, the shadows of surrounding trees and buildings make the situation more difficult for correct foreground/background extractions. The image segmentation is not easy in the presence of some moving tree braches in front of the surveillance camera. The bridges in the monitored scene also produce strong shadows over water surface. Consequently, in the moving background the objects can only be detected by virtue of their existence in the multiple consecutive frames. Furthermore, counting of the number of fallen trees that are passing through some strategically important places during flood requires that the waves present in the river must be separated from the fallen tree or wood. The tracking of the foreground objects in this case has some constraints too. The water waves and wood that are moving with the same speed make it difficult to distinguish between the two ones. The motions of wood and water waves inside the river are not linear. For good tracking of the moving objects it is necessary that the objects should be present in the multiple consecutive frames. The water waves during the floods are so large that they submerge the fallen tree branches, and the size of the objects does not remain the same in consecutive frames. In case of small wood pieces or debris, these water waves totally submerge them and they appear in one frame and remain submerged in the next two or three frames. Finally, due to remote location of the monitoring scene and limitations of transfer rate of data networks, the frame rate (fps) in the video is very low ( 4 fps). Consequently, the object motion is larger in consecutive frames. This paper is organized as follows. Section 2 presents the review of relevant works in similar situations and highlights the constraints and technical difficulties in our case. In section 3, the proposed methodology for detection is described. In section 4 the experimental results and comparison of the results with statistical data obtained manually is presented. 2 Related Works Automatic segmentation and recognition of wood in the river is relatively new field of research. There are not many articles in the literature in such type of application. In this section we present the previous works on the detection of foreground objects in the non-stationary backgrounds. For foreground detection the adaptive background model is proposed for non-stationary backgrounds [2]. The background model plays the role of reference image in background subtraction techniques. It is constructed by adapting the changes during the training period. The construction of background model is based on the different image features (spectral features, spatial features and temporal features). For construction of background model based on the

580 I. Ali and L. Tougne spectral characteristics, Gaussian Mixture Model (GMM) method is used by most of the researchers [3,4,5], where one or more Gaussians are used to represent the spectral features at each background pixel. All these methods are used in situations of very small dynamic background movements. The GMM method leads to misclassification when the background scene is complex [6], [7]. In [1] the method of spatio-temporal filtering is proposed for compensation of the limitations of region-based blocks of images. This method is applied for detection of swimmers in the swimming pool. The spatial features are extracted by gradient analysis, which gives the information of movements in the images. The mixture of spatial features with spectral features extracted from the image for foreground extraction is used by [8]. Our method is inspired from these but gradient analysis alone is not sufficient in our case because the water waves and wood both have strong gradients. For moving object detection in the video temporal characteristics are very important. The optical flow technique proposed by [9] is largely used for this purpose. Many researchers use this technique. [10,11] used the estimation of the consistency of optical flows over a short duration of time. But the consistency of local optical flows requires small displacements from one frame to another. In our case, the videos are having very low frame rate ( 4 fps), due to which there is large displacement of wood from one frame to other and also the motion of the wood is not linear. Hence for object segmentation and recognition, spectral and spatial features must be incorporated with temporal features. Notice that due to dynamic nature of our application we cannot construct background model. As a matter of fact the background is dynamic with water waves and wood moving with the same speed. A framework is proposed in next section, which uses the spectral and spatial features for detection and segmentation and temporal features for tracking the objects in the video for counting the number of fallen tree, branches or stem depending on their appearance in the videos. 3 Proposed Methodology The detection of wood in river contains two steps: image segmentation and recognition of wood. The outdoor environments have constraints of sudden appearance/disappearance of sunshine. This fact is shown in Figure 2. First row represents the original images from the video of flood. The presence of bridge (top left corner of images), moving branches of tree before camera (right middle portions of images) and the shadows of surrounding trees over the river are evident from these images. The proposed methodology for detection of wood in river composed of two major parts: 1) detection and recognition of wood in river, 2) separation of wood and water waves by tracking them in consecutive sequence of images, with the architecture as presented in Figure 1. The following two subsections describe the proposed methodology in details.

Unsupervised Video Analysis for Counting of Wood in River during Floods 581 Frame 1 Frame 2 Frame 3 Temporal difference (df1) Temporal difference (df2) Intensity Mask Gradient Mask Intensity Mask Gradient Mask (MI) (MG) (MI) (MG) Resulting contours Resulting contours MI MG df1 MI MG df2 Find barycenter of center of mass Find barycenter of center of mass Resumed Image Fig. 1. Outline of proposed methodology for detection and segmentation of wood in the river water 3.1 Detection and Recognition of Wood The automatic detection of wood begins from automatic segmentation of image. The flow chart in Figure 1 shows that each frame is treated for two segmentation processes. One is named as intensity mask (MI) and other is gradient mask (MG). They are the result of images segmentation based on intensity histogram thresholding and edge-based gradient technique respectively. IntensityMask(MI). Gray-level histograms of image intensity are calculated for every incoming frame. Histogram thresholding is among the most popular techniques for segmenting gray-level images and several strategies have been proposed to implement it [12], [13]. In fact, peaks and valleys of the 1D brightness histograms can be easily identified, respectively, with objects and backgrounds of gray-level images. In the absence of sun shine the water in the river and wood has a difference intensity levels. But, the intensity of water waves and wood resemble one another both in gray level and in color RGB values. This fact is shown in Figure 2. The Fisher linear discriminate technique is used for histogram thresholding. This technique produces very good segmentation of images in the absence of sunshine. In Figure 2, the first two images in second row are the results of our algorithm in the presence of sunlight. The last two images in second row are the results of intensity based segmentation in the absence of sunshine that shows the efficiency of this technique Figure 2. Gradient Mask (MG). The spectral analysis, as described above, is working well in the absence of sunshine. In the presence of sunshine the shadows of surrounding trees and building over the river make the segmentation based on the histogram thresholding very difficult. Therefore it is necessary to integrate spatial features of the image with spectral features to avail meaningful

582 I. Ali and L. Tougne Fig. 2. The representation of various steps involved in the segmentation, images in first row represent original images of moving wood in water, the images in the second row are intensity masks, the images in the third row show gradient mask of corresponding images and resulting combinations of all segmentations are shown in the last row segmentation. By this it means that wood must be separated from the water. The branches and debris moving under the shadows of the surrounding trees cannot be separated from each other. So for this reason, segmentation by detecting the edges among regions is applied with intensity histogram thresholding. This approach has been extensively investigated for gray-level images [12], [13]. Algorithms have also been proposed for the detection of discontinuities within color images [14]. This technique gives the image segmentation based on spatial features. The resulting image is named as gradient mask (MG) in the Figure 1. The resulting images of this method are shown in third row of Figure 2. Temporal Difference (df). The image segmentation is done by two different methods. The histogram thresholding technique based on the spectral analysis separates the wood from water in the absence of sunshine but fails to detect woods under the shadows of surrounding trees in the presence of sunshine. The gradient analysis separates the objects in motions from the rest of the scene. As in our case, both water waves and moving wood have strong gradients, resulting image contains both of them. The advantage of using gradient mask is that it detects the objects under the shadows of surrounding trees and buildings. The wood and water waves can only be separated from one another by virtue of their

Unsupervised Video Analysis for Counting of Wood in River during Floods 583 existence in the consecutive frames of video. The majority of water waves that are dispersed in two consecutive frames are automatically suppressed by taking such inter-frame differences. The Resulting Combination. The intersection of the spectral segmentation based intensity mask, the edge based gradient mask and temporal inter-frame difference are combined in a manner to give a resulting image. This image is a binary image that represent the detected wood along with some water waves. The combination images are shown in the last row of Figure 2. Fig. 3. The moving fallen tree as original video, the combination image showing the detected contours of tree 3.2 The Separation of Wood and Water Waves Here the main goal is to detect and count the number of fallen trees and debris in the flood that passes through the river. The water waves in the flood, in the absence of sunshine, resemble the wood. So, for counting the number of fallen trees and debris, the decision cannot be made on single image segmentation. The water waves and wood forming contours must be tracked in the consecutive frames of video. First constraints of tracking the wood is that the floating fallen trees are not having the same length from one frame of video to another. The water waves in the flood often submerge the wood. Secondly, the movements of fallen trees are not linear and also the water waves exist for longer duration in the videos, therefore, sometimes detected as wood in the many consecutive frames. So to avoid loosing the counts of the wood it is important to find some mechanism than minimizes the false detection.the method of counting the fallen trees is explained in this section. Figure 3 shows the fallen tree in the river with corresponding combination image. The Barycentre of Mass Centers. The fallen trees in the river have many branches and appear in the video as different closed contours as shown in Figure 3. To cope with first constraint the multiple contours of the same object must be grouped together to avoid false detection. Every resulting contour has some area and center of mass. So the centers of mass of the contours are grouped on the basis of closeness of them in the image to give barycenter of mass centers. These barycenters of mass centers are stored in the summery image.

584 I. Ali and L. Tougne Counting the Number of Woods. In order to count the number of fallen trees, bushes, stems of trees, roots and debris that are passing through the river, we propose to represent the presence of barycenters in a summary image. The barycenter of the object (wave or wood) that is present in the consecutive two frames make a pair of barycenters in the summery image and a trace is formed on the summery image. If the object is wood then these centers of masses must be present continuously from left to the right of the screen (as motion of river water is from left to right). This means that if the object is not totally submerged in the water it will be present in more than four continuous frames. So the wood is detected and counted on this basis.(see an example of such image in Figure 4). Fig. 4. Example of resumed image 4 Experimental Results A monitoring system has been set up on the river Ain France. The videos of the flood during recent years are recorded. The number of fallen trees, bushes, branches and roots of trees are counted manually by Geographers. The results are qualitatively evaluated by visual inspection. The quantitative evaluation is computed as the true positive, false negative and false positive of the wood detected in the videos. Figure 5 shows a glimpse of some difficult situations. The first scenario presents two very small wood pieces moving at the same time. These two pieces are segmented and counted as two different objects. The second one shows that the detection is done even if there is shadow. In addition to qualitative evaluation, Table 1 shows the quantitative evaluation in terms of wood pieces actually present and counted as wood, the number of wood pieces that are not counted by our algorithm and the number of waves that are detected as wood pieces. The separation of wood and water waves depends on the presence of wood in the consecutive frames. The parameters are tested for different type of situations and different length of wooden objects. To count the wood pieces present in the videos the number of continuous frames are optimized to five. Geographers obtain the ground truth through visual inspection. They have manually gone through 5400 frames to derive the reported detection rate.

Unsupervised Video Analysis for Counting of Wood in River during Floods 585 Fig. 5. The segmentation of wood on sample frames captured from different challenging scenarios at different time intervals in the absence and presence of sun light. Odd rows: Samples frames captured. Even rows: Corresponding segmentation results. The algorithm is applied to seven videos of flood. Total duration of seven videos is thirty-six minutes. The number of wood detected is clearly higher percentage than the number of false detection of waves as wood. The brightness of waves are very close to wood pieces and have strong gradient, moreover the water waves last for more than five frames in some cases. If the water waves are continuously present in five frames false detection occurs but the percentage of such false detection is not very important. Moreover, the wood pieces sometimes appear in some number of consecutive frames and disappear for one or two frames. Such type of wood pieces cannot be detected. The detection rate is nearly 98% while successful counting rate is 90%. The number of detected wood (Nd), number of non-detected wood (Npd), number of water waves detected as wood (Nw) of the seven videos of total duration of thirty six minutes are summarized in Table 1. The numbers of non-detected wood (Npd) are those wooden objects that appeared in the videos for less than five and more than two consecutive frames. The results are shown in Figure 6, which clearly indicate that the algorithm count the wooden objects in difficult scenario with high success rate.

586 I. Ali and L. Tougne Table 1. Quantitative evaluation of proposed algorithm in terms of number of true detection Nd, number of non-detected wood Npd and number of waves detected as wood Nw Total frames Duration Nd Npd Nw (min) (%) (%) (%) Video 1 650 4 00 95 5 6 Video 2 900 5 23 91 9 13 Video 3 860 5 36 81 19 19 Video 4 750 5 11 90 10 7 Video 5 550 4 02 76 24 2 Video 6 800 5 52 93 7 14 Video 7 880 6 05 91 9 19 Total 5390 36 05 90 10 15 Fig. 6. Results of counting the number of wood in videos, white bars represents number of true wood pieces, number of wood non-detected are represented by black bars, number of waves detected as wood are represented by grey bars 5 Concluding Remarks In this paper, the problem of automated monitoring based on video surveillance in highly dynamic environment of river has been discussed. The nature of problem is such that a background model can not be created. There is a need of an algorithm that detects the wood by using different features of images. In particular this paper has addressed the two fundamental issues: 1) unsupervised segmentation of wood in river 2) the method to count the number of wooden material in river during floods. The first issue has been addressed by using the spectral features of images with spatial features. The two types of features help in great deal in unsupervised segmentation of wood and water waves in the river from rest of the water. As the water waves and wooden objects both are present in the segmented image, the separation of wood from water waves need tracking the wooden objects in the consecutive frames. The fallen tree or bushes can only be detected if some part of it remains above the water level in the river. If the wood submerges in some frames and appear in next frame then such wooden

Unsupervised Video Analysis for Counting of Wood in River during Floods 587 objects cannot be detected. Moreover, during heavy cloudy environment the water waves resemble the wood in color. The water waves during flood stays longer time, produces false detection of them as wood. The experimental results indicate that the proposed algorithm detect and count the number of wood with reasonably good percentage. References 1. Eng, H.L., Wang, J., Wah, A.H.K., Yau, W.: Robust human detection within a highly dynamic aquatic environment in real time. IEEE Tran. on Image Processing 15, 1583 1600 (2006) 2. Li, L., Huang, W.M., Gu, I.H., Tian, Q.: Statistical modeling of complex background for foreground object detection. IEEE Trans. Image Process. 13, 1459 1472 (2004) 3. Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: realtime tracking of the human body. IEEE Trans. Pattern Anal Machine Intell. 19, 780 785 (1997) 4. Vacavant, A., Chateau, T.: Realtime head and hands tracking by monocular vision. In: IEEE International Conference on Image Processing 2005, ICIP 2005 (2005) 5. Stauffer, C., Grimson, W.: Learning patterns of activity using real-time tracking. IEEE Trans. Pattern Anal. Machine Intell. 22, 747 757 (2000) 6. Boult, T.: Frame-rate multi-body tracking for surveillance. In: DARPA Image Understanding Workshop (1998) 7. Gao, X., Boult, T., Coetzee, F., Ramesh, V.: Error analysis of background adoption. In: IEEE Conf. Computer Vision and Pattern Recognition, June 2000, pp. 503 510 (2000) 8. Mittal, A., Paragios, N.: Motion-based background subtraction using adaptive kernel density estimation. In: CVPR, pp. 302 309 (2004) 9. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artificial Intelligence 17, 185 203 (1981) 10. Iketani, A., Nagai, A., Kuno, Y., Shirai, Y.: Deteching persons on changing background. In: Int. Conf. Pattern Recognition, vol. 1, pp. 74 76 (1998) 11. Wixson, L.: Detecting salient motion by accumulating directionary-consistent flow. IEEE Tran. Pattern Anal. Machine Intell. 774 780(22) (August 2000) 12. Fu, K., Mui, J.: A survey on image segmentation. Pattern Recognition 13, 3 16 (1981) 13. Rosenfeld, A., Kak, A.: Digital picture processing, 2nd edn., vol. 2. Academic Press, New York (1982) 14. Zhao, A.: Robust histogram-based object tracking in image sequences. Digital Image Computing Techniques and Applications, 45 52 (2008)