Context analysis: sky, water and motion

Transcription

1 Context analysis: sky, water and motion Solmaz Javanbakhti Svitlana Zinger Peter H.N. de With Eindhoven University of Technology Electrical Engineering department P.O. Box 513, 5600 MB Eindhoven, The Netherlands Abstract Interpreting the events present in the video is a complex task, and the same gesture or motion can be understood in several ways depending on the context of the event and/or the scene. Therefore the context of the scene can contribute to the semantic understanding of the video. In this paper, we present our research on context analysis on video sequences. By context analysis we mean not only determining the general conditions such as daytime or nighttime, indoor or outdoor environments, but also region labeling [1] and motion analysis of the scene. This paper reports on our research results on sky and water labeling and on motion analysis for determining the context. Later, this can be extended with regions such as roads, greenery, buildings, etc. Experiments based on the above detection techniques show that we achieve results comparable with other state-of-the-art techniques for sky and water detection, although in our case the color information is poor. To evaluate results, we use the Coverability Rate (CR) which measures how much of the true sky or water is detected by the algorithm. The obtained average of CR for water detection is about 96.6% and for sky detection it is about 98%. 1 Introduction Content analysis of digital images and video sequences is widely used, with applications ranging from high-level image understanding and semantic-driven image- and video retrieval, to pixel-level applications like object recognition and local picture quality improvement. Sky is one of the important visual cues, which appears frequently in video and images [2]. Sky detection provides the context information for further image analysis, and it helps to extract information about weather and illumination conditions [3]. Accurate identification of the sky area also enhances advanced object recognition algorithms [3]. Due to sky variations, systems which restrict themselves to blue sky detection have limited practical value [3]. From [2] we learn that the color of the sky changes gradually from dark blue in the zenith to a bright, sometimes almost white, blue near the horizon. In this case, we can model the values in three color channels along straight lines from zenith to horizon as three one-dimensional functions for further analysis. Sky detection in [2] is based on calculating an initial sky belief map, followed by a selection of connected areas based on texture and color analysis and the fitting degree to a two-dimensional (2D) model [2]. An alternative approach is based on the analysis of color, position and shape properties of color-homogeneous spatially connected regions [3]. Another research direction for outdoor environment analysis is the localization of water regions. This analysis involves several influencing factors, like day/night time, reflection at the water surface, relative size of the water region, wave state and possible The research of this paper is part of the ITEA 2 ViCoMo research project.

2 occurrence of material at the water surface [4]. Matthies et al. [5] developed a color image classifier based on a mixture of Gaussians to exploit mean and standard deviation of brightness and saturation, and they trained this classifier on water regions in the RGB color space. In our research, we also perform context analysis based on motion, which can be used to annotate roads and to restrict the computationally heavy search for moving object to the areas where the motion is detected. Most motion analysis techniques are based on comparing the current video frame with a previous frame or with the background [6]. In [7], the motion is detected by comparing neighboring frames when a video stream is recorded in real-time. In this paper, we present and evaluate our approaches to sky and water detection when the color information is poor. The sky detection algorithm [2] that we apply has two phases: (1) training phase which defines the color model, texture properties at multi-resolution and vertical position, (2) detecting phase which adapts the color model, vertical position and calculates texture properties. The water detection algorithm that we present in this paper also consists of two parts: (1) graph-based image segmentation, which generates initial regions, and (2) SVM-based region recognition. Normalized RGB color information is used as a feature for SVM, and we evaluate the entropy of pixels as an additional metric. We also consider the location of pixels, but this feature reduces the flexibility. To analyze motion, we develop a heat map, which is a 2D histogram indicating main regions of motion activity [8], and identify the direction of the movement in the scene using optical flow [8]. The structure of this paper will start by presenting water, sky and motion detection algorithms. Sections 3 will show the evaluation method which was used in this work then the different results of this work. 2 Description of context detection algorithms 2.1 Sky detection algorithm Our sky detection algorithm is based on [9]. It assumes that sky regions are smooth and are found around the top of the image. An initial sky probability is calculated based on color, texture and vertical position, and the features of high probability areas are used to compute a final sky probability. The applied algorithm has clearly lower false detection/rejection rates compared to state-of-the-art algorithms. The improvements are primarily due to an extensive multi-scale texture analysis, adaptive thresholds and a spatially-adaptive color model. Next to position and color features, the key assumption of our system is that sky has a smooth texture and shows limited luminance and chrominance gradients, which are different in the horizontal and vertical directions. Using predefined settings for the color, vertical position, texture, and horizontal and vertical gradients, an initial sky probability is calculated for each image pixel by [9]: P sky = P color P position P texture Q sky, (1) where P color is computed by a three-dimensional Gaussian function in the YUV color space, having a fixed variance, and centered at a predetermined color. Parameter P position is defined by a function that emphasizes the upper parts of the image. For the texture probability P texture, a multi-scale analysis is performed on the image, using an analysis window of 5 5 pixels. Furthermore, we compute a confidence metric Q sky, that prevents small sky-blue objects from being accepted as sky, in images where large areas with high sky probability are absent [9]. 2.2 Water detection algorithm Finding water regions in the image provides a useful context for image understanding. For example, to improve the robustness of ship detection, it can be helpful to locate

3 the water region in the image. The idea is that the detected ship can be confirmed if its major part is included within the water region. Our system basically consists of three components. First, each image is segmented in terms of the color uniformity of a region. In order to choose a proper segmentation algorithm, several popular algorithms, including mean shift, graph cut, normalized cut and graph-based algorithms, are investigated and compared. The graph-based method is proven to be suited for our application due to both its real-time processing capabilities and sufficient accuracy. Let us now explain the first component - segmentation - of our water detection algorithm. The whole image is treated as a graph {V, E}. Each pixel is regarded as a node v i V, and for each adjacent pixel v i,j, there is an edge v i,j E. The weight of the edge is the Euclidean distance of the two nodes in color space, specified by: W i,j = (R i R j ) 2 + (G i G j ) 2 + (B i B j ) 2. (2) Here, two kinds of weights are defined for regions: inner-region weight and inter-region weight. the inner-region weight is defined as the maximum edge weight within a region, and the inter-region weight is specified as the minimum edge weight that connects two regions. The basic idea behind this graph-based method as discussed in [10] is that pixels within one region are closer in feature space (color space in this case) than pixels from different regions. The difference between inter-region weight and inner-region weight is not a fixed number. Instead, there are high variance regions and smooth regions, so that the image cannot be segmented simply according to the absolute edge weight. However, there should always be significant weight changes if we go from one region to another. Such a change depends also on the size of a region. We assume that a larger region has a bigger tolerance while a smaller region can hardly have a large variance. Or if a region is large, it typically incorporates more neighboring pixels. In this algorithm, the Euclidean distance in RGB space is used as a metric. However, for this application, when there is strong disturbance in the river, usually caused by the movement of ships, the river tends to be broken into different regions. Instead, the desired result is to segment the whole river as one region. The idea is to reduce the distance in gray scale and increase the distance in color scale. As an extreme, we could segment mainly according to its color. This is because one object usually has one color but its appearance may change because of the light, shadow and reflection conditions. Moreover, usually the color change caused by the environment does not have a high saturation. Therefore, we explored the normalized RGB color to segment the image. This leads to a weight W i,j = ( Ri R ) 2 j + L i L j ( Gi G ) 2 ( j Bi + B ) 2 j, (3) L i L j L i L j where L i is the brightness, L i = max{r i, G i, B i } in pixel i. It can be seen that the normalized RGB is less sensitive to the brightness, which is good for detecting river and mush regions. However, the sand bank and ships sometimes also merge with the river and sky. Therefore, we combine the normalized difference with the brightness as the weight of edge. This results in the final definition of the weight, giving W i,j = (1 α) ( Ri R ) 2 j + L i L j ( Gi G ) 2 ( j Bi + B ) 2 j + α. L i L j, (4) L i L j L i L j where α is used to adjust the ratio of brightness and normalized RGB. Graph-based segmentation can be performed with fast processing and its result is still acceptable. This explains why we finally adopt a graph-based segmentation technique. In the second processing stage of water detection, visual features of each region, such

4 as the color feature (RGB) and the texture feature (entropy) are extracted and analyzed. It is possible to sample the region and calculate the mean saturation in an HSV histogram to distinguish the regions. However, this method may not robust. Therefore we explore the entropy as a feature, which is defined by E = p log p, where p is the probability in the local histogram or occurrence matrix. Here we use 5 5 neighborhoods to calculate the entropy. Generally, the sky is a smooth area with a low entropy, mush has a large entropy, while water has a moderate entropy. We have found that the entropy of the pixels does not achieve much improvement in segmentation. For this reason, the final model only uses RGB as input vectors. Finally in the third stage, a Support Vector Machine (SVM) classifier is used to recognize the water region. The classifier is trained off-line, based on the image samples captured at the same harbor in different weather conditions. 2.3 Motion detection algorithm Context analysis based on motion is another goal of this work which can be used to annotate roads and to restrict the computationally heavy search for moving objects to the areas where the motion is detected. To analyze and visualized the motion, we apply the concept of a heat map. A heat map is a 2D histogram indicating main regions of motion activity [8]. We also identify the direction of the movement in the scene using optical flow [8]. Our approach is based on motion variation in regions of interest. First, a motion intensity heat map is computed. The motion heat map represents hot and cold areas on the basis of motion intensities. The hot areas are the zones of the scene where the motion is high. The cold areas are regions of the scene where the motion intensities are low. This histogram can be designed from the accumulation of binary blobs of moving objects, which are extracted following the background subtraction method [8]. The obtained heat map can be used as a mask to define regions of interest for a following step of the processing algorithm, such as change detection. The use of the heat map image improves the quality of the results and reduces processing time which is an important factor for real-time applications. We track a moving object over the succeeding frames using the optical flow technique. For this tracking, we have employed the Kanade-Lucas-Tomasi (KLT) feature tracker [8]. After matching an object between frames, the result is a set of vectors which indicate the direction of that moving object. Optical flow is an approximation of the local image motion, based upon local derivatives in a given sequence of images. Thus, the computation of differential optical flow is essentially a two-step procedure [11]. First, it involves a measurement of the spatio-temporal intensity derivatives, which is equivalent to measuring the velocities normal to the local intensity structures. Second, it includes the integration of normal velocities into full velocities, for example, either locally via a least squares calculation, or globally via a regularization. Assume I(x, y, t) is the center pixel in a n n neighborhood and moves by δx, δy in time δt to I(x + δx, y + δy, t + δt). Since I(x, y, t) and I(x + δx, y + δy, t + δt) are the images of the same point (and therefore identical) we conclude: I(x, y, t) = I(x + δx, y + δy, t + δt). (5) After performing a Taylor series expansion of I(x, y, z) in Equation (5) on n n n 3D block, we obtain a 3D motion constraint which is used in the KLT algorithm, specified by: I x V x + I y V y + I z V z = I t, (6) where I x,i y, I z and I t are the 3D intensity derivatives in an n n n neighborhood centered at a voxel (x, y, z), so that I x = δi δx, I y = δi δy, I t = δi δt. (7)

5 A constant velocity (V x, V y, V z ) in that neighborhood is solved by: v = [A T W 2 A] 1 A T WB, (8) where the diagonal elements of W are the N = n n n 3D Gaussian coefficients. The N rows of A consist of I x, I y and I z for each (x, y, z) position and the N rows of B consist of the I t values for those (x, y, z) positions, see [11] for further details. The previous considerations lead to the following matrices: A = [ I(x 1, y 1, z 1 ),..., I(x N, y N, z N )], W = diag[w (x 1, y 1, z 1 ),..., W (x N, y N, z N )], B = (I t (x 1, y 1, z 1 ),..., I t (x N, y N, z N )). (9) 3 Experimental results To express the performance of our evaluated results, we use the Coverability Rate (CR), which measures how much of the true sky or water is detected by the algorithm [3]. This rate is computed by O GT CR(O, GT ) =, (10) GT where Ground Truth (GT) is manually annotated water or sky, and O is the area detected as sky or water. For quality evaluation, we have manually annotated sky on 17 images and water on 15 images. Let us first discuss the sky detection results. Fig. 1 (left) shows the original image of our data set which is one frame of a video sequence. As can be observed, this data set does not provide sufficient information in terms of color. Fig. 1 (middle) visualizes the probability map of the sky region which is achieved by the sky detection algorithm on the original image, and Fig. 1 (right) shows the result of applying a threshold on the found probabilities shown on Fig. 1 (middle). For sky detection, the average CR is about 98%. For water detection, we use RGB colors as a feature. We omitted the pixel location as a feature, because this feature reduces the flexibility. Hence, the final model only uses normalized RGB as input vectors. In the region recognition stage, SVM is used to classify the regions using the normalized RGB feature. Fig. 2 shows the original image and the obtained detection results of the detection algorithm. Fig. 3 shows an alternative case, where a ship is entering the camera scene. In both cases the water region is correctly found and the obtained average of CR for water detection is about 96.6%. For motion analysis, we have applied background subtraction on our video sequence to compute a heat map. The threshold that we impose on the gray scale values during background subtraction is 120. The resulting heat map corresponding with our video sequence expressed in gray values, is shown in Fig. 4 (left). We have applied KLT technique to pairs of smoothed neighboring frames of our data set, after which velocity vectors of each pixel are computed. If velocity vectors are zero, we ignore them to achieve better visualization. The result of the motion detection is shown in Fig. 4 (rigth). At the right side of the Figure, we show the direction of the moving object. 4 Conclusions In this paper, we have presented our research on context analysis for video sequences. The purpose of a context detection system is to provide additional background in-

6 Figure 1: Original image (left). Probability of the sky region (middle). Result of the sky detection after thresholding (right). Figure 2: Original image (left). Result of water detection algorithm (right). Figure 3: Original water vision image with ship (left). Result of water detection algorithm (right).

7 Figure 4: Result of gray scale heat map on video of a moving ship ( The white area indicates region with highest movement) (left). Result of optic flow KLT on a moving ship, arrow shows the direction of ship s movement(right). formation to the already existing foreground object detection, to facilitate further interpretation about event classification in the scene. This view point is shared in the research program of the European ViCoMo project. Our research concentrates on sky and water labeling and on motion analysis for determining contextual information of the scene. For sky detection, we have adopted a detection algorithm from earlier work, based on a probability map that jointly uses a color model, texture properties at multi-resolution and the vertical position [9]. We have designed a supplementary water detection algorithm, combining segmentation and region recognition. Water segmentation is carried out at the first stage to increase the robustness of detection and to decrease the calculation complexity in region recognition. Besides this, the usage of the normalized color space and also an additional parameter measuring the intensity difference of two neighboring pixels are introduced to the graph-based segmentation, in order to improve the algorithm with respect to light reflections at the water surface and sub region scattering. In the region recognition stage, color is analyzed, and SVM is used to classify the regions using the RGB pixel values. To analyze motion, we compute a heat map. This information provides context for identifying regions in the scene where the motion occurs and it helps in fast searching of moving objects. As a result, less video processing is required when this context is used. We use a method for motion detection that is sensitive to velocity and direction. This method is based on optical flow and the KLT tracker. To evaluate the results, we have computed the Coverability Rate (CR). The obtained CR average for water detection is about 96.6% and for sky detection it is approximately 98%. The advantage of this work is reusing the several detectors in one system to achieve parallel context analysis. It will be interesting to explore our concept with another real object detection such as group detection to estimate the abnormality of the scene events.

8 References [1] J. Fan, Y. Gao, H. Luo, Multi-Level Annotation of Natural Scenes Using Dominant Image Components and Semantic Concepts, Multimedia 04: Proceedings of the 12th annual ACM international conference on Multimedia, New York, USA, October 2004 [2] B. Zafarifar and P. H. N. de With, Adaptive Modeling of Sky for Video Processing and Coding Applications, WIC, 2006 [3] F. Schmitt, L. Priese, Sky Detection in CSC-segmented Color Images, In fourth International Conference on Computer Vision Theory and Applications (VISAPP), Lisboa, Portugal, 2009 [4] M. Iqbal, O. Morel, F. Meriaudeau, A survey on outdoor water hazard detection, Communication Technology and Systems (ICTS), Indonesia, 2009 [5] L. Matthies, P. Bellutta and M. McHenry, Detecting water hazards for autonomous offroad navigation, In Proceedings of SPIE Conference 5083: Unmanned Ground Vehicle Technology V, 2003 [6] A. Kirillov, Motion Detection Algorithms, http : // video/motion D etection.aspx., last access 20 Jan 2011 [7] M. AzamQsman, A. Zawawi Talib, T. Kian Lam, W. Poh Lee, M. Sabudin Vehicle monitoring system using motion detection and character recognition algorithms for USM campus, Proceedings of the third IMT-GT Regional Conference on Mathematics, Statistics and Applications Universiti Sains Malaysia, Dec 2007 [8] N. Ihaddadene, C. Djeraba, Real-time Crowd Motion Analysis, 19th International Conference on Pattern Recognition (ICPR 2008) , Tampa, Florida - USA, December, [9] B. Zafarifar, P. H. N. de With, Blue Sky Detection for Content-based Television Picture Quality Enhancement, IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, [10] S. Liu, J. Han, Camera-based water region detection in harbor monitoring, personal comunication, [11] Tutorial on Optical Flow, University of Manchester, last access 14 Jan 2011.