Figure 1 shows unstructured data when plotted on the co-ordinate axis

Transcription

1 7th International Conference on Computational Intelligence, Communication Systems and Networks (CICSyN) Key Frame Extraction and Foreground Modelling Using K-Means Clustering Azra Nasreen Kaushik Roy Kunal Roy Global Academy of Technology, Shobha G Abstract - In this paper, we have proposed and implemented a novel robust key frame extraction and foreground isolation method using k-means clustering and mean squared error method for variable frame rate videos. We also isolated foreground objects in the video whilst eliminating the noise generated in the recording. The flickering of the frames caused as a result of variable frame rate in a recorded video is reduced by a considerable degree using this method. Also, the k-means clustering is performed on Apache s hadoop infrastructure to make the results of the computation faster. We have implemented this method and obtained results to be clear enough to extract meaningful detail from the frames. The results of the method have been compared to similar results obtained using well- known techniques such as the Gaussian Mixture Model and have been shown to be better. Keywords Background Subtraction; Key Frame Extraction; Foreground modeling ; Hadoop; Distributed Computing I. INTRODUCTION In today s world, image processing concepts such as foreground extraction and key frame extraction are used extensively to reduce computational burden on post processing of the video. Post processing of the video can mean to extract features of objects of interest, for example face recognition or perhaps, for surveillance in cameras or perhaps, for analysis in detection of traffic violations, to name a few. In any case one problem that many image processing techniques face today is variable frame rates and large amounts of background noise that is created during recording of the video being analyzed. The variable frame rate and noise hinders key frame extraction by resulting in frames that have no discernible foreground movement. Therefore, in this paper we propose a method to extract key frames from a variable frame rate video and also isolate foreground objects using a combination of k- means clustering and mean squared error method, k- means clustering being performed on Apache s hadoop configuration [1][2][3]. In this method, first the background is modeled using a running average of all the frames. Statistically, the rate of averaging to decide the rate at which a workable model of the background in the video scene is obtained is determined [4]. Second, foreground objects contained in the video are obtained by using background subtraction. Next, the obtained foreground objects frame are color quantized using k- means clustering of the frames. It was found that frames that don t possess any significant motion and can be of no value in the post processing phase are complete blackened out after clustering. Such frames are filtered and sorted by a measure of black in the frame and obtain the most black frame. Finally, we do a mean squared error comparison of all the frames with the most black frame and extract the corresponding colored frame if the value returned is greater than a threshold [5]. A value of 0.0 means that the frames are similar. Thus only frames with significant foreground movement are extracted resulting in a method of key frame extraction which is computationally simple and highly time efficient. This paper contains seven sections. The first section introduces the topic and the proposed method. The second section provides some background that can aid with the understanding of the proposed method. The third section explains about how the background is modeled for background subtraction. The fourth section explains how the background subtraction phase is carried out and also how the clustering on a MapReduce setup is accomplished. The fifth section describes how finally the blank frames or black frames are filtered out using the mean squared error similarity checker. The sixth section illustrates the results obtained and compares them to similar results obtained using the well-known Gaussian Mixture Model. The seventh section presents the conclusion of the paper along with some insight into how we plan to further our research. Finally, references that have greatly helped us carry out the research have been listed. II. BACKGROUND A. Background modeling It refers to modeling of the background contained in the video scene without all the foreground objects. This background can later be used for extracting foreground objects of interest using techniques such as background /15 $ IEEE DOI /CICSyN

2 subtraction. Background modeling can be carried out using various methods. Some of the more popular methods include the running average method where frames are averaged to obtain a background of the scene dynamically, the fuzzy running average method, Morphological background modeling, etc. [6][7] B.K-means clustering K-means clustering refers to the clustering or grouping of dataset of point s representative of some data into the defined number of clusters. The grouping, as the name suggests is done by calculating the measure of the average of all the points representing the data when mapped into a co-ordinate system. The dataset is collected using statistical methods and once the desired clusters are obtained, the characteristics that define a particular cluster can be used to further process the meaning of the clustered data and also, draw logical co-relation between the data-points contained in the dataset. Figure 2: Clustered Data. Figure 2 shows the results of clustering. After grouping into clusters in this manner meaningful common characteristics can be identified among data points and processed. C. Mean squared error method Mean squared error is used to measure the average of the squares of the errors of an estimator or in simple terms, the difference between the entity that is estimating and what is estimated. The square root of the mean squared error is equivalent to what is known as the standard deviation in statistics. Figure 1: Unstructered data. Figure 1 shows unstructured data when plotted on the co-ordinate axis The above equation (1) represents the mean square error of a set of points. mn are the total number of points and the bracket represents the squares of the difference of the data points to the mean. All these differences are added to get the mean squared error. D. Key frame extraction When a video is recorded and frames are extracted. Upon careful analysis it is revealed that many of these frames are redundant frames in that, they contain no significant movement of objects in the scene. During the post processing phase of these frames for further analysis depending on the application, processing of the same type of frame repetitively introduces a sizeable amount of overhead that can be avoided by filtering out such frames. E. Foreground isolation Foreground objects in a scene are those objects that move against a background modeled in the scene. This background can be modeled using techniques already

3 discussed. To obtain foreground objects as defined, each frame of a video needs to be subtracted from the modeled background thus, leaving only objects that are in motion against that background and blacking out the rest of the frame. III. MODELING THE BACKGROUND USING RUNNING AVERAGE METHOD In this paper, this method has been used to model the background of the recorded video. First, the frames of the video are extracted and averaged as they are extracted. The above equation (2) represents the running average method. F n is the new or destination frame. F 0 is the old frame or source frame and, r is a regulator value which regulates the rate at which the foreground objects are deleted from the background in the scene. Figure 3: Foreground objects. Figure 3 shows the foreground image of objects moving in the background. In the above figure the foreground is that of a vehicle and people on two wheelers at a cross roads. F. MapReduce on the Hadoop Distributed File System MapReduce is a software paradigm that operates on the data that is stored on the Hadoop distributed cluster framework, basically targeted at making the processing of large amounts of data easier. Hadoop provides distributed storage which does away with much of the complexities of storing huge amounts of data. But since it is so scattered, we need an effective tool to navigate the data, process the data and aggregate the results. The distributions are commonly referred to as nodes. MapReduce provides this mechanism of collating node results. A master node holds a copy of all tasks (distributed across the framework) and the resources needed to perform these tasks. The other nodes, or the agent nodes of the master node (called the name node) have multiple copies of the data (relating to tasks and resources) to allow easy recovery from a malfunction or failure. The name node, however, is the single point of failure of the framework. There is no recovery mechanism if the name node fails. The unstructured data is read (typically from a file) by the mapper and is reshuffled to form <key,value> pairs of data points. The mapper then proceeds to assort these data points in some order of significance or by imposing some other such structure of meaning to make it easy for the reducer to process. The Reduce function aggregates the data points along the single line for the repeated occurrences, resulting in an output. Therefore by using a high value of r, non-static background objects are also considered when modeling the background. Thus only foreground objects that move for longer periods of time are averaged out of the modeled background. Figure 4: Foreground object being averaged out. Figure 4 shows the foreground object being deleted from the background using the running average method

4 for the reducer to concurrently re-calculate the centroids upon clustering. Cluster1 and Cluster2 is grouped as per the Euclidean distance between the centroids and the pixel points as shown in equation (3) and (4). The new centroids after grouping the pixel point is calculated by averaging the intensities of all pixel points belonging to or grouped under that cluster. Figure 5: Modeled background. Figure 5 shows the background modeled after sustained averaging for prolonged periods of time. IV. BACKGROUND SUBSTRACTION AND CLUSTERING USING MAP REDUCE After modeling the background, the frames of the video are subtracted from the background to isolate the foreground objects. However, there is noise present in the resulting foreground noise. To eliminate this noise to some extent, a bilateral filter is applied to the frame. This helps blur the noise and thus make the foreground objects more clear. After obtaining and de-noising the foreground mask, color quantization using k-means clustering is performed. This process reduces the number of distinct colors in the image and thus completely eliminates background noise. First, the width and height of the foreground mask image is obtained. Second, the image is converted from the RGB color space to the LAB color space to make use of the Euclidean distance between colors for easier implementation of the k-means clustering algorithm. Next, we reshape the frame of M x N pixels and 3 channels into a feature vector to convert the image into a two-dimensional array representation for clustering. Finally, we cluster using 2 clusters, and predict which quantized color the original color in the foreground mask is going to be. This is done by determining which centroid out of the two clusters, the input pixel is closest to. The clustering of all these pixels is carried out using the MapReduce implementation of k-means. The Mapper first chooses two initial colors as centroids to get started. Next, by comparing the Euclidean distance from each of the remaining colors to the two initial colors in consideration and in doing so, choosing the color that is closer, it proceeds to align the newly chosen color with the closest cluster. This data is replicated in the HDFS nodes to allow V. MEAN SQUARED ERROR SIMILARITY CHECK AND KEYFRAME EXTRACTION After clustering of the foreground masks, the frames that cause flicker as a result of the frame rate appear as completely black frames. All such frames are sorted according to the number of black pixels in them and a mean squared error comparison of the foreground mask frames and the black frames is done. If dissimilar it is considered a key frame. To carry out the mean squared error comparison, we begin by converting the frames or images from 8-bit unsigned integers to floating point. Next the pixel intensities of the two images are subtracted and squared. This difference is summed for all pairs of images to calculate the mean squared error. If the value returned by the mean squared error comparison is close to 0.0 the images are considered to be similar. Thus when the foreground masks are found dissimilar to the black frame, the value returned by the mean square error function was significantly greater than zero. VI. RESULTS This experiment was carried out with a MPEG-4 format and an AVI format video with 3 channel images. The frame rate of the video was 10 frames per second. Also, the same was carried out with the same video running at a frame rate of 25 frames per second. Figure 6: Foreground frames extracted using GMM

5 Figure 6 above shows foreground frames extracted using the Gaussian Mixture model. As is evident, the frame rate introduces a blank frame in between two frames of recorded movement. Also, the noise is not completely filtered out and there is still exists a small amount of background noise in the frames extracted.. VII. CONCLUSION AND FUTURE WORK In this paper, a novel approach to key frame extraction and foreground isolation, making use of MapReduce, has been shown to effectively and accurately model foreground objects and record actual foreground movement for further processing of the video. Furthermore, the proposed method also performs the required task in a timely manner with the trade off in time as compared to traditional methods being compensated by the higher accuracy and clarity achieved. The proposed technique is slower by only 5 seconds but filters out all blank frames and eliminates noise at a higher degree of efficiency compared to the Gaussian Mixture Model. We are also trying to obtain faster execution of the entire model by exploiting GPU architecture or CUDA architecture. REFERENCES Figure 7: Foreground Mask frames extracted using the proposed method. As Figure 7 shows, the frame extracted using the proposed method filters out the blank frames, eliminates noise and records only actual foreground movement. Also the proposed method only takes 30 seconds more to complete the frame extraction when compared to the frames extracted using the Gaussian Mixture Model. However, this deficiency is overcome using the MapReduce implementation of the k-means Clustering algorithm to only 5 seconds. Therefore for a trade-off of only 5 seconds, considerably more accuracy and clarity is achieved. [1] Apache Hadoop. apache. org/ [2] T.White, Hadoop: The Definitive Guide. O'Reilly Media, Yahoo! Press, June 5, 2009 [3] J. Dean and S. Ghemawat. Mapreduce: Simplied data processing on large clusters. Communications of the ACM, 51(1): , 2008 [4] M. Piccardi, "Background subtraction techniques: a review," IEEEInternational Conference on Systems, Man and Cybernetics, 2004: [5] L. Bruzzone and D. Prieto, "Automatic analysis of the difference image for unsupervised change detection," IEEE Trans. Geosci.Remote Sens.,vol. 38, no. 3, pp , May 2000 [6] M.HoseynSigari N Mozayani H Reza Rpurrerza Fuzzy Running Average and Fuzzy Background Subtraction:Concepts and Application, International Journal of Computer Science andnetwork Security, 2005 [7] Sen-Ching S. Cheung and Chandrika Kamath, "Robust Techniques for Background Subtraction in Urban Traffic Video," EURASIP Journal on Applied Signal Processing, vol.2005, pp , Jan