Indoor Outdoor Image Classification

Transcription

1 Indoor Outdoor Image Classification Bachelor Thesis Fei Guo Department of Computer Science, ETH Zurich Advisor: Lukas Bossard Supervisor: Prof. Dr. Luc van Gool August 18, 2011

2

3 Abstract Because of the increasing need for automatic photo organization, the indoor outdoor image classification as a very basic aspect of it is going to be investigated in this work. We are going to use color, texture, shape and metadata features and their combinations which could be directly extracted and inferred from images to help identify if an image shows an indoor or outdoor scene. Furthermore, we are going to divide the images into subblocks in different ways and evaluate the classification on each of them. In addition, the images will be rescaled to different resolutions and at each resolution the classification is performed. For classification, the K-Nearest Neighbor and Support Vector Machine are used. Through a series of experiments we found that the low-level feature combination of color, texture, spatial envelope gave the accuracy 78.17% and camera metadata alone achieved 79.33%.

4

5 Acknowledgements I would like to take this opportunity to sincerely thank my advisor Lukas Bossard for his patience, his help of building the image data set, labeling the ground truth and also for his weekly discussion with me. And of course, I must thank my parents who gave me financial support for the whole bachelor study in Switzerland.

6

7 Contents 1 Introduction 1 2 Background Features Color Histogram Texture Features Spatial Envelope Metadata Classification K-nearest Neighbor (KNN) Support Vector Machine (SVM) Related Work Experiments and Results Image Data Set Experiment Protocol Experiments with Single Image Features Experiments with Feature Combinations Experiments Integrated with Metadata Experiments with Different Image Dimensions Experiments with Different Division Strategies Experiments with Two Stage Support Vector Machine Conclusion 19 I

8 CONTENTS II

9 List of Figures 2.1 Two-level Wavelet Pyramid Structure Two-level Wavelet Decomposition Distribution of Flash Distribution of Exposure Time [sec] Distribution of Aperture Value Distribution of ISO Speed Rating KNN Classification SVM Separating Hyperplane Ambiguous Image Examples Histogram of Image Resolutions Color Feature and Texture Feature x3 vs. 4x Different Feature Extraction Strategies Illustration of the two-stage classification approach III

10 LIST OF FIGURES IV

11 List of Tables 3.1 Results of Single Image Features Percentage of images which have Metadata Results of Feature Combinations Results of Single Features with Metadata Results of Feature Combinations with Metadata Comparison between Normal SVM and Two Stage SVM V

12 LIST OF TABLES VI

13 Chapter 1 Introduction Nowadays, there are a lot photos being taken by different photographers at different places all over the world. Images are accumulated more and more, which raises the need for automatic photo organization. We put this work in the context of attribute based image classification. The exact content of an image must not be understood by us, e.g. a person eating in a kitchen or a car on a street. Instead, we focus on indoor/outdoor classification which can be used for organizing photo collections. Although a lot of work for image classification have been done before, we are going to select some interesting features from different works and use combinations of them. In this work, most of the relevant features are taken under consideration. Color feature can be extracted by means of histogram. Texture feature are collected by discrete wavelet decomposion. Structure of scenes will be estimated by spatial envelope. EXIF information consists of different tags, e.g. flash, exposure time, aperture value and ISO speed rating, which can be easily fetched from images. We will exploit single features, feature combinations, feature extraction strategies and different image resolutions to train different classifiers and see how these different factors affect the success rate of the classification. 1

14 CHAPTER 1. INTRODUCTION 2

15 Chapter 2 Background In general, it is very common for indoor/outdoor classification to divide images into subblocks and extract different image features from each subblock instead of the whole image. Afterwards the feature vectors for each subblock are going to be concatenated to one feature vector. In the end, the feature vectors of all images of the training set are used for training a classifier. The classifier s performance is then evaluated on the test set. 2.1 Features Many different features can be extracted from an image. In this work, we selected some interesting and important features which help us to distinguish between indoor and outdoor images Color Histogram Color histograms describe how the colors of an image are distributed. We compute the histogram of each color channel as color feature on a region of the image. According to the Benchmark for image classification [7], the Ohta color space is used for computing the color features in our case. It is a linear transformation of the RGB space. Its color channels are used by Szummer and Picard in [10] as: I1 = R + G + B I2 = R B I3 = R 2G + B In this work, we used 16-bin histograms for each channel as a feature, with in total 48 dimensions Texture Features With the texture features we are able to know the information about how pixel intensities spatially vary. The texture features are computed by means of two level wavelet decomposition as described by Serrano et al. [9]. The decomposition is performed on the I1 channel of the Ohta color space the with Daubechies 4-tap filters, where low-pass filter is: h(n) = [ ] 3

16 CHAPTER 2. BACKGROUND and the high-pass filter is: g(n) = [ ] According to the two-level wavelet pyramid structure described in their work, after the first level decomposition, all the coefficients are used for computing texture features except for the upper left one. The second decomposition is performed on the upper left part. Then, the texture features are obtained by first filtering Figure 2.1: Two-level Wavelet Pyramid Structure the low-frequency coefficients c 5 (see Figure 2.1) using the Laplacian filter and then use the measure of the sub-band energy for all wavelet coefficients which is defined as: e k = 1 MN M i=1 j=1 N c k (i, j) 2, k = 2, 3,... 4K, where M and N are the image dimensions of the coefficient c k and K is the number of decomposition levels (in this case K = 2) Spatial Envelope Besides color and texture, also the structure of an image consisting of the spatial properties of the scene, is also an important factor to be considered for image classification. In the work of Oliva and Torralba [6], they used a very low dimensional representation of the scene which is called spatial envelope for recognition of real world scenes. The spatial envelope captures the holistic property of an image. Hence, segmentation and processing of individual objects and regions does not need to be taken under consideration here anymore. The authors defined five different perceptual properties (Naturalness, Openness, Roughness, Expansion and Ruggedness) and designed features which can be used to characterize different scene categories. If an image has high degrees of Naturalness, Ruggedness and Expansion, it probably describes a natural landscape. Therefore, we can then consider the spatial envelope of an image as an important factor for indoor/outdoor classification Metadata EXIF data saved by the camera does not depend on the image content and provides information on how the images were taken. We could take advantages of such data to improve the performance of indoor/outdoor classification. Out of the variety of EXIF fields, only the tags which expose some condition with the indoor/outdoor class need to be considered for classification. It is worth noticing that not all images in our data set contain all of the considered data fields. Hence, we set the corresponding data to zero, when they are not present. 4

17 CHAPTER 2. BACKGROUND (a) Second Level DWT of Indoor Image (b) Second Level DWT of Outdoor Image Figure 2.2: Two-level Wavelet Decomposition 5

18 CHAPTER 2. BACKGROUND Sometimes when we want to take a picture of a person in a bar or a club, which is usually dark inside, we may want to fire the flash. Hence, the camera s flash is more often used, when taking an image inside as it is brighter outside. As shown in Table 2.3, we can see the percentages of flash firing in different scenes are different Indoor Outdoor 0.8 Percentage Flash off (0) or Flash on (1) Figure 2.3: Distribution of Flash Exposure time is defined as the time duration from opening to closing the shutter. It could be very large if we take a picture of a star orbit. It could also be very short when a clear picture of a fast moving object is going to be taken. The larger the value of exposure is, the longer the lights will be captured, which is suitable for the situation with bad illumination. However, small exposure time is perfect for a brighter scene. Figure 2.4 shows that images with exposure times of less than sec. are more likely to be of outdoor scenes. Aperture values illustrates the amount of the light that can be captured in a fixed amount of time. The bigger the aperture is, the more light can reach the sensor if the exposure time is constant. On the contrary, the smaller it is, the less will be absorbed. Artificial light is not as bright as sunlight. In this case, aperture values are considered to be useful because in general they get smaller when people take photos outside and bigger inside. See Figure 2.5. ISO speed rating describes the film speed, i.e. how much light the sensor needs to produce a picture. Higher ISO value allows people shooting pictures in a lower light or with small aperture value. But there would be some noise generated on the images. Thus, higher ISO value is suitable for the scene low of illumination which are likely to be indoor scenes. 2.2 Classification Classification is the problem of identifying the groups to which new individual observations belong. A classifier should be able to learn this mapping, based on existing samples, which are already labeled. In our work, we focus on binary classification. 6

19 CHAPTER 2. BACKGROUND P(ET indoor) P(ET outdoor) Figure 2.4: Distribution of Exposure Time [sec] P(AV indoor) P(AV outdoor) Figure 2.5: Distribution of Aperture Value K-nearest Neighbor (KNN) K-nearest neighbor algorithm is one of the simplest method in the field of machine learning. The idea of this classifier is that the training samples of the same category are grouped together in the multidimensional feature space. If most of the K nearest neighbors of a test sample belong to a single category, this test sample can also be included in this category. K can be determined by the user in the classification phase and usually it is an odd number which will not give any ties. As we can see in Figure 2.7 [1], the green circle can be 7

20 CHAPTER 2. BACKGROUND P(ISO indoor) P(ISO outdoor) Figure 2.6: Distribution of ISO Speed Rating classified as group of triangles if k = 3, since there are 2 triangles and only 1 square. If k = 5, the circle will then be classified as group of squares, since 3 squares win over 2 triangles. Figure 2.7: KNN Classification Support Vector Machine (SVM) Support vector machine is another very popular method in machine learning theory. A (p 1)-dimensional hyperplane is used to separate all p-dimensional data vectors into different groups. This is called a linear SVM. Assume we have some training data points D which have the form: 8 D = {(x i, y i ) x i R p, y i { 1, 1}} n i=1,

21 CHAPTER 2. BACKGROUND where x i is the p-dimensional vector and y i describes to which class the vector x i belongs. The hyperplane that separates the vectors can be expressed as: w x b = 0, where b is the bias value and w is normal vector of the hyperplane which can be calculated as: w = i α i y i x i where α is the Lagrange multiplier. There are a lot of hyperplanes available to be chosen. The best choice is the one whose distance or margin to the nearest data vector on each side is maximized, as we can see from the Figure 2.8 [2]. 1 w Figure 2.8: SVM Separating Hyperplane 2.3 Related Work Many researchers have worked on this interesting topic to detect whether an image was taken indoor or outdoor. Hence, many different techniques are proposed and evaluated. Most of them gave significant results by measuring different kinds of features. They extracted different image features by means of basic computer vision methods such as wavelet decomposition and used single feature or feature combinations to train classifiers. These classifiers were then used perform the final classification. First of all, color and texture are the most commonly used features. Szummer and Picard [10] used histograms of Ohta and RGB color space for extracting color features and it turns out that Ohta color space performs better than RGB, while in the work of Serrano et al. [9] the LST color space is used. Ohta and LST color spaces are almost the same, except for a scale factor. Serrano et al. [9] used 16 bin histograms instead of 32 bins in [10] to reduce the dimensionality by one half. On the other hand, the texture features are obtained from a two-level wavelet decomposition in [9] which gives better results than from multi-resolution simultaneous autoregressive model (MSAR) in [10]. 9

22 CHAPTER 2. BACKGROUND Secondly, edge analysis has also been used for indoor/outdoor classification. Payne and Singh [8] proposed that indoor images have more synthetic objects, therefore they have a greater proportion of edges that are straight compared to outdoor images. They used edge straightness as a feature and applied this method in a real-time system. Since most of the features which are directly extracted from image content are very extensively researched, exploiting EXIF metadata to classify indoor/outdoor images becomes a new perspective to improve the classification accuracy. In the work of Boutell and Luo [3] they took advantages of subject distance, exposure time, flash fired, aperture value and also combination of them. Moreover, they also added the metadata to content based cues which gives the highest accuracy. A number of classification techniques are exploited, such as K-Nearest Neighbors (KNN) [8] [10], support vector machine (SVM) [8] [9], neural network [11] and baysian network structure learning [4]. Szummer and Picard [10] used intersection norm for KNN instead of Euclidean norm for measuring the distance between histograms. But K-Nearest Neighbors is slow when the data sets are very large and feature dimension is high. Also, the optimal value of k is difficult to determine. However, SVM only needs to store the support vectors. Thus, its memory consumption is smaller than KNN method, but SVM has been criticized [5] for being too large to be used in a practical system with limited memory. 10

23 Chapter 3 Experiments and Results In our experiments, all low-level features (color, texture, spatial envelope) are extracted first and then we use every single feature as a classification criterion as in the work of Payne and Singh [7]. In a second batch of experiments, different feature combinations are used to train the classifier. Furthermore, among all the previous classification results the best combination is chosen to be combined with the metadata and this combination is used as a new classification criterion. Some other factors (different image division strategies and different resolutions) can be considered to affect the performance of the classification as well. Thus, a series of diverse experiments have been carried out in order to see how much impact these factors on the classification performance. 3.1 Image Data Set For the purpose of diversity, generalization and high accuracy, an image data set which consists of 5000 geotagged images from was exploited. The ground truth was manually labeled. There certainly exist images which we think their indoor-outdoor distinction is unclear, for example, a very close view of an object as shown in Figure 3.1(a) and Figure 3.1(b) or really ambiguous image as in Figure 3.1(c). Such images were omitted from the data set, which leaves us with 4525 images, in which 1395 images are indoor and 3130 images are outdoor. There is a class prior of 1 3 in our dataset. To let the result be more convincing, for evaluating the performance, we removed the class prior and the amounts of indoor and outdoor training images was balanced, i.e. 50% indoor images and 50% outdoor images. The scenes of the images range from indoor such as, living rooms, kitchens, concerts, clubs etc. to outdoor such as, sunset, lake, appearance of building, forest etc. 3.2 Experiment Protocol Before classification process begins, the whole data set is split into 3 parts: training set, validation set and test set. The test set is randomly selected with 300 indoor images and 300 outdoor images. The rest of the data set are for training and validation which contains 2190 images. The test set is totally independent from the training/validation set. If this would not be the case, it would lead to the situation that data from test set also contributes to the training process which could increase the accuracy implicitly. It is also worth noticing that the test, training and validation set were fixed for all experiments. Otherwise, the results would not be consistent and not comparable. Furthermore, some feature vectors are of higher order of magnitude and with higher dimension, such as color feature vectors. Some of them are of smaller one and with lower dimension, 11

24 C HAPTER 3. E XPERIMENTS AND R ESULTS (a) Example 1 (b) Example 2 (c) Example 3 Figure 3.1: Ambiguous Image Examples such as the texture feature vectors. It is useless if we concatenate them directly together, because the features with higher order of magnitude and higher dimension will dominate the performance. Therefore, we need to normalize feature vectors before combining them using the formula as follows 0 v = 12 v µv, σv

25 CHAPTER 3. EXPERIMENTS AND RESULTS where µ v is mean value of the old data and σ v is standard deviation of the old data. Memory consumption was measured by the size of the training data in the classification phase for the KNN classifier and the size of support vectors for the SVM classifier. 3.3 Experiments with Single Image Features We extracted color, texture, spatial envelope and camera meta data from images and used these features to train SVM and KNN classifiers. As shown in Figure 3.2, the dimensions of the images are various, the properties are not comparable when the images are of different scales. Therefore we need to rescale all the images to the same resolution and then compare the properties. In this experiment, we normalized all Number of Images Image Resolution Figure 3.2: Histogram of Image Resolutions images to maximal 512 pixels on the dimension and also divided the images into 4x4 subblocks for feature extraction. It would be interesting to be noted that the values of meta data are unique no matter how the images are rescaled and divided, i.e. values of meta data and image size are independent. We can see from Table 3.1, spatial envelope performed better than other low-level features for both classifiers. Unlike color or texture feature, spatial envelope described the holistic property of an image, it gave more accurate information of the images. The precision of outdoor scenes is better than indoor scenes for spatial envelope. This is because the properties (Naturalness, Openness, Roughness, Expansion and Ruggedness) described in the work of Oliva and Torralba [6] contribute more to outdoor scenes. The image which possess less degree of such properties would probably be labeled as indoor. Therefore, the outdoor scenes are better classified. Also, the metadata alone performed best among other low-level features, even better than spatial envelope. The accuracy is not so good as stated 91.51% in the work [3] on our dataset. This is because the number of images who contained meta data shown in Table 3.2 is not the same in both works and not every image in our data set contain all types of meta data. 13

26 CHAPTER 3. EXPERIMENTS AND RESULTS Feature Classifier Indoor Outdoor Accuracy Color KNN SVM Texture KNN SVM Spatial Envelope KNN SVM Metadata KNN SVM Table 3.1: Results of Single Image Features Meta data Flash Exposure Time Aperture Value ISO Speed Rating All 4 meta data Percentage 92.11% 91.87% 53.81% 76.66% 42.30% Table 3.2: Percentage of images which have Metadata 3.4 Experiments with Feature Combinations We may be also concerned about the performance of the feature combination. The results are shown in Feature Classifier Accuracy Color KNN % 8.83% 2. Texture SVM % 1.17% 1. Color KNN % 3.33% 2. Spatial Envelope SVM % -3.82% 1. Texture KNN % -0.34% 2. Spatial Envelope SVM % -0.32% 1. Color, 2. Texture KNN % 17.50% 4.00% 3. Spatial Envelope SVM % 6.00% 1.18% Table 3.3: Results of Feature Combinations the Table 3.3. i (i = 1,2,3) means the accuracy increment corresponding to the i-th component. When features were combined and the dimensionality increased, the performance would then be improved. Thus, color-texture performed better than single color or texture feature. Color-texture-spatial envelope performed better than color-texture, color-spatial envelope and texture-spatial envelope. We have also noticed that the performance of single texture feature was not very great. However, after combining with other low-level features, the performance was significantly improved. 14

27 3.5 Experiments Integrated with Metadata CHAPTER 3. EXPERIMENTS AND RESULTS The meta data alone had the best results among all other low-level features. We are also interested in the performance of low-level features after integrating with meta data. In general, the performance was improved for most features. But for spatial envelope with meta data when classified by KNN classifier, there is a slight reduction on the performance as shown in Table 3.4. For feature combinations as shown in Table 3.5, the Feature + Metadata Classifier Accuracy Color KNN % SVM % Texture KNN % SVM % Spatial Envelope KNN % SVM % Table 3.4: Results of Single Features with Metadata performance were also improved for most cases. But the combination of color and spatial envelope became the best combination when classified by KNN, although there was a slight accuracy reduction. For SVM classifier, color-texture-spatial envelope still performed best. Feature + Meta data Classifier Accuracy Color-Texture KNN % SVM % Color- KNN % Spatial Envelope SVM % Texture- KNN % Spatial Envelope SVM % Color-Texture- KNN % Spatial Envelope SVM % Table 3.5: Results of Feature Combinations with Metadata 3.6 Experiments with Different Image Dimensions Since the dimensions of the images are various, it is not sufficient that we normalized all images only to one particular size. The time consumption was increasing while the resolution of images increases, since the images got bigger, it took longer to read and extract features from images. Hence, we expected that we can train and evaluate the performance with a relatively low image dimension without much influence on the final performance. We then normalized all images to different resolutions according to the maximal value of height and width which are 128, 256, 512, 1024, 2048 to see if the results are significantly different due to different resolutions. For spatial envelope, the code the authors provided in [6] resized all images to 15

28 CHAPTER 3. EXPERIMENTS AND RESULTS Color Features Texture Features 0.72 KNN SVM 0.72 KNN SVM Accuracy 0.64 Accuracy Image resolution Image resolution Figure 3.3: Color Feature and Texture Feature resolution 256x256 and camera metadata are independent on image dimensions, therefore we only evaluated color and texture features. As we can see from Figure 3.3, the accuracies don not change very much while the image dimension decreases. Especially, for color feature there is a slight increase at resolution 256. Hence, it would be a benefit to reduce image to a smaller dimension. 3.7 Experiments with Different Division Strategies Different parts of images give different semantic information. The small square which contains the black Figure 3.4: 3x3 vs. 4x4 point shown in Figure 3.4 contributes to the upper part when we use 4x4 division strategy. However, if the 3x3 division strategy is exploited, this part contributes to the middle part. The information it contains would influence the final prediction of the classifiers according to different spatial positions. We have carried out experiments, which divided images into 3x3 subblocks, 4x4 subblocks, 3 columns and 3 rows individually. 16

29 CHAPTER 3. EXPERIMENTS AND RESULTS For each strategy, we compute the feature vector for the whole image by concatenating the feature vectors of each block. We would like to investigate how spatial positions could influence the high-level decision. Color Features Texture Features 0.72 KNN SVM 0.72 KNN SVM Accuracy 0.64 Accuracy by 3 4 by 4 column wise row wise Different Feature Extraction Strategies 3 by 3 4 by 4 column wise row wise Different Feature Extraction Strategies Figure 3.5: Different Feature Extraction Strategies As we can see from Figure 3.5, the 4x4 strategy gave the best results for color features when classified by KNN, while column wise gave the best when classified by SVM. For texture features, column wise strategy was the best for KNN classifier and 3x3 was the best for SVM. 3.8 Experiments with Two Stage Support Vector Machine Figure 3.6: Illustration of the two-stage classification approach A two stage support vector machine approach was also employed as a comparison to the work of Serrano et al. [9]. The first stage trains the SVM with image features and we compute the distance between feature vectors and separating hyperplane denoted by f(x), which can be calculated as, f(x) = w x + b, where x is the feature vector, b is the bias value. In the second stage, a new SVM will be trained using the distance values f(x) obtained from the first stage 17

30 CHAPTER 3. EXPERIMENTS AND RESULTS and it will produce final classification results. The whole procedure is illustrated in Figure 3.6. In our experiments, for each block we gathered the color and texture feature vectors of all images separately. In the training process, we trained the first SVM of each block and gained the f(x i ) values, where i indicates the block number. Then for each image, we summed f(x i ) together and used this sum f(x i ) as training i data for the second SVM. Normal SVM Two Stage SVM Accuracy Memory Consumption MB MB Time Consumption s 12154s Table 3.6: Comparison between Normal SVM and Two Stage SVM As we can see from Table 3.6, compared to the method which concatenated color texture vectors together, the two stage vector machine achieves a better performance but it occupied 3 times the memory and took 10 times longer compared to the normal SVM. Compared to the work of Serrano et.al in [9], the performance was not so good as theirs on our dataset. Since we used the same technique as used in [9], we could conclude that the images from our dataset are more difficult to be identified as indoor or outdoor. 18

31 Chapter 4 Conclusion After these step by step experiments, we have shown that we can gain the high-level image property (indoor/outdoor) by integrating camera meta data with low-level image properties color, texture and shape. Additionally, we also considered some other factors such as, subblock division strategy and different resolutions of images, which could play important roles in this classification problem. We have seen that for single features, the spatial envelope performed best among all low-level features and camera meta data performed even better than spatial envelope. The feature combination with color, texture and spatial envelope had better accuracy than other combinations. After integrated with meta data, in some cases the accuracies decreased and color spatial envelope outperformed over other features. When the resolutions of images increased, the classification accuracy did not have significant growing tendency. Therefore, for the purpose of efficiency, we could reduce images to a relatively small dimension without much influence on the accuracy. With the experiments with two level support vector machine, we found that the performance on our data set is not so good compared to other image data set. For future work, we may need to build a common data set for evaluation of different methods, otherwise we cannot compare the performance from different works. And we believe that if more ground truth for more diverse image scenes are provided and more advanced classifiers are used, better accuracy would be achieved. 19

32 CHAPTER 4. CONCLUSION 20

33 Bibliography [1] neighbor algorithm. [2] vector machine. [3] Matthew R. Boutell and Jiebo Luo. Photo classification by integrating image content and camera metadata. In ICPR (4), pages , [4] Michael J. Kane and Andreas E. Savakis. Bayesian network structure learning and inference in indoor vs. outdoor image classification. In ICPR (2), pages , [5] Lei Zhang Mingjing. Boosting image orientation detection with indoor vs. outdoor classification, [6] Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42: , [7] Andrew Payne and Sameer Singh. A benchmark for indoor/outdoor scene classification. In ICAPR (2), pages , [8] Andrew Payne and Sameer Singh. Indoor vs. outdoor scene classification in digital photographs. Pattern Recognition, 38(10): , [9] Navid Serrano, Andreas E. Savakis, and Jiebo Luo. A computationally efficient approach to indoor/outdoor scene classification. In ICPR (4), pages 146, [10] Martin Szummer and Rosalind W. Picard. Indoor-outdoor image classification. In CAIVD, pages 42 51, [11] Li Tao, Yeong-Hwa Kim, and Yeong-Taeg Kim. An efficient neural network based indoor-outdoor scene classification algorithm. In Consumer Electronics (ICCE), 2010 Digest of Technical Papers International Conference on, pages , jan