Road Sign Visualization with Principal Component Analysis and Emergent Self-Organizing Map

Transcription

1 Road Sign Visualization with Principal Component Analysis and Emergent Self-Organizing Map H6429: Computational Intelligence, Method and Applications Assignment One report Written By Nguwi Yok Yen G J October 2006 School Of Computer Engineering

2 Contents LIST OF FIGURES 3 LIST OF TABLE 4 1. INTRODUCTION 5 2. PRINCIPAL COMPONENT ANALYSIS (PCA) 7 3. EMERGENT SELF-ORGANIZING MAP (ESOM) U-MAP (DISTANCE-BASED VISUALIZATION) P -MAP (DENSITY BASED VISUALIZATION) U* -MAP (DISTANCE AND DISTANCE BASED VISUALIZATION) OTHER VISUALIZATION EXPERIMENTAL RESULTS THE DATASET VISUALIZATION PCA FIRST PRINCIPAL COMPONENT SECOND PRINCIPAL COMPONENT THIRD PRINCIPAL COMPONENT TENTH PRINCIPAL COMPONENT FIRST VERSUS SECOND PRINCIPAL COMPONENT FIRST VERSUS TENTH PRINCIPAL COMPONENT SUMMARY ESOM ESOM WITH FIRST PRINCIPAL COMPONENT ESOM WITH FIRST AND SECOND PRINCIPAL COMPONENT ESOM WITH TEN PRINCIPAL COMPONENT CONCLUSION 30 BIBLIOGRAPHY 31

3 List of Figures Figure 1 Structures of quad and hex grid, and topologies of bounded and boundless...10 Figure 2 Chain Link with ward clustering...13 Figure 3 The corresponding chain link s U-Matrix, P-Matrix, and U* Matrix...13 Figure 4 Stop sign and give way sign...14 Figure 5 Screenshot of the dataset...15 Figure 6 First principal component of dataset...16 Figure 7 Second principal component of dataset...18 Figure 8 Third principal component of dataset...19 Figure 9 Tenth principal component of dataset...20 Figure 10 First versus second principal component...21 Figure 11 First versus tenth principal component...22 Figure 12 Stop sign s box plot of all ten principal components...23 Figure 13 Give way sign s box plot of all ten principal components...23 Figure 14 Tiled U-Map of first principal component...25 Figure 15 Tiled P-Map of first principal component...26 Figure 16 Tiled U-Map of first and second principal component...27 Figure 17 Tiled P-Map of first and second principal component...27 Figure 18 Tiled U-Map of ten principal component...28 Figure 19 U-Map of ten principal component...29 Figure 20 Tiled P-Map of ten principal component...29

4 List of Table Table 1 Summary of ten principal components... 24

5 1. Introduction Data visualization is an important research field. With lots of useful data sets and information, how do we analyze them, and in which dimension that it shows interesting results? Human observation combines with computational software tools can achieve differing views of the same visualization data. This report presents the visualization of road sign images through the use of PCA and ESOM. The observation obtained through the visualization process will be discussed. Two classes of road signs are investigated; they are namely stop sign and give way sign. The focus of this work is to show how well PCA and ESOM work hand in hand to segregate the two classes. This work attempts to study how the data shows better visualization result through the combination of PCA and ESOM rather than using either one of the technique solely. Road sign images are taken on Singapore roads by the author. The gray scaled properties of the training images results in high dimension features. The features are then processed by PCA to shrink down the dimensions and further analyzed by an ESOM tool. PCA is a useful technique that has been widely used in vast applications for reducing dimensionality, visualization, and finding patterns or clusters in high dimension data. The similarities and differences of data can be shown through the use of PCA. The maps used by most SOM applications are usually small and face significant performance degradation when come to large data set. Using small SOM is similar to k-means clustering with k equal to the number of nodes in the map. Emergent SOM is an extension of SOM. Emergent phenomena involve by defining a large number of individualism, where large means at least a few thousands. It has been demonstrated that using ESOM is a significantly different

6 process from using k-means. ESOM is a powerful tool for clustering, visualization, and classification. The next chapter presents some theoretical background of PCA. Chapter 3 discusses the brief details of ESOM. We shall then evaluates and review the test methodology and visualization results. Lastly, we draw the conclusion of this work.

7 2. Principal Component Analysis (PCA) Principal components analysis is a quantitatively rigorous method to simplify the data structure. The method generates a new set of variables, called principal components. Each principal component is a linear combination of the original variables. All the principal components are orthogonal to each other so there is no redundant information. The principal components as a whole form an orthogonal basis for the space of the data. PCA can be used to find patterns in data, and compress the data. Assuming we have a data vector X with p variables. The idea is to locate a linear combination of the variables such that to achieve large variance in new variable. The j-th principal component PCi is the linear combination of the original variables X1, X2, Xp: Under the condition of (1) Where are coefficients assigned to the original p variables for PCj. (2) The following algorithm is the general process of PCA: Data preparation The data is usually in large dimensional or large features. These data must be pre-processed before computing the various matrixes. Mean subtraction The mean for a given sample set X, is computed using the following formula.

8 (3) However, the mean only shows some centre point of the whole dataset. It is not sufficient to show the nature of the data. We have to consider how spread out the data is. Compute co-variance matrix Variance is the spread of data. Variance and standard deviation only show the relationship of data within its own dimension. Covariance matrix shows the variance between dimensions. The covariance can be computed as follows: (4) The result shows how similar or dissimilar the data are. Positive result shows that the dimensions increase together. Negative result shows that the dimensions decrease together. With covariance equals to zero, it shows that two dimensions are independent of each other. Given a matrix with n rows and n columns, and is xth dimension, the covariance matrix is given by: (5) Compute eigenvectors and eigenvalues Eigenvectors of transformation are vectors which are either left unaffected or simply multiplied by a scale factor after the transformation. The eigenvalue of an eigenvector is the scale factor by which it has been multiplied [1]. (6) Where λ is an eigenvalue of C and v is the associated eigenvector of λ. Eigenvector with the highest eigenvalue is generally the principal component of the data set. It is the most significant relationship between the data dimensions. Determine the component(s) and construct feature vector

9 Feature vector is constructed by taking the eigenvectors of interest, and forms a matrix in columns. The feature vector is then multiplied by the original dataset. (7)

10 3. Emergent Self-Organizing Map (ESOM) Self-Organizing maps (SOM), or Self-Organizing Feature Maps (SOFM), are popular tool for clustering and visualization of high dimensional data [2]. It implements an orderly mapping from a high dimensional data distribution onto a regularly low-dimensional grid of neurons. The goodness of its topological and metric relationship of the input data set is preserved. The ability of self-organizing according to neuron s neighbourhood Euclidean distance is the key feature of SOM. However, the power of self-organizing that allows the emergence of structure in data is often neglected [3]. The maps used in most SOM research are usually very small. The concept of boundless maps (eg toroids map) to avoid border effect is rarely used. These capture great interest by the Data Bionics Research Group who developed the Emergent Self-Organizing Map (ESOM) tool [4]. Emergent Self-Organizing Map may be regarded as a non-linear projection technique using neurons arranged on a map. There are mainly two types of ESOM grid structures in use: hexgrid (honeycomb like) and quadgrid (trellis like) maps. Figure 1 shows the above mentioned grid and topology. Figure 1 Structures of quad and hex grid, and topologies of bounded and boundless [5]

11 ESOM forms a low dimensional grid of high dimensional prototype vectors. The height and distances between matrices are taken into account generally for better visualization. ESOM map consists of U-Map (from U-Matrix), P-Map (from P-Matrix) and U*-Map(combines U and P map). The three maps show the floor space layout for a landscape like visualization o distance and density structure of the high dimensional data space. Structures emerge on top of the map by the cooperation of many neurons. These emerging structures are the main concept of ESOM. How the different maps of ESOM achieve visualization is described below: 3.1 U-Map (Distance-based Visualization) The U-height for each neuron ni is the average distance of ni s weight vectors to the weight vectors of its immediate neighbors N(i). The U-height uh(i) is calculated as follows: A display of all U-heights on top of the map is called a U-Matrix [6]. (8) The height value will be large in area where no or few data points reside, creating mountain ranges for cluster boundaries. The sum will be small in area of high densities.u-map is a visualization of distance measure. The local distance structure is displayed at each neuron as a height value creating a 3D landscape of high dimensional data space. 3.2 P -Map (Density based Visualization) The P-height ph(i) for a neuron ni is a measure of the density of data points in the vicinity of wi: ph(i) = {x E d(x, wj) < r >0, r R}. A display of all P-heights on top of the grid G is called a P-Matrix [7]. Distance based methods usually work well for clearly separated clusters, problems can occur with slowly changing densities and overlapping clusters. Density-based methods more directly

12 measure the density in the data space sampled at the prototype vectors. The P- matrix displays the local density measures with Pareto Density Estimation (PDE) [8]. 3.3 U* -Map (Distance and Distance based Visualization) U*-matrix combines the distance-based U-matrix and the density-based P-matrix. The U*-matrix shows significant improvement over U-matrix in dataset with clusters that are not clearly separated in the high dimensional space. 3.4 Other Visualization The Databionic ESOM tool provides several other visualization functions like Topology-based visualization, Bestmatch visualization, Component visualization and etc. Topology-based visualization shows the topographical errors, like forward projection error and backward projection error. Bestmatch visualization shows each neuron that is the bestmatch for a data point. For demonstration purpose, we adopt chain-link problem set. The chain-link data set [9] consists of 1000 points in A^3 space arranged such that they form the shape of two intertwined three-dimensional rings. The two rings can be thought of as two links of a chain with each one consisting of 500 data objects. This problem illustrates the capabilities of finding the structure of the data even for non-spherical, complex shaped and non-linearly separable clusters. Figure 2 shows the SOM disentangle cluster structures that are not linearly separable. Figure 3 shows the corresponding distance and density measures. Take U*- matrix for example, the chain link data now shows two disentangled clusters.

13 Figure 2 Chain Link with ward clustering Figure 3 The corresponding chain link s U-Matrix, P-Matrix, and U* Matrix

14 4. Experimental Results 4.1 The Dataset The efficiency of PCA and ESOM are tested using road sign images. The images are collected by the author. The dataset consists of two classes of road signs, namely stop sign and give way sign as shown in Figure 4. The images are extracted from road scene images; they are processed manually using image processing applications to make them appear differently. The processes are listed below: Apply different lighting conditions to the images. Add brightness or contrast to the images. In this way, neurons learn to recognise images under varying weather. Rotate the images from the angle of 1 degree to 30 degree. And rotate images in both clockwise and anti-clockwise directions. This allows neuron to recognise tilted road signs. Add different levels of noise to the image. Apply different colours to the background of road sign images. Apply different threshold values to the images. Add some blurring or motion effect on to the images. Sharpen or enhance the pixel values. The dataset contains 157 stop signs and 135 give way signs. They are normalized to the size of 30x30. The gray scaled properties of the image are used. The snapshot of the dataset is shown in Figure 5. Figure 4 Stop sign and give way sign

15 Figure 5 Screenshot of the dataset. 4.2 Visualization The dataset is processed by PCA first to extract ten principal components that correspond to ten dimensions. The aim is to show how many major principal components are sufficient to solve this two class problem PCA First principal component The PCA has been developed under Matlab environment. Figure 6 shows the result of first principal component of the dataset. The first box subplot shows the Quantiles of the data. It shows that the median of give way and stop signs are and respectively. Main distribution of give way sign (within 25%~75% quantiles) falls between to Main distribution of stop sign (within 25%~75% quantiles) falls between to For PC1 (Principal Component 1) function, most

16 of the data points of give away sign is higher than most of the data points of stop sign. The stop sign has a much better distribution comparing to give way sign as the standard deviation for stop sign is smaller. From the second normal subplot, the blue reference line of stop sign fits the data point much better than give way sign. The last plot shows the T-test result. The two circles are very far apart that shows that the two classes get separated very well; very significant different in statistic. Figure 6 First principal component of dataset

17 Second principal component Figure 7 shows the result of second principal component of the dataset. It shows that the median of give way and stop signs are and respectively. From the second sub-plot, it shows that ~10% of the data points for give way sign and stop sign overlap with each other. However, Main distribution of give way sign (within 25%~75% quantiles) falls between the value of to Main distribution of stop sign (within 25%~75% quantiles) falls between to Thus, these two distributions are still significantly different for PC2 (Second Principal Component). The give way sign has a much better distribution comparing to stop sign as the standard deviation for give way sign is smaller. From the second normal subplot, the blue reference line of stop sign fits is steeper than the blue reference line of give way sign. This implies that data points from stop sign are higher than the data points from give way sign. The last plot shows the T- test result. The two circles are still quite far apart that shows that the two classes get separated well but not as far apart as the first principal component result.

18 Figure 7 Second principal component of dataset

19 Third principal component Figure 8 shows the result of third principal component of the dataset. It shows that the median of give way and stop signs are and respectively. From the second sub-plot, the blue reference line of give way sign crosses over the blue reference line of stop sign at 50% quantitles. In the last T-test result, the two circles are not far apart that shows that the two classes are comparable to each others and they are inseparable. Figure 8 Third principal component of dataset

20 Tenth principal component Figure 9 shows the result of last principal component of the dataset. It shows that the median of give way and stop signs are and respectively. From the second sub-plot, it shows that most of the data points are give way sign is comparable to stop sign. In the last T-test result, the two circles overlap with each other. Thus, these two classes are comparable to each other and inseparable. Figure 9 Tenth principal component of dataset

21 First versus second principal component Figure 10 shows the relationship of first and second principal components. It can be seen that there are some red data point crossing over to the green boundary. The green data point of stop sign has wider spread. For give way sign, F1 function showed strong positive correlation to F2 function; the R square is Their fit model can be explained by the equation, F1 = F2. For stop sign, F1 function showed minor positive correlation to F2 function; the R square is quite low, Their fit model can be explained by the equation, F1 = F2. Figure 10 First versus second principal component

22 First versus tenth principal component Figure 11 shows the relationship of first and tenth principal components. The figure shows that there is no correlation for PC1 and PC10 for both give way and stop sign as both their R square values are very low. Figure 11 First versus tenth principal component Summary Figure 13 and Figure 13 give an overview of all the ten principal components for stop sign and give way sign respectively. It is significant from the plots that the first two components show larger variation comparing to the rest which vary very narrowly. They are generated using the software GhostMiner [10]. Table 1 summarizes all the related statistics figures for the ten principal components.

23 Figure 12 Stop sign s box plot of all ten principal components Figure 13 Give way sign s box plot of all ten principal components

24 Table 1 Summary of ten principal components ESOM To investigate the behavior of different principal components when apply to ESOM, we will first look at ESOM with first principal component, ESOM with first and second principal components, and lastly ESOM with all the ten principal components ESOM with first principal component As shown in Figure 14 is the training result of applying first principal component result to ESOM after 200 epochs. Yellow color shows the distribution of stop sign, red color corresponds to give way sign. It can be shown that there exist many give way sign data points that are position in the stop sign area which confuses the result. The mis-classify data points are circled in black in the figure. U-Map shows the distance measure between data points. Large U-Matrix corresponds to dark colour (blue in the earth colour scheme adopted), it

25 means that there are many data points surrounding that area. Small U-Matrix corresponds to light colour (white in the earth colour scheme adopted), it means that there are very few data points surrounding that area. Figure 15 shows the density measure, the P-Map. It works in the similar manner as in U-Map. Figure 14 Tiled U-Map of first principal component

26 Figure 15 Tiled P-Map of first principal component ESOM with first and second principal component As shown in Figure 16 is the training result of applying first and second principal component result to ESOM after 200 epochs. Yellow colour shows the distribution of stop sign, red colour corresponds to give way sign. The mis-classify data points as circled in black are significantly reduced comparing to the result of previous section. There are eight circled outliers in the figure, but this map is a tiled version of four grids so the actual outlier points are two. Figure 17 shows the density measure, the P-Map.

27 Figure 16 Tiled U-Map of first and second principal component Figure 17 Tiled P-Map of first and second principal component

28 ESOM with ten principal component As shown in Figure 18 is the training result of applying ten principal component results to ESOM after 200 epochs. Yellow colour shows the distribution of stop sign, red colour corresponds to give way sign. The result is very encouraging; it clearly shows the clusters of the two classes without a single outlier. The two classes are separated successfully with the use of ten principal components with ESOM. Figure 19 displays the original U-Map without tiled. The borderless map which tiled four copies of the grid can show better visualization because clusters stretching over the edge of the grid image can be seen connected. Figure 20 shows the density measure, the P-Map. It can be seen that most of the yellow stop sign data points are very densely on top. Figure 18 Tiled U-Map of ten principal component

29 Figure 19 U-Map of ten principal component Figure 20 Tiled P-Map of ten principal component

30 5. Conclusion This assignment presents the visualization of using PCA and ESOM. The basic theory of PCA and ESOM was described in Chapter 2 and 3. The visualization result was presented in the previous chapter. This work uses a two class road sign problem set as data which consists of a total of 292 stop sign and give way sign images. Principal components analysis is a quantitatively rigorous method to simplify the data structure. The method generates a new set of variables, called principal components. Each principal component is a linear combination of the original variables. In this work, we have seen the visualization result of PCA on its own. Also, the visualization result of applying PCA prior to ESOM was reviewed. ESOM is an extension of SOM which is significantly different from the concept of K-Means clustering. Emergent phenomena involve by defining a large number of individual, it allows the emergence of intrinsic structural features of the data space. ESOM overcomes the finite border effect of general SOM. ESOM achieve the finite but borderless map by connecting the top row of the map to the bottom row of the map, and left to right within the lattice. We have seen in the last section of Chapter 4 that the result of applying ten principal components to ESOM in tiled U-Map and P-Map. The tiled U-map shows the linking border which is hard to visualize without tiling. ESOM results demonstrate that the first and first two principal components are insufficient to segregate the two class road sign problem set. With ten principal components, U-Map and P-Map show excellent clustering picture as shown in Figure 18 and Figure 20.

31 Bibliography [1] Wikipedia, [Online]. Available: [2] T. Kohonen, The self-organizing map, Neurocomput., vol. 21, pp. 1 6, [3] Ultsch, A. & Mörchen F., ESOM-Maps: tools for clustering, visualization, and classification with Emergent SOM, Technichal Report No. 46, Dept. of Mathematics and Computer Science, University of Marburg, Germany [4] Databionics ESOM Tools. [Online]. Available: [5] Databionics Research Group, Emergent Self-Organizing Map [Online]. Available: [6] A.Ultsch, H.P.Siemon, Kohonen's Self Organizing Feature Maps for Exploratory Data Analysis, Proc. Intern. Neural Networks, Kluwer Academic Press, Paris, , [7] A.Ultsch, Maps for the Visualization of high-dimensional Data Spaces, Proc. WSOM, Kyushu, Japan, , 2006 [8] Ultsch, A.: Pareto Density Estimation: A Density Estimation for Knowledge Discovery, Baier D., Wernecke K.D. (Eds), In Innovations in Classification, Data Science, and Information Systems - Proceedings 27th Annual Conference of the German Classification Society (GfKL) 2003, Berlin, Heidelberg, Springer, pp , 2003 [9] Ultsch, A. Self-Organizing Neural Networks for Visualization and Classification. In: O. Opitz et al. (Eds). Information and Classification. Springer, Berlin, pp , [10] Fujitsu, Ghost Miner, [Online]. Available: