A Study of the Relationship Between Support Vector Machine and Gabriel Graph

Size: px

Start display at page:

Download "A Study of the Relationship Between Support Vector Machine and Gabriel Graph"

Prudence Jacobs
5 years ago
Views:

1 A Study of the Relationship Between Support Vector Machine and Gabriel Graph Wan Zhang and Irwin King {wzhang, Departent of Coputer Science & Engineering The Chinese University of Hong Kong Shatin, NT., Hong Kong Abstract - In recent years, Support Vector Machine (SVM) has becoe a very dynaic and popular topic in the Neural Network counity for its abilities to perfor classification, estiation, and regression. One of the ajor tasks in the SVM algorith is to locate the discriinant boundary in classification task. It is crucial to understand various approaches to this particular task. In this paper, we survey several different ethods of finding the boundary fro different disciplines. In particular, we exaine SVM fro the statistical learning theory, the Convex hull proble fro the Coputational Geoetry s point of view, and Gabriel graph fro the Coputational Geoetry perspective to describe their theoretical connections and practical ipleentation iplications. Moreover, we ipleent these ethods and deonstrate their respective results on the classification accuracy and run tie coplexity. Lastly, we conclude with soe discussions about these three different techniques. I. Introduction Given a data set which contains the data belonging to two or ore different classes, either linearly separable or nonseparable, the proble is to find the optial separating hyperplane (Decision Boundary) to separate the data according to their class type. Support Vector Machines (SVMs) have attracted wide interest as a ean to ipleent structural risk iniization for the proble of classification and regression estiation introduced by Vapnik in the late seventies [8]. This led to a recent explosion of applications and deepening theoretical analyses, that has now established SVMs along with neural networks as one of the standard tools for achine learning and data ining. There are three key points needed to understand SVM: (1) axiizing argins, () the dual forulation, and (3) kernel functions. Most people intuitively grasp the idea that axiizing argins should help iprove generalization. Kernel functions are used to solve the probles in high-diensional space. But changing fro the prial to dual forulation is typically black agic for those unfailiar with duality theory. If the prial proble P has its dual proble, it eans the solution of one proble could be recovered fro that to the other. For exaple, Lagrangian Dual proble is forulated with the Lagrangian dual objective function which statisfies the constraints of the prial proble, and aintains any other constraints within the data set. To better understand SVMs, a geoetric interpretation fro the dual perspective along with a atheatically rigorous derivation of the ideas behind the geoetry, can be useful. Convex hull could be a siple intuitive geoetric explanation of SVM []. The convex hull of a set of points is the sallest convex set which contains all the points. It is also the fundaental construction for atheatics and coputational geoetry. It turns out that finding the nearest points of the two convex hulls is siilar to finding the separating hyperplane with axiu argin in the SVM case []. Fro [1], we know that Convex hull could be used to solve any probles, such as half space intersection, Delaunay triangulation, Voronoi diagras, etc. The Voronoi diagra and Delaunay triangulation are two of the possible representations for K-Nearest neighbor rule. The K-Nearest neighbor rule can be seen as the non-paraetric decision rule which needs no prior knowledge of the distributions [3]. Decision rules are used in any areas such as pattern recognition and database. They are used to deterine the class ebership for a point based on soe coputational easureents for the point. Despite siplicity and good perforance of K-Nearest neighbor rule, the traditional criticis of the ethod is that it needs a large storage space for the entire training data and the necessity to query the entire training set in order to ake a single ebership classification. As a result, there has been considerable interest

2 in editing the training set to reduce its size. Just like SVMs choose support vectors which are a sall part of the whole training set to find the separating hyperplane, different proxiity graphs (such as Delaunay triangulation and Gabriel graph) provide efficient geoetric apparatus for solving the proble and find the decision boundary. The Gabriel graph of a set of points is a subgraph of Delaunay triangulation for that set, which is a dual of Voronoi diagra [3]. This paper tries to bring all these concepts to a unified application doain. We use Convex hull to be a bridge which connects SVM and Gabriel graph. Both of the could be used in the classifier and have a good perforance not only on accuracy but also on tie coplexity. We also perfored experients to show the perforance the two ethods and ade the discussion of their advantages and drawbacks. In the next section, we will present the three ain concepts for coparison: (1) SVM, () Convex hull, and (3) Gabriel graph. In Section 3, we conduct a series of experients that will use the different data sets and deonstrate the perforance of the different ethods. The discussion of these results is in Section 4. Lastly, we conclude and ake soe final rearks in Section 5. II. Related Background In this section, we plan to take a survey on three different concepts for finding the separating hyperplanes in a data set. These concepts coe fro different disciplines in Coputer Science, ranging fro Coputational Geoetry to statistical learning theory. We want to show the siilar relationship aong these different concepts arising fro different disciplines. We will start with the siplest case linear separable data. Later we will see that the analysis for the general case, nonlinear-separable data, results in a very siilar prograing proble. A. Support Vector Machine Given the training data, x i, y i, i = 1,, l, y i { 1, 1}, x i R n, suppose there exists a hyperplane which could separate the positive fro the negative data set. It eans that the points x which lie on the hyperplane satisfy ω x+b = 0, where ω is noral to the hyperplane, b / ω is the perpendicular distance for the hyperplane to the origin, and ω is the Euclidean nor of ω. Define the argin is the su of the distance of the separating hyperplane to the closest positive and negative points [4]. For the linearly separable case, SVM siply finds the separating hyperplane with the largest argin. It could be forulated as a set of linear constraints for all the Fig. 1. Linear separating hyperplanes for the separable case training data points as: y i (ω x i + b) 1 0, i. (1) Fro Fig. 1 and Eq. (1), we can calculate the argin as / ω. Then we can find the separating hyperplane with the largest argin under the constraint in Eq. (1) by iniizing ω in in 1 ω () subject to y i (x ω + b) 1, i = 1,,. Let α = α 1, α,, α be the nonnegative Lagrange ultipliers, one for each inequality constraints in Eq. (1), the solution to Eq. () equals to the solution to the constrained quadratic optiization proble using the Wolfe dual theory [4] as, L P 1 α i α j y i y j x i x j i,j i subject to 0 α i 1, i α i y i = 0. α i (3) In the soft argin (nonlinear separable case) forulation of [9], the generalized optial separating hyperplane is regarded as the solution to Eq. (4) as follows, in( 1 ω + C ξ i ) (4) i=1 subject to y i (x ω+b) 1 ξ i, ξ i 0, i = 1,,, C > 0. Siilarly, the corresponding Lagrangian forulation is defined as L P = 1 ω + C i i ξ i (5) α i {y i (x i ω) + b) 1 + ξ i } i µ i ξ i.

3 The optial solution to this proble satisfies the following Karush-Kuhn-Tucker (KKT) conditions [9]: α i {y i (ω ξ i + b) 1 + ξ i } = 0, µ i ξ i = 0. So its Wolfe dual proble could be defined as in 1 α i α j y i y j x i x j i,j i α i (6) subject to 0 α i C, i α i y i = 0. For solving high diension probles, SVM aps the space of covariates X to a Hilbert space H of a higher diension (aybe infinite), and fits an optial linear classifier in H. It does so by choosing a apping function φ:r n H in such a way that φ(x) φ(y) = K(x, y) for soe known and easy to evaluate set of functions, K. Sufficient conditions for the existence of such a ap are provided by the Mercer s theore [9]. An exaple of the apping function can be described as follows. Set Q ij = y i y j K(x i, x j ), such that α y = 0, 0 α i C, i = 1,,. Due to the new apping function, the objective function is changed as follows R(α) = 1 α (Q α) α. (7) Fro a Coputational Geoetric point of view, the solution to the Convex hull proble provides a way to locate support vectors []. In the next section, we survey soe interesting properties on the coputation of Convex hull. B. Convex hull The separating plane is the hyperplane which is orthogonal to the line segent and bisects the line segent which connects the nearest points of the two convex hulls of the data sets. In Fig., the convex hull of class A(B) consists of all the points which could be written as convex cobination of the points in A(B). A convex cobination of points in A, u, is denoted by u= i A β ix i, where i A,β i 0, and i A β i=1 and the convex cobination of the points in B, v, is denoted by v= j B β jx j, where j B, β j 0, and j B β j=1. Assue U is the convex hull of A, V is the convex hull of B [7]. The proble of finding two nearest points in the convex hulls could be written as follows: in u v (8) Fig.. Two closest points of the two convex hulls deterine the separating plane such that u U, and v V. If (ω, b) is the optial solution of Eq. (3), and (u, v) is the optial solution of Eq. (8), with the fact that the axiu argin of the two sets=/ ω = u v, and ω has the sae direction with (u v), so ω = (u v), u v b = v u u v. We could introduce a new variable, σ, to Eq. (3) and then rewrite i α iy i = 0 into two constraints α i y i = σ, i A Now, we define β rewritten as follows in σ = α σ α i y i = σ. i B,, so Eq. (3) could be β β k y y k x x k σ (9) such that 0 β i 1, i A k β i = 1, j B β j = 1. When σ =, it is equivalent to k ββ ky y k x x k the following proble: in 1 β β k y y k x x k (10) such that 0 β i 1, i A k β i = 1, j B β j = 1.

4 We now define the atrix P with coluns: y 1 x 1, y x,, y x. With this, we can obtain the following equation fro Eq. (10) as β β k y y k x x k = P β. k If β satisfies the constraints, then P β = u v,where u U and v V. So Eq. (3) is equivalent to Eq. (8). Except the algebraic description for convex hull, We also could represent a convex hull with a set of facets and a set adjacency lists giving the neighbors and vertices for each facet. The boundary eleents of a facet are denoted as ridges. Each ridge signifies the adjacency of two facets. In R 3 and general position, facets are triangles and ridges are edges. A Delaunay triangulation in R d could be coputed fro a convex hull in R d+1. To deterine the Delaunay triangulation of a set of points, we can calculate it by lifting the points to a paraboloid and coputing their convex hull. The set of ridges of the lower convex hull is the Delaunay triangulation of the original points [1]. Gabriel graph can be coputed by discarding edges fro Delaunay triangulation. However, according to the tie consued, this approach to obtain the Gabriel graph is not a very attractive one when nuber of diensions is large. C. Gabriel graph In K-Nearest neighbor classification, we classify an object (point) in d-diensional space according to the doinant class aong its k-nearest neighbors fro the training data set. It is useful if we can find soe representatives fro the training set to classify new point while preserving a high accuracy. Both Voronoi diagra and Gabriel graph can be used for such purpose. The general idea is found in [3]. A Voronoi diagra is a partition of special points into regions such that each region consists of points closer to one particular node than to any other nodes. For exaple, a set of points P := p 1, p,... such that for each cell corresponding to point p i, the points q in that cell are nearer to p i than to any other point in P. In other words, the points q satisfies the following inequality: dist(q, p i ) < dist(q, p j ), p i, p j P, j i. Therefore, a new point in a Voronoi region ust be closer to the region s node than to any other nodes. So, we can assign the new point to the class represented by the region s node. Moreover, the boundaries of the Voronoi regions separating those regions whose nodes are of different class can be used as the decision boundary of the clas- Fig. 3. Gabriel graph sifier. However, it is clear that the nodes whose boundaries did not contribute to the decision boundary are redundant and can be safely deleted fro the training data. Gabriel graph of a set, S, of points has an edge between points p and q in S if and only if the diaetral sphere of p and q does not contain any other points. The resulting points fro the above process ake up of the Gabriel edited set. We shall see that decision boundary can be constructed fro those Gabriel neighbors (p and q) such that p and q are of different classes (see Fig. 3). The Gabriel edited set is always a subset of the Voronoi edited set because of the fact that a Gabriel graph of a set of points is a subgraph of Delaunay triangulation for that set. Thus, Gabriel editing which is the procedure of finding the Gabriel neighbors, reduces the size of the training set ore than Voronoi editing. Although, the resulting Gabriel editing does not preserve the original decision boundary, the changes occur ainly outside of the zones of interest. The Gabriel editing algorith can be forulated as follows: 1. Copute the Gabriel graph for the training set.. Visit each node, arking it if all its Gabriel neighbors are of the sae class as the current node. 3. Delete all arked nodes, exiting with the reaining ones as the edited training set. Clearly, the Gabriel graph can be coputed by brute force if for every potential pair of neighbors A and B, we just verify if any other point X is contained in the diaetral sphere such that L (A, X) + L (B, X) < L (A, B) where L is the square of the distance between the two points. A. Tie Coplexity III. Experients and Results Table I suarizes the theoretical analyses fro the research result of Hush and Scovel [6] and Bhattacharya [3], where n is the nuber of input points, d is the diension. Here, ɛ n is obtained through an appropriate

5 TABLE I Tie Coplexity Methods Best Average Worst Case Case Case SVM O(n) uncertain O(n 5 log n /ɛ n) Gabriel graph O(dn ) O(dn 3 ) O(dn 3 ) noralization of objective function, R, and depends on n. The Best Case for any algorith is to visit each data point once. For average case analysis with SVM, it typically requires soe knowledge of the distribution over proble instances. Moreover, it is not uncoon to see run tie estiates of to 3 reported fro experients with these types of algoriths [6]. But the bounds of iteration steps to find the optial solution could not be ade certain. The Gabriel graph algorith requires O(n ) operations to yield O(n ) pairs of Gabriel neighbors. For each such pair of points (A, B), the algorith requires O(nd) operations. Hence the overall average coplexity of the algorith is O(dn 3 ). Fro Table I, we see that Gabriel graph is ore stable in different cases (linear-separable or nonlinear-separable cases and different data sets) but has poor perforance with data having high diension. Even now we have not found an exact way to easure the iterations that the algorith will take to copute the optial solution. In other words, the result of SVM is data-sensitive, i.e., it changes with different data sets. The diension has little effect on SVM. The worst case for SVM is that the algorith of SVM drives the criterion function to the optiu solution in O( Cn4 ɛ n ) iterations. In each step it will take O(n log n ) tie to deterine whether the result satisfies the constraints for the optial solution. Although this does not happen often in the real world, it is a point to consider when ipleenting the algorith. The following epirical experients evaluate the perforance of SVM and Gabriel graph on two siple data sets: (1) Iris data set and () Spiral data set. First, we use Libsv [5], an integrated software for support vector classification, (C-SVC, nu-svc), regression (epsilon- SVR, nu-svr) and distribution estiation (one-class SVM). It supports ulti-class classification. The basic algorith is a siplification of both SMO by Platt and SVMLight by Joachis. It provides both C++ and Java source codes. Second, we ipleent the Gabriel graph [3] algorith with Matlab. We run the two algoriths under WINNT operating syste and record the result. In these experients, the SVM paraeters are set according to Table II. TABLE II SVM paraeters setting Paraeter Spiral Data Iris Data Kernel Function Radial Basis Function Dot Product Error Penalty(C) Gaa for RBF 50 Default TABLE III Experients on the Iris Data and the Spiral Data Method Size Support Gabriel of Data set Vectors Edited Set Iris Data 100 Spiral Data B. Iris Data and Spiral Data The Iris data set consists of four easureents ade on each of 150 flowers. There are three pattern classes: Virginica, Setosa, and Versicolor corresponding to three different types of Iris. In this case, the reference set consists of 150 feature vectors in 4-space each of which is assigned to one of the three above entioned classes. In these experients, we only choose two features of the two classes (Iris-setosa and Iris-versicolor) of the Iris data: petal length and petal width as the test data. Table III shows the results when the Gabriel graph algorith and SVM are applied to the Iris data. We record the nuber of support vectors and the size of Gabriel edited set. In the second exaple, we illustrate the two ethods on a two-spiral benchark proble. The training data with two classes indicated with the different labels are in a two diensional input space. (see Fig. 4 and Fig. 5)The excellent generalization perforance is clear fro the decision boundaries shown in the two ethods experient. The results are shown in Table III. IV. Discussions As described in above sections, the Convex hull could be one intuitive geoetric explanation for SVM and it also can be used to solve the Voronoi diagra proble. In fact, the edited set of Voronoi diagra is the super set of the edited set of Gabriel graph. There exist soe relationships between SVM and Gabriel graph. Observation #1 Fro the two experients, we observe that the nuber of support vectors is always saller than the size of the Gabriel graph edited

Conclusion In this paper, we have deonstrated how the SVM and Gabriel graph can be used for solving the classification proble.

6 Fig. 4. The boundary found by Gabriel graph evaluate the average case of soe SVM algoriths. Observation #3 For the siple case, we can observe that SVM and Gabriel graph could solve the proble perfectly, if we select the proper paraeter for SVM. V. Conclusion In this paper, we have deonstrated how the SVM and Gabriel graph can be used for solving the classification proble. Moreover, we have tried to show the relationships aong these two algoriths. The Gabriel graph will render a superset of the SVM results through epirical observations. We could try to iprove SVM s perforance in general by using the Gabriel graph s training data set reduction algorith. In light of this, we plan to investigate the following in the future: 1. Given the sae input data set, copare which algorith is faster and ore accurate in obtaining the optial decision boundary.. Theoretically prove that support vectors are the subset of Gabriel edited set. Acknowledgeent This research is supported in part by an Eararked Grant fro the Hong Kong s University Grants Coittee (UGC), CUHK #4407/99E. Fig. 5. The boundary found by SVM set. Suppose support vectors are indeed the subset of the Gabriel edited set, we could iprove SVM with Gabriel graph algorith as follows: 1. Map the data to soe other higher, possibly infinite, diension space, H and fit an optial linear classifier in that space.. Use the Gabriel graph algorith to reduce the size of the training data. 3. Using the SVM s optiization steps to obtain the solution to the quadratic proble and find the separating plane. As a result of applying the steps above, the reduced training data will accelerate the convergence speed of finding the optial quadratic solution. Observation # Moreover, the Gabriel graph s forulation is ore rigid in the sense that it is not paraetrizable so that there is only a single solution to Gabriel graph for a given input set. On the other hand, SVM can be paraeterized to achieve different results for a given input set. For exaple, error penalty can be odified to suit a user s requireent. Nonetheless, the Gabriel graph algorith provides a stable solution which can be used as a reference to References [1] C. Bradford Barber, David P. Dobkin, and Hannu Huhanpaa. The quickhull algorith for convex hulls. ACM Transactions on Matheatical Software, (4): , [] Kristin P. Bennett and Erin J. Bredensteiner. Duality and Geoetry in SVM Classifiers. Morgan Kaufann, San Francisco, CA, 000. [3] Binay K. Bhattacharya, Ronald S. Poulsen, and Godfried T. Toussaint. Application of Proxiity Graphs to Editing Nearest Neighbor Decision Rule. International Syposiu on Inforation Theory, Santa Monica, [4] Christopher J. C. Burges. A tutorial on support vector achines for pattern recognition. Data Mining and Knowledge Discovery, ():11 167, [5] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a Library for Support Vector Machines (Version.31). Technical report, Departent of Coputer Science and Inforation Engineering, National Taiwan University, Taipei, Taiwan, 001. [6] Don Hush and Clint Scovel. Polynoial-tie decoposition algoriths for support vector achines. Technical report, Los Alaos National Laboratory, USA, 000. [7] S.S. Keerthi, S.K. Shevade, C. Bhattacharyya, and K.R.K. Murthy. A Fast Iterative Nearest Point Algorith for Support Vector Machine Classifier Design. Technical Report TR- ISL-99-03, Intelligent Systes Lab, Dept. of Coputer Science and Autoation, Indian Institute of Science, Bangalore, India, 000. [8] V. N. Vapnik. Estiation of dependencies based on epirical Data. (in Russian), Nauka, Moscow, [9] Vladiir N. Vapnik. The nature of statistical learning theory. Springer Verlag, Heidelberg, Gerany, 1995.

Gromov-Hausdorff Distance Between Metric Graphs

Gromov-Hausdorff Distance Between Metric Graphs Groov-Hausdorff Distance Between Metric Graphs Jiwon Choi St Mark s School January, 019 Abstract In this paper we study the Groov-Hausdorff distance between two etric graphs We copute the precise value