Blind Data Classification using Hyper-Dimensional Convex Polytopes

Transcription

1 Blin Data Classification using Hyper-Dimensional Convex Polytopes Brent T. McBrie an Gilbert L. Peterson Department of Electrical an Computer Engineering Air Force Institute of Technology 9 Hobson Way Wright-Patterson AFB, OH -776 Abstract A blin classification algorithm is presente that uses hyperimensional geometric algorithms to locate a hypothesis, in the form of a convex polytope or hyper-sphere. The convex polytope geometric moel provies a well-fitte class representation that oes not require training with instances of opposing classes. Further, the classification algorithm creates moels for as many training classes of ata as are available resulting in a hybri anomaly/signature-base classifier. A metho for hanling non-numeric ata types is explaine. Classification accuracy is enhance through the introuction of a tolerance metric, ß, which compensates for intra-class isjunctions, statistical outliers, an input noise in the training ata. The convex polytope classification algorithm is successfully teste with the voting atabase from the UCI Machine Learning Repository (Blake an Merz 998, an compare to a simpler geometric moel using hyper-spheres erive from the k-means clustering algorithm. Testing results show the convex polytope moel s tighter fit in the attribute space provies superior blin ata classification. Introuction Many popular ata classification methos, such as Artificial Neural Networks, Support Vector Machines, an Decision Trees, are not blin. This inicates that on a ecision with two or more classifications, they must be traine upon instances of the various ata classifications against which they will be teste. If they are teste against an unfamiliar class instance, the learne hypothesis woul not be able to reliably istinguish the foreign instance from the classes of the training set. A blin classification metho recognizes that a foreign instance is not a member of any of its training classes an ientifies it as an anomaly. Anomaly etection is useful when there is incomplete omain knowlege available for training, as in one sie of a two sie classification problem. This paper presents a blin classification metho using hyper-imensional geometric constructs to create a class moel without referencing other classes. Section introuces applicable geometric concepts generalize to Copyright 00, American Association for Artificial Intelligence ( All rights reserve. arbitrary imensions. Section escribes how class instance ata is formatte for use in the geometric classifier. Sections an present two ifferent hypergeometric construct-base classifiers, one using convex polytopes an the other using hyper-spheres. Section 6 covers the testing methoology use to evaluate both classifiers an section 7 presents the testing results. Applicable Geometric Concepts Central to the primary geometric classifier algorithm is the concept of a polytope. A -polytope is a close geometric construct boune by the intersection of a finite set of hyperplanes, or halfspaces, in imensions. It is the generalize form of a point, line, polygon, an polyheron in zero, one, two, an three imensions, respectively (Coxeter, 97, but it is also efine for arbitrarily higher imensions. As the number of imensions rises, a polytope s structure becomes increasingly complex an unintuitive for the human brain accustome to moeling objects in the three-imensional physical worl (see Table. Dimensions Polytope Name 0 point line polygon polyheron Example polychoron N/A -polytope N/A Table - Dimensional Progression of Polytopes A polytope is convex if a line segment between any two points on its bounary lies either within the polytope or on its bounary. A convex hull of a set, S, of points in

2 imensions is the smallest convex -polytope that encloses S (O'Rourke 998. Each vertex of this enclosing polytope is a point in S. As a two imensional analogy, imagine that a set of nails (points is riven into a table or other flat surface (plane. Next, a single rubber ban is place tightly aroun all of the nails. The polygon outline by the rubber ban is the convex hull of the points marke by the nails (see Figure a. The nails touching the rubber ban where it bens are the vertices of the polygon. Each of the polytopes rawn in the thir column of Table is a special kin of polytope calle a simplex. A -simplex is the simplest (i.e. has the smallest number of vertices, eges, an facets possible polytope that can exist in imensions. A -simplex is always convex an contains exactly + non-coplanar vertices. Thus, a - imensional convex hull cannot be compute for fewer than + points. (a -D (b -D (Lambert 998 Figure - Convex Hulls The seconary classifier makes use of the generalize circle, or hyper-sphere. A -sphere is a hyper-sphere in imensions that is efine simply by a center point an a raius. This construct is significantly easier to eal with than the convex polytope. This fact has both its avantages an isavantages, as will be explore in a later section. Both geometric classifiers evaluate separations between points in -imensional space using generalize Eucliean istance. The Eucliean istance between any two points in a continuous -space is erive from the Pythagorean Theorem (Weisstein, 999 an is given by Equation. It is common to compare istances square to avoi performing a computationally expensive square root calculation. ( pi qi q R : ist( q = ( i= Data Representation The ata on which the hyper-geometric classifier trains an tests is comprise of a set of instances. Each instance belongs to a class an possesses an orere list of attributes. Each class instance with attributes is mappe to a real-value -vector which represents a point, in a continuous -space. Point p is efine symbolically as: { p p, } R p =,, L The attribute values are mappe to the i.many classification problems involve non-numeric attributes. For example, an attribute escribing the weather coul contain values such as sunny, clouy, or rainy. Such non-numeric attribute values may be arbitrarily mappe to real numbers, but the chosen mapping must be consistently applie. This arbitrary mapping leas to equally arbitrary point coorinate assignments an cluster isjunctions within the attribute space. These isjunctions, if not ares se, can negatively affect the classification accuracy of the erive geometric class moel. A metho to hanle these isjunctions is explaine in the next section. p Convex Polytope Classifier p R Once the training instances for class C have been mappe to a set, T, of -vectors, a geometric class moel is create. If the esire geometric shape is a convex - polytope, then the convex hull of T is compute: ConvexHull( T H T For classification purposes, the convex hull of T is represente as a set of vertices, H, which are the vertices of the smallest convex -polytope that encloses T. Note that this is a blinly-create moel because it is erive solely from instances of class C. It knows nothing about the attribute values of other classes. A istinct test point, is eclare to be a match (member of class C iff it is boune by the polytope efine by H. This is etermine by computing: ConvexHull ( H { p} p D match( C D p D match( C If p is not boune by H s polytope, then D will represent a larger polytope of which p is a vertex. If p is boune by the polytope, then D H. An alternate metho, which hanles the special case of p H (i.e. p nee not be istinct, is to compare the hyper-volumes of the polytopes represente by D an H. If volume ( D > volume( H, then p oes not match the moel an is eclare anomalous with respect to class C. Spatial isjunctions of ata clusters an statistically extreme points cause by input noise or other factors can result in a convex hull moel that is too voluminous,

3 enclosing much of the attribute hyper-space that oes not rightly efine class C. This extra space translates to a poorly fitte moel that is highly susceptible to eclaring false positives. Convex Hull Tolerance Parameter: ß To compensate for isjunctions an lessen the impact of statistical outliers, a tolerance feature controlle by parameter 0 β is ae. It breaks up training points into groups of smaller convex polytopes an provies a tighter fit aroun the training ata, as follows:. Select values MIN an MAX, such that MIN < MAX.. Scale each imension of the vertices in T between MIN an MAX. All points in T then lie insie of a - hypercube with opposing extreme points at {MIN } an {MAX }. The istance square between these two extreme points: ist ({ MIN },{ MAX } = ( MAX MIN provies the upper boun on the istance square between any two points in T: p, q T : ist( q ( MAX MIN. Let G be an unirecte, unweighte graph. For each vertex in T, a a corresponing noe to G. Create an ege between each istinct pair of noes, p an q, in G where: ist ( q β ( MAX MIN. Partition G into unconnecte sets, such that in each set every noe is connecte by at least one ege an no eges cross between sets. Throw out any set with fewer than + non-coplanar points (the minimum neee to create a simplex in -space--this is where statistical outliers are iscare. The multiple convex hulls constructe aroun the partitione sets of G comprise the moel of class C. Test point p is then scale by the same factors use on T in step an is eclare a match iff it resies within any of C s convex polytope moels. With ß=, the algorithm behaves as the unmoifie version an creates a single convex polytope moel. As ß ecreases, the potential number of smaller polytopes will increase an their combine hyper-volume in the attribute space will ecrease. In testing, this leas to a lower probability of false positives at the expense of a higher probability for false negatives, or a loss of generality. At ß=0, no convex hull moels are create an all test points are rejecte. Fining the right ß value for each class moel to fit the training ata properly an achieve a goo balance between false positives an negatives requires experimentation. If instances from multiple classes are available for training, then hyper-geometric moels, or signatures, are create for each class. Test instances are compare against each of the moels an classifie as an anomaly if no matches are foun. This classifying algorithm is thus a hybri signature/anomaly etector. Classification accuracy an specificity increases with the level of training ata available. When the classifier contains signatures for more than one class, moel overlapping is possible. This is especially true when poorly iscriminating attributes are use or when a moel s ß value is too high. Multiple class matches may be acceptable in the case of non-mutually exclusive classes (i.e.: a person with ual citizenship. For mutually exclusive classes, the accuracy of overlapping matches may be improve with a tie-breaking protocol, perhaps relying upon the test point s istance from each geometric moel s center or nearest bounary. There are many freely available programs that compute convex hulls in high imensions. (Avis, Bremner, an Seiel 997 evaluate the performance of many of the most popular programs, incluing c+, PORTA, qhull, an lrs. The qhull program (Barber an Huhanpaa 00, version 00., is use with this convex polytope classifier, in part because it can compute hyper-volumes. Qhull has a time complexity of / Ο ( n, for n input points in -space (Barber, Dobkin, an Huhanpaa 996. Hyper-sphere Classifier: k-means Hyper-spheres may also be use as the geometric class moel, in lieu of convex polytopes. Using this paraigm, a class is escribe by a set, S, of k hyper-spheres. The k- means clustering algorithm partitions the training set, T, into k ifferent clusters. Each cluster has a centroi, the average of all points in the cluster. The k-means algorithm attempts to minimize the sum of square within group errors, or the sum of the istance square between each point an the centroi of its assigne cluster. k behaves as a tolerance parameter for the hypersphere classification algorithm by controlling the partitioning of T. The cluster centrois prouce by the k- means algorithm become the center points of the k hyperspheres in S. The raius of each hyper-sphere is given by the istance between the corresponing centroi an the most istant point in its cluster. Point p is boune by a hyper-sphere with center point c an raius r iff ist ( c r. A point is eclare a member of class C iff it is enclose by any of the k hyper-spheres in S. The hyper-sphere classifier uses the SimpleMeans program of the WEA java suite, version.. (Witten an Frank 00. The time complexity of the k-means algorithm is Ο (knr, for k clusters, n points, an r iterations (Wong, Chen, an Yeh 000. Testing a point for inclusion in S s k hyper-spheres takes Ο (k time. The obvious avantage the hyper-sphere moel has over a convex polytope is that its time complexity is linear, not exponential, in. Thus, a hyper-sphere can create a moel with much higher imensionality than is feasible with a convex polytope. The main avantage the convex polytope paraigm hols is that it can create much tighter-fitting moels than are possible using a hyper-

4 sphere an important requirement for goo blin classification, as testing will show. Testing Methoology Both the convex polytope an the hyper-sphere geometric classifiers are teste with the voting atabase from the UCI Machine Learning Repository (Blake an Merz, 998. It contains the voting recors of members of the 98 U.S. House of Representatives on 6 key votes. Each instance in the atabase represents an iniviual member of congress who belongs to one of two classes: Republican or Democrat. The instance attributes are the iniviual s choices on the 6 votes. Each attribute has one of three values: yea (vote for, paire for, or announce for, nay (vote against, paire against, or announce against, an unknown (vote present, vote present to avoi a conflict of interest, or i not vote. The non-numeric attribute values nay, unknown, an yea are arbitrarily mappe to real numbers -, 0, an, respectively. Due to the imensional complexity of the convex hull algorithm, the classifiers train an test with only the first seven of the 6 available attributes. The testing regimen creates a total of 6,0 blin moels of the two classes, using the two ifferent hypergeometric structures an sixteen ifferent values for their applicable tolerance parameters (ß or k. These moels are teste for matches a total of 7,00 t imes with test points from both classes. The results are isplaye in the box-whisker graphs of Figure through Figure. Each box-whisker structure isplays the maximum, r quartile, meian, st quartile, an minimum match percentages over 00 trials with a given tolerance parameter, training class, an testing class. In each trial, 9 of the training class instances are ranomly selecte for training. The testing set is comprise of the remaining of the training class (top graphs plus a ranom of the non-training class (bottom graphs. Next, the best ß values for both moels are use in a test to gauge how overall etection accuracy is affecte by the training set size, as a percentage of the whole atabase. Overall accuracy is efine as average probability of the classifier correctly ientifying a test instance. The results are given in Figure 6. Results Analysis Strong test results are represente by a high mean match percentage for the top graphs (correct matches an a low mean percentage on the bottom graphs (false positives. A narrow inner-quartile range (istance between the Q an Q bars reflects consistent/stable performance. As expecte, the convex polytope moel s tighter fit of the ata space prouces better results than that achieve with hyper-spheres. Matche (correct Matche (error Figure - Republican Moel (Convex Polytopes Matche (correct Matche (error Figure - Democrat Moel (Convex Polytopes

5 Matche (correct Matche (correct Matche (error Matche (error Figure - Republican Moel (Hyper-Spheres The best ß values for the Republican blin moel (Figure range roughly between an.0. The best ß for the Democrat moel (Figure falls at about. At these ß values both moels exhibit goo, stable classification accuracy with low incience of false positive an false negative matching errors. This is especially impressive consiering that the classifiers use less than half of the available attributes! Classification Accuracy Training Sample Size Figure 6 - Training Sample versus Detection Accuracy Figure - Democrat Moel (Hyper-Spheres The hyper-sphere moels (Figure an Figure o not fare so well. Their performance is less stable, as evience by their wier bar-whisker graphs. At every value of k the moels exhibit inferior balancing of false positive an false negative errors. Figure 6 shows that the overall classification accuracy for the convex polytope moel eclines only graually as a smaller percentage of the atabase is use for training. Note that an overall accuracy of is equivalent to classification via the flip of a coin. The classifier rops to this level when the training sample size is not large enough to form a working convex hull moel. Preliminary testing with a few other UCI Machine Learning Repository atabases suggests that the tight fit provie by a convex polytope oes not perform as well for training sets requiring more generality. For example, for each of the monks, monks, an monks atabases, which were esigne to test inuction algorithms, the convex polytope classifier achieve a blin classification accuracy of about 7. Accuracy was increase to about 8 by applying some omain knowlege in the form of two class moels using the previously escribe tiebreaking protocol. On the other han, the classifier achieve 99% accuracy on the Iris atabase. Since a blin classifier oes not, by efinition, compare an contrast attribute values between opposing classes, it is at a clear isavantage for sparse training sets (requiring greater generality, compare to more

6 traitional non-blin classifiers. This is the traeoff for being able to recognize anomalous classes. The blin hyper-geometric classifier performs best on iverse training sets whose points are well-representative of the class true attribute space topology. In other wors, a goo blin class moel can only be achieve if the class is well known. Future Possibilities The negative impact of the convex hull complexity limitation on the number of imensions may be lessene by creating multiple classifiers, each using ifferent partitions of the attribute space, an then boosting or bagging to combine the collective results. Alternate implementations of a tolerance feature can be explore to perhaps increase the generality of the hyper-geometric classifier. It may be possible to reuce the hyper-sphere moel over-inclusiveness problem by scaling own the raii of the hyper-spheres. Structures other than convex polytopes an hyperspheres may be use in a hyper-geometric classifier. A polytope create by the alpha-shapes algorithm may be especially promising. Such a polytope nee not be convex, potentially allowing for an even better spatial fit aroun training ata. Further, the algorithm alreay has a built-in tolerance value in alpha. However, the alphashapes algorithm suffers from similar imensional complexity issues as the convex hull algorithm. While an alpha-shape is possible in hyper-imensional space (Eelsbrunner an Mucke 99., the authors have yet to fin an alpha-shape tool that can excee three imensions. Conclusion To be an effective blin classifier, a geometric construct must be versatile enough to create a goo fit aroun a wie variety of training ata topologies in the attribute space. Testing results on the voting atabase show that a hyper-imensional convex polytope can offer goo blin classification, even when ealing with non-numeric attribute values or training on only a portion of the attribute space. Further testing on a wier variety of stanarize atabases from the UCI Machine Learning Repository an other sources will etermine the true range of this metho s usefulness. Acknowlegements The work on this paper was supporte (or partially supporte by the Digital Data Embeing Technologies group of the Air Force Research Laboratory, Rome Research Site, Information Directorate, Rome NY. The U.S. Government is authorize to reprouce an istribute reprints for Governmental purposes notwithstaning any copyright notation there on. The views an conclusions containe herein are those of the authors an shoul not be interprete as necessarily representing the official policies, either expresse or implie, of Air Force Research Laboratory, or the U.S. Government References Avis, D; Bremner, D., an Seiel R How Goo are Convex Hull Algorithms? ACM Symposium on Computational Geometry, 0-8. Barber, C.B., Dobkin, D.P., an Huhanpaa, H.T The Quickhull Algorithm For Convex Hulls. ACM Trans. on Mathematical Software., Barber, C.B an Huhanpaa. 00. Qhull, version Blake, C.L. & Merz, C.J UCI Repository of Machine Learning Databases. University of California, Department of Information an Computer Science, Irvine, CA. Coxeter, H. S. M. 97. Regular Polytopes, r e. New York: Dover. Eelsbrunner, H. an Mucke, E. 99. Three-imensional Alpha Shapes. ACM Transactions on Graphics, (:- 7. Lambert, Tim Convex Hull Algorithms applet. UNSW School of Computer Science an Engineering O'Rourke, J Computational Geometry in C, n e. Cambrige, Englan: Cambrige University Press. Weisstein, E Distance. Wolfram Research, CRC Press LLC. Witten I.H. an Frank E. 00. WEA, version.., Java Programs for Machine Learning University of Waikato, Hamilton, New Zealan. Wong C., Chen C., an Yeh S Means-Base Fuzzy Classifier Design, The Ninth IEEE International Conference on Fuzzy Systems, vol., pp. 8-.