Learning Polynomial Functions. by Feature Construction

Transcription

1 I Proceeings of the Eighth International Workshop on Machine Learning Chicago, Illinois, June Learning Polynomial Functions by Feature Construction Richar S. Sutton GTE Laboratories Incorporate 40 Sylvan R., Waltham MA suttongte.com Christopher J. Matheus GTE Laboratories Incorporate 40 Sylvan R., Waltham MA matheusgte.com Abstract We present a metho for learning higherorer polynomial functions from examples using linear regression an feature construction. Regression is use on a set of training instances to prouce a weight vector for a linear function over the feature set. If this hypothesis is imperfect, a new feature is constructe by forming the prouct of the two features that most eectively preict the square error of the current hypothesis. The algorithm is then repeate. In an extension to this metho, the specic pair of features to combine is selecte by measuring their joint ability to preict the hypothesis' error. 1 INTRODUCTION We present a metho for learning higher-orer polynomial functions from examples using linear regression an feature construction. The linear regression algorithm takes a training set an returns a hypothesis represente as the coecients for a linear function over the feature set. If this hypothesis is less than perfect when applie to the training set, a new feature is constructe by forming the prouct of the two features that most eectively preict the square error of the current hypothesis; an then the process is repeate. In an extension to this metho, the specic pair of features to combine is selecte by measuring their joint ability to preict the hypothesis' error. Preliminary experiments with this approach are promising an suggest some extensions. We begin this paper with a escription of the learning problem followe by a general explanation of our metho. We then escribe the algorithm an illustrate its use on two simple examples. We conclue with a short iscussion of anticipate extensions of this research. 2 THE PROBLEM The problem we are intereste in is learning polynomial functions from examples: Given a set of realvalue vectors fy; x1; :::x n g, learn the function y = f(x). This task represents a subset of the general problem of multivariate function approximation (see (Breiman et al., 1984; Frieman, 1988)). When f(x) is a rst-orer polynomial it can be represente as a linear function over the inepenent variables x: nx f(x) = w0 + w i x i i=0 We can erive the coecient vector w in this equation by oing a linear regression on the training instances. Equivalently, we coul inuce the coecients by training a simple linear network using the LMS rule, w i = x i ( ^f(x) y), where is the learning rate an ^f(x) is the network's estimate of f(x): 1 w0 x1 w1 ^f(x) :::: w n x n w i = x i ( ^f(x) y) Either of these methos will n the set of coecients that form the optimal linear approximation ^f(x) in terms of the minimum mean square error on the training instances. Unfortunately, linear functions are inaequate for representing higher-orer polynomials. To learn these we must either use a learning metho that searches a more complex hypothesis space (e.g., a multi-layere connectionist network) or change the set of features use as the basis for the linear function. The metho escribe in this paper implements the latter approach through a new form of feature construction.

2 I 3 THE METHOD As a simple example, consier the function y = 2x1 + 5x2x3. Clearly, y is not linear over the set of primitive features fx1; x2; x3g. If, however, we a a new feature x 0 = x2x3, then y can be represente as a linear function over this new basis: y = 2x1 + 5x 0. The problem with this sort of feature construction is knowing which new features to construct. For example, for a polynomial of orer with training instances escribe by n inepenent features an with c constructive operators, the carinality of the space of potentially useful new features is of orer c(n + 1). In our metho we control the complexity of feature construction with two major constraints. First, we use multiplication as our only constructive operator. For polynomial functions this single operator is sucient; its use for this problem omain can be viewe as a piece of omain knowlege. Secon, the multiplication operator we use is binary. This means we can combine only two features at a time, an must apply the operator iteratively to construct higher-orer features. For example, to construct the new feature x1x2x3 woul require two iterations of feature construction, e.g. x 0 = prouct(x1; x2) followe by x 00 = prouct(x 0 ; x3). Having just this single binary operator reuces the number of caniate new features at any given moment to n2 2. This number is still rather large to simply a all caniates, most of which woul be useless anyway. Instea, we selectively construct new features one at a time by rating all features iniviually an then forming the prouct of the pair of features with the highest ratings. In our metho, features are rate accoring to their ability to preict the square error of the hypothesis (cf., Sanger, 1991). The rational behin this way of rating features can be explaine by viewing the moel in terms of a single unit network. After the network is traine on a set of examples it will accurately preict y for some training instances but not for others. On the instances where its preiction is poor, the square error (the square of the ierence between the preicte an actual values) will be high. When the square error is high, the largest ajustments are mae by the learning rule, an the largest of these are mae in the coecients whose features have the largest magnitues at the time (recall w i = x i ( ^f(x) y)). If, after the network coecients have stabilize, a given feature x j still tens to be high when the error is high, this implies that this unit is relevant to the target function but by itself it is unable to accurately contribute to the preiction of the function's value. One way to measure the ability of a feature to preict the square error is to measure the correlation between the square error an the square of the feature's value: correlation i =E[e 2 x 2 i ] where E stans for the expecte value. This is the approach propose by Sanger (1991), who notes that this correlation is proportional to the variance of the coecients in the limit as they approach stability. 1 We have use this feature-rating metho successfully, but we have also ha a problem with it: if a new feature such as x 2 i is constructe, its correlation is typically very similar to that of its single parent x i, an therefore it is rate highly for inclusion in a further new features. Consequently, the system procees to construct more an more complex features in the same terms, contributing little new to the feature set. We have moie Sanger's iea to give it a competitive quality that iscourages the formation of features that o not improve the preiction of the square error. Instea of using the correlations between square error an square value we use the coecients from a regression of the square error over the square values: Regress(e 2 jx 2 ). The equivalent network is as shown below: 1 p0 p1 x 2 1 be 2 :::: p n x 2 n p i = x 2 i ( b e 2 e 2 ) We refer to these coecients p as \potentials" because we use them to etermine a feature's relative potential for being useful as part of a new feature. To construct a new feature we take the prouct of the two features with the highest potentials. 2 4 THE ALGORITHM The algorithm to implement our metho is written as pseuo-coe in Figure 1. After normalizing the training instances such that all features have zero mean an variance of one, a regression is performe (step 1) on the training set to learn the best set of coecients for preicting y given x. If the error (the resiual) is approximately zero, then the function has been learne an the algorithm halts; otherwise feature construction is initiate (step 2). A secon regression is performe to erive the potentials p for preicting the square error from the (normalize) values of x 2 i. The prouct of the two features with the highest potentials becomes the enition of a new feature. This feature 1 Note that this iers from \cascae correlation" (Fahlman an Lebiere, 1990) in that we correlate with the square of the error, rather that with the error itself. 2 If the two features with the highest potentials have alreay been use to ene a feature then the next best pair is use.

3 step 0: initialize variables I = number of instances n = number of features X = norm(ffx 1 1:::x 1 ng:::fx I 1; :::x I ngg) Y = norm(fy 1 :::y I g) F eatureset = fx1:::x ng step 1: o the regression w = Regress(Y jx) P P I SquareError = i=1 ((w0 + I j=1 wjxi j) y i ) 2 if SquareError 0 return results an STOP. step 2: construct new feature X 2 = norm(ff(x 1 1) 2 :::(x 1 n) 2 g:::f(x I 1) 2 ; :::(x I n) 2 gg) p = Regress(SquareErrorjX 2 ) Inex1 = inex(heighest(p)) Inex2 = inex(secon heighest(p)) NewF eature = prouct(x inex1 ; x inex2 ) F eatureset = F eatureset + NewF eature n = n + 1 X = norm(ffx 1 1; :::x 1 n1; prouct(x 1 inex1 ; x1 inex2 )g... fx I 1; :::x I n1; prouct(x I inex1 ; xi inex2 )gg) GOTO step 1. Figure 1: The Potentials Algorithm is ae to the feature set an all the instances are upate an renormalize. The algorithm then returns to step 1 an the process is repeate. 5 AN EXAMPLE Figure 2 shows a simple example illustrating the algorithm an some of its problems. For this example 200 training instances were create by ranomly generating ve real-value variables (range = [0,1]) an calculating their values for the function y = 2 + 3x1x2 + 4x3x4x5. In aition to y an these ve relevant variables x1; x2; x3; x4; an x5; four istracter variables x6; x7; x8; an x9 were ae to each instance (ranomly generate in the same way). In this example, the algorithm erive the correct function in the seventh cycle, after creating six new terms: x5x5, x4x5, x4x4x5x5, x3x4x5, x2x2, an x1x2. The output from each cycle reports the following information: Coes the coecients for each feature resulting from the linear regression Potentials the potentials use to select the two features to combine into a new term Resiual the total square error of the best linear moel of the target function as erive by the regression algorithm New Terms the new term that was constructe One thing to notice in this example is that the algorithm completely isregars the istracter variables (a characteristic share by the correlation metho). This has been a consistent result in all our tests in which the training set was suciently large for the complexity of the terms in the function. As the number of primitive features in a term increases, each iniviual feature's contribution to the function's value ecreases; as this contribution becomes small it becomes icult to separate from the ranom noise of the istracters, unless the size of the training set is appropriately increase. There are two problems with this example, one obvious an the other more subtle. First, four unnecessary features were constructe: x5x5, x3x3, x4x4x5x5, an x1x1. These i not prevent solution of the problem, but they i slow own the process; for more complex functions these aitional terms coul become problematic. Secon, the target function is such that the potentials of the relevant features are signicantly ifferent epening upon which of the two terms of the function they participate in. As a result, the algorithm is able to etermine which relevant features go together. So for example, x1 is in a term of two features having a coecient of 3 whereas x3 is in a term of three features weighte by 4; consequently their initial potentials are quite ierent. If we ha use the function y = x1x2 + x3x4 the algorithm woul not have been able to tell that x1 shoul be paire with x2 rather than with x3 or x4, other than by chance. 6 JOINT POTENTIALS The example iscusse above shows that the metho of potentials is goo at ientifying which terms are relevant to the function, but not at etermining which combinations of these terms are relevant. To remey this shortcoming we evelope the notion of \joint potentials," that is, the potentials of pairs of features. To compute the potentials of all pairs of n features is not feasible as it woul require a regression on n 2 variables. Instea, we restrict attention to the pairs of features that appear most promising accoring to the prouct of the potentials of their constituent features. For the m most promising pairs, for m << n 2, we perform another regression onto the square error. The coecients of this regression are calle the joint potentials. The pair with the highest joint potential is then selecte as the next new feature. This is basically a traitional generate-an-test approach to feature construction, 3 but guie by the potential ratings of the iniviual original features. The example run in Figure 3 shows the results of using this metho on the training set from the rst example (using n = m = 9). This time the algorithm constructs the minimum number of new features require to learn 3 As in, e.g., (Ivakhnenko, 1971).

4 Coeffs: Potentials: Resiual: 193 New Terms: X5X5 Cycle 2: Coeffs: Potentials: Resiual: 193 New Terms: X4X5 Cycle 3: Coeffs: Potentials: Resiual: 193 New Terms: X4X4X5X5 Cycle 4: Coeffs: Potentials: Resiual: 189 New Terms: X3X4X5 Cycle 5: Coeffs: Potentials: Resiual: 69 New Terms: X2X2 Cycle 6: Coeffs: Potentials: Resiual: 68 New Terms: X1X2 Cycle 7: Coeffs: Resiual: 0 Solution: (+ 2.0 (* 4.0 X3 X4 X5) (* 3.0 X1 X2) ) Figure 2: Application of the algorithm to the target function y = 2 + 3x1x2 + 4x3x4x5. The training set inclue four istracter variables an 200 examples. the target function. The information isplaye on each cycle now inclues the top four joint potentials (limite to four for isplay purposes). The example run in Figure 4 shows how the joint potentials metho also solves the problem of etermining which pairs of relevant features belong together. Recall our statement that functions of the form y = x1x2 + x3x4 (where each term is of equal weight an length) are icult for the straight potential metho because it has no way of eciing how to pair features. This thir example emonstrates that the joint potential selection metho can correctly ecie which of the relevant features to combine. Notice that if the algorithm ha use the (non-joint) potentials to pick the new feature pair on cycle 1, it woul have create the useless new feature x1x4 since x1 an x4 have the two highest potentials. Using joint potentials, the relevant feature x1x2 is constructe instea. 7 EXTENSIONS This research is still preliminary. We are in the process of esigning more thorough tests of this approach an we are eveloping a better theoretical explanation for when an why this approach works. In aition we are exploring several new ieas, among them: An incremental algorithm. The current algorithm assumes that all the instances are available at the outset. We inten to make it able to construct features an learn functions incrementally, i.e., by processing each example only once. Comparisons. We are currently working with Sanger to obtain irect comparisons with his original metho on the basis of learning rate an computational requirements. Multiple new features per cycle. Rather than aing just a single feature per cycle we are working on a metho that as a set of features base on their potentials or joint potentials. Application to more complex functions. Because we have constraine our metho to the use of a multiplication operator the omain of solvable problems is quite limite. We are exploring the use of aitional constructive operators, such as ivision, roots, an transcenental functions. Pruning of unnecessary features. As we saw in the above examples, the metho of potentials is quite goo at ientifying irrelevant features. It shoul be a simple matter to use this information to prune the feature set an thereby reuce the computation at each cycle.

5 Coeffs: Potentials: Joints: (( ) ( ) ( ) ( )) Resiual: 193 New Terms: X4X5 Cycle 2: Coeffs: Potentials: Joints: (( ) ( ) ( ) ( )) Resiual: 193 New Terms: X3X4X5 Cycle 3: Coeffs: Potentials: Joints: (( ) ( ) ( ) ( )) Resiual: 69 New Terms: X1X2 Cycle 4: Coeffs: Resiual: 0 Solution: (+ 2.0 (* 4.0 X3 X4 X5) (* 3.0 X1 X2) ) Figure 3: Application of the algorithm to the target function y = 2 + 3x1x2 + 4x3x4x5 using Joint Potentials. The training set inclue four istracter variables an 200 examples. Coeffs: Potentials: Joints: (( ) ( ) ( ) ( )) Resiual: 189 New Terms: X1X2 Cycle 2: Coeffs: Potentials: Joints: (( ) ( ) ( ) ( )) Resiual: 89 New Terms: X3X4 Cycle 3: Coeffs: Resiual: 0 Solution: (+ (* 1.0 X1 X2) (* 1.0 X3 X4) ) Figure 4: Application of the algorithm to the target function y = x1x2 + x3x4 using Joint Potentials. The training set inclue four istracter variables an 200 examples. Acknowlegments We woul like to thank Terry Sanger for his encouragement an iscussion of this work. Thank you also to Rich Branau for his helpful comments. References L. Breiman, J. H. Frieman, R. A. Olshen, an C. J. Stone. (1984) Classication an Regression Trees. Wasworth & Brooks, Belmont, California. S. E. Fahlman an C. Lebiere. (1990) The cascaecorrelation learning architecture. In D. S. Touretzky, (e.), Avances in Neural Information Processing Systems 2, pages 524{532. Morgan Kaufmann. J. H. Frieman. (1988) Multivariate aaptive regression splines. Technical Report TR No. 102, Laboratory for Computational Statistics, Stanfor University, Stanfor, CA. A. G. Ivakhnenko. (1971) Polynomial theory of complex systems. IEEE Trans. Systems, Man, an Cybernetics, SMC-1(4):364{378. T. D. Sanger. (1991) A tree-structure algorithm for reucing computation in networks with separable basis functions. Neural Computation, 3(1):67{78.