Boosted Random Forest

Transcription

1 Boosted Random Forest Yohei Mishina, Masamitsu suhiya and Hironobu Fujiyoshi Department of Computer Siene, Chubu University, 1200 Matsumoto-ho, Kasugai, Aihi, Japan {mishi, Keywords: Abstrat: Boosting, Random Forest, Mahine Learning, Pattern Reognition. he ability of generalization by random forests is higher than that by other multi-lass lassifiers beause of the effet of bagging and feature seletion. Sine random forests based on ensemble learning requires a lot of deision trees to obtain high performane, it is not suitable forimplementingthealgorithmonthesmall-sale hardware suh as embedded system. In this paper, we propose a boosted random forests in whih boosting algorithm is introdued into random forests. Experimental results show that the proposed method, whih onsists of fewer deision trees, has higher generalization abilityomparingtotheonventionalmethod. 1 IRODUCIO Random forest(breiman, 2001) is a multi-lass lassifier that is robust against noise, has high disrimination performane, and is apable of training and lassifying at high speed. It is therefore attrating attention in many fields, inluding omputer vision, pattern reognition, and mahine learning(amit and Geman, 1997), (Lepetit and p. Fua, 2006), (J. Shotton and Cipolla, 2008), (J. Shotton et al., 2011), (Gall et al., 2011). Random forest ontrols loss of generalization in training due to overfitting by introduing randomness in bagging and feature seletion(ho, 1998) when onstruting an ensemble of deision trees. Eah deision tree in random forest is independent, so high speed an be attained by parallel proessing in tree training and lassifiation. Boosting(Freund and Shapire, 1995) is a typial ensemble training algorithm that is used to sequentially onstrut lassifiers for random forest that involve independent deision trees. Boosting is an ensemble training algorithm that ombines weak learners, whih individually have low disriminating performane, to onstrut a lassifier of higher disriminating performane. Boosting generally attains high disrimination by sequential training of lassifiers in whih the training sample for whih the previous lassifier produed lassifiation errors is used to train the subsequent lassifier to produe orret lassifiation. However, boosting tends to overfit the training sample. Random forest, on the other hand, uses randomness in the onstrution of the deision trees, thus avoiding overfitting to the training sample. For that reason, a large number of deision trees must be onstruted to obtain high generality. However, inreasing the number of deision trees inreases the memory requirements, so that approah is not suited to implementation on small-sale hardware suh as embedded system. We therefore propose a boosted random forest method, in whih a boosting algorithm is introdued in random forest. he proposed method onstruts omplementary lassifiers by suessive deision tree onstrution and an yield lassifiers with smaller deision trees while maintaining disrimination performane. 2 RADOMFORES Random forest is an ensemble training algorithm that onstruts multiple deision trees. It suppresses overfitting to the training samples by random seletion of training samples for tree onstrution in the same way as is done in bagging(breiman, 1996),(Breiman, 1999), resulting in onstrution of a lassifier that is robust against noise. Also, random seletion of features to be used at splitting nodes enables fast training, even if the dimensionality of the feature vetor is large. 2.1 raining Proess In the training of random forest, bagging is used to reate sample sub sets by random sampling from the training sample. One sample set is used to onstrut one deision tree. At splitting node n, samplesets n is split into sample sets S l and S r by omparing the

2 value of feature quantity x i with a threshold value τ. he splitting funtion of the splitting node selets ombinations that an partition the most samples from among randomly seleted features {f k } K k=1 and threshold {τ h } H h=1 for eah lass. he reommended number of feature seletions, K, isthesquarerootof the feature dimensionality. he evaluation funtion used for seleting the optimum ombination is the information gain, ΔG. he splitting proessing is repeated reursively until a ertain depth is reahed or until the information gain is zero. A leaf node is then reated and the lass probability P( l) is stored. 2.2 Classifiation Proess An unknown sample is input to all of the deision trees, and the lass probabilities of the leaf nodes arrived at are output. he lass that has the largest average of the lass probabilities obtained from all of the deision trees, P t ( x), aordingtoeq.(1)isthe lassifiation deision. P( x)= 1 P t ( x) (1) 2.3 umber of Deision rees and Disriminating Performane Random forest ahieves generality by using a large number of deision trees for ensemble training. However, it is diffiult to obtain the optimum number of trees for training, so there are many redundant deision trees that onsume a large amount of memory. 3 BOOSED RADOMFORES Random forest is robust against noise and has high generality beause of random training. However, it requires many deision trees, as using fewer deision trees redues performane. It therefore annot maintain generality when implemented on small-sale hardware. For that reason, boosting is introdued to random forest. he purpose of using boosting is to maintain generality even with a small number of deision trees by using the fat that sequential training onstruts omplementary deision trees for the training samples. 3.1 raining Proess he proposed training algorithm involves one proedure for when the sample weighting is updated and another proedure for when there is no Algorithm 1: Proposed method. Require: raining samples {x 1,y 1,w 1 },...,{x,y,w }; x i X,y i {1,2,...,M},w i Init: Initialize sample weight w i : w (1) i 1. Run: for t = 1: do Make subset S t from training samples. ΔG max. for k = 1:K do Random sampling from feature f k. for h = 1:H do Random sampling from threshold τ h. Split S n into S l or S r by f k and τ h. Compute information gain ΔG: ΔG = E(S n ) Sl S E(S n l) Sr S E(S n r). if ΔG > ΔG max then ΔG max ΔG if ΔG max = 0orreahamaximumdepththen Store the probability distribution P( l) to leaf node. else Generating a split node reursively. if Finished training of deision tree then Estimate lass label ŷ i : ŷ i = arg maxp t ( l). Compute error rate of deision tree ε t : ε t = i / i:y i ŷ i i=1 i. Compute weight of deision tree α t : α t = 1 (M 1)(1 εt ) 2 log ε t if α > 0 then Update weight { of training sample w i,t+1 : w (t+1) i = i exp (α t ) if y i ŷ i i exp ( α t ) otherwise. else Rejet a tree updating. First, a training sample of size, {x 1,y 1,w 1 },...,{x,y,w },thathaveafeatureof dimension d and lass labels y M are prepared. he training sample weight, w,isinitializedto 1.Sample sets are reated by random sampling from the training sample. Deision trees are onstruted using the sample sets in the same way as in random forest. Proposed algorithm 1 is desribed in above ode Splitting he flow of the proposed method is illustrated in Fig. 1. he splitting funtion selets ombinations of randomly-prepared features and thresholds that have

3 Figure 1: raining algorithm of the proposed method. the highest information gain. he information gain ΔG is omputed by ΔG = E(S n ) S l S n E(S l) S r S n E(S r), (2) where S n is sample set at node n, S l is sample set at left hild node, S r is sample set at right hild node, and E(S) is entropy omputed by E(S)= M j=1 P( j )logp( j ). (3) In alulating the information gain, the samples are prioritized by largest weight and the probability of lass j, P( j ),isalulatedusingtheweightofsample i, w i omputed using P( j )= w i / w i, (4) i S y i = j where, S is the sample set that arrived at node. A leaf node is reated when reursive splitting has developed the deision tree to a ertain depth or when the information gain of a sample set that has reahed a node is zero. he leaf node stores the lass probability P() obtained with Eq. (4) Deision ree Weighting In the same way as for multi-lass boosting(kim and Cipolla, 2008), (Saberian and Vasonelos, 2008), the deision tree weight, α t,isalulatedby i S α t = 1 2 log (M 1)(1 ε t) ε t, (5) where, ε t is the error rate of the deision tree and M is the number of lasses. he expeted value for the suessful lassifiation rate in random lassifiation is 1 M. If the lassifiation error rate exeeds 1 1 M, the value of α is negative in Eq. (5) and the deision tree is disarded. he training sample is lassified by the onstruted deision trees and the error rate is alulated from the weights of the inorretly lassified samples as ε t = i / i:y i ŷ i i=1 i. (6) Updating raining Sample Weights Deision trees that easily orret lassifiation of the samples that have been inorretly lassified in the next step are onstruted by making the weights of inorretly lassified samples large as w (t+1) i = { i exp(α t ) if y i ŷ i exp( α t ) otherwise, i where ŷ i is estimated lass label using (7) ŷ i = arg maxp t ( l). (8) After updating the training sample weights, the weights are normalized to. Construting the deision trees and updating the training sample weights in that way is repeated to obtain deision trees and weighted deision trees. After all deision trees have been onstruted, the deision tree weights are normalized. 3.2 Classifiation Proess An unknown sample is input to all of the deision trees as shown in Fig. 1, and the lass probabilities that are stored in the arrived-at leaf nodes are output. han, the outputs of the deision trees, P t ( x), are weight-averaged using the deision tree weights as P( x)= 1 α t P t ( x). (9) he lass that has the highest probability ŷ is output as the lassifiation result by ŷ = arg maxp( x). (10) 4 EXPERIMEALRESULS o show the effetiveness of the proposed method, the number of nodes are ompared for the proposed method and the onventional method at the same level of generalization ability. For the proposed method, we investigated proedures with and without sequential sample weight updating.

4 4.1 Data Set he evaluation experiments used four data sets, Pendigits, Letter, Satelite, and Spambase, from those published by the UCI Mahine Learning Repository as a set of mahine training algorithm benhmarks. he data sets are desribed briefly in able 1. able 4: Error rate by boosted random forest [%] able 1: Data sets. Dataset raining ests Class Dim. Pendigits 7,494 3, Letter 10,000 10, Satelite 4,435 2, Spamebase 3,221 1, raining Parameter In this experiment, the depth of the deision trees for training parameters was fixed at 5, 10, 15, and 20. We ompared the minimum value of the miss rate by hanging the number of deision trees. he number of andidates for the split funtion was 10 times the square root of the number of feature dimensions. 4.3 Experimental Results he error ratio for eah dataset of the onventional random forest(rf) and for the proposed method(boosted random forest with or without sample weight updating, BRF, BRF w/o updating) are shown in able 2, 3, and 4 respetively. he miss able 2: Error rate by random forest [%]. Figure 2: Generalization error of pendigits able 3: Error rate by boosted random forest w/o updating [%] rate of the proposed method was lower than that of the random forest when the number of depth for deision trees was shallower. In ontrast, when the number of depth for deision trees was deeper, the miss rate of the proposed method was equal to or lower than that of the random forest. It is lear that the lassifiation performane of the boosted random forest with updating sample weight is superior. Figure 2 shows the miss rate for eah depth in the ase of the Pendigits dataset. From Fig. 2, we realized that the random forest requires more depth in order to obtain a higher lassifiation performane than the proposed method. In ontrast, the proposed boosted random forest method does not require more depth beause it has been trained by hoosing the split funtion with diffiult training samples at upper nodes, whih have aheaviersampleweight. 4.4 Memory Usage for Deision rees For the implementation of deision trees on smallsale hardware, less memory is better. herefore, in this setion we ompare the amount of memory needed for eah method. For eah node, the total memory of a split funtion is 11 bytes, of whih the seleted feature dimension is 1 byte, the threshold is

5 2bytes,andthepointerforahildnodeis8bytes. For eah leaf node, total memory is estimated by the number of lass bytes. hus, we estimate the amount of memory B required for deision trees by B = ( s,t 11 + l,t M), (11) where the number of trees is,numberofsplitnodes is s,t,numberofleafnodesis l,t and number of lasses is M. Figure 3 shows the amount of memory for eah method with minimum error rate and the redution ratio of the proposed method ompared to the random forest. he amount of memory required by the Figure 3: Amount of memory and redution rate. proposed boosted random forest method is signifiantly redued while maintaining the higher lassifiation performane. Due to sequential training, the boosted random forest onsists of omplementary deision trees that enable the final lassifier to be onstruted in favor of those instanes mislassified by previous deision trees. REFERECES Amit, Y. and Geman, D. (1997). Shape quantization and reognition with randomized trees. In eural Computation. MIPress. Breiman, L. (1996). Bagging preditors. In Mahine Learning. Breiman, L. (1999). Using adaptive bagging to debias regressions. In ehnial Report.StatistisDept.UCB. Breiman, L. (2001). Random forests. In Mahine learning. Freund, Y. and Shapire, R. (1995). A deision-theoreti generalization of on-line learning and an appliation to boosting. In Computational learning theory. Gall, J., Yao, A., Razavi,., Van Gool, L., and Lempitsky, V. (2011). Hough forests for objet detetion, traking, and ation reognition. In Pattern Analysis and Mahine Intelligene. IEEE. Ho,. K. (1998). he random subspae method for onstruting deision forests. In Pattern Analysis and Mahine Intelligene. IEEE. J. Shotton, a. A. F., Cook, M., Sharp,., Finohio, M., Moore, R., Kipman, A., and Blake, A. (2011). Realtime human pose reognition in parts from single depth images. In Computer Vision and Pattern Reognition. IEEE. J. Shotton, M. J. and Cipolla, R. (2008). Semanti texton forests for image ategorization and segmentation. In Computer Vision and Pattern Reognition. IEEE. Kim,.-K. and Cipolla, R. (2008). Mboost: Multiple lassifier boosting for pereptual o-lustering of images and visual features. In Advanes in eural Information Proessing Systems. Lepetit, V. and p. Fua (2006). Keypoint reognition using randomized trees. In Pattern Analysis and Mahine Intelligene. IEEE. Saberian, M. and Vasonelos,. (2008). Multilass boosting: heory and algorithms. In Advanes in eural Information Proessing Systems. 5 COCLUSIOS We have proposed a boosted random forest in whih aboostingalgorithmisintroduedtoaonventional random forest. he boosted random forest maintains ahighlassifiationperformane,evenwithfewer deision trees, beause it onstruts omplementary lassifiers through sequential training by boosting. Experimental results show that the total memory required by the boosted random forest is 47% less than that of the onventional random forest. It is thus suited to implementation in low-memory, smallsale hardware appliations suh as embedded systems. Our future work inludes experimental evaluation for image reognition problems that are urrently diffiult to lassify.