A new Unsupervised Clustering-based Feature Extraction Method

Transcription

1 A new Unsupervsed Clusterng-based Feature Extracton Method Sabra El Ferchch ACS Natonal School of Engneerng at Tuns, Tunsa Salah Zd AGIS lle Unversty of Scence and Technology, France Kaouther aabd ACS Natonal School of Engneerng at Tuns, Tunsa ABSTRACT In manpulatng data such as n supervsed or unsupervsed learnng, we need to extract new features from the orgnal features for the purpose of reducng the dmenson of feature space and achevng better performance. In ths paper, we nvestgate a novel schema for unsupervsed feature extracton for classfcaton problems. We based our method on clusterng to acheve feature extracton. A new smlarty measure based on trend analyss s devsed to dentfy redundant nformaton n the data. Clusterng s then performed on the feature space. Once groups of smlar features are formed, lnear transformaton s realzed to extract a new set of features. The smulaton results on classfcaton problems for expermental data sets from UCI machne learnng repostory and face recognton problem show that the proposed method s effectve n almost cases when compared to conventonal unsupervsed methods le PCA and ICA. General Terms Pattern Recognton, Machne earnng, Feature Extracton, and Data mnng. Keywords Unsupervsed feature extracton, smlarty measure, clusterng, face recognton and classfcaton problems. 1. INTRODUCTION Feature extracton s an mportant step n data analyss. It can be used as a preprocessor for applcatons ncludng vsualzaton, classfcaton, detecton, and verfcaton. Heren, we are nterested n applyng feature extracton to classfcaton problems and n partcular for face recognton problem. Actually, as the amount of collected data becomes larger, extractng the rght features becomes necessary for an effcent classfcaton process. However, when desgnng a classfcaton system, pror nowledge about effectve features n classfcaton processng, are usually unnown. Although the classfcaton performance s a non-decreasng functon of the number of features, usng a large number of features can be wasteful of both computatonal and memory resources [1]. In addton, tranng a classfer wth a large number of features descrbng a fnte amount of data can, eventually, degrade the classfer accuracy [2]. Two dfferent approaches exst to address ths ssue: feature selecton whch conssts on selectng only relevant attrbutes through a pre-defned crteron for feature relevance [3]; and feature extracton whch apples a lnear or nonlnear transformatons to the orgnal set of features to construct new ones [4]. In fact, dfferent Feature Selecton strateges lead ether to a combnatoral problem or suboptmal soluton [4]. For ths very reason, fndng a transformaton to lower dmensons mght be easer than selectng features, gven an approprate obectve functon. Accordng to whether the class labels are used, Feature Extracton methods can be dvded nto supervsed and unsupervsed ones. When labeled data are suffcent, supervsed feature extracton methods usually outperforms unsupervsed feature extracton methods [5]. However, n many cases obtanng class label s expensve and the amount of labeled tranng data s often very lmted, supervsed feature extracton methods may fal on such small labeled-sample problem. In ths wor, we are rather nterested n unsupervsed feature extracton, where no pror nowledge about pdfs of data or about ts class-dstrbuton s avalable. It has been reported through some experments that the performance of the classfer system can be deterorated as new rrelevant features are added [7]. Ths problem can be avoded by extractng new features contanng the maxmal nformaton about the data. One well nown unsupervsed feature extracton method s Prncpal Component Analyss (PCA) [6]. It s derved from egenvectors correspondng to the largest egenvalues of the covarance matrx for data. PCA sees to optmally represent the data n terms of mnmal mean square error between the representaton and the orgnal data. The man drawbac of ths method s that the extracted features are not nvarant under transformaton. Merely scalng the attrbutes changes the resultng features [7]. However, t may be useful n reducng nose n the data by separatng sgnal and nose subspace. Another unsupervsed method Independent Component Analyss (ICA), has been proposed as a tool to fnd nterestng proectons of the data [7]. It has been devsed for blnd source separaton problems, and has demonstrated a lot of potental n varous applcatons. It produces statstcally ndependent components based on negentropy (dvergence to a Gaussan densty functon) maxmzaton. Hence, feature extracton methods le PCA and ICA and feature selecton methods tres ether to fnd new drectons that are statstcally ndependent, or to elmnate totally the so ugged rrelevant or redundant features wthn a specfc crteron. Alternatve approach s to reduce feature dmensonalty by groupng smlar features nto a much smaller number of feature-clusters, and use these clusters as features. Hence, nformaton contaned n redundant features could be preserved whle the sze of the model s reduced and good performance s mantaned. The crucal stage n such procedures s how to determne the smlarty of features. Ths technque has been used by other authors especally for text classfcaton problems [8, 9] and proten sequences analyss [10]. The frst researches nvestgate the use of a clusterng technques based on a smlarty measure based- Informaton Bottlenec for text categorzaton and the latter use a bologcally motvated smlarty measure based on the 43

2 contact energy of the amno acd and the poston n the sequence for the predcton of HIV protease resstance to drugs. Hence, ths wor ams to develop a new method whch nvestgates the use of a new smlarty-based clusterng technque, to perform unsupervsed feature extracton for pattern classfcaton problems and face recognton n partcular. The new smlarty measure that we propose n ths wor has more general capabltes than the ones dscussed above. Actually, n hgh dmensonal data there are many features that have smlar tendences all along the nstances of the data set: they descrbe smlar varatons of monotoncty (ncreasng or decreasng). Thus, these features may gve related or the same dscrmnatve nformaton for the learnng process. Ths maes such features redundant and useless. Analyzng varatons of monotoncty of each feature along the data set can lead us to determne a form of redundancy between features. By usng trend analyss, each feature wll be totally descrbed by ts sgnature whch s statstcally dstngushed from random behavor. Intutvely, once groups of smlar features have been settled, feature extracton can be realzed. A lnear transformaton s appled on each dentfed cluster of smlar features. The rest of ths paper s organzed as follows: n secton 2, we ntroduce the new unsupervsed method to extract features based on analyzng ther tendences along the data set. A clusterng technque based on a new measure of smlarty between features s used to dentfy and gather smlar features. In secton 3, performance of the proposed Feature Extracton Method based on Clusterng (FEMC) s assessed through classfcaton problem and face recognton tas. We compare our method to conventonal lnear unsupervsed method PCA and ICA. In secton 4, we offer some conclusons and suggestons for future wor. 2. FEATURE EXTRACTION METHOD BASED CUSTERING In ths secton, we ntroduce our new unsupervsed feature extracton method based on clusterng (FEMC) for pattern classfcaton problems. In fact, redundant nformaton s an ntrnsc characterstc of hgh dmensonal data whch may complcate learnng tas and degrade classfer performance [7]. Hence, dentfyng redundant features n data can lead to reduce the dmenson wthout loss of some mportant nformaton. Exstng feature selecton solutons use a flter method to select relevant features and elmnate totally redundant features. Although redundant nformaton s not necessarly relevant for establshng dscrmnatng rules of classfcaton, but t has an underlyng nteracton wth the rest of features. Elmnatng them totally from feature space may lead to elmnate some predctve nformaton of the nherent structure of data and thereby nduce a less accurate classfer. Our goal s then to dentfy and then transform ths form of redundancy n a way that conserves nherent structure of data, and mnmzes the amount of predctve nformaton lost. Intutvely, smlar features descrbe very smlar or mostly the same varatons of monotoncty along the data set. They mght ncorporate the same dscrmnatng nformaton. Identfyng ths form of smlarty, through an approprate metrc, groupng them nto dfferent clusters and applyng a lnear transform on each group, are the dfferent steps of our feature extracton approach. Hence, we based our method on clusterng algorthm based a new smlarty measure to dentfy lnear or complex relatons that would exst between features. Clusterng s supposed to dscover nherent structure of data [11-12]. It s a method to creatng groups of obects, or clusters, n such a way smlar obects are grouped together whle those that are dfferent are segregated n ther dstnct samples but to approxmate the true partton of the functon Q.The new smlarty measure used n feature clusterng s devsed based on trend analyss of each feature vector along the data set. 2.1 Problem Formulaton A feature extracton process transforms, lnearly or nonlnearly, a set of orgnal features nto new ones. The new features are supposed to descrbe completely the nstances of a database and to be useful ether supervsed or unsupervsed classfcaton tas. Merely, consderng a set of D-dmensonal data D x, x,... x, a feature extracton process tres to fnd a 1 2 transformaton T to apply, such that: y T x ;1. (1) d D The transformed sample y s composed of d D new features v 1, v 2,... vd. Each feature vector v s consttuted by the dfferent values correspondng to each nstance or sample x 1. A new metrc s devsed to characterze smlarty between features and a clusterng algorthm s appled on the feature space v 1, v 2,... vd to gather smlar features. The new smlarty measure formulaton used to characterze smlarty between features, s nspred from trend analyss. It wll be explaned n the next secton. Hence, the obectve functon Q to be optmzed by the clusterng procedure can be defned by: d Q d, (2) 1 1 The dstance d between two feature vector v and v has to be approprately defned to get to an optmum result. Hence, clusterng process fnds smlar features and form d clusters where d D, represented each by ts centrod: g f S, (3) Where, S s the set of n feature vectors vs S belongng to the cluster C and the transformaton f s defned by: f S s 1 s1 n w v. (4) Fnally, the obtaned set of centrods g s the set of the d new features that wll re-descrbe the data set D x, x,... x Smlarty Measure Dstance or smlarty relatonshps between pars of patterns are the most mportant nformaton n clusterng process, to approxmate true partton n a data set. FECM focuses on defnng a smlarty measure that characterzes smlarty n the behavor of each par of features. We propose to analyze ther tendences through studyng and comparng ther varatons of monotoncty along the data set rather than dfference between ther real values. Thus, conventonal 44

3 dstance le Eucldean dstance used normally n clusterng algorthm s not sutable for our obectve. Usng Eucldean dstance may lead to erroneous results snce t computes the mean of dfference between each value of two vectors wthout use of tendency nformaton about them. In fact, two features may have the same mean (or closer means) but they dffer completely n ther trend. To descrbe the monotoncty of each feature vector, we used trend analyss. Actually, a trend s sem-quanttatve nformaton, descrbng the evoluton of the qualtatve state of a varable, n a tme nterval, usng a set of symbols such as {Increasng, Decreasng, Steady}[13]. In our case, the trend descrbes the evoluton of each feature n a fnte set of samples. We proceed by computng the correspondng frst order dervatve of a feature vector v at each pont or sample x : (5) dv v v dx x x 1 1 Then, we determne the sgn of the dervatve n each pont by (6). A feature vector v s then beng represented as an - dmensonal vector composed of a new set of varables 1, 1,0. Dstance functon devsed to compare two feature vectors reles on verfyng dfference n the sgn of tendency between two feature vectors. dv 0 dx 1 decrease dv dv f 0 then sgn 1 ncrease. dx dx dv 0 steady 0 dx (6) It s the squared sum of the absolute dfference between the occurrence of a specfed value of for two gven feature vectors. It was nspred from the Value Dfference Metrc (VDM) [14]. Thus, the locaton of a feature vector wthn the feature space s not defned drectly by the values of ts components, but by the condtonal dstrbutons of the extracted trend n each component. That maes the dstance between two features s ndependent from the order of data. The dstance or smlarty measure s gven by: (7) Where d v, v v, v v, v v, v, v v pv pv, / /, (8) Occurrence of n v pv /. (9). pv / s determned by countng haw many tmes the value occurs n the feature vector v for the learnng data set. In fact, n ths wor we have computed the occurrence of the par of varables, 10,11,1 1, 10, 11, 11,00,01,0 1 n each vector v nstead of computng only the probablty of the sngle varable 1, 1,0. A smlarty matrx D between all features vectors s then generated such that D, d v, v ;, 1... n. 2.3 Feature Extracton Schema Feature extracton algorthm that we propose n ths wor, s gven by the Fgure 1. Clusterng algorthm based on the smlarty measure descrbed below, s used to form clusters of smlar features. It conssts at a frst step on computng the smlarty matrx M between each one of the feature set, based on the metrc defned prevously by (7). The second step s to perform clusterng strategy whch s based on C- means clusterng. We fx the number of clusters, whch correspond ndrectly to the number of new extracted features such that: d ndv, where the threshold s chosen emprcally. Intally, clusters are ntalzed randomly. Then each cluster C s formed sequentally by selectng the frst raned features n the smlarty matrx M. They correspond to the set of the most smlar features to the correspondng centrod g. The cluster center s then updated by computng the new centrod: where W features C and g W V (10) 1.., d 1 I s the transform matrx appled to the set of n V of the cluster. n s the cardnalty of the cluster V s the matrx contanng the features vectors n C. Hence, as ths process allows an overlap between clusters, snce features could be assgned to more than one cluster, we dentfy common feature between each par of the obtaned clusters. Each common feature s then assgned to the closest cluster accordng to the Eucldean dstance: 1 2 1, 2 v C C, v C arg mn g v. (11) h Where h s the ether the ndex 1 or 2. Clusters centers are then re-computed usng the current cluster membershps and the process s stopped when all d clusters are constructed. The qualty of extracted feature s quantfed through evaluatng classfcaton accuracy of the transformed data set accordng to a K-fold cross valdaton schema for pattern classfcaton tas and leave one out schema for face recognton tas. 2 45

4 {Input: Raw Data} {Output: cluster centers} dv Compute the sgn of frst order Dervatve sgn dx Compute Proporton pv / Compute matrx of dstance Clusterng: { d ndv: number of clusters : number of preselected features n: ntal number of features Idx: ndex of ntal centrod Whle n > 1 Dst= sort M D, d v, v ;, 1... n Cluster C = select the frst features from Dst Data sets Table 1. Data sets nformaton No. of features No. of nstances No. of classes Intal classfcaton accuracy Sonar Pma Breast cancer Ionosphere A 13-fold cross valdaton schema has been used for the sonar data set and 10-fold cross valdaton was used for the others as n the ref [7] Sonar data set Ths data set was constructed to dscrmnate between the sonar returns bounced off a metal cylnder and those bounced off a roc. It conssts of 208 nstances, wth 60 features and two output classes: mne and roc. In our experment, we used 13-fold cross valdaton n gettng the performances as follows: the 208 nstances are randomly dvded nto 13 sets wth 16 nstances n each. n= n- Update Idx End Intersecton between d fnal clusters} Compute clusters centres g New features= {clusters centres} Fg 1: Pseudo-code for FECM algorthm 3. EXPERIMENTA RESUTS Frst, we have conducted the proposed extracton method (FECM) for some well nown and largely used data sets from UCI machne learnng repostory [15]. Second, we have acheved experments on the face recognton problem through Yale and OR databases. 3.1 Results on UCI Databases We have used 4 datasets: Sonar, Pma, Breast cancer and Ionosphere. Some characterstcs of these data sets are shown n the Table 1. We have conducted conventonal unsupervsed feature extracton method PCA and ICA on these data sets and have compared ther classfcaton performances wth our proposed method FECM for dfferent number of extracted features. We have used Support Vector Machnes SVM [16], as a classfer. No pre-treatment has been done for these bases before conductng feature extracton on them. For SVM, we have used Matlab Toolbox to mplement t. The ernel functon used s Gaussan ernel and s set after varous tests on 10, 1 or Fg 2: Classfcaton accuracy on Sonar Data set For each experment, 12 of these sets are used as tranng data, whle the 13th s reserved for valdaton. The experment s repeated 13 tmes so that every case appears once as part of a test set. For SVM parameters, we have set to 1. The Fgure 2 shows classfcaton accuracy for dfferent number of extracted features. We can see that the performances of FECM are far better than PCA and ICA except for the case of a very low dmenson case of only one extracted feature where ICA outperforms. Snce the concept of our approach s to form groups of smlar features; extractng a very low number of features means gatherng all features n a few numbers of clusters. That can be delcate for some data sets as n ths data set. In Ths case, FECM sn t the most effectve method, nevertheless t stll have better classfcaton accuracy than PCA and better than ICA for hgher dmenson. We can note also that n the case of dmenson 9 and 12, FECM s beyond the ntal error rate of 82% whch s far better than ICA and PCA Pma Indan Dabetes data set It conssts of 768 nstances n whch 500 are class 0 and the others are class 1. It has eght numerc features wth no mssng value. It has numerous nvald data ponts, e.g., features that have a value of 0 even though a value of 0 s not meanngful or physcally possble. These correspond to ponts n feature space that are statstcally dstant from the mean calculated 46

5 usng the remanng data (wth the ponts n queston removed). We removed the ponts n queston from the data set. We have appled PCA, ICA and FECM and compared ther performances as we have done for Sonar data set. In tranng, we have used SVM and the parameter was set to 10. A 10- fold cross valdaton has been used for valdaton process. In followng Fgure 3, classfcaton performances are presented. We can note that the performances of PCA and ICA get closer as the number of extracted features becomes larger. In ths data set, ICA outperforms both PCA and FECM for dfferent numbers of features especally for the lower ones such as dmenson 1 and 2. Fg 4: Classfcaton accuracy on Breast cancer Data set We have used a 10-fold cross valdaton as a strategy of verfcaton. For the SVM classfer, the parameter sgma was set to 10 after we have conducted varous experments. The results show that, wth only one extracted feature, FECM can largely outperform ICA and PCA. Hence, for lager number of extracted features, FECM gets ether smlar or better performance and acheve the best accuracy wth 12 features. Fg 3: Classfcaton accuracy on Pma Data set The proposed method FECM stll perform better than PCA and approaches the performances of ICA for hgher dmenson such as 3 and Wsconsn Breast cancer data set Ths data set conssts of nne attrbutes and two classes; whch are bengn and malgnant. It contans 699 nstances wth 458 bengn and 241 malgnant. There are 16 mssng values and we replaced these wth average values of correspondng attrbutes as n ref [7]. We compared classfcaton performance of our proposed method FECM wth those of ICA and PCA for dfferent number of extracted features. Results are shown n the Fgure 4. We have used a 10-fold cross valdaton as a strategy of verfcaton. For the SVM classfer, the parameter was set to 0.01 after we have conducted varous experments. The results show that, wth only one extracted feature, FECM can get the maxmum classfcaton performance. Hence, for lager number of extracted features, PCA outperform both ICA and FECM Ionosphere data set Ths radar data was collected by a system n Goose Bay, abrador. Ths system conssts of a phased array of 16 hghfrequency antennas. Receved sgnals were processed usng an autocorrelaton functon whose arguments are the tme of a pulse and the pulse number. Instances n ths database are descrbed by 2 attrbutes per pulse number, correspondng to the complex values returned by the functon resultng from the complex electromagnetc sgnal. We compared classfcaton performance of our proposed method FECM wth those of ICA and PCA for dfferent number of extracted features. Results are shown n the Fgure 5. Fg 5: Classfcaton accuracy on Ionosphere Data set 3.2 Results on Face Recognton Problem We have used two datasets: Yale and OR data sets from UCI machne learnng repostory [15]. The Fgure 6 shows some samples from Yale and OR data sets respectvely Yale database It contans 165 GIF mages of 15 subects (subect01, subect02, etc.). There are 11 mages per subect, one for each of the followng facal expressons or confguratons: centerlght, w/glasses, happy, left-lght, w/no glasses, normal, rghtlght, sad, sleepy, surprsed, and wn. The sze of each mage s 320 x 243, composed of pxels. In fact, there are many methods to determne the features of the mage. Fg 6: Samples of Yale and OR databases 47

6 The most ntutve way s to use each pxel as one feature. The nput space becomes too large to be handled. To overcome ths constrant we had to accept the loss of some nformaton. Each mage was reszed to get 783 pxels. The classfcaton performances of extracted features from PCA, ICA and FECM methods were obtaned by the leave-one-out schema. The number of extracted features by ICA s the same as that of PCA (PCA s used as a pre-processor for ICA). In the experments, PCA was ntally conducted on 784 pxels and the frst 30 PCs were used. The ICA was appled to these 30 PCs. Table 2 shows the classfcaton error rates correspondng to each of these methods. The recognton was performed usng the K nearest neghbor classfer (KNN). PCA and ICA produce the same number of features whch s 30, but ICA s slghtly better than PCA n classfcaton accuracy. It can be seen that our proposed approach outperforms PCA and ICA usng only 14 features. Table 2. Classfcaton performance on Yale Data set Methods Error rate (%) No. of (KNN) features PCA ICA FECM KNN OR database The AT&T database of faces conssts of 400 mages, whch are 10 dfferent mages for 40 dstnct ndvduals. It ncludes varous lghtng condtons, facal expressons and facal detals. The mages were cropped nto get 952 pxels for computatonal effcency. The experments were performed exactly the same way as n the Yale database. The results n the Table 3 are from the leave-one-out test wth the one nearest neghbor classfer. The frst 40 PCs are retaned as nput for ICA. PCA and ICA produces smlar results. Our approach s close to PCA and ICA n terms of classfcaton accuracy wth smaller number of features whch s 23. Table 3. Classfcaton performance on OR Data set Methods Error rate No. of features (%) (KNN) PCA ICA FECM KNN CONCUSION Ths wor focuses on developng a general feature extracton approach based on clusterng technque for pattern classfcaton tas and face recognton n partcular. The man motvaton behnd t was to dentfy redundancy n feature space and reduce ts effect wthout losng some mportant nformaton for classfcaton process. Smlar feature are recognzed through analyzng ther tendences along the data set. Although, trend analyss was used to devse the new smlarty measure, t conserves ts ndependency from the order of samples. In the frst stage, the proposed approach FECM extracts feature-clusters as features by applyng clusterng technque, based on the new smlarty measure. In the second stage, these features are used for the classfcaton of patterns. Results obtaned from experments conducted on several data sets, obtaned from UCI machne learnng repostory, showed that ths representaton of patterns mproves over PCA and ICA n almost of cases, especally when proectng to low dmensons. Except for Pma data set, FECM was not able to mprove over other methods n lower dmenson, but gets to smlar results n hgher dmensonal proectons. For the face recognton tas, FECM gets to the lowest dmenson wth the best accuracy for Yale data set. In the case of OR data set, FECM gets the lowest dmenson wth a lower accuracy of 1%. The transformaton appled to the obtaned clusters was a lnear combnaton wth equal weght appled to each of ts correspondng component. However, the transformaton we have to apply s a very mportant step snce t has to determne a representatve feature of each group. It has to preserve the man characterzes of each group of features and ncorporate them nto the new representatve feature. Thus, an approprate transformaton has to be defned n further wor. It would be based on a sophstcated statstcal measure of dependency such as Mutual nformaton. In addton, t would be nterestng to ncorporate label nformaton n the FECM approach for sem-supervsed or supervsed learnng tass. 5. REFERENCES [1] Hld II K. E., Erdogmus D., Torola K. and Prncpe J. C Feature extracton usng nformaton-theoretc learnng. IEEE Trans. on pattern analyss and machne ntellgence, 28 (September 2006), [2] Rpley B.D, Pattern recognton and neural networs. In Cambrgde Unv. Press [3] Guyon I., Elsseeff A An ntroducton to varable and feature selecton. J. Mach. earn. 3(March 2003), [4] Torola K Feature extracton by non-parametrc mutual nformaton maxmzaton. J. Mach. earn. 3 (2003), [5] u X., Tang J., u J., Feng Z., "A sem-supervsed relef based feature extracton algorthm", n second nternatonal conference on future generaton communcaton and networng symposa, [6] Saul. K., Wenberger K. Q., Sha F., Ham J. and ee D. D. Spectral Methods for Dmensonalty Reducton. n O. Chapelle, B. Schoelopf, and A. Zen (eds.), Sem supervsed earnng, MIT Press. Cambrdge, MA [7] Kwa N. and Chong-Ho C "Feature extracton based on ICA for bnary classfcaton problems". IEEE Trans. on nowledge and data engneerng, 15 (November 2003), [8] S. Noam and T. Naftal "The power of word clusters for text classfcaton", 23 rd European colloquum on nformaton retreval research. [9] Baer,. D. and McCallum A. K., "Dstrbutonal clusterng of words for text classfcaton", 21 st Annual nternatonal ACM SIGIR conference on research and development n nformaton retreval. [10] Bonet I., Sayeys Yvan, Grau Abalo R., Garca M.M, Sanchez R., and Van de Peer Y Feature extracton usng clusterng of proten. In proceedngs of the 11 th Iberoamercan congress n pattern recognton CIARP, [11] Fern X.Z., Brodeley C.E Cluster Ensembles for Hgh Dmensonal Clusterng: an emprcal study. Techncal report CS [12] Von uxburg U., Bubec S., Jegela S. and Kaufmann M Consstent mnmzaton of clusterng obectve functons. Neural nformaton processng systems NIPS. 48

7 [13] Charboner S. and Gentl S A trend-based alarm system to mprove patent montorng n ntensve care unts, n Control engneerng practce, 15, [14] Payne T.R. and Edwards P "Implct feature selecton wth the value dfference metrc". 13 th European conference on artfcal ntellgence, pp [15] [16] Cortes C. and Vapnc V Support-vector networs. Machne learnng. 20, pp