Feature Reduction and Selection

Transcription

1 Feature Reducton and Selecton Dr. Shuang LIANG School of Software Engneerng TongJ Unversty Fall, 2012

2 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

3 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

4 Introducton In practcal multcategory applcatons, t s not unusual to encounter problems nvolvng tens or hundreds of features. Intutvely, t may seem that each feature s useful for at least some of the dscrmnatons. In general, f the performance obtaned wth a gven set of features s nadequate, t s natural to consder addng new features. Even though ncreasng the number of features ncreases the complety of the classfer, t may be acceptable for an mproved performance.

5 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

6 Problems of Dmensonalty Unfortunately, t has frequently been observed n practce that, beyond a certan pont, addng new features leads to worse rather than better performance. Ths s called the curse of dmensonalty. There are two ssues that we must be careful about: How s the classfcaton accuracy affected by the dmensonalty (relatve to the amount of tranng data)? How s the complety of the classfer affected by the dmensonalty?

7 Problems of Dmensonalty Potental reasons for ncrease n error nclude wrong assumptons n model selecton, estmaton errors due to the fnte number of tranng samples for hgh-dmensonal observatons (overfttng). Potental solutons nclude reducng the dmensonalty, smplfyng the estmaton.

8 Problems of Dmensonalty Dmensonalty can be reduced by redesgnng the features, selectng an approprate subset among the estng features, combnng estng features. Estmaton errors can be smplfed by assumng equal covarance for all classes (for the Gaussan case), usng regularzaton, usng pror nformaton and a Bayes estmate, usng heurstcs such as condtonal ndependence,.

9 Problems of Dmensonalty Problem of nsuffcent data s analogous to problems n curve fttng. The tranng data (black dots) are selected from a quadratc functon plus Gaussan nose. A tenth-degree polynomal fts the data perfectly but we prefer a second-order polynomal for better generalzaton.

10 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

11 Feature Reducton One way of copng wth the problem of hgh dmensonalty s to reduce the dmensonalty by combnng features. Issues n feature reducton: Lnear vs. non-lnear transformatons. Use of class labels or not (depends on the avalablty of tranng data). Tranng objectve: mnmzng classfcaton error (dscrmnatve tranng), mnmzng reconstructon error (PCA), mamzng class separablty (LDA), retanng nterestng drectons (projecton pursut), makng features as ndependent as possble (ICA), embeddng to lower dmensonal manfolds (Isomap, LLE).

12 Feature Reducton Lnear combnatons are partcularly attractve because they are smple to compute and are analytcally tractable. Lnear methods project the hgh-dmensonal data onto a lower dmensonal space. Advantages of these projectons nclude reduced complety n estmaton and classfcaton, ablty to vsually eamne the multvarate data n two or three dmensons.

13 Feature Reducton Two classcal approaches for fndng optmal lnear transformatons are: Prncpal Components Analyss (PCA): Seeks a projecton that best represents the data n a least-squares sense. Lnear Dscrmnant Analyss (LDA): Seeks a projecton that best separates the data n a least-squares sense. Besdes, we can also resort to statstcal methods to perform feature reducton.

14 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

15 Statstc Methods Calculate feature values Inner-class varance 1 c 1 V nner c 1 N j D Inter-class varance V nter 1 c c 1 ( ) 2 2 ( ) j j Weghtng scheme w V V nter nner w w sum _ w

16 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

17 Prncpal Components Analyss Objectve Seek a projecton that best represents the data whle preservng nformaton Fnd a project Mnmze reconstructon error + ' 1 1, 1 ˆ d n u a n m a u m n d J 1 2 ' ˆ + ' 1 1, 1 ˆ d n u a n m a u m

18 Eamples Scatter plot (red dots) and the prncpal aes for a bvarate sample. The blue lne shows the as e 1 wth the greatest varance and the green lne shows the as e 2 wth the smallest varance. Features are now uncorrelated

19 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components Analyss Lnear Dscrmnant Analyss

20 Lnear Dscrmnant Analyss Objectve Seek a projecton that best separates the data whle preservng nformaton wthn-class scatter between-class scatter D D t c W n m m m S S S 1 ) )( ( 1 c c t B n n n n ) )( ( m m m m m m S

21 Lnear Dscrmnant Analyss For projecton y or t w y W 1,, d' t Seek a transformaton matr W Mamze the rato of the between-class scatter to the wthn-class scatter J ( W ) ~ S ~ S B W W W t t S S B W W W

22 Lnear Dscrmnant Analyss Projecton of the same set of samples onto two dfferent lnes n the drectons marked as w. The fgure on the rght shows greater separaton between the red and black projected ponts.

23 Eample 1 Scatter plot and the PCA and LDA aes for a bvarate sample wth two classes. Hstogram of the projecton onto the frst LDA as shows better separaton than the projecton onto the frst PCA as.

24 Eample 2 Scatter plot and the PCA and LDA aes for a bvarate sample wth two classes. Hstogram of the projecton onto the frst LDA as shows better separaton than the projecton onto the frst PCA as.

25 Eample 3 A satellte mage and the frst s PCA bands (after projecton). Hstogram equalzaton was appled to all mages for better vsualzaton.

26 Eample 3 A satellte mage and the s LDA bands (after projecton). Hstogram equalzaton was appled to all mages for better vsualzaton.

27 Eample 4 A satellte mage and the frst s PCA bands (after projecton). Hstogram equalzaton was appled to all mages for better vsualzaton.

28 Eample 4 A satellte mage and the s LDA bands (after projecton). Hstogram equalzaton was appled to all mages for better vsualzaton.

29 Summary Feature selecton leads to savngs n computatonal costs and the selected features retan ther orgnal physcal nterpretaton. Feature reducton wth transformatons may provde a better dscrmnatve ablty but these new features may not have a clear physcal meanng. Whereas PCA seeks drectons that are effcent for representaton, dscrmnant analyss seeks drectons that are effcent for dscrmnaton.