Ameran Journal of Appled Senes (0): 445-455, 005 ISSN 546-939 005 Sene Publatons Pattern Classfaton: An Improvement Usng Combnaton of VQ and PCA Based Tehnques Alok Sharma, Kuldp K. Palwal and Godfrey C. Onwubolu Sgnal Proessng Lab., Grffth Unversty, Brsbane, Australa Department of Engneerng, Unversty of the South Paf, Suva, F Abstrat: Ths study frstly presents a survey on bas lassfers namely Mnmum Dstane Classfer (MDC), Vetor Quantzaton (VQ), Prnpal Component Analyss (PCA), Nearest Neghbor (NN) and K-Nearest Neghbor (KNN). Then Vetor Quantzed Prnpal Component Analyss (VQPCA) whh s generally used for representaton purposes s onsdered for performng lassfaton tasks. Some lassfers aheve hgh lassfaton auray but ther data storage requrement and proessng tme are severely expensve. On the other hand some methods for whh storage and proessng tme are eonomal do not provde suffent levels of lassfaton auray. In both the ases the performane s poor. By onsderng the lmtatons nvolved n the lassfers we have developed Lnear Combned Dstane (LCD) lassfer whh s the ombnaton of VQ and VQPCA tehnques. The proposed tehnque s effetve and outperforms all the other tehnques n terms of gettng hgh lassfaton auray at very low data storage requrement and proessng tme. Ths would allow an obet to be aurately lassfed as qukly as possble usng very low data storage apaty. Key words: VQPCA, Classfaton auray, LCD, total parameter requrement, proessng tme INTRODUCTION Pattern lassfaton/reognton s an area where we learn how to better famlarze the obets to the mahne and get atons or desons based on the observed ategores of the pattern. A pattern ould be human fae, sampled speeh, handwrtten or prnted dgts, any letter, gesture, spoken word, fnanal data, bometr data or any statstal data. Humans naturally lassfy/reognze patterns from the envronment n everyday lfe. A fve year old kd an adapt to dfferent type of obets or patterns and reat aordngly. Ths adaptaton s taken for granted untl we ome to teah a mahne to lassfy/reognze and provde atons or desons on the same patterns. The more the patterns avalable, the better the deson would be. Ths gves hope to desgn a lassfer system. For the last fve deades researh s gong on n ths feld to provde an optmum lassfer/reognzer. But the lassfer performane s stll far behnd the perepton of a human bran. However, pattern lassfaton/reognton plays a rual role n the areas lke bankng, multmeda ommunaton, data synthess, speeh or mage proessng, forens senes, omputer vson and remote sensng, data mnng, robots and artfal ntellgene. It emerged as an essental and ntegral part of daly lfe. The evolvng omputatonal demand n pattern lassfaton makes ths feld very hallengng and thus open for researh. For example n mage reognton, several thousands of multdmensonal patterns are requred for proessng whh makes the mplementaton of the lassfer system qute mpossble. There are two man ategores of pattern lassfaton () supervsed lassfaton: where the state of nature for eah pattern s known and () unsupervsed lassfaton: where the state of nature s unknown and learnng s based on the smlarty of patterns []. In ths study only supervsed pattern lassfaton proedures have been onsdered. A supervsed lassfaton ould be subdvded nto two man phases namely tranng phase and testng phase. In the tranng phase the lassfer s learned by known ategores (lasses) of patterns and n the lassfaton or the testng phase unknown patterns whh were out of the tranng datasets assgn lass labels of tran patterns for whh the dstane from the test pattern to the prototype(s) s mnmzed. The performane of a lassfer depends upon several fators. Some of the man fators are () number of tranng samples avalable to the lassfer. () Generalzaton ablty.e. ts performane n lassfyng test patterns whh were not used durng the tranng stage. () Classfaton error-some measured value based on the norret deson of the lass labellng of any gven pattern. (v) Complexty - n some ases (due to lassfer desgn) the number of features or attrbutes (dmensons) are relatvely larger than the number of tranng samples usually referred as the urse of dmensonalty, (v) speed-proessng speed of tranng and/or testng phase(s) and (v) storage-amount of parameters requred to store after the tranng phase, for lassfaton (testng) purposes []. For a gven lassfer model and a fxed number of tranng samples, the performane may depend on the Correspondng Author: Alok Sharma, Sgnal Proessng Lab., Grffth Unversty, Brsbane, Australa 445
generalzaton apablty (auray), speed and mplementaton ost (due to storage of nformaton). The number of parameters requred to perform lassfaton task (testng) after the tranng proedure, s referred as total parameters. For a gven lassfer we an assoate the total parameters to the mplementaton ost of the lassfaton system and the generalzaton apablty may depend upon the type of parameters (dstrbuton, values et.) Used. The hgher the total parameters requred for lassfaton task the ostler the system would be. Another mportant fator n lassfer desgn s the speed or the proessng tme requred to do the task. It s possble n a lass that n two dfferent nstanes the total parameter requrement s same but the proessng tme dffers. We therefore want to redue the total parameters and proessng tme but at the same tme last sarfe the lassfaton auray. In other words, we searh for the optmal lassfaton auray or least lassfaton error, nvolvng as mnmum total parameters and proessng tme as possble. Ths would allow the system to lassfy/reognze an obet as qukly as possble at mnmum ost. Nearest neghbor (NN) lassfer [] s the most smple lassfer found up tll now. In NN lassfer no speal proedure s requred to do the tranng. All the avalable data (as maxmum as possble) are stored to perform lassfaton, where eah test pattern s ompared for smlarty wth all the avalable tranng data (pattern). The test pattern s assgned the lass label of that tranng pattern, whh s the losest to the test pattern. A maor drawbak of NN approah s ts large total parameter requrement to perform the lassfaton task. For example, a dataset wth 0 lasses, havng 5000 vetors or patterns n eah lass wth 64 attrbutes or dmensons would requre total parameters as follows: Am. J. Appl. S., (0): (0): 445-455, 005 same as that of NN approah exept for the omputatonal demand, whh s severe n the former approah. The mplementaton ost of the lassfaton system ould be redued by estmatng eah lass by a sngle prototype, usually a entrod. Ths would help n dereasng the total parameter requrement for the lassfaton task but ould be at the pre of lassfaton auray. Ths type of lassfer s known as Mnmum Dstane Classfer (MDC). The goal of MDC s to orretly label as many patterns as possble. It provdes the mnmal total parameter requrement and omputatonal demand. The MDC method fnds entrod of lasses and measures dstanes between these entrods and the test pattern. In ths method, the test pattern belongs to that lass whose entrod s the losest dstane to the test pattern. Takng the same above example of 0 lasses, the total parameter requrement for the MDC would be ust 640, whh s about /5000 as ompared to NN approah. Usually lassfaton auray s sarfed to get ths advantage of extremely low proessng tme and total parameter requrement. MDC s used n many pattern lassfaton applatons [3-7] nludng dsease dagnosts [8], lassfaton of dgtal mammography mages [9] and optal meda nspeton [0]. The natural extenson of sngle prototype s mult prototype, where eah lass s estmated by several prototypes lke n Vetor Quantzaton (VQ) [,]. VQ based lassfers are also referred as loal lassfers sne ther partton eah lass nto several dsont regons or loal regons and estmate eah regon by a prototype (entrod) usually referred as a odeword. The set of odewords s known as odebook of the system. The am of VQ tehnque s to fnd the odebook that mnmzes the expeted dstorton between pattern x and the entrod of total parameters = lass NoOfVe dmenson dsont 6 regon ( µ ).e. D = E[mn( x µ )] where E[ ]. denotes = 0 5000 64 = 3. 0 expetaton wth respet to x. So the tranng proedure If the dmenson s very hgh (e.g., n mage), then s to fnd the odebook and store t for lassfaton the total parameter requrement for the NN approah tasks. Inreasng the number of odewords per lass wll be even more severe whh would restrt the would nrease the performane up to some extent but t pratal applaton of suh approah. It an also be would also augment the total parameter requrement seen that an nrease n the total parameter does not and proessng tme. VQ tehnque s appled n several always lead to better performane. When tran patterns areas of pattern ompresson and lassfaton [3], and test patterns are losely mathed then auray whh nlude mage lassfaton [4] speeh odng or obtaned by NN approah s good. But when the test speeh ompresson [5], speaker reognton [6], hgh patterns do not math wth tran patterns, NN approah range resoluton sgnature dentfaton [7] and mage provdes poor performane (n terms of auray). In odng [8]. the unmathed pattern ase the performane of the Another way of performng lassfaton s by lassfer system does not mprove by nreasng the utlzng lnear subspae lassfers [9,0]. Here eah total parameters. lass s represented by ts Karhunen-Loéve transform The lassfaton auray of NN approah an be (KLT) [] or Prnpal Component Analyss (PCA). The mproved by makng the deson of a test pattern for obetve of PCA s to fnd a global lnear transform of lass labellng based on k nearest patterns. Ths method gvng patterns n the feature spae and produe lassndependent s known as k-nearest Neghbor (KNN) [] tehnque. or lass-dependent bass vetors. The frst The total parameter requrement for knn approah s bass vetor s n the dreton of maxmum varane of 446
the gven data. The remanng bass vetors are mutually orthogonal and n order, maxmze the remanng varanes subet to the orthogonalty ondton. The prnpal axes are those orthonormal axes onto whh the remanng varables under proeton are maxmzed. These orthonormal axes are gven by the domnant egenvetors (.e. those wth the largest assoated egenvalues) of the ovarane matrx. Class-ndependent PCA fnds those h orthonormal axes (subspae dmenson) from d-dmensonal datasets ( h < d ), where h domnant egenvetors are from the t KLT of the data orrelaton matrx Σ = E[xx ] whh s n fat a ovarane matrx wth zero mean []. Classndependent PCA annot be used for lassfaton purposes sne all the lasses are sattered over the feature spae wth dfferent entrod values or mean and varanes for eah lass makng mpossble to preserve the ndvdual lass nformaton by a sngle KLT for the entre tran samples. Therefore domnant egenvetors are taken for eah lass separately (lass-dependent). For a -lass problem, ovarane matrx wll be gven by: Σ = E[(x µ )(x µ ) ] for =,,... t where only those x, that belong to the th lass have been taken n the expetaton funton at a tme. It has been seen that the subspae lassfaton s further mproved by ts loal lnear extenson []. Here the performane depends upon the subspae dmenson and the number of loal regons. Kambhatla and Leen [] and Kambhatla [3] have shown loal lnear PCA or VQPCA for representaton purposes. The goal of VQPCA s to mnmze the mean squared reonstruton error E[ x x ˆ ] where ˆx s the reonstruted pattern of x. Kambhatla [3] showed VQPCA usng Euldean dstane (VQPCA-Eu) and VQPCA usng reonstruton dstane (VQPCA-re). VQPCA-re s a better tehnque than VQPCA-Eu for representaton purposes n terms of ahevng lesser reonstruton error, but ths ahevement omes wth the expense of hgher total parameter requrement and omputatonal demand. For example, takng the same 0 lass problem, where eah lass s subdvded nto 4 dsont regons (loal regons), ths would requre storage of dxd (64x64) egenvetor set for eah dsont regon together wth other parameters (entrod of dsont regon).e.: total parameters = parameters due to egenvetors + parameters due to entrod total parameters (VQPCA-re) = (d d)*lass*level + (d )*lass *level total parameters (VQPCA-Eu) = (d h)*lass*level + (d )*lass*level Am. J. Appl. S., (0): (0): 445-455, 005 where the term level s the number of dsont regons or loal regons per lass and h<d. Ths yeld total parameter requrement for VQPCA-re.66x0 5 (for d=64), whereas 7680 (for h = ) for VQPCA-Eu whh d+ s /( ) ompared to VQPCA-re. Although the h+ VQPCA-re model exhbts slght mprovement over VQPCA-Eu model, t severely nreases the total parameter requrement and omputatonal demand. Ths would nrease the mplementaton ost and proessng tme of the lassfaton system. Consderng the mplementaton ost and omputatonal demand we opted for an eonomal model (VQPCA-Eu) to tran the system. Hereafter VQPCA-Eu model wll be referred as VQPCA model. Some modfaton s requred n VQPCA model pror to use as a lassfer. The urrent VQPCA model frst parttons the data spae nto dsont regons and then performs loal PCA about eah luster (referred as a dsont regon of a lass) enter. Ths s deal for representaton purposes but for the lassfaton task a mnor hange n dstane measurement s requred whh should reflet the dstane of a test pattern from the entrod and domnant egenvetors of eah dsont regon onurrently. The VQPCA model as a lassfer does not exhbt very enouragng results but stll an be used to perform the lassfaton task. Nonetheless t an be shown that VQPCA model as a lassfer behaves satsfatorly n terms of obtanng reasonably well perentage auray at low total parameter requrements and proessng tme. The performane of VQPCA as a lassfer ould be sgnfantly mproved by ombnng the lnear dstanes of VQ and VQPCA. The normalzed reonstruton dstane measure x x ˆ and the normalzed dstane between the test pattern and the enter of dsont regon x µ, are ombned lnearly to form a new dstane measure for the lassfaton. Ths dstane measure would mnmze the ombnaton of the mean squared reonstruton error (MSE) E[ x x ˆ ] and the expeted dstorton E[ x µ ]. Eah dstane added together may have ts own loal regons n the feature spae where t performs the best. We have ntrodued ths lnear ombnaton of dstane (LCD) tehnque and shown n ths study that t s a better lassfer wth no extra total parameter requrement than VQPCA. Classfaton results obtaned by LCD exhbt sgnfant mprovement over MCD, VQ, VQPCA, NN and knn lassfers n terms of ahevng hgher perentage auray or lower lassfaton error and at the same tme mantanng the total parameters requrement and proessng tme as mnmum as possble. Consequently, ths would allow lassfaton or reognton of the obets as qukly as possble at mnmum ost. Conventonal lassfers: The style of notatons s adopted from Duda and Hart [4]. In all the dsussons 447
ω denotes the state of nature or lass label of th lass n a -lass problem, χ denotes the set of n tran samples, Ω = { ω : =,,,} be the fnte set of states of nature and let θ be the lass label of tran pattern or prototype suh that θ Ω.The set χ an be separated by lass nto subsets χ, χ,, χ, wth the samples n χ belongng to ω : χ = x, x,, x } where x R d (d-dmensonal { n hyperplane): Am. J. Appl. S., (0): (0): 445-455, 005 χ χ and χ χ χ = χ Let n denote the number of samples n the subset χ, therefore n = n. = Fgure llustrates the lass labellng of a test pattern and the relatonshp between the label of the prototype ( θ ) and the label of the lass ( ω ). The prototype ould be a tran pattern, a entrod, a KLT or a group of entrod and KLT dependng upon the type of lassfer s used. In Fg. two-lass problem s onsdered where eah lass onssts of 3 prototypes. Eah of the lass s assgned a unque label namely ωp and ωq suh that ( ωp, ωq ) Ω. The lass labels of the prototypes are θ, θ, θ k and θ l, θ m, θ n suh that: ω = θ = θ = θ and ω = θ = θ = θ p k q l m n The lass label of prototype s assgned to a test pattern x whh s the losest to the prototype based on some dstane measurements or ondtonal probabltes. Therefore f L(x) denotes the lass label of a test pattern x then from the fgure L(x) = θ l = ωq. NN lassfer: The proedure for NN lassfer an be subdvded nto two man phases namely, tranng phase and testng or lassfaton phase. In the tranng phase all the avalable patterns χ wth ther orrespondng lass label nformaton are stored for lassfaton purpose. The total parameter requrement for the NN approah s gven by: total parameters = d n = d n () = Fg. : Class labellng of a test pattern n a two-lass problem knn lassfer: knn lassfer s a generalzed form of NN lassfer. In ths approah k nearest tran patterns to a test pattern x s olleted. The test pattern s assgned the lass label whh has the maorty of k olleted patterns. The tranng phase of the knn lassfer s smlar to NN lassfer where all the tranng patterns together wth ther lass label nformaton are stored for the later use. The total parameter requrement s also same as NN approah. The proessng speed of KNN lassfer s slower than NN lassfer due to the searhng of k nearest patterns for eah of the test patterns. The lassfaton auray may mprove wth the nrease n the value k. Ths mprovement s usually observed when the test patterns and the tran patterns are losely mathed. However, n some ases when the test patterns and the tran patterns do not math the lassfaton auray s poor. In ths ase nreasng the value k may not mprove the lassfaton auray of the system. MDC lassfer: In MDC lassfer eah lass χ s represented by sngle prototype, whh s usually the entrod of the lass n the feature spae. It requres a mnmal total parameter requrement and least omputatonal demand. The total parameter requrement for MDC s: total parameters = d Whh s / n as ompared to NN or knn = lassfer. Ths advantage of the lower total parameter requrement and fast omputaton may aheve by sarfng some lassfaton auray. VQ lassfer: VQ lassfer s the further extenson of It an be seen from equaton that total parameters MDC lassfer. Here eah lass s represented by depend upon the attrbute or dmenson d, number of multple prototypes. VQ parttons a lass nto several lass and number of tranng patterns. In many pratal dsont regons n the feature spae usually known as applatons the values of d and n are very large whh Vorono regons []. The enter of Vorono regons severely affets the storage requrements and (prototype) s referred as odeword of the lassfer and proessng tme, nreasng the ost and redung the a set of odewords s known as odebook of the speed of the lassfer system. lassfer system. The am of VQ s to produe a 448
Am. J. Appl. S., (0): (0): 445-455, 005 odebook that mnmzes the expeted dstorton E[ x µ ]. See Lnde et al. [] for detals. The total parameter requrement s d (Q ) where Q s the level of lassfer.e. Number of dsont regons or ode words for eah of the lasses. PCA lassfer: Class dependent PCA s onsdered for lassfaton where eah lass s represented by ts KLT. In a d-dmensonal feature spae left Σ and µ denote ovarane matrx and entrod, f lass χ n a -lass problem respetvely, xˆ be the reonstruted pattern of x, then the goal of the tranng phase of PCA lassfer s to fnd egenvetors w suh that the followng rtera s satsfed: Σ w = λ w () where, λ denotes egenvalues orrespondng to whh s obtaned by mnmzng MSE E[ x x ˆ ] total parameter requrement for PCA lassfer s: total paramaters = entrod _ paramters + egenvetor _ parameters total paramaters = d + (d h) = d(h + ) where, h < d s the number of egenvetors used. w. The VQPCA as a lassfer: In ths approah, frstly, the set of tran patterns are parttoned nto dsont regons by applyng the VQ tehnque for eah lass separately and then KLT s performed on eah of the dsont regons or loal regon enter []. The am of VQPCA s to mnmze MSE E[ x x ˆ ] n the loal regons. To llustrate tranng and lassfaton proedures let Q be the number of dsont regons or levels per lass. (Detals of the tranng proedure are gven n Kambhatla [3]. VQPCA an also be traned usng splttng tehnque [5] ). Step 5: Store W and µ wth ther orrespondng lass nformaton for lassfaton. The total parameter requrement for VQPCA an be gven by: total paramters = parameters _ entrods + paramters _ egenvetors total paramters = Q d + Q (d h) = Qd(h + ) Whh s Q tmes the total parameter requrement of PCA lassfer. If VQPCA s used for representaton purposes then n the deodng step (here lassfaton) frstly the losest dsont regon to a test pattern x s omputed. One the losest regon s obtaned, the next step s to use ts orrespondng egenvetor and entrod nformaton to ompute reonstruted pattern xˆ. For lassfaton VQPCA proedure would provde no better performane than VQ tehnque sne the deson would le only on the losest dsont regon to the test pattern x and the omputaton of KLT for dsont regons may beome redundant. Therefore a proedure for deson makng of a test pattern should be adopted that uses both the entrod and dreton (egenvetor) nformaton n parallel. Classfaton: Step : Compute reonstruton dstane test pattern x and ts reonstruted pattern ˆx : δ = x x ˆ = (I W W )(x µ ) for =,,...,(Q ) t δ between a Step : Fnd the argument for whh the reonstruton dstane s mnmzed: Q k = arg mn δ = Step 3: Assgn lass label ω = r θ k to the test pattern x, where θ Ω. k Thus, t an be seen that step omputes the error of reonstruton dstane by usng dreton and entrod nformaton n one sngle step for the lassfaton. Tranng Step : Take tran patterns χ χ of lass label ω at a tme for onsderaton, where =,,...,. Step : Apply VQ tehnque and partton χ nto Q dsont regons; for all =,,...,. Step 3: For eah dsont regon ompute entrod and ovarane matrx Σ where =,,...,( Q). Step 4: Evaluate d h retangular matrx of egenvetors W = {w l : l =,,...,h} for eah dsont regon where h<d and w s from equaton ; arrange the obtaned egenvetors suh that ts orrespondng egenvalues are n desendng order. Let the lass label of egenvetor set W beθ Ω. µ 449 LCD lassfer: The LCD s a ombnaton of VQ and VQPCA tehnques. Empral results show sgnfant mprovement of LCD lassfer over prevously dsussed lassfers n terms of gettng hgher perentage auray wth the total parameter requrement no more than VQPCA approah. In our approah the tranng phase of the lassfer s dental to VQPCA lassfer thus the total parameter requrement for LCD approah s same as VQPCA approah. However the lassfaton proedure dffers. In the lassfaton phase the dstane used n VQ lassfaton and the dstane used n VQPCA lassfaton s added together wth some weghtng to form a new dstane measure. Ths ombnaton or addton may redue expeted dstorton E[ x µ ]
Am. J. Appl. S., (0): (0): 445-455, 005 and MSE or root-mse E[ x x ˆ ], overall produng mproved results for the ombnaton. The mproved results aheved ould be due to eah of the onsttuent dstane performng the best n ther loal regons n the feature spae. The generalzaton apablty or lassfaton auray of a lassfer depends on the type of dstrbuton or values used for tranng and/or testng the lassfer. For e.g. If tranng patterns of eah lass are spherally dstrbuted, dense, well separated wth eah other and test pattern are losely mathed wth ther tran patterns then tehnques suh as MDC, VQ, NN and kn may perform better; f outlers are present n the tranng patterns then tehnques suh as PCA or VQPCA may gve poor performane. However for Gaussan data wth mathng tran and test ondtons PCA may provde reasonably hgh lassfaton auray [] and VQPCA and LCD may provde even better performane than PCA. In the presene of outlers and omplex dstrbutons (unmathed tran and test ondtons) LCD may provde better performane than other tehnques. The onept of ombnaton of multple lassfers has been prevously appled by Xu et al. [6] for handwrtng reognton. They have llustrated the ombnaton usng some bas lassfers suh as Bayesan and knn and shown three ategores of ombnaton whh depend upon the levels of nformaton avalable from the lassfers. Jaobs et al. [7] suggested supervsed learnng proedure for systems omposed of many separate expert networks. Ho et al. [8] used multple lassfer system to reognze degraded mahne-prnted haraters and words from large lexons. Tresp and Tanguh [9] presented modular ways for ombnng estmators. Woods et al. [30] and Woods [3] presented a method for ombnng lassfers that use estmates of eah ndvdual lassfer s loal auray n small regons of feature spae surroundng a test pattern. Zhou and Ima [3] showed a ombnaton of VQ and multlayer pereptron (MLP) for Chnese syllable reognton. Almoglu and Alpaydn [33] used the ombnaton of two MLP neural networks for handwrtten dgt reognton. Kttler et al. [34,35] developed a ommon theoretal framework for ombnng lassfers whh use dstnt pattern representatons. Breukelen van and Dun [36] showed the use of ombned lassfers for the ntalzaton of neural network. Alexandre et al. [37] ombned lassfers usng weghted average after Turner and Gosh [38]. Ueda [39] presented lnearly ombnng multple neural network lassfers based on statstal pattern reognton theory. Senor [40] used ombnaton of lassfers for fngerprnt reognton. Le et al. [4] demonstrated a ombnaton of multple lassfers for handwrtten Chnese harater reognton and Yao et al. [4] used a ombnaton based on fuzzy ntegral and 450 Bayes method. Smlarly several other researh work on ombnatonal lassfers have been reported n the lterature. In our approah the tranng phase parameters µ (entrod) and lass label W (egenvetor set) are stored wth the θ Ω nformaton for the use n the lassfaton phase whh s same as the tranng phase of VQPCA approah. Let n a -lass problem eah Class s separately parttoned nto Q dsont regons then the lassfaton phase of the LCD approah an be llustrated as follows: Classfaton Step : Compute the dstane x and the entrod µ of the dsont regon: δ = x µ for =,,...,(Q ) δ between a test pattern Step : Compute the reonstruton dstane δ between a test pattern x and ts reonstruted pattern ˆx : δ = x x ˆ = t (I W W )(x µ ) for =,,...,(Q ) Step 3: Normalze dstane δ and δ to elmnate the dfferene n ther ampltudes that would allow them to ontrbute equally n deson makng. Q = Q ˆ δ = δ / max( δ ) and ˆ δ = δ / max( δ ) = Step 4: Add dstane ˆ δ and ˆ δ : ˆ δ = αδˆ + ( α) ˆ δ for =,,...,(Q ), where α s a weghtng onstant n the range[0,]. Step 5: Fnd the argument for whh the ombned dstane s mnmzed: Q k = arg mnδˆ = Step 6: Assgn lass label ωr = θ k to the test pattern x, where θ k Ω. The lassfaton phase of LCD tehnque s smple, omputatonally nexpensve and attans hgh lassfaton auray or low lassfaton error. The dstane ˆ δ n the lassfaton phase depends on the weghtng onstant α and the two normalzed dstane ˆ δ and ˆ δ. The weghtng onstant α (n step 4) s a postve onstant n the range [0,]. The approprate value for α should be taken sne bad seleton may lead to poor lassfaton auray. The two normalzed dstanes ˆ δ and ˆ δ are lassfaton dstane of VQ and VQPCA tehnques respetvely. Choe of α: The optmum or lose to optmum performane by LCD lassfer an be obtaned by
Am. J. Appl. S., (0): (0): 445-455, 005 seletng the approprate value of α emprally. We have used speeh data [43] and mage data [44,45] to selet the value of α. In ths study we have taken α as a numeral onstant, however, one an also take α as a probablst model whh would depend on a test pattern and the dstrbuton of tran patterns. Ths may nrease the omputaton and storage requrements. The dsusson about α as a probablst model s beyond the sope of ths study. In Fg. and 3 lassfaton auray for LCD tehnque s omputed for dmenson h and level Q, where h =,...,4 and Q =,, 4,8,6. The values of α are 0.,0.,...,0.9, where hoosng α values lose to 0. and 0.9 wll gve performane smlar to VQPCA approah and VQ approah respetvely. Fg. : Classfaton auray for dfferent values of α on mage data Dvertng ether upwards ( α = 0.6,..., 0.9 ) or downwards ( α = 0.4,...,0. ) from the enter value of α (0.5) wll make the dstane ˆ δ based for ˆ δ or ˆ δ respetvely. It an be observed from Fg. and 3 that at α = 0.5 lassfatothe LCD tehnquetaned by LCD tehnque (n Fg. and 3 denoted by bold lnes) s lose to optmum. Ths mples that when the dstane ˆ δ and ˆ δ ontrbute equally n the deson makng for a test pattern n the feature spae then the lassfaton auray s lose to optmum. Thus we have takenα = 0.5. Expermentaton: For all the experments two sets of mahne learnng orpuses have been utlzed namely TIMIT database [43] for speeh lassfaton and Sat- Image dataset [44,45] for mage lassfaton. From the TIMIT orpus a set of 0 dstnt monothongal vowels s extrated, then eah vowel s dvded nto three segments and eah segment s used n gettng Melfrequeny Cepstral oeffents wth energy-deltaaeleraton (MFCC_E_D_A) feature vetors [46]. A total of 9357 MFCC_E_D_A vetors of dmenson 39 for tranng sessons and a separate set of 3 vetors for lassfaton are utlzed. The seond dataset s Sat- Image whh onssts of 6 dstnt lasses wth 36 dmensons. A sum of 4435 feature vetors s used to tran the lassfer and a dfferent set of 000 vetors s used for verfyng the performane of the lassfer. In the frst part of the expermentaton, lassfaton auray s measured for all the lassfers gven some fxed parameters. Here the auray s a funton of dmenson h and level Q, where Q =,, 4, 8, 6 and h =,,..4 for all the levels, exept for Q = 8, where h =,,...,0. Level 8 (Q = 8) s taken at random for dmenson h =,,...,0 to get a general understandng of how the dmenson affets the lassfaton auray f t s nreased ontnuously. Fg. 4: Classfaton auray vs. dmensons and Fg. 3: Classfaton auray for dfferent values of levels usng MDC, VQ, PCA, VQPCA and α on speeh data LCD on mage data sets 45
Am. J. Appl. S., (0): (0): 445-455, 005 Not all the tehnques depend upon both the dmenson h and level Q; VQ depends upon levels, PCA depends upon the dmensons, MDC, NN and knn depend nether upon dmensons nor on levels, only VQPCA and LCD depend upon dmensons as well as levels. Fg. 4 (mage dataset) and Fg. 5 (speeh dataset) llustrates the lassfaton auray for MDC, VQ, PCA, VQPCA and LCD tehnques and Table depts lassfaton auray for NN and knn tehnques. Usually the MDC tehnque s a speal ase of VQ when Q =, that s why t s represented n the olumn of Level n Fg. 4 and 5. Table : Classfaton auray for NN and knn tehnques on mage and speeh datasets Tehnque Classfaton auray Classfaton usng mage dataset auray usng speeh dataset NN 90.30 74.05 knn 3 90.45 75.67 5 89.70 76.8 7 90.05 77.56 9 90.05 78.5 89.35 78.34 Fg. 5: Classfaton auray vs. dmensons and levels usng MDC, VQ, PCA, VQPCA and LCD on speeh dataset It an be observed from Fg. 4 (mage datasets) that MDC s gvng better lassfaton auray than PCA; VQ s produng a hgher lassfaton auray at Level and Level 4 than VQPCA, but VQPCA s showng mprovement over VQ tehnque at level 8 and level 6. It s also lear that LCD s performng better than MDC, VQ, PCA and VQPCA at all the levels and dmensons. Inreasng the dmenson at any gven level s mprovng the lassfaton auray of LCD tehnque. At level 8 and dmenson 0 the lassfaton auray of an LCD s 89.% whh s very lose to NN and knn tehnques. It should be noted that NN and knn tehnques produe smlar lassfaton auray as LCD tehnque but ther proessng tme and total parameter requrement are severely expensve. Furthermore, t an be observed from the experment on speeh data (Fg. 5) And Table that MDC s gvng better lassfaton auray than NN tehnque; PCA s mprovng at dmenson over MDC tehnque; VQPCA s produng a better lassfaton auray over VQ tehnque at levels and 4 for dmenson but deteroratng at level 8 and level 6. LCD s exhbtng better performane than all the tehnques nludng NN and knn. The lassfaton auray s mprovng wth the nrease n dmenson at any gven level. The lassfaton auray by NN and knn s qute poor for speeh data. Ths may be due to the testng data not mathng wth ther tranng data. In the seond part of expermentaton, lassfaton auray s omputed as a funton of total parameters and proessng tme. Ths would gve 3D plot where x and y axes represent total parameters and proessng tme and z-axs represents lassfaton auray. For smplty, a 3D plot s splt nto two D plots, where one plot shows lassfaton auray versus total parameters and the other plot shows lassfaton auray versus proessng tme for the orrespondng values of total parameters. Fg. 6.: Classfaton auray vs. log 0 (total Fg. 6.: Classfaton auray vs. Proessng tme on parameters) on mage datasets mage datasets 45
Am. J. Appl. S., (0): (0): 445-455, 005 The level s taken as Q =,, 4, 8, 6 and data sesson respetvely and the total parameter requrement for h =,,...,0 for mage data set and h =,,..., for both the tehnques s 0 5.03, whh s qute expensve speeh dataset. Fgure 6. and 6. show lassfaton as ompared to LCD and other tehnques. Fgure 7. auray versus total parameters n logarthm sale and 7. show lassfaton auray vs. total and lassfaton auray versus proessng tme parameters on logarthm sale and lassfaton respetvely, usng all the tehnques on mage dataset. auray vs. Proessng tme respetvely for all the For LCD tehnque, as presented on the Fg. 6. tehnques on speeh dataset. The plottng sheme s and 6., the frst value of lassfaton auray s smlar to that appled for Fg. 6. and 6.. 8.3% at total parameter 0.636 (Fg. 6.) whh takes It s evdent from Fg. 7. and 7. that LCD tehnque s performng better than all the other proessng tme of.94 unts (Fg. 6.). The next tehnques nludng NN and knn n terms of reported value of lassfaton auray n Fg. 6. and ahevng hgher lassfaton auray at low total 6. s only those whh provde better lassfaton parameter requrement and low proessng tme. The auray than the present value,.e. Those values are lassfaton auray of NN tehnque s even poorer plotted next n the fgures whh are gven the than MDC, PCA and VQ tehnques; ths means that mprovement n lassfaton auray ompared to the nreasng total parameters does not always help n prevous value. Ths would help to desrbe that to mprovng the lassfaton auray. The maxmum aheve a ertan range of lassfaton auray what lassfaton auray for LCD tehnque s 84.% n s the total parameter requrement and ts orrespondng 0 3.670 usng 8.74 unts proessng tme, whereas the proessng tme. A smlar strategy s opted for VQPCA nearest tehnque n terms of auray s knn whh s and PCA tehnques. For VQ tehnque there are only gvng 78.3% (for k = ) n 0 5.56 usng 794.08 unts four levels and all of them are gven whh are denoted proessng tme. by,4,8 and 6 n the Fg. 6. and 6.. MDC and NN have only one value and knn has got 5 values for k = 3,5,7,9, whh s depted n the same fgures. It an be observed from the Fg. 6. and 6. that the MDC has a mnmal total parameter requrement and proessng tme but the lassfaton auray s qute poorly around 76.6%. The other tehnques wth the same total parameter requrement but wth dfferent proessng tmngs are PCA, VQ and LCD (at level ). Though the proessng tme s very low for PCA (around.53 to.99 tme unts), the performane s qute poor gvng lassfaton auray n the range of 69.4% to 73.3% whh s even lower than MDC. Wth the same total parameter requrement VQ gves muh better performane than PCA n terms of auray but the proessng tme nreases as the levels nrease towards 6. The lassfaton auray of VQPCA s Fg. 7.: Classfaton auray vs. log 0 (total qute poor at the begnnng. As the total parameter parameters) on speeh dataset requrement nreases t gves reasonably good results but at the expense of hgh proessng tme. It s evdent that LCD tehnque gves hgh lassfaton auray at low total parameter requrement and proessng tme, for e.g. t gves 85.4% auray at 0 3.033 total parameters usng only 3.00 unts proessng tme whereas the maxmum auray obtaned by VQ s 85.% at 0 3.539 total parameters usng 3.4 unts proessng tme and VQPCA gves 84.9% at 0 3.840 usng 3.8 unts proessng tme. The maxmum auray aheved by the LCD tehnque (when Q< 6 and h 0 ) s 90.0% at 0 4.580 usng 48.the NN tehnquessng tme whh s very lose to NN KNNhnque (90.3%) and lose to the maxmum of knn (for k = 3 ) tehnque (90.5%). However the proessng tme for NN and knn tehnques are 93.37 unts and Fg. 7.: Classfaton auray vs. Proessng tme on from 96.89 to 0.0 unts (for k = 3, 5, 7, 9, ) speeh dataset 453
It an be onluded from the experments on mage data set and speeh dataset that LCD tehnque outperforms MDC, PCA, VQ, VQPCA, NN and KNN tehnques n terms of gettng reasonably aepted lassfaton auray and at the same tme mantanng the mnmal total parameter requrement and proessng tme. Ths would enable the user to lassfy a gven obet aurately and qukly wth mnmal mplementaton ost. CONCLUSION A survey on bas lassfers namely MDC, VQ, PCA, NN and knn was gven. Ther lassfaton proedures were llustrated. Then we looked at VQPCA tehnque whh s normally used for representaton purposes. We showed how to use VQPCA for lassfaton purposes. However, we found that VQPCA dd not gve a very enouragng performane as a lassfer but ths gave us ntatve to develop ombned lassfers. Next we presented LCD tehnque whh s the ombnaton of VQ and VQPCA tehnques. By ombnng the lassfers we found that the performane mproved sgnfantly whh was not possble by usng ether VQ or VQPCA ndvdually. The performane of LCD tehnque s found to be better than all the other presented tehnques. Thus t an lassfy a gven obet more aurately at very low mplementaton ost and proessng tme, whh was demonstrated usng speeh and mage datasets. It was found that when the weghtng oeffent α was lose to 0.5 the LCD tehnque gave lose to optmum performane,.e. when VQ and VQPCA tehnques ontrbute equally n the deson makng of a test pattern then the performane s lose to optmum. REFERENCES. Jan, A.K., R.P.W. Dun and J. Mao, 000. Statstal pattern reognton: a revew. IEEE Trans. Pattern Anal. Mahne Intellgene, : 4-37.. Fukunaga, K., 990. Introduton to Statstal Pattern Reognton. Aadem Press In., Hartourt Brae Jovanovh, Publshers. 3. D Mao V. and F. Marano, 003. Automat lassfaton of neural spke atvty: an applaton of mnmum dstane lassfers. Cybernets and Systems, 34: 73-9. 4. Palk, P. and R.P.W. Dun, 003. Dssmlartybased lassfaton of spetra: Computatonal ssues. Real-tme Imagng, 9: 37-44. 5. Sahn, F., 000. A radal bass funton approah to a olor mage lassfaton problem n a real tme ndustral applaton. PhD Thess, State Unversty, Vrgna. Am. J. Appl. S., (0): (0): 445-455, 005 454 6. Datta, P. and D. Kbler, 997. Symbol nearest mean lassfers. Pro. of the 4th Natl. Conf. On Artfal Intellgene, San Mateo, CA, pp: 8-87. 7. Grguolo, S., 994. Pxel-by-pxel lusterng for vegetaton montorng. Intl. Conf. on Alerte préoe et suv de l'envronment, Namey, Nger. 8. Lewensten, K. and M. Chonak, 004. Mnmum dstane lassfers n oronary artery dsease dagnosng. Modellng n Mehatrons, Kazmerz Dolny, Poland. 9. Lambrou, T., A.D. Lnney, R.D. Speller and A. Todd-Pokropek, 00. Statstal lassfaton of dgtal mammograms usng features from the spatal and wavelet domans. Medal Image Understandng and Anal., Portsmouth, UK. 0. Toth, D., A. Condurahe and T. Aah, 00. A two-stage-lassfer for defet lassfaton n optal meda nspeton. 6th Intl. Conf. On Pattern Reognton (ICPR'0), 4: 373-376.. Lnde, Y., A. Buzo and R.M. Gray, 980. An algorthm for vetor quantzaton desgn. IEEE Trans. On Comm., COM-8, : 84-94.. Gray, R.M., 984. Vetor quantzaton. IEEE ASSP Magazne, pp: 4-9. 3. Wesel, R.D. and R.M. Gray, 994. Bayes rsk weghted VQ and learnng VQ. Pro. Data Compresson Conf. (DCC 94), UT, USA, pp: 400-409. 4. Potlapall, H., M.Y. Jasmha, H. Barad, A.B. Martnez, M.C. Lohrenz, J. Ryan and J. Pollard, 989. Classfaton tehnques for dgtal map ompresson. Pro. Of the st Southeastern Symp. On System Theory, Tallahassee, FL, USA, pp: 68-7. 5. Makhoul, J., S. Rouos and H. Gsh, 985. Vetor quantzaton n speeh odng. Pro. Of the IEEE, 73: 55-588. 6. Soong, F.K., A.E. Rosenberg and B. Juang, 987. A vetor quantzaton approah to speaker reognton. AT&T Tehnal Jnrl., 66: 4-6. 7. Dsung, T.P., 998. Applatons of unsupervsed lusterng algorthms for arraft dentfaton usng hgh range resoluton radar. Pro. IEEE Natonal Aerospae and Eletrons Conf., OH, USA, pp: 8-35. 8. Arvnd, R. and A. Gersho, 986. Low-rate mage odng wth fnte-state vetor quantzaton. In Pro. ICASSP pp: 37-40. 9. Oa, E., 983. Subspae Methods of Pattern Reognton. Researh Studes Press, New York. 0. Oa, E. and J. Parkknen, 984. On subspae lusterng. Seventh Intl. Conf. On Pattern Reognton, : 69-695.. Oa, E., 99. Prnpal omponents, mnor omponents and lnear neural networks. Neural Networks, 5: 97-935.. Kambhatla, N., Leen, T.K., 997. Dmensonalty reduton by loal PCA. Neural Computaton, 9: 493-56.
Am. J. Appl. S., (0): (0): 445-455, 005 3. Kambhatla, N., 995. Loal models and Gaussan mxture models for statstal data proessng. PhD Thess, Oregon Graduate Inst. Of S. and Tehnology. 4. Duda, R.O. and P.E. Hart, 973. Pattern Classfaton and Sene Analyss. John Wley and Sons, New York. 5. Sharma, A., K.K. Palwal and G.C. Onwubolu, 006. Splttng tehnque ntalzaton n loal PCA. J. Computer S., : 53-58 (n prnt). 6. Xu, L., A. KrzyŜak and C.Y. Suen, 99. Methods of ombnng multple lassfers and ther applatons to handwrtng reognton. IEEE Trans. On Systems Man. and Cybernets, : 48-435. 7. Jaobs, R.A., M.I. Jordan, S.J. Nowlan and G.E. Hnton, 99. Adaptve mxtures of loal experts. Neural Computaton, 3: 79-87. 8. Ho, T.K., J.J. Hull and S.N. Srhar, 994. Deson ombnaton n multple lassfer systems. IEEE Trans. On Pattern Anal. and Mahne Intellgene, 6: 66-75. 9. Tresp, V. and M. Tanguh, 995. Combnng estmators usng non-onstant weghtng funtons. In G. Tesauro, D.S. Touretzky, T.K. Leen (Eds). Advanes n Neural Info. Proessng Systems 7, MIT press, Cambrdge. 30. Woods, K., K. Bowyer and W.P. Kegelmeyer, 996. Combnaton of multple lassfers usng loal auray estmates. IEEE Comp. So. Conf. Computer Vson and Pattern Reognton CVPR 96, pp: 39-396. 3. Woods, K., 997. Combnaton of multple lassfers usng loal auray estmates. IEEE Trans. Pattern Anal. Mahne Intellgene, 9: 405-40. 3. Zhou, L. and S. Ima, 996. Chnese all syllables reognton usng a ombnaton of multple lassfers. ICASSP, 6: 3494-3497. 33. Almoglu, F. and E. Alpaydn, 997. Combnng multple representatons and lassfers for penbased handwrtten dgt reognton. Intl. Conf. Doument Analyss and Reognton, : 637-640. 34. Kttler, J., M. Hatef, R.P.W. Dun and J. Matas, 996. On ombnng lassfers. Intl. Conf. Pattern Reognton, : 897-90. 35. Kttler, J., M. Hatef, R.P.W. Dun and J. Matas, 998. On ombnng lassfers. IEEE Trans. Pattern Anal. Mahne Intellgene, 0: 6-39. 36. Breukelen van, M. and R.P.W. Dun, 998. Neural network ntalzaton by ombnng lassfers. Intl. Conf. Pattern Reognton, : 5-8. 37. Alexandre, L.A., A.C. Camplho and M. Kamel, 000. Combnng ndependent and unbased lassfers usng a weghted average. Intl. Conf. Pattern Reognton, : 495-498. 38. Turner, K. and J. Gosh, 999. Lnear and order statsts ombners for pattern lassfaton. In A. Sharkey, (Ed.). Combnng Artfal Neural Nets, Sprnger-Verlag, pp: 7-6. 39. Ueda, N., 000. The optmal lnear ombnaton of neural networks for mprovng lassfaton performane. IEEE Trans. Pattern Anal. Mahne Intellgene, : 07-5. 40. Senor, A., 00. A Combnaton Fngerprnt Classfer. IEEE Trans. Pattern Anal. and Mahne Intellgene, 3: 65-74. 4. Le, L., W. Xao-Long and L. Bng-Quan, 00. Combnng multple lassfers based on statstal methods for handwrtten Chnese harater reognton. Intl. Conf. Mahne Learnng and Cybernets, : 5-55. 4. Yao, M., X. Pan, T. He and R. Zhang, 00. An mproved ombnaton method of multple lassfers based on fuzzy ntegrals. World Congress on Intellgent Control and Automaton, 3: 445-447. 43. Garofalo, S.G., L.F. Lor, F.M. Wllam, F.G. Jonathan, P.S. Davd and D.L. Nany, 986. The DARPA TIMIT aoust-phonet ontnuous speeh orpus CD-ROMs. NIST. 44. Blake, C.L. and C.J. Merz, 988. UCI repostory of mahne learnng databases. http://www.s.u.edu/~mlearn, Irvne, CA, Unversty of Calf., Dept. Of Informaton and Comp. Sene. 45. Mhe, D., D.J. Spegelhalter and C.C. Taylor (Eds.), 994. Mahne Learnng, Neural and Statstal Classfaton. Ells Horwood. 46. Young, S., G. Evermann, T. Han, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valthev and P. Woodland, 00. The HTK Book Verson 3., Cambrdge, England, Cambrdge Unversty. 455