Three supervised learning methods on pen digits character recognition dataset
|
|
- Dulcie Cobb
- 6 years ago
- Views:
Transcription
1 Three supervsed learnng methods on pen dgts character recognton dataset Chrs Flezach Department of Computer Scence and Engneerng Unversty of Calforna, San Dego San Dego, CA Satoru Fukushma Department of Computer Scence and Engneerng Unversty of Calforna, San Dego San Dego, CA Introducton Supervsed learnng s a broad feld that encompasses a number of methods, whch can generally be classfced nto two categores: parametrc and nonparametrc. In the parametrc methods, t s assumed that the forms of the underlyng densty functons are known. The problem of estmatng unknown functons can be reduced to estmatng some values of parameters. In contrast, n the nonparametrc method, there s no assumpton on the form of the underlyng denstes. The parametrc category s dvded further nto two subcategores: generatve and dscrmnatve. In the generatve method, we estmate P(X Y), whch descrbes how to generate X gven Y, whle n the dscrmnatve method, we drectly estmate P(Y X). It s our goal to compare classfcaton results and characterstcs of learnng methods n the dfferent categores. Bayesan classfcaton wth a mxture of Gaussans, logstc regresson, and k nearest-neghbor classfcaton were mplemented, and ther results usng a pen dgts character recognton dataset are analyzed. 2 Bayesan classfcaton wth a mxture of Gaussans Usng mxtures of probablty densty functons for estmatng lkelhoods s a practcal technque when a class s not easly descrbed by one probablty densty functon. Such a case may arse when a class contans two centers of concentrated actvty, such as a bmodal dstrbuton. Combnng dstrbutons can offer a better approxmaton of a true modelng functon. In the project, we used a mxture of Gaussans to model each class. To generate the Gaussans parameters, the expectaton maxmzaton process was employed, and further tempered by determnstc annealng. The frst step n generatng the probablty dstrbuton was to create a covarance matrx and a mean for each class of data n the tranng set. We assumed that the devaton would not change, and were able to leave the covarance matrx stable once created. The mean
2 though was modfed to better ft the data through the teratve process of EM. Intally, a dfferent set of mean values were created for each Gaussan component by random perterbaton around the mean per class. The expectaton process then took each tranng example from that class and calculated the probablty usng all the Gaussans n the mxture model. Each probablty was also annealed by takng t s root to the tenth power, whch amed to lengthen the teratve process to acheve a truer representaton of the actual model. Each Gaussan had a weght assocated wth t and that weght was used to determne the percentage of partcpaton the Gaussan would have n determnng the probablty for the example. Then the maxmzaton step calculated new weghts and new means for each Gaussan by determnng how much partcpaton each had n the fnal answer. The process was repeated teratvely untl the values of the weghts and the means converged wth.01 of the prevous teraton. Ths number was chosen snce all values of data were between 0 and 100, and the sad convergence lmt would amount to approxmately.01% dfference n many cases. Wth the converged values of the weghts and means for each Gaussans, we then appled those to the test data to obtan the partcpaton probablty of a testng example n a mxture of Gaussans. That was not the fnal class predcton though, as Bayes theorem was used to fnd the probablty. The value from the Gaussan was used as the P(X Y) component n a Bayes equaton, where X was the testng example and Y was the class. The other numerator, P(Y) was calculated by countng classes. The denomnator, P(X), was the same for all Gaussans, so we merely needed to multply P(Y)*P(X Y) and compare t wth all classes. The maxmum of whch was used to predct whch class the data lay wthn. The most mportant queston we faced was how many component Gaussans should be used to buld the densty functon. To determne ths value, we used ten fold cross valdaton on our tranng set and averaged the accuracy results as the number of Gaussans ranged from one to ten. The results are plotted n Fgure 1. Interestngly, there s very lttle dfference n the accuracy when more than two Gaussans were used. Ths may be explaned by the fact that two Gaussans cover almost all the data n the class and addtonal Gaussans do not fnd other centers of data to ft, effectvely contrbutng zero to the total probablty for almost all data. Even though wth a larger number of Gaussans there seems to be a slghtly small uptck, we chose to use two Gaussans snce t was possble that more than two Gaussans would overft the data. Fgure 1: Accuracy vs. number of Gaussans for mxture modelng Usng two Gaussans, the accuracy for the test set was 95.88%.
3 The tme complexty of usng a mxture of models was qute small, snce the data had all been generated and thus needed to only be appled. In the case of two Gaussans, ths meant the Gaussan was appled twce for each class, for each example of data. The testng tme could be consdered to be O(c*p), where c was the number of classes and p was the number of features. To actually generate the model though, takes longer, snce t depended on a convergence process to generate the values needed for the Gaussan. Once agan, the Gaussan must be appled twce to each tranng example, for each class. But, mportantly, the values for each set of Gaussans must be teratvely recalculated untl t convergences. In practce, the convergence process took on average ten tmes, but t was a number that clearly depended on the data nvolved and choces n the startng means and weghts. 3 Logstc regresson Logstc regresson s a parametrc, dscrmnatve classfcaton algorthm, and drectly estmate P(Y X). For ths project, we used a bnary classfer for each dgt. In the tranng phase, the classfer was gven a label 1 for a sample of the dgt and 0 for other dgts. In the testng phase, each classfer output the probablty that the sample represented the dgt and was multpled by a pror of each dgt, and then the dgt whose classfer s resulted value was hghest was chosen as the fnal predcted dgt. We used the functon, fmnsearch n the optmzaton toolbox n Matlab, for maxmzng the log of condtonal lkelhood over weght coeffcents W, l(w ) = n Y n (w 0 + w X n ln(1 + exp(w 0 + w X n )) where p s the number of features and X n represents the value of the vector X for the n-th tranng sample. The classfcaton accuracy when the classfer was traned wth a whole tranng dataset and tested on the testng set was 81.85%. To mprove ts accuracy, we exploted regularzaton, whch reduces overfttng by peneralzng large values of W. The revsed log lkelhood we used was, l(w ) = n Y n (w 0 + w X n ln(1 + exp(w 0 + w X n )) λ W 2 2 where p s the number of features and X n s the value of the vector X for the n-th tranng sample. For fgurng out whch value of λ works well, we conduct 2 fold cross valdatons wth 8 dfferent λ values from 2 9, 2 8,..., and 2 3. The reason we chose 2 fold, not 10 fold and we just examned these 8 values was the tme constrant. 10 fold cross valdaton mght have produced more accurate estmaton. As Fgure 2 shows, when λ was 2 4, ts resulted accuracy was maxmzed. Hence, we used 2 4 as the value of λ. However, the accuracy on a whole testng set was deterorated to 79.56% from 81.85% wthout any regularzaton. To fgure out the best value on the testng set, we further used the same 8 λ values on the testng set, and Fgure 3 shows, the best accuracy, 82.42%, was produced when λ was 2 8. There are possble reasons for the relatvely poor accuracy. The frst one s that we used 2 fold cross valdatons, not 10 fold, whch mght make ts estmaton deterorated. The second s that the range of used λ values was lmted. It mght be possble that other λ values would produce more accurate predctons. Both of these two reasons were caused manly by the neffcency of fmnsearch algorthm, that s, t took too much tme to converge. In addton, the fact we needed to termnate before the convergence of fmnsearch algorthm s another possble reason for ths result.
4 Fgure 2: Accuracy vs. dfferent values of λ on cross valdatons 4 K nearest-neghbor classfcaton K-nearest neghbor classfcaton algorthms explctly gnore parametrc modelng n decdng whch class a data pont les wthn. Ths has the effect of performng a hard classfcaton on each data pont, wthout the ablty to nuance and massage parameters to tune to specfc problems. The basc dea n K nearest neghbors s that for an nstance x of the testng set, the dstance between all tranng ponts s calculated. The dstance functon s defned as the Eucldean dstance, whch has the convenent property of workng for any dmensonal set of data. In our case, the data had 16 features, so the dfference between each feature was taken and squared, summed wth all other features and the square root was taken. Wth the dstance calculated between each pont, the class pluralty s taken of closest k neghbors. Although the complexty of the algorthmc s qute lmted, t s remarkably accurate for certan sets of data dependng on the degree of k whch s chosen. The tme complexty of the algorthm s an unfortunate drawback of usng k-nearest neghbors. Although there s no tranng phase per se, each data pont from the test set must be compared separately wth every value n the tranng set. That number mght be reduced through samplng, f the tranng set s too large, but that may not be desrable n many stuatons. If we say n s the tranng sze, p s the number of features and k s the number of neghbors, then the runnng tme s O(n,p,k) = ( n*p *k + k ). The frst component of the sum, n*p*k, s the tme requred to calculate the dstance between every tranng example, whch then has to be compared aganst the top k neghbors to determne f t s a closer neghbor then exstng neghbors. The last term of the sum s the tme requred to count the class that had a pluralty amongst all neghbors. As k s usually qute small, the formula mght better be wrtten as O(n,p) = n * p. To effectvely choose whch k should be employed, 10 fold cross valdaton was used on the tranng set. Thus, each tenth of the tranng set was used as a test set, whle the remanng porton was used to determne class membershp. The accuracy was averaged over each cross valdaton run. Ths was done for all k from 1 to 20. The best results were obtaned when k=1. There was a notceable declne n accuracy as k ncreased, ndcatng
5 Fgure 3: Accuracy vs. dfferent values of λ on the testng set that more neghbors were not better, most lkely because the classes were relatvely close to each other n the Eucldean sense. When more neghbors were used, more classes were brought nto the equaton whch affected the overall predcton. Fgure 4 demonstrates the deteroratng qualty as k ncreased. Fgure 4: Accuracy vs. number of neghbors for K-nearest neghbors Usng k=1 on the entre tranng set and the testng set resulted n an accuracy of 97.86%. 5 Dscusson 5.1 Accuracy Table 1 shows the comparsons between the three classfcaton algorthms when they were traned wth a whole tranng dataset and tested on the entre testng set. As for accuracy, the K nearest neghbor algorthm produced the best result among three. Bayesan classf-
6 caton wth a mxture Gaussans was close to t, and logstc regresson was the worst. The possble reason for the poor performance of logstc regresson was due to the numercal optmzaton method used, fmnsearch functon. It tred to fnd the coeffcent values whch mnmzed a functon value, but t ddn t converge wthn a small number of teratons. So, wth the relatvely lmted tme we had, we needed to termnate t the maxmum number of functon evaluatons, 3400, whch was the default value of MaxFunEvals n optons for fmnsearch n Matlab. If t had run longer, ts accuracy could have been mproved. Bayesan w/mx Gauss Logstc regresson K nearest neghbor Accuracy 95.88% 82.42% 97.86% Tme for tranng N.A. N.A. 0 Tme for testng O(c*p) O(c*p) O(p*n) Space O(c*p 2 ) O(c*p) O(p*n) Table 1: Comparsons between the three classfcaton methods (c s the number of class, p s the number of features, and n s the number of tranng data.) 5.2 Tme complexty The k nearest neghbor algorthm doesn t requre a tranng phase, but takes a long tme n ts testng phase snce t needs to examne all data ponts. On the other hand, Bayesan classfcaton wth a mxture of Gaussans and logstc regresson both must be traned, but then they can conduct testng much faster than k nearest neghbor. Whle n the tranng of the Bayesan classfer, the estmated parameter values converged relatvely quckly, the tranng for the logstc regresson took a much longer tme snce fmnsearch, the Matlab functon used for numercal optmzaton, was not effcent. A more effcent method such as teratve reweghted least squares would reduce ts tme complexty. Tme complextes are shown n Table 1, where c s the number of classes, p s the number of features, and n s the number of tranng examples. For logstc regresson and Bayesan classfcaton wth a mxture of Gaussans, we could not provde a closed form bg-o notaton due to the convergence propertes of both algorthms. 5.3 Space complexty Snce the k nearest neghbor algorthm needs to examne all data ponts when a new example s classfed, all data needs to be stored. Its space complexty s represented as O(p*n), where p s the number of features and n s the number of tranng examples. On the other hand, the Bayesan method and logstc regresson only need to store several values of parameters. In the Bayesan method, the space complexty was O(c*(j*p + p 2 )), where j was the number of Gaussans and p was the number of features. In ths nstance, p represents the array of mean values for each Gaussan and p 2 represents the covarance matrx for each class. In logstc regresson, the space complexty was O(c*p), where c s the number of classes and p s the number of features. The latter two algorthms space complexty should always be much smaller than O(p*n), the space complexty for k nearest neghbors, snce the sze of the tranng dataset should always be much larger than the other parameters. 5.4 Characterstcs of the classfers As mentoned n the subsecton about accuracy above, the nferor performance of logstc regresson mght be caused by early termnaton of the numercal optmzaton functon wthout t havng truly converged. Wth ths n mnd, we dscuss several characterstcs of the classfcaton algorthms.
7 The accuracy of k nearest neghbor was the best among three. The observaton can best be explaned by the flexblty the algorthm has n examnng other neghbors. Whle choosng whch value of k worked best, we notced ts accuracy deterorated as k ncreased, as seen n 4. Ths ndcates that for the dataset, the best predctor of an example s class was fndng the class of another example whch had nearly the same values for each feature. In contrast, the other two classfcaton algorthms do not have ths flexblty and are forced nto usng parameters amed at coverng the entre range of examples. Even n the case of the mxture of components, selectng the example that s closest may do better for a varety of reasons. For example, the nfluence of nearby Gaussans from other classes may overrde a Gaussan component whch has lttle weght wthn t s own class. Although the k nearest neghbor algorthm performed the best, t s man drawback s that t takes much longer to classfy a test example compared to the other two methods. Ths characterstc prohbts the k nearest neghbor algorthm from beng used n certan knds of applcatons whch requre classfcaton n realtme. Between the two parametrc methods, the number of parameters are dfferent. For the Bayesan, there are O(c*(j*p + p 2 )), whle logstc regresson only had O(c*p). Hence, when the number of features, p, s large, logstc regresson may be preferred. 6 Concluson Expermentaton wth the three methods of classfcaton revealed a number of nsghts. As the number of features ncreasng, the classfcaton problem becomes ntractable n many respects. The prevous dataset had nearly 800 features and could not be used n many formulas wthout overflow or underflow and, specfcally, n logstc regresson, convergence tme took too long to be useful. The power of the conceptually smplstc k nearest neghbor model was a surprse and demonstrated that for many datasets, a smpler approach may be just as vald as a parametrc approach. Even more nterestng was that as more neghbors were used, the accuracy actually decreased. One mght be led to beleve that as more data was examned, the accuracy would rse correspondngly, snce a better nformed judgement could be made.
Support Vector Machines
/9/207 MIST.6060 Busness Intellgence and Data Mnng What are Support Vector Machnes? Support Vector Machnes Support Vector Machnes (SVMs) are supervsed learnng technques that analyze data and recognze patterns.
More informationOutline. Type of Machine Learning. Examples of Application. Unsupervised Learning
Outlne Artfcal Intellgence and ts applcatons Lecture 8 Unsupervsed Learnng Professor Danel Yeung danyeung@eee.org Dr. Patrck Chan patrckchan@eee.org South Chna Unversty of Technology, Chna Introducton
More informationUnsupervised Learning
Pattern Recognton Lecture 8 Outlne Introducton Unsupervsed Learnng Parametrc VS Non-Parametrc Approach Mxture of Denstes Maxmum-Lkelhood Estmates Clusterng Prof. Danel Yeung School of Computer Scence and
More informationFeature Reduction and Selection
Feature Reducton and Selecton Dr. Shuang LIANG School of Software Engneerng TongJ Unversty Fall, 2012 Today s Topcs Introducton Problems of Dmensonalty Feature Reducton Statstc methods Prncpal Components
More informationCS 534: Computer Vision Model Fitting
CS 534: Computer Vson Model Fttng Sprng 004 Ahmed Elgammal Dept of Computer Scence CS 534 Model Fttng - 1 Outlnes Model fttng s mportant Least-squares fttng Maxmum lkelhood estmaton MAP estmaton Robust
More informationUser Authentication Based On Behavioral Mouse Dynamics Biometrics
User Authentcaton Based On Behavoral Mouse Dynamcs Bometrcs Chee-Hyung Yoon Danel Donghyun Km Department of Computer Scence Department of Computer Scence Stanford Unversty Stanford Unversty Stanford, CA
More informationClassifier Selection Based on Data Complexity Measures *
Classfer Selecton Based on Data Complexty Measures * Edth Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad Natonal Insttute for Astrophyscs, Optcs and Electroncs, Lus Enrque Erro No.1 Sta.
More informationMachine Learning. Topic 6: Clustering
Machne Learnng Topc 6: lusterng lusterng Groupng data nto (hopefully useful) sets. Thngs on the left Thngs on the rght Applcatons of lusterng Hypothess Generaton lusters mght suggest natural groups. Hypothess
More informationUnsupervised Learning and Clustering
Unsupervsed Learnng and Clusterng Why consder unlabeled samples?. Collectng and labelng large set of samples s costly Gettng recorded speech s free, labelng s tme consumng 2. Classfer could be desgned
More informationLearning the Kernel Parameters in Kernel Minimum Distance Classifier
Learnng the Kernel Parameters n Kernel Mnmum Dstance Classfer Daoqang Zhang 1,, Songcan Chen and Zh-Hua Zhou 1* 1 Natonal Laboratory for Novel Software Technology Nanjng Unversty, Nanjng 193, Chna Department
More informationMathematics 256 a course in differential equations for engineering students
Mathematcs 56 a course n dfferental equatons for engneerng students Chapter 5. More effcent methods of numercal soluton Euler s method s qute neffcent. Because the error s essentally proportonal to the
More informationSLAM Summer School 2006 Practical 2: SLAM using Monocular Vision
SLAM Summer School 2006 Practcal 2: SLAM usng Monocular Vson Javer Cvera, Unversty of Zaragoza Andrew J. Davson, Imperal College London J.M.M Montel, Unversty of Zaragoza. josemar@unzar.es, jcvera@unzar.es,
More informationMULTISPECTRAL IMAGES CLASSIFICATION BASED ON KLT AND ATR AUTOMATIC TARGET RECOGNITION
MULTISPECTRAL IMAGES CLASSIFICATION BASED ON KLT AND ATR AUTOMATIC TARGET RECOGNITION Paulo Quntlano 1 & Antono Santa-Rosa 1 Federal Polce Department, Brasla, Brazl. E-mals: quntlano.pqs@dpf.gov.br and
More informationBiostatistics 615/815
The E-M Algorthm Bostatstcs 615/815 Lecture 17 Last Lecture: The Smplex Method General method for optmzaton Makes few assumptons about functon Crawls towards mnmum Some recommendatons Multple startng ponts
More informationSubspace clustering. Clustering. Fundamental to all clustering techniques is the choice of distance measure between data points;
Subspace clusterng Clusterng Fundamental to all clusterng technques s the choce of dstance measure between data ponts; D q ( ) ( ) 2 x x = x x, j k = 1 k jk Squared Eucldean dstance Assumpton: All features
More informationBOOSTING CLASSIFICATION ACCURACY WITH SAMPLES CHOSEN FROM A VALIDATION SET
1 BOOSTING CLASSIFICATION ACCURACY WITH SAMPLES CHOSEN FROM A VALIDATION SET TZU-CHENG CHUANG School of Electrcal and Computer Engneerng, Purdue Unversty, West Lafayette, Indana 47907 SAUL B. GELFAND School
More informationEECS 730 Introduction to Bioinformatics Sequence Alignment. Luke Huan Electrical Engineering and Computer Science
EECS 730 Introducton to Bonformatcs Sequence Algnment Luke Huan Electrcal Engneerng and Computer Scence http://people.eecs.ku.edu/~huan/ HMM Π s a set of states Transton Probabltes a kl Pr( l 1 k Probablty
More informationAnnouncements. Supervised Learning
Announcements See Chapter 5 of Duda, Hart, and Stork. Tutoral by Burge lnked to on web page. Supervsed Learnng Classfcaton wth labeled eamples. Images vectors n hgh-d space. Supervsed Learnng Labeled eamples
More informationNAG Fortran Library Chapter Introduction. G10 Smoothing in Statistics
Introducton G10 NAG Fortran Lbrary Chapter Introducton G10 Smoothng n Statstcs Contents 1 Scope of the Chapter... 2 2 Background to the Problems... 2 2.1 Smoothng Methods... 2 2.2 Smoothng Splnes and Regresson
More informationMachine Learning. Support Vector Machines. (contains material adapted from talks by Constantin F. Aliferis & Ioannis Tsamardinos, and Martin Law)
Machne Learnng Support Vector Machnes (contans materal adapted from talks by Constantn F. Alfers & Ioanns Tsamardnos, and Martn Law) Bryan Pardo, Machne Learnng: EECS 349 Fall 2014 Support Vector Machnes
More informationCHAPTER 3 SEQUENTIAL MINIMAL OPTIMIZATION TRAINED SUPPORT VECTOR CLASSIFIER FOR CANCER PREDICTION
48 CHAPTER 3 SEQUENTIAL MINIMAL OPTIMIZATION TRAINED SUPPORT VECTOR CLASSIFIER FOR CANCER PREDICTION 3.1 INTRODUCTION The raw mcroarray data s bascally an mage wth dfferent colors ndcatng hybrdzaton (Xue
More informationInvestigating the Performance of Naïve- Bayes Classifiers and K- Nearest Neighbor Classifiers
Journal of Convergence Informaton Technology Volume 5, Number 2, Aprl 2010 Investgatng the Performance of Naïve- Bayes Classfers and K- Nearest Neghbor Classfers Mohammed J. Islam *, Q. M. Jonathan Wu,
More informationA Statistical Model Selection Strategy Applied to Neural Networks
A Statstcal Model Selecton Strategy Appled to Neural Networks Joaquín Pzarro Elsa Guerrero Pedro L. Galndo joaqun.pzarro@uca.es elsa.guerrero@uca.es pedro.galndo@uca.es Dpto Lenguajes y Sstemas Informátcos
More informationSupport Vector Machines
Support Vector Machnes Decson surface s a hyperplane (lne n 2D) n feature space (smlar to the Perceptron) Arguably, the most mportant recent dscovery n machne learnng In a nutshell: map the data to a predetermned
More informationExercises (Part 4) Introduction to R UCLA/CCPR. John Fox, February 2005
Exercses (Part 4) Introducton to R UCLA/CCPR John Fox, February 2005 1. A challengng problem: Iterated weghted least squares (IWLS) s a standard method of fttng generalzed lnear models to data. As descrbed
More informationCS246: Mining Massive Datasets Jure Leskovec, Stanford University
CS46: Mnng Massve Datasets Jure Leskovec, Stanford Unversty http://cs46.stanford.edu /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, http://cs46.stanford.edu Perceptron: y = sgn( x Ho to fnd
More informationS1 Note. Basis functions.
S1 Note. Bass functons. Contents Types of bass functons...1 The Fourer bass...2 B-splne bass...3 Power and type I error rates wth dfferent numbers of bass functons...4 Table S1. Smulaton results of type
More information6.854 Advanced Algorithms Petar Maymounkov Problem Set 11 (November 23, 2005) With: Benjamin Rossman, Oren Weimann, and Pouya Kheradpour
6.854 Advanced Algorthms Petar Maymounkov Problem Set 11 (November 23, 2005) Wth: Benjamn Rossman, Oren Wemann, and Pouya Kheradpour Problem 1. We reduce vertex cover to MAX-SAT wth weghts, such that the
More informationData Mining: Model Evaluation
Data Mnng: Model Evaluaton Aprl 16, 2013 1 Issues: Evaluatng Classfcaton Methods Accurac classfer accurac: predctng class label predctor accurac: guessng value of predcted attrbutes Speed tme to construct
More informationProblem Definitions and Evaluation Criteria for Computational Expensive Optimization
Problem efntons and Evaluaton Crtera for Computatonal Expensve Optmzaton B. Lu 1, Q. Chen and Q. Zhang 3, J. J. Lang 4, P. N. Suganthan, B. Y. Qu 6 1 epartment of Computng, Glyndwr Unversty, UK Faclty
More informationImplementation Naïve Bayes Algorithm for Student Classification Based on Graduation Status
Internatonal Journal of Appled Busness and Informaton Systems ISSN: 2597-8993 Vol 1, No 2, September 2017, pp. 6-12 6 Implementaton Naïve Bayes Algorthm for Student Classfcaton Based on Graduaton Status
More informationComplex Numbers. Now we also saw that if a and b were both positive then ab = a b. For a second let s forget that restriction and do the following.
Complex Numbers The last topc n ths secton s not really related to most of what we ve done n ths chapter, although t s somewhat related to the radcals secton as we wll see. We also won t need the materal
More informationLecture 5: Multilayer Perceptrons
Lecture 5: Multlayer Perceptrons Roger Grosse 1 Introducton So far, we ve only talked about lnear models: lnear regresson and lnear bnary classfers. We noted that there are functons that can t be represented
More informationMachine Learning 9. week
Machne Learnng 9. week Mappng Concept Radal Bass Functons (RBF) RBF Networks 1 Mappng It s probably the best scenaro for the classfcaton of two dataset s to separate them lnearly. As you see n the below
More informationA Robust Method for Estimating the Fundamental Matrix
Proc. VIIth Dgtal Image Computng: Technques and Applcatons, Sun C., Talbot H., Ourseln S. and Adraansen T. (Eds.), 0- Dec. 003, Sydney A Robust Method for Estmatng the Fundamental Matrx C.L. Feng and Y.S.
More informationCluster Analysis of Electrical Behavior
Journal of Computer and Communcatons, 205, 3, 88-93 Publshed Onlne May 205 n ScRes. http://www.scrp.org/ournal/cc http://dx.do.org/0.4236/cc.205.350 Cluster Analyss of Electrcal Behavor Ln Lu Ln Lu, School
More informationLecture 4: Principal components
/3/6 Lecture 4: Prncpal components 3..6 Multvarate lnear regresson MLR s optmal for the estmaton data...but poor for handlng collnear data Covarance matrx s not nvertble (large condton number) Robustness
More informationSmoothing Spline ANOVA for variable screening
Smoothng Splne ANOVA for varable screenng a useful tool for metamodels tranng and mult-objectve optmzaton L. Rcco, E. Rgon, A. Turco Outlne RSM Introducton Possble couplng Test case MOO MOO wth Game Theory
More informationSimulation: Solving Dynamic Models ABE 5646 Week 11 Chapter 2, Spring 2010
Smulaton: Solvng Dynamc Models ABE 5646 Week Chapter 2, Sprng 200 Week Descrpton Readng Materal Mar 5- Mar 9 Evaluatng [Crop] Models Comparng a model wth data - Graphcal, errors - Measures of agreement
More informationAn Optimal Algorithm for Prufer Codes *
J. Software Engneerng & Applcatons, 2009, 2: 111-115 do:10.4236/jsea.2009.22016 Publshed Onlne July 2009 (www.scrp.org/journal/jsea) An Optmal Algorthm for Prufer Codes * Xaodong Wang 1, 2, Le Wang 3,
More informationClassifying Acoustic Transient Signals Using Artificial Intelligence
Classfyng Acoustc Transent Sgnals Usng Artfcal Intellgence Steve Sutton, Unversty of North Carolna At Wlmngton (suttons@charter.net) Greg Huff, Unversty of North Carolna At Wlmngton (jgh7476@uncwl.edu)
More informationHermite Splines in Lie Groups as Products of Geodesics
Hermte Splnes n Le Groups as Products of Geodescs Ethan Eade Updated May 28, 2017 1 Introducton 1.1 Goal Ths document defnes a curve n the Le group G parametrzed by tme and by structural parameters n the
More informationContent Based Image Retrieval Using 2-D Discrete Wavelet with Texture Feature with Different Classifiers
IOSR Journal of Electroncs and Communcaton Engneerng (IOSR-JECE) e-issn: 78-834,p- ISSN: 78-8735.Volume 9, Issue, Ver. IV (Mar - Apr. 04), PP 0-07 Content Based Image Retreval Usng -D Dscrete Wavelet wth
More informationX- Chart Using ANOM Approach
ISSN 1684-8403 Journal of Statstcs Volume 17, 010, pp. 3-3 Abstract X- Chart Usng ANOM Approach Gullapall Chakravarth 1 and Chaluvad Venkateswara Rao Control lmts for ndvdual measurements (X) chart are
More informationThe Greedy Method. Outline and Reading. Change Money Problem. Greedy Algorithms. Applications of the Greedy Strategy. The Greedy Method Technique
//00 :0 AM Outlne and Readng The Greedy Method The Greedy Method Technque (secton.) Fractonal Knapsack Problem (secton..) Task Schedulng (secton..) Mnmum Spannng Trees (secton.) Change Money Problem Greedy
More informationNUMERICAL SOLVING OPTIMAL CONTROL PROBLEMS BY THE METHOD OF VARIATIONS
ARPN Journal of Engneerng and Appled Scences 006-017 Asan Research Publshng Network (ARPN). All rghts reserved. NUMERICAL SOLVING OPTIMAL CONTROL PROBLEMS BY THE METHOD OF VARIATIONS Igor Grgoryev, Svetlana
More informationA New Approach For the Ranking of Fuzzy Sets With Different Heights
New pproach For the ankng of Fuzzy Sets Wth Dfferent Heghts Pushpnder Sngh School of Mathematcs Computer pplcatons Thapar Unversty, Patala-7 00 Inda pushpndersnl@gmalcom STCT ankng of fuzzy sets plays
More informationOutline. Self-Organizing Maps (SOM) US Hebbian Learning, Cntd. The learning rule is Hebbian like:
Self-Organzng Maps (SOM) Turgay İBRİKÇİ, PhD. Outlne Introducton Structures of SOM SOM Archtecture Neghborhoods SOM Algorthm Examples Summary 1 2 Unsupervsed Hebban Learnng US Hebban Learnng, Cntd 3 A
More informationCSCI 5417 Information Retrieval Systems Jim Martin!
CSCI 5417 Informaton Retreval Systems Jm Martn! Lecture 11 9/29/2011 Today 9/29 Classfcaton Naïve Bayes classfcaton Ungram LM 1 Where we are... Bascs of ad hoc retreval Indexng Term weghtng/scorng Cosne
More informationWishing you all a Total Quality New Year!
Total Qualty Management and Sx Sgma Post Graduate Program 214-15 Sesson 4 Vnay Kumar Kalakband Assstant Professor Operatons & Systems Area 1 Wshng you all a Total Qualty New Year! Hope you acheve Sx sgma
More informationUnderstanding K-Means Non-hierarchical Clustering
SUNY Albany - Techncal Report 0- Understandng K-Means Non-herarchcal Clusterng Ian Davdson State Unversty of New York, 1400 Washngton Ave., Albany, 105. DAVIDSON@CS.ALBANY.EDU Abstract The K-means algorthm
More informationCourse Introduction. Algorithm 8/31/2017. COSC 320 Advanced Data Structures and Algorithms. COSC 320 Advanced Data Structures and Algorithms
Course Introducton Course Topcs Exams, abs, Proects A quc loo at a few algorthms 1 Advanced Data Structures and Algorthms Descrpton: We are gong to dscuss algorthm complexty analyss, algorthm desgn technques
More informationBAYESIAN MULTI-SOURCE DOMAIN ADAPTATION
BAYESIAN MULTI-SOURCE DOMAIN ADAPTATION SHI-LIANG SUN, HONG-LEI SHI Department of Computer Scence and Technology, East Chna Normal Unversty 500 Dongchuan Road, Shangha 200241, P. R. Chna E-MAIL: slsun@cs.ecnu.edu.cn,
More informationImproving Low Density Parity Check Codes Over the Erasure Channel. The Nelder Mead Downhill Simplex Method. Scott Stransky
Improvng Low Densty Party Check Codes Over the Erasure Channel The Nelder Mead Downhll Smplex Method Scott Stransky Programmng n conjuncton wth: Bors Cukalovc 18.413 Fnal Project Sprng 2004 Page 1 Abstract
More informationy and the total sum of
Lnear regresson Testng for non-lnearty In analytcal chemstry, lnear regresson s commonly used n the constructon of calbraton functons requred for analytcal technques such as gas chromatography, atomc absorpton
More informationThe Codesign Challenge
ECE 4530 Codesgn Challenge Fall 2007 Hardware/Software Codesgn The Codesgn Challenge Objectves In the codesgn challenge, your task s to accelerate a gven software reference mplementaton as fast as possble.
More informationDetermining the Optimal Bandwidth Based on Multi-criterion Fusion
Proceedngs of 01 4th Internatonal Conference on Machne Learnng and Computng IPCSIT vol. 5 (01) (01) IACSIT Press, Sngapore Determnng the Optmal Bandwdth Based on Mult-crteron Fuson Ha-L Lang 1+, Xan-Mn
More informationProper Choice of Data Used for the Estimation of Datum Transformation Parameters
Proper Choce of Data Used for the Estmaton of Datum Transformaton Parameters Hakan S. KUTOGLU, Turkey Key words: Coordnate systems; transformaton; estmaton, relablty. SUMMARY Advances n technologes and
More informationCollaboratively Regularized Nearest Points for Set Based Recognition
Academc Center for Computng and Meda Studes, Kyoto Unversty Collaboratvely Regularzed Nearest Ponts for Set Based Recognton Yang Wu, Mchhko Mnoh, Masayuk Mukunok Kyoto Unversty 9/1/013 BMVC 013 @ Brstol,
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 15
CS434a/541a: Pattern Recognton Prof. Olga Veksler Lecture 15 Today New Topc: Unsupervsed Learnng Supervsed vs. unsupervsed learnng Unsupervsed learnng Net Tme: parametrc unsupervsed learnng Today: nonparametrc
More informationMachine Learning. K-means Algorithm
Macne Learnng CS 6375 --- Sprng 2015 Gaussan Mture Model GMM pectaton Mamzaton M Acknowledgement: some sldes adopted from Crstoper Bsop Vncent Ng. 1 K-means Algortm Specal case of M Goal: represent a data
More informationParallel Numerics. 1 Preconditioning & Iterative Solvers (From 2016)
Technsche Unverstät München WSe 6/7 Insttut für Informatk Prof. Dr. Thomas Huckle Dpl.-Math. Benjamn Uekermann Parallel Numercs Exercse : Prevous Exam Questons Precondtonng & Iteratve Solvers (From 6)
More informationHybridization of Expectation-Maximization and K-Means Algorithms for Better Clustering Performance
BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 16, No 2 Sofa 2016 Prnt ISSN: 1311-9702; Onlne ISSN: 1314-4081 DOI: 10.1515/cat-2016-0017 Hybrdzaton of Expectaton-Maxmzaton
More informationAdaptive Transfer Learning
Adaptve Transfer Learnng Bn Cao, Snno Jaln Pan, Yu Zhang, Dt-Yan Yeung, Qang Yang Hong Kong Unversty of Scence and Technology Clear Water Bay, Kowloon, Hong Kong {caobn,snnopan,zhangyu,dyyeung,qyang}@cse.ust.hk
More informationOutline. Discriminative classifiers for image recognition. Where in the World? A nearest neighbor recognition example 4/14/2011. CS 376 Lecture 22 1
4/14/011 Outlne Dscrmnatve classfers for mage recognton Wednesday, Aprl 13 Krsten Grauman UT-Austn Last tme: wndow-based generc obect detecton basc ppelne face detecton wth boostng as case study Today:
More informationA Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems
A Unfed Framework for Semantcs and Feature Based Relevance Feedback n Image Retreval Systems Ye Lu *, Chunhu Hu 2, Xngquan Zhu 3*, HongJang Zhang 2, Qang Yang * School of Computng Scence Smon Fraser Unversty
More information5 The Primal-Dual Method
5 The Prmal-Dual Method Orgnally desgned as a method for solvng lnear programs, where t reduces weghted optmzaton problems to smpler combnatoral ones, the prmal-dual method (PDM) has receved much attenton
More informationTerm Weighting Classification System Using the Chi-square Statistic for the Classification Subtask at NTCIR-6 Patent Retrieval Task
Proceedngs of NTCIR-6 Workshop Meetng, May 15-18, 2007, Tokyo, Japan Term Weghtng Classfcaton System Usng the Ch-square Statstc for the Classfcaton Subtask at NTCIR-6 Patent Retreval Task Kotaro Hashmoto
More informationHelsinki University Of Technology, Systems Analysis Laboratory Mat Independent research projects in applied mathematics (3 cr)
Helsnk Unversty Of Technology, Systems Analyss Laboratory Mat-2.08 Independent research projects n appled mathematcs (3 cr) "! #$&% Antt Laukkanen 506 R ajlaukka@cc.hut.f 2 Introducton...3 2 Multattrbute
More informationEmpirical Distributions of Parameter Estimates. in Binary Logistic Regression Using Bootstrap
Int. Journal of Math. Analyss, Vol. 8, 4, no. 5, 7-7 HIKARI Ltd, www.m-hkar.com http://dx.do.org/.988/jma.4.494 Emprcal Dstrbutons of Parameter Estmates n Bnary Logstc Regresson Usng Bootstrap Anwar Ftranto*
More informationEdge Detection in Noisy Images Using the Support Vector Machines
Edge Detecton n Nosy Images Usng the Support Vector Machnes Hlaro Gómez-Moreno, Saturnno Maldonado-Bascón, Francsco López-Ferreras Sgnal Theory and Communcatons Department. Unversty of Alcalá Crta. Madrd-Barcelona
More informationFuzzy Logic Based RS Image Classification Using Maximum Likelihood and Mahalanobis Distance Classifiers
Research Artcle Internatonal Journal of Current Engneerng and Technology ISSN 77-46 3 INPRESSCO. All Rghts Reserved. Avalable at http://npressco.com/category/jcet Fuzzy Logc Based RS Image Usng Maxmum
More information12/2/2009. Announcements. Parametric / Non-parametric. Case-Based Reasoning. Nearest-Neighbor on Images. Nearest-Neighbor Classification
Introducton to Artfcal Intellgence V22.0472-001 Fall 2009 Lecture 24: Nearest-Neghbors & Support Vector Machnes Rob Fergus Dept of Computer Scence, Courant Insttute, NYU Sldes from Danel Yeung, John DeNero
More informationFast Feature Value Searching for Face Detection
Vol., No. 2 Computer and Informaton Scence Fast Feature Value Searchng for Face Detecton Yunyang Yan Department of Computer Engneerng Huayn Insttute of Technology Hua an 22300, Chna E-mal: areyyyke@63.com
More informationThe Research of Support Vector Machine in Agricultural Data Classification
The Research of Support Vector Machne n Agrcultural Data Classfcaton Le Sh, Qguo Duan, Xnmng Ma, Me Weng College of Informaton and Management Scence, HeNan Agrcultural Unversty, Zhengzhou 45000 Chna Zhengzhou
More informationA Background Subtraction for a Vision-based User Interface *
A Background Subtracton for a Vson-based User Interface * Dongpyo Hong and Woontack Woo KJIST U-VR Lab. {dhon wwoo}@kjst.ac.kr Abstract In ths paper, we propose a robust and effcent background subtracton
More informationIncremental MQDF Learning for Writer Adaptive Handwriting Recognition 1
200 2th Internatonal Conference on Fronters n Handwrtng Recognton Incremental MQDF Learnng for Wrter Adaptve Handwrtng Recognton Ka Dng, Lanwen Jn * School of Electronc and Informaton Engneerng, South
More informationOnline Detection and Classification of Moving Objects Using Progressively Improving Detectors
Onlne Detecton and Classfcaton of Movng Objects Usng Progressvely Improvng Detectors Omar Javed Saad Al Mubarak Shah Computer Vson Lab School of Computer Scence Unversty of Central Florda Orlando, FL 32816
More informationParameter estimation for incomplete bivariate longitudinal data in clinical trials
Parameter estmaton for ncomplete bvarate longtudnal data n clncal trals Naum M. Khutoryansky Novo Nordsk Pharmaceutcals, Inc., Prnceton, NJ ABSTRACT Bvarate models are useful when analyzng longtudnal data
More informationOn Some Entertaining Applications of the Concept of Set in Computer Science Course
On Some Entertanng Applcatons of the Concept of Set n Computer Scence Course Krasmr Yordzhev *, Hrstna Kostadnova ** * Assocate Professor Krasmr Yordzhev, Ph.D., Faculty of Mathematcs and Natural Scences,
More informationAnalysis of Continuous Beams in General
Analyss of Contnuous Beams n General Contnuous beams consdered here are prsmatc, rgdly connected to each beam segment and supported at varous ponts along the beam. onts are selected at ponts of support,
More informationRelated-Mode Attacks on CTR Encryption Mode
Internatonal Journal of Network Securty, Vol.4, No.3, PP.282 287, May 2007 282 Related-Mode Attacks on CTR Encrypton Mode Dayn Wang, Dongda Ln, and Wenlng Wu (Correspondng author: Dayn Wang) Key Laboratory
More informationTsinghua University at TAC 2009: Summarizing Multi-documents by Information Distance
Tsnghua Unversty at TAC 2009: Summarzng Mult-documents by Informaton Dstance Chong Long, Mnle Huang, Xaoyan Zhu State Key Laboratory of Intellgent Technology and Systems, Tsnghua Natonal Laboratory for
More informationVirtual Memory. Background. No. 10. Virtual Memory: concept. Logical Memory Space (review) Demand Paging(1) Virtual Memory
Background EECS. Operatng System Fundamentals No. Vrtual Memory Prof. Hu Jang Department of Electrcal Engneerng and Computer Scence, York Unversty Memory-management methods normally requres the entre process
More informationCompiler Design. Spring Register Allocation. Sample Exercises and Solutions. Prof. Pedro C. Diniz
Compler Desgn Sprng 2014 Regster Allocaton Sample Exercses and Solutons Prof. Pedro C. Dnz USC / Informaton Scences Insttute 4676 Admralty Way, Sute 1001 Marna del Rey, Calforna 90292 pedro@s.edu Regster
More informationArtificial Intelligence (AI) methods are concerned with. Artificial Intelligence Techniques for Steam Generator Modelling
Artfcal Intellgence Technques for Steam Generator Modellng Sarah Wrght and Tshldz Marwala Abstract Ths paper nvestgates the use of dfferent Artfcal Intellgence methods to predct the values of several contnuous
More informationSVM-based Learning for Multiple Model Estimation
SVM-based Learnng for Multple Model Estmaton Vladmr Cherkassky and Yunqan Ma Department of Electrcal and Computer Engneerng Unversty of Mnnesota Mnneapols, MN 55455 {cherkass,myq}@ece.umn.edu Abstract:
More informationTECHNIQUE OF FORMATION HOMOGENEOUS SAMPLE SAME OBJECTS. Muradaliyev A.Z.
TECHNIQUE OF FORMATION HOMOGENEOUS SAMPLE SAME OBJECTS Muradalyev AZ Azerbajan Scentfc-Research and Desgn-Prospectng Insttute of Energetc AZ1012, Ave HZardab-94 E-mal:aydn_murad@yahoocom Importance of
More informationLearning-Based Top-N Selection Query Evaluation over Relational Databases
Learnng-Based Top-N Selecton Query Evaluaton over Relatonal Databases Lang Zhu *, Wey Meng ** * School of Mathematcs and Computer Scence, Hebe Unversty, Baodng, Hebe 071002, Chna, zhu@mal.hbu.edu.cn **
More informationReducing Frame Rate for Object Tracking
Reducng Frame Rate for Object Trackng Pavel Korshunov 1 and We Tsang Oo 2 1 Natonal Unversty of Sngapore, Sngapore 11977, pavelkor@comp.nus.edu.sg 2 Natonal Unversty of Sngapore, Sngapore 11977, oowt@comp.nus.edu.sg
More informationSome Advanced SPC Tools 1. Cumulative Sum Control (Cusum) Chart For the data shown in Table 9-1, the x chart can be generated.
Some Advanced SP Tools 1. umulatve Sum ontrol (usum) hart For the data shown n Table 9-1, the x chart can be generated. However, the shft taken place at sample #21 s not apparent. 92 For ths set samples,
More informationDeep Classification in Large-scale Text Hierarchies
Deep Classfcaton n Large-scale Text Herarches Gu-Rong Xue Dkan Xng Qang Yang 2 Yong Yu Dept. of Computer Scence and Engneerng Shangha Jao-Tong Unversty {grxue, dkxng, yyu}@apex.sjtu.edu.cn 2 Hong Kong
More informationClassification Based Mode Decisions for Video over Networks
Classfcaton Based Mode Decsons for Vdeo over Networks Deepak S. Turaga and Tsuhan Chen Advanced Multmeda Processng Lab Tranng data for Inter-Intra Decson Inter-Intra Decson Regons pdf 6 5 6 5 Energy 4
More informationFast Sparse Gaussian Processes Learning for Man-Made Structure Classification
Fast Sparse Gaussan Processes Learnng for Man-Made Structure Classfcaton Hang Zhou Insttute for Vson Systems Engneerng, Dept Elec. & Comp. Syst. Eng. PO Box 35, Monash Unversty, Clayton, VIC 3800, Australa
More informationRandom Kernel Perceptron on ATTiny2313 Microcontroller
Random Kernel Perceptron on ATTny233 Mcrocontroller Nemanja Djurc Department of Computer and Informaton Scences, Temple Unversty Phladelpha, PA 922, USA nemanja.djurc@temple.edu Slobodan Vucetc Department
More informationA MOVING MESH APPROACH FOR SIMULATION BUDGET ALLOCATION ON CONTINUOUS DOMAINS
Proceedngs of the Wnter Smulaton Conference M E Kuhl, N M Steger, F B Armstrong, and J A Jones, eds A MOVING MESH APPROACH FOR SIMULATION BUDGET ALLOCATION ON CONTINUOUS DOMAINS Mark W Brantley Chun-Hung
More informationSequential search. Building Java Programs Chapter 13. Sequential search. Sequential search
Sequental search Buldng Java Programs Chapter 13 Searchng and Sortng sequental search: Locates a target value n an array/lst by examnng each element from start to fnsh. How many elements wll t need to
More informationImprovement of Spatial Resolution Using BlockMatching Based Motion Estimation and Frame. Integration
Improvement of Spatal Resoluton Usng BlockMatchng Based Moton Estmaton and Frame Integraton Danya Suga and Takayuk Hamamoto Graduate School of Engneerng, Tokyo Unversty of Scence, 6-3-1, Nuku, Katsuska-ku,
More informationParallelism for Nested Loops with Non-uniform and Flow Dependences
Parallelsm for Nested Loops wth Non-unform and Flow Dependences Sam-Jn Jeong Dept. of Informaton & Communcaton Engneerng, Cheonan Unversty, 5, Anseo-dong, Cheonan, Chungnam, 330-80, Korea. seong@cheonan.ac.kr
More informationJournal of Chemical and Pharmaceutical Research, 2014, 6(6): Research Article. A selective ensemble classification method on microarray data
Avalable onlne www.ocpr.com Journal of Chemcal and Pharmaceutcal Research, 2014, 6(6):2860-2866 Research Artcle ISSN : 0975-7384 CODEN(USA) : JCPRC5 A selectve ensemble classfcaton method on mcroarray
More information