Preface to the Second Edition. Preface to the First Edition. 1 Introduction 1

Transcription

1 Preface to the Second Edition Preface to the First Edition vii xi 1 Introduction 1 2 Overview of Supervised Learning Introduction Variable Types and Terminology Two Simple Approaches to Prediction: LeastSquaresandNearestNeighbors Linear Models and Least Squares Nearest-Neighbor Methods From Least Squares to Nearest Neighbors Statistical Decision Theory LocalMethodsinHighDimensions Statistical Models, Supervised Learning and Function Approximation A Statistical Model for the Joint Distribution Pr(X, Y ) Supervised Learning Function Approximation StructuredRegressionModels Difficulty of the Problem... 32

2 xiv 2.8 Classes of Restricted Estimators Roughness Penalty and Bayesian Methods Kernel Methods and Local Regression Basis Functions and Dictionary Methods Model Selection and the Bias Variance Tradeoff Bibliographic Notes Exercises Linear Methods for Regression Introduction LinearRegressionModelsandLeastSquares Example: Prostate Cancer The Gauss Markov Theorem Multiple Regression from Simple Univariate Regression Multiple Outputs Subset Selection Best-Subset Selection Forward- and Backward-Stepwise Selection Forward-Stagewise Regression Prostate Cancer Data Example (Continued) ShrinkageMethods Ridge Regression The Lasso Discussion: Subset Selection, Ridge Regression andthelasso Least Angle Regression Methods Using Derived Input Directions Principal Components Regression Partial Least Squares Discussion: A Comparison of the Selection andshrinkagemethods Multiple Outcome Shrinkage and Selection More on the Lasso and Related Path Algorithms Incremental Forward Stagewise Regression Piecewise-Linear Path Algorithms The Dantzig Selector The Grouped Lasso Further Properties of the Lasso Pathwise Coordinate Optimization Computational Considerations Bibliographic Notes Exercises... 94

3 xv 4 Linear Methods for Classification Introduction Linear Regression of an Indicator Matrix Linear Discriminant Analysis Regularized Discriminant Analysis Computations for LDA Reduced-Rank Linear Discriminant Analysis Logistic Regression Fitting Logistic Regression Models Example: South African Heart Disease Quadratic Approximations and Inference L 1 Regularized Logistic Regression Logistic Regression or LDA? Separating Hyperplanes Rosenblatt s Perceptron Learning Algorithm Optimal Separating Hyperplanes Bibliographic Notes Exercises Basis Expansions and Regularization Introduction Piecewise Polynomials and Splines Natural Cubic Splines Example: South African Heart Disease (Continued) Example: Phoneme Recognition Filtering and Feature Extraction SmoothingSplines Degrees of Freedom and Smoother Matrices Automatic Selection of the Smoothing Parameters Fixing the Degrees of Freedom The Bias Variance Tradeoff Nonparametric Logistic Regression Multidimensional Splines Regularization and Reproducing Kernel Hilbert Spaces Spaces of Functions Generated by Kernels Examples of RKHS WaveletSmoothing Wavelet Bases and the Wavelet Transform Adaptive Wavelet Filtering Bibliographic Notes Exercises Appendix: Computational Considerations for Splines Appendix: B-splines Appendix: Computations for Smoothing Splines

4 xvi 6 Kernel Smoothing Methods One-Dimensional Kernel Smoothers Local Linear Regression Local Polynomial Regression SelectingtheWidthoftheKernel Local Regression in IR p Structured Local Regression Models in IR p Structured Kernels Structured Regression Functions LocalLikelihoodandOtherModels Kernel Density Estimation and Classification Kernel Density Estimation Kernel Density Classification The Naive Bayes Classifier Radial Basis Functions and Kernels Mixture Models for Density Estimation and Classification Computational Considerations Bibliographic Notes Exercises Model Assessment and Selection Introduction Bias, Variance and Model Complexity The Bias Variance Decomposition Example: Bias Variance Tradeoff Optimism of the Training Error Rate Estimates of In-Sample Prediction Error TheEffectiveNumberofParameters TheBayesianApproachandBIC Minimum Description Length Vapnik Chervonenkis Dimension Example (Continued) Cross-Validation K-Fold Cross-Validation The Wrong and Right Way to Do Cross-validation Does Cross-Validation Really Work? Bootstrap Methods Example (Continued) Conditional or Expected Test Error? Bibliographic Notes Exercises Model Inference and Averaging Introduction

5 xvii 8.2 TheBootstrapandMaximumLikelihoodMethods A Smoothing Example Maximum Likelihood Inference Bootstrap versus Maximum Likelihood BayesianMethods Relationship Between the Bootstrap and Bayesian Inference The EM Algorithm Two-Component Mixture Model The EM Algorithm in General EM as a Maximization Maximization Procedure MCMCforSamplingfromthePosterior Bagging Example: Trees with Simulated Data Model Averaging and Stacking StochasticSearch:Bumping Bibliographic Notes Exercises Additive Models, Trees, and Related Methods Generalized Additive Models Fitting Additive Models Example: Additive Logistic Regression Summary Tree-Based Methods Background Regression Trees Classification Trees Other Issues Spam Example (Continued) PRIM:BumpHunting Spam Example (Continued) MARS: Multivariate Adaptive Regression Splines Spam Example (Continued) Example (Simulated Data) Other Issues HierarchicalMixturesofExperts MissingData Computational Considerations Bibliographic Notes Exercises Boosting and Additive Trees Boosting Methods Outline of This Chapter

6 xviii 10.2 Boosting Fits an Additive Model Forward Stagewise Additive Modeling Exponential Loss and AdaBoost Why Exponential Loss? Loss Functions and Robustness Off-the-Shelf Procedures for Data Mining Example: Spam Data Boosting Trees Numerical Optimization via Gradient Boosting Steepest Descent Gradient Boosting Implementations of Gradient Boosting Right-Sized Trees for Boosting Regularization Shrinkage Subsampling Interpretation Relative Importance of Predictor Variables Partial Dependence Plots Illustrations California Housing New Zealand Fish Demographics Data Bibliographic Notes Exercises Neural Networks Introduction Projection Pursuit Regression Neural Networks Fitting Neural Networks Some Issues in Training Neural Networks Starting Values Overfitting Scaling of the Inputs Number of Hidden Units and Layers Multiple Minima Example: Simulated Data Example: ZIP Code Data Discussion Bayesian Neural Nets and the NIPS 2003 Challenge Bayes, Boosting and Bagging Performance Comparisons Computational Considerations Bibliographic Notes

7 xix Exercises Support Vector Machines and Flexible Discriminants Introduction The Support Vector Classifier Computing the Support Vector Classifier Mixture Example (Continued) Support Vector Machines and Kernels Computing the SVM for Classification The SVM as a Penalization Method Function Estimation and Reproducing Kernels SVMs and the Curse of Dimensionality A Path Algorithm for the SVM Classifier Support Vector Machines for Regression Regression and Kernels Discussion Generalizing Linear Discriminant Analysis Flexible Discriminant Analysis Computing the FDA Estimates Penalized Discriminant Analysis Mixture Discriminant Analysis Example: Waveform Data Bibliographic Notes Exercises Prototype Methods and Nearest-Neighbors Introduction Prototype Methods K-meansClustering Learning Vector Quantization Gaussian Mixtures k-nearest-neighborclassifiers Example: A Comparative Study Example: k-nearest-neighbors and Image Scene Classification Invariant Metrics and Tangent Distance Adaptive Nearest-Neighbor Methods Example Global Dimension Reduction fornearest-neighbors Computational Considerations Bibliographic Notes Exercises

8 xx 14 Unsupervised Learning Introduction Association Rules Market Basket Analysis The Apriori Algorithm Example: Market Basket Analysis Unsupervised as Supervised Learning Generalized Association Rules Choice of Supervised Learning Method Example: Market Basket Analysis (Continued) Cluster Analysis Proximity Matrices Dissimilarities Based on Attributes Object Dissimilarity Clustering Algorithms Combinatorial Algorithms K-means Gaussian Mixtures as Soft K-means Clustering Example: Human Tumor Microarray Data Vector Quantization K-medoids Practical Issues Hierarchical Clustering Self-Organizing Maps Principal Components, Curves and Surfaces Principal Components Principal Curves and Surfaces Spectral Clustering Kernel Principal Components Sparse Principal Components Non-negative Matrix Factorization Archetypal Analysis Independent Component Analysis and Exploratory Projection Pursuit Latent Variables and Factor Analysis Independent Component Analysis Exploratory Projection Pursuit A Direct Approach to ICA Multidimensional Scaling Nonlinear Dimension Reduction and Local Multidimensional Scaling The Google PageRank Algorithm Bibliographic Notes Exercises

9 xxi 15 Random Forests Introduction Definition of Random Forests Details of Random Forests Out of Bag Samples Variable Importance Proximity Plots Random Forests and Overfitting Analysis of Random Forests Variance and the De-Correlation Effect Bias Adaptive Nearest Neighbors Bibliographic Notes Exercises Ensemble Learning Introduction Boosting and Regularization Paths Penalized Regression The Bet on Sparsity Principle Regularization Paths, Over-fitting and Margins Learning Ensembles Learning a Good Ensemble Rule Ensembles Bibliographic Notes Exercises Undirected Graphical Models Introduction Markov Graphs and Their Properties Undirected Graphical Models for Continuous Variables Estimation of the Parameters whenthegraphstructureisknown Estimation of the Graph Structure Undirected Graphical Models for Discrete Variables Estimation of the Parameters whenthegraphstructureisknown Hidden Nodes Estimation of the Graph Structure Restricted Boltzmann Machines Exercises High-Dimensional Problems: p N When p is Much Bigger than N

10 xxii 18.2 Diagonal Linear Discriminant Analysis andnearestshrunkencentroids Linear Classifiers with Quadratic Regularization Regularized Discriminant Analysis Logistic Regression with Quadratic Regularization The Support Vector Classifier Feature Selection Computational Shortcuts When p N Linear Classifiers with L 1 Regularization Application of Lasso toproteinmassspectroscopy The Fused Lasso for Functional Data Classification When Features are Unavailable Example: String Kernels and Protein Classification Classification and Other Models Using Inner-Product Kernels and Pairwise Distances Example: Abstracts Classification High-Dimensional Regression: Supervised Principal Components Connection to Latent-Variable Modeling Relationship with Partial Least Squares Pre-Conditioning for Feature Selection Feature Assessment and the Multiple-Testing Problem The False Discovery Rate Asymmetric Cutpoints and the SAM Procedure A Bayesian Interpretation of the FDR Bibliographic Notes Exercises References 699 Author Index 729 Index 737

11