Bing Liu. Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data. With 177 Figures. Springer

Transcription

1 Bing Liu Web Data Mining Exploring Hyperlinks, Contents, and Usage Data With 177 Figures Springer

2 Table of Contents 1. Introduction What is the World Wide Web? A Brief History of the Web and the Internet Web Data Mining What is Data Mining? What is Web Mining? Summary of Chapters How to Read this Book 11 Bibliographie Notes 12 Part I: Data Mining Foundations 2. Association Ruies and Sequential Patterns Basic Concepts of Association Ruies Apriori Algorithm Frequent Itemset Generation Association Rule. Generation Data Formats for Association Rule Mining Mining with Multiple Minimum Supports Extended Model Mining Algorithm Rule Generation Mining Class Association Ruies Problem Definition Mining Algorithm Mining with Multiple Minimum Supports 37

3 XII Table of Contents 2.6. Basic Concepts of Sequential Patterns Mining Sequential Patterns Based on GSP GSP Algorithm Mining with Multiple Minimum Supports Mining Sequential Patterns Based on PrefixSpan PrefixSpan Algorithm Mining with Multiple Minimum Supports Generating Ruies from Sequential Patterns Sequential Ruies Label Sequential Ruies Class Sequential Ruies 51 Bibliographie Notes Supervised Learning Basic Concepts Decision Tree Induction Learning Algorithm Impurity Function Handling of Continuous Attributes Some Other Issues Classifier Evaluation Evaluation Methods Precision, Recall, F-score and Breakeven Point Rule Induction Sequential Covering Rule Learning: Learn-One-Rule Function Discussion Classification Based on Associations Classification Using Class Association Ruies Class Association Ruies as Features Classification Using Normal Association Ruies Naive Bayesian Classification Naive Bayesian Text Classification Probabilistic Framework Naive Bayesian Model Discussion Support Vector Machines Linear SVM: Separable Case 99

4 Table of Contents XIII Linear SVM: Non-Separable Case Nonlinear SVM: Kernel Functions K-Nearest Neighbor Learning Ensemble of Classifiers Bagging Boosting 114 Bibliographie Notes 115 Unsupervised Learning Basic Concepts K-means Clustering K-means Algorithm Disk Version of the K-means Algorithm Strengths and Weaknesses Representation of Clusters Common Ways of Representing Clusters Clusters of Arbitrary Shapes Hierarchical Clustering Single-Link Method Complete-Link Method Average-Link Method Strengths and Weaknesses Distance Functions Numeric Attributes Binary and Nominal Attributes Text Documents Data Standardization Handling of Mixed Attributes Which Clustering Algorithm to Use? Cluster Evaluation Discovering Holesand Data Regions 146 Bibliographie Notes 149 Partially Supervised Learning Learning from Labeled and Unlabeied Examples EM Algorithm with NaTve Bayesian Classification 153

5 XIV Table of Contents Co-Training Self-Training Transductive Support Vector Machines Graph-Based Methods Discussion Leaming from Positive and Unlabeied Examples Applications of PL) Learning Theoretical Foundation Building Classifiers: Two-Step Approach Building Classifiers: Direct Approach Discussion 178 Appendix: Derivation of EM for Naive Bayesian Classification -179 Bibliographie Notes 181 Part II: Web Mining 6. Information Retrieval and Web Search Basic Concepts of Information Retrieval Information Retrieval Models Boolean Model Vector Space Model Statistical Language Model Relevance Feedback Evaluation Measures Text and Web Page Pre-Processing Stopword Removal Stemming Other Pre-Processing Tasks for Text Web Page Pre-Processing Duplicate Detection Inverted Index and Its Compression Inverted Index Search Using an Inverted Index Index Construction Index Compression 209

6 Table of Contents XV 6.7. Latent Semantic Indexing Singular Value Decomposition Query and Retrieval An Example Discussion WebSearch Meta-Search: Combining Multiple Rankings Combination Using Similarity Scores Combination Using Rank Positions Web Spamming Content Spamming Link Spamming Hiding Techniques Combating Spam 234 Bibliographie Notes 235 Link Analysis Social Network Analysis Centrality Prestige Co-Citation and Bibliographie Coupling Co-Citation Bibliographie Coupling PageRank PageRank Algorithm Strengths and Weaknesses of PageRank Timed PageRank HITS HITS Algorithm Finding Other Eigenvectors Relationships with Co-Citation and Bibliographie Coupling Strengths and Weaknesses of HITS Community Discovery Problem Definition Bipartite Core Communities Maximum Flow Communities Communities Based on Betweenness Overlapping Communities of Named Entities 270

7 XVI Table of Contents Bibliographie Notes Web Crawling A Basic Crawler Algorithm Breadth-First Crawlers Preferential Crawlers Implementation Issues Fetching Parsing Stopword Removal and Stemming Link Extraction and Canonicalization Spider Traps Page Repository Concurrency Universal Crawlers Scalability Coverage vs Freshness vs Importance Focused Crawlers Topical Crawlers Topical Locality and Cues Best-First Variations Adaptation Evaluation Crawler Ethics and Conflicts Some New Developments 318 Bibliographie Notes Structured Data Extraction: Wrapper Generation Preliminaries Two Types of Data Rieh Pages Data Model HTML Mark-Up Encoding of Data Instances Wrapper Induction Extraction from a Page Learning Extraction Rules Identifying Informative Examples Wrapper Maintenance 338

8 Table of Contents XVII 9.3. Instance-Based Wrapper Learning Automatic Wrapper Generation: Problems Two Extraction Problems Patterns as Regulär Expressions String Matching and Tree Matching String Edit Distance Tree Matching Multiple Alignment Center Star Method Partial Tree Alignment Building DOM Trees Extraction Based on a Single List Page: Fiat Data Records Two Observations about Data Records Mining Data Regions Identifying Data Records in Data Regions Data Item Alignment and Extraction Making Use of Visual Information Some Other Techniques Extraction Based on a Single List Page: Nested Data Records Extraction Based on Multiple Pages Using Techniques in Previous Sections RoadRunner Algorithm Some Other Issues Extraction from Other Pages Disjunction or Optional A Set Type or a Tuple Type Labeling and Integration Domain Specific Extraction Discussion 379 Bibliographie Notes Information Integration Introduction to Schema Matching Pre-Processing for Schema Matching Schema-Level Match 385

9 XVIII Table of Contents Linguistic Approaches Constraint Based Approaches Domain and Instance-Level Matching Combining Similarities :m Match Some Other Issues Reuse of Previous Match Results Matching a Large Number of Schemas Schema Match Results User Interactions Integration of Web Query Interfaces A Clustering Based Approach A Correlation Based Approach An Instance Based Approach Constructing a Unified Global Query Interface Structural Appropriateness and the Merge Algorithm Lexical Appropriateness Instance Appropriateness 409 Bibliographie Notes Opinion Mining Sentiment Classification Classification Based on Sentiment Phrases Classification Using Text Classification Methods Classification Using a Score Function Feature-Based Opinion Mining and Summarization Problem Definition Object Feature Extraction Feature Extraction from Pros and Cons of Format Feature Extraction from Reviews of of Formats 2 and Opinion Orientation Classification Comparative Sentence and Relation Mining Problem Definition Identification of Gradable Comparative Sentences 435

10 Table of Contents XIX Extraction of Comparative Relations Opinion Search Opinion Spam Objectives and Actions of Opinion Spamming Types of Spam and Spammers Hiding Techniques Spam Detection 444 Bibliographie Notes Web Usage Mining Data Collection and Pre-Processing Sources and Types of Data Key Elements of Web Usage Data Pre-Processing Data Modeling for Web Usage Mining Discovery and Analysis of Web Usage Patterns Session and Visitor Analysis Cluster Analysis and Visitor Segmentation Association and Correlation Analysis Analysis of Sequential and Navigational Patterns Classification and Prediction Based on Web User Transactions Discussion and Outlook 482 Bibliographie Notes 482 References 485 Index 517