CONTENTS IN DETAIL PART I AN INTRODUCTION TO SPAM FILTERING INTRODUCTION 1 THE HISTORY OF SPAM 3 2 HISTORICAL APPROACHES TO FIGHTING SPAM 25

Transcription

1 CONTENTS IN DETAIL INTRODUCTION xvii PART I AN INTRODUCTION TO SPAM FILTERING 1 THE HISTORY OF SPAM 3 The Definition of Spam... 4 The Very First Spam... 4 Spam: The Early Years... 7 Jay-Jay s College Fund... 7 The Jesus Spam... 9 Canter & Siegel Cancelmoose Jeff Slaton, the Spam King Krazy Kevin Lipsitz Stanford Wallace, Cyber Promotions Floodgate The First Spamware Other Significant Events in War Waged on Spam Spamhaus Unsolicited Commercial Spam Out of Control , 1999, and 2000: Three Years of War on Spam Network Solutions to the Present: Exponential Spam Growth Final Thoughts HISTORICAL APPROACHES TO FIGHTING SPAM 25 Primitive Language Analysis Blacklisting Propagation and Maintenance Problems Heuristic Filtering Brightmail SpamAssassin Drawbacks to Heuristic Filtering Maintenance Headaches Scoring... 32

2 Whitelisting A Little Too Effective Forgeries Challenge/Response Problems with Challenge/Response Throttling TarProxy Other Throttling Tools Collaborative Filtering Address Obfuscation New Standards Authenticated SMTP Sender Policy Framework Litigation Spammer Fingerprinting Intellectual Property Final Thoughts LANGUAGE CLASSIFICATION CONCEPTS 45 Understanding Accuracy Machine Learning Concept Learning Using Language Classification to Fight Spam Training Statistical Filtering and Bayesian Analysis Components of a Language Classifier The Historical Dataset The Tokenizer The Analysis Engine Providing Feedback Training Train-Everything (TEFT) Train-on-Error (TOE) Train-Until-Mature (TUM) Train-Until-No-Errors (TUNE) When to Train An Example of a Filter Instance Step 1: Tokenize the Message Step 2: Build a Decision Matrix Step 3: Evaluate the Decision Matrix Step 4: Train the Message Step 5: Correct Errors Efficacy of Statistical Filtering The Future of Language Classification The Sovereignty of Statistical Filtering Final Thoughts x

3 4 STATISTICAL FILTERING FUNDAMENTALS 63 An Imperfect Solution Building a Historical Dataset Corpus Feeding Starting from Scratch Correcting Errors The Tokenizer and Calculating Token Values Single-Corpus Tokens A Biased Filter Hapaxes Final Product The Analysis Engine Sorting Statistical Combination Bayesian Combination (Paul Graham) Bayesian Combination (Brian Burton) Robinson s Geometric Mean Test Fisher-Robinson s Inverse Chi-Square Improvements to Statistical Analysis Improving the Decision Matrix Improvements to Tokenization Statistical Sedation Iterative Training Learning New Tricks Final Thoughts PART II FUNDAMENTALS OF STATISTICAL FILTERING 5 DECODING: UNCOMBOBULATING MESSAGES 87 Introduction to Encoding Decoding Message Body Encodings Quoted-Printable Encoding Base64 Encoding Custom Encodings Message Header Encodings HTML Encodings Message Actualization Supporting Software Final Thoughts xi

4 6 TOKENIZATION: THE BUILDING BLOCKS OF SPAM 97 Tokenizing a Heuristic Function Basic Delimiters Redundancy Other Delimiters Exceptions Token Reassembly Degeneration Header Optimizations URL Optimizations HTML Tokenization Word Pairs Sparse Binary Polynomial Hashing Internationalization Final Thoughts THE LOW-DOWN DIRTY TRICKS OF SPAMMERS 111 Successful Filtering No More Headaches A Weak Link in Statistical Filters? Attacks on Tokenizers Encoding Abuses Header Encodings Hypertextus Interruptus ASCII Spam Text-Splitting Table-Based Obfuscation URL Encodings Symbolic Text Just Plain Dumb Attacks on the Dataset Mailing List Attacks Bayesian Poisoning Empty but Not Empty Probes Attacks on the Decision Matrix Image Spams Random Strings of Text Word Salad Directed Attacks Final Thoughts DATA STORAGE FOR A ZILLION RECORDS 141 Storage Considerations Disk Space Speed xii

5 Locking Portability Statefulness Recovery I/O Contention Random-Access Features Ease of Use Storage Framework Third-Party Storage Solutions Stateless Database Implementations Stateful SQL-Based Solutions Peter Graf s PBL ISAM Library SQLite Proprietary Implementations Final Thoughts SCALING IN LARGE ENVIRONMENTS 157 Requirements Assessment Total Disk Space Requirements Total Processing Power Parallelization versus Serialization Operating System Requirements High Availability I/O Bandwidth Requirements Features End-User Support Sizing Machine Capacity General Resource Planning Assessing Resource Utilization Building a Distributed Model Round-Robin Distributed Networking Distributed BGP Networking Final Thoughts PART III ADVANCED CONCEPTS OF STATISTICAL FILTERING 10 TESTING THEORY 177 The Challenge of Testing Message Continuity Archive Window Purge Simulation Interleave Corrective Training Delay xiii

6 Types of Simulations Measuring the Accuracy of a Specific Filter Test Criteria Performing the Test Measuring Adaptation in Chaotic Environments Test Criteria Performing the Test Testing the Effectiveness of Multiple Filters Test Criteria Performing the Test Comparing Features in a Single Filter Test Criteria Performing the Test Testing Caveats Corrective Training Purge Simulations Test Messages Presuppositions Final Thoughts CONCEPT IDENTIFICATION: ADVANCED TOKENIZATION 197 Chained Tokens Case Study Analysis Pattern Identification Differentiation HTML Classification Contextual Analysis Other Uses Administrative Concerns Supporting Data Summary Sparse Binary Polynomial Hashing Supporting Data Summary Karnaugh Mapping Final Thoughts FIFTH-ORDER MARKOVIAN DISCRIMINATION 215 Markov s Great Advance Hidden Markov Models (HMMs) Using Markov Models to Model Text Classic Bayesian Spam Filter Bayesian versus Markovian Classification Storage Concerns Purging Old Data xiv

7 Floating-Point Renormalization and Underflow Final Thoughts INTELLIGENT FEATURE SET REDUCTION 227 Calibration Algorithms Bayesian Noise Reduction (BNR) Instantiation Phase Training Phase Dubbing Phase Examples End Result Efficacy Final Thoughts COLLABORATIVE ALGORITHMS 241 Message Inoculation Supporting Data External Inoculation Classification Groups Collaborative Neural Meshes Neural Declustering Machine-Automated Blacklists Streamlined Blackhole List Weighted Private Block List Distributed Attacks Filters That Fight Back Fingerprinting Probing Automatic Whitelisting URL Blacklisting Minefields Final Thoughts APPENDIX SHINING EXAMPLES OF FILTERING 257 POPFile: The POP3 Proxy About POPFile Accuracy Interview with the Author SpamProbe: A Modified Approach About SpamProbe Accuracy Interview with the Author xv

8 TarProxy: IANA Spam Filter About TarProxy Accuracy Interview with the Author DSPAM: A Large-Scale Filter About DSPAM Accuracy Interview with the Author The CRM114 Discriminator About CRM Under the Hood Accuracy Interview with the Author INDEX 275 xvi