Logging and Log Management

Transcription

1 Logging and Log Management The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management Dr. Anton A. Chuvakin Kevin J. Schmidt Christopher Phillips Partricia Moulder, Technical Editor AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINCAPORE SYDNEY TOKYO ELSEVIER Syngress is an Imprint of Elsevier

2 Contents ACKNOWLEDGMENTS ABOUT THE AUTHORS ABOUT THE TECHNICAL EDITOR FOREWORD PREFACE xv xvii xix xxi xxv CHAPTER 1 Logs, Trees, Forest: The Big Picture 1 Introduction 1 Log Data Basics 2 What Is Log Data? 2 How is Log Data Transmitted and Collected? 4 What is a Log Message? 6 The Logging Ecosystem 7 A Look at Things to Come 15 Logs Are Underrated 16 Logs Can Be Useful 17 Resource Management 17 Intrusion Detection 18 Troubleshooting 21 Forensics 21 Boring Audit, Fun Discovery 22 People, Process, Technology 23 Security Information and Event Management (SIEM) 24 Summary 27 CHAPTER 2 What is a Log? 29 Introduction 29 Definitions 29 Logs? What logs? 32 Log Formats and Types 34 Log Syntax 40 Log Content 44

3 Criteria of Good Logging 46 Ideal Logging Scenario 47 Summary 48 CHAPTER 3 Log Data Sources 51 Introduction 51 Logging Sources 51 Syslog 52 SNMP 58 The Windows Event Log 62 Log Source Classification 63 Security-Related Host Logs 64 Security-Related Network Logs 68 Security Host Logs 68 Summary 70 CHAPTER 4 Log Storage Technologies 71 Introduction 71 Log Retention Policy 71 Log Storage Formats 73 Text-Based Log Files 73 Binary Files 76 Compressed Files 76 Database Storage of Log Data 78 Advantages 78 Disadvantages 78 Defining Database Storage Goals 79 Hadoop Log Storage 81 Advantages 82 Disadvantages 82 The Cloud and Hadoop 82 Getting Started with Amazon Elastic MapReduce 83 Navigating the Amazon 83 Uploading Logs to Amazon Simple Storage Services (S3) 84 Create a Pig Script to Analyze an Apache Access Log 86 Processing Log Data in Amazon Elastic MapReduce (EMR) 87 Log Data Retrieval and Archiving 89 Online 90

4 Contents Near-line 90 Offline 90 Summary 90 CHAPTER 5 syslog-ng Case Study 93 Introduction 93 Obtaining syslog-ng 93 What Is syslog-ngsyslog-ng? 94 Example Deployment 95 Configurations 96 Troubleshooting syslog-ng 99 Summary 101 CHAPTER 6 Covert Logging 103 Introduction 103 Complete Stealthy Log Setup 105 Stealthy Log Generation 105 Stealthy Pickup of Logs 106 IDS Log Source 106 Log Collection Server 107 "Fake" Server or Honeypot 109 Logging in Honeypots 110 Honeynet's Shell Covert Keystroke Logger Ill Honeynet's Sebek2 Case Study 112 Covert Channels for Logging Brief 113 Summary 114 CHAPTER 7 Analysis Goals, Planning, and Preparation: What Are We Looking For? 115 Introduction 115 Goals 115 Past Bad Things 115 Future Bad Things, Never Before Seen Things, and All But the Known Good Things 117 Planning 117 Accuracy 117 Integrity 118 Confidence 119 Preservation 119 Sanitization 120 Normalization 120 Challenges with Time 121

5 i. Contents Preparation 122 Separating Log Messages 122 Parsing 122 Data Reduction 122 Summary 125 CHAPTER 8 Simple Analysis Techniques 127 Introduction 127 Line by Line: Road to Despair 127 Simple Log Viewers 129 Real-Time Review 129 Historical Log Review 130 Simple Log Manipulation 131 Limitations of Manual Log Review 134 Responding to the Results of Analysis 135 Acting on Critical Logs 135 Acting on Summaries of Non-Critical Logs 137 Developing an Action Plan 138 Automated Actions 140 Examples 140 Incident Response Scenario 140 Routine Log Review 141 Summary 142 CHAPTER 9 Filtering, Normalization, and Correlation 145 Introduction 145 Filtering 147 Artificial Ignorance 147 Normalization 148 IP Address Validation 150 Snort 150 Windows Snare 150 Generic Cisco IOS Messages 151 Regular Expression Performance Concerns 152 Correlation 154 Micro-Level Correlation 155 Macro-Level Correlation 157 Using Data in Your Environment 161 Simple Event Correlator (SEC) 161 Stateful Rule Example 163 Building Your Own Rules Engine 169 Common Patterns to Look For 178

6 Contents I The Future 178 Summary 180 CHAPTER 10 Statistical Analysis 181 Introduction 181 Frequency 181 Baseline 182 Thresholds 186 Anomaly Detection 186 Windowing 187 Machine Learning 187 k-nearest Neighbor (knn) 188 Applying the k-nn Algorithm to Logs 188 Combining Statistical Analysis with Rules-Based Correlation 190 Summary 191 CHAPTER 11 Log Data Mining 193 Introduction 193 Data Mining Intro 194 Log Mining Intro 198 Log Mining Requirements 200 What We Mine For? 201 Deeper into Interesting 203 Summary 205 CHAPTER 12 Reporting and Summarization 207 Introduction 207 Defining the Best Reports 208 Authentication and Authorization Reports 208 Network Activity Reports 211 Why They Are Important 211 Specifics Reports 212 Who Can Use These Reports 213 Resource Access Reports 213 Why They Are Important 213 Specifics Reports 213 Who Can Use These Reports 214 Malware Activity Reports 215 Why They Are Important 215 Specific Reports 215 Who Can Use These Reports 216

7 Critical Errors and Failures Reports 216 Why They Are Important 216 Specifics Reports 216 Who Can Use These Reports 217 Summary 217 CHAPTER 13 Visualizing Log Data 219 Introduction 219 Visual Correlation 219 Real-Time Visualization 220 Treemaps 221 Log Data Constellations 222 Traditional Log Data Graphing 227 Summary 229 CHAPTER 14 Logging Laws and Logging Mistakes 231 Introduction 231 Logging Laws 231 Law 1 Law of Collection 232 Law 2 Law of Retention 232 Law 3 Law of Monitoring 233 Law 3 Law of Availability 233 Law 4 Law of Security 233 Law 5 Law of Constant Changes 234 Logging Mistakes 234 Not Logging at All 235 Not Looking at Log Data 236 Storing for Too Short a Time 237 Prioritizing Before Collection 239 Ignoring Application Logs 240 Only Looking for Known Bad Entries 241 Summary 241 CHAPTER 15 Tools for Log Analysis and Collection 243 Introduction 243 Outsource, Build, or Buy 243 Building a Solution 244 Buy 245 Outsource 246 Questions for You, Your Organization, and Vendors 246 Basic Tools for Log Analysis 247 Grep 247 Awk 249

8 Contents Microsoft Log Parser 251 Other Basic Tools to Consider 252 The Role of the Basic Tools in Log Analysis 254 Utilities for Centralizing Log Information 254 Syslog 254 Rsyslog 256 Snare 256 Log Analysis Tools Beyond the Basics 257 OSSEC 257 OSSIM 261 Other Analysis Tools to Consider 261 Commercial Vendors 262 Splunk 263 NetlQ Sentinel 264 IBM qllabs 264 Loggly 265 Summary 265 CHAPTER 16 Log Management Procedures: Log Review, Response, and Escalation 267 Introduction 267 Assumptions, Requirements, and Precautions 268 Requirements 269 Precautions 269 Common Roles and Responsibilities 269 PCI and Log Data 270 Key Requirement Other Requirements Related to Logging 275 Logging Policy 277 Review, Response, and Escalation Procedures and Workflows 278 Periodic Log Review Practices and Patterns 279 Building an Initial Baseline Using a Log Management Tool 283 Building an Initial Baseline Manually 285 Main Workflow: Daily Log Review 286 Exception Investigation and Analysis 289 Incident Response and Escalation 291 Validation of Log Review 293 Proof of Logging 294 Proof of Log Review 294 Proof of Exception Handling 294

9 Logbook Evidence of Exception of Investigations 296 Recommended Logbook Format 296 Example Logbook Entry 297 PCI Compliance Evidence Package 299 Management Reporting 300 Periodic Operational Tasks 300 Daily Tasks 300 Weekly Tasks 300 Monthly Tasks 301 Quarterly Tasks 302 Annual Tasks 303 Additional Resources 303 Summary 303 CHAPTER 17 Attacks Against Logging Systems 305 Introduction 305 Attacks 305 What to Attack? 306 Attacks on Confidentiality 307 Attacks on Integrity 313 Attacks on Availability 318 Summary 327 CHAPTER 18 Logging for Programmers 329 Introduction 329 Roles and Responsibilities 329 Logging for Programmers 331 What Should Be Logged? 332 Logging APIs for Programmers 333 Log Rotation 335 Bad Log Messages 336 Log Message Formatting 337 Security Considerations 340 Performance Considerations 341 Summary 342 CHAPTER 19 Logs and Compliance 343 Introduction 343 PCI DSS 344 Key Requirement ISO2700x Series 350 HIPAA 353

10 Contents FISMA 360 NIST Logging Guidance 361 Summary 366 CHAPTER 20 Planning Your Own Log Analysis System 367 Introduction 367 Planning 367 Roles and Responsibilities 368 Resources 368 Goals 370 Selecting Systems and Devices for Logging 371 Software Selection 371 Open Source 371 Commercial 372 Policy Definition 374 Logging Policy 374 Log File Rotation 375 Log Data Collection 375 Retention/Storage 375 Response 376 Architecture 376 Basic 376 Log Server and Log Collector 377 Log Server and Log Collector with Long-Term Storage 378 Distributed 378 Scaling 378 Summary 379 CHAPTER 21 Cloud Logging 381 Introduction 381 Cloud Computing 381 Service Delivery Models 382 Cloud Deployment Models 383 Characteristics of a Cloud Infrastructure 384 Standards? We Don't Need No Stinking Standards! 385 Cloud Logging 386 A Quick Example: Loggly 388 Regulatory, Compliance, and Security Issues 390 Big Data in the Cloud 392 A Quick Example: Hadoop 394 SIEM in the Cloud 395

11 M Contents Pros and Cons of Cloud Logging 396 Cloud Logging Provider Inventory 396 Additional Resources 396 Summary 398 CHAPTER 22 Log Standards and Future Trends 401 Introduction 401 Extrapolations of Today to the Future 402 More Log Data 402 More Motivations 404 More Analysis 405 Log Future and Standards 406 Adoption Trends 410 Desired Future 410 Summary 411 INDEX 413