INTRUSION DETECTION IN DISTRIBUTED SYSTEMS An Abstraction-Based Approach

Transcription

1 INTRUSION DETECTION IN DISTRIBUTED SYSTEMS An Abstraction-Based Approach

2 Library of Congress Cataloging-in-Publication ISBN ISBN (ebook) DOI / Copyright 2004 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 2004 Softcover reprint ofthe hardcover lst edition 2004 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photo-copying, microfilming, recording, or otherwise, without the prior written permission of the publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Permissions for books published in the USA: permissions@wkap.com Permissions for books published in Europe: permissions@wkap.nl Printed on acid-free paper.

3 INTRUSION DETECTION IN DISTRIBUTED SYSTEMS An Abstraction-Based Approach by PengNing North Carolina State University, U.S.A. Sushil Jajodia George Mason University, U.S.A. X. Sean Wang University of Vermont, u.s.a. SPRINGER SCIENCE+ BUSINESS MEDIA, LLC

4 Advances in Information Security Sushil Jajodia Consulting editor Center for Secure Information Systems George Mason University Fairfax, VA The goals of Kluwer International Series on ADVANCES IN INFORMATION SECURITY are, one, to establish the state of the art of, and set the course for future research in information security and, two, to serve as a central reference source for advanced and timely topics in information security research and development. The scope of this series includes all aspects of computer and network security and related areas such as fault tolerance and software assurance. ADVANCES IN INFORMATION SECURITY aims to publish thorough and cohesive overviews of specific topics in information security, as well as works that are larger in scope or that contain more detailed background information than can be accommodated in shorter survey articles. The series also serves as a forum for topics that may not have reached a level of maturity to warrant a comprehensive textbook treatment. Researchers as well as developers are encouraged to contact Professor Sushil Jajodia with ideas for books under this series. Additional titles in the series: SECURE ELECTRONIC VOTING edited by Dimitris A. Gritzalis; ISBN: DISSEMINATING SECURITY UPDATES AT INTERNET SCALE by Jun Li, Peter Reiher, Gerald J. Popek; ISBN: SECURE ELECTRONIC VOTING by Dimitris A. Gritzalis; ISBN: APPUCATIONS OF DATA MINING IN COMPUTER SECURITY, edited by Daniel Barbara, Sushil Jajodia; ISBN: MOBILE COMPUTATION WITH FUNCTIONS by Zeliha Dilsun Ktrh, ISBN: TRUSTED RECOVERY AND DEFENSIVE INFORMATION WARFARE by Peng Liu and Sushil Jajodia, ISBN: RECENT ADVANCES IN RSA CRYPTOGRAPHY by Stefan Katzenbeisser, ISBN: X E-COMMERCE SECURITY AND PRIVACY by Anup K. Ghosh, ISBN: INFORMATION HIDING: Steganography and Watermarking-Attacks and Countermeasures by Neil F. Johnson, Zoran Duric, and Sushil Jajodia, ISBN: Additional information about this series can be obtained from

5 To my grandma Huijun Wu, and parents Changcheng Ning and Kuiling Ao. -PN To my parents and my wife. -SJ To my children, with joy. -XW

6 Contents Dedication List of Figures List of Tables Preface Acknow ledgments 1. INTRODUCTION Computer Security and Intrusion Detection Intrusion Detection in Distributed Systems Summary of Contributions Organization 2. AN OVERVIEW OF RELATED RESEARCH 3. SYSTEM VIEW AND EVENT HISTORY 1 System View and Event History 1.1 Qualitative Temporal Relationships between Events V Xl xiii xv xvii MODELING REQUEST AMONG COOPERATING INTRUSION DETECTION SYSTEMS Query 1.1 Query Result Scaling to Large and Heterogeneous Environments 2.1 Expected View and Provided View 2.2 Mismatch and Mismatch Resolution Discussion 3.1 Comparison with Alternative Approaches 3.2 Relationship with Signature-based Intrusion Detection

7 viii 3.3 Implementation Issues EXTENDING COMMON INTRUSION DETECTION FRAMEWORK (CIDF) TO SUPPORT QUERIES 37 1 Background Common Intrusion Specification Language 39 2 A Query Facility for CIDF S-Patterns Format of Returning Message An Example - Tracing Suspicious Users 50 3 Impact on CIDF A HIERARCHICAL MODEL FOR DISTRIBUTED ATTACKS 55 1 Misuse Signature 56 2 Defining System Views Using Signatures: A Hierarchical Model 62 3 Discussion Extensions to ARMD Generic and Specific Signatures Clock Discrepancy DECENTRALIZED DETECTION OF DISTRIBUTED ATTACKS 71 1 Serializable Signatures 71 2 Detection Task and Workflow Tree 73 3 Execution of Detection Tasks 79 4 Optimization 84 5 Generating Workflow Tree A Heuristic Approach CARDS: AN EXPERIMENTAL SYSTEM FOR DETECTING DISTRIBUTED ATTACKS 91 1 CARDS Architecture Signature Manager Monitor Directory Service 94 2 System Design Issues Internal Languages Specific Signature Generation 96

8 Contents ix 2.3 Specific Signature Decomposition 3 Prototype Implementation 3.1 Directory Service and DirHelper 3.2 Signature Manager 3.3 Monitor 3.4 Limitations 9. CONCLUSION Appendices A Document Type Definitions (DTDs) Used in CARDS The DID for System Views The DTD for Signatures The DID for Detection Tasks 115 B Sample System Views, Signatures and Detection Tasks in CARDS System Views The System View DOSAttacks The System View LocalTCPConn The Generic Signature for the Mitnick Attack One Specific Signature for the Mitnick Attack The Detection Tasks for the Specific Signature of the Mitnick Attack Detection Task nl Detection Task n Detection Task n3 124 References Index

9 List of Figures 4.1 The query for tracing suspicious users The query for detecting a distributed port scanning attack A query aggregating TCP 3-way handshake into establish_conn events The signature for the Mitnick attack Events on the system views The view definition for deriving SYN flooding events from TCP packets A hierarchy of system views and signatures A non-serializable signature Examples of workflow trees The algorithm for executing detection task Event processing in a detection task Comparason of two workflow trees Generation of a workflow tree from a specific signature The CARDS architecture A typical configuration The monitor architecture The system view TCPDOSAttacks The generic signature for the Mitnick attack (some details omitted) Detection task n2 of a specific signature for the Mitnick attack (some details omitted) The screen shot of a signature manager The screen shot of a monitor 104

10 List of Tables 3.1 An event history on the system view TCPDOSAttacks An event history provided by a host-based IDS The qualitative temporal relationships between two events Events on host B Result of the query shown in figure Result of the query shown in figure Derivation of implied events Events in the derived history A list of monitors, probes and their system views 99

11 Preface Intrusions in an information system are the activities that violate the security policy of the system, and intrusion detection is the process to identify intrusions. Intrusion detection has been studied for over 20 years. It is based on the beliefs that an intruder's behavior will be noticeably different from that of a legitimate user and that many unauthorized actions will be detectable. Intrusion detection systems (IDSs) are usually deployed along with other preventive security mechanisms, such as access control and authentication, as a second line of defense that protects information systems. Intrusion detection complements the protective mechanisms to improve the system security. Moreover, even if the preventive security mechanisms can protect information systems successfully, it is still desirable to know what intrusion attempts have happened or are happening, so that the users can understand the security threats and risks, and thus be better prepared for future attacks. Intrusion detection techniques are traditionally categorized into two classes: anomaly detection and misuse detection. Anomaly detection is based on the normal behavior of a subject (e.g., a user or a system); any action that significantly deviates from the normal behavior is considered intrusive. Misuse detection catches intrusions in terms of the characteristics of known attacks or system vulnerabilities; any action that conforms to the pattern of a known attack or vulnerability is considered intrusive. Alternatively, IDSs may be classified into host-based IDSs, distributed IDSs, and network-based IDSs according to the sources of the audit information used by each IDS. Host-based IDSs get audit data from host audit trails, usually aiming at detecting attacks against a single host; distributed IDSs gather audit data from multiple hosts and possibly the network that connects the hosts, aiming at detecting attacks involving multiple hosts; network-based IDSs use network traffic as the audit data source, relieving the burden on the hosts that usually provide normal computing services.

12 xvi This monograph presents the research contributions in three areas with respect to intrusion detection in distributed systems. The first contribution is an abstraction-based approach to addressing heterogeneity and autonomy of distributed environments. Specifically, the concept of system view is introduced to provide an abstract interface between different systems. On the one hand, system views hide the difference between heterogeneous systems; on the other hand, they describe what information an autonomous system is willing to provide to other systems. The second contribution is a formal framework for modeling requests among cooperative IDSs and its application to Common Intrusion Detection Framework (CIDF). The first problem is how to enable.idss to request specific information from other IDSs. To address this problem, the proposed technique represents a request to an IDS as a pattern plus a transformation rule, where the pattern specifies the events that the requesting party is interested in and the transformation rule extracts interesting information from the events. The formal approach is also used to add a query facility to the Common Intrusion Detection Framework (CIDF), which allows an IDS to form flexible requests to other systems. The third contribution is a novel approach to coordinating different IDSs for distributed event correlation. The proposed technique represents the event correlation to be performed as a pattern (called a signature) among distributed events. A decentralized method is then presented for autonomous but cooperative IDSs to perform the event correlation specified by signatures. Specifically, a signature is decomposed into finer units called detection tasks, each of which represents the activity to be monitored in one place. The IDSs (involved in a signature) then perform the detection tasks cooperatively according to the "dependency" relationships among these tasks. Our approach is superior to the existing centralized or hierarchical approaches in that (1) communication is more efficient by having different IDSs communicate with each other only when necessary and (2) no centralized or hierarchical trust is required. As an important application of distributed event correlation, this approach can be used to represent and detect distributed (or coordinated) attacks that cannot be detected from a single place. An experimental system called CARDS has been implemented to test the feasibility of the proposed approaches. PENG NING, SUSHIL JAJODIA, AND X. SEAN WANG

13 Acknowledgments We are grateful to Joe Giordano of the Air Force Research LaboratorylRome, David Hislop of the Army Research Office, and Maria Zemankova of the National Science Foundation for sponsoring our research presented in this volume. It is also a pleasure to acknowledge the Association for Computing Machinery for allowing us to use material from ''Abstraction-based intrusion detection in distributed environments," ACM Transactions on Information and System Security, Vol. 4, No.4, November 2001, pages , and Elsevier Science for permission to use material from "Modeling requests among cooperating intrusion detection systems," Computer Communications, Vol. 23, No. 17, November 2000, pages , and "Design and Implementation of A Decentralized Prototype System for Detecting Distributed Attacks," Computer Communications, Vol. 25, No. 15, September 2002, pages

14 Series Foreword ADVANCES IN INFORMATION SECURITY Sushil Jajodia Consulting Editor Center for Secure Information Systems George Mason University Fairfax, VA Welcome to the ninth volume of the Kluwer International Series on ADVANCES IN INFORMATION SECURITY. The goals of this series are, one, to establish the state of the art of, and set the course for future research in information security and, two, to serve as a central reference source for advanced and timely topics in information security research and development. The scope of this series includes all aspects of computer and network security and related areas such as fault tolerance and software assurance. ADVANCES IN INFORMATION SECURITY aims to publish thorough and cohesive overviews of specific topics in information security, as well as works that are larger in scope or contain more detailed background information than can be accommodated in shorter survey articles. The series also serves as a forum for topics that may not have reached a level of maturity to warrant a comprehensive textbook treatment. The success of this series depends on contributions by researchers and developers such as you. If you have an idea for a book that is appropriate for this series, I encourage you to contact me. I would be happy to discuss any potential projects with you. Additional information about this series can be obtained from SUSHIL JAJODIA Consulting Editor