Huawei esight LogCenter Technical White Paper Issue 1.0 Date 2013-12-03 PUBLIC HUAWEI TECHNOLOGIES CO., LTD.
2013. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd. Trademarks and Permissions and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders. Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied. Huawei Technologies Co., Ltd. Address: Website: Email: Huawei Industrial Base Bantian, Longgang Shenzhen 518129 People's Republic of China http://www.huawei.com support@huawei.com i
Contents Contents 1 Summary... 1 2 Introduction... 2 3 Solution... 3 3.1 LogCenter Overview... 3 3.1.1 Functions... 3 3.1.2 System Composition... 4 3.2 Typical Networking... 5 3.2.1 Centralized Networking... 5 3.2.2 Distributed Networking... 6 3.3 Key Technical Principles... 7 3.3.1 Unified Log Management... 7 3.3.2 Log Storage Solution... 8 3.3.3 Intelligent Log Search... 8 4 Experience... 9 4.1 Application Scenarios... 9 4.1.1 Network-wide Log Audit... 9 4.1.2 NAT Source Tracing... 10 4.1.3 Security Service Analysis (Including Online Behavior Management)... 11 4.1.4 Authentication Fault Locating... 12 4.2 Performance Indicators... 13 5 Conclusion... 14 A Acronyms and Abbreviations... 15 ii
1 Summary 1 Summary The LogCenter is Huawei log management platform in the B/S architecture. It collects logs from devices, especially security devices, and analyzes security services. Therefore, it applies to many application scenarios, such as unified log management and analysis, network address translation (NAT) source tracing, and enterprise employee online behavior analysis. This document describes LogCenter functions and solutions. 1
2 Introduction 2 Introduction The LogCenter provides log management for Huawei all series devices and a large number of third-party devices by collecting logs from the devices and analyzing the logs in a fine-grained manner. Based on high-performance session log processing, the LogCenter provides an industry-leading NAT source tracing solution. The distributed architecture provides log collection, storage, and audit for an entire enterprise network. The LogCenter supports massive logs and helps rapidly understand network security conditions and user online behavior by providing service analysis reports based on device logs. Security threats can be rapidly detected and removed, improving enterprise working efficiency. 2
3 Solution 3 Solution 3.1 LogCenter Overview 3.1.1 Functions Based on high-performance session log processing, the LogCenter provides an industry-leading NAT source tracing solution. The distributed architecture provides log collection, storage, and audit for an entire enterprise network. The LogCenter supports massive logs and helps rapidly understand network security conditions and user online behavior by providing service analysis reports based on device logs. Security threats can be rapidly detected and removed, improving enterprise working efficiency. NAT source tracing The LogCenter provides an industry-leading NAT source tracing solution. This solution applies to wireless and fixed networking scenarios, using call detail records (CDRs) to trace users. Log audit The distributed architecture allows the LogCenter to collect, store, and audit logs on an entire enterprise network. In addition, the LogCenter supports massive logs. Manageable log sources cover Huawei all series devices and a large number of third-party devices. The LogCenter supports most popular log formats and can rapidly collect, store, and audit logs. Security service analysis (including online behavior management) By efficiently collecting device logs, the LogCenter enables you to learn the running status of devices in a timely manner, trace user online behavior, and rapidly identify and eliminate security threats. Syslogs on Huawei firewalls are collected and analyzed in real time. If an attack occurs or abnormal traffic is detected, an alarm is sent to help rapidly identify and eliminate security threats. By collecting and analyzing session and security logs from Huawei firewalls, the LogCenter can trace online behavior of point-to-point (P2P), email, hypertext transfer protocol (HTTP), Microsoft service network (MSN), and Tencent QQ users, analyze online behavior and application duration and traffic volume, and manage online behavior. Diversified reports are provided for Huawei firewalls to help understand real-time and historical security information about the devices and network attack conditions. 3
3 Solution 3.1.2 System Composition Figure 3-1 shows LogCenter composition. Figure 3-1 LogCenter composition Table 3-1describes component functions. Table 3-1 Component functions and external interfaces Component Function External Interface Graphical user interface (GUI) Log server Log collector Is a unified platform for you to query logs, alarms, and reports. You can configure policies on the GUI. Provides log query, secondary log collection and analysis, statistical reports, security policy management, and user management. Collects, classifies, filters, sums up, and collects statistics on security device logs. Hypertext transfer protocol (HTTP)/Hypertext transfer protocol secure (HTTPS) Simple object access protocol (SOAP) 4
3 Solution 3.2 Typical Networking 3.2.1 Centralized Networking Figure 3-2 LogCenter server and collector deployed in centralized mode The centralized networking applies to scenarios where there are a few devices deployed comparatively close to each other with less than 160,000 Events Per Second (EPS) session logs or 7000 EPS text logs. In this case, the LogCenter server and collector are deployed on one device. The centralized networking is cost-effective and applies to small networks on which devices are deployed in centralized mode. Consider the following factors when selecting this networking mode: Networking of managed NEs In centralized networking, it is recommended that managed NEs be deployed on the same local area network (LAN). If the NEs are deployed on a wide area network (WAN), mass log information occupies WAN bandwidth, affecting service running. Number of logs It is recommended that the number of logs generated on managed NEs does not exceed the processing capability of a log collector in centralized networking. If the log information amount on the live network exceeds the processing capability of a log collector, the server and the log collector must be deployed in distributed mode. One log collector can process 160,000 EPS session logs, 7000 EPS text logs (including syslogs and file transfer protocol (FTP)/secure file transfer protocol (SFTP) logs), or 20,000 EPS binary dataflow logs. 5
3 Solution 3.2.2 Distributed Networking Figure 3-3 LogCenter server and collector deployed in distributed mode The distributed networking applies to scenarios where there are many scattered NEs with more than 160,000 EPS logs. The network administrator can configure one LogCenter server and more than 30 log collectors, which can be horizontally extended. Adding one collector can process more 160,000 EPS logs. The distributed networking applies to medium and large networks. Consider the following factors when selecting this networking mode: Networking of managed NEs NEs are distributed in multiple areas that are connected through a WAN. A log collector is deployed in each area to prevent mass log information from occupying bandwidth and reduce the cost of bandwidth rental. Number of logs If the log information amount on the live network exceeds the processing capability of a log collector, the server and the log collector must be deployed in distributed mode. One log collector can process 160,000 EPS session logs, 7000 EPS text logs (including syslogs and FTP/SFTP logs), or 20,000 EPS binary dataflow logs. Number of external disk chassis The storage cost of the server is lower than that of disk chassis. You can configure multiple log collectors to increase the storage space of the server and avoid or reduce disk chassis. 6
3 Solution 3.3 Key Technical Principles 3.3.1 Unified Log Management Logs are from different types of devices and application programs. Log formats and collection modes differ a lot, such as syslogs, NAT logs, SFTP, and FTP. Therefore, unified log management is necessary. Figure 3-4 Log processing As shown in the previous figure, logs are processed in the following steps: Log collection The LogCenter receives and collects the logs generated on devices and application programs by multiple means. It also supports non-proxy log collection. It can use FTP and SFTP to collect binary and text logs. Log classification The LogCenter provides a simple and effective log classification method based on accumulated experience on device and application system logs. Logs of the same class use the same log structure, facilitating log query and analysis. Log formatting The LogCenter uses patented technology to format logs. Formatting rules support rapid upgrades. Log formatting converts heterogeneous logs into a unified log format. Log filtering The LogCenter can filter logs based on configured filtering policies. Unnecessary information is discarded to save disk space and provide log analysis performance. Log storage The LogCenter stores logs in the file database. Compared with the relational database, the file database has higher throughput and consumes less resources, meeting the massive data storage requirement. Log statistics The LogCenter needs to export a large number of log analysis reports complying with preventive maintenance inspection requirements and regulations. By analyzing formatted log data, the LogCenter records analysis results in the database, supporting rapid log report export. 7
3 Solution Log analysis 3.3.2 Log Storage Solution Rapid detecting security events from massive logs is a major function of the LogCenter. The LogCenter provides policy-based log correlation analysis. If one or multiple logs complying with a policy are detected, the LogCenter notifies the administrator of the events by means of email messages, short messages, or alarm sound. To store massive logs and save disk space, the LogCenter uses a 3-level log storage solution to ensure that log data can be stored for a long time and reduce log storage costs. On the LogCenter, log files are classified into the following types based on log lifetime: Online log Log data is not compressed. You can rapidly query online logs, but they consume large disk space. This storage mode applies to new logs. You can set the storage period for online logs based on user query habits. Dumped log When online logs reach a configured threshold (the storage period or disk space usage), the LogCenter compresses and dumps them. Generally, dumped logs consume only 1/4 of the disk space consumed before they are compressed. Backup log To meet long-term log storage requirements, you can store logs to cheaper storage devices. The LogCenter can re-import backup logs for query. The following figure shows the log file conversion process. Figure 3-5 3-level log storage solution 3.3.3 Intelligent Log Search The LogCenter supports keyword-based word segmentation and indexes. This function is implemented on log collectors that can be deployed in distributed mode. In addition, the LogCenter provides log query similar to that on a search engine and can drill and collect statistics on query results. 8
4 Experience 4 Experience 4.1 Application Scenarios 4.1.1 Network-wide Log Audit Many routers, switches, and firewalls are deployed on an enterprise network. Different types of NEs generate logs in different formats, causing poor log readability. Massive logs are hardly stored, and the logs cannot be managed in a unified manner. As a result, the network management system cannot rapidly detect critical security threats from logs. The LogCenter can manage logs in a unified manner. It can collect binary and text logs using SFTP and FTP. In addition, the LogCenter can collect, classify, filter, sum up, analyze, store, and monitor logs reported by NEs to help you manage massive logs, understand NE operating status, trace user online behavior, and rapidly identify and eliminate security threats. Based on log management, the LogCenter analyzes logs in real time to provide real-time alarms. Figure 4-1 Security event management scenario 9
4 Experience 4.1.2 NAT Source Tracing NAT is widely used on various types of Internet access modes and networks. NAT addresses the IP address shortage issue and effectively prevents external network attacks by hiding internal network PCs. NAT brings another issue: Hosts in an enterprise share one IP address, and therefore it is difficult to trace the responsible person for an online violation. The LogCenter collects and analyzes session logs from gateways, such as Eudemons, Unified Security Gateways (USGs), MA5200Gs, NE40s, NE80s, and ME60s, to obtain NAT information, including destination IP addresses, destination ports, source IP addresses before NAT, and protocols. With the help of traffic logs, the LogCenter can trace online behavior of users in the enterprise. Figure 4-2 NAT source tracing scenario NAT source tracing involves multiple IP addresses and ports, including the source IP address, source port, destination IP address, destination port, source IP address after NAT, source port after NAT, destination IP address after NAT, and destination port after NAT. The following example explains field meanings in port address translation (PAT) mode. Figure 4-3 IP address translation example for source tracing The internal user at 172.18.11.2 accesses the external HTTP server at 10.78.18.20 through NAT. The addresses and ports in the session are as follows: 10
4 Experience Source IP address/source port: 172.18.11.2:64244 Destination IP address/destination port: 10.78.18.20:80 Source IP address after NAT/source port after NAT: 10.78.18.11:57841 Figure 4-4 IP address translation example in NAT server mode The external user at 10.78.18.20 accesses the internal HTTP server at 172.18.11.2. The NAT device has mapped the internal HTTP server. The addresses and ports in the session are as follows: Source IP address/source port: 10.78.18.20:57841 Destination IP address/destination port: 10.78.18.11:8080 Destination IP address after NAT/destination port after NAT: 172.18.11.2:80 NAT can be implemented in three modes: static NAT, dynamic NAT, and PAT. Static NAT: Private IP addresses are translated into public IP addresses. A private IP address corresponds to a fixed public IP address. Dynamic NAT: Private IP addresses are randomly translated into public IP addresses. A private IP address of a user is translated into any valid public IP address before the user accesses the Internet. PAT: The source port is a packet destined for the Internet is translated into another. All hosts on an internal network can share one valid public IP address for Internet access, greatly saving IP address resources and hiding internal hosts to effectively prevent attacks from the Internet. This mode applies to most networks. 4.1.3 Security Service Analysis (Including Online Behavior Management) To manage enterprise employee online behavior, the LogCenter collects and analyzes session and security logs from NEs, such as firewalls, to trace and analyze employee online behavior (top N online traffic, top N online duration, web access analysis, and email analysis). The LogCenter provides user-based analysis and query of online traffic, online duration, searched keywords, web access, email sending and receiving, online applications, network threats, and file sending. The enterprise management personnel can manage employees based on online behavior analysis results. 11
4 Experience Figure 4-5 Enterprise employee online behavior management scenario 4.1.4 Authentication Fault Locating Customer pain points: The employees of an enterprise must access a remote authentication dial in user service (RADIUS) server through access devices before accessing their enterprise network. There are many access devices, including switches and wireless local area network (WLAN) access controllers (ACs), generating a large number of logs. Logs cannot be queried on one device. Access device logs and RADIUS server authentication logs cannot be associated for query. If a fault occurs (for example, an access failure), the maintenance personnel must obtain logs from multiple devices and manually associate and analyze the logs with low efficiency. The customer hopes that there is a centralized device storing all switch authentication logs, WLAN authentication logs, and RADIUS authentication logs, providing keyword-based rapid query, and associating access device authentication logs and RADIUS server authentication logs. LogCenter solution: The LogCenter provides keyword-based indexes and intelligent search capabilities. It allows unified indexes among switch authentication logs, WLAN authentication logs, and RADIUS server authentication logs. The LogCenter has a search engine-like GUI, allowing you to query multiple types of logs within a given time range. In addition, query conditions can be 12
4 Experience regular expressions, associating multiple types of logs. Query results can be drilled and counted based on log types. 4.2 Performance Indicators Table 4-1 LogCenter technical specifications Item Sub-item Description Log performance indicator Log processing capability Binary session logs: 160,000 EPS Text logs: 7000 EPS Binary dataflow logs: 20,000 EPS You can add hardware to improve the log processing capability. One standard-configured log collector can collect 160,000 EPS binary logs or 7000 EPS text logs. 13
5 Conclusion 5 Conclusion The LogCenter supports log management for Huawei all series devices. It provides comprehensive log collection and service analysis capabilities. Therefore, it applies to many application scenarios for unified log management and analysis, NAT source tracing, and enterprise employee online behavior analysis. Deploying the LogCenter reduces IT system maintenance costs, enhances timely response capabilities to faults and security events occurring on Huawei security devices, helps enterprises cope with regulation compliance checks, and improves IT system auditability. 14
HUAWEI esight Technical White Paper (LogCenter) A Acronyms and Abbreviations A Acronyms and Abbreviations Figure A-1 Acronyms and abbreviations for the LogCenter Abbreviation EPS NAT FTP SFTP SOAP HTTP HTTPS Full Spelling Event per second Network Address Translation File Transfer Protocol Secure File Transfer Protocol Simple Object Access Protocol Hypertext Transfer Protocol Hypertext Transfer Protocol over Secure Socket Layer 15