Detecting Anomalies in Service Logs by Using Geographic Tagging

Transcription

1 Detecting Anomalies in Service Logs by Using Geographic Tagging Roman Tekhov Abstract. This article explains what relevant information can be obtained from log files of well known formats, tells how geolocation information can be retrieved, briefly describes existing solutions, introduces the program created to solve the problem and summarizes result obtained from experiments.. Keywords: log analysis, geolocation 1 Introduction The user s credentials may get stolen and the person who accomplished this may use this information in order to impersonate the real owner. In today s globalized world, however, there is a great possibility that the real user and the impostor are physically separated by large distances, possibly located in different countries or even continents. If we could analyze the history of user s requests in terms of locations from which those requests were made, then we could probably detect cases of credential leakage. Service logs typically contain such information, so the problem boils down to the following series of actions: 1. Parse various log files. 2. Retrieve user request information from the logs, i.e. the source host or IP address and the time of each request. 3. Determine the geolocation of each source host or IP address. 4. Compose a scheme of user s movements and detect suspicious cases such as: a. Moving speed that is too large to be really feasible. b. Frequent random moving between different locations, especially distinct ones. c. Requests made from countries that are not considered trust-worthy. d. Movement activity that is too rapid when compared to an average user. Even if the movement takes place in one single city it might indicate that the account is used by multiple people. 2 Retrieving Information from Log Files In order to be useful for our case the log file must contain separate entries about client requests. Each such log entry must contain three things: the user identifier, the source host or IP from which the user performed the request and the exact time of the

2 request. Luckily all major web servers log their data using the corresponding structure. 2.1 Common Log File Format and Apache/NCSA Combined Log Format Common format is a W3C standard format which is used by the Apache server for example. The entries have the following structure: host rfc931 authuser [date] request status bytes The so-called combined log format can be used by Apache as an alternative. The structure is similar to the common format but three additional fields are added: host rfc931 authuser [date] request status bytes refferer user_agent cookie We are only interested in fields host (remote host from which the request was recieved), auhtuser (client username) and date (time of the request) 2.2 W3C Extended Log Format Another W3C standardized format which is used by the Microsoft Internet Information Server for example. This is format is highly customizable and can output lots of different information in custom order. Once again we are only interested in the following fields: c-ip (remote host or IP address), cs-username (username), two separate fields date and time (time of the request). 2.3 OpenVPN Status Log Format OpenVPN is a popular Virtual Private Network implementation. OpenVPN outputs information concerning its currently connected users to a so-called status log file. In case this log is rotated in order to save the history of user actions it is also theoretically possible to analyze the past entries. The structure of the status log entry is the following: Virtual Address,Common Name,Real Address,Last Ref Common Name identifies the user; Real Address represents the remote host and Last Ref the time of the last request.

3 3 Geolocation Once the IP address is obtained it becomes possible to analyze it in order to retrieve the geographical location that this address belongs to. The geolocation software is used to achieve this. Typically such software contains a database of corresponding IP addresses and geographical locations. When the user has an IP address and wishes to know what region it belongs to then he or she submits a query via some interface and the software performs a lookup in the database to retrieve the most probable match. There are several ways how to create such databases: Collect user-submitted data about their location together with the address they submitted from. E.g. in case of a weather forecast site people will most frequently request information concerning their actual whereabouts. Collect the IP addresses from which users connect most frequently together with addresses registered in their account profiles. Use databases of Internet Service Providers. Combine data from multiple different sources. Popular vendors of the geolocation software include Quova IP2Location MaxMind GeoIP All of these vendors offer commercial products. MaxMind also offers an alternative free database called GeoLite that is less accurate but is available for everyone to download. It also offers a number of open-source API-s for working with the database using various languages, e.g. C, Java and Python. Proxy Servers. If the client refers to the server not directly but via a proxy server, then the geolocation will not produce the right result since the real source IP address stays unknown. It is possible, however, to detect the usage of proxies and to report that fact as a sign of suspicion of its own. 4 Created Program The program called FraudDetector got implemented in Java. 4.1 Log File Format Configuration Section 2 of this report only describes some of the most widely spread formats that servers might use to log their data. In practice there are numerous possible ways how such data might be structured so the software has to be configurable enough to support any custom format. All standard formats, like those listed previously, have to be predefined to allow out-of-the-box usage.

4 Format Descriptions. Supported formats are described using a special XML structure in formats.xml file. So far only such formats where a separate single line represents a single user request are supported. Here is a sample description corresponding to Common Log File Format: <format> <name>common</name> <reg-exp> ^([\d.]+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(.+?)" (\d{3}) (\d+) </reg-exp> <host-part>1</host-part> <user-part>3</user-part> <date-part>4</date-part> <date-format>dd/mmm/yyyy:hh:mm:ss Z</date-format> <date-locale>en</date-locale> </format> Most of the fields provide instructions about how each line of the log should be parsed. The idea is to break the line into logically separated parts. The reg-exp element contains the Java regular expression used to do exactly that. Each part is included in parenthesis which is a way to tell the Java pattern matching mechanism to distinguish a separate group of characters. The host-part, user-part and date-part contain numbers of parts which contain the corresponding data portions. Elements date-format and date-locale serve the purpose of date parsing and contain the date pattern and locale to use. For example consider the following log entry: roman [02/Mar/2010:22:30: ] "GET /test.html HTTP/1.1" It will be broken in 7 parts from which are interested in 3: host ( , part 1), user (roman, part 3) and date (02/Mar/2010:22:30: , part 4). 4.2 Log File Reading and Processing The program can read data from one or multiple log files specified in configuration file. The files might be of different format: format.common=some-common.log, other-common.log format.combined=some-combined.log, other-combined.log The program processes each log in a separate thread parsing each line using the corresponding format description. In keeps track of all encountered users and populates a registry of them and all their requests discovered in logs.

5 4.3 Geolocation Once the data is gathered the program starts to analyze it. If some user has referred from multiple different hosts then there is a possibility of fraud. In this case geolocation of all that users remote hosts is performed using MaxMind database. It is possible to specify the location of the binary database file in the configuration. For development and experiments we use the freely available GeoLite City database. MaxMind open-source Java API is used for actual querying. The retrieved Location object contains information concerning the most probable match obtained from the database. This information includes the following useful fields: Country name and code. City name. Geographical coordinates (latitude and longitude). This information is already enough for applying some analysis metrics listed in introduction, e.g. determine whether the user has referred from lots of different distinct locations or from non trust-worthy countries. 4.4 Movement Calculation Since we have the exact time of each request and the coordinates of the location from which that request was made, it is possible to determine the speed of movements between those locations. Suppose we have 2 consecutive requests from the same user: 1. IP , date :30:23 2. IP , date :35:34 Time between these two requests is 5 minutes and 11 seconds or 0,086 hours. The first IP originates from Tartu, Estonia. The second belongs to Oslo, Norway. The distance between those cities is 926,559 km. It means that the speed of moving from Tartu to Oslo had to be 10725,443 km/h which is not likely to be feasible so there is a large possibility of fraud. At this stage all such suspicious cases are just printed to a specified file as an output. 5 Experiments Work in progress, summary will be present in the final version. 6 Conclusion Conclusion will be based on experiment results.

6 References 1. Logging Control In W3C httpd, 2. Extended Log File Format, 3. MaxMind,