Creating Customized Whitelist Domains from DNS Traffic

Transcription

1 White Paper Security Creating Customized Whitelist Domains from DNS Traffic

2 Table of Contents page Abstract... 1 Introduction... 1 Background... 2 The Proposed Method Experiments Experiment Results and Discussions... 9 System Architecture Conclusion... 17

3 Abstract DNS Malware Analytics (DMA) is a Micro Focus Security solution which provides a better understanding of DNS traffic to alert customers to advanced cyber-threats, using in-depth cutting-edge intelligent analytical and statistical tools. One of the challenges organizations face is how they can quickly narrow down the scope of evaluated data to the suspicious DNS requests only. In healthy organizations, the expected amount of traffic to malicious domains should be relatively small, so finding them is like finding a needle in a haystack. Traditional ways of finding malicious DNS traffic rely significantly on subscription to third-party blacklists. Those lists are costly, very dependent, and not customizable. This paper presents a new solution to this problem via identifying normal or benign traffic and creating a customized whitelist pertaining to the related DNS traffic. Leveraging such a whitelist will help security professionals focus the investigation effort only on the unknown and suspicious traffic. The proposed algorithm automatically identifies whitelist domains based on a given DNS traffic dataset and a streamlined system to continuously generate a customized whitelist. The system was validated based on extensive experiments on 16 months of internal DNS traffic data. Results have suggested that the solution works very well: covering about 30 percent of current unfiltered data with the preferred parameters identified in this whitepaper. One of the quality metrics used to validate the strength of this method is the ratio of malicious domains before the final output whitelist. The estimated ratio of malicious domains is less than 1.3 percent before executing the validation and elimination process. This result is lower than one of the main lists currently used in the industry Alexa Top Sites 1 which is reported at about 1.5 percent in a similar setup environment described in this whitepaper. The paper ends with future suggestions for improvements and general integrations with additional threat indicators from other security products to achieve greater value and impact. Introduction 1 The Domain Name System (DNS) is one of the most vital protocols of the modern Internet. This protocol resolves human-readable hostnames (e.g., microfocus.com) into machine-readable IP addresses (like ) and its traffic has been ubiquitously flowing in every enterprise s network infrastructure. However, DNS data in many cases has been overlooked as a source to identify security incidents, due to the fact that other systems are more accessible and more familiar to small or less-experienced companies (e.g. network firewall and end-point protection such as antivirus software); moreover, analyzing DNS traffic or protecting at the DNS level requires advanced knowledge and experience. This situation has been significantly leveraged by attackers, who use more sophisticated strategies around DNS protocol to launch advanced cyber-attacks. The need for more research and best practices to inspect and protect DNS traffic has rarely been more urgent and important. 1

4 White Paper Creating Customized Whitelist Domains from DNS Traffic In an enterprise network environment, DNS traffic is ubiquitous and represents a core portion of network communication. For example, the Micro Focus core data centers experience an average of 18 to 20 billion DNS requests daily. However, the expected amount of DNS requests to malicious domains is extremely small, and finding them is particularly challenging. This fact is likely one reason that DNS service is involved in the initial part of cyber-attack vectors. Hence, effective solutions are needed to detect and block suspicious or malicious DNS activities. One popular way of blocking malicious DNS activities is simply to subscribe to blacklists from third-party security vendors. However, these solutions might be costly, very dependent, uncontrollable, and not customizable. Techniques that are more intelligent have been used to identify malicious DNS requests, such as Domain Generation Algorithm (DGA) detection, or DNS exfiltration detection. As the detection techniques get more complicated, the load of traffic data being inspected greatly affects the performance of the analysis. Therefore, reduction of the inspected data set is crucial, and the usage of whitelist as a data filter is required. One source for such whitelist is Alexa Top Sites, which can manage to filter the majority of traffic in typical enterprise DNS data. 1 This paper presents a system to identify normal or benign traffic and create a customized whitelist pertaining to the related DNS traffic. By using this method, the rest of the traffic is condensed, allowing for further investigation to identify the actual maliciousness. In particular, this study focuses on identifying benign domains (beyond the Alexa Top Sites) and creating a customized whitelist for a given DNS traffic dataset to filter out benign DNS data and expose malicious traffic. Studying this problem can help reduce the DNS traffic needed to be examined, and build more advanced downstream analytics, such as improving performance of extracting threat alerts, or exposing new types of malicious activities. Additionally, although this study is inspired by the DMA product with experiments based on the data and infrastructure in DMA systems, the problem and methodology are generic, and may be applicable to other similar environments. Background Rationales The following list articulates different reasons for conducting research on ways to identify benign DNS traffic: a. Existing whitelist (Alexa Top Sites) is insufficient in certain scenarios. I. Alexa provides a popular list of HTTP traffic, not necessarily a domain-name whitelist. II. Even after filtering out the Alexa top one million website domains from Micro Focus, around 20 billion DNS records per day, there are still around million DNS records left each day. III. Alexa top one million domains do not include some subdomains of top websites, such as google. doubleclick.net, static.alibaba.com, translate.google.nl, maps.live.com, appsearchcdn.baidu.com, app.weibo.com, and others

5 IV. Individual users may frequently visit a low-ranked website, or have special DNS traffic patterns. In this sense, Alexa is too broad to be responsible for a particular DNS traffic. V. Manually adding whitelist domains is inconvenient. Given a fixed or timeseries dataset of DNS traffic D, containing timestamp, source IP, and requested domain, design a strategy to generate a whitelist of benign domains W from the traffic (benign means the domain is not associated with malicious intention, such as phishing, serving malwares, etc.) Bilge L, Kirda E, Kruegel C, Balduzzi M. EXPOSURE: Finding Malicious Domains Using Passive DNS Analysis. In NDSS 2011 Feb 6. 3 Yadav S, Reddy AK, Reddy AL, Ranjan S. Detecting algorithmically generated malicious domain names. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement 2010 Nov 1 (pp ). ACM. 4 Antonakakis, M., Perdisci, R., Nadji, Y., Vasiloglou, N., Abu-Nimeh, S., Lee, W., and Dagon, D., From Throw-Away Traffic to Bots: Detecting the Rise of DGA-Based Malware, Proceedings of 21th USENIX Security Symposium (USENIX Security 12), b. Better focus on real attack traffic. I. Using a customized whitelist to remove obviously benign DNS traffic allows customers to focus on plausible or bad traffic. c. Storage concern in general. I. Save space by safely filtering benign traffic. Other Related Work Micro Focus conducted a brief preliminary investigation on how the industrial and academic researchers solved the problem of identifying malicious DNS traffic. The majority of existing studies tried to find such data by directly creating blacklists, especially by discovering automatically generated domains (AGDs) 2, 3, 4 through intelligent methods such as machine learning and network traffic analysis. Methods of directly creating whitelist domains to expose suspicious DNS traffic do have some discussions scattered around, which mainly fall into the following ways: 1. Domain owners manually register domains to a reputable central location, such as dnswl.org. Normally those locations are community-owned or trustable organizations so that register process is not abused by spammers. 2. Large Internet Service Providers (ISPs) maintain their own whitelists through either manual or automatic process from the DNS traffic they can access. 3. The Alexa Top Sites. 1 Alexa provides a list of popular websites that represents the behavior of people globally on a daily updating basis. Be aware that the Alexa domain is not necessarily a whitelist, but rather, it is based on global users browsing behavior; so, it could contain malicious domains. In summary, creating customized whitelist domains directly from DNS traffic has very rarely been discussed openly. A systematic solution on how to solve this problem is needed. Problem Statement Let s now formalize the problem in a clear way, and set up the assumption. At a high level, the problem is how to identify benign domains to create a whitelist from the DNS traffic dataset. Scientifically speaking, the problem statement is: Goal The goal of this study is to develop a strategy to generate a whitelist of benign domains from existing DNS traffic data. From a DMA perspective, this capability will be implemented per customer instance and not as a general shared list for all. Micro Focus conducted comprehensive experiments to develop this strategy. Three main questions to answer: 3

6 White Paper Creating Customized Whitelist Domains from DNS Traffic Question 1: How to generate the candidate whitelist domains from DNS traffic only, and what is the statistical measurement for a domain to be considered a candidate for the whitelist? Question 2: How to determine if a domain is malicious (e.g., serving phishing or spamming contents, or malware, or drive-by downloads, etc.) or benign (free of malicious contents)? Domains can constantly change, serving benign or malicious contents at different times, making this a very challenging question. Additionally, this type of knowledge is out of scope for DNS traffic data only, meaning a reliable third-party verification mechanism might need to be established. However, such lists will be considered acceptable in this study when looking for a set of statistically sound whitelist domains, as long as the majority of the extracted domains are benign or exhibit no signs of malicious activities. In short, this study estimates the quality of the whole list of extract domains with the best knowledge available. Question 3: How soon should the whitelist be expired, and how frequently should it be updated with new contents? Given domains constantly change and DNS traffic is time-sensitive data, expiring an old whitelist and creating a new whitelist is an appropriate choice for more accurate results. Remember that even the Alexa top domain list is updated daily. Updating and expiring the customized whitelist based on DNS data is a reasonable strategy to keep a high quality of whitelist domains. Considering the DNS traffic is continuous and streamlines data, the new whitelist can be partially created day-by-day for convenience, while expiring them gradually. Hence, frequency of updating the whitelist can differ from their expiration date range. For example, a monthly whitelist can be roughly created in four consecutive weeks one partial update each week, and set the expiration date as a month later. The Proposed Method Thought Process A simple and straightforward idea on creating customized whitelist domain is to check every single DNS request automatically to see whether it is benign or malign. However, the study found this idea was not practical due to the following reasons: 1. The unique number of domains is very big. Within Micro Focus s daily DNS traffic, there are about million events after filtered with Alexa top one million sites, containing about 1 4 million unique domains per day. Verifying millions of domains through an automatic system or a third party is not practical in terms of cost and time. 2. Majority of the unique domains, more than 55 percent, are only requested once or twice in the Micro Focus data. Validating such rarely requested domains brings little value to future DNS traffic. 4

7 Based on these numbers, there is a need to create a whitelist that covers as much DNS traffic as possible and uses as little validation effort as possible. Another thought process comes from the fact that Micro Focus had more than 300,000 employees worldwide (at the time DNS data was collected in 2015). The DNS domains are very diverse, with so many different locations and a huge number of users. Creating a whitelist has naturally come down to how popular the domains are that have been requested. The more unique users (e.g. IPs) requesting a domain, the more likely it is benign, and the more it impacts DNS traffic. In this sense, it seems to delegate or crowdsource the process of creating the whitelist to the users. Of course, the list of popular domains collected is not necessarily all benign, extra constraints and verification processes are required to improve the quality. This thought process has naturally brought up the following ideas. Core Ideas Generating a high-quality whitelist is based on one assumption: if a domain is requested by multiple clients (IPs) within a narrow time window, then the domain is most likely benign. Therefore, the rule to generate whitelist domains is: The rule is straightforward, but requires several parameters to be experimentally verified in order to validate the rule for practical use. Doing so brings a few additional questions and concerns to mind. Some Concerns Given a dataset of DNS traffic D, if a domain d is queried by x number of IPs within a time interval window t, this domain is considered as a candidate of whitelist. The candidate whitelist domains are further evaluated by reputation service to generate final whitelist. 5 a. Regarding the threshold x, is it still possible that a domain requested by more than x clients or IPs can be malicious? I. The chances are low, because the probability of many employees getting infected by malware and visiting the related malignant domains in a short period of time is very unlikely. Also, the more employees, the less chance the domain is malignant. In addition, a safe way is to increase x until low or no false positive domains, given a validation dataset. To determine how low the false positive be, need to compare that with other open whitelists, such as Alexa top list. II. To remove all potential malicious domains, Micro Focus suggests verifying all whitelist domains through a reliable third-party service (e.g., VirusTotal 5 ) to remove malignant domains. Or if the whitelist is too big, random sampling for verification can be used to gain a statistical confidence, and compare such results with Alexa Top Sites maliciousness quality. III. If clients get dynamic IPs from DHCP/Proxy/VPN, could the x multiple IPs be the same client? Again, reducing the time window t to a reasonable period of time can lower the false positive rates. b. What is the appropriate response if a domain is verified as malicious during the verification process? I. This malignant candidate domain will be removed from the whitelist. c. Visitor behavior and domains constantly change requiring the whitelist to regularly self-update, but at what frequency? How are expiration dates determined? I. There is an expiration date for the whitelist denoted in parameter exp_days. However, the active whitelist can be updated at once, or partially; hence, there is an update frequency freq, which can be shorter than exp_days. Different experiments are performed to determine which fits the best. 5

8 White Paper Creating Customized Whitelist Domains from DNS Traffic Algorithm Design Let s now design the detailed algorithm for the proposed idea and tackle each of those concerns mentioned above. There are three main steps in the algorithm and several related smaller subtasks in each of them. Step 1: Identifying good candidates for the whitelist domains. Three subtasks in this step includes: 1. Given DNS traffic data D, containing timestamp, source IP, requested domain per record then find the domains requested by x number of unique source IPs in a given time window t. The main solution to this problem is easily implemented with SQL syntax when database has properly been set up. In this study, the dataset researched was filtered out of Alexa Top Sites, internal blacklist, and traffic that was identify using DGA. The SQL syntax looks like the following: [Image_1]. SQL syntax 2. Determine the number of x and time interval window t. Various experimental settings are tried to optimize the parameters in this study. 3. Aggregate the popular domain lists in each t-hours window into a longer list, where the length is the expiration date exp_days. As argued previously, the final whitelist will have an expiration date, so the list needs to be aggregated in every t-hours segment into the targeted expiration date period. The algorithm in this task is simple: for each popular list of domains in a t-hour window, add them to the candidate whitelist. For example, union domains in all popular lists of 720 one-hour intervals together to the monthly list of April. (Adjusts for the actual number of calendar days). Then remove duplicated domains from the candidate list. Step 2: Validation of candidate whitelist domains via reputation services. This step presents a process to verify statistically the benign quality of the complied list. It is possible to use a manual process to validate the maliciousness of a domain by searching any related information on the Internet. Obviously, a manual process is impractical, and designing a service to verify a domain from scratch, by checking its Whois info, web, or service information, IP information is also cumbersome. Note: <starttime>, <stoptime> is a timeinterval window; <threshold> is a parameter 6

9 The study utilized many existing third-party domain reputation services to perform this task. Three related questions during validation needed to be addressed. 1. Is there a need to verify every domain in the candidate list (if the list is too big)? Two reasonable strategies are discussed here: Strategy 1: Yes, verify every domain via the reputation system if possible. In this study, Micro Focus researchers are able to query millions of domains per day using the subscription of VirusTotal to validate all the candidates and check the reputation. In addition, a cumulative verification process can be used to lower the requirement of getting a high-limit subscription, e.g., gradually verify each t-hour segment list (instead of verifying the whole candidate list at once). Strategy 2: Random sampling to pick up N (up to or over 10,000 or the limit permitted) number of domains in the candidate list for verification. The percentage of malicious domains is recorded to measure the quality of the whole candidate whitelist. In this sense, the final whitelist may still contain malicious ones (with a quality metric). Overall, users should avoid this strategy unless there are no other choices. Meanwhile, a cache system can be used to save reputation results to reduce repeatedly requested domains, which can speed up the validation process, as well as lower the chances of reaching query limits. 2. What to do with domains if the reputation service returns nothing? Here is the proposed process to validate a domain (or URL): I. Convert the domain (or URL) to canonical name (remove http and/or www). II. Search for the domain name from the reputation service API. III. If the API returns a lower score, the domain is marked as malicious. If the API returns nothing, try one domain level higher and check again until the second-level domain. IV. If there are still no results for the second-level domain, mark it as benign. When using the VirusTotal reputation system to confirm a malicious domain, Micro Focus suggests using their domain search endpoint, and the condition of either a webutation score of less than 50 or has at least two infected scan hits in the last year. 5. What to do with domains verified as malicious? When a domain is marked as malicious via reputation score, remove it from the list before generating the final whitelist. Step 3: Determine the whitelist expiration date exp_days, and updating frequency freq and strategy. To determine the best whitelist expiration date range exp_days, in terms of quality and workload, the Micro Focus research team conducted various experiments to find out. A few exp_days settings are seven days (a week), 14 days (two weeks), 30 days (a month), 60 days (two months), and so on. 7

10 White Paper Creating Customized Whitelist Domains from DNS Traffic For an efficient updating strategy, the research team proposed a partial update strategy: expiring the oldest part of the active whitelist, and appending them with a daily new whitelist. The benefits of this strategy are: a) at any time, the active whitelist set is still the whole whitelist generated within the expiration date range; and b) the daily new whitelist is smaller and easier to verify fully in the validation process. However, one condition that such strategy holds its ground on is that the targeted whitelist can be split into smaller (or daily) whitelists. This requirement indeed holds up in the algorithm design: the active whitelist is a union set of all whitelist domains extracted from each time window t within the expiration date range. Experiments Experiment Design Internally, the DNS research environment containing Micro Focus s core DNS traffic data has been robustly created by Hewlett Packard Labs and the DMA team. The Vertica Big Data analytics database is used to support the infrastructure. The data used in this study contains DNS traffic within Micro Focus s network from 01 January 2015 to 30 April 2016 (missing a month of data in June 2015). Although each day Micro Focus s core network generated billion DNS requests, the list of Alexa Top Sites with one million domains has been applied to the DNS capture device before outputting requests to the Vertica database (DNS replies are also saved in the same database, but are not the focus in this study yet). Still there are about million DNS request packages per day. Here is a summary statistics of the data. Year-month Number of DNS requests Year-month Number of DNS requests ,054,488, ,730,465, ,699,849, ,912,902, ,727,746, ,058,802, ,768,156, ,189,060, ,462,672, ,288,034, ,225,139, ,107,450, ,106,043, ,463,884, ,433,696,577 Table 1. Summary statistics of DNS requests (After Alexa Top Sites filtered) in this study Based on this dataset, the research team designed the following experiments to address the discussions mentioned in the above algorithm design. E1. Test the impacts of different threshold x to whitelist. The underlying expectation is the higher threshold x is, the smaller the candidate whitelist is, delivering lower traffic and less chance of malicious domains. Impacts are measured by the size of the whitelist generated, the DNS request traffic it covers, the percentage of (estimated) malicious domains, and the similarity between one whitelist to the next active whitelist. 8

11 E2. Test the domain quality of Alexa Top Sites with a reputation system. This experiment shows if there are malignant domains in Alexa list, and how high the percentage is. E3. Compare percent malicious domains between the whitelist and Alexa list. By comparing the percentage of malicious domains in the whitelists (ranked with domain popularity) to Alexa directly, it gives a statistical confidence on how well the whitelist performed. E4. Test the impacts of different time interval window t. Time interval window t is the time range that is used to find out how many clients are requesting a unique domain, in order to determine its candidacy in the whitelist. Here, a reasonable interval window can be one hour, two hours, four hours, eight hours, and 12 hours. E5. Test what is the best expiration date length of whitelist domains and updating frequency. Again, a few reasonable date lengths are tested, e.g., two months, a month, two weeks, and one week. The metric to choose the best length is using percentage of malicious domains in the list. E6. Test what is better for updating frequency to the whitelist. Updating frequency should less than expiration date. So a few updating frequencies are two weeks, one week, or one day. The average number of new domains being added in consecutive lists can be used to choose the best. Experiment Results and Discussions This section discusses the exact settings, results, and conclusions of the preceding experiments. ER1. Test the impacts of different threshold x to whitelist. Settings: in this experiment, threshold x is changing, while interval window t and expiration date is fixed. Based on the experiments and preliminary results, here it is using t = 1 hour and exp_days as 30 days, and x is choosing among 4, 8, 12, 16, 20, 24, 32, 64, 128. This shows results with t = 2 hours. Due to the size of candidate domains, use random sampling (sample size = 500) to estimate the size of malign domains in the candidate list. 9

12 White Paper Creating Customized Whitelist Domains from DNS Traffic Settings: interval=1 hour, threshold=12 IPs, expiration day = 30 Number of requests by candidate of requests by candidate Year-month Number of candidate domains domains domains of estimated malign domains ,152 85,608,115 22% 1.00% ,195 83,234,182 23% 2.50% 40% ,324 86,215,294 27% 0.50% 36% ,572 64,700,702 21% 2.50% 37% ,808 86,854,898 20% 0.50% 41% , ,532,177 22% 0.50% 34% , ,011,907 27% 2.00% 33% , ,564,914 31% 0.50% 40% , ,353,881 31% 0.50% 37% , ,957,381 36% 0.50% 26% , ,846,029 32% 0.50% 57% , ,270,356 27% 0.60% 41% , ,570,835 29% 0.40% 47% , ,412,313 27% 0.60% 47% , ,510,553 42% 0.40% 42% Average 86, ,109,569 28% 0.90% 40% Table 2. Impact of whitelist generated by x=12 with interval 1 hour, expiration 30 days of similarity to previous list Settings: interval=1 hour, expiration days=30 Number of requests by candidate of requests by candidate of estimated malign Number of estimated malign Threshold Number of candidate domains domains domains domains domains 4 1,195,640 1,065,915,389 46% 1.03% 12,355 32% 8 261, ,239,709 34% 1.10% 2,879 37% 12 86, ,886,923 28% 0.90% % 16 47, ,418,167 23% 0.88% % 20 29, ,782,099 21% 1.30% % 24 20, ,872,540 20% 1.45% % 32 11, ,115,170 18% 1.78% % 64 4, ,035,772 14% 2.58% % 128 1, ,302,341 12% 3.17% 62 55% of similarity to previous list Table 3. Comparison of whitelist impacts generated by x=4, 8, 12, 16, 20, 24, 32, 64, and 128 with one-hour intervals and 30 days expiration. 10

13 Figure 1. Comparison of whitelist impacts generated by x=4, 8, 12, 16, 20, 24, 32, 64, and 128, with one-hour intervals and 30-day expiration. Settings: interval=2 hour(s), expiration days=30 Number of requests by candidate of requests by candidate of estimated malign Number of estimated malign Threshold Number of candidate domains domains domains domains domains 4 1,332,728 1,125,775,955 48% 0.90% 11,995 34% 8 320, ,581,029 37% 0.93% 2,993 39% , ,798,725 30% 1.36% 1,551 42% 16 64, ,855,031 25% 1.02% % 20 42, ,926,729 23% 1.07% % 24 29, ,733,611 21% 1.22% % 32 17, ,814,430 19% 1.98% % 64 6, ,609,598 15% 2.33% % 128 3, ,637,903 13% 2.72% 85 56% of similarity to previous list Table 4. Comparison of whitelist impacts generated by x=4, 8, 12, 16, 20, 24, 32, 64, and 128 with two-hour intervals and 30-days expiration. Conclusions: From tables 3 and 4, the research team s expectation does match the results: the higher the threshold is, the smaller the size of candidate whitelist, and the less DNS traffic the whitelist covers. However, the percentage of estimated malignant domains is not linearly decreasing, and in most cases, it is increasing. One explanation is that the real number of malicious domains is relatively constant, and the larger the whitelist is, the lower percentage of malignant domains. Fortunately, the absolute number of estimated malignant domains is constantly decreasing due to the significant decreased size of candidate whitelists. This phenomenon suggests that if possible, validating all domains instead of sampling should be used to generate the final whitelist. 11

14 White Paper Creating Customized Whitelist Domains from DNS Traffic Meanwhile, from the results we have just seen in the preceding tables and others not shown here, threshold 12 seems to provide the best balance between number of estimated malignant domains, covered DNS traffic, and similarity to previous expired list. ER2. Test the domain quality of Alexa Top List with reputation system. Settings: Collected three sets of Alexa Top Sites with one million domains in March 2015, Jan 2016, and Apr 2016, sliced them into 10 piles (e.g. 100k domains per pile), and randomly sampled about 200 domains in each pile for reputation verification. Three independent runs had been conducted for each set, and the final results were averaged together. In addition, the top 100k domains were sliced into 10 piles (e.g., 10k domains per pile) and used for similar experiments. Pile size 100k Pile size 10k Range Mar 2015 Jan 2016 Apr 2016 Range Mar 2015 Jan 2016 Apr k k k 200k k 20k k 300k k 30k k 400k k 40k k 500k k 50k k 600k k 60k k 700k k 70k k 800k k 80k k 900k k 90k k 1m k 100k Average Average Table 5. of malignant domains in three sets of Alexa top lists verified by VirusTotal. Conclusions: Surprisingly, Alexa top lists also contain a relative number of malignant domains. More interestingly, as shown on the left side of the table, the ranking is not linearly correlated with the percentage of malignant domains. However, from the right side, it seems that the sites ranking within 100k had higher average percentage of malignant domains than those in the 1 million range. Those results again suggest that random estimation via a reputation system for a large pile of domains is not reliable. A whitelist needs to get all the domains verified before being put into use. ER3. Compare percentage of malicious domains between the candidate whitelist and Alexa list. Settings: Ranked the monthly candidate whitelist domains based on their DNS traffic size and then split them into 100k a pile for reputation validation with random sampling size at 500. Each pile is then compared to the same pile in Alexa Top Sites list. The interval hours are fixed at two hours and threshold at 12. The following table shows the months that generated at least 1 million candidate domains for fair comparison. Note: Each number in the cell is a percentage, averaged from three independent runs. 12

15 Settings: interval = 2 hours, threshold = 12, expiration days = 30 Pile Jan 2015 F Conclusions: Obviously, the candidate whitelist generated in the experiments mostly had a smaller percentage of estimated malicious domains compared to the same level of Alexa Top Sites. This proves that the method works, as it generally provides a good quality of whitelist domain candidates, comparing the industry standards. Feb 2015 Mar 2015 May 2015 Oct 2015 Nov 2015 Dec 2015 Feb 2016 WL Size 3080k 3898k 1186k 1497k 994k 1175k 1377k 1491k Horizontal average pile-100k pile-200k pile-300k pile-400k pile-500k pile-600k pile-700k pile-800k pile-900k pile-1m Average sets of Alexa Top 1m Table 6. Side-by-side comparisons of estimated malicious domains (in percentage) from the candidate whitelist vs. Alexa top list ER4. Test the impacts of different time interval window t. Settings: The time interval windows t is set at one hour, 2, 4, 8, and 12 hours to limit reaching the threshold x. Threshold x is set at 12 IPs and expiration days exp_days is 30, and size of randomly verified domains is 500. The experiment also used other parameters, but only this group of results is shown here. Note: Each number in the cell is a percentage. Numbers with bold font means the percent of malicious domains in the candidate list is higher than that of averaged 3 sets of Alexa list in the same pile. The candidate list ranking is based on its covered DNS traffic size in the expiration date range. Settings: interval=2 hour(s), expiration days=30 interval=12 hours of DNS covered Yearmonth Estimated percentage of bad domains Estimated percentage of bad domains Estimated percentage of bad domains Estimated percentage of bad domains interval=1 hour of DNS covered interval=2 hours of DNS covered interval=4 hours of DNS covered interval=8 hours of DNS covered % 1.0% 24.3% 1.5% 25.8% 1.8% 27.3% 1.0% 28.1% 1.8% % 2.5% 26.6% 2.5% 28.3% 1.5% 29.8% 1.0% 30.6% 0.5% % 0.5% 30.1% 1.0% 32.0% 0.8% 34.3% 1.3% 36.1% 1.0% % 2.5% 23.9% 0.5% 26.1% 1.3% 29.2% 1.5% 32.2% 1.0% % 0.5% 21.3% 2.0% 22.7% 2.0% 25.4% 0.0% 28.1% 1.5% % 0.5% 24.2% 2.0% 26.4% 0.5% 31.0% 0.8% 36.2% 0.3% % 2.0% 29.7% 2.5% 33.8% 0.5% 39.5% 1.0% 43.5% 0.8% Estimated percentage of bad domains Continued on the next page 13

16 White Paper Creating Customized Whitelist Domains from DNS Traffic interval=1 hour interval=2 hours interval=4 hours interval=8 hours interval=12 hours % 0.5% 34.5% 1.5% 38.1% 0.3% 43.1% 1.0% 46.2% 1.5% % 0.5% 33.8% 1.5% 36.5% 0.8% 39.2% 0.5% 40.7% 1.5% % 0.5% 38.6% 1.0% 41.0% 0.8% 43.5% 1.5% 45.0% 0.8% % 0.5% 34.7% 1.0% 37.2% 0.3% 39.8% 1.8% 41.4% 0.3% % 0.6% 29.0% 0.8% 31.4% 1.2% 34.0% 0.8% 35.6% 0.4% % 0.4% 30.9% 1.2% 33.2% 1.0% 35.6% 0.6% 37.1% 0.8% % 0.6% 29.8% 0.8% 32.5% 0.8% 35.3% 1.2% 36.9% 0.2% % 0.4% 43.8% 0.6% 45.7% 0.8% 47.7% 0.6% 49.0% 0.4% Average 28.0% 0.9% 30.4% 1.4% 32.7% 0.9% 35.6% 1.0% 37.8% 0.8% Avg. size whitelist 86, , , , ,230 Table 7. Comparing impacts of different time interval windows to candidate whitelists Conclusions: Time interval window has a linear relation to the size of the candidate whitelist and its DNS coverage: the larger it is, the bigger the size of the whitelist domains, and their DNS traffic. The percentage of estimated malicious domains is not linearly correlated with time intervals. ER5. Test what is the best expiration date length of whitelist domains. Settings: To generate all customized whitelists from Jan 2015 to April 2016, 60 days, 30 days, 14 days, and seven days were used as the expiration date. Then the researchers calculated the size of those whitelists, percentage of DNS traffic they covered, estimated percentage of malignant domains, and the similarity between two neighboring whitelists. Average results are shown here since there were multiple whitelists being created. Results are summarized in table 8. Settings: threshold=12 IPs, interval=1 hour Number of whitelist of DNS Estimated percentage of similarity with Expiration days domains covered of bad domains next WL , % 1.1% 37.2% 30 86, % 0.9% 40.0% 14 50, % 1.1% 45.9% 7 35, % 1.2% 52.0% Table 8. Comparing impacts of different whitelist expiration dates Conclusions: This group of results is as expected. First, size of the whitelist is decreasing with respect to expiration days, which is as expected. The covered DNS is around the 27 percent to 28 percent range, which is mainly determined by the fixed threshold and interval, as expected too. Estimated malignant domains are relatively stable around 1 percent, due to the fixed threshold and interval. Finally, similarity of the next whitelist in the sequential list is increasing when the expiration days are shorter, which is explainable the 14

17 shorter expiration days, the more likely the DNS traffic is similar. Considering the size of whitelist domains and DNS it covers, expiration days set at 30 are still very reasonable and balanced. ER6. Test what is better for updating frequency to the whitelist Settings: Updating frequency at two weeks, seven days, and one day are the considering choices. Settings: t=12 IPs, interval=1 hour, expiration days=30 Number of new domains Update frequency Number of whitelist domains to previous list 14 days 50,951 27,346 7 days 35,223 16,880 1 day 13,111 5,617 Conclusions: The results in this experiment were also as expected. If updating the whitelist partially in a longer range, the size of the whitelist is indeed larger, and compared to the last active whitelist, the new domains to be added are larger. Therefore, if using a short updating frequency e.g., every day only about 5k new domains are added to the list researchers only need to verify 5k domains each day from the reputation system. Hence, updating daily is the preferred strategy, as it will have the lowest requirement for a reputation service subscription. (Notice that it is also a good strategy to update the whitelist at every t interval hour to generate faster results, which has same effect as daily update.) Preferred parameters after experiments Based on the above experiments, the researchers concluded the following best practices for preferred parameters: a. Threshold x = 12 b. Time interval window t = one hour c. Expiring date exp_days = 30 days d. Updating frequency freq = one day (or one hour if need faster effect) System Architecture System Design Once the algorithm and validation designs are complete, it is time to define the formal process for implementing the algorithm, verifying and saving the whitelist, and providing guidelines to integrate results from other research or products. Micro Focus proposes the system architecture and data flow shown in figure 2 to implement the algorithm and strategy discussed in the preceding paragraphs. 15

18 White Paper Creating Customized Whitelist Domains from DNS Traffic Database Daily DNS traffic For every t-hours, extract domains queried by >t IPs, and is not blacklisted/dga Aggregate all t-hour domains into daily (24-hours) whitelist DNS traffic DNS traffic table Customized whitelist table Expiring whitelist older than exp_days Customized whitelist Verify with reputation services to remove malign domains Figure 2. Data flow and system architecture of the customized whitelist creation Whole algorithm setup and running process a. Pre-requirement I. Create new whitelist domain table WhiteListDomains, including fields: rank (integer), tag (varchar 255), domain (varchar 255), top_domain (varchar 255), d0 (varchar 255), d1 (varchar 255), d2 (varchar 255), created_at (timestampz), expire_days (integer), num_ip_threshold (integer). II. Create an independent table VTCheckResults to save results searched by VirusTotal, need fields: domain, reputation_score, and search_time. b. Initial execution process I. Run the WL creation SQL and aggregation algorithm for a one-month date range from a date point, e.g. 01 May 2016, with the given optimized parameters. II. Check every single candidate domain with VirusTotal (and save all search results to local storage table VTCheckResults), and remove malign domains to get the final customized whitelist. III. Insert the whitelist into WhiteListDomains, with rank as 1,000,001, tag as 30days-t12, created_at date as 30 May 2016, expire_days as 30, and num_ip_threshold as 12. c. Daily execution process I. Run the WL creation SQL algorithm every day, e.g. every night at 12:05 a.m., or every hour, with the preferred parameters to generate candidate whitelist. II. Check every single candidate domain with VirusTotal (and save search results into local storage VTCheckResults table), and remove malignant domains to finalize the daily customized whitelist. III. Insert this daily whitelist into WhitelistDomains, and set created_at date as the day. Note: d0, d1, and d2 are the top-level, secondlevel, and third-level domain string of a fullnamed domain. e.g., news.bbc.co.uk, d0 is co.uk, d1 is bbc, and d2 is news 16

19 Conclusion This whitepaper describes a way to create customized whitelist domains from a fixed or streamlined DNS traffic data, producing whitelists with a high quality of benign domains and decent coverage of traffic. Experimental studies on 16 months of DNS traffic proved that the proposed method indeed generated high-quality customized whitelists compared to industrial standard lists. The Micro Focus researchers also provided preferred parameters of the proposed method so that application of the algorithm can be easily adopted to other DNS traffic data. A few additional thoughts and improvements need to be discussed here to fully explore the value of the proposed methods in DNS analysis. 1. Scalability. Are the method and parameters scalable to other sizes of DNS traffic? Since the experimental data is million DNS requests per day (after Alexa list filtering) from hundreds thousands of IPs, it is relatively large compared to other organizations data. Hence, the method and preferred parameters can be directly applicable to other comparable sizes of DNS data. To leverage the value of this method fully, Micro Focus suggests applying additional optimization strategy. 2. Generality. Micro Focus proposes this customized whitelist method to be as general as possible and use the minimum information needed, so the same idea and process can be applied to other similar problems. For example, VirusTotal is the reputation system used in the experiments, but it is replaceable with other reliable ones. The DNS data can also come from HTTP or proxy traffic, as long as it has timestamp, request, and source IP. 3. Improvement to the proposed method. For example: Using dynamic threshold x, or soft threshold x. Dynamic means the x depends on active IPs in the traffic, and soft means using a percentage of threshold (e.g., 0.9*x) and then ranking them with DNS traffic size. Using the malignant domains identified in verification process. As those domains were requested from multiple (more than thresholds x) IPs, it is worth further investigation to find out if there is a true malware infection for those IPs. 4. Identify patterns of whitelist domains with clustering algorithms. Predicting new whitelist domains. If the created whitelist domains can predict new whitelist domains, it will save the effort to calculate or verify the whitelist; for example, there are many internal IP-bound domains in the whitelist, hence it makes more sense to write regex rules to recognize such internal domains. Clustering domains. Performing domain name clustering on the customized whitelist or to a general domain list to find new malware patterns could be much more valuable to all other security analysis. Learn More At /securitysolutions 17

20 Additional contact information and office locations: AA H 04/ Micro Focus. All rights reserved. Micro Focus and the Micro Focus logo, among others, are trademarks or registered trademarks of Micro Focus or its subsidiaries or affiliated companies in the United Kingdom, United States and other countries. All other marks are the property of their respective owners.