CHAPTER 5 URL ANALYSIS

Size: px

Start display at page:

Download "CHAPTER 5 URL ANALYSIS"

Lynette Richardson
5 years ago
Views:

1 112 CHAPTER 5 URL ANALYSIS 5.1 INTRODUCTION The Web has become a platform for supporting a wide range of criminal enterprises such as spam-advertised commerce, financial fraud and as a vector for propagating malware. The precise commercial motivations behind these schemes may differ but the common thread among them is the requirement that unsuspecting users visit their sites. These visits can be driven by , web search results or links from other Web pages, but all require the user to take some action, such as clicking, that specifies the desired Uniform Resource Locator (URL) and obtains sensitive information. In order to overcome this problem, the security community has responded by developing blacklisting services encapsulated in toolbars, appliances and search engines that provide an alert or warning precisely as feedback. Many malicious sites are not blacklisted either because they are too new, or never evaluated, or not evaluated incorrectly. In order to address this problem, some client-side systems analyze the content or behavior of a Web site as it is visited which causes runtime overhead due to browser based vulnerabilities. Phishing attacks are referred as Lure, Hook and Catch (Jacobsson and Myers 2007). s addressed to the victims seem to come from legitimate company addresses but in fact they are spoofed. These addresses are called the Lure. Usually, the s contain URLs that refer to the actual phishing sites which are clones of legitimate websites and lure the

2 113 users into entering sensitive information. The actual phishing websites are the Hook which obtains the private information from the user. In order to make sure that the innocent user doesn t suspect the to be fraudulent one, the text of the should be legitimate. The attacker create various plausible conditions by a message such as account suspension, failed transaction or even upgrading of the user s account to the newly installed security feature. Once the user clicks the link in the , it is automatically taken to the fake phishing site. It is referred as Catch. The legitimacy of the website may or may not be displayed by the browser depending upon a number of heuristics used by the browser to detect phishing. In some cases, the user also overrides the browsers decision Need for URL Analysis The websense report 2012 states that the spam has been increasing at regular paces and hackers adopt new techniques every day. The shift to blended threats using as a lure and web links remains strong as 92% of spam contains a URL. The websense report 2011, the spam statistics states that 12% of spam messages refer to shopping, 84% of all messages are spam, 89.9% of unwanted links to spam sites or malicious websites, 85% of malicious s contain a web link and 9% of data-stealing attacks occur over . spam correlated to phishing came at 1.62% while virus-related spam was 0.4%. 5.2 EXISTING SOLUTIONS Blacklists are often used by filters and browsers to block users from the malicious content (e.g., messages and websites). PhishNet (Pawan et al 2010) enhances existing blacklists by discovering related malicious URLs. One major problem with blacklists is that they fail to identify phishing URLs in the early hours of a phishing attack because their update process is insufficiently fast. Phishing campaigns have an average life

3 114 of less than two hours (Sheng et al 2009) and by the time a phishing website is positively identified and blacklisted, it would have most probably has ended and a new one started. Detecting phishing websites by is a helpful phishing countermeasure and researchers have attempted to detect phishing websites using features extracted from the URL. Illustrations of URL based features include but are not limited to the number of dots in the URL, length of the machine names, number of special characters, presence of hexadecimal characters or IP addresses instead of machine name, and length of the URL (Garera et al 2007, Justin Ma et al 2009). Garera et al (2007) extracted 18 host- and URL-based features from potential phishing URLs and classify the features using Logistic Regression. On a data set of 2,508 URLs, of which 1,245 were phish, the classifier provided a 95.8% true positive rate and 1.2% false-positive rate. Colin Whittaker et al (2010) discussed a scalable machine learning algorithm to automatically classify phishing pages by training the classifier on noisy dataset. The classifier is used to maintain Google s phishing blacklist automatically by analyzing millions of pages a day, examining the URL and the contents of a page and maintains a false positive rate below 0.1%.Justin Ma et al (2009) discuss a method to detect malicious websites by analyzing features indicative of suspicious URLs. They used passive aggressive algorithm and explored online learning approaches for detecting malicious websites using lexical and host-based features of the associated URLs. The improvement can be obtained by analyzing the features of page content and page rank. Zhang et al (2007) proposed a content-based method using a simple linear classifier on top of eight features, achieving 89% TP and 1% FP on 100 phishing URLs and 100 legitimate URLs. CANTINA+ (2010) classifies phishing URLs and the feature set is more exhaustive and obtained a classification accuracy of 92.3%. There exist various related researches and case studies conducted on analyzing the feature set required to reduce the exhaustiveness and time consumption. The usability

4 115 study experiment to evaluate the accuracy and the precision of various phishing website features were previously collected and analyzed by Maher Abburrous et al (2010). The set of phishing attacks and tricks to measure their effectiveness and influence were collected from the APWG s archive (2011) and Phishtank archive (2012). The purpose is to find the most common and essential phishing clues that appear in the scenarios, to determine what aspects of a website effectively convey authenticity and to identify which malicious strategies and attack techniques are successful at deceiving general users and why. Some of the observation on the high impact features for phishing attacks is listed in Table 5.1. Table 5.1 High impact features on URL phishing instances S.No. Phishing Features No. of appearances Appearance % 1. Using the IP address Abnormal request URL Abnormal URL of anchor Abnormal DNS record Abnormal URL Using SSL certificate Certification authority Abnormal cookie Distinguished Names Certificate (DN) Redirect pages Straddling attack Pharming attack Using on MouseOver to hide the link Server Form Handler (SFH) Spelling errors Copying website Using forms with Submit button Using Pop-Ups windows Disabling right click Long URL address Replacing similar characters for URL Adding prefix or suffix Using symbol to confuse Using hexadecimal character codes Much emphasis on security and response Buying time to access accounts

5 116 The above selected features ( ) display high impact in various studies as mentioned in the literature and hence the feature set comprises features whose impact is greater than 20%. This involved the host based features, lexical features, page rank and suspicious keywords in the mail for better performance. 5.3 URL ANALYZER Phishing URLs can be analyzed based on the lexical features and host based features of the URL and the structures are shown in Figure 5.1. The lexical feature analyses the format of the URL. An URL consists of two parts the hostname and the path. As an example, with the URL the hostname portion is and the path portion is emmrc/emmrc.html Figure 5.1 Feature analyzer The proposed methodology analyses the hosts based features such as Pagerank and age of domain, various lexical based features such as URL encoding, presence of suspicious characters, hexadecimal character or malicious IP addresses to hide them and analyses the word probabilities to

6 117 find whether the contains any suspicious links to avoid end users falling by phishing attacks as illustrated in Figure 5.2. This method is quite useful as illegitimate users spoof their identities and it may pass authentication tests and during content analysis also it may get escaped by avoiding spam keywords. Some s may not contain any message in the body except some malicious links in it urging the users to click them leading to fraudulent websites. Figure 5.2 URL feature extraction Lexical Features (F 1 ) Lexical features are the textual properties of the URL itself (not the content of the page it refers). These properties include the length of the hostname, the length of the entire URL, as well as the number of dots in the URL, binary feature for each token in the hostname (delimited by. ) and in the path URL (strings delimited by /,?,., =, - and _ ). This is also known as a bag-of-words IP address Phishing URLs often contain IP addresses to hide the actual URL and domain of the website. For instance, a website URL may be extremely long and look suspicious such as

7 118 markswebsite/ todaysphishingpage.html but the URL that contains the IP address is typically shorter and more standard such as URL detection methods looks for an IP address in the URL and add to a phishing score if one is found. However, the legitimate websites also sometimes use IP addresses especially for internal private devices that aren t accessible to the public. Network devices such as routers, servers, and networked printers are often accessed using an IP address Hexadecimal characters The URL can be represented with a numeric value, each character on the keyboard that the computer understands. This numeric decimal value can easily be converted into hexadecimal base. Web browsers can understand hexadecimal values and they can be used in URLs by preceding the hexadecimal value with a % symbol. For instance, the value %20 is the hexadecimal equivalent of the space character on the keyboard Suspicious character Spoofguard (Neil Chou et al 2004) identified two characters common in phishing URLs, and - character. The username proceeds symbol and the destination URL follows symbol. symbol in a URL causes the string to the left to be disregarded, with the string on the right treated as the actual URL for retrieving the page which is a phishing site. For example the URL phishingsite.com will navigate to the destination URL which is phishingsite.com and will attempt to login using as the username. Hence, the actual URL of the website is disguised and when combined with an IP address it can really hide the phishing site while the URL appears to be legitimate.

8 Number of dots in URL This feature counts the number of dots in the URL. Phishing pages tend to use more dots in their URLs than the legitimate sites. All the lexical features are denoted as a single feature set F 1. After examining the dataset, 1000 phishing mails and 1000 legitimate mails the occurrence of the lexical feature is as follows in the Table 5.2. Table 5.2 Number of occurrence of lexical features in training samples IP address More Dots Encoded Symbol Suspicious characters Phishing mail Legitimate mail Host Based Features Host based features can describe where malicious sites are hosted, who own them, and how they are managed. The following are properties of the hosts (there could be multiple) that are identified by the hostname as part of the URL Age of domain (F 2 ) This feature checks the age of the webpage domain name. Many phishing sites are hosted on recently registered domains, and as such have a relatively young age. In order to exploit that property, this feature measures the number of months since the domain name is first registered. The WHOIS 6 lookups on the WHOIS server is used to retrieve the domain registration date, and if the domain registration entry is not found on the WHOIS server, this 6 WHOIS - Internet service that finds information about a domain name or IP address.

9 120 feature will simply return-1, deeming it suspicious. The occurrence of the feature in the training sample is as in Table 5.3. Table 5.3 Number of occurrence of Age of Domain feature in training samples Dataset Age of the domain Phishing mail 750 Legitimate mail Page rank (F 3 ) Page rank represents the relative importance of a page within a set of web pages. The higher the page rank, the more important is the page. Phishing web pages are short lived and thus either have a very low page rank or their page rank does not exist. Page rank is a link analysis algorithm first used by Google, in which each document on the web is assigned a numerical weight from 0 to 10, with 0 indicating least popular and 10 meaning most popular. A score value of 1 is assigned when the page rank value for a particular webpage is not available. The occurrence of the page rank feature in the training sample is as in Table 5.4 and Figure 5.3. Table 5.4 Number of occurrence of Page rank feature in training samples Phishing mail Legitimate mail

10 121 MAILS PAGE RANK Phishing mail Legitimate mail Figure 5.3 Page rank feature in training samples Among the training samples, the percentage of s matching the Lexical and Host based features are listed in Table 5.5. Table 5.5 Percentage of s matching the lexical and host based features Feature Non-phishing Phishing Matched Matched Has IP Address 0% 0.04% Has Hexadecimal Character 0% 0.01% Has suspicious symbol 0% 0.01% More No. of Dots 0.01% 0.06% Suspicious Age of Domain 35% 75% Page rank< 3 feature 1.2% 88% Number of Sensitive Words in URL Individual occurrences of suspicious phishing keywords (F 4 ) Abu-Nimeh et al (2007) adopted the bag-of-words strategy and simply used a list of 43 most frequent words as features in a machine learning approach. Garera et al (2007) summarized a set of eight sensitive words such as secure, account, update, login, sign-in, banking, confirm and Verify that

11 122 frequently appear in phishing URLs. The system is trained with 1000 phishing s to give weights to the suspicious words found in the phishing s. The count of most occurring words as in Table 5.6 and Figure 5.4 in the phishing mail is analyzed and hence these words can be assumed as suspicious words by which the phishing mails can be identified. Table 5.6 Number of occurrences of suspicious phishing keywords Keywords No. of Occurrences Secure 570 Account 750 Update 240 Login 150 Signin 60 Banking 220 Confirm 320 Verify 330 Notify 130 Click 340 Inconvenient 250 Password 580 No.Of OCccurences No. of Occurences of Suspicious Keywords Suspicious Keywords Figure 5.4 Number of occurrences of suspicious phishing keywords

12 Co occurrences of suspicious keywords(f 5 ). The table 5.7 shows the count of prominent words in 1000 phishing mails, the correlation between the words and used their correlation as a score to classify the s by counting their number of occurrences. Table 5.7 Number of co-occurrences of suspicious phishing keywords Secure -- Account Secure Account Up date Update Log In Log in Sign In Banking Confirm Verify Sign in Banking Confirm Verify Notify Click In Convenient Notify Click In Convenient The content is parsed at that instant the content is checked for the presence of any embedded forms and then the words in the are checked if it contains any suspicious words. If any suspicious words are found, the score for each word is calculated and correlation score of those words are also calculated and are added up to the total score LOGIN FORM DETECTION (F 6 ) Almost all phishing attacks try to trick people into sharing their information through a fake login form. A login form is characterized by FORM tags, INPUT tags, and LOGIN keywords such as password, PIN, etc. INPUT fields are usually used to hold user input. Usually, form tags, input

13 124 tags and login keywords appear in the DOM. Login keywords are searched in the text nodes as well as the alt and title attributes of element nodes of the sub tree rooted at the form node. Consider when form and input tags are found, but login keywords exist outside the sub tree rooted at the form node f. Examine whether the form f is a search form by searching for keyword search. Iffis not a search form, traverse the DOM tree up forklevels +1 to ancestor noden, and search login keywords under the subtree rooted atn. 5.4 APPROACH Training Set Bayes Classifier Commonly used in spam filters, the bayes model assumes that for a given label, the individual features of URLs are distributed independently of the values of other features. Bayes theorem provides a way to calculate the probability of a hypothesis, for the event B, given the observed training data, represented as A: P(B A) = ( ) ( ) ( ) (5.1) This simple formula has enormous practical importance in many applications. It is often easier to calculate the probabilities, ( ), P(A), P(B) for the probability ( ) that is required. Extrapolating Bayes rule, assuming that malicious and legitimate Web sites occur with equal probability, compute the posterior probability that the feature vector x belongs to a malicious URL as P(B = 1 A) = ( ) ( ) ( ) (5.2) P(B A) = ( ) ( ) ( ) (5.3) P(B A) = ( ) ) ( ) ( ) ( ) ) (5.4)

14 125 where, P(A) = Probability of feature F in phishing and legitimate dataset. P(B ) = Legitimate dataset. P(B) = Phishing dataset. P(B(Phishing)) = P(B (Legitimate)) = 0.5 The classifier has a training dataset of malicious phishing URLs and legitimate URLs. The probability occurrence of each feature in the dataset are calculated and their respective scores are obtained (i.e) Count up occurrence of features in the dataset and calculate the cumulative score. If Cumulative score > Threshold, consider as phishing URL else legitimate URL as illustrated in Figure 5.5. Figure 5.5 Phishing URL classifications a) How many times does feature F(F 1,F 2,F 3,F 4,F 5, F 6 ) appear in phishing dataset? b) How many times does feature F(F 1,F 2,F 3,F 4,F 5, F 6 ) appear in legitimate dataset? Let F 1 = Lexical features F 2 = Age of the domain factor of URLs F 3 = Occurrence of Pagerank < 3 in phishing and legitimate dataset

15 126 F 4 = Individual Occurrence of suspicious keywords F 5 = Co Occurrences of suspicious keywords F 6 = Login Form detection Calculating Probability In order to calculate the probability of a specific feature in the phishing dataset, consider 2000 URLs 1000 phishing URLs and 1000 legitimate URLs. Feature F 1 (Lexical features) The feature F 1 involves the occurrence of lexical features that appeared in 120 phishing URLs and 20 legitimate URLs. Hence its probability is calculated as follows. P(A B) P(B A) = P(A B) + P(A B ) P(B A) = ( ) ( ) ( ) = 0.86, since P(B(Phishing)) = P(B (Legitimate)) = 0.5. Feature F 2 (Age of Domain) The feature F 2 (Age of domain) appeared in 750 phishing URLs and 350 legitimate URLs. Hence its probability is calculated as follows. P(A B) P(B A) = P(A B) + P(A B ) P(B A) = ( ) ( ) ( ) = 0.68.

16 127 Feature F 3 (Page rank) The feature F 3 (Page rank) appeared in 880 phishing URLs and 120 legitimate URLs. Hence, its probability is calculated as follows. P(A B) P(B A) = P(A B) + P(A B ) P(B A) = ( ) ( ) ( ) = DATA SETS The datasets are obtained from two sources viz DMOZ Open Directory Project and Phishtank (2012). Phishtank is a blacklist of phishing URLs consisting of manually-verified user contributions. Phishtank focuses on phishing URLs advertised in spam (phishing, pharmaceuticals, software, etc.). Both sources include URLs crafted to evade automated filters, while phishing URLs in particular visually tricks the users as well. Phishtank consists of phishing instances, a large community-based anti-phishing service with active accounts and verified phishes. 5.6 RESULTS Test Cases An server has been configured with hmail 7 named as SSE Mail Server for the testing purposes,. The system is tested against phishing URLs present in the and the feature found in each URL is noted. This is repeated for 1000 phishing URLs from which weights for each feature has been calculated. 7 hmailserver - Free server for Microsoft Windows used by Internet service providers and companies supporting the common protocols (IMAP, SMTP and POP3) and can easily be integrated with many existing web mail systems.

17 128 Table 5.8 Performance analysis with the existing systems Technique Cantina (Existing) (with n1 features) Cantina+ (Existing)(with n2 features) URL Classifier (Proposed)(with m features) Number of features (n) TPR (%) FPR (%) n1(20) 89 1 O(n1) Time Complexity n2(27) O(n2) (n1<n2) m(14) O(m) (m<n2) The false positive rate corresponds to the proportion of legitimate s classified as phishing s, and false negative rate corresponds to the proportion of phishing s classified as legitimate. The Table 5.8 shows that out of 1000 Phishing mails with malicious URLs, the above results were obtained for identifying various lexical and host based features. The following are the sample shots for malicious URLs embedded in s such as encoded URLs and embedded forms as in Figure 5.6 and Figure 5.7 respectively. Figure 5.6 A snapshot showing encoded URL in

129 Figure 5.7 A snapshot showing embedded forms in E-Mail 5.

18 129 Figure 5.7 A snapshot showing embedded forms in 5.7 CONCLUSION Hackers bypass anti-spam filtering techniques by embedding malicious URL in the content of the messages. Hence the URL analyzer method with the help of minimized phishing feature set identifies the malicious URL in the s.

IJSRD - International Journal for Scientific Research & Development Vol. 4, Issue 03, 2016 ISSN (online):

IJSRD - International Journal for Scientific Research & Development Vol. 4, Issue 03, 2016 ISSN (online): 2321-0613 Tweet Analysis for Malicious Content using Hybrid System Aditya Jadhav 1 Shital Salve