Cloak of Visibility. -Detecting When Machines Browse A Different Web. Zhe Zhao

Transcription

1 Cloak of Visibility -Detecting When Machines Browse A Different Web Zhe Zhao

2 Title: Cloak of Visibility -Detecting When Machines Browse A Different Web About Author: Google Researchers Publisher: IEEE 2016 Symposium on Security and Privacy Copyright Date: 2016

3 What is Cloaking?

4 Contribution - We provide the first broad study of blackhat cloaking techniques and the companies affected - We build a distributed crawler and classifier that detects and bypasses mobile, search, and ads cloaking, with 95% accuracy and a false positive rate of 0.9% - We measure the most prominent search, and ads cloaking techniques in the wild; we find 4.9% of ads and 11.7% of search results cloak against Google s generic crawler - We determine the minimum set of capabilities required of security crawlers to contend with cloaking today

5 Background & Related work Web Cloaking Incentives - Search Engine Optimization (SEO) - Advertisements - Drive-by Downloads* Prevalence of Cloaking - 2.2% of Google Search - 61% related to certain merchandise - 31% related to pharmaceutical keywords

6 Underground Perspective of Cloaking Select 10 prominent cloaking software - Exhaustively crawled and indexed a selection of underground forums - Webpage process - key words frequency - Range of price : $167 - $13,188

7 Analyzing goals 1. Fingerprinting capabilities 2. Switch logic for displaying targeted content 3. Other built-in SEO capabilities a. Content spinning b. Keyword stuffing Initial result - Various language, C++, Perl, JavaScript, and PHP - Sophistication range, drop-in scripts and plugins for Wordpress, custom compilation of Nginx

8 Cloaking Techniques 1. Network Fingerprinting - IP Address (54,166 unique ip)

9 Cloaking Techniques cont. 2. Browser Fingerprinting - Http header : User-Agent - E.g. googlebot, bingbot,msnbot etc.

10 Cloaking Techniques cont. 3. Redirection techniques - Stopped at a doorway - Redirected to an entirely different domain 4. Build-in SEO services - Automatic content generation - Targeted search terms from software s operator - Query popular search engine

11 Detecting Cloaking 1. Aggregating a diverse sample of URLS 2. Fetch each of URLS via multiple browser profiles & network vantage point 3. Compare the content, structure and redirect graph associated with each fetch 4. Feeding these features into a classifier to detect the presence of blackhat cloaking

12 URL selection

13 Browser Configuration

14 Context selection & Network Selection Key words - Non-ads based URL - frequency of top 3 words - Ads based URL Network - Rely on the keywords the advertiser bids on for targeting ( gather from Google AdWords) - Google s network, AT&T or Verizon, Google Cloud datacenter, residential IP

15 Similarity Content similarity - Visible text comparison Screenshot similarity - Visual differences in the layout and media presented to browsing profile of the same window dimension Element similarity - Extract the set of embedded images per document, calculate the difference in media content (Jaccard similarity coefficient)

16 Similarity Cont. Topic Similarity - Extracting representative topics based on visible text

17 Classification Extremely Randomized Trees - Ensemble, non-linear, supervised learning model - Candidate features and thresholds are selected entirely at random - Normalize all features into a range (0,1) - Rely on ten-fold cross validation

18 Performance

19 Performance

20 Summary - Explored the cloaking arms race playing out between security crawlers and miscreants seeking to monetize search engines and ad networks. - Informed our system s design by directly engaging with blackmarket specialists selling cloaking software and services to obtain ten offerings - Developed our crawler into the wild to scan 135,577 unknown URLs, we found 11.7% of the top 100 search results related to luxury products and 4.9% of advertisements targeting weight loss and mobile applications