data analysis - basic steps Arend Hintze

Transcription

1 data analysis - basic steps Arend Hintze

2 1/13: Data collection, (web scraping, crawlers, and spiders) 1/15: API for Twitter, Reddit 1/20: no lecture due to MLK 1/22: relational databases, SQL 1/27: SQL, Overview data analysis techniques 1/29: data preprocessing 2/3: data visualization 2/5: project discussion 2/10: Intro to hadoop 2/12: hadoop getting started 2/17: hadoop programming 2/19: Intro to pig 2/24: More on pig 2/26: Intro to hive 3/3: BREAK 3/5: BREAK 3/10: midterm 3/12: classification 3/17: classification 3/19: regression time series prediction 3/24: market basket analysis 3/26: network analysis 3/31: cluster analysis 4/2: cluster analysis 4/7: outlier detection 4/9: project discussion 4/14: Intro to mahout 4/16: More on mahout 4/21: presentations 4/23: presentations 4/30: final exam

3 analysis methods we will talk about preprocessing, cleaning, normalization, transformation, selection, feature extraction visualization (get a peek, understand the raw data) classification ( you belong in this class! ) regression, time series prediction ( and what comes next? ) market basket analysis ( you should also buy: XYZ! ) cluster analysis ( where should I cut this mess? ) outlier detection ( you do not belong here! )

4 optional topics? information theory social network analysis compression text corpus analysis RFID, mobile devices, GPS, gadgets, sensors

5

6

7 IPv4 -> IPv6

8

9 big data methods: HADOOP (low level map reduce programming) PIG (high level map reduce interface) HIVE (perform SQL like database queries) MAHOUT (machine learning in the cloud)

10 data collection in the web - hacker style Arend Hintze

11 the internet

12 URLs

13 HTML <link> <link> HTML <link> <link> HTML <link> <img> HTML <link> <link> HTML <link> <link>.jpg

14 browser takes this:.jpg.jpg HTML <link> <link> <script> <img> <style> more.css

15 turns it into this:

16 however HTML is just a text file a text file written in HyperText Markup Language

17 HTML

18 static vs. dynamic pages static pages are indeed HTML only content never changes, or is changed by changing the file a program (php, server) creates a HTML file on the fly the HTML contains scripts (javascript) or (flash, media player ) a dynamic website can take parameters from the URL

19 turning the web into the data scrape, crawl, spider websites and extract the data automated programs load page after page (scrape) follow up on links (crawl, spider) search engines do this to get content use provided API (Application Programming Interface) websites provide dedicated interfaces (url schemes) often there are many modules/libraries for various languages usually require some kind of credentials get data directly from devices check card readers, RFID, cell phones, cameras usually requires dedicated hardware/software solution, no free/open access

20 but what is data? A collection of objects (items) and their associated attributes an object specifies an entity to be described: customer, product, page, day, location an attribute describes or characterizes the object: eye color, temperature, value, price, time

21 10 table format Attributes Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No Objects 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes

22 let s take a look at MapQuest search term: Starbucks URL?

23 search term zip code goto example->

24 JSON java script object notation allows you to transform an object into a JSON string allows you to transform a JSON string into a JS object each language (more or less) has it s own wrapper and parser tutorial:

25 anatomy of a json string [] LIST : Key:Value pair separator, enumerator {} nesting element u name :u Arend key=u name :value=u Arend u address :{u street :u Broad Street,u town :u East-Lansing } u Phonenr :[ , , ]

26 [ ] { }, { } name : Ernie, phone : { home : , mobile : } name : Bert, phone : { home : , mobile : } book[0][ name ] -> Ernie book[1][ phone ][ home ] = book[0][ phone ][ home ]

27 [ { name : Ernie, phone : { home : , mobile : } }, { name : Bert, phone : { home : , mobile : } } ] <XML> <LIST> <DICT> <KEY> <STRING>name</STRING> </KEY> <VALUE> <STRING>Ernie</STRING> </VALUE> <KEY> <STRING>phone</STRING> </KEY> <VALUE> <DICT> <KEY> <STRING>home</STRING> </KEY> <VALUE> <STRING> </STRING> </VALUE> <KEY> <STRING>mobile</STRING> </KEY> <VALUE> <STRING> </STRING> </VALUE> </DICT> </VALUE> </DICT> </LIST> </XML>

28 Regular Expressions

29 Regular Expressions

30 Regular Expressions I really only use google to find the answer that approach works 95% of the time

31 discussion: difference between RE and JSON approach to scraping MapQuest?

32 MapQuest has an API! Application Programming Interface the company wants you to use those not every API is free usually fast, reliable, and supported some APIs work on one platform only iphone SDK works only in objective C more about this next Monday

33 crawler / spider each HTML page has links to other pages: <a href=" W3Schools</a> following up on these links allows to automatically gather data

34 <a href > <a href > <a href > <a href > <a href >! <a href > <a href > <a href > <a href > <a href > <a href > <a href > <a href >! <a href > <a href > <a href > <a href > <a href > <a href > <a href > <a href >

35 crawler constraints do not load the same page twice branching factor determines if you come to an end or have exponential growth load new pages either breadth first or depth first? when to stop?

36 anatomy of a web crawler Initialize: append seed URLs to Queue Terminate? yes Terminated no Dequeue: remove a URL from Queue Fetch: retrieve web page associated with URL Parse: extract URLs from retrieved web page Enqueue: append extracted URLs to queue depth first or breadth first?

37 let s do it

38 was that allowed? Robot Exclusion Protocol! each site can have a robot.txt file which specifies if crawling is permitted legally binding? even if the data is public, keep in mind that you are using someone else's resources! allows and disallows crawling

39

40 Meta Tags META tags on a webpage also tell a crawler what not to do Meta tags are placed between <head> </head> tags in HTML <META NAME="ROBOTS" CONTENT="NOFOLLOW">! To not follow links on this page <META NAME="ROBOTS" CONTENT= NOINDEX">! To not appear in Google s index <META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">! To not archive copy in search results 1 h#p://googleblog.blogspot.com/2007/02/robots5exclusion5protocol.html:

41 Wget A freely available GNU utility for web crawling Supports both HTTP and FTP Can recursively traverse the structure of HTML documents and FTP directory trees Can specify wildcards to match certain types of files Can restrict the maximum depth of the directory traversed Available with both command line argument and graphical user interface Included in most Unix and Linux systems

42 Wget example wget Retrieve the index.html file from wget -t 30 Retry 30 times if access fails wget -r Recursively retrieve files under the hierarchy structure of (default: recurse up to 4 levels) wget -o log.txt Direct output messages to log.txt file For more examples:

43 exercise!