An Introduction to Web Scraping with Python and DataCamp

Transcription

1 An Introduction to Web Scraping with Python and DataCamp Olga Scrivner, Research Scientist, CNS, CEWIT WIM, February 23,

2 Objectives Materials: DataCamp.com Review: Importing files Accessing Web Review: Processing text Practice, practice, practice! 1

3 Credits Hugo Bowne-Anderson - Importing Data in Python (Part 1 and Part 2) Jeri Wieringa - Intro to Beautiful Soup 2

4 Importing Files

5 File Types: Text Text files are structured as a sequence of lines Each line includes a sequence of characters Each line is terminated with a special character End of Line 4

6 Special Characters: Review 5

7 Special Characters: Answers 6

8 Modes Reading Mode r Writing Mode w 7

9 Modes Reading Mode r Writing Mode w Quiz question: Why do we use quotes with r and w? 7

10 Modes Reading Mode r Writing Mode w Quiz question: Why do we use quotes with r and w? Answer: r and w are one-character strings 7

11 Open - Close Open File - open(name, mode) name = filename mode = r or mode = w 8

12 Open New File 9

13 Open New File 9

14 Read File Read the Entire File - filename.read() Read ONE Line - filename.readline() Read lines - filename.readlines() - Return the FIRST line - Return the THIRD line 10

15 Read File Read the Entire File - filename.read() Read ONE Line - filename.readline() Read lines - filename.readlines() - Return the FIRST line - Return the THIRD line What type of object and what is the length of this object? 10

16 Python Libraries

17 Import Modules (Libraries) Beautiful Soup urllib More in next slides... For installation - intro-to-beautiful-soup 12

18 Review: Module I To use external functions (modules), we need to import them: 1. Declare it at the top of the code 2. Use import 3. Call the module 13

19 Review: Modules II To refer and import a specific function from the module 1. Declare it at the top pf the code 2. Use from import 3. Call the randint function from random module: random.randint() 14

20 How to Import Packages with Modules 1. Install via a terminal or console Type command prompt in window search Type terminal in Mac search 15

21 How to Import Packages with Modules 1. Install via a terminal or console Type command prompt in window search Type terminal in Mac search 2. Check your Python Version 3. Click return/enter 15

22 Python 2 (pip) or Python 3 (pip3) pip or pip3 - a tool for installing Python packages To check if pip is installed: 16

23 Web Scraping Workflow

24 Web Concept 1. Import the necessary modules (functions) 2. Specify URL 3. Send a REQUEST 4. Catch RESPONSE 5. Return HTML as a STRING 6. Close the RESPONSE 18

25 URLs 19

26 URLs 1. URL - Uniform/Universal Resource Locator 2. A URL for web addresses consists of two parts: 2.1 Protocol identifier - http: or https: 2.2 Resource name - datacamp.com 19

27 URLs 1. URL - Uniform/Universal Resource Locator 2. A URL for web addresses consists of two parts: 2.1 Protocol identifier - http: or https: 2.2 Resource name - datacamp.com 3. HTTP - HyperText Transfer Protocol 4. HTTPS - more secure form of HTTP 5. Going to a website = sending HTTP request (GET request) 6. HTML - HyperText Markup Language 19

28 URLLIB package Provide interface for getting data across the web. Instead of file names we use URLS Step 1 Install the package urllib (pip install urllib) Step 2 Import the function urlretrieve - to RETRIEVE urls during the REQUEST Step 3 Create a variable url and provide the url link url = Step 4 Save the retrieved document locally Step 5 Read the file 20

29 Your Turn - DataCamp DataCamp.com - create a free account using IU 1. Log in 2. Select Groups 3. Select RBootcampIU - see Jennifer if you do not see it 4. Go to Assignments and select Importing Data in Python 21

30 Today s Practice 22

31 Importing Flat Files urlretrieve has two arguments: url (input) and file name (output) Example: urlretrieve(url, file.name ) 23

32 Importing Flat Files 24

33 Opening and Reading Files read_csv has two arguments: url and sep (separator) pd.head() 25

34 Opening and Reading Files read_csv has two arguments: url and sep (separator) pd.head() 26

35 Importing Non-flat Files read_excel has two arguments: url and sheetname To read all sheets, sheetname = None Let s use a sheetname

36 Importing Non-flat Files 28

37 HTTP Requests read_excel has two arguments: url and sheetname To read all sheets, sheetname = None Let s use a sheetname

38 GET request Import request package 30

39 HTTP with urllib 31

40 HTTP with urllib 32

41 Print HTTP with urllib Use response.read() 33

42 Print HTTP with urllib 34

43 Return Web as a String Use r.text 35

44 Return Web as a String 36

45 Scraping Web - HTML 37

46 Scraping Web - HTML 37

47 Scraping Web - BeautifulSoup Workflow 38

48 Many Useful Functions soup.title soup.get_text() soup.find_all( a ) 39

49 Parsing HTML with BeautifulSoup 40

50 Parsing HTML with BeautifulSoup 41

51 Turning a Webpage into Data with BeautifulSoup soup.title soup.get_text() 42

52 Turning a Webpage into Data with BeautifulSoup 43

53 Turning a Webpage into Data - Hyperlinks HTML tag - <a> find_all( a ) Collect all href: link.get( href ) 44

54 Turning a Webpage into Data - Hyperlinks 45