Review files from last week:

Transcription

1 Tinyurl for this etherpad: Review files from last week: Week 6 slides & tutorials: /week6 Selectors for HTML scraping - XPATH vs. CSS Multiple selector syntaxes, xpath, css, jquery, custom selectors. Openrefine uses jsoup which has CSS/jquery style lxml supports xpath, css, and its own language We'll use css selectors to keep things simple Firefox uses Xpath and CSS, Chrome only uses Xpath. API Basics "Application programming interface" Specially-constructed urls return data in a variety of different formats. JSON is easier to work with in python. Some APIs require authentication or keys. Keys are becoming more common. This provides an opportunity to understand who's using the API and an opportunity to identify bad applications, so they can revoke your API key. Most APIs have rate limits. These can be by IP as well as by dev key. It's a good idea to check and see if you're getting these bouncebacks somewhere in your code. API listings: API listings for library-related APIs: OpenLibrary API (astronomy subject query): OpenLibrary API documentation: It's possible to build an interface that uses jquery and other back-end services to make a search box with auto-suggestions. "OCLC: not as bad as it used to be"--but still pretty bad. API queries start with a "?", and generally look like this: /?q=library&format=json Parameters are seperated by ampersands If you're familiar with HTTP GET requests, this should look familiar. 1 of 7 10/17/13 11:47 PM

2 Before you start coding, you should always read the API documentation and look at some examples to see what you might want to return. Documentation: and /ois/systems/webservices/lookup-userguide/wwhelp/wwhimpl/js/html/wwhelp.htm Harvard Art Museums API documentation: /edit#heading=h.oraz453q9hth Fetching Data from APIs Here is an example jsonp API call for Hollis: /isbn/ ?jsonp=ws_result not all interpreters handle jsonp Version Control History Provenance Manage collaboration Revert if you screw up GitHub (or anything else with social infrastructure) helps to facilitate collaboration with notifications, etc Git has GUI options: desktop client, web application List of Git GUIs: Live Coding!!! url to cut & paste: size=20 If you missed the last class and need to install, see: /urls/dl.dropboxusercontent.com/u/ /dst4l-projects/week5/web_scraping_tutorial- TotalFilm_50_Adaptations.ipynb type into python editor: response = requests.get(' /search?q=constituentid:22730&size=20') json = response.json() print "Total records: %d" % json['info']['totalrecords'] 2 of 7 10/17/13 11:47 PM

3 pip installers for windows: Also you can install pip for windows by running the command prompt as an Admin and typing easy_install pip next url example: /isbn/ ?jsonp=callback Using the JSON version of the Hollis API... type into python editor: import json response = requests.get(' /isbn/ ?jsonp=callback') content = response.content[9:-2] json = json.loads(content) title = json['oai_dc:dc']['dc:title'] creator = json['oai_dc:dc']['dc:creator'] print '"%s" by %s' % (title, creator) you can use single or double quotes if you nest one inside the other, the inside ones will be intrepreted like regular characters print '"%s" by %s' % (title, creator) returns "Girl with the dragon tattoo" by Larsson, Stieg Note: in JSON, things in square brackets are arrays. You can have arrays of lists. It's fun. namespaces strip end of url to get xml instead of json: /isbn/ import xml.etree.elementtree as ET NAMESPACES = { 'dc': ' } using the XML version of the Hollis API... type into python editor: import xml.etree.elementtree as ET NAMESPACES = { 3 of 7 10/17/13 11:47 PM

4 'dc': ' } response = requests.get(' xml = ET.fromstring(response.content) content = response.content[9:-2] title = xml.find('dc:title', NAMESPACES) creator = xml.find('dc:creator', NAMESPACES) print '"%s" by %s' % (title.text, creator.text) published_in= &limit=5000 _cache requests_cache.install_cache('ol_cache') response = requests.get(' ebooks=true&published_in= &limit=5000') works = response.json()['works'] print len(works) openlibrary "subjects" are actually subjects + keywords + genre terms + other properties lots of different vocabularies together Slightly different syntax than Tom uses: This chunk of code creates a dictionary of authors, including a count of how many works they have created, then prints out all authors who've created more than 10 works. _cache requests_cache.install_cache('ol_cache') response = requests.get(' ebooks=true&published_in= &limit=5000') works = response.json()['works'] authors={} for work in works: for author in work['authors']: key=author['key'] if not key in authors: authors[key] = {'name':author['name'], 'works':1} else: authors[key]['works'] += 1 print '%d unique authors' % len(authors) print('the following authors have more than 10 works:') for author in authors: 4 of 7 10/17/13 11:47 PM

5 if authors[author]['works'] > 10: print authors[author]['works'],authors[author]['name'].encode('utf-8') reminder: indentation matters in python! also: dicts use {squiggly braces}; tuples use (parens); and lists use [square brackets] This chunk of code uses the 'Counter' function from the 'collections' module to give us the 10 most common subjects in our json. from collections import Counter authors = {} subjects = Counter() for work in works: for author in work['authors']: key=author['key'] if not key in authors: authors[key] = {'name':author['name'], 'works':1} else: authors[key]['works'] += 1 subjects.update(work['subject']) for subject, count in subjects.most_common(10): print(count, subject) New code chunk! Just the new stuff! BASE_URL = ' for author in authors: if authors[author]['works'] > 10: url = BASE_URL + author + '.json' response = requests.get(url) print url author_data = response.json() for k in ('birth_date','death_date'): if k in author_data: authors[author][k]=author_data[k] if 'photos' in author_data and author_data['photos']: authors[author]['photo']=' [0])+'-M.jpg' print authors[author] Output: {'works': 11, 'name': u'joannes de Sacro Bosco'} of 7 10/17/13 11:47 PM

6 {'birth_date': u'1564', 'photo': ' 'works': 26, 'name': u'galilei, Galileo', 'death_date': u'1642'}... PDF Table Extraction some PDFs are just scanned images some are scanned images backed with OCR text PDFs do not have any concept of words, lines, etc. Interpreting PDFs requires looking at characters and where they are positioned in relation to each other Tabula: - new open source project showing promise. Commercial service: ScraperWiki also has pdf extraction Python libraries Other tools dedupe - Natural language processing: NLTK python module - Book overview with examples as an introduction to NLTK, free online. We will come back to it later. PyMARC pip install PyMARC or easy_install PyMARC PyMARC has some documentation on its GitHub page: looks like someone taught a course on PyMARC Unix command line sampling cut -f 3 -d ' ' head looks at third column dlimited by a space cut -f 3-5 -d ' ' head 6 of 7 10/17/13 11:47 PM

7 looks at third through fifth column Belfer Center Group Example (Harvard Library's A-Z list) 7 of 7 10/17/13 11:47 PM