Part 1. Learn how to collect streaming data from Twitter web API.

Transcription

1 Tonight Part 1. Learn how to collect streaming data from Twitter web API. Part 2. Learn how to store the streaming data to files or a database so that you can use it later for analyze or representation purpose Part 3. Lean how to play with stored data in Python environment

2 Part 1. Introduction to Twitter Tweets, Retweets, Mentioning, Hashtag Intro to Twitter Dev Intro to HCDE user module

3 Outline What is Twitter? UI and Interactions Twitter Dev Reading the REST API specifications Installing HCDE python user module

4 What is Twitter? Micro blog Status messages 140 characters max Links Social Follow (be followed) Groups

5 What is Twitter? Login screen Create account

6 What is Twitter? Your main page Public Timeline People you *follow* Promoted content Advertisements Trending topics Recommendations

7 What is Twitter? Getting to your tweet history

8 What is Twitter? Getting to your tweet history

9 What is Twitter? Your tweet history

10 What is Twitter? Your friends The people who you follow

11 What is Twitter? Your followers The people who follow you

12 What is Twitter? Compose, write a tweet Write a status update

13 What is Twitter? Compose, write a tweet

14 What is Twitter? You can direct message people who follow you To have a direct message conversation you need to have a reciprocal follow!

15 What is Twitter? Yes, you can search Tweet content People Organizations Lists

16 Twitter Conventions A mention or a directed

17 Twitter Conventions A mention or a directed A re-tweet (or) Old style almost never see this More and more tools automate this, Now part of the metadata

18 Twitter Conventions Categorization, tagging #category URL Shortening URLs are long, too many characters Services like bitly.com, tinyurl.com, Twitter has t.co

19 Intro to Twitter Development How to read the specifications

20 Goals Set up things to develop Twitter Applications How to read and understand the specification REST REpresentational State Transfer Web model, where the state of the communication or the state of the communication/application is transferred each time a client interacts with a host API Application Programming Interface

21 Twitter Dev

22 Twitter Dev Create a Developer account using your own twitter account Allows you to create new twitter applications Read the documentation

23 Twitter Dev Create a Developer account using your own twitter account Allows you to create new twitter applications Read the documentation

24 Twitter Dev I have a lot of twitter applications that I use long list of my twitter applications I created two for this course: HCDE530Test01 read only HCDE530Test02 read/write

25 Twitter Dev A twitter app has two keys essential for the Twitter authentication model (OAuth2) Consumer Key Consumer Secret When a user authorizes your app, that user also gets two keys

26 Twitter API Documentation This course focuses on the REST API Documentation: Data that you want to get is a Resource Tweets, user profiles, friends, followers You access a "resource" through a URL Almost like requesting a web page Twitter resources require authentication (OAuth2) The URL query string includes special parameters

27 Twitter API Documentation Browsing to the API documentation takes two clicks!

28 Twitter API Documentation Browsing to the API documentation takes two clicks!

29 Twitter API Documentation Left side List of API "calls" The possible operations we might request of Twitter Let's look at the "search" call

30 Twitter API Documentation Example of the "search/tweets" API documentation Includes things you need to know for using this API call

31 Twitter API Caveat All Resources have to be authenticated In the best case this is just annoying In the worst case this is frustrating to newcomers Open Authentication version 2 (OAuth2) Great idea in principle Different implementations (versions) of the same specification

32 Twitter API Caveat Twitter provides an "API Console Tool" for testing API calls Never seems to work (doesn't load right)

33 Demo Look through REST API Use API Console Tool to make Query to Twitter and get the data

34 Search Tweet Resource Resource access point

35 Search Tweet Resource Resource access point domain

36 Search Tweet Resource Resource access point domain version

37 Search Tweet Resource Resource access point domain version resource

38 Search Tweet Resource Resource access point domain version resource data return type

39 Search Tweet Resource Digression Return Type JSON JavaScript Object Notation (JSON) Simple text based data structure Collection of Key:Value pairs (like a python dictionary) Lists (like a python list) Keys are text (words) Values are strings, int, real, collection, list Python has a json module slow There are several other json implementations - ultrajson

40 Search Tweet Resource Resource access point Parameters of the request q query parameter, term phrase count number of tweets to return lang restrict tweets to the indicated ISO 2 character code result_type popular, recent, mixed since_id return tweets more recent than the one given max_id return tweets older than the one given

41 Intro to HCDE User Module

42 Basic Steps 1. Install distribution modules: pymysql, requests, oauthlib, request_oauthlib 2. Download the HCDE python user module 3. Unzip 4. Copy into your python directory The PYTHONPATH directory 5. Start using code Authenticate both HCDE applications with your twitter user Try some samples

43 Download the HCDE user module Visit the course web page Click on the "Resources" tab (link) About the middle of the page is a link to a zip file for the HCDE user module

44 Examples Authenticate Apps with Login.py python Login.py auth HCDE530Test01 -user <your_twitter_user> Finding Trending Topics with Trend.py python Trends.py -auth HCDE530Test01 -user <your_twitter_user> -place seattle python Trends.py -auth HCDE530Test01 -user <your_twitter_user> -woeid

45 Week 3 In-class activity #1 Meet with your team mates! python Login.py auth HCDE530Test01 -user <your_twitter_user>

46 Week 3 In-class activity #1 Installing Python packages requests oauthlib Networkx nltk

47 Week 3 In-class activity #1 Install HCDE module wi16/resources/index.html

48 Week 3 In-class activity #1 Log-in to HCDE530Test01 Use search.py, Trends.py, TwitterLimits.py

49 Week 3 In-class activity #1 Take your time to get familiarized to HCDE530Test01 module