Twitter Data Collection and Analysis

Transcription

1 Twitter Data Collection and Analysis Tutorial Session EDEE CSM Course Darshan Santani April

2 Outline Twitter API Basics Applications API (REST vs. Streaming) Descriptive Analysis Authentication Localization (Geo) Rate Limits and TOS Text Analysis (NLP) Data Access Language Topic Modeling Networks

3 Twitter API Basics

4 Twitter API API: Application Programming Interface REST vs. Streaming API REST Non-persistent Streaming Persistent Source:

5 Authentication App registration at Oauth Authentication Protocol Application level authentication Consumer Key Consumer Secret User level authentication Access Token Access Secret Source:

6 Rate Limits and Bias Imposes clear rate limits on REST interface No clear documentation on Streaming interface Can potentially vary from 1% to X% (potentially unknown) At most 1% of all the tweets produced on Twitter Representativeness Sample bias Law of large numbers Firehose 100% of all public tweets Cost and Computing resources Source:

7 Sample Bias F Morstatter,et al. Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose. International AAAI Conference on Web and Social Media, 2013.

8 Terms of Service (TOS) Not allowed to share the dataset publicly Can share derivative and aggregated analysis Student License Agreement for data sharing with CSM class students Source:

9 Data Access Data access via Streaming API Free version! Streaming endpoints Public streams: public data flowing through Twitter User streams: single-user streams Site streams: multi-user version of user streams Public Streams Establishing too many connections with the same credentials Reading data too slowly message queue overflows Error Code: 420 Source:

10 Streaming Request Parameters track comma-separated list of keywords incl. hashtags a keyword may be one or more terms keywords' order does not matter case agnostic Source:

11 Streaming Request Parameters locations comma-separated list of {longitude, latitude} pairs specifying a set of bounding boxes to filter tweets southwest corner of the bounding box comes first bounding boxes do not act as filters for other filter parameters More request parameters: Source:

12 Response Object (JSON) JSON, or JavaScript Object Notation Key-value pair Human-readable format Platform independent Twitter-specific Attributes of a JSON-encoded object are unordered Unexpected or missing fields Source:

13 Tweet Attributes Entities Media Urls User Mentions Hashtags (#) Places are named locations with geo coordinates Tweets associated with places are not necessarily issued from that location Sources:

14 User Attributes Source:

15

16 Practical Notes Decouple collection and processing of high volume streams When requesting high-volume streams, request compressed streams (GZIP) ~1/5th the size of an uncompressed stream Accept-Encoding: deflate, gzip Twitter API support in different languages: Source:

17 Hands-on Tutorial

18 Applications

19

20

21 Topic Modeling Topic 1 Transport Topic 7 Junk Food Topic 3 Italian Cuisine airport train swiss ticket hour don t zurich security class chf wi-fi area long buy shop switzerland shopping expensive open trains burger burgers de drink forget fries chips kind generous dont cheese owners backpacker hostel pub guinness romandie big live hamburgers pizza italian pizzas pasta pie tiramisu shopping pretty terrible restaurant chocolate vis al store bernerhof made special gnocchi manor luxemburgerli Darshan Santani, Daniel Gatica-Perez, Speaking Swiss: Languages and Venues in Foursquare, ACM MM 2013

22 Lausanne Gare Zurich HB

23

24 Q&A