Collecting social media data based on open APIs

Transcription

1 Collecting social media data based on open APIs Ye Li With Qunyan Zhang, Haixin Ma, Weining Qian, and Aoying Zhou

2 Outline Social Media Data Set Data Feature Data Model Data Collecting and Limitations Motivation Methods Applications on Social Media Data Set

3 Sina Weibo: Largest microblog service in China A Twitter-like service with rapid growth of active users

4 A short review of our work on Social media data collecting and analytics : Birth of Sina Weibo (Chinese Twitter) : Research focused on Sina Weibo : Continuous data crawling with a distributed crawler : Various work on social media data analytics : Microblog Cube: An online collective behavior analytics portal : Work focused on real time data crawling and On-line analysis : RCBA: A system on real-time social collective behavior analytics

5 The features of social media data: Large-scale, Rapid growth, Real-time Event Monitoring Data Analysis Social CRM Emotion Sensing

6 Why collecting social media data? Sensing the world Lots of hot events happen every day Who are talking about? What are they talking about? Why are they talking about them?

7 Data Model: Social Stream Global Stream: 1. High density 2. Rapid growth 3. Large scale User Stream: 1. High quantity 2. Quite large scale

8 Two ways to collect Weibo data Crawl the web page (too hard) Using open APIs

9 How to collect the Weibo data? Using open APIs Weibo provides a lot of APIs to developer: Status (the content of tweets) Comments (the comments of tweets) Users (the profile of Weibo users) Friendships (the relationships between Weibo users) Links: 微博 API

10 How to collect the Weibo data? Using open APIs The process of collect data: 1. Become a developer (or a partner with Sina Weibo) 2. Create your application 3. Get users authorization 4. Collect the data Links: 新手指南

11 The limitation of open APIs Frequency limitation Request n times per hour for an application Request m times per day for an application Request x times per hour for an IP address n, m, x are determined by the quality of application Proportion limit For a specific API, we can t collect the whole data For example, we can only collect the recent 2000 retweets with the repost API

12 What we have done on data collecting? Off-line collecting and analyitcs Distributed crawler Data include: social network, tweets, user profiles Pros and Cons Entire data set Out-of-date Off-line data set

13 Application of the data set One of the largest social media corpus in universities All tweets of 2 million users (before Dec. 2013) Continuously updating Billions of tweets Their profiles and social networks 1 Billion followship relationships 10+TB raw data

14 Application of the data set An online collective behavior analytics portal (193 events from to )

15 Next step we want to: collect the data for real-time analysis We have to deal with the following problems: What type of data we should collect? How to collect the data as much as possible? (API limitations) How to detect the hot event?

16 Framework Real-time Sampler: Collect data with specific strategies Event Monitor Monitor the daily hotspot on internet Data Analyzer Analyze the updated data in real-time Real-time Sampler Event Monitor Database Data Analyzer Online System

17 Real-time data sampler Multi-threads crawler Collect threads: Global Timeline Data Opinion Leader Data Hotspot Data Manage threads: Resource Dispatcher Data Filter Data Monitor Resource Dispatcher Global Timeline Sampler Opinion Leader Sampler Hotspot Sampler Database Data Filter Data Monitor

18 Sample Strategy Adaptive dispatch Time Hotspot Repeat filter Real-time monitoring Monitor new retweets Monitor potential hotspot Resource Dispatcher Global Timeline Sampler Opinion Leader Sampler Hotspot Sampler Database Data Filter Data Monitor

19 According the data feature in real world, we can: Dispatch the resource of data sampler Compare the data feature which we sample with the real data set

20 Data scale and effectiveness Data scale(per day): Tweets (700M+) and Retweets 3+ million tweets and 10+ million retweets 1+ million users Compare with the off-line data set The tweets of the Malaysian airline event

21 Event Monitor and Data Analyzer Event Monitor: Get the hot events information through the Internet Data Analyzer: Analyze the hot events with the event related tweets Event Monitor Input: Web resource Output: Event Information (Title, Introduction, News, Images, Videos )

22 Application of real-time data set RCBA: A system on real-time social collective behavior monitor and analytics

23 Event Time Series

24 User Analytics

25 Location Discussion

26 Word cloud and popular mood

27 Data Report

28 Summary Social media data Data crawling strategies Applications on the data set Off-line analysis On-line analysis

29 Thanks!