Center for the Management of Information Technology March 9, 2012

Transcription

1 Big comscore Turning Billions of Rows into Market Insight Michael Brown March 9 th, 2012 comscore is a Global Leader in Measuring the Digital World NASDAQ Clients SCOR Employees Headquarters Global Coverage worldwide Reston, VA 170+ countries under measurement; 43 markets reported Local Presence 32 locations in 23 countries 2 V1011 University of Virginia - McIntire School of Commerce 1

2 Some of our Clients Media Agencies Telecom/Mobile Financial Retail Travel CPG Pharma Technology 3 V1011 The Trusted Source for Digital Intelligence Across Vertical Markets 9 out of the top 10 INVESTMENT BANKS 4 out of the top 4 WIRELESS CARRIERS 47 out of the top 50 ONLINE PROPERTIES 45 out of the top 50 ADVERTISING AGENCIES 9 out of the top 10 MAJOR MEDIA COMPANIES 9 out of the top 10 AUTO INSURERS 11 out of the top 12 INTERNET SERVICE PROVIDERS 14 out of the top 15 PHARMACEUTICAL COMPANIES 11 out of the top 12 CONSUMER FINANCE COMPANIES 8 out of the top 10 CPG COMPANIES 4 V1011 University of Virginia - McIntire School of Commerce 2

3 Average Records Captured per Day ( ) 1,800,000,000 1,600,000,000 1,400,000,000 1,200,000,000 1,000,000, ,000, ,000, ,000, ,000,000-5 Unified Digital Measurement (UDM) Establishes Platform For Panel + Census Data Integration Global PERSON Measurement Global DEVICE Measurement PANEL CENSUS Unified Digital Measurement (UDM) Patent-Pending Methodology Adopted by 90% of Top 100 U.S. Media Properties 6 V0411 University of Virginia - McIntire School of Commerce 3

4 Percentage of Machines Included in UDM Measurement North America Canada 93% United States 90% Latin America Argentina 94% Brazil 97% Chile 95% Colombia 96% Mexico 94% Peru 97% Puerto Rico 89% Venezuela 93% Europe Austria 81% Belgium 88% Denmark 89% Finland 83% France 92% Germany 89% Ireland 89% Italy 85% Netherlands 88% Norway 89% Poland 81% Portugal 90% Spain 92% Sweden 90% Switzerland 85% Turkey 95% United Kingdom 91% Middle East & Africa Israel 92% South Africa 74% Asia Pacific Australia 89% China 69% Hong Kong 91% India 90% Indonesia 90% Japan 73% Malaysia 91% New Zealand 91% Philippines 92% Singapore 90% South Korea 59% Russia 63% Taiwan 80% Vietnam 93% 7 August 2011 Penetration Data V1011 Worldwide Traffic per Day 35,000,000,000 8,000,000,000 30,000,000,000 7,000,000,000 25,000,000,000 6,000,000,000 # of census records 20,000,000,000 15,000,000,000 5,000,000,000 4,000,000,000 3,000,000,000 # of panel records 10,000,000,000 2,000,000,000 5,000,000,000 1,000,000,000 0 Jul 2009 Aug 2009 Sep 2009 Oct 2009 Nov 2009 Dec 2009 Jan 2010 Feb 2010 Mar 2010 Apr 2010 May 2010 Jun 2010 Jul 2010 Aug 2010 Sep 2010 Oct 2010 Nov 2010 Dec 2010 Jan 2011 Feb 2011 Mar 2011 Apr 2011 May 2011 Jun 2011 Jul 2011 Aug 2011 Sep 2011 Oct 2011 Nov 2011 Dec 2011 Jan 2012 Feb Beacon Records Panel Records 8 University of Virginia - McIntire School of Commerce 4

5 Worldwide Tags per Month 1,000,000,000, ,000,000, ,000,000, ,000,000,000 # of records 600,000,000, ,000,000, ,000,000, ,000,000, ,000,000, ,000,000,000 0 Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Panel Records Beacon Records 9 High Level Data Flow Panel ETL Tagging Delivery 10 University of Virginia - McIntire School of Commerce 5

6 Product: Device Essentials Smartphone and ISP market share from R&D with distributed SQL on structured data 11 Device Essentials helps you answer key questions, such as: 1. What types of devices are used to access the web across countries? 2. What is the relationship between content, device and country? 3. How much smartphones traffic is on WiFi vs. wireless carrier by OS? 4. Which wireless networks carry the largest share of traffic by device? 12 University of Virginia - McIntire School of Commerce 6

7 What types of devices are used to access the web across countries? Connected Non-PC Devices Devices Share Share of Total of Total Traffic Traffic US 6.2% Singapore 1.6% 5.9% UK 5.3% Japan 4.6% Australia 4.3% Canada 3.5% India 3.4% Spain 2.3% Chile 2.0% Germany 1.6% France 1.5% Argentina 1.1% Brazil 0.6% Mobile Tablet Other 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% % of country total page requests % of country total page requests 13 Source: comscore Device Essentials TM International Data May 2011 What is the relationship between content, device and country? 10% 9% Share of Total Traffic Overall and Newspaper Category 8% 7% 2.8% 6% 5% 4% 1.4% 1.8% 1.6% Other 3% 2% 1.2% 1.9% Tablet 1% Mobile 0% Total Newspapers Total Newspapers Total Newspapers Canada US Singapore 14 Source: comscore Device Essentials, May 2011 University of Virginia - McIntire School of Commerce 7

8 How much smartphones traffic is on WiFi vs. wireless carrier by OS? Smartphone Connection Type by OS - US iphone 47% Mobile WiFi/LAN Android 22% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % of device total page requests 15 Source: comscore Device Essentials TM International Data May 2011 Which wireless networks carry the largest share of traffic by device? 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Provider Share of Traffic 9% 15% 13% 10% 21% 20% 27% Mobile Total iphone Android Other (Wireline) Other (Wireless) T-Mobile Sprint PCS Verizon Wireless AT&T Wireless 16 Source: comscore Device Essentials TM US Data June 2011 University of Virginia - McIntire School of Commerce 8

9 Device Essentials What do we measure from the tag? From each event on our network comscore receives Event IP Address User Agent Cookie Timestamp URL Event Type Page view Ad impression Video play Survey Campaign ISP Geography Country City Local Market Wi-Fi vs. Non-WiFi OS Device Browser Type 3 rd party cookie Persistent over time and sites 17 Device Essentials Data Review 12 TBs per day of new compressed data (372TB per month) 30 billion events per day 1 million domains 150 million user agents per month 15 billion cookies per month Full page and referring URL as longest strings Distributed across cluster by cookie value to limit skew URL classification by company and category 18 University of Virginia - McIntire School of Commerce 9

10 Device Essentials Methodology Design in Greenplum SELECT 1 week of data to work on a complete subset 1 hour GROUP BY country, ISP, region, zip, device Coding 2 hours 30 min Processing 2 hours AGGREGATE traffic across groups Total 4 hours 10 min JOIN on ISP and device lookups 20 min STORE back to Greenplum cluster (distribute randomly give data skew) 10 min EXPORT directly to corporate file systems and other systems 10 min Greenplum SQL access raw column data. Rapid iteration! 19 Device Essentials - Methodology Design in Greenplum Initial R&D for proof of concept ~100 lines of code written in PSQL Generated 1 week WW and 1 month by country examples Available on demand Dynamic iteration for new features, QA and client requests Identify mobile carrier ISPs and IP address ranges by mobile device usage mobile optimized site consumption cookie behavior User agent classification Distributed MaxMind apply Traffic Aggregations 20 No Engineering! University of Virginia - McIntire School of Commerce 10

11 Greenplum cluster hardware 100 Node Cluster: 1 Master; 6 ETL; 108 Workers Using Dell R510 with GB 15K RAID, 64GB RAM, 24 cores (HT) 2,400 total CPUs; 6.4TB total memory; 540TB total disk space 10GB Ethernet Loads 400 billion rows in 8 hours PSQL, Perl, Python Compiled C libraries Dozens of users 21 Source: Greenplum Whitepager Device Essentials Production Solution Tags User agent Dictionary URL Dictionary IP Dictionary Production Processing Distribute Tag Processing Storage Aggregation Sybase Data Warehouse 22 University of Virginia - McIntire School of Commerce 11

12 SOCIAL ESSENTIALS Full insight into the composition, behavior and value of consumers reached by brands on Facebook Introducing Social Essentials Strategic challenges addressed by Social Essentials insights: Compare the reach and frequency of your brand s Facebook presence to other media channels. Determine whether you are reaching your target consumer. Tailor marketing campaigns and partnerships to your Facebook audience s interests and passions. Benchmark your brand s presence on Facebook to that of your competitors. 24 University of Virginia - McIntire School of Commerce 12

13 Segments are created by classifying the: Message & Location News Feed Users are classified as exposed when they see a message in the news feed or wall Profile Pages If the page belongs to the current user, they are a FAN. When fan information appears less prominently in areas the user may not typically see, we classify as identified If the page does not belong to the current user, they are a FRIEND. 25 Social Essentials uses comscore s PC based panel. Messages seen on a mobile phone are not included in this analysis. comscore Social Network Segments Brand s Total Facebook Network All Fans All Friends Exposed Fans Identified Fans Exposed Friends Identified Friends Newsfeed exposed Profile (wall) exposed Profile (wall) identified Newsfeed exposed Profile (wall) exposed Profile (wall) identified 26 University of Virginia - McIntire School of Commerce 13

14 The vast majority of brand exposure happens in the Newsfeed and on Profile pages In May, Starbucks received 156 impressions for every 1 page view on their brand page Starbucks, Southwest & Bing: Ra>o of Facebook Impressions for Fans & Friends of Fans to Page Views on Brand Fan Pages Source: comscore Social Essen>als, U.S., May Starbucks Southwest Bing 27 Social Essentials provides full insight into a brand s presence on Facebook Key Measures ü Topline data providing insight into the reach & frequency of a brand on Facebook ü Can include both unpaid and paid brand exposure ü Segments: fans, friends of fans, sponsored stories and social ads Profiling ü Insight into each segments demographics, interests, search and internet usage intensity Fan Value ü Brand engagement: brand search and site visitation indices ü Target alignment: primary & secondary ü Spending propensity: total and category level Competitive Insight ü 1 competitor (more can be added at additional cost) ü Key Measures comparison ü Fan base overlap ü Advertiser category engagement 28 University of Virginia - McIntire School of Commerce 14

15 Brands should focus on news feed and less on fan page Ratio of News Feed Exposures to Fan Page Views 42 to 1 45 to to 1 29 Fans and Friends visit brand s site more frequently than average Percent of segment visiting the brand s site Index in () Starbucks (331) (522) Bing (133) (155) Friends Fans Avg Internet User Southwest (266) (463) 0% 10% 20% 30% 40% 50% 60% 70% 30 Source: comscore Social Essen0als, US, May 2011 University of Virginia - McIntire School of Commerce 15

16 Bing Fans outperform the average Bing visitor in Search Bing : Fans and Friends Search Ac>vity vs. Average Bing Searcher Searches Per Searcher Searches Per Session Searches Per Usage Day Total Internet Fan Friend of Fan Search Usage Days 31 Source: comscore Social Essen0als, US, May 2011 Starbucks Fans & Friends spend more money and spend more often at Starbucks than the typical Starbucks consumer Starbucks: Fans & Friends Purchase Ac>vity vs. Average Internet User 12% 11% 10% 8% 8% 6% 4% 2% 0% Spending per Buyer Transactions per Buyer 32 Source: comscore Social, Banking and Credit Card data, U.S., May 2011 University of Virginia - McIntire School of Commerce 16

17 Social Essentials Fans are heavy Facebookers Avg sessions per person Avg Facebook user Southwest Bing Starbucks 33 Source: comscore Social Essen0als, US, May 2011 Daily Volume of Extracted Facebook Items Daily Volume of Extracted Entities (MM) Ads Brand-News Brand-Profile Ads Brand-News Brand-Profile 34 University of Virginia - McIntire School of Commerce 17

18 Social Essentials Data Review Social Network page content collected as panelists browse Fewer events than tagging (~2M panelists, ~160 raw FB pages/panelist/day) Average page-size: ~15 KB Hadoop (with MapR DFS) Data is too large (~5 TB / day) to analyze without massively parallel processing Processing day of data sequentially on a single machine could take ~19 days! Engineers analyze text in Pig, Perl, Java, Python Panelist + cproxy Panelist + cproxy Facebook Page Content Collection Servers Hadoop DFS Panelist + cproxy 35 Social Essentials Methodology Design Hadoop streaming + Perl s regex = Fast text processing Page content data has the raw, un-rendered HTML Use regular expressions to identify: News feed, ads, fan pages, and other pages Brand exposure Method of exposure Building more extensible parser Rapid testing in Hadoop of new processing code Social Dictionary maps page content to brands 36 University of Virginia - McIntire School of Commerce 18

19 Challenges in Extracting Social Networking Entities ~320 million social pages per day 5 TB per day Need to process ~3,700 pages / sec just to keep up Client NameNode Avg page-size ~15KB + time to load to HDFS, buffer time for errors, etc Hadoop cluster used to extract entities from streaming data 300 machines together process ~44K pages per second Each machine processes ~145 pages / sec Daily Facebook entity extraction completed in ~3 hours 37 Hadoop-1 Hadoop-2 Hadoop-3 Hadoop-N HDFS Load FB Pages NTFS Hadoop cluster hardware and processing overview Production Hadoop Cluster 75 nodes: Mix of Dell R710 and R total CPUs; 4.7 TB total memory; 1.1PB total disk space, 1GB Ethernet Hadoop DFS Social Entity- Stream Extraction Social Entity- Partitions NTFS Filter News & Profile Social Entities Daily: 2 Hr / 70G 1 Hr / 15Gx 0.5 Hr / 15Gx 0.5 Hr / 15Gx 38 University of Virginia - McIntire School of Commerce 19

20 Some Tips & Tricks 39 Compression w/sorting Compress Log Files when processing large volumes of log data Several advantages to Sorting Data First: Reduces the size of the data Improves application performance Examples: 1 Hour of our data (313 GB raw, 815 million rows) Standard compression of time ordered data is 93GB (30% of original) Standard compression on a 2 key sorted set is 56GB (18% of original) For one day it saves 800GB For one month it saves 25 TB For 90 days it saves 75TB 40 University of Virginia - McIntire School of Commerce 20

21 Splitting of Data We took a sample of our census log data for one hour Row count was 49,404,267 which consumed 28,451,294,113 bytes raw. When GZIPed the size was 3,446,775,819; 12.11% of original We broke the file into smaller files to test parallism Test results of a simple line count program written in Pig Time (seconds) Rows/Second One raw file 49 1,008,250 One compressed file ,453 Split Compressed (4MM) x ,906 Split Compressed (2MM) x ,372, Useful Factoids Colorful, bite-sized graphical representations of the best discoveries we unearth. Visit or for the latest gems. 42 University of Virginia - McIntire School of Commerce 21

22 Thank You! Michael Brown CTO comscore, Inc. 43 University of Virginia - McIntire School of Commerce 22