Big Data Integration for Data Enthusiasts. Jayant Madhavan Structured Data Research Google Inc.

Transcription

1 for Data Enthusiasts Jayant Madhavan Structured Data Research Google Inc.

2 Big Data Challenge Running computations over ginormous datasets Petabytes, Exabytes, maybe more! Only one aspect of the challenge!

3 Big Data Challenge What about the data enthusiasts? Data experts, but without technical expertise Journalists, Social Scientists, NGOs, High School Teachers and Students, etc. Motivated by advocacy, i.e., doing good through data awareness Tools that help data enthusiasts work effectively Analysis: explore, clean, analyze and summarize Storytelling: visualize and publish Integration: share, find, and combine

4 Google Fusion Tables Data Management / Integration in the cloud Sharing, Collaborating, Exploring, Visualizing and Publishing Ease of use with focus on data enthusiasts Launched in Labs June Now part of Google Apps. Many million uploaded tables Embedded maps popular with journalists SQL API used widely to access reference tables, e.g., CA DMV Search to find public tables and tables extracted from the Web Bigdata Storytelling through Interactive Maps [Madhavan+, DeBull 2012] Dated system overviews [SoCC 2010, SIGMOD 2010]

5 Upload, share, explore, visualize, publish

6 Upload, share, explore, visualize, publish

7 Upload, share, explore, visualize, publish

8 Upload, share, explore, visualize, publish

9 Upload, share, explore, visualize, publish

10 Upload, share, explore, visualize, publish

11 Thousands of points -- not a simple mashup anymore

12 Table Facts: Wikileaks Iraq War Diaries: 52,000 incidents

13 Table Facts: Texas Counties 2010 Census: 254 counties with vertices Colored based on various demographics

14 Integration by Map Layers Table Facts: Earthquakes since 1973 (USGS): 174,000 incidents of magnitude 4.5+ displayed as a heatmap Nuclear Power Stations (IAEA): 248 locations with with active nuclear energy generation

15 Table Facts: English poverty rates: 32,000 wards with a total of 1.8 million vertices Colors indicate poverty levels 2011 Rioting: 2100 incidents Colors indicate addresses of Rioting and Rioters Best UK Internet Journalist Knight-Batten Award for Innovations in Journalism

16 Interactive Visualizations on Large Datasets Interactive responses necessitate Fast server-side retrieval Fast visualization rendering Low network delays Low server response time In-memory column database Specialized spatial index Details in [Lee, Sarma, Gonzalez, Lam, Madhavan, Roy, In Prep] Low rendering time à Small server response sizes Constrained sampling to limit points according to level of detail Details [Sarma, Lee, Gonzalez, Halevy, Madhavan, SIGMOD 2012]

17 The F in Fusion Tables Support for Merged tables Virtual tables that are the join of two or more underlying data tables Full first class citizens sharing, exploration, visualization and publishing Potentially critical to data enthusiasts Data from multiple sources that complete a story Requirements: Joins that just work, i.e., approx matching, entity-based matching, etc.

18 Use Case 1: Merging Complementary Datasets Firearm Deaths by Country Population by Country

19 Use Case 2: Merging with Reference Datasets Cost by Country Geometry boundaries per County

20 Use Case 2: Merging with Reference Datasets

21 Merges and their consequences: Community Benefits Fosters an eco-system for data reuse High quality reference tables used by many, but managed by a dedicated few Extent of data reuse serves as a crowdsourced quality signal Per-table permissions lead to sophisticated sharing models

22 Merges and their consequences: Performance Benefits Efficient in-memory optimizations Multiple visualizations share in-memory indices reducing their combined footprint Splitting tables into fixed and changing subtables leads to higher update rates

23 Table Facts: Terrestrial Eco Regions: regions and 4 million vertices Colored based on choice of ground water, cleared forests, human appropriations, etc.

24 The Merge Search Problem Problem: Given a table and a keyword, find table(s) that can be merged with the table Input table + looking for: population à Tables that include population for all/most of the countries in the input table

25 Detour: Table Search Problem: Find tables on the web that match keyword search queries firearm deaths by state à Table search available at: research.google.com/tables

26

27 Table Search Problem: Find tables on the web that match keyword search queries Challenges Extraction: Not everything within a <table> is a data table Identifying data tables from navigation and formatting ones Ranking: Not as simple as restricting Google.com results Content outside table might be necessary, yet misleading

28

29 Table Search Class-property queries are data-seeking and plentiful and they can be improved over web search Detecting subject columns and their corresponding semantic classes can improve search quality of results Detecting header columns and their corresponding properties can improve results [Venetis+, VLDB 2011] [Cafarella+, VLDB 2008]

30 Back to Merge Search Matching join columns Coverage: Entity overlap Matching keyword queries Use token-based matching, synonymy, etc. Subtle, yet critical, difference from web search Recall more important than precision Traditional IR optimizations can come in the way

31

32

33 Future: can we make data research easier for data enthusiasts? Can we automatically suggest Datasets that complement theirs E.g., tables with other attributes relevant to firearms

34 Future: can we make data research easier for data enthusiasts? Can we automatically suggest Datasets that complete theirs E.g., tables with the same data, but countries not in the table Datasets that contradict / support theirs E.g., tables with the same information for earlier years Visualizations that highlight trends in their data E.g., charts that demonstrate a correlation in data

35 Structured Google Sreeram Balakrishnan Johnny Chen Alon Halevy Felix Halim Boulos Harb Karen Jacqmin-Adams Hector Gonzalez Nitin Gupta Heidi Lam Anno Langen Hongrae Lee Rod McChesney Afshin Rostamizadeh Rebecca Shapley Warren Shen Steven Whang Kenneth Wilder Fei Wu Cong Yu and others

36 Structured Google Google Fusion Tables Data management for data enthusiasts Analysis, storytelling, and integration made easy Table Search Finding data tables on the Web Making data tables useful for other search activities