New Technologies for Data Management

Transcription

1 New Technologies for Data Management Chaitan Baru

2 2 2 Why new technologies? Big Data Characteristics: Volume, Velocity, Variety Began as a Volume problem E.g. Web crawls 1 spb-100 spb in a single cluster Velocity became an issue E.g. Clickstreams at Facebook 100,000 concurrent clients 6 billion messages/day Variety is now important Integrate, fuse data from multiple sources Synoptic view; solve complex problems

3 3 3 Varying opinions on Big Data It s all about velocity. The others issues are old problems. Variety is the important characteristic. Big Data is nothing new.

4 4 4 Expanding Enterprise Systems: From OLTP to OLAP to Catchall OLTP: OnLine Transaction Processing E.g., point-of-sale terminals, e-commerce transactions, etc. High throughout; high rate of single shot read/write operations Large number of concurrent clients Robust, durable processing

5 5 5 Expanding Enterprise Systems: OLAP OLAP: Decision Support Systems requiring complex query processing Fast response times for complex SQL queries Queries over historical data Fewer number of concurrent clients Data aggregated from multiple systems Multiple business systems, e.g. Sales, Manufacturing, Financial Multiple locations, e.g. branches across the country, world Create Data Warehouses Leading to data marts

6 6 6 Expanding Enterprise Systems: Catchment Area Capture all data that an enterprise might care about E.g. information about customers from social networks, other contextual data Data catchment area now an element of enterprise architectures Big data is about late binding

7 7 7 Example from an industry standard TPC: Transaction Processing Performance Council TPC-Decision Support (TPC-DS)

8 8 8 Extending TPC-DS for Big Data Add semistructured and unstructured data Incorporate data mining operations in queries

9 9 9 Another Big Data Use Case: Deep Analytics Pipelines Sequence of processing steps: From data ingestion to data cleaning and transformation (ELT, sorting, SQL queries) To Machine Learning and Predictive Analytics Feed data from one step to the next Acquistion/ Recording Extraction/ Cleaning/ Annotation Integration/ Aggregation/ Representation Analysis/ Modelin g Interpretation

10 10 10 Pipeline Example: User Modeling Based on clickstream processing Data Acquisition Collect data from Web logs across all servers Sessionization Pull together all user data for a single session Feature and Target Generation Targets are, say, clicks on ads of interest Features are the prefix operations which lead to that click Model Training Offline Scoring & Evaluation Batch Scoring & Upload to serving

11 11 11 Two Approaches to Representing Big Data Data Warehousing Structured data repository With extensions for semistructured and unstructured data Pipeline / Catchment Unstructured data repository Acquistion/ Recording Extraction/ Cleaning/ Annotation Data structured according to needs of an application ( late binding ) Integration/ Aggregation/ Representation Analysis / Modelin g Interpretation

12 12 12 Transforming Data ETL vs ELT (vs NoETL) ETL Extract data from sources Transform data to fit a schema Load data into data management system ELT Extract data Load into data management system Transform data as needed by application(s)

13 13 13 ELT Example: SciSIP Project NSF Science of Science and Innovation Policy project Compare trajectory of research productivity among San Diego, Philadelphia, and St. Louis regions over a given period of time Compute research spending Measure patent production and other output quantities Perform comparative study Data acquisition Download social science data from Federal Government sources, e.g. research spending data from USASpending.gov; patent data from USPTO; Analysis Analysis to be performed according to MSA s (Metropolitan Statistical Areas), but data provided according to state/county Question: What is San Diego? What is Philadelphia? What is St. Louis?

14 14 14 San Diego County is an MSA ELT Map (State, County Name) MSA Some counties in PA Some counties in NJ Some counties in MD Some counties in DE

15 15 15 SciSIP: Map institution names Map names to Burnham Institute

16 16 16 Big Data: Agile Applications Data structuring determined by needs of agile applications Need for loose (flexible) schemas, and late binding of schemas Extract-Load-Transform (ELT) rather than Extract- Load-Transform Support for processing pipelines Data runs through multi-step processing pipelines Building and execution of machine learning models using data Used for event detection User clicks Device failures Hospital re-admissions

17 17 17 Hadoop Ecosystem Designed to deal with large amounts of semi/unstructured data Potentially a step backwards Exposes many internals of the system Can expect next generation of Hadoop technologies to bring back higher abstractions and performance optimizations

18 18 18 Hadoop Ecosystem Components HDFS Distributed File System (across a cluster) MapReduce Parallel execution environment, operating atop HDFS Pig Workflow based system; specifies data processing workflows (assembler for data) Hbase Column-based data management system Hive SQL-like (lite) interface to Hbase. Not for OLTP. Uses MR and sequential scans with HDFS. Slow.

19 19 19 Hadoop Ecosystem Components Mahout Machine learning libraries on Hadoop Recommendation mining, clustering, classification, frequent itemset mining Cassandra Distributed key-value store. With an implementation on Hadoop. YARN: MR2 Ambari: Hadoop cluster manager Avro: Data serialization system Chukwa: Monitoring system Zookeeper: Config services

20 20 20 NoSQL Databases Not Only SQL Misnomer Should have been named according what they are, rather than what they are not What are they? Data stores designed to operate at large scale By relaxing (not incorporating) a number of features of relational databases By incorporating extensible storage mechanisms Storage abstractions Key-value stores Column-oriented stores Distributed storage Simple SQL operators, e.g. Select, Project, Group By, simple Join

21 21 21 Column Stores Column store systems BigTable, Hbase, Vertica, MonetDB Store data not by rows but by column groups/families Don t store duplicate values Apply compression Utilize reduction in storage space for data replication

22 22 22 Column Store Example BigTable Komadinovic Vanja, Google

23 23 23 Hbase: Column-oriented store Storing data at scale

24 24 24 Vertica vs Postgres: From the Tropical Ecology Monitoring and Assessment Project (TEAM) Conservation International, Smithsonian Institution, World Wildlife Fund SELECT COUNT(*) FROM VIEW Vertica (sec) Postgres (sec) 1/ VIEW iucn_species_data /VIEW liana_info_ /VIEW netstats_climate_days_by_year / VIEW netstats_climate_records_by_site 1 3 5/ VIEW vegbank_tree1ha_taxonomies 1 3 6/ VIEW sampling_unit_observed_time 1 5 7/ VIEW sampling_unit_observed_time_version / VIEW sampling_unit_sampling_time / VIEW site_protocol_block_event_record_number_v / VIEW site_protocol_block_event_record_number_v9 1 57

25 / VIEW iucn_species_data CREATE VIEW iucn_species_data AS SELECT DISTINCT x.red_list_status_id, n.class, n.order_team AS "order", n.family, (((n.genus)::text ' '::text) (n.species)::text) AS species, a.year, a.unit_name AS "camera trap sampling unit id", a.latitude, a.longitude, a.country_name AS "country name", a.continent_name AS "continent name", a.short_name AS "TEAM site" FROM ( ( ( taxonomy_scientific_name n JOIN ( SELECT DISTINCT p.genus, p.species, s.short_name, r.country_name, t.continent_name, u.unit_name, u.latitude, u.longitude, date_part('year'::text, h.taken_time) AS YEAR FROM ( ( ( ( ( ( tv_photo_animal p JOIN tv_photo h ON ((p.photo_id = h.id)) ) JOIN tv_camera_trap_data c ON ((c.id = h.camera_trap_data_id)) ) JOIN sampling_units u ON ((u.id = c.camera_trap_id)) ) JOIN sites_team s ON ((u.site_id = s.site_id)) ) JOIN countries r ON ((s.country_id = r.country_id)) ) JOIN continents t ON ((r.continent_id = t.continent_id)) ) ) a ON ((((n.genus)::text = (a.genus)::text) AND ((n.species)::text = (a.species)::text))) ) JOIN taxonomy_scientific_name_with_association x ON ((x.scientific_name_id = n.id)) ) JOIN taxonomy_other_information o ON ((o.id = x.other_information_id)) ) ORDER BY a.short_name, n.class, n.order_team, n.family, (((n.genus)::text ' '::text) (n.species)::text);

26 26 26 SQL on Hadoop HIVE A data warehouse on top of Hadoop Supports basic DDL and SQL Select, Project, Join, Group By Commercial Offerings Cloudera: Supports Apache. Impala Hortonworks Data Platform: Apache. Tez Pivotal: HAWQ MapR: Modified storage layer. Drill.

27 27 27 Apache Impala Almost same as DB2 Parallel From 1995!

28 28 28 Pivotal HAWQ

29 29 29 Hortonworks Tez: Under Apache incubation vote

30 30 30 MapR Drill

31 31 31 SciDB: Array-based DBMS Unbounded non-uniform dimensions with holes Arbitrary nesting New operations such as regrid (mapping one array onto another) User-defined functions (UDF) (first-class citizens in SciDB) Sophisticated storage representations, including overlapping chunks AQL, an array query language that is similar to SQL. AFL, a functional language that provides the same capabilities as AQL but with a functional syntax

32 32 32 MongoDB Next talk