Building an Integrated Big Data & Analytics Infrastructure September 25, 2012 Robert Stackowiak, Vice President Data Systems Architecture Oracle

Transcription

1 Building an Integrated Big Data & Analytics Infrastructure September 25, 2012 Robert Stackowiak, Vice President Data Systems Architecture Oracle Enterprise Solutions Group

2 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

3 Sources for the Data are Growing 383+ Million Twitter accounts (100m+ tweeting) 835+ Million Facebook subscribers 1.2+ Billion Mobile Web users Sensors everywhere

4 Structured Data & Big Data Structured data from applications. Semi-structured Big Data from social media and logs, sensors, feeds, etc.

5 How Big Data Fills Out the Complete Picture AUTOMOTIVE Auto sensors reporting location, problems COMMUNICATIONS Location-based advertising CONSUMER PACKAGED GOODS Sentiment analysis of what s hot, problems FINANCIAL SERVICES Risk & portfolio analysis EDUCATION & RESEARCH Experiment sensor analysis HIGH TECHNOLOGY / INDUSTRIAL MFG. Mfg quality Warranty analysis LIFE SCIENCES Clinical trials Genomics MEDIA/ ENTERTAINMENT Viewers / advertising effectiveness ON-LINE SERVICES / SOCIAL MEDIA People & career matching Web-site optimization HEALTH CARE Patient sensors, monitoring, EHRs Quality of care OIL & GAS Drilling exploration sensor analysis RETAIL Consumer sentiment Optimized sales & marketing TRAVEL & TRANSPORTATION Sensor analysis for optimal traffic flows Customer sentiment UTILITIES Smart Meter analysis LAW ENFORCEMENT & DEFENSE Threat analysis - social media monitoring, photo analysis Challenged by: Data Volume, Velocity, Variety in finding Value

6 Typical Stages in Analytics Choosing the Right Solutions for Right Data Needs Discover and Explore Growing investment here Query and Analyze Dashboard and Report Model and Plan Predict Growing investment here

7 Challenges & Strategies CHALLENGES Fragmented Solutions Difficulty of Self-Service BI Data Not Current Time to ROI / Development Time Rapidly Growing Diverse Data & User Communities Deployment Manageability, Security & Expense STRATEGIES Specialized but integrated data stores and tools Flexible, guided, automated, easy-to-use tools, data discovery Solutions for Just-in-Time well-understood data Horizontal and industry pre-built solutions, appliance-like solutions & Cloud solutions Enterprise class solutions serving 1000s of users optimized for diverse workloads and providing petabytes of data Pre-integrated solutions that are centrally managed with advanced security / governance; Consolidation where possible to reduce platform footprint space & power

8 BI Abstraction & Query Federation An Information Architecture that includes Big Data Source Data Layer Information Access Processes Enterprise Data Warehouse Staging Data Layer Performance Management COTS/ERP External Social/Text Sensors Strongly Typed Data Data Quality Weakly Typed Data Foundation Layer Enterprise Data with full history Performance Layer Embedded Data Marts Alerts, Dashboards, Reporting Services Information Discovery Streaming Knowledge Discovery Layer Data Mining Sandbox Rapid Dev Sandbox Advanced Analysis & Data Science Security and Metadata Data Integration

9 Oracle Data Integrator / Connectors Oracle Analytics Software Components Unstructured Data / Sparse Data of Value Cloudera Hadoop Oracle NoSQL DB Endeca Information Discovery Structured Data / Highly dense data Oracle Transactional Database & Applications Acquire Organize Oracle Data Warehouse & Embedded Analytics Oracle BI Foundation Suite Analyze & Decide

10 Oracle Data Integrator / Connectors Exalytics In-Memory Machine & Engineered Systems Unstructured Data / Sparse Data of Value Cloudera Hadoop Big Data Appliance Oracle NoSQL DB Endeca Information Discovery Structured Data / Highly dense data Oracle Transactional Database & Applications Exadata Platforms Acquire Organize Oracle Data Warehouse & Embedded Analytics Oracle BI Foundation Suite Analyze & Decide

11 Oracle s Analytics Platforms Oracle Big Data Appliance Oracle Exadata Oracle Exalytics InfiniBand InfiniBand Stream Acquire Organize Analyze & Visualize Expedited time to value Easier to manage and upgrade Lower cost of ownership Reduced change management risk One-stop support Extreme performance

12 Big Data Appliance Big Data for the Enterprise Foundation Software: Oracle Linux Oracle Java VM Cloudera Apache Hadoop Distribution Cloudera Manager Oracle NoSQL Database Community Edition Application Software: Oracle NoSQL Database Enterprise Edition New Oracle Big Data Connectors - New Oracle Loader for Hadoop Oracle Direct Connector for HDFS Oracle Data Integrator Application Adapter for Hadoop Oracle R Connector for Hadoop 18 Sun X4270 M2 Servers 48 GB memory per node = 864 GB memory 12 Intel cores per node = 216 cores 36 TB storage per node = 648 TB storage 40 Gb/sec InfiniBand 10 Gb/sec Ethernet

13 Cloudera Distribution Including Apache Hadoop Distribution Details Apache Hadoop Apache Sqoop Apache Hive Apache Mahout Apache Pig Apache Whirr Apache HBase Apache Oozie Apache Zookeeper Fuse-DFS Apache Flume Hue Plus Cloudera Manager

14 Hadoop Software Layout Node 1: M: Name Node, Balancer & HBase Master S: HDFS Data Node, NoSQL DB Storage Node Node 2: M: Secondary Name Node, Management, Zookeeper, MySQL Slave S: HDFS Data Node, NoSQL DB Storage Node Node 3: M: JobTracker, MySQL Master, ODI Agent, Hive Server S: HDFS Data Node, NoSQL DB Storage Node Node 4 18: S: HDFS Data Nodes, Task Tracker, HBase Region Server, NoSQL DB Storage Nodes Your MapReduce runs here!

15 Time (hours) Time (hours) Big Data Appliance Performance Comparisons 7x faster than custom 20-node Hadoop cluster for large batch transformation jobs 2.5x faster than 30-node Hadoop cluster for tagging and parsing text documents Big Data Appliance DIY Hadoop Cluster 0 Big Data Appliance Cloud-based Hadoop

16 Oracle NoSQL DB A distributed, scalable key-value database Simple Data Model Key-value pair with major+sub-key paradigm Read/insert/update/delete operations Scalability Dynamic data partitioning and distribution Optimized data access via intelligent driver High availability One or more replicas Disaster recovery through location of replicas Resilient to partition master failures No single point of failure Transparent load balancing Reads from master or replicas Driver is network topology & latency aware Application NoSQLDB Driver Storage Nodes Data Center A Application NoSQLDB Driver Storage Nodes Data Center B

17 Note: For illustration purposes only! Big Data Appliance System Layout NoSQL DB Master Node Replicas

18 Price Comparison Oracle Big Data Appliance Year 1 Year 2 Year 3 Total Build-Your-Own HP hardware and Cloudera Year 1 Year 2 Year 3 Total BDA Cost $450,000 Servers and switches $428,220 Support Cost $54,000 $54,000 $54,000 Support Cost $136,233 $72,000 $72,000 On-site Installation $14,150 Total $518,150 $54,000 $54,000 $626,150 Installation & configuration not included Total $564,453 $72,000 $72,000 $708,453 Full details at

19 Justifying Big Data Projects & the BDA Incremental Business Value? Customer sentiment Web-site / promotion optimization Location based services & advertising Fraud detection Other

20 The BDA vs. Build Your Own Big Data Hardware Physical Installation (10 racks) Electricians Network Engineers Storage Engineers System Administrators 286 hours 236 hours, 616 cables 264 hours, 864 cables 320 hours, 576 cables 232 hours Totals: 1338 people hours, 677 elapsed hours, 2344 cables Note: For illustration purposes only!

21 Input Oracle Loader for Hadoop Partition and transform into Oracle ready format Load Query.. Table Input Oracle Loader for Hadoop

22 Oracle s Big Data Platform Big Data Connectors Oracle Loader for Hadoop (OLH) A MapReduce utility to optimize data loading from HDFS into Oracle Database Oracle Direct Connector for HDFS Access data directly in HDFS using external tables ODI Application Adapter for Hadoop ODI Knowledge Modules optimized for Hive and OLH

23 Oracle Loader for Hadoop Use The Cluster ORACLE LOADER FOR HADOOP MAP MAP MAP SHUFFLE /SORT REDUCE REDUCE Last stage in MapReduce workflow MAP REDUCE Partitioned and nonpartitioned tables MAP MAP SHUFFLE /SORT REDUCE REDUCE Online and offline loads

24 Oracle Data Integrator & OLH

25 Oracle Direct Connector for HDFS Direct Access from Oracle Database HDFS Oracle Database SQL Query SQL access to HDFS External Table External table view Data query or import Infini Band DCH DCH DCH HDFS Client

26 Performance Oracle Big Data Appliance to Oracle Exadata (two database instances) with InfiniBand Oracle Direct Connector for HDFS Load text files from HDFS: 12 TB/hour Oracle Loader for Hadoop and Oracle Direct Connector for HDFS Load data pump files from HDFS (not including Hadoop time): 12 TB/hour 50% less database CPU time than loading text files Oracle Loader for Hadoop (tested with one database instance) Create Oracle-ready format and load (including time on Hadoop): 2 TB/hour 75% less database CPU time than loading text files Comparison with third party products Apples-to-apples comparison: Oracle Direct Connector for HDFS is 5x faster than Fuse-DFS Oracle Loader for Hadoop online option is 12x faster than SQOOP Oracle Loader for Hadoop online JDBC (our slowest) is 2x faster than SQOOP Our fastest option is 20x faster than SQOOP, 5x faster than Fuse-DFS

27 Oracle R Connector for Hadoop R package that provides an interface between the local R environment, Oracle Database, and Hadoop Using simple R functions, copy data between R memory, the local file system, Oracle Database, HDFS Schedule R programs to execute as Hadoop MapReduce jobs - return the results to any of the locations

28 Oracle Database 11g Data Warehousing The Leading Database for Data Warehousing Key Data Warehousing Capabilities Flexible Model Deployment Embedded Analytics Advanced Analytics (R & Data Mining) OLAP Single Point of Management Secure 24X7 Availability Optimal Storage Management Scaled to petabytes & large business analyst communities

29 Exadata Hardware Architecture Database Grid 8 x Dual-processor x64 database servers OR 2 x Eight-processor x64 database servers InfiniBand Network 3 x 36-port 40Gb/s switches Unified server & storage network Intelligent Storage Grid 14 High-performance low-cost storage servers 100 TB High Performance disks, or 504 TB High Capacity disks 22.4 TB PCI Flash Intelligent Storage Server Software

30 Exadata Storage Server Software Innovations Intelligent storage Scale-out InfiniBand storage Smart Scan query offload Hybrid Columnar Compression 6-10x compression for warehouses 10-15x compression for archives Smart PCI Flash Cache Accelerates random I/O up to 30x Triples data scan rate Data remains compressed for scans and in Flash uncompressed compress primary DB Benefits Cascade to Copies standby test dev backup

31 Roles for Data Warehouse & Middle Tier BI Data Warehouse Optimized storage for enterprise data volumes Exceeds performance & availability SLAs Persistent & secure version of the truth Flexible schema IT timeframes for solutions Middle-Tier BI Optimized for information delivery Quality of data visualization is key Discovery, scenario modeling, scorecards Dimensional-style self-guided analysis Easy to add new sources of data

32 Oracle Exalytics & BI Foundation Suite Platform TimesTen for Exalytics Adaptive In-Memory Tools 1 TB RAM 40 Processing Cores High Speed Networking Essbase In-Memory Oracle Business Intelligence Foundation Suite In-Memory Analytics Exalytics Hardware

33 TimesTen In-Memory Database for Exalytics Better Analytics Support OLAP Grouping Operators: CUBE, ROLLUP, GROUPING SETS WITH Clause Analytic Functions: RANK, DENSE_RANK, SUM, AVG, ORDER BY NULLS FIRST LAST Time functions: TIMESTAMPADD, TIMESTAMPDIFF Columnar Compression

34 Matured vs. New Data Analysis Processes DECIDE ACQUIRE DECIDE ACQUIRE Matured New ANALYZE ORGANIZE ORGANIZE ANALYZE

35 Oracle Endeca Information Discovery Helps organizations quickly explore ALL relevant data Unified Querying Endeca Information Discovery Interactive Exploration Endeca Server App Composition Faceted Data Model Integration Enrichment Combines structured & unstructured data from disparate systems Automatically organizes information for search, discovery & analysis

36 Oracle Exalytics & Endeca Platform Deep Search Contextual Navigation Endeca Server In-Memory 1 TB RAM 40 Processing Cores High Speed Networking Visual Analysis Oracle Endeca Information Discovery In-Memory Data Discovery Exalytics Hardware

37 In-Memory Data Discovery: Endeca Server Unstructured & Structured Data in Memory Specify data sources for loading into Endeca Server High performance data discovery for all types of data in the Endeca Server Manual configuration 1TB RAM In-memory Unstructured & Structured data Data sources

38 Making Sense of Diverse Data Sources Data Warehouse Shopping Cart Site Sensors Website Logs & Data NoSQL DB

39 Determine Value of Data of All Types Knowledge Discovery Engine Structured Data Warehouse Unstructured Semi-structured High Volume Distributed File System Sensors Website Logs & Data NoSQL DB

40 Valuable Data Found Now Store it Securely Persistent Data Store for All Data of Value Knowledge Discovery Engine Data Warehouse Discoveries High Volume Distributed File System MapReduce code separates valued data, then sent to via specialized adapters to Data Warehouse Sensors Website Logs & Data NoSQL DB

41 Deploy Widely Available Reports & Analytics Persistent Data Store for All Data of Value + In-DB Analytics Knowledge Discovery Engine Data Warehouse BI Tools and Dashboards Enterprise-class for reporting & analysis High Volume Distributed File System MapReduce code separates valued data, then sent to via specialized adapters to Data Warehouse Sensors Website Logs & Data NoSQL DB

42 Feed the Recommendation Engine Persistent Data Store for All Data of Value + In-DB Analytics Knowledge Discovery Engine Data Warehouse BI Tools and Dashboards Update Website Recommendations High Volume Distributed File System MapReduce code separates valued data, then sent to via specialized adapters to Data Warehouse Real-Time Analytics and Recommendations Sensors Website Logs & Data NoSQL DB

43 Make Well-Tuned Real-Time Recommendations Persistent Data Store for All Data of Value + In-DB Analytics Knowledge Discovery Engine Data Warehouse BI Tools and Dashboards High Volume Distributed File System MapReduce code separates valued data, then sent to via specialized adapters to Data Warehouse Location & User Profile Real-Time Analytics and Recommendations Recommend Sensors Website Logs & Data NoSQL DB

44 Oracle Footprint for Entire Solution Fast, Intuitive Data Discovery Endeca Information Discovery on Exalytics Unstructured Data Analysis Structured Data Analysis Reliable, Available, Secure Source of Truth Oracle Database DW on Exadata Advanced Analytics Analyst Friendly Reporting Query and Analysis Tools Oracle BI Foundation Suite on Exalytics Real-time Recommendations Cloudera HDFS on Big Data Appliance Oracle ERP & CRM Solutions on Exadata Oracle Real-Time Decisions Unstructured Data Analysis Sensors Website Logs & Data Oracle NoSQL DB

45 Unstructured Semistructured Structured Oracle Big Data Architecture Capabilities Management Security, Governance Data Acquire Organize Analyze Decide Master & Reference Transactions Machine Generated Social Media Text, Image Video, Audio DBMS (OLTP) Files NoSQL HDFS ETL/ELT ChangeDC Real-Time Message- Based Hadoop (MapReduce) ODS Warehouse Streaming (CEP Engine) In-Database Analytics Reporting & Dashboards Alerting & Recommendations EPM, BI, Social Applications Text Analytics and Search Advanced Analytics Interactive Discovery Specialized Hardware Big Data Cluster High Speed Network RDBMS Cluster In Memory Analytics

46 Oracle Delivers Value from ALL Data Best for Business, Best for IT Better Insights, Decisions, Actions From measurement to analysis, forecasting & optimization Insights across time, functions and roles Persistent version of the truth for ALL DATA Most Complete, Open, Integrated From Discovery to Dashboards to Analytics to Data Management Standards based & blending of Open Source components Optimized integrated Engineered Systems & software World Class Analytics Infrastructure Best of Breed capabilities at each layer of the stack Uniquely enables complete analysis of ALL DATA Enterprise Architecture: scalable, reliable, manageable & secure

47 Big Data Contacts at Oracle Robert Stackowiak, Oracle ESG Phone: Gokula Mishra, Oracle BI & Performance Management Phone:

48