基于 Hadoop 和 RDBMS 的 Oracle 大数据分析 Corey Wei 技术顾问甲骨文公司. 版权所有 2014,Oracle 和 / 或其关联公司 保留所有权利

Transcription

1 基于 Hadoop 和 RDBMS 的 Oracle 大数据分析 Corey Wei 技术顾问甲骨文公司

2 Agenda Big Data Solution Overview Big Data Appliance Oracle NoSQL Database Big Data SQL Big Data Connectors Oracle Advanced Analytics Case Study 2

3 Big Data Solution Overview 3

4 Big Data Definition Big Data: Techniques and Technologies that Make Handling Data at Extreme Scale Economical. Brian Hopkins and Boris Evelson, Forrester Research, Expand Your Digital Horizons with Big Data (September 2011) 4

5 Applications In-Database Analytics Oracle Big Data Approach - Functional View Decide Real-Time Dashboards, Reporting & Query Information Discovery Event / Stream Data Capture Hadoop NoSQL Bridge Unstructured/ Structured Predictive Analytics Data Warehouse Log / File Data Capture Predictive Analytics ETL Data Marts / ODS Stream Acquire Organize Analyze 5

6 Applications In-Database Analytics Oracle Big Data Approach - Product View Decide Oracle Real-Time Decisions Oracle BI Enterprise Edition Endeca Information Discovery Oracle Event Processing Cloudera Hadoop Oracle NoSQL Database Oracle Big Data Connectors Oracle Advanced Analytics Oracle Industry Data Model(s) Apache Flume Oracle R Distribution Oracle Data Integrator Data Warehouse Oracle Database Stream Acquire Organize Analyze 6

7 Applications In-Database Analytics Oracle Big Data Approach Engineered Systems Decide Oracle Real-Time Decisions Oracle BI Enterprise Edition Endeca Information Discovery Oracle Event Processing Cloudera Hadoop Complete Oracle Big Data Connectors Oracle Advanced Analytics Apache Flume Oracle NoSQL Database Oracle R Distribution Integrated Oracle Data Integrator Scalable Data Warehouse Oracle Database Stream Acquire Organize Analyze 7

8 Big Data Appliance 8

9 Big Data Appliance X4-2 Sun Oracle X4-2L Servers with per server: 2 * 8 Core Intel Xeon E5 Processors 64 GB Memory 48TB Disk space Integrated Software: Oracle Linux Oracle Java VM Cloudera Distribution of Apache Hadoop (CDH) Cloudera Manager All Cloudera Options Oracle R Distribution Oracle NoSQL Database All integrated software (except NoSQL DB CE) is supported as part of Premier Support for Systems and Premier Support for Operating Systems 9

10 Big Data Appliance Product Family Starter Rack is a fully cabled and configured for growth with 6 servers In-Rack Expansion delivers 6 server modular expansion block Full Rack delivers optimal blend of capacity and expansion options Grow by adding rack up to 18 racks without additional switches 10

11 Big Data Appliance Engineered Systems Benefits Lower TCO than DIY Hadoop Clusters Faster Time to Value Higher Performance out-of-box Lower Management Overhead Integrated and Comprehensive Security Tight Integration with your Infrastructure 11

12 Cumulative Cost and Savings Engineered Systems Benefits TCO Data Points: 18 servers (DL380 vs. X4-2L) $1,400,000 $1,200,000 List Price Comparisons 864TB Raw Storage 288 Cores 1152GB Total Memory Cloudera Enterprise Subscription with all options Subscription vs. Perpetual Equivalent Installation Cost Not calculated: $1,000,000 $800,000 $600,000 $400,000 $200,000 Oracle BDA HP + Cloudera Savings Soft Cost (people and time to value) Data integration licenses $0 Year 1 Year 2 Year 3 Year 4 Year 5 12

13 Engineered Systems Benefits Management Console Single Command Patching and Upgrade Full Stack Patching and Upgrading Automatic Cluster Re- Configuration Security (AAA) out-of-box Encryption out-of-box (network and at-rest) InfiniBand + Optimizations Stack Tuning (OS, Java, Hadoop) BDA 3.0 DIY CDH

14 BDA Security Overview Authentication through Kerberos Authorization through Apache Sentry Auditing through Oracle Audit Vault Encryption for Data-at-Rest Network Encryption 14

15 Integrated Management Framework Management Infrastructure combines EM and CM Quick view of Hardware and Software status in Oracle Enterprise Manager 15

16 Oracle NoSQL Database 16

17 Oracle NoSQL Database Scalable, Highly Available, Key-Value Database Features Key-value, JSON & RDF data Large Object API BASE & ACID Transactions Data Center Support Online Rolling Upgrade Online Cluster Management Table data model Secondary Indices Secondary Zones (Data Centers) Security Application Application NoSQL DB Driver Storage Nodes Datacenter A Application Application NoSQL DB Driver Storage Nodes Datacenter B 17

18 Features - Failover Automatic Failover Automatic election of new Master Replication factor = 5 Rejoining nodes automatically synchronize with the Master Isolated nodes can still service reads All nodes are symmetric Rep Node Master Rep Node Replica Rep Node Replica Rep Node Replica Rep Node Replica New Master 18

19 Features Flexible Data Model Key-Value pairs Simple data model key-value pair (major+minor-key paradigm) Simple operations read/insert/update/delete, RMW support Major key: hashed to a Shard (partition), Minor key Btree within a Shard Raw Key/Value and JSON schema APIs supported Strings Major key: Minor key: subscriptions userid address picture Byte Array Value: expiration date phone # id.jpg Value Options: Key-Value JSON RDF Triples Tables/Rows 19

20 Features Flexible Data Model NoSQL DB Table Model Benefits Lower barrier to adoption, shorter time to market Simplified application modeling Uses familiar table concepts Features Layered on top of distributed key-value model Compatible with Release 2.0 JSON schemas Supports table evolution, retains flexible client access Sets foundation for future capabilities 20

21 Features Configurable Transactions Greater Flexibility Configurable Durability per operation Configurable Consistency per operation ACID by default Transaction scope is single API call Records share same major key Multiple operations supported 21

22 Features Elasticity On-Demand Cluster Expansion On Demand Application NoSQL DB Driver Increase Data Capacity Add more storage nodes New shards automatically created Increase Data Throughput More shards = better write throughput More replicas/shard = better read throughput Master Replica Replica Shard-1 Master Replica Replica Shard-2 StorageNode StorageNode StorageNode 22

23 Features Automatic Rebalancing Improve Performance Storage Node 1 Storage Node 2 Storage Node 3 Supports heterogeneous storage topology Replicas move from over-utilized to under-utilized storage nodes Number of shards and replication factor remain unchanged Represents a partition 23

24 Time to Upgrade (min) Features Online Rolling Upgrades Ever tried to upgrade a 200 node system while it s active? What s the Big Deal 17.5 Online Rolling Upgrade We did do it! Admin commands available to describe safe upgrade order Scripted available hands-free upgrade experience Read/Write availability throughout the upgrade process (24x3) 144 (48x3) 216 (72x3) Total Nodes (Shards x Rep. Factor) 24

25 Features Integration Oracle NoSQL Database: Integrated out of the box Query NoSQL data from Oracle Database Access NoSQL data from Hadoop for DW and analytics Share data with Coherence for extensible in-memory cache grid Persist history & event streams for processing with OEP Store & query RDF data using Oracle RDF for NoSQL 25

26 Throughput (ops/sec) Average Latency (ms) Benchmark Results - YCSB (Yahoo Cloud Scalability Benchmark) Mixed Throughput 1.25M ops/sec 2 billion records 2 TB of data 95% read, 5% update Low latency High Scalability 1,400,000 1,200,000 1,000, , , , , (2x3) 12 (4x3) 24 (8x3) 30 (10x3) Cluster Size Throughput (ops/sec) Read Latency (ms) Write Latency (ms) 26

27 Big Data SQL 27

28 Strengths of Both Systems Hadoop is good at some things Databases are good at others SQL is very important 28

29 Embrace Innovation and Integrate Big Data Appliance + Hadoop & NoSQL Unify Development languages Security Administration Support Workload management Lifecycle management Availability Exadata + Oracle Database 29

30 Publish Hadoop Metadata to Oracle Catalog Big Data Appliance + Hadoop Hive metadata HDFS Name Node HDFS Data Node HDFS Data Node create table customer_address ( ca_customer_id number(10,0), ca_street_number char(10), ca_state char(2), ca_zip char(10) ) organization external ( TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ACCESS PARAMETERS (com.oracle.bigdata.cluster hadoop_cl_1) LOCATION ('hive://customer_address') ) Oracle Catalog Hive metadata External Table External Table Exadata + Oracle Database 30

31 Executing Queries on Hadoop Hive metadata HDFS Name Node HDFS Data Node HDFS Data Node HDFS Data Node HDFS Data Node Determine: Data locations Data structure Parallelism Send to specific data nodes: Data request Context Select c_customer_id, c_customer_last_name, ca_county From customers, customer_address where c_customer_id = ca_customer_id and ca_state = CA Oracle Catalog Hive metadata External Table External Table 31

32 Executing Queries on Hadoop Hive metadata Do I/O and Smart Scan: Filter rows Project columns Select c_customer_id, c_customer_last_name, ca_county From customers, customer_address where c_customer_id = ca_customer_id and ca_state = CA Oracle Catalog HDFS Name Node HDFS Data Node HDFS Data Node HDFS Data Node HDFS Data Node Move only relevant data Relevant rows Relevant columns Hive metadata External Table External Table Apply join with database data Tables 32

33 Optimizing Scans on Hadoop Hive metadata HDFS Name Node HDFS Data Node HDFS Data Node HDFS Data Node HDFS Data Node Storage Indexes Min Max Min Max Min Max Automatically collect and store the minimum and maximum value within a storage unit Before scanning a storage unit, verify whether the data requires falls within the Min- Max If not, skip scanning the block and reduce scan time Blocks 33

34 Oracle Big Data SQL One Query Spanning Oracle Database, Hadoop & NoSQL Query Data in RDBMS, Hadoop & NoSQL Fast Oracle SQL Massive Parallelism Storage Indexes Filtered Locally Oracle NoSQL DB HDFS Data Node Oracle Database Storage Server Minimized Data Movement Oracle NoSQL DB HDFS Data Node Oracle Database Storage Server 34

35 Big Data Connectors 35

36 Oracle Loader for Hadoop Partition, sort, and convert into Oracle data types on Hadoop ORACLE LOADER FOR HADOOP MAP MAP MAP SHUFFLE /SORT Connect to the database from reducer nodes, load into database partitions in parallel (JDBC or direct path) REDUCE REDUCE Offloads data preprocessing from the database server to Hadoop Works with a range of input data formats Automatic balancing in case of skew in input data MAP REDUCE Online and offline modes MAP REDUCE MAP SHUFFLE /SORT REDUCE Kerberos authentication 36

37 Oracle SQL Connector for HDFS Use Oracle SQL to Access Data on HDFS Generate external table in database pointing to HDFS data Load into database or query data in place on HDFS Fine-grained control over data type mapping Parallel load with automatic load balancing Kerberos authentication Hadoop Access or load into the database in parallel using external table mechanism Oracle Database OSCH OSCH OSCH OSCH HDFS Client External Table SQL Query 37

38 Oracle R Advanced Analytics for Hadoop R Analytics leveraging Hadoop and HDFS Oracle R Client Linearly Scale a Robust Set of R Algorithms MAP MAP REDUCE MAP REDUCE MAP HDFS Hadoop Leverage MapReduce for R Calculations Compute Intensive Parallelism for Simulations 38

39 Oracle Data Integrator Application Adapters for Hadoop Transforms Via MapReduce(HIVE) Oracle Data Integrator Oracle Loader for Hadoop Activates Loads Benefits Consistent tooling across BI/DW, SOA, Integration and Big Data Reduce complexities of processing Hadoop through graphical tooling Improves productivity when processing Big Data (Structured + Unstructured) Oracle Database Improving Productivity and Efficiency for Big Data 39

40 Oracle XQuery for Hadoop OXH is a transformation engine for Big Data XQuery language executed on the Hadoop XQuery for $ln in t ext:collect ion() let $f := t okenize($ln) where $f[1] = 'x' return text:put($f[2]) Map/Reduce Execut ion Plan M/R M/R M/R Oracle Big Data Connectors Oracle Data Integrator M/R Map/Reduce Worker Nodes OXH Engine HDFS Oracle Loader for Hadoop Acquire Organize Analyze 40

41 Oracle Data Integrator Simplify Map Reduce Automatically generates MapReduce code High performance loads into Data Warehouse leveraging both OLH and OSCH Manages the process across platforms Oracle Data Integrator OLH & OSCH 41

42 Oracle Advanced Analytics 42

43 Oracle Advanced Analytics Classification Function Algorithms Applicability Logistic Regression (GLM) Decision Trees Naïve Bayes Support Vector Machines (SVM) In-Database Data Mining Algorithms Classical statistical technique Popular / Rules / transparency Embedded app Wide / narrow data / text Regression Linear Regression (GLM) Support Vector Machine (SVM) Classical statistical technique Wide / narrow data / text Anomaly Detection One Class SVM Unknown fraud cases or anomalies Attribute Importance Association Rules A1 A2 A3 A4 A5 A6 A7 Minimum Description Length (MDL) Principal Components Analysis (PCA) Apriori Attribute reduction, Reduce data noise Market basket analysis / Next Best Offer Clustering Hierarchical k-means Hierarchical O-Cluster Expectation-Maximization Clustering (EM) Product grouping / Text mining Gene and protein analysis Feature Extraction F1 F2 F3 F4 Nonnegative Matrix Factorization (NMF) Singular Value Decomposition (SVD) Text analysis / Feature reduction 43

44 Oracle Advanced Analytics Oracle R Enterprise Compute Engines R Engine Other R packages Oracle R Enterprise packages User R Engine on desktop SQL Results Oracle Database R Open Source User tables?x Database Compute Engine R Results R Engine Other R packages Oracle R Enterprise packages R Engine(s) spawned by Oracle DB R-SQL Transparency Framework intercepts R functions for scalable in-database execution Function intercept for data transforms, statistical functions and advanced analytics Interactive display of graphical results and flow control as in standard R Submit entire R scripts for execution by database Scale to large datasets Access tables, views, and external tables, as well as data through DB LINKS Leverage database SQL parallelism Leverage new and existing in-database statistical and data mining capabilities Database can spawn multiple R engines for database-managed parallelism Efficient data transfer to spawned R engines Emulate map-reduce style algorithms and applications Enables lights-out execution of R scripts 44

45 Oracle Enabling Technologies Unified access model supporting all analysys capabilities: SQL, R & MR Unified Analytics API SQL R MR Hadoop RDBMS IB Management Framework and Tools Unified Analytics Processing Platform 45

46 Case Study 46

47 Current ( as-is ) Architecture Siloed Operational Systems with complex, heavy and slow data transformation and flows to Data Marts Current ( as-is ) architecture is based on a years 90s design: siloed datamarts with complex and expensive provision systems, unable to respond to the new business requirements with agility 1. Provision systems are complex, expensive and inefficient 2. Lack of business agility and very long time-to-market and time-to-value (6-12 months) 3. Business users are by-passing IT corporate systems 4. Datamarts are strongly siloed with no interoperability 5. Complex Operations with very limited backup/recovery and no HA capabilities 6. Unstructured information not managed 7. Lack of Advanced Analytic capabilities 47

48 MQ Data Factory Engine ODI MQFTE Data Factory Engine Data Factory Engine ODI BI Abstraction & Query Federation BI Server ODI MQFTE Data Pool Logical Architecture Source Data Layer Transformed data Data Marts SAP RRHH Information Management Staging & Raw Data Layer Level Performance Management Mainframe Interfases Otros Social/Text GG High Density Data Quality Low Density Foundation Layer Level High Density Low Density Access & Level 4 Performance Layer Embedded Data Marts Alerts, Dashboards, Reporting Services Sensors Diario Electronico Streaming Knowledge Discovery Area Analytical Discovery Sandbox Rapid Development Sandbox Advanced Analysis & Data Science (Discovery) Security and Metadata Data Integration : Data Factory Engine & ODI + Metadata Information Access 48

49 Data Pool Hardware Architecture UAT PRD DC1 BDR DC2 PRD UAT Oracle DataGuard IB IB IB IB SAN Replication SAN IB IB Data Pool Data Pool Backup ZS-3 Backup Oracle RMAN 10GbE TSM FC VTL VTL FC TSM 10GbE ZS-3 Backup Oracle RMAN Backup Snapshot ZFS Replication Snapshot 49

50