<Insert Picture Here> Introduction to Big Data Technology

Transcription

1 <Insert Picture Here> Introduction to Big Data Technology

2 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remain at the sole discretion of Oracle.

3 Agenda What is Big Data Real World Use of Big Data Architecture Overview Oracle Big Data Solution Summary

4 WHAT IS BIG DATA?

5 Why Is Big Data Important? US HEALTH CARE MANUFACTURING GLOBAL PERSONAL LOCATION DATA EUROPE PUBLIC SECTOR ADMIN US RETAIL Increase industry value per year by Decrease dev., assembly costs by Increase service provider revenue by Increase industry value per year by Increase net margin by $300 B 50% $100 B 250 B 60+% McKinsey Global Institute Source: * McKinsey Global Institute: Big Data The next frontier for innovation, competition and productivity (May 2011)

6 Data are Growing 400+ Million Twitter accounts (100m+ tweeting) 900+ Million Facebook subscribers 1.2+ Billion Mobile Web users Sensors everywhere

7 What is Big data Big data analytics is concerned with the analysis of large volumes of transaction/event data and behavioral analysis of human/human a human/system interactions. (Gartner) Big data represents the collection of technologies that handle large data volumes well beyond that inflection point and for which, at least in theory, hardware resources required to manage data by volume track to an almost straight line rather than a curve. (IDC)

8 What is Big data SOCIAL BLOG SMART METER VOLUME VELOCITY VARIETY VALUE

9 Structured.. Non Structured Non Structured Level Structured Semi-structured Quasi-structured Unstructured Example Relational database XML data files Text documents Images and video A new class of problems has emerged which demands an ability to accept and manage data without advanced knowledge of its structure or format.

10 Real world use of big data Today s Challenge Healthcare Expensive office visits Manufacturing In-person support Location-Based Services Based on home zip code Public Sector Standardized services Retail One size fits all marketing New Data Remote patient monitoring Product sensors Real time location data Citizen surveys Social media What s Possible Preventive care, reduced hospitalization Automated diagnosis, support Geo-advertising, traffic, local search Tailored services, cost reductions Sentiment analysis segmentation

11 Public Healthcare Management of Outbreak Through Early Detection of Clusters

12 3. Public/Private Hospital executes Health Program integrated EHR/EMR Systems 4. Health Tracking Health check up records 1. Blood Pressure Sleep Tracking 4. Health Tracking Health check up records 1. Blood Pressure Heart rate Tracking 4. Health Tracking Health check up records Government Officer 2. Government creates/revises National Health Program Personal Health Improvement Cloud Suggested Health Improvement (Secured Personal Access) Big Data Doctor Nurse Officer 1. Blood Pressure Coach Tracking Integrated Medical Device Integrated Medical Device Integrated Medical Device Integrated Medical Device Integrated Medical Device

13 What does a Big Data World look like? Utilities What they collect Smart Metering -Monitors power usage How they use it Better demand planning Better targeted marketing Better targeted products based on individuals power needs Big Data means The ability to predict demand at household level Reduce exposure to spot market

14 Real world use of big data AUTOMOTIVE Auto sensors reporting location, problems COMMUNICATIONS Location-based advertising CONSUMER PACKAGED GOODS Sentiment analysis of what s hot, problems FINANCIAL SERVICES Risk & portfolio analysis EDUCATION & RESEARCH Experiment sensor analysis HIGH TECHNOLOGY / INDUSTRIAL MFG. Mfg quality Warranty analysis LIFE SCIENCES Clinical trials Genomics MEDIA/ ENTERTAINMENT Viewers / advertising effectiveness ON-LINE SERVICES / SOCIAL MEDIA People & career matching Web-site optimization HEALTH CARE Patient sensors, monitoring, EHRs Quality of care OIL & GAS Drilling exploration sensor analysis RETAIL Consumer sentiment Optimized sales & marketing TRAVEL & TRANSPORTATION Sensor analysis for optimal traffic flows Customer sentiment UTILITIES Smart Meter analysis LAW ENFORCEMENT & DEFENSE Threat analysis - social media monitoring, photo analysis Challenged by: Data Volume, Velocity, Variety in finding Value

15 ARCHITECTURE OVERVIEW

16 Basic Architecture Client

17 Basic Architecture HDFS Client Name Node

18 Basic Architecture HDFS Client Name Node Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker

19 Basic Architecture Client HDFS Name Node Map Reduce Job Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker

20 Basic Architecture Client HDFS Map Reduce Name Node Job Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker

21 The concept in not new

22 The concept in not new

23 The concept in not new

24 The concept in not new

25 The concept in not new

26 The concept in not new

27 The concept in not new

28 The concept in not new

29 The concept in not new

30 Mapping to world of Big data Data

31 Mapping to world of Big data Data Non Structured Data (Input)

32 Mapping to world of Big data Data Integration

33 Mapping to world of Big data Data Integration Integration (Data Import, Log Collection)

34 Mapping to world of Big data Data Management

35 Mapping to world of Big data Data Management Data Management File System Data Management DBMS

36 Mapping to world of Big data Data Processing

37 Mapping to world of Big data Data Processing Processing Map () Processing Reduce () Processing Serialization Processing Output

38 Mapping to world of Big data Data Query

39 Mapping to world of Big data Data Query Data Query

40 Mapping to world of Big data Analysis

41 Mapping to world of Big data Analysis Analysis(statistical computing and graphics env) Analysis(machine learning and data mining library ) Analysis (displaying, monitoring & analyzing)

42 Mapping to world of Big data Management and Co-ordination

43 Mapping to world of Big data Management and Co-ordination Reliable coordination service for distributed computing environments workflow/coordination/ job scheduling service to orchestrate dependencies between jobs

44 Big Data Functional Building Blocks Ecosystem Development Fx Script based log processing Display & Monitoring SDK Analysis Fx machine learning & data mining Lib statistical computing and graphics env Mgmt Fx High Level Data Processing API NLP and Math Lib X-Language service Dev Script based Data flow / transformation Processing Fx Coordination Workflow Job: Input -> Plot (key,data) -> Process -> Output Data Serialization Data Management Fx Distributed File System Distributed Database Monitor all Building Blocks Block based Query Language Streaming Data Collection Integration Fx Data Import Log Collection

45 Big Data Solution Building Blocks Technologies Development Fx SAWZALL CHUKWA Analysis Fx MAHOUT R Mgmt Fx CACADING CLOJRE ZOO KEEPER Processing Fx MAP REDUCE OOZIE Cloudera Mgmt Tools THRIFT AVRO PIG HDFS,Mogile FS,Lustre Data Management Fx Cassandra,Hbase,BigTable HIVE Integration Fx FLUME SQOOP SCRIBE

46 ORACLE BIG DATA SOLUTION

47 Oracle Big Data Appliance Big Data for the Enterprise HDFS Map Reduce Ecosystems Enterprise Support

48 Big Data: Infrastructure Requirements Acquire Organize Analyze Low, predictable Latency High Transaction Volume Flexible Data Structures High Throughput In-Place Preparation All Data Sources/Structures Deep Analytics Agile Development Massive Scalability Real Time Results

49 Divided Solution Spectrum Data Variety Schema DBMS (OLTP) ETL DBMS (DW) Advanced Analytics Dynamic Schema Acquire Organize Analyze

50 Divided Solution Spectrum Data Variety Schema DBMS (OLTP) ETL DBMS (DW) Advanced Analytics Dynamic Schema Distributed File Systems Transaction (Key-Value) Stores MapReduce Solutions Acquire Organize Analyze

51 Divided Solution Spectrum Data Variety Schema Oracle Database (OLTP) Oracle ODI Oracle Database (DW) Oracle BI EE Map Reduce Dynamic Schema HDFS Oracle NoSQL DB Oracle Loader for Hadoop Oracle Data Integrator In-DB Analytics R Mining Acquire Organize Analyze

52 Divided Solution Spectrum Data Variety Schema Oracle Database (OLTP) Oracle ODI Oracle Database (DW) Oracle BI EE Dynamic Schema Oracle BigData Hadoop NoSQL Database Oracle Loader for hadoop Oracle Data Integrator, R Acquire Organize Analyze

53 Divided Solution Spectrum Data Variety Schema Oracle Exadata OLTP & DW Data Mining & Oracle R Semantics Spatial Oracle BI EE Dynamic Schema Oracle BigData Hadoop NoSQL Database Oracle Loader for hadoop Oracle Data Integrator, R Acquire Organize Analyze

54 Divided Solution Spectrum Data Variety Schema Dynamic Schema Oracle Exadata OLTP & DW Data Mining & Oracle R Semantics Spatial Oracle BigData Hadoop NoSQL Database Oracle Loader for hadoop Oracle Data Integrator, R Oracle Exalytic Acquire Organize Analyze

55 Oracle Big Data Appliance Hardware 18 Sun X4270 M2 Servers 48 GB memory per node = 864 GB memory 12 Intel cores per node = 216 cores 24 TB storage per node = 432 TB storage 40 Gb p/sec InfiniBand 10 Gb p/sec Ethernet Name Node Name Node Name Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node

56 Oracle Big Data Appliance Software Oracle Linux 5.6 Apache Hadoop Distribution R Distribution Oracle NoSQL Enterprise Edition Oracle Data Integrator Application Adapter for Hadoop Oracle Loader for Hadoop

57 Scale Out to Infinity Scale out by connecting racks to each other using Infiniband Expand up to eight racks without additional switches Scale beyond eight racks by adding an additional switch

58 Big Data Appliance Batch Usage Model Oracle Big Data Appliance Oracle Exadata Oracle Exalytics InfiniBand InfiniBand Acquire Organize Analyze

59 SUMMARY

60 Big Data, An Integrated Architecture Capture Store/Proces s Integrate Organize Analyze Gover n Structured Master & Ref Data Transaction Data Machine Generated Social Media Text, Image Video, Audio DBMS (OLTP) Hadoop Cluster w MapReduce Key-Value Data Store DB Replic ETL/ELT ChangeDC Real-Time Unstructured Semistructured Message- Based ODS Data Warehouse Data Marts Streaming (CEP Engine) Reporting & Dashboards Alerting EPM BI Applications Text Analytics and Search In-Database Analytics Advanced Analytics Visual Discovery Management Security, Governance

61