Maximize efficiency of Data" with Data Integration. Pramote Trinaksit Senior Technical Consultant May 10, 2016

Transcription

1 Maximize efficiency of Data" with Data Integration Pramote Trinaksit Senior Technical Consultant May 10, 2016

2 Agenda o Why Integration? o Today Data Integration Solutions o Simplify Data Consolidation o Master Your Data with MDM 2

3 Why Integration? 3

4 Silos Business Applications Today is world of heterogeneity. We have different technologies. We operate on different platforms. We have large amount of data being generated everyday in all sorts of organizations and Enterprises. Data Stores And we do have problems with data. 4

5 Internet of Things (IoT) 5

6 Internet of Things (IoT) 6

7 Where the Wireless Things Are Information and Analysis Automation and Control 7

8 Big Data Drives Big Benefits 8

9 Big Data Landscape 9

10 Why Data Integration? NEED Business Intelligence Information How and Where you Want It Corporate Performance Management Business Process Management Business Activity Monitoring Data Integration Migration Data Warehousing Master Data Management Data Synchronization Federation Real Time Messaging Data in Disparate Sources HAVE Legacy --- ERP CRM Best-of-breed Applications 10

11 Data Integration Trends Data Integration Tools Today Need to Do More 11

12 Conventional ETL Design Staging Area Legacy Application Coding Oracle, MSSQL,DB2 1 SQL, Procedure, Function SQL, Procedure, Function Text, Excel, CSV 3 Lookup ODBC, SQL Loader, Coding 12

13 Today Data Integration Solutions 13

14 Data Integration Strategy 14

15 Data Integrator Tool 15

16 Differentiator: Declarative Design Specify ETL Data Flow Graph Developer must define every step of Complex ETL Flow Logic Traditional approach requires specialized ETL skills And significant development and maintenance efforts Declarative Set-based Design Simplifies the number of steps Automatically generates the Data Flow whatever the sources and target DB Benefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-it pros Conventional ETL Design ODI Declarative Design 1 2 Define What You Want Automaticall y Generate Dataflow Define How: Built-in Templates 16

17 1 Define Design Transformations What You Want 3 Automatically Generate Data flows Oracle Data Integrator Interface Declarative Design 2 Define How to Do It: Select Template Bulk Load Changed Data Capture Incremental Update Slowly Changing Dimension 17

18 Orchestrate Data Flows 1. Sequence Transformations 2. Leverage OracleDI Tools Data Quality Processes Files/Archives Management Send/Receive s Web Services Invokation Event Detection Create your Own Tools 3. Use Control Structures Loops Conditions Error Handling 18

19 Data Integration Strategy 19

20 Today - Strategy for Real-Time Analytics 20

21 Real Times Replication Tool 21

22 Flexible Deployment Models Unidirectional Query Offloading, DR Site, Migration Bi-Directional Active-Active for HA Peer-to-Peer Load Balancing, Multi-Master Broadcast Data Distribution Integration/Consolidation Data Warehouse Cascading Data Marts 22

23 Data Consolidation & Distribution Architecture Customer Real-time Data replication Real-time Data replication Region 2 HQ Region 1 Customer Real-time Data replication Remote Site Region 3 Customer DR Customer Region 4 Remote Site Real-time Data replication Central Site Reporting DR Site 23

24 Real-Time Data Integration Platform EMP Source 1 DEPT Oracle GoldenGate Real time extracts from transactional systems Non-invasive on sources Continuous streaming load into ODS Schema of target Latency in seconds Oracle Data Integrator EE High Performance E-LT on target data warehouse Periodic mini-batches (15 min) Transform in the Database Never go back to sources Oracle Data Integrator On-Disk Logs Source 2 Oracle GoldenGate DIM FACT DIM EMP DEPT EMP DEPT DIM DIM Oracle GoldenGate On-Disk Logs ODS Schema DW Schema 24 24

25 Data Integration Strategy 25

26 Why Impact of Data Quality on BI? 26

27 Your Data Contains Errors and Inconsistencies 27

28 Companies Core Problem: Data Degrades Individuals Products In one hour In one hour In one year 240 businesses will change addresses 150 business telephone numbers will change or be disconnected 112 directorship (CEO, CFO, etc.) changes will occur 20 corporations will fail 12 new businesses will open their doors 5,769 individuals in the US will change jobs 2,748 individuals will change address 515 individuals will get married 263 individuals will get divorced 186 individuals will declare a personal bankruptcy Source: D&B, US Census Bureau, US Department of Health and Human Services, Administrative Office of the US Courts, Bureau of Labor Statistics, Gartner, A.T Kearney, GMA Invoice Accuracy Study Data changes at rate of 2% per month. On average 20% duplicates in product data 90% of new product introductions fail Retailers lost 40 billion or 3.5% of total sales lost each year due to item info inefficiencies 60% error rate for all invoices generated Global Data Sync will realize 30% lower IT costs Compounded, 2% monthly change is 27% per year, 61% in two years, 104% in three years!!! 28

29 Data Governance Life Cycle PREVENTION IS BETTER THAN CURE 29

30 Profiling & Discover Column Profiling New! Column & Rule Profiling Visual of Nulls, Unique & Non-Unique Value & Pattern Freqs Drill Down Results 30

31 Manage Flow and Exceptions Sample Workflow with tasks. Human Task is made up of steps. 31

32 Data Quality Use cases 32

33 Best Real-Time Data Integration Platform 33

34 Data Integration Strategy 34

35 System Components - Intelligent Data Warehouse/Data Mart & Business Analytics Platform End-to-End Security Management Font Channels & Data Sources 35

36 Simplify Data Consolidation 36

37 Phases of IT Optimization 37

38 Why need database consolidation? 1. Reduce complexity Servers running standard configurations 2. Improve efficiency Drive up hardware utilization rates 4. Simplify maintenance Fewer servers in fewer locations 5. Better security Smaller security perimeter 3. Lower costs Hardware/Software (License and Maintenance) Energy and floor space 38

39 Resource Management 39

40 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 40

41 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 41

42 Consolidate with Oracle Multitenant New architecture for consolidating databases and simplifying operations DW ERP CRM Complementary to VMs Self-contained PDB for each application Applications run unchanged Rapid provisioning (via clones) Portability (via pluggability) Common operations performed at CDB level Manage many as one (upgrade, HA, backup) Granular control when appropriate Shared memory and background processes More applications per server 42

43 Mixed Workloads OLTP peak time is day DWH peak time is night Pre-12c DBs high resource allocation for peak use Low resource usage during non-peak activity OLTP day night day DWH day night day Pre-12c OLTP + DWH in the same CDB on the same Hardware = efficient use of resources c day night day 43

44 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 44

45 High Availability Production Site RAC Scalability Server HA Flashback Human error correction Application Continuity Application HA Global Data Services Service Failover / Load Balancing Active Replica Active Data Guard Data Protection, DR Query Offload GoldenGate Active-active Heterogeneous Edition-based Redefinition, Online Redefinition, Data Guard, GoldenGate Minimal downtime maintenance, upgrades, migrations RMAN, Oracle Secure Backup Backup to tape / cloud 45

46 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 46

47 Consolidated Database Performance Testing 47

48 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 48

49 Enterprise Manager Cloud Control 12c 49

50 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 50

51 Data Breach 67 % Records breached from servers 76 % Breached using weak or stolen credentials 69 % Discovered by an external party 97 % Preventable with basic controls 51

52 From MISTAKES to MALICIOUS Basic security not enough for today s business Accidents Disclosures Privilege Abuse Curiosity Data Leakage Social Engineering Sophisticated Attacks Business Data Theft Reputation Loss 52

53 Why Are Databases Vulnerable? 80% of IT Security Programs Don t Address Database Security Network Security Authentication & User Security SIEM Security Database Security Endpoint Security 53

54 Completely Database Security Prevent Data Breaches and Satisfy Regulatory Requirements 54

55 Um How to Start? 6-Step for Comprehensive Database Security 1. Understanding what data needs to be protected 2. Understanding applicable regulatory compliance requirements 3. Performing an inventory of all databases, including nonproduction 4. Discovering and classifying databases based on sensitivity of data 5. Establishing security policies for all databases 6. Taking appropriate security measures 55

56 Data Consolidation Strategy 1. Resource Management 2. High Availability 3. Testing 4. Operation 5. Secure Data 6. Separate of Duties 56

57 What s Cloud Computing On demand network access Shared pool of configurable computing resources Provisioned by the Service Provider 57

58 Changes the way we use IT resources 58

59 Changes the way we use IT resources 59

60 Types of Cloud Computing 60

61 Benefits Summary 61

62 Master Your Data with MDM 62

63 The Information Challenge 63

64 Real Life Meet Adam A long serving, high ARPU customer of Service Provider QuadPlay My name is Adam Jiminez, I live in Bridgeport I ve been with my Service Provider QuadPlay for 8 years now They provide me and my family with mobile, broadband and home phone services They keep trying to sell me broadband which I already have but what I really want is a discounted TV package Data Records for Adam from CRM Quadplay: Name Address DoB Address Contact Number Products Customer Since Marketing Opt in Billing Date Avg Mth Spend Credit Card Adam Jiminez 716 Mollis Street, Bridgeport, USA /20/1970 adams Mobile40 4GB 10/10/2012 Day 10 $ Mr A Jiminez 716 Mollis Street, Bridgeport, USA 10/20/1970 a.jiminez@gmail.com HomeFone Std 27/03/2007 Yes Day 27 $50 Adam Jimminez Mollis Street, Bridgeport, USA /20/1970 adamjimminez@quadplay.com BBUnlimited 03/07/2010 Day 3 $50 Adam N Jiminez 716 Mollis Street, Bridgeport, USA jiminez@yahoo.com PrePay Freedom 14/09/2015 N/A $10 Name & Address Variations Non Standard format Missing Data? $10 ARPU or $310 ARPU? 64

65 Implementation Step james smith th avenue suite 7 nyc new york First Name: James james Last Name: Smith smith AddressL1: 1008 Avenue 6 th avenue of the Americas AddressL2: Suite suite 7 City: New nyc York State: NY new york Zip Code: System X Jim J. Smyth New York, NY jsmyth@mywork.com (212) Profile Parse Correct Standardize Match Merge Enrich First Name: Jim Mid Name: J. Last Name: Smyth AddressL1: 1008 Avenue of the Americas AddressL2: Suite 7 City: New York State: NY Zip Code: Phone: (212) jsmyth@mywork.com First Name: Jim Mid Name: J. Last Name: Smyth AddressL1: 1008 Avenue of the Americas AddressL2: Suite 7 City: New York State: NY Zip Code: Longitude: Latitude: Phone: (212) jsmyth@mywork.com C_Category: Affluent Couples & Families C_Group: Affluent Families 65

66 Core Capabilities of the MDM 66

67 How Does Help Customers with MDM? 67

68 Contact: Website: Tel: