Hochverfügbare Real-Time Datenreplikation auf MQ-Basis

Transcription

1 Patrick Hempeler Technical Sales Professional zim Hochverfügbare Real-Time Datenreplikation auf MQ-Basis

2 Disclaimer Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, DB2, InfoSphere, Cognos, and InfoSphere Warehouse on System z are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others

3 Bereitstellung von Massendaten Gängige Konzepte Kopie aus Primär- in Sekundärsystem Batch-Style in entsprechenden Zeitfenstern Hohe Latenz (täglich, wöchentlich, monatlich) Hohes Datenvolumen im Rahmen der Komplettladung Strukturgleichheit der Quell- und Zieltabellen Performance durch Ulitity-Operationen Transformation mittels Extraktprogrammen Batch-Style in entsprechenden Zeitfenstern Hohe Latenz (täglich, wöchentlich, monatlich) Hohes Datenvolumen im Rahmen der Komplettladung Transformation via Programmlogik Transformation mittels ETL-Tooling Performance via spezialisierte Parallelisierung (s. DataStage) Hohe Latenz (täglich, wöchentlich, monatlich) Transformation via Tool-Logik Hohes Maß an Flexibilität (Quellen, Ziele, Operatoren, grafisches Paradigma) im Rahmen der Komplettladung In der Regel Komplettladung, Delta nur wenn die Quellen es erlauben Delta-Replikation mittels Replication Software Geringe Latenz (Asynchron, aber Near Real-Time Verarbeitung) Geringes Datenvolumen da nur Delta verarbeitet werden muss Transformationen im Replication Tooling eingeschränkt möglich Geringst-möglicher Einfluss auf das Quell- und Zielsystem

4 Business & IT Challenges 1. Lower TCO through low impact, data capture solutions. 2. Reduce the time and costs associated with batch windows and data movement processes. 3. Improve business insight by providing continuous access to real-time information. 4. Ability to synchronize data in silos in real-time. Increase Revenue Increase Productivity Reduce Costs Smart Business New Intelligence I need insight Reduce Risk Smarter businesses are able to leverage information to drive smarter business outcomes

5 Challenges with Synchronizing Data to Applications/End-Users Reduce the time and costs associated with batch windows and data movement processes. Improve business insight & agility by providing continuous access to real-time information. Business Need Get information on demand for access to the most current / fresh information needed 1 for in daily 3 decision making and to improve customer service. business leaders frequently make critical decisions without the information they need i.e. depending on the industry, up to the minute reporting may be needed for fraud detection or risk avoidance Ensure the right information is available to the right person at the right time to enable timely decision-making Business Pain Decision makers and business users are working with aged data, risking errors or missing opportunities. IT Need Break out of reliance on nightly batch windows. Optimize the workload balance for the transactional systems avoiding performance degradation for operational transactions. Reduce cost of information infrastructure processing & hardware IT Pain Minimize cost of data processing. Reduce strain on operational systems to capture and synchronize data in silos Business Agility Business Insight Operational Costs Do More With Less

6 IBM Replication Technologies Multi-Tier Staging Concept (via CCD) Simple to implement Fan-Out to huge number of targets Pull Logic Circular connections Low latency, near-realtime, high volume replication Typically used for high availability, workload distribution, application integration Strong in DB2 marketplace Cross Plattform DB2 Active/Active Solution Can be linked to InfoSphere Classic Replication Bi-directional Replication Near-realtime via TCP/IP Strong in heterogeneous marketplace Can be linked to InfoSphere IMS Replication Java-based GUI for configuration, administration, and monitoring 3 Modes of Replication Continous/Periodic Mirroring, Refresh

7 Q-Replication Architecture DB2 Target DB2 Source IMS Source VSAM Source Log Log Control Capture Admin Utilities IBM WebSphere MQ Control Apply Sybase Target Oracle Target Informix Target Teradata Target SQL Server Target DB2 LUW Target IDMS Source Adabas Source Each message represents a transaction Highly parallel apply process Update Anywhere - Differentiated conflict detection and resolution for DB2 sources and targets Sources: DB2 (z/os & LUW), VSAM, IMS, IDMS, ADABAS Targets: DB2 (z/os & LUW), Informix, Oracle, Sybase, Microsoft SQL Server, Teradata Guaranteed one-time delivery of data SSL and security options for encryption, digital signatures, etc An outage at one site doesn t prevent progress on the other; example: Capture and MQ continue even when target system down Work queued up for target - no need to restart at source when outage ends Nicknames Federation Layer

8 DB2 Q-Replication Process Flow SOURCE SOURCE2 METADATA SOURCE1 METADATA Q Apply Browser DB2 Log Q Capture Apply Agent Apply Agent ADMINISTRATION Apply Agent TGT3 Replication Center Replication Monitor TARGET TGT1 TGT

9 Performance & Health Monitoring Reduction Click to add an outline Available through: Database queries Dashboards Dashboards: Web-based Live and historical graphs At-a-glance health summary Drill down for details Reports for SLA compliance Customers: We need help Performance advice How to handle recovery

10 User Interfaces Replication Center GUI Launchpads, Wizards, Online Help Definitions, Operations, Monitoring Command Line Interface Scripts or interactive mode ASNCLP Java APIs Typically used when replication is embedded

11 Key Elements Of Incremental Data Delivery Value Proposition IMPACT (Lowers Risk and Cost) 1. Lower cost of changed data access. 2. Use of native DB logs, lower overhead. 3. Non intrusive to applications and databases. 4. No use of database triggers. 5. Management easily integrated into existing IT operations. 6. Reduces risk to operational systems. Low Impact Change Data Capture Low Latency LATENCY (Increases Business Visibility) 1. Near zero latency for pervasive integration projects. 2. ETL can also deliver low latency but at significantly higher impact to production systems and mission-critical applications. Consistent Delivery CONSISTENT DATA DELIVERY (Trusted Delivery) 1. Data pushed, delivered in continuous stream, continuous with business operations. 2. Transaction consistency maintained to preserve units of work, referential integrity. 3. Full transaction granularity, before and after image of all transactional changes. 4. Data event aware, can be used to trigger specific business processes. 5. Fault tolerance, recover to last committed transaction

12 Technical challenges with incremental data delivery Performance Can the solution handle massive, increasing data volumes with low latency? Impact on source system How does the solution impact source system CPU or MIPS consumption? Breadth of heterogeneous support Can the solution support different platforms and databases? Supported topologies Is the solution flexible enough to support varying architectural topologies? Extensibility Can the solution be integrated with other applications or initiatives that require incremental data delivery? Recoverability and transactional integrity Will data be lost in the event of a network outage or system failure?

13 Technical challenges with incremental data delivery IBM change data capture & replication solutions Performance Log-based change data capture and optimized applies help ensure scalability and maximum throughput Impact on source system Breadth of heterogeneous support Supported topologies Extensibility Direct and out of the box integrations for delivering data to ETL or Data Quality processes, MDM systems, warehouses, and JMS queues for SOA environments Recoverability and transactional integrity Native, proprietary (not third party) log-based capture agents have minimal impact and risk on source systems Expansive breadth of OS and native database support ranging from DB2 on all platforms, Informix, Oracle databases, Sybase, SQL Server, Teradata, VSAM, IMS, Adabas, IDMS, Datacomm Supports all major topologies for flexibility and minimizing IT infrastructure changes Built-in recoverability and designed for exceptional data delivery to ensure data/transactional integrity and resiliency to network and system failures

14 Workload Scenarios Critical data protection and high availability Build a dynamic ODS for operational reporting or for Audit database E-business application synchronization Complete an existing architecture Data synchronization for upgrades, migrations and Workload Balancing Deport SQL statements of Production and Reporting

15 Critical data protection and high availability Solutions that enable companies to have a copy of critical data as a backup solution or a high availability solution. Production Server Availability Server OLTP Finance Partition 1 Backup Native DB Log QREP Backup Continuous (to backup instance) Partition 2 High availability solution in Unix/Oracle environments InfoSphere QREP process Other Technology Creation of backup database on another server or on the same server but on another LPAR. In case of two or several LPARs, only one license is mandatory. InfoSphere QREP license

16 Build a dynamic ODS for operational reporting or for Audit database. Solution deployed to improve business visibility in the companies who have some need of operational business intelligence (BI) and data audit. Production Server ERP OLTP (On-Line Transaction Processing) Native DB Log QREP Continuous (to Dynamic ODS ) Operational Data Store ODS Manufacturing OLTP Production Server Native DB Finance Log Manufacturing QREP Continuous (to Dynamic ODS ) Process InfoSphere QREP Other technology License InfoSphere QREP All operations as insert, update & delete are converted in inserts to maintain a complete history of transactions. Additional information as timestamp, transaction type, source system ID, user ID who has done the transaction.

17 E-business application synchronization Solution deployed to bring to customers in real time a visibility on sales and item s availability in e-business applications domain. Website QREP Continuous (bi-directional) Native DB OLTP Production Server Log OLTP Native Inventory Log DB QREP Continuous (bi-directional) Sale site Retail Native DB Log OLTP InfoSphere QREP process Other Technology InfoSphere QREP license Permit a synchronization in real time and in bi-directional mode between web applications and critical production servers

18 Complete an existing architecture Solution deployed to improve the business visibility (Dynamic Warehousing) and reduce the impact of Extract Transform and Load ( ETL) on critical systems Production Server ETL Server Data Warehouse Point Of Sale OLTP Native DB Log QREP Continuous Stage ETL Scheduled Batch EDW Retail InfoSphere QREP process Other Technology InfoSphere QREP license Target can be: 1. Relational table 2. Message Queue Complete ETL technologies : 1. IBM DataStage (native integration) 2. Business Objects Data Integrator 3. Ab Initio 4. Informatica Power Center

19 Data synchronization for upgrades, migrations and Workload Balancing Solutions that allow IT to manage the migrations of applications, databases and platforms. Production Server Testing Server ERP ERP OLTP Native Upgrades, Migrations Native OLTP DB Log Workload Balancing DB Log Manufacturing Manufacturing InfoSphere QREP process Other Technology InfoSphere QREP license Permit to synchronize data between actual production server and other deployed server to test a new version or a new upgrade. Capacity to do some load balancing with an integrated conflict management

20 Deport SQL statements of Production and Reporting Solutions that allow companies to deport the generated impact by heavy SQL statements of reporting to a less critical system. Production Server(s) OLTP Native DB Log Finance 1 QREP Continuous InfoSphere QREP process Other Technology InfoSphere QREP license Reporting Server Services OLTP Finance 2 Table Copy Report Query Services QREP Continuous OLTP Native Finance 3 A Reporting server may be used for consolidation needs. For example: to consolidate financial information coming from different sources to the same target. DB Log Services The replication mode can vary: real time, batch mode or refresh mode

21 Customer Examples German Health Insurance German Financial Market

22 Customer Example German Health Insurance

23 Customer Example German Health Insurance

24 Deutschen WertpapierService Bank AG A seamless switch to back-up systems prevents losses Business challenge dwpbank's core business revolves around processing trades and financial transactions. Any system outages - from planned downtime to disasters cost a significant amount of money and damage to the bank s reputation. The bank had fail-safes in place but felt its precautions were not enough to seamlessly switch over to back-ups without interrupting operations. dwpbank needed a redundant IT solution that would create a fully synchronized, independent failover environment for its core trade processing activities. Solution With help from IBM Global Business Services, dwpbank deployed a business continuity solution based on IBM WebSphere Information Integrator V8.2 for z/os and Unix. The new infrastructure synchronizes information between the primary and back-up systems using low latency queue replication, allowing the company to immediately switch to failover systems without losing data. The Global Business Services team introduced the new technology to dwpbank, developed the IT architecture and implemented the solution. IBM also provides ongoing support at the client's site. Benefits Regular operations can continue in the event of a system outage Crucial financial data preserved in the event of disaster Provides extra processing capability on a normal basis

25 Announcement Letter IBM InfoSphere Data Replication for DB2 for z/os, V10.1 delivers high-volume, low-latency data replication technologies and solutions for enhanced flexibility and choice Links InfoSphere Replication Server for z/os InfoSphere Change Data Capture for z/os InfoSphere Data Event Publisher for z/os

26 Kontakt Patrick Hempeler IBM-Allee 1 Patrick Hempeler IBM-Allee 1 Technical Sales Professional D Ehningen Technical Sales Professional D Ehningen z/ Information Management z/ Information Management IBM Software Group Mobil: 0151 / IBM Software Group Mobil: 0151 / patrick.hempeler@de.ibm.com patrick.hempeler@de.ibm.com

27 Backup

28 Q-Replication process types SOURCE CD1 SOURCE CD1 COPY COPY COPY COPY COPY SOURCE CD1 SOURCE CD1 SOURCE CD1 Uni-directional Data Distribution (1:some) Data Consolidation (n:1) REPLICA REPLICA REPLICA REPLICA PRIMARY CD1 SECONDARY CD1 Subsets Transformations Conflict Detection/Resolution Updateable Predicates Updateable Primary Keys High-Availability Applications Peer to Peer Bidirectional (Bidirectional and Peer-to-Peer Currently Not Supported for Classic Sources)

29 Multi-Directional Conflicts detection & Recursion Read/Write Applications Read/Write Applications Q Apply Q Capture Primary Database Q Capture Q Apply Secondary Database Replication processes and subscriptions are defined in both directions and data changes flow in both directions Recursion is stopped by Capture, which reads special logged events created by Apply Conflict detection is typically necessary, unless the application is carefully designed to completely avoid conflicts

30 Conflict Detection & Resolution Unidirectional 2 participating nodes Changes are replicated in one direction between two servers (i.e. from source to target) Changes can be filtered and transformed No Conflicts Bidirectional (Value based conflict detection) 2 participating nodes Changes are replicated in both directions between two servers Minimal overhead Utilizes VALUE based conflict detection Peer to peer (Version based conflict detection) 2 or more participating nodes (practical limit around 6) Requires extra columns and triggers Most robust conflict detection and resolution (with some restrictions) Utilizes VERSION based conflict detection Important for Active standby systems Workload balancing

31 Performance Advisor What it does: Analyzes historical data Provides recommendations V10 focus is reducing latency For an 'alert' For trends over time Launch points New tabs for advisors Directly from alert 13 tuning recommendations possible

32 Recovery Advisor Provides help with five scenarios E.g. for failover and switchback in continuous availability configurations Two ways to start Select a recovery scenario Analyze a configuration for any issues that need to be addressed Steps are provided Explanation for each Description of user action When needed, a full command that can be copied and run

33 Event Publishing Conceptually, Q Replication without the Q Apply Change data goes into WebSphere MQ queues MQ delivers data to consuming applications Example InfoSphere DataStage TARGET Source SOURCE1 SOURCE2 WebSphere MQ SOA/User Application Capture WBI Event Broker User Application TGT1 DB2 Log DataStage TGT2 TGT