Frank Peters, Schenker AG, Information Mgmt. Dr. Dierk Hahn, Capgemini, Insights & Data June 2015

Transcription

1 Hochautomatisierte Generierung einer kompletten Applikation-Staging-Synchronisation Architektur inklusive CDC The BackRoom Staging solution for DB Schenker Logistics Frank Peters, Schenker AG, Information Mgmt. Dr. Dierk Hahn, Capgemini, Insights & Data June 2015

2 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 2

3 DB Schenker is the transport & logistics division of Deutsche Bahn AG Deutsche Bahn AG Chairman (G) Finances (F) Compliance / Legal (C) Human Resources (H) Technology (T) DB Mobility ML AG Logistics AG CEO(G) Finances (F) Compliance / Legal (C) Human Resources (H) Technology (T) Passenger transport (P) Transport&Logistics (L) Infrastructure (I) DB... Schenker Schenker... Rail DB Logistics Source: DB AG 3

4 Global transport networks for all modes of transport strengthen the market position of DB Schenker Source: DB 1 According to tkm 2 According to revenue 3 According to t 4 According to TEU 5 According to revenues 4

5 DB Schenker Logistics offers global transport and logistics solutions onshore, by sea and in the air DB Schenker Logistics Employees: 64,810 Revenues (EUR): EBIT (EUR): Land transport (shipments): Air freight (t) 1 : Ocean freight (TEU) 1 : CL/SCM (revenues): 14.9 bn 332 m 98.9 m 1.11 m 1.98 m 2.0 bn Own locations Partner 1 Export 5

6 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 6

7 The BackRoom Staging platform is a building block in the reusable platform approach for Information Management FrontRooms Benefits & Synergy Opportunities Special purpose and local Data Marts Regional Data Mart Cross Business Unit Data Mart Customer Data Mart Others Differentiating, target oriented FrontRooms BackRoom Consistent Approach & Methodology Data Warehouse Data Mart 1 Data Mart 2 Data Mart 3 Other Consistent Data Access Consistent Data Integration Integrated / Operational Data Store ODS A ODS B IDS C IDS D Other Reusable Components Shared Platform Approach Staging Area BackRoom Staging - Real-time with OGG Application A Staging Application B Staging Classical Staging - Batches Application D Staging Application D Staging Other Failover and scalability optimization of infrastructure (RAC + GRID clusters) Enterprise Licensing Source Systems Application A Application B Application C Application D Other 7

8 Demand for flexible real-time staging solution raised from real-time operational reporting needs decoupled from sources Demands Increasing need for operational real-time reporting Real-time staging solution decoupled from / not impacting source systems Increasing demands of target system regarding reporting Increasing business unit demands for real-time visibility Operational reporting more integrated in operational processes and systems Real-time reporting requires an efficient, lean delta handling Reporting / staging decoupled from operational systems No system load impact on operational systems (sources) No impact on operational systems service level agreements No impact on source systems development (e.g. CDC) High availability and failover Staging solution needs to automatically follow a source system failover event High availability of staging solution by a clustered setup Flexibility in regard to operational system development Continuous development and extension of operational systems Staging solution must be able to flexibly and quickly follow source system data model changes 8

9 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 9

10 BackRoom Staging is a reusable platform providing decoupled, real-time access to application data and changes ODS A ODS B In nitial Delta In nitial Delta Interface Views 1:1 CL (Change Log Table) Interface Views 1:1 CL (Change Log Table) BackRoom Staging OGG OGG Application A Application B 10

11 BackRoom Staging enriches the source data and provides a change log to ease delta load for target applications 1:1 Replication Area. Change Log Area Change Data Capture out of the box All data entries (1:1 Replication and Change Log) enriched with additional timestamp and source information 1:1 Replication Area represents a selected real-time synchronization of the source system Change Log Area provides all data changes as delta history (enables very fast delta processing) Lifecycle / kept history in Change Log configurable to control data volume Insert Update Delete Primary key changes on source side are converted to a delete and insert combination in Change Log 11 Equal treatment for all applications / generic architecture and design

12 BackRoom Staging Builder generates all necessary scripts and configurations just based on table list and source DB Interface Matrix (Table List) BackRoom Staging Builder Initial Load/Data Migratio on Source Database Source Schema Staging Database Repl Area Interface Area Redo Log CDC Area OGG Replication SYNC Mining DB 12 High flexibility and easy adoption by use of BackRoom Staging Builder Target application using BackRoom Staging provides interface matrix (list of source tables to replicate) Source definitions are read from source DB BackRoom Staging Builder then generates from this all necessary Staging DB and Oracle GoldenGate scripts and configurations Generated scripts are deployed, initial load or data migration is processed, finally real-time replication is started Important is the tight alignment and synchronization with the source system deployment / release

13 BackRoom Staging provides flexible, real-time access to operational data with minimal impact to source application Advantages Minimal source system impact Minimal loss of performance No modification of the source No active components on source beside backup Solution is flexible and reusable Most DBMS could be integrated Change log is provided for all source systems OGG guarantees data completeness Operation and administration Shared real-time replication platform unifies operations Administrators are used to deployments and migrations OGG automatically follows source database failover Shared platform reduces costs Shared infrastructure and hardware Only one target DB to be (OGG) licensed Shared service with uniform processes enables synergies Pays out if multiple operational systems with real-time reporting demand exist 13

14 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 14

15 15

16 16

17 17

18 18

19 19

20 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 20

21 The different demands leads to a stable platform architecture with a industrializing custom developed tool Demands Customer Demands Development effort Often source modifications Customer Infrastructure Industrialization of development and deployment GOAL: Real-time staging solution decoupled from / not impacting source systems Audit of data Integrate in Customer chosen landscape: OGG, staging.. Mile stones due alignment with external projects Many similar configuration details (SQL,OGG,lists) OGG uses detailed configuration files, no GUI Elimination of Rework Error free creation of file set for deployment Data model coupled to source Short reaction times Multi stage environment Outsourced deployment Usage of CR and approve processes General Architec ture Tool based develop. deployment method 21

22 In this project the focus was set on the Staging Area 22

23 The stable architecture reflects all demands form source and target application and provides real time data High flexibility and easy adoption by use of BackRoom Staging Builder Provide the complete source data Provide all changes of data (CDC) Staging Database Repl Area Interface Area CDC Area Provide security options Provide data selection options OGG Replication Provide real time data SYNC No impact on source system No impact on source design Source Database Source Schema Mining DB Redo Log 23

24 SOURCE: ORACLE How ORACLE Golden gates works Extract: committed transactions are captured (and can be filtered) by reading the transaction logs. Trail: stages and queues data for routing. OGG also supports Downstream Capture for less impact on source system Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Replicat: applies data with transaction integrity, transforming the data as required. Extract Trail Files Pump Pump Trail Files Replicat Source Oracle & Non-Oracle Database(s) Replicat Trail Files Pump LAN / WAN / Internet Over TCP/IP Bi-directional (possible, not used) Pump Trail Files Extract Target Oracle & Non-Oracle Database(s) 24

25 SOURCE: ORACLE ORACLE Golden Gate with Integrated Capture uses Downstream technique with ORACLE Data Guard Observer Primary Database Standby Database Redo Transport (SYNC or ASYNC) Redo Transport OCI Connection (ASYNC) LogMining Server File I/O Integrated Capture 25

26 The synchronisation architecture creates the replication and CDC area using customized mapping processes Golden gate configuration The mining DB acts as source proxy Staging Database Interface Area The data selection is done by the Collect process Repl Area CDC Area Data pumps doubles all selected data Delivery of data over wide area possible Apply processes integrate the data in the target DB Special mapping is done for audit functionality Special mapping is done for CDC functionality Initial Load/Data Migration OGG Environment running on target database Source Database Replicat/ Apply Repl. Area Datapump Repl. Area SYNC Replicat/ Apply CDC Area Datapump CDC Area Extract / Collect Mining DB Parameters Table Lists Source Schema Redo Log 26

27 Many deliverables are necessary to build a complete harmonized BackRoom Stage Deployment configuration 1. The source data base is configured to deliver all synchronization data 2. Data of selected tables are collected 3. Data pumps guarantee the delivery of the right data CDC and Repl can differ 4. The apply processes map the source data to the target data base incorporate the CDC and meta data 5. The DDLs create the needed stage Migration scenarios can be used Initial Load/Data Migration DDL Migration Data fill Mappings Table Mappings Lists Migration/ Initial Load 6 Supplemental logging OGG Environment running on target database 4 5 Staging Database Repl Area Replicat/ Apply Repl. Area Interface Area CDC Area Extract / Collect 5 5 Replicat/ Apply CDC Area 4 3 Datapump Datapump 3 Repl. Area CDC Area SYNC 2 1 Source Database Mining DB Parameters Table Lists Source Schema Redo Log 27

28 The BackRoom Staging Builder generates all data base and OGG configuration just by a button press TableList once per Application Release Modules, Attribute-exclude/extension, where-clauses Config INPUT Application: Source DB schema with Definitions Deliver: source_ definition.csv Config DEF Write DEF Read DEF BackRoom Staging Builder DEF Application Tables Automated generation Complete deployment is generated by button press Table selection is included from configuration Per table options are incorporated Working Tables SQL-Cmd Deliver Staging DDLs Deliver OGG Lists+ Mappings Deliver Admin Params OGG Environment running on target database Source Database Source Schema Staging Database Repl Area Replicat/ Apply Repl. Area Datapump Repl. Area Interface Area SYNC CDC Area Replicat/ Apply CDC Area Datapump CDC Area Extract / Collect Mining DB Redo Log 28

29 The modules of the BRSB generate predefined functionality using easy table-specific configuration Data consumer Hide all columns with specific globally defined name pattern (e. g. use for hiding private data) BackRoom Staging Interface Views BackRoom Staging ExcludeBlobsFromView: hide blob columns in interface view add synchronization info (Load timestamp and commit sequence number) to tables (e.g. for use for incremental loading) 1:1 Change- Replicated Log Application Tables Tables AddCL: add a change log table and a corresponding view (use for change tracking) Create CTAS statements for this table (create table as select) for deployment method Migration with CTAS OGG: Apply ExcludeBlobsFromMapping: columns with data type blob will not be synchronized (use to save space and performance on incompatible or nonreportable information) One button press creates all deliverables 29

30 The BackRoom Staging Builder resists on a separated system using reporting demands and source definitions HTML-Documentation Deployment files for one version Deployments Configurator START- BUTTON Source code Helper Docu- StartButton running 30

31 Deployment Scenarios Main use cases BRS Greenfield Deployment: New BRS installation on a clean BRS environment Deployment methods Initial Load during Freeze Initial Load without Freeze BRS Stage Update No changes on source to consider Changes to scope of replication or other BRS reconfigurations Source Update New source application release Hotfixes on source with data model changes Migration with CTAS/part dump Migration with source scripts Limited Migration (no Freeze) recommend possible if necessary 31

32 Migration concepts integrate the steps of a source release and the BackRoom STG release to minimize freeze times Release notes The data model of the source and the target are connected Release process is perfectly aligned Synchronization database maintenance can be done after application migration on work days Synchronization is kept by stored log files Migration can be applied directly or by partial loading updated data/tables 32

33 Using this architecture even more complex infrastructures can be build and maintained COLLECT DATA-PUMP APPLY Source 1 (INT 5) Source 1 TableList 1 PAR REPL PAR CL PAR REPL CL FRI SE (PAR) Source 2 (INT 4 ) Logging Source 2 TableList 2 COLLECT INT REPL INT CL INT REPL CL FRI SE (JIT) TableList 3 Source 3 (DEV) Appl.-DB Source 3 Mining DB Trail-File DEV REPL DEV CL Trail-Files DEV REPL CL FRI CG Source-Server 1 Example of a running environment Source Logging Source COLLECT MM REPL MM REPL CL FRI MM Appl.-DB Mining DB TableList MM CL Staging INT Source-Server 2 Schema Trail-File Process Target Server 33

34 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 34

35 BackRoom Staging solution works well for real-time staging demands for multiple operational systems Summary Real-time Staging Solution Enables real-time reporting decoupled from source application Almost no impact to source application performance Change Log provides easy access for subsequent delta loads Only very limited configuration changes on source database BackRoom Staging Builder Concept Generation concept enables quick adaption to source changes and target requirement changes, in particular important for source applications under development Effective and quick release and deployment preparation Pays out for multiple source application with real-time reporting demand Lessons learned Oracle GoldenGate provides real-time replication, initial or migration loads one need to take care himself Failover and high availability requirements increase complexity and need to be considered right from the beginning Tight alignment and integration with source application release and deployment processes necessary and crucial 35

36 BackRoom Staging provides flexible, real-time access to operational data with minimal impact to source application Advantages Minimal source system impact Minimal loss of performance No modification of the source No active components on source beside backup Solution is flexible and reusable Most DBMS could be integrated Change log is provided for all source systems OGG guarantees data completeness Operation and administration Shared real-time replication platform unifies operations Administrators are used to deployments and migrations OGG automatically follows source database failover Shared platform reduces costs Shared infrastructure and hardware Only one target DB to be (OGG) licensed Shared service with uniform processes enables synergies Pays out if multiple operational systems with real-time reporting demand exist 36

37 Agenda1 Presentation DB Schenker Logistics Demand and context for a real-time staging solution BackRoom Staging solution overview Presentation Capgemini BackRoom Staging solution deep dive Conclusion Questions & Answers 37

38 Frank Peters DWH & BI Architect Schenker AG Name Last Name Senior Managing Consultant Capgemini Deutschland 38