XDM TEST DATA PROVISION & DATA MIGRATION. UBS Hainer GmbH, the IBM Business Partner

Transcription

1 TEST DATA PROVISION & DATA MIGRATION

2 Decoupling: Data Security & Consistency Production DB2, Oracle, etc. Clone Preproduction DB2, Oracle, etc. Copies Test Environments 2

3 Row Level Processing Unit Tests Table Copying Component / Integration Tests Database / Subsystem Cloning Release / System Tests / Preproduction 3

4 Component Overview 4

5 COMPONENT OVERVIEW DATABASE CLONING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 5

6 COMPONENT OVERVIEW DATABASE CLONING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 6

7 What does XDM do? 7

8 WHAT DOES XDM DO? XDM is a convenient data mover for relational data. It allows you to copy within and between DBMS instances. 8

9 WHAT DOES XDM DO? Goals Simplify & automate the data provision procedure Save expert time Speed up execution Separate test from production Increase operational reliability 9

10 WHAT DOES XDM DO? DB2 for z/os DB2 for z/os Supported Platforms DB2 for Linux / Unix / Windows Oracle on Linux / Unix / Windows MS SQL Server 10

11 WHAT DOES XDM DO? Work is organized in tasks Task definition Select source and target DBMS (Prod/Test) Select objects / select rename rule Select copy options Task administration graphical client Execution: Command line or scheduled 11

12 WHAT DOES XDM DO? Work is organized in tasks Task definition Select source and target DBMS (Prod/Test) Select objects / select rename rule Select copy options Task administration graphical client Execution: Command line or scheduled 12

13 WHAT DOES XDM DO? Work is organized in tasks Task definition Select source and target DBMS (Prod/Test) Select objects / select rename rule Select copy options Task administration graphical client Execution: Command line or scheduled 13

14 WHAT DOES XDM DO? Work is organized in tasks Task definition Select source and target DBMS (Prod/Test) Select objects / select rename rule Select copy options Task administration graphical client Execution: Command line or scheduled START COPY 14

15 COMPONENT OVERVIEW DATABASE CLONING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 15

16 SCHEDULED CLONING How does a clone become consistent? NIGHT SHIFT Time 16

17 SCHEDULED CLONING How does a clone become consistent? POINT IN TIME Time 17

18 SCHEDULED CLONING How does a clone become consistent? Inflight Copy Rollforward of logs Time Transactions while copying 18

19 DUPLICATE SOURCE DATABASE A typical scenario: How can I get a preproduction of this A typical scenario: production environment? ORA 1 DB2 z/os ORA 2 DB2 LUW 19

20 DUPLICATE SOURCE DATABASE How does a clone of such an environment become consistent? ORA 1 ORA 1 ORA 2 ORA 2 DB2 LUW DB2 LUW DB2 z/os DB2 z/os 20

21 DUPLICATE SOURCE DATABASE How does a clone of such an environment become consistent? ORA 1 ORA 1 ORA 2 ORA 2 DB2 LUW DB2 LUW DB2 z/os DB2 z/os START CLONE END CLONE 21

22 DUPLICATE SOURCE DATABASE How does a clone of such an environment become consistent? ORACLE AIX DB2 LINUX DB2 z/os START CLONE END CLONE 22

23 DUPLICATE SOURCE DATABASE How does a clone of such an environment become consistent? ORACLE AIX DB2 LINUX DB2 z/os START CLONE END CLONE 23

24 WORKFLOW SOURCE EXTRACT STRUCTURE DDL COPY DATA SYNCRONIZE COPY LOGS TARGET DROP DATABASE LOAD DATA CREATE DATABASE DROP DATABASE 24

25 BENEFITS Consistent preproduction Refreshing preproduction on demand Cloning without stopping the source Synchronized Cloning Base for other test environments Environment for system and release tests 25

26 COMPONENT OVERVIEW DATABASE CLONING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 26

27 Choose source and destination P PP T1 T2 D2 D1 27

28 Choose data to be copied Select schemas and/or tables Exclude rules 28

29 Specify instructions to rename objects Oname Oname SCHEMA SCHEMA Schema, owner, tables, table index', table space, distinct object 29

30 No Specify manual processing intervention rules No weekend work Options Create missing databases Create missing tables Treatment of structural deviations 30

31 XDM Table Copying - Features 31

32 FEATURES WORKFLOW BENEFITS Verification XDM TC 32

33 FEATURES WORKFLOW BENEFITS Verification DDL XDM TC 33

34 FEATURES WORKFLOW BENEFITS Verification DDL XDM TC Move data 34

35 FEATURES WORKFLOW BENEFITS Verification DDL XDM TC Fully automated scheduler enabled process Move data Make usable 35

36 XDM Table Copying - Workflow 36

37 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA 37

38 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Deploy selection/exclude rules Fetch source structures 38

39 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Fetch object information from target objects 39

40 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Compare structures and generate reports Generate DDL 40

41 FEATURES WORKFLOW BENEFITS ********************************************************************** * TABLESPACE COMPARISON REPORT ********************************************************************** Tablespace PROD.USERSPACE1 TMF25.USERSPACE1 Different + PAGESIZE Different EXTRACT DDL ********************************************************************* * TABLE COMPARISON REPORT ********************************************************************* Table PROD.JOBFLATFILES TMF25.JOBFLATFILES Equal Table PROD.HISTORY TMF25.HISTORY Different Column - NEW_COL Missing Table PROD.PRODUCTS TMF25.PRODUCTS Equal Table PROD.RESULT - Missing 41

42 FEATURES WORKFLOW BENEFITS Differences between DBMS': Partitioned on DB2 z/os Partitioned on DB2 LUW DDL syntax differs between platforms 42

43 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Execute DDL on target system 43

44 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Extracts data, considers stop mode Deploys multi-thread processing Uses the most efficient method to move data 44

45 FEATURES WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL LOAD DATA Load data to the target system Update indentity values Fix integrity state 45

46 XDM Table Copying - Benefits 46

47 FEATURES WORKFLOW BENEFITS Maximum reduction of manual work Scheduler based, automated process Platform independent Saves expert time Automatic restart of interupted copy tasks Notifications via Use of existing infrastructure for fastest copy method 47

48 COMPONENT OVERVIEW DATABASE CLONING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 48

49 Source Target Complex environments must be copied in development systems 49

50 Choose source and destination P PP T1 T2 D2 D1 50

51 Select data to be copied P1 C1 P2 C2 Where do the relations come come from? from? From DBMS CAT or imported or manually supplied Start-Table Select from table where condition Select from table where condition 51

52 No Specify manual integration intervention rules for target side No weekend work Options Create missing tables Replace target table(s)' content entirely Replace rows by keys Merge data Append rows Delete rows Embed data into new structure 52

53 The MERGE Option Production Test/ Development 53

54 Specify instructions to change data for Jim Jack Anonymization of personal data (privacy) 54

55 Specify instructions to change data for Date shifts to allow time journey tests 55

56 No Specify manual integration intervention rules for target side No weekend work Options Create missing tables Replace target table(s)' content entirely Replace rows by keys Merge data Append rows Delete rows Embed data into new structure 56

57 DELETE in target 57

58 No Specify manual integration intervention rules for target side No weekend work Options Create missing tables Replace target table(s)' content entirely Replace rows by keys Merge data Append rows Delete rows Embed data into new structure 58

59 Embed data into new structure Data in source system ID INTEGER NOT NULL FIRSTNAME VARCHAR(256) NOT NULL LASTNAME VARCHAR(256) NOT NULL Copied data in target system + additional data from second source ID INTEGER NOT NULL FIRSTNAME VARCHAR(256) NOT NULL LASTNAME VARCHAR(256) NOT NULL DATE_OF_BIRTH DATE NOT NULL _ADDRESS VARCHAR(256) NOT NULL 59

60 XDM Row Level Processing - Workflow 60

61 WORKFLOW BENEFITS SOURCE EXTRACT DDL EXTRACT DATA TARGET EXTRACT DDL COMPARE GENERATE EXECUTE DDL APPLY DATA Extract DDL and data from source Extract DDL and compare on target Execute DDL and populate data on target 61

62 XDM Row Level Processing - Benefits 62

63 WORKFLOW BENEFITS Select and copy specific test cases Detect and transfer all relevant RI from source to target Create an environment Control execution with a scheduler Command language Mask and modify data for target system 63

64 COMPONENT OVERVIEW OPTIONAL FEATURE DATABASE CLONING TABLE COPYING Row Level Processing XDM BASE 64

65 COMPONENT OVERVIEW OPTIONAL FEATURE DATABASE CLONING ICEBOX MASKING TABLE COPYING ROW LEVEL PROCESSING XDM BASE 65

66 MASKING ICEBOX Why is Masking necessary? Statutory provisions Data privacy Industrial espionage Credibility / reputation 66

67 MASKING ICEBOX What needs to be masked? Address Telephone number, fax , web Date of birth, age Occupation Gender Religion, ethnicity, etc 67

68 MASKING ICEBOX Methods Translation, Encryption Data scrambling Nulling, Truncating Delete values Substitution Replace entries Shuffling Exchange fields Smith 1gt5q Jonson dv41v4 Meyer Gurus Albright 7fws5jq5 Black Wf7zc 68

69 MASKING ICEBOX Methods Translation, Encryption Data scrambling Nulling, Truncating Delete values Substitution Replace entries Shuffling Exchange fields Smith *null Jonson *null Meyer *null Albright *null Black *null 69

70 MASKING ICEBOX Methods Translation, Encryption Data scrambling Nulling, Truncating Delete values Smith Jonson Meyer Albright Black Mouse Fox Cat Horse Dog Substitution Replace entries Shuffling Exchange fields 70

71 MASKING ICEBOX Methods Translation, Encryption Data scrambling Nulling, Truncating Delete values Substitution Replace entries Shuffling Exchange fields Smith Black Jonson Meyer Meyer Jonson Albright Black Smith 71

72 MASKING ICEBOX Complete Anonymization? Is anonymization secure? Intranet, Internet? Against whom, what for? 72

73 MASKING ICEBOX Quality Assurance needs: Same data as in production Preserve special cases Preserve sums Preserve diversification Preserve structural relations 73

74 MASKING ICEBOX Short summary: There is no correct or false anonymization Fuzzy method Depends strongly on special needs and the environment Risk / Effort / Cost / Effect 74

75 COMPONENT OVERVIEW OPTIONAL FEATURE DATABASE CLONING OPTIONAL MASKING ICEBOX FEATURE TABLE COPYING ROW LEVEL PROCESSING XDM BASE 75

76 MASKING ICEBOX Application Version 1 GIT / SVN / CVS ID FIRSTNAME LASTNAME INTEGER VARCHAR(256) VARCHAR(256) Test Case 0815 DB2 Oracle Application Version 2 ID INTEGER FIRSTNAME VARCHAR(256) LASTNAME VARCHAR(256) DATE_OF_BIRTH DATE _ADDRESS VARCHAR(256) GIT / SVN / CVS Test Case 0815 DB2 Oracle 76

77 MASKING ICEBOX Application Version 1 GIT / SVN / CVS ID FIRSTNAME LASTNAME INTEGER VARCHAR(256) VARCHAR(256) Test Case 0815 DB2 Oracle Application Version 2 GIT / SVN / CVS ID INTEGER FIRSTNAME VARCHAR(256) LASTNAME VARCHAR(256) DATE_OF_BIRTH DATE _ADDRESS VARCHAR(256) Test Case 0815 V1 V2 77

78 MASKING ICEBOX Make Tests Reproducible 78

79 MASKING ICEBOX The Dilemma Test Env.1 Modifies data during test execution Test Case 0815 Tests should run under same preconditions Tests modify the data 79

80 MASKING ICEBOX FROZEN TEST DATA Test Env.1 BACKUP BEFORE TEST EXECUTION REGULAR 80

81 MASKING ICEBOX FROZEN TEST DATA Test Env.1 AFTER EACH TEST EXECUTION REGULAR ON DEMAND Test Env.2 81

82 MASKING ICEBOX Preserve Test Case Data 82

83 MASKING ICEBOX 83

84 MASKING ICEBOX PRODUCTION Refresh from production drops all valuable test data 84

85 MASKING ICEBOX FROZEN TEST DATA BACKUP RELEVANT TEST DATA 85

86 MASKING ICEBOX FROZEN TEST DATA PRODUCTION BACKUP RELEVANT TEST DATA 86

87 MASKING ICEBOX FROZEN TEST DATA PRODUCTION BACKUP RELEVANT TEST DATA 87

88 MASKING ICEBOX FROZEN TEST DATA RESTORE TEST DATA BACKUP RELEVANT TEST DATA 88

89 You are keen to see what you can accomplish with XDM in your installation? 89

90 Order a free trial and find out how much valuable expert time you can save. 90

91 Thank you for your attention! Your UBS Hainer Team 91