Change Data Capture OpenEdge 11.7

Transcription

1 Change Data Capture OpenEdge 11.7 Workshop and Discussion Brian Bowman Product Management, OpenEdge November 2017

2 Login Information On your page is the Remote Desktop Login Login: \Administrator Password: EMEAPug2017 OE Management Login: admin OE Management Password: examplecdc 2

3 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL User ID Cleanup and Management Additional topics 3

4 LAB Agenda Configuring DB for CDC OEM Command Line Working with CDC Policies Setting up a Policy Enabling a Policy Policy Granularity Modifying a Policy generating changes Viewing changes Working with CDC in ETL Accessing captured data ABL Code SQL Code 4

5 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 5

6 Different product scenarios OpenEdge Workgroup RDBMS OpenEdge CDC for Workgroup OpenEdge Explorer < During install checked on by default > OpenEdge Enterprise RDBMS OpenEdge CDC for Enterprise OpenEdge Management OpenEdge Advanced Enterprise RDBMS 6

7 Dictionary of Terms Change Data Capture (CDC) An industry term that describes the duplication (capture) of subsets of OLTP data Captured data is a relatively up to date version of the relational data ETL Extract, Transform, Load The process of taking changes in a database, moving them out of the database, changing them (including aggregation) and placing them somewhere else. Policy Defining the rules around what is captured, how it is captured and where it is captured 7

8 Introduction - Change Data Capture Why do we need to capture changed data? Big data Data Warehousing Consolidated Repository Drives strategic business decisions Replication of all data to Data warehouse Time consuming data is irrelevant to business needs. A tailored approach is needed ABL or SQL One Solution needed! Data sources Flat Files OLTP Source tables OLAP analysis Data Warehouse Reporting Data Mining 8

9 OpenEdge Change Data Capture Change Data Capture Capture of changed data from OLTP actions Data captured can be stored Optional ETL (Extract Transform-Load) tool can transform captured data further. Data source OLTP Source Tables ETL Data Warehouse Capture Alternatives Delta file Transaction log Replication Trigger based Capture process Changed Data Process OLAP analysis Reporting Data Mining 9

10 OpenEdge Change Data Capture Internal trigger based capture Captures Create/Update/Delete record operations Stores changed data in relational form Defined through policies Change Tracking and Change Data capture Extract, Transform, Load (ETL) SQL and ABL. Record operation OLTP Source Tables CDC Database triggers Change Tables 10

11 OpenEdge Change Data Capture - Overview Enable CDC on source table Source Table (Customer) User Data Record operations: create, update, delete (CUD) _Cdc-Table-Policy, _Cdc-Field-Policy CDC Policies Cache CDC Internal Database Triggers Change Tracking Table (_Cdc-Change-Tracking) Change Table (Customer) CDC_Customer(captured data) 11

12 OpenEdge Tables Source Table (Customer) User Data Source tables: OpenEdge database user tables Identified in a CDC policy as the source captured data Source tables exist prior to implementing a CDC policy Source tables are not changed when the CDC policy is implemented 12

13 Policy Tables CDC Policy Tables: _Cdc-Table-Policy, _Cdc-Field-Policy Policy tables are created when CDC is enabled on an OpenEdge database CDC Table Policy Table _Cdc-Table-Policy Contains one record for each policy defined on a source table There may be multiple policies defined for a source table (pending, current, previous) Relationship: A table policy record is parent of zero or more field policy records CDC Field Policy Table _Cdc-Field-Policy Contains one record for each tracked field of a CDC table policy Note if a table policy is set to level 0 there are no field policy records 13

14 Change Tracking Table Change Tracking Table (_Cdc-Change-Tracking) _Policy-Id _Tran-id _Time-Stamp _Change-Seq _Op CDC Change Tracking Table : _Cdc-Change-Tracking Table created when CDC is enabled on the database Contains a record for each CDC tracked operation on a source table Operations are captured for the current policy on the source table when that policy is active 14

15 CDC Change Tables Change Table Change Table (CDC_Customer) Meta-data Columns User Data (captured data) CDC Change Tables : Contains change data as prescribed in the CDC policy tables Includes both meta data and user data (which are identified in the CDC policy table) A CDC change table is created the first time a policy with level 1 or higher is defined There is at most one CDC Change Table for a CDC source table May be associated with different CDC policies over the course of its lifetime 15

16 Change Tables Meta-data columns: Identification Sorting Split record Meta-data Columns This example shows a change table named CDC_Order User data columns: Fields from source table identified in the CDC policy User Data Columns 16

17 Execution Process for ABL Statement Start Transaction Map Policies to Source Table Record Buffer Created Original Data Capture DB Engine Trigger Execution Record Created Apply CDC Policy Write Change Tracking Record End Transaction Commit All Records Write Record To Change Table 17

18 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 18

19 Configuring Change Data Capture Configuring CDC is done by creating CDC policies Three ways to configure CDC: 1. OpenEdge Management 2. OpenEdge Explorer 3. Change Data Capture ABL API Support for CDC is identical in OpenEdge Management and OpenEdge Explorer Scripting happens through ABL that make calls to the CDC ABL API 19

20 CDC Database Feature Database Administration Console contains support for Change Data Capture Select the console by clicking the Database Administration 20

21 Enabling Change Data Capture 21

22 Enable CDC On-line and Off-line Change Data Capture can be enabled whether the database is on-line or off-line When the database is off-line the Area selection lookup is not available When working off-line type the names existing type II database areas for both the Data area and the Index area The search option is not available when the database is off-line 22

23 Lab One Configuring DB for CDC 20 Minutes NOTE: Structure file may not match the picture

24 Questions

25 Questions What are 4 ways to enable CDC?

26 Questions What are 4 ways to enable CDC? Is CDC running?

27 Questions What are 4 ways to enable CDC? Is CDC running? What is being captured?

28 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 28

29 Create a Change Data Capture Policy 29

30 Generating a Policy Program 30

31 Creating a Policy Policy Names must be unique Table based Every table has one policy active 31

32 Creating a Policy When the policy Level set to 1, 2 or 3 Additional fields and selections are required 32

33 Identifying Fields for a CDC Policy Defines how you want to identify and access data Up to 15 Identifying fields can be selected 33

34 Selecting Identifying Fieldes 34

35 Policy Instances and Policy States Pending (Inactive) Create new Policy with Inactive state Current (Inactive) Inactive Active Current (Active) Create new Policy with Active state Previous (Inactive) Policy Instances: Pending policy (2) Current policy (0) Previous policy (1) Policy States: Active Inactive 35

36 Lab Two Working with CDC Policies 25 minutes

37 Questions

38 Questions How can I enable CDC while the database is down?

39 Questions How can I enable CDC while the database is down? What are the 4 policy states?

40 Questions How can I enable CDC while the database is down? What are the 4 policy states? What is the extension of the CDC export policy?

41 Table Policy Levels Minimal (0) A Minimal with Field Map (1) is NOT created and change table records are not written Same as Minimal, but includes a field map of changed fields (_Change-FieldMap ) A Change Table IS created, but change table records are not written Medium (2) Create a Change Table Change Table record with the current values Maximum (3) Create a change table record with the current values (same as Medium) For updates, the previous values and current values are recorded Creating two Change Table records 42

42 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 43

43 CDC Support in Promon If cache Misses increases periodically, increase the size of CDC cache to improve performance. 44

44 New Virtual System Table (VST) for Shared Caches There is a new Virtual System Table (VST) for shared caches Users can use this table to monitor the status and activities of the caches The new VST is named _Cache To use the new VST, users must update the database to have the latest VST definitions: proutil <dbname> -C updatevst Like most VST tables, updates on _Cache records are not allowed This is the case since most of the fields contain system information that cannot be altered 45

45 VST _Cache Table Definition Field Name Data Type Initial Value Format Description _Cache-Id Integer 0 >>> ID of the cache. Each supported cache has a different ID. _Cache-Name Char X(16) Name of the cache. Each supported cache has a different name _Cache-Size Integer 0 >>>>>>> Size of the cache in kbytes _Cache-Entries Integer 0 >>>>> Number of entries stored in the cache _Cache-Hash Integer 0 >>>>> Hash table size or hash prime of the cache _Cache-Hits Int64 0 >>>>>>>>>>> Total of cache hits since the last reset _Cache-Misses Int64 0 >>>>>>>>>>> Total of cache misses since the last reset 46

46 Indexes on the Change Tracking Table Index Name Index Field(s) Attribute _Sequence-Id 1 _Source-Table-Number 2 _Change-Sequence (These two fields together point to a single instance of the table) Unique Primary _Time-Stamp-Seq 1 _Source-Table-Number 2 _Time-Stamp 3 _Change-Sequence Non-Unique _Part-Rec-Id 1 _Source-Table-Number 2 _Partition-Id 3 _Recid 4 _Change-Sequence Unique 47

47 Indexes on Change Tables Index Name Index Field(s) Attribute _Change-Sequence-Id _Identifying-Fields 1 _Change-Sequence 2 _Operation 3 _Continuation 1 15 User Defined Columns (15 maximum) 2 _Change-Sequence (Last column always added) Unique Primary Non-Unique _Time-Sequence 1 _Time-Stamp 2 _Change-Sequence Non-Unique 48

48 Lab Three Creating CDC Policies with varying granularity 20 minutes

49 Questions

50 Questions When do you generate a Change Table record?

51 Questions When do you generate a Change Table record? How many indexes are there on the Change Table?

52 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 53

53 ETL Coding Considerations The tables that are accessed are the Change Tracking Table and CDC change tables There is a relationship between the records in these two tables Each change tracking table record has zero, one, or two associated change table records depending on the Level of the CDC policy Relationship between these two tables: The type of change activity (operation) is in the Change Tracking Table The data resulting from the operation is in the change table, plus some meta data Efficient access of change data requires using the correct index based on the needs of the application When an ETL program reads the data it often needs to read the data in the order it occurred Order by the sequence of entry, from first to last 54

54 ABL Language Enhancements for CDC Some ABL classes and enumerations have been added to support ABL developers writing ETL programs Classes can be used to extract data from the CDC Change Tracking and from CDC change tables The extracted data is processed and copied to an external data source The classes: 1. Make it easy to work with the field map in column _Change-FieldMap 2. Provide support for rejoining split records There is no requirement to use these classes to process CDC change data These classes are provided as a convenience functionality that users might not require 55

55 Working with the Field Map When a source table is modified: An internal database CDC trigger writes a row to the CDC Change Tracking Table This includes writing the field map to the Change FieldMap column (_Change-FieldMap) Each field map reflects the fields from the change table that changed The _Change-FieldMap is written only when the policy Level is greater than 0 Functionality is provided to: Get a list of the fields that changed in the row based on the _Change-FieldMap column Determine whether a particular field changed in the row based on the _Change-FieldMap 56

56 Using SQL Scalar Functions to Identify and Access Modified Fields The _Change-FieldMap field in the Change Tracking record is used to indicate which fields in the source table will be captured _Change-FieldMap (type VARBINARY) On (1) indicates that the field is defined in the policy, and that the data is captured to the change table when the level is 2 or 3. Off (0) means that the field is not identified in the policy, and data for that field is never captured. OpenEdge SQL provides two scalar functions to aid in interpreting _Change-FieldMap CDC_is_column_changed CDC_get_changed_columns 57

57 Return if a Column Was Modified CDC_is_column_changed (Change table name, column name, _Change-FieldMap) Scalar function CDC_is_column_changed is called to return whether or not a the value for a column has been modified Returns: 1 - the field value changed 0 - field value did NOT changed Examples: Change table pub.cdc_customer; fields CustNum, Name, Address, State, Phone Change Tracking Table field _Change-FieldMap contains Returns value 1 CDC_is_column_changed (pub.cdc_customer, Address, _ Change-FieldMap ) Returns value 0 CDC_is_column_changed (pub.cdc_customer, State, _ Change-FieldMap ) 58

58 Return Changed Columns CDC_get_changed_columns (change table name, _Change-FieldMap) Scalar function CDC_get_changed_columns returns the modified columns based on the information stored in the _Change-FieldMap field Returns an array of field names For each bit having value 1 in _Change-FieldMap the corresponding field name is added to the array Examples: Change table pub.cdc_customer; fields CustNum, Name, Address, State, Phone Change Tracking Table field _Change-FieldMap contains Returns string value Address, Phone CDC_is_column_changed (pub.cdc_customer, _ Change-FieldMap ) 59

59 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 60

60 Split Records In rare cases, CDC change table records can be split across two change table records This is done to avoid exceeding the maximum OpenEdge Database record size The maximum record size is bytes It is very rare for records to be near the limit To exceed this limit most if not all fields would be captured The addition of the meta-data columns would need to push the size over this limit OpenEdge 11.7 includes a CDC ABL class that rejoins split records into a single record 61

61 Will Split Records Occur? Calling the method to join split records each time a record is read can impact performance It is very rare for database records to contain records that are near the maximum database size A DBA can evaluate the database to determine if any records are near the limit The analysis should be done on fields that will be included in the CDC policy, instead of all fields If it is known that CDC change table records will never be split: There is no need to call the method for joining split records This maximizes performance of reading the data If it is expected that splitting of records could occur: The method to join split records should be called to avoid any loss of data Call the join method only for changed tables that have been identified in the analysis 62

62 Split Records in a Change Table Meta-data Columns User Data Columns _TranId Timestamp _Change- Sequence _Continuation- Position _ArrayIndex _Fragment _Operation Fld1 Fld2 The maximum record size in an OpenEdge database is 32,000 bytes When the size exceeds the maximum, the OpenEdge database automatically and transparently creates two records to hold the data for one database change Split records are also known as continuation records 63

63 Misc. Items User Miscellaneous Field User ID 64

64 CDC and other RDBMS Features - Auditing Both Auditing and CDC can be configured for the same data tables Auditing of CDC data tables is not supported CDC of audit policy tables is not supported 65

65 CDC and other RDBMS Features - Encryption OpenEdge TDE encryption can occur on the area or table level Source table being considered for CDC has ether been encrypted or it has not Case 1 Table is already encrypted: When a CDC policy is created for the Table to insure the security of the data, the policy creator should mark the _Encrypted-Policy field on This ensures that new change table records remain encrypted at the same level as the source table Case 2 Table is not encrypted: Encrypt the source table before creating the CDC Table Policy Remember to set the _Encrypted-Policy field to true to provide seamless security of your data. 66

66 CDC and other RDBMS Features - Replication Step 1: When a database is already setup for Replication, all that is necessary to enable the CDC feature is first making sure the areas you will designate as in USE by CDC exist on both databases Step 2: Enable CDC on the source database As the replication process occurs from the source to the target, the enablement will occur on the target through AI replicated notes Step 3: Table/Field Policies are created on source database and again through the replication process created on the target database All CDC specific activities driven by database utilities, ABL and SQL are available ONLY on the source replication database 67

67 CDC and other RDBMS Features - MT, TP and Security Multi-tenancy CDC and Multi-tenancy coexist in a database without issue. Table partitioning CDC and Table Partitioning coexist in a database without issue. Security Security roles for CDC are not necessary and are not provided However, if the Change Tracking Table and Change table(s) need additional security, you can add CAN constraints to these tables 68

68 Lab Four Working with CDC in ETL (ABL and SQL) 20 minutes

69 Questions

70 Questions When is the CDC Cache Used?

71 Questions When is the CDC Cache Used? Name three tools you can you use for ETL with CDC?

72 Questions When is the CDC Cache Used? Name three tools you can you use for ETL with CDC? What ETL tools will you use?

73 Documentation on CDC OpenEdge RDBMS OpenEdge Getting Started: Change Data Capture OpenEdge Data Management: Database Administration OpenEdge Management / OpenEdge Explorer OpenEdge Management: Managing Change Data Capture in Databases & Online help OpenEdge SQL OpenEdge Data Management: SQL Development OpenEdge Data Management: SQL Reference OpenEdge ABL OpenEdge Development: Programming Interfaces (Appendix C) Data Administration, Data Dictionary Online help & OpenEdge Development: Basic Database Tools 74

74 On-line OpenEdge Documentation OpenEdge Product Documentation on-line links: Most recent (current) release: OpenEdge documentation sets for the current and previous releases: openedgegeneral/1329.openedge-product-documentation-overview 75

75

76 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 77

77 Renaming Fields in a CDC Source Table Renaming a field in a source table triggers a corresponding rename of same field in a change table For example if field Location is renamed to HomeLocation in the source table, it is automatically renamed to the same name in the change table 78

78 Purging Data From the Change Tracking Table and From Change Tables Purging CDC data: Data in CDC tables grows based on the amount and type of activity in the source tables DBAs monitor the growth of these tables and periodically purge data to recover space Tables that need to be monitored are The Change Tracking Table Each of the change tables One method to control the growth of change data is to purge it as it is extracted This works in environments where captured data is no longer needed after it has been extracted 79

79 Deactivating and Deleting CDC Policies Deactivating CDC policies: CDC policies should be deactivated when newly captured data is no longer needed Deactivating the policy stops the capture of data and frees up resources CDC policies can be deactivated using the Database Administration Console Set the policy status for the current policy to inactive to deactivate the policy The policy remains available, but will not capture data unless it is reactivated Deleting CDC policies: CDC policies with a policy state of previous are candidates for deletion Other policies can also be deleted if they will not be needed in the future 80

80 Policy Maintenance Done Offline There are a few cases where CDC policy maintenance requires single user (offline) access Offline means no-one else can be running ABL or SQL against the database Examples of policy maintenance that require single user access: Deleting a current CDC policy with Level greater that zero This requires deleting the change table, and deleting a table is an offline operation Deleting the current CDC policy on a table is done when changes are no longer being tracked Deactivating indices on the CDC tables For performance reasons a DBA may choose to deactivate indexes on CDC tables For change tables, the _Identifying-Fields index can be deactivated For the Change Tracking Table, all but the primary index can be deactivated. 81

81 Deleting CDC Policies CDC policies with a policy state of previous are candidates for deletion Other policies can also be deleted if they will not be needed in the future To delete policies developers write a program using ABL or SQL 82

82 Details on Deleting a CDC Policy Deleting a CDC policy (_Cdc-Table-Policy) is allowed: Only if there are no Change Tracking (_Cdc-Change-Tracking) rows for the policy Only when there are no rows in the change table (Policy Level is greater than 0) Deleting a CDC policy (_Cdc-Table-Policy) row also: Deletes the change table for that policy if one exists Removes Field Policy (_Cdc-Field-Policy) rows for that policy (Policy Level is greater than 0) Deleting a _Cdc-Table_Policy has no impact on the _Cdc-Change- Tracking table However, as noted earlier, all records in the Change Tracking table for this policy must be deleted prior to deleting the CDC policy 83

83 Agenda Introduction to CDC Configuring CDC Policies for CDC What now? ETL Cleanup and Management Additional topics 84

84

85 Code Example May Be Split Records 86

86 Coding Example No Split Records 87

87 Example 01 - Finding Changed Data References select ct.* -- specify change tracking fields here from pub."_file" f inner join pub."_cdc-change-tracking" ct on f."_file-number" = ct."_source-table-number" where f."_file-name" = customer and f."_owner" = pub order by ct."_source-table-number", ct." _Change-Sequence"; -- sort keys map to index _Sequence-Id 88

88 Example 02 - Finding Changed Data References select ct.cust-num, ct.name, ct.address, ct.state, ct.phone from pub."_file" f inner join pub."_cdc-table-policy" tp on f.rowid = tp."_source-file-recid" and tp."_policy-instance" = 0 -- Policy is current inner join pub."_cdc-change-tracking" ct on f."_file-number" = ct."_source-table-number" and tp."_policy-id" = ct."_policy-id" where f."_file-name" =? -- table name parameter from ETL app and f."_owner" =? -- table owner parameter from ETL app order by ct."_source-table-number", ct." _Change-Sequence"; 89