Tracking and defending changes in an ESRI Geodatabase

Transcription

1 Tracking and defending changes in an ESRI Geodatabase 2017 ESRI Petroleum User Group (PUG) Conference Jeff Wilcox Kinder Morgan Peter Veenstra - TRC

2 Introduction and Overview Introductions Jeff Wilcox Kinder Morgan Pete Veenstra - TRC Problem Statement Technical overview of: the different strategies, the pros/cons what has worked or not worked in our experience tracking edit changes and history

3 Straw Poll Who is managing production data in a Geodatabase? Who is utilizing a versioned Geodatabase? Who is tracking history and recording/archiving changes? What methods are you utilizing to track history/changes? Geodatabase Archiving? In-line history? Code-based method? Triggers on compress? Database back-up and restore? Other?

4 Problem Statement How can I track the history of edits in my Geodatabase? Why do I care to track the history of edits in my Geodatabase? Forensics - Who did the edit, what did they do, how did it happen and why was it done? Quality Security Governance Defensibility Audit Defensibility (recall the state of condition of the geodatabase as a particular time Caveats and Assumptions Versioning is the most powerful function of the Geodatabase We are not going to discuss viewing history but it needs to work with the ESRI Time Slider

5 Approach Methods In-line History/Auditing Fields Geodatabase Archiving Trapping Changes before POST RDBMS Triggers set at Compress Database Back-ups Review Criteria How database deltas are stored and how data are retrieved. Impact on database performance Pros and Cons

6 Anatomy of a Geodatabase RDBMS SDE Software Base Tables Register as Versioned A and D tables Compress Server RDBMS SOFTWARE Base Tables ArcGIS Server Software (SDE) A&D Tables

7 What is a Version? A version is a wrapper within a Geodatabase that tracks a series of edits (add, delete, update) against data in a geodatabase over a period of time Called a Long Transaction A user creates a version from another version in the geodatabase When a table or feature class is registered as versioned a set of A and D tables are created for each table (that is versioned, in the geodatabase) As a user adds data, the changes between the parent version and the child version are stored in the A tables per table or feature class per version. As a user deletes data, the changes between the parent version and the child version are stored in the D tables per table or feature class per version. As a user updates table, first a delete is performed (stored in the D table) and then an add is performed (stored in the A table).

8 Reconcile, Post and Compress When a user has completed the transaction they reconcile the changes in the child version against that of the parent version Reconcile checks the records in the parent version to see if any changes have been performed on those records that conflict with any changes made in the child version If the parent record is different then the original parent record and the child records a different version of that truth then a conflict occurs Conflicts can be resolved in favor of the parent or the child. Once conflicts are resolved then the changes in the child are written or posted to the parent in an edit session. Once the changes are posted then the child version can be deleted.

9 Example of Versioning Resolution Conflicts! DEFAULT Reconcile Post Reconcile... Post... CHILD V1 CHILD V2 Reconcile Post Reconcile Post CHILD V1 CHILD V2_A

10 ESRI Versioning Documentation Versioning Versioning Workflows

11 Post to DEFAULT and COMPRESS Post to DEFAULT triggers Archiving if it is enabled After Post to DEFAULT the changes are not committed to the base tables until a COMPRESS is affected COMPRESS cleans up the database It tidy s up the A and D tables and any unneeded rows within them And it performs a RDBMS level write or commit the base tables Since this is an RDBMS function it invokes the standard RDBMS events <before/on/after> <add/delete> An update trigger never fires. ESRI actually treats updates as a delete and an insert. A trigger is a RDBMS block of code that can be written on these events to execute additional structured query language (SQL) statements Post to DEFAULT and Compress can occur with outstanding versions remaining. Reconciling all version prior to compress will help tighten up the A and D tables. Create Version, Delete Version, Reconcile, Post and Compress can be scripted using ArcToolbox tools including functions for conflict resolution

12 In-line History/Auditing Fields How database deltas are stored and how data are retrieved. Edits are tracked using fields within each schema Created By, Last Modified, EffectiveFromDate, EffectiveToDate, HistoricState, ParentEventID A database add will: Insert a new record populating CreatedBy, LastModified, EffectiveFromDate with the current editor, current date/time, HistoricState is set to Current or Active No database deletes are performed A database update will: Updates to LastModify to the current user and EffectiveToDate to the current date/time and HistoricState is set to Historic or In-active Insert a new record populating LastModified, EffectiveFromDate with the current editor, current date/time, HistoricState is set to Current or Active A database retire or replacement with a more current record will: Set the LastModified, EffectiveToDate to the current date/time and HistoricState is set to Historic or In-active, ParentEventID is set to the EventID of the retired feature.

13 In-line History/Auditing Fields Pros Simple approach easy to understand logic and process Uses single table, no need for duplicate schema Built in ESRI Editor Tracking functionality (Does not cover all of logic) Showing the state of the data at a specific point in time involves a query Showing or tracing the provenance or origin of a feature is complicated This the approach supported by ESRI ArcGIS for Pipeline Referencing to enable time-slide functionality within ArcGIS Pro (slide the bar over time and see the changes) Cons Requires user to understand process and pay attention to detail A replacement may only involve a spatial part of a parent feature requiring logic to manage the resultant parts Users and logic must be used to query for current records only Does NOT track/store SHAPE at time of change Impact on database performance Database table size increases as more edits are added Process itself does not affect performance (no more than regular batch edits)

14 Geodatabase Archiving How database deltas are stored and how data are retrieved. Native ESRI Geodatabase technology Archiving is turned on similar to versioning, edits (adds, deletes) are preserved in a hidden mirror schema when user posts to defaults Time-stamp fields are added and applied Tracks the differences to the DEFAULT version over time ESRI Tools manage the fields for tracking FromDate and ToDate allowing data to be immediately filtered ESRI allows users to view the state of data between two dates when adding a layer or data to a map DEFAULT keeps only the most current and active data

15 Geodatabase Archiving Connect to a Transactional Version View the version in ArcMap (Geodatabase History Toolbar)

16 Geodatabase Archiving Version is created Edits are performed Post to DEFAULT Change is written to ARCHIVE

17 Geodatabase Archiving Pros Native to the Geodatabase and ArcGIS Desktop (and Pro?) software Can be scripted Easy to use tools for retrieving and viewing versions Tracks/Records SHAPE at time of change Cons Effectively duplicates the schema (although hidden through the standard ESRI interfaces) Not overly adaptable to schema changes (the mysterious _H tables) If a data model is updated (PODS to UPDM or APDM 4.0 to APDM 6.0) the archive must be migrated Difficult to capture or trap POST to DEFAULT event (Geodatabase/ArcObjects issue) Works on Enterprise ArcGIS Server (SDE) instances only Impact on database performance A large POST may slow the database down but none-appreciably

18 Trapping Changes before POST How database deltas are stored and how data are retrieved. Custom Written application Tracks changes between two versions Tracks changes in what-ever tables are specifieddelta (add, update, delete) is written to a log file (CSV, XML, JSON) Log-file is archived and/or posted to another process and/or written to a database Very powerful construct for notifying other systems of changes If DEFAULT is protected then a check can be put into place to verify that delta have been posted and changes QA/QC ed Becomes part of workflow

19 Trapping changes before POST Version is created Edits are performed Changes are recorded Edits posted to DEFAULT

20 Trapping changes before POST Pros Isolates delta between any two versions across all tables Allows for QA/QC on changes only Provides entry into database synchronization with other systems Provides notification and entry into other workflows Can be conducted before or after RECONCILE (makes more sense after ) Can be configured within a larger workflow Cons Required ArcObject coding Requires manual operation by end user Does NOT track/record SHAPE at time of change. Captures spatial edits but does not add them to the log file Impact on database performance Massive edits to the database across many will slow down the application

21 RDBMS Triggers at Compress How database deltas are stored and how data are retrieved. When the Geodatabase is compressed a trigger after-add or after-delete execute The RDBMS trigger writes the new and deleted records to a duplicate table to indicate changes Additional POST process can create SHAPE and store it Version is created Edits are performed Edits posted to DEFAULT COMPRESS is performed Triggers fire at write to BASE Table

22 RDBMS Triggers at Compress Pros Captures all changes (even Spatial) Occurs automatically, not manual process required Utilizes database triggers, which in turn and kick off other workflows Allows for complete database forensic Allows the addition of other fields that can be populated with logic. Allows storing shape on event based features where the original event table did not have shape (do this as a post process) When it was compressed and who compressed it. Cons Each trigger is written manually for a specific table (could be advantage) or could be written as a procedure that adds triggers to any table at will. Requires duplicate schema Impact on database performance The impact is minimal as the triggers only fire at compress which could be ran during non-peak hours.

23 Database Back-up and Restore How database deltas are stored and how data are retrieved. Database is backed up on a nightly basis Database is archived or replicated in real-time

24 Database Back-up and Restore Pros Uses RDBMS Technology Cons Only as good as your last backup latency Only as good as your restore protocol Size requirements for managing backups how far back to you go? Impact on database performance None

25 What if you do not use Versioning? Native RDBMS software have built-in archiving and record tracking Most use a method of duplicate or archive schema Not using versioning offers no more protection than regular database back-ups

26 Measure by Performance None of the methods other than in-line history affect GDB performance because the work can be scheduled. Any table or FeatureClass that has a FromDate/ToDate can be used with the ESRI historical time slider

27 Conclusions and Summary Triggers on the COMPRESS event write to base tables provides the best approach for tracking history Allows for customized archive of the feature or record Stores the shape change for events (not treated as records) Allows for the notification of other systems or the triggering of other processes No approach is perfect Tracking history and changes provides governance, forensic and audit defensibility

28 Thank You Jeff Wilcox Peter Veenstra