Overcoming Challenges While Migrating SDE Versions

Transcription

1 Copyright 2016, Saudi Aramco. All rights reserved. Overcoming Challenges While Migrating SDE Versions Jameel Al-Khalaf ESRI User Conference June 28, 2016

2 Agenda Project Overview Project Objectives Migration Methods Challenges Benefits Lessons Learned 2

3 Why Did We Initiate This Project? Meeting IT Demand 3

4 Project Objectives System Upgrade Hardware & Software Raster & Vector Data Migration 4

5 System Upgrade Source Geodatabase (Our Data Center) Target Geodatabase (IT Data Center) Cluster_Server_1 Cluster_Server_2 Cluster_Server_1 Cluster_Server_2 ArcSDE ArcSDE ArcSDE 10.2 ArcSDE 10.2 Database Database Upgrading H/W & S/W Database Database Server 1 Server 2 Upgraded Server 1 Upgraded Server 2 Old Cluster Environment New Cluster Environment SAN 7 TB SAN 15 TB 5

6 Data Migration Source Geodatabase (Our Data Center) Target Geodatabase (IT Data Center) Cluster_Server_1 Cluster_Server_2 Cluster_Server_1 Cluster_Server_2 SDE SDE 1 ArcSDE ArcSDE ArcSDE ArcSDE 10.2 QC QC QC QC QC QC Database Database Raster & Vector (SDE Versions) Database Database Server 1 Server 2 Upgraded Server 1 Upgraded Server 2 Old Cluster Environment New Cluster Environment SAN 7 TB SAN 15 TB 6

7 Clearing The Path S D E V e r s i o n s M i g r a t i o n P r o j e c t U p g r a d e D e p e n d e n c e s U p g r a d e & D a t a M i g r a t i o n Updating TNS File Upgrading Clients App. Compatibility System Upgrade SDE Versions Migration Raster 7

8 Data Migration Methods No ESRI Documented Oracle Procedure FGDB Aramco Compress GDB to trim state tree back to zero Back up the entire DB & restore it on the new server Works only at one version below SDE Develop tool to Export all SDE versions & Import it on the target 8

9 Features of Aramco Versions Migration Tool Listing FCs feature count Creating Versions Exporting DSs/FCs/Tables Listing Versions Listing Versions changes count Exporting output logfile Exporting Deltas Importing Deltas 9

10 1. Listing Versions Export Datasets_DataOwner 3. Data Statistics_DataOwner 4. Listing Versions Changes Count 5. Create Version on Target DB 6. Exporting Versions Deltas

11 IT Saudi Aramco: Company General Use Technical Challenges Lack of Admin Privilege (TNS) DownloadCenter (ArcGIS & Ext.) 11

12 Download Center 12

13 IT Saudi Aramco: Company General Use Technical Challenges Lack of Admin Privilege (TNS) New Tool (Lots of Issues) DownloadCenter (ArcGIS & Ext.) 13

14 Aramco Geodatabase Versioning Tree 14

15 Exporting Main SDE Versions Version Name 4. Listing Versions changes 5. Create Version on 6. Exporting Versions count Target Deltas DCs_Duba_Corr 0:05:37 Created 00:09:24 DCS_NorthWest_PipeLine 0:08:39 Created 00:10:08 DCS_Shaybah Wildlife 0:00:37 Created 0:04:45 EWG-1 PL Corr 0:05:25 Created 0:12:04 Ghinah_Reservation 00:13:44 Created Harmaliyah_Reservation 00:05:11 Created 00:08:00 Jaladi_Reservation 01:15:59 Created Jauf_Reservation 00:17:17 Created Jizan_Refinery 00:03:17 Created 00:03:27 KAPSARC 00:04:19 Created 00:04:29 MNFG-1 00:08:09 Created Riyadh Refinery Mapping 00:14:43 Created Wasea_Bulk_Plant 00:12:35 Created 15

16 IT Saudi Aramco: Company General Use Technical Challenges Lack of Admin Privilege (TNS) New Tool (Lots of Issues) DownloadCenter (ArcGIS & Ext.) Finding Alternative Solutions (Free Tools) 16

17 In-House Tools & Apps Section Tools Source Action / Alternative Solution LSS G&PSS GSITU LSS Tools (As-Built, Proposed Pipeline, RTP) Extraction (GDB to DGN) Developed In-House WDMS Recompile for ArcGIS 10.2 EditorID Developed In-House Z-Management 3D Snapping 3D Snapping Free ESRI Production Mapping Extension PLTS ESRI Extension ESRI Data Reviewer Extension Raster Clipping Free ESRI Clip Data Management Tool Text2GDB In House Developed TBC/new developed procedure Geodetic Manager Developed In-House Compatible with ArcGIS 10.2 Adding Ortho SWO Metadata ASCOT Loader GDB Data Refresh SSD Data Synchronization Tools Developed In-House Recompile for ArcGIS 10.2 Aerial Photo Publisher Automate Reconcile & Post Job 17

18 Hardware Software Saudi Aramco: Company General Use Data Project s Benefits Better Performance Increased storage capacity Bug-Fixes Cope with new Technology Enhanced editing functionalities House Cleanup (Versions) Fixed R_Pipeline Dataset New geoprocessing tools Upgrade legacy tools 18

19 Lessons Learned 1. Corporate IT 2. Capitalize on the Upgrade Opportunity 3. Creating Checklists 4. Practice makes perfection 19

20 Lessons Learned [ 1 ] Corporate IT - Identify the group/individual - Meet ahead of time & make any necessary requests - Lookout for any future IT policies/procedures to be implemented 20

21 Lessons Learned [ 2 ] Capitalize on the Upgrade Opportunity - Get rid of free/unsupported tools & use maintained S/W - Standardized our installation process (ArcGIS & TNSNames.ora) - Updated our inventory lists (users, workstations, S/W licenses) 21

22 Lessons Learned [ 3 ] Creating Checklists - Important to keep track of all implementation procedures and details Example: Data Migration Checklist 22

23 DATA MIGRATION CHECKLIST Preparation: 1 Re-compiling G&PSS and LSS existing 9.3 tools 2 Availabily of ESRI Extensions 3 15 Installing Disconnect ArcGIS all users on all G&PSS and LSS client machines 4 16 Availability Make a of backup SDE Media, of SDE SDE (Using License, XML recordset) new cluster environment 5 17 Develop Analyze the (Versions to update Migration GDB statistics Tool) to Read & Write other versions 6 18 List of Compress 20 all headcounts Prepare the GDB the (Schemas server to (Operating be migrated, System, FC, Database, FDS, Versions) ArcSDE ) 7 19 Prepare Export 21 MXD Pipeline Copy for QCing Database dataset migrated Users, as XML data Roles, for each Profiles, version Tablespaces from Source to Target 8 Create 22 User.Test Verify Version the count Under of Default imported on (Users, Source Roles, and make Profiles, few changes Tablespaces) (Add, Modify a name) 9 Create 23 MyUser.JDK Copy 37 the Version Verify DEFAULT the Under head version Default count ( Source on of Source (Feature ) schema and Classes, make (Excluding few Datasets, changes Pipeline Versions, (Add, DS) Modify delta changes) a name) 10 Synch 24 the databases Import 38 the MXD: (Replication) Pipeline Check the DS XML imported Workspace versions file by to selecting target DB random schema areas with its on data each version (source vs. target) 11 Stop and 25 break Copy 39 the other Replication Connect schemas QCUser process ( all data (G&PSS) owners ) to check migrated versions and data 12 Send the 26 announcement Default 40 QCUser data QCwill edit on QC.Test_Version to test the new environment (Add, Delete) 13 Make an 27 Oracle Role 41 dump Assignments QCUser (a restorable (G&PSS): (grant backup) Oracle Reconcile Roles and access Post QC.Test_Version permission to the to FDS) SDE.Default 14 Reconcile, 28 Post, Register 42 and Other Delete Datasets QCall Users As-Versioned child to versions test by to creating the DEFAULT new versions, version. editing, (Up to the reconciling Project Level) and posting sample data 23 Source Geodatabase: Target Geodatabase: QC (Data on Target): 29 Import 43 MyUser.JDK Send the announcement version (using to the connect Version all Migration users Tool) Saudi Aramco: Company General Use

24 Lessons Learned [ 4 ] Practice makes perfection - Once decided on the migration method: Conducted many tests Making different scenarios Utilizing different environments Developed validation procedures 24

25 Version Comparison DCs_Duba_Corr SOURCE TARGET Matching R_AVT_AIRPORT_PT - (11) R_AVT_AIRPORT_PT - (11) TRUE R_AVT_AVIATON_PL - (3) R_AVT_AVIATON_PL - (3) TRUE R_BMC_POST_PT - (197) R_BMC_POST_PT - (197) TRUE R_HYD_HYDRO_LN - (6555) R_HYD_HYDRO_LN - (6555) TRUE R_HYD_HYDRO_PT - (66) R_HYD_HYDRO_PT - (66) TRUE R_LDF_EXCAVATION_PL - (112) R_LDF_EXCAVATION_PL - (112) TRUE R_LDF_SURFACE_PL - (171) R_LDF_SURFACE_PL - (171) TRUE R_OAG_OILGAS_PL - (9) R_OAG_OILGAS_PL - (9) TRUE R_OAG_TANK_PL - (12) R_OAG_TANK_PL - (12) TRUE R_OAG_WELLPAD_PL - (6) R_OAG_WELLPAD_PL - (6) TRUE R_OAG_WELL_PT - (1) R_OAG_WELL_PT - (1) TRUE R_TRN_HIGHWAYSIGN_PT - (180) R_TRN_HIGHWAYSIGN_PT - (180) TRUE R_TRN_ROADCNTLINE_LN - (13) R_TRN_ROADCNTLINE_LN - (13) TRUE R_TRN_ROAD_LN - (1741) R_TRN_ROAD_LN - (1741) TRUE R_TRN_TRANSPORTSTRUCTURE_PL - (8) R_TRN_TRANSPORTSTRUCTURE_PL - (8) TRUE R_UDF_UNDEFINED_LN - (16) R_UDF_UNDEFINED_LN - (16) TRUE R_UDF_UNDEFINED_PL - (94) R_UDF_UNDEFINED_PL - (94) TRUE R_UTL_SUPPORTSTRUCTURE_PT - (2937) R_UTL_SUPPORTSTRUCTURE_PT - (2937) TRUE R_UTL_TRANSMISSIONTOWER_PL - (371) R_UTL_TRANSMISSIONTOWER_PL - (371) TRUE R_UTL_UNDGRNDCBLMARKER_PT - (868) R_UTL_UNDGRNDCBLMARKER_PT - (868) TRUE R_UTL_UNDGRNDCBL_LN - (13) R_UTL_UNDGRNDCBL_LN - (13) TRUE R_UTL_UTILITY_LN - (17) R_UTL_UTILITY_LN - (17) TRUE R_UTL_UTILITY_PT - (427) R_UTL_UTILITY_PT - (427) TRUE R_UTL_UTILITY_PL - (12) R_UTL_UTILITY_PL - (12) TRUE 25

26 Wasea_Bulk_Plant Source Target Diff. B_MAPPINGBOUNDARY_PL - (9) B_MAPPINGBOUNDARY_PL - (9) B_PLACE_PT - (2) B_PLACE_PT - (2) R_AVT_AVIATON_PL - (4) R_AVT_AVIATON_PL - (4) R_BMC_POST_PT - (221) R_BMC_POST_PT - (221) R_HYD_HYDRO_LBL - (1) R_HYD_HYDRO_LBL - (1) R_HYD_HYDRO_LN - (1342) R_HYD_HYDRO_LN - (1341) 1 R_HYD_HYDRO_PT - (35) R_HYD_HYDRO_PT - (35) R_TRN_CAMELCROSSING_PL - (82) R_TRN_CAMELCROSSING_PL - (82) R_TRN_GASOLINE_PT - (13) R_TRN_GASOLINE_PT - (13) R_TRN_HIGHWAYSIGN_LN - (2) R_TRN_HIGHWAYSIGN_LN - (2) R_TRN_HIGHWAYSIGN_PT - (24) R_TRN_HIGHWAYSIGN_PT - (24) R_TRN_ROADCNTLINE_LN - (4489) R_TRN_ROADCNTLINE_LN - (4489) R_TRN_ROAD_LN - (3159) R_TRN_ROAD_LN - (3157) 2 R_TRN_TRANSPORTSTRUCTURE_LN - (172) R_TRN_TRANSPORTSTRUCTURE_LN - (172) R_TRN_TRANSPORTSTRUCTURE_PL - (77) R_TRN_TRANSPORTSTRUCTURE_PL - (77) R_UTL_UTILITY_PL - (186) R_UTL_UTILITY_PL - (186) R_UTL_WATERTANK_PL - (346) R_UTL_WATERTANK_PL - (346) R_VEG_VEGETATION_PT - (889) R_VEG_VEGETATION_PT - (889) R_VEG_VEGETATION_PL - (121) R_VEG_VEGETATION_PL - (121) 3 26 Saudi Aramco: Company General Use

27 Wasea_Bulk_Plant R_TRN_ROAD_LN Source Geodatabase Target Geodatabase 27

28 Team IT Infrastructure IT Support App. Development CSL CAD Geodatabase QC Data Captures Rapid Update 28

29 29 Saudi Aramco: Company General Use