SANDAG LANDCORE GEODATABASE MIGRATION AND WORKFLOWS

Transcription

1 SANDAG LANDCORE GEODATABASE MIGRATION AND WORKFLOWS December 2005 Revision 1.1 October 2007 Revision 1.2 February 2008 Paul Hardwick Steve Kunkel

2 SANDAG LANDCORE GEODATABASE MIGRATION AND WORKFLOWS TABLE OF CONTENTS December 2005 Paul Hardwick Steve Kunkel

3

4

5

6 TABLE OF CONTENTS EXECUTIVE SUMMARY Background... 1 PROJECT PHASES... 4 Geodatabase Design... 4 Data Migration... 5 SanGIS Parcel Integration... 5 The Landcore Attribute Editor... 6 Scenarios... 7 GPALL Process... 7 CONCLUSION... 7 THE LANDCORE GEODATABASE DESIGN THE LANDCORE GEODATABASE DESIGN... 1 Geodatabase Structure Overview... 1 Landcore Geodatabase Schema Maintenance... 5 Relationship Classes in Landcore The Landcore Class Extension Description of the Landcore Class Extension Technical Reference for the Landcore Class Extension Landcore Class Extension Programming Notes THE LANDCORE DATA MIGRATION INTRODUCTION... 3 PART I... 3 PART II... 3 Part I, Step A. Apply CASE tools to generate a new, empty geodatabase... 4 Part I, Step B. Set data privileges PART II, STEP A. DATA LOADING PROCEDURES... 8 iii

7 Initial Setup Steps... 8 Data Loading... 9 Final Steps PART II, STEP B. CREATE TOPOLOGY AND VALIDATE PART II, STEP C. REGISTER AS VERSIONED THE LANDCORE ATTRIBUTE EDITOR INTRODUCTION... 1 SETTING UP A COMPUTER TO ACCESS LANDCORE AND USE THE ATTRIBUTE EDITOR... 1 Setup: Installing the Landcore Attribute Editor... 1 Setup: A Database Connection to Landcore... 2 Setup: Landcore Class Extension... 3 GETTING STARTED... 5 USING THE LANDCORE ATTRIBUTE EDITOR... 8 Introduction... 8 The Landuse Tab The Planned Land Use Tab The Ownership Tab The General Plan Tab The Forecast Tab The Edits Tab The Validation Tab SPECIAL LANDCORE BEHAVIORS THAT AFFECT EDITING PROGRAMMING DOCUMENTATION FOR THE LANDCORE ATTRIBUTE EDITOR Development Environment User Interface Development Programming Code Development Code Organization Special Coding SPECIAL INSTRUCTIONS: HOW TO ADD A CONTROL THE SANGIS / LANDCORE INTEGRATION THE SANGIS / LANDCORE INTEGRATION... 1 Introduction... 1 Description... 1 Model and Scripts Overview... 2 Model and Scripts Detailed Descriptions... 6 iv

8 1. Prepare New SanGIS Parcel Base Landcore Parsing SG Parse and Attribute Part I SG Parse and Attribute Part II New Parcel Attributes Merge SanGIS and Landcore Features Landcore Preparation Setup New Landcore Training Requirements Issues Gaps and Overlaps from SANGIS-Moved Parcels Stacked Features in the New SANGIS Shapefile (Condos and Timeshares) Large dataset geoprocessing Technical/Software THE GENERAL PLAN ALL PROCESS THE GENERAL PLAN ALL PROCESS... 1 INTRODUCTION... 1 GROWTH FORECAST INPUTS... 2 PROCESS OVERVIEW... 3 PROCESS DETAILS... 4 THE LANDCORE SCENARIO ENVIRONMENT INTRODUCTION... 1 Terminology... 1 Mission Critical Goals... 1 Non-Mission Critical Goals... 2 GENERAL APPROACH... 2 GIS CONFIGURATION... 3 Privileges... 3 Posting and Reconciling... 3 Versioning Tree... 4 THE GROWTH FORECAST WORKFLOW... 6 Landcore Maintenance Process... 7 Creating the Forecast Geodatabase... 7 v

9 Create a New Scenario... 7 Gpall Process... 7 Growth Forecast Modeling... 8 WORKING WITH SCENARIOS... 9 Creating a New Scenario... 9 Connecting to and Modifying a Scenario Deleting a Scenario Combining Features from Different Scenarios CONFLICT DETECTION APPENDICES APPENDIX A. LAYERFILE TABLES... 3 APPENDIX B. GEODATABASE DESIGN DIAGRAMS... 6 APPENDIX C. LANDCORE DATA MIGRATION APPENDIX D. TRAINING DOCUMENT - GEODATABASE DESIGN USING VISIO APPENDIX E. GPALL MODELS AND SCRIPTS APPENDIX F. SANGIS/LANDCORE INTEGRATION PROCESS vi

10 THE LANDCORE GEODATABASE DESIGN December 2005 Revision 1.1 December 2007 Revision 1.2 February 2008 Paul Hardwick Steve Kunkel

11

12 TABLE OF CONTENTS THE LANDCORE GEODATABASE DESIGN... 1 Geodatabase Structure Overview... 1 Landcore Geodatabase Schema Maintenance... 5 Relationship Classes in Landcore The Landcore Class Extension Description of the Landcore Class Extension Technical Reference for the Landcore Class Extension Landcore Class Extension Programming Notes iii

13

14 THE LANDCORE GEODATABASE DESIGN GEODATABASE STRUCTURE OVERVIEW This section provides an overview of the different components of the Landcore geodatabase. The reader will be referred to the database diagrams in Appendix B for more details and will be provided with details and a discussion of the various aspects of the geodatabase that are not shown in the diagrams. The main parts of the Landcore geodatabase and their properties are listed below. 1. LandCoreMaint feature dataset. LandCoreMaint is the main feature dataset which holds the spatial reference for features in Landcore. The dataset is designed in Visio, however the parameters of spatial reference are not set until the geodatabase schema is created through CASE tools. SANDAG has determined the parameters for the spatial reference which are: Coordinate System: California Stateplane, zone 6, feet. Precision: 1000 Min X: Min Y: Max X: (determined automatically) Max Y: (determined automatically) Z domain: default values M domain: default values This spatial reference is the same as that which is used by SanGIS. 2. Landcore feature class Landcore is the main polygon feature class in the Landcore geodatabase. The landcore feature class contains parcel data with land use, ownership, and other attributes maintained by SANDAG. The landcore parcels relate to records in other tables in the geodatabase. The geometry for these parcels originate from SanGIS however SANDAG may have split the original polygons into multiple parts based on land use considerations. These are single-part polygons. 3. Topology Class A geodatabase topology class was created to ensure correct polygon topology in the landcore feature class. The topology class has the following properties: Name: LandCoreMaint_Topology (default name) Cluster tolerance: (default) 1

15 Participating feature classes: landcore Rules: 1. landcore must not overlap 2. landcore must not have gaps The topology is deleted when Landcore undergoes major data loads and then later recreated and validated. Topology should be used and routinely validated as editors perform edits on the landcore feature class. 4. Existing Land use subtypes Generalized existing land use is used as a basis for the subtypes in Landcore. These are shown on the database diagram Figure 1 in Appendix B. Overall, subtypes are used in a minimal way in Landcore much of the functionality possible by using subtypes is not needed for Landcore. 5. Related Data tables There are ten data tables in Landcore. These tables and their fields are shown on the database diagram Figure 1 in Appendix B. Some of these tables function as lookup tables, like the original xref info tables, while others are used to store optional attributes of the Landcore polygons. 6. Relationship Classes There are six relationship classes in Landcore that serve to automatically relate 6 of the 10 tables to the landcore feature class. These are shown on the database diagram Figure 1 in Appendix B. The Landcore database design poster gives more detail about each relationship class, such as cardinality and key fields. 7. Attribute Domains Many attribute domains have been created in Landcore. Most of these are coded-value domains, their original source being the xref tables that were formerly used as lookup tables. These are listed in Figures 2 through 4 in Appendix B. 8. Custom Class Extension A custom class extension was developed for the Landcore feature class as a means to enforce certain behavior related to editing polygons. This is described in section IV The Landcore Class Extension in this document. This extension has particular significance because the Landcore feature class references this extension and thus Landcore cannot be viewed or edited without this extension present and installed on the user s computer. 9. Layer files While not actually part of the Landcore geodatabase, layer files are worth mentioning here because they help with the geodatabase design. For instance, they alleviate the need for symbol fields or subsetted feature classes. Some layer files have been developed for Landcore and more continue to be developed. These are stored in M:\RES\GIS\Layerfiles\LandcoreLayerfiles on the SANDAG file system network. These layer files use the standard color schemes created by SANDAG. Since they are centrally shared, they should not be modified by most users. 2

16 The steps to insert these layer files in your.mxd file are: Add the layer from M:\RES\GIS\LayerFiles\LANDCORELayerFiles. The layer will probably come into the table of contents (TOC) with the red exclamation point which means that you will have to repair the data source in order to use it. This is because the layer files are stamped with the user name and password of the person who made the file. Right click the layer name in the TOC. Right click on Data. Click Repair Data Source Navigate to the Landcore feature class in the LandCoreMaint feature dataset in the LandCore geodatabase. Click Add. All the layer files referencing the LandCore feature class that you added will be instantly repaired. 3

17 The following table summarizes the layer files that have been created for Landcore. Filename Layer Name scale dependency Definition Query symbolize on field transparency Landcore.GIS.Landcore.lyr Landcore.GIS.Landcore < 24,000 - General Land Use 90% Existinglanduse.lyr Existing land use < 24,000 - Existing Land Use 50% BaseYearLanduse.lyr Base Year land use < 24,000 - Base Year Land Use 50% PlannedLanduse.lyr Planned land use < 24,000 - Planned Land Use 50% LanduseChange.lyr Land use change < 24,000 BASELU <> LU Existing Land Use 50% BaseLUtoPLUchg.lyr BLU to PLU change < 24,000 BASELU <> PLU Planned Land Use 50% ExistingOwnership.lyr Existing ownership < 24,000 - Existing Ownership 50% BaseYearOwnership.lyr Base Year ownership < 24,000 - Base Own 50% OwnershipChange.lyr Ownership change < 24,000 BASEOWN <> OWN Existing Ownership 50% Redevelop_Infill.lyr Redevelop/Infill < 24,000 REDEVINF > 0 Planned Land Use 50% Site-Specific.lyr Site-Specific < 50,000 SITEID > 0 Planned Land Use 50% Numerous other layer files have been created, which use feature classes in the LIS geodatabase and file system imagery. These are shown in the tables in Appendix A. 4

18 LANDCORE GEODATABASE SCHEMA MAINTENANCE The schema for the Landcore geodatabase was originally designed and created using Unified Modeling Language (UML) in Microsoft Visio 2002 (later upgraded to 2003). The schema design process is explained in detail in the training document found in Appendix D, entitled Geodatabase Design using Visio. The fact that the schema was designed using UML, has importance for ongoing maintenance of the Landcore geodatabase, because changes to the Landcore schema should be carried out through a process involving the Visio drawing file. This is not a mandatory requirement, but rather a policy decision that SANDAG has decided to follow. This policy allows SANDAG to keep an active record of the Landcore schema design. This section of the Landcore geodatabase design document describes the process of making updates to the Landcore schema. The schema maintenance process should be followed when any of the following apply: A table is added or removed. A feature class is added to Landcore. A field is added to or removed from a table. A relationship class is added, removed, or modified. A domain code is added, removed, or changed. The domain description for a code is changed. A new person will be editing the Landcore geodatabase. The Landcore class extension is changed and recompiled. All the above reasons constitute a schema change. The schema maintenance process does not need to be followed when: Data is loaded into a table or feature class. The topology is deleted and rebuilt. The database is compressed or a table is unversioned. Edits occur on tables or features. The process of applying schema changes to the Landcore geodatabase is a fairly simple with a few exceptions. Steps: 1. Make the schema changes to the Visio diagram. The Landcore Visio diagram file is located on the SANDAG file system at: M:\Res\GIS\LandcoreSystemLibrary\schema\ Open this file using Visio 2003 and make the appropriate changes to this file. (Consult Appendix D and other training materials on how to do this.) Save this file with the filename incremented by one number. Export the file as an XMI format file to 5

19 M:\RES\GIS\LandcoreSystemLibrary\schema\ and run the semantics checker. Fix any errors and export to XMI again. 2. Make sure no users are connected to the Landcore geodatabase. 3. Using ArcCatalog connect to the Landcore geodatabase as the GIS user. 4. Use the CASE Schema Creation tool to step through the wizard, using the new XMI file as input. 5. Existing tables and feature classes will appear with a red outline as shown below. Click the Next button. If a new field is added, you will be prompted to go to the Exists tab of the Properties dialog. Select the table, and click Properties. Open the Exists tab and scroll the list of fields to the new field. Select Add Field. If the new field is actually just a renaming of an existing field, then choose the field which the new field is to replace/rename instead of choosing Add Field. 6. Click Next and Finish. 7. By applying the CASE Schema Creation tool to Landcore all the privileges (beside those for the database owner) are cleared. Therefore, you will have to reestablish the SDE privileges on the object classes in Landcore. This should be done for the feature dataset and all the tables except the vcounter table. To do this: a. Right-click each table, and select Privileges. 6

20 The process outlined above will work for nearly all schema modifications except if a domain code or description is to change. If a domain code is added, then the process above will work. To use the CASE Schema Creation wizard to change a domain code or description, you will first have to run a macro on Landcore before following the process outlined above. This macro is a workaround to bypass a bug in ArcGIS 9.2 and may not be necessary in later versions of ArcGIS. The bug is that the CASE Schema Creation tool will only modify a coded value domain if the number of codes in a domain are different in the geodatabase compared to the same domain in the XMI file. The workaround macro will add a new code and description to all the coded value domains in the geodatabase thereby making them different. The new code that is added is and the description is Bogus Code. This added code will later be removed when the CASE Schema Creation tool is run. The text for this macro is in the file TweakDomains_code.txt and is shown below. Private Sub LCDomainTweaks_Click() ' This is a workaround for a bug. ' When: To be used before the schema wizard is used to reapply the schema to landcore. ' Purpose: Make the domains "different" so that they will be updated by the CASE tool. ' Method: Add a bogus value to all the coded-value domains in the selected geodatabase. ' Dim sokdomainnamearray() As String 'Use later to restrict the changes to only the domains in this list. ' Dim pgxapplication As IGxApplication Dim pgxobject As IGxObject Set pgxapplication = Application Set pgxobject = pgxapplication.selectedobject Dim pgxdatabase As IGxDatabase Dim pworkspace As IWorkspace ' Make sure a geodatabase is selected If TypeOf pgxobject Is IGxDatabase Then Set pgxdatabase = pgxobject 'QI Set pworkspace = pgxdatabase.workspace Else MsgBox "Please Select A Personal/ArcSDE Geodatabase", vbexclamation Exit Sub End If ' Warn the user and allow then to exit. Dim swarn As String 7

21 swarn = "This command will change ALL the coded-value domains in the geodatabase " & pgxobject.name & "." & _ " You cannot undo this. Are you sure you want to do this?" If Not MsgBox(sWarn, vbyesnocancel, "Tweak Coded Domains") = vbyes Then Exit Sub End If ' DOMAINS Dim pdomain As IDomain Dim pcodedvaluedomain As ICodedValueDomain Dim pworkspacedomains As IWorkspaceDomains Set pworkspacedomains = pworkspace 'QI ' Setup for going throgh all the domains. Dim vboguscode As Variant Dim sbogusdescription As String vboguscode = sbogusdescription = "Bogus Code" Dim lcodes As Long Dim bfoundbogus As Boolean Dim bfoundanybogus As Boolean bfoundanybogus = False Dim pschemalock As ISchemaLock Dim i As Integer Dim penumdomain As IEnumDomain ' Get the Domains Set penumdomain = pworkspacedomains.domains If Not penumdomain Is Nothing Then ' get the first domain Set pdomain = penumdomain.next ' Loop through the domains Do Until pdomain Is Nothing ' if it is a coded-value domain If pdomain.type = esridtcodedvalue Then 8

22 Set pcodedvaluedomain = pdomain 'QI ' Loop through the values checking for any "99999" codes lcodes = pcodedvaluedomain.codecount bfoundbogus = False For i = 0 To lcodes - 1 If pcodedvaluedomain.value(i) = vboguscode Then bfoundbogus = True bfoundanybogus = True ' Exit For End If Next i ' If isn't in the domain. If Not bfoundbogus Then ' We probably need an exclusive schema lock so: Set pschemalock = pcodedvaluedomain 'QI pschemalock.changeschemalock esriexclusiveschemalock ' Add the bogus code and value pcodedvaluedomain.addcode vboguscode, sbogusdescription ' Release the exclusive lock pschemalock.changeschemalock esrisharedschemalock End If End If ' Next domain Set pdomain = penumdomain.next Loop End If 9

23 If Not bfoundanybogus Then MsgBox "Finished adding a Bogus code to all the coded-value domains in " & pgxobject.name & "." & _ " Geodatabase is ready for reapplying the schema from the XMI file." Else MsgBox "Finished adding a Bogus code to all the coded-value domains in " & pgxobject.name & _ ", however some or all of these already had a code of These were not altered." & vbcr & _ "Geodatabase is ready for reapplying the schema from the XMI file." End If End Sub Steps 1. Open this file with a text editor such as Wordpad or Notepad and copy all the text to the clipboard. 2. Open the VBA environment in ArcCatalog and paste this text into the bottom of ThisDocument code module. Close the VBA environment (File Close and Return to ArcCatalog). 3. Make sure no users are connected to Landcore. 4. Connect to Landcore as the GIS user. Select the Landcore geodatabase in ArcCatalog and run this macro (Tools-Macros-Macros). 5. The macro will issue a warning to the user as a safety precaution. It is recommended that this macro not be run from a UICommand button to avoid inadvertently running the code. 10

24 However, if the code is accidentally started, the following message provides a way to exit the macro. Clicking No or Cancel will cause the macro to stop. 6. A message box will popup indicating when the macro is finished running. Since there are a large number of domains in Landcore, the macro may take up to a minute to run. Click OK. Now that this macro has been run, the standard procedure for updating the Landcore schema (outlined at the beginning of this section) can be followed. This will update all the Landcore coded value domains with the coded value domains in the UML Visio drawing. 11

25 RELATIONSHIP CLASSES IN LANDCORE Introduction The Landcore geodatabase is a relational database, stored in a relational database management system, or RDBMS, called SQL Server. There are many parts to Landcore but the main parts are the tables, which, like any other table, consist of columns and rows. In GIS terms, the columns are fields and the rows are records. In the SQL Server geodatabase, the landcore feature class consists of several tables which are maintained and served to the user via specialized Server software called Spatial Database Engine (commonly referred to as SDE). Records in the landcore feature class are referred to as features. In Landcore, features may be related to records in the tables, or conversely table records may be related to landcore features. For instance, certain parcels have activity centers on them. Information about these activity centers is stored in records in the ActivityCenters table. By knowing which features relate to which table records, one can retrieve the table information for a set of features. In the geodatabase, this relationship is described and stored as a Relationship Class, which can then later be used to traverse from a set of selected parcels to a set of related records. For instance parcels can be selected that have at least one Activity Center with more than 3 office floors. Relationship classes store information about how the two tables relate to one another. The essential pieces of information are the names of the fields in each table which are used to relate records, the name of the origin table, and the cardinality of the relationship. In addition, the geodatabase relationship classes store the destination table name, the type of relationship, and some labeling text for traversing forward or backward. The relationship classes in Landcore and their properties are listed below. Landcore Relationship Classes Landcore contains six relationship classes which relate six tables to the landcore feature class. The name and description of each is listed below. Relationship class:: Activity Type :: Simple Cardinality :: 1-M Origin :: Landcore Origin Primary Key : LCKey Origin Foreign Key : LCKey Destination :: ActivityCenters Forward Label :: Activity Center Info Backward Label:: Activity LC Poly Relationship class:: Proprecords(Attributed) Type :: Simple 12

26 Cardinality :: M-N Origin :: Landcore Origin Primary Key : OBJECTID Origin Foreign Key : objectidlc Destination :: MPR Destination Primary Key : OBJECTID Destination Foreign Key : objectidmpr Forward Label :: Property Records Backward Label:: MPR Parcels Relationship class:: SiteSpecific Type :: Simple Cardinality :: 1-M Origin :: SiteSpec Origin Primary Key : siteid Origin Foreign Key : siteid Destination :: Landcore Forward Label :: SiteSpec LC Poly Backward Label:: SiteSpec Information Relationship class:: Plan Type :: Simple Cardinality :: 1-M Origin :: GenPlan Origin Primary Key : planid Origin Foreign Key : planid Destination :: Landcore Forward Label :: Plan LC Poly Backward Label:: GP, CPA, Sphere, SPA Info Relationship class:: Edit_History Type :: Simple Cardinality :: 1-M Origin :: Landcore Origin Primary Key : LCKey Origin Foreign Key : LCKey 13

27 Destination :: Edits Backward Label:: Edited Parcels Forward Label :: Edit History Relationship class:: OwnerInfo Type :: Simple Cardinality :: 1-M Origin :: Ownid Origin Primary Key : ownid Origin Foreign Key : ownid Destination :: Landcore Forward Label :: Owner Parcels Backward Label:: Owner Information In the geodatabase design diagram, the relationship classes appear as a heavy green line which connects landcore and a table. Their cardinality and forward/backward labels are indicated near where they connect to each table. The cardinality defines the number of features/records that will be related to a number of other features/records. In the diagram, a convention was used to place the origin tables to the left of the relationship class and destination tables to the right. All relationship classes in landcore are simple relationship classes (no composite relationships) and to optimize performance messaging is not used. In Landcore, the relationship classes are created at the same time the other objects in the geodatabase are created when the geodatabase schema is applied using CASE tools in ArcCatalog. Five of the six relationship classes have a 1-M cardinality and therefore, once created, do not require any additional preparation to be used. The Proprecords relationship class, however, has an M-N (many-to-many) cardinality and thus requires some additional steps to prepare it for use. These are discussed in the following section. The Property Records Relationship Class The relationship between the landcore parcels and the MPR (Master Property Records) table is based on common values in the parcelid field. The parcelid field in the MPR table is populated by SanGIS. SanGIS also populates the parcelid field in San Diego County parcels feature class, under their ongoing maintenance and updates. These parcels are the data source for features in Landcore. As mentioned earlier, the cardinality of the Proprecords relationship class is M-N. This means that it is possible for one or several polygons in the landcore feature class to be related to one or several records in the MPR Master Property Records (MPR) table. This can occur under the following two situations: 14

28 1. A parcel is split by SANDAG based on land use or other considerations. This causes the parcelid field to be duplicated in the two resulting polygons. Both of these polygons are related to the same record in the MPR table. This is a many-to-one relationship. 2. Timeshare or condominium parcels cause 1-M cardinality. With timeshare or condominium parcels, a single polygon is used to represent the property of a number of property owners. For instance an entire condominium complex is a single polygon in the landcore feature class yet it has associated records for all the different property owners in the MPR table. This is a one-tomany relationship. By definition a one-to-many and a many-to-one relationship between two of the same tables denotes a many-to-many (M-N) relationship. This is the situation between the landcore feature class and the MPR table. To accommodate this, the M-N relationship class Proprecords was designed in Visio and has been created in Landcore. In the geodatabase design diagram this is shown as the green line Proprecords with the green class box that is also called Proprecords. This is also known as an Attributed Relationship Class because it allows for the storage of additional attributes that describe the relationship between individual records. For more information about M-N relationships and how to create them consult the following ESRI documents: Building a Geodatabase Defining Relationship Classes Building Geodatabases with CASE Tools For an M-N relationship class to work, an attributed relationship class must have a table that essentially acts as a bridge between the two related tables. This table stores the foreign keys from both tables. Although the landcore features relate to the MPR records based on common parcelids, the parcelid field could not be used as a foreign key. This is because ArcGIS/ArcMap assumes that the primary keys are unique and the parcelid field is not unique in either table. This is important because ArcMap will make edits to this table in very special ways whenever features or records are added or removed. For instance, when a feature is deleted ArcMap will automatically delete any records in the relationship class table which are related to the feature because ArcMap assumes that these records are no longer needed. Through testing we found that using the parcelid as a common field (primary and foreign keys) produced very undesirable results. When a feature was deleted, the link for any remaining parcels with the same parcelid was lost. This confirms normal ArcMap and M- N relationship class behavior. Therefore, to ensure that the primary keys are unique, the objectid fields in landcore and the MPR table are used as primary keys in the Proprecords relationship class. Since the Proprecords relationship class is an attributed relationship class with a table, the table must be populated with records much like any other table. Unlike 1-M and 1-1 relationship classes, this additional step is required after the relationship class is created. There are three options available to populate the Proprecords relationship class; ArcMap, Table To Relationship Class Tool, or CreateMNRelationships Script. Either the Table To Relationship Class Tool, or CreateMNRelationships Script must be used after any data loading occurs on either the landcore feature class or the MPR table. We expect this to be no more often than quarterly, when the SanGIS parcel update is performed. The reason for the tools 15

29 are that the object ids (the primary key field) have changed during the update process outside of an ArcMap edit session. Any normal edits within an ArcMap edit session will not adversely affect the relationship class, even though the objectid field does indeed change. This is because ArcMap has built in processes which will automatically update the keys in the relationship class table. ArcMap To accomplish this task within an ArcMap edit session the editor can select the associated records in each table, opening the attributes dialog, right-clicking on the relationship name and selecting Add Selected, which will add one or perhaps several records to the relationship class table. This is desirable when adding features to Landcore in an edit session and is discussed later in this section. But, to establish the relationship between entire MPR table and Landcore feature class this manual task would have to be repeated over 800,000 times. Create MPR Relationship Class Tool As previously stated, the Proprecords relationship class needs to be set up so that the Landcore objectids and matching MPR objectids are joined on where there are matching parcelids in the two tables. This can be accomplished in SQL Server through an SQL query that writes results to a table that can be loaded into the relationship class. To simplify the process, a Python script and script tool were developed. The script executes the SQL query, takes the output table and registers it to the geodatabase, and runs the Table to Relationship Class geoprocessing tool. The SQL query selects Landcore objectids and matching MPR objectids, joined on parcelid where parcleid is not 0 in either table. The query as seen in SQL Query Analyzer looks as follows. SELECT gis.landcore.objectid AS objectidlc, gis.mpr.objectid AS objectidmpr INTO gis.tempmprrelatetable FROM gis.landcore INNER JOIN gis.mpr ON gis.landcore.parcelid = gis.mpr.parcelid AND gis.landcore.parcelid <> 0 AND gis.mpr.parcelid <> 0 ORDER BY gis.landcore.objectid To register the table to the Landcore geodatabase the Table to Table tool from the Conversion Toolbox in ArcToolbox is run. This tool copies the table to a new table and adds an ObjectId (OID) that allows ArcGIS to register the new table called MPRRelateTable with the geodatabase. The Table to Relationship Class Tool can be found in the Data Management Toolbox in ArcToolbox. This tool takes the MPRRelateTable and loads it into the PropRecords relationship class. The script tool found in the Landcore/SciptTools Toolbox has four required inputs. The first is the Landcore feature class. The second is the MPR table. The third is the PropRecords relationship class. And, the fourth is the name of the Landcore geodatabase. A browse tool can be used to enter the first three inputs. The last input must be typed in and is just the name of the geodatabase i.e. Landcore. 16

30 The CreateMNRelationships Script The CreateMNRelationships is a VBA macro written with ArcObjects. This macro was written by TAIC. It accomplishes the same process as the above script tool, but runs much slower because the use of ArcObjects requires each record to be processed separately. To use the CreateMNRelationships tool follow these steps: Steps 1. Get the text file called CreateM-NRelationships.txt and open it in Wordpad or Notepad. 2. Copy all the text in the CreateM-NRelationships.txt file to your clipboard (select and copy). 3. Open ArcCatalog. 4. Select Visual Basic Editor from the Tools Macros menu. 5. Open the ThisDocument code module: 6. Position the cursor at the very bottom of the ThisDocument code module and select Paste from the Edit menu. 7. Close the VBA Editor by selecting Close and Return to ArcCatalog from the File menu. 8. Connect to the Landcore geodatabase. 9. Select the Landcore geodatabase in ArcCatalog. 10. Select Macros from the Tools Macros menu. 11. Select the macro called ThisDocument.CreateMNRelationships. Macros Dialog 17

31 12. Enter the name of the landcore feature class (Landcore.GIS.Landcore). 13. Enter the name of the MPR Table (Landcore.GIS.MPR). 14. Enter the name of the relationship class (Proprecords). 15. Enter the name of the parcelid field (parcelid). 16. The next input box is an option that is useful to testing. It will allow only the first X number of features to be processed and then stop the macro. Enter a number or click the Cancel button to process all the features. A message will appear similar to the following when the tool has finished running. You can view the relationship class table by selecting the relationship class in ArcCatalog and then opening the Preview pane. There should be a record for each relationship, with the object ids of landcore and the MPR table. Do not run this macro more than once unless all records are first deleted from this table as it will append records to the end of the table. In the ArcToolbox the Delete Rows tool under Data Management - Table can be used to remove all the records from the relationship class table. Just drag the relationship class into the tool s Input Rows box. Editing Splitting Features There is a second design element of the Landcore geodatabase which forms an important part of ensuring that the Proprecords relationship class table remains updated. This design element is the landcore class extension, which is fully described in the Landcore Class Extension section of this document. The class extension ensures that when a polygon is split, both resulting polygons will automatically be related to the appropriate MPR records (assuming the original polygon was related to a record in the MPR table to begin with). This action is performed automatically and requires no additional action by the editor. It is the esrirsppreserveonall property that makes this possible. Editing Merging Features Merging features requires no additional steps by the editor. The normal ways that ArcMap automatically updates the Proprecords relationship class table are sufficient to keep the relationship class table current. Editing Adding Features When new features are added to the Landcore feature class, they will not have a relationship to the MPR table even if the parcelid field is calculated. This is because the M-N relationship table has not 18

32 been updated. If these new features should be related to the MPR table, the editor needs to take the following steps outlined below: Steps 1. Select the new feature. 2. Select the records in the MPR table that should be related to the new feature. This may be done by opening the MPR table, finding the records and selecting them by hand or by using the Select by Attributes dialog to select records by parcelid. 3. Open the Attributes dialog. 4. Expand the attributes until the Proprecords relationship is visible. 5. Right-click on Proprecords and select Add Selected. The steps above will have to be repeated for each new feature. 19

33 THE LANDCORE CLASS EXTENSION The Landcore Class Extension performs the following functions: Automatically prevents values in the LCKEY field from being cleared as a by-product of specific editing actions. Automatically prevents values in the PlanID, ownid, and siteid fields from being cleared when a parcel is split. Automatically updates values in the split field. These functions are essentially built into the Landcore geodatabase and are always present and operating whenever Landcore is viewed or edited using ArcGIS software. Description of the Landcore Class Extension The Landcore geodatabase makes uses of some special programming known as a Class Extension. A class extension can be defined as : Custom programming that is linked to an object class (tables or feature classes) in the geodatabase, in such a way that it automatically runs in response to particular events occurring on that object class. Thus it adds to, or extends, the normal behavior of an object class. In Landcore, the main polygon feature class called landcore uses this class extension so that it will respond in very specific ways when certain edits are performed on it. Putting this abstract definition into concrete terms, here is an example. A class extension can be used to automatically update values in the date and editorname fields in a geodatabase feature class to reflect the most recent date and editor s name. This special update behavior affects only the field of interest in the feature class. More significant is the fact that it is not controlled or dependent upon the user using a special tool to perform an edit. The update will occur using the standard editing tools available in ArcMap and any other programs such as ArcServer or ArcEngine applications which are capable of editing geodatabase feature classes. An alternate approach to using a class extension to accomplish automatic custom behavior, is to use custom programming within the application. There are two different yet similar ways that this can be done. The first way is to program custom tools which contain the code for the behavior and require the user to always use these. For instance, when the user digitizes a feature, they must use a special tool for digitizing. This tool will have code for digitizing a feature and additional code for performing some behavior. The second way to implement custom behavior is to write code for the behavior and associate it with the normal actions of the editing application (ArcMap). This code in effect waits and watches for the editing application to do a particular action on some data, at which time it will then execute to accomplish a custom behavior. Implementing the custom behavior using either of these methods does not guarantee that the behavior will always occur, because the user is free to work with the feature class outside of these applications. 20

34 To compare the application-based approach with the class extension approach, ESRI has provided a comparison table showing the pros and cons of each. This is shown below. Class extensions Application customization Advantages Disadvantages Database customization is always available it is not dependent on a particular application such as ArcMap being present. This can be important for feature classes accessed from ArcGIS Engine or ArcGIS Server. Business logic is stored closely to the data. A level of encapsulation is guaranteed. All ArcGIS users require access to the customization DLL, even to view the data. If the customization fails at runtime, the data cannot be accessed from ArcGIS (this can also be considered an advantage that ensures data integrity). The developer cannot make any assumptions that a particular application will be running - this can limit functionality. An object class can only have one class extension. You cannot easily extend annotation feature classes or dimension feature classes. Easy to implement and tightly integrated with application user interface. If customization fails then user can access important data with other tools. The DLL is only required by those users who need the specific customization functionality. There is a possibility that users could avoid business rules by running the application without the customization. Implementation is sometimes duplicated amongst several applications. The customization is only available when the application is running. Comparison of class extensions vs. custom applications. There is one additional difference between the two approaches which should be mentioned here: that of performance. Depending on the specific programming needed, a class extension may result in slower overall performance compared to the same behavior implemented through application customization. The decision was made to use a class extension instead of custom application programming primarily because this would better solve a specific problem, which is described in detail below. An important fact about this problem was that it compromised the integrity of the Landcore geodatabase by causing some data loss. Therefore we needed a robust solution that would reduce or eliminate any possibility of this occurring. We chose to use a class extension primarily because it would be present and effective whenever the user is working with Landcore. In fact once the class extension is tied to the landcore feature class, the user actually cannot access the feature class without first having the class extension installed and working on their computer. 21

35 A secondary consideration in the decision to implement a class extension for Landcore is by introducing a class extension into the geodatabase now, we felt that this would provide a mechanism for future development and offer SANDAG more options on how future development of the Landcore geodatabase could occur. As needs may arise for more functionality in Landcore, these needs could be met by either expanding the Attribute Editor (an application-based solution) or by adding to the existing Landcore class extension. The class extension which has been implemented in the Landcore Geodatabase is only for the landcore feature class. The problem that we have been discussing up to this point is the way certain fields in the geodatabase will drop their values when a polygon is split. This affects fields that are used as foreign keys in a geodatabase relationship class. Specifically, when a split occurs the smaller of the two resulting polygons will drop their foreign key values. This is documented normal geodatabase behavior. For the Landcore geodatabase, this means that the PlanID, SiteID, and the OwnID fields of the smaller parcel took on null values. This is very significant because this behavior caused landcore parcels to become disassociated with records in the GenPlan, OwnID, and SiteSpec Tables. To demonstrate, the attributes in a landcore parcel, before and after the split, are shown in the screenshots below. Figure. Attributes before the parcel is split. 22

36 Figure. Attributes in the smaller feature created from the split parcel. Figure. Attributes in the larger feature created from the split parcel. Notice the values in the Plan ID field in the top screenshot (before the split) compared to the middle screenshot (new smaller feature). To avoid this loss of data, we needed to introduce custom code into a class extension that will cause the original foreign key values to be duplicated. This class extension code would be specific to the landcore feature class and would execute whenever a parcel is split. The code is relatively simple, returning a value to the application that informs the application that the feature class has a custom split policy for relationships and setting a property that this custom split policy is to 23

37 preserve the original values. The property used is CustomSplitPolicyForRelationship which is set to PreserveOnAll. The code is shown below. Implements IClassExtension Implements IObjectClassExtension Implements IFeature classedit Implements IObjectClassEvents Private m_pclshelper As IClassHelper ' HRESULT constant for returning errors Private Const E_FAIL As Long = &H Const c_classextename = "Landcore Class Extension" ''Private m_papp As IApplication ''Private m_peditor As IEditor Private Sub IClassExtension_Init(ByVal pclasshelper As esrigeodatabase.iclasshelper, ByVal pextensionproperties As esrisystem.ipropertyset) Set m_pclshelper = pclasshelper ' The class for this extension End Sub Private Sub IClassExtension_Shutdown() Set m_pclshelper = Nothing End Sub Private Property Get IFeature classedit_caneditwithprojection() As Boolean End Property Private Function IFeature classedit_hascustomsplitpolicyforrelationship() As Boolean IFeature classedit_hascustomsplitpolicyforrelationship = True End Function Private Property Get IFeature classedit_customsplitpolicyforrelationship(byval Row As esrigeodatabase.irow, ByVal relclass As esrigeodatabase.irelationshipclass) As esrigeodatabase.esrirelationshipsplitpolicy IFeature classedit_customsplitpolicyforrelationship = esrirsppreserveonall End Property Private Sub IObjectClassEvents_OnChange(ByVal obj As esrigeodatabase.iobject) End Sub Private Sub IObjectClassEvents_OnDelete(ByVal obj As esrigeodatabase.iobject) End Sub 24

38 Through testing, we observed that the difference in performance between splitting a parcel with this class extension and without it to be minimal. LCKEY Field Automatic Calculation An additional capability that has been worked into the Landcore Class Extension is to automatically calculate the LCKEY field in landcore whenever a feature is created. This occurs when new features are added or a feature is split (two adds and one delete). There were two reasons for adding this capability to the class extension. The first reason is that this will ensure that the LCKEY is unique for all polygons, no longer requiring the user to remember to assign an LCKEY using a special button on the Landcore Attribute Editor. This is of special importance because, the LCKEY field is a primary key used by the two relationship classes in Landcore, Activity Centers and Edit History and in the forecasting process. If the LCKEY is not unique, these relationship classes will cause the corresponding foreign keys to be assigned a null value whenever landcore features are deleted. We found this to occur when polygons, having non-unique LCKEY values, were merged. The second, and lesser important, reason for including this capability in the class extension, is to avoid an LCKEY value of 0, which was assigned by default when a new feature is added. When automatically calculating the LCKEY value, the class extension will retrieve the next available LCKEY from the Vcounter table, calculate the LCKEY field for the new feature using this value, and then recalculate an incremented LCKEY value back into the Vcounter table. It is important to be aware that by calculating the LCKEY, the class extension affects related records in the Edits and ActivityCenters tables. Both of these tables use one-to-many relationship classes to relate to landcore features. It is the LCKEY in the tables that is the foreign key field in these relationships. When a feature is split, new LCKEYs will be assigned to the two resulting features. If the original feature (before the split) was related to a record(s) in either of these two tables, then the table records related to the larger of the two resulting features will also be reassigned the new LCKEY. Thus, the resulting larger feature will continue to be related to records in the ActivityCenters table and/or the Edits tables. If, after splitting a polygon, the ArcMap user wants the smaller portion, rather than the larger portion, to be related to the ActivityCenter or Edtis record(s), it is the user s responsibility to manually recalculate the LCKEY appropriately. This can be done by simply switching the LCKEY values in the Landcore feature class between the larger and smaller portions, using the Attributes dialog in ArcMap. This is probably the easiest method, instead of changing the LCKEY in the related table records, since possibly more than one table records may be related and thus need to be changed. The Visual Basic / ArcObjects code that carries out the automatic LCKEY calculation is shown below. This block of code was inserted into existing code in the IObjectClassEvents:On_Create event procedure. Also, a short reusable function called getprefix was created which is also shown below. ' CALC A NEW LCKEY ' Get the LCKEY Value from the feature. If prow.fields.findfield("lckey") = -1 Then MsgBox c_classextename & vbnewline & "LCKEY Field is missing. " 25

39 Exit Sub End If Dim strlckey As String strlckey = prow.value(prow.fields.findfield("lckey")) ' Set the name of the Activity Center and VCounter tables. Dim stracttablename As String Dim strvcountertablename As String If thegdbtype = "personal" Then stracttablename = "ActivityCenters" strvcountertablename = "VCounter" Else ' Get the prefix from the name of the feature class name ' and add it to the table names. Dim pfcdataset As IDataset Set pfcdataset = pfeature class 'QI Dim ptableprefix As String ptableprefix = getprefix(pfcdataset.name) stracttablename = ptableprefix & ".ActivityCenters" strvcountertablename = ptableprefix & ".VCounter" End If ' Get the next lckey from the vcounter table Dim pworkspace2 As IWorkspace2 Set pworkspace2 = pworkspace 'QI If Not pworkspace2.nameexists(esridttable, strvcountertablename) Then MsgBox "table with a name of " & strvcountertablename & " does not exist.", vbinformation, c_classextename Exit Sub End If Dim pfeatureworkspace As IFeatureWorkspace Set pfeatureworkspace = pworkspace ' QI Dim ptable As ITable Set ptable = pfeatureworkspace.opentable(strvcountertablename) Dim strnewlckey As String Dim pcursor As ICursor 26

40 Set pcursor = ptable.search(nothing, False) Dim pvrow As IRow Set pvrow = pcursor.nextrow ' Get the values. Do Until pvrow Is Nothing strnewlckey = pvrow.value(pvrow.fields.findfield("lckey")) Set pvrow = pcursor.nextrow Loop ' Increment the LCKEY the VCounter table. Set pcursor = ptable.update(nothing, False) Set pvrow = pcursor.nextrow Do Until pvrow Is Nothing pvrow.value(pvrow.fields.findfield("lckey")) = strnewlckey + 1 pcursor.updaterow pvrow Set pvrow = pcursor.nextrow Loop ' CALC the feature's LCKEY to the new lckey prow.value(prow.fields.findfield("lckey")) = strnewlckey Event procedure code continues here. Private Function getprefix(strinput As String) As String ' delimiter Dim strdel As String strdel = "." ' position Dim ldelpos As Long ldelpos = InStrRev(strInput, strdel) ' get the prefix If ldelpos > 0 Then getprefix = Left(strInput, ldelpos - 1) End If End Function 27

41 Split Field Update The Landcore class extension is also used to automatically update the split field in the Landcore feature class. This automatic update is secondary in importance yet it helps show the potential of developing this class extension. The split field in Landcore indicates if a parcel is considered a split parcel. A split parcel is defined as a parcel composed of multiple polygons. The Landcore Attribute Editor has two buttons to calculate the split for polygons: the -Ext for all polygons in the visible extent and the -Sel for the selected polygons. However, the user needs to remember to click these buttons after they have split a parcel. This assumes that the user is actually using the Attribute Editor. To make this update automatic, code was added to the Landcore class extension to automatically calculate the split field whenever a new feature is created. The calculation is performed only on the new features as well as features adjacent to them. Thus, when a parcel is split (which is two creates and one delete) the two resulting halves will be attributed as split. This was tested on 12/11/2007 and the split field was calculated correctly only for the new (smaller polygon). The split field in the larger polygon, which retained the original LCKEY was null. The code for this part of the class extension is shown below. Private Sub IObjectClassEvents_OnCreate(ByVal obj As esrigeodatabase.iobject) On Error GoTo ErrHandler ' This was just for testing: ' SplitIsFive obj Dim strdupfieldname As String strdupfieldname = "apn8" ' ' Get the Feature Class Dim pfeature class As IFeature class Set pfeature class = m_pclshelper.class ' Get the new feature. Dim prow As IRow Set prow = obj Dim pfeature As IFeature Set pfeature = prow 'QI '' ' - get the editor '' Dim pid As New UID '' pid = "esrieditor.editor" '' Dim papp As IApplication '' Set papp = Application '' Dim peditor As IEditor 28

42 '' Set peditor = papp.findextensionbyclsid(pid) ' Determine if SDE or Personal GDB Dim thegdbtype As String Dim pfeaturedataset As IFeatureDataset Set pfeaturedataset = pfeature class.featuredataset Dim pworkspace As IWorkspace Set pworkspace = pfeaturedataset.workspace ' type inheritance of this property If pworkspace.type = esriremotedatabaseworkspace Then thegdbtype = "SDE" End If If pworkspace.type = esrilocaldatabaseworkspace Then thegdbtype = "personal" End If ' Check for the required fields. Dim pfields As IFields Set pfields = prow.fields Dim lngdupfieldindex As String lngdupfieldindex = pfields.findfield(strdupfieldname) If lngdupfieldindex = -1 Then MsgBox "The field " & strdupfieldname & " is not present in the feature class. " & c_classextename ' Err.Raise E_FAIL,, "The field " & strdupfieldname & " is not present in the feature class. " & c_classextename Exit Sub End If Dim lngsplitfieldindex As Long lngsplitfieldindex = pfields.findfield("split") If lngsplitfieldindex = -1 Then MsgBox "The split field is not present in the feature class. " & c_classextename Exit Sub End If 29

43 ' =============================== end setup checks ========================================== ' Get the TOPOLOGY Dim ptopologycontainer As ITopologyContainer Set ptopologycontainer = pfeaturedataset ' QI Dim ptopology As ITopology Set ptopology = ptopologycontainer.topology(0) ' Assumes the FIRST topology. ' VALIDATE the topology for that feature. Dim penvelope As IEnvelope Set penvelope = pfeature.shape.envelope If pworkspace.type = esrilocaldatabaseworkspace Then Dim penvelope_validated As IEnvelope 'peditor.startoperation Set penvelope_validated = ptopology.validatetopology(penvelope) 'peditor.stopoperation "Validated Topology operation" End If ' Get the TopologyGraph Dim ptopograph As ITopologyGraph Set ptopograph = ptopology.cache ' Get the Topology Edges Dim pedges As IEnumTopologyEdge ' Build the Topology Graph ptopograph.build penvelope, False Set pedges = ptopograph.edges '' MsgBox "There are " & pedges.count & " edges in the topology graph." ' Get the first Edge Dim pedge As ITopologyEdge 30

44 Set pedge = pedges.next ' Parent Features Dim pparents As IEnumTopologyParent Dim ptopoparent As tagesritopologyparent ' Dim variables for finding the splits. Dim pfcursor As IFeatureCursor Dim pqfilter As IQueryFilter Dim lngparentid As Long Dim i As Integer Dim thevalue As String Dim strvaluesarray() As String Dim strdupvaluesarray() As String Dim intj As Integer Dim blnwasduped As Boolean Dim thedupvalue As String Dim bnoparent As Boolean bnoparent = False ' For each Edge Do Until pedge Is Nothing ' Get the parents of the current edge Set pparents = pedge.parents If pparents.count = 0 Then bnoparent = True End If ' Clear the duplicated values for the parents of the previous edge. ReDim strdupvaluesarray(0) 31

45 ' Get the first parent. ptopoparent = pparents.next ' Do Until ptopoparent = Is Nothing For i = 0 To pparents.count - 1 ' Get the Parent Object ID lngparentid = ptopoparent.m_fid ' MsgBox "Parent with id of " & theparentid ' Need to get an attributes based on the Parent's OID ' Create the feature cursor Set pqfilter = New QueryFilter If thegdbtype = "personal" Then pqfilter.whereclause = "[OBJECTID] = " & lngparentid Else pqfilter.whereclause = "OBJECTID = " & lngparentid End If ' Feature Update Cursor Set pfcursor = pfeature class.update(pqfilter, False) Set pfeature = pfcursor.nextfeature Do Until pfeature Is Nothing ' get the field value thevalue = pfeature.value(pfcursor.findfield(strdupfieldname)) ' Calc the feature as not split, but don't override a poly that ' has been calced as split. If pfeature.value(pfcursor.findfield("split")) = "0" Then pfeature.value(lngsplitfieldindex) = 9 pfcursor.updatefeature pfeature 32

46 End If ' Add the Value to an array that will hold the values of the parents. ' This array is used to compare the value in each parent in order to determine ' if the parent has a value that has already been found. If i = 0 Then ' If the first parent, then just add it to the array. ReDim strvaluesarray(0) strvaluesarray(ubound(strvaluesarray)) = thevalue Else ' See if this Value has been encountered earlier by going through ' all the array (other parent's) looking for it's value. blnwasduped = False For intj = LBound(strValuesArray) To UBound(strValuesArray) If strvaluesarray(intj) = thevalue Then ' thevalue is duplicated! If strdupvaluesarray(0) = "" Then ' Add it to the dup array which has just one empty element. strdupvaluesarray(0) = thevalue blnwasduped = True Else ' Expand the array and add it to the duplicated values array. ReDim Preserve strdupvaluesarray(ubound(strdupvaluesarray) + 1) strdupvaluesarray(ubound(strdupvaluesarray)) = thevalue End If End If Next intj ' Add it to the array if it has not yet been encountered If Not blnwasduped Then ReDim Preserve strvaluesarray(ubound(strvaluesarray) + 1) strvaluesarray(ubound(strvaluesarray)) = thevalue End If 33

47 End If ' next feature Set pfeature = pfcursor.nextfeature Loop ' Next Parent ptopoparent = pparents.next Next i ' next parent counter '' If bnoparent Then '' MsgBox "Could not get the parents for one of the edges." '' End If ' NOW see if duplicates were found for that edge. ' IF duplicates were found, go back through the parents and calc those parents ' that have a duplicated value. If strdupvaluesarray(0) <> "" Then For intj = LBound(strDupValuesArray) To UBound(strDupValuesArray) thedupvalue = strdupvaluesarray(intj) '' MsgBox "Duplicate ---> " & thevalue ' Get the parents of the current edge. Set pparents = pedge.parents ' Get the first parent. ptopoparent = pparents.next For i = 0 To pparents.count - 1 ' Get the Parent's Object ID 34

48 lngparentid = ptopoparent.m_fid ' MsgBox "Checking parent " & lngparentid & " for a value of " & thedupvalue ' Query Filter Set pqfilter = New QueryFilter If thegdbtype = "personal" Then pqfilter.whereclause = "[OBJECTID] = " & lngparentid Else pqfilter.whereclause = "OBJECTID = " & lngparentid End If ' Feature Update Cursor Set pfcursor = pfeature class.update(pqfilter, False) Set pfeature = pfcursor.nextfeature Do Until pfeature Is Nothing ' get the field value thevalue = pfeature.value(pfcursor.findfield(strdupfieldname)) If thevalue = thedupvalue Then ' Calc the Split field as "split" ' MsgBox "Calc split as Split for polygon for " & thevalue pfeature.value(lngsplitfieldindex) = 1 pfcursor.updatefeature pfeature End If ' next feature Set pfeature = pfcursor.nextfeature Loop ' Next Parent ptopoparent = pparents.next 35

49 Next i Next intj End If ' next edge Set pedge = pedges.next Loop ' each edge '' ' REFRESH '' Dim pactiveview As IActiveView '' Set pactiveview = pmap ' QI '' pactiveview.partialrefresh esriviewgeography, pgeofeaturelayer, Nothing ' player, Nothing ' MsgBox "Finished" ErrHandler: ' SendErrorMessage c_modulefilename, "Landcore", Err.Number, Err.Source, Err.Description Exit Sub End Sub The actual code in class extension that calculates the split field is rather lengthy and complex and therefore a slight performance hit is noticeable. Our testing showed a 2-3 second delay between the time the parcel is split and when the code finishes. Since splitting parcels is not a very common task, we felt that this delay is tolerable. 36

50 Technical Reference for the Landcore Class Extension This section of the Landcore documentation is to explain how the Landcore class extension was setup and what technical steps may be required in the future to keep it setup. 1. Hooking the class extension to Landcore (this process is seldom done) The Landcore class extension must be hooked (for lack of a better term) to the landcore feature class. Without doing this the code is not associated with landcore and will not automatically run whenever it needs to. This hooking process needs to be done whenever the landcore feature class is recreated. For instance, landcore is recreated when the SanGIS parcel update is performed. The landcore class extension does not need to be rehooked to the landcore feature class when the schema is reapplied using CASE tools, since the class extension is part of the geodatabase design in Visio. Steps 1. Start ArcCatalog. 2. Open the VBA Editor (Tools-Macros-Visual Basic Editor). 3. Open the This Document code module by double clicking on the This Document icon shown below: 4. In the Normal.gxt window, confirm that the SetClassExtension macro is not already present in the code module by selecting General from the left dropdown and checking the right dropdown for SetClassExtension. If it is not there, then follow steps 5 and 6 below. 5. Using Wordpad or Notepad, open the file SetClassExtension_VBA.txt which is located at M:\RES\GIS\LandcoreSystemLibrary\schema\LandcoreClassExtension. Copy all this text onto the clipboard and close the file. 6. Go back to the VBA editor and scroll the Normal.gxt window to the bottom. Insert the cursor there and select Paste from the Edit menu. 7. Close the VBA Editor by selecting Close and Return to ArcCatalog from the File menu. 8. In ArcCatalog, connect to the Landcore geodatabase. Open the feature dataset and select the landcore feature class. 9. Verify that no other users are connected to this Landcore geodatabase. 37

51 10. Select Macros from the Tools Macros menu. 11. Select the macro ThisDocument.SetClassExtension and click Run. 12. Enter {4963F554-E891-4B5C-905D-93026F04AF7C} in the GUID dialog and click OK. This is the GUID that was used when the class extension was compiled. It may change in future updates of the class extension. 13. You will be notified when the class extension has been changed. Click OK. For reference, the code for the SetClassExtension is shown below. However, we recommend that you do not copy this code from this document, but rather copy the text from the text file provided. ' Implementing a class extension is an ideal way to extend the default geodatabase behavior. 'This VBA script lets you set the class extension of an existing feature class or object class. 'To do this you must have already implemented a class extension suitable for the dataset. 'You should be cautious when using this script, since incorrectly applying class extensions can lead to unexpected 'behavior. ' 38

52 'How to use: 'Paste the code into the ArcCatalog VBA environment. ' 'Find the CLSID of the class extension you have developed. To do this search the registry for the name of your 'class and copy the CLSID to the clipboard. ' 'In ArcCatalog, select the appropriate feature class or object class, then run the script 'Paste in the CLSID, and press OK. ' 'If you enter the string 'Nothing' instead of a CLSID, any current class extension will be cleared. 'If you press Cancel, the current CLSID will be shown. ' Public Sub SetClassExtension() On Error GoTo ErrH Dim pgxapp As IGxApplication Set pgxapp = Application Dim pgxobject As IGxObject If pgxobject Is Nothing Then Set pgxobject = pgxapp.selectedobject End If If Not TypeOf pgxobject Is IGxDataset Then MsgBox "A geodatabase feature class or table is not selected. Quitting.", vbinformation, "Set Class Extension" Exit Sub End If Dim pgxdataset As IGxDataset Set pgxdataset = pgxobject ' QI If Not TypeOf pgxdataset.dataset Is IClass Then MsgBox "A geodatabase feature class or table is not selected. Quitting.", vbinformation, "Set Class Extension" Exit Sub End If Dim pclass As IClass Set pclass = pgxdataset.dataset Dim strguid As String strguid = InputBox("Enter GUID", "Set class extension for " & pgxobject.name) If Len(strGUID) <> 38 And UCase(strGUID) <> "NOTHING" Then 39

53 ' Show the current extension Dim strcurrent As String If pclass.extclsid Is Nothing Then strcurrent = "Current class extension is nothing" Else strcurrent = "Current class extension is: " & pclass.extclsid End If MsgBox "No valid GUID entered." & vbnewline & strcurrent Exit Sub End If Dim puid As New UID If UCase(strGUID) = "NOTHING" Then Set puid = Nothing Else puid.value = strguid End If Dim pclassschemaedit As IClassSchemaEdit Set pclassschemaedit = pclass Dim pschemalock As ISchemaLock Set pschemalock = pclassschemaedit pschemalock.changeschemalock esriexclusiveschemalock pclassschemaedit.alterclassextensionclsid puid, Nothing pschemalock.changeschemalock esrisharedschemalock MsgBox "Class extension changed for " & pgxobject.name Exit Sub ErrH: ' SendErrorMessage "", "SetClassExtension", Err.Number, Err.Source, Err.Description MsgBox "An error occurred in the setclassextension Macro" End Sub 40

54 I. Removing (unhooking) the Class Extension from a feature class. Although rare, there may be times when the Class Extension needs to be removed from a feature class. This may happen if the landcore feature class is copied from one geodatabase into another. The copying process also copies the settings in the feature class that tells ArcGIS to use the Class Extension. To unhook the Class Extension, follow the same steps for hooking but instead of entering the GUID number (step 12 above), enter the word NOTHING (all upper case). Code in the macro will disassociate the class extension with the feature class. II. Installing the Class Extension on a User s Computer (repeated when class extension updates are made) It is a requirement that all computers which will need to view the Landcore geodatabase have the Landcore Class Extension installed. Without the extension they will not be able to view the landcore feature class or any other feature classes within the same feature dataset. This is because ArcMap or ArcCatalog will attempt to load the extension when the feature class is loaded. If the extension is not installed, the user will receive the following message: The procedure to install the Landcore class extension on a computer is a relatively simple one and should be within the grasp of most users. To make this as easy as possible, a software installer is used which will copy the necessary files (just one dll file), register the dll, and to register the class with the correct COM Component. Installation Steps: Close any ArcMap or ArcCatalog sessions that are connected to the Landcore geodatabase. First, check that the Class Extension is not already installed on the computer. To do this Open the Control Panel (Start Control Panel). Double-click Add or Remove Programs. Scroll the list down to where the program SANDAG Landcore Class Extension would appear (alphabetically sorted). See below. If the SANDAG Landcore Class Extension is on the list, then it is installed. It must be uninstalled before installing a new version. To uninstall, select the SANDAG Landcore Class Extension and Click Change/Remove. See below. 41

55 Click Yes on the Application Removal dialog. When finished, close the Add/Remove Programs window and the Control Panel. Next, install the SANDAG Landcore Class Extension onto the computer. 1. Go to m:\res\gis\landcoresystemlibrary\ where the installer for the SANDAG Landcore Class Extension is located. If needed, extract the contents of the zip file setup.zip. 2. Double-click the file setup.exe. This will start the installer. 3. Click OK. 4. Designate the folder on the computer s local drive by clicking the Change Directory button. Browse to C:\ArcGIS\addons and double-click the LandcoreClassExtension folder or type \LandcoreClassExtension after the C:\ArcGIS\addons path to create the new folder. 5. Click the large icon on the dialog to install the class extension. 6. When finished, click OK. The computer does not need to be restarted. Once installed, Landcore Class Extension will appear as an installed program under the Add/Remove Programs (Control Panel Add/Remove Programs) with the name SANDAG Landcore Class Extension. Also, if the Developer Kit is installed, the class will appear under the ESRI GeoObject Class Extension COM Component in the Component Category Manager (see below). Uninstalling the Landcore Class Extension To uninstall the Landcore Class Extension, use the Add/Remove Programs in the Control Panel. 42

56 Landcore Class Extension Programming Notes The Landcore class extension was developed in Visual Basic 6 using ArcObjects from ArcGIS 9.0. The source code for this Visual Basic project is located in: Several ESRI-provided Visual Basic add-ons were used in development. These include the Compile and Register and the Interface Implementer. The Compile and Register add-on will compile and register the dll and will register the new class into the designated COM component categories. Of course this is only on the development machine, which was useful for testing. The VB debugger was not used for testing. The Compile and Register add-on also creates the <project>.reg file which is important later for deployment. Various methods exist to package and deploy a class extension. In the initial stages of development a simple yet technical method was used which involves registering the dll and then running the <project>.reg file to register the class to the components. However for more widespread deployment we felt that this was too advanced for many users, so we chose to deploy via an installer. The Package and Deploy wizard in Visual Basic 6 was used to create the setup.exe file which the user will simply run to install the Landcore class extension. Further Help and Documents While some experience and training is practically a must, here are some places to get help: ArcGIS Developer Help VB6 Help Extending ArcObjects Chapter 7 ESRI Discussion Forum ArcObjects General 43

57 THE LANDCORE DATA MIGRATION December 2005 Revision 1.1-February 2008 Paul Hardwick Steve Kunkel

58

59 TABLE OF CONTENTS INTRODUCTION... 3 PART I... 3 PART II... 3 Part I, Step A. Apply CASE tools to generate a new, empty geodatabase... 4 Part I, Step B. Set data privileges PART II, STEP A. DATA LOADING PROCEDURES... 8 Initial Setup Steps... 8 Data Loading... 9 Final Steps PART II, STEP B. CREATE TOPOLOGY AND VALIDATE PART II, STEP C. REGISTER AS VERSIONED iii

60

61 THE LANDCORE DATA MIGRATION INTRODUCTION Following the process of designing the Landcore geodatabase in Microsoft Visio, the Landcore geodatabase was created through a process that had two major parts. The first major part is the creation of the entire schema of the geodatabase. The Landcore geodatabase feature dataset, feature class, tables, and domains were created using CASE tools and the appropriate user privileges are set. This process is described in greater detail below. The second major part of the data migration is the data loading process. In this process, some pre-planning was done to map the existing GIS tables and fields to the new geodatabase schema. Next, the data were loaded using the simple object loader. To complete the data migration process, the topology class is created and validated and some tables are registered as versioned. Throughout this document the naming convention for the Landcore geodatabase will be the proper case word Landcore, occasionally followed by the word geodatabase. The naming convention for the landcore feature class will be the lowercase word landcore, occasionally followed by the words feature class. Outline of Data Migration PART I Step A. Apply CASE tools to generate a new, empty Landcore geodatabase Step B. Set database privileges on all new object classes PART II Step A. Execute the data loading procedures Step B. Create topology class and validate Step C. Register tables as versioned The Landcore geodatabase was designed in Visio using UML, which was then exported to an XMI file. The XMI file contains all the information about the structure of the geodatabase. This file, in conjunction with the Date Type Dictionary file uml.dtd residing in the same folder, is used by the CASE tools subsystem of ArcGIS to create the new geodatabase. 3

62 Landcore Data Migration Apply CASE Tools Part I, Step A. Apply CASE tools to generate a new, empty geodatabase The first step is to generate the Landcore schema using the following procedures: 1. Start ArcCatalog. 2. An SDE service to the Landcore geodatabase has already been created by the database administrator. The connection properties are shown in the database connection dialog below. Connection to the Landcore geodatabase 3. Connect to the Landcore geodatabase as the GIS user. 4. Next, if you haven t previously done this, add the Case Schema Creation button to a toolbar in ArcCatalog. Open the Customize dialog and add the Case Schema Creation wizard to the ArcCatalog toolbar. Schema Wizard in the Customize Dialog 4

63 Landcore Data Migration Apply CASE Tools 5. Select the Landcore geodatabase, which you just connected to and click the Case Schema Creation button ( ). 6. Click Next in the wizard dialog. 7. Click the Browse button and select the XMI file that was exported from Visio. 8. If the next dialog presents the following message, select Use Default Values and click Next. 9. Select the LandCoreMaint feature dataset and click Properties. 10. In the Feature Dataset Properties dialog, click the Edit button. 11. In the Spatial Reference dialog, click Import. 12. Prior to this, a template personal geodatabase was created for the purpose of storing the spatial reference information. Browse to that geodatabase and select the feature dataset. Click Add. 13. Click Ok. 14. Select the Landcore feature class and click Properties. 15. On the General tab, enter the following Spatial Index grid sizes: Grid 1 =

64 Landcore Data Migration Apply CASE Tools Grid 2 = 0 Grid 3 = Click Next. 17. Click Finish. The CASE Schema creation tool then creates the entire schema for the Landcore geodatabase. This includes all the tables, domains, and relationship classes. The spatial grid size of 4000 was derived through calculations based on a typical viewable extent of landcore and the size of the polygons in landcore. The grid size is used as a type of indexing to help speed retrieval and drawing in ArcMap. At this time, we have found that the drawing speed, while performing routine maintenance tasks on Landcore, is acceptable. 6

65 Part I, Step B. Set data privileges. Different levels of user access, or privileges, can be established for each object class, with enterprise geodatabases. These levels are: select, update, insert, and delete. In step 1, the gis user was connected to Landcore when they created the object classes (tables, datasets, feature classes) using the CASE Schema Creation wizard. Therefore, gis is the owner of these objects in Landcore and automatically has full privileges. Previously, the SANDAG SDE database administrator had created a user role called Editors and added the SANDAG GIS staff to this role. Now, the Editors role needs to be given all four privileges for all the tables and the feature dataset in Landcore. To set these privileges: 1. Open ArcCatalog. 2. Connect to the Landcore geodatabase as the gis user. 3. Right-click each object (tables and feature dataset) in ArcCatalog and select privileges. 4. Type Editors as the user name. 5. Check the boxes next to all the privileges. 6. Click Ok. Privileges granted to a feature dataset will be inherited by feature classes within that dataset. 7

66 Landcore Data Migration Data Loading PART II, STEP A. DATA LOADING PROCEDURES The Landcore geodatabase is essentially an empty shell up to this point. Features need to be added to the feature class and records added to the tables. In this step, a simple data loader is used to populate the geodatabase classes. However, first some planning is needed to map the data from the source to the destination. In doing this, data loading steps were formalized and documented for each object class and used as reference while loading the data into the geodatabase. The source data for Landcore existed in the landcore ArcInfo coverage and numerous xref info tables located on the SANDAG network. In anticipation of the data loading process, in which a wizard is used to map source fields to destination fields, a table listing the landcore geodatabase fields and the corresponding fields in the source tables was developed. This table is shown in Appendix C. This table also proved to be useful in early stages of the geodatabase design to record field types and aliases, which were then later coded into the UML classes in Visio. The data loading process is one where records are copied from a source table and loaded into a destination table. Thus all fields in each record are populated at the time of the load. By creating the table just described, some pre-processing needs were identified where fields needed to be combined or reformatted in order to fit into the new schema. This pre-load processing was done by SANDAG staff in the ArcInfo Workstation environment. A set of detailed data loading instructions were developed by piloting the data loading on a test personal geodatabase. These instructions are grouped by the table to be loaded and are formatted as a checklist outline. They are shown in their original format on the following four pages. Data Loading Check Sheet Initial Setup Steps Open ArcCatalog and connect to the Landcore Geodatabase as the GIS user. This user will own Landcore. Delete the Landcore feature class in the LandCoreMaint feature dataset, any existing tables that are part of the landcore geodatabase schema. Do not delete any of the other tables or feature classes that are in the geodatabase. Copy the latest XMI file of the Landcore geodatabase schema to Make sure a uml.dtd file is also in that folder. In ArcCatalog click the "Case Schema Creation" Use CASE tools to create the schema in this geodatabase. If you do not have this button, then add it to a toolbar using the customize dialog. The tool is located in Category: Case Tools: Commands: Schema Wizard. Follow the Schema Wizard. Define the spatial reference by importing from the SpatialReferenceSD feature dataset LIS geodatabase. Change the spatial index grid to Grid1 = 4000 Finish the Schema Wizard. Open the excel table FieldInfo_FieldMapping.xls. This will be referred to throughout the loading 8

67 Landcore Data Migration Topology process as a guide for the source data. This spreadsheet defines the source tables and fields for the Landcore geodatabase feature class and tables. This is called the Mapping Table. Notes: ALL the source data is located in: C:\agis\Projects\SANDAG\DataLoading\SourceData\prefinalLoad Data Data Loading Landcore Feature class Load Target: landcore.gis.landcoremaint.landcore (feature class) A geodatabase feature class called newlcall in the LandcoreCovImport dataset has already been prepared as a data source for loading. This contains all features with fields formatted for loading. Connect to this at kona.landcore.gis.sde (5155 service) as the GIS user. An AML has been run on the original coverage which has calculated a field called split that has values to indicate if a parcel is a landcore split parcel. The LCKey field has been calculated to be unique for all features. Right-click the landcore feature class and select Load Data. Follow the wizard making sure the fields are mapped correctly, according to the Mapping Table. Edits Table Load Target: landcore.gis.landcoremaint.edits Refer to the excel FieldInfo_FieldMapping.xls for the source table and fields. Since this table is related using a 1 to many relate and many polygons do not contain any edit history, use the query USER <> to load only records with an edit history. Right-click the edits table and select Load Data. Follow the wizard making sure the fields are mapped correctly, according to the Mapping Table. ActivityCenters Table Load Target: landcore.gis.landcoremaint.activitycenters Some overlay analysis is needed to transfer the LCKEY from Landcore to the activity centers table. The activity centers points will be overlaid onto the landcore feature class, using the Identity tool. Open the ArcToolbox. Open the Identity tool from the Overlay tool set in the AnalysisTools toolbox. Set the following parameters: Input: the activity Centers point coverage. (\DataLoading\SourceData\pre-finalLoadData\activitycov) Identity: The Landcore feature class (DatabaseConnections\kona.gis.landcore.sde\landcore.GIS.LandCoreMaint\landcore.GIS.Landcore) 9

68 Landcore Data Migration Topology Output: A temporary point feature class. Call this activelandcore and put this into the LandCoreMaint dataset. The next major step is to calculate the LC Parcel ID field in the ActivityCenters table. This will be done using a relate in ArcMap. Start ArcMap. Add the activelandcore feature class to the map. Add the source ActivityCenters table to the map (refer to the Mapping table). Join the activelandcore _landcore feature class to the source ActivityCenters table based on the "ID" Field. Open the source ActivityCenters table and calculate the activitycenters:lckey equal to the LCKEY field in landcore. You can do this outside an edit session. Close ArcMap. The final major step is to load records into the ActivityCenters table in the geodatabase. Use the Simple Data loader (as you did before). Refer to the Mapping table for field mapping. Clean up by deleting the activelandcore feature class with ArcCatalog. OWNID Table Load Target: landcore.gis.landcoremaint.ownid Right-click the Ownid table and select Load Data. Follow the wizard making sure the fields are mapped correctly, according to the Mapping Table. SiteSpec Table Load Target: landcore.gis.landcoremaint.sitespec Right-click the SiteSpec table and select Load Data. Follow the wizard making sure the fields are mapped correctly according to the Mapping Table. GenPlan Table Load Target: landcore.gis.landcoremaint.genplan Right-click the GenPlan table and select Load Data. Follow the wizard making sure the fields are mapped correctly according to the Mapping Table. MarketStat Table Load Target: landcore.gis.landcoremaint.marketstat Right-click the MarketStat table and select Load Data. Follow the wizard making sure the fields are mapped correctly according to the Mapping Table. 10

69 Landcore Data Migration Topology MPR Table Load Target: landcore.gis.landcoremaint.mpr Right-click the MPR table and select Load Data. Follow the wizard making sure the fields are mapped correctly according to the Mapping Table. Final Steps Now that the new geodatabase is complete and loaded, some permissions need to be granted to the users/roles and topology created. For the Landcore geodatabase, users in the editors role will get all privileges. Set these for: LandCoreMaint feature dataset SiteSpec GenPlan OwnID Validation Devcode MarketStat Edits ActivityCenters MPR Landcore uses some topology rules to detect overlapping polygons and gaps between polygons. First, make sure the dataset is not versioned. If it is, unregister it as versioned, by right-clicking on the dataset and selecting Unregister as versioned. Add a topology class to the LandCoreMaint dataset with the following settings: Name: LandCoreMaint_Topology Cluster tolerance: accept the default (it should be ) Landcore feature class participates Rank: 1 Rule: Landcore Must not Overlap Rule: Landcore Must not have gaps Validate the topology. Register the LandCoreMaint feature dataset as versioned. 11

70 Landcore Data Migration Topology PART II, STEP B. CREATE TOPOLOGY AND VALIDATE. After all the data were loaded into the Landcore geodatabase, a topology class was created. This takes place prior to versioning. The following steps were used: 1. Open ArcCatalog 2. Connect to the Landcore geodatabase as the GIS user. 3. Right-Click on the LandCoreMaint feature dataset and select New Topology. 4. Click Next on the first dialog. 5. Accept the default name and cluster tolerance and click Next. 6. Check the landcore feature class and click Next. 7. No ranking was needed in the topology. Click Next on the ranking dialog. 8. Add the following two new rules: landcore must not overlap landcore must not have gaps. 12

71 Landcore Data Migration Topology 9. Click Next. 10. Click Finish. 11. Answer No to validating the topology at this time. Eventually the topology needs to be validated however this can be a lengthy process. Therefore, this process should be started at the end of the day so that it may run overnight. 13

72 Landcore Data Migration Register as Versioned PART II, STEP C. REGISTER AS VERSIONED. For classes in an SDE to be editable, they must be versioned. This applies even if there is no intention of creating multiple versions, as is the case with Landcore. After creating the topology, the feature dataset and all the tables except the vcounter table were versioned. IMPORTANT: the vcounter table should NOT be versioned. The vcounter table is not versioned because editors working on landcore at the same time need to be able to see the very latest values in this table. If it were versioned, the editor would only see the current state of the vcounter table in the version that they are editing and would not see other user s edits until they did a version refresh. The VCounter table is used to keep a running count of the highest LCKEY value. This is very important because it provides a means to keep values in the LCKEY field in the landcore feature class unique. The Attribute Editor handles this and will edit this table automatically when needed. The following steps were used to register the tables as versioned. 1. Start ArcCatalog. 2. Connect to the Landcore Geodatabase as the GIS user. 3. Right-click each table (except vcounter) and select Register as Versioned. 14

73 THE LANDCORE ATTRIBUTE EDITOR December 2005 Revision 1.1 February 2008 Paul Hardwick Steve Kunkel

74

75 TABLE OF CONTENTS INTRODUCTION... 1 SETTING UP A COMPUTER TO ACCESS LANDCORE AND USE THE ATTRIBUTE EDITOR... 1 Setup: Installing the Landcore Attribute Editor... 1 Setup: A Database Connection to Landcore... 2 Setup: Landcore Class Extension... 3 GETTING STARTED... 5 USING THE LANDCORE ATTRIBUTE EDITOR... 8 Introduction... 8 The Landuse Tab The Planned Land Use Tab The Ownership Tab The General Plan Tab The Forecast Tab The Edits Tab The Validation Tab SPECIAL LANDCORE BEHAVIORS THAT AFFECT EDITING PROGRAMMING DOCUMENTATION FOR THE LANDCORE ATTRIBUTE EDITOR Development Environment User Interface Development Programming Code Development Code Organization Special Coding SPECIAL INSTRUCTIONS: HOW TO ADD A CONTROL iii

76

77 Landcore Attribute Editor Setup: Landcore Class Extension THE LANDCORE ATTRIBUTE EDITOR INTRODUCTION The Landcore Attribute Editor, or Attribute Editor for short, was developed as part of SANDAG s strategic move from ArcInfo Workstation to ArcGIS Desktop. The Attribute Editor is Visual Basic and ArcObjects code that runs within ArcMap. It replaces the set of custom AMLs and form menus that formerly were used within ArcEdit to perform editing tasks on Landcore. The Attribute Editor is designed to edit only the Landcore geodatabase, which is served by the SDE server Kona at SANDAG. Some tasks that were formerly handled by the custom ArcEdit forms, such as drawing different layers and changing their symbols, are not part of this Attribute Editor but are handled by standard tools in ArcMap. SETTING UP A COMPUTER TO ACCESS LANDCORE AND USE THE ATTRIBUTE EDITOR There are three things that need to be done to an editor s computer before the Landcore Attribute Editor can be used. These are: 1. Install the Attribute Editor on the computer, 2. Create a database connection to Landcore, and 3. Install the Landcore Class Extension. Instructions for each of these are given on the following pages. Setup: Installing the Landcore Attribute Editor The Attribute Editor must be installed on each computer that will be using it. It is packaged and distributed as a DLL (Dynamic Linked Library), which is a file with a.dll extension. The installation process is as follows: 1. Copy the Landcore.dll file to a local drive on your computer. C:/ArcGIS/addons is the common location that SANDAG has established for this and other ArcGIS custom files. 2. Start ArcMap. 3. Select Customize from the Tools menu. 4. Select the Commands tab. 5. There are two options on where ArcMap will make the Attribute Editor available. a. To make it available in only the currently open map document, select the name of the map document in the dropdown next to Save in: b. To make it always available in ArcMap, select Normal.mxt from the dropdown next to Save in: 6. Click the Add from File button. 1

78 Landcore Attribute Editor Setup: Landcore Class Extension 7. Browse to the location of the Landcore.dll file and select it. 8. Click OK to the small dialog. 9. A new category called SANDAG will be added to the Categories listbox. Select this if it is not selected. 10. A new command Landcore Attributes is added to the Commands listbox. Click on this command to select it. See the figure below. The Customize dialog. 11. Drag this command from the Customize dialog onto an existing toolbar. 12. Click the Close button on the Customize dialog. 13. If you used option 5a above, save your map document. There is no need to run any scripts or to manually register the DLL ArcMap automatically registers the DLL when it is added through the Customize dialog. Occasionally, new versions of the Attribute Editor application will become available as it is enhanced or modified. These also will be distributed as a new DLL file. To install a new update, simply replace the existing Landcore.dll file with the new file. Then, repeat all the steps in the setup process above. Setup: A Database Connection to Landcore In order to view the Landcore geodatabase, the user must first establish a connection to it. Use ArcCatalog to create a new Spatial Database connection with the following properties: 2

79 Landcore Attribute Editor Setup: Landcore Class Extension For User Name use your SDE login name (sandagnet\<login> ) and your password. For added security, do not check the box to save your password. If you do not have a username and password, see your SDE database administrator (John Hofmockel). Setup: Landcore Class Extension The Landcore Class Extension is a small program (developed in Visual Basic) which is independent of the Landcore Attribute Editor but must be installed on each computer that will be used to view Landcore, with or without the Landcore Attribute Editor. Details about this program are discussed in Landcore Class Extension section of the Geodatabase Design document. Without the extension the user will not be able to view the Landcore feature class or any other feature classes within the LandCoreMaint feature dataset because ArcMap or ArcCatalog attempts to load the extension whenever the Landcore feature class is accessed. If the extension is not installed, the user will receive the following message: Installing the Landcore Class Extension The procedure to install the Landcore class extension on a computer is relatively simple and should be within the grasp of most users. To make this as easy as possible a software installer is used. This will copy the necessary files (just one dll file) and do the registration needed for the class extension to work. 3

80 Landcore Attribute Editor Setup: Landcore Class Extension Installation steps: 1. Go to m:\res\gis\landcoresystemlibrary\ where the installer for the SANDAG Landcore Class Extension is located. If needed, extract the contents of the zip file setup.zip. 2. Double-click the file setup.exe. This will start the installer. 3. Click OK. 4. Designate the folder on the computer s local drive by clicking the Change Directory button. Browse to C:\ArcGIS\addons and double-click the LandcoreClassExtension folder or type \LandcoreClassExtension after the C:\ArcGIS\addons path to create the new folder. 5. Click the large icon on the dialog to install the class extension. 6. When finished, click OK. The computer does not need to be restarted. Once installed, the Landcore Class Extension will appear as an installed program under the Add/Remove Programs (Control Panel Add/Remove Programs) with the name SANDAG Landcore Class Extension. Also, if the Developer Kit is installed, the class will appear under the ESRI GeoObject Class Extension COM Component in the Component Category Manager (see below). Uninstalling the Landcore Class Extension To uninstall the Landcore Class Extension, use the Add/Remove Programs windows Utility in the Control Panel. 4

81 Landcore Attribute Editor Getting Started GETTING STARTED Once the Attribute Editor has been setup with ArcMap on the computer, it can be accessed by selecting the tool and clicking on a parcel. The tool will be inactive and appear grayed-out if the current Map does not have a layer called Landcore.GIS.Landcore (case sensitive), which must be the landcore feature class from the Landcore geodatabase. For security reasons, the user must add this layer to their map using their own SDE connection log in. In addition, the user needs to add the following tables from the Landcore geodatabase to their map: ActivityCenters Devcode Edits GenPlan MarketStat MPR Ownid SiteSpec Validation VCounter (All these tables can be added at one time by holding the Ctrl key while selecting them.) When the user applies the tool to a parcel, if these required tables are not loaded, a message similar to the following will appear: 5

82 Landcore Attribute Editor Getting Started and be followed with a further warning: In such cases, the user should immediately close the Attribute Editor and load the appropriate table. Then try using the tool again. Once the dialog opens (with no preceding error messages), the Attribute Editor can be used and will appear as shown below. 6

83 Landcore Attribute Editor Getting Started The Attribute Editor dialog Using the Attribute Editor for Viewing Landcore Attributes The Attribute Editor may be used simply for the purpose of viewing attributes in Landcore, without making any edits to the data. An open edit session on landcore is not required. Steps: 1. Select the tool. 2. The mouse cursor will change to a. This indicates that the tool is the active tool. Click on a parcel (polygon in Landcore.GIS.Landcore). 3. The Attribute Editor dialog will open displaying the attributes for the parcel that was clicked on. 4. Since the polygon is not actually selected or drawn distinctively, the B button can be used to blink the polygon to remind the user of what polygon they clicked on. Clicking the button will cause the feature to blink once. Note that the Attribute Editor dialog can only display attributes for one polygon at a time. The Attribute Editor dialog may be closed at any time or can be left open to perform other actions in ArcMap or to view more attributes of Landcore by applying the tool ( cursor) to other polygons. 5. To close the Attribute Editor dialog, click the Cancel button in the lower right corner of the dialog. 7

84 USING THE LANDCORE ATTRIBUTE EDITOR Introduction The main purpose of the Attribute Editor is to view and edit attributes of the landcore feature class. These attributes are stored in fields within the landcore feature class and in several associated tables within the Landcore geodatabase. There are approximately 90 fields in the Landcore geodatabase that are related to the landcore feature class. Attribute domains and relationship classes are used throughout the geodatabase to manage this data. The Landcore Attribute Editor was created to shield the ArcMap user from having to deal with much of this complexity by providing a way to easily and quickly view relevant attributes for landcore features. For an identified feature, it will perform processing in the background necessary to pull information from a variety of tabular sources and present them together on a dialog to the user in an easily readable form. The user can then change most of these attributes by simply selecting or typing in a new value and clicking the Update Attributes button. Since there are a large number of fields in Landcore, they are organized on the Attribute Editor under different tabs, grouped according to their thematic similarities (Landuse, Ownership, etc) and in order to display information which the user may want to see at the same time (for instance base year landuse and existing landuse). The general logic and thought processes which the SANDAG landcore editors had been using, as they regularly performed maintenance edits on Landcore, were factored into how the field information is arranged and displayed on the Attribute Editor. The tabs along the top of the Attribute Editor form act like turning the page of a book, changing the information displayed on the form as a different tab is selected. The tabs are: Landuse information about current landuse. Planned Landuse information about planned and changing landuse. Ownership information about land ownership. General Plan information from a jurisdictions General Plan. Forecast relevant information summarized for the growth forecast. Edits information about the edits performed on the polygon. Validation query tool for performing cross-field validation. Each of these tabs are described in greater detail later in this document. 8

85 Although the tabs show different information they share some common characteristics. Each tab uses objects called controls to display information from the database fields. There are several types of controls which consistently appear a certain way and act in a certain manner. These are described below. This is a listbox that will highlight the value from a field for the identified parcel. The text above the listbox indicates the name of the field, which is actually the alias name. In this example the Generalized Land Use field is displayed. A listbox will also allow the user to change the selected value by simply clicking on a different value. Only one value may be selected at a time. With the exception of one listbox control on the Attribute Editor, listboxes always list the description from the attribute domain that is used by the field. This is an editable textbox control which will display values from a field in the database and will allow the user to type in a different value. field. This is for display only. This is a textbox control that will only display values from a The field name text next to an editable textbox or a listbox behaves in a certain way to remind the user that they have changed a value in the control. The field name text will turn blue when the user changes a value in a textbox or makes a selection in a listbox, as is shown below. There are a few exceptions to the field name text turning blue, which are explained on a case by case basis later in this document. There are two things that the user should keep in mind when looking at controls and values in the controls. The first is the fact that often an attribute domain is used by a control to translate field values into more descriptive text. For instance, the text JUNKYARD/DUMP/LANDFILL will be shown for Existing Landuse instead of code The second thing to keep in mind is that, as mentioned earlier, field aliases are used on the Attribute Editor instead of the actual field names. Using the example above, Existing Landuse is the field alias for the lu field. We expect that these two facts should not lead to any confusion for the user since, typically, it is the field aliases and the coded domain descriptions that the user sees when identifying a feature or viewing a table. 9

86 There are two direction arrow buttons that appear on several of the Attribute Editor tabs. These are the forward and backward arrow buttons that appear as. These will be positioned next to a textbox which displays values from a field for which there may be more than one value related to the identified feature. Clicking these buttons will show the next or the previous value in the textbox. The bottom portion of the Landcore Attribute Editor has controls that will be visible at all times and thus these controls can be interacted with while the contents of any tab are visible. This portion is shown below: The button used to calculate fields in the Landcore database. After the user has identified a parcel and changed values on the Attribute Editor dialog, clicking this button will recalculate the appropriate fields in Landcore. For efficiency sake, unchanged values are not recalculated in the database. The button is used to blink the identified feature. This will cause the feature to briefly flash, reminding the user which feature they are viewing attributes for. The button is to close the Attribute Editor dialog. The button will open a help file (winhelp *.hlp format). Currently this file does not exist but will be created at a later date. The buttons are used to calculate the field in landcore called split. This field is used to track which polygons have been split into more than one polygon. This is determined by comparing values in the parcelid field between adjacent polygons. Those with the same parcelid are considered split and the split field will be assigned a value of 1. These may be referred to as split parcels. SanGIS parcels can consist of more than one polygon, when for instance they are bisected by a road right-of-way. We do consider these as split parcels as well as the ones that we have split through our editing process. The two buttons perform the split field calculation for a specific set of features, as explained below. = calculate splits for all polygons in the visible extent of the map. 10

87 = calculate splits for the selected polygons. With the implementation of the Landcore Class Extension (developed after the Attribute Editor), the two calculate split buttons are essentially obsolete and should seldom need to be used. This is because, through the Landcore Class Extension, the split field calculation occurs automatically whenever a Landcore feature is created or a polygon is split (which actually is two creates and one delete). The class extension is described in greater detail in the Landcore Class Extension section of the Geodatabase Design document. The button is used to recalculate the LCKEY field. Like the two split buttons discussed above, this tool was developed before the Landcore Class Extension and is now basically obsolete. The Landcore Class Extension contains code that automatically calculates the LCKEY field when needed. Refer to page 31 of this document for a detailed explanation. Only under special circumstances should this button be used. If it is determined that this tool needs to be used (see page 31) then select the feature(s) which need to have a new LCKEY and click this button. The next available unique LCKEY will be assigned to the LCKEY field. If more than one feature is selected, they all will get unique values. The two remaining controls on this bottom portion of the Attribute Editor are radio buttons which are shown below. These two radio buttons can be called the Calculate Mode buttons. They determine which features the Attribute Attributes button will calculate and are thus very important for the user to understand so that they can avoid calculating the wrong features. Only one of these buttons can be turned on at a time. Selecting the Selection button will automatically unselect the ID button, and vice versa. Selecting the ID button will set the mode for the Calculate Attributes button to calculate only the identified feature. The identified feature is the one that will blink using the Blink ( ) button (the feature that was clicked on using the tool). Selecting the Selection button will set the mode for the Calculate Attributes button to calculate all the selected features. These are the features that are selected in the Landcore.GIS.Landcore layer and would normally be highlighted with the layer s highlight color (usually a cyan color). In the button screenshot shown above, there are six selected features. The text following the Calculate Mode buttons will automatically update to reflect the number of selected features in the Landcore layer. 11

88 Landcore Attribute Editor The Forecast Tab The Landuse Tab All the controls on the Land Use tab are for the purpose of displaying or changing values in the geodatabase. They are formatted and function in the standard way described for listboxes and textboxes with a few exceptions which are described below. The Generalized Land Use listbox is associated with genlu field in the landcore feature class. This very important field defines the subtype for a feature. This listbox shows the names of the subtypes, which incidentally are the coded value domain descriptions used by the Generalized Land Use (genlu) field. In the geodatabase data model, subtypes can regulate which domains are used by certain fields. In Landcore, this is particularly true for the Existing Land Use ( lu ) field. In the screenshot above, note that the feature has a Generalized Land Use of Educational. For this subtype, only education-related existing land uses are valid and thus only values from an education existing land use domain are displayed in the Existing Land Use listbox, directly below the Generalized Land Use listbox. This ability of subtypes to control the domains of different fields is modeled in the Attribute Editor by enabling the Generalized Land Use listbox to change values in other controls. By selecting a 12

89 Landcore Attribute Editor The Forecast Tab different value in the Generalized Land Use listbox, thus changing the subtype, the list of choices in the Existing Land Use listbox automatically change. Also, upon doing this, the field name text above the Existing Land Use listbox will become red. The screen shot below shows these two controls when the subtype has been changed by the user. The red text is to notify the user that they need to select a value in the Existing Land Use listbox. In general, it is saying something isn t right here. In a similar way, red text is used elsewhere by other controls on the Attribute Editor to alert the user. In the screenshot above, the subtype was changed from Educational to Public Facilities/Hospital/Cemetery. The user must now select a new Existing Land Use because the existing land use attribute on the feature is no longer valid. The red Existing Land Use text also automatically triggers a change to the Attribute Editor dialog. The icon is drawn just below the Update Attributes button. Since the user is able to go to a different tab, thus hiding the red Existing Land Use text, this icon is to remind them that there is still a control somewhere that needs their attention. They should find this control and do what they need to do before clicking the Update Attributes button. Once the user takes the appropriate action, the icon will disappear. On the Land Use and Planned Land Use tabs there is a small purple arrow button. Generally this button is used to select a value in one listbox that is already selected in another listbox. The arrow points away from the listbox that has the selected value and toward the listbox that will be selected from. When the mouse cursor is hovered over this button, it will pop up a help tip describing specifically what it will do. This arrow button on the Land Use tab, will select the same Base Year Land Use as is selected in the Existing Land Use listbox. 13

90 Landcore Attribute Editor The Forecast Tab Only numbers can be entered into the textboxes in the Units Group shown below. Text, such as two or 20-30, cannot be typed into these unit textboxes. The Planned Land Use Tab All the controls on the Planned Land Use tab are for the purpose of displaying or changing values in the geodatabase. They are formatted and function in the standard way described for listboxes and textboxes. 14

91 Landcore Attribute Editor The Forecast Tab The button selects the same Planned Land Use that is selected in the Existing Land Use listbox (located on the Land Use tab). This calculation has been part of the SANDAG landcore maintenance editing workflow. The Ownership Tab All the controls on the Ownership tab are for the purpose of displaying or changing values in the geodatabase. They are formatted and function in the standard way described for listboxes and textboxes with the following exceptions. When a selection is made from the Existing Ownership listbox, the Owner ID text will change to red, as shown below: As is the convention, red text is used to alert the user that they need to change something in a control. In this case, a new Owner ID needs to be changed to an Owner ID that is compatible with 15

92 Landcore Attribute Editor The Forecast Tab the selected Existing Ownership category. For instance, if a general ownership category for a public agency is selected then the Owner ID should not be a number for PRIVATE (50000). The Owner ID textbox can be edited by the user. This control will dictate what is shown in the Name of Ownership textbox. When the Owner ID is changed, by the user, the Name of Ownership listbox will update appropriately. When a valid owner ID number is entered, the Owner ID text will turn blue (indicating a change). The owner ID numbers and descriptions are taken from the Ownid table in the geodatabase. If the user enters an owner ID that does not exist in this table, the Owner ID text will turn red and the Name of Ownership textbox will appear blank. The Attribute Editor does not have checks to make sure the owner ID is actually compatible with the general Ownership category. It assumes that the user will enter a value that makes sense. To build this into the Attribute Editor would have required additional tables and complexity in the geodatabase. It was determined that this complexity would not be worth the benefit gained. The General Plan Tab 16

93 Landcore Attribute Editor The Forecast Tab The controls on the General Plan tab are for the purpose of displaying or changing values in the geodatabase. About half of these controls are formatted and function in the standard way described for listboxes and textboxes. The remaining controls have special behavior and are described below. To help understand how the controls on the General Plan tab work, it is best to understand something about the structure of the Landcore geodatabase. The main purpose of the General Plan tab is to display information from the GenPlan table that is related to the identified landcore polygon. Landcore features are related to the GenPlan records using the field planid in both tables. Values in this field are formatted in a very specific way, being composed by concatenating values in the following manner: Sphere + spa + gp code = planid Thus the plan id is an attribute for a parcel that has a general plan code that is geographically located within a Specific Planning Area (SPA) (or possibly not in a SPA) which is also geographically located within a specific Sphere of Influence (sphere). It is important to remember that these are actual combinations as they are actually occurring in San Diego County and not just random combinations. Since these are actual, we use the concept of a valid planid. The GenPlan table stores these valid planid codes and information that describes them, such as Sphere, SPA, and gp code as well as a general plan designation and the planned housing densities. Besides displaying field values to the user, the main purpose of the General Plan tab is to change the planid in the landcore feature class. This is done so that the identified feature/parcel will relate to a different record in the GenPlan table. The planid is not changed in the GenPlan table. Since, as already explained, the planid is a composite of Sphere, SPA, and GP code, controls are provided on the General Plan tab to change these parts of the planid. These are the Sphere of Influence listbox, the Specific Plan listbox, and the GP/SPA Code textbox. These are the only controls that can be directly edited by the user. When the user changes the values in any of these three controls, the Plan ID textbox will be updated accordingly. An important additional step is taken to check that the new planid is a valid one (actually occurring). If it is not valid, the text Invalid ID will appear next to the Plan ID. The user should double-check the combination of Sphere, SPA, and GP code and either choose a different combination or speak with SANDAG GIS Staff about adding this code to the GenPlan table. It is intended that the GenPlan table be fairly static, and thus is not directly editable by the Attribute Editor tool. Only staff that understands the GenPlan table should attempt editing it. In addition to what is discussed above, the GP/SPA Code textbox has some other special behavior which will change the values in the Label and Description textboxes. As the user changes the value in the GP/SPA Code textbox, the Label and Description values will also change appropriately. 17

94 Landcore Attribute Editor The Forecast Tab If an invalid GP/SPA Code is entered, which is one that does not actually occur in the selected SPA and Sphere combination, then the Label and Description will become blank and the red Invalid ID text will appear next to the Plan ID, as shown below. The Forecast Tab All the controls on the Forecast tab are for the purpose of displaying values in the Landcore geodatabase. 18

95 Landcore Attribute Editor The Edits Tab The Edits Tab The purpose of the Edits tab is to show the edit history for the identified landcore polygon. This history includes the name of the person who made the last edits, when they were made, the type of edit made, and why the edit was made. The controls on the Edits tab behave slightly different than those on the other tabs and are described in detail below. The Edits tab is divided into two parts. The upper portion, enclosed by the Last Edits box, is used to view the edit history. The lower portion, enclosed by the New Edits Information, is used to add a edit history record for the identified feature. Since a polygon may have had one or several edits performed over the course of its lifetime, there may be one or more records in the edit history table associated with that polygon. To help the user view each of these records in succession, yellow forward and backward arrow buttons are provided in the upper right corner. These buttons will only appear if there is more than one edit history record available. To view the next record click the forward button and to view the previous 19

96 Landcore Attribute Editor The Edits Tab record click the backward button clicked.. They will appear and disappear automatically as they are The New Edits Information portion of the Edits tab is for the user to add an edit history record for the identified polygon. To do so, the user selects a general Reason category and types in some descriptive information about the edit in the Explanation text box. This is limited to 50 characters. This is shown below. The Attribute Editor will automatically use the current date and the user s SDE login to populate the edit date and the name fields in the Edits table. As is the norm with all the other textbox and listbox controls, these changes (new edit history information) will be added to Landcore when the user clicks the Update Attributes button. It is significantly different however, in that it does not update existing records, but rather adds a new record to the Edits table. The Attribute Editor does not have the ability to change an edit history record that has already been entered. 20

97 Landcore Attribute Editor The Validation Tab The Validation Tab The Validation Tab The Validation tab differs fundamentally from the other tabs in that it is not used to view or change attributes in Landcore. This tab is basically an interface for running a utility that can be used to check for conflicting values across different fields in Landcore. We call this Cross-Field Validation. The Cross-Field Validation process will compare values in two fields using a defined rule, and determine if the values are permitted or if they violate the rule. The fields which are compared by the validation may be in the landcore feature class or in tables which are related to the landcore feature class. The existing relationships classes defined in the Landcore geodatabase are used by the Attribute Editor in the validation process to relate records between landcore and the various tables. For easy reference for the user, each rule is given a name which is displayed in the Rule Names listbox. 21

98 Landcore Attribute Editor The Validation Tab Performing a Validation To run a validation, follow these steps. 1. Select a set of parcels to validate. 2. Select the rule to use to validate the parcels with. 3. Click the Validate button. While the validation process is running, the text Running Cross-field Validation will appear in the Violations listbox. The time required for the process to run depends greatly on the number of parcels which are being validated more parcels resulting in longer time. When the validation process is finished, the text in the Violations listbox will change to indicate if the rule was violated. If the rule was not violated, then the text No rules were violated will appear in the listbox and the selection will be cleared. If the rule was violated, then the name of the rule will be displayed in the listbox and the parcels which violated the rule will be selected. Defining the Validation Rules The validation rules are stored in fields in the Validation table. The process to create new rules or alter existing rules involves directly editing records in this table. Specific parts of the rule are stored in different fields and follow some formatting conventions which is illustrated with the following example: Rule Name = Undeveloped land that has employers Rule expression = If [genlu] = 17 in Landcore, then parcels cannot also have [empden] > 0 in SiteSpec This rule is stored in the validation table as: Conventions: Field Rule Name Table1 Value Undeveloped land that has employers Landcore Query1 [genlu] = 17 Table2 sitespec Query2 [empden] > 0 The rule name can be any string. It is used for display purposes only. The rule is always expressed in the negative terms that is if X then parcels cannot have Y. (Positive terms would be if X then parcels must have.y ) The rule can work on no more than two different tables. Compound expressions can be used in a rule so that many fields can be included. (fieldx = A AND fieldy = B and fieldz = C) The word Landcore is used for the landcore feature class (not Landcore.GIS.Landcore). 22

99 Landcore Attribute Editor The Validation Tab The name of the table without the Landcore.GIS, or any other prefix, is used for the name of the table. (GenPlan not Landcore.GIS.GenPlan) Rules can be used to validate values in fields that are in the same table or feature class or in different tables. For example the rules: If baselu = 1100 AND plu = 1200 in Landcore, then parcels cannot also have redevinf <> 8 in Landcore validates fields in the same table, while If empden > 0 in SiteSpec, then parcels cannot also have gplu = 7603 in GenPlan validates fields in different tables. The query expression that is stored in the Query1 and Query2 fields should be strictly formed. A convenient method to form these expressions is to use the Select by Attributes dialog in ArcMap (shown below). The expression can then be copied and pasted into the appropriate field in the Validation table. The expression can be as complex or as simple as needed. It is up to the person making the rule to construct the query expression properly. The Attribute Editor has no checks for this and will throw an error if the expression is badly formed. The Select by Attributes dialog. 23

100 Landcore Attribute Editor The Validation Tab To display the expression to the user, the Attribute Editor will extract the values from the validation table and construct the rule in plain English, inserting the phrase, then parcels cannot also have for readability. This occurs when the user selects a rule from the Rule Names dialog. This way the user can click on each rule name, examining each rule expression, before choosing which one to use. 24

101 SPECIAL LANDCORE BEHAVIORS THAT AFFECT EDITING The Landcore feature class uses a custom class extension to override some default geodatabase behaviors and achieve some custom behaviors. Unless the user is aware of how this affects Landcore, they may end up a little perplexed at times. For instance, the values in some fields will seem to inexplicably change. This section of the Attribute Editor documentation will discuss the Landcore Class Extension in terms of how it affects particular fields and when this occurs. A more in depth discussion of the Landcore Class Extension can be found in the Landcore Geodatabase Design document. LCKEY Field Automatic Calculation The Landcore Class Extension will automatically calculate the LCKEY field in landcore whenever a feature is created. This occurs when new features are added or a feature is split (which is really two adds and one delete). This is to ensure that the LCKEY is unique for all polygons, no longer requiring the user to use the LCKEY button, and to avoid having an LCKEY of 0. A by-product of this is that the LCKEY is also recalculated for related records in the Edits and the ActivityCenters tables. This is important because if the original feature (before the split) was related to a record(s) in either of these two tables, then the table records related to the larger of the two resulting features will also be reassigned the new LCKEY. Thus, the resulting larger feature will continue to be related to records in the ActivityCenters table and/or the Edits tables and the smaller feature will not. If, after splitting a polygon, the ArcMap editor wants the smaller portion, rather than the larger portion, to be related to the ActivityCenter or Edtis record(s), it is the user s responsibility to manually recalculate the LCKEY appropriately. This can be done by simply switching the LCKEY values landcore between the larger and smaller portions, using the Attributes dialog in ArcMap. This is probably the easiest method, instead of changing the LCKEY in the related table records, since possibly more than one table records may be related and thus need to be changed. Split Field Automatic Calculation The Landcore Class Extension will automatically calculate the Split field in landcore whenever a feature is created. This was mentioned earlier in this document when the buttons were explained. This occurs when new features are added or a feature is split (which is really two adds and one delete). In actuality, the only consequence for the user who is performing the edits is that they no longer need to manually calculate the split field. The user may notice that values in this field are changing ( 0 or 1 or 9 ) from time to time. The user should not manually change these values, but rather leave that up to the Class Extension. However, no harm will be done if the user manually calculates the split field. Preserving Key fields on Split operations The Landcore Class Extension has settings that override default geodatabase relationship class behavior which determines how fields that are foreign keys in relationships are maintained when a 25

102 feature is split. The default behavior is for the smaller resulting feature to be assigned a null value in the foreign key field and the larger feature to retain the key value. The Landcore Class Extension overrides this behavior so that both features will retain their original foreign key values. The fields in landcore that are foreign keys are planid, siteid, and ownid. For the editor, this behavior should be transparent, and for most folks it s actually what they expect. However, for those editors that know about and are expecting the default behavior, they should know that the class extension is overriding it. 26

103 PROGRAMMING DOCUMENTATION FOR THE LANDCORE ATTRIBUTE EDITOR Development Environment The Landcore Attribute Editor ArcGIS application was developed with ArcObjects and Visual Basic. It was originally developed in the VBA environment within ArcMap for ease of testing. Later, it was ported over to the Visual Basic 6 development environment so that it could be distributed as a Dynamic Linked Library COM component, which makes for easier updates for the users at SANDAG. While porting to VB6, much of the original VBA code did not require any changes, however the VBA user form had to be completely reconstructed as a VB form, using its own activex controls. In addition, some code had to be moved to different events because we found that some control events differed between VBA and VB6. Some minor language differences had to be dealt with and the logistics of hooking to the application had to be setup. Overall, these coding changes were significant enough to decide to not to continue to maintain this application in VBA. User Interface Development At the initial stages of development, the design and layout of the user interface (the VB Form) was drafted by SANDAG GIS staff using Microsoft PowerPoint. An example is shown below. The development team met with SANDAG to discuss the general functionality that was expected and specifics related to the data in the Landcore geodatabase. At this point in time the major phases of the geodatabase design were complete and a draft geodatabase was available to develop and test against. The development team then developed a prototype of the first tab on the Attribute Editor, demonstrated this, and received comments. Next the remaining tabs were developed and the application was delivered to SANDAG as a custom ArcMap document (the application being within in the VBA environment in ArcMap). 27

104 While testing the application on a pre-production version of the Landcore geodatabase, SANDAG requested various changes to the interface such as the addition of controls or moving controls from one tab to another. Most of these changes were incorporated and frequently delivered to SANDAG as updated ArcMap documents. Programming Code Development Currently the SANDAG GIS staff lacks the training or experience in using the Visual Basic programming language and using ArcObjects, however it is their desire to be able to do the work maintaining the Landcore Attribute Editor. Therefore, from the onset the development team saw the importance of using a coding strategy which would help SANDAG achieve this goal. SANDAG s desire to maintain the application and the simple fact that the original developers of the application are not in-house staff necessitated a programming approach that would make the coding most easily approachable and understandable by the uninitiated or those new to the application. To guide the programming of the Landcore Attribute Editor the development team came up with the following programming strategy which consists of a set of guidelines: I. Anticipate some typical ways the application will need to be altered in the future and structure the coding so that making those changes will be as easy as possible. II. Keep the complexity down or find ways to shelter the developer from having to deal with overly complex routines. III. Include comments and explanation in the code to help the programmer. Other minor points in the programming strategy are: IV. Maximize code reuse. V. Since Landcore is a new GDB, as much as possible, allow for the structure of the geodatabase to change, or at least minimize the impact any changes will have on the code. VI. Use common accepted coding conventions and style (variable names, etc). Code Organization The programming code is stored and organized into the following places: AttributeEditor This is the Class module for the Attribute Editor. All the usual procedures for creating a custom UI control class which implements ICommand and ITOOl class are here. LandUseForm contains the Landcore Attribute Editor form, code associated with the controls on the form, and Landcore-specific procedures. 28

105 This is a form module, which contains the form, the event procedures, and other subprocedures and functions. Special effort was made here to place only code that is specific to the form or specific to Landcore in this module. Utilities contains reusable code functions and procedures that are for accessing various ArcObjects and their properties. This is a very important code module in the application. The code in this module is written generically such that it is not specific to Landcore. The procedures are called throughout the application to perform very common and repeated tasks, such as getting and returning an ArcObject object with a given name. VBUtilities contains code for basic manipulating intrinsic visual basic objects. The intention is that this code module would contain procedures to perform basic Visual Basic tasks. This would not include ArcObjects or anything specific to Landcore. This module has three procedures for sorting a listbox (a complex operation in VB6) and for copying the selection from one listbox to another. moderrorhandler contains code for returning error messages. This code module has one public procedure for returning a descriptive message about the error. This procedure is called by the error handling in all the procedures in the project. ErrorHandling contains code for returning error messages. While porting the application from VBA to VB6, this code module was automatically generated by using the ESRI Interface Implementer add-in to stub-out the procedures for the class. This module should not be necessary since our own moderrorhandler module is used for reporting errors. modwindhelp contains code for invoking the Windows Help. This code module contains a function for opening a Windows Help dialog (*.hlp). This is for the purpose of displaying a Landcore Attribute Editor help file. Special Coding There are a few pieces of information about the programming in the Landcore Attribute which would be helpful for the developer, who is coming to this application for the first time, to know. Although this information is not hidden within the application, and would eventually be discovered by looking through the code, they are simply stated here to save the developer some time. There are some coding techniques that are not necessarily always present in a VB6 ArcMap application and thus can be considered special. Since the Landcore Attribute Editor is a modeless form the IModelessFrame interface is used. In the Form code module, refer to the general declarations, the Frame public function, as well as the ICommand_OnCreate and general declarations in the Class module. 29

106 The problem of making the code in the form s code module aware of the application hook is handled by using an accessor function, which creates an Application property of the form. In the class module, when the command is created (ICommand_OnCreate), the form s Application property is pointed to the application hook. The VB6 project uses the Resource Editor to store the bitmap for the command button. Since the query expressions are different between personal and enterprise geodatabases a modulelevel variable is created in the form s initialize event procedure. Originally this was set automatically, but since this event fires when the class is initialized (due to the necessity of the modeless frame), and if there is no layer called Landcore.GIS.Landcore, an error would be thrown. To get around this, this variable is hardcoded. Therefore, two different DLL files must be compiled, one for personal GDB and one for enterprise GDB. SPECIAL INSTRUCTIONS: HOW TO ADD A CONTROL Adding a control that will simply display values from a field and optionally allow those values to be edited should be a fairly commonplace development task. In keeping with the development strategy guidelines, the Landcore Attribute Editor application was developed in such a way to help streamline this task. This task can easily be accomplished by a Visual Basic 6 / ArcObjects developer. However, it is the hope that, by following the steps outlined below, this task can be accomplished by persons with only some general GIS and development skills. What follows are the outlined steps required to add a control which will display Landcore field values and optionally allow those values to be changed. Very specific coding conventions were used in developing the Attribute Editor to make this possible, with the intention to minimize the amount of code that has to be written. Most of the new code that needs to be created is simply copied from existing code and slightly changed. Steps: 1. Open the LandUseForm form and add a new control to the form. If the control is to display values from a field which has a coded-value domain use a listbox control. If the values should be editable use a textbox control. For display only values use a label control. We advise not copying and pasting a control that is already on the form but instead create a new control from the control Toolbox. Use the same formatting properties used by the same type of control that is already on the form. Set the name property of the control using a suffix to indicate the type of control: txt = Textbox Control lst = listbox control lbl = label control The remainder of the name should be fairly descriptive such as PlanLU indicating planned landuse information. Since often the same information occurs on many of the tabs, and control 30

107 names cannot be duplicated, the control names have an additional suffix indicating the tab that they are on. For instance, lblplanlu_gp is a label control which is on the General Plan tab. 2. Set the Tag property. The tag for the control is a string consisting of 5 possible elements, which are single-word strings. The control s tag property is used by the update and calculate procedures to determine a variety of things, including which values to display, which fields to calculate, and how an associated table relates to the feature class. The syntax for the tag property is: {TABLE} <field name> <domain name> <related table name> <table field name><nocalc option> To number the elements for reference: 1. TABLE or field name 2. domain name 3. related table name 4. table field name 5. NOCALC Element #1 If the control to be added is to display records from a table, such as a lookup table, then this first element should be the word TABLE. This will cause the code to handle this control very differently than it would other listbox controls. In these cases (a listbox to display table records), elements #2 and #3 must be the name of the table and the name of the field in the table. If the control is to simply display field values from a field that is in the landcore feature class, then <field name> is the field in the feature class. If the control is to display values from a field which is not in the landcore feature class, but in a related table, then <field name> is the name of the field that is used to relate landcore to the related table. For instance it may be LCKEY. Element #2 If the field uses a domain, then <domain name> is the name of the domain. Element #3 <related table name> is the name of the related table, and <table field name> is the field in the related table that the control displays. Element #4 31

108 If the control displays values from a field which is in a related table and this field uses a domain, then <domain name> is the name of the domain, otherwise <domain name> is # (used as a placeholder for the 2 nd element). Element #5 In some special cases, such as on the General Plan Tab, some controls are editable, but they should not be used to calculate values in the table fields that they display. For these controls, the 5 th tag string, <nocalc option>, should be set to the word nocalc. Otherwise, the 5 th tag string is omitted. For example, the Specific Plan listbox control on the General Plan tab, has the tag property set to planid LCSpa GenPlan spa. This listbox control displays values from the LCSpa domain. It shows (highlighted) values in the spa field in the GenPlan table, as they are related to the landcore feature class on the planid field. Examples: A textbox which displays values from dwellunits in the landcore Feature class. Tag Property = dwellunits A listbox which displays values from the lu field which has a coded value domains called LCLanduse. Tag Property = lu LCLanduse A label which displays values from edittype in the Edits table. This field uses a domain called LCEdits. The Edits table is related to the feature class on the LCKey field. Tag Property = LCKey LCEdits Edits edittype 3. Edit code in the change event procedure. For most controls this is a relatively short block of code which can be copied from other controls. Special behavior may require more programming. 4. Add the text control that is to be associated with the control you just added. This is the text that is positioned next to the other control which has the name of the field, or describes the control. This must be a label control. Do not copy an existing control, but make a new label control and set its font accordingly. This control must be named in a specific way so that will be associated with the primary control. A suffix of lbl is used to indicate it is a label control. The remainder of the name is the name of the primary control, without the preceeding txt or lst. For example lblgenerallu is the label control that is associated with the lstgenerallu listbox control. 32

109 5. Add a block of code to the ControlUpdater procedure. The ControlUpdater procedure is used by all controls to update their contents. Copy the code from a similar control. For most controls, this is just 6 lines of code. In the procedure the controls are grouped together by the tab that they are located on. 33

110 THE SANGIS/LANDCORE INTEGRATION December 2005 Revision 1.1 February 2008 Paul Hardwick Steve Kunkel

111

112 TABLE OF CONTENTS THE SANGIS/LANDCORE INTEGRATION... 1 Introduction... 1 Description... 1 Model and Scripts Overview... 2 Model and Scripts Detailed Descriptions Prepare New SanGIS Parcel Base Landcore Parsing SG Parse and Attribute Part I SG Parse and Attribute Part II New Parcel Attributes Merge SanGIS and Landcore Features Landcore Preparation Setup New Landcore Training Requirements Issues Gaps and Overlaps from SANGIS-Moved Parcels Stacked Features in the New SANGIS Shapefile (Condos and Timeshares) Large dataset geoprocessing Technical/Software iii

113

114 THE SANGIS/LANDCORE INTEGRATION INTRODUCTION The SanGIS/Landcore Integration is a process to update the Landcore geodatabase with a new version of parcels for San Diego County. For use in the integration process, on approximately a quarterly cycle, SANDAG will obtain new parcels from SanGIS, the agency that maintains a GIS parcels database for San Diego County. The SanGIS/Landcore Integration methodology and GIS tools are presented and explained in this document. The SanGIS/Landcore Integration process is significantly different from the integration process that has been used by SANDAG in the past to update Landcore with new parcel boundaries. Besides the obvious differences in the new ArcGIS environment and the specialized GIS tools, all of these explained in greater detail later in this document, the two integration processes take fundamentally different approaches in how to go about merging the SanGIS and SANDAG parcel layers. The former integration process would involve a staff editor performing tedious manual edits and adjustments to the landcore (SANDAG source) polygon geometry in order to best match any changed line work in the new SanGIS parcels. The new integration process takes an approach that replaces Landcore parcel polygons with new SanGIS parcels polygons. The biggest difference between the two integration processes is how many features are affected in the end result. The former process would result in a fraction of the landcore parcels being modified, while the new integration process changes a much greater number of landcore polygons. The characteristics of the new SanGIS/Landcore Integration process that set it apart from the old SANDAG integration methodology are listed below. The SanGIS/Landcore Integration process is primarily automated using ArcGIS models and Python scripts to process and assemble a large volume of data. has a small amount of manual editing which is performed using the latest ArcGIS tools, including the ArcMap editing environment and geodatabase topology. results in landcore features being spatially coincident with SanGIS parcels. This eliminates the phenomenon of drifting line work seen in the past. DESCRIPTION The SanGIS/Landcore Integration Process is carried out in the ArcGIS Desktop environment using a set of ArcGIS models and Python scripts and followed by some ArcMap editing. Essentially, the models and scripts perform many GIS processing operations, generally referred to as geoprocessing, to combine features and attributes from the new SanGIS parcels and the Landcore parcels, ultimately producing a new version of the Landcore geodatabase. 1

115 The models were developed using the Model Builder environment in ArcGIS. This open development environment is included with ArcGIS and is within the intellectual grasp of most GIS users. Models consist of a sequence of GIS procedures that are graphically arranged and thus are naturally well documented showing the process as a flow of data as it passed through various GIS procedures. The self documenting nature of models and their accessibility to a larger number of users played a part in deciding to use ArcGIS models instead of developing in Visual Basic and ArcObjects. The SanGIS/Landcore Integration process uses the standard tools in the ArcGIS 9.0 toolbox and some custom Python scripts. The Python scripts were developed in order to overcome some technical challenges and to provide more efficient means to accomplish a task. These scripts are implemented as tools in the models. The entire SanGIS/Landcore Integration Process is encapsulated as a set of interrelated models, each of these processing and producing output data that is in turn used as input for another model. The entire SanGIS/Landcore Integration Process is shown, with each connected model and the flow of data, in the diagram on the following page. Each of the large, numbered boxes on the diagram corresponds to a model. The GIS processing operations are depicted in a generalized way on this diagram, with the details being available in the documentation for each model. The set of models and scripts, comprising the SanGIS/Landcore Integration, were developed, tested, and refined on a subset of the Landcore geodatabase that was approximately 1/10 the size (number of features) of the entire landcore feature class. In development, every effort was made to avoid using spatially-intensive operations, such as GIS overlays, knowing that the county-wide landcore feature class has over 800,000 features. Nevertheless, we have found new technical challenges when attempting to run these same procedures on the entire landcore feature class. Diagrams of each model can be found in Appendix F. The names of each procedure in the Overview diagram will correspond to the name of the model in the Appendix. MODEL AND SCRIPTS OVERVIEW The SanGIS/Landcore Integration Process Overview diagram on the following page illustrates graphically how the overall process is composed of many different parts, each of which are arranged in a sequence and are interrelated through the sharing of data. Note the legend box in the lower right of the diagram. Each of the large boxes represents a model, with its name appearing near the top of the box. The sequence of the models is shown by the number preceding the model s name and by the heavy blue lines running from one model to the next model in the sequence. Black lines running from a data box in one model to a process box in another model shows how data flows out of one model and into another. 2

116 3

117 The SanGIS/Landcore Integration process contains the following models and Python scripts. Models Prepare New SanGIS Data Landcore Parsing SG Parse and Attribute Part 1 SG Parse and Attribute Part 2 New Parcel Attributes Merge SanGIS and Landcore Landcore Preparation Setup New Landcore Python Scripts HideAllFieldsExcept.py (the KeepFields tool ) UnjoinNOutput.py (the Unjoin and CopyFeatures tool) JoinNCleanFields.py (the Clean Add Join tool) reindex.py (the Reindex tool) FieldToField.py (the Calculate Field to Field tool) getmaximumvalue.py (the Get Max and Calc tool) Since the models have input and output parameters, it is possible to create a single model that runs all eight of the SanGIS/Landcore Integration models. However, we have had better success running each model individually. Shown below is a model that is configured to run the first six models, each tied together via their input and output parameters, with the temporary sg_b feature class being the exception. 4

118 A discussion of the general approach of the SanGIS/Landcore Integration process, and the methodology used to implement this approach, is helpful in that it provides the basis for the detailed geoprocessing steps within the individual models. The overarching approach in the SanGIS/Landcore Integration process is to use the geometry from all the SanGIS parcels to construct a new version of landcore, transferring over the attributes from the prior version of landcore. The goal is to completely update the geometry in landcore so that it matches the geometry in the SanGIS parcels. However, some of the landcore parcels have been split by SANDAG, based on land use or other considerations and to replace these split parcels would lose the split. Therefore a compromise is reached in which the geometries of only the non-split parcels are updated. The geometries for the non-split parcels in landcore are replaced with the geometries of SanGIS parcels and the geometries of the remaining split parcels are retained and carried through the end result. To bring this about, geoprocessing steps are used to separate features from landcore and the SanGIS parcels into temporary feature classes, based upon whether or not they are split or correspond to landcore split parcels. Also, new feature classes are created from the SanGIS parcels based upon whether or not the features are new (have a new parcelid) or if they previously existed. The five feature classes listed below are created, with each of these undergoing different geoprocessing steps. Ultimately, these four feature classes are merged together to form a new landcore feature class. 1. Coastal landcore polygons (these have a negative LCKEY attribute) 2. Split landcore parcels (based on the split field) that have a corresponding feature in the SanGIS parcels. 3. New SanGIS parcels. 4. Previously existing SanGIS parcels. Also split landcore parcels that have been replaced with new SanGIS parcels are identified and saved into a temporary feature class. These features are examined later in ArcMap editing sessions. In the modeling diagrams these are called orphan splits. Each of the four feature classes listed above are shown on the Overview Diagram with heavy outlines. The combination of values from the parcelid and apn8 fields is used in the process as a unique identifier. This is used to associate the incoming SanGIS parcels with the existing Landcore polygons to determine which SanGIS parcels are new and which previously existed. SanGIS parcels that do not have a matching Landcore parcel (based on parcelid and apn8) are considered new. The parcelid field is maintained by SanGIS with the intention to uniquely identify each parcel, however we found there to be exceptions, which is why the combination of the parcelid and apn8 field was used. 5

119 The models and scripts are stored on the SANDAG file server under M:/RES/ArcGIS/ArcGISTesting. Python scripts can be found in the scripts folder. ArcToolbox tools developed to run these scripts are found in the Scripts Tools toolset within the Landcore Tools toolbox. The models for the SanGIS/Landcore Integration are within the SanGIS Update Tools toolset. MODEL AND SCRIPTS DETAILED DESCRIPTIONS 1. Prepare New SanGIS Parcel Base A diagram of the Prepare New SanGIS Data model is printed on page 36 in Appendix F, with its associated Python scripts. Two shapefiles were downloaded from SanGIS prior to running this model. The shapefiles are the right_of_way.shp and the parcels.shp, which form the basis for the update. This is the first model of the SanGIS/Landcore Integration process. The purpose of this model is to format the new files acquired from SanGIS in a way that is more compatible with Landcore, and for further processing. Also the feature dataset lcupdate is created as a temporary container for input/output feature classes for the SanGIS/Landcore Integration models. Then, the new road rightof-ways and the parcels are combined into a single feature class. Temporary fields are added to track the apn and the parcelid. An important attribute in the overall integration process is the combination of values from the parcelid and apn8 fields. This unique combination is used to relate features in Landcore with features in the incoming SanGIS parcels. The table below lists the input, output data and the Python scripts used in the Prepare New SanGIS Data model. Input K:\box1\gis\parcel\right_of_way.shp K:\box1\gis\parcel\parcels_all.shp Output lcupdate/landcore.gis.parcelsall Scripts HideAllFieldsExcept.py (the KeepFields tool) 6

120 2. Landcore Parsing A diagram of the Landcore Parsing model is printed on page 39 in Appendix F, with its associated Python scripts. The main purpose of the Landcore Parsing model is to create three feature classes, each of these being treated differently in the SanGIS/Landcore integration process. Coastal polygons in landcore are selected based on an LCKEY of < 0 (negative number). Landcore features that are split (split field = 1) and have corresponding parcels in SanGIS are identified, by relating the apn8 and parcelid in the two, and are saved to lcsplit_ok. Landcore split parcels that have no corresponding SanGIS feature are identified and saved to lcsplit_orp (Orphaned Splits). The orphaned split parcels cannot ultimately be used to construct the new Landcore feature class, since new parcels from SanGIS will replace them. So that the integration of the new parcels does not result in a loss of splits, these features are saved to a feature class which will be manually examined at the end of the integration process. If necessary, these splits will have to be re-edited back into the new landcore. Since the likelihood of splits occurring where SanGIS has created new parcels is low, the effort to manually edit in these splits back into landcore should be minimal. ArcMap tracing and other sketch tools should help streamline this task. Some specialized Python scripts are used. The script run by the Clean Add Join tool is used to create joins in which the field names in the resulting layer are not prefixed with the geodatabase and owner name (Landcore.GIS). This is so that when the layer is output to a feature class, the original field names will be preserved. This tool is necessary to overcome a persistent obstacle of changed field names that frequently appears in other models and to overcome the 31-character limit to field names in SDE geodatabases. The second script, which is run by the UnJoin and Copy Features tool, is used to remove a join from a layer and save the layer as a feature class. The reason behind this tool is one of convenience as it combines two steps into one. The table below lists the input, output data and the Python scripts used in the Landcore Parsing model. Input Landcore.GIS.Landcore Landcore.GIS.cnparcel Output lcupdate/landcore.gis.lcsplit_orp lcupdate/landcore.gis.lc_coast lcupdate/landcore.gis.lcsplit_ok Scripts UnjoinNOutput.py (the Unjoin and CopyFeatures tool) JoinNCleanFields.py (the Clean Add Join tool) 7

121 3. SG Parse and Attribute Part I A diagram of the SG Parse and Attribute Part I model is printed on page 46 in Appendix F, with its associated Python scripts. The main purpose of this model is to create a feature class of SanGIS parcels that have the parcel geometry that will be used to construct the new landcore feature class. For simplicity sake, these are referred to as the good SanGIS parcels. These parcels are those that do not match the landcore split parcels in lcsplit_ok. First, a join between lcsplit_ok and cnparcel and a reselect identifies which features are present in both, and then the inverse is selected from cnparcel which is then output to the sg_b feature class. The ReIndex python script tool is used update the indexes (tables) which reside in the SQL Server geodatabase in order to make reselects more efficient. This is needed when the old indexes, which were based on the whole dataset, become stale. The table below lists the input, output data and the Python scripts used in the SG Parse and Attribute Part I model. Input lcupdate/landcore.gis.cnparcel lcupdate/landcore.gis.lcsplit_ok Output lcupdate/landcore.gis.sg_b Scripts UnjoinNOutput.py (the Unjoin and CopyFeatures tool) JoinNCleanFields.py (the Clean Add Join tool) reindex.py (the Reindex tool) 4. SG Parse and Attribute Part II A diagram of the SG Parse and Attribute Part II model is printed on page 49 in Appendix F, with its associated Python scripts. The SG Parse and Attribute Part II model has two purposes. The first purpose is to identify which of the good SanGIS parcels previously existed and which are new, and to create two new feature classes based on this. The sg_same feature class has the good SanGIS parcels that are not new and the sg_new feature class has the new good SanGIS parcels. The second purpose of the model is to finalize the sg_same feature class by attaching the Landcore attributes to it. The table below lists the input, output data and the Python scripts used in the SG Parse and Attribute Part II model. Input lcupdate/landcore.gis.sg_b Landcore.GIS.landcore 8

122 Output lcupdate/landcore.gis.sg_same lcupdate/landcore.gis.sg_new Scripts UnjoinNOutput.py (the Unjoin and CopyFeatures tool) JoinNCleanFields.py (the Clean Add Join tool) 5. New Parcel Attributes A diagram of the New Parcel Attributes model is printed on page 50 in Appendix F, with its associated Python scripts. The purpose of this lengthy model is to attach landcore attributes to the new good SanGIS parcels. This is done with a commonly-used sequence of a point-in-poly overlay and table join. Essentially, by intersecting sg_new centroid points with landcore, the underlying landcore attributes are identified and then later joined to the sg_new polygons. An important step at the end of the model uses the Python script getmaximumvalue.py to determine and calculate the unique LCKEY values in these new features that will ultimately be added to the new landcore feature class. The fieldtofield.py Python script is used as an efficient means to calculate several fields in one step. The table below lists the input, output data and the Python scripts used in the New Parcel Attributes I model. Input lcupdate/landcore.gis.sg_new Landcore.GIS.landcore lcupdate/landcore.gis.mgra (polygons) Output lcupdate/landcore.gis.sg_newparcels Scripts FieldToField.py (the Calculate Field to Field tool) getmaximumvalue.py (the Get Max and Calc tool) 6. Merge SanGIS and Landcore Features A diagram of this model Merge SanGIS and Landcore is printed on page 54 in Appendix F. The main purpose of this model is to create a new version of the landcore feature class by combining the feature classes that have been created up to this point by the models. These feature classes contain some features of the prior landcore feature class and some features of the new SanGIS parcels, based on specific selection criteria and geoprocessing steps. It is important to recognize that the features across these four feature classes do not significantly overlap. More 9

123 accurately, features (specific parcels) are not duplicated across these four feature classes. This being the case, a simple append operation is used to combine the features by loading them into the new_landcore feature class. The table below lists the input, output data and the Python scripts used in the Merge SanGIS and Landcore model. Input lcupdate/landcore.gis.lcsplit_ok lcupdate/landcore.gis.llc_coast lcupdate/landcore.gis.sg_same lcupdate/landcore.gis.sg_newparcels Output lcupdate/landcore.gis.new_landcore Scripts none 7. Landcore Preparation A diagram of this model Landcore Preparation is printed on page 55 in Appendix F. The main purpose of this model is to replace the old version of the landcore feature class with the new landcore feature class. Before deleting the old landcore, it is copied into a temporary dataset. Some Boolean dependencies ensure that the copy operation was successful before the old landcore is deleted. The table below lists the input, output data and the Python scripts used in the Landcore Preparation model. Input LandCoreMaint/Landcore.GIS.landcore (existing) Output LandCoreMaint/Landcore.GIS.landcore (new) Scripts none 8. Setup New Landcore A diagram of this model Setup New Landcore is printed on page 56 in Appendix F. The finishing touches are put on the landcore feature class in this final model in the SanGIS/Landcore Integration process. A new topology class is created and configured. Before running this model, the user needs to manually reapply the schema in order to recreate the relationship classes, which were automatically deleted when the old landcore was deleted in the 10

124 prior model. Instructions for reapplying the schema can be found in the Landcore Geodatabase Design document. The table below lists the input, output data and the Python scripts used in the Setup New Landcore model. Input LandCoreMaint/Landcore.GIS.landcore Output LandCoreMaint/Landcore.GIS.landcore Scripts none TRAINING REQUIREMENTS Using and maintaining the models in the SanGIS/Landcore Integration process comes down to a question of training and skill level. Although Model Builder is a standard part of ArcGIS 9.0, it is still fairly new and not many GIS analysts use it regularly. Of the current GIS staff at SANDAG, one person has the skills necessary to run and to make changes to the models and Python scripts. It is recommended that several other GIS analysts receive Model Builder and Python training and, more importantly, spend time gaining experience in building models. Also, it will be very important for them to explore the contents of the SanGIS/Landcore Integration models to better understand the GIS operations employed in these. Some creative and innovative approaches were used in these models (these not being covered in training materials) to overcome software limitations or to improve efficiency. It is not the intention of this document to provide a step-by-step explanation of how to run and modify the SanGIS/Landcore Integration models, as that would ultimately amount to Model Builder and Python training. Rather, the following sources of training are recommended: ArcGIS 9.0 Documentation (available as PDF download) Geoprocessing in ArcGIS Writing Geoprocessing Scripts with ArcGIS Training Courses ESRI Virtual Campus: Geoprocessing with ArcGIS 9 for ArcInfo Published Books Learning Python, 2 nd ed, O Reilly Media Inc Python Pocket Reference, O Reilly Media Inc 11

125 ISSUES Gaps and Overlaps from SANGIS-Moved Parcels One of the tasks that SanGIS performs on the parcels as they maintain the parcel base for San Diego County, is to make minor spatial adjustments to the parcel geometries. In doing this new parcels are not created thus the parcelid and APN remain the same. Furthermore, SanGIS does not track, via a polygon attribute, which parcels have been adjusted since the last parcel release. Therefore, the adjusted parcels cannot be readily identified by SANDAG. Following the SanGIS/Landcore Integration process, the new version of landcore is essentially a composite of geometries from SanGIS and from Landcore; split parcel geometries coming from landcore and the remainder coming from the SanGIS. In instances where a spatially-adjusted parcel (adjusted since the last integration process) is directly adjacent to a parcel that originated from landcore (a split parcel), gaps and overlaps can occur. Landcore geodatabase topology is used to identify these as topology errors. The final steps in the SanGIS/Landcore integration process are manual ones, in which the user uses ArcMap to locate these errors and systematically correct them. The likelihood of these topology errors occurring is low and initial testing on a sample area approximately 1/10 th the size of the full landcore confirmed this. One other factor that may influence how the SanGIS parcel geometry meshes with the landcore parcel geometry is that of spatial reference, but more specifically spatial precision. Much forethought was put into this in designing the landcore geodatabase; and at this time the landcore spatial reference is configured to be the same as in the SanGIS parcel maintenance geodatabase. It is expected that by using the same spatial reference, which includes precision parameter, feature misalignments will be avoided. Stacked Features in the New SANGIS Shapefile (Condos and Timeshares) Occasionally SanGIS changes their data model for various reasons. SanGIS has adopted a new data model that uses a one-to-one relationship between features and APN. This means that certain features are multi-part shapes and some shapes may be overlapping. The overlapping shapes, or stacked features, can be seen in instances where a single polygon represents multiple condominium and timeshare parcels. A parcel is defined as a geographic unit that has one title of ownership. Multi-part shapes can be seen where a parcel is bisected by another feature such as a road right-of-way. Coincidentally, this data model change occurred after the SanGIS/Landcore Integration models were created. Obviously, the models have to make assumptions about input data consistently being in a certain state. Therefore, some additional development is needed to create methods and/or tools to accommodate this data model in Landcore. Large dataset geoprocessing Technical/Software Currently with ArcGIS there are known issues with performing GIS overlay analysis operations on very large sets of data. As a workaround, ESRI has developed a set of tools and python scripts that 12

126 are specifically designed to be capable of executing spatial overlays with large datasets. These can be downloaded from The full landcore feature class, at over 800,000 features, can be considered a large dataset. While knowing about the large overlay issue and for performance considerations, every effort was made to avoid including spatial overlays in the SanGIS/Landcore Integration models. However, a point-inpoly Identity (probably the most benign of all) was found to be necessary and is therefore part of the New Parcel Attributes model. All models were successfully tested against a sample landcore database that was approximately 1/10 th the size of the full landcore. The large overlay tools were not used in this testing; but it is anticipated that they will be required in order to run the model on the full landcore. There have been mixed reviews from users who have tried using the large overlay tools. 13

127 THE GENERAL PLAN ALL PROCESS December 2005 Revision 1.1 February 2008 Paul Hardwick Steve Kunkel

128

129 TABLE OF CONTENTS THE GENERAL PLAN ALL PROCESS... 1 INTRODUCTION... 1 GROWTH FORECAST INPUTS... 2 PROCESS OVERVIEW... 3 PROCESS DETAILS... 4 iii

130

131 THE GENERAL PLAN ALL PROCESS INTRODUCTION The General Plan All process, henceforth called the GPALL process, is the set of GIS processing steps and procedures that are used to create the inputs for the SANDAG growth forecast modeling. The GPALL process takes the form of a set of ArcGIS Model Builder models and Python scripts, which were developed and tested in house by SANDAG staff, and are described in detail in this document. The GPALL process creates a new feature class called LCGpall through a series of overlays and calculations using constraints, general plan information, and employment information. Various constraints GIS layers are used to calculate the maximum percent constraint for each landcore polygon. Information from general plans and the Landcore feature class are used to find parcels that are vacant and developable. Also, employment information from the State of California Employment Development Department is attached to LCGpall. Each of these overlay and calculation processes are contained within models which are generally described in the following pages and can also be found in Appendix E. 1

132 GPALL Process: Growth Forecast Inputs GROWTH FORECAST INPUTS The following table lists the fields that are output by the GPALL process to support SANDAG growth forecast modeling. The fields which are generated by the GPALL process are shown in bold type. The growth forecast modeling process will access these fields through queries on the LCGPALL feature class and the SiteSpec table found in the Forecast geodatabase. Attribute Source Table Source GDB Destination GDB & Table lckey landcore Landcore Forecast:LCGpall baselu landcore Landcore Forecast:LCGpall plu landcore Landcore Forecast:LCGpall planid landcore Landcore Forecast:LCGpall redevinf landcore Landcore Forecast:LCGpall siteid landcore Landcore Forecast:LCGpall phase landcore Landcore Forecast:LCGpall baseown landcore Landcore Forecast:LCGpall mktstat landcore Landcore Forecast:LCGpall dwellunits landcore Landcore Forecast:LCGpall Office landcore Landcore Forecast:LCGpall commercial landcore Landcore Forecast:LCGpall industrial landcore Landcore Forecast:LCGpall sgra landcore Landcore Forecast:LCGpall scenarioid landcore Landcore Forecast:LCGpall sphere genplan Landcore Forecast:LCGpall loden genplan Landcore Forecast:LCGpall hiden genplan Landcore Forecast:LCGpall pctconstrained consttable Forecast Forecast:LCGpall empciv emptable Forecast Forecast:LCGpall empmil emptable Forecast Forecast:LCGpall acres calculated Forecast Forecast:LCGpall siteid sitespectable Landcore Forecast:SiteSpec sitename sitespectable Landcore Forecast:SiteSpec sqft sitespectable Landcore Forecast:SiteSpec empden sitespectable Landcore Forecast:SiteSpec civemp sitespectable Landcore Forecast:SiteSpec milemp sitespectable Landcore Forecast:SiteSpec sfu sitespectable Landcore Forecast:SiteSpec mfu sitespectable Landcore Forecast:SiteSpec mhu sitespectable Landcore Forecast:SiteSpec civgq sitespectable Landcore Forecast:SiteSpec milgq sitespectable Landcore Forecast:SiteSpec 2

133 GPALL Process: Process Overview PROCESS OVERVIEW An overview of the GPALL process can be described by showing the flow of GIS data and the GIS processing that is performed on that data. This is shown in the GPALL Process Overview diagram below. 3

134 GPALL Process: Process Details PROCESS DETAILS This section discusses each step in the GPALL process in greater detail. As this section is read, the reader should refer to the individual model diagrams and scripts found in Appendix E. Step: Create the Constraint Composite This step combines all the constraint layers into a single composite layer which is used in the steps that follow. This process has not been automated in a model or script. In general, the process is to use the ArcToolbox Union tool to combine the different constraint feature classes except for slope. Since the Union tool creates multipart shapes, the multipart shapes are next exploded to single parts so that it can be used in the Large Overlay Tools. The slope constraints are combined with the constraints composite using the Large Overlay Union Tool. This also results in multipart polygons that are exploded into single part polygons using the Multi to Single Part Tool. Inputs: Many constraints shapefiles and coverages. Output: Forcast.GIS.ConstAll Step: Create the Developable Lands Feature Class In this relatively short step, a new feature class is created by selecting polygons out of the landcore feature class which represent vacant, developable lands. The SQL query expression below is used to select the features based upon their existing land use at the time of the forecast s baseline year. ((BASELU = 9101) AND ( PLU <> 7603 AND PLU <> 7609)) OR (BASELU = 1000 AND PLU <> 1000) OR (BASELU = 2201 AND PLU <> 2201) OR (BASELU = 2301 AND PLU <> 2301) OR (BASELU = 4114 AND PLU <> 4114) OR (BASELU = 7204 AND PLU <> 7204) OR ((BASELU >= 8000 AND BASELU <= 8003) AND (PLU < 8000 OR PLU > 8003)) This step is shown in the GPALL Overview diagram as the Vacant Developable Lands box. Model: createlcvacdev Inputs: Forecast.GIS.Landcore Output: Forecast.GIS.lcVacDev Fields Added: pctconstrained Fields Calculated: none 4

135 GPALL Process: Process Details Step: Combine Constraints with Vacant Developable Lands The purpose of this step is to attach the constraints fields with each vacant developable land polygon. In this step, the vacant developable lands (lcvacdev) are intersected with the constraints composite (ContsAll) to produce lcvacdevconst. This step is represented in the GPALL Overview diagram as the DL + Constraints box. Model: createlcvacdevconst Inputs: Forecast.GIS.lcVacDev Forcast.GIS.ConstAll Output: Forecast.GIS.lcVacDevConst Fields Added: none Fields Calculated: none Step: Create the Landcore features with information for the Growth Forecast modeling. This step in the GPALL process is the longest and most complex, involving some specialized processing which uses python scripts. Refer to the model diagram creategpallfc and associated python scripts in Appendix E. The purpose of this step is to use the constraints information, which has previously been joined to the developable lands features, to calculate the greatest constraining factor. For each landcore polygon, which may have many constraints on it, the maximum constraining factor is identified and the portion (area) of the feature is output to a table. The python script percentconstrained is used to very efficiently make this calculation, which can be found in Appendix E. The table is then joined to a copy of the landcore feature class. This step is represented in the GPALL Overview diagram as the % Constrained Calculations box. Model: creategpallfc Inputs: Forcast.GIS.lcVacDevConst Developable land with constraints fields Forecast.GIS.lcVacDev Developable land Forecast.GIS.Landcore Landcore feature class for forecast baseline. Forecast.GIS.GenPlan Table with general plan codes and their levels of constraint for each different constraining factor. Output: Forecast.GIS.lcGpall Fields Added: acres Fields Calculated: pctconstrained, acres 5

136 GPALL Process: Process Details Step: Get Employment Information This final step in the GPALL process is used to attach employment information to the landcore features. This is done by performing a spatial identity overlay with the employment points onto the lcgpall from the prior step. Civilian and military employment information is then assigned back to lcgpall and calculated into permanent fields in the feature class. This step is represented in the GPALL Overview diagram as the Employment box. Model: empptsassignment Scripts: KeepFields TwoPointFix empptsassignment Inputs: Forecast.GIS.lcGpall Output: Forecast.GIS.lcGpall Fields Added: empmil, empciv, Fields Calculated: empmil, empciv 6

137 THE LANDCORE SCENARIO ENVIRONMENT December 2005 Revision 1.1 February 2008 Paul Hardwick Steve Kunkel

138

139 TABLE OF CONTENTS INTRODUCTION... 1 Terminology... 1 Mission Critical Goals... 1 Non-Mission Critical Goals... 2 GENERAL APPROACH... 2 GIS CONFIGURATION... 3 Privileges... 3 Posting and Reconciling... 3 Versioning Tree... 4 THE GROWTH FORECAST WORKFLOW... 6 Landcore Maintenance Process... 7 Creating the Forecast Geodatabase... 7 Create a New Scenario... 7 GPALL Process... 7 Growth Forecast Modeling... 8 WORKING WITH SCENARIOS... 9 Creating a New Scenario... 9 Connecting To and Modifying a Scenario Deleting a Scenario Combining Features from different Scenarios CONFLICT DETECTION iii

140

141 THE LANDCORE SCENARIO ENVIRONMENT INTRODUCTION The Landcore database includes baseline information on essential land use features (i.e., existing land use, planned land use, residential densities, and generalized land ownership) that are used by growth forecasting and transportation modeling. The information included in Landcore is collected from established data sources and adopted plans, and represents SANDAG s best attempt to accurately reflect the existing and planned land use conditions. SANDAG business processes often require the baseline Landcore data be adjusted or overridden in order to run alternative modeling scenarios. Modeling scenarios take a variety of forms at SANDAG and are implemented for different purposes. A couple of examples may include: A growth forecast scenario uses an alternative pattern of planned land uses and higher residential densities clustered along transit routes, than those in the current General Plan. A growth forecast scenario assumes the lower range of residential densities within the City of Poway rather than the mid range. A transportation modeling scenario uses a detailed site plan for a proposed future development. Terminology The following are terms as they apply to the scope of Landcore Scenario Environment. Landcore The existing parcel-level SANDAG geodatabase, served as service 5151 by the Kona SQL Server. This region-wide database contains important information to support growth forecasting and transportation modeling and is continually edited and updated by SANDAG GIS staff. Scenario An alternate version of the Landcore geodatabase, which includes all spatial and nonspatial data tables. This alternate version may differ from the baseline landcore in large or small ways. Mission Critical Goals A Scenario should: accommodate changes to attributes in Landcore accommodate changes to the geometry (parcels) in Landcore be able to exist over a period of years work within the limits of a finite data storage space and backup system 1

142 Non-Mission Critical Goals have a clear system to manage and keep track of different scenarios GENERAL APPROACH Scenarios will be implemented by using Versioning, which is an editing environment made possible by the Spatial Database Engine, or SDE, enterprise geodatabase server. In the versioning environment, separate versions are created which can be described as: an alternative, independent, persistent view of the database that does not involve creating a copy of the data and supports multiple concurrent editors. SANDAG uses Versioning such that a version represents a Landcore Scenario. In a sense, the terms version and scenario can be used synonymously and there is a one to one correspondence between the two. A version is the GIS data for the Scenario. There are several advantages with using SDE versioning to create Landcore scenarios. Versioning: will leverage new available ArcGIS technology which is already in use at SANDAG. will require no custom programming to create, modify, and manage scenarios. includes capabilities to limit user access to each version. has native capabilities to store descriptive information about the version. It may be argued that the concepts behind SDE Versioning technology are complex and require more training for users. This is only partly true. Compared to alternative approaches that require all users to use special custom tools and follow a set of procedures to create, modify, and manage Landcore scenarios, the SDE Versioning approach is much simpler and more robust. In addition, the underlying complexity that is internal to SDE Versioning is actually exposed to a smaller set of users; these being the SDE database administrator and SANDAG GIS technical staff. As far as the end users at SANDAG are concerned - the transportation planners and growth forecast modelers - they only need to learn how to use a relatively simple set of ArcGIS versioning tools, as they create and manage their own versions. The basic task of making a unique land use scenario is accomplished by making edits to a version using the standard set of ArcMap editing tools. These tools are discussed later in the Working with the Forecast Geodatabase section of in this document. 2

143 GIS CONFIGURATION Scenarios are implemented in a special geodatabase called Forecast. This geodatabase contains a copy of Landcore, with all feature classes, tables, and relationship classes as they existed in Landcore at the time the latest Regional Growth Forecast was conducted. Since Forecast is a copy, created on a specific date, it represents a snapshot in time and can be referred to as the Landcore baseline. The baseline concept is a familiar one for Growth Forecasting at SANDAG and fits well into their current workflow. One of the characteristics of SDE Versioning is that all object classes (feature classes and tables) which are registered as versioned automatically participate in all versions in the geodatabase. All tables are registered as versioned in Forecast, with the exception of the Vcounter table. It is important to recognize and understand that, as viewed through each version, the object classes (tables and feature classes) are represented according to the changes that have been performed on each. These changes may be both spatial, as when features are added, moved, or deleted, or tabular, as when values are changed in tables. Thus, the landcore feature class may show changes to its geometry and/or changes to values its related tables. Connecting to the Forecast geodatabase is done by establishing a database connection using ArcCatalog or ArcMap. It is served by the Kona SDE server as SDE Instance Privileges All SANDAG staff who need to work with the Forecast geodatabase use SDE user accounts which derive database privileges from a specific user SQL Server role. This role has been granted full privileges to all object classes in the Forecast geodatabase.. The originator of the version grants other users access to the version. This should be as open as possible and should be set to select, insert, update, and delete. Posting and Reconciling Posting is the process that migrates the edits in a version (adds and deletes) to the parent version. Although the parent version can be another version, most of the time it will be the default version in the Forecast geodatabase. Posting to the default version will make the default and the version show the same information. In light of the way SANDAG uses SDE Versioning to create and manage land use scenarios; posting to the default version is not undesirable and should never be done. In order to keep each scenario unique, Posting should not be performed on the Forecast geodatabase. 3

144 Reconciling is the process that moves changes from a parent version into the edit session of a child version. In light of the way SANDAG uses SDE Versioning to replicate scenarios, it may be necessary at times to reconcile a version in order to move changes from a parent version/scenario into a child scenario. As shown in the figure below, the transportation center polygons in Scenario #1 may need to be moved into Scenario #2. One way to do this is to create Scenario #2 based upon Scenario #1. This is shown in the diagram below. If the transportation centers were present in Scenario #1 at the time when Scenario #2 was created, then they will be present in Scenario #2. If not, then reconciling Scenario #2 with Scenario #1 will move the transportation center into Scenario #2. An example of scenarios and their relationship to one another and to the default version is illustrated in the following figure. Forecast GDB Save Add Major Transportaion Center Scenario #1 Create Version Default Version Start Editing Create Version Create Version Save Save Change GP High Densities Start Editing Scenario #2 (child of #1) Has Transportation Centers and Changed High Densities Scenario #3 Start Editing Change Landuses Figure 1. Different versions in the Forecast Geodatabase Versioning Tree A multiple tier versioning tree is used in the Forecast geodatabase, which gives the user the flexibility to be able to: maintain persisting scenarios create scenarios based upon changes in other scenarios There is one common use of SDE Versioning that is not being used in the Forecast geodatabase; that is to create versions as a temporary means of holding a user s edits for a short time until they are posted back to the parent version. This practice is typically done in instances where ongoing edits are considered draft and should in effect be hidden from a larger group of users until these edits 4

145 reach a state of completion, at which time they are then posted up to the parent version, allowing other users to see the database with the recently posted edits. After posting these edits back to the parent, the temporary version is then deleted. While this type of versioning is possible, given how we anticipate the way different scenarios will be developed and used internally in SANDAG, it should not be necessary. 5

146 THE GROWTH FORECAST WORKFLOW The following diagram illustrates how the Scenarios in the Forecast geodatabase interact with the SANDAG Growth Forecast Modeling process. 6

147 The five large boxes on the previous diagram represent distinct SANDAG workflows that in some way involve the Forecast geodatabase. Each of these is discussed in more detail below. Landcore Maintenance Process The Landcore Maintenance Process box on the diagram summarizes the process in which the Landcore geodatabase is maintained by the SANDAG GIS staff. This includes keeping Landcore as current as possible by performing ongoing daily edits as well as performing some large data loading from time to time. Attributes about ownership, historic, and current land use as well as many others are continually updated as the staff consults different ancillary data sources. Creating the Forecast Geodatabase The New Forecast Baseline Process box contains the workflow that occurs when it is time to create a new Forecast baseline. This occurs roughly at the beginning of the SANDAG Growth Forecast cycle, approximately every five years. In this workflow, a new Forecast geodatabase is created by the SDE Administrator. The database schema is applied to Forecast and records are imported from the Landcore geodatabase. During this process, all active editing on Landcore stops so that tables and feature classes do not become out of sync. After the data loading, the object classes in Forecast are registered as versioned. Create a New Scenario The box Scenario #1 Creation and Changes contains the workflow showing when a new scenario is created and edited. In this workflow, the user will create a new scenario by creating a new SDE version, using ArcCatalog, and give it a name, description, and set the level of user access. The version (scenario) may be based upon the baseline or upon changes in a version that have already been created. This type of parent-child relationship is explained later in this document. Permissions for the version should be either Private or Public, allowing other users access to use the version in the workflows that follow. GPALL Process The GPALL Process workflow box on the diagram contains the GIS processing that creates the input data tables for the Growth Forecast model. In this workflow, SANDAG GIS Staff will first establish a database connection to a specific version of the Forecast geodatabase. They will do this by specifying the version name as an additional parameter in the Database Connection properties dialog in ArcCatalog or ArcMap. This is shown in the figure below. 7

148 The Change button is used to change the Database Connection configuration to a different version. On the example above, the Database Connection is configured to connect to a version with the name Scenario #3. Next, while still connected to a specific version, the user will run a sequence of GIS operations and automated Model Builder models and python scripts. In the first of these models, a query is performed to create a new feature class that consists only of parcels which are vacant and developable. In subsequent steps, this feature class and the Forecast.GIS.landcore feature class is used to create a new feature class called LCGpall. The important thing here is that during this entire process, GPALL is using the version, which is acting as a kind of filter so that it reflects a given land use scenario. Growth Forecast Modeling The Growth Forecast Modeling box on the diagram shows a very generalized workflow in which the SANDAG Growth Forecast model(s) is/are run to produce a table of the Growth Forecast results. SANDAG Growth Forecast staff use the Forecast.GIS.LCGpall feature class (produced by the GPALL process) and the Forecast.GIS.SiteSpec table as inputs to their Growth Forecast models, producing a new DevCode table. As was done in the GPALL process, the modelers establish a connection to a specific version in order to obtain the correct version of the SiteSpec and LCGPALL object classes for a given scenario. It is very important to note that connecting directly to the database tables using SQL Server will not yield the correct version (i.e. scenario) of the database. This is because SDE versioning works by storing the changes particular to each version (scenario) in separate tables, known as the delta tables. SDE combines information in the delta tables with the business tables in order to serve versions of a database to the end user. SQL Server queries will not do this. At the end of the Growth Forecast model run, a connection to the correct version is established so that the correct version of the DevCode table is replaced or edited. This can be done using a data loading process in ArcMap or by calculating fields in DevCode based on joined records from the modeling output table. 8

149 WORKING WITH SCENARIOS This section provides some step-by-step instructions on how to create and manage scenarios. In ArcGIS there are several ways to accomplish these tasks, and all are not provided here. The user is advised to refer to the ArcGIS documentation and ArcGIS online help to learn more about the various ways to work with SDE versioning. Creating a New Scenario 1. Start ArcCatalog 2. Connect to the Forecast Geodatabase 3. Right-click the database connection and select Versions. 9

150 4. In the Version Manager dialog, select the version that will be the parent version to the new version. In most cases, this will be the DEFAULT version. Right-click this version and select New. 5. Type in a name, a short description of the Scenario, and set the Permission to Public. These properties can later be changed at any time by selecting Properties on the Version Manager dialog. 6. Click Ok. 7. Close the Version Manager. Connecting To and Modifying a Scenario You can connect to a version either through ArcCatalog or ArcMap. In the steps shown below, it is assumed that the user is connecting through ArcMap for the sake of making changes to a version. 1. Start a new ArcMap blank document or open an existing MXD file. 2. Open the Versioning toolbar (show below) 10

151 3. If not already loaded, add the feature class and tables from the Forecast geodatabase to the Map. 4. Go to the Source tab in the Table of Contents. 5. In the Table of Contents, select the source geodatabase. 6. Click the Change Version button on the Versioning toolbar. 7. Select the version that you want to edit and click OK. 8. The Name of the Geodatabase source will change to reflect the name of the version. 9. Start Editing, selecting the Forecast geodatabase as the edit source. The user may make changes to the landcore feature class or any of the tables in the Forecast geodatabase. This includes geometry changes, such as splitting, merging, reshaping, or adding polygons, or attribute changes. Deleting a Scenario Deleting a version is done through the Version Manager. 1. In ArcCatalog, right click the database connection to Forecast and select versions. 2. In the Version Manager, right-click the version that you want to delete and select delete. If an attempt is made to delete a version that is a parent to another version, the following dialog will appear informing the user that this cannot be done. 11

152 The child version must first be deleted. Warning: If changes in a child version are not posted to the parent version, those changes will be lost when the child version is deleted. There is no warning dialog to alert the user prior to deleting a version, informing them that there are un-posted records in a version s delta tables. Combining Features from different Scenarios There may be times when a Scenario contains particular changes that should be incorporated into another Scenario. If there is a parent child relationship between these two versions and the parent version contains the changes that need to be seen by the child version, then the process is relatively simple and is outlined below. Option #1 Move changes from one Scenario to another using Reconcile (For a parent-child relationship only) Steps: 1. Start Editing on the child version. 2. Reconcile the child against the parent version. This process is explained in more detail by way of example below. Step 1 New Scenario #1 is created and a new Transportation Center is added (shown in gray). 12

153 Step 2 New Scenario #2 is created from parent Scenario #1. Open space is developed to Multifamily Residential (shown in orange). Step 3 In Scenario #1 the Transportation Center is expanded. 13

154 Step 4 The expanded Transportation Center is not automatically in Scenario #2. Reconcile button on the versioning toolbar is used to reconcile Scenario #2 against Scenario #1. Scenario #1 is chosen. Step 5 The expanded transportation center from Scenario #1 is included in Scenario #2 through the reconcile process. 14

155 Note that the method which is explained above, to move changes from one version to another, will only move ALL changes in the parent version to the child version. Different methods, involving copy and paste operations (explained below), should be used to move only some changes from one version to another. You cannot reconcile one version with a non-parent version. The two versions must have a parentchild lineage in order to be able to reconcile. However you can reconcile a child with a grandparent version. This is shown graphically: Changes are copied from Scenario #1 to Scenario #3 Reconcile Scenario #1 1. Create Version Scenario #2 2. Create Version Scenario #3 Save Start Editing 3. Edits Option #2: Copy and Paste Operations to move changes between versions This method is a manual one in which features from one version are copied and pasted into another version. There are two advantages of using this method to move changes. The first is that changes can be copied across versions for which there is no a parent-child lineage. The second advantage is that individual changes can be copied. A disadvantage is that this is a very manual process and the user needs to be aware of, and be able to select, each feature to be copied. The steps for using this method are outlined below. Steps 1. Bring both versions into ArcMap. The layers from each version will have the same names. Rename them for easier identification later. 2. Start Editing on the child version. 3. Delete the features that need to be replaced with features from the parent version. 4. Select features to be copied in the parent version. 5. Select Copy from the Edit menu. 6. Set the Edit target to the child version (if needed). 7. Select Paste from the Edit menu. 15

156 Other suggestions The Transfer Attributes tool on the Spatial Adjustment can be used to transfer attributes from one version to another. CONFLICT DETECTION When the changes in one version are brought into another version using the Reconcile process, some conflicts may be found. This is when the same record or feature has been edited in both versions but to different values. When a conflict occurs, a process and dialog is presented to the user to resolve this conflict by determining which of the two values to accept. An example of this is shown below. For the workflow in Option #1 described above, where the user wants all the changes in the parent version/scenario to go to the child version/scenario, the user should select Replace with Conflict Version as shown below. Replacing the edit version feature with the feature in the parent version. Accessing Versions in SQL Server 16

157 To be able to see a version in SQL Server a multi-versioned view will need to be created in SDE. Currently this will need to be done with SDE commands from the command prompt window on the SDE server. The syntax for the command is: sdetable -o create_mv_view T<the name of the view to create> -t<the name of the feature class> - i<instance #> -D<geodatabase name> -u<user name> -p<password> The sdetable command has many options. It is recommended that these options be reviewed in the ArcSDE documentation before using the command. Once the above command has been executed a multi-versioned view will be available for use in SQL Server. A multi-versioned view looks and works like any other SQL Server table, but is actually a combination of the adds and deletes tables that are associated with the version. Use the following syntax to access and query the multi-versioned view in SQL Query Analyzer: EXECUTE sde.set_current_version <owner.arcgis version name> ; SELECT * FROM <owner.view name>; The <owner.arcgis version name> is the owner and name that was given at the time the version was created and is seen in the ArcCatalog version manager. The <owner.view name> is the name of the version that was given in the sdetable -o create_mv_view command and is seen in SQL Query Analyzer under the geodatabase views. 17

158 SANDAG LANDCORE GEODATABASE MIGRATION AND WORKFLOWS APPENDICES December 2005 Revision 1.1 February 2008 Paul Hardwick Steve Kunkel

159

160 APPENDIX A. LAYERFILE TABLES Landcore Layer Files Filename Layer Name scale dependency Definition Query symbolize on field transparency join table labeling label scale Landcore.GIS.Landcore.lyr Landcore.GIS.Landcore < ? Existinglanduse.lyr Existing land use < 24,000 - Existing Land Use 50% - apn8 - off - BaseYearLanduse.lyr Base Year land use < 24,000 - Base Year Land Use 50% - apn8 - off - PlannedLanduse.lyr Planned land use < 24,000 - Planned Land Use 50% - apn8 - off - LanduseChange.lyr Land use change < 24,000 BASELU <> LU Existing Land Use 50% - apn8 - off - ExistingOwnership Existing ownership < 24,000 - Existing Ownership 50% - apn8 - off - BaseYearOwnership Base Year ownership < 24,000 - Base Own 50% - apn8 - off - OwnershipChange.lyr Ownership change < 24,000 BASEOWN <> OWN Existing Ownership 50% - apn8 - off - Redevelop_Infill.lyr Redevelop/Infill < 50,000 REDEVINF > 0 Planned Land Use 50% - apn8 - off - Site-Specific.lyr Site-Specific < 50,000 SITEID > 0?

161 Non-Landcore Layer Files Filename Layer Name featureclass SanGIS_BusSites.lyr SANGIS.BUS_SITES (SDE: "sangis") SANGIS.BUS_SITES SanGIS_ParcelsAll.lyr SANGIS.PARCELS_ALL SANGIS.PARCELS_ALL Group Layer Files Filename Layer Name featureclass Activities.lyr Schools LIS.GIS.Schools Colleges LIS.GIS.Colleges Hospitals LIS.GIS.Hospital Activity Centers LIS.GIS.act Land Use LIS.GIS.Comp Public Ownership LIS.GIS.Comp2004 Admin2000.lyr MSA SRA Census Designated Place Census Block Census Block Groups Census Tract Zip Code SGRAs LIS.GIS.msa2000 LIS.GIS.sra2000 LIS.GIS.place2000 LIS.GIS.block2000 LIS.GIS.bkgp2000 LIS.GIS.ct2000 LIS.GIS.zip2001 LIS.GIS.sgra Basemap.lyr Parcels Coast County CPA City CPA Sphere City County CPA Non-county LIS.GIS.parcel LIS.GIS.coast ccpa citycpa sphere citycpa county noncnty Imagery.lyr Color APU 2005 West Color APU/Lenska All Color Apu/Lenska/SanGISSDE All Color Lenska West Color Landiscor West CIR West CIR East BW SDCIty CIR West CIR East K:\IMAGERY\SIDS\SanDiegoWest2005.ecw K:\imagery\apu2004Image\SanDiego-2f-0404_2021.Map SDE - RASTER.ORTHO_COUNTYWIDE_COLOR_2003 K:\IMAGERY\SIDS\SanDiegoWest2003.ecw K:\IMAGERY\SIDS\SanDiego02.ecw K:\IMAGERY\SIDS\cir2000west.sid K:\IMAGERY\SIDS\cir2000east.sid K:\IMAGERY\SIDS\sd99.sid K:\IMAGERY\SIDS\cir1996west.sid K:\IMAGERY\SIDS\cir1996east.sid 4

162 Transportation.lyr Freeway Shields State Route Shields Freeway Future Freeways State Route Major Roads Future Major Roads Roads Road Names Railroads Runways LIS.GIS.fwyshpt2 LIS.GIS.rdshpt2 LIS.GIS.fwy LIS.GIS.fwyfut LIS.GIS.statertes LIS.GIS.majorrds LIS.GIS.majfut LIS.GIS.roadg LIS.GIS.roadg LIS.GIS.rr LIS.GIS.runways 5

163 APPENDIX B. GEODATABASE DESIGN DIAGRAMS The following pages consist of geodatabase diagrams created in Microsoft Visio. These Static Structure Diagrams are from the UML Model that was used to design the Landcore geodatabase. For more information about how to interpret these diagrams and how to design a geodatabase using Visio, refer to the documents listed below. These are available as PDF files from the ESRI Support Center ArcGIS 9.0 Documentation Building Geodatabases with Visio Introduction to CASE Tools White Papers CASE Tools Tutorial (advanced content) 6

164 7

165 8

166 9

167 10

168 Create MPR Relationship Class A script tool in the Landcore Toolbox that populates the attributed PropRecords Relationship Class. #FileName: CreateMPRRelationship.py #Author: Paul Hardwick #Date: 11/9/2005 #Description: Utility that populates the PropRecords relationship class. The utility # executes an SQL query that creates a temporary table. The temporary table # is registered with the geodatabase to add an OID. The registered table is # loaded into the relationship class with the tabletorelationshipclass # geoprocessing tool #Comments The user inputs the Landcore feature class, the table to be related, the existing # PropRecords relationship class, and types in the name of the geodatabase # User cannot have the geodatabase open when running tool. The temporary # SQL table is deleted upon program completion or crash. # The mprrelatetable must remain in the geodatabase. # # Import system modules import sys import string import os import mx.odbc.windows import win32com.client import traceback import re # Create the Geoprocessor object gp = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if gp.exists(r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if gp.exists(r"c:/arcgis/arctoolbox/toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" gp.addtoolbox(systbxpath + "/Data Management Tools.tbx") gp.addtoolbox(systbxpath + "/Analysis Tools.tbx") #Script arguments lcfc = sys.argv[1] #Landcore Feature Class mprtablename = sys.argv[2] #Master Property Records Table relationshipname = sys.argv[3] #PropRecords relationship class gdbname = sys.argv[4] #Geodatabase Name try: #SQL Server ODBC connection setup. connectionstring = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=" + gdbname sqlconnection = mx.odbc.windows.driverconnect(connectionstring) cursor = sqlconnection.cursor() #creates a SQL cursur #Get table names from input arguments #User inputs feature class or shape file. Fully qualified path name split into a list lcfcstring = lcfc.split(".") #returns [user].[feature class name] lcfeaturetable = lcfcstring[len(lcfcstring) -2] + "." + lcfcstring[len(lcfcstring) -1] #User inputs feature class or shape file. Fully qualified path name split into a list mprstring = mprtablename.split(".") 11

169 #returns [user].[feature class name] mprtable = mprstring[len(mprstring) -2] + "." + mprstring[len(mprstring) -1] #create path variable for relationship table tablepath = "Database Connections\\gis." + gdbname + ".kona.sde\\" + gdbname + ".gis." #Create path variable for GDB gdbpath = "Database Connections\\gis." + gdbname + ".kona.sde" gp.addmessage("table Path is " + tablepath) print gp.getmessages() #SQL commmand to select Landcore objectids and matching MPR objectids on #parcelid where parcelid is not 0 gp.addmessage("") print gp.getmessages() gp.addmessage("slecting Landcore objectids and matching MPR objectids on parcelid where parcelid is not 0") print gp.getmessages() cursor.execute("select " + lcfeaturetable + ".objectid as objectidlc, " + \ mprtable + ".objectid as objectidmpr into gis.tmpreltab from " + \ lcfeaturetable + " inner join " + mprtable + " on " + lcfeaturetable + \ ".parcelid = " + mprtable + ".parcelid and " + lcfeaturetable + \ ".parcelid <> 0 and " + mprtable + ".parcelid <> 0 order by " + \ lcfeaturetable + ".objectid") sqlconnection.commit() #Import the tmpreltab with the tabletotable geoprocessing tool #This will register the table with the GDB and create and populate an OID field gp.addmessage("") print gp.getmessages() gp.addmessage("registering table with " + gdbname + " geodatabase") gp.getmessages() gp.refreshcatalog(gdbpath) gp.tabletotable_conversion(tablepath + "tmpreltab", gdbpath, "mprrelatetable") #Use the gis.mprrelatetable to populate the proprecords relationship class #with the tabletorelationshipclass geoprocessing tool gp.addmessage("") print gp.getmessages() gp.addmessage("running TableToRelationshipClass Geoprocessing Tool") print gp.getmessages() gp.tabletorelationshipclass_management(lcfc, mprtablename, relationshipname, "SIMPLE", \ "Property Records", "MPR Parcels", "NONE", \ "MANY_TO_MANY", tablepath + "mprrelatetable", \ "objectidlc;objectidmpr", "objectid", "objectidlc", \ "objectid", "objectidmpr") #Delete the tmpreltab from SQL Server gp.addmessage("") print gp.getmessages() gp.addmessage("droping temporary SQL table") print gp.getmessages() cursor.execute("drop table gis.tmpreltab") sqlconnection.commit() sqlconnection.close() except: cursor.execute("drop table gis.tmpreltab") 12

170 sqlconnection.commit() sqlconnection.close() gp.addmessage(gp.getmessages(2)) print gp.getmessages() gp.adderror('-'*60) gp.adderror(' '*20 + "CRASHED") gp.adderror("error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) gp.adderror('-'*60) 13

171 APPENDIX C. LANDCORE DATA MIGRATION Table used to Map Source Data to Landcore The following table was used as a planning tool as part of the data migration process to map the fields in the existing tables to fields in the new Landcore geodatabase. Please note that fields in the current Landcore geodatabase may be different from those shown in the table because the Landcore schema underwent changes after the data loading process was completed. Gray columns indicate source tables that were not used. Gray rows indicate fields which are new to the landcore database and do not have any corresponding source data. 14

172 15

173 16

174 17

175 APPENDIX D. TRAINING DOCUMENT - GEODATABASE DESIGN USING VISIO The document in this appendix was originally created as a training document for a half day training session conducted by TAIC for SANDAG. This was a technology transfer to instruct and equip SANDAG GIS staff with the methods and means to design a geodatabase using Microsoft Visio SANDAG staff should be able to carry out the process of changing the Landcore geodatabase schema by modifying the UML in the Visio drawing file. In addition, SANDAG has future plans to design other new geodatabases using UML/Visio. This CDROM contains files that were used in the Visio training. This includes various setup files, a sample diagram, and additional instruction materials. Affix CDROM envelope here 18

176 Geodatabase Design using Visio SANDAG Training Document Introduction The knowledge and skills of how to design geodatabases in Visio and construct them from this design has direct applicability for SANDAG GIS staff. The Landcore geodatabase, SANDAG s core database supporting growth forecasting and SANDAG s role as a regional GIS data distributor, was originally designed in Visio. SANDAG has committed to keeping this Visio design current as a way to clearly document the geodatabase and as a mechanism to introduce schema changes to the Landcore geodatabase. In addition, SANDAG plans to design and implement future geodatabases through UML geodatabase design methods. This document describes the basic steps required to design a geodatabase using Microsoft Visio. It is written as step-by-step instructions, with checkboxes provided, and is composed of four sections. The first section addresses how to setup up a computer for geodatabase design with Visio. The second section covers how to begin the geodatabase design process by setting up a project workspace. The third topic covers how to design within Visio by outlining the start-to-finish tasks. The final topic explains how to create a new, or update an existing, geodatabase from the Visio design. Some additional training documents, published by ESRI, on geodatabase design using Visio and CASE tools are cited in the first section and are highly recommended. Section 1 - Setup: What You Need to Setup Your Computer You need Visio 2002 or 2003 If you have Visio 2002, then you must have Service Pack 1 installed. This can be downloaded from You need ArcGIS Desktop 9x. If you have ArcGIS Desktop 8.2, then you must download and install the XMI CASE tools upgrade. SANDAG currently has ArcGIS Desktop 9.x installed. Setup the XMI Export Visio plugin (See instruction below) For VISIO 2002 Only Install the XMI Export a. This can be downloaded at: b. Run the xmiexport.exe. Copy the resulting XMIExport.dll to: C:\Program Files\Microsoft Office\Visio10\DLL. 19

177 Install the ESRI XMI Export add on. a. Copy the <ArcGIS Install folder>\casetools\utilities\esri XMI Export.vsl to C:\Program Files\Microsoft Office\Visio10\1033\Solutions\Visio Extras. For VISIO 2003 Only Install the ESRI XMI Export add on. a. Copy the C:\ArcGIS\CaseTools\Utilities\ESRI XMI Export.vsl into the C:\Program Files\Microsoft Office\Visio11\1033 folder. Install the XMIExprt add on. a. Download the XMIExprt.exe from: a868-d f57d&displaylang=en. b. Verify the file has been downloaded and unzipped and then copy the file XMIExprt.dll into the C:\Program Files\Microsoft Office\Visio11\DLL folder. (Note: XMIExprt.dll is not a typo.) Since 3 rd party plugins are detected differently in Visio 2003 than they were in Visio 2002, you have to tell Visio where to load plugins from. Follow the steps below. 1. Start Visio 2003 Professional. 2. Select the Tools menu and select Options. 3. Change security settings for Macros to Medium (will prompt you to enable macros). 4. Select the Advanced Tab in the Options dialog, and click the 'File Paths...' button. 5. Select the browse button next to the Add-ons field and add both: C:\Program Files\Microsoft Office\Visio11\1033 and C:\Program Files\Microsoft Office\Visio11\DLL\XMASamples 6. Click OK to close the File Paths and Options dialogs. 7. Restart Visio Professional for the changes to take effect. Setup the Schema Wizard You just need to add the Schema Wizard button to a toolbar in ArcCatalog. Start ArcCatalog and open the Customize dialog. Go to the CASE Tools command group and drag the Schema Wizard button onto a toolbar (suggest placing it on the Standard toolbar). Get other Training Documents There are three Instruction documents that are available from ESRI. These are: - Building Geodatabases with CASE Tools - ArcGIS9 Introduction to CASE Tools 20

178 - CASE Tools Tutorial Creating custom features and geodatabase schemas The first two documents listed above are recommended. The third document covers how to create custom features using C++ programming a very advanced topic. Section 2 - Get Organized (do this for each geodatabase design effort). Create project file folders Create a logical folder structure to keep track of the various file formats you will be creating. Most likely, you will be creating many different versions of your schema. You will be creating Visio diagrams (vsd), XMI export files (xmi), personal test geodatabases (mdb), and probably some PDF figures of the diagrams (pdf). For simplicity, put these into separate folders and use file name/numbering to keep track of them. Suggested Folders: visio2002uml the Visio diagram (vsd) files. (or visio2003uml if you have Visio2003) xmi_files xmi files exported from Visio. xmi_gdb test personal geodatabases. Get the Visio Template If you have Visio2002, copy <ArcGIS Install Folder>\CaseTools\Uml Models\ArcInfo UML Model (Visio 2002).vst to your visio2002uml folder that you just created. If you have Visio 2003, then copy <ArcGIS Install Folder>\CaseTools\Uml Models\ArcInfo UML Model (Visio 2003).vst to your visio2003uml folder that you just created. Rename the template file to reflect the geodatabase you are about to design, (version #1). (Note: I had to download this from ESRI.) When starting a diagram from scratch, this is the file you will always start with. Get the uml.dtd Copy the <ArcGIS Install Folder >\ArcGIS\CaseTools\Utilities\UML.DTD file into your xmi_files folder. Later, this file is used by the Semantics Checker, and must be in the same folder as the exported schema (xmi) files. Section 3 - Using Visio for Geodatabase Design For this overall topic, the ESRI documents provide excellent training material. A good short introductory document ArcGIS 9 Introduction to CASE Tools gives short descriptions of the different elements in a Visio geodatabase design and how they relate to the different objects in the geodatabase. It is useful if the reader has some understanding on object-oriented concepts such as class inheritance but not necessary to understand the material. In addition, a few pages cover how to use CASE tools to generate or update an existing geodatabase, which will be covered in the next section. 21

179 The second document Building Geodatabases with CASE Tools (formerly Building Geodatabases with Visio) gives a thorough description of how to design a geodatabase within Visio, with step-bystep instructions and screenshots, given for nearly all possible tasks within Visio. The introductory narrative in this document duplicates all the text in the short Introduction document. What is lacking in this document is structure to the presentation, which would tie together all the individual tasks as a linear sequence leading up to a designed and implemented geodatabase. Start to Finish Outline Below is an outline that lists a recommended sequence of steps to design a geodatabase within Visio. The specific details of each step are not included since they are better covered in the ESRI document Geodatabases with CASE Tools. 1. Start your Model. Open the <project name>.vst file in your visio2003uml folder. This is the template that will form the basis for your model. From Visio, save this file as a *.vsd in your visio2002uml or visio2003uml folder. 2. Turn on Visio Windows You will need to see three windows: 1. UML Static Structure stencil in the Shapes window,2. Model Explorer window (UML View Model Explorer ), and 3. a diagram. Make these visible and arrange them to your liking. 3. Create Packages A package is simply a container to hold your different classes. Create these under the Workspace package. Give your packages descriptive names since the package name will appear in the diagram next to the class name (see example below). Package name 22

180 Suggested package names: Table Feature class Domain Feature Dataset (use actual name) 23

181 4. Create any Feature Dataset Packages Datasets are a special case of packages which are stereotyped as FeatureDataset. Create these under the Workspace package and select the FeatureDataset stereotype in its UML Package Properties dialog. Also create/move any feature class packages to be within their respective Feature Dataset package. 5. Setup the Main Diagram. You can create diagrams within any package. It is recommended that you create a package called Main and place a new diagram there which will be the main diagram that summarizes the structure of your model. This is the one that you will be printing and will care about how it looks. 6. Create additional Static Structure diagrams. Create diagrams within each of your packages. This is optional, but it provides a means to get your classes into their package. These diagrams may not be needed for anything beyond this, with the exception being the diagram in the Domains package. 7. Create the Domains. You can do this now or later, after setting up the table classes, but it is more efficient to do this now so that you don t have to go back to the table classes. Domains are special classes that are provided for you under the Workspace package (named Template Domain). Since the package name always appears before the class, it is advantageous to have your domains with a package called domain. You can accomplish this within the Model Explorer window. Rightclick the template domain and select duplicate. Then, drag the new domain into your Domain package. Using the properties window for each domain, change their names and set their properties. Later, as you create classes and set their properties, you can associate these domains with specific fields. If you have a lot of domains, you may elect to print your domain classes on a separate diagram. If so, create a diagram in this Domain package and drag all your domains onto this diagram. 8. Setup Abstract Classes. Put any special abstract classes on your main diagram. Usually you will need at least the Object and the Feature classes. Drag these out of the ESRI Classes package. These are abstract classes, meaning that they are simply used to pass on inherited properties to classes that you will create later. Note that they are connected using a generalization connector/association because the feature class inherits from the object class. 9. Create your Classes From the UML Static Structure stencil drag the Class class onto an appropriate diagram. For example, if you are going to make a feature class, then put this class on the diagram in your Featureclass package. 24

182 10. Edit your Classes. Using each UML Class Properties dialog, set the name, documentation, and the Tagged values. Refer to the ESRI documentation for instructions on how to do this. 11. Compose the Main Diagram. Drag each of your classes from its package (in the Model Explorer window) onto the main diagram. 12. Set Class Inheritance. Connect each class using the Generalization association. If it is just a table, connect it to the Object class. If a feature class then connect it to the Feature class. 13. Create Relationship Classes. The relationship class is a UML Binary Association which has very specific tagged values set. From the UML Static Structure stencil, drag Binary Associations onto your main diagram and connect this to the origin and foreign tables. Then set the name and tagged values (refer to the ESRI document for the list of required tags). Also the relationship cardinality and the end names need to be set. 14. Format the main diagram. You can spend a little or a lot of time formatting all the classes on the diagram. A well formatted diagram can greatly help readability for users that are not accustomed to viewing UML diagrams. Here are some general recommended guidelines: a. Use color to differentiate the different types of classes. For example, color the tables yellow and the feature classes green. b. Arrange related classes with the primary class on the left and the foreign class on the right. c. Try to avoid associations crossing over one another or running behind classes. 15. Export your model to XMI. Select Tools Addons ESRI XMI Export Name the file with a version number and put the XMI file into your xmi_files folder. 16. Run the Semantics Checker. Select Tools Macros ESRI Semantics Checker. 17. Fix any errors. Usually the error report will give you enough information so that you can quickly locate and fix the error. Special Notes If you delete an object on a diagram, it will remain in the package in your Model Explorer. Delete it there also. 25

183 You can copy fields from one class to another by duplicating the field (property) and then dragging the copy onto another class. If you have not properly connected your objects using the Generalization association, the object will not be created in your geodatabase. The semantics checker will not catch this. Set the AllowNulls tagged value on all fields to true. There are tagged values that need to be set at the class level for feature classes. Section 4 - Creating or Updating the Geodatabase In this final step, a special tool which is often referred to as CASE Tools is used to create a new geodatabase or update and existing geodatabase from the schema, now stored in the XMI file. The ArcGIS Desktop Help has a topic called Building Geodatabases with CASE Tools, located under Building a Geodatabase, has a good detailed step-by-step instructions on how to use the Schema Wizard. 26

184 APPENDIX E. GPALL MODELS AND SCRIPTS The GPALL process has been created as a series of automated processes and manual procedures. The automated processes take the form of ArcGIS Model Builder models and python scripts, which are shown on the following pages. For a discussion of the GPALL process, refer to the GPALL Process document. GPALL: CreateLcVacDev Model The CreateLCVacDev model is essentially the second major GIS processing step in the GPALL process and is shown as the Vacant and Developable Lands (DL) box on the overview diagram found on page 3 of the GPALL documentation. 27

185 GPALL: creategpallfc Model The creategpallfc model is essentially the fourth major GIS processing step in the GPALL process and is shown as the % Constrained Calculations box on the overview diagram found on page 3 of the GPALL documentation. 28

186 GPALL: Python Script percentconstrained.py A tool which runs this python script percentconstrained.py is used in the creategpallfc Model shown on the previous page. #FileName: percentconstrainedreal #Author: Paul Hardwick/Daniel Flyte #Date: 05/05/05 #Description: Utility that calculates total percent constrained for vacant # developable parcels #Comments: Need to check the SQL Server database connection string to # make sure you are connecting to the right database. This utility # uses an SQL stroed procedure in Forecast named constcreatetable # #Assign units to Landcore #Import system modules import sys import string import os import mx.odbc.windows import win32com.client import traceback import re # Create the Geoprocessor object gp = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") #Script Variables #User inputs feature class. Fully qualified path name split into a list #Needed so that utility could capture the shape.area of the feature classes lcvacdevfc = sys.argv[1].split(".") #returns [user].[feature class name] vacdevtable = lcvacdevfc[len(lcvacdevfc) -2] + "." + lcvacdevfc[len(lcvacdevfc) -1] vacdevconnect = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=forecast" lcvacdevconstfc = sys.argv[2].split(".") lcconsttable = lcvacdevconstfc[len(lcvacdevconstfc) -2] + "." + \ lcvacdevconstfc[len(lcvacdevconstfc) -1] lcconstconnect = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=forecast" lcconstfields = "objectid,lckey,planid,conother,wet,flood,cntypreserve,ag20e," +\ "apz,tier1,tier2,faults,fci,froads,pama,gw05,gw067,gw125,gw2,gw25,gw5," +\ "regionpreserve1,regionpreserve2" ##lcconstfields = "objectid,lckey, planid, slope15,slope25,slope50,conother,wet,flood,habpres,ag20e," +\ ## "apz,tier1,tier2,faults,fci,froads,pama,gw05,gw067,gw125,gw2,gw25,gw5," +\ ## "regionpreserve1,regionpreserve2" genplanconnect = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=Landcore" genplantable = "gis.genplan" genplanfields = "conother,wetlands,floodplains," +\ "habitatpreserves,ageastcwa,airporthazzardzones,vegetation1," +\ "vegetation2,faults,fci,futureroads,pama,gw05,gw067,gw125,gw2,gw25," +\ "gw5,regionpreserves1,regionpreserves2" ##genplanfields = "slope15,slope25,slope50,conother,wetlands,floodplains," +\ ## "habitatpreserves,ageastcwa,airporthazzardzones,vegetation1," +\ ## "vegetation2,faults,fci,futureroads,pama,gw05,gw067,gw125,gw2,gw25," +\ ## "gw5,regionpreserves1,regionpreserves2" pctconstbylckey = "\\\\kalani\\shared\\res\\arcgis\\arcgis9testing\\constraints\\pctconstbylckey.txt" constbylckeytable = "gis.constbylckey" outtableview = sys.argv[3] 29

187 try: #SQL Server ODBC connection setup. NOTE: May need to change Database sqlconnection1 = mx.odbc.windows.driverconnect(lcconstconnect) lcconstcursor = sqlconnection1.cursor() #creates a SQL cursur sqlconnection2 = mx.odbc.windows.driverconnect(genplanconnect) genplancursor = sqlconnection2.cursor() gp.addmessage("compiling dictionaries, Please wait") print gp.getmessage #Create a dictionary that contains lckey and total lcparcel area from vacdevfc & vacdevconst lcparceldict = {} #Creates empty dictionary #vacdevcursor.execute("select lckey,'shape.area' from " + vacdevtable) #vacdevrecord = vacdevcursor.fetchone() vacdevcursor = gp.searchcursor(sys.argv[1]) vacdevrecord = vacdevcursor.next() while(vacdevrecord): lcparceldict[vacdevrecord.getvalue("lckey")] = vacdevrecord.getvalue("shape.area") vacdevrecord = vacdevcursor.next() constparceldict = {} #Creates empty dictionary lcvacdevconstcursor = gp.searchcursor(sys.argv[2]) lcvacdevconstrecord = lcvacdevconstcursor.next() while(lcvacdevconstrecord): constparceldict[lcvacdevconstrecord.getvalue("objectid")] = lcvacdevconstrecord.getvalue("shape.area") lcvacdevconstrecord = lcvacdevconstcursor.next() #Create text file to hold percent constrained by lckey constbylckeyfile = open(pctconstbylckey, 'w') constbylckeyfile.write("lckey,pctconst\n") constbylckeyfile.flush #Establish local variables and lists lcconstcursor.execute("select " + lcconstfields + " from " + lcconsttable + " order by lckey") lcconstrecord = lcconstcursor.fetchone() currentlckey = lcconstrecord[1] lastlckey = currentlckey pctconst = 0 constpolyarea = 0 count = 1 ## gp.addmessage(str(lcconstrecord)) ## print gp.getmessage #Get the first genplan record query = "select " + genplanfields + " from " + genplantable +\ " where planid = " + str(lcconstrecord[2]) genplancursor.execute(query) genplanrecord = genplancursor.fetchone() ## gp.addmessage(query) ## print gp.getmessage ## gp.addmessage(str(genplanrecord)) ## print gp.getmessage gp.addmessage("") gp.addmessage("calculating Percent Constrained") print gp.getmessage #Loop through constraint polys for each lckey while (lcconstrecord): currentlckey = lcconstrecord[1] 30

188 #Keeps track of number of records processed if (count % 1000 == 0): gp.addmessage ("Processed " + str(count + 1)) # + " of " + len(constparceldict)) print gp.getmessage #Get percent constrained from the genplan table if (currentlckey!= lastlckey): genplancursor.execute("select " + genplanfields + " from " + genplantable +\ " where planid = " + str(lcconstrecord[2])) genplanrecord = genplancursor.fetchone() if (currentlckey == lastlckey): constpolyarea = constparceldict[lcconstrecord[0]] maxconst = 0 for i in range(0, len(genplanrecord)): if (lcconstrecord[i+3] == "YES"): if (genplanrecord[i] > maxconst): maxconst = genplanrecord[i] ## gp.addmessage(str(currentlckey) + " " + str(maxconst)) ## print gp.getmessage if (maxconst == 100): break ## gp.addmessage("next Poly") ## print gp.getmessage pctconst += constpolyarea * (long(maxconst)/100.0) lcconstrecord = lcconstcursor.fetchone() else: #Get total area of lckey and calc total percent constrained lcparcelarea = lcparceldict[lastlckey] totalpctconst = ((pctconst / long(lcparcelarea)) * 100.0) ## gp.addmessage("totalpctconst = " + str(totalpctconst)) ## print gp.getmessage #Round totalpctconst and write lckey and total percent constrained to file totalpctconst = int(round(totalpctconst)) constbylckeyfile.write(str(lastlckey) + "," + str(totalpctconst) + "\n") pctconst = 0 #Reset pctconst to 0 lastlckey = currentlckey count = count + 1 #increment counter by 1 ## sqlconnection1.close() sqlconnection2.close() #Get total area of lckey and calc total percent constrained for last record lcparcelarea = lcparceldict[lastlckey] totalpctconst = ((pctconst / long(lcparcelarea)) * 100.0) #Round totalpctconst and write lckey and total percent constrained to file totalpctconst = int(round(totalpctconst)) constbylckeyfile.write(str(currentlckey) + "," + str(totalpctconst) + "\n") constbylckeyfile.close() #Create constbylckey table from a stored procedure and bulk insert the constbylckeyfile lcconstcursor.execute("execute constcreatetable") sqlconnection1.commit() lcconstcursor.execute("bulk insert " + constbylckeytable + " from '" + \ str(pctconstbylckey) + "' with (fieldterminator = ','," + \ " rowterminator = '\n', firstrow = 2)") sqlconnection1.commit() sqlconnection1.close() 31

189 #gets fully qualified pathname to table. Strips out FDS name. Then creates table view output #Table path should = Database Connections\kona.forecast.gis.sde\Forecast.GIS.constByLckey newtable = sys.argv[1][0: sys.argv[1].rfind(lcvacdevfc[len(lcvacdevfc) -3])] + "constbylckey" ## gp.addmessage(newtable) ## print gp.getmessage gp.maketableview(newtable, outtableview) except: gp.addmessage(gp.getmessages(2)) print gp.getmessages() gp.adderror('-'*60) gp.adderror(' '*20 + "CRASHED") gp.adderror("error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) gp.adderror('-'*60) 32

190 GPALL: empptsassignment Model The empptsassignment model is essentially the fifth and final major GIS processing step in the GPALL process and is shown as the Employment box on the overview diagram found on page 3 of the GPALL documentation. 33

191 GPALL: Python Script TwoPoinFix.py A tool which runs this python script TwoPoinFix.py is used in the empptsassignment Model shown on the previous page. #FileName: TwoPointFix.py #Author: Paul Hardwick/Daniel Flyte #Date: 3/15/2005 #Description: Utility that removes duplicated points created through the Identity or # Intersect geoprocessing tools.there is no logic as to which points gets # removed it is just based on whether the unique ID is replicated in the file. #Comments The input points must have a unique ID field # # Import system modules import sys import string import os import mx.odbc.windows import win32com.client import traceback import re # Create the Geoprocessor object gp = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if gp.exists(r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if gp.exists(r"c:/arcgis/arctoolbox/toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" gp.addtoolbox(systbxpath + "/Data Management Tools.tbx") gp.addtoolbox(systbxpath + "/Analysis Tools.tbx") #Script arguments inputfeature = sys.argv[1] uniquepointid = sys.argv[2] duppointfeature = sys.argv[3] dbname = sys.argv[4] #Find total number of features in the input and the identity output intotalfeatures = gp.getcount(inputfeature) gp.addmessage("original Number of Points " + str(intotalfeatures)) print gp.getmessages outtotalfeatures = gp.getcount(duppointfeature) gp.addmessage("number of Resulting Points " + str(outtotalfeatures)) print gp.getmessages try: if intotalfeatures!= outtotalfeatures: #SQL Server ODBC connection setup. connectionstring = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=" + dbname sqlconnection = mx.odbc.windows.driverconnect(connectionstring) cursor = sqlconnection.cursor() #creates a SQL cursur #Get table names from input arguments #User inputs feature class or shape file. Fully qualified path name split into a list 34

192 inputstring = inputfeature.split(".") #returns [user].[feature class name] infeaturetable = inputstring[len(inputstring) -2] + "." + inputstring[len(inputstring) -1] #User inputs feature class or shape file. Fully qualified path name split into a list outputstring = duppointfeature.split(".") #returns [user].[feature class name] outfeaturetable = outputstring[len(outputstring) -2] + "." + outputstring[len(outputstring) -1] featurelist = [] #Fill feature list with unique IDs ## gp.addmessage(str(uniquepointid) + " " + str(infeaturetable)) ## print gp.getmessages #SQL commmand to select unique parcelids cursor.execute("select distinct " + uniquepointid + " from " + infeaturetable) uniqueid = cursor.fetchone() #Loop to fill the feature list while(uniqueid): featurelist.append(int(uniqueid[0])) uniqueid = cursor.fetchone() numuniquefeatures = len(featurelist) updatecur = None for i in range(0,numuniquefeatures): if i % 100 == 0: #Paul Change the counter to 1000 gp.addmessage("processing record " + str(i) + " of " + str(numuniquefeatures)) print gp.getmessages whereclause = uniquepointid + "=" + str(featurelist[i]) updatecur = gp.updatecursor(duppointfeature, whereclause) cursor.execute("select count(*) from " + outfeaturetable + " where " + whereclause) numrecords = cursor.fetchone()[0] if numrecords > 1: gp.addmessage(" deleting Record ") print gp.getmessages curfeature = updatecur.next() curfeature = updatecur.next() while curfeature: updatecur.deleterow(curfeature) curfeature = updatecur.next() gp.setparameterastext(4, "True")#Sets boolean output to True sqlconnection.close() #Closes the SQL Server connection #Skips the rutine if the inputs have the same number of features and sets the boolean to true else: gp.addmessage("the Input Feature Classes have the same number of Features") print gp.getmessages gp.addmessage("move On!") print gp.getmessages gp.setparameterastext(4, "True")#Sets boolean output to True except: gp.addmessage(gp.getmessages(2)) print gp.getmessages() gp.adderror('-'*60) gp.adderror(' '*20 + "CRASHED") gp.adderror("error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) gp.adderror('-'*60) gp.setparameterastext(4, "False")#Sets boolean output to False 35

193 GPALL: Python Script empptsassignment.py A tool which runs this python script empptsassignment.py is used in the empptsassignment Model shown on the previous page. #FileName: empptsassignment.py #Author: Paul Hardwick/Daniel Flyte #Date: 3/16/2005 #Description: Utility that assigns the employment points to the Gpall file #Comments: Need to check the SQL Server database connection string to # make sure you are connecting to the right database. # # Import system modules import sys import string import os import mx.odbc.windows import win32com.client import traceback import re # Create the Geoprocessor object gp = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if gp.exists(r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if gp.exists(r"c:/arcgis/arctoolbox/toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" gp.addtoolbox(systbxpath + "/Data Management Tools.tbx") gp.addtoolbox(systbxpath + "/Analysis Tools.tbx") #Script arguments gpallfeature = sys.argv[1] empptsfeature = sys.argv[2] try: #SQL Server ODBC connection setup. NOTE: May need to change Database connectionstring = "driver={sql Server};server=kona;uid=gis;pwd=gis;database=forecast" sqlconnection = mx.odbc.windows.driverconnect(connectionstring) cursor = sqlconnection.cursor() #creates a SQL cursur #Get table names from input arguments gpallstring = gpallfeature.split(".") #User inputs feature class or shape file. Fully qualified path name split into a list gpalltable = gpallstring[len(gpallstring) -2] + "." + gpallstring[len(gpallstring) -1] #returns [user].[feature class name] empstring = empptsfeature.split(".") #User inputs feature class or shape file. Fully qualified path name split into a list emptable = empstring[len(empstring) -2] + "." + empstring[len(empstring) -1] #returns [user].[feature class name] emplist = [] #Fill feature list with unique LcKeys cursor.execute("select distinct LcKey from " + emptable) #SQL commmand to select unique parcelids lckeyid = cursor.fetchone() 36

194 #Fill the emplist with records while(lckeyid): emplist.append(lckeyid[0]) lckeyid = cursor.fetchone() sqlconnection.close() #Closes the SQL Server connection gp.addmessage("closed Connnection") print gp.getmessage totalemppts = gp.getcount(empptsfeature) #Get the total number of employment points gp.addmessage("total number of employment points = " + str(totalemppts)) print gp.getmessage numuniquefeatures = len(emplist) updatecur = None gp.addmessage("total number of unique LcKeys = " + str(numuniquefeatures)) print gp.getmessage for i in range(0, numuniquefeatures): whereclause = "LcKey = " + str(emplist[i]) searchcur = gp.searchcursor(empptsfeature, whereclause) military = 0 civilian = 0 searchcuremp = searchcur.next() if searchcuremp.getvalue("lckey") > 0: #This is for Tile8, should not need this for all of Landcore while searchcuremp: if searchcuremp.getvalue("naics_ss") == 14: military += searchcuremp.getvalue("emp_adj") else: civilian += searchcuremp.getvalue("emp_adj") searchcuremp = searchcur.next() ## command = "update gis.lcgpall set empciv = " + str(int(civilian)) + ", empmil = " + str(int(military)) + " where " + whereclause ## gp.addmessage(command) ## print gp.getmessage ## sqlcursor.execute(command) ## sqlcursor.commit() updatecur = gp.updatecursor(gpallfeature, whereclause) updatecurgpall = updatecur.next() updatecurgpall.setvalue("empmil", military) updatecur.updaterow(updatecurgpall) updatecurgpall.setvalue("empciv", civilian) updatecur.updaterow(updatecurgpall) if (i + 1) % 250 == 0: gp.addmessage("processed " + str(i + 1) + " of " + str(numuniquefeatures) + " LcKeys") print gp.getmessage gp.setparameterastext(2, "True")#Sets boolean output to True ## sqlconnection.close() except: gp.addmessage(gp.getmessages(2)) print gp.getmessages() gp.adderror('-'*60) gp.adderror(' '*20 + "CRASHED") gp.adderror("error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) gp.adderror('-'*60) gp.setparameterastext(2, "False")#Sets boolean output to False 37

195 APPENDIX F. SANGIS/LANDCORE INTEGRATION PROCESS Model: Prepare New SanGIS Data 38

196 # # HideALlFieldsExcept.py # # Created on: 10/18/04 by Greg Nichols # # Import system modules import sys, win32com.client, traceback, re, string # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") try: # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") # Script arguments... infc = sys.argv[1] keepfields = sys.argv[2] outfl = sys.argv[3] GP.addMessage(inFC + " is input layer") print GP.getMessages() keeplist = keepfields.split(";") GP.addMessage(str(len(keepList)) + " fields to keep.") print GP.getMessages() #try: # Get the Field Names thefields = GP.listfields(inFC,"","") GP.addMessage("Got the list of fields") print GP.getMessages() # First Field thefields.reset afield = thefields.next() # See if the field is one to keep visible hideoption = "HIDDEN" newname = afield.name for akfield in keeplist: if akfield == afield.name: hideoption = "VISIBLE" # special option to rename the parcelid field to make it unique. if string.upper(afield.name) == "PARCELID": newname = afield.name + "sg" GP.addMessage("Found parcelid") print GP.getMessages() GP.addMessage("Found visible field " + afield.name) print GP.getMessages() 39

197 # Add fielnames to the Value Table #vtab.addrow(afield.name fixname "Visible") fieldinfo = afield.name + " " + newname + " " + hideoption # Next afield = thefields.next() # ITERATE through all fields while afield: # See if the field is one to keep visible hideoption = "HIDDEN" newname = afield.name for akfield in keeplist: if akfield == afield.name: hideoption = "VISIBLE" # special option to rename the parcelid field to make it unique. if string.upper(afield.name) == "PARCELID": newname = afield.name + "sg" GP.addMessage("Found parcelid") print GP.getMessages() GP.addMessage("Found visible field " + afield.name) print GP.getMessages() # Add fieldnames to the Field Options fieldinfo = fieldinfo + ";" + afield.name + " " + newname + " " + hideoption # Next afield = thefields.next() # Process: Make Feature Layer GP.addMessage("Creating " + outfl + "Layer.") print GP.getMessages() GP.MakeFeatureLayer_management(inFC, outfl, "", "", fieldinfo) #### #### # Pass out the derrive Layer: #### GP.setParameterAsText(2,"SanGIS Shapefile Layer") except: GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 40

198 Model: Landcore Parsing 41

199 # # UnJoinNOutput.py # # Created on: 10/20/04 by Greg Nichols # # Import system modules import sys, string, os, win32com.client, traceback # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") inlayer = sys.argv[1] #layer or table view joinname = sys.argv[2] # name of the join to remove outfc = sys.argv[3] # output Feature Class try: # Get the input layers's featureclass desc = GP.Describe(inLayer) infcbasename = os.path.basename(desc.catalogpath) infcname = desc.catalogpath # Message GP.addMessage("") print GP.getMessages() # PROCESS: Remove the joins ## GP.Describe(inLayer) ## theindexes = GP.ListIndexes(inLayer) GP.RemoveJoin_management(inLayer, joinname) GP.addMessage(joinName + " Join Removed.") print GP.getMessages() ## # Delete output featureclass if it already exists ## if GP.Exists(outFC): ## GP.Delete_management(outFC) GP.RefreshCatalog # Process: Make Feature Layer GP.CopyFeatures_management(inLayer, outfc) GP.addMessage(outFC + " Featureclass Created.") print GP.getMessages() except: GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 42

200 # # JoionNCleanFields.py # # Created on: 10/20/04 by Greg Nichols # # Import system modules import sys, string, os, win32com.client, traceback, re # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") inlayer = sys.argv[1] #layer or table view infield = sys.argv[2] # field in layer or table view if str(sys.argv[3]) == "#": insuffix = None else: insuffix = sys.argv[3].strip() joinlayer = sys.argv[4] # Join layer or table view joinfield = sys.argv[5] # field in layer or table view if str(sys.argv[6]) == "#": joinsuffix = None else: joinsuffix = sys.argv[6].strip() GP.addMessage("Join Suffix " + joinsuffix) print GP.getMessages outlayer = sys.argv[7] # output layer name ##GP.addMessage(inLayer) ##print GP.getMessages() ##GP.addMessage(inField) ##print GP.getMessages() ##GP.addMessage("Input Suffix Option = " + str(insuffix)) ##print GP.getMessages() ##GP.addMessage(joinLayer) ##print GP.getMessages() ##GP.addMessage(joinField) ##print GP.getMessages() ##GP.addMessage("Join Suffix Option = " + str(joinsuffix)) ##print GP.getMessages() try: # Get the source featureclasses for later desc = GP.Describe(inLayer) infcname = os.path.basename(desc.catalogpath) desc = GP.Describe(joinLayer) joinfcname = os.path.basename(desc.catalogpath) 43

201 # Get the Field Names fs = GP.listfields(inLayer,"","") fs.reset af = fs.next() afl = [] while af: afl.append(af.name) af = fs.next() GP.addMessage("Fields in Input Layer = " + str(len(afl))) print GP.getMessages() # Inform number of fields before the join fs = GP.listfields(joinLayer,"","") fs.reset af = fs.next() afl = [] while af: afl.append(af.name) af = fs.next() GP.addMessage("Fields in Join Layer = " + str(len(afl))) print GP.getMessages() # PROCESS: join the layers GP.AddJoin_management(inLayer, infield, joinlayer, joinfield, "KEEP_ALL") # Inform the number of fields after the join fs = GP.listfields(inLayer,"","") fs.reset af = fs.next() afl = [] while af: afl.append(af.name) af = fs.next() GP.addMessage("Fields in Input Layer after the join = " + str(len(afl))) print GP.getMessages() # START PROCESS TO ALTER THE FIELD NAMES # Inform Progress GP.addMessage("Stripping all Field prefixes.") print GP.getMessages() # Get the Field Names thefields = GP.listfields(inLayer,"","") # First Field thefields.reset afield = thefields.next() # default is no change to the field name and visible field fixname = afield.name visoption = "VISIBLE" # Strip the prefix if found if re.compile("objectid").search(afield.name): if re.compile(infcname).search(afield.name): # Joined from the Input table fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] 44

202 fixname = newname # + "_XI" GP.addMessage("**Strip Here 1**" + fixname) print GP.getMessages() elif re.compile(joinfcname).search(afield.name): # Joined from the Join Table fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] fixname = newname + "_XJ" visoption = "HIDDEN" GP.addMessage("**Strip Here 2**" + fixname) print GP.getMessages() else: # If it was from a join that occured before this join GP.addMessage("Found other object id field") print GP.getMessages() newname = afield.name.split(".")[fnlength - 1] fixname = newname + "_XX" visoption = "HIDDEN" GP.addMessage("**Strip Here 3**" + fixname) print GP.getMessages() elif re.compile(".").search(afield.name): fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] fixname = newname GP.addMessage("**Strip Here 4**" + fixname) print GP.getMessages() # Print the string that is being stripped ## thediff = afield.name.replace(fixname,"") ## GP.addMessage("Stripping " + thediff) GP.addMessage("Suppressed the repeating message about fields being stripped.") print GP.getMessages() # Check for Input suffix option and the field is from the Input Table if insuffix and re.compile(infcname).search(afield.name): fixname = newname + "_" + insuffix # Check for Join suffix option and the field is from the Join Table if joinsuffix and re.compile(joinfcname).search(afield.name): fixname = newname + "_" + joinsuffix GP.addMessage("Join Suffixed Name 1 " + fixname) print GP.getMessages() # Add fielnames to the Value Table #vtab.addrow(afield.name fixname "Visible") fieldmvstring = afield.name + " " + fixname + " " + visoption # Next afield = thefields.next() # ITERATE through all fields while afield: # default is no change to the field name and visible field fixname = afield.name visoption = "VISIBLE" # Strip the prefix if found if re.compile("objectid").search(afield.name): if re.compile(infcname).search(afield.name): 45

203 # Joined from the Input table fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] fixname = newname # + "_XI" GP.addMessage("**Strip Here 5**" + fixname) print GP.getMessages() elif re.compile(joinfcname).search(afield.name): # Joined from the Join Table fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] fixname = newname + "_XJ" visoption = "HIDDEN" GP.addMessage("**Strip Here 6**" + fixname) print GP.getMessages() else: # If it was from a join that occured before this join GP.addMessage("Found other object id field") print GP.getMessages() newname = afield.name.split(".")[fnlength - 1] fixname = newname + "_XX" visoption = "HIDDEN" GP.addMessage("**Strip Here 7**" + fixname) print GP.getMessages() elif re.compile(".").search(afield.name): fnlength = len(afield.name.split(".")) newname = afield.name.split(".")[fnlength - 1] fixname = newname GP.addMessage("**Strip Here 8**" + fixname) print GP.getMessages() # Print the string that is being stripped ## thediff = afield.name.replace(fixname,"") ## GP.addMessage("Stripping " + thediff) ## print GP.getMessages() # Check for Input suffix option and the field is from the Input Table if insuffix and re.compile(infcname).search(afield.name): spam = newname + "_" + insuffix fixname = spam # Check for Join suffix option and the field is from the Join Table if joinsuffix and re.compile(joinfcname).search(afield.name): spam = newname + "_" + joinsuffix fixname = spam GP.addMessage("Join Suffixed Name 2 " + fixname) print GP.getMessages() # Add fieldnames to the Value Table #vtab.addrow(afield.name fixname "Visible") fieldmvstring = fieldmvstring + ";" + afield.name + " " + fixname + " " + visoption # Next afield = thefields.next() # Inform Progress GP.addMessage("Creating " + '"' + outlayer + '"' + " Layer.") print GP.getMessages() # Process: Make Feature Layer GP.addMessage(fieldMVString) 46

204 print GP.getMessages() GP.MakeFeatureLayer_management(inLayer, outlayer, "", "", fieldmvstring) except: GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 47

205 Model: SG Parse and Attribute Part I 48

206 # # reindex.py # # Created on: 2/11/2005 by Paul Hardwick # # Import system modules import sys, string, os, win32com.client, traceback, re # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") GP.AddToolbox(sysTbxPath + "/Analysis Tools.tbx") #Script arguments fc = sys.argv[1] nameindex = sys.argv[2] newindex = sys.argv[3] try: #Check to see if indexes exist GP.addMessage("Get Index List") print GP.getMessages existindex = GP.listIndexes(fc) #Delete existing indexesand add new indexes curindex = existindex.next() if curindex!= None: while curindex!= None: ## GP.addMessage(curIndex.Fields.Next().Name) ## print GP.getMessages indexfieldname = curindex.fields.next().name if indexfieldname!= "OBJECTID" and indexfieldname!= "SHAPE": #Remove existing attribute indexes GP.RemoveIndex_management(fc, curindex.name) GP.addMessage("Removing attribute index " + indexfieldname) print GP.getMessages curindex = existindex.next() #Create new attribute indexes GP.addMessage("Adding indexes on the following fields " + newindex) print GP.getMessages GP.AddIndex_management(fc, newindex, nameindex) GP.addMessage("Index " + nameindex + " Added") print GP.getMessages else: #Create new attribute indexes if none existed GP.addMessage("No existing index found" "\n" "Adding indexes on the following fields " + newindex) print GP.getMessages GP.AddIndex_management(fc, newindex, nameindex) GP.addMessage("Index " + nameindex + " Added") print GP.getMessages 49

207 except: GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 50

208 Model: SG Parse and Attribute Part II 51

209 Model: New Parcel Attributes 52

210 # # FieldToField.py # # Created on: 10/20/04 by Greg Nichols # # Import system modules import sys, string, os, win32com.client, traceback # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") GP.AddToolbox(sysTbxPath + "/Analysis Tools.tbx") # Script arguments... intabtocalc = sys.argv[1] tofields = sys.argv[2].split(";") fromfields = sys.argv[3].split(";") try: ucur = GP.UpdateCursor(inTabToCalc) GP.addmessage("--- opened update cursor") print GP.getMessages() rec = ucur.next() i = 0 for tofield in tofields: fromfield = fromfields[i] GP.addMessage("calculate " + tofield + " = " + fromfield) print GP.getMessages() i = i + 1 while rec: # Calculate the value i = 0 for tofield in tofields: fromfield = fromfields[i] thevalue = rec.getvalue(fromfield) rec.setvalue(tofield,thevalue) i = i + 1 ucur.updaterow(rec) rec = ucur.next() except: GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 53

211 # # getmaximumvalue.py # # Created on: 10/20/04 by Greg Nichols # # Import system modules import sys, string, os, win32com.client, traceback # Create the Geoprocessor object GP = win32com.client.dispatch("esrigeoprocessing.gpdispatch.1") # Load required toolboxes... if GP.Exists(r"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/program Files/ArcGIS/ArcToolbox/Toolboxes" if GP.Exists(r"C:/ArcGIS/ArcToolbox/Toolboxes"): systbxpath = r"c:/arcgis/arctoolbox/toolboxes" GP.AddToolbox(sysTbxPath + "/Data Management Tools.tbx") GP.AddToolbox(sysTbxPath + "/Analysis Tools.tbx") # Script arguments... intab = sys.argv[1] thefieldname = sys.argv[2] intabtocalc = sys.argv[3] infieldtocalc = sys.argv[4] # CALCULATE AND GET THE MAXIMUM VALUE try: # Run the Statistics GP.Workspace = "c:/temp/" GP.addMessage("Starting to run statistics") print GP.getMessages() GP.statistics(inTab, "maxstat.dbf", thefieldname + " " + "MAX") GP.addMessage("Done running statistics") print GP.getMessages() statfield = "max_" + thefieldname # Open the Cursor cur = GP.SearchCursor("c:/temp/maxstat.dbf") GP.addmessage("--- opened cursor") print GP.getMessages() rec = cur.next() while rec: thevalue = rec.getvalue(statfield) rec = cur.next() GP.addMessage("Found max value of " + str(thevalue)) print GP.getMessages() except: GP.addMessage(GP.getMessages(2)) GP.addMessage("Error in block to retrieve maximum value",2) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") 54

212 GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) # START WITH THE MAXIMUM VALUE, CALCULATE AND INCREMENT A FIELD try: ucur = GP.UpdateCursor(inTabToCalc) GP.addmessage("--- opened update cursor") print GP.getMessages() # increment the value thevalue = thevalue + 1 rec = ucur.next() while rec: # calculate rec.setvalue(infieldtocalc, thevalue) # increment the value thevalue = thevalue + 1 # next ucur.updaterow(rec) rec = ucur.next() except: GP.addMessage(GP.getMessages(2)) GP.addMessage("Error in calculating the value into ",2) GP.addMessage(GP.getMessages(2)) print GP.getMessages() GP.addError('-'*60) GP.addError(' '*20 + "CRASHED") GP.addError("Error On Line: " + str(traceback.tb_lineno(sys.exc_info()[2]))) GP.addError('-'*60) 55

213 Model: Merge SanGIS and Landcore 56

214 Model: Landcore Preparation 57