Example Procedure Log

Size: px

Start display at page:

Download "Example Procedure Log"

Rolf Harris
5 years ago
Views:

1 CLKIMMEL.COM Example Procedure Log From City of Raleigh Detailed Intersection Project Charles Lee Kimmel 3/27/2017 This is an example procedure log from my work on the Detailed Intersection Project with the City of Raleigh. I created this project after a need was expressed to me that the detailed intersection files were a mess and slowing down efficiency with the GIS staff. Below illustrates how to replicate the project on a click by click basis. It may be useful to apply the principles and thought processes to another project based on these instructions. This information is 2017 clkimmel.com and City of Raleigh. DO NOT DUPLICATE OR SHARE WITHOUT EXPRESSED PREMISSION FROM AUTHOR OR CITY OF RALEIGH.

1 2 3 4 5 6 7 8 9 10 City of Raleigh Public Utilities GIS Capital City Improvements Prepared by: Charles Lee Kimmel Date: 1 /22/2017 Procedure log for the Detailed Intersection Project: Brief

2 City of Raleigh Public Utilities GIS Capital City Improvements Prepared by: Charles Lee Kimmel Date: 1 /22/2017 Procedure log for the Detailed Intersection Project: Brief background: The Raleigh PUGIS utilizes historical drawings referred to as detailed intersection drawings (DID) of waterlines along roadways or more commonly road intersections. These historic documents were archived in books and as microfilm. The drawings were later scanned by the department for geocoding into ESRI ArcMap so the PUGIS team could utilize them to aid in the development of the waterline feature class. Problems: The DID are not properly geocoded in ArcMap. Many of the points are scattered, overlap or do not match their intersection. The DID are utilized throughout the PU department and the GIS team and there are many duplicates of these files which should be removed for consistency. Organizational and naming convention issues are abundant with the files. It was discovered that the scanned files are missing various pages from the books. In an attempt to fix this the microfilm (previously thought to be complete) were scanned as well to complete the digital archive. It has also been determined that each set is incomplete and can also contain multiple copies of files. There are now two separate incomplete sets of DID files. Also within each set the filenames differ greatly because the people scanning the files did not follow the same standard filename convention. Efficiency for the GIS staff to utilize these files to aid in the modification or addition of waterline has been degraded. 23 Tasks: Locate all known DID files Organize the files, remove and archive duplicate files (i.e. if one file is duplicated from the scans and the microfilm then the microfilm will be archived into a folder named duplicates with (2).pdf added the filename). Standardize filenames: o The files will be sorted alphabetically and evaluated for standard naming errors. o All the files will be standardized with the correct road type o Separators will be standardized to & o Names of roads will be evaluated for correctness if they are unable to geocode Geocode files: o Files will be geocoded via a.csv by the QGIS MMQGIS geocoding plugin o Develop new schema for the feature class o Utilize Google to locate intersections o Store filenames that could not be located in a notfound.csv file o Manually add not found files via the ArcMap editor o Double check precision of the QGIS geocode output in ArcMap Final product

3 o o o Add the final geocoded feature class to the SDE Upload all files to the main network drive to final DID_Project folder Remove all previous DID files from the main network drive and send out an memorandum order to remove old DID files from personal computers and network folders Software Needed: ArcGIS, MS Excel, Notepad++ (or any text editor that can block select), QGIS Begin Procedure Log Section 1: Locating files and Updating Filename Convention Locate all DID files Search and find all the detailed intersection files on the main network drive The files should be divided into two types Scanned files Microfiche (microfilm) Once located move them into */ directory for processing Update files: Filenames must be updated and will currently appear as follows 56 Scanned files: This That Rd Micofilm files: THIS THAT RD This will help distinguish between the two sets of files while updating names in the future. Open command window (CMD): Right Click + Shift on the folder for the microfilm files and the command window will appear. In the command prompt type: dir /b > _list.txt This will generate a list of the files in that folder Repeat this step for the scanned folder Open the folder for the microfilm files then open _list.txt into notepad++.exe Erase any non-did files that may exist (i.e. NOTADIDFILE.BAT) from _list.txt including _list.txt Double Check that files already look like the above example: THIS THAT RD If not press ctrl + A and right click on the selection and press uppercase to convert all the text and then save Open the folder for the scanned files then open _list.txt into notepad++.exe Erase any non-did files that may exist (i.e. notadidfile.bat) from _list.txt including _list.txt

4 Double Check that files already look like the above example: This That Rd If not press alt and block highlight all the letters of a word except the first letter and convert it to upper or lowercase whichever is needed. After all names have been fixed then save. *NOTE* this step should not be needed for the scanned files but this is listed just in case Create folders: Create a main folder named DID_Project Create folders: Create subfolders in \DID_Project for each letter of the alphabet, duplicates, Micofilm and Scanned Move all files into either Scanned or Micofilm directory. Create a new _list.txt for each the Scanned and Micofilm folder following previous steps Open the included MS Excel template file with the VBA macros named Rename_List_Macro.xlsm or create a new excel sheet and manually update records. Note the VBA code has been included at the end of this document along with more instructions to create a new macro file. Open the _list.txt file from each Scanned or Micofilm into NotePad and remove all non-did filenames from the list. Past the original filenames into column A and B. Column C should have the following code for each filename: =CONCATENATE("ren """,A#,""""," ","""",B#,"""") update the # for the appropriate row number. Run the standardizing macros for the Microfilm (MICROFICHE) and the macro for Scanned (ODID) Here are a few examples of the following items that will be updated (for a complete list see VBA at, &, and, or To: & Rd, St, Blvd, Ct, Cir, Ln, etc To: Road, Street, Boulevard, Court, Circle and Lane Copy column C to column D and paste as values only to remove the formula Open command window (CMD): Right Click + Shift on the folder for the microfilm files and the command window will appear. Copy column D in excel and Right Click on the CMD window and select paste This will rename all the files to the new names from Excel column B

5 Move and combine the scanned and microfilm files together into their appropriate letter subdirectory in this order so the microfilm files will be the last ones added Beginning with folder A: Any duplicated files that are microfilm at this point will be prompted to be automatically named to EXAMPLE (2).pdf and then should be moved to the duplicated folder. Open command window (CMD): Right Click + Shift on the folder and the command window will appear. In the command prompt type: dir /b > _list.txt This will generate a list of the files in that folder Repeat these steps for each letter subfolder Now all files have been located and stored on the network drive under /DID_Project *(to be updated when finished) Section 2: Standardizing Filenames Further and Creating the.csv Geocoding Template Open _list.txt from folder A into NotePad and remove all non-did files. Open MS Excel and name the default worksheet GEOCODE_A Create the following column headers: FID, ADDRESS, GEOCODING_C, LOCATED, GRID_TYPE, DUPLICATED, CITY, STATE, COUNTRY, HYPERLINK, CODE_L Create another worksheet in excel named WORKING and add the following column headers: FID, INTERSECTIONS, DUPLICATED, GEOCODE, ORIGINAL, NEW FILE Copy the filenames from _list.txt into worksheet WORKING under column INTERSECTIONS and column ORIGINAL Examination of file names: Highlight column INTERSECTIONS and select conditional formatting > highlight cell rules > duplicate values At this point if there are any duplicates in the column they will be highlighted red (although this should not be the case at this time because you should have already been prompted for duplicates when copying and combining the files to their subfolders). Populate the FID field giving each file a unique number Highlight the column INTERSECTIONS and press Ctrl + H Find and Replace the following in brackets with what is in curly brackets in the following order (these are case sensitive): [.PDF] {.pdf}, [ ] { }, [, ] { & }, [U.S. #1] {US 1}, [Gird] {GRID], [Stirp Map] {Strip Map} Now manually examine the filenames for:

6 Spelling mistakes Parenthesis that are broken A common issue with the microfilm files can be that they do not have any road type indicator after the road name so this has to be manually added Any other issues that may prevent duplicate files for matching As these issues are fixed the column should begin to highlight the new duplicated filenames Once the entire list has been examined sort the column INTERSECTIONS by cell color and A-Z Fill in the column DUPLICATED : Any cell that is NOT highlighted = N Any highlighted cell that is All Caps for microfilm = 2 Any highlighted cell that is Normal text for Scanned = Y After this is complete sort by column DUPLICATED and highlight any cell in column INTERSECTIONS that has a 2 in column DUPLICATED associated with it. Find and Replace the following in brackets with what is in curly brackets (these are case sensitive): [.pdf] { (2).pdf} Copy column INTERSECTIONS to column GEOCODE here Find and Replace the following in brackets with what is in curly brackets (these are case sensitive): [ (Grid*] {}, [ (Strip*] {}, [ (Zone*] {} and [.pdf] {} This will clear the cells of any information after the road names Reopen: Rename_List_Macro.xlsm Past column ORIGINAL into column A and column INTERSECTIONS into column B Copy column C to column D and past as values only and then copy column D again Open command window (CMD): Right Click + Shift on folder A and the command window will appear. 157 In the command prompt: Right click and select paste This will now update all the names in folder A Populate the worksheet GEOCODE_A: Copy from WORKING Sheet TO GEOCODE_A Sheet FID = Give sequential numbers for each file name i.e ADDRESS = Copy column INTERSECTIONS here GEOCODING_C = Copy column GEOCODE here LOCATED = Assign all cells to M GRID_TYPE = Copy column INTERSECTIONS here this will be the type of drawing or grid number Do a Find and Replace the following in brackets with what is in curly brackets (these are case sensitive): [*(Grid] { GRID}, [*(Strip] { Strip}, [*(Zone] { Zone} and [).pdf] {} DUPLICATED = Copy column DUPLICATED here

7 CITY = Raleigh STATE = North Carolina COUNTRY = USA HYPERLINK = *\A\Example Road & This Street (Grid 000).pdf The easiest way to generate this is to prefill the standard directory name in this column and then concatenate. Enter =J2&A2 into cell L and copy the quick fill the formula down and copy and paste values only into column HYPERLINK CODE_L = The Letter of the Folder the drawings are from i.e. A Save the file as GEOCODE_A.xlsx and another as GEOCODE_A.csv Open QGIS Import the MMQGIS PLUGIN Open the MMQGIS PLUGIN and select GEOCODE > Geocode CSV with Google / OpenStreetMap Geocode CSV with Google / OpenStreetMap Tool: To utilize this tool click Browse > find and select GEOCODE_A.csv > then fill in the fields as follows: Address Field = GEOCODING_C, City Field = CITY, State Field = STATE, Country Field = COUNTRY, Web Services = Google Maps Google API Key = (none). Output Shapefile = *\A\Shapefile\GEOCODE_A.shp Not found Output List = *\A\Shapefile\notfound.csv See Figure 1 for detailed layout of QGIS and Figure 2 for the geocoding tool. QGIS will show the geocoding progress in the lower left corner of the window. Once this is complete two output files will be generated. 1. A shapefile containing the newly geocoded points and 2. a comma delimited values file (.csv) that will contain the intersections that could not be located.

188 189 190 Figure 1: QGIS program layout.

8 Figure 1: QGIS program layout. A: is the location of the MMQGIS plugin that the geocoding tool is located. B: represents the progress of the geocoding tool during processing. 191

192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 Figure 2: Geocode CSV with Google / OpenStreetMap Tool.

9 Figure 2: Geocode CSV with Google / OpenStreetMap Tool. To utilize this tool select Browse for select GEOCODE_A.csv > then fill in the fields as shown above. Address Field = GEOCODING_C, City Field = CITY, State Field = STATE, Country Field = COUNTRY, Web Services = Google Maps Google API Key = (none). Output Shapefile = *\A\Shapefile\GEOCODE_A.shp Not found Output List = *\A\Shapefile\notfound.csv Press OK after all the fields have been filled in QGIS will geocode all the files from the GEOCODING_C column Open ArcGIS and add the newly generated GEOCODE_A.shp to the data view. Update the symbology to make the points more visible Open the attribute table From the WAKE SDE add the following features: WAKE.HYDRO_LINE WAKE.PROPERTY WAKE.RAILROAD WAKE.STREET_ALL WAKE.YE_OLE_TAX_GRID From the RPUD SDE add the following features: RPUD.wAbandonedLine RPUD.wControlValve RPUD.wFitting RPUD.wHydrant RPUD.wLateralLine RPUD.ssManhole RPUD.wNetworkStructure RPUD.wPressureMain

10 Apply the appropriate symbology to each feature class (See Figure 3 for an example) From ArcGIS services maps.raleighnc.gov add all the ortho-images to the data view Begin an edit session on GEOCODE_A.shp Select points from GEOCODE_A and the intersecting roads under that point Check that the filename matches the intersecting road names If they do then type Y into the LOCATED field for that feature If not then Open the pdf file that was not properly located and examine the names on the drawing for to make sure there are no spelling errors in the filename and possible road realignments or name changes. Utilize Google Maps to double check the road names and locations. Fix any errors found and update all occurrences of the filename i.e. in all entries in ArcMap and the actual file. To locate the correct intersection or roads from the previous step do the following In ArcGIS Search by attribute WAKE.STREETS_ALL and enter the following CARTONAME LIKE %EXAMPLE 1% OR CARTONAME LIKE %EXAMPLE 2% DO NOT ENTER THE NAMES AS FOLLOWS: CARTONAME LIKE %EXAMPLE 1 ROAD% OR CARTONAME LIKE %EXAMPLE 2 STREET% The % symbol is a wildcard character this will allow the SQL query to look for any strings associated with the name i.e. CARTONAME LIKE %EXAMPLE 1% can return many possible selections for example: EXAMPLE 1 ST, EXAMPLE 1 RD, EXAMPLE 1 BLVD, W EXAMPLE 1 ST, and etc. Utilize this operation to narrow the search that most logically fits the intersection names. If the correct intersection is located after searching move the point over the newly selected intersection or location If the intersection still could not be located then further analysis is needed either enter N into the LOCATED field or try and locate the intersection through other historical documentation. Follow these procedures to enter and double check all geocoded points. Save edits Open the *\notfound.csv file in Excel Open Google Maps or utilize any files with historic road names Analyze historic documents and Google Maps to locate any of the intersections in the notfound file. Select create feature and manually add the notfound intersections to the shapefile Copy and paste the attributes from the notfound.csv file into the shapefile attribute table

249 250 251 252 253 Add all the notfound intersections manually until completed Click save edits to finalize the changes NOTE: it is a good management practice to save edits often to prevent data

11 Add all the notfound intersections manually until completed Click save edits to finalize the changes NOTE: it is a good management practice to save edits often to prevent data loss or degradation All the steps from Section 2 need to be repeated for each subfolder (i.e. folders A, B, C,, Z) until all the files are properly geocoded Figure 3: ArcMap layout. This represents the general layout of ArcMap during this project. Note the symbology of roads and DID points. The utilization of max and min visibility of RPUD features and the orthoimages was set to 1:20,000 which will greatly boot map refresh performance.

258 259 260 Figure 4: Utilize the attribute table in quadrants.

12 Figure 4: Utilize the attribute table in quadrants. This will allow all the important feature attributes open to help locate known and unknown intersections.

13 Detailed Intersection Drawing Project Workflow Start Develop a VBA script to automate the standardization. Start workflow Key to Workflow Multiple documents Locate all Scanned files Microfilm files No Are all DID files located? Do most the files need to be fixed? Manually fix the few filename problems Prepare action Decision Process Termination point Yes Filenames have been standardized Are filenames in a standardized convention? No Rename and organize files to prepare for geocoding Yes Repeat process until A-Z files have been geocoded Are filenames clear enough to be geocoded? Yes No Open ArcMap and QA/QC that QGIS geocoded correctly Sort all shapefiles and insure that they are complete No Begin an edit session and manually correct errors or add files that were not located by QGIS geocoding Create directories for letters A-Z and sort all files in to their alphabetical directory Geocode files using QGIS alphabetically output will be GEOCODE_%.shp

14 MS Excel VBA Code: This section will discuss the creation of VBA macros in MS Excel. To create new macros in Excel open the visual basic editor. To do this select the file tab and then click the options button (see figure 5). Figure 5: Excel 2010 menu layout and location of macros button. Then click Customize Ribbon and under Customize the Ribbon: select Developer (see figure 6) and press OK.

15 Figure 6: Location of the Developer tab to access the visual basic editor. Click on the visual basic button to open the editor window (figure 7). Figure 7: Access the VBA editor in Excel. Inside the project window right click and select Insert > Module (figure 8).

Figure 8: The visual basic editor in MS Excel. Create 2 new modules and paste the Microfilm VB Code into module 1 and then paste the Scanned Files VB Code into module 2.

16 Figure 8: The visual basic editor in MS Excel. Create 2 new modules and paste the Microfilm VB Code into module 1 and then paste the Scanned Files VB Code into module 2. Brief understanding of the code: Columns("X:X").Select = Where X is any letter that represents the desired column Selection.Copy = This command copies the selected cells ActiveSheet.Paste = This will paste the clipboard into selected cells Application.CutCopyMode = False = prevents the data from being cut or deleted from copy Selection.Replace What:="X ", Replacement:="Y", This command will replace X with Y in a selection. Copy this whole segment, change X and Y to the desired stings for search and replace. LookAt =xlpart is used to search for any part of the string. Use xlwhole to search the entire cell for matching cases. SearchOrder by rows searches across rows then down to the next row, by columns searches down columns then to the next column. MatchCase matches the case of the text being searched. Search format looks for a predefined format when searching. = Replaces the format of a selected cell. Sub SOMENAME() = This defines the start of a script End Sub = This defines the end of a script The remaining code should be copied as typed here is no further understanding of VBA exists.

17 Microfilm VBA Macro Code: Sub MICROFICHE() ' ' MICROFICHE Macro ' ' Columns("A:A").Select Selection.Copy Columns("B:B").Select ActiveSheet.Paste Application.CutCopyMode = False Columns("B:B").Select Selection.Replace ", Replacement:=" & ", Selection.Replace What:=". & ", Replacement:=" & ", Selection.Replace What:=". - ", Replacement:=" - ", Selection.Replace What:=". ", Replacement:=" & ", Selection.Replace What:=".,", Replacement:=" & ", Selection.Replace What:="_", Replacement:=" ", LookAt:=xlPart, _ Selection.Replace What:=" ST ", Replacement:=" STREET ", Selection.Replace What:=" AVE ", Replacement:=" AVENUE ", LookAt:=xlPart _, Selection.Replace What:=" BLVD ", Replacement:=" BOULEVARD ", LookAt:= _ xlpart, Selection.Replace What:=" BLVD,", Replacement:=" BOULEVARD,", LookAt:= _ xlpart, Selection.Replace What:=" LN ", Replacement:=" LANE ", Selection.Replace What:=" PL ", Replacement:=" PLACE ", Selection.Replace What:=" RD ", Replacement:=" ROAD ", Selection.Replace What:=" TR ", Replacement:=" TRAIL ", Selection.Replace What:=" TRL ", Replacement:=" TRAIL ", Selection.Replace What:=" DR ", Replacement:=" DRIVE ", Selection.Replace What:=" CT ", Replacement:=" COURT ", Selection.Replace What:=" CIR ", Replacement:=" CIRCLE ", LookAt:=xlPart _, Selection.Replace What:=" PL ", Replacement:=" PLACE ", Selection.Replace What:=" ST.", Replacement:=" STREET.", Selection.Replace What:=" DR.", Replacement:=" DRIVE.", Selection.Replace What:=" LN.", Replacement:=" LANE.", Selection.Replace What:=" TR.", Replacement:=" TRAIL.",

18 Selection.Replace What:=" BLVD.", Replacement:=" BOULEVARD.", Selection.Replace What:=" PL.", Replacement:=" PLACE.", Selection.Replace What:=" AVE.", Replacement:=" AVENUE.", LookAt:=xlPart _, Selection.Replace What:=" CT.", Replacement:=" COURT.", Selection.Replace What:=" CIR.", Replacement:=" CIRCLE.", LookAt:=xlPart _, Selection.Replace What:=" RD.", Replacement:=" ROAD.", Selection.Replace What:=" DR.", Replacement:=" DRIVE.", Selection.Replace What:=", ", Replacement:=" & ", Sheets("File_List_Working").Select Selection.Replace What:=" - ", Replacement:=" (GRID ", Selection.Replace What:=".pdf", Replacement:=").pdf", Columns("C:C").Select Selection.Copy Columns("D:D").Select Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, SkipBlanks _ :=False, Transpose:=False End Sub Scanned Files VBA Macro Code: Sub ODID() ' ' ODID Macro ' ' Columns("A:A").Select Selection.Copy Columns("B:B").Select ActiveSheet.Paste Application.CutCopyMode = False Columns("B:B").Select Selection.Replace ", Replacement:=" & ", Selection.Replace What:=" Tr ", Replacement:=" Trail ", Selection.Replace What:=" Tr.", Replacement:=" Trail.", Selection.Replace What:=" Tr,", Replacement:=" Trail,", Selection.Replace What:=" Rd ", Replacement:=" Road ", Selection.Replace What:=" Dr ", Replacement:=" Drive ", Selection.Replace What:=" St ", Replacement:=" Street ", Selection.Replace What:=" Ave ", Replacement:=" Avenue ", LookAt:=xlPart _, Selection.Replace What:=" Ave, ", Replacement:=" Avenue, ", LookAt:=xlPart _, Selection.Replace What:=" Ln ", Replacement:=" Lane ", Selection.Replace What:=" Cir ", Replacement:=" Circle ", LookAt:=xlPart _, Selection.Replace What:=" Blvd ", Replacement:=" Boulevard ", LookAt:= _

19 xlpart, Selection.Replace What:=" Blvd.", Replacement:=" Boulevard.", LookAt:= _ xlpart, Selection.Replace What:=" Blvd,", Replacement:=" Boulevard,", LookAt:= _ xlpart, Selection.Replace What:=" Tr ", Replacement:=" Trail ", Selection.Replace What:=" Trl ", Replacement:=" Trail ", Selection.Replace What:=" Rd.", Replacement:=" Road.", Selection.Replace What:=" Rd,", Replacement:=" Road,", Selection.Replace What:=" Dr,", Replacement:=" Drive,", Selection.Replace What:=" Ct,", Replacement:=" Court,", Selection.Replace What:=" Ct ", Replacement:=" Court ", Selection.Replace What:=" Cir,", Replacement:=" Circle,", LookAt:=xlPart _, Selection.Replace What:=" Ln,", Replacement:=" Lane,", Selection.Replace What:=" St,", Replacement:=" Street,", Selection.Replace What:=" Pl,", Replacement:=" Place,", Selection.Replace What:=" Pl ", Replacement:=" Place ", Selection.Replace What:=" Dr.", Replacement:=" Drive.", Selection.Replace What:=" Trl ", Replacement:=" Trail ", Selection.Replace What:="_", Replacement:=" ", LookAt:=xlPart, _ Selection.Replace ", Replacement:=" & ", Selection.Replace What:=". & ", Replacement:=" & ", Selection.Replace What:=". - ", Replacement:=" - ", Selection.Replace What:=". ", Replacement:=" & ", Selection.Replace What:=".,", Replacement:=" & ", Selection.Replace What:=", ", Replacement:=" & ", Selection.Replace What:="_", Replacement:=" ", LookAt:=xlPart, _ ActiveWindow.ScrollColumn = 2 Columns("C:C").Select Selection.Copy Columns("D:D").Select Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, SkipBlanks _ :=False, Transpose:=False End Sub

CEU Online System, The Friday Center for Continuing Education, UNC-Chapel Hill How to Obtain Participant IDs for Awarding of CEUs

CEU Online System, The Friday Center for Continuing Education, UNC-Chapel Hill How to Obtain Participant IDs for Awarding of CEUs The Friday Center for Continuing Education has the responsibility of approving continuing education activities for which CEUs are recorded and maintained as a permanent record for individual participants.