Oracle Spatial Summit

Transcription

1 Oracle Spatial Summit 2015 Oracle Spatial 12c as an Applied Science for Solving Today's Real-World Engineering Problems Tracy McLane, GIS Manager Bechtel Infrastructure

2 Speaker Bio Tracy McLane is currently the GIS Manager for the Bechtel Infrastructure Global Business Unit (GBU), where she also serves as the GIS Technical Discipline Lead for the company. She is responsible for the development of the GIS Technical Center of Excellence for Bechtel and has implemented an Enterprise GIS database and centralized GIS Knowledge Bank for the Bechtel GIS user community. Tracy was named as one of ten Bechtel Distinguished Scientists in the Bechtel Distinguished Engineers and Scientists (BDES) program in 2012, and currently serves as the Vice Chairman on the Oracle Spatial Special Interest Group (SIG) Board of Directors in the Independent Oracle Users Group (IOUG). Ms. McLane has worked in the GIS industry for more than 20 years, and has provided GIS support for each of Bechtel s Global Business Units (GBUs) during her 17+ years with the company. Ms. McLane s previous GIS experience includes work at the Tennessee Valley Authority and the U. S. Department of Energy Bechtel Savannah River Site before taking her current position with Bechtel in March of Ms. McLane holds a Masters of Science degree in Geography from the University of Tennessee in Knoxville, Tennessee and a Bachelor of Arts degree in International Business from Eckerd College in Saint Petersburg, Florida.

3 Bechtel Corporate Enterprise GIS (BecGIS) OVERVIEW Company: Bechtel Top U.S. construction contractor by Engineering News- Record (ENR) for 17 straight years Bechtel has worked on more than 25,000 projects in 160 countries on all seven continents. Bechtel has over 40 offices and 58,000 employees in nearly 40 countries. Signature Projects: Hoover Dam, English Channel Tunnel, Hong Kong International Airport, Ivanpah Solar Energy Site System: BecGIS Internal users for multiple Global Business Units (GBUs) Development and Production Environments GIS Architecture on Bechtel Network CHALLENGES / OPPORTUNITIES GIS Data Configuration Management Processes Spatial Data Content Sharing and the User Experience Scalable, Multi-Tiered Data Security Integration of Disparate Data Effective Spatial Data Process Workflows Data Integrity and Documentation of Accuracies SOLUTIONS Oracle Database 12c Enterprise Edition Use of Spatial Views for Filtering and Analysis Oracle Automation with PL/SQL Use of JSON Parsing to Create SDO Geometries RESULTS Centralized spatial data repository with feature-level metadata and reduced data maintenance costs More effective Big Data handling and processing Leveraging Oracle database automation to reduce data analysis costs

4 Objectives Bechtel's large engineering and construction projects require a vast amount of varied spatial data for real-world problem solving. The ability to manage and manipulate these data on an Enterprise basis is well-served in an Oracle Spatial and Graph 12c database environment, where the spatial data that technical specialists and engineers need can easily be analyzed to support the work that they do. JSON data file loading and parsing for indoor location tracking earthquake catalog classification for seismic hazard study analyses by the seismologist dynamic hydraulic gradient calculations used for groundwater modeling by the hydrologist dynamic commodity cost calculations for construction materials for the field engineer This presentation will take an in-depth look at Oracle Spatial technologies as an applied science in the world of engineering using the above case studies Bechtel 4 Level 4 - Bechtel Public

5 Handling of Unstructured Data With the variety of data available for spatial analysis, not all of it is structured in such a way to make it readily usable for the creation of spatial geometry and analytics. A common format for today s unstructured data is JSON, with GeoJSON it geospatial counterpart for GIS users. However, not all unstructured data is readily available in GeoJSON, which then limits the ability to integrate the data with other spatial datasets for analytics, and in the new world of Big Data, this is especially important Bechtel 5 Level 4 - Bechtel Public

6 Copying the JSON Files to the Server One of the first steps is to get the JSON files to a location the database. The use of a database-defined external directory on the Oracle database server is one way to do this. Example: Copy JSON from GIS Share Drive to Oracle Server (Copy_JSON.bat Code by Tracy McLane) copy <server input directory>\*.json <server output directory>\*. \. del <server input directory>\*. \*.json copy <share drive directory>\*.json <server input directory>\. sqlplus <user>@<oracle <input directory>\parse_json.sql" The batch file to copy the JSON files can be put on a server scheduled task: Example: Scheduled Task Copy JSON File 1-minute intervals) Code by Tracy McLane schtasks /create /sc minute /mo 1 /tn "Copy JSON Files" /tr <server input directory>\copy_json.bat 2016 Bechtel 6 Level 4 - Bechtel Public

7 Loading the JSON Data One way of handling large JSON datasets that lack a standardized structure is to load the files into a JSON table in an Oracle Spatial database from files located in a database-defined external directory ( Creating an External Table, Oracle Database 10g: The Complete Reference, p 479). Example: Code by Tracy McLane CREATE TABLE <json_table> (json_document CLOB) ORGANIZATION EXTERNAL (TYPE ORACLE_LOADER DEFAULT DIRECTORY <external directory> ACCESS PARAMETERS (RECORDS DELIMITED BY 0x'0A' DISABLE_DIRECTORY_LINK_CHECK BADFILE <external directory>: 'JSONDumpFile.bad' LOGFILE <external directory>: 'JSONDumpFile.log' FIELDS (json_document CHAR(5000))) LOCATION (<external directory>:'*.json')) PARALLEL REJECT LIMIT UNLIMITED; The Oracle table JSON column must be defined as a CLOB type, to accept whatever format the JSON file may use, although additional columns may be added to the Oracle JSON table as well Bechtel 7 Level 4 - Bechtel Public

8 Parsing the JSON Data Once the JSON data is loaded as an Oracle database table, the next step is to parse out the coordinate information from the table using a declared cursor to loop through the data. Below is an example excerpt from PL/SQL scripting to do this using the Oracle 12c (12.1) function JSON_VALUE*: Example: Code by Tracy McLane OPEN <cursor> FOR SELECT JSON_VALUE(json_document,'$.id'), JSON_VALUE(json_document,'$.position_x'), JSON_VALUE(json_document,'$.position_y'), JSON_VALUE(json_document,'$.position_z'), FROM <json_table> ORDER BY JSON_VALUE(json_document,'$.ID'); Source: Usage of JSON_VALUE can be found in the JSON_TABLE section of the online Oracle Help Center Database SQL Language Reference at *Note: In Oracle 12c (12.2), there is a new option when using JSON_VALUE to parse and extract the SDO_GEOMETRY directly from the JSON file, but this is not available in earlier versions of the Oracle database Bechtel 8 Level 4 - Bechtel Public

9 Creating SDO Geometry Points Finally, once the data has been read into the cursor, the records can be retrieved and loaded into variables for input into Oracle Spatial SQL statements to create spatial features in an Esri geodatabase, where the feature class was created using an ArcCatalog configuration keyword to define the geometry type as SDO_GEOMETRY. Below is an example of PL/SQL scripting using the FETCH function for a cursor to create point SDO_GEOMETRY of indoor tracking positions (which is why an Oracle Spatial Reference ID (SRID) is not defined in the PL/SQL script example shown below). Example: LOOP FETCH <cursor> INTO irecid,icoordx,icoordy,icoordz; Code by Tracy McLane EXIT WHEN currow%notfound; INSERT INTO <SDO_GEOMETRY feature class> (OBJECTID, COORD_X, COORD_Y,COORD_Z,UPLOAD_TIME,SHAPE) VALUES (irecid,icoordx,icoordy,icoordz,systimestamp, SDO_GEOMETRY (3001,null, SDO_POINT_TYPE (icoordx,icoordy,icoordz), null,null) ); isdonum := isdonum + 1; END LOOP; Source: Usage of Oracle PL/SQL cursors can be found in Cursors, Oracle Database 12c SQL: Master SQL and PL/SQL, p 390) Bechtel 9 Level 4 - Bechtel Public

10 Next, a spatial view can be created to filter out the most recent record for each unique entity in the dataset. In the case of GPS/GNSS tracking data, this can be used to select the most recent location of a person or object you are tracking: Example: Filtering the SDO Geometry Points with Spatial Views One set of spatial views to show most recent location A second set of spatial views to show recent tracking history Spatial Views GIS Client 2016 Bechtel 10 Level 4 - Bechtel Public

11 Process Improvement for Current Earthquake Catalog Data Process Spatial Views are used to filter earthquake data by magnitude for map visualization of event data. Different source data contain different types of magnitude information: Mw moment magnitude Ms surface magnitude Mb body-wave magnitude Emb - The Emb magnitude Smb - The Smb magnitude Rmb - The Rmb magnitude Umag - UMS Magnitude Scale Reference: Earthquake Catalog Data used to Support Seismic Hazard Study in Papua New Guinea Earthquake symbology can be shown by magnitude intensity, magnitude class or depth to the earthquake epicenter 3D Visualization of Oracle Spatial Earthquake Point Geometries Displayed with their Distance to the Earthquake Epicenter (KM) 2016 Bechtel 11 Level 4 - Bechtel Public

12 Applying the JSON Process to Improve a Handling of Earthquake Catalog Data Earthquake catalog data comes in a variety of data formats, and for mapping purposes, it is necessary to standardize data to a common data model so sources can be merged. The GIS team previously worked with the seismologists within Bechtel to develop a GIS web mapping application to allow the user to upload their earthquake catalog data and manually map the columns of the source files to the data model target. The tool then creates point geometries from the source data, and allows users to merge and filter the data points for analysis and mapping. Catalog A Catalog B Catalog C The JSON process above offers a more automated way to parse and load these disparate and large data into a single earthquake data model as SDO points geometries. In addition, the use of spatial views can provide users a way to automate some of the filtering processes currently being defined on a more manual basis Bechtel 12 Level 4 - Bechtel Public

13 Spatial Views for Dynamic Hydraulic Gradient Calculations Dynamic hydraulic gradient calculations used for groundwater modeling Spatial Views with calculated columns of SDO Geometries: h = (from_elev-to_elev)/(sdo_geom.sdo_length(shape,0.001) L = SDO_GEOM.SDO_LENGTH(shape,0.001) 2015 Bechtel 13 Level 4 - Bechtel Public

14 Spatial Views for Commodity Cost Calculations for Construction Materials Using spatial views to aggregate construction materials geographically Decoding values from look-up table to add commodity description for linking with the unit rates table DECODE (fence_line.material_d,d_fentyp.domainvalu,d_fentyp.domaindesc,'unknown') material, Calculating contract awarded versus contingency commodity quantities SUM(VTBL_FIELD_QUANT_REPORT.FEAT_QUANT) QUANT_AWRD, SUM(UTELECTG.CTG_QUANT) QUANT_CTG, SUM(UTELECTG.CTG_QUANT+VTBL_FIELD_QUANT_REPORT.FEAT_QUANT) TOTAL_AWRD 2015 Bechtel 14 Level 4 - Bechtel Public

15 Conclusion By leveraging the database for what is does best...the data...technical specialists and engineers alike can spend less time on data manipulation and analysis, and even improve the quality of their deliverables through database automation of processes that are data-intensive and often subject to human error. Oracle Spatial Technology Focus:» Data File Handling and Database Automation» Spatial View Filters, Calculations and Analytics The case studies shared in this presentation focus on leveraging the use of database automation and spatial views at Bechtel to support some of the work needed to engineer and construct large infrastructure around the globe Bechtel 15 Level 4 - Bechtel Public

16 Questions?

17 Contact Information: Tracy J. McLane Bechtel Infrastructure, GIS Manager