Addressing the Need for a Bathymetric Data Management System

Transcription

1 Addressing the Need for a Bathymetric Data Management System Authors: Karen Cove, CARIS Fred Lavoie, Canadian Hydrographic Service Abstract The implementation of a bathymetric data management system provides opportunities for an organization to better meet the demand of its clients. This can be accomplished by providing hydrographic information to governmental and non-governmental agencies from a single, controlled source. CARIS has addressed this need by developing the Bathy DataBASE, a bathymetric data management solution. The system components and architecture, as well as, atypical workflow are described. This paper looks at testing done by the Canadian Hydrographic Service (CHS) Quebec region to improve the management of its bathymetric data. As with most hydrographic offices, volumes of data produce by new sonar technology (MBES) and requests for bathymetric data are increasing at the CHS. The Quebec region of the CHS is evaluating Bathy DataBASE to improve the management of the bathymetry results from various types of surveys (singlebeam, MBES, lead-line, etc). The goal is to respond, in the most efficient way, to all requests from its clients. Bathymetry extracts can be tailored to meet a specific set of criteria (epoch, resolution, attribution, etc.) and provided to clients for applications such as navigation, coastal zone management, emergency decision support, etc. Introduction Current assessment, storage and management needs for bathymetric data are pushing the hydrographic organization, and other large holder of bathymetric data, to investigate new data management solutions. Many organizations have a need to assess large volumes of bathymetric data coming in from various sources for possible threats to safety of navigation. When the data is deemed valid, tools are needed to prepare and format the data prior to storage. CARIS Bathy DataBASE is a bathymetric data management system that provides these kinds of tools along with an effective retrieval and distribution system. The storage and management of datasets in a single repository facilitates the distribution of the entire bathymetric resources in an area to the most clients within and external to an organization. Users can connect to a single source and determine what resources they want to access in order to best meet their needs. Ideally, data is stored at optimal resolution; not limited to set levels of resolution. The maintenance of the data at its optimal resolution allows the data to be exploited for the greatest number of applications, including applications such as engineering work, coastal studies and navigation. A bathymetric data management system can provide the underlying infrastructure needed to make the delivery of products more flexible and efficient. Like other hydrographic organizations, Canadian Hydrographic Service (CHS) is facing increasing size and coverage of MBES, increased demand for and distribution of products and a desire to make use of new and historical data in the creation of products. The CHS has a mandate to manage hydrographic data and to make it accessible to its clients. In an effort to continue to meet this goal in light of new challenges the CHS faces, database driven CARIS 1

2 technology has been implemented for the management of hydrographic features with the Hydrographic Production Database (HPD) and Bathy DataBASE is being considered for the management of the source bathymetry from which features are derived. Ping to Chart Processing The need for tools that that streamline the flow of hydrographic information from acquisition through to the creation of products has been brought forward by the hydrographic community over the last several years. CARIS has undertaken a major initiative to address this need by developing software solutions that work together to improve the Ping to Chart production process, quality and timeline (Peyton, 2005). The key software components of the CARIS solution for a complete Ping to Chart workflow are HIPS and SIPS, Bathy DataBASE, Notebook and HPD. Bathy DataBASE is the solution for addressing the need to store and manage surface density bathymetry. In terms of this workflow, Bathy DataBASE sits between the data processing and validation system, i.e. HIPS and SIPS, and the hydrographic data management and production system, i.e. HPD. See Figure 1. Figure 1 - Bathy DataBASE in the CARIS Software Suite The data is maintained in a client-server environment where it can be easily accessed for multiple products by multiple clients over time. The focus of this investigation is the bathymetry data management portion of the solution, CARIS Bathy DataBASE. In the development of a bathymetric data management system, CARIS has focused on the management and storage of bathymetric surfaces that represent the data in a gridded format at the resolution that best reflects the spatial accuracy of the sensor used to acquire the data. The system was also designed to support the storage of data in non-gridded format. This storage 2

3 format would be appropriate for the storage of sparse and/or randomly spaced collections of soundings, for example single point, trackline data, or digitized chart soundings. This allows for the seamless integration of over-sampled data from modern multibeam and LiDAR systems with the under-sampled sources often found in historical archives. The CARIS Bathy DataBASE Solution CARIS has undertaken the task of providing a series of production tools that cover the flow of hydrographic information from acquisition to production. Bathy DataBASE plays a key role in this process. The objective of the system is to provide a toolset that facilitates the analysis, compilation and management of different sources of bathymetric data. The Bathy DataBASE software suite is composed of several application modules, as seen in Figure 2. The available modules include: Figure 2 - CARIS Bathy DataBASE modules BASE Editor - BASE Editor is a standalone desktop application used to validate 3 rd party data and perform surface operations. Note that the application does not interact with the DataBASE Server. BASE Manager - BASE Manager is the primary client to the DataBASE Server. The BASE Manager application has all of the functions found in BASE Editor and provides all the tools needed to load, query and access data on the Server. Bathy DataBASE Server - The DataBASE Server allows for the attributed objects in the database to be stored and managed. The BASE Manager application provides the interface to the Server but many of the processing functions are executed on the Server itself. Administration Tools - The Administration Tools provide control over the DataBASE Server. The tools allow for database creation, configuration and log and user role management. 3

4 In the future, the Bathy DataBASE Server could be accessed by other client applications such as HPD or an external party via an API. This will allow the resources in the database to be used directly by other applications, potentially further streamlining workflow processes. The Server can be accessed by multiple clients simultaneously. Workflow An intrinsic part of the Bathy DataBASE system is the flexible toolset provided to evaluate and edit datasets from third party sources, preparing the data to meet the standards of an organization so it can be used to enhance the organization s data holdings and coverage. All valid data can potentially be used in the creation of products such as nautical charts. A typical workflow, as seen in Figure 3, would have data coming into the system from various sources with some data preparation and validation taking place using the tools in BASE Manager. Validated data and metadata would be loaded into the Bathy DataBASE Server. Products such as Soundings, Contours and Depth Areas would be generated from the source data. Figure 3 Typical workflow in Bathy DataBASE Each of these basic areas in the workflow is described in greater detail below. Import Sources In order to meet the needs of the hydrographic organization, the system needs to support the input of data from a wide a variety of sources as possible. This would include the ingesting of data surveyed to the high standards required for navigation (i.e. IHO Order 1) and data collected for other purposes (ship trackline data or scientific studies) that may not meet any significant standards at all. Some hydrographic organizations receive large numbers of nonstandard 3 rd party source data that has to be evaluated carefully (Howlett, 2003). The import of many widely used file formats is supported (XYZ, GSF, HTF, etc ). Sources can be viewed and validated in context with background raster imagery and vector files. Storage Solutions Bathy DataBASE supports the management of bathymetric surfaces derived from cleaned and processed full density surveys. Surface density data is managed in formats such as the CARIS format known as the BASE Surface and the open-source BAG format. Source data that does not lend itself to gridding, such as data from single beam echo sounders, ship 4

5 tracklines, and leadline soundings, can be maintained as a point data in the Sounding Set format. BASE Surface - The BASE Surface (Bathymetry Associated with Statistical Error) (Gourley et al, 2003) is an implementation of the Navigation Surface concept (Smith, 2003). The goal of the BASE Surface is to provide statistically valid representation of the seafloor. Along with depth values, an uncertainty estimate and other statistical attributes can also be calculated for each node in the gridded surface. The Uncertainty is modeled based on the total influence of error estimates provided from individual sensors used for acquisition, or the Total Propagated Error. The other statistical attributes such as Shoalest, Deepest, Density and Standard Deviation are calculated from all soundings used in the population of a grid node. In the case where additional attribute information is available in the source data, they can be maintained through the import process and can be incorporated into the BASE Surface as attribute layers. The attributes provide information to the cartographer or hydrographer important for safety of navigation and for the use of data for other applications. A BASE Surface can be generated from new and old data sources. Using the HIPS software, unprocessed data from all sensors is imported, cleaned and gridded. In cases where the data has already been processed, the data can be imported directly into Bathy DataBASE. In either case, the data is gridded at an optimal resolution. This can be defined as a resolution that best represents the spatial resolution of the sensors used in acquisition and a resolution that will fully support the creation of any vector or raster hydrographic product. BAG - The Bathymetry Attributed Grid or BAG format is the result of an open-source software initiative by the Open Navigation Surface Working Group (Calder et al, 2005). The goal of the project is to provide a freely available, source-code library for the encapsulation of gridded bathymetric surfaces. The BAG format supports both depth and uncertainty attributes. In addition to these basic attributes, metadata such as authorizing agency, author, etc. can be populated on the file header. For more information on the Open Navigation Surface Working Group and the BAG format, visit Sounding Set - Where it is not desirable to maintain data in a gridded format, data can be stored in the Sounding Set format. Each line in an imported source data file (XYZ, GSF, HTF, etc.) is stored in a point store. Attribution such as uncertainty, source, date, etc. can be maintained at the point level. Meta data such as coordinate system and reference datum is also maintained at the dataset level. Sounding Sets can viewed as points or as a Triangular Irregular Network (TIN). A gridded BASE Surface can be interpolated at a user defined resolution if desired. Tools are available to edit individual or groups of points. Analysis, Editing and Validation Tools A variety of editing, analysis and validation tools are provided in BASE Manager to help the user prepare the data for integration with other sources and future use in the creation of products. One of the benefits of using a system with integrated editing tools, is that less stringently collected survey data can be prepared and incorporated with sources that meet charting standards to get the maximum coverage. Uncertainty is maintained where possible so that future users can evaluate whether they want to use data in an area from a source depending on its uncertainty not just its availability. All supported storage formats can be 5

6 viewed simultaneously and integrated together into a single surface that represents the optimal data from all sources over the maximum coverage area Included in the application are tools for: Differencing to compare overlapping sources Shifting to apply a single or model vertical shift Extracting to create a subset of data Combining to de-conflict overlapping sources Area based cleaning with the Subset Editor TIN editing Interpolating, including Uncertainty model estimation Viewing of source data with raster and vector backdrop in 2D and 3D The tools provided allow a great deal of flexibility in the format and nature of data that can be brought into, stored and used by the system. Figure 4, is a BASE Surface draped with an ENC displayed in 3D. Figure 4 - Vector map draped over BASE Surface The 3D graphics tools allows the user to validate surfaces by providing fly-through navigation and recording capabilities. Datasets can be distributed as movies recorded in the application (wmv or avi). Surfaces can also be easily distributed and viewed in 2D or 3D using CARIS Easy View, a free viewer that displays BASE Surfaces, BAG files, ENCs, and many other vector and raster formats. Easy ENC is available for download on the CARIS corporate website 6

7 Generate Products Bathymetric features are generated from the data held in Bathy DataBASE and made available to HPD for analysis and possible inclusion or updating of existing chart products. The user can take advantage of two powerful pieces of functionality to make the creation of bathymetric products (contours, depth areas and soundings) more efficient: surface combining and surface generalization. Bathy DataBASE provides a system for de-confliction on multiple sources of data based on supersede rules. The function can support the combination of sources in all supported formats (BASE Surface, Sounding Set and BAG). The result surface offers maximum coverage in the area of interest and the populated by the best data available from all sources. The best data is defined by a hierarchy of user defined supersede rules. This function gives the user the flexibility to create contours, depth areas and soundings from an individual survey or from a de-conflicted surface of all suitable sources in the database. Prior to the generation of bathymetric products, the user can apply generalization to a surface (single source or multiple de-conflicted sources). The generalization is applied to a surface to smooth it to the degree that cartographically correct contours can be generated appropriate for a scaled product. The generalization of a surface is a two step process of 3D double buffering and horizontal de-focusing. The user defines the amount of generalization applied. The result is a generalized surface from which contours that require no further editing can be generated. If the surface is not smoothed prior to the generation of contours, the contour lines then have to be altered to make them suitable for use on a chart (for example, edited or smoothed using Douglas Peuker filter). If the generalized surface is also used in the sounding selection then the chance of soundings falling on the wrong side of contours is eliminated. This situation often occurs when contours are smoothed due to the change in position of the contour lines. Data sources held in BDB can be exploited as needed by the organization or by the requirements of a particular product. Database Architecture The CARIS Bathy DataBASE system is based on a client-server design. The system supports the storage and management of bathymetric data in the BASE Surfaces, BAG files and Sounding Sets formats. Data is loaded into the Bathy DataBASE Server as survey and surface objects. Each survey and surface object has associated meta data based on S-57 attribution, see Figure 5. The catalogues defining the objects and their attributes are in xml format and are customizable by the user. 7

8 Figure 4 - Surface object with associated attribution and bounding polygon The object meta-data and system generated bounding polygon coordinates are stored in the database. The points/nodes that populate the dataset are stored in a file/folder system. All interaction between the client application and the objects in the database are managed by the Server software. The Server software handles transactions requested by the client application (primary client being BASE Manager) for data loading, data extraction, object editing, etc. Server resources are also used in the creation of generalized and de-conflicted surfaces and the generation of bathymetric products (i.e. contours) off of sources in the database. Also stored in the database are the objects resulting from the de-confliction process. Both the resulting surface and the supersede decisions defined for the de-confliction are maintained; the surface is stored as a surface object and the supersede decisions are stored in a product profile object. The product profile object carries information about the area which has been superseded and the parameters of the supersede decision. The CARIS Bathy DataBASEsolution was designed to be scalable in terms of accommodating the types and volume of data currently being surveyed with multibeam and LiDAR systems. The scalability also addresses the need for coverage on a global scale as needed by some organizations that have a mandate beyond sovereign waters. Distribution The logical extension of any data management system is the distribution of those resources to clients. Due to the massive improvements in communication systems over the past several years, delivery over the World Wide Web has become an efficient mechanism for disseminating geo-spatial information. The CARIS solution for the distribution spatial data on the web is Spatial Fusion. This technology provides an interface to spatial data, including hydrographic information and 8

9 bathymetric data. Through the web interface, as seen in Figure 5, the user would have the ability to view, query, and select data sources. Figure 5 Access to hydrographic data through the Web Spatial Fusion implements the Open Geospatial Consortium Inc (OGC) specifications for Web Map Service (WMS) (CARIS, 2006). This allows data to be easily shared among organizations and allows connections to external web sites for viewing additional sources of data. Pilot Project - CHS Quebec Region As part of a larger initiative to adopt database driven technology in its processing and management of bathymetry and hydrographic data, CARIS Bathy DataBASE is being evaluated by the CHS Quebec region. The CHS is already in the process of implementing HPD for the chart production component of this initiative. The CHS Quebec region is currently in the early stages of evaluating Bathy DataBASE to improve the management of the bathymetry results from various types of surveys. Sources of bathymetry include new and historical single-beam, MBES, leadline, and other. The CHS has a mandate to manage all bathymetry sources for Canada and to provide this data to its clients. The goal of investigating new ways of managing the data is to respond, in the most efficient way, to all requests from its clients. Using a bathymetry data management system like Bathy DataBASE, allows bathymetry extracts to be tailored to meet a specific set of criteria. The criteria may include specifications for epoch, resolution, attribution, etc. The data is provided to a wide variety of clients for applications in navigation, science and engineering. 9

10 File Based Workflow The current practice in place for the management of bathymetric data is the use of NTX files (CARIS exchange format). All data are converted to a common datum and cartographic projection to make the future analysis easier. In the case of multibeam data, there is an extra step in the process. The original file is decimated to a sounding density that makes the files usable without losing too much detail. After the conversion to a common format, analysis is done by comparison to files that cover the same area. A decision is made to modify the status (suppressed or not-suppressed) of data if a change is required. Once this analysis is complete, the new file is accessible to other users. All the steps, as described above, need to be recorded manually in the CHS metadata database (CHSDir) for tracking. To access the data, the user needs to use CHSDir to make a query on the attribute that they are looking for (type of survey, geographic query, etc). Even if only a small area of a large file is needed, the entire file needs to be imported. The user needs to manipulate the files to access the area of interest. Figure 6 is a simplified representation of the file based workflow for the bathymetric data management. CHS Data External data provider conversion to a common format Figure 6 CHS1: Actual Workflow Interaction with CHSDir makes the management complex and restrictive in term of access and distribution to all clients. The file management is restricted in the interaction with the database providing only one way communication. Bathy DataBASE Workflow The goal of this project is to explore the possibilities offered in the new technology developed by CARIS to complete the Ping to Chart workflow in a production environment. CHS would like to improve its bathymetric data management by: developing interaction with its internal databases (two ways communication) and improving accessibility to the community. See Figure 7. 10

11 CHS Data External data provide Web Access Figure 7 Bathy DataBASE and HPD Workflow A bathymetric data management must have some basic qualities in order to fit into the CHS workflow process. The following is a list of criteria considered important for CHS Quebec. Interaction with existing databases (CHSDir, CARIS HPD); Seamless integration with the technologies used in adjacent processes (upstream (HIPS), down stream (HPD Suite)); Datum adjustment (management); Minimal loss of information (as close as possible to the source); Traceability of data from source to product; Easy access to the data for internal and external users (Web access); Performance and robustness of the application; Evaluation of the export format by the clients. Implementation of new technology often requires an organization to redefine its methodology. This list was defined to make sure that even if the methodology is modified, the actual and future needs will still be covered. Investigation of Bathy DataBASE The effective management of bathymetric data is crucial to the CHS. The development of new tools and data management solutions allows the CHS to explore new opportunities to upgrade or adjust workflow practices. 11

12 Two philosophies are possible for the management of data. Load data as a navigation surface (BASE Surface) or a point stored file (Sounding Set). We tested both possibilities to evaluate which of those two possibilities will best address our needs. The area coverage selected for the tests was limited to Quebec harbor which contains around 50 source documents, from leadline to multibeam data. Before loading both test databases, all the files were converted to a common datum and coordinate system. No datum adjustment or validation of the data was needed because those historical files have already been normalized. In the case of new sources, those steps would be necessary. Historical sparse data were converted to Soundings Set and BASE Surface format. Data with enough sounding density were converted directly to BASE Surfaces. After the conversion, a survey was created for each of the files and metadata were filled manually using information contained in CHSDir. The data was loaded into the database attached to the survey previously defined. Once the data was completely loaded into both databases, we begin to test the superseding rules at the extraction. The access to the data is very efficient and it is easy to use. But some of the initial tests didn t give us the results were expecting. We were expecting that the conflicts would be solved at a dataset level instead of at the node level. We need to look back at the results and determine if the old methods are still valid or necessary. Discussion with other CHS regions or even other hydrographic organizations will be necessary to know how to proceed. New methods usually come with new technology implementation. A meticulous analysis of needs will be necessary before asking for any modifications (if needed) to the application. Some additional work remains to complete the evaluation the criteria previously defined: Datum adjustment (transformation and management); Interaction with databases in CHS (CHSDir and CARIS HPD); Performance robustness of the application; Web Access to the data. Future developments The project is on-going until the end of September We will need to cover all the criteria to make sure that the implementation of this technology at CHS Quebec will improve the management of bathymetric data and complete the Ping to Chart workflow. The conclusions of the project will be presented at another hydrographic conference or journal article. References Calder, B., LCDR Richard T. Brennan, Barry Gallagher, Shannon Byrne, James D. Case, David Fabre, R. Wade Ladner, Bill Lamey, Friedhelm Moggert, Mark Paton, and The Open Navigation Surface Working Group. The Open Navigation Surface Project, U.S Hydro, San Diego, California, March CARIS Spatial Fusion Enterprise, Brochure, Gourley, Mike, Shawn Mitchel and Hugh Astle, "Management of Bathymetric 12

13 Data using BASE Surfaces", Proc. 3rd International Conference On High- Resolution Surveys in Shallow Water, Sydney, Australia, November Howlett, C., New Bathymetric Data Assessment at the UK Hydrographic Office, Hydro International Vol. 7 No. 4 May Peyton, D., A Database Driven Hydrographic Information System, International ENC Seminar and EAHC Extraordinary Conference Palace Hotel, Seoul, Korea, February 21-24, Smith, LT S.M., "The Navigation Surface: A Multipurpose Bathymetric Database", MSc. Thesis, Center for Coastal and Ocean Mapping & Joint Hydrographic Center, University of New Hampshire, Durham NH 03824, Author contact information: Karen Cove, CARIS Fredericton, New Brunswick, Canada karen.cove@caris.com Frederic Lavoie, Canadian Hydrographic Service Mont Jolie, Quebec, Canada LAVOIEF@dfo-mpo.gc.ca 13