SDI Tecnological Components and Standards

Transcription

1 SDI Tecnological Components and Standards Salvador Bayarri World Bank Consultant

2 Contents The SDI architecture model Software components: the SDI stack Basic services Metadata SDI access control Current trends in SDI technology

3 SDI Architecture Model SDI uses a three-tier service-oriented model Standardized Source: mapit.biz

4 SDI Architecture Model Same thing with a component-oriented view: Access to transformed data, pictures, maps, reports, multi-media content Metadata search and retrieval for data and services SIG Service chaining: search, display, access, e-commerce,. Clients Middleware Catalogs Metadata update Geo-processing and catalog Services Direct data access Content Repositories Other data e.g., administrative, statistical, env. reporting Coverages Features Servers

5 SDI Architecture Model The use of a communication layer based on standardized Web services ensures: Transparent access to data, regardless of format, technology but we still need to harmonize the data content Interoperability between different brands of servers and clients Distributability of components in a network. The SDI philosophy is that each responsible organization maintains and publishes its datasets. Multiple clients, catalogs, etc. are possible

6 Software components: SDI stack

7 SDI stack: Spatial Databases Spatial DB = Relational DB + Geometry representation. Standard is SFS (Simple Feature Specification) Spatial Index. Speeds up search and operators, sorting objects by location Spatial operators. Can be invoked in SQL sentences to measure, perform geometric operations (e.g. intersection, reprojection), create new objects, etc. Essential for data quality assurance. Needed for large datasets, many simultaneous accesses (e.g. concurrent editing) Relational databases also used for metadata Files still used for raster data (images, elevation grids)

8 SDI stack: Spatial Databases Example of spatial table Encoded geometry field

9 SDI stack: Spatial Databases Simple Feature Specification (OGC) for SQL

10 SDI stack: Spatial Databases Well-Known Binary representation for simple features (OGC)

11 SDI stack: Spatial Databases Open-source (OS) implementations SpatiaLite (extension to sqlite), supported by qgis MySQL spatial extensions (not fully standard) PostGIS (extension to PostgreSQL) Excellent support for standards Widely supported in OS GIS, ArcGIS, Intergraph Used by GeoNode Friendly administration tools (pgadmin) Interesting extensions (pgroute for routing) The whole UK Ordnance Survey is stored in PostGIS ( 1,000,000 savings compared to Oracle Spatial), as well as the national topographic data of France for the Institut Géographique National

12 SDI stack: Spatial Databases Proprietary implementations Oracle Spatial MS SQL Server 2008 and later ArcSDE (now ArcGIS Server Basic) Wraps spatial or non-spatial DBs for use in ESRI software, getting good performance Some support in OS GIS, but In general, not useful if you need to use non-esri software with the DB

13 SDI stack: GeoServers A variety of OS and proprietary technologies support OGC standards for geoservices

14 SDI stack: GeoServers MapServer (OS). Excellent performance, quality and symbology for Map services (image) GeoServer (OS). Excellent performance and support for Feature services (data), great integration with PostGIS, even for remote editing. Easy administration. Used in GeoNode

15 SDI stack: Catalog Servers Geonetwork (OS) Reference OGC implementation for catalog service Includes powerful Web metadata editing and administration Extensively used in catalog networks (UN, many SDIs) Used in GeoNode

16 SDI stack: Catalog Servers ESRI s Geoportal Server Free and Open source, but Meant to be used with ArcGIS Online and ArcGIS for Server

17 SDI stack: Web clients Intergraph s SDI Portal Geospatial Portal Bundles OpenLayers with GeoExt UI to provide ready-touse components

18 SDI stack: Web clients Usually combine: A live-map component (map viewer) which can: Display data from standard Web geoservices Navigate, query data Sometimes provide data editing Web user-interface UI technology for legend, search, symbology, tools, etc. Today these are based on AJAX, Flex, Silverlight or other highly interactive framework for Web apps

19 SDI stack: Web clients OpenLayers (OS) Most used map viewer Supports Google Maps and other map providers Excellent performance, using tiled images Supports feature data, editing with WFS-T Used in GeoNode

20 GeoExt (OS) SDI stack: Web clients Based on JavaScript toolkit ExtJS, applied to Web geoportals Highly interactive table of contents, menus, tools, dialogs, etc. Integrates OpenLayers. It easily extended and customized Used in GeoNode

21 SDI stack: Web clients ESRI Web Mapping API Bundles ESRI s map components with JavaScript, Flex or Silverlight UI. Includes Web editing functionality.

22 SDI stack: desktop clients GIS (Geographical Information Systems) Heavy-duty data analysis and processing are still run in desktop applications All of them support OGC standards, so they can use Web geoservices and catalogs as source for data (but compliance and compatibility is not always good) The best practice is not to download data locally, but work with remote data that is official and up-to-date The SDI model, together with GIS, can also be used to implement information systems within organizations, sharing common data and metadata

23 SDI stack: desktop clients qgis (FOSS) We ll use this one in the workshop

24 SDI stack: desktop clients udig (FOSS)

25 SDI stack: desktop clients ArcGIS

26 SDI stack: desktop clients GeoMedia

27 Basic SDI Services Web Map Service (WMS) Get maps as images Get attribute information about features in a specific map location Web Feature Service (WFS) Get vector data with geometry and attributes based on filters. Not meant for large data downloads! Remote feature editing (WFS-T) Web Coverage Service (WCS) Similar to WFS, for raster data (multiband images, elevation grids - DEMs) Catalog Service for Web (CSW) Search and read metadata Harvest metadata to other catalogs Remotely edit medatada

28 Basic SDI Services Web Map Service (WMS) requests: GetCapabilities: returns the description of the service (layers, styles, coordinate systems ) as XML text GetMap: returns an image (jpeg, png) of a map composed of one or more layers, in the requested coordinate system. The layers and style (legend) are defined in the server but can be selected and changed (the legend) in the request IRE&WIDTH=640&HEIGHT=320&FORMAT=image/png&SRS=EPSG:4326&BBOX=-180.0,-90.0,180.0, OTEGIDAS%2CETIQUETAS&TRANSPARENT=true&VERSION=1.1.1&FORMAT=image%2Fpng&SERVICE=WMS &REQUEST=GetMap&STYLES=&SRS=EPSG%3A97057&BBOX= , , , &WIDTH=256&HEIGHT=256

29 Basic SDI Services Web Map Service (WMS) requests: GetFeatureInfo request: returns the attributes of one or more objects of a specified layer at a certain location in the map. Meant as a click-on query on a map viewer mania_water_bodies&styles=&service=wms&version=1.1.1&request=getfeatureinfo&bbox= , , , &FEATURE_COUNT=10&HEIGHT=256&WIDTH=512&FORMAT=image/gif&INFO_FORMAT=text/html& SRS=EPSG:4326&X=258&Y=188 The output format can be changed (plain text, XML ) to process it in our client application.

30 Basic SDI Services WMS GetFeatureInfo Exercise Try it live at Use your Web Browser to spy on network requests and responses Chrome: Tools /Developer Tools / Network tab Firefox: Tools / Web Developer / Web Console

31 Basic SDI Services Web Feature Service (WFS) requests: GetCapabilities: returns the description of the service (layers, filter types, coordinate systems ) as XML text DescribeFeatureType: returns a description of the feature for a specified layer etype&typename=usa:states

32 Basic SDI Services Web Feature Service (WFS) requests: GetFeature: returns the features that comply with a certain condition (filter), based on location or attribute values (it is not meant for massive data dowload!) ename=usa:states&maxfeatures=10&filter= <ogc:filter xmlns:ogc=" <ogc:bbox> <ogc:propertyname>the_geom</ogc:propertyname> <gml:envelope xmlns:gml=" srsname="epsg:4326"> <gml:lowercorner> </gml:lowerCorner> <gml:uppercorner> </gml:upperCorner> </gml:envelope> </ogc:bbox> </ogc:filter>

33 Basic SDI Services GetFeature response in GML format <?xml version="1.0" encoding="utf-8"?> <wfs:featurecollection numberoffeatures="1"> <gml:featuremembers> <usa:states gml:id="states.20"> <usa:the_geom> <gml:multisurface srsdimension="2"> <gml:surfacemember> <gml:polygon> <gml:exterior> <gml:linearring> <gml:poslist> </gml:poslist> </gml:linearring> </gml:exterior> </gml:polygon> </gml:surfacemember> </gml:multisurface> </usa:the_geom> <usa:state_name>south Carolina</usa:STATE_NAME> <usa:persons> </usa:persons> </usa:states> </gml:featuremembers> </wfs:featurecollection> Test this live at

34 Basic SDI Services Web Feature Service (WFS) requests for remote editing (WFS-T): LockFeature: prevents other clients from editing selected features while we work at them Transaction: creates, modifies or deletes features. Each transaction consists Insert, Update, and Delete elements, performed in order Not all servers or clients implement these requests!

35 Basic SDI Services Web Coverage Service (WCS): Similar to WMS but returns raster data with multiple band information, elevation values, etc. Catalog Service for the Web (CSW): Provides remote access to a DB of metadata records (catalog) DescribeRecord - returns info about the format of metadata records GetRecords - search for records, returning record IDs GetRecordsById - returns records, specified by their ID Harvest (optional) - create/update metadata by asking the server to 'pull' metadata from somewhere (other CSW server, a Web folder ) Transaction (optional) - create/edit metadata by 'pushing' the metadata to the server

36 Basic SDI Services Catalog Service for the Web (CSW): GetCapabilities and DescribeRecord examples: GetRecords example: RSION=2.0.2&OUTPUTSCHEMA= GUAGE=CQL_TEXT&RESULTTYPE=results&TYPENAMES=csw:Record A constraint or filter is usually added to get only the metadata that we are searching (for instance, containing certain words, or for a certain geographic area)

37 Basic SDI Services The CSW service, through the Harvest request, allows the federation and integration of metadata catalogs, automatically collecting metadata from some catalogs to another (for instance, to specific organizations to a national catalog) Metadata editor Web or desktop client Local/entity Catalog Server Harvest National Catalog Server Metadata DB Metadata DB

38 Basic SDI Services How they all work together (Animation) Geoportal, GIS client Search Viewer File Web Metadata Metadata Service Add WFS WMS layer File Main Catalog Metadata CSW request Query WFS result WMS WFS (data) request CSW Result Servidor entidad WMS result Data visualization (image) Data download Entity metadata Entity metadata Metadata Entity data Geodata Archivo

39 Other Basic SDI Services Direct file download Use it for geodata download instead of WFS or WCS, which should be only for limited query Use it for documents and other files that cannot be accessed through standard Web services The download address can be written in the metadata, as we do for Web services News and alerts (RSS, GeoRSS, Atom ) Conventional Web clients can get updated messages and news Users can subscribe to receive them in a variety of readers

40 What is "Metadata"? Metadata is a formal data documentation and it is critical to data discovery A metadata record is a set of information fields, usually presented as an XML document, which captures the basic characteristics of data as an information resource It represents the who, what, when, where, why and how of the resource

41 What is "Metadata"? Metadata as seen in Geonetwork

42 What is "Metadata"? Metadata as seen in ESRI s ArcCatalog

43 Biodiversity & Environmental Resource Data System of Belize

44 Main uses of Metadata Organize and maintain an organization's internal investment in spatial data Provide information about an organization's data holdings to data catalogs, clearinghouses, and portals (it is the entry point of an SDI and for each SDI node) Provide information to process and interpret data received through a transfer from an external source

45 Value of Metadata Facilitates data maintenance Facilitate data discovery Enables reuse of data Informs potential users of inappropriate uses of a dataset

46 Challenges of using Metadata Exchanging records -> Use standards! Creating metadata takes time and patience -> Use templates and tools that automatically extract or set metadata (like GeoNode). Imperfect metadata is better than none Maintaining metadata is essential, especially for Web services and downloads (beware of URL changes!) -> set up processes or tools for automatic checkups

47 Metadata standards Some standards define the metadata content (fields, values, mandatory v. optional) The most comprehensive is ISO From this standards, custom profiles can be defined for specific countries, themes, etc. Old metadata standards like FGDC or Dublin Core are now profiles of ISO Other standards define appropriate exchange formats The most accepted is ISO 19139, which defines an XMLbased format

48 Metadata content: Identification Important for data search

49 Metadata content: Identification Information in abstract: data model Entity types and their attributes and the domains from which attribute values may be assigned The names and definitions of features, attributes, and attribute values

50 Metadata content: Identification

51 Metadata content: Distribution information How to access the information? Multiple transfer options are possible Contact for physical media Fees

52 Metadata content: Data quality Positional and attribute accuracy Completeness and consistency, Sources of information Methods used to produce the data Appropriate uses (scales)

53 Metadata Content: Spatial Reference

54 Metadata content: Metadata information Language, date, update status Who is responsible for the metadata?

55 Metadata content: Access and use constraints Use constraint statements can be crafted to express scale, geographic, or temporal limitations to the data Liability statements should be written by legal staff to ensure that the legal requirements for use of the data are fully outlined In general, it is far better to publish your dataset with limitations within your metadata than to later attempt to generate them in response to an inquiry or lawsuit.

56 Access and use control Use constraints and disclaimers Web clients typically include liability limitations (disclaimers) and licenses that users must accept But geoservices can be accessed directly! Therefore, use constraints must be specified in the metadata and GetCapabilities response And services must be protected from non-authorized access Within an organization: generic IP filtering and firewalls External services: need an authentication mechanism

57 Access control for geoservices There are generic mechanisms for Service-Oriented Architectures Based on XML standards like SAML and XACML However, these standards are not easily used in SDIs because Current geoservices do not use SOAP for passing information Server and client software should be modified to comply

58 Access control for geoservices Specific SDI solutions have been proposed and used Existing clients and server don t have to be modified, only intercepting components need to be added Access can be controlled for specific layers, scales, features and attributes See

59 Access control for geoservices In practice, however, most systems use simple HTTP authentication that provides a simple on/off swith for services depending on the user credentials Web browsers have a built-in implementation to ask for user and password

60 Current trends in SDI technology Gazetteer services (search location by name), based on WFS-G profile

61 Current trends in SDI technology Web Processing Service (WPS). Standard for discovering and calling remote processing modules Used for typical GIS analysis like spatial analysis, running distributed models Used for operations that are not easy to standardize in a specific Web service Spatial Coordinates transformation Geocoding (address finding) See, for instance: &request=getcapabilities

62 Current trends in SDI technology Tiled maps Cached and tiled Map Services (WMTS, WMS-C )

63 Current trends in SDI technology Tiled maps Static maps can be pre-calculated and tiled at different scales, like a raster pyramid Tiles can be directly server through tile services like WMTS, WMS-C, Google Earth Pre-calculated tiles can be used to quickly respond to conventional WMS requests An example is the GeoWebCache technology, used by GeoNode, which can be integrated with regular WMS servers like GeoServer

64 Current trends in SDI technology Tiled maps

65 Current trends in SDI technology Tiled maps A problem with tiled map providers like Google, Bind, OpenStreetMap, is that they only provide tiles with a specific ad-hoc coordinate system (used to be EPSG:900913, now it is officially EPSG:3857) It is difficult to overlay this data layer on other coordinate systems

66 Current trends in SDI technology Remote on-line editing with WFS-T

67 Current trends in SDI technology Remote collaborative editing (OpenStreetMap)

68 Current trends in SDI technology Remote geoprocesses (coordinate transformation, geocoding) with WPS

69 Current trends in SDI technology Real-time data integration (RSS, Common Alert Protocol)

70 Current trends in SDI technology Standard Web services for reading and controling remote sensors (SWE, SOS )

71 Current trends in SDI technology Integration with SOLAP data mining

72 Current trends in SDI technology SDI Web services in the cloud

73 Current trends in SDI technology New data sources (LiDAR, georeferenced Video & street-level images)