MASSACHUSETTS INSTITUTE OF TECHNOLOGY Lexington, Massachusetts. Prepared for the Federal Aviation Administration Washington, DC 20591

Transcription

1 Project Report ATC-354 NextGen Network-Enabled Weather Metadata Guidelines for the 4-D Weather Data Cube Oliver J. Newell Gregory W. Rappa Brett Levasseur April 24, 2009 Lincoln Laboratory MASSACHUSETTS INSTITUTE OF TECHNOLOGY Lexington, Massachusetts Prepared for the Federal Aviation Administration Washington, DC This document is available to the public through the National Technical Information Service, Springfield Virginia

2 ABSTRACT Data from most systems can be divided into two categories, data and metadata, where metadata are loosely defined as data about data. Given a set of temperature observations, for example, the actual temperature measurements made during an observation can be considered to be data, while information about the time and/or location of the observation is often considered to be metadata. In the earth science community, the volume of data available from sensor and computer models is growing rapidly, raising the overall importance of metadata to allow for efficient access and interpretation. Metadata standards and associated best practices play a key role in the sharing of data on Internet scales. Though numerous metadata standards exist today, there is significant flexibility with respect to the choice of standards, and how the standards are used. For any given community-of-interest (COI), additional guidance is generally needed to ensure that the interoperability goals within that community are achieved. This document describes a set of metadata guidelines developed by the NextGen Network-Enabled Weather (NNEW) Program. The guidelines have been developed based on the results of the research and prototyping efforts conducted by the NNEW Program since early The document focuses on discovery metadata for weather datasets and weather data access services, as well as structural metadata related to weather dataset access. Guidelines for sensor description metadata are also provided, but in less detail. It is expected that this document will be expanded and revised over time to reflect not only the lessons learned by the NNEW program, but by the NextGen program as a whole. iii

3

4 TABLE OF CONTENTS Page ABSTRACT List of Illustrations List of Tables iii vii ix 1. INTRODUCTION 1 2. NEXTGEN NETWORK-ENABLED WEATHER PROGRAM BACKGROUND The 4-D Weather Data Cube NNEW Program Relationship To Other NextGen Programs NNEW Architecture 5 3. METADATA OVERVIEW Categories of Metadata Levels of Metadata DataSet Discovery And Access METADATA STANDARDS ISO / ISO / ISO Metadata Standards Background Web Services Description Language NetCDF Climate and Forecast Conventions Sensor Markup Language ebxml Registry Information Model Time Metadata Uniform Resource Name Namespace for Federated Content GUIDELINES FOR METADATA STANDARDS IN THE CUBE Mapping of Metadata Standards To Cube Capabilities ISO High Level Aggregate Dataset Metadata WSDL Service Metadata ISO Service Metadata ebxml Taxonomy Metadata 44 v

5 TABLE OF CONTENTS (Continued) Page 5.6 NetCDF-CF Dataset Structural Metadata Sensor Markup LANGUAGE SENSORDescription Metadata SUMMARY AND FUTURE WORK 67 APPENDIX A. ISO METADATA EXAMPLES 69 APPENDIX B. SENSOR METADATA EXAMPLES 83 APPENDIX C. EBXML TAXONOMY EXAMPLE 89 Glossary 92 References 95 vi

6 LIST OF ILLUSTRATIONS Figure No. Page 1 The NextGen 4-D Weather Data Cube. 3 2 NNEW relationship to other NextGen programs. 5 3 Sample Nodes in a hub and spoke architecture. 6 4 Example of product/dataset/dataset series partitioning for CIWS system Role of metadata in weather data discovery and access. Metadata are important to all three phases of the process Evolution of U.S. geospatial metadata standards [1] Block diagram illustrating the three key metadata documents related to dataset discovery and access Illustration of the separation of service interfaces from service implementations Time Expression diagram. 51 vii

7

8 LIST OF TABLES Table No. Page 1 NNEW Program Usage of the NetCDF-4 Climate and Forecast Convention Attributes 47 2 SensorML System Elements 58 ix

9 1. INTRODUCTION Data from most systems can be divided into two categories, data and metadata, where metadata are loosely defined as data about data. Given a set of temperature observations, for example, the actual temperature measurements made during an observation can be considered to be data, while information about the time and/or location of the observation is often considered to be metadata. The distinction between data and metadata is not always clear cut, and can depend on the viewpoint of a particular data consumer. In the above example, a scientist trying to pinpoint the location of a region with certainly daily temperature characteristics may consider the location of the measurement to be data rather than metadata. In general, one can say that for any given data/application combination, the data can be considered to be that which is of direct interest to the data consumer, while the metadata can be considered to be the information that allows the user to discover, access, and interpret that data. In the earth science community, the volume of data available from sensor and computer models is growing rapidly, raising the overall importance of metadata to allow for efficient access and interpretation. Metadata standards and associated best practices play a key role in the sharing of data on Internet scales. Though numerous metadata standards exist today, there is significant flexibility with respect to the choice of standards, and how the standards are used. For any given community of interest (COI), additional guidance is generally needed to ensure that the interoperability goals are achieved. This document describes a set of metadata guidelines developed by the NextGen Network-Enabled Weather (NNEW) Program. The guidelines have been developed based on the results of the research and prototyping efforts conducted by the NNEW Program since early This document currently focuses on discovery metadata for datasets and services, as well as structural metadata related to dataset access. Guidelines for sensor description metadata are also provided, but in less detail. It is expected that as the NNEW Program proceeds further, guidelines related to other areas, will be developed. In addition, the metadata standards being leveraged by the NNEW Program will themselves will be refined and extended, requiring that the guidance be revised over time. This document will be periodically updated throughout the life of the Program to capture these changes. 1

10

11 2. NEXTGEN NETWORK-ENABLED WEATHER PROGRAM BACKGROUND Metadata are used in a variety of ways. To put things in context, this section provides background on the NNEW Program and the role played by metadata in a number of key usage scenarios. 2.1 THE 4-D WEATHER DATA CUBE Weather data distribution in today s National Airspace System (NAS) is distributed across multiple systems; each using its own set of data formats and data distribution mechanisms. In the NextGen vision, these capabilities are integrated together using a common framework based on a relatively small set of standardized data formats and distributed data access services. This distributed weather database, depicted in Figure 1, is commonly referred to as the 4-D Weather Data Cube, or simply, the Cube. Figure 1. The NextGen 4-D Weather Data Cube. 3

12 Figure 1 illustrates a number of the key weather data providers and consumers at the conceptual level. Weather observations are provided to the Cube from a wide variety of weather sensors, including satellites, weather radars, aircraft, and surface sensors. Forecasting systems act first as data consumers, using observations to set the initial conditions for their models, and subsequently as data providers, making generated forecast data available to other consumers. Higher-level integration functions access weather data, producing information in a variety of forms for end users. All data providers provide not only the data, but enough accompanying metadata to discover and access the weather datasets. All data consumers are provided with the means to search data provider metadata to locate datasets of interest. This is accomplished via the use of a registry/repository that acts as a warehouse for the metadata and provides the necessary query capabilities. One key concept associated with the Cube is the notion of the Single Authoritative Source, or SAS. For any given weather data type, there may exist multiple data products for that type. For example, there may be two gridded temperature data products based on two different forecast models. An overarching goal for NextGen is support for shared situational awareness, which requires that there be a common operational picture available to all NextGen participants. Within the Cube, there exists a subset of data products that are categorized as the SAS product for their particular data type. Metadata will be used to provide much of the necessary SAS categorization information. 2.2 NNEW PROGRAM RELATIONSHIP TO OTHER NEXTGEN PROGRAMS At its core, the NNEW Program is an infrastructure program, providing common weather data capabilities to higher-level NextGen programs. The NNEW Program is utilizing lower-level infrastructure provided by the System-Wide Information Management (SWIM) and Federal Aviation Administration Telecommunications Infrastructure (FTI) programs to build the FAA portion of the Cube. The overall picture can be viewed as an infrastructure stack, with each layer providing a subset of the overall required functionality. This is illustrated in Figure 2. As shown in the figure, the NNEW Program provides data format standards, data access services, and registry extensions necessary for implementation of the 4-D Weather Data Cube. SWIM provides the cross-domain infrastructure such as messaging, monitoring, security, core registry functionality, and a common software infrastructure to ease system development and improve overall interoperability. The FTI layer supplies the fault-tolerant network communications backbone used by all SWIM-compliant applications. Note that the overall partitioning of responsibilities among the three programs is still evolving, and this description is based on the current understanding of program direction for the upcoming 3-5 year time frame. 4

13 CIWS ITWS WMSCR Weather Systems/Programs (Data Producers/ Consumers)..... NNEW Weather-Specific Data Formats & Access Services Weather-Specific Registry Extensions Deployed weather data distribution hubs Weather Domain SOA Infrastructure SWIM Messaging, Security, Monitoring, Core Registry FTI Basic IP Connectivity Fault-Tolerant Backbone Cross-Domain SOA Infrastructure Physical Network Layer Infrastructure Figure 2. NNEW relationship to other NextGen programs. 2.3 NNEW ARCHITECTURE Historically, the Federal Aviation Administration (FAA) and Department of Defense (DOD) have relied on weather data provided by the National Weather Service (NWS), but both agencies have also maintained significant weather data producing systems of their own. The trend in NextGen is that within the Contiguous United States (CONUS), the responsibility for generating operational weather products specific to aviation will shift away from the FAA to a set of NWS and/or commercial data providers. Though these external (w/respect to the FAA) providers will bear responsibility for generation of weather data products, providing the data to consumers in the NAS in a timely, cost-effective manner remains the responsibility of those organizations. The combined NNEW/SWIM/FTI infrastructure will provide this capability in the FAA domain. The idea of a scalable, efficient data distribution infrastructure is not new, and is often referred to as a Content Delivery Network (CDN). A variety of CDNs exist today (e.g., Akamai) and are in wide use at Internet scales. These existing solutions, however, tend to depend on caching of relatively static content, and are therefore not well-aligned with the more dynamic weather and/or surveillance data distribution requirements. An architecture with an increased focus on dynamic content is needed. The overall Cube architecture is still under active development, but the current thinking focuses on a hub and spoke architecture with distribution nodes that intelligently adapt to data access requirements and minimize overall bandwidth demands. This is especially important near the network edge, since costs to end consumers tend to be driven by the cost of the last mile. The notional high-level architecture is 5

14 depicted in Figure 3. Data are produced at a given location and made available by a data provider. The source of the data is termed the Origin Server. In this example, the Origin Server is a node in the NWS network. Other nodes are simply termed distribution nodes in the diagram, and can be of varying size and be positioned in the network core as well as closer to the network edge. Weather data nodes are capable of data fan-in and/or fan-out to make efficient use of the underlying physical network. As clients in the FAA network make requests for data, the Cube infrastructure dynamically adapts, requesting data from the NWS origin server and providing it to multiple consumers within the NAS based on the latency and reliability requirements of the requesting clients. Other Users FAA-Hosted Weather Data Distribution Node + Registry Weather Data Client (Regional Airport) NWS Weather Data Origin Server + Registry FTI WAN Weather Data Distribution Node Router Weather Data Distribution Node Weather Model Supercomputer(s) Weather Data Clients (TRACON/ARTCC) Weather Data Clients (TRACON/ARTCC) Figure 3. Sample Nodes in a hub and spoke architecture. The combined NNEW/SWIM/FTI infrastructure insulates individual data providers from the need to support a potentially very large number of clients at varying levels of service. With respect to data streams from multiple sensors (e.g., TDWR), the concept of data aggregation is also supported by the data distribution nodes. In general, the nodes can be viewed as very generic entities that support both data fan-in and fan-out. This idea builds upon the very successful Local Data Manager 6

15 (LDM) software infrastructure, a Unidata technology that is widely used in the scientific community today to disseminate weather data of all types. Additions to the LDM concept in the NNEW architecture include dynamic discovery of available data products and data access services, and on-demand filtering of data using spatial and temporal filtering attributes. The NNEW architecture is aligned with the principles behind Service-Oriented Architecture (SOA), and data formats and data access services are designed using a variety of composable building blocks. The result is a very flexible architecture that can be mapped to a variety of system topologies as the needs of the system evolve over time. A significant number of the NNEW data models and data access services are based on standards from the ISO and Open Geospatial Consortium (OGC) communities. Included in the list is the Geography Markup Language (GML), which is both an ISO and an OGC standard, as well as the OGC Web Feature Services (WFS) and Web Coverage Services (WCS). The ISO and OGC geospatial standards communities are closely related, and the prevalence of their use by NNEW drives the choice metadata standards towards those that are well supported by ISO and OGC. This will be discussed further in section 3. A key component in this architecture with respect to metadata is the registry/repository (henceforth simply referred to as the registry. The registry is the primary mechanism used to provide dataset and/or service discovery, leveraging a variety of metadata. The registry is based on the Organization for the Advancement of Structured Information Standards (OASIS) Electronic Business using extensible Markup Language (ebxml) Registry/Repository standard, version 4 of which is due out in late The ebxml registry is highly extensible, making it a good choice to implement the wide variety of data access queries that will be required as NextGen moves forward. 7

16

17 3. METADATA OVERVIEW 3.1 CATEGORIES OF METADATA Metadata comes in a variety of forms. Some of the more common categories of metadata are: 1. Structural Metadata - How data assets are physically composed (e.g., type of file: GIF, JPEG,...) and relationships between specific parts of the data asset 2. Discovery Metadata - Key attributes and concepts of a data asset used for discovery; this includes the means to enable a user to discriminate between individual elements of a data asset or across data assets 3. Data Access Metadata - Data needed to acquire a dataset (When the mechanism is a service, this can also be qualified as service metadata.) 4. Security Metadata - Information (e.g., security and privacy markings consistent with applicable standards) through which systems will be able to control access to assets based on classification metadata and enable typically inaccessible assets to be available to users and applications that have appropriate access needs 5. Data Lineage Metadata - Allow for identification of the author, publisher, and sources contributing to the data, allowing users and applications to assess the derivation of the data 6. Other Metadata - Vocabularies, taxonomic structures used for organizing data assets, interface specifications, mapping tables, etc. While examples of all these types of metadata are provided in this guidance document, the primary development work to date has focused on the first three categories listed above. The next section discusses in more detail the use of these types of metadata within the Cube. 3.2 LEVELS OF METADATA In addition to falling into different categories, metadata also exist at multiple levels. Using the analogy of a traditional file system, a set of files in a directory hierarchy may be owned by the same user, in which case the user metadata can be considered to exist at the top-level directory level. Individual files within the directory tree will often each have a unique time, in which case those metadata exist at the file level. The separation of metadata into a set of levels helps to avoid unnecessary duplication. In the geospatial realm, data and metadata are grouped together into datasets using a variety of physical and/or logical criteria. One common dataset grouping is based on the physical grouping of data within a disk file, where the disk file may contain data for a 10-minute time period. A higher-level logical dataset grouping might consist of the set of all the disk file-based datasets for a one-year period. If used in conjunction with finer-grained dataset terminology in the same application, these higher-level groupings 9

18 are referred to as an aggregate dataset, or to use the official ISO terminology, a dataset series. The partitioning of data and metadata into datasets and dataset series is dependent on the particular application, and can be done across a number of dimensions, including product type, time, and space. Key drivers of the partitioning choices include efficient access to the underlying data, as well as efficient representation of metadata that are largely static (to avoid redundancy). To make the concept of dataset vs. dataset series more concrete, consider the partitioning of Corridor Integrated Weather System (CIWS) data into datasets and dataset series as illustrated in Figure 4. A CIWS dataset is considered to be the combination of metadata and data for a single product type at a single CIWS output time. A Vertically Integrated Liquid (VIL) dataset, for example, consists of all of the gridded VIL product data at one of the 2.5 minute CIWS update intervals. Datasets are grouped into logical high-level dataset series along the time dimension. The VIL dataset series represents all VIL datasets available from the CIWS system, bounded only by the length of time the data product is considered to be the same in terms of the algorithm used to produce it and available inputs. If an experimental version of a VIL algorithm were running in parallel with an officially released version, they would be considered to be logically separate dataset series. Dataset Dataset Series CIWS Processing Cluster... VIL Echo Tops Lightning Dataset Storage Product Times VIL VIL Echo Tops Echo Tops Lightning Lightning Product Types Figure 4. Example of product/dataset/dataset series partitioning for CIWS system. Though a dataset can contain multiple weather products, the Cube convention is to prefer a fine-grained dataset that acts as a container for a single weather product. This simplifies data access for consumers. In Cube, a dataset series most commonly refers to a logical collection of datasets over a period of time (potentially on a multi-year timescale). 10

19 Metadata at the dataset series level provides the information needed for high-level discovery of a particular type and/or source of data, including: 1. Product Type 2. Time Extent 3. Geospatial Properties (extent, geographical map projection, etc.) 4. Data Quality 5. Data Lineage 6. Access Restrictions (security, commercial licensing, etc.) 7. Native Format of the Dataset 8. Data Classifications (e.g., Single Authoritative Source for a given product type) This information is of relatively low volume relative to the size of the entire dataset series, and is appropriate for storage in a shared, domain-wide registry/repository. Metadata at the individual dataset level is, by its nature, of much higher volume than the metadata at the dataset series level. In the case of CIWS, for example, storage of metadata at the dataset level, for a oneyear period, results in a storage requirement of approximately 2.1 million datasets (10 products x 576 outputs/day x 365days/year). Storing metadata at this level of granularity in a shared registry/repository is possible, but often not practical. It is essentially redundant since the metadata are already typically stored along with the data in data files and/or database tables maintained by a particular system. A commonly-used approach is to support the finer-grained metadata via the data access services provided by a particular system. These services are tightly-coupled to the underlying data/metadata, and can easily provide efficient access. The OGC WFS and WCS services provide this type of functionality. 3.3 DATASET DISCOVERY AND ACCESS Metadata are used most often to discover and gain access to weather datasets that reside in the 4-D Weather Data Cube. This usage is the current focus of this document. A diagram depicting the role of metadata in the discovery and access process is provided in Figure 5. 11

20 1 2 Provider publishes What, Where, When metadata for weather datasets and their associated data access services Registry/ Repository Consumer discovers weather datasets and associated data access services using What, Where, When metadata within discovery request Weather Data Provider Consumer contacts provider s data access service and access subsets of available realtime and/or archived data using What, When, Where metadata within request message 3 Weather Data Consumer Figure 5. Role of metadata in weather data discovery and access. Metadata are important to all three phases of the process. 12

21 4. METADATA STANDARDS The following table summarizes the key metadata-related standards discussed in this document. Though this list is expected to grow in subsequent releases of this document, those listed will likely remain key standards for the foreseeable future. Standard Name ISO ISO ISO Web Services Description Language (WSDL) NetCDF-4 CF ebxml Registry Information Model (ebrim) SensorML ISO 8601 RFC 4198 Description General metadata standards for datasets of all types. This is an abstract specification, describing the data model in Unified Modeling Language (UML). Extended Service metadata. This is an abstract specification, describing the data model in UML. Standard XML schema for the abstract ISO 19115/19119 metadata models Basic service metadata, including the abstract service interface as well as instance endpoint information. Earth science community data format with associated metadata. Generalized information model for ebxml registry/repository. Standard metadata for describing sensors and their capabilities. Standard for representation of time. Time is a common element used by many of the above standards. Uniform Resource Name (URN) namespace for federated content. 4.1 ISO / ISO / ISO METADATA STANDARDS BACKGROUND Beginning in 1994, the Federal Geographic Data Committee (FGDC) led the development of the Content Standard for Digital Geospatial Metadata, or CSDGM. This standard subsequently became the base for many other national standards, including the international ISO metadata standard. A relative timeline of the relationship between the FGDC work and ISO is shown in Figure 6. As can be seen from the diagram, the standard continues to be extended and refined, and this is expected to continue as experience with metadata continues to grow in the geospatial data community. 13

22 ISO profiles are created in recognition of the fact that the generic standard is very broad, and provides significant flexibility in terms of how it is actually used. This flexibility is desirable in some respects, but can also lead to interoperability issues if users within a particular community use the standard in significantly different ways. A profile essentially consists of a set of restrictions on how the standard is used, as well as a set of extensions specific to a particular domain. For example, some metadata elements considered optional may become mandatory in a profile (restriction), and additional data categories specific to North American usage may be defined (extension). Figure 6. Evolution of U.S. geospatial metadata standards [1]. Since the release of the first version in 2003, ISO is on track to become the official U.S. metadata standard for geospatial metadata. As part of this effort, a North American Profile (NAP) of ISO is being established. This is a collaboration between the United States and the Canadian government. On the U.S. side, the FGDC is in charge of the ISO NAP development. 14

23 The NAP of ISO is currently in draft status [2], and is expected to be finalized in the coming year. The guidance is for NextGen participants to conform to this profile when it becomes available. A significant portion of the draft profile is currently specified as best practices, rather than mandatory rules. This reflects the draft status, and the relative lack of experience within the geospatial community with specifying and using ISO metadata at a very detailed level. It is expected that the NAP will change significantly over the next three to five years as the community gains more experience. Though actually quite general in many respects, ISO is primarily concerned with metadata about geospatial datasets. A complementary specification, ISO 19119, defines metadata related to geospatial services. Using the combination of the two specifications, it is possible to locate datasets using a variety of search criteria, and subsequently discover information about what services can work with, or operate on, those datasets. The primary services of interest in the current NNEW realm are data access services, the services that allow data to be accessed and filtered based on spatial and/or temporal criteria. It is worth noting that ISO capabilities overlap with a number of the capabilities embodied in the WSDL. For the Cube, the two are being used in a complementary fashion. This is discussed in more detail in section Lastly, ISO and ISO are both abstract metadata standards, specifying the metadata and its associated relationships as a UML model rather than a concrete schema. In 2007, ISO became available, defining a concrete realization of ISO and ISO as an XML schema. ISO is being rapidly adopted in domains that are leveraging XML. One example of this adoption is version 3.2 of the Geography Markup Language (GML), which has integrated the ISO schema definition as a base-level building block. 4.2 WEB SERVICES DESCRIPTION LANGUAGE The WSDL describes services in terms of the content of messages passed back and forth to invoke service operations, as well as the message exchange patterns (MEPs) themselves. When compared with ISO 19119, WSDL metadata provides basic service information, while ISO supports richer metadata more specific to the geospatial domain. Two versions of WSDL are in current usage, 1.1 and 2.0. WSDL 2.0 addresses a number of shortcomings in WSDL 1.1, particular with respect to extensibility, and is the preferred choice for future Cube applications. It is expected that NNEW metadata guidance in the 2010 time frame will focus on WSDL 2.0. In 2009, however, WSDL 1.1 is an acceptable choice, as some of the SOA tools do not yet provide mature support for WSDL

24 4.3 NETCDF CLIMATE AND FORECAST CONVENTIONS Network Common Data Format, version 4 (NetCDF-4) is the preferred data format for gridded datasets in the Cube. NetCDF-4 provides support for structural metadata such as grid sizes and layout, but has relatively little guidance relative to more detailed metadata, such as metadata describing the coordinate reference system for the grid. The situation is analogous to the ISO standard in the sense that NetCDF-4 is very general and flexible, but lacking further guidance for a given COI, interoperability problems can result if individual members of the community come up with their own metadata variations. In the NetCDF earth science COI, a set of additional conventions (i.e., a profile) has been established to help ensure interoperability. The NetCDF Climate and Forecast (CF) metadata conventions specify how to specify dataset coordinate reference systems, units of measure, temporal reference frames, data types, and a variety other miscellaneous metadata. Additional information regarding the CF conventions can be found at [3]. 4.4 SENSOR MARKUP LANGUAGE SensorML is an OGC standard used to describe sensors. It supports encoding of sensor type, sensor location, physical sensor characteristics, as well as information related to the data produced by the sensor. In the Cube environment, examples of sensors of interest are weather radar systems (e.g., NEXRAD, TDWR, ASR-9), weather ground stations, and onboard aircraft instrumentation. 4.5 EBXML REGISTRY INFORMATION MODEL In order for metadata to be used for dataset and service discovery, it must be queryable. In NNEW, the primary query mechanism for discovery is the ebxml registry. The ebxml registry supports a core Registry Information Model (ebrim) which can be extended as needed to support metadata of various types. In order to support queries, metadata of different types (such as ISO 19115) is typically mapped to the Registry Information Model. The registry implementation, typically backed by a relational database, is then able to efficiently store and query the metadata. In ebxml terminology, extensions are supported via ebxml Registry extension packages. In the NNEW effort to date, work has been done on extension packages for ISO 19115/19119 metadata, and, to a lesser extent, on a SensorML package. Some of this work has built upon previous work done within the OGC [4, 5]. It is expected that these extension packages will become standardized over time, likely within the OGC. 16

25 4.6 TIME METADATA Time metadata within the ISO and NetCDF-CF datasets conform to the ISO 8601 standard. This standard allows time to be expressed in two primary formats: the basic format with a minimal number of separators and the extended format with separators added to enhance human readability. An example of the basic format, when fully specifying a timestamp down to 1-second resolution, is T101600Z. An example of the extended format for the same time is T10:16:00Z. 4.7 UNIFORM RESOURCE NAME NAMESPACE FOR FEDERATED CONTENT In a distributed data environment, unique identifiers are commonly used to allow information to be tracked throughout its lifecycle and cross-referenced to other information. Due to the fact that multiple organizations are involved in a federated environment, some conventions are required to guarantee that the identifiers are unique. RFC 4198 [6] provides a set of URN conventions deemed appropriate for the Cube. It leverages existing unique domain names already employed by the federation participants, and requires little in the way of centralized governance. The need for RFC 4198 is based on limitations with the core URN syntax. That syntax is: urn:<nid>:<nss> where <NID> is the Namespace Identifier and <NSS> is the Namespace Specific String [7] In order to be considered an official URN, the Namespace Identifier must be registered with the Internet Assigned Numbers Authority (IANA). Registration of a URN namespace is a non-trivial activity and can take multiple years. URN namespace identifiers also do not support existing domain names such as faa.gov, since the period character is disallowed. This conflicts with the desire in most federated environments to use a more lightweight convention and leverage existing domain names. RFC 4198 addresses this issue by specifying a single federated data content namespace, fdc, along with a set of conventions for the Namespace Specific String. The conventions include existing domain names and a time stamp with resolution appropriate to the data item being referenced. The syntax is: urn:fdc:<domainname>:<time>:<community-specific string> Adhering to this syntax, an example of a URN for an FAA gust front dataset would be: urn:fdc:faa.gov: :datasetseries:gustfront-001 Similarly, an example of a URN for a NWS-hosted data access service would be: urn:fdc:nws.noaa.gov: :service:webcoverageservice

26 Additional information regarding RFC 4198 can be found at: 18

27 5. GUIDELINES FOR METADATA STANDARDS IN THE CUBE Given the set of metadata standards presented in section 4, how is the combination of them used to address the requirements of the Cube? This section describes some of the key usage scenarios and provides mappings of the standards to them. The mappings are based on lessons learned during the NNEW prototyping efforts conducted during 2007 and Following the introductory mapping discussion, the use of each standard is described in detail, accompanied by real-world examples. For convenience, the examples in their original form (primarily XML files, with one exception) are also provided in Appendices at the end of this document. 5.1 MAPPING OF METADATA STANDARDS TO CUBE CAPABILITIES Dataset Discovery and Access Perhaps the primary usage scenario for metadata in the Cube relates to weather dataset discovery and access. The most common flow of events is as follows: 1. Data provider publishes metadata about one or more dataset series (an aggregation of individual datasets) to the registry. The metadata includes a variety of classification information, one aspect of which describes if the dataset is considered to be the SAS for its weather product type. The ISO metadata standard is used for the dataset series metadata. As part of the publishing operation, the registry maps the metadata to the ebrim in support of efficient queries. 2. Data provider publishes metadata about a data access service that provides access to the dataset series. A combination of WSDL and ISO is used for the service-level metadata. WSDL is used for the low-level definition of the messages/operations supported for a service, while ISO is used for higher-level metadata, such as information regarding which weather datasets the service provides access to. Again, as a part of the publishing operation, the registry maps the WSDL and ISO metadata to the ebrim standard in support of efficient queries. 3. Data consumer searches for datasets of interest, using a flexible find datasets query supported by the registry. 4. Data consumer selects one or more particular datasets from the query result set. 5. Consumer then issues a registry query to find out what data access services are available for dataset, and is returned a set of one or more services 6. Consumer selects an appropriate service, connects to it, and begins accessing data. 7. Consumer uses structural metadata embodied in each individual received dataset to unpack the data as needed. The preferred data format for gridded data products is 19

28 NetCDF, in which case the structural data are represented using the NetCDF CF conventions. To summarize, steps 1-6 use a combination of ISO 19115, ISO 19119, WSDL, and ebrim metadata, and deal with discovery and basic access of datasets. Step 7 uses NetCDF-CF structural metadata to unpack desired portions of individual datasets into memory, and actually make use of the data. The combination of metadata types stored in the registry for dataset series access and discovery is illustrated in the Figure 7. Dataset Series e.g. Precip Data for most recent one year period Dataset Metadata Metadata Metadata ISO Dataset Series Metadata NNEW Registry/Repository Operates On Data Access Service (e.g. OGC WCS or WFS) WSDL Service Metadata Auxiliary ISO Service Metadata Figure 7. Block diagram illustrating the three key metadata documents related to dataset discovery and access. For each aggregate dataset (dataset series) in the Cube, an ISO document is produced and published to the registry. For each service capable of providing access to the dataset, a combination of a WSDL file and an auxiliary ISO service metafile are created and published to the registry. 20

29 5.1.2 Sensor Discovery A second usage scenario for the Cube is the discovery of a weather sensor via the registry. In NextGen, discovery of a weather sensor may enable the sensor's status to be monitored in real time, or may provide an alternate pathway to discovery of a data feed associated with the sensor. The sensor discovery scenario is supported in NNEW via use of metadata that conforms to the SensorML standard, in conjunction with a mapping of SensorML to the ebrim standard to support the necessary registry queries. Note that the guidance related to the use of SensorML is considered less mature than in the case of the ISO and WSDL specifications described above, as SensorML has yet to be incorporated into a demonstration. 5.2 ISO HIGH LEVEL AGGREGATE DATASET METADATA Creation of an ISO metadata file for an aggregate weather datasets is the first step in enabling dataset discovery. The ISO standard has a lot of flexibility with respect to how it is used, requiring additional guidance. The subsequent sections describe the NNEW guidelines for the subset of the ISO data elements that are considered required for NNEW. Additional metadata over and above the elements described is considered optional. Throughout the following section, sample fragments of an actual ISO based dataset description are use for illustration purposes. The dataset description used as the source for the fragments is included in its entirety in Appendix A File Identifier Metadata The identifier for a dataset metadata file is a unique id that refers to the metadata file, not the dataset itself. The NNEW convention is to construct this id using the unique dataset id as the base, followed by the metadata suffix. Given the dataset URN convention described in section 4.7, this becomes: urn:fdc:<domain>:<time>:datasetseries:<dataset short id>:metadata An example of file identifier metadata for a FAA VIL dataset series encoded using ISO is shown below. <!-- Creating unique metadata file id from UUID of the dataset with a ':metadata" suffix. <gmd:fileidentifier> <gco:characterstring> urn:fdc:faa.gov: :datasetseries:vil-001:metadata </gco:characterstring> </gmd:fileidentifier> 21

30 5.2.2 Language and Character Set Metadata The language and character set elements are mandatory in ISO These elements represent the language and character set used within the metadata file itself, rather than the dataset being described. For the Cube the convention is to always use English as the character set and UTF-8 as the encoding. An example of the ISO encoding of these elements is shown below. <gmd:language> <gco:characterstring>eng</gco:characterstring> </gmd:language> <gmd:characterset> <gmd:md_charactersetcode codelist=" codelistvalue="utf8" /> </gmd:characterset> Dataset Hierarchy Level Metadata Metadata can exist at many levels. ISO supports numerous levels, including attribute, attributetype, collectionhardware, collectionsession, dataset, series, nongeographicdataset, dimensiongroup, feature, featuretype, propertytype, fieldsession, software, and service. See ISO for more details on these different levels. To date, NNEW has used only two of these levels, series, for dataset series metadata and service, for service-related metadata. An example of the ISO encoding for a dataset series is shown below. The hierarchylevel element is considered mandatory for NNEW. <gmd:hierarchylevel> <gmd:md_scopecode codelist=" codelistvalue="series" /> </gmd:hierarchylevel> Dataset Contact Metadata For each NNEW dataset, one or more contacts are provided, each with a particular role. Roles supported by ISO include resourceprovider, custodian, owner, user, distributor, originator, pointofcontact, principalinvestigator, processor, publisher, and author. Of primary interest with respect to an operational system is the pointofcontact. At a minimum, Cube conventions require a single contact with that role. Additional contact information is considered optional. 22

31 Below is an example of the ISO encoding of a point of contact for a dataset series. The individual fields are self-explanatory. <gmd:contact> <gmd:ci_responsibleparty> <gmd:organisationname> <gco:characterstring>mit Lincoln Laboratory</gco:CharacterString> </gmd:organisationname> <gmd:contactinfo> <gmd:ci_contact> <gmd:phone> <gmd:ci_telephone> <gmd:voice> <gco:characterstring> </gco:characterstring> </gmd:voice> <gmd:facsimile> <gco:characterstring> </gco:characterstring> </gmd:facsimile> </gmd:ci_telephone> </gmd:phone> <gmd:address> <gmd:ci_address> <gmd:deliverypoint> <gco:characterstring>244 Wood St.</gco:CharacterString> </gmd:deliverypoint> <gmd:city> <gco:characterstring>lexington</gco:characterstring> </gmd:city> <gmd:administrativearea> <gco:characterstring>ma</gco:characterstring> </gmd:administrativearea> <gmd:postalcode> <gco:characterstring>02146</gco:characterstring> </gmd:postalcode> <gmd:country> <gco:characterstring>usa</gco:characterstring> </gmd:country> <gmd:electronicmailaddress> <gco:characterstring>olivern@ll.mit.edu</gco:characterstring> </gmd:electronicmailaddress> </gmd:ci_address> </gmd:address> </gmd:ci_contact> </gmd:contactinfo> <gmd:role> <gmd:ci_rolecode codelist=" codelistvalue="pointofcontact" /> </gmd:role> </gmd:ci_responsibleparty> </gmd:contact> 23

32 5.2.5 Metadata Time Stamp Each metadata file has a mandatory time stamp element. This time stamp refers to the time of creation of the metadata file, not necessarily the creation of the dataset described by the file. In many instances, these two times will co-exist, but not in every case. The guidance is to always use a fully-specified time stamp using GMT, as shown below. <gmd:datestamp> <gco:datetime> t09:00:00z</gco:datetime> </gmd:datestamp> Coordinate Reference System Metadata Datasets can be viewed as independent of a particular coordinate reference system (CRS). Data access services may, for example, provide the same data set using multiple geographic projections, performing coordinate system transformations on the fly as needed. This situation is expected to exist within the Cube, and will be supported. Though support for multiple CRSs is provided at the data access service level, datasets will nevertheless have a native projection the projection used for the underlying storage of the master copy. It is useful to specify this projection using metadata, since access to data using non-native projections can sometimes introduce distortion during the reprojection process. One set of reference systems very common in the geospatial world is the set available from the European Petroleum Survey Group (EPSG). A complete listing of the CRSs supported by EPSG may be found at: The sample below illustrates the ISO encoding of the most commonly used EPSG reference system, the WGS 84 latitude, longitude CRS. The most common <gmd:referencesysteminfo> <gmd:md_referencesystem> <gmd:referencesystemidentifier> <gmd:rs_identifier> <gmd:code> <gco:characterstring> urn:ogc:def:crs:epsg:4326</gco:characterstring> </gmd:code> </gmd:rs_identifier> </gmd:referencesystemidentifier> </gmd:md_referencesystem> </gmd:referencesysteminfo> 24

33 In the weather domain, parameterized CRSs are often used in place of the more static, map-focused EPSG definitions. Representation of a parameterized CRS in ISO is considered a work in progress and will be described in a future release of this guidelines document Dataset Identification Metadata This section describes the ISO identificationinfo section in the context of a dataset. Given the complexity of this section, the description is broken down by subsection. Each dataset is assigned a unique identifier. The guidance for the unique id is to use a URN that conforms to RFC 4198, as well as the additional naming conventions described in section 4.7. An example of the encoding of the dataset identifier is shown below. It is important to note that the critical aspect of the identifier is that it is globally unique, and that is what the NNEW guidelines are intended to provide help with. It is not intended that the URNs be parsed to extract information, and in fact that is considered a bad practice. Structured information is better managed via an XML schema. <gmd:identificationinfo> <gmd:md_dataidentification uuid="urn:fdc:faa.gov: :datasetseries:vil-001"> Some citation information is associated with each NNEW dataset. At a minimum, NNEW guidelines require that there be a single citation information block with a title and a creation date. Additional citation information is optional. The example below illustrates an example containing a citation with a publication date as well as a creation date. Standard ISO date type codes include creation, publication, and revision. Like all of the ISO code lists, the date type code list may be extended if needed for a particular application. <gmd:citation> <gmd:ci_citation> <gmd:title> <gco:characterstring>vertically Integrated Liquid (VIL)</gco:CharacterString> </gmd:title> <gmd:date> <gmd:ci_date> <gmd:date> <gco:datetime> t14:27:49</gco:datetime> </gmd:date> <gmd:datetype> <gmd:ci_datetypecode codelist=" sts.xml#ci_datetypecode" codelistvalue="creation" /> </gmd:datetype> </gmd:ci_date> 25

34 </gmd:date> <gmd:date> <gmd:ci_date> <gmd:date> <gco:datetime> t13:00:20</gco:datetime> </gmd:date> <gmd:datetype> <gmd:ci_datetypecode codelist=" delists.xml#ci_datetypecode" codelistvalue="publication" /> </gmd:datetype> </gmd:ci_date> </gmd:date> </gmd:ci_citation> </gmd:citation> High-level abstract and purpose information are also required for each dataset. The abstract should provide a high-level description of the dataset content, while the purpose provides the intended use of the data. An example encoding is shown below. <gmd:abstract> <gco:characterstring> Integrated reflectivity within a column of air. </gco:characterstring> </gmd:abstract> <gmd:purpose> <gco:characterstring>weather monitoring and prediction</gco:characterstring> </gmd:purpose> Standard ISO status values span the lifecycle of a dataset and include required, planned, underdevelopment, ongoing, completed, historicalarchive, and obsolete. During the NNEW prototyping phase, the convention is to use underdevelopment. For operational datasets, such as those available in the time frame of the Cube Initial Operating Capability (IOC), the guidance is to use ongoing, since real-time datasets are continually being updated. When given datasets are retired (perhaps superseded by another dataset that is now considered to be the SAS), it is expected that they will be reclassified using the historicalarchive status. These guidelines do not currently prescribe how to make use of the other status values. <gmd:status> <gmd:md_progresscode codelist="./resources/gmxcodelists.xml#md_progresscode" codelistvalue="underdevelopment" /> </gmd:status> 26

35 A set of one or more keywords is supported by the ISO standard. Current guidance for the Cube is to use the "Weather keyword as a minimum, and in addition, use the Aviation keyword if the weather data are specifically targeted towards aviation. <!-- Ordinary (english) keywords for dataset <gmd:descriptivekeywords> <gmd:md_keywords> <gmd:keyword> <gco:characterstring>aviation</gco:characterstring> </gmd:keyword> <gmd:keyword> <gco:characterstring>weather</gco:characterstring> </gmd:keyword> </gmd:md_keywords> </gmd:descriptivekeywords> Datasets can be classified in multiple ways, using a variety of taxonomies. In NNEW, for example, a taxonomy describing the different domains (e.g., SAS) in the Cube will be used for a variety of purposes. The current version of ISO does not, however, have explicit support for general classification schemes. A topiccategory element exists, but it is tied to the ISO topic category taxonomy. A separate strategy is needed to support general classification. The ISO specification provides an example in Annex I that supports an external taxonomy by leveraging the descriptive keywords information section. The example is quite complex given the relatively straightforward goal of representing a given node in a taxonomy. Current guidance is to use a simplified version of the approach in ISO 19115, Annex I. The recommended encoding of taxonomy information for the Cube is to use the ebrim classification scheme representation from the ebxml registry information model. The example below illustrates the recommended approach to represent an ebxml dataset classification using the ISO keywords information section. The classification in this example labels the dataset as being in the SAS domain of the Cube. Note that ebxml classification nodes use single URNs that incorporate the classification scheme, so a single keywords section with a keyword type of classificationnodeid can be used to hold multiple classifications (only a single one is shown in this example) A modification to ISO to support general-purpose classifications more explicitly would be of use to the Cube. This feedback will be provided to the ISO community by the NNEW team. <gmd:descriptivekeywords> <gmd:md_keywords > <gmd:keyword > <gco:characterstring> urn:ogc:def:ebrim-classificationnode:faa:datacube:unrestricted:sas 27

36 </gco:characterstring> </gmd:keyword> <!-- Keyword type specifying extended version of codelist that includes the 'classificationnodeid' type code <gmd:type> <gmd:md_keywordtypecode codelist=" codelistvalue="classificationnodeid" /> </gmd:type> </gmd:md_keywords> </gmd:descriptivekeywords> Spatial representation types defined in ISO include vector, grid, texttable, tin, stereomodel, and video. Datasets in the Cube are currently only making use of the first two types, vector, and grid. As one would expect, the grid data type is used for gridded data, and the vector data type is used for non-gridded data types, such as lightning strike data. Both gridded and non-gridded datasets tend to have a native spatial resolution. This information is considered mandatory for the Cube. In the case of non-gridded data containing position measurements, the resolution is considered to be the precision of position data (a lightning sensor might have a resolution of 100m, for example) Below is an example encoding for a gridded data product with a 1 km native resolution. <gmd:spatialrepresentationtype> <!-- Spatial representation types include: { vector, grid, texttable, tin, stereomodel, video } <gmd:md_spatialrepresentationtypecode codelist=" codelistvalue="grid" /> </gmd:spatialrepresentationtype> <!-- Dataset has 1-km grid cell resolution <gmd:spatialresolution> <gmd:md_resolution> <gmd:distance> <gco:distance uom="m">1000.0</gco:distance> </gmd:distance> </gmd:md_resolution> </gmd:spatialresolution> ISO allows the language used within the dataset to be specified separately from the language used within the metadata file (see section 5.2.2). This element is mandatory in ISO The guideline for the Cube is to always specify English as the language, as shown below. <gmd:language> <gco:characterstring>eng</gco:characterstring> 28

37 </gmd:language> ISO supports a fixed topic category taxonomy. The primary topic category for Cube datasets is climatologymeteorologyatmosphere. Additional topic categories may be specified if desired. Consult the ISO specification for the entire list of topic categories. <gmd:topiccategory> <gmd:md_topiccategorycode>climatologymeteorologyatmosphere</gmd:md_topiccategorycode> </gmd:topiccategory> The spatial and temporal extent elements are considered mandatory for Cube datasets. The spatial extent for datasets is specified as a simple bounding box using latitude, longitude coordinates. The temporal extent is specified using a time period. For dataset series of the online, real-time variety, the begin time of the dataset is the time of the earliest data in the series, while the end time of the dataset is specified as now using the GML encoding for the current time. This reflects the fact that the dataset is in fact being continually updated with fresh data, without requiring that the metadata about the dataset be continually updated. The basic idea is to prevent the need to continually post new information to the high-level metadata registry just to update the dataset end time, even though all the other information in the metadata is remaining static. <gmd:extent> <gmd:ex_extent> <gmd:geographicelement> <gmd:ex_geographicboundingbox> <gmd:westboundlongitude> <gco:decimal>-120.0</gco:decimal> </gmd:westboundlongitude> <gmd:eastboundlongitude> <gco:decimal>-70.0</gco:decimal> </gmd:eastboundlongitude> <gmd:southboundlatitude> <gco:decimal>25.0</gco:decimal> </gmd:southboundlatitude> <gmd:northboundlatitude> <gco:decimal>40.0</gco:decimal> </gmd:northboundlatitude> </gmd:ex_geographicboundingbox> </gmd:geographicelement> <gmd:temporalelement> <gmd:ex_temporalextent> <gmd:extent> <gml:timeperiod gml:id="timeperiod-001"> <gml:beginposition> t09:00:00z</gml:beginposition> <gml:endposition indeterminateposition= now /> </gml:timeperiod> </gmd:extent> </gmd:ex_temporalextent> 29

38 </gmd:temporalelement> </gmd:ex_extent> </gmd:extent> Content Information Metadata There are two forms of content information, one for ISO Coverages and one for ISO Features. In both cases, the critical information to include in the metadata is the type or types of data in the dataset. For the most part, Cube datasets are partitioned based on a single data type, so only a single data type need be encoded. In some cases, such as a coverage dataset containing a wind field broken down into north-south and east-west components, or a coverage dataset containing a grid of data values and a corresponding grid of flags values associated with the data values, there may be multiple data types in the dataset. Below is an example of an ISO content information section for a grid coverage. Unfortunately, the ISO coverage description is not fully aligned with the ISO coverage specification in terms of the underlying coverage model. In particular, an ISO coverage supports multiple properties at each spatiotemporal location (the multiple properties are considered to be the range of the coverage in ISO terminology), and that is only partially supported in ISO NNEW will be working with the ISO community to address this shortcoming. In the meantime, NNEW is adopting the following conventions for the coverage description section: The <attributedescription> and <contenttype> elements are set to nil (no value specified), with a nilreason of inapplicable. The reason these elements are set to nil is that they have a cardinality of 1, and can therefore not represent a coverage with multiple properties. Note that these elements are mandatory in ISO 19115, so they cannot simply be omitted. Use the dimension attribute of the coverage description to describe the coverage range set (see ISO for more details regarding coverage range sets). Each property, or data type, in the coverage is specified as an ontology reference using Web Ontology Language (OWL). This strategy aligns with that used to represent coverages in other OGC specifications, such as the Observations & Measurements specification. Coverage datasets can often contain related attributes. A common case is for a dataset variable (e.g., precipitation) to have a related status flags attribute that qualifies sensor coverage, data quality, or likelihood of precipitation phase. In the NetCDF Climate and Forecast convention, these related attributes are declared as ancillary variables and they are associated with a data variable, as such, along with a standard name consisting of the data variable standard name followed by a standard name modifier, e.g., status_flag. In a data model sense, these related attributes can be thought of as properties of a class. In many compute languages, properties of a class are represented using a '.' separator. The same convention is adopted below to represent the flags field associated with the core data type field in the OWL syntax below. This convention is the current guidance for the Cube, but is subject to change in future revisions of this document. 30

39 <gmd:contentinfo> <gmd:md_coveragedescription> <! These elements cannot be used in cases where a coverage has multiple properties, and are therefore not used in the Cube (pending changes to ISO 19115) <gmd:attributedescription gco:nilreason="inapplicable" /> <gmd:contenttype nilreason= inapplicable /> <!-- ISO Coverage 'rangeset' specified using one or more dimension elements. OWL references used to describe the properties (data types) in the range set. <gmd:dimension> <gmd:md_rangedimension> <gmd:descriptor> <gco:characterstring> String> </gmd:descriptor> </gmd:md_rangedimension> </gmd:dimension> <gmd:dimension> <gmd:md_rangedimension> <gmd:descriptor> <gco:characterstring> co:characterstring> </gmd:descriptor> </gmd:md_rangedimension> </gmd:dimension> </gmd:md_coveragedescription> </gmd:contentinfo> Data Quality Metadata Guidance for specification of data quality information over and above the core ISO specification is currently limited. The scope element is required, and for dataset series will be set to the same value as the dataset series hierarchylevel ('series'). Guidance for the lineage element is to include a simple statement regarding how the dataset was produced, at a minimum. In the future it is expected that datasets produced via fusion of other datasets will include information about those original source datasets. The source element in the data quality section is envisioned to be useful for that purpose, specifying references to source datasets via their unique identifiers. For now, the source element is considered optional for the Cube, and it is omitted from the sample below. <!-- Data Quality field not yet filled in <gmd:dataqualityinfo> <gmd:dq_dataquality> <gmd:scope> <gmd:dq_scope> <gmd:level> 31

40 <gmd:md_scopecode codelist=" codelistvalue="series" /> </gmd:level> </gmd:dq_scope> </gmd:scope> <gmd:lineage> <gmd:li_lineage> <gmd:statement> <gco:characterstring>vil data are being produced from radar data which has been quality edited</gco:characterstring> </gmd:statement> </gmd:li_lineage> </gmd:lineage> </gmd:dq_dataquality> </gmd:dataqualityinfo> Constraints Metadata Constraints on the usage of a dataset exist in a variety of forms, including legal constraints and security constraints. The ISO representation of these data is relatively high level. Current metadata guidance for the Cube is to specify the security classification using the encoding below. Valid values for security classification include unclassified, restricted, confidential, secret, and topsecret. The legal constraints block is considered optional, and if omitted, will be taken to imply that there are no legal constraints on the use of the data set. Valid values for legal constraints include copyright, patent, patentpending, trademark, license, intellectualpropertyrights, restricted, and otherrestrictions. It is expected that this metadata area, in particular, will be affected by the overall adoption of role-based access policies throughout the NAS in the NextGen environment. The NNEW program will track the evolution of the role-based access policies and adjust the constraints metadata guidance accordingly over time. <gmd:metadataconstraints> <!-- Restriction codes include: { copyright, patent, patentpending, trademark, license, intellectualpropertyrights, restricted, otherrestrictions } <gmd:md_legalconstraints> <gmd:accessconstraints> <gmd:md_restrictioncode codelist=" codelistvalue="restricted" /> </gmd:accessconstraints> </gmd:md_legalconstraints> </gmd:metadataconstraints> <gmd:metadataconstraints> <gmd:md_securityconstraints> <gmd:classification> <gmd:md_classificationcode codelist=" 32

41 codelistvalue="unclassified" /> </gmd:classification> </gmd:md_securityconstraints> </gmd:metadataconstraints> Maintenance Metadata It is useful to know how often metadata are maintained, in order for client applications to know how often to check for metadata updates. ISO provides support for a variety of common metadata update frequencies, including continual, daily, weekly, fortnightly, monthly, quarterly, biannually, annually, asneeded, irregular, notplanned, unknown. In the case of relatively static, real-time aggregate datasets, the high-level metadata need not change very often. In such cases, the guidance is to use asneeded for the update frequency. For the non realtime case, other update frequencies may be more appropriate. <gmd:metadatamaintenance> <gmd:md_maintenanceinformation> <gmd:maintenanceandupdatefrequency> <gmd:md_maintenancefrequencycode codelist=" codelistvalue="asneeded" /> </gmd:maintenanceandupdatefrequency> </gmd:md_maintenanceinformation> </gmd:metadatamaintenance> 5.3 WSDL SERVICE METADATA In the Cube, WSDL is used to define the core service information, while ISO provides complementary, higher-level, metadata. The core information contained in service descriptions falls into two broad categories, abstract service interface information and specific service instance information. The abstract service interface describes the message and transport details (bindings) that apply to all instances of a particular service type. Service instance information describes properties of a service that vary with each instance, such as the endpoint at which the service can be reached. It is generally considered a good practice to split up service descriptions along the lines of the abstract service interface information, transport binding information, and service instance information. This approach, illustrated in the Figure 8, maximizes the reuse of service definition information, and helps ensure that service instances all incorporate common definitions of the underlying interfaces. WSDL supports this practice via its support for the import element. An instance WSDL constructed in this manner will typically import a bindings WSDL, which in turn imports the transport-independent interface WSDL. 33

42 Common WSDL Service Interface Definition Service Operations Operation GetMetadata Message GetMetadataRequest Message GetMetadataResponse Operation GetData Message GetDataRequest Message GetDataResponse SOAP / HTTP Service Bindings HTTP SOAP/ JMS Implements Operations/Binding(s) Implements Operations/Binding(s) Endpoint Service Instance #1 Endpoint Service Instance #2 Service Instances WSDL Service Implementation Definition - Provider <N> WSDL Service Implementation Definition for Data Provider <N> Figure 8. Illustration of the separation of service interfaces from service implementations. A service interface defines the messages and message transport details for a service, whereas an implementation represents a particular instance or related set of instances that implement the service interface An example of partitioning service metadata across multiple WSDL files, based on abbreviated versions of WSDL files included with the OASIS ebxml Registry/Repository specification, is shown in the following sections. Section describes the generic interface definitions. Section contains details related to a particular service interface binding (in this case the SOAP/HTTP binding). Section

43 defines the endpoint information for a sample service instance. Finally, section provides an example of an OGC Web Feature Service WSDL encoding for a given service instance Modular WSDL Abstract Interface Example <?xml version = "1.0" encoding = "UTF-8"?> <definitions name="regrep-server-interface" targetnamespace="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:interfaces:4.0" xmlns=" xmlns:lcm="urn:oasis:names:tc:ebxml-regrep:xsd:lcm:4.0" xmlns:mime=" xmlns:query="urn:oasis:names:tc:ebxml-regrep:xsd:query:4.0" xmlns:rs="urn:oasis:names:tc:ebxml-regrep:xsd:rs:4.0" xmlns:soap=" xmlns:tns="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:interfaces:4.0" xmlns:xsd=" <documentation> $Header:$ Author: Matt MacKenzie, Farrukh Najmi This is the normative abstract interface definition in WSDL for the OASIS ebxml Registry services. This WSDL file defines the messages and porttypes needed to communicate with a compliant ebxml Registry. </documentation> <!-- Messages for all service operations are defined in schemas external to WSDL file <types> <xsd:schema> <!-- Import the rs.xsd, lcm.xsd and query.xsd. <xsd:import namespace="urn:oasis:names:tc:ebxml-regrep:xsd:rs:4.0" schemalocation="../../xsd/rs.xsd"/> <xsd:import namespace="urn:oasis:names:tc:ebxml-regrep:xsd:lcm:4.0" schemalocation="../../xsd/lcm.xsd"/> </xsd:schema> </types> <!-- Request/response messages for Life Cycle Manager (lcm) operations. Common response message is shared by all Life Cycle Manager (lcm) operations. <message name="msgsubmitobjectsrequest"> <part element="lcm:submitobjectsrequest" name="partsubmitobjectsrequest"/> </message> <message name="msgupdateobjectsrequest"> <part element="lcm:updateobjectsrequest" name="partupdateobjectsrequest"/> </message> <message name="msgremoveobjectsrequest"> <part element="lcm:removeobjectsrequest" name="partremoveobjectsrequest"/> </message> <message name="msgregistryresponse"> <documentation>defines a RegistryResponse message.</documentation> <part element="rs:registryresponse" name="partregistryresponse"/> </message> <!-- Exception message <message name="msgregistryexception"> <part element="rs:registryexception" name="fault"/> 35

44 </message> <!-- Life Cycle Manager Interface <porttype name="lifecyclemanager"> <documentation> The porttype for LifecycleManager abstract interface. </documentation> <operation name="removeobjects"> <input message="tns:msgremoveobjectsrequest"/> <output message="tns:msgregistryresponse"/> <fault name="registryexception" message="tns:msgregistryexception"/> </operation> <operation name="submitobjects"> <input message="tns:msgsubmitobjectsrequest"/> <output message="tns:msgregistryresponse"/> <fault name="registryexception" message="tns:msgregistryexception"/> </operation> <operation name="updateobjects"> <input message="tns:msgupdateobjectsrequest"/> <output message="tns:msgregistryresponse"/> <fault name="registryexception" message="tns:msgregistryexception"/> </operation> </porttype> <!-- Additional details omitted for this example... </definitions> Modular WSDL Service Bindings Example <?xml version="1.0" encoding="utf-8"?> <definitions name="regrep-server-binding" targetnamespace="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:bindings:4.0" xmlns=" xmlns:bindings="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:bindings:4.0" xmlns:interfaces="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:interfaces:4.0" xmlns:mime=" xmlns:soap=" xmlns:tns="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:bindings:4.0" xmlns:xsd=" <documentation> Author: Matt MacKenzie, Farrukh Najmi This is the normative SOAP binding in WSDL for the OASIS ebxml Registry services. </documentation> <!-- Import the WSDL interface definition. The common interface may be shared by numerous binding variants (SOAP, HTTP, etc...) <import location="regrep-server-interface.wsdl" namespace="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:interfaces:4.0"/> <!-- Life Cycle Manager binding <binding name="lifecyclemanagersoapbinding" type="interfaces:lifecyclemanager"> <documentation> SOAP Binding for LifecycleManager interface. </documentation> <soap:binding style="document" transport=" <operation name="removeobjects"> <soap:operation 36

45 soapaction="urn:oasis:names:tc:ebxmlregrep:wsdl:registry:bindings:4.0:lifecyclemanager#removeobjects"/> <input> <soap:body use="literal"/> </input> <output> <soap:body use="literal"/> </output> <fault name="registryexception"> <soap:fault name="registryexception" use="literal"/> </fault> </operation> <operation name="submitobjects"> <soap:operation soapaction="urn:oasis:names:tc:ebxmlregrep:wsdl:registry:bindings:4.0:lifecyclemanager#submitobjects"/> <input> <soap:body use="literal"/> </input> <output> <soap:body use="literal"/> </output> <fault name="registryexception"> <soap:fault name="registryexception" use="literal"/> </fault> </operation> <operation name="updateobjects"> <soap:operation soapaction="urn:oasis:names:tc:ebxmlregrep:wsdl:registry:bindings:4.0:lifecyclemanager#updateobjects"/> <input> <soap:body use="literal"/> </input> <output> <soap:body use="literal"/> </output> <fault name="registryexception"> <soap:fault name="registryexception" use="literal"/> </fault> </operation> </binding> <!-- Additional details omitted for this example... </definitions> Modular WSDL Service Instance Example An example of a service instance WSDL for an ebxml RegRep service is shown below. It is relatively standard with respect to established industry practice. One key thing to note is the use of a unique identifier for the particular service instance as the target namespace. The guidance is to use this approach for all service instance WSDLs, as it is used to associate the WSDL with its sibling ISO metadata file. 37

46 <?xml version="1.0" encoding="utf-8"?> <definitions name="regrep-server-service-instance01" targetnamespace="urn:fdc:faa.gov:service:registry:instance01:4.0" xmlns=" xmlns:bindings="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:bindings:4.0" xmlns:spib="urn:oasis:names:tc:ebxml-regrep:wsdl:spi:bindings:4.0" xmlns:notb="urn:oasis:names:tc:ebxml-regrep:wsdl:notificationlistener:bindings:4.0" xmlns:soap=" xmlns:xsd=" <documentation> Example of a WSDL describing a service instance compliant with the compliant with the OASIS ebxml Registry specification A WSDL describing a service instance specifies the generic service interface (which includes the binding type as the term is used here) and adds information about the specific endpoint for this service instance. There will typically be multiple instances of a single service type in a given network, each with its own endpoint information. The details related to the generic service interface are imported from common WSDL files. </documentation> <!-- Import the service transport-specific binding WSDL, which in turn imports The transport-independent interface WSDL <import location="regrep-server-binding.wsdl" namespace="urn:oasis:names:tc:ebxml-regrep:wsdl:registry:bindings:4.0"/> <!-- Describe the endpoint details for the Life Cycle Manager service for this particular registry service instance <service name="lifecyclemanagersoapservice"> <documentation> A service providing SOAP endpoints for LifecycleManager interface. </documentation> <port binding="bindings:lifecyclemanagersoapbinding" name="lifecyclemanagerport"> <documentation> A template SOAP endpoint for ebxml RegRep LifecycleManager interface. </documentation> <soap:address location=" </port> </service> </definitions> Data Access Service WSDL Example The previous examples illustrated the partitioning of WSDLs into abstract interface and binding WSDLs, and service instance WSDLs, using the WSDLs that accompany the ebxml Registry/Repository service specification. This example describes a particular instance of a Web Feature Service, using the same methodology. Note that the abstract service information is simply imported from the OGC web site, and need not be managed within the Cube. This demonstrates the benefit of the modular WSDL approach, as all WFS implementations can easily share the identical abstract service definitions. As described in the previous section, the guidance is to use the service's unique identifier (URN) for the service instance as the target namespace. 38

47 <?xml version="1.0" encoding="utf-8"?> <wsdl:definitions targetnamespace="urn:fdc:ll.mit.edu: :service:wfs-01" xmlns:wsdl=" xmlns:http=" xmlns:wfs-soap=" xmlns:soap=" <wsdl:documentation xmlns:dc=" <dc:date> </dc:date> <dc:description> This WSDL document defines the service-specific properties of a MyService WFS implementation; it specifies available endpoints and alternative bindings. </dc:description> </wsdl:documentation> <!-- Import abstract service interface details from separate WSDL file <wsdl:import namespace=" location=" <!-- Add endpoint details for this service instance <wsdl:service name="mit_webfeatureservice-01"> <wsdl:documentation> MIT Web Feature Service instance. Supports SOAP/HTTP interface </wsdl:documentation> <wsdl:port name="wfs-soap-port" binding="wfs-soap:wfs-soap"> <soap:address location=" </wsdl:port> </wsdl:service> </wsdl:definitions> 5.4 ISO SERVICE METADATA ISO service metadata (which itself builds on ISO 19115) augments the core WSDL metadata with contact information, legal and security constraints, and provides the information needed to link services with the datasets they provide. The sections below describe the different sections in an ISO compliant metadata file. The ISO encoding schema for ISO 19115/ISO abstract data models is used throughout. A number of the metadata sections for an ISO service description are essentially identical to the ISO dataset description provided in section 5.2. In some cases the equivalent subsection in section 5.2 is simply referenced for reasons of conciseness. As in the dataset case, the entire sample schema used as the source for the included XML fragments is provided in Appendix B File Identifier Metadata As was the case for dataset metadata, the identifier for a service metadata file is a unique id that refers to the metadata file, not the service itself. The convention for the Cube is to construct this id using the unique service id as the base, followed by the metadata suffix. Following the service URN convention described in section 4.7 this becomes: 39

48 urn:fdc:<domain>:<time>:service:<service short id>:metadata An example of file identifier metadata for a WCS series encoded using ISO is shown below. <!-- Creating unique metadata file id from UUID of the service with a ':metadata" suffix. <gmd:fileidentifier> <gco:characterstring>urn:fdc:ll.mit.edu: :service:wcs-01:metadata</gco:characterstring> </gmd:fileidentifier> Language and Character Set Metadata The language and character set elements are mandatory in ISO 19119, which builds upon ISO The guidance is the same for the service metadata as it is for the dataset metadata. English is used for the character set and UTF-8 is used for the encoding. <gmd:language> <gco:characterstring>eng</gco:characterstring> </gmd:language> <gmd:characterset> <gmd:md_charactersetcode codelist=" codelistvalue="utf8" /> </gmd:characterset> Service Hierarchy Level Metadata ISO supports numerous levels, as described in section For a service, the guidance is straightforward a hierarchy level of service should be used. An example is provided below. <gmd:hierarchylevel> <gmd:md_scopecode codelist=" codelistvalue="service" /> </gmd:hierarchylevel> Service Contact Metadata Service contact metadata are of the same form as dataset metadata, and the same guideline applies. At a minimum, a contact with the role pointofcontact must be specified. Refer to section for an example. 40

49 5.4.5 Time Stamp Metadata The service metadata time stamp is of the same form as dataset metadata time stamp. The time stamp refers to the time of creation of the metadata file, not necessarily the creation of the service described by the file. In many instances, these two times will coexist, but not in every case. <gmd:datestamp> <gco:datetime> t09:00:00z</gco:datetime> </gmd:datestamp> Service Identification Metadata Each service is assigned a unique identifier. The guidance for the unique id is to use a URN that conforms to RFC 4198, as well as the additional naming conventions described in section 4.7. An example of the encoding of the service identifier for an OGC Web Coverage Service is shown below. Current guidance requires that the service URN be identical to the target namespace for the service instance WSDL document associated with these ISO metadata. See sections and for additional details. When the transition is made from WSDL 1.1 to WSDL 2.0, it is expected that a WSDL 2.0 extension will be created to make the linkage between the WSDL file and the ISO metadata more explicit. <gmd:identificationinfo> <srv:sv_serviceidentification uuid=" urn:fdc:ll.mit.edu: :service:wcs-01"> Guidelines for citation data embedded within the service identification metadata block are identical to the guidelines for datasets. At a minimum, a title and creation date for the service is specified. The creation date of the service represents when the service first became operational. An example of a citation section for a Web Coverage Service is shown below. <gmd:citation> <gmd:ci_citation> <gmd:title> <gco:characterstring>mit LL Web Coverage Service</gco:CharacterString> </gmd:title> <gmd:date> <gmd:ci_date> <gmd:date> <gco:datetime> t13:00:20</gco:datetime> </gmd:date> <gmd:datetype> <gmd:ci_datetypecode codelist=" delists.xml#ci_datetypecode" 41

50 codelistvalue="creation"/> </gmd:datetype> </gmd:ci_date> </gmd:date> </gmd:ci_citation> </gmd:citation> High-level abstract information is also required for each service. The abstract should provide a high-level description of the service, approximately a paragraph in length. An example encoding is shown below. <gmd:abstract> <gco:characterstring>web Coverage Service (WCS) providing access to data sets generated by the Corridor Integrated Weather System (CIWS). Available data sets include Vertically Integrated Liquid (VIL) and Echo Tops, each with a set of accompanying forecasts</gco:characterstring> </gmd:abstract> ISO provides a standard service taxonomy, which includes definitions of the OGC WCS and WFS services being adopted by the NNEW Program. The service type and version should be included in every service metadata file, as shown below. Note that the service version should conform to the 2-digit {major}.{minor} numbering scheme shown, and not be extended to use additional digits. <srv:servicetype> <gco:localname codespace=" </srv:servicetype> <srv:servicetypeversion> <gco:characterstring>1.1</gco:characterstring> </srv:servicetypeversion> Cube services are primarily data access services. In the envisioned network topology for FAA s portion of the Cube, data access services are most commonly tightly-coupled to their underlying data store, for efficiency reasons. They may also be more loosely-coupled in some cases, such as a floating data access service that is capable of mediation among a variety of formats, or is acting as a caching proxy for another data access service. Valid values for the ISO coupling type include tight and loose. The majority of data access services will specify a coupling of tight. The guidance requires only that one or the other be specified. <srv:couplingtype> <srv:sv_couplingtype codelist=" codelistvalue="tight"/> </srv:couplingtype> 42

51 ISO is capable of describing service operations at a high level. This functionality, however, overlaps with core functionality available in WSDL. For the Cube, the more mainstream WSDL approach is being used to describe service operations, and the ISO metadata are considered to be the mechanism by which richer, higher-level metadata are provided. For this reason, the mandatory containsoperations metadata element is set to nil (no value) with a reason of inapplicable, as shown below. <srv:containsoperations gco:nilreason="inapplicable"/> Perhaps the key added value of ISO service descriptions, when considered with WSDL alone, is the ability to link a service to the datasets that it operates on. With respect to Cube data access services this is equivalent to the ability to link the data access service to the datasets it provides. The guidance for encoding this relationship is very straightforward, and leverages the uuidref attribute of the operateson element to provide the link to the dataset s unique id. An example for a service that provides access to two weather datasets is shown below. The number of datasets that can be associated with the service in this manner is unbounded. <srv:operateson uuidref="urn:fdc:ll.mit.edu: :dataset:vil-001"/> <srv:operateson uuidref="urn:fdc:ll.mit.edu: :dataset:echotops-001"/> Distribution Information Section Data access services will often provide access to datasets using multiple formats. In the Cube, for example, there will be a need to support NetCDF-4, as well as older formats such as NetCDF-3 and GRIB. When requesting data, the desired format will be typically be specified in the request. Service metadata must include information as to what formats are supported, so clients can be aware of the supported formats. For the Cube, the ISO distribution element is being leveraged to define the set of supported formats for a given data access service. An example for a service that supports NetCDF-4 as well as NetCDF-3 is shown below. This section is required for all Cube data access services. It is likely that data access performance for one of the supported formats is better than for the others. This will typically be the case for a format that is identical to the format of the underlying data store, since the need to do a translation is eliminated. Current guidance is to list the format used by the underlying data store first as a hint to clients as to which format provides the best access speed. This approach is not considered ideal, and will almost certainly be refined and enhanced in future versions of the guidelines. 43

52 <gmd:distributioninfo> <gmd:md_distribution> <gmd:distributionformat> <gmd:md_format> <gmd:name> <gco:characterstring>application/x-netcdf</gco:characterstring> </gmd:name> <gmd:version> <gco:characterstring>4</gco:characterstring> </gmd:version> </gmd:md_format> </gmd:distributionformat> <gmd:distributionformat> <gmd:md_format> <gmd:name> <gco:characterstring>application/x-netcdf</gco:characterstring> </gmd:name> <gmd:version> <gco:characterstring>3</gco:characterstring> </gmd:version> </gmd:md_format> </gmd:distributionformat> </gmd:md_distribution> </gmd:distributioninfo> Service Constraints Metadata The same guidance provided for datasets applies in the case of services as well. See section Maintenance Metadata The same guidance provided for datasets applies in the case of services as well. See section EBXML TAXONOMY METADATA Management of weather information often requires that datasets be classified based on multiple criteria, an example of which would be a dataset classified as being in the Cube SAS domain. Classification is most commonly based around the idea of a taxonomy, a set of concepts organized in a simple hierarchy. More general classification can be implemented via the use of an ontology, which has a structure that is not constrained to the parent-child relationships of a fixed hierarchy. The guidance for representing ontology-based classification metadata is to use the OWL. OWL ontology representations are fairly standardized, and additional guidance is not provided in this version of the document. 44

53 For taxonomy metadata, the current guidance is to use the classification scheme representation available in ebrim. A sample fragment of the Cube domain classification scheme used to date in NNEW demonstrations is shown below. The classification scheme is included in its entirety in Appendix D. The ebrim representation is practical in the sense that taxonomies can easily be published to the registry being used by the NNEW Program, since ebrim is the registry's native data model. As ontologies take on a larger role in the Cube, it may be more practical to use an OWL representation for simple taxonomies as well as ontologies, just to unify the representation. This will require a well-defined registry profile for OWL, which is a subject for future work in the NNEW Program. <rim:registryobject xsi:type="rim:classificationschemetype" id="urn:ogc:def:ebrim-classificationscheme:faa:datacubedomains" objecttype="urn:oasis:names:tc:ebxml-regrep:objecttype:registryobject:classificationscheme" isinternal="true" nodetype="urn:oasis:names:tc:ebxml-regrep:nodetype:uniquecode"> <rim:name> <rim:localizedstring xml:lang="en" value="faa/noaa/dod Data Cube Domain Taxonomy"/> </rim:name> <rim:description> <rim:localizedstring xml:lang="en" value="this taxonomy is used to classify data sets according to the domain(s) of the data cube to which they belong. Initial use is focused on weather datasets. Datasets containing surveillance and other aeronautical information may also make use of this taxonomy" /> </rim:description> <!-- Single top-level node <rim:classificationnode code="datacube" objecttype="urn:oasis:names:tc:ebxml-regrep:objecttype:registryobject:classificationnode" parent="urn:ogc:def:ebrim-classificationscheme:faa:datacubedomains" id="urn:ogc:def:ebrim-classificationnode:faa:datacube"> <rim:name> <rim:localizedstring xml:lang="en" value="datacube"/> </rim:name> <rim:description> <rim:localizedstring xml:lang="en" value="entire Data Cube (the 'All' domain)"/> </rim:description> <rim:classificationnode code="restricted" objecttype="urn:oasis:names:tc:ebxml-regrep:objecttype:registryobject:classificationnode" parent="urn:ogc:def:ebrim-classificationnode:faa:datacube:" id="urn:ogc:def:ebrim-classificationnode:faa:datacube:restricted"> <rim:name> <rim:localizedstring xml:lang="en" value="restricted"/> </rim:name> <rim:description> <rim:localizedstring xml:lang="en" value="restricted Data Rights"/> </rim:description> <rim:classificationnode code="government" objecttype="urn:oasis:names:tc:ebxml-regrep:objecttype:registryobject:classificationnode" parent="urn:ogc:def:ebrim-classificationnode:faa:datacube:restricted" id="urn:ogc:def:ebrim-classificationnode:faa:datacube:restricted:government"> <rim:name> <rim:localizedstring xml:lang="en" value="restricted-government"/> 45

54 </rim:name> <rim:description> <rim:localizedstring xml:lang="en" value="restricted Data Rights - Government"/> </rim:description> </rim:classificationnode> Additional detail omitted for example 5.6 NETCDF-CF DATASET STRUCTURAL METADATA NetCDF is a set of software libraries and data formats that support the creation, access, and sharing of scientific data. NetCDF files are self-describing in that they contain enough information in file headers to describe the layout and content of the file, including data arrays and metadata, associated with those arrays, in the form of name/value attributes. NetCDF on its own is quite general, in the same sense that ISO is very general. The NetCDF Climate and Forecast (CF) conventions provide the necessary additional guidance to achieve broad interoperability in the climate and forecast community. The NNEW Program adheres to the NetCDF CF conventions very closely. There is little in the way of additional guidance needed. The single exception is the omission of a latitude-longitude reference grid with each individual NetCDF file. The purpose of the extra grid (in addition to the data grid) is to unambiguously map grid points to latitude, longitude locations, independent of geographic projection used. The information is however, redundant given that the mathematical information for the projections is included with each file, and the mapping grid may be created in software using standard projection equations. Given that the inclusion of the extra reference grid more than doubles the size of most NetCDF files, it is felt that this information can be safely omitted. This feedback is being provided to the CF community. Table 1, below, describes the CF-compliant metadata attributes currently used by the NNEW Program for NetCDF files containing gridded data products. The examples in the right hand column represent metadata for the Corridor Integrated Weather System (CIWS) VIL data product. Additional details regarding the NetCDF-CF convention can be found at More detailed discussions of NNEW-specific NetCDF components appear in subsections below. 46

55 TABLE 1 NNEW Program Usage of the NetCDF-4 Climate and Forecast Convention Attributes CF Attributes Description Example(s) _FillValue A value used to represent missing or -1 undefined data. add_offset If present for a variable, this number is to 0.0 be added to the data after it is read by an application axis Identifies latitude, longitude, vertical, or Z time axes. calendar Calendar used for encoding time axes. gregorian comment A global file attribute that provides miscellaneous information about the data or the methods used to produce them Mosaic of VIL derived from NEXRAD, TDWR and Canadian radar systems Conventions A global file attribute that names the CF-1.3 conventions followed by the dataset. earth_radius Used to specify the radius, in metres, of the spherical figure used to approximate the shape of the Earth false_easting The value added to all abscissa values in 0.0 the rectangular coordinates for a map projection false_northing The value added to all ordinate values in 0.0 the rectangular coordinates for a map projection grid_mapping Identifies a variable that defines a grid grid_mapping0 mapping. grid_mapping_name Contains a grid mapping's name lambert_azimuthal_equal _area history institution latitude_of_projection_origin long_name A global file attribute that lists the applications that have modified the original data. A global file attribute that describes where the original data was produced. The latitude chosen as the origin of rectangular coordinates for a map projection A descriptive name which may, for example, be used for labeling plots File created T14:47:31Z on machine compute-7-5.local by ProductAdapterVILForec ast Data produced by the MIT Lincoln Lab Weather Sensing Group 38.0 Vertically Integrated Liquid (VIL) 47

56 CF Attributes Description Example(s) longitude_of_projection_origi The longitude chosen as the origin of n rectangular coordinates for a map projection positive Direction of increasing vertical coordinate up value. references A global file attribute that describes the data or methods used to produce it. scale_factor If present for a variable, the data are to be multiplied by this factor after the data are read by an application source standard_name title A global file attribute that describes the method of production of the original data. A standard name that references a description of a variable"s content A global file attribute that describes the file contents. units Units of a variable s content. kg m-2 valid_range Smallest and largest valid values of a 0, variable. National CIWS Data Stream: StormForecast:CombineF inalforecast:forecastima ges:ciwsrelay atmosphere_cloud_liquid _water_content time altitude projection_x_coordinate projection_y_coordinate forecast_reference_time forecast_period National CIWS VIL Mosaic Product minutes Global Attributes Some of the attributes appearing in NetCDF files are global, and represent high-level metadata that applies to all data variables within the NetCDF file. Global attributes are provided to describe where the data have come from and what has been done to it. Two additional (optional) global attributes, FileType and FileFormat, are included in weather data product files to identify the file type, e.g., NetCDF, and format, e.g., Classic, Netcdf4, Offset64Bits, Netcdf4Classic. // global attributes: :Conventions = "CF-1.3" ; :history = "File created T10:07:39Z on machine compute-7-4.local by ProductAdapterVIL 1.0.0" ; :institution = "Data produced by the MIT Lincoln Lab Weather Sensing Group" ; 48

57 :references = " ; :source = "National CIWS Data Stream: MosTileAsm:Mosaic:CiwsRelay" ; :title = "National CIWS VIL Mosaic Product minutes" ; :comment = "Mosaic of VIL derived from NEXRAD, TDWR and Canadian radar systems." ; :FileType = "NetCDF" ; :FileFormat = "Netcdf4" ; All other attributes, listed in Table 1 and discussed in subsections to follow, are associated with data and/or coordinate variables in NNEW NetCDF files Dimensions and Coordinate Variables Dimensions in NetCDF files define the index space of data variables. For example, VIL(y,x) may have dimensions y=3520 and x=5120 to express a grid containing 3520 rows and 5120 columns. Coordinates are the independent variables on which the data depend. Currently, gridded NNEW NetCDF files contain data variables declared with four coordinate variables: time, altitude, projected latitude (y), and projected longitude (x). dimensions: time = 1 ; z0 = 1 ; y0 = 3520 ; x0 = 5120 ; variables: double time(time) ; time:standard_name = "time" ; time:long_name = "Product validity time" ; time:units = "seconds since T00:00:00Z" ; time:calendar = "gregorian" ; time:string = " T18:20:00Z" ; double z0(z0) ; z0:standard_name = "altitude" ; z0:long_name = "Product altitude" ; z0:units = "meters" ; z0:axis = "Z" ; z0:positive = "up" ; double y0(y0) ; y0:standard_name = "projection_y_coordinate" ; y0:long_name = "Distance from projection reference point latitude" ; y0:units = "meters" ; double x0(x0) ; x0:standard_name = "projection_x_coordinate" ; x0:long_name = "Distance from projection reference point longitude" ; x0:units = "meters" ; Time Expressions NNEW NetCDF files contain a number of expressions for time, most of which are assigned standard names that follow CF convention. These time elements are double-precision floating point values that 49

58 express the number of seconds since the UNIX Epoch ( :00:00Z). However, for the benefit of human readers, an additional attribute, named string, provides the extended format ISO-8601 expression for all variables expressing time in NetCDF files. A summary of time variables used in NetCDF files is listed here: time The coordinate variable named time expresses a product validity time as a single value or a number of values, as defined for a set of forecasts. double time(time) ; time:standard_name = "time" ; time:long_name = "Product validity time" ; time:units = "seconds since T00:00:00Z" ; time:calendar = "gregorian" ; time:string = " T18:20:00Z" ; double time(time) ; time:standard_name = "time" ; time:long_name = "Product validity time" ; time:units = "seconds since T00:00:00Z" ; time:calendar = "gregorian" ; time:string = " T18:20:00Z/ T20:15:00Z" ; start_time, stop_time The start and stop time describe the collection time interval from which the data sample was gathered. double start_time ; start_time:long_name = "Data observation start time" ; start_time:units = "seconds since T00:00:00Z" ; start_time:calendar = "gregorian" ; start_time:string = " T18:05:00Z" ; start_time:comment = "Data observation start time is the time data collection began" ; double stop_time ; stop_time:long_name = "Data observation stop time" ; stop_time:units = "seconds since T00:00:00Z" ; stop_time:calendar = "gregorian" ; stop_time:string = " T18:15:00Z" ; stop_time:comment = "Data observation stop time is the time data collection ended" ; forecast_reference_time The time of the analysis from which the forecast was made (only included in NetCDF files containing forecast products). double forecast_reference_time ; forecast_reference_time:standard_name = "forecast_reference_time" ; forecast_reference_time:long_name = "Forecast reference time" ; forecast_reference_time:units = "seconds since T00:00:00Z" ; forecast_reference_time:calendar = "gregorian" ; 50

59 forecast_reference_time:string = " T18:15:00Z" ; forecast_reference_time:comment = "Forecast reference time is the time of the analysis from which the forecast was made" ; forecast_period The time interval between the forecast reference time and the product validity time (only included in NetCDF files containing forecast products). int forecast_period(time) ; forecast_period:standard_name = "forecast_period" ; forecast_period:long_name = "Time interval between the forecast reference time and the validity time" ; forecast_period:units = "seconds" ; Figure 9 illustrates how these times relate to each other with respect to data collection, processing, and forecast. Figure 9. Time Expression diagram. Notes: [1] The stop_time may drift beyond the forecast_reference_time in systems that continue collecting data while processing existing data sets. [2] The forecast periods may be computed using: forecast_period i = time i forecast_reference_time However, the forecast_period values are included in the NetCDF files as a convenience for file clients. [3] For current (non-forecast) data sets, time 0 is identical to the forecast_reference_time and the forecast_period is zero. 51