Juvenile Migration Data Exchange Implementation

Transcription

1 Northwest Indian Fisheries Commission Washington Department of Fish and Wildlife Juvenile Migration Data Exchange Implementation Flow Configuration Document Version 1.4 February 28, 2012 Prepared For: Northwest Indian Fisheries Commission 6730 Martin Way East Olympia, Washington Prepared By: 4386 SW Macadam Ave. Suite 101 Portland, OR (503)

2 THIS PAGE INTENTIONALLY LEFT BLANK

3 Version Control Date Author Changes Version 4/1/2011 Windsor Solutions Initial version /20/2011 Windsor Solutions Finalized document 1.0 2/28/2012 Windsor Solutions Adjusted the Submission Information section to describe the structure the XML submission file now that the schema includes binary objects for reference documents. Added information to the Submission Processing and Feedback section to describe the execution log / validation errors reports that are returned from a receiving Node. 1.4

4 THIS PAGE INTENTIONALLY LEFT BLANK

5 Contents ACKNOWLEDGEMENTS... 1 INTRODUCTION... 3 FLOW SUMMARY INFORMATION... 4 SCHEMA INFORMATION... 5 SUBMISSION INFORMATION... 7 DATA SERVICE INFORMATION... 9 EXCHANGE NETWORK HEADER INFORMATION PREPARED BY WINDSOR SOLUTIONS, INC PAGE: V

6 THIS PAGE INTENTIONALLY LEFT BLANK PAGE: VI PREPARED BY WINDSOR SOLUTIONS, INC

7 Acknowledgements The document was developed with input and support from following individuals: Northwest Indian Fisheries Commission (NWIFC) Bruce Jones Ron McFarlane Osa Odum Washington Department of Fish and Wildlife (WDFW) Brodie Cox Matt Klungle Mick Morbitzer Max Pham Mara Zimmerman PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 1

8 THIS PAGE INTENTIONALLY LEFT BLANK PAGE: 2 PREPARED BY WINDSOR SOLUTIONS, INC

9 Introduction Background Under the sponsorship of the Puget Sound Partnership (PSP), representatives from the Northwest Indian Fisheries Commission (NWIFC), Tribes; State and Federal fish & wildlife agencies; water quality and quantity agencies; land managers, and agencies providing restoration and protection action grants have worked together to design and establish the Puget Sound Data Exchange (PSDE) as well as the Juvenile Migration Data Exchange (JMX). Using the Exchange Network (EN), participants are collaborating to standardize, integrate and exchange information as part of their shared efforts to monitor and restore the health of the Puget Sound watershed. The EN provides tools and approaches for data to be shared efficiently and securely over the Internet, providing real-time access to higher-quality data while saving time and resources for partners. The NWIFC and the Washington Department of Fish and Wildlife (WDFW) are the main collectors of this juvenile migrant salmon information and will be providing access to this information to inform decisions and activities designed to restore and protect the Puget Sound watershed. The Juvenile Migrant Data Flow Development Workgroup (JMWG) was formed to enable the sharing, interpretation, and integration of juvenile migrant salmon data from multiple sources. This JMX exchange is intended to focus on the use of the Exchange Network to collect and integrate Washington State s juvenile migrant salmon information. The NWIFC and WDFW aggregate and share JMX data amongst participating Tribal Nations and share this with PSP and the National Oceanic and Atmospheric Administration s National Marine Fisheries Service (NMFS). The data is exchanged via the JMX Data Exchange through the National Environmental Information Exchange Network (NEIEN). The purpose of the JMX Data Exchange is to: Improve sharing of juvenile migrant salmon information, using the EN, between WDFW, NWIFC, participating Tribal Nations, the National Oceanic and Atmospheric Administration s National Marine Fisheries Service (NMFS) and the PSP. Improve data management of juvenile migrant salmon information within and among WDFW, NWIFC, Tribal Nations, NMFS, and PSP. Provide the tools to exchange juvenile migrant salmon information among participating organizations. How to use this FCD This Flow Configuration Document (FCD) describes how Exchange Network (EN) partners shall make available and exchange JMX data. This document applies to JMX data providers and data consumers. PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 3

10 Flow Summary Information Flow Identification Flow Name: Juvenile Migration (JMX) Data Exchange Flow Description: Exchange of juvenile migrant salmon monitoring events and abundance estimates. Flow Steward: Northwest Indian Fisheries Commission (NWIFC) 6730 Martin Way East Olympia, Washington Main: Flow Contact: Bruce Jones Data Flow Overview The JMX data exchange supports these central goals: 1. Provide a mechanism for the Washington State partners to provide juvenile migrant monitoring event and abundance estimate information to a central location (at NWIFC and/or WDFW) where data will be warehoused for aggregation and tabulation. The provision of data is performed as data providers believe it is needed and as defined in their Trading Partner Agreement (TPA). The submission format is outlined in the Submission Information section of this document. 2. Provide a mechanism for NWIFC and WDFW to synchronize received data, on a regular basis. The submission format is outlined in the Submission Information section of this document. 3. Provide a mechanism for Washington State partners to query juvenile migrant monitoring event and abundance estimate information from the centralized (at NWIFC and/or WDFW) juvenile migrant data warehouse. 4. Provide a mechanism for authorized EN users to query abundance estimate summary information from the centralized (at NWIFC and/or WDFW) juvenile migrant data warehouse, on an ad hoc basis. The query services are outlined in the Data Service Information section of this document. This document describes the configuration of the JMX data exchange to meet these goals. Flow Access and Security The JMX flow implements standard NAAS security. For Query requests, each flow implementer may choose the appropriate level of security to apply for the needs of their agency. For submission of JMX data to the NWIFC and WDFW Nodes, Submit privileges will be granted only to authorized partner agencies. PAGE: 4 PREPARED BY WINDSOR SOLUTIONS, INC

11 Schema Information Schema Structure The JMX schema describes the organization, monitoring event details, abundance estimate details, abundance estimate summary and protocols relating to juvenile migrant activities. The following diagram illustrates the major components of the JMX schema. Double blocks represent components that may be repeated. Dotted lines represent optional components. JMX Organization Facility Site All Detail Details Monitoring Event Detail Monitoring Event Measurement Fish Summary Fish Identifier Fish Condition Abundance Estimate Detail Strata Precision Abundance Estimate Summary Precision Protocol Component Name Legend Component Method Required Relationship Optional Relationship Diagram of the JMX v1.0 Schema Components PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 5

12 Schema Components The following list describes the major data blocks within the JMX XML schema. Organization Describes the organization that undertakes juvenile migrant salmon data collection or abundance estimation activities. The organization component describes the organization JMX ID, organization name, phone number and web address. Monitoring Event Detail Describes the details of a specific juvenile migrant catch or release monitoring events undertaken by an organization. The Monitoring Event Details component describes the timing and location of the event, the details of the trap equipment, any measurements captured during the event, the details of the fish processed in the event (e.g., life stage, species, identifiers, conditions, sizes, etc.) and the protocol used to define the manner in which monitoring events will be conducted. Abundance Estimate Detail Describes the details of a juvenile migrant abundance estimate which has been derived by the organization. The Abundance Estimate Details component describes the year the estimate applies, the life stage and species of the fish, the location the estimate relates to, the estimates for each strata (i.e., estimate for a specific period), the precision of the strata estimate, and the protocol used to define the manner in which abundance estimates will be calculated. Abundance Estimate Summary Describes a summary of a juvenile migrant abundance estimate which has been derived by the organization. The Abundance Estimate Summary component describes the year the estimate applies, the life stage and species of the fish, the location the estimate relates to, and the precision of the estimate. Protocol Describes a protocol used to define the manner in which a monitoring event will be conducted or an abundance estimates will be calculated. The Protocol component describes a collection of methods for a particular protocol. PAGE: 6 PREPARED BY WINDSOR SOLUTIONS, INC

13 Submission Information The JMX exchange is used as a mechanism to submit juvenile migration data from an authorized partner to the NWIFC/WDFW Node. The frequency of data submission will be determined in a Trading Partner Agreement (TPA) between each data provider and the NWIFC/WDFW. Submission Structure Submissions shall utilize the Network Submit primitive method. The submit operation s dataflow element must be set to JMX_v1. The submission may be in either XML or ZIP format and must contain a single JMX document that conforms to the JMX XML schema. A submission can also contain zipped binary objects (such as protocol reference documents or images), in which case the entire submission must be in ZIP format. In this case the binary objects are referenced by name in the JMX document and are provided as individual documents wrapped inside the ZIP submission file at the root level. In order to assemble all related documents for a submission, each JMX submission is exchanged as in a single compressed file in ZIP format. This also improves transmission efficiency due to the significantly smaller size of compressed files. The following diagram describes the structure of a submission ZIP file for JMX: Submission File (.ZIP) JMX Document (.XML) JMX Data Attached Binary Object 1 (.PDF for example) Attached Binary Object 2 (.DOC for example) Submission Data Processing Identifying Organization NWIFC/WDFW will prescribe a unique identifier to each submitting organization, ensuring that each submission to NWIFC/WDFW only updates the submitter s JMX data. This element must be set to the identifier assigned by NWIFC/WDFW for the submitting agency. The identifier must be included in the OrganizationDetails component as the OrganizationIdentifier element. There is a certain known fallibility to this approach. It is possible that a submitter could overwrite another agency s JMX data if another agencies OrganizationIdentifier is supplied, but since the users must be authorized to submit on behalf of particular OrganizationIdentifier at the NWIFC/WDFW Nodes, and since the Submit operation of the JMX exchange is only available to authorized NAAS accounts, this risk is perceived to be minimal. PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 7

14 Data Processing The first component in each of the XML schemas for JMX is the Organization Details. The unique Organization Identifier included in this component for an organization and must be assigned by the NWIFC/WDFW before that organization can submit data to the NWIFC/WDFW Nodes. Before processing any submission file, the NWIFC/WDFW Nodes will first validate that the Organization Identifier in the JMX file is valid and that the User used to authenticate and submit the JMX file to NWIFC/WDFW Node has been authorized to modify data for that organization. The following section will describe how data will be aggregated with the existing dataset. Full Replace Transactions Once the user and organization have been validated, the NWIFC/WDFW Nodes will proceed to aggregate the data provided in the submission with the existing dataset. JMX data received from a partner will be aggregated following a full replace process. Each submission will include one or more organizations. If an organization already exists at the receiving partner, it will be updated with the received properties. If an organization does exist at the receiving partner, it will be created. All components below the organization (e.g., Monitoring Event Details, Abundance Estimate Details and Protocol) will be deleted and replaced, if they are found to exist. For example, if a monitoring event is received which already exists, the monitoring event and all its related child data will be deleted and replaced with the received data. If a monitoring event is received which doesn t yet exist, a monitoring event will be created with the received data. The NWIFC/WDFW Nodes will automatically determine if a component already exists and where it requires replacement. The following procedure is used to make this determination: the unique identifier for each Monitoring Event, Abundance Estimate and Protocol in the payload is compared with the identifiers for these respective items in the JMX database. If the identifier is found, the Replace is performed where the existing record (and its related data) will be deleted and the record will be replaced with the record in the database with the copy from the payload. If it is not found, an Insert is performed, creating a new record in the database. Accurately updating the JMX data at NWIFC/WDFW is only possible if you have uniquely identified each of the primary components in your submission file. When a Monitoring Event and/or Abundance Estimate is inserted or replaced in the database, the Last Changed Date for that respective component will be updated with the current date and time value. Submission Processing and Feedback After a submission has been performed, the partner agency may call the GetStatus operation on the receiving Node (NWIFC/WDFW) to obtain the processing status of the submission. The Node will return one of the following three results: Received Indicates that the file has been received but not yet been processed. Processing Indicates that the file is currently being processed by the NWIFC/WDFW Node. Completed Indicates that the file has been successfully processed by the NWIFC/WDFW Node. Failed Indicates that an error was encountered during the file processing. Contact the NWIFC/WDFW Node administrator to perform troubleshooting steps. Once a submission reports a status of Completed or Failed at the receiving Node, the Download method can be called to the retrieve the Execution Log file (if submission completed) or Validation Errors file (if failure occurredxml validation failed). PAGE: 8 PREPARED BY WINDSOR SOLUTIONS, INC

15 Data Service Information Three data services are defined for the JMX flow. These data services are optional and will be made available at the discretion of the data provider. The first service, JMX.GetMonitoringEventDetails_v1, returns a detailed set of catch and/or release monitoring events and related protocols that match a set of given criteria. The second service, JMX.GetAbundanceEstimateDetails_v1, returns a detailed set of abundance estimates and related protocols that match a set of given criteria. The third service JMX.GetAbundanceEstimateSummary_v1, returns a summary of abundance estimates that match a set of given criteria. Data Services Service Name JMX.GetMonitoringEventDetails_v1 JMX.GetAbundanceEstimateDetails_v1 JMX.GetAbundanceEstimateAndMonitoringEventDetails_v1 JMX.GetAbundanceEstimateSummary_v1 Service Type Query/Solicit Query/Solicit Query/Solicit Query/Solicit JMX.GetMonitoringEventDetails_v1 Data Service Type: Query, Solicit Data Service Parameters, Order, and Format: OrganizationIdentifier Position 1 xsd:string 30 characters Treat search string as equal. No wildcard character supported. AreaNumber Position 2 xsd:integer N/A Treat search string as equal. No wildcard character supported. Waterbody Position 3 xsd:string PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 9

16 Position 4 Position 5 Position 6 Position characters Treat search string as equal. No wildcard character supported. Species xsd:string 25 characters Treat search string as equal. No wildcard character supported. StartDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as greater than or equal to the monitoring event start date provided. EndDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as less than or equal to the monitoring event end date provided. DataChangedAfterDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as greater than or equal to the monitoring event last changed date provided. Payload Format: This service shall return an XML file conforming to the JMX_JMX_v1.0.xsd schema. The result file shall contain a root Organization element, zero or more MonitoringEventDetails elements and zero or more Protocol elements which are associated to the provided MonitoringEventDetails for each of the MonitoringEventDetails that match the search criteria provided. The service shall not return any AbundanceEstimateDetails or AbundanceEstimateSummary data. PAGE: 10 PREPARED BY WINDSOR SOLUTIONS, INC

17 Data Service-level Business Rules: At least one parameter must be provided. XML Header Usage: The EN Header is not implemented for JMX Query service. JMX.GetAbundanceEstimateDetails_v1 Data Service Type: Query, Solicit Data Service Parameters, Order, and Format: OrganizationIdentifier Position 1 xsd:string 30 characters Treat search string as equal. No wildcard character supported. AreaNumber Position 2 xsd:integer N/A Treat search string as equal. No wildcard character supported. Waterbody Position 3 xsd:string 120 characters Treat search string as equal. No wildcard character supported. Species Position 4 xsd:string 25 characters Treat search string as equal. No wildcard character supported. StartYear Position 5 xsd:integer PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 11

18 4 Position 6 4 Position 7 If provided, treat as greater than or equal to the abundance estimate start year provided. EndYear xsd:integer If provided, treat as less than or equal to the abundance estimate end year provided. DataChangedAfterDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as greater than or equal to the monitoring event last changed date provided. Payload Format: This service shall return an XML file conforming to the JMX_JMX_v1.0.xsd schema. The result file shall contain a root Organization element, zero or more AbundanceEstimateDetails elements and zero or more Protocol elements which are associated to the provided AbundanceEstimateDetails for each of the AbundanceEstimateDetails that match the search criteria provided. The service shall not return any MonitoringEventDetails or AbundanceEstimateSummary data. Data Service-level Business Rules: At least one parameter must be provided. XML Header Usage: The EN Header is not implemented for JMX Query service. JMX.GetAbundanceEstimateAndMonitoringEventDetails_v1 Data Service Type: Query, Solicit Data Service Parameters, Order, and Format: Position 1 OrganizationIdentifier xsd:string 30 characters Treat search string as equal. No wildcard character supported. PAGE: 12 PREPARED BY WINDSOR SOLUTIONS, INC

19 Position 2 Position 3 Position 4 Position 5 Position 6 AreaNumber xsd:integer N/A Treat search string as equal. No wildcard character supported. Waterbody xsd:string 120 characters Treat search string as equal. No wildcard character supported. Species xsd:string 25 characters Treat search string as equal. No wildcard character supported. StartDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as greater than or equal to the monitoring event start date or abundance estimate year provided. Only the year is used for abundance estimates. EndDate xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as less than or equal to the monitoring event end date or abundance estimate year provided. Only the year is used for abundance estimates. DataChangedAfterDate PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 13

20 Position 7 xsd:string conforming to MM/DD/YYYY format N/A If provided, treat as greater than or equal to the monitoring event or abundance estimate last changed date provided. Payload Format: This service shall return an XML file conforming to the JMX_JMX_v1.0.xsd schema. The result file shall contain a root Organization element, zero or more MonitoringEventDetails elements, zero or more AbundanceEstimateDetails elements, and zero or more Protocol elements which are associated with the MonitoringEventDetails or AbundanceEstimateDetails that match the search criteria provided. The service shall not return any AbundanceEstimateSummary data. Data Service-level Business Rules: At least one parameter must be provided. XML Header Usage: The EN Header is not implemented for JMX Query service. JMX.GetAbundanceEstimateSummary_v1 Data Service Type: Query, Solicit Data Service Parameters, Order, and Format: OrganizationIdentifier Position 1 xsd:string 30 characters Treat search string as equal. No wildcard character supported. AreaNumber Position 2 xsd:integer N/A Treat search string as equal. No wildcard character supported. Waterbody Position 3 xsd:string 120 characters PAGE: 14 PREPARED BY WINDSOR SOLUTIONS, INC

21 Position 4 Position 5 4 Position 6 4 Treat search string as equal. No wildcard character supported. Species xsd:string 25 characters Treat search string as equal. No wildcard character supported. StartYear xsd:integer If provided, treat as greater than or equal to the abundance estimate start year provided. EndYear xsd:integer If provided, treat as less than or equal to the abundance estimate end year provided. Payload Format: This service shall return an XML file conforming to the JMX_JMX_v1.0.xsd schema. The result file shall contain a root Organization element, zero or more AbundanceEstimateSummary elements and zero or more Protocol elements which are associated to the provided AbundanceEstimateSummary for each of the AbundanceEstimateSummary that match the search criteria provided. The service shall not return any MonitoringEventDetails or AbundanceEstimateDetails data. Data Service-level Business Rules: At least one parameter must be provided. XML Header Usage: The EN Header is not implemented for JMX Query service. PREPARED BY WINDSOR SOLUTIONS, INC PAGE: 15

22 Exchange Network Header Information An exchange header will not be required or used for the JMX exchange. PAGE: 16 PREPARED BY WINDSOR SOLUTIONS, INC