CMDP-STATE DATABASE INTERFACE CONTROL DOCUMENT

Transcription

1 CMDP-STATE DATABASE INTERFACE CONTROL DOCUMENT Prepared for: WILL BOWMAN PRODUCT OWNER U.S. EPA OFFICE OF WATER KRISTEN GASTNER PROJECT MANAGER U.S. EPA OFFICE OF WATER Prepared by: ATTAIN 1600 TYSONS BLVD. SUITE 1400 MCLEAN, VIRGINIA Contract Number: Contract Name: Job Number:

2 Modification History Version Number Date of Revision Description of Changes Revision Entered By /02/2018 Switch A.1.8 Get Job Samples and A.1.9 Get Jobs for more logical order Update heading A.1.9 Add Section 1.10 Correct references to Content-Type\json to Content- Type\application/xml Attain, LLC Page ii

3 TABLE OF CONTENTS INTRODUCTION... 1 About This Document... 1 Background... 1 A.1.1 Intended Audience... 1 A.1.2 Acronyms and Definitions... 1 CMDP Overview... 2 A.1.3 Supporting Systems and Services... 3 CMDP WEB SERVICES... 4 CMDP Web Services Usage Overview... 4 A.1.4 Download Sample Results from CMDP System... 4 A.1.5 Load Inventory Data to CMDP System... 6 A.1.6 Load Laboratory Data to CMDP System... 8 CMDP Web Service Security... 8 A.1.7 Authentication... 8 Web Service Methods A.1.8 Get Jobs A.1.9 Get Job Samples (in edwr schema) A.1.10 Get Job Samples (in CMDP Schema) A.1.11 Update Job A.1.12 Create / Update Water System A.1.13 Retrieve Water System A.1.14 Create / Update Facility A.1.15 Retrieve Facility A.1.16 Create / Update Facility Sampling Point A.1.17 Retrieve Facility Sampling Point A.1.18 Create / Update Laboratory A.1.19 Retrieve Laboratory Page iii

4 LIST OF EXAMPLES Example 1 Authorization Request Header (for User ID usera with Password password ).. 9 LIST OF TABLES Table 1 List of Acronyms and Definitions Commonly Used throughout the Document... 1 Table 2 Authorization Request Header... 9 Table 3 Unsuccessful Authentication Scenarios Table 4 Get Job Samples (in edwr Schema) Requirements Table 5 Get Job Samples (in CMDP Schema) Requirements Table 6 Update Job Requirements Table 7 Create / Update Water System Web Service Requirements Table 8 Retrieve Water System Web Service Requirements Table 9 Create / Update Facility Web Service Requirements Table 10 Retrieve Facility Web Service Requirements Table 11 Create / Update Facility Sampling Point Web Service Requirements Table 12 Retrieve Facility Web Service Requirements Table 13 Create / Update Laboratory Web Service Requirements Table 14 Retrieve Facility Web Service Requirements LIST OF FIGURES Figure 1 CMDP Architecture and SDWIS State Web Services... 2 Figure 2 Workflow to Download Sample Results from CMDP System... 5 Figure 3 Workflow to Download Inventory Data from CMDP... 7 Figure 4 Workflow to Download Laboratory Data from CMDP... 8 Figure 5 REST API Authentication... 9 Page iv

5 INTRODUCTION ABOUT THIS DOCUMENT The Compliance Monitoring Data Portal (CMDP) Web Services State Database Interface Control Document (ICD) outlines the processes for Primacy Agencies not using Safe Drinking Water Information System (SDWIS) State to leverage CMDP State Database web services through legacy drinking water compliance systems in two capacities: Download public drinking water system (PWS) sampling data Create and update PWS inventory, contacts, and laboratory data in CMDP using the agency s existing compliance system This document includes instructions for non-sdwis State users describing how to connect their existing compliance systems to CMDP. Primacy Agencies using SDWIS State can skip this document since the DSE provides a solution to connect SDWIS State to CMDP. See DSE ICD for details on installing and using the DSE. BACKGROUND CMDP, a part of EPA s SDWIS modernization effort, will be used by laboratories to electronically report samples and results data and by water systems to inform pertinent Primacy Agencies. CMDP State Database web services allow Primacy Agencies to synchronize their legacy compliance systems with CMDP. A.1.1 Intended Audience The intended audiences of the CMDP Web Services State Database ICD are Primacy Agencies (e.g., States, Tribes, Regions, and Territories) not using SDWIS State, and their IT vendors. A.1.2 Acronyms and Definitions Table 1 List of Acronyms and Definitions Commonly Used throughout the Document Acronym EPA CMDP CROMERR ICD LIMS PWS SCS SDWIS UI Definition Environmental Protection Agency Compliance Monitoring Data Portal Cross-Media Electronic Reporting Rule Interface Control Document Laboratory Information Management System Public Water System Shared CROMERR Services Safe Drinking Water Information System User Interface Page 1

6 CMDP OVERVIEW CMDP is a web-based application and suite of web services aimed at providing the following: Electronic reporting of compliance sample results from laboratories and public water systems Certification of sample results via EPA's CROMERR Reporting of certified samples and operational data to primacies Primacy access to certified sampling data to support compliance determinations and other purposes The figure below depicts the high-level architecture of CMDP: SCS/ CROMERR Services sync LIMS CMDP CMDP/ SDWIS Database SDWIS Prime Inventory & Laboratory Sampling Results Labs PWS Primacy Agencies Figure 1 CMDP Architecture and SDWIS State Web Services The area bounded by a green dashed line shows the interaction between the primacy agencies compliance systems and the CMDP database through web services, and is the focus of this document. Basic CMDP State Database web services usage workflow (usage by Primacy Agencies): 1. Primacy Agencies enter State CMDP Administrator or State Compliance Officer SCS credentials to access the CMDP application 2. Primacy Agencies check that their inventory and laboratory information exists and is correct in CMDP a. If not, the primacies create/update inventory and laboratory information using the CMDP State Database Web Services described in this document b. Only users with State CMDP Administrator SCS Credentials can create/update inventory and laboratory profile information. 3. PWSs, laboratories, and state laboratories enter sample results data into CMDP 4. Primacy Agencies use the CMDP State Database Web Services to synchronize updated sample results data in their local existing compliance system Page 2

7 A.1.3 Supporting Systems and Services Existing Compliance Database Systems Existing systems used by Primacy Agencies to store, update, and report inventory, laboratory, and compliance data related to drinking water quality. 1. SDWIS State (SDWA schema 3.2, 3.4, etc.): legacy compliance system. Note: Legacy SDWIS State-supported systems can be leveraged with CMDP using the web services outlined in this document. SDWIS State users will implement the DSE solution, using the CMDP State Database Web Services to connect CMDP to their local SDWIS State system no further actions is needed from Primacy Agencies using SDWIS State. See DSE ICD for more information. 2. Non-SDWIS State Compliance System: A custom compliance system in place of SDWIS State. This document is focused on Primacy Agencies using non-sdwis State compliance systems. Data Synchronization Engine (DSE) Synchronizes CMDP with SDWIS State 3.2/3.3, including inventory, laboratory, and sampling data. This application is loaded in the Primacy Agency s environment on the same server as SDWIS State. Note: XML Sampling must also be loaded. See DSE ICD for more information. XML Sampling Validates and routes all the sample results (chemical and microbiological, not operational) to the appropriate SDWIS State tables. Shared CROMERR Services (SCS) A suite of four CROMERR web services that support centralized, cloud-based, state-managed CROMERR processes, including: 1. Registration and Account Management 2. Identity Management 3. Signature Device 4. Signature and Copy of Record Generation Laboratory Information Management System (LIMS) Many private laboratories, state laboratories, and water utilities with laboratories have acquired and are using LIMS. Laboratories can use their LIMS to report data to CMDP electronically using web services or data exchange templates. Page 3

8 CMDP WEB SERVICES USAGE OVERVIEW CMDP WEB SERVICES The usage for CMDP State Database web services for Non-SDWIS State compliance systems is classified using two main categories: 1. Download (pull) sampling data from the CMDP system. See the CMDP Web Services Sampling XML Schema Definitions and related documents, found in the CMDP Help Center, for more information about sample data available. 2. Update/create (push) water system inventory and laboratory information to CMDP. See the CMDP Web Service Inventory and Lab Information supporting document for more information about inventory and laboratory data accepted by CMDP. Primacy Agencies will call the web services outlined in this document to push data to and pull data from CMDP. A.1.4 Download Sample Results from CMDP System The following services are available to download certified and signed sample results and supplemental files from CMDP System: 1. Get Certified Jobs 2. Get Job Samples 3. Update Job Sample results are available by job, and a single job may include one or more samples. See the CMDP User Manual for more information on sample results and jobs in CMDP. See the CMDP Web Service Samples Information supporting document for more information about available sample data. Page 4

9 Authenticate User with SCS User Credential < Userid, Password > < Token > Authenticate() Get List of Certifed Jobs For User Organization < User id, Token Id, PrimacyAgency> < List of Certified Jobs for working Organization > GetCertifiedJobs() Get Sample Results for a Certified Job < User id, Token Id,PrimacyAgency,Job Id > Samples (Paginated) GetJobSamples() Update the Status of Job Accepted/Rejected < User id, Token Id, PrimacyAgency,Job Id, Status > < Success > UpdateJob() End Transaction Session < User id, Token Id > < Success > EndUserSession() Figure 2 Workflow to Download Sample Results from CMDP System Page 5

10 A.1.5 Load Inventory Data to CMDP System The following services are available to create and update inventory data in CMDP: 1. Create / Update Water System 2. Retrieve Water System 3. Create / Update Facility 4. Retrieve Facility 5. Create / Update Sampling Point 6. Retrieve Sampling Point See the CMDP Web Service Inventory and Lab Information supporting document for more information about inventory data accepted by CMDP. Page 6

11 SDWIS State Authenticate User With SCS User Credential < Userid, Password > < Token > CMDP Authenticate() Create WS < XML Content > < Success > CreateWaterSystem() Retrieve WS < WaterSystemId > < XML Content > RetrieveWaterSystem() Authenticate User With SCS User Credential < Userid, Password > < Token > Authenticate() Create Facility < XML Content> < Success > CreateFacility() Retrieve Facility < WSID, FacilityId > < XML Content > RetrieveFacility() Authenticate User With SCS User Credential < Userid, Password > < Token > Authenticate() Create SamplingPT < XML Content> < Success > CreateSamplingPT() Retrieve SamplingPT < WSID, FacilityID, SamplingPTId > < XML Content > RetrieveSamplingPT() Figure 3 Workflow to Download Inventory Data from CMDP Page 7

12 A.1.6 Load Laboratory Data to CMDP System The following services are available to create and update laboratory data in CMDP: 1. Create / Update Laboratory 2. Retrieve Laboratory See the CMDP Web Service Inventory and Lab Information supporting document for more information about laboratory data accepted by CMDP. SDWIS State Authenticate User With SCS User Credential < Userid, Password > < Token > CMDP Authenticate() Create LE < XML Content > < Success > CreateLE() Retrieve LE < LEId, PrimacyAgency > < XML Content > RetrieveLE() CMDP WEB SERVICE SECURITY A.1.7 Authentication Figure 4 Workflow to Download Laboratory Data from CMDP A DESCRIPTION Authentication occurs as part of a single web service request, not as a separate request. The Basic Authentication mechanism provided by Spring Security was implemented for CMDP Web Services. As part of the web service request, security credentials are passed from the client to the server via the HTTP request header that contains 'userid:password' encoded in base64. Users will obtain their credentials through SCS registration and it will be the same userid:password used to login to CMDP. For more information about SCS and interactions with CMDP, please see the SCS portion of the CMDP User Manual. In order to perform any operations using CMDP Web Services, all requests must contain the user s security credentials in the request header. Only after authentication is successful will the web request be passed to the REST API and be processed. Page 8

13 PLEASE NOTE: The security credentials (required) are part of a request, not a separate request. Figure 5 REST API Authentication Note: The following are not scenarios for a specific web service endpoint (method), but for the purposes of illustrating the responses from the authentication process that occurs as part of a request. A REQUEST Table 2 Authorization Request Header Name Required Definition / Constraints Authorization Yes SCS user ID and password encoded in base64 Example 1 Authorization Request Header (for User ID usera with Password password ) Authorization: Basic dxnlcke6cgfzc3dvcmq= userid:password encoded in base64 A RESPONSE If authentication is successful, the request will be processed and a response returned. If authentication fails, the response will raise an exception and return an error message. Page 9

14 Table 3 Unsuccessful Authentication Scenarios Scenario HTTP Response Status Code Response No security credentials provided Invalid Security Credentials (e.g., user ID not valid) Valid security credentials, but user not authorized (based on roles) 401 Unauthorized <?xml version="1.0" encoding="utf-8" standalone="yes"?> <response> <endrow>0</endrow> <errormessage>full authentication is required to access this resource</errormessage> <queuestatus>0</queuestatus> <startrow>0</startrow> <status>401</status> <totalrows>0</totalrows> </response> 401 Unauthorized <?xml version="1.0" encoding="utf-8" standalone="yes"?> <serverresponse> <response> <endrow>0</endrow> < errormessage>invalid User: usera</errormessage> <queuestatus>0</queuestatus> <startrow>0</startrow> <status>401</status> <totalrows>0</totalrows> </response> </serverresponse> 401 Unauthorized <?xml version="1.0" encoding="utf-8" standalone="yes"?> <serverresponse> <response> <endrow>0</endrow> <errormessage>requested resource ' Data' is not available for this user: stadmin. Applicable Roles for This Resource: [ROLE_WS_MODE, ROLE_LB_MODE, ROLE_LS_MODE]</errorMessage> <queuestatus>0</queuestatus> <startrow>0</startrow> <status>401</status> <totalrows>0</totalrows> </response> </serverresponse> Page 10

15 WEB SERVICE METHODS A.1.8 Get Jobs A DESCRIPTION The Get Jobs operation is for State Compliance User/System; this operation will retrieve a list of all jobs. A ARGUMENTS None. A REQUEST Pre-Production URL: Production URL: HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Get Job operation is successful, the response will return a SUCCESS message with a list of job IDs. If the Get Job Samples operation fails, the response will raise an exception and return a standard error message. A.1.9 Get Job Samples (in edwr schema) A DESCRIPTION The Get Job Samples operation is for State Compliance User/System; this operation will retrieve a list of sample results or Operational Data results with details associated with a particular job in XML format based on the edwr schema (primarily used for XML Sampling). This operation only downloads Sample or Operational Data results. There is no size limit associated with download process and each Sample or Operational Sample job is downloaded as a separate file. Page 11

16 A ARGUMENTS Table 4 Get Job Samples (in edwr Schema) Requirements Element Required Definition / Constraints jobid Yes Job ID from the list of jobs from Get Jobs A REQUEST Pre-Production Sample URL: Production Sample URL: Production Operational Data Sample URL: webservice/api/dse/jobs/opdata/{jobid} Pre-Production Operational Data Sample URL: Pre-Production Sample URL: Production Sample URL: HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Get Job Samples operation is successful, the response will return a SUCCESS message with a list of samples for the requested job. If the Get Job Samples operation fails, the response will raise an exception and return an error message. Page 12

17 A.1.10 Get Job Samples (in CMDP Schema) A DESCRIPTION The Get Job Samples operation is for State Compliance User/System; this operation will retrieve a list of Sample and Operational Data results with details associated for a particular job in XML format based on the CMDP schema. This operation only downloads Sample and Operational Data results. There is no size limit associated with download process and each Sample or Operational Sample job is downloaded as a separate file. A ARGUMENTS Table 5 Get Job Samples (in CMDP Schema) Requirements Element Required Definition / Constraints jobid Yes Job ID from the list of jobs from Get Jobs A REQUEST Pre-Production URL: Production URL: HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Get Job Samples operation is successful, the response will return a SUCCESS message with a list of samples for the requested job. Snippet of an example XML Response: <?xml version="1.0" encoding="utf-8" standalone="yes"?> <serverresponse> <response> <data> <job> <createdt> </createdt> <createid>329</createid> <fieldvalidationerrors xmlns:xsi=" Page 13

18 xmlns:xs=" xsi:type="xs:string"></fieldvalidationerrors> <updatedt> </updatedt> <updateid>329</updateid> <version>6</version> <certifiedbyid>329</certifiedbyid> <certifieddt> t16:36:42-05:00</certifieddt> <certifiedon>02/24/2018</certifiedon> <createdon>02/24/2018</createdon> <description>new Job using files</description> <documentid>4444</documentid> <filename>opdata 377 V2.xml</filename> <jobid>10810</jobid> <ownerorgcode>x1tpws002</ownerorgcode> <ownerorgtype>ws</ownerorgtype> <ownerprimacyagency>x1</ownerprimacyagency> <reviewedbyid>329</reviewedbyid> <reviewedon>02/24/2018</reviewedon> <samples>... too large to display here... </samples> <status>accepted by State</status> </job> </data> <endrow>0</endrow> <errormessage>success:null</errormessage> <queuestatus>0</queuestatus> <startrow>0</startrow> <status>0</status> <totalrows>1</totalrows> </response> </serverresponse> Please see the following supporting documentation: CMDP Web Services Sampling XML Schema Definition.docx: details related to the file structure of sample results data (found under the <samples> tag in example above). These details include attribute names, data types, field validations, and descriptions. CMDP Web Service Example Sample Data CMDPSchema document.docx: Example response XML file If the Get Job Samples operation fails, the response will raise an exception and return an error message. A.1.11 Update Job A DESCRIPTION The Update Job operation updates the CMDP job status from Submitted to Accepted once the Primacy Agency s Compliance System has accepted the new sampling data. The Accepted status indicates that the State downloaded and received all the sampling data relevant to a job. Using the DSE as an example (for SDWIS State users only), once the DSE sends sampling data Page 14

19 to XML Sampling successfully, it automatically sends a message back to CMDP using the Update Job service to update the CMDP database status to Accepted (from submitted). A ARGUMENTS Table 6 Update Job Requirements Element Required Definition / Constraints jobid Yes Job ID A REQUEST Pre-Production URL: Production URL: HTTP Method: POST Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Update Job operation is successful, the response will return a SUCCESS message. If the Update Job operation fails, the response will raise an exception and return a standard error message. A.1.12 A Create / Update Water System DESCRIPTION If the request contains a water system ID that does not exist in CMDP, a new water system profile is created in CMDP. If water system exists in CMDP, then water system profile is updated with changes in the request. The request payload will contain a string representation of the data to be posted. Page 15

20 A ARGUMENTS Table 7 Create / Update Water System Web Service Requirements Element Required Definition / Constraints String Yes JSON Content A REQUEST Pre-Production URL: Production URL: HTTP Method: POST Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): JSON string representation of data to be loaded A RESPONSE If the Create / Update Water System operation is successful, the response will return a SUCCESS message. If the Create / Update Water System operation fails, the response will raise an exception and return a standard error message. A.1.13 Retrieve Water System A DESCRIPTION Retrieves water system data from CMDP in XML format. A ARGUMENTS Table 8 Retrieve Water System Web Service Requirements Element Required Definition / Constraints watersystemid Yes Water System ID A REQUEST Pre-Production URL: Production URL: Page 16

21 HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Retrieve Water System operation is successful, the response will return a SUCCESS message with the requested water system data. If the Retrieve Water System operation fails, the response will raise an exception and return an error message. A.1.14 A Create / Update Facility DESCRIPTION If the request contains a Facility that does not exist in CMDP, a new Facility profile is created in CMDP. If Facility exists in CMDP, then Facility profile is updated with changes in the request. The request payload will contain a string representation of the data to be posted. Note: Must know Facility Category and submit appropriate schema (other, source, treatment). A ARGUMENTS Table 9 Create / Update Facility Web Service Requirements Element Required Definition / Constraints String Yes JSON Content A REQUEST Pre-Production URL: Production URL: HTTP Method: POST Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): JSON string representation of data to be loaded Page 17

22 A RESPONSE If the Create / Update Facility operation is successful, the response will return a SUCCESS message. If the Create / Update Facility operation fails, the response will raise an exception and return a standard error message. A.1.15 Retrieve Facility A DESCRIPTION Retrieves facility data from CMDP in XML format. A ARGUMENTS Table 10 Retrieve Facility Web Service Requirements Element Required Definition / Constraints watersystemid Yes Water System ID facilityname Yes State Assigned Facility ID A REQUEST Pre-Production URL: Production URL: HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): None A RESPONSE If the Retrieve Facility operation is successful, the response will return a SUCCESS message with data for the requested facility. If the Retrieve Facility operation fails, the response will raise an exception and return a standard error message. Page 18

23 A.1.16 A Create / Update Facility Sampling Point DESCRIPTION If the request contains a Facility Sampling Point that does not exist in CMDP, a new Facility Sampling Point profile is created in CMDP. If Facility Sampling Point exists in CMDP, then Facility Sampling Point profile is updated with changes in the request. The request payload will contain a string representation of the data to be posted. A ARGUMENTS Table 11 Create / Update Facility Sampling Point Web Service Requirements Element Required Definition / Constraints String Yes JSON Content A REQUEST Pre-Production URL: Production URL: HTTP Method: POST Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): JSON string representation of data to be loaded A RESPONSE If the Create / Update Facility Sampling Point operation is successful, the response will return a SUCCESS message. If the Create / Update Facility Sampling Point operation fails, the response will raise an exception and return a standard error message. A.1.17 Retrieve Facility Sampling Point A DESCRIPTION Retrieves facility sampling point data from CMDP in XML format. Page 19

24 A A DEFINITION ARGUMENTS Table 12 Retrieve Facility Web Service Requirements Element Required Definition / Constraints watersystemid Yes Water System ID facilityname Yes State Assigned Facility ID samplingpointid Yes Sampling Point ID A REQUEST Pre-Production URL: Production URL: HTTP Method: GET Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/xml Request Body (Payload): None A RESPONSE If the Retrieve Facility Sampling Point operation is successful, the response will return a SUCCESS message with requested Facility Sampling Point data. If the Retrieve Facility Sampling Point operation fails, the response will raise an exception and return a standard error message. A.1.18 A Create / Update Laboratory DESCRIPTION If the request contains a Laboratory that does not exist in CMDP, a new Laboratory profile is created in CMDP. If Laboratory exists in CMDP, then Laboratory profile is updated with changes in the request. The request payload will contain a string representation of the data to be posted. Page 20

25 A ARGUMENTS Table 13 Create / Update Laboratory Web Service Requirements Element Required Definition / Constraints String Yes JSON Content A REQUEST Pre-Production URL: Production URL: HTTP Method: POST Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/application/xml Request Body (Payload): JSON string representation of data to be loaded A RESPONSE If the Create / Update Laboratory operation is successful, the response will return a SUCCESS message. If Create / Update Laboratory operation fails, the response will raise an exception and return a standard error message. A.1.19 Retrieve Laboratory A DESCRIPTION Retrieves laboratory data from CMDP in XML format. A ARGUMENTS Table 14 Retrieve Facility Web Service Requirements Element Required Definition / Constraints labname Yes Laboratory Name A REQUEST Pre-Production URL: Production URL: HTTP Method: GET Page 21

26 Request Header: Authorization: Basic bgfiywrtaw46ihrlc3q= Content-Type: application/xml Request Body (Payload): None A RESPONSE If the Retrieve Laboratory operation is successful, the response will return a SUCCESS message with Laboratory data. If the Retrieve Laboratory operation fails, the response will raise an exception and return a standard error message. Page 22