Continuous Monitoring Data in AWQMS. A Comprehensive Review as of 8/20/2018

Transcription

1 Continuous Monitoring Data in AWQMS A Comprehensive Review as of 8/20/2018 The newer versions of AWQMS have significant enhancements both in terms of data storage and user interface; and there are more enhancements for continuous data coming in the next version of AWQMS. 1. Improved equipment meta-data tracking and added the concept of equipment limits and deployments in relation to continuous results. Equipment is under the metadata menu in AWQMS. Figure 1 My organization's continuous monitoring equipment list. Figure 2 Opening up the detection limits list for a piece of continuous monitoring equipment Figure 3 The detection limits list for a piece of continuous monitoring equipment

2 Figure 4 Opening the deployments list for a piece of continuous monitoring equipment Figure 5 The deployments list for a piece of continous monitoring equipment 2. Redesigned data structures and business objects to separate storage of continuous results from discrete results in the AWQMS database. This was critical for both data storage optimization and application speed reasons. a. Included a new, much more efficient import capability for continuous results. Figure 6 Navigating to the new import mechanism for continuous monitoring data

3 b. Example File and Import Configuration Figure 7 Example of continuous data for DO and Water Temperature (has ~4,300 results over approximately a month and a half period)

4 Figure 8 A continuous monitoring data import configuration Figure 9 Navigating to the new import mechanism for continuous monitoring data

5 Figure 10 Getting ready to import the file of continuous monitoring data Figure 11 Importing the continuous monitoring data

6 Figure 12 Import successful with no errors; migrating the data. Figure 13 Dataset migrated into the AWQMS database successfully with no errors. 3. Enhanced the "only send summary statistics for continuous data to WQX" option to provide more flexibility; a. Included addition of a new outbound web service specifically for getting AWQMS continuous monitoring data)

7

8 4. Made major revisions to the graphs, Google Maps, exports and some reports in the AWQMS data analysis tools to improve their usefulness with continuous data. Example: Single Parameter Line Graph Figure 14 Navigating to the Data Analysis Tools Figure 15 Selecting the Single Parameter Line Graph

9 Figure 16 Choosing the default search criteria for running the Single Parameter Line Graph Figure 17 Determining the dataset to analyze - Step 1: Decide which locations

10 Figure 18 Determining the dataset to analyze - select and accept the locations Figure 19 Navigating to step 2: Other Search Criteria Figure 20 Narrowing in on the desired data set for analysis

11 Figure 21 Step 3: What parameters do I have data for at that location within that time frame? Figure 22 Step 3: Choose the Parameter of Interest, select applicable criteria, generate the graph Figure 23 Graph of the continuous monitoring Water Temperature (degrees F) results

12 Figure 24 Changing the color scheme

13 Figure 25 Changing the header text

14 Figure 26 Changing the header text (continued)

15 Figure 27 You can save the image as a PDF or Other file format; and export the data to Excel

16 Figure 28 Displaying the AWQMS Graphs Help Figure 29 Displaying the AWQMS Graphs Help pop-up (continued)

17 Figure 30 Hover you mouse over a point to view additional information Figure 31 Hover your mouse over a series name in the legend to highlight the series on the graph

18 Figure 32 Click on a series name in the legend to hide/show the series in the graph (currently, Zirconia10 location is hidden) Figure 33 Scrolling the mouse wheel forward zooms in on the graph

19 Figure 34 While zoomed in you can click and drag to reposition your view of the graph Figure 35 Scrolling the mouse wheel backward zooms out on the graph

20 Figure 36 Hold down the Shift key and then click and drag with your mouse to zoom to a specific area Example: Same data with multi-parameter line graph (Single Monitoring Location)

21 Figure 37 Zoomed in on a specific area of the graph Figure 38 Navigating back to the Data Analysis Tools Page

22 Figure 39 Navigating to the Multi-Parameter Line Graph Page

23 Figure 40 Select the "Latest Search Criteria" to run this graph for the same dataset Figure 41 Adding Dissolved Oxygen to the graph and navigating to the Options tab

24 Figure 42 Adding a second scale and creating the graph Figure 43 Changed the color scheme and the header

25 Figure 44 Turning on the data grid capability

26 Figure 45 Selecting a point in a series and then viewing the surrounding data Example 3 Box and Whisker Plot

27 Figure 46 Navigating to the Box and Whiskers Plot in the Data Analysis Tools page Figure 47 Selecting "Latest Search Criteria" to run the analysis on the same data set

28 Figure 48 Selecting the parameter to graph and navigating to the Options tab Figure 49 Generating one box plot per month

29 Figure 50 The monthly box plot for Water Temperature (F) (Mouse hovered over the May, 2017 series) 5. Easy Batch Delete of Continuous Data (so you can re-import it without duplicates). Figure 51 Navigating to the Continuous Results Batch Deletion page

30 Figure 52 Deleting a batch of continuous monitoring data for a given date range and monitoring location Figure 53 Deleting our continuous result data set

31 Figure 54 The continuous data was successfully deleted We are also currently working on: 1. An enhancement to allow for improved integrated review and QC of continuous data during the import process (or even after the fact). The plan is to include an ability to adjust values for drift and capture rationale behind the adjustment. See Appendix A for more information about what is envisioned. 2. AWQMS Sensor Observation (web) Services (SOS) to allow integration of AWQMS as a "Data Appliance" in the EPA Interoperable Watersheds Network (IWN), if desired. We are mostly done with these but are debugging them. We are awaiting some troubleshooting assistance from EPA personnel. See Appendix B for more information.

32 Appendix A AWQMS Integration with EPA s Interoperable Watersheds Network (IWN) AWQMS-Cloud environment will be registered in the EPA central catalog and sensor discovery tool. AWQMS will implement web services and become a Data Appliance

33

34

35

36 Appendix B An Improved Process for Continuous Monitoring Data An Improved Process for Continuous Monitoring Data AWQMS users can currently bring continuous data into the system in at least two ways: manually triggered batch file imports of continuous sample data automatic batch file imports of continuous sample data retrieved periodically from a logger over the Internet (in the case of Bishop Paiute). Manually Triggered Batch Import Allowed values for Result Status are: Rejected Reported result has not been accepted Not released to public Preliminary Internal use only Not released to public Unreviewed Reported result has not been reviewed Is available to the public Provisional Reported result has not been verified, not undergone all Is available to the public QC procedures. Validated Reported result has been verified and reviewed Is available to the public Accepted Reported result has been accepted Is available to the public Final Publishable, released to public Is available to the public Unreviewed was added specifically as a way to be able to have continuous data be both preliminary but also publicly visible. The assumption is that, for CMD batch-imported data, the result status is currently brought into the system as either Preliminary or Unreviewed and would eventually be changed to some other result status such as Final by either reviewing/updating the result values one by one (which is never going to happen) or by reimporting a modified copy of the original dataset. There are obvious flaws and inefficiencies with this approach that we ll handle below. Data is currently sanity checked as it is imported into the system. First, the parameter for each result value imported into the system is compared against the user-specified Expected Parameter list in the organization s QC Thresholds list. Note that, in AWQMS, a parameter is more than just the characteristic name; it s the combination of the characteristic name, method speciation, sample fraction, and measurement unit. If an unexpected parameter (e.g. degrees F when we expect degrees C) is imported into the system a warning is generated that the user can review during the import process. Although a warning is generated, the data point is still allowed into the system. Next, the result values are checked as to whether they exceed user-provided minimum and/or maximum reasonable values for the given parameter (QC Threshold(s)). For each exceedance, a warning is generated. What other sanity/quality checks could/should the system be able to perform? Dramatic swing from last N data points? (Maybe in the future need to re-watch the Aq. video) Would there be any point in providing notifications for errors/warnings during a manually triggered batch file import? (Probably not, as they can see the errors in the data set summary page). For either type of sanity check, the system can be configured to error; rather than warn. In the case of errors, the data point would only be allowed into the system if it's corrected to be within tolerances; or if the status is changed to 'Rejected'. Do they want to change their settings to error, rather than only to warn? Not at this time

37 There is a user preference available to allow automatic migration of imported data that has no errors or warnings. However, that isn t quite what is needed. Ideally, the system would automatically migrate all of the values into the system and then: set the status of the values that passed both sanity checks as Provisional (or maybe there could be a perparameter preference in the QC thresholds page [E.g. ph the last thing we want to show the public is that our ph meter is reading 15]); Ask: couldn t this be handled by setting the QC threshold tighter? (as opposed to a parameter by parameter )? set the status of any values that didn t pass either or both sanity checks as Preliminary (or maybe there could be a preference); flag the values that didn t pass either or both sanity checks with warnings for whichever checks they didn t pass. Ideally, the user would be able to see that they have errors/warnings through multiple convenient methods: During the import / migrate process; similarly to how the user can immediately resolve domain value or other such errors. After the fact. For example, they kick off the import/migrate and then shut down their computer and leave on vacation for a week. I could envision the user either going back to the dataset summary page for the specific import/migrate or going to a new My Notifications page to see what is on their hot list. This last part might come in later as part of integrating in the auto-polling. Also, the envisioned My Notifications page. Maybe that is the same thing as a filtered version of the existing Event Log; but it would have links to the migrated data. The user could review the data for a parameter at a site over a date range and be able to easily see from the resulting data grid which values are still flagged as needing review.

38 Regardless of how the user navigates to a given error or warning notification, the user would be able to see the details of the data point and which specific sanity checks the value didn t pass. Additionally, the user would be able to see the data point in context of the surrounding data points for the same parameter (on a graph and in a grid of data): They should be able to compare the parameter series against another co-sampled parameter (so they can see if other parameters are also off, or maybe if, say, the temperature affected the DO). Ideally, be able to plot a best fit trend line for the primary series. Future vision thing

39

40 The user would be able to perform the following actions on the data point. Unflag it as needing attention Change the result status (e.g. Rejected) Provide a review comment. A comment would only be needed if they are changing the status to Rejected or Preliminary. Provide or calculate an adjusted value (e.g. to account for drift). [Ryan, consider a status history child table with a comment field]

41 Ideally, the user would be able to perform those actions on multiple user-selected data points at a time; so they don t have to do the same thing point by point. Created by and last changed by (and date) are already captured. However, it doesn t currently capture who changed what when. Is that needed? Automatically Retrieved and Imported Periodic Logger Files (Only one pilot tribe in R 9 thus far) The result status is currently brought into the system as Unreviewed. Currently, if an unexpected parameter (any) error occurs, the value is not migrated into AWQMS but the value is included in a rejection file that can be reviewed, fixed, and reimported manually later. If a QC exceedance occurs, the data point is migrated into the AWQMS database with a status of Unreviewed and an notification is sent out to up to 3 recipients. Ideally, the values that exceed the QC threshold(s) for their parameters would be flagged as Needs Review and could be found and handled either through a My Notifications hot list, or through reviewing the data for a given site/parameter as described above. (and set the status to whatever the org decides in their preferences (e.g.

42 preliminary ). They say this is sometimes a parameter by parameter decision. Might need to be in the QC thresholds list. - What about missing data? (Currently no way to handle within the system). This is not an issue for this discussion. Not something they say they ever do. Adjustment of Continuous Data Based on Pre/Post Calibration Value Differences (Drift Correction) One tribe says they re-calibrate their dedicated continuous probes every 4 to 6 weeks. They take a pre-cleaning reading, they then do a field cleaning of the dedicated field sensor/probes. They then take a near-simultaneous reading with both a clean, calibrated probe/sensor and the dedicated, field-cleaned probe/sensor. That determines the amount of bio-fouling drift since last calibration. Then they bring both sondes w/sensors back to the lab and they calibrate the dedicated sensors. Difference between the initial reading and the second reading is the bio-fouling drift. Time Drift Since Last Calibration = Difference between the standard solution value and the actual reading in the solution at the lab (prior to pushing the calibrate button). Cumulative Drift = Time Drift since last calibration + Bio-Fouling Drift. One tribe says they currently don t load their calibration data into AWQMS. Ideally, they could load/enter their data into AWQMS and then AWQMS would automatically calculate and apply the needed drift correction. This might necessitate changes to allow for tagging of discrete values as calibration data (maybe using activity type.) There might be a need to be able log calibration events and then the associated readings. At a minimum, though, they need to be able to enter drift corrections from calibration point to calibration point. - Are there other types of drift we should be thinking about? E.g a comparison of voltage reading VS what we know the reading should be I think that was in the Aquarius video. Right now this is the only type of drift we are thinking about.

43