Climate Data Screener and Summarizer, version 3.2

Transcription

1 Climate Data Screener and Summarizer, version 3.2 Instruction Manual May 2011 Program Written By: Michael T. Tercek, Ph.D. WALKING SHADOW ECOLOGY PO Box 1085 Gardiner, Montana USA Conceptual Advice and Software Testing By: Stephen T. Gray, Ph.D. Department of Civil and Architectural Engineering Water Resources Data Systems Wyoming State Climate Office University of Wyoming, Laramie, Wyoming USA 1

2 TABLE OF CONTENTS INTRODUCTION 3 SETTING UP THE PROGRAM 5 DOWNLOADING CLIMATE DATA AND MAKING SURE IT IS READY 6 SETTING UP A FILE THAT CONTAINS 30-YEAR AVERAGES 18 QA/QC OF DATA 22 ANALYZING THE DATA 26 WHERE DO GRAPH TITLES COME FROM? 32 LITERATURE CITED 33 2

3 1. Introduction The Climate Data Screener and Summarizer is a stand-alone, GUI-driven analysis package designed to facility the quality-checking / quality assurance (QC/QA) and summary of climate data. Version 3.0 reads data from the three types of climate stations: the Cooperative Observer Program (COOP) administered by the National Weather Service, the Snowpack Telemetry (SNOTEL) network run by the Natural Resources Conservation Service, and stream gauges operated by the United States Geological Survey. Nothing needs to be installed on the user s computer. You need only Windows XP, Windows Vista, or Windows 7. There are 5 different summary tables and 18 graph types available to the user, as well as a small set of file handling tools that facilitate the presentation and pre-processing of data files. Most of the program outputs focus on the calculation of monthly totals / averages and their departure from 30-year averages. However, rather than blindly calculating these summary statistics, the software uses a set of decision rules to determine whether there is enough data of sufficient quality to justify such a calculation. Months that do not pass quality standards are reported as missing. In addition to these basic quality checks, this software provides a series of diagnostic plots that allow the user to screen the data for outliers and logical inconsistencies. For COOP data, the program provides a list of dates that require closer examination before analysis should proceed. In many cases, the user will want to consult the original, hand-written station records (B-91 forms) to verify that the data points identified by the program are valid. The authors wish to emphasize the importance of the GIGO principle. GIGO stands for Garbage In = Garbage Out. In other words, if you feed bad data into this program, you will get useless results. Any tool can be used for both good and evil. We hope that this tool will be used properly, as a force for good. A lot of effort has been devoted to error checking the calculations in this program, and in some cases, the user is warned against common data handling mistakes. But each individual user must accept personal responsibility for the decisions that they make while using this program. In particular, we wish to stress the importance of completing the QA/QC steps for COOP data before producing summary tables or graphs. Failure to do so will produce bad results. In any type of data analysis, it is critically important to understand what you are doing with the data, even when you use automated tools. We wish to thank and acknowledge the open source software community for their development of the Python programming language ( In addition, we would like to thank the authors of the following open source Python add-on packages: Matplotlib (Hunter 2007; Numpy, and Scipy ( This software would have been impossible without the amazing programming tools that numerous open source programmers have developed. We welcome suggestions and constructive criticism. Would you like to see a new feature in the program? Maybe you are interested in analyzing other climate parameters like snowpack? Do you need the capability to read other data types? Have you found a bug? Send to: Information@YellowstoneEcology.com 3

4 Funding for this work was provided in part by the US National Park Service (NPS) Inventory and Monitoring Program. We thank the following NPS staff, who have provided helpful suggestions during various phases of the project: Isabel Ashton, Stacey Ostermann-Kelm, Rob Daley, and Cathie Jean. This software is free and may be distributed to anyone. The authorship notice may not be removed and the program may not be modified before distribution. If the program is modified, the authors accept no responsibility for any errors that may be introduced. 4

5 2. Setting Up The Program Unzip the file that you downloaded and double-click on the automatic installer that is inside: Install_Climate_Data_Summarizer_3_1 Details: Make sure you unzip the archive file and copy the resulting installer to a new location on your hard drive. NOTE: WHEN THE INSTALLER RUNS, YOU SHOULD PROBABLY LET IT USE DEFAULT SETTINGS. DO NOT CHANGE THE DESTINATION DIRECTORY. IF YOU FORCE THE PROGRAM TO INSTALL IN A FOLDER THAT RESTRICTS WRITE PRIVILEGES, SUCH AS THE Program Files FOLDER, THE PROGRAM WILL NOT WORK. This is because some versions of windows (particularly Windows Vista) do not allow files to be created or edited in the program files folder for security reasons. Since the Climate Data Summarizer spools data to the hard drive while it makes calculations, the program will not run properly in a folder that has security restrictions. This program probably will not run remotely over a network. You should have a copy of the program folder on the computer that you are personally using to do the analysis. It is OK to have the data files somewhere else on the network. The file paths that you specify will track across the network, but the graphical user interface generated by the program will appear only on the computer that contains the executable file. On some older computers, when you run the program you may receive an error message that states The installation information is incorrect. This is due to your computer missing some proprietary MS windows library files. In order to install these files, visit the following web link, download the Visual C++ drivers, and install them: For 32 bit versions of windows, such as windows XP, visit this link: In the unlikely event that you encounter this error on a 64 bit version of windows, visit the following link and follow the instructions: 4B40-8C14-157CFDFFEE4E&displaylang=en Once you have installed these drivers, you may need to re-install the Climate Data Summarizer. 5

6 3. Downloading Climate Data and Making Sure That It Is Ready The web links provided here were accurate at the time of writing. Keep in mind that these are just guidelines. There are other ways to obtain climate data. The goal is to download files that are formatted in the same way as the example data files that shipped with this program. Even if the location and organization of the web pages change in the future, you can still analyze your data by obtaining files from the relevant agencies and saving them unedited on your hard drive. In the case of SNOTEL files, you may want to merge files from multiple water years into a single file. The Climate Data Summarizer provides a tool to do this automatically. See below for details. A. Cooperative Observer Network (COOP) Weather Data Go to the following web link: At the time of writing, the page looked like this: 6

7 Pick a state. Click on List Stations. On the page that appears, click on the link for your weather station of interest. You should get a page like this: 7

8 Click on the link for DATA indicated by the red arrow above. You should get a page like this: 8

9 Click on the link for Digital ASCII Files indicated by the red arrow above. You should get a page like this: 9

10 Click on the link for Daily Surface Data indicated by the red arrow above. You should get a page like this: 10

11 Click on the link for Continue with Advanced Options indicated by the red arrow above. You should get a page like this: 11

12 This step is important. You must do all of the following: 1. You MUST choose Delimited Station Names (top right arrow). 2. Select comma with data flag (bottom right arrow). 3. Choose the type of data that you want in the file. To select more than one data type, hold down the control key while you click on the elements (left red arrow in the picture above). Select PRCP, TMAX, and TMIN. It doesn t matter if you get other data types in there too. The Climate Data Summarizer will ignore them during analysis. 4. Choose the dates for which you want the data (top right of the screen). 5. When you have all these choices in place, click Continue. 12

13 Access to this type of data is free if you have a university or government internet connection. Otherwise, you will be charged a fee for each data file. You will be asked for your address and sent a link that allows you to download the data. When the data appears you have two options: (A) On the File menu of your browser, select "Save Page As" -- If you are asked to specify a file type, save it as a text file Wait!! Your browser doesn t have a File menu? Well, if use Internet Explorer, you can make a traditional File menu appear by going to the Tools dropdown box at the top of the screen. Then select Toolbars and check Menu Bar. If you can t find a Tools drop-down box, you can also right-click near the top of your browser window, next to one of the browser tabs. This should give you a list of options that can be either selected or un-selected. Check the menu bar option. (B) Cut and paste the data from your browser into a text editor or word processor and save the resulting file in text format. Be sure to save the files in text format and not as web pages (mht or html files). ** You do not need to make any changes to the data file before analysis. You should, however, take a look at the data and make sure that has the example files provided with this program. In particular, make sure that the arrangement of columns is the same. Is the station name in the first column? Is the station ID number in the second column? All of the columns must be in exactly the order shown in the example data file. Also make sure that you have comma separated values (csv) rather than tab delimited data. If you followed the instructions above, your file should be formatted correctly. B. USGS Stream Data for the Current Water Year Note that the graphs generated by the Climate Data Summarizer are designed to depict a water year, which runs from September 30 - October 1. Keep this in mind when you specify the start and end dates of your data files. Method # 1 1. Go to the web address: 2. Click on the map to get to your stream gauge. 3. On the drop-down box, select "Time Series: Daily Data." -- Click "GO" 13

14 4. On the page that appears, enter the following information: -Parameter of interest: Discharge (mean) -Data format: Tab separated -The start and end dates for your data file- Click "GO" 5. When the data appears you have two options: (A) On the File menu of your browser, select "Save Page As" -- If you are asked to specify a file type, save it as a text file Wait!! Your browser doesn t have a File menu? Well, if use Internet Explorer, you can make a traditional File menu appear by going to the Tools dropdown box at the top of the screen. Then select Toolbars and check Menu Bar. If you can t find a Tools drop-down box, you can also right-click near the top of your browser window, next to one of the browser tabs. This should give you a list of options that can be either selected or un-selected. Check the menu bar option. (B) Cut and paste the data from your browser into a text editor or word processor and save the resulting file in text format. Be sure to save the files in text format and not as web pages (mht or html files). Method #2 1. Go to the following web page: 2. Select "Daily Data" 3. Check the box for either "Site Number" or "Site Name" - Click "submit" 4.On the page that appears, enter the following: - either the gauge number or name - check the box for "Streamflow = ft3/s" - Under output options, select "Table of Data" - The start and end dates for your data file - Click "Submit" 5. At this point, you can follow the instruction described in Method # 1 starting with step 3. ** You don t need to make any changes to the USGS stream files before analysis. 14

15 C. USGS Historical Stream Data 1. Follow the steps needed to identify the gauge of interest in either of the above methods for obtaining data for the current water year. In other words: If you are following Method #1, complete steps 1 and 2 If you are following Method #2, complete steps On the drop-down box, select "Time Series: daily statistics." Notice that this is different from "Time Series Daily Data" -- Click GO 3. Select "0060 Discharge (Mean)" 4. Click the button for "Table" under "output format" 5. Enter the start and end dates for your data file. 6. Click submit 7. Save the file as described above. D. SNOTEL DATA 1. Go to the following web site: 2. Click on the map to find the SNOTEL station of interest 3. Click on the link for Historical, Daily, Tab Formatted Data as shown below. 15

16 4. You clicked on the link for *DAILY*, Historical, and Tab Formatted data... right? 5. Pick the water year that interests you. Notice that if you select the 2008 water year, for example, your data file will run from October 1, 2007 September 30, If you want to analyze data by CALENDAR year for example January 1, 2008 December 31, 2008, then you will need to download files for two water years. In this example you would need data from both the 2008 and 2009 water years. Don t worry, the Climate Data Summarizer has a tool that automatically joins these files for you. See step 7 for details. 6. Save the data. Your data should be on the screen. Right click on the link that says Right click here then choose save to download this data. If that doesn t work or the link doesn t appear, go the file menu of your browser and select Save as. Wait!! Your browser doesn t have a File menu? Well, if use Internet Explorer, you can make a traditional File menu appear by going to the Tools dropdown box at the top of the screen. Then select Toolbars and check Menu Bar. If you can t find a Tools drop-down box, you can also right-click near the top of your browser window, next to one of the browser tabs. This should give you a list of options that can be either selected or un-selected. Check the menu bar option. 16

17 7. Use the Climate Data summarizer to clean up your SNOTEL data files and merge them. *The Climate Data Summarizer has a tool that automatically cleans up your SNOTEL files and combines multiple files into a single file for analysis. You may have downloaded 2 files for a single SNOTEL station. For example, you might have downloaded data for both the 2008 and 2009 water years. The first file runs from October 2007 September 2008 and the second file runs from October 2008 September If you want to generate tables and graphs for the calendar year 2008 (January December 31, 2008), you need to combine these files into a single table. Also, if you look at the data files that you downloaded, you might also find that there is extraneous text before the actual data. To get rid of this extra text and combine the files, go to the File Menu of the Climate Data Summarizer and select Merge SNOTEL files into a single file. You can build a list of SNOTEL files in the window that appears, as shown in the screenshot below. The order in which you list the data files is not important. The Climate Data Summarizer will sort the data chronologically before analysis. When you click Save List of Files to Merge, you will be allowed to specify where you want to save the merge file. You don t need to edit this merged file. It is ready for analysis. 17

18 4. Setting Up a File That Contains 30-year Normals (Monthly Averages). Monthly normals (averages) are traditionally re-calculated every 10 years. At the time of writing, the normals used by most climate scientists are calculated for the time period Prior to this, the standard averages were for the period In a year or so, new averages calculated for the period will become the standard 30-year normals If you want the Climate Data Summarizer to produce graphs and tables that show departure from 30-year averages, you must store these monthly averages in a file on your hard drive. You will need to update this file every 10 years to reflect changes in the normals that are currently being used. The format is shown in an example file that ships with program: Greater_Yellowstone_1971_2000_Averages.csv. It looks like this: The file should be saved in comma-separated value (csv) format. The order of the weather stations is not important, but you must have the data for each month in the proper column as shown. The headers (column names) do not have to be the same as shown, but you must have the columns in the exact order shown by the example file, **INCLUDING ALL BLANK COLUMNS BETWEEN VALUES**. You must have the station name in the first column. The second column must contain either a station ID number or some other indication of station affiliation (e.g. SNOTEL ). The third column must contain precipitation averages for January, etc. SINCE THE DATA IN THIS TABLE IS IN COMMA-SEPARATED FORMAT, YOU CANNOT HAVE COMMAS IN THE STATION NAMES. ADDING EXTRANEOUS COMMAS WILL CREATE NEW COLUMNS AND MISALIGN YOUR DATA. If you happen to be doing climate data analysis for the Yellowstone area, the work has been done for you. Just use the example file that ships with the program. IT DOESN T MATTER WHERE YOU SAVE THIS FILE ON YOUR HARD DRIVE. THE PROGRAM WILL ASK YOU WHERE IT IS LOCATED DURING ANALYSIS. The name of the file doesn t matter, but we recommend giving it a.csv extension. The Climate Data Summarizer will automatically match the proper monthly normals to the data that you are analyzing. To do this, the program compares both the station ID number and station name to the data file being analyzed. During analysis, a window will appear asking you to confirm the matches 18

19 that have been made. If a match is not made for a particular data file, you will be allowed to manually specify which row in the file matches each weather station. NOTE ON SNOTEL DATA: Some climate scientists question the quality of the temperature data collected by the SNOTEL network, and they argue against calculating 30-year normals for these stations. You will notice that the example normal file that ships with this program (Greater_Yellowstone_1971_2000_Averages.csv) does not contain monthly temperature averages for SNOTEL stations. As a result, some graphs and tables produced by the program will show blank/ missing values for SNOTEL departure from 30-year averages. If you want to calculate the departure from 30-year averages for SNOTEL stations, you will need to fill in the blank SNOTEL values in your normal file. Getting the Normal Data MONTHLY AVERAGES You can get the values for the normal table from the internet. If you are looking for normals from a COOP weather station, go the Western Regional Climate Center web site and click on the link for your weather station. A good starting point would be here: This page should show a list of weather stations. If you click on the name of your station, you should see a link for NCDC normals. NOTE THAT THESE NORMALS ARE NOT THE SAME AS THE MONTHLY AVERAGES THAT APPEAR WHEN YOU FIRST VIEW THE WEB PAGE FOR YOUR STATION. Click on the link indicated by the red arrow below to get the correct normal data. FREEZE / FROST DATES On the same web page, click on the link for Spring freeze probabilities (see left arrow below). You should see a graph. Below the graph, click on the link for tabular output to get a page like the one below: 19

20 In the row for 32.5 degrees, find the column for 50% probability. In the example shown above, this yields the date 6/15 (right arrow). In your normal table, this date goes into the last date in spring <= 32 F. Now on this page find the date in the row for 28.5 F and column for 50% probability. This date goes into last date in spring <=28F. Once you have your spring dates, click on the link for Fall Freeze Probabilities, which is shown just below the left red arrow in the screenshot above. Follow the procedure just described to fill in your fall frost / freeze probabilities. Accumulated Growing Degree Days 20

21 As shown below, click on the click for Growing Degree Days. The number for AGDD40 (2730) is shown by the top, right arrow in the screenshot below. AGDD50 (1200) is indicated by the bottom right arrow. Extreme Day Totals As shown below, click on the link for Temperature under General Climate Summary Tables. The numbers that you need are in the Annual row as shown by the right arrow below. Important: the numbers that you get may be for the period of record for the weather station, rather than When you report your results, be sure to specify the time period for your extreme day totals if it differs from

22 To get normals for SNOTEL stations, go to the following web page: Follow the links to the SNOTEL station of interest. Your normals should appear at the bottom of the data page for your station. NOTICE THAT THE MONTHLY VALUES ARE ARRANGED BY WATER YEAR (OCTOBER SEPTEMBER). MAKE SURE THAT YOU ENTER THEM INTO YOUR NORMAL TABLE IN THE CORRECT ORDER JANUARY DECEMBER. If you don t live in the western US, contact the National Climatic Data Center or the National Weather Service for more information on obtaining monthly normals for your area. 5. QA/QC of COOP Data Introduction to QA/QC As mentioned in the Introduction, QA/QC of your data is very important, particularly for COOP data. Let s take a look at an example COOP data file: 22

23 Each line in the data file contains one month of data for either precipitation, daily maximum temperature (Tmax), or daily minimum temperature (Tmin). Each column in a line contains either a data point for a specific day or supplementary information, such as data quality flags. Since COOP data are often recorded manually on paper data sheets, the transition from the original data to computer files is fraught with errors. The data flags are designed to indicate the level of confidence that the National Climatic Data Center (NCDC) has in a particular data point and they often indicate what exactly might be wrong with the data. In addition to the errors noted by the data flags, there are other common problems, which include duplicated lines of data and most importantly, missing values. THE CLIMATE DATA SUMMARIZER DOES NOT MAKE ANY CHANGES TO YOUR DATA FILE, but it does ignore a few common data flaws as it reads the data into the computer s memory, including duplicated data lines. THE USER IS NOTIFIED OF ALL THE PRELIMINARY SCREENING THAT IS TAKING PLACE AS THE FILE IS BEING READ INTO MEMORY. The details of this screening for a particular file will appear in the main text window (see below). Importantly, the program will not let you calculate a precipitation total for a given month if there are more than three days of missing data, and it prevents the calculation of Tmax/Tmin averages in months that have more than 5 missing days of temperature data. Starting The QA/QC Procedure To start the QA/QC of a data file, go to the FILE MENU of the Climate Data Summarizer, and click on Open Single COOP Data File 23

24 Once you have specified the location of your data file, the program will give you text output highlighting particular dates that need closer examination. If you click on the Create Graphs menu and select Graph Data from just one COOP or SNOTEL station you can also generate diagnostic plots of daily temperature and precipitation data for each month in your file. You might get output something like this: 24

25 Notice that the text output in the window has identified a problem with the precipitation data for 8/31/1956 and the graphs are showing transposed values for Tmax/Tmin and an outlier precipitation value on January 1, These diagnostic plots are not meant for publication. They just help the user visualize the data... They are easier to look at than the messy data file shown at the beginning of this section. In some cases, the user will want to consult the original station data forms (B-91 forms) in order to correct the flaws identified by the program. These forms are available online ( ), but as you can see below, they are sometimes difficult to read. Luckily, the user doesn t need to read through every single data form for every weather station. The user only needs to look at the forms for months that have been identified by the program. In this artificial example, the user would want to look at the forms for January 1894 and August Specifically, they would need to examine the line on the data form for January 1, 1894 and August 31,

26 In our experience, there are often values on these forms that can be added to the data file, eliminating many of the missing values that were flagged by the program. Missing values are perhaps the most serious problem with COOP data and they are often easily fixed. By adding missing data to the computer files, you can produce graphics that had fewer missing values. Remember that the program will not let you calculate totals / averages with more than 5 missing temperature or 3 missing precipitation values in any specific month...once the user adds missing data back into the files the program will allow the new totals / averages to appear in the final output when it is appropriate. On the other hand, if your manual examination of the original station record did not help you confirm that a particular data point is valid, you might consider replacing it with a missing value (-99999). By adding missing values to your data, you will trigger the Climate Data Summarizer s quality thresholds. If you end up with more than 3 missing precipitation or 5 missing temperature data points in a particular month, the program will automatically report the entire month missing, instead of calculating an inaccurate mean /total. Once the user has corrected a data file, they should save it under a new name on their hard drive. It is important to save the original data file in case someone ever wants to undo or redo your QA/QC. 6. Analyzing the data. Generating Tables and Graphs. In some of our past work, we have spent a lot of time combining individual graphics into images that would display properly on an 8.5 x 11 sheet paper. It doesn t make sense to put just one graph on a page, but it often isn t obvious how many to put on a page or how big to make each graph. Will the font be too small if we put three graphs on a page? If the ratio of width length isn t right, will the graph be smashed into a thin, narrow, template? In light of this experience, the Climate Data Summarizer has been designed to produce finished pages with graphs for multiple weather stations arranged in a grid. In order to produce these graphics, the user is guided through the selection of station data files. Notice in the screenshot below that two stations have been added to the list so far. The user can choose to add either COOP or SNOTEL files as needed. The program has built-in error checking routines that try to prevent the user from opening the wrong type of file or invalid files. You can access the station list window shown below from either the File menu or by clicking the graph / table of your choice in the other menus. If you haven t specified a station list when you try to make a graph or table, the program will prompt you to do so. 26

27 Once the user clicks Save Station List they get another window that looks like the one shown below. Using a series of routines, the program has matched the station data files with the appropriate averages (30-year normals). The user is asked to verify that the matches are correct. If the matches are wrong or if there is an unmatched station (shown deliberately for this example in the case of Canyon SNOTEL) then the user can fix the problem. 27

28 If the user needs to fix one of the matches, another window will open that provides a list of the entries in the normal table, allowing for an easy change (not shown). Once the matches have been approved, the user gets a window that looks like the one below. The user can change the color for each weather station and specify the number of rows x columns in the graph grid. 28

29 Once the user saves the settings, they get a graphic similar to the one shown below. If the user chooses to create 8 graphs, they fit perfectly on an 8.5 x 11 inch of paper. If fewer graphs are selected, each graph will still be the same size. The graph grid will just fill less space on the page. 29

30 Figure 1. Example output showing precipitation departure from averages for 8 weather stations. 30

31 Different types of graphs come with different formatting requirements. Boxplots, for example, have a larger width: length ratio, so only six will fit on a page. Don t worry, the program will give you more detail on formatting requirements in the text box while you are producing the graphs. If you follow the guidelines that the program gives you, your graphics should fit correctly on a page. For more information, you may also consult the document Example Graphics From the Climate Data Summarizer version 3.docx, which ships with the program. This Microsoft Word document contains all the possible graphics that may be generated, and its margin / text wrap settings can be used as a model for how to fit graphics onto a page without reducing their quality. The program also generates a series of summary tables. To make them, the user navigates a series of windows very similar to those just described. Example output is shown below. These tables can be viewed either as excel files or as html documents (shown above). The html documents look nicer, can be edited in Microsoft Word and posted directly to the internet. Notice that in the example table (above) there were many missing values in the data, despite QA/QC. The program has prevented the table from containing inaccurate values by reporting only for fields that met the missing value criteria described. Only some of the weather stations in this table had enough data to calculate all the summaries in accordance with the data quality criteria. The program has options that will allow you to relax how many missing values are permitted, thus producing more complete tables. The restrictive version is shown above. Notice also that Ashton, Idaho isn t reporting 30-yr averages for temperature, so the program just leaves those fields blank. Even though the examples shown here focus on data for just one year (2009), the program can read data files of unlimited length and produce tables that span many years. To generate a multi-year table, go Create Tables menu and click on the first option, as shown below: 31

32 7. Where Do The Graph Titles Come From? Graphs that depict departure from 30-year averages will have titles matching the station names in your table of normals (see section 4). All the other graphs will take their titles from the name of their respective data files. File extensions and extended path names are automatically removed. For example, you might have a file name that looks like this: C:/Users/big_computer/Desktop/Climate Data Summarizer version 3/Yellowstone data - example files/2009 COOP files/alta.csv And it will show up in your list of weather stations like this: But your graph title will be Alta. The program strips off the.csv and everything that appears before the last slash. If you want to change the graph title to Alta 1NNW, you should change the file name on your computer like this: 32

33 8. Literature Cited Hunter, J.D Matplotlib, a 2D graphics environment. Computing in Science and Engineering 9: