1 Organizing time series in MATLAB structures

Transcription

1 1 Organizing time series in MATLAB structures The first step is to store your time series and associated metadata in MATLAB structures. This step provides some uniformity to the format of the time series data, and is necessary to ensure that all MATLAB scripts and functions written for the course will run on each student s data without the need for reformatting. A structure is a MATLAB variable-type similar to a database in that the contents are accessed by textual field designators. A structure can store data of different forms. For example, one field might be a numeric time series matrix, another might be text describing the source of data, etc. In this first lesson you will run a MATLAB script that stores your data in structures. In subsequent lessons you will apply time series methods to the data simply by running MATLAB scripts on those structures. As a first step, you must organize your time series and associated metadata in one of several possible formats. The input time series files for the first script can be space-delimited ascii, Excel spreadsheet, or.mat file a MATLAB storage file. The metadata must be in an ascii file. This lesson describes the preparation of your time series for use in the course, and introduces computation and graphics with MATLAB. 1.1 Data types: V1, V2, V3 You must prepare three types of time series variables. Types V1 and V2 are best thought of as nominal output from and input to a causal system. V1 contains output variables and V2 contains input variables. For example, V2 might be precipitation series and V1 tree-ring index chronologies. Or V2 might be precipitation and V1 might be streamflow. The designation as input and output is nominal in that your time series might not fall neatly into a causal system. If not, simply one group of time series as input series and another group as output series. The series in V1 and V2 are assumed to be stationary in the mean -- that is, either to have been detrended beforehand or to not require detrending. This assumption is necessary because some of the methods assume stationarity. The third type of data, V3, is allowed to have -- and preferably does have -- a trend in mean. Variables V3 are used to demonstrate detrending methods. One example of a suitable V3 variable is measured tree-ring width, which typically has a growth trend. Another example is a temperature series for a station affected by urban warming. You will need at least three V1 time series one V2 and one V3 time series for the course assignments. 1.2 Time step and sample size A time series consists of some variable observed at different times. A critical requirement for the time series selected for V1, V2, and V3 is a uniform time step, or time spacing of observations. The class examples and data sets used by most students have a time step of one year (tree-ring indices, summer-total rainfall, etc.). Other time steps are acceptable as long the time step is constant and the same for all time series. What s critical is that the time step is constant and is the same for all data in sets V1, V2 and V3. Sample size is important for some time series applications. To avoid problems with inadequate sample size, the all time series in V1 and V2 should overlap by at least 50 observations. In other words, there must be a common data period of 50 observations for all series in V1 and V2. The series in V3 are used independently of those in V1 and V2, and so are not required to overlap at all with V1 and V2 series. Moreover, it is not necessary that the individual series in V3 overlap one another. Notes_1, GEOS 585A, Spring

2 The data selected for V1, V2, and V3 should be continuous in the sense that there are no internally missing values. This means that the all observations must be complete from the first data point to the last data point. Some interesting data sets with missing data might still be usable, but would require preliminary data estimation to fill in the missing values. 1.3 Input format of time series You need to organize your time series into a time series matrix (one variable per column, one observation per row) in preparation for input to MATLAB. A function geosa1.m reads the data into MATLAB, and will accept input data in any of the following forms: Ascii, with spaces separating columns; file with suffix.dat.mat, a specific MATLAB storage format; file with suffix.mat.xls, an Excel spreadsheet, file with suffix.xls Whatever the format used, the first column should be the time variable (e.g., year) and the other columns should be the individual time series. I assume that if your time series matrix is already stored in a.mat file you are familiar with that sort of storage. For ascii data, the matrix must be all-numeric -- every element must be a number; no letters are allowed. For.xls format, the part of the spreadsheet holding the time series must similarly be allnumeric. An alphanumeric header line is allowed in the.xls file. A message box appears when you run geosa1.m listing the series headers, if any, read from the xls file. These should agree in order with the series names and ids as listed in the txt file accompanying the xls file. 1.4 Missing data The time series matrix must be full (no blanks), but because all series might not cover the same period, some elements of the matrix will likely be missing-value codes. The only restriction on missing values is that missing values cannot be imbedded within a stretch of valid data for any series. For example, an annual time series covering years with data missing for 1955 is not allowed; you should have filled in any such internal missing data with estimated data beforehand. The time series matrix might of course have missing values at the start and end of each series. Any missing data must be coded either as NaN or as some numeric missing-value code (e.g., -999). Use the same code for all missing data within a given data set V1, V2 or V3. NaN is the IEEE standard representation for a missing number. Geosa1.m reformats your data on input to replace your missing-value code with NaN. 1.5 File naming A separate time series matrix must be prepared for each of the data sets V1, V2, and V3 described in 1.1. The files should be named consistently for ease of reference, and must have the appropriate suffix (.dat,.mat, or.xls) depending on the format. For example, you might have spreadsheets v1.xls, v2.xls and v3.xls. Or, more informatively, you might name the files v1index.xls, v2precip.xls, v3width.xls. I suggest sticking with the convention of using v1, v2 and v3 as the first two characters of the filenames, as that will make it easier for the file location functions used in some of the scripts. 1.6 Supporting information for time series the metadata The MATLAB scripts need supporting information on the time series in order to produce intelligible labeling of series and graphics, and to facilitate stratification into data subsets for some analyses. Geosa1.m reads this information from an ASCII text file (.txt file) that you must Notes_1, GEOS 585A, Spring

3 build. You can easily build the required.txt file using a text editor (e.g., notepad). I prefer vedit plus ( for this task, but any editor that produces ASCII output will do. The metadata.txt file should have eight initial lines of information that apply to all time series in the time series matrix, plus one additional line for each time series in the matrix. Let s say you have stored eight tree-ring index time series as V1 data in the spreadsheet V1SAMPLE.XLS. Make an accompanying.txt file V1SAMPLE.TXT with the metadata. The first eight lines of metadata should have the following information: 1 Name of the data file with the time series matrix 2 Type of data 3 Data label... short label used on axes of plots 4 Units 5 Time increment of observations 6 Missing value code 7 Three category names, separated by '$' 8 Beginning column of individual-series fields (see below) Subsequent lines should contain individual series information, one line per series, starting in columns denoted on line 8 above. For example, one line would contain the following information: 1 series id, 12 or fewer characters 2 first category 3 second category 4 third category 5 expanded series name, maximum of 30 chars 6 latitude, in decimal degrees, with + for N and for S 7 longitude, in decimal degrees, with for W and + for E 8 elevation above mean sea level, in meters The series id is self explanatory -- a short code that makes sense to you in identifying the series. This id should not contain any blanks. The categories are codes, preferably 4 or fewer characters, included to allow the possibility of selecting series interactively depending on some quality of the data (e.g., species). None of the class scripts currently used refer to these codes, and so dummy values will suffice though for your own reference you may want to use meaningful codes. For the dummy option, simply assign fake 4-or-fewer character codes as the names of the categories and the values of the categories for each series. The codes in this case might all be the same, say XXX. The expanded series name (item 5 above) allows you to put more detailed series identification than can be included in the short (12 chars or fewer) id. This expanded name is not used for labeling, but can be queried to spark your memory of the series. The latitude, longitude and elevation information are include to allow flexibility to do spatial analysis on the data. It is acceptable to use other consistent x and y coordinates (e.g., UTM coordinates of easting and northing in place of longitude and latitude. The only requirement is that the equivalent latitude increase to the north and the longitude increase to the east. Notes_1, GEOS 585A, Spring

4 1.7 Sample input files A simple example using fictional data will serve to illustrate the format and content of the input data for script geosa1.m. Consider a V1 data set consisting of 8 tree-ring index chronologies in an Excel spreadsheet. The associated ascii text file v1sample.txt has the following lines: v1sample.xls Ring-Width Index Index Dimensionless Year -999 Species $ Measurement Type $ Age Class BOLES JUOC TW X Boles Creek, CA ANTEL PIPO TW X Antelope Lake, CA JACKSON JUOC TW X Jackson Mtns, NV STEENS JUOC TW X Steens Mtn, OR SARDINE JUOC TX X Sardine Point, CA CARSON JUOC TX X Carson Pass/Caples LK, CA MEANTREE MIX TW X Six-Site Mean PC#1TREE MIX TW X Six-Site PC#1 Scores As specified by the first line in v1sample.txt, the time series matrix is an Excel spreadsheet v1sample.xls. Line 2 specifies that the type of data is Ring-Width Index. Line 3 specifies that the short label Index is to be used in labeling the y axis of time series plots. Line 4 specifies that the ring-width index is dimensionless (units). Line 5 specifies that the time step or spacing between observations is a year. Lin6 6 specifies that any missing values in v1sample.xls are coded as Recall that that no internal missing values are allowed. Line 7 specifies that Species, Measurement Type and Age Class are the category names. Line 8 specifies that the starting columns for series-specific metadata in subsequent lines is to start in columns 1, 14, 19, 22, 24, 50, 59 and 68. For example: column 1 has the B in Boles, column 14 has the J in JUOC, column 19 has the T in TW, etc. Similarly, the V2 time series are in v2sample.xls, with associated metadata in v2sample.txt; and the V3 series are in v3sample.xls, with associated metadata in v3sample.txt. Please look at those sample files as a guide in setting up your own data. An easy way to do this is to use the sample.txt files as templates and edit in the information relevant to your own series. 1.8 Running geosa1.m First make sure that the required input data files are in the current MATLAB working directory (Appendix A). At the MATLAB prompt, enter >>geosa1 The script begins running, and a prompt appears asking whether the data is V1, V2 or V3. Choose V1. A file selection window appears asking you to pick (click on) the desired.txt file. Here the file is v1index.txt, so click on v1index.txt. Notes_1, GEOS 585A, Spring

5 Information relevant to the V1 series appears in the command window and a menu window asking whether the data looks OK so far appears. If displayed information correct, click that menu choice. A dialog box appears asking you to specify the name of the output.mat file in which to store the output time series structure. If this is the first run of geosa1.m, type in the name of a.mat file to create (e.g., meko.mat). If you have already stored one or more time structures in the.mat file, click on the file name in the dialog window. You will be asked whether you want to overwrite the.mat file. Click Yes. A message box appears telling you what you are about to do. Click OK to get rid of the message box. A menu asks whether to continue saving or abort. Unless you have made a mistake, click Continue saving. You are returned to the opening menu, Select type of data or Quit. Here you may repeat the above steps for time series V2 and V3. Or if you decide to stop with V1 for now and enter the other data later, select Quit. 1.9 Checking that the structure variable has been created and stored You should now have a.mat file (e.g., meko.mat) in your current working directory, and that.mat file should hold vlist -- a list of variables in the.mat file, and a subset of time series structure variables V1, V2, V3. Assume the.mat file is meko.mat. Check out contents of the.mat file by typing commands: >> what shows a list of.mat files in the command window. You should see meko.mat in the list. >> clear >> load meko clears the workspace of any existing variables and loads v1index.mat contents >> whos...lists variables in the workspace. You should see vlist and one or more of V1, V2, V3 there now. >>vlist... lists the contents of variable vlist >>V1 Typing a variable s name displays the contents of the variable in the command window. Names are CASE SENSITIVE. When you type V1, you see a list of all the fields in the structure variable V1. Type the structure followed by a period and a fieldname and you get the expanded contents of the field. So, for example, >>V1.catname lists the category names, >>V1.name lists the names of the five time series, and >>V1.tsm lists the time series themselves. Note that V1.tsm has only the time series data, not the vector of years. V1.time is the year vector. You can generate a quick time series plot of the first and third series, with a legend of ids, by entering this sequence of commands >>t=v1.time ; Notes_1, GEOS 585A, Spring

6 >>x1=v1.tsm(:,1); >>x3=v1.tsm(:,3); >>plot(t,x1, -,t,x3, -- ); >>ylabel(v1.label); >>xlabel(v1.increment); >>legend(v1.id([1 3])); The first three commands just rename variables so that the plot syntax is less cumbersome. The next commands plot the series and label the plot. Entering the commands one by one at the command prompt is one way to use MATLAB. A better way is to organize the commands in an ascii file called a script and then run the script from the command prompt. Using scripts avoids having to re-enter commands each time your want to repeat an analysis or modify some part of the analysis or the plots. The interactive tools at the top of the figure window are another way to change attributes of plots. Please refer to the MATLAB help files for more information on graphics. MATLAB functions, like scripts, are also ascii files containing commands, but are more general in that they may be applied to different problems merely by changing the input arguments. The plot command above is actually a function with input arguments the x-data, y data and line-type for the plot. MATLAB has many built-in functions for data analysis, and you can extend this capability with your own functions or user-written functions available over the Web. It is helpful to learn some basic elements of MATLAB programming if you want to apply MATLAB functions other than those covered in the course, or if you want to tailor your own data analysis. But you do not need to program in MATLAB for this course. The script files and functions needed for all analyses have already been written and are provided for the course Turning in assignments as figures and captions Assignments for the course consist of running MATLAB scripts demonstrating the time series methods and writing up brief interpretations of the results. Each assignment should be turned in as a word-processing document with figures and captions only. The figures are generated by the MATLAB scripts. The captions are your answers to the questions in the assignment. This is easily done with a word processor in combination withy copy/paste from MATLAB s figure window. For example, open a new blank document in MS Word, click View/PrintLayout, set the view to whole page, and click Insert/TextBox to set a box to hold the figure. In MATLAB, use the following two commands to make a dummy figure: >x=rand(10,1); >plot(x) You can use the figure-window annotation features (e.g., arrows and text) to point out items of interest in the figure. On the MATLAB figure window menu, click Edit/CopyFigure, then inside the box in MS Word click Paste. You now have the figure in a text box in MS Word. Pull down the bottom of the text box to allow room for a caption. Right-click on the figure inside the text box to bring up a menu that includes Caption. Click Caption, and then OK when the figure box comes up with a caption number. Now can enter whatever text you want as the caption. Repeat the procedure for as many figures are included in the assignment. Save the file with as a.doc file with the filename including your last name and the assignment number. For example, smith1.doc. Zip the.doc file together with any additional requested files into a.zip file of the same name (e.g., smith1.zip). Notes_1, GEOS 585A, Spring

7 me the.zip file as an attachment. I will return the assignment as a.pdf with imbedded comments and the grade (number of points) coded into a header. Notes_1, GEOS 585A, Spring