Penetrating the Matrix Justin Z. Smith, William Gui Zupko II, U.S. Census Bureau, Suitland, MD

Transcription

1 Penetrating the Matrix Justin Z. Smith, William Gui Zupko II, U.S. Census Bureau, Suitland, MD ABSTRACT While working on a time series modeling problem, we needed to find the row and column that corresponded to the minimum of a dataset. Processing the dataset via SAS 1 DATA step methods proved somewhat inefficient for the task because SAS is designed to run along observations, but not through variables without considerable manipulation of the dataset. By reading the dataset into PROC IML and treating it as a matrix, we were able to pinpoint the specific location with ease. PROC IML was able to penetrate the matrix and create macro variables that told us where in the matrix our minimum value was located. These macro variables were then used to identify the tentative autoregressive and moving average orders for time series modeling using PROC ARIMA. DISCLAIMER This report is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. INTRODUCTION SAS is designed to process observations in a dataset in the DATA step. Although there are many procedures available to check through variables, we encountered a problem when we tried to locate a specific number within a dataset. The twist with our situation is that we needed to obtain the location of the minimum value of the dataset which was more important than the minimum value itself. Finding the minimum of a matrix is simple enough, but what do you do if you need to find the row and column that correspond to the minimum of a matrix? Using publically available historical shipments and inventories data from the U.S. Census Bureau s office of Manufacturers Shipments, Inventories, and Orders (M3) 2, we created an autoregressive integrated moving average (ARIMA) model to try to predict the value of shipments (VS) / total inventory (TI) ratio over time. An ARIMA model involves autoregressive (AR) terms, which are past values of the series, and moving average (MA) terms, which are past values of random errors. The number of AR terms is denoted by p, and the number of MA terms is denoted by q. There are several methods to tentatively identify suitable p and q for a model. This paper focuses on using the minimum information criterion 3 to accomplish this. The minimum information criterion is a statistic based on the residuals from the model and an added penalty which is a function of the number of model parameters. By using the MINIC option in the IDENTIFY statement, PROC ARIMA will give a matrix, where the rows are an inputted range of p s, the columns are an inputted range of q s, and the entries are values of the information criterion (Brocklebank, 2005). 1 SAS and all other SAS Institute Inc. product or service names are registered trademarks of SAS Institute Inc. in the USA and other countries. A indicates USA registration. Other brand and product names are trademarks of their respective companies. 2 Estimates of manufacturers shipments, inventories, and orders are subject to survey error and revision. For further details on survey design, methodology, and data limitations, see 3 See for detailed information. 1

2 EXPLANATION OF CODE We used the following PROC ARIMA code to produce the matrix shown in Table 1. Because PROC ARIMA outputs its results directly to the output window, we needed to use ODS Output to create a dataset that will house our matrix. We called this dataset MINICDATA. ODS Output takes the results from the output window and creates a dataset with that information. In this case, we wanted the information provided in the Minimum Information Criterion. ods output "Minimum Information Criterion"=minicdata; proc arima data=mergevsti; identify var=ratio(1) minic p=(1:7) q=(1:7) alpha=.1; quit; ods output close; To get a fairly wide range of possible p and q values to test, we programmed PROC ARIMA calculate the information criterion using p and q both ranging from 1 to 7. Usually, these values are restricted by business rules, which are processing criteria specific to a given survey, to be below a certain order, for example p 3 and q 4, and other restrictions, such as not having both p and q in the same model (a mixed model ). The p values represent the number of autoregressive terms, and are the rows in the matrix. The q values represent the number of moving average terms, and are the columns in the matrix. Note that the ratio(1) part of the code takes the first difference of the series. We used this method with this series to achieve stationarity. Table 1 Minimum Information Criterion Lags MA 1 MA 2 MA 3 MA 4 MA 5 MA 6 MA 7 AR AR AR AR AR AR AR We then manipulated the matrix, deleting the first row, due to a business rule for a prediction to be based on at least two prior observations. This gave us the matrix shown in Table 2. Table 2 Minimum Information Criterion Lags MA 1 MA 2 MA 3 MA 4 MA 5 MA 6 MA 7 AR AR AR AR AR AR

3 By observation, we can see that the minimum of this matrix is the value , which we ve highlighted in Table 2, corresponding to the AR 3 and MA 1 location. Therefore, we want our macro variables to be p = 3 and q = 1. Our first solution was to make a dataset for each column, calculate the minimum in each dataset using PROC MEANS, and keep the column with the minimum of the minimums. Then we d loop through the columns and note the row for which the minimum occurred. This was a slow process for the simple matrix we were using and it would be even slower using matrices with many more rows and columns. Our second solution was to use PROC IML. We read in the MINICDATA dataset, and looped through the rows (i) and columns (j), storing i and j where the matrix entry was the minimum. PROC IML is different from other PROCs. It does not use the data or set statements in the DATA step. While the use statement is like the set statement, no columns are read into IML unless the read statement is used. We used the read all into command to bring in the MINICDATA dataset into IML as a matrix. We needed to tell PROC IML the name of the rows and columns. We named the matrix pqmat, with the observations becoming the rowname and the variable names starting with the string ma. proc iml; use minicdata; read all into pqmat[rowname=rowname colname=ma_]; Next, we obtained the number of rows and columns. These values were used in the DO loops over the rows and columns. We calculated the minimum of the matrix here. Note that calculating the minimum outside the DO loops is more efficient than calculating the minimum inside the DO loops because we only needed to calculate it once, not repeatedly. nrow=nrow(pqmat); ncol=ncol(pqmat); minmatrix=min(pqmat); Below are the DO loops. Here we stored the values of i and j to macro variables that (i,j) corresponded to the minimum identified above. The print option is not required in this step, but it is useful to confirm that the matrix was created correctly. We printed the nrow and ncol to confirm that we had the correct number of rows and columns in the matrix. This gave us further confirmation that the data was created properly. print pqmat nrow ncol; do i=1 to nrow; do j=1 to ncol; if pqmat[i,j]=minmatrix then call symput('i',left(char(i))); if pqmat[i,j]=minmatrix then call symput('j',left(char(j))); end; end; quit; Next, we used i and j to create the macro variables p and q. This correction step was needed because we started the range of p and q at 1. Since we have a business rule where future observations in our time series must be based on at least two past observations, p must be at least two. This value of p is then calculated by increasing the value of i by one. Because q starts at one, no additional correction was made to q. However, we left this portion of code in for q to remind us that if we start at a different range for q, or have other business rules for q, then we would need to modify j to calculate q as well. For the dataset 3

4 above, the minimum was identified at i = 2 and j = 1. Therefore, this correction step will calculate the correct p = 3 and q = 1. %let p=%eval(&i+1); %let q=%eval(&j); Last, we outputted the values of p and q to the log window for review. %put p=&p; %put q=&q; Our code determined p = 3 and q = 1 as the model with the minimum information criterion, and is therefore the preferred tentative model. Using PROC ARIMA with the ESTIMATE option with macro variables evaluating to p = 3 and q = 1, the output window showed the parameter estimates for the model. Figure 1 shows the output screen. The IML code not only printed out the rows and columns, but the variables we created showing how many rows and columns are in the matrix. Fig. 1 A modification to the code would be necessary if, for example, the matrix had more than one cell containing the minimum. In that case, our code would merely store the last (p, q) pair where the minimum occurred. In our context, models that would happen to have the same value of the minimum information criterion would be statistically indistinguishable. The modification would be to insert a counter to total the number of minimums identified, and to store the different p and q by incorporating this counter as a macro variable in the names. By storing these values as macros, we were able to use them to set up parameters for later use. To see this demonstrated in an entire process, please refer to our parent paper Creating and Displaying an Econometric Model Automatically paper # 52, by the same authors. CONCLUSION Finding the minimum of a matrix is a simple task, but finding the row and column corresponding to the minimum of the matrix is somewhat more involved. PROC IML gives us the flexibility to search a SAS dataset through its observations, and also through its variables. Programmers often need greater flexibility than row-based calculations or comparisons. PROC IML grants that flexibility using a matrix for its calculations, which allows programmers to compare observations and columns. We learned that some care needs to be taken to identify the correct row and column if rows or columns are being deleted. Additionally, the code needs to be modified if there is more than one minimum. Note, that while we worked with the minimum function, the method is general and many criteria can be used depending on the application. 4

5 REFERENCES Brocklebank, John C. and Dickey, David (2005). SAS for Forecasting Time Series, Cary, NC, SAS Institute Inc. ACKNOWLEDGEMENTS The authors would like to thank Courtney Harris, Jan Lattimore, and Amy Newman-Smith for their reviews and suggestions which greatly improved this paper. CONTACT INFORMATION If you would like to submit comments and questions, please contact the authors at: Justin Z. Smith U.S. Census Bureau 4600 Silver Hill Road, Room 7K072D Suitland, MD, William Gui Zupko II U.S. Census Bureau 4600 Silver Hill Road, Room 7K072D Suitland, MD,

6 Appendix A: Input Data /*M3 data, seasonally adjusted VS and TI at the total mfg level*/ data mergevsti; input date:monyy5. vs_total ti_total; format date monyy5.; datalines; jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb

7 mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov

8 dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug

9 sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may jun jul aug sep oct nov dec jan feb mar apr may

10 jun jul aug sep oct nov dec jan feb mar apr may jun jul ; /*create the ratio of value of shipments over total inventory*/ data mergevsti; set mergevsti; ratio=vs_total/ti_total; 10

11 Appendix B: PROC ARIMA and PROC IML code /*get the minimum information criterion of the first differenced series*/ ods output "Minimum Information Criterion"=minicdata; proc arima data=mergevsti; identify var=ratio(1) minic p=(1:7) q=(1:7) alpha=.1; /*took the first difference*/ quit; ods output close; /*clean up the minicdata dataset. This is because we want a forecast based on at least 2 past observations*/ data minicdata; set minicdata; if rowname in('ar 1') then delete; /*use proc IML to manipulate dataset to get p and q for suggested AR(p) and MA(q) process*/ proc iml; use minicdata; read all into pqmat[rowname=rowname colname=ma_]; /*get num of rows and columns. This will be used in the DO loop*/ nrow=nrow(pqmat); ncol=ncol(pqmat); minmatrix=min(pqmat); /*calculate minimum of matrix*/ /*display matrix, nrow, and ncol, get i and j*/ print pqmat nrow ncol; do i=1 to nrow; do j=1 to ncol; if pqmat[i,j]=minmatrix then call symput('i',left(char(i))); if pqmat[i,j]=minmatrix then call symput('j',left(char(j))); end; end; quit; /*this adjustment is done to get the correct p and q*/ /*this is because we want a forecast based on at least 2 past observations*/ %let p=%eval(&i+1); %let q=%eval(&j); %put p=&p; %put q=&q; /*estimate the model using the identified p and q*/ proc arima data=mergevsti; identify var=ratio; estimate p=&p q=&q; quit; 11