Statistics, Data Analysis & Econometrics

Transcription

1 ST001 A SAS MACRO FOR THE CONTROLLED ROUNDING OF ONE- AND TWO-DIMENSIONAL TABLES OF REAL NUMBERS Robert D. Sands Bureau of the Census Keywords: Controlled Rounding, Tabular data, Transportation algorithm ABSTRACT A SAS Macro is described that solves the problem of rounding the real number entries of a tabular array to adjacent integers while preserving the additive structure of the table marginals. The controlled rounding is also the optimal solution in that the total change in the original numbers is minimized. The Macro is designed to have functionality similar to that provided by the conventional SAS ROUND Function. This is to say that a round-off unit argument is provided which will produce a table rounded to the nearest integer, as well to the nearest ten, hundred, tenth or hundredth. INTRODUCTION This paper describes the background, theory, and implementation of the controlled rounding macro. Appendix B contains the macro in its entirety. BACKGROUND The general algorithm employed in the Macro is based on a paper by Cox and Ernst (1982) describing the solution to the Controlled Rounding problem. The first solution to the controlled rounding problem which employed SAS PROC NETFLOW was created by Brian Richards and Jim Fagan (1997) at the Bureau of the Census. This macro did a controlled rounding to integers of a fixed size 7x7 matrix of real numbers. The macro produced a solution with the same minimum cost as an earlier FORTRAN language controlled rounding program created by the same authors. Applications of controlled rounding include rounding to integers the real number estimates of population which had been adjusted subject to factors determined by a population coverage survey (Wolter, 1986). This was the application for the 2000 Census coverage survey that motivated the author to produce the macro discussed in this paper. Another application could, for example, be the disposition of the cents portion of a table of financial figures in the most judicious manner while maintaining the additive structure of the table. THEORY The theory for controlled rounding is based on the capacitated transportation problem characterized as a network (Cox and Ernst, 1982). This theory is presented in detail in Dantzig (1963). The mathematical theory underlying the transportation problem and linear programming (LP) problems in general concerns the solution of a system of linear inequalities that also minimizes a linear form. An excellent treatment of linear programming (LP), the simplex method of solving LP problems, as well as the distribution method of solving the specific transportation type of LP problem can be found in Siemens, et al, The transportation problem consists of a list of sources of a commodity and a list of destinations for a commodity. Each link or arc between a source and destination has a cost per unit shipped of the commodity as well as a minimum and maximum flow for the commodity. Each source node has a limited capacity to produce the commodity while each destination node has a particular demand for the commodity. The objective is to fulfill the demands of all destinations while not exceeding the supplies of each of the sources and to do so at a minimum distribution cost. In the controlled rounding problem, the distribution cost for a cell of the matrix is a function of the square of the decimal portion of each real number. For example, the number, 2.31, is rounded up to 3 at a cost of ( ) 2 - (0.31) 2 = 0.38 whereas the number, 4.62, is rounded up to 5 at a cost of ( ) 2 - (0.62) 2 = The objective is to round (distribute) the decimal portion of each number in the interior cells and in the marginal row and column to an adjacent integer subject to a number of constraints. The controlled rounding must result in a matrix in which : 1) all the numbers sum to the marginal row, column, and grand total (tabular structure is maintained), 2) a number that is already an integer is unchanged, and 3) the total amount of change to the numbers is minimized.

2 The third constraint deserves some more discussion. The function for determining the cost is in terms of rounding up. The rounding up operation has a zero or negative cost when the number has a decimal portion >=.5 and a positive cost if the number to be rounded up has 0 < decimal portion <.5. The roundings selected by the systematic methodology of linear programming are most often conventional and therefore have a negative cost. Frequently, however, there are instances where the additive structure cannot be maintained at minimum overall cost without unconventionally rounding some individual table entries. That is, some entries which have 0 < decimal portion <.5 must be rounded up to sacrifice for the greater good of the matrix. For example, consider the sequence of Tables 1a-j in Appendix A, representing a 7x7 table of real numbers (Table 1a). In Table 1b, the marginal row and column totals as well as the grand total entries are included. With the marginals added the 8x8 table of real numbers can begin to be characterized in the form of a transportation problem matrix. This will be explained in more detail in the following paragraphs. When the integer portion of the original 7x7 internal table entries is removed and the marginal sums included, the matrix in Table 1c is produced. This table concerns only the decimal portion of the original table (Table 1a) because the controlled rounding algorithm concerns only the costs of rounding up the decimal portion of the number. Thus, the integer portions of the real numbers in the original table are arbitrary to the rounding solution. Once the controlled rounding solution is determined by the transportation algorithm the integer portions are merely substituted back into the solution. The highlighted entries of Table 1d are added (folded-in) with the highlighted marginal row and column of Table 1c to produce the marginal row and column of Table 1d. In Table 1d the marginal (ninth) column represents the supply held by each row while the marginal (ninth) row of Table 1d represents the demand required by each column. The supply and demand marginals of Table 1d form the right-hand side of a system of linear functions depicted in Table 1f. Each variable, x ij, in Table 1f represents a cell in Table 1b. In the control rounding solution, x ij = 0 means round down and x ij = 1 means round up. The 8x8 sub-table entries (excluding the marginal row and column and grand total) in Table 1d determine the cost of rounding up each cell of Table 1b. The cost of rounding up each interior cell and the marginal cells is given in Table 1e. Refer to the discussion of cost earlier in this section to review the function that uses the decimal portion of the real number to compute the cost of rounding. The transportation algorithm operates, generally, in two phases. In the first phase, an initial solution to the controlled rounding problem is found. This initial feasible solution is depicted in Table 1g as a 0 1 solution to the transportation matrix of Table 1f. The actual rounded entries corresponding to the initial solution of Table 1g are given in Table 1h. Incidentally, the initial solution was found the Northwest corner method (Seimans et al, 1973) which inserts 1s as values of the x ij variables by starting in the upper left corner of the table and proceeding through the cells of the table in a stair step manner until all the marginal constraints are fulfilled. This approach yields a controlled rounding of the table but not necessarily the lowest cost (optimal) controlled rounding. The second phase of the transportation algorithm employs the cell rounding costs of Table 1e to determine a controlled rounding of Table 1b in which the total cost of changing the numbers is minimized. This optimal solution in shown in Tables 1i and 1j. Employing a method based on the original Simplex method of Dantzig (1963) the cost of controlled rounding is reduced from 4.76 of the Initial Solution to of the optimal solution. IMPLEMENTATION The SAS macro program for controlled rounding (CTRLROUND) described in this paper was created by the author in 1998 and was used to produce rounded small area estimates in preparation for the 2000 Census population coverage survey. The author used the SAS program, which control rounded a fixed size 7x7 matrix, created by Brian Richards and Jim Fagan, and generalized it. This Macro will produce a controlled rounding to adjacent integers of an arbitrary m x n matrix of real numbers input as either a text file or as a SAS data set. It can handle round-off units in powers-of-ten. The Macro is hardware platform independent. The PC (Pentium III) implementation can process a 1,000 cell table in 1 second, a 10,000 cell table in approximately 70 seconds and a 100,000 cell table requires approximately 70 minutes. The complete SAS macro program code (CTRLROUND.SAS) is given in Appendix B. It can be cut and pasted as is from an electronic version of this paper into the SAS Program Editor window on any hardware platform running SAS Version 8, the appropriate home file directory path defined for the home macro variable, and run.

3 Continuing the example of the previous section and referring again to given to Tables 1a-j, the input to PROC NETFLOW is held in two data sets. Each of these data sets must be constructed from the table to be control rounded using the methodology discussed in the previous section. The two input data sets 1 created by the CTRLROUND macro are the NODED and ARCD1 data sets shown in Appendix C. The control rounded solution data set output by PROC NETFLOW is the SOLUTION data set shown in Appendix C. CONTACT INFORMATION Robert D. Sands, DSSD, Room , US Census Bureau, Washington, DC 20233, (301) robert.d.sands@census.gov REFERENCES Cox, Lawrence H. and Ernst, Lawrence R. (1982), "Controlled Rounding", INFOR (20)4, Dantzig, George B. (1963), Linear Programming and Extensions, Princeton, NJ: Princeton University Press. SAS Institute Inc. (1999), SAS/IML User s Guide, Version 8, Cary, NC: SAS Institute Inc. SAS Institute Inc. (1999), SAS/OR User s Guide, Version 8, Cary, NC: SAS Institute Inc. Siemens, Nicolai, Marting, C.H., and Greenwood, Frank (1973), OPERATIONS RESEARCH, New York: THE FREE PRESS. Wolter, Kirk, M. (1986), Some Coverage Error Models for Census Data, Journal of the American Statistical Association, 81, The _CO ST_ data values in the SAS data sets of Appendix C are, when divided by 10000, equivalent to the costs given in Table 1e of Appendix A.

4 APPENDIX A Table 1a. Original Table without marginals Table 1f. Transportation Matrix of CR Problem x 11 x 12 x 13 x 14 x 15 x 16 x 17 1-x x 21 x 22 x 23 x 24 x 25 x 26 x 27 1-x x 31 x 32 x 33 x 34 x 35 x 36 x 37 1-x x 41 x 42 x 43 x 44 x 45 x 46 x 47 1-x x 51 x 52 x 53 x 54 x 55 x 56 x 57 1-x x 61 x 62 x 63 x 64 x 65 x 66 x 67 1-x x 71 x 72 x 73 x 74 x 75 x 76 x 77 1-x x.1 1-x.2 1-x.3 1-x.4 1-x.5 1-x.6 1-x.7 x Table 1b. Original Table with marginals Table 1g. Initial Solution of Transportation Matrix Table 1c. Fractional Portion of Table with marginals Table 1d. Folded-in Table Table 1e. Rounding Cost for each Table Cell Table 1h. Initial Solution of CR Problem cost = 4.76 Table 1i. Optimal Solution of Transportation Matrix Table 1j. Optimal Solution of CR Problem cost = -6.56

5 APPENDIX B /* ctrlround */ /* Robert D. Sands, U.S. Bureau of the Census, Washington, DC, June 13, 2003 */ /* Controlled rounding program - SAS data set input */ /* This program performs the controlled rounding of an m x n matrix of real numbers. The general algorithm is based on a paper by L.H. Cox and L.R. Ernst (1982) "Controlled Rounding", INFOR (20)4. The algorithm solves the problem of rounding to adjacent integers of the real number entries in a tabular array such that the total change to the table cells is minimized while preserving the tabular structure of the input array. The problem is represented as a capacitated transportation problem as described in George B. Dantzig (1963) Linear Programming and Extensions, pgs The solution to the controlled rounding problem using SAS PROC NETFLOW was first created by Brian Richards and Jim Fagan (1997) at the U.S. Census Bureau and operated on a fixed size 7 x 7 input matrix. The current SAS program, which reads an arbitrary m x n matrix from a text file or a SAS data set and creates the output matrix and SAS data set, was compiled by Robert Sands (1998) at the Census Bureau. The first two data steps, which read in the arbitrarily sized matrix, were written by Timothy J. Braam (1998) at the Census Bureau. In addition, this Macro (Robert Sands, 2003) is designed to have functionality similar to that provided by the conventional SAS ROUND Function. This is to say that a round-off unit argument is provided which will produce a table rounded to the nearest integer, as well to the nearest ten, hundred, tenth or hundredth. */ ******************************************************************************************************* To run controlled rounding SAS macro : (A) If input table is text file (row(s) of numbers delimited by space(s) : %CTRLROUND (TEXTFILEIN=<full text file path, filename, extension>, DSOUT=<output data set name>); OR (B) If input table is SAS data set (requires column variables to be named : col1-col&cols) : %CTRLROUND (DSIN=<input data set name>, DSOUT=<output data set name>); ******************************************************************************************************* /* signd */ /* Find the most significant decimal digit (places to the right of the decimal point) in all the numbers in the tabular array contained in &ds */ %macro signd (ds=,maxwidth=,maxsignd=); data signd (keep = signd); retain signd; signd = 0; if _N_ eq 1 then do until (last); set &ds end=last; %do j = 1 %to &cols; decimals = substr(put(col&j,&maxwidth..&maxsignd),%eval(&maxwidth - &maxsignd + 1),&maxsignd); hold = &maxsignd; found = 0; /* Find most significant digit position in col&j */ do while (hold > 0 and not found); found = substr(decimals,hold,1) ne "0"; if not found then hold = hold - 1; /* do while (hold > 0) */ /* Check if col&j has a more significant digit than previous canidate */ if hold > signd then signd = hold; /* put " col&j = " col&j " hold = " hold " signd = " signd; */ % end /* do until (last) */; else do; stop;

6 run; %mend signd; /* prcprint */ %macro prcprint (lib=, ds=, vars=, state=); proc print data = &lib&ds&state heading = v width = min; title "&lib&ds&state"; format &vars ; run; title " "; %mend prcprint; %macro ctrlround (textfilein=, roundoff=, dsin=, dsout=); /* Determine whether input table is in the form of a (1) a text file or (2) a SAS data set */ %if &textfilein ne %then %do; data _NULL_ ; infile "&textfilein" end=done truncover obs=1 ; ; okay=1 ; do while (okay=1) ; input ; i + 1 ; if col lt 0 then do ; call symput('cols', left(put(i, 4.))) ; okay=0 ; end ; end ; run ; data set1; infile "&textfilein" end=done truncover ; input col1-col&cols ; run ; % %else %if &dsin ne %then %do; data set1; set &dsin; /* Change variable names to col1-col&cols */ run ; % /* %prcprint (ds=set1, vars=col1); */ /* Multply entries of input data set by inverse of the roundoff unit */ data set1 (drop = total); set set1 end=last; file PRINT; total + col1; %do n = 1 %to &cols; col&n = col&n * (1/&roundoff); % /* if last then put "total = " total 25.12; */ run ; /* Find the number in the input data set with the most significant digits to the right of a decimal point */ %signd (ds=set1,maxwidth=25,maxsignd=12);

7 /* Process matrix with input data table into data sets needed for the PROC netflow in order to do controlled rounding of the table */ proc iml; /* Read the most significant decimal digit for the numbers in the input data set */ use signd; read all var _num_ into signd; /* print "IML : signd = " signd; */ B = 10**signd; /* Put the base value into a data set */ create B var {B}; app /* Read data set with table of real numbers into a matrix */ use set1; read all var _num_ into table; tot = sum(table); /* Multply all input matrix enties by base value, B */ table = table*b; /* Determine the number of rows of the input matrix */ m = nrow(table); /* Put the number of rows value into a data set */ create m var {m}; app /* Determine the number of columns of the input matrix */ n = ncol(table); /* Put the number of columns value into a data set */ create n var {n}; app /* Define the total matrix to hold the input matrix with the marginal totals */ ttable = j(m+1,n+1,0); ttable[1:m,1:n] = table; /* Calculate row marginal totals */ do i = 1 to m ; ttable[i,n+1] = sum(table[i,]); /* Calculate column marginal totals */ do j = 1 to n ; ttable[m+1,j] = sum(table[,j]); ttable[m+1,n+1] = sum(table); /* Create data set of matrix with marginal totals */ create ttable from ttable; append from ttable; /* reset autoname; print "ttable = " ttable; */ /* Create the 'Folded-in' table from the total table */ ftable = j(m+2,n+2,0); ftable[1:m,1:n] = mod(table,b); /* Calculate row node and folded-in values */ do i = 1 to m ; ftable[i,n+2] = ceil(sum(ftable[i,1:n])/b)*b; ftable[i,n+1] = ftable[i,n+2] - sum(ftable[i,1:n]); /* Calculate column node and folded-in values */ do j = 1 to n ; ftable[m+2,j] = ceil(sum(ftable[1:m,j])/b)*b; ftable[m+1,j] = ftable[m+2,j] - sum(ftable[1:m,j]); ftable[m+1,n+2] = ceil(sum(ftable[m+1,1:n])/b)*b; ftable[m+1,n+1] = ftable[m+1,n+2] - sum(ftable[m+1,1:n]); ftable[m+2,n+1] = ceil(sum(ftable[1:m,n+1])/b)*b; /* Create data set of matrix with folded-in values */ create ftable from ftable; append from ftable;

8 /* reset autoname; print "ftable = " ftable; */ /* Create data sets for input to PROC NETFLOW */ /* Create arrays containing a supply node name and value representing each interior row (and one for the folded-in row) and a demand node name and value for each interior column (and one for the folded-in column) */ _node_ = j(m+1 + n+1,1,' '); _sd_ = j(m+1 + n+1,1,0); /* Load supply nodes name and value */ do i = 1 to m+1 ; _node_[i,1] = char(i,8); _sd_[i,1] = ftable[i,n+2]/b; /* Add identifying characters to supply nodes of node data */ /* Change all blanks to zeros */ call change (_node_," ","0",0); /* Change 1st 3 characters to 'row' */ call change (_node_,"000","row"); /* Load demand nodes name and value */ do j = 1 to n+1 ; _node_[m+1 + j,1] =char(j,8); _sd_[m+1 + j,1] = (ftable[m+2,j]/b) * -1; /* Add identifying characters to demand nodes of node data */ demnode = _node_[m+2:m+1+n+1,1]; call change (demnode," ","0",0); call change (demnode,"000","col"); _node_[m+2:m+1+n+1,1] = demnode; /* Create nodedata data set from arrays for input to PROC NETFLOW */ create noded var {_node sd_}; app /* Create arrays containing the arc information: from, to, cost, capacity */ _cost_ = j((m+1)*(n+1),1,0); _from_ = j((m+1)*(n+1),1,' '); _to_ = j((m+1)*(n+1),1,' '); _capac_ = j((m+1)*(n+1),1,0); capacity = 1; p = 2; do i = 1 to m+1; do j = 1 to n+1; _cost_[(i-1)*(n+1)+j,1] = (B - ftable[i,j])**p - (ftable[i,j])**p; _from_[(i-1)*(n+1)+j,1] = char(i,8); _to_[(i-1)*(n+1)+j,1] = char(j,8); _capac_[(i-1)*(n+1)+j,1] = capacity; /* Add identifying characters to _from_ and to _to_ nodes of arc data */ call change (_from_," ","0",0); call change (_from_,"000","row"); call change (_to_," ","0",0); call change (_to_,"000","col"); /* Create arcdata data set from arrays for input to PROC NETFLOW */ create arcd1 var {_from to cost capac_}; app quit; /* Produce solution of controlled rounding using node and arc data sets */ /* The _flow_ variable in the solution dataset output by PROC NETFLOW contains eith a 0 or a 1 for each cell on the real number matrix designating whether it should be rounded the floor integer or rounded up to the ceiling integer */ proc netflow nodedata = noded arcdata = arcd1 arcout = solution;

9 reset maxit1 = ; run; /* Output a portion of the controlled rounding solution data set */ /* proc print data = solution uniform heading = v width = min; sum _fcost_; title 'Controlled Rounding Solution'; var _from to flow cost fcost_; run; */ /* Interpret output of PROC NETFLOW and create rounded table */ proc iml; /* Get number of rows variable */ use m; read point 1; /* Get number of columns variable */ use n; read point 1; /* Get base variable */ use B; read point 1; /* Read input table real numbers into a matrix */ use ttable; read all var _num_ into ttable; ttable = ttable/b; /* Print input table of real numbers with marginal totals */ reset autoname; /* print ttable; */ /* Create matrix for rounded table */ rtable = j(m+1,n+1,0); /* Get solution values from PROC NETFLOW output */ use solution; read all var {_flow_} into flow; /* Interpret output of PROC NETFLOW and create rounded table */ do j = 1 to n+1; do I = 1 to m+1; if ((I <= m & j <= n) (I = m+1 & j = n+1)) then /* Create interior cells and total cell from _flow_ variable */ do; if (flow[(j-1)*(m+1)+i,] = 0) then rtable[i,j] = floor(ttable[i,j]); else /* flow = 1*/ rtable[i,j] = ceil(ttable[i,j]); else if ((I = m+1 j = n+1) & ^(I = m+1 & j = n+1)) then /* Create row and column marginal cells from _flow_ variable */ do; if (flow[(j-1)*(m+1)+i,] = 0) then rtable[i,j] = ceil(ttable[i,j]); else /* flow = 1*/ rtable[i,j] = floor(ttable[i,j]); /* Print (control) rounded table of integers */ reset autoname; ttable = ttable*(&roundoff); print ttable; rtable = rtable*(&roundoff); print rtable; /* Create data set from rounded matrix */ create &dsout from rtable; append from rtable; quit; %mend ctrlround;

10 /* Platform-dependent file directory path macro variable */ %let home = %str(c:\); /* PC Windows */ /* %let home = %str(/home/sands003/); */ /* Unix */ /* %let home = %str(dssd_se_sam:[r_sands]); */ /* Compaq Alpha */ %let textfilein = %str(matrix.txt); libname user "&home"; /* Input matrix example */ data _null_; file "&home&textfilein"; put " "; put " "; put " "; put " "; put " "; put " "; put " "; stop; run; /* Macro Invocation using input matrix above */ %ctrlround (textfilein=&home&textfilein, roundoff=1, dsin=, dsout=user.a);

11 APPENDIX C NODED Data Set OBS _NODE SD_ 1 row row row row row row row row col col col col col col col col

12 ARCD1 Data Set OBS _FROM TO COST CAPAC_ 1 row00001 col row00001 col row00001 col row00001 col row00001 col row00001 col row00001 col row00001 col row00002 col row00002 col row00002 col row00002 col row00002 col row00002 col row00002 col row00002 col row00003 col row00003 col row00003 col row00003 col row00003 col row00003 col row00003 col row00003 col row00004 col row00004 col row00004 col row00004 col row00004 col row00004 col row00004 col row00004 col row00005 col row00005 col row00005 col row00005 col row00005 col row00005 col row00005 col row00005 col row00006 col row00006 col row00006 col row00006 col row00006 col row00006 col row00006 col row00006 col row00007 col row00007 col row00007 col row00007 col row00007 col row00007 col row00007 col row00007 col row00008 col row00008 col row00008 col row00008 col row00008 col row00008 col row00008 col row00008 col

13 SOLUTION Data Set OBS _FROM TO COST FLOW FCOST_ 1 row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col row00001 col row00002 col row00003 col row00004 col row00005 col row00006 col row00007 col row00008 col Cost =

14