RANKINGS AND MULTI-CRITERIA EVALUATION OF ALTERNATIVES: USAGE OF OBJECTIVE DATA, SUBJECTIVE MEASUREMENTS AND EXPERT JUDGMENT

Transcription

1 RANKINGS AND MULTI-CRITERIA EVALUATION OF ALTERNATIVES: USAGE OF OBJECTIVE DATA, SUBJECTIVE MEASUREMENTS AND EXPERT JUDGMENT Pavel Brusilovskiy and Robert Hernandez Risk Management, Conrail Corporation Abstract There are many problems that require full utilization of all available and useful information in order to achieve a comprehensive and systemic evaluation of alternatives. These sources are objective data, subjective measurements (i.e. survey), and expert judgment. Each type of information is reflected in the ordinal scale as an individual ranking (or rating) of the alternatives. Two different cases are considered: a) all individual rankings are treated equally important; b) each individual ranking has its own ranked weights. In the latter case the weights reflect in corresponding ordinal scale of a decision maker s preferences (confidence, reliability and/or importance) of the different types of information obtained. The paper is dedicated to better management decisions through the aggregation of different types of information. Introduction Ranking within uni-criterion evaluation of alternatives is a frequently used approach to assess performance. Some common examples of this approach are to assess the best books to read through the use of the best selling books list or to evaluate salespeople s performance based on number of sales orders. The uni-criterion evaluation for these uncomplicated issues offers a fair performance representation. However, many evaluation problems are complex and require a multi-criteria evaluation approach instead of a uni-criterion technique. Complex issues necessitate the utilization of all available and useful sources and types of information. Generally, they are objective data, subjective measurements and expert judgment. Each type of information offers valuable insight to the issues under consideration, but independently none of them provide decision makers with complete information. The goal of this article is to present a simple and general approach to multi-criteria evaluation of alternatives for non-statisticians. This approach is valuable for integrating qualitative and quantitative information in diverse fields, and in particular related to quality performance, marketing and risk. For example: Quality Of Health Care Among Hospitals There are many sources of information to evaluate the quality of health care among hospitals. Objective data could be treatment success measures and corresponding cost related measures. Subjective measurements consist of special patient survey results. Expert judgment can be obtained from Quality Health Care professionals. Market Segmentation and Product Positioning There are many sources of information to evaluate existing products for each market segment. Objective data could be related to product attributes. Subjective measurements are consumers preferences, and expert judgment can be provided by professionals in appropriate field. A Risk Assessment and Risk Reduction example should clarify the importance of viewing the information at a comprehensive level. Risk Assessment and Risk Reduction Example Let s examine a risk management problem associated with the personal injury risk assessment in a large industrial company. Our major issue is to identify the safety districts with the highest degree of risk. An assessment procedure must take into account the two dimensions of risk. The first dimension reflects likelihood of occurrence of unfavorable event and the second dimension reflects consequences of this event [6]. Data and information sources for these two dimensions consist of three possible types which are objective data, subjective measurements and expert judgment. Each type of information offers valuable insight to a company s risk exposure, but independently none of these types of information can provide decision makers with a comprehensive risk exposure background. 1

2 Objective data Degree of risk can be characterized for each safety district in the company by several objective attributes: Personal injury rate reflects the probability of unfavorable event occurrence (first dimension of risk) Average settlement cost reflects medical and litigation expenses (one component associated with the second dimension of risk) Average amount of days disabled reflects lost manhours (another component associated with the second dimension of risk) Subjective measurements Important aspects of the personal injury process are safety training, work organization, behavioral features including inclination to risk and management-employee relations. Employee surveys can be used to evaluate the personal injury process. Subjective measurements as corresponding scores gauge important insight of personal injury process from employees point of view. This type of information represents local and subjective vision of the personal injury process, who are not experts in Risk Management. Expert Judgment Experts are individuals that have a special education, deep knowledge, and possess global vision of the personal injury process at a company level and within each safety district. They are able to take in account production or service technology, workers union affiliation structure, geographical location of a safety district, safety culture, management style and other latent conditions not found in objective data and subjective measurements. From this global perspective, the experts would rank safety districts according to the degree of personal injury risk. Each of these attributes or criteria are different and relatively independent components of the personal injury process. Separately, none of these criterion have the capability of providing decision makers with a comprehensive reflection to the degree of personal injury risk for each safety district. An adequate approach has to combine information from different sources. Our approach described in the following sections fulfills this requirement and is practical and non-consuming. Original Data Description The appropriate objective data, subjective measurements and expert judgment were collected for 14 safety districts within the company. Objective Data are presented by three interval-scaled variables INJ_RATE, SET_COST and DIS_DAYS. INJ_RATE measures number of injuries in a safety district for the last two years divided by total labor hours for the last two years. SET_COST measures average settlement cost among personal injuries in each safety district for the last two years. DIS_DAYS measures average number of days disabled in each safety district for the last two years. Subjective measurement data are depicted by one ordinally-scaled variable EMP_RATG. This variable measures average safety satisfaction for all employee responses in each safety district. The safety satisfaction is estimated on 100 points scale and the question is simply, How you would rate your satisfaction with safety where you work? The higher satisfaction rate a safety district has the less degree of risk associated with that district. Note: Ordinal scale means there is an ordering to levels, but distance between adjacent levels is not distinct. Expert judgment data are characterized by one ordinally scaled variable EXP_JUDG. This variable represents expert s safety district ranking. The higher rank a safety district has the greater degree of risk associated with that district. Raw Data We read the raw data into a SAS data set named PI_ASSES: libname risk c:\ INJURY ; options nodate linesize=76; data risk.pi_asses; input sftydist $ 25. inj_rate set_cost dis_days emp_ratg exp_judg; datalines; CAR SHOP LOCOMOTIVE SHOP LOCOMOTIVE ENGINEERS ; proc print data=risk.pi_asses; title Original Data For Risk Assessment ; These data and proc steps produced Output 1 on Page 3. 2

3 OUTPUT 1 Original Data For Risk Assessment OBS SFTYDIST INJ RATE SET COST DIS DAYS EMP RATG EXP JUDG 1 CAR SHOP LOCOMOTIVE SHOP MAINTENANCE OF WAY STORE ROOM MECHANICAL ENGINEERING INDUSTRIAL ENGINEERING CAR INSPECTORS LOCOMOTIVE INSPECTORS CIVIL ENGINEERING ROAD SIGNAL DATS GANG BRIDGE INSPECTORS TRACK GANG LOCOMOTIVE ENGINEERS Conversion to Rankings Most numeric variables in the risk.pi_asses data set will be converted to rank variables. The values of a rank variable are ranks, from 1 to n. To perform this conversion process, we need to exploit PROC RANK from the SAS Base software: /* Rank conversion for variables INJ_RATE, */ /* SET_COST, DIS_DAYS and EXP_JUDG */ /* is performed according to the rule: Original variable*/ /* values will be ranked from smallest to largest, */ /* assigning the rank 1 to the smallest value, 2 to the */ /* next smallest, and so on up to rank n */ /* *******************************************/ proc rank data =risk.pi_asses out=pi_rank1; ranks r_injury r_cost r_days r_judg; var INJ_RATE SET_COST DIS_DAYS EXP_JUDG; /* Rank Conversion for variable EMP_RATG is */ /* according to the rule: Original variable values will */ /* be from largest to smallest, assigning rank 1 to */ /* the largest value, 2 to next largest, and so on up to */ /* rank n */ proc rank data=risk.pi_asses out=pi_rank2 descending; ranks r_emp ; var EMP_RATG ; /* Merge pi_rank1 and pi_rank2 datasets into one */ /* dataset named risk.pi_ranks */ data risk.pi_ranks; title 'Personal Injury Rankings Data'; merge pi_rank1 pi_rank2; /* Prefix R indicates rank variable */ keep sftydist r_injury r_cost r_days r_emp r_judg; proc print data=risk.pi_ranks; These data and proc steps produced Output 2 on Page 4. 3

4 OUTPUT 2 Personal Injury Rankings Data OBS SFTYDIST R INJURY R COST R DAYS R JUDG R EMP 1 CAR SHOP LOCOMOTIVE SHOP MAINTENANCE OF WAY STORE ROOM MECHANICAL ENGINEERING INDUSTRIAL ENGINEERING CAR INSPECTORS LOCOMOTIVE INSPECTORS CIVIL ENGINEERING ROAD SIGNAL DATS GANG BRIDGE INSPECTORS TRACK GANG LOCOMOTIVE ENGINEERS Decision Making a) Aggregation of individual rankings: each criterion is equally weighted in importance Reviewing output 2, we can see that the Store Room safety district has the lowest degree of personal injury risk according to the injury rate (R_INJURY) and employee rate (R_EMP). But according to the criterion disabled days (R_DAYS), this district has a high level of personal injury risk compared to the other safety districts. The two other criteria R_COST and R_JUDG have lower ranks than R_DAYS, but higher ranks then R_INJURY and R_EMP. This is a common situation when a value for one criterion contradicts a value of another one. To overcome the contradiction problem, we need to employ one of the rankings aggregation method. A simple but rather useful method for this problem is the average ranking technique. The technique is a two step process when we are under the assumption that importance is equally weighted for each criterion. First step is to calculate average rank for each alternative (safety district) and then the second step is to construct their final multi-criteria rankings. data final1; set risk.pi_ranks; avgrank1=mean( r_injury, r_cost, r_days, r_emp, r_judg); keep sftydist avgrank1; 4 proc rank data=final1 out=risk.pifinal1; title Aggregation of Individual Ranking: Equal Weights of the Criteria ; ranks r_final1; var avgrank1; proc print data=risk.pifinal1; This data and proc steps produce the Output 3. OUTPUT 3 Aggregation Of Individual Ranking: Equal Weights Of The Criteria OBS SFTYDIST AVGRANK1 R FINAL1 1 CAR SHOP LOCOMOTIVE SHOP MAINTENANCE OF WAY STORE ROOM MECHANICAL ENGINEERING INDUSTRIAL ENGINEERING CAR INSPECTORS LOCOMOTIVE INSPECTORS CIVIL ENGINEERING ROAD SIGNAL DATS GANG BRIDGE INSPECTORS TRACK GANG LOCOMOTIVE ENGINEERS The final multi-criteria ranking in Output 3 shows that the Bridge Inspector safety district has lowest degree of risk and Industrial Engineering safety district has the highest degree of risk. There are also three safety districts, Locomotive Shop, Maintenance of Way and Track

5 Gangs, with the same degree of personal injury risk (tied ranks). The ranking approach is more than just recognizing who is the best and the worst. The intent of this approach is to provide decision makers with a comprehensive (multi-criteria) view to make better decisions. In output 3, the multi-criteria ranking information clearly illustrate that Industrial Engineering and Road Signal have the highest overall personal injury risk. For a substantial reduction in overall personal injury risk, management must at least have some action plans associated with the several safety districts holding the highest overall personal injury risk. b) Aggregation of individual rankings: each criterion has its own weight In the previous paragraph, we considered each criterion to be equally weighted in importance. Let us assume in the next example that each criterion holds different levels of importance or priority. Decision Maker may decide that operating cost has the highest priority within the company and prioritize the criteria accordingly. In our example, this operating cost initiative would mean that we would place higher priorities on settlement costs and disabled days. If technology is the number issue then we would place a higher priority on Expert Judgment. The information below is based on operating cost as the highest priority for the company and is provided by Decision Maker: Table 1. Decision Maker Priorities of Criteria Criterion Priority INJ_RATE 4 SET_COST 1 DIS_DAYS 2 EMP_RATG 3 EXP_JUDG 5 Note: Criteria with lower rank has a higher priority. This information can be converted to weights w(i) according to the rank-sum rule [1]: n+ 1 i 2*( n+ 1 i) wi () = = j n*( n+ 1) i=1,...,n For our example n=5, and DIS_DAYS has a priority of 2. Its calculated weight is the following: w( 2) = 2*( ) = *( 5+ 1) To calculate weights using formula (1), we need to submit the following code: data weights; title 'Criteria Weights'; do priority =1 to 5; w=2*(5+1-priority)/(5*6); output; end; proc print data=weights; The results are presented in Output 4. OUTPUT 4 Criteria Weights OBS PRIORITY W Now we need to calculate linear combinations of all rankings in Output 2 (Page 5) with the criteria weights in Output 4. The results are the weighted averages from the criteria rankings. The next step will encompass the conversion of the weighted averages from the individual criteria rankings to multi-criteria ranking. We will again use the RANK procedure to obtain the multi-criteria ranking. data risk.final2; set risk.pi_ranks; array weight(5) w1-w5; array r_criter(5) R_COST R_DAYS R_INJURY R_EMP R_JUDG; where n is the number of criteria, i is a priority of ith criterion. (1) (See the rest of code on the next page) 5

6 (continued from previous page) /* The order of the R_variables in the array statement */ /* array r_criter(5) has to correspond to the ranks of the*/ /* criteria in Table 1. */ avgrank2=0; do priority=1 to 5; weight(priority)=2*(5+1-priority)/(5*6); avgrank2= weight(priority)*r_criter(priority) + avgrank2; end; keep sftydist avgrank2; proc rank data=risk.final2 out=risk.pifinal2; ranks r_final2; var avgrank2; proc print data=risk.pifinal2; title 'Aggregation of Individual Rankings: Nonequal Weights of the Criteria'; OUTPUT 5 Aggregation Of Individual Rankings: Nonequal Weights Of The Criteria OBS SFTYDIST AVG_RANK2 R_FINAL2 1 CAR SHOP LOCOMOTIVE SHOP MAINTENANCE OF WAY STORE ROOM MECHANICAL ENGINEERING INDUSTRIAL ENGINEERING CAR INSPECTORS LOCOMOTIVE INSPECTORS CIVIL ENGINEERING ROAD SIGNAL DATS GANG BRIDGE INSPECTORS TRACK GANG LOCOMOTIVE ENGINEERS Naturally, the final ranking from R_FINAL2 depends on the decision maker s priorities. We can determine the correlation between equally weighted and non-equally weighted rankings. R_FINAL2 strongly correlates with R_FINAL1 on Output 2 (Page 6). The Spearman s correlation coefficient between these two variables is Below is the SAS coding to check the correlation between the two final ranking variables. In our example, multi-criteria rankings have basically the same rank order. However, other priorities may have multi-criteria rankings that do not strongly correlate with one another. If this is the case then priority information from the decision maker can be essential for multi-criteria ranking. Comparing equally weighted and non-equally weighted rankings improve the decision maker s understanding of the complex issues in terms of priority and ranking data allfinal; merge risk.pifinal1 risk.pifinal2; keep r_final1 r_final2; proc corr data=allfinal spearman; var r_final1; with r_final2; Discussion In this section we will discuss advantages and disadvantages of the approach described, some generalization and related literature. Advantages The approach works well also in the present of rank ties. Rank ties are interpreted as the same level of importance for decision maker priorities and/or criteria preferences for alternatives under consideration. There might be a requirement to add more information from several experts and/or subjective attributes. This approach easily can take in account this modification. A more sophisticated approach like path analysis can be applied to objective analysis of subjective measurements. Several path coefficients could be included in the multi-criteria ranking process of the alternatives. See [3] for more details on path analysis. Disadvantage Our general concern with the conversion process is the lost of some useful information through ranking. In the example, all variables except for EXP_JUDG variable can be less informative after the conversion process. The EXP_JUDG variable is measured in rank scale and there is no loss of information at all. Loss of information during the conversion process mainly depends on the degree of accuracy and objectivity of the measured variable in its primary scale. In our case, interval scaled variable INJ_RATE is measured without any random error and therefore is considered to be 100 percent accurate. The high accuracy level may lead to some shrinkage of useful information during the conversion process. Interval scaled variables SET_COST and DIS_DAYS have a subjective component related to 6

7 plaintiff compensation practices in local jurisdictions. This subjective component in these variables lower the objectivity level. Conversion of these variables will probably have no deterioration of information due to the low degree of measurement objectivity. Overall, we intentionally sacrifice some information but in return gain more simplicity to work with multi-criteria issues. Optimal properties of multi-criteria ranking The multi-criteria ranking possesses the following two properties: The mean value of Spearman correlation coefficients between each criterion ranking in Output 2 and final ranking (R_FINAL1) in Output 3 is maximum. The multi-criteria ranking is also optimal as a least square estimator. See footnote [5] for more details Other approaches to multi-criteria ranking There are several different approaches to multi-criteria ranking. Another popular approach of multi-criteria evaluation in ranking scale is the so called axiomatic one. It is based on the concept of Kemeny median. Difference between the approach described in this paper and the axiomatic approach is approximately the same as the difference between mean and median in the location parameter estimation problem. See [4] for more details. Weights and Priorities Several different methods are developed to take criteria priorities into account. To calculate weights w(i), we have used one of the simplest formula. See [1] for comparison of different weighting methods. Criteria rankings concordance In order to test a concordance among separate criteria rankings, it is necessary to use Kendall's W coefficient of concordance. The SAS system does not provide this test but it can be easily performed within the SAS system independently See [6] for more detail on Kendall s W coefficient of concordance. Objective data and expert judgment References 1. Barrett Bruce E.and F.H. Barron, "Decision Quality Using Ranked AttributeWeights," Management Science, November 1996, pp Brusilovskiy Pavel.M. and Tilman Leo.M., "Incorporating Expert Judgement into Multivariate Polynomial Modeling," Decision Support Systems, Vol. 18 (1996), pp Hatcher, Larry. A Step-by-step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling. Cary, NC: SAS Institute Inc., Kemeny J.G. and Snell J.L. Mathematical Models in the Social Sciences. New York: Blaisdell Publishing Company. 5. Kendall,Maurice.G. Rank Correlation Methods. Griffin London, Moore, Peter G. The Business of Risk. Great Britian: Cambridge University Press, SAS Institute Inc., SAS Procedures Guide, Version 6, Third Edition, Cary, NC: SAS Institute Inc., 1990, pp 750. Author Contacts Pavel Brusilovskiy Conrail Corporation 2001 Market Street 6-D P.O. Box Phildelphia, PA (215) pbrusilovs@aol.com Robert Hernandez Conrail Corporation 2001 Market Street 14-C P.O. Box Philadelphia, PA (215) RHer1@aol.com See [2] for a more sophisticated usage of objective data and expert judgment. 7