Pairwise Comparisons

Transcription

1 Version for CS/SE 3RA3 Ryszard Janicki Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada Ryszard Janicki 1/12

2 Weights and Ranking Subjective Weights Weights or weighted parameters/attributes are part of most measurement and/or classification techniques. However when judgments are subjective, weight assignment and its consistency is problematic. A ranking or preference is usually defined as a weakly ordered relationship between a set of items such that, for any two items, the first is either less preferred, more preferred or indifferent to the second one. The ranking is numerical if numbers are used to measure importance and to create the ranking relation. Consistent subjective ranking is problematic and assigning numerical importance is even more problematic. Ryszard Janicki 2/12

3 Subjective Weights The pairwise comparisons method is based on the observation that it is much easier to rank the importance of two objects than it is to rank the importance of several objects. This very old idea goes back to Ramon Llull in the end of XIII century. Its modern version is due to 1785 influential paper by Marquis de Condorcet, where he used this method in the election process where voters rank candidates based on their preference, and 1860 paper by Fechner. Modern version follows from Thurstone (1927) and Saaty (1977), and it is one of many methods used in multicriteria decision making and analysis. Ryszard Janicki 3/12

4 Analysis Let C 1,..., C n be a finite set of objects (called criteria, attributes, etc.) to be judged and/or analyzed. The relationship between features may be qualitative (relational) or quantitative (numerical). A quantitative relationship between features C i and C j is represented by a number a ij. We assume a ij > 0 and a ij = 1 a ji, for i, j = 1,..., n (which implies a ii = 1 for all i). If a ij > 1 then C i is more important (preferred, better, etc.) than C j and a ij is a measure of this relationship (the bigger a ij, the bigger the difference), if a ij = 1 then C i and C j are indifferent. The matrix of such relative comparison coefficients, A = [a ij ] n n, is called a pairwise comparison matrix. Ryszard Janicki 4/12

5 Analysis Table : Non-linear comparison scale proposed by Janicki and Zhai in Value and range of a ij relation Definition of intensity range starting value symbol or importance (C i vs C j ) C i C j indifferent C i C j slightly in favour C i C j in favour C i > C j strongly better C i C j extremely better Ryszard Janicki 5/12

6 Analysis Since the features C 1,..., C n characterize the same entity, the values of a ij should somehow be consistent. A pairwise comparison matrix A = [a ij ] n n is consistent if and only if, for i, j, k = 1,..., n, a ij a jk = a ik. Saaty s Theorem states that a pairwise comparison matrix A is consistent if and only if there exists positive numbers w 1,..., w n such that a ij = w i /w j, i, j = 1,..., n. The values w i are unique up to a multiplicative constant. They are called weights, interpreted as a measure of importance and often scaled to w w n = 1 (or 100%). In practice, the values a ij are very seldom consistent, so some measurements of inconsistency are needed. Ryszard Janicki 6/12

7 Inconsistency Index Analysis Saaty in 1977 proposed an inconsistency index based on the value of the largest eigenvalue of A. However this method does not give any clue where most inconsistent values of A are located, so we will not use it. In 1993, Koczkodaj provided a distance-based inconsistency index, which is based on the analysis of all triads a ij, a jk and a ik from A = [a ij ] n n. and defined as follows: ( ( cm A = max min 1 a ij (i,j,k) a ik a kj, 1 a )) ika kj a ij In this case the most inconsistent triad is localized. We will use this index for our purposes in this paper. Ryszard Janicki 7/12

8 Lowering Inconsistency Analysis Acceptable levels of inconsistency depend on the inconsistency index definition and particular interpretation of C 1,..., C n. When distance-based inconsistency index is used, since the biggest troublemakers are localized, we can improve consistency step by step, by small changes of values of the triple that results in the maximal inconsistency index. Dependently on the area of application one may either try to reduce the inconsistency index to zero or to some acceptable level. Ryszard Janicki 8/12

9 Weights and Inconsistency Analysis There are two popular methods for deriving a suitable value w i from an inconsistent, but with acceptable level of inconsistency, matrix A. 1 The weights w 1,..., w n are defined as the principal eigenvector of the matrix A. 2 The weights w 1,..., w n are defined as the geometric means of columns (or equivalently, rows) of the matrix A, i.e., w i = n n j=1 a ij, for i = 1,..., n. The geometric means are used in this paper. For small values of cm A, the differences are negligible anyway. Ryszard Janicki 9/12

10 Initial Judgments and Weights Initial Weights Final Weights Table : Initial qualitative judgments of key attributes from ISO/IEC 9126 standard Attribute Name C 1 C 2 C 3 C 4 C 5 C 6 Functionality C 1 > Reliability C 2 > > Usability C 3 Efficiency C 4 Maintainability C 5 < < Portability C 6 < Table : Initial quantitative judgments. Attribute Name C 1 C 2 C 3 C 4 C 5 C 6 Functionality C Reliability C Usability C 3 1/2.6 1/ / Efficiency C 4 1/2.6 1/ Maintainability C 5 1/4.7 1/ / Portability C 6 1/7.0 1/4.7 1/1.6 1/ inconsistency coefficient cm A = 0.46, unacceptable as greater than 0.3. Ryszard Janicki 10/12

11 Final Weights Table : Consistent (i.e. with acceptable inconsistency) pairwise comparisons matrix. Attribute Name C 1 C 2 C 3 C 4 C 5 C 6 Functionality C Reliability C 2 1/ Usability C 3 1/3.25 1/ / Efficiency C 4 1/2.5 1/ Maintainability C 5 1/4.85 1/4.1 1/1.5 1/ Portability C 6 1/5.9 1/4.95 1/1.8 1/2.34 1/ weights of criteria w 1 w 2 w 3 w 4 w 5 w 6 values 34% 29% 11% 14% 7% 6% inconsistency coefficient cm A = < 0.3, practically consistent. Table : Final qualitative judgments of key attributes. Corrected values are red. Attribute Name C 1 C 2 C 3 C 4 C 5 C 6 Functionality C 1 > > > Reliability C 2 > > Usability C 3 < Efficiency C 4 Maintainability C 5 < < Portability C 6 < <

12 This study demonstrates how the use of theoretical results to relate such intangible software attributes as resilience or reusability can result in computing more adequate and trustworthy weights for software evaluation. Ryszard Janicki 12/12