On the Evaluation of Boolean Operators in the Extended Boolean Retrieval Framework

Transcription

1 On the Evaluation of Boolean Operators in the Extended Boolean Retrieval Framework Joon Ho Lee, Won Yong Kim, Myoung Ho Kim and Yoon Joon Lee Department of Computer Science and Center for Artificial Intelligence Research Korea Advanced Institute of Science and Technology 373-1, Kusung-dong, Yusung-gu, Taejon, , Korea ABSTRACT The retrieval models based on the extended boolean retrieval framework, e.g., the fuzzy set model and the extended boolean model have been proposed in the past to provide the conventional boolean retrievat system with the document ranking facility. However, due to undesirable properties of evaluation formulas for the AND and OR operations, the former generates incorrect ranked output in certain cases and the latter suffers from the complexity of computation. There have been a variety of fuzzy operators to replace the evaluation formulas. In this paper we first investigate the behavioral aspects of the fuzzy operators and address important issues to affect retrieval effectiveness. We then defiie an operator class called positively compensatory operators giving high retrieval effectiveness, and present a pair of positively compensatory operators providing high retrieval efficiency as well as high retrieval effectiveness. All the claims are justifkxt through experiments. 1. INTRODUCTION Information Retrieval (IR) systems represent, store and retrieve the input information, which is likely to include the natural language text of documents or of document exceqms and abstracts [Bart85]. The output of IR systems in response to a search request consists of references. These references are intended to provide the system users with the information about items of potential interest. A major role of IR systems, however, is not just to generate a set of relevant references, but to help determine which documents are most likely to be relevant to the given requirements. IR systems should present to users a sequence of documents ranked in decreasing order of query-document similarities. Users are able to minimize their time spent to find useful information by reading the top-ranked documents fust. Permission to copy without fee all or part of thk material is granted provided thet the oopies are not made or distributed for direct commercial advantage, the ACM copyright notica and the title of the publication and its date appear, and notice is given that copying ie by permission of tha Association for Computing Machinery. To copy otherwise, or to republish, raquires a fea and/or specific permission. ACM-SlGlR 93-6/93 /Pittsburgh, PA, USA a 1993 ACM /93/0006/ $l.50 Boolean retrieval systems have been most widely used among commercially available IR systems due to efficient retrieval and easy query formulation. In conventional boolean retrieval systems, however, document ranking is not supported and similarity coefficients cannot be computed between queries and documents. The fuzzy set model [Bue181, Rade79, Sach76, Watt79] and the extended boolean model [Salt83, Satt85, Satt89] have been proposed to overcome this problem. They are logical extensions of the boolean retrieval system because they reduce to the boolean model when document term weights are restricted to zero or one, In this paper we do not consider query weighting schemes and focus on evaluation formulas for the AND and OR operations. When disregarding query weights, the fuzzy set model and the extended boolean model can be explained within the same framework, which will be called the Extended Boolean Retrieval Framework (EBRF). Formally, an IR system based on EBRF is defined by the quadruple <T, Q, D, F>, where (1) (2) (3) (4) T is a set of index terms that are used to represent queries and documents. Q is a set of queries that can be recognized by the system. Each query q e Q is a legitimate boolean expression composed of index terms and logical operators AND, OR and NOT, D is a set of documents. Each document d e D is represented by { (tl, W1),... (tn, wj} where wi designates the weight of term ti in d and wi may take any vatue between zero and one, i.e., O < wi <1. F is a retrieval function F: DxQ+[O, l] which assigns to each pair (d, q) a number in the closed interval [0, 1]. This number is a measure of similarity between the document d and the query q, and is called the document value for document d with respect to query q. The retrievat function F(d, q) is defined as follows: i) For each term &in a query, the function F(d, ti) is ii) defined as the weight of ti in document d, i.e., wi. Logical operators are then evatuated by applying the corresponding formulas. For instance, the fuzzy set model uses the F operators given in 291

2 Figure 1 and the extended boolean model uses the E operators given in Figure 2. For boolean queries containing more than one logical operator, the evaluation proceeds recursively from the innermost clause. (Fm) F(d, tl AND tz) = MIN (F(d, t,), F(d, t2)) FOR) F(d, tl OR tz) = MAX Old, tl), F(d, tz)) (FN(x) F(d, NOT tl) = 1- F(G tl) Figure 1. Similarity evaluation formulas in the fuzzy set model (EM) F(d, tl AND tz ) 1 ~(1-F(Ltl))p + (1- F(d,t,))p 1 =. [ 2 1lip F(d, tl )P + F(d, tz )P (EOll) F(d, tl OR tl) = 2 [ (ENo~) F(d, NOT tl) = 1- F(d,tl) Figure 2. Similarity evaluation formulas in the extended boolean model Though the fuzzy set model and the extended boolean model are elegant approaches, they have some problems due to undesirable properties of evaluation formulas for the AND and OR operations as follows: (i) The fuzzy set model has been criticized to generate incorrect ranked output in certain cases because the MIN and MAX operators have properties adverse to retrieval effectiveness [Book80, Lee93]. (ii) Although the extended boolean model has overcome the problems of the fuzzy set model by utilizing the E- and EOR OPtX%ttO13,it SufferS from the computation complexity of the operators [Salt89]. retrieval effectiveness when they are adopted in EB RF to evaluate boolean operators. We also describe that behavioral aspects of positively compensatory operators are suitable for achieving high retrieval effectiveness. In section 4 we evaluate retrieval effectiveness and efficiency of the proposed scheme. Finally concluding remarks are given in section CLASSIFICATION OF FUZZY OPERATORS We utilize the operator graph that is useful to analyze the behavioral properties of various operators. The operator graph is one way of representing the characteristics of an operator Lee92]. The operator graph is constructed for the given two operand graphs. The operator and operand graphs are represented by lines and dotted lines, respectively. Figure 3 shows how the operator graph is constructed for the operand graphs A and B, where the verticzd axis denotes the degree of membership and the horizontal denotes a set of objects. The operator graph is a set of points that are computed by applying the operator to the values of A and B at each element in the set of objects. For example, a point y is computed by applying the operator to the values of two operands et and ~. AB Figure 3. Creating the operator graph Since the first introduction of fuzzy set theory [Zade65], a variety of fuzzy operators have been proposed for the AND and OR operations. We first investigate their operational properties by using operator graphs, and address important issues to affect retrieval effectiveness. We then define an operator class called positively compensatory operators and propose to use them in EBRF for the AND and OR operations. We present two pairs of positively compensatory operators; one is A~,AN~ and Ad,oR and the other is EAND and EOR. Note that EAND and EOR are the operators of the extended boolean model. We also show through experiments that positively compensatory operators give higher retrieval effectiveness than the others and the Ad operators provide similar retrieval effectiveness and higher retrieval efficiency in comparison with the E operators. The remainder of this paper is organized as follows. Section 2 gives classification of fuzzy operators. In section 3 we analyze the effect of various fuzzy operators on The development of fuzzy set theory has allowed researchers to apply set theoretic concepts to sets of objects whose membership values vary in the interval [0, 1]. This applicability has been achieved by defining new operators for classical set theory operators. Often there is more than one fuzzy operator corresponding to a given classical operator, and the different operators have different operational characteristics. A variety of fuzz y operators have been developed for the AND and OR operations in the literature. They are classified into two groups such as T- Tolerators and averaging operators [Zimm91], which will be denoted by T and A, res~ctively, for simplicity. The T-operators, namely T-norms and T-conorms originated from the studies of probabilistic metric spaces. Later, it was proposed that T-norms and T-conorms could be used for the AND and OR operations, respectively, in the fuzzy set theory. The MIN operator belongs to T-norms 292

3 T-norm [AND] T-conorm [OR] F MIN(x, y) MAX(X, y) T1 x. y X+y-xy T2 MAX(x+y -1, O) MIN(x + y, 1) (Al ) (x+ y -x y) (x y)( - ) (A2) 7. MAX(X, y) + (1-7) MIN(x, y) (As) yo(x+y-xoy) + (1-y) o(xoy) (1-y) (x+y) (Al.-) Y M~(x, y) + (1-y; (x+y) (A4.0~) 1 MAX(X, y) + Xy x+y-2xy (a) The averaging opemtors (O<@21) X+y-xy l-xy x ify=l x ify=o (PO.8) (7=0.8) T4 y ifx=l y ifx=o 3 {[ O otherwise 1 otherwise (a) The T-operators T F.OR Al (y=o.2) (TO.8) &.0R(@Q5) A F.AN AA 1. 2.AN T T 3.OR 4.OR 3.AN.A AA ~ (b) The operator graphs for the T-operators In fuzzy decision theory the decision has been viewed as the intersection or the union of fuzzy sets, and T- Tolerators have been used to model human decisions in many cases. However, it has been noted that neither T- norms nor T-conorms are appropriate to model managerial decisions. Averaging operators have been developed to overcome this problem Wem88, Zimm80]. In averaging operators the resulting value is controlled by a parameter y. In Figure 5 three general operators, i.e., A1-A3 and a pair of operators distinguishing the AND operator, i.e., &.m and the OR operator, i.e., Aq,o ~ are shown with the corresponding operator graphs. T 1.OR T 2.OR (b) The operator gmphs for the averaging operators Figure 5, The averaging operators and the corresponding operator graphs Figure 4. The T-operators and the corresponding operator graphs and the MAX operator belongs to T-conorms. Figure 4 shows some T-operators and their operator graphs. (For more T-operators, see [Gupt91].) 3. ANALYSIS OF FUZZY OPERATORS FOR HIGH RETRIEVAL EFFECTIVENESS In the remainder of this paper, EBRFOP denotes the retrieval model that uses the operator op in EBRF to evaluate the AND and OR operations. Note that EBRFF and EB RFE are equivalent to the fuzzy set model and the extended boolean model, respectively. 293

4 3.1 Critics against T-Operators EBRF~ generates in certain cases incorrect document rankings not to agree with humans intuition [Book80, Lee93]. This is beeause the MINI and MAX operators give the resulting value that depends on only one operand without considering the other. The problem resulting from the single operand dependency is illustrated in the following example. Although we explain only problems incurred by the AND operation, it must be noted that the OR operation causes similar problems. Sinde Ooerand Deoe ndencv Problem : Suppose that we have two documents are represented dl and dz shown below. The dcmnnents by two pairs of an index term and its weight. dl = { (Thesaurus, 0.40), (Clustering, 0.40) } k= { (Thesaurus, 0.99), (Clustering, 0.39)} ql = Thesaurus AND Clustering When the MIN operator is used for the AND operation, the document values of dl and d2 for the query ql are evaluated as 0.40 and 0.39, respectively. Hence, dl is retrieved with a higher rank than dj. Most people, however, will obviously deeide that dz rather than dl is more similar to ql. The T-operators except MIN and MAX have the following common properties [Zimm91]: (i) When one operand value of the two is zero or one, the resulting value is equal to one of two operand values. (ii) They allow some compensation between two operand values in other cases, and the resulting value is less than the lower value of the two, or greater than the higher value. Although they are used in EBRF, the first common property still causes the single operand dependency problem. The second common property, i.e., compensatory effect, on the other hand, alleviates the problem. For example, when the product operator replaces the MIN operator in the above example, the document values of dl and dz are evaluated as 0.16 and 0.39 respectively, and hence d2 is retrieved with a higher rank than dl. However, the compensated value, that is less than the lower value or greater than the higher value, may result in the additional problem called negative compensation. ~ative Compensation Problem : Suppose a document d3 and two queries ql and q? are given as follows: d~= { (Thesaurus, 0.70), (Clustering, 0.70), (System, 0.70) } ql = Thesaurus AND Clustering ~= System Using other types of T-norms except the MIN operator will decide that dq is more similrw to qz than ql. For instance, if the product operator is applied, the document value of dg is 0.49 for ql and 0.70 for qz. Note that the similarity between ql and d~ is less than that between qz and d~, which clearly does not agree with most people s decision. 3.2 Suitability of Positively Compensatory Operators Four averaging operators have been described in section 2. In this section we investigate their behavioral properties and propose to use the averaging operatom A2 and A. in EBRF for the evaluation of boolean operators. We first indicate that the averaging operators Az and & are mathematically equivalent though they are independently developed by different researchers at different time. The distinction of the AND and OR operations separates the averaging operator A2 into two parts as follows: (A2.ANo) TMAX(x,y) + (1-y)*M~(x,y), OSySO.5 (z42.ofj ~MAX(x,y) + (1-y)@M~(x,y), 0.5S7< 1 In order to coincide the value range of the parameter of A2W with that of the parameter of A2,0R, we change the operator A2.m to a different form having the same value. By replacing y with 1-y, we obtain the following expression. (A2 ~) ~MIN(x,y) + (1-y) *MAx(x,y), 0.5<7< ] Then we can Wm.sform A2,Am and Az,o~ into &w and AA, OR, respectively, by replacing y with (y+ 1)/2. In the remainder of this paper, we will consider only the Al operator. From Figure 5-(b) we can easily see that Al and As allow negative compensation in some value ranges and Al always gives zero regardless of the other operand when one of operands is zero. Hence, EBRFA1 does not avoid the single operand dependency and negative compensation problems, and EBRFA~ causes the negative compensation problem. The operator graphs of Ad,w and &o~ show that the resulting value is always greater than the lower value of two operand values and less than the higher value with the exception that the values of both operands are equivalent. The operators having the aforementioned properties will be called positively compensato~ operators because their compensatory effects overcome the single operand dependency and negative compensation problems. Consequently, we insist that positively compensatory operators should provide higher retriewd effectiveness than the others. 3.3 The Operators of the Extended Boolean Model The extended boolean model has overcome the single operand dependency problem of the fuzzy set model by developing the EM and Eo~ operators for the evaluation of the AND and OR operations, respectively. The E and Ad 294

5 operators have the following common characteristics: (i) All of them are parametrized where the results are controlled by the associated parameter. (ii) Because the E operators are also positively compensatory, the output value is always in the range between MIN and MAX. MIN(x, y) < AA.m(x, Y) S &.or(x, Y) ~ MM(x, Y) MIN(x, y) < E-(x, y) < ~R(X, y) < MM(x, y) Figure 6 shows the operator graphs of the E operators. The behavior of the E operators ahnost coincides with that of the A4 operators, which can be easily known by comparing their operator graphs. Ap=l documents by particular authors. Both collections also contain relevance assessment of each document with respect to each query. The retrieval model based on EBRF exploits document term weights to rank documents. The 1S1 and CACM collections do not have specific information about the actual importance of the terms. In this case, the weights of document terms can be derived from their occurrence frequency such as Inverse Document Frequency (IDF) and Term Frequency (TF) [Salt83]. If N is the number of documents in a collection and n~ is the number of documents in which term k occurs, then the inverse document frequency of term k, IDF~, is defined as log(n/nj. The term frequency ~ik means the occurrence frequency of term k in document i. The weight w& of term k in document i can be defined as IDFk TFik. Since document term weights in the fuzzy set model must be in the interval [0, 1], the weight wik is normalized as follows: mik IDFk normalized Wik = maximum TF maximum IDF in document i in document i 4.2 Experimental Results A p.cw Figure 6. The operator graphs for the extended boolean model 4. PERFORMANCE EVALUATION We insisted that EBRFPositivelY_comPensatorY_oPerators should achieve high retrieval effectiveness. This fact is shown clearly through experiments in this section, We fwst evaluate retrieval effectiveness Of EB RFF, EB ~T 1,... EBRFT4, EBRFA1, EBRFA3, EBRFA4 and EBRFE. We then compare retrieval effectiveness and efficiency of EBRFA4 with those of EBRFE in more detail. 4.1 Experimental Collections We use two different document collections covering items in library science and computer science [Salt83]. The library science items, designated as 1S1 1460, cover highly cited items extracted from the Social Science Citation Index. The CACM 3204 collection covers articles published between 1959 and 1979 in the Communications of the ACM. Queries were formulated fwst in natural language for each of these collections, and later in boolean form by graduate students and staff of Cornell University. The 1S1 collection consists of 1460 documents and 35 queries. The CACM collection consists of 3204 documents and 52 queries, and we used only the 50 queries that do not request To evaluate the effectiveness of an IR system, it is customary to compute values of the recall and precision [Salt89]. In general, users want to retrieve most everything relevant and to reject most everything extraneous - to get high recall and high precision. Documents are ranked in decreasing order of query-document similarities, that is, the most important items are obtained frost. The ranked output then makes it possible to compute a recall and a precision value after the retrieval of each item. By interpolation the precision values can be calculated for fixed values of the recall, e.g., 0.1, 0.2 and so on up to 1.0. Figure 7 shows retrieval effectiveness of EBRFF, EBRFT1,... EBRFT4, EBRFA1, EBRFA3, EBRFA4 and EBRF~. We calculated a single precision value representing the average precision at three typical recall levels (0.25, 0.50, 0.75). For some parameterized operators we computed the precision value as changing the corresponding parameter in its legal range and then chose the maximum value. Figure 8 shows retrieval effectiveness of EBRFA4 and EBRF~ in more detail, The horizontal coordinate denotes the value range of the associated parameter, and the vertical coordinate denotes the precision. It should be noted that retrieval effectiveness of EBRFA4 ahnost coincides with that of EBRFE. 295

6 precision precision 0 3 ~ 0 3 ~ FT1T2T3T4AIA3A4E 0.0 FTIT2T3T4AIA3A4 E (a) The CACM 3204 collection (b) The 1S collection Figure 7. The performance of i uzzy operators pmeision preeision 0 3EBRF~ (a) The CACM Iomp 3204 collection precision precision 0 3EBRFA4 0 3 EBR~ Y mcop (b) The 1S collection Figure 8. Retrieval effectiveness of the EBRFAd and EBRF~ 296

7 To compare retrieval efficiency of EBRFAd and EBRF~, we measured computation time spent in evaluating tl AND t2 and tl OR t2 10,000 times. It took 0.04 and 1.54 seconds for EBRFA~ and EBRF~, respectively. The ~ operators are much more efficient than the E operators. It is indicated in the original proposal of the extended boolean model that the complexity of the END and ~R operators is a serious problem. In prwticular, recursive application of these operators produces complicated expressions for boolean queries with a large number of boolean operators. The AA operators overcome the computational inefficiency problem successfidly. 5. CONCLUDING REMARKS IR systems should be designed to aid users in determining which documents of those retrieved are most likely to be relevant to the given queries. Boolean retrieval systems have been most widely used among commercially available IR systems. Although conventional boolean retrieval systems accomplish efficient document retrievals, they suffer from an inability to rank the retrieved documents. The EBRF-based retrieval models such as the fuzzy set model and the extended boolean model have been proposed in the past to provide the document ranking facility. The fuzzy set model has the single operand dependency problem caused by the MIN and MAX operators, and the extended boolean model suffers from the computation complexity of the EM and ~R OpeEMOrS. In recent years a variety of fuzzy operators have been developed which are classified into T-operators and averaging operators. We have investigated the effect of the fuzzy operators on retrieval effectiveness and have presented the operational properties adverse to effective document ranking, i.e., single operand dependency and negative compensation. We have also described that the positively compensatory operators such as A4 and E overcome the single operand dependency and negative compensation problems. Experimental results have shown that the positively compensatory operators provide higher retrieval effectiveness than the others. They have also shown that EBRFA4 achieves similar retrieval effectiveness and higher retieval effickiicy in COInPZUiSOII With EB RFE. REFERENCES ~art 85] M. Bartschi, An Overview of Information Retrieval Subjects, IEEE Computer, pp , A. Bookstein, Fuzzy Requests: An Approach to Weighted Boolean Searches, Journal of the American Society for Information Science, Vol. 31, No. 4, pp , Euel 81] D.A. Buell, A General Model of Query Processing in Information Retriewd System, [Gupt 91] Lee 92] Lee 93] l-l?ade79] [Sach 76] [Salt 83] [Salt 85] [Salt 89] mall 79] lwem 88] [Zade 65] [Zimm 80] [Zimm 91] Information Processing & Management, Vol. 17, No. 5, pp , M.M. Gupta and J, Oi, Theory of T-Norms and Fuzzy Inference Methods, Fuzzy Sets and Systems, Vol. 40, No. 3, pp , J.H. Lee, M.H. Kim and Y.J. Lee, Enhancing the Fuzzy Set Model for High Quality Document Rankings, Proceedings of the 19th Euromicro Conference, Paris, France, , J.H. Lee, M.H. Kim and Y.J. Lee, Ranking Documents in Thesaurus-Based Boolean Retrieval Systems, Information Processing & Management. (to appear) T. Radecki, Fuzzy Set Theoretical Approach to Document Retrieval, Information Processing & Management, Vol. 15, No. 5, pp , W.M. Sachs, An Approach to Associative Retrieval through the Theory of Fuzzy Sets, Journal of the American Society for Information Science, Vol. 27, pp , G. Salton, E.A. Fox, and H. Wu, Extended Boolean Information Retrieval, Communications of the ACM, Vol. 26, No. 11, pp , G. Salton, E.A. Fox, and E. Voorhees, Advanced Feedback Methods in Information Retrieval, JournaI of the American Society for Information Science, Vol. 36, No. 3, pp , G. Salton, Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer, Addison Wesley, W.G. Wailer and D.H. Kraft, A Mathematical Model of a Weighted Boolean Retrieval System, Information Processing & Management, Vol. 15, pp , B. Werners, Aggregation Models in Mathematical Programming, in Mathematical Models for Decision Support, G. Mitra cd., pp , L.A. Zadeh, Fuzzy Sets, Information and Control, Vol. 8, pp , H.J. Zimmerman and P. Zysno, Latent Connective in Human Decision Making, Fuzzy Sets and Systems, Vol. 4, No. 1, pp , H.J. Zimmerman, Fuzzy Set Theory and Its Applications, 2nd cd., Kluwer Academic Publishers,