1 Extended Boolean Model

Size: px

Start display at page:

Download "1 Extended Boolean Model"

Patrick Stevenson
6 years ago
Views:

1 1 EXTENDED BOOLEAN MODEL It has been well-known that the Boolean odel is too inflexible, requiring skilful use of Boolean operators to obtain good results. On the other hand, the vector space odel is flexible but not precise enough. Is there a iddle ground? 1 Extended Boolean Model 1.1 What is wrong with the Boolean and vector space odels? We can see that the Boolean odel has extreely rigid conditions that the returned docuents ust satisfy. On the other hand, the pure vector space odel essentially applies the OR operation to the query keywords, atches the keywords against a docuent, and sus up the score of each query ter that occurs in a docuent. Thus, the vector space odel can theoretically rank at the top a docuent that atches only one query ter. As discussed earlier in the vector space odel, this does not eet the expectation of the searchers. To add ranking capability to the Boolean odel, we saw previously that the Boolean odel can be used as a pre-filter to restrict the docuents to a subset that satisfies the Boolean condition and the vector space odel can then be applied to the subset to produce a ranked list of results. This approach is suitable for (and indeed largely adopted by) web search engines because all web search engines nowadays assue by default the AND operator between the query keywords when users don t specify any Boolean operators in the query (99% of users don t!). When the web has an abundance of inforation, the Inforation default AND operator works very well. 1 The Boolean pre-filter inherits the restriction of the Boolean odel in that when the user specifies an AND operator in the query, the condition ay already be too restricted that the result had excluded any useful docuents which would otherwise ranked very high in the result. When the user uses an OR operator then it has no difference fro the siple vector space odel. Again, such design assues that users are able to specify a good Boolean expression for his query, which is in general far fro the truth. 1. Integrating Boolean and ranking The Extended Boolean Model is an interesting extension of the Boolean odel that cobines the control of the Boolean odel and the ranking capability of the vector spaceodel into a unifor fraework. The idea is very siple. Users can use AND and OR in the queries (as in the Boolean odel) but keywords are weighted (as in the vector space odel). Furtherore, docuents are ranked by a siilarity function that is designed to exhibit the following behaviors: 1 In fact, Google was one of the first, if not the first, search engines that assues an AND operator between query keywords, when at least Alta Vista insisted an OR operator for a long tie until Google had proven to be a threat to all first-generation search engines. abundance on the web changed the way search engines should work. 1 c Suntek Coputer Systes

2 1.3 Disjunctive queries 1 EXTENDED BOOLEAN MODEL For conjunctive queries, docuents containing all keywords are awarded higher scores, while docuents that don t contain all of the query keywords still receive non-zero scores and as such have a chance to be ranked high should the atching query keywords have heavy enough weights. For disjunctive queries, docuents containing all keywords are awarded with higher scores and docuents that don t contain all of the query keywords shall receive non-zero weights so that they too have a good chance to be ranked high in the result. The two cases are very siilar. The only difference is that in the conjunctive case, the issing of one query keyword in a docuent results in a heavy penalty on the docuent s score but in the disjunctive case, the penalty is relatively sall. Now, let s look at how the siilarity forula achieves these effects. In the following, we assue that the weight of a keyword is noralized to [0,1]. For exaple, the tfxidf weighting forula can be odified as: w x,j = tf x,j idf x (1) ax l tf l,j ax i idf i where w x,j is the weight of ter x in docuent j, the ter frequency tf x,j is noralized by the highest tf in docuent j, andtheidf of ter x is noralized by the highest idf aong all keywords in the collection. 1.3 Disjunctive queries Given a disjunctive query x y, and the weights of x and y in docuent j are w x and w y, respectively, the siilarity between the query and docuent j is defined as follows. ( ) w x,j + w 0.5 y,j si(q or,d j )= () The geoetric interpretation of the function is shown in Figure 1(a). In essence, a docuent has the highest possible weight at (1,1), eaning that it contains both x and y and their weights in the docuent are highest possible (i.e., equal to 1). The siilarity function easures the vector length of docuent d fro the origin. 1.4 Conjunctive queries Given a conjunctive query x y, the siilarity between q and docuent j is defined as follows. ( (1 wx,j ) +(1 w y,j ) ) 0.5 ) si(q and,d j )=1 (3) The siilarity function easures the copleent of the vector length d fro the coordinates (1,1). Intuitively, a docuent at (1,1) would be the best atch c Suntek Coputer Systes

3 1.5 Observations 1 EXTENDED BOOLEAN MODEL (a) x y. (b) x y. Figure 1: Geoetric interpretation. d j Ter Docuent Score contains weights si(x y, d j ) si(x y, d j ) xx..xx wt x,j = xx..xx wt x,j = xx..yy wt x,j = wt y,j = xx..yy wt x,j = wt y,j = Table 1: The variation of docuent score against ter weights. to the query, and the farther away a docuent is fro this ideal position, the lower is its score with respect to q. Figure 1(b) is the geoetric interpretation of the siilarity function. 1.5 Observations Let s look at how the score of a docuent varies according to (i) the ters it contains and the ter weights, and (ii) the type of the query. Table 1 shows the variation of the score of docuent d j for both disjunctive and conjunctive queries when: (i) d j contains only one query keyword (x is assued), and (ii) when d j contains both x and y. We can see that if a docuent contains both ters, the docuent score increases rapidly to the axiu (i.e., 1). The behavior is the sae for both conjunctive and disjunctive queries. However, when a docuent contains only one ter, the existence of the ter gives a gain to an OR query but only 0.93 to an AND query. In other words, issing a ter gives a uch saller penalty to an OR query than an AND query. This is consistent with the intuition that when the user specifies an AND query, he expects to see both ters in a docuent. Thus, when a docuent contains only one ter it should be heavily penalized. This is not the case for an OR query when the user expects The Extended Boolean odel is consistent with user s expectation. 3 c Suntek Coputer Systes

4 1.6 Generalization of the Extended Boolean 1 EXTENDED odel BOOLEAN MODEL to see only one of the two ters. Therefore, a docuent containing only one of the two query ters should get a reasonably high score whereas a docuent containing both ters exceeds the user s expectation and thus should get the highest possible score (i.e., 1). 1.6 Generalization of the Extended Boolean odel The description of the Extended Boolean Model is based on two query keywords. Needless to day, the odel has to be able to handle ore than two query keywords. The extended siilarity functions for query keywords are as follows. ( x si(q or,d)= 1 + x ) 0.5 x (4) ( (1 x1 ) +(1 x ) (1 x ) ) 0.5 si(q and,d)=1 (5) We can see that so far the siilarity function is expressed based on the Euclidean lengths of the docuent vectors, i.e., the L nor. In fact, the siilarity function can be further generalized into the p-nor odel by replacing the L nor with p-nor. ( x p 1 si(q or,d)= + xp ) 1/p xp (6) ( (1 x1 ) p +(1 x ) p (1 x ) p ) 1/p si(q and,d)=1 (7) When the value of p varies, the odel degenerates into different special cases. When p =1,si(q or,d)=si(q and,d)=(x 1 + x x )/. The odel degenerates into the inner product siilarity function in the vector-space odel. When p =, si(q or,d) = ax(x i ); si(q and,d) = in(x i ). This is equivalent to the fuzzy logic odel. In the p-nor odel, the AND and OR operators in the sae query can be associated with different p values. For exaple, the query (x y) z eans the following: d ust contain z because when z does not exist, si() = 0 When x and y both do not exist, si() = 0 When z exists, si() = in(w z,si(x y, d)) By assigning different p values to the Boolean operators, a keyword can be interpreted differently in different operator. For exaple, y has a L nor in the AND operator but an infinite nor in the OR operator and thus creating different effects. It is clear that the Boolean expression is not coutative any ore and that despite its flexibility and elegance, the generalized p-nor odel is very difficult to use. It is not clear what ipacts the p value has on 4 c Suntek Coputer Systes

5 1.7 Questions 1 EXTENDED BOOLEAN MODEL the overall retrieval quality. In order to ake the odel useful for the general users, perhaps the Boolean expression and the value of p ust be obtained and adapted using achine learning ethods. 1.7 Questions 1. At the tie of writing, none of the ajor search engines ake use of the Extended Boolean odel, why? 5 c Suntek Coputer Systes

The Internal Conflict of a Belief Function

The Internal Conflict of a Belief Function Johan Schubert Abstract In this paper we define and derive an internal conflict of a belief function We decopose the belief function in question into a set of