Information Retrieval and Web Search

Transcription

1 Information Retrieval and Web Search IR models: Boolean model IR Models Set Theoretic Classic Models Fuzzy Extended Boolean U s e r T a s k Retrieval: Adhoc Filtering Browsing boolean vector probabilistic Structured Models Non-Overlapping Lists Proximal Nodes Algebraic Generalized Vector Lat. Semantic Index Neural Networks Probabilistic Inference Network Belief Network Browsing Flat Structure Guided Hypertext Slide 1 Baili Zhang/ Southeast 1

2 The Boolean Model Simple model based on set theory Queries specified as boolean expressions precise semantics neat formalism q = ka (kb kc) Terms are either present or absent. Thus, wij {0,1} Consider q = ka (kb kc) vec(qdnf) = (1,1,1) (1,1,0) (1,0,0) Each query can be transformed in DNF form Slide 2 The Boolean Model q = ka (kb kc) Ka (1,0,0) (1,1,0) (1,1,1) Kb Kc sim(q,dj) = 1, if document satisfies the boolean query 0 otherwise - no in-between, only 0 or 1 Slide 3 Baili Zhang/ Southeast 2

3 Exercise D 1 = computer information retrieval D 2 = computer retrieval D 3 = information D 4 = computer information Q 1 = information retrieval Q 2 = information computer Slide 4 Exercise ((chaucer OR milton) AND (NOT swift)) OR ((NOT chaucer) AND (swift OR shakespeare)) Slide 5 Baili Zhang/ Southeast 3

4 Drawbacks of the Boolean Model Retrieval based on binary decision criteria with no notion of partial matching No ranking of the documents is provided (absence of a grading scale) Information need has to be translated into a Boolean expression which most users find awkward The Boolean queries formulated by the users are most often too simplistic As a consequence, the Boolean model frequently returns either too few or too many documents in response to a user query Slide 6 The Boolean model imposes a binary criterion for deciding relevance The question of how to extend the Boolean model to accomodate partial matching and a ranking has attracted considerable attention in the past Two extensions of boolean model: Fuzzy Set Model Extended Boolean Model Slide 7 Baili Zhang/ Southeast 4

5 Extended Boolean Model Boolean model is simple and elegant. But, no provision for a ranking As with the fuzzy model, a ranking can be obtained by relaxing the condition on set membership Extend the Boolean model with the notions of partial matching and term weighting Combine characteristics of the Vector model with properties of Boolean algebra Slide 8 The Idea The extended Boolean model (introduced by Salton, Fox, and Wu, 1983) is based on a critique of a basic assumption in Boolean algebra Let, q = kx ky Use weights associated with kx and ky In boolean model: wx = 1; Or 0: all other documents are irrelevant Slide 9 Baili Zhang/ Southeast 5

6 The Idea ky q AND = kx ky; w xj = x and w yj = y (1,1) dj+1 AND y = w yj dj (0,0) x = w xj kx We want a document to be as close as possible to (1,1) Slide 10 The Idea ky q or = kx ky; w xj = x and w yj = y (1,1) dj+1 OR y = w yj dj (0,0) x = w xj kx We want a document to be as far as possible from (0,0) Slide 11 Baili Zhang/ Southeast 6

7 Generalizing the Idea We can extend the previous model to consider Euclidean distances in a t-dimensional space This can be done using p-norms which extend the notion of distance to include p-distances, where 1 p is a new parameter A generalized conjunctive query is given by qor = k1 pk2 p... pkt p km A generalized disjunctive query is given by qand = k1 p k2 p... p kt p km Slide 12 Generalizing the Idea If p = 1 then (similar to vectorial model) sim(q or,dj) = sim(q and,dj) = x xm m Slide 13 Baili Zhang/ Southeast 7

8 Extended Boolean Model Model is quite powerful Properties are interesting and might be useful Computation is somewhat complex However, distributivity operation does not hold for ranking computation: q1 = (k1 k2) k3 q2 = (k1 k3) (k2 k3) sim(q1,dj) sim(q2,dj) Slide 14 Fuzzy Set Model Queries and docs represented by sets of index terms: matching is approximate from the start This vagueness can be modeled using a fuzzy framework, as follows: with each term is associated a fuzzy set each doc has a degree of membership in this fuzzy set This interpretation provides the foundation for many models for IR based on fuzzy theory In here, the model proposed by Ogawa, Morita, and Kobayashi (1991) Slide 15 Baili Zhang/ Southeast 8

9 Fuzzy Set Theory Framework for representing classes whose boundaries are not well defined Key idea is to introduce the notion of a degree of membership associated with the elements of a set This degree of membership varies from 0 to 1 and allows modeling the notion of marginal membership Thus, membership is now a gradual notion, contrary to the notion enforced by classic Boolean logic Slide 16 Fuzzy Set Theory Definition A fuzzy subset A of U is characterized by a membership function (A,u) : U [0,1] which associates with each element u of U a number (u) in the interval [0,1] Definition Let A and B be two fuzzy subsets of U. Also, let A be the complement of A. Then, ( A,u) = 1 - (A,u) (A B,u) = max( (A,u), (B,u)) (A B,u) = min( (A,u), (B,u)) Slide 17 Baili Zhang/ Southeast 9

10 Fuzzy Information Retrieval Fuzzy sets are modeled based on a thesaurus This thesaurus is built as follows: Let vec(c) be a term-term correlation matrix Let c(i,l) be a normalized correlation factor for (ki,kl): c(i,l) = n(i,l) ni + nl - n(i,l) - ni: number of docs which contain ki - nl: number of docs which contain kl - n(i,l): number of docs which contain both ki and kl We now have the notion of proximity among index terms. Slide 18 Fuzzy Information Retrieval The correlation factor c(i,l) can be used to define fuzzy set membership for a document dj as follows: (i,j) = 1 - (1 - c(i,l)) kl dj - (i,j) : membership of doc dj in fuzzy subset associated with ki The above expression computes an algebraic sum over all terms in the doc dj A doc dj belongs to the fuzzy set for ki, if its own terms are associated with ki If doc dj contains a term kl which is closely related to ki, we have c(i,l) ~ 1 (i,j) ~ 1 Slide 19 Baili Zhang/ Southeast 10

11 Fuzzy Information Retrieval Disjunctive set: algebraic sum (cc1 cc2 cc3, j) = 1 - (1 - (cc i, j)) Conjunctive set: algebraic product (cc1 & cc2 & cc3,j) = ( (cc i, j)) Slide 20 Fuzzy IR: An Example Ka cc3 cc2 cc1 Kb q = ka (kb kc) vec(qdnf) = (1,1,1) + (1,1,0) + (1,0,0) = vec(cc1) + vec(cc2) + vec(cc3) (q,dj) = (cc1+cc2+cc3,j) = 1 - (1 - (cc i, j)) = 1 - (1 - (a,j) (b,j) (c,j)) * (1 - (a,j) (b,j) (1- (c,j))) * (1 - (a,j) (1- (b,j)) (1- (c,j))) Kc Slide 21 Baili Zhang/ Southeast 11