vector space retrieval many slides courtesy James Amherst

Transcription

1 vector space retrieval many slides courtesy James Amherst 1

2 what is a retrieval model? Model is an idealization or abstraction of an actual process Mathematical models are used to study the properties of the process, draw conclusions, make predictions Conclusions derived from a model depend on whether the model is a good approximation of the actual situation Statistical models represent repetitive processes, make predictions about frequencies of interesting events Retrieval models can describe the computational process e.g. how documents are ranked Note that how documents or indexes are stored is implementation Retrieval models can attempt to describe the human process e.g. the information need, interaction Few do so meaningfully Retrieval models have an explicit or implicit definition of relevance 2

3 retrieval models today boolean vector space latent semnatic indexing statistical language inference network hyperlink based 3

4 outline review: geometry, linear algebra vector space model vector selection similarity weighting schemes latent semantic indexing 4

5 linear algebra 5

6 vectors 6

7 subspaces 7

8 linear independence, base, dimension, rank! 0$)1.&! ",!"#$%& '$($#'$#1.2 0$)1.&, " 3 # " 4 # $$$# " % "2 15$&$ $6",1, &$%! #7-+$&, & 3 # & 4 # $$$# & %,7)5 15%1! 8 & 3 " 3 9 & 4 " 4 9 $$$& % " %! +%,$.2 % 0$)1.&"%!,(%)$ 8 -%6"-%!,$1.2!"#$%& "#'$($#'$#1 0$)1.&,: ;!! +%,$,.2 % <"0$#,(%)$ 5%0$ 15$,%-$ '"--$#,".# ='"--$#,".#.2 15$,(%)$>! &%#/='> 8 -%6"-7- #7-+$&.2 &%?,@).!7-#,!"#$%& "#'$($#'$#1! &%#/='> 8 '"-$#".#.2 15$,7+,(%)$,(%##$' +A ' 8

9 matrix multiplication 9

10 dot product, norm!!"# $%"!&'# "+, -.*/!0*/)-0").%% *$21 #3/ *.#%04 $%"!&'# 50#3 %/-&2#. %/.2 )&*6/%!! 7 8! 9 "!, " ###"!$:; % 7 8% 9 " %, " ###" %$: #3/) &! " % '7!! % ( 7! $ )79! ) % )! *, )"%* < ##!## 7 " &! "! '! )"%*.20=.#0")<! 7! ##!## ; ##!##

11 cosine 11

12 cosine computation cos(θ) = a b a b cos(θ) = cos(β α) = cos(β) cos(α) + sin(β) sin(α) 12

13 orthogonality 13

14 projections 14

15 A=LDU factorization!!"# $%&!'" ($)#*' +, )-.#..'*/)/ $ 0.#(12 )$)*"% ($)#*' #, $ 3"4.# )#*$%513$# ($)#*' $ 4*)- 1%*) 6*$5"%$3 $%6 $%!'".7-.3"% ($)#*' % /17- )-$) # & 8 $%!!"# $%& "'" ($)#*' +, )-.#..'*/)/ $,% 3"4.# $%6 100.# )#*1%5-*13$# 4*)- 1%*) 6*$5"%$3/,' $ 6*$5"%$3 ($)#*' "9 0*:")/ $%6 # $ 0.#(1)$)*"% ($)#*' /17- )-$) # & 8 $'%! <9 & */ /&((.)#*7 =& 8 & ( > )-.% )-.#. */ %" %..6 9"# # $%6 % 8 $ (? & 8 $'$ (

16 eigenvalues, eigenvectors!! ") &$!"#!$%&'(! /-. 0&,."1! "2 "#$3!!!%4 5 6!!%!.7!"#!$%&'(! 8&) & +-..!)9-$*!$, $-$: ;!.-!"#!$%!+,-. &,8&, )&,")<!) 3!!!%4& !& 5!& "$ -,8!. =-.*)!& &$* & 8&%! )&0! *".!+,"-$! )(0 -/!"#!$%&'(!) 5,.&+!3!4 5 )(0 -/ *": &#-$&'! 9.-*(+, -/!"#!$%&'(!) 5 *!,3!4!!"#!$%&'(!) -/ & (99!.>'-=!.,."&$#('&. 0&:,."1 &.!,8! *"&#-$&'!$,."!) 16

17 matrix diagonal form! %,! -". +%*/"$0 %*'/1/*'/*# /%(/*2/3#)$. " 4 # " 5 # $$$# " % "*' & %. #-/!"#$%& -"2%*( #-)./ ". 3)+6!*.7 & 8 9" 4 " 5 $$$" % :7! #-/* & %. %*2/$#%;+/ $ "*'! 4 &!4!!& 8 < 8 " 5 % 7 #-/ '%"()*"+ # $$$ &!%!"#$%& ), /%(/*2"+6/. ),!=!! 8 &<&!4! *) $/1/"#/' /%(/*2"+ " %*'/1= /%(/*2/3#!!.0!/#$%3! ' 8! " & )$#-)()*"+> & ' & 8 4! & %. *)# 6*%?6/!!& 8 &< -)+'. %@ & -". /%(/*2/3# ". 3)+6!*.! *)# "++!"#$%3/. "$/ '%"()*"+%A";+/ 17!

18 singular value decomposition! "1! "! " " #$ " % # (*'&.'0("2 03*# "0,'# 4* +*,-./-!*+ '!! 5 & '( ) 63*(*! & "! " " #7 '$ ( '(* # " #! &$ ( '(* -(03-$-#'& 8 & ) & 5 ( ) ( 5 9 #"#! ' "! +"'$-#'&: "0! *#0("*! '(* 03*!;%(* (--0! -1 *"$*#)'&%*! -1! )! 18

19 outline review: geometry, linear algebra vector space model vector selection similarity weighting schemes latent semantic indexing 19

20 vector space represent documents and queries as vectors in the term space issue : find the right coefficients use a geometric similarity measure, often angle-related issue: normallization 20

21 mapping to vectors terms: an axis for each term vectors corresponding to terms are canonical document = sum of vectors corresponding to terms contained in doc queries treated the same as documents 21

22 coefficients The coefficients (vector lengths, term weights) represent term presence, importance, or aboutness Magnitude along each dimension Model gives no guidance on how to set term weights Some common choices: Binary: 1 = term is present, 0 = term not present in document tf: The frequency of the term in the document tf idf: idf indicates the discriminatory power of the term Tf idf is far and away the most common Numerous variations 22

23 raw tf weights cat cat cat cat cat cat cat lion lion cat cat lion dog cat cat lion dog dog 23

24 tf = term frequency raw-tf (tf)=count of term in document Robertson tf (okapi tf) based on a set of simple criteria loosely connected to 2-Poisson model popular k=0.5; c=1.5 basic formula is tf /(k+tf) document length = verbosity factor many variants tf tf + k + c doclen avg.doclen 24

25 Robertson tf 25

26 IDF weights Inverse Document Frequency Used to weight terms based on frequency in the corpus Fixed, it can be precomputed for each term basic formula IDF (t) = log( N ) N t N= # of docs Nt= #of docs containing term t 26

27 tf-idf tf * idf the weight on every term is tf(t,d)*idf(t) sometimes variants on tf, IDF no satisfactory model behind these combinations 27

28 outline review: geometry, linear algebra vector space model vector selection similarity weighting schemes latent semantic indexing 28

29 common similarity measures 29

30 vector similarity: cosine θ 1 θ 2 30

31 similarity, normalized!"#"$%&"'(!!!"#$%&$'#!("!!&$# "!"!&$# #! same intersection area but different fraction of sets the size of intersection alone is meaningless often divided by sizes of sets same for vectors, using norm by normalizing vectors, cosine does not change 31

32 cosine similarity: example 32

33 cosine example, normalized round-off error, should be the same 33

34 similarity example 34

35 tf-idf base similarity formula!!!!" "#$%&!!"!#$" "#$%&!!""!!!" '()!!"!#$" '()!!"" ""'()""!"""#$%&"" many options for TF query and TF doc raw tf, Robertson tf, Lucene etc try to come up with yours some options for IDF doc IDF query sometimes not considered normalization is critical 35

36 Lucene comparison 36

37 complicated formulas 37

38 outline review: geometry, linear algebra vector space model vector selection similarity weighting schemes latent semantic indexing 38

39 LSI Variant of the vector space model Use Singular Value Decomposition (a dimensionality reduction technique) to identify uncorrelated, significant basis vectors or factors Rather than non-independent terms Replace original words with a subset of the new factors (say 100) in both documents and queries Compute similarities in this new space Computationally expensive, uncertain effectiveness 39

40 dimensionality reduction when the representation space is rich but data is lying in a small subspace that is when some eigenvalues are zero non-exact: ignore smallest eigenvalues, even if they are not zero 40

41 LSI T0, D0 orthogonal with unit length columns T0 * T0 T =1 S0 = diagonal matrix of eigenvalues m = rank of X 41

42 LSI: example 42

43 LSI: example 43

44 LSI!!!"# $%&!$'$(") *(+&,)-('&!.$) /!!!! 0 12! "!"# $%&!$'$(")*(+&,)-('&!.$) /"!"! 0 12! # 3+"'$(") 4"&%+5 $6 -+'-( 7")*-#! $ +# &!- %"(8 $6 %! & 0 9 $6 %$:# +( %! ' 0 9 $6.$)*4(# +( %! ( 0.!$#-( (*4;-% $6 3+4-(#+$(# $6 %-3*.-3 4$3-) 44

45 using LSI 45

46 LSI: example 46

47 original vs LSI 47

48 using LSI D is new doc vectors (k dimensions) T provides term vectors Given Q=q 1 q 2 q t want to compare to docs Convert Q from t dimensions to k!! )! " *"#! " #"$! % "* $"$ Can now compare to doc vectors Same basic approach can be used to add new docs to the database 48

49 LSI : does it work? Decomposes language into basis vectors In a sense, is looking for core concepts In theory, this means that system will retrieve documents using synonyms of your query words The magic that appeals to people From a demo at lsi.research.telcordia.com They hold the patent on LSI 49

50 vector space: summary Standard vector space Each dimension corresponds to a term in the vocabulary Vector elements are real-valued, reflecting term importance Any vector (document,query,...) can be compared to any other Cosine correlation is the similarity metric used most often Latent Semantic Indexing (LSI) Each dimension corresponds to a basic concept Documents and queries mapped into basic concepts Same as standard vector space after that Whether it s good depends on what you want 50

51 vector space : disadvantages Assumed independence relationship among terms Though this is a very common retrieval model assumption Lack of justification for some vector operations e.g. choice of similarity function e.g., choice of term weights Barely a retrieval model Doesn t explicitly model relevance, a person s information need, language models, etc. Assumes a query and a document can be treated the same (symmetric) 51

52 vector space: advantages Simplicity Ability to incorporate term weights Any type of term weights can be added No model that has to justify the use of a weight Ability to handle distributed term representations e.g., LSI Can measure similarities between almost anything: documents and queries documents and documents queries and queries sentences and sentences etc. 52