Level of analysis Finding Out About Chapter 3: 25 Sept 01 R. K. Belew

Transcription

1 Overview The fascination with the subliminal, the camouflaged, and the encrypted is ancient. Getting a computer to munch away at long strings of letters from the Old Testament is not that different from killing animals and interpreting the entrails, or pouring out tea and reading the leaves. It does add the modern impersonal touch a computer found it, not a person, so it must be really there. But computers find what people tell them to find. As the programmers like to say, prophesy in, prophesy out.

2 Level of analysis

3 Zipf s Law Frequency of words Zipf's first law Rank order of words too common significant too rare

4 Zipfian Distribution of AIT Words

5 Principle of Least Effort

6 Other very clever people have provided other cognitive/linguistic explanations

7 Or, not!

8 WWW surfing behavior

9 Consequences of lexical decisions

10 Consequences... (cont) Token Freq Unstem-f the of and a to in system is model for de network this base 9838 that 9820 are 9792 learn 9293 world 8103 la 7678 author 7615 an 7593 Token Freq Unstem-f knowledg neural with 7197 as 6964 on 6920 by 6886 process design del 6178 be 6045 develop 5891 integr 5633 domain 5630 based 5326 use 5226 intellig 5197 which 5158 control expert comput 4851 mechan 4818 escolar 4728

11 Consequences... (cont2) Token Freq Unstem-f approach from 4587 classifi 4556 algorithm final 4436 systems 4387 can 4370 code 4116 robot 4103 intern 4097 applic 4055 perform 4051 percept 4047 method enabl 4036 data make 3984 increm 3947 incomplet 3890 secondli 3765 mo 3733 it 3697 used 3594 problem 3520 Token Freq we 3276 these 3268 using 3268 learning 3266 was 3205 has 3051 or 2859 been 2715 research 2622 have 2609 two 2601 developed 2550 information 2461 networks 2449 time 2370 s 2350 new 2293 also 2259 performance 2244 results 2239 were 2216 such 2165 problems 2133 analysis 2045 models 2000

12 Function words follow Poisson distribution λ Pr(n occur of w) = e λ w λ w n n!

13 Two-Poisson model λ λ

14 Resolving power

15 Resolving Power upper cut-off lower cut-off Frequency of words Frequency of words too common Resolving power significant Rank order of words too rare Zipf's first law

16 Indexing term distribution

17 Exhaustivity: Number of topics indexed

18 Specificity: ability to describe FOA information need precisely

19 Index: A balance between user and corpus Specificity Exhaustivity Query INDEX Corpus

20 Not too exhaustive, not too specific... Specificity Exhaustivity Query Discriminability of INDEX Representation of Corpus few doc/jkw Hi Precision leads to many kw/doc Hi Recall

21 Factors in index weighting w kd freq kd discrim k freq kd N(occurrences of word k in doc d )

22 Indexing Graph

23 Information is reduction in uncertainty

24 Hypothetical Word Distributions

25 Separate informative words from noise Noise k = NDoc d=1 freq kd freq k log freq k freq kd Signal k = freq k Noise k w kd = freq kd * Signal k

26 3.3.7 Inverse document frequency Doc k N(documents containing word k ) w = freq * log Norm + 1 kd kd Doc k Norm = Ndoc [Sparck - Jones' 72] argmax k Doc k [Sparck - Jones' 79]

27 Fig 3.7 Vector Space

28 Inter-document similarity Sim(d i,d j ) "Similarity" twix documents D * Centroid; average document Sim 1 NDoc NDoc 2 i, j = α Sim( d i, D * ) i=1 Sim(d i,d j )

29 Removing keyword collapses document space Sim k Sim when term k removed Disc k Sim k Sim w kd = freq kd * Disc k

30 Length Normalization of Vector Space

31 Sensitivity of IDF to Document Size

32 Pivot-Based Document Length Normalization

33 Summary: SMART Weighting Specification w kd = freq kd collect k norm

34 Frequency of KW in DOC freq kd = {0,1} binary freq kd max norm max( freq kd ) k freq kd augmented 2 max( freq kd ) k ln( freq kd ) + 1 log

35 Collection statistics of KW freq kd = {0,1} binary freq kd max norm max( freq kd ) k freq kd augmented 2 max( freq kd ) k ln( freq kd ) + 1 log

36 Normalization norm = vector w i w 2 i vector sum cosine w 4 i fourth vector max vector ( ) max w i

37 3.5.1 Measures of association Q = {kw query} D = {kw document} Q D 2 Q D Q + D Q D Q D Shared features Dice coefficient Cosine coefficient

38 3.5.2 Cosine Similarity

39 Dissimilarity as distance S D D S

40 3.7 Computing Partial Match Scores