Information Retrieval. CS630 Representing and Accessing Digital Information. IR Basics. User Task. Basic IR Processes

Transcription

1 CS630 Repesenting and Accessing Digital Infomation Infomation Retieval: Basics Thosten Joachims Conell Univesity Infomation Retieval Basics Retieval Models Indexing and Pepocessing Data Stuctues ~ 4 lectues Based on slides fom Pof. Jamie Callan and Pof. Claie Cadie IR Basics Task definition Evaluation Statistical popeties of text Use Task Retieval The field of infomation etieval deals with the epesentation, stoage, oganization of, access to infomation items. Bowsing Database Basic IR Pocesses Task Definition: Ad-hoc Retieval Infomation Need Repesentation Document Repesentation Seach a lage collection of documents to find the ones that satisfy an infomation need I.e., find elevant documents Sometimes called achival etieval Example: Web seach engines Quey Indexed Objects Compaison Evaluation/Feedback Retieved Objects

2 Settings fo Ad-hoc Retieval Unanked ad-hoc etieval Retun an unodeed set of documents that satisfies the quey Usually used on in Boolean etieval systems (which you ll hea about soon enough) Disadvantages: Impotant to ceate a good quey, so that the etieved set is small Small set may not have enough elevant documents??? Advantages??? Settings fo Ad-hoc Retieval Ranked ad-hoc etieval Retun a set of documents that satisfies the quey odeed by (pesumed) elevance Advantages Lage etieved sets ae not a poblem Less time spent cafting queies and eading documents Disadvantages Good queies ae still impotant??? Settings fo Ad-hoc Retieval Coss-lingual etieval (CLIR) Quey in one language (e.g. English) Retun documents in othe languages (e.g. Koean, Geek, Tamil) Sometimes called tanslingual etieval Settings fo Ad-hoc Retieval Distibuted etieval Ad-hoc etieval in a distibuted computing envionment many text collections eside on diffeent machines possibly diffeent IR system fo each machine Issues to addess include Database selection Meging esults fom diffeent databases IR Basics Task definition Evaluation Issues Test collections Metics Statistical popeties of text Evaluation in IR: Histoy Expeimental methodology has been a pominent component of IR eseach since 1960 s Ealy wok compaed manual vs. automatic indexing

3 Evaluation in IR: Histoy Manual vs. automatic indexing Could automatic indexing appoach manual quality? Issue: Humans ae not as consistent as they think! IR field developed methods of compaing oveall system pefomance Batch Inteactive Until 1990s, poblems of scale Types of Evaluation IR components that might be evaluated Ability to assist fomulating queies Speed of etieval Computing esouces equied Ability to find elevant documents Evaluation geneally compaative System A vs. system B System A vs. system A Most common evaluation measue Retieval effectiveness Ad-hoc Retieval Example Quey: ski aeas in New Yok Results: GoSki New Yok New Yok ski aeas, snow NY ski aeas on I Love NY touism guide Ski aeas in the Adiondack egion Pess Releases Lake Placid Ski aeas in Cental NY Ski aeas in Cotland County Ski aeas in the United States Nodic skiing ski aeas wap up season Geek Peek AYH nea ski aeas Relevance Relevance is difficult to define satisfactoily A elevant document is one judged useful in the context of a quey Who judges? What is useful? Issue of seendipitous utility Humans aen t consistent in thei judgments Judgment depends on moe than the document and quey With eal collections, the full set of elevant documents is neve known All etieval models include an implicit definition of elevance Test Collections Sample Test Collections Retieval pefomance is compaed using a test collection Set of documents, set of queies, set of elevance judgments To compae two techniques Each technique is used to evaluate queies Results (set o anked list) compaed using some metic Most common measues: pecision, ecall Usually use multiple measues to get diffeent pespectives Usually test with multiple test collections because pefomance is collection-dependent to some extent Canfield Size (documents) 1,400 Size (MB) 1.5 Yea ceated 1968 Wod stems 8,226 Stem occuences 123,200 Avg DocLen (wods) 88 Queies 225 CACM 3, , ISI 1, ,448 98, TREC2 742,611 2, ,040, ,800,

4 Finding Relevant Documents Evaluation Metics: Pecision and Recall Fo small test collections, can eview all documents fo a quey Not pactical fo lage collections Pooling Retieve documents using seveal techniques Judge top n documents fo each technique Relevant set is union of elevant documents fom each technique Relevant set is a subset of the tue elevant set Possible to estimate size of tue elevant set by sampling When testing: How should unjudged documents be teated? How might this affect the esults? Recall Pecentage of all elevant documents that ae found by a seach # of elevant items etieved R = # of elevant items in collection Pecision Pecentage of etieved documents that ae elevant # of elevant items etieved P = # of items etieved etieved R = 5/10 = 50% P = 5/8 = 62.5% Evaluation Metics: Pecision and Recall Pecision and ecall ae well-defined fo unanked etieval Unanked etieval poduces a set of documents Fo anked etieval The entie collection is anked (in theoy) Compute P at fixed ecall points (e.g. pecision at 20% ecall) Compute P at fixed ank cutoffs (e.g. pecision at ank 20) Recall Pecision Tables Pecision Method A Recall Method B 100 Pecision at Fixed Rank Cutoffs F-measue hamonic aveage of pecision and ecall F = 2*(PRECISION RECALL) (PRECISION RECALL) ewads esults that keep ecall and pecision close togethe R=40, P=60. R/P aveage = 50. F-measue= 48 R=45, P=55. R/P aveage = 50. F-measue= 49.5

5 BeakEvenPoint beak even point is the point at which ecall equals pecision IR Basics Task definition Evaluation Statistical popeties of text Zipf s Law Collocations and Co-occuences Statistical Popeties of Text Thee ae stable, language-independent pattens in how people use natual language A few wods occu vey fequently; most occu aely most common wods fom Tom Sawye In geneal Top 2 wods ~ 10-15% of all wod occuences Top 6 wods ~ 20% of all wod occuences Top 50 wods ~ 50% of all wod occuences 1 14 The And A To Of Tom With peposition Statistical Popeties of Text The most fequent wods in one copus may be ae wods in anothe copus Example: compute in CACM vs. National Geogaphic Each copus has a diffeent, faily small woking vocabulay These popeties hold in a wide ange of languages Zipf s Law Zipf s Law elates a tem s fequency to its ank fequency 1/ank Thee is a constant k such that fequency * ank = k Rank the tems in a vocabulay by fequency, in descending ode f : fequency of tem at ank N : total numbe of wod occuences Zipf s Law p =f / N and V p = = 1 1 Empiical obsevation: p = A /, A 0.1 Hence: f A p = = f = AN k N/10 fo English N

6 Pedicting Occuences of Fequencies A wod that occus n times has ank Example: n=50, A=0.1, N=100,000 n = , = 200 n = AN n Seveal wods may occu n times; assume ank n applies to last wod that occus n times n wods occu moe than n times n1 wods occu moe than n1 times Pedicting Occuences of Fequencies The numbe of wods that occu exactly n times is: Highest anking tem occus once and has ank max =AN/1 Popotion of wods with fequency 1 is: ( n( 1) ) In = n n 1 = AN n AN ( n 1) = AN n I n ( n( 1) ) max = 1 n (independent of text length and A) Popotion of wods occuing once is 1/2 Statistical Popeties of Text Summay: Tem usage is highly skewed, but in a pedictable patten Why is it impotant to know the chaacteistics of text? Optimization of data stuctues Statistical etieval algoithms depend on them Statistical Pofiles Can act as a summaization device Indicate what a document is about Indicate what a collection is about Collocations and Co-occuences A collocation is an expession consisting of two o moe wods that occu in a paticula ode and coespond to some conventional way of saying things Noun phases (e.g. a stiff beeze, weapons of mass destuction) Phasal vebs (e.g. to make up) Stock phases (e.g. the ich and famous, vim and vigo) Two wods co-occu if they appea in the same context (in geneal) o the same text (in IR) Co-occuence pattens docto with nuse, honoay, dentist, teat, examined, bills, etc. people and companies Ted Tune with Tune Boadcasting, Atlanta Baves, etc.