Index Construction Introduction to Information Retrieval INF 141 Donald J. Patterson
|
|
|
- Meghan O’Brien’
- 6 years ago
- Views:
Transcription
1 Index Construction Introduction to Information Retrieval INF 141 Donald J. Patterson Content adapted from Hinrich Schütze
2 Index Construction Overview Introduction Hardware BSBI - Block sort-based indexing SPIMI - Single Pass in-memory indexing Distributed indexing Dynamic indexing Miscellaneous topics
3 Indices The index has a list of vector space models Every 1 Her 1 I 1 I'm 1 Jensen's 2 Julie 1 Letter 1 Most 1 all 1 allegedly 1 back 1 before 1 brings 2 brothers 1 could 1 days 1 dead 1 death 1 everything 1 for 1 from 1 full 1 happens 1 haunts 1 have 1 hear 3 her 1 husband 1 if 1 it 1 killing 1 letter 1 nothing 1 now 1 of 1 pray 1 read, 1 saved 1 sister 1 stands 1 story 1 the 2 they 1 time 1 trial 1 wonder 1 wrong 1 wrote
4 Indices Term-Document Matrix Capture Keywords A Column for Each Web Page (or Document ) A Row For Each Word (or Term ) This picture is deceptive it is really very sparse Our queries are terms - not documents We need to invert the vector space model To make postings
5 Introduction Terms
6 Introduction Terms Inverted index
7 Introduction Terms Inverted index (Term, Document) pairs
8 Introduction Terms Inverted index (Term, Document) pairs building blocks for working with Term-Document Matrices
9 Introduction Terms Inverted index (Term, Document) pairs building blocks for working with Term-Document Matrices Index construction (or indexing)
10 Introduction Terms Inverted index (Term, Document) pairs building blocks for working with Term-Document Matrices Index construction (or indexing) The process of building an inverted index from a corpus
11 Introduction Terms Inverted index (Term, Document) pairs building blocks for working with Term-Document Matrices Index construction (or indexing) The process of building an inverted index from a corpus Indexer
12 Introduction Terms Inverted index (Term, Document) pairs building blocks for working with Term-Document Matrices Index construction (or indexing) The process of building an inverted index from a corpus Indexer The system architecture and algorithm that constructs the index
13 Indices The index is built from term-document pairs (TERM,DOCUMENT) (1998, (Every, (Her, (I, (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly, (back, (before, (brings, (brothers, (could, (days, (dead, (death, (everything, (for, (from, (full, (happens, (haunts, (have, (hear, (her, (husband, (if, (it, (killing, (letter, (nothing, (now, (of, (pray, (read,, (saved, (sister, (stands, (story, (the, (they, (time, (trial, (wonder, (wrong, (wrote,
14 Indices The index is built from term-document pairs (TERM,DOCUMENT) (1998, (Every, (Her, (I, (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly, (back, (before, (brings, (brothers, (could, (days, (dead, (death, (everything, (for, (from, (full, (happens, (haunts, (have, (hear, (her, (husband, (if, (it, (killing, (letter, (nothing, (now, (of, (pray, (read,, (saved, (sister, (stands, (story, (the, (they, (time, (trial, (wonder, (wrong, (wrote,
15 Indices The index is built from term-document pairs (TERM,DOCUMENT) (1998, (Every, (Her, (I, (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly, (back, (before, (brings, (brothers, (could, (days, (dead, (death, (everything, (for, (from, (full, (happens, (haunts, (have, (hear, (her, (husband, (if, (it, (killing, (letter, (nothing, (now, (of, (pray, (read,, (saved, (sister, (stands, (story, (the, (they, (time, (trial, (wonder, (wrong, (wrote, Core indexing step is to sort by terms
16 Indices Term-document pairs make lists of postings (TERM,DOCUMENT, DOCUMENT, DOCUMENT,...) (1998, (Every, news.bbc.co.uk) (Her, (I, ) (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly,
17 Indices Term-document pairs make lists of postings (TERM,DOCUMENT, DOCUMENT, DOCUMENT,...) (1998, (Every, news.bbc.co.uk) (Her, (I, ) (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly, A posting is a list of all documents in which a term occurs.
18 Indices Term-document pairs make lists of postings (TERM,DOCUMENT, DOCUMENT, DOCUMENT,...) (1998, (Every, news.bbc.co.uk) (Her, (I, ) (I'm, (Jensen's, (Julie, (Letter, (Most, (all, (allegedly, A posting is a list of all documents in which a term occurs. This is inverted from how documents naturally occur
19 Introduction Terms How do we construct an index?
20 Introduction Interactions An indexer needs raw text We need crawlers to get the documents We need APIs to get the documents from data stores We need parsers (HTML, PDF, PowerPoint, etc.) to convert the documents Indexing the web means this has to be done web-scale
21 Introduction Construction Index construction in main memory is simple and fast. But: As we build the index we parse docs one at a time Final postings for a term are incomplete until the end. At postings per term, large collections demand a lot of space Intermediate results must be stored on disk
22 Index Construction Overview Introduction Hardware BSBI - Block sort-based indexing SPIMI - Single Pass in-memory indexing Distributed indexing Dynamic indexing Miscellaneous topics
23 Hardware in 2007 System Parameters Disk seek time = sec Transfer time per byte = sec Processor clock rate = sec Size of main memory = several GB Size of disk space = several TB
24 Hardware in 2007 System Parameters Disk Seek Time The amount of time to get the disk head to the data About 10 times slower than memory access We must utilize caching No data is transferred during seek Data is transferred from disk in blocks There is no additional overhead to read in an entire block 0.2 seconds to get 10 MB if it is one block 0.7 seconds to get 10 MB if it is stored in 100 blocks
25 Hardware in 2007 System Parameters Data is transferred from disk in blocks Operating Systems read data in blocks, so Reading one byte and reading one block take the same amount of time
26 Hardware in 2007 System Parameters Data transfers are done on the system bus, not by the processor The processor is not used during disk I/O Assuming an efficient decompression algorithm The total time of reading and then decompressing compressed data is usually less than reading uncompressed data.
27 Index Construction Overview Introduction Hardware BSBI - Block sort-based indexing SPIMI - Single Pass in-memory indexing Distributed indexing Dynamic indexing Miscellaneous topics
28 BSBI Reuters collection example (approximate # s) 800,000 documents from the Reuters news feed 200 terms per document 400,000 unique terms number of postings 100,000,000
29 BSBI Reuters collection example (approximate # s)
30 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time.
31 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time
32 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time Sort posting entries by term
33 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time Sort posting entries by term Then by document in each term
34 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time Sort posting entries by term Then by document in each term Doing this with random disk seeks is too slow
35 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time Sort posting entries by term Then by document in each term Doing this with random disk seeks is too slow e.g. If every comparison takes 2 disk seeks and N items need to be sorted with N log2(n) comparisons?
36 BSBI Reuters collection example (approximate # s) Sorting 100,000,000 records on disk is too slow because of disk seek time. Parse and build posting entries one at a time Sort posting entries by term Then by document in each term Doing this with random disk seeks is too slow e.g. If every comparison takes 2 disk seeks and N items need to be sorted with N log2(n) comparisons? 306ish days?
37 BSBI Reuters collection example (approximate # s)
38 BSBI Reuters collection example (approximate # s) 100,000,000 records
39 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons
40 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2
41 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds
42 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds = 442, minutes
43 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds = 442, minutes = 7, hours
44 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds = 442, minutes = 7, hours = days
45 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds = 442, minutes = 7, hours = days = 84% of a year
46 BSBI Reuters collection example (approximate # s) 100,000,000 records Nlog2(N) is = 2,657,542, comparisons 2 disk seeks per comparison = 13,287, seconds x 2 = 26,575, seconds = 442, minutes = 7, hours = days = 84% of a year = 1% of your life
47 BSBI - Block sort-based indexing Different way to sort index 12-byte records (term, doc, meta-data) Need to sort T= 100,000,000 such 12-byte records by term Define a block to have 1,600,000 such records can easily fit a couple blocks in memory we will be working with 64 such blocks Accumulate postings for each block (real blocks are bigger) Sort each block Write to disk Then merge
48 BSBI - Block sort-based indexing Different way to sort index Block Block Merged Postings (1998, (Every, (Her,news.google.com) (I'm,news.bbc.co.uk) (1998,news.google.com) (Her,news.bbc.co.uk) (I, (Jensen's, (1998, (1998,news.google.com) (Every, (Her,news.bbc.co.uk) (Her,news.google.com) (I, (I'm,news.bbc.co.uk) (Jensen's, Disk
49 BSBI - Block sort-based indexing BlockSortBasedIndexConstruction BlockSortBasedIndexConstruction() 1 n 0 2 while (all documents not processed) 3 do block ParseNextBlock() 4 BSBI-Invert(block) 5 WriteBlockToDisk(block, f n ) 6 MergeBlocks(f 1, f 2..., f n, f merged )
50 BSBI - Block sort-based indexing Block merge indexing Parse documents into (TermID, DocID) pairs until block is full Invert the block Sort the (TermID,DocID) pairs Compile into TermID posting lists Write the block to disk Then merge all blocks into one large postings file Need 2 copies of the data on disk (input then output)
51 BSBI - Block sort-based indexing Analysis of BSBI The dominant term is O(TlogT) T is the number of TermID,DocID pairs But in practice ParseNextBlock takes the most time Then MergingBlocks Again, disk seeks times versus memory access times
52
Index Construction Introduction to Information Retrieval INF 141/ CS 121 Donald J. Patterson
Index Construction Introduction to Information Retrieval INF 141/ CS 121 Donald J. Patterson Content adapted from Hinrich Schütze http://www.informationretrieval.org Index Construction Overview Introduction
Reuters collection example (approximate # s)
BSBI Reuters collection example (approximate # s) 800,000 documents from the Reuters news feed 200 terms per document 400,000 unique terms number of postings 100,000,000 BSBI Reuters collection example
Information Retrieval and Organisation
Information Retrieval and Organisation Dell Zhang Birkbeck, University of London 2015/16 IR Chapter 04 Index Construction Hardware In this chapter we will look at how to construct an inverted index Many
Information Retrieval
Introduction to Information Retrieval Lecture 4: Index Construction Plan Last lecture: Dictionary data structures Tolerant retrieval Wildcards This time: Spell correction Soundex Index construction Index
Index Construction. Dictionary, postings, scalable indexing, dynamic indexing. Web Search
Index Construction Dictionary, postings, scalable indexing, dynamic indexing Web Search 1 Overview Indexes Query Indexing Ranking Results Application Documents User Information analysis Query processing
Index construction CE-324: Modern Information Retrieval Sharif University of Technology
Index construction CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2014 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276, Stanford) Ch.
index construct Overview Overview Recap How to construct index? Introduction Index construction Introduction to Recap
to to Information Retrieval Index Construct Ruixuan Li Huazhong University of Science and Technology http://idc.hust.edu.cn/~rxli/ October, 2012 1 2 How to construct index? Computerese term document docid
Index construction CE-324: Modern Information Retrieval Sharif University of Technology
Index construction CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2016 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276, Stanford) Ch.
Information Retrieval
Introduction to Information Retrieval Lecture 4: Index Construction 1 Plan Last lecture: Dictionary data structures Tolerant retrieval Wildcards Spell correction Soundex a-hu hy-m n-z $m mace madden mo
Index Construction 1
Index Construction 1 October, 2009 1 Vorlage: Folien von M. Schütze 1 von 43 Index Construction Hardware basics Many design decisions in information retrieval are based on hardware constraints. We begin
PV211: Introduction to Information Retrieval
PV211: Introduction to Information Retrieval http://www.fi.muni.cz/~sojka/pv211 IIR 4: Index construction Handout version Petr Sojka, Hinrich Schütze et al. Faculty of Informatics, Masaryk University,
3-2. Index construction. Most slides were adapted from Stanford CS 276 course and University of Munich IR course.
3-2. Index construction Most slides were adapted from Stanford CS 276 course and University of Munich IR course. 1 Ch. 4 Index construction How do we construct an index? What strategies can we use with
Information Retrieval
Information Retrieval Suan Lee - Information Retrieval - 04 Index Construction 1 04 Index Construction - Information Retrieval - 04 Index Construction 2 Plan Last lecture: Dictionary data structures Tolerant
Index construction CE-324: Modern Information Retrieval Sharif University of Technology
Index construction CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2017 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276, Stanford) Ch.
Introduction to Information Retrieval
Introduction to Information Retrieval http://informationretrieval.org IIR 4: Index Construction Hinrich Schütze, Christina Lioma Institute for Natural Language Processing, University of Stuttgart 2010-05-04
CSCI 5417 Information Retrieval Systems Jim Martin!
CSCI 5417 Information Retrieval Systems Jim Martin! Lecture 4 9/1/2011 Today Finish up spelling correction Realistic indexing Block merge Single-pass in memory Distributed indexing Next HW details 1 Query
CSE 7/5337: Information Retrieval and Web Search Index construction (IIR 4)
CSE 7/5337: Information Retrieval and Web Search Index construction (IIR 4) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Schütze Institute for Natural
Introduc)on to. CS60092: Informa0on Retrieval
Introduc)on to CS60092: Informa0on Retrieval Ch. 4 Index construc)on How do we construct an index? What strategies can we use with limited main memory? Sec. 4.1 Hardware basics Many design decisions in
Introduction to Information Retrieval
Introduction to Information Retrieval http://informationretrieval.org IIR 4: Index Construction Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-16 Schütze:
Information Retrieval. Danushka Bollegala
Information Retrieval Danushka Bollegala Anatomy of a Search Engine Document Indexing Query Processing Search Index Results Ranking 2 Document Processing Format detection Plain text, PDF, PPT, Text extraction
Introduction to Information Retrieval
Introduction to Information Retrieval http://informationretrieval.org IIR 4: Index Construction Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-16 1/54 Overview
Building an Inverted Index
Building an Inverted Index Algorithms Memory-based Disk-based (Sort-Inversion) Sorting Merging (2-way; multi-way) 2 Memory-based Inverted Index Phase I (parse and read) For each document Identify distinct
Index construc-on. Friday, 8 April 16 1
Index construc-on Informa)onal Retrieval By Dr. Qaiser Abbas Department of Computer Science & IT, University of Sargodha, Sargodha, 40100, Pakistan qaiser.abbas@uos.edu.pk Friday, 8 April 16 1 4.1 Index
INDEX CONSTRUCTION 1
1 INDEX CONSTRUCTION PLAN Last lecture: Dictionary data structures Tolerant retrieval Wildcards Spell correction Soundex a-hu hy-m n-z $m mace madden This time: mo among amortize Index construction on
Information Retrieval
Introduction to CS3245 Lecture 5: Index Construction 5 CS3245 Last Time Dictionary data structures Tolerant retrieval Wildcards Spelling correction Soundex a-hu hy-m n-z $m mace madden mo among amortize
Introduction to. CS276: Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan. Lecture 4: Index Construction
Introduction to Information Retrieval CS276: Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 4: Index Construction 1 Plan Last lecture: Dictionary data structures
Information Retrieval
Introduction to CS3245 Lecture 5: Index Construction 5 Last Time Dictionary data structures Tolerant retrieval Wildcards Spelling correction Soundex a-hu hy-m n-z $m mace madden mo among amortize on abandon
Introduction to Information Retrieval
Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Hamid Rastegari Lecture 4: Index Construction Plan Last lecture: Dictionary data structures
Index Construction. Slides by Manning, Raghavan, Schutze
Introduction to Information Retrieval ΕΠΛ660 Ανάκτηση Πληροφοριών και Μηχανές Αναζήτησης ης Index Construction ti Introduction to Information Retrieval Plan Last lecture: Dictionary data structures Tolerant
Introduction to Information Retrieval (Manning, Raghavan, Schutze)
Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 3 Dictionaries and Tolerant retrieval Chapter 4 Index construction Chapter 5 Index compression Content Dictionary data structures
Informa(on Retrieval
Introduc*on to Informa(on Retrieval CS276: Informa*on Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 4: Index Construc*on Plan Last lecture: Dic*onary data structures Tolerant retrieval
Document Representation : Quiz
Document Representation : Quiz Q1. In-memory Index construction faces following problems:. (A) Scaling problem (B) The optimal use of Hardware resources for scaling (C) Easily keep entire data into main
Index construc-on. Friday, 8 April 16 1
Index construc-on Informa)onal Retrieval By Dr. Qaiser Abbas Department of Computer Science & IT, University of Sargodha, Sargodha, 40100, Pakistan qaiser.abbas@uos.edu.pk Friday, 8 April 16 1 4.3 Single-pass
The Anatomy of a Large-Scale Hypertextual Web Search Engine
The Anatomy of a Large-Scale Hypertextual Web Search Engine Article by: Larry Page and Sergey Brin Computer Networks 30(1-7):107-117, 1998 1 1. Introduction The authors: Lawrence Page, Sergey Brin started
Introduction to Information Retrieval
Introduction to Information Retrieval http://informationretrieval.org IIR 5: Index Compression Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-17 1/59 Overview
Lecture 3 Index Construction and Compression. Many thanks to Prabhakar Raghavan for sharing most content from the following slides
Lecture 3 Index Construction and Compression Many thanks to Prabhakar Raghavan for sharing most content from the following slides Recap of the previous lecture Tokenization Term equivalence Skip pointers
MapReduce: Simplified Data Processing on Large Clusters. By Stephen Cardina
MapReduce: Simplified Data Processing on Large Clusters By Stephen Cardina The Problem You have a large amount of raw data, such as a database or a web log, and you need to get some sort of derived data
Crawler. Crawler. Crawler. Crawler. Anchors. URL Resolver Indexer. Barrels. Doc Index Sorter. Sorter. URL Server
Authors: Sergey Brin, Lawrence Page Google, word play on googol or 10 100 Centralized system, entire HTML text saved Focused on high precision, even at expense of high recall Relies heavily on document
Querying Introduction to Information Retrieval INF 141 Donald J. Patterson. Content adapted from Hinrich Schütze
Introduction to Information Retrieval INF 141 Donald J. Patterson Content adapted from Hinrich Schütze http://www.informationretrieval.org Overview Boolean Retrieval Weighted Boolean Retrieval Zone Indices
Web Information Retrieval. Lecture 4 Dictionaries, Index Compression
Web Information Retrieval Lecture 4 Dictionaries, Index Compression Recap: lecture 2,3 Stemming, tokenization etc. Faster postings merges Phrase queries Index construction This lecture Dictionary data
Information Retrieval
Introduction to Information Retrieval CS3245 Information Retrieval Lecture 6: Index Compression 6 Last Time: index construction Sort- based indexing Blocked Sort- Based Indexing Merge sort is effective
Indexing Web pages. Web Search: Indexing Web Pages. Indexing the link structure. checkpoint URL s. Connectivity Server: Node table
Indexing Web pages Web Search: Indexing Web Pages CPS 296.1 Topics in Database Systems Indexing the link structure AltaVista Connectivity Server case study Bharat et al., The Fast Access to Linkage Information
Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval
Information Retrieval INFO 4300 / CS 4300! Web crawlers Retrieving web pages Crawling the web» Desktop crawlers» Document feeds File conversion Storing the documents Removing noise Desktop Crawls! Used
Chapter 2. Architecture of a Search Engine
Chapter 2 Architecture of a Search Engine Search Engine Architecture A software architecture consists of software components, the interfaces provided by those components and the relationships between them
The Right Read Optimization is Actually Write Optimization. Leif Walsh
The Right Read Optimization is Actually Write Optimization Leif Walsh leif@tokutek.com The Right Read Optimization is Write Optimization Situation: I have some data. I want to learn things about the world,
Administrative. Distributed indexing. Index Compression! What I did last summer lunch talks today. Master. Tasks
Administrative Index Compression! n Assignment 1? n Homework 2 out n What I did last summer lunch talks today David Kauchak cs458 Fall 2012 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture5-indexcompression.ppt
EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling
EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling Doug Downey Based partially on slides by Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze Announcements Project progress report
Multimedia Information Extraction and Retrieval Term Frequency Inverse Document Frequency
Multimedia Information Extraction and Retrieval Term Frequency Inverse Document Frequency Ralf Moeller Hamburg Univ. of Technology Acknowledgement Slides taken from presentation material for the following
Information Retrieval
Information Retrieval Suan Lee - Information Retrieval - 05 Index Compression 1 05 Index Compression - Information Retrieval - 05 Index Compression 2 Last lecture index construction Sort-based indexing
Instructor: Stefan Savev
LECTURE 2 What is indexing? Indexing is the process of extracting features (such as word counts) from the documents (in other words: preprocessing the documents). The process ends with putting the information
Index Compression. David Kauchak cs160 Fall 2009 adapted from:
Index Compression David Kauchak cs160 Fall 2009 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture5-indexcompression.ppt Administrative Homework 2 Assignment 1 Assignment 2 Pair programming?
How TokuDB Fractal TreeTM. Indexes Work. Bradley C. Kuszmaul. MySQL UC 2010 How Fractal Trees Work 1
MySQL UC 2010 How Fractal Trees Work 1 How TokuDB Fractal TreeTM Indexes Work Bradley C. Kuszmaul MySQL UC 2010 How Fractal Trees Work 2 More Information You can download this talk and others at http://tokutek.com/technology
Analyzing the performance of top-k retrieval algorithms. Marcus Fontoura Google, Inc
Analyzing the performance of top-k retrieval algorithms Marcus Fontoura Google, Inc This talk Largely based on the paper Evaluation Strategies for Top-k Queries over Memory-Resident Inverted Indices, VLDB
Efficiency. Efficiency: Indexing. Indexing. Efficiency Techniques. Inverted Index. Inverted Index (COSC 488)
Efficiency Efficiency: Indexing (COSC 488) Nazli Goharian nazli@cs.georgetown.edu Difficult to analyze sequential IR algorithms: data and query dependency (query selectivity). O(q(cf max )) -- high estimate-
Indexing. Jan Chomicki University at Buffalo. Jan Chomicki () Indexing 1 / 25
Indexing Jan Chomicki University at Buffalo Jan Chomicki () Indexing 1 / 25 Storage hierarchy Cache Main memory Disk Tape Very fast Fast Slower Slow (nanosec) (10 nanosec) (millisec) (sec) Very small Small
TREC 2003 Notebook Paper: Document Structure with IRTools
TREC 2003 Notebook Paper: Document Structure with IRTools Gregory B. Newby * Arctic Region Supercomputing Center University of Alaska Fairbanks Abstract The IRTools software toolkit was modified for 2003
Full-Text Indexing For Heritrix
Full-Text Indexing For Heritrix Project Advisor: Dr. Chris Pollett Committee Members: Dr. Mark Stamp Dr. Jeffrey Smith Darshan Karia CS298 Master s Project Writing 1 2 Agenda Introduction Heritrix Design
THE WEB SEARCH ENGINE
International Journal of Computer Science Engineering and Information Technology Research (IJCSEITR) Vol.1, Issue 2 Dec 2011 54-60 TJPRC Pvt. Ltd., THE WEB SEARCH ENGINE Mr.G. HANUMANTHA RAO hanu.abc@gmail.com
CS November 2018
Bigtable Highly available distributed storage Distributed Systems 19. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
CS November 2017
Bigtable Highly available distributed storage Distributed Systems 18. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
Search Engines. Information Retrieval in Practice
Search Engines Information Retrieval in Practice All slides Addison Wesley, 2008 Web Crawler Finds and downloads web pages automatically provides the collection for searching Web is huge and constantly
Logistics. CSE Case Studies. Indexing & Retrieval in Google. Review: AltaVista. BigTable. Index Stream Readers (ISRs) Advanced Search
CSE 454 - Case Studies Indexing & Retrieval in Google Some slides from http://www.cs.huji.ac.il/~sdbi/2000/google/index.htm Logistics For next class Read: How to implement PageRank Efficiently Projects
CSCI 5417 Information Retrieval Systems! What is Information Retrieval?
CSCI 5417 Information Retrieval Systems! Lecture 1 8/23/2011 Introduction 1 What is Information Retrieval? Information retrieval is the science of searching for information in documents, searching for
Information Retrieval
Multimedia Computing: Algorithms, Systems, and Applications: Information Retrieval and Search Engine By Dr. Yu Cao Department of Computer Science The University of Massachusetts Lowell Lowell, MA 01854,
COMP6237 Data Mining Searching and Ranking
COMP6237 Data Mining Searching and Ranking Jonathon Hare jsh2@ecs.soton.ac.uk Note: portions of these slides are from those by ChengXiang Cheng Zhai at UIUC https://class.coursera.org/textretrieval-001
Cache Optimisation. sometime he thought that there must be a better way
Cache sometime he thought that there must be a better way 2 Cache 1. Reduce miss rate a) Increase block size b) Increase cache size c) Higher associativity d) compiler optimisation e) Parallelism f) prefetching
ECE331: Hardware Organization and Design
ECE331: Hardware Organization and Design Lecture 25: Multilevel Caches & Data Access Strategies Adapted from Computer Organization and Design, Patterson & Hennessy, UCB Overview Last time: Associative
Information Retrieval II
Information Retrieval II David Hawking 30 Sep 2010 Machine Learning Summer School, ANU Session Outline Ranking documents in response to a query Measuring the quality of such rankings Case Study: Tuning
Midterm 2, CS186 Prof. Hellerstein, Spring 2012
Midterm 2, CS186 Prof. Hellerstein, Spring 2012 Please put all answers on the answer sheets. Make sure your name is on each answer sheet! B+- Trees Assume we have the following B+ Tree of order 2 in which
Introduction to Information Retrieval
Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Center for Information and Language Processing, University of Munich 2014-04-09 Schütze: Boolean
Advance Indexing. Limock July 3, 2014
Advance Indexing Limock July 3, 2014 1 Papers 1) Gurajada, Sairam : "On-line index maintenance using horizontal partitioning." Proceedings of the 18th ACM conference on Information and knowledge management.
Ges$one Avanzata dell Informazione Part A Full- Text Informa$on Management. Full- Text Indexing
Ges$one Avanzata dell Informazione Part A Full- Text Informa$on Management Full- Text Indexing Contents } Introduction } Inverted Indices } Construction } Searching 2 GAvI - Full- Text Informa$on Management:
Corso di Biblioteche Digitali
Corso di Biblioteche Digitali Vittore Casarosa casarosa@isti.cnr.it tel. 050-315 3115 cell. 348-397 2168 Ricevimento dopo la lezione o per appuntamento Valutazione finale 70-75% esame orale 25-30% progetto
Elementary IR: Scalable Boolean Text Search. (Compare with R & G )
Elementary IR: Scalable Boolean Text Search (Compare with R & G 27.1-3) Information Retrieval: History A research field traditionally separate from Databases Hans P. Luhn, IBM, 1959: Keyword in Context
Information Retrieval
Introduction to Information Retrieval Information Retrieval and Web Search Lecture 1: Introduction and Boolean retrieval Outline ❶ Course details ❷ Information retrieval ❸ Boolean retrieval 2 Course details
Information Retrieval. Lecture 3 - Index compression. Introduction. Overview. Characterization of an index. Wintersemester 2007
Information Retrieval Lecture 3 - Index compression Seminar für Sprachwissenschaft International Studies in Computational Linguistics Wintersemester 2007 1/ 30 Introduction Dictionary and inverted index:
Indexing. Lecture Objectives. Text Technologies for Data Science INFR Learn about and implement Boolean search Inverted index Positional index
Text Technologies for Data Science INFR11145 Indexing Instructor: Walid Magdy 03-Oct-2017 Lecture Objectives Learn about and implement Boolean search Inverted index Positional index 2 1 Indexing Process
Column Stores vs. Row Stores How Different Are They Really?
Column Stores vs. Row Stores How Different Are They Really? Daniel J. Abadi (Yale) Samuel R. Madden (MIT) Nabil Hachem (AvantGarde) Presented By : Kanika Nagpal OUTLINE Introduction Motivation Background
Information Retrieval
Introduction to Information Retrieval CS276 Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 1: Boolean retrieval Information Retrieval Information Retrieval (IR) is finding
Informatica Data Explorer Performance Tuning
Informatica Data Explorer Performance Tuning 2011 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)
MapReduce Algorithms
Large-scale data processing on the Cloud Lecture 3 MapReduce Algorithms Satish Srirama Some material adapted from slides by Jimmy Lin, 2008 (licensed under Creation Commons Attribution 3.0 License) Outline
1 o Semestre 2007/2008
Efficient Departamento de Engenharia Informática Instituto Superior Técnico 1 o Semestre 2007/2008 Outline 1 2 3 4 5 6 7 Outline 1 2 3 4 5 6 7 Text es An index is a mechanism to locate a given term in
SOURCERER: MINING AND SEARCHING INTERNET- SCALE SOFTWARE REPOSITORIES
SOURCERER: MINING AND SEARCHING INTERNET- SCALE SOFTWARE REPOSITORIES Introduction to Information Retrieval CS 150 Donald J. Patterson This content based on the paper located here: http://dx.doi.org/10.1007/s10618-008-0118-x
MG4J: Managing Gigabytes for Java. MG4J - intro 1
MG4J: Managing Gigabytes for Java MG4J - intro 1 Managing Gigabytes for Java Schedule: 1. Introduction to MG4J framework. 2. Exercitation: try to set up a search engine on a particular collection of documents.
CS 134: Operating Systems
CS 134: Operating Systems More Memory Management CS 134: Operating Systems More Memory Management 1 / 27 2 / 27 Overview Overview Overview Segmentation Recap Segmentation Recap Segmentation Recap Segmentation
Reducing Hit Times. Critical Influence on cycle-time or CPI. small is always faster and can be put on chip
Reducing Hit Times Critical Influence on cycle-time or CPI Keep L1 small and simple small is always faster and can be put on chip interesting compromise is to keep the tags on chip and the block data off
Parametric Search using In-memory Auxiliary Index
Parametric Search using In-memory Auxiliary Index Nishant Verman and Jaideep Ravela Stanford University, Stanford, CA {nishant, ravela}@stanford.edu Abstract In this paper we analyze the performance of
Recap: lecture 2 CS276A Information Retrieval
Recap: lecture 2 CS276A Information Retrieval Stemming, tokenization etc. Faster postings merges Phrase queries Lecture 3 This lecture Index compression Space estimation Corpus size for estimates Consider
HTRC Data API Performance Study
HTRC Data API Performance Study Yiming Sun, Beth Plale, Jiaan Zeng Amazon Indiana University Bloomington {plale, jiaazeng}@cs.indiana.edu Abstract HathiTrust Research Center (HTRC) allows users to access
Information Retrieval
Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 1: Boolean retrieval Information Retrieval Information Retrieval (IR)
Part 2: Boolean Retrieval Francesco Ricci
Part 2: Boolean Retrieval Francesco Ricci Most of these slides comes from the course: Information Retrieval and Web Search, Christopher Manning and Prabhakar Raghavan Content p Term document matrix p Information
Time-Series Data in MongoDB on a Budget. Peter Schwaller Senior Director Server Engineering, Percona Santa Clara, California April 23th 25th, 2018
Time-Series Data in MongoDB on a Budget Peter Schwaller Senior Director Server Engineering, Percona Santa Clara, California April 23th 25th, 2018 TIME SERIES DATA in MongoDB on a Budget Click to add text
An Overview of Search Engine. Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia
An Overview of Search Engine Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia haixu@microsoft.com July 24, 2007 1 Outline History of Search Engine Difference Between Software and
Text Analytics. Index-Structures for Information Retrieval. Ulf Leser
Text Analytics Index-Structures for Information Retrieval Ulf Leser Content of this Lecture Inverted files Storage structures Phrase and proximity search Building and updating the index Using a RDBMS Ulf
L02 : 08/21/2015 L03 : 08/24/2015.
L02 : 08/21/2015 http://www.csee.wvu.edu/~adjeroh/classes/cs493n/ Multimedia use to be the idea of Big Data Definition of Big Data is moving data (It will be different in 5..10 yrs). Big data is highly
Advanced Retrieval Information Analysis Boolean Retrieval
Advanced Retrieval Information Analysis Boolean Retrieval Irwan Ary Dharmawan 1,2,3 iad@unpad.ac.id Hana Rizmadewi Agustina 2,4 hagustina@unpad.ac.id 1) Development Center of Information System and Technology
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 3: Programming Models RCFile: A Fast and Space-efficient Data
Gerald's Column by Gerald Fitton
Gerald's Column by Gerald Fitton I've had quite a lot of feedback this month about some recent and some long ago articles; I wasn't sure whether to do a PipeDream/Fireworkz article, a domain name or email
ECE 3055: Final Exam
ECE 3055: Final Exam Instructions: You have 2 hours and 50 minutes to complete this quiz. The quiz is closed book and closed notes, except for one 8.5 x 11 sheet. No calculators are allowed. Multiple Choice
ΕΠΛ660. Ανάκτηση µε το µοντέλο διανυσµατικού χώρου
Ανάκτηση µε το µοντέλο διανυσµατικού χώρου Σηµερινό ερώτηµα Typically we want to retrieve the top K docs (in the cosine ranking for the query) not totally order all docs in the corpus can we pick off docs