Efficient Top-k Algorithms for Fuzzy Search in String Collections
|
|
|
- Dortha Rachel Hancock
- 6 years ago
- Views:
Transcription
1 Efficient Top-k Algorithms for Fuzzy Search in String Collections Rares Vernica Chen Li Department of Computer Science University of California, Irvine First International Workshop on Keyword Search on Structured Data, 2009 Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
2 Outline 1 Motivation 2 Efficient Top-k Algorithms Problem Formulation Algorithms Overview Top-k Single-Pass Search Algorithm 3 Experimental Evaluation Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
3 Need for Approximate String Queries ID FirstName LastName # Movies 10 Al Swartzberg 1 11 Hanna Wartenegg 1 12 Rik Swartzwelder Joey Swartzentruber 1 14 Rene Swartenbroekx 4 15 Arnold Schwarzenegger Luc Swartenbroeckx 1 Figure: Actor names and popularities Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
4 Need for Approximate String Queries ID FirstName LastName # Movies 10 Al Swartzberg 1 11 Hanna Wartenegg 1 12 Rik Swartzwelder Joey Swartzentruber 1 14 Rene Swartenbroekx 4 15 Arnold Schwarzenegger Luc Swartenbroeckx 1 Figure: Actor names and popularities SELECT * FROM Actors WHERE LastName = Shwartzenetrugger ORDER BY # Movies DESC LIMIT 1; Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
5 Need for Approximate String Queries ID FirstName LastName # Movies 10 Al Swartzberg 1 11 Hanna Wartenegg 1 12 Rik Swartzwelder Joey Swartzentruber 1 14 Rene Swartenbroekx 4 15 Arnold Schwarzenegger Luc Swartenbroeckx 1 Figure: Actor names and popularities SELECT * FROM Actors WHERE LastName = Shwartzenetrugger ORDER BY # Movies DESC LIMIT 1; 0 Results Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
6 Need for Ranking ID FirstName LastName # Movies ED 10 Al Swartzberg Hanna Wartenegg Rik Swartzwelder Joey Swartzentruber Rene Swartenbroekx Arnold Schwarzenegger Luc Swartenbroeckx 1 9 Figure: Actor names, popularities, and edit distances to a query string Shwartzenetrugger Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
7 Need for Ranking ID FirstName LastName # Movies ED 10 Al Swartzberg Hanna Wartenegg Rik Swartzwelder Joey Swartzentruber Rene Swartenbroekx Arnold Schwarzenegger Luc Swartenbroeckx 1 9 Figure: Actor names, popularities, and edit distances to a query string Shwartzenetrugger Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
8 Need for Ranking ID FirstName LastName # Movies ED 10 Al Swartzberg Hanna Wartenegg Rik Swartzwelder Joey Swartzentruber Rene Swartenbroekx Arnold Schwarzenegger Luc Swartenbroeckx 1 9 Figure: Actor names, popularities, and edit distances to a query string Shwartzenetrugger Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
9 Need for Ranking ID FirstName LastName # Movies ED 10 Al Swartzberg Hanna Wartenegg Rik Swartzwelder Joey Swartzentruber Rene Swartenbroekx Arnold Schwarzenegger Luc Swartenbroeckx 1 9 Figure: Actor names, popularities, and edit distances to a query string Shwartzenetrugger Which one result should the system return? Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
10 Need for Ranking ID FirstName LastName # Movies ED 10 Al Swartzberg Hanna Wartenegg Rik Swartzwelder Joey Swartzentruber Rene Swartenbroekx Arnold Schwarzenegger Luc Swartenbroeckx 1 9 Figure: Actor names, popularities, and edit distances to a query string Shwartzenetrugger Which one result should the system return? Which value is more important, # Movies or similarity? Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
11 Top-k Similar Strings Given: Weighted string collection eg, actors LastName and # Movies Query string eg, Shwartzenetrugger Similarity function eg, Edit Distance Scoring function (score of a string in terms of similarity and weight) eg, linear combination of similarity and popularity Integer k Return: k best strings in terms of overall score to the query string Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
12 Top-k Similar Strings Given: Weighted string collection eg, actors LastName and # Movies Query string eg, Shwartzenetrugger Similarity function eg, Edit Distance Scoring function (score of a string in terms of similarity and weight) eg, linear combination of similarity and popularity Integer k Return: k best strings in terms of overall score to the query string Advantages over Range Search: specify k instead of a similarity threshold guaranteed k results; a range search might have 0 results Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
13 Algorithms Overview Range Search Range Search Iterative Range Search Single-Pass Search Two-Phase Search Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
14 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
15 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
16 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
17 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
18 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
19 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn,ni Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
20 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn,ni,ic Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
21 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn,ni,ic,ca} Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
22 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn,ni,ic,ca} Veronica {Ve,er,ro,on,ni,ic,ca} Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
23 q-grams: Overlapping substrings of fixed length Find similar strings: eg, Vernica and Veronica q-gram: substring of length q of a string: eg, q = 2 Vernica {Ve,er,rn,ni,ic,ca} Veronica {Ve,er,ro,on,ni,ic,ca} Similar strings share a certain number of grams Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
24 q-gram Inverted List Index q = 2 ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
25 q-gram Inverted List Index q = 2 ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
26 q-gram Inverted List Index q = 2 ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Query string: bcd Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
27 q-gram Inverted List Index q = 2 ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Query string: bcd Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
28 q-gram Inverted List Index q = 2 ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Query string: bcd Identified strings are verified by computing the real similarity Verification is usually an expensive process Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
29 Top-k Single-pass Search Algorithm ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
30 Top-k Single-pass Search Algorithm Setup Assign IDs st ascending order of IDs descending order of weights ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
31 Top-k Single-pass Search Algorithm Setup Assign IDs st ascending order of IDs descending order of weights Sort the IDs on each list in ascending order ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
32 Top-k Single-pass Search Algorithm Setup Assign IDs st ascending order of IDs descending order of weights Sort the IDs on each list in ascending order Scan the lists corresponding to the grams in the query eg, bcd ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
33 Top-k Single-pass Search Algorithm Naïve approach: Round-Robin Scan all the lists in the same time Maintain a list of open IDs (might still appear on some of the lists) Store the best k closed IDs in a top-k buffer Stop when the top-k buffer cannot improve ID String Weight 1 ab ccd cd abcd bcc 040 Figure: Dataset ab cc cd bc Figure: Gram inverted-list index Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
34 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
35 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 1 Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
36 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 1 2 Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
37 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
38 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
39 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 1 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
40 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 1 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
41 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 1 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
42 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = 1 20 Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
43 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
44 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
45 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
46 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
47 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = 1 20 Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
48 Top-k Single-pass Search Algorithm Heap-based Traverse the lists in a sorted order using a heap on the top IDs of the lists Advantages: 1 No need to maintain the list of open IDs 2 Skip elements ab bc cd Figure: Gram inverted-lists for query abcd 2 Figure: Top-k buffer, k = 1 20 Figure: Minheap Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
49 Algorithms Overview Range Search Range Search Iterative Range Search Single-Pass Search Two-Phase Search Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
50 Experimental Setting Datasets: IMDB Actor Names 1 actor names and the number of movies they played in 12 million actors, average name length 15 weight is the number of movies (log normalized) WEB Corpus Word Grams 2 sequences of English words and their frequency on the Web 24 million sequences, average sequence length 20 weight is the frequency (log normalized) Jaccard similarity and normalized edit similarity, q = 3 Index and data are stored in main memory at all times Implemented in C++ (GNU compiler) on Ubuntu Linux OS Intel 240GHz PC, 2GB RAM LDC2006T13 Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
51 Benefits of Skipping Elements 50 Time (ms) SPS SPS* Average running time for top-10 queries IMDB dataset with Jaccard similarity Single-Pass Search (SPS) algorithm and SPS without skipping (SPS*) Dataset Size (millions) Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
52 Potential of the Two-Phase Algorithm 40 Time (ms) Running time for 3 top-10 queries WEB Corpus dataset with normalized edit similarity Two-Phase algorithm with different initial thresholds 0 Q1 Q2 Q3 Queries Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
53 Optimum Initial Threshold for the Two-Phase Algorithm Time (ms) SPS 2PH 2PH Opt Dataset Size (millions) Average running time for top-10 queries Web Corpus dataset with normalized edit similarity Single-Pass Search (SPS) algorithm, Two-Phase (2PH) algorithm, 2PH with the optimum initial threshold (2PH Opt) The Iterative Range Search algorithm was to expensive to be plotted The average running time was around 5s Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
54 Summary Approximate ranking queries in string collections Useful when mismatch between query and data Propose three approaches to solve the problem: 1 Use existing approximate range search algorithms as a black box Proves to be the most expensive 2 Use particularities of the top-k problem Proves to be very efficient 3 Combine (1) and (2) sequentially Proves to be slightly more efficient Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
55 The Flamingo Project This work is part of The Flamingo Project at UC Irvine Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
56 A Quick Note on Related Work Fagin et al [1] similarity on multiple numerical attributes traverse list of IDs one list per attribute lists sorted on similarity to that attribute lists have different orders of IDs all IDs appear on all the lists Our Setting similarity on one string attribute traverse list of IDs one list per q-gram lists sorted on global weight lists have the same order of IDs a subset of IDs appear on each list [1] RFagin, A Lotem, and M Naor Optimal Aggregation Algorithms for Middleware In PODS, 2001 Rares Vernica (UC Irvine) Top-k Fuzzy String Search KEYS / 17
Combining Fuzzy Information - Top-k Query Algorithms. Sanjay Kulhari
Combining Fuzzy Information - Top-k Query Algorithms Sanjay Kulhari Outline Definitions Objects, Attributes and Scores Querying Fuzzy Data Top-k query algorithms Naïve Algorithm Fagin s Algorithm (FA)
Mind the Gap: Large-Scale Frequent Sequence Mining
Mind the Gap: Large-Scale Frequent Sequence Mining Iris Miliaraki Klaus Berberich Rainer Gemulla Spyros Zoupanos Max Planck Institute for Informatics Saarbrücken, Germany SIGMOD 2013 27 th June 2013, New
modern database systems lecture 5 : top-k retrieval
modern database systems lecture 5 : top-k retrieval Aristides Gionis Michael Mathioudakis spring 2016 announcements problem session on Monday, March 7, 2-4pm, at T2 solutions of the problems in homework
Fast Algorithms for Top-k Approximate String Matching
Fast Algorithms for Top-k Approximate String Matching Zhenglu Yang # Jianjun Yu Masaru Kitsuregawa # # Institute of Industrial Science, The University of Tokyo, Japan {yangzl, kitsure}@tkliisu-tokyoacjp
Analyzing the performance of top-k retrieval algorithms. Marcus Fontoura Google, Inc
Analyzing the performance of top-k retrieval algorithms Marcus Fontoura Google, Inc This talk Largely based on the paper Evaluation Strategies for Top-k Queries over Memory-Resident Inverted Indices, VLDB
Tree-Pattern Queries on a Lightweight XML Processor
Tree-Pattern Queries on a Lightweight XML Processor MIRELLA M. MORO Zografoula Vagena Vassilis J. Tsotras Research partially supported by CAPES, NSF grant IIS 0339032, UC Micro, and Lotus Interworks Outline
Integrating rankings: Problem statement
Integrating rankings: Problem statement Each object has m grades, oneforeachofm criteria. The grade of an object for field i is x i. Normally assume 0 x i 1. Typically evaluations based on different criteria
Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval
Basic Tokenizing, Indexing, and Implementation of Vector-Space Retrieval 1 Naïve Implementation Convert all documents in collection D to tf-idf weighted vectors, d j, for keyword vocabulary V. Convert
Set Cover Algorithms For Very Large Datasets
Set Cover Algorithms For Very Large Datasets Graham Cormode Howard Karloff AT&T Labs-Research Tony Wirth University of Melbourne Set Cover? Given a collection of sets over a universe of items Find smallest
The interaction of theory and practice in database research
The interaction of theory and practice in database research Ron Fagin IBM Research Almaden 1 Purpose of This Talk Encourage collaboration between theoreticians and system builders via two case studies
Indexing in Search Engines based on Pipelining Architecture using Single Link HAC
Indexing in Search Engines based on Pipelining Architecture using Single Link HAC Anuradha Tyagi S. V. Subharti University Haridwar Bypass Road NH-58, Meerut, India ABSTRACT Search on the web is a daily
Indexing Variable Length Substrings for Exact and Approximate Matching
Indexing Variable Length Substrings for Exact and Approximate Matching Gonzalo Navarro 1, and Leena Salmela 2 1 Department of Computer Science, University of Chile gnavarro@dcc.uchile.cl 2 Department of
Query Answering Using Inverted Indexes
Query Answering Using Inverted Indexes Inverted Indexes Query Brutus AND Calpurnia J. Pei: Information Retrieval and Web Search -- Query Answering Using Inverted Indexes 2 Document-at-a-time Evaluation
Efficient Computation of Data Cubes. Network Database Lab
Efficient Computation of Data Cubes Network Database Lab Outlines Introduction Some CUBE Algorithms ArrayCube PartitionedCube and MemoryCube Bottom-Up Cube (BUC) Conclusions References Network Database
Pathfinder/MonetDB: A High-Performance Relational Runtime for XQuery
Introduction Problems & Solutions Join Recognition Experimental Results Introduction GK Spring Workshop Waldau: Pathfinder/MonetDB: A High-Performance Relational Runtime for XQuery Database & Information
Keyword Search Using General Form of Inverted Index
Keyword Search Using General Form of Inverted Index Mrs. Pratiksha P. Nikam Prof. Srinu Dharavath Mr. Kunal Gawande Lecturer Professor(Guide) Software developer GSMCOE, Pune,india GSMCOE, Pune,india Mumbai,
tablename ORDER BY column ASC tablename ORDER BY column DESC sortingorder, } The WHERE and ORDER BY clauses can be combined in one
} The result of a query can be sorted in ascending or descending order using the optional ORDER BY clause. The simplest form of an ORDER BY clause is SELECT columnname1, columnname2, FROM tablename ORDER
Parallel Similarity Join with Data Partitioning for Prefix Filtering
22 ECTI TRANSACTIONS ON COMPUTER AND INFORMATION TECHNOLOGY VOL.9, NO.1 May 2015 Parallel Similarity Join with Data Partitioning for Prefix Filtering Jaruloj Chongstitvatana 1 and Methus Bhirakit 2, Non-members
Chapter 5, Data Cube Computation
CSI 4352, Introduction to Data Mining Chapter 5, Data Cube Computation Young-Rae Cho Associate Professor Department of Computer Science Baylor University A Roadmap for Data Cube Computation Full Cube Full
Learning Ontology-Based User Profiles: A Semantic Approach to Personalized Web Search
1 / 33 Learning Ontology-Based User Profiles: A Semantic Approach to Personalized Web Search Bernd Wittefeld Supervisor Markus Löckelt 20. July 2012 2 / 33 Teaser - Google Web History http://www.google.com/history
Indexing and Searching
Indexing and Searching Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University References: 1. Modern Information Retrieval, chapter 8 2. Information Retrieval:
Data Warehousing and Data Mining
Data Warehousing and Data Mining Lecture 3 Efficient Cube Computation CITS3401 CITS5504 Wei Liu School of Computer Science and Software Engineering Faculty of Engineering, Computing and Mathematics Acknowledgement:
Popularity of Twitter Accounts: PageRank on a Social Network
Popularity of Twitter Accounts: PageRank on a Social Network A.D-A December 8, 2017 1 Problem Statement Twitter is a social networking service, where users can create and interact with 140 character messages,
Parallelizing Frequent Itemset Mining with FP-Trees
Parallelizing Frequent Itemset Mining with FP-Trees Peiyi Tang Markus P. Turkia Department of Computer Science Department of Computer Science University of Arkansas at Little Rock University of Arkansas
Ingo Brenckmann Jochen Kirsten Storage Technology Strategists SAS EMEA Copyright 2003, SAS Institute Inc. All rights reserved.
Intelligent Storage Results from real life testing Ingo Brenckmann Jochen Kirsten Storage Technology Strategists SAS EMEA SAS Intelligent Storage components! OLAP Server! Scalable Performance Data Server!
Fast-Join: An Efficient Method for Fuzzy Token Matching based String Similarity Join
Fast-Join: An Efficient Method for Fuzzy Token Matching based String Similarity Join Jiannan Wang Guoliang Li Jianhua Feng Tsinghua National Laboratory for Information Science and Technology, Department
CS 171: Introduction to Computer Science II. Binary Search Trees
CS 171: Introduction to Computer Science II Binary Search Trees Binary Search Trees Symbol table applications BST definitions and terminologies Search and insert Traversal Ordered operations Delete Symbol
Quadrant-Based MBR-Tree Indexing Technique for Range Query Over HBase
Quadrant-Based MBR-Tree Indexing Technique for Range Query Over HBase Bumjoon Jo and Sungwon Jung (&) Department of Computer Science and Engineering, Sogang University, 35 Baekbeom-ro, Mapo-gu, Seoul 04107,
Query Languages. Berlin Chen Reference: 1. Modern Information Retrieval, chapter 4
Query Languages Berlin Chen 2005 Reference: 1. Modern Information Retrieval, chapter 4 Data retrieval Pattern-based querying The Kinds of Queries Retrieve docs that contains (or exactly match) the objects
Analysis of Algorithms
Algorithm An algorithm is a procedure or formula for solving a problem, based on conducting a sequence of specified actions. A computer program can be viewed as an elaborate algorithm. In mathematics and
Birkbeck (University of London)
Birkbeck (University of London) MSc Examination for Internal Students Department of Computer Science and Information Systems Information Retrieval and Organisation (COIY64H7) Credit Value: 5 Date of Examination:
Indexing and Searching
Indexing and Searching Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University References: 1. Modern Information Retrieval, chapter 9 2. Information Retrieval:
Assignment 3 ITCS-6010/8010: Cloud Computing for Data Analysis
Assignment 3 ITCS-6010/8010: Cloud Computing for Data Analysis Due by 11:59:59pm on Tuesday, March 16, 2010 This assignment is based on a similar assignment developed at the University of Washington. Running
CS222P Fall 2017, Final Exam
STUDENT NAME: STUDENT ID: CS222P Fall 2017, Final Exam Principles of Data Management Department of Computer Science, UC Irvine Prof. Chen Li (Max. Points: 100 + 15) Instructions: This exam has seven (7)
Visual C# 2012 How to Program by Pe ars on Ed uc ati on, Inc. All Ri ght s Re ser ve d.
Visual C# 2012 How to Program 1 99 2-20 14 by Pe ars on Ed uc ati on, Inc. All Ri ght s Re ser ve d. 1992-2014 by Pearson Education, Inc. All 1992-2014 by Pearson Education, Inc. All Although commonly
Comparison of of parallel and random approach to
Comparison of of parallel and random approach to acandidate candidatelist listininthe themultifeature multifeaturequerying Peter Gurský Peter Gurský Institute of Computer Science, Faculty of Science Institute
ARM: Authenticated Approximate Record Matching for Outsourced Databases. Boxiang Dong, Wendy Wang Stevens Institute of Technology Hoboken, NJ
ARM: Authenticated Approximate Record Matching for Outsourced Databases Boxiang Dong, Wendy Wang Stevens Institute of Technology Hoboken, NJ Approximate Record Matching DD: dataset ss qq : target record
Efficient Dynamic Pruning with Proximity Support
Efficient Dynamic Pruning with Proximity Support Nicola Tonellotto Information Science and Technologies Institute National Research Council Via G. Moruzzi, 56 Pisa, Italy nicola.tonellotto@isti.cnr.it
SEASHORE / SARUMAN. Short Read Matching using GPU Programming. Tobias Jakobi
SEASHORE SARUMAN Summary 1 / 24 SEASHORE / SARUMAN Short Read Matching using GPU Programming Tobias Jakobi Center for Biotechnology (CeBiTec) Bioinformatics Resource Facility (BRF) Bielefeld University
Predictive Indexing for Fast Search
Predictive Indexing for Fast Search Sharad Goel, John Langford and Alex Strehl Yahoo! Research, New York Modern Massive Data Sets (MMDS) June 25, 2008 Goel, Langford & Strehl (Yahoo! Research) Predictive
AN IMPROVISED FREQUENT PATTERN TREE BASED ASSOCIATION RULE MINING TECHNIQUE WITH MINING FREQUENT ITEM SETS ALGORITHM AND A MODIFIED HEADER TABLE
AN IMPROVISED FREQUENT PATTERN TREE BASED ASSOCIATION RULE MINING TECHNIQUE WITH MINING FREQUENT ITEM SETS ALGORITHM AND A MODIFIED HEADER TABLE Vandit Agarwal 1, Mandhani Kushal 2 and Preetham Kumar 3
A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK
A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK Qing Guo 1, 2 1 Nanyang Technological University, Singapore 2 SAP Innovation Center Network,Singapore ABSTRACT Literature review is part of scientific
Just Sort. Sathish Kumar Vijayakumar Chennai, India (1)
Just Sort Sathish Kumar Vijayakumar Chennai, India satthhishkumar@gmail.com Abstract Sorting is one of the most researched topics of Computer Science and it is one of the essential operations across computing
GLADE: A Scalable Framework for Efficient Analytics. Florin Rusu University of California, Merced
GLADE: A Scalable Framework for Efficient Analytics Florin Rusu University of California, Merced Motivation and Objective Large scale data processing Map-Reduce is standard technique Targeted to distributed
Information Science 2
Information Science 2 - Basic Search Algorithms- Week 04 College of Information Science and Engineering Ritsumeikan University Agenda Week 03 review Search algorithms on linear data structures - Linear
Leveraging Set Relations in Exact Set Similarity Join
Leveraging Set Relations in Exact Set Similarity Join Xubo Wang, Lu Qin, Xuemin Lin, Ying Zhang, and Lijun Chang University of New South Wales, Australia University of Technology Sydney, Australia {xwang,lxue,ljchang}@cse.unsw.edu.au,
CPS510 Database System Design Primitive SYSTEM STRUCTURE
CPS510 Database System Design Primitive SYSTEM STRUCTURE Naïve Users Application Programmers Sophisticated Users Database Administrator DBA Users Application Interfaces Application Programs Query Data
CSC Web Programming. Introduction to SQL
CSC 242 - Web Programming Introduction to SQL SQL Statements Data Definition Language CREATE ALTER DROP Data Manipulation Language INSERT UPDATE DELETE Data Query Language SELECT SQL statements end with
MG4J: Managing Gigabytes for Java. MG4J - intro 1
MG4J: Managing Gigabytes for Java MG4J - intro 1 Managing Gigabytes for Java Schedule: 1. Introduction to MG4J framework. 2. Exercitation: try to set up a search engine on a particular collection of documents.
Supporting Fuzzy Keyword Search in Databases
I J C T A, 9(24), 2016, pp. 385-391 International Science Press Supporting Fuzzy Keyword Search in Databases Jayavarthini C.* and Priya S. ABSTRACT An efficient keyword search system computes answers as
Diversification of Query Interpretations and Search Results
Diversification of Query Interpretations and Search Results Advanced Methods of IR Elena Demidova Materials used in the slides: Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova,
Reduce and Aggregate: Similarity Ranking in Multi-Categorical Bipartite Graphs
Reduce and Aggregate: Similarity Ranking in Multi-Categorical Bipartite Graphs Alessandro Epasto J. Feldman*, S. Lattanzi*, S. Leonardi, V. Mirrokni*. *Google Research Sapienza U. Rome Motivation Recommendation
Text Analytics. Index-Structures for Information Retrieval. Ulf Leser
Text Analytics Index-Structures for Information Retrieval Ulf Leser Content of this Lecture Inverted files Storage structures Phrase and proximity search Building and updating the index Using a RDBMS Ulf
A Graph Theoretic Approach to Image Database Retrieval
A Graph Theoretic Approach to Image Database Retrieval Selim Aksoy and Robert M. Haralick Intelligent Systems Laboratory Department of Electrical Engineering University of Washington, Seattle, WA 98195-2500
Applied Databases. Sebastian Maneth. Lecture 7 Simple SQL Queries. University of Edinburgh - February 1 st, 2016
Applied Databases Lecture 7 Simple SQL Queries Sebastian Maneth University of Edinburgh - February 1 st, 2016 Outline 2 1. Structured Querying Language (SQL) 2. Creating Tables 3. Simple SQL queries SQL
Efficient in-network evaluation of multiple queries
Efficient in-network evaluation of multiple queries Vinayaka Pandit 1 and Hui-bo Ji 2 1 IBM India Research Laboratory, email:pvinayak@in.ibm.com 2 Australian National University, email: hui-bo.ji@anu.edu.au
Induction of Association Rules: Apriori Implementation
1 Induction of Association Rules: Apriori Implementation Christian Borgelt and Rudolf Kruse Department of Knowledge Processing and Language Engineering School of Computer Science Otto-von-Guericke-University
Applied Databases. Sebastian Maneth. Lecture 7 Simple SQL Queries. University of Edinburgh - February 6 st, 2017
Applied Databases Lecture 7 Simple SQL Queries Sebastian Maneth University of Edinburgh - February 6 st, 2017 Outline 2 1. Structured Querying Language (SQL) 2. Creating Tables 3. Simple SQL queries SQL
Similarity Joins in MapReduce
Similarity Joins in MapReduce Benjamin Coors, Kristian Hunt, and Alain Kaeslin KTH Royal Institute of Technology {coors,khunt,kaeslin}@kth.se Abstract. This paper studies how similarity joins can be implemented
MASSACHUSETTS INSTITUTE OF TECHNOLOGY Database Systems: Fall 2017 Quiz I
Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.830 Database Systems: Fall 2017 Quiz I There are 15 questions and 12 pages in this quiz booklet. To receive
End-Term Examination Second Semester [MCA] MAY-JUNE 2006
![End-Term Examination Second Semester [MCA] MAY-JUNE 2006](/thumbs/79/79983336.jpg "End-Term Examination Second Semester [MCA] MAY-JUNE 2006")
(Please write your Roll No. immediately) Roll No. Paper Code: MCA-102 End-Term Examination Second Semester [MCA] MAY-JUNE 2006 Subject: Data Structure Time: 3 Hours Maximum Marks: 60 Note: Question 1.
Computing n-gram Statistics in MapReduce
Computing n-gram Statistics in MapReduce Klaus Berberich (kberberi@mpi-inf.mpg.de) Srikanta Bedathur (bedathur@iiitd.ac.in) n-gram Statistics Statistics about variable-length word sequences (e.g., lord
Connecting Software Connect Bridge [Performance Benchmark for Data Manipulation on Dynamics CRM via CB-Linked-Server]
![Connecting Software Connect Bridge [Performance Benchmark for Data Manipulation on Dynamics CRM via CB-Linked-Server]](/thumbs/88/115772065.jpg "Connecting Software Connect Bridge [Performance Benchmark for Data Manipulation on Dynamics CRM via CB-Linked-Server]")
Connect Bridge [Performance Benchmark for Data Manipulation on Dynamics CRM via CB-Linked-Server] Document History Version Date Author Changes 1.0 21 Apr 2016 SKE Creation Summary [This document provides
Information Retrieval. (M&S Ch 15)
Information Retrieval (M&S Ch 15) 1 Retrieval Models A retrieval model specifies the details of: Document representation Query representation Retrieval function Determines a notion of relevance. Notion
ΕΠΛ660. Ανάκτηση µε το µοντέλο διανυσµατικού χώρου
Ανάκτηση µε το µοντέλο διανυσµατικού χώρου Σηµερινό ερώτηµα Typically we want to retrieve the top K docs (in the cosine ranking for the query) not totally order all docs in the corpus can we pick off docs
Ranked Retrieval. Evaluation in IR. One option is to average the precision scores at discrete. points on the ROC curve But which points?
Ranked Retrieval One option is to average the precision scores at discrete Precision 100% 0% More junk 100% Everything points on the ROC curve But which points? Recall We want to evaluate the system, not
Natural Language Processing
Natural Language Processing Language Models Language models are distributions over sentences N gram models are built from local conditional probabilities Language Modeling II Dan Klein UC Berkeley, The
Market baskets Frequent itemsets FP growth. Data mining. Frequent itemset Association&decision rule mining. University of Szeged.
Frequent itemset Association&decision rule mining University of Szeged What frequent itemsets could be used for? Features/observations frequently co-occurring in some database can gain us useful insights
CS 222/122C Fall 2017, Final Exam. Sample solutions
CS 222/122C Fall 2017, Final Exam Principles of Data Management Department of Computer Science, UC Irvine Prof. Chen Li (Max. Points: 100 + 15) Sample solutions Question 1: Short questions (15 points)
Information Retrieval Rank aggregation. Luca Bondi
Rank aggregation Luca Bondi Motivations 2 Metasearch For a given query, combine the results from different search engines Combining ranking functions Text, links, anchor text, page title, etc. Comparing
Motivation for Sorting. External Sorting: Overview. Outline. CSE 190D Database System Implementation. Topic 3: Sorting. Chapter 13 of Cow Book
Motivation for Sorting CSE 190D Database System Implementation Arun Kumar User s SQL query has ORDER BY clause! First step of bulk loading of a B+ tree index Used in implementations of many relational
On Multiple Query Optimization in Data Mining
On Multiple Query Optimization in Data Mining Marek Wojciechowski, Maciej Zakrzewicz Poznan University of Technology Institute of Computing Science ul. Piotrowo 3a, 60-965 Poznan, Poland {marek,mzakrz}@cs.put.poznan.pl
Query Evaluation Strategies
Introduction to Search Engine Technology Term-at-a-Time and Document-at-a-Time Evaluation Ronny Lempel Yahoo! Research (Many of the following slides are courtesy of Aya Soffer and David Carmel, IBM Haifa
Parallelizing String Similarity Join Algorithms
Parallelizing String Similarity Join Algorithms Ling-Chih Yao and Lipyeow Lim University of Hawai i at Mānoa, Honolulu, HI 96822, USA {lingchih,lipyeow}@hawaii.edu Abstract. A key operation in data cleaning
A Combined Semi-Pipelined Query Processing Architecture For Distributed Full-Text Retrieval
A Combined Semi-Pipelined Query Processing Architecture For Distributed Full-Text Retrieval Simon Jonassen and Svein Erik Bratsberg Department of Computer and Information Science Norwegian University of
Dynamic Grouping Strategy in Cloud Computing
IEEE CGC 2012, November 1-3, Xiangtan, Hunan, China Dynamic Grouping Strategy in Cloud Computing Presenter : Qin Liu a Joint work with Yuhong Guo b, Jie Wu b, and Guojun Wang a a Central South University,
Outline. Database Management and Tuning. Index Data Structures. Outline. Index Tuning. Johann Gamper. Unit 5
Outline Database Management and Tuning Johann Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE Unit 5 1 2 Conclusion Acknowledgements: The slides are provided by Nikolaus Augsten
A Case Study in Optimizing GNU Radio s ATSC Flowgraph
A Case Study in Optimizing GNU Radio s ATSC Flowgraph Presented by Greg Scallon and Kirby Cartwright GNU Radio Conference 2017 Thursday, September 14 th 10am ATSC FLOWGRAPH LOADING 3% 99% 76% 36% 10% 33%
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing
Mining Frequent Patterns without Candidate Generation
Mining Frequent Patterns without Candidate Generation Outline of the Presentation Outline Frequent Pattern Mining: Problem statement and an example Review of Apriori like Approaches FP Growth: Overview
Guoping Wang and Chee-Yong Chan Department of Computer Science, School of Computing National University of Singapore VLDB 14.
Guoping Wang and Chee-Yong Chan Department of Computer Science, School of Computing National University of Singapore VLDB 14 Page 1 Introduction & Notations Multi-Job optimization Evaluation Conclusion
Parallel Merge Sort with Double Merging
Parallel with Double Merging Ahmet Uyar Department of Computer Engineering Meliksah University, Kayseri, Turkey auyar@meliksah.edu.tr Abstract ing is one of the fundamental problems in computer science.
Face Image Quality Assessment for Face Selection in Surveillance Video using Convolutional Neural Networks
Face Image Quality Assessment for Face Selection in Surveillance Video using Convolutional Neural Networks Vignesh Sankar, K. V. S. N. L. Manasa Priya, Sumohana Channappayya Indian Institute of Technology
An Approach for Privacy Preserving in Association Rule Mining Using Data Restriction
International Journal of Engineering Science Invention Volume 2 Issue 1 January. 2013 An Approach for Privacy Preserving in Association Rule Mining Using Data Restriction Janakiramaiah Bonam 1, Dr.RamaMohan
Apriori Algorithm. 1 Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke
Apriori Algorithm For a given set of transactions, the main aim of Association Rule Mining is to find rules that will predict the occurrence of an item based on the occurrences of the other items in the
Applying Multi-Core Model Checking to Hardware-Software Partitioning in Embedded Systems
V Brazilian Symposium on Computing Systems Engineering Applying Multi-Core Model Checking to Hardware-Software Partitioning in Embedded Systems Alessandro Trindade, Hussama Ismail, and Lucas Cordeiro Foz
Similarity search in multimedia databases
Similarity search in multimedia databases Performance evaluation for similarity calculations in multimedia databases JO TRYTI AND JOHAN CARLSSON Bachelor s Thesis at CSC Supervisor: Michael Minock Examiner:
Outline 1 Motivation 2 Theory of a non-blocking benchmark 3 The benchmark and results 4 Future work
Using Non-blocking Operations in HPC to Reduce Execution Times David Buettner, Julian Kunkel, Thomas Ludwig Euro PVM/MPI September 8th, 2009 Outline 1 Motivation 2 Theory of a non-blocking benchmark 3
Efficient Common Items Extraction from Multiple Sorted Lists
00 th International Asia-Pacific Web Conference Efficient Common Items Extraction from Multiple Sorted Lists Wei Lu,, Chuitian Rong,, Jinchuan Chen, Xiaoyong Du,, Gabriel Pui Cheong Fung, Xiaofang Zhou
Predictive Indexing for Fast Search
Predictive Indexing for Fast Search Sharad Goel Yahoo! Research New York, NY 10018 goel@yahoo-inc.com John Langford Yahoo! Research New York, NY 10018 jl@yahoo-inc.com Alex Strehl Yahoo! Research New York,
D. Θ nlogn ( ) D. Ο. ). Which of the following is not necessarily true? . Which of the following cannot be shown as an improvement? D.
CSE 0 Name Test Fall 00 Last Digits of Mav ID # Multiple Choice. Write your answer to the LEFT of each problem. points each. The time to convert an array, with priorities stored at subscripts through n,
COMPARATIVE ANALYSIS OF POWER METHOD AND GAUSS-SEIDEL METHOD IN PAGERANK COMPUTATION
International Journal of Computer Engineering and Applications, Volume IX, Issue VIII, Sep. 15 www.ijcea.com ISSN 2321-3469 COMPARATIVE ANALYSIS OF POWER METHOD AND GAUSS-SEIDEL METHOD IN PAGERANK COMPUTATION
Instructor: Craig Duckett. Lecture 03: Tuesday, April 3, 2018 SQL Sorting, Aggregates and Joining Tables
Instructor: Craig Duckett Lecture 03: Tuesday, April 3, 2018 SQL Sorting, Aggregates and Joining Tables 1 Assignment 1 is due LECTURE 5, Tuesday, April 10 th, 2018 in StudentTracker by MIDNIGHT MID-TERM
Exploiting Index Pruning Methods for Clustering XML Collections
Exploiting Index Pruning Methods for Clustering XML Collections Ismail Sengor Altingovde, Duygu Atilgan and Özgür Ulusoy Department of Computer Engineering, Bilkent University, Ankara, Turkey {ismaila,
Ranking Spatial Data by Quality Preferences
International Journal of Engineering Research and Development e-issn: 2278-067X, p-issn: 2278-800X, www.ijerd.com Volume 10, Issue 7 (July 2014), PP.68-75 Ranking Spatial Data by Quality Preferences Satyanarayana
CS570 Introduction to Data Mining
CS570 Introduction to Data Mining Frequent Pattern Mining and Association Analysis Cengiz Gunay Partial slide credits: Li Xiong, Jiawei Han and Micheline Kamber George Kollios 1 Mining Frequent Patterns,
The Threshold Algorithm: from Middleware Systems to the Relational Engine
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL.?, NO.?,?? 1 The Threshold Algorithm: from Middleware Systems to the Relational Engine Nicolas Bruno Microsoft Research nicolasb@microsoft.com Hui(Wendy)
GLADE: A Scalable Framework for Efficient Analytics. Florin Rusu (University of California, Merced) Alin Dobra (University of Florida)
DE: A Scalable Framework for Efficient Analytics Florin Rusu (University of California, Merced) Alin Dobra (University of Florida) Big Data Analytics Big Data Storage is cheap ($100 for 1TB disk) Everything
Big Data Analytics Influx of data pertaining to the 4Vs, i.e. Volume, Veracity, Velocity and Variety
Holistic Analysis of Multi-Source, Multi- Feature Data: Modeling and Computation Challenges Big Data Analytics Influx of data pertaining to the 4Vs, i.e. Volume, Veracity, Velocity and Variety Abhishek
Introduction to Information Retrieval (Manning, Raghavan, Schutze)
Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 3 Dictionaries and Tolerant retrieval Chapter 4 Index construction Chapter 5 Index compression Content Dictionary data structures