TREC OpenSearch Planning Session
|
|
|
- Silvester Lang
- 5 years ago
- Views:
Transcription
1 TREC OpenSearch Planning Session Anne Schuth, Krisztian Balog
2 TREC OpenSearch OpenSearch is a new evaluation paradigm for IR. The experimentation platform is an existing search engine. Researchers have the opportunity to replace components of this search engine and evaluate these components using interactions with real, unsuspecting users of this search engine. TREC OpenSearch 2
3 TREC OpenSearch OpenSearch is a new evaluation paradigm for IR. The experimentation platform is an existing search engine. Researchers have the opportunity to replace components of this search engine and evaluate these components using interactions with real, unsuspecting users of this search engine. TREC OpenSearch 2
4 TREC OpenSearch OpenSearch is a new evaluation paradigm for IR. The experimentation platform is an existing search engine. Researchers have the opportunity to replace components of this search engine and evaluate these components using interactions with real, unsuspecting users of this search engine. TREC OpenSearch 2
5 TREC OpenSearch OpenSearch is a new evaluation paradigm for IR. The experimentation platform is an existing search engine. Researchers have the opportunity to replace components of this search engine and evaluate these components using interactions with real, unsuspecting users of this search engine. TREC OpenSearch 2
6 TREC OpenSearch For OpenSearch is a new evaluation paradigm for IR. The experimentation now, only platform reranking is an existing search engine. Researchers component have the opportunity to replace components of this search engine and evaluate these components using interactions with real, unsuspecting users of this search engine. TREC OpenSearch 2
7 Evaluation Setup TREC OpenSearch 3
8 Evaluation Setup Search Engine TREC OpenSearch 3
9 Evaluation Setup query Search Engine TREC OpenSearch 3
10 Evaluation Setup query ranking Search Engine TREC OpenSearch 3
11 Evaluation Setup query ranking click Search Engine TREC OpenSearch 3
12 Evaluation Setup query ranking click Living Labs API Search Engine TREC OpenSearch 3
13 Evaluation Setup Likely to be issued again in the future Queries query click ranking Living Labs API Search Engine TREC OpenSearch 3
14 Evaluation Setup Documents Queries query click ranking Living Labs API Search Engine TREC OpenSearch 3
15 Evaluation Setup Documents Queries REST-full API that speaks JSON Documents Queries query click ranking Participant Living Labs API Search Engine TREC OpenSearch 3
16 Evaluation Setup Documents Queries Documents Queries query click ranking Participant Living Labs API Search Engine TREC OpenSearch 3
17 Evaluation Setup Documents Queries Documents Queries query click ranking Participant Living Labs API Search Engine ranking TREC OpenSearch 3
18 Evaluation Setup Documents Queries Documents Queries query click ranking Participant Living Labs API Search Engine ranking TREC OpenSearch 3
19 Evaluation Setup Documents Queries Documents Queries query click ranking query Participant Living Labs API Search Engine ranking TREC OpenSearch 3
20 Evaluation Setup Documents Queries Documents Queries query click ranking query Participant No need to ask the Living Labs APIparticipant! Search Engine ranking ranking TREC OpenSearch 3
21 Evaluation Setup Documents Queries Documents Queries query click ranking query Participant Living Labs API Search Engine ranking ranking TREC OpenSearch 3
22 Evaluation Setup Documents Queries Documents Queries query click ranking query Participant Living Labs API Search Engine ranking ranking TREC OpenSearch 3
23 Evaluation Setup Documents Queries Documents Queries query click ranking query Participant Living Labs API click Search Engine ranking ranking TREC OpenSearch 3
24 Evaluation Setup Documents Queries Documents Queries query click ranking click query Participant Living Labs API click Search Engine ranking ranking TREC OpenSearch 3
25 Evaluation Setup Documents Queries Documents Queries query click ranking click query Participant Living Labs API click Search Engine ranking ranking TREC OpenSearch 3
26 Evaluation Setup Documents Queries Documents Queries query click ranking click query Participant Participant Participant Living Labs API click Search Engine Site Site ranking ranking TREC OpenSearch 3
27 Evaluation Setup Rankings shown to users are interleaved Documents Queries Documents Queries query click ranking click query Participant Participant Participant Living Labs API click Search Engine Site Site ranking ranking TREC OpenSearch 3
28 Evaluation Setup - Interleaving A doc 1 B doc 2 doc 2 doc 4 doc 3 doc 7 doc 4 doc 1 doc 5 doc 3 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
29 Evaluation Setup - Interleaving A B doc 3 doc 4 doc 5 doc 4 doc 7 doc 3 doc 1 doc 2 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
30 Evaluation Setup - Interleaving A B doc 1 doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
31 Evaluation Setup - Interleaving A B doc 1 doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
32 Evaluation Setup - Interleaving Infer winner: B > A doc 1 doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
33 Evaluation Setup - Interleaving Infer winner: B > A Well tested in practice Used at Bing, Yandex, Seznam doc 1 doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
34 Evaluation Setup - Interleaving Infer winner: B > A Well tested in practice Used at Bing, Yandex, Seznam Participants are not compared head to head but transitivity holds in practice: if A > B and C < B then A > C doc 1 doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
35 Evaluation Setup - Interleaving Infer winner: B > A Well tested in practice Used at Bing, Yandex, Seznam Participants are not compared head to head but transitivity holds in practice: if A > B and C < B then A > C Metric: fraction of wins doc against 1 the search engine doc 2 doc 4 doc 3 doc 7 F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM TREC OpenSearch 4
36 Test/Train Queries/Periods Query Type Train Test Period Train Test - Feedback Available - Individual Feedback - Update possible - Feedback Available - No Individual Feedback - Update possible - No Feedback Available - No Individual Feedback - Update not possible TREC OpenSearch 5
37 Test/Train Queries/Periods Query Type Train Test Period Train Test - Feedback Available - Individual Feedback - Update possible - Feedback Available - No Individual Feedback - Update possible - No Feedback Available - No Individual Feedback - Update not possible TREC OpenSearch 5
38 Test/Train Queries/Periods Query Type Train Test Period Train Test - Feedback Available - Individual Feedback - Update possible - Feedback Available - No Individual Feedback - Update possible - No Feedback Available - No Individual Feedback - Update not possible TREC OpenSearch 5
39 Documentation TREC OpenSearch 6
40 Documentation TREC OpenSearch 6
41 Documentation TREC OpenSearch 6
42 Advantages of this setup Realistic evaluation with real users No real-time requirement on participants A single implementation to experiment on many search engines TREC ad-hoc style queries + documents TREC OpenSearch 7
43 Limitations of this setup No tail queries No sessions No context TREC OpenSearch 8
44 Task Ad-hoc Academic Literature Search TREC OpenSearch 9
45 Task Ad-hoc Academic Literature Search Easy to comprehend TREC OpenSearch 9
46 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough TREC OpenSearch 9
47 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research TREC OpenSearch 9
48 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research E.g. entity linking, normalization TREC OpenSearch 9
49 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research E.g. entity linking, normalization Extendable in future years TREC OpenSearch 9
50 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research E.g. entity linking, normalization Extendable in future years related literature TREC OpenSearch 9
51 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research E.g. entity linking, normalization Extendable in future years related literature author/venue recommendation TREC OpenSearch 9
52 Task Ad-hoc Academic Literature Search Easy to comprehend The setup is already different enough Connects to current research E.g. entity linking, normalization Extendable in future years related literature author/venue recommendation subtasks in the form of several search engines TREC OpenSearch 9
53 Task - Academic Search Engines TREC OpenSearch 10
54 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing TREC OpenSearch 10
55 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR TREC OpenSearch 10
56 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR OpenEdition TREC OpenSearch 10
57 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR OpenEdition arxiv TREC OpenSearch 10
58 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR OpenEdition arxiv CiteSeerX TREC OpenSearch 10
59 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR OpenEdition arxiv CiteSeerX Google Scholar TREC OpenSearch 10
60 Task - Academic Search Engines contact enthusiasm commit match timeline implement clicks flowing SSOAR OpenEdition arxiv CiteSeerX Google Scholar MS Academic Search TREC OpenSearch 10
61 Timeline TREC OpenSearch 11
62 Timeline December: finalize search engine agreements TREC OpenSearch 11
63 Timeline December: finalize search engine agreements January: query + document selection TREC OpenSearch 11
64 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines TREC OpenSearch 11
65 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines March 1: release train queries TREC OpenSearch 11
66 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines March 1: release train queries March 15: clicks start flowing TREC OpenSearch 11
67 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines March 1: release train queries March 15: clicks start flowing May 15: release test queries TREC OpenSearch 11
68 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines March 1: release train queries March 15: clicks start flowing May 15: release test queries Jun 1: test period begins TREC OpenSearch 11
69 Timeline December: finalize search engine agreements January: query + document selection March 1: finalize guidelines March 1: release train queries March 15: clicks start flowing May 15: release test queries Jun 1: test period begins July 15: test period ends TREC OpenSearch 11
70 Discussion Task details Query selection Relevance assessments (!) Simulations Reproducibility TREC OpenSearch 12
71 Discussion - Task Details Document format to be discussed full articles may not (always) be possible metadata Collection Statistics? Author graph? Historical interactions?? TREC OpenSearch 13
72 Discussion - Query Selection Volume Head but few + higher sensitivity - not so interesting? Torso but many + more interesting? - lower sensitivity Mix of both? Language Type TREC OpenSearch 14
73 Discussion - Relevance Assessments Relevance assessments (for some queries) Compare Cranfield-style evaluation to OpenSearch-style evaluation Participants bidding on queries? No assessments (for the first year)? TREC OpenSearch 15
74 Discussion - Simulations Simulate a search engine Queries sampled according to frequency from log Noisy click model to produce interactions So that participants can verify their systems TREC OpenSearch 16
75 Discussion - Repeatability/Reproducibility Share all interactions after the test period? Have multiple test periods?? TREC OpenSearch 17
Modern Retrieval Evaluations. Hongning Wang
Modern Retrieval Evaluations Hongning Wang CS@UVa What we have known about IR evaluations Three key elements for IR evaluation A document collection A test suite of information needs A set of relevance
Historical Clicks for Product Search: GESIS at CLEF LL4IR 2015
Historical Clicks for Product Search: GESIS at CLEF LL4IR 2015 Philipp Schaer 1 and Narges Tavakolpoursaleh 12 1 GESIS Leibniz Institute for the Social Sciences, 50669 Cologne, Germany firstname.lastname@gesis.org
Jan Pedersen 22 July 2010
Jan Pedersen 22 July 2010 Outline Problem Statement Best effort retrieval vs automated reformulation Query Evaluation Architecture Query Understanding Models Data Sources Standard IR Assumptions Queries
Automatic Search Engine Evaluation with Click-through Data Analysis. Yiqun Liu State Key Lab of Intelligent Tech. & Sys Jun.
Automatic Search Engine Evaluation with Click-through Data Analysis Yiqun Liu State Key Lab of Intelligent Tech. & Sys Jun. 3th, 2007 Recent work: Using query log and click-through data analysis to: identify
Information Retrieval
Introduction Information Retrieval Information retrieval is a field concerned with the structure, analysis, organization, storage, searching and retrieval of information Gerard Salton, 1968 J. Pei: Information
Large-Scale Validation and Analysis of Interleaved Search Evaluation
Large-Scale Validation and Analysis of Interleaved Search Evaluation Olivier Chapelle, Thorsten Joachims Filip Radlinski, Yisong Yue Department of Computer Science Cornell University Decide between two
A Comparative Analysis of Interleaving Methods for Aggregated Search
5 A Comparative Analysis of Interleaving Methods for Aggregated Search ALEKSANDR CHUKLIN and ANNE SCHUTH, University of Amsterdam KE ZHOU, Yahoo Labs London MAARTEN DE RIJKE, University of Amsterdam A
Best Practice Guidelines for the Development and Evaluation of Digital Humanities Projects
Best Practice Guidelines for the Development and Evaluation of Digital Humanities Projects 1.0. Project team There should be a clear indication of who is responsible for the publication of the project.
Informativeness for Adhoc IR Evaluation:
Informativeness for Adhoc IR Evaluation: A measure that prevents assessing individual documents Romain Deveaud 1, Véronique Moriceau 2, Josiane Mothe 3, and Eric SanJuan 1 1 LIA, Univ. Avignon, France,
doi: / _32
doi: 10.1007/978-3-319-12823-8_32 Simple Document-by-Document Search Tool Fuwatto Search using Web API Masao Takaku 1 and Yuka Egusa 2 1 University of Tsukuba masao@slis.tsukuba.ac.jp 2 National Institute
Learning to Rank for Faceted Search Bridging the gap between theory and practice
Learning to Rank for Faceted Search Bridging the gap between theory and practice Agnes van Belle @ Berlin Buzzwords 2017 Job-to-person search system Generated query Match indicator Faceted search Multiple
Entity and Knowledge Base-oriented Information Retrieval
Entity and Knowledge Base-oriented Information Retrieval Presenter: Liuqing Li liuqing@vt.edu Digital Library Research Laboratory Virginia Polytechnic Institute and State University Blacksburg, VA 24061
Comparative Analysis of Clicks and Judgments for IR Evaluation
Comparative Analysis of Clicks and Judgments for IR Evaluation Jaap Kamps 1,3 Marijn Koolen 1 Andrew Trotman 2,3 1 University of Amsterdam, The Netherlands 2 University of Otago, New Zealand 3 INitiative
CS6200 Information Retrieval. Jesse Anderton College of Computer and Information Science Northeastern University
CS6200 Information Retrieval Jesse Anderton College of Computer and Information Science Northeastern University Major Contributors Gerard Salton! Vector Space Model Indexing Relevance Feedback SMART Karen
Retrieval Evaluation. Hongning Wang
Retrieval Evaluation Hongning Wang CS@UVa What we have learned so far Indexed corpus Crawler Ranking procedure Research attention Doc Analyzer Doc Rep (Index) Query Rep Feedback (Query) Evaluation User
Fighting Search Engine Amnesia: Reranking Repeated Results
Fighting Search Engine Amnesia: Reranking Repeated Results ABSTRACT Milad Shokouhi Microsoft milads@microsoft.com Paul Bennett Microsoft Research pauben@microsoft.com Web search engines frequently show
Building Test Collections. Donna Harman National Institute of Standards and Technology
Building Test Collections Donna Harman National Institute of Standards and Technology Cranfield 2 (1962-1966) Goal: learn what makes a good indexing descriptor (4 different types tested at 3 levels of
Learning Temporal-Dependent Ranking Models
Learning Temporal-Dependent Ranking Models Miguel Costa, Francisco Couto, Mário Silva LaSIGE @ Faculty of Sciences, University of Lisbon IST/INESC-ID, University of Lisbon 37th Annual ACM SIGIR Conference,
Information Retrieval
Introduction to Information Retrieval CS3245 Information Retrieval Lecture 9: IR Evaluation 9 Ch. 7 Last Time The VSM Reloaded optimized for your pleasure! Improvements to the computation and selection
A Large Scale Validation and Analysis of Interleaved Search Evaluation
A Large Scale Validation and Analysis of Interleaved Search Evaluation OLIVIER CHAPELLE, Yahoo! Research THORSTEN JOACHIMS, Cornell University FILIP RADLINSKI, Microsoft YISONG YUE, Carnegie Mellon University
Information Retrieval
Information Retrieval ETH Zürich, Fall 2012 Thomas Hofmann LECTURE 6 EVALUATION 24.10.2012 Information Retrieval, ETHZ 2012 1 Today s Overview 1. User-Centric Evaluation 2. Evaluation via Relevance Assessment
Information Retrieval
Introduction to Information Retrieval CS276 Information Retrieval and Web Search Chris Manning, Pandu Nayak and Prabhakar Raghavan Evaluation 1 Situation Thanks to your stellar performance in CS276, you
WHOIS Studies Update. Liz Gasster
WHOIS Studies Update Liz Gasster WHOIS Topics WHOIS Studies 4 studies: Misuse of public data Registrant Identification Proxy/Privacy Abuse Proxy/Privacy Relay and Reveal WHOIS Service Requirements Other
Evaluation. David Kauchak cs160 Fall 2009 adapted from:
Evaluation David Kauchak cs160 Fall 2009 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture8-evaluation.ppt Administrative How are things going? Slides Points Zipf s law IR Evaluation For
Information Retrieval
Information Retrieval Test Collections Gintarė Grigonytė gintare@ling.su.se Department of Linguistics and Philology Uppsala University Slides based on previous IR course given by K.F. Heppin 2013-15 and
IPv6 at Google. Lorenzo Colitti
IPv6 at Google Lorenzo Colitti lorenzo@google.com Why IPv6? IPv4 address space predictions (G. Huston) Why IPv6? Cost Buying addresses will be expensive Carrier-grade NAT may be expensive Lots of session
Evaluating Aggregated Search Using Interleaving
Evaluating Aggregated Search Using Interleaving Aleksandr Chuklin Yandex & ISLA, University of Amsterdam Moscow, Russia a.chuklin@uva.nl Pavel Serdyukov Yandex Moscow, Russia pavser@yandex-team.ru Anne
Advanced Topics in Information Retrieval. Learning to Rank. ATIR July 14, 2016
Advanced Topics in Information Retrieval Learning to Rank Vinay Setty vsetty@mpi-inf.mpg.de Jannik Strötgen jannik.stroetgen@mpi-inf.mpg.de ATIR July 14, 2016 Before we start oral exams July 28, the full
Using Historical Click Data to Increase Interleaving Sensitivity
Using Historical Click Data to Increase Interleaving Sensitivity Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, Iadh Ounis Yandex, Moscow, Russia School of Computing Science, University of Glasgow,
Overview of the NTCIR-13 OpenLiveQ Task
Overview of the NTCIR-13 OpenLiveQ Task Makoto P. Kato, Takehiro Yamamoto (Kyoto University), Sumio Fujita, Akiomi Nishida, Tomohiro Manabe (Yahoo Japan Corporation) Agenda Task Design (3 slides) Data
arxiv: v1 [cs.ir] 11 Oct 2018
![arxiv: v1 [cs.ir] 11 Oct 2018](/thumbs/89/98418313.jpg "arxiv: v1 [cs.ir] 11 Oct 2018")
Offline Comparison of Ranking Functions using Randomized Data Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork Google Mountain View, CA {agaman,xuanhui,chgli,bemike,najork}@google.com
Information Retrieval
Multimedia Computing: Algorithms, Systems, and Applications: Information Retrieval and Search Engine By Dr. Yu Cao Department of Computer Science The University of Massachusetts Lowell Lowell, MA 01854,
ICFHR 2014 COMPETITION ON HANDWRITTEN KEYWORD SPOTTING (H-KWS 2014)
ICFHR 2014 COMPETITION ON HANDWRITTEN KEYWORD SPOTTING (H-KWS 2014) IOANNIS PRATIKAKIS 1 KONSTANTINOS ZAGORIS 1,2 BASILIS GATOS 2 GEORGIOS LOULOUDIS 2 NIKOLAOS STAMATOPOULOS 2 1 2 Visual Computing Group
Managing Complex Link Building Campaigns.
Managing Complex Link Building Campaigns http://dejanseo.com.au Chapter One Lovely Shades of Grey High Risk Competitor Competitor Competitor You Low Risk Competitor Competitor Competitor You Competitor
CS506/606 - Topics in Information Retrieval
CS506/606 - Topics in Information Retrieval Instructors: Class time: Steven Bedrick, Brian Roark, Emily Prud hommeaux Tu/Th 11:00 a.m. - 12:30 p.m. September 25 - December 6, 2012 Class location: WCC 403
Search Evaluation. Tao Yang CS293S Slides partially based on text book [CMS] [MRS]
![Search Evaluation. Tao Yang CS293S Slides partially based on text book [CMS] [MRS]](/thumbs/75/72136869.jpg "Search Evaluation. Tao Yang CS293S Slides partially based on text book [CMS] [MRS]")
Search Evaluation Tao Yang CS293S Slides partially based on text book [CMS] [MRS] Table of Content Search Engine Evaluation Metrics for relevancy Precision/recall F-measure MAP NDCG Difficulties in Evaluating
Supporting Multi-Year Data in Ed-Fi Technology. Chris Moffatt Director of Technology, Ed-Fi Alliance
Supporting Multi-Year Data in Ed-Fi Technology Chris Moffatt Director of Technology, Ed-Fi Alliance chris.moffatt@ed-fi.org Session Abstract This session will provide a review of the activities of the
arxiv: v1 [cs.ir] 22 Feb 2018
![arxiv: v1 [cs.ir] 22 Feb 2018](/thumbs/96/127590528.jpg "arxiv: v1 [cs.ir] 22 Feb 2018")
Generating High-Quality Query Suggestion Candidates for Task-Based Search Heng Ding 1,2, Shuo Zhang 2, Darío Garigliotti 2, and Krisztian Balog 2 1 Wuhan University, Wuhan, China 2 University of Stavanger,
Distributed Time Travel for Feature Generation. Prasanna Padmanabhan DB Tsai Mohammad H. Taghavi
Distributed Time Travel for Feature Generation Prasanna Padmanabhan DB Tsai Mohammad H. Taghavi Turn on Netflix, and the absolute best content for you would automatically start playing Everything is a
SME License Order Working Group Update - Webinar #3 Call in number:
SME License Order Working Group Update - Webinar #3 Call in number: Canada Local: +1-416-915-8942 Canada Toll Free: +1-855-244-8680 Event Number: 662 298 966 Attendee ID: check your WebEx session under
WEB REDESIGN PROJECT. presented to President s Cabinet. presented by Eric Turner, Web and Portal Services Uyen Mai, Marketing and Communication
WEB REDESIGN PROJECT presented to President s Cabinet presented by Eric Turner, Web and Portal Services Uyen Mai, Marketing and Communication October 28, 2014 Web Redesign Goals More Audience Focused More
Learning Socially Optimal Information Systems from Egoistic Users
Learning Socially Optimal Information Systems from Egoistic Users Karthik Raman Thorsten Joachims Department of Computer Science Cornell University, Ithaca NY karthik@cs.cornell.edu www.cs.cornell.edu/
Towards Predicting Web Searcher Gaze Position from Mouse Movements
Towards Predicting Web Searcher Gaze Position from Mouse Movements Qi Guo Emory University 400 Dowman Dr., W401 Atlanta, GA 30322 USA qguo3@emory.edu Eugene Agichtein Emory University 400 Dowman Dr., W401
Adaptive Search at Essex
Adaptive Search at Essex Udo Kruschwitz School of Computer Science and Electronic Engineering University of Essex udo@essex.ac.uk 7th October 2011 Adaptive Search - LAC Day - 7 October 2011 1 (Source:
Release Notes December 2016
Release Notes December 2016 About the Release Notes... 3 Release Overview... 3 Other Announcements... 3 Enhancements... 4 Doc Launcher Forms... 4 External Review... 6 Multiple Documents in E-Signature...
Advanced Search Techniques for Large Scale Data Analytics Pavel Zezula and Jan Sedmidubsky Masaryk University
Advanced Search Techniques for Large Scale Data Analytics Pavel Zezula and Jan Sedmidubsky Masaryk University http://disa.fi.muni.cz The Cranfield Paradigm Retrieval Performance Evaluation Evaluation Using
Perspectives on Open Data in Science Open Data in Science: Challenges & Opportunities for Europe
Perspectives on Open Data in Science Open Data in Science: Challenges & Opportunities for Europe Stephane Berghmans, DVM PhD 31 January 2018 9 When talking about data, we talk about All forms of research
Letter Pair Similarity Classification and URL Ranking Based on Feedback Approach
Letter Pair Similarity Classification and URL Ranking Based on Feedback Approach P.T.Shijili 1 P.G Student, Department of CSE, Dr.Nallini Institute of Engineering & Technology, Dharapuram, Tamilnadu, India
This is an author-deposited version published in : Eprints ID : 15246
Open Archive TOULOUSE Archive Ouverte (OATAO) OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible. This is an author-deposited
Information Search in Web Archives
Information Search in Web Archives Miguel Costa Advisor: Prof. Mário J. Silva Co-Advisor: Prof. Francisco Couto Department of Informatics, Faculty of Sciences, University of Lisbon PhD thesis defense,
University of Virginia Department of Computer Science. CS 4501: Information Retrieval Fall 2015
University of Virginia Department of Computer Science CS 4501: Information Retrieval Fall 2015 2:00pm-3:30pm, Tuesday, December 15th Name: ComputingID: This is a closed book and closed notes exam. No electronic
Overview of the NTCIR-13 OpenLiveQ Task
Overview of the NTCIR-13 OpenLiveQ Task ABSTRACT Makoto P. Kato Kyoto University mpkato@acm.org Akiomi Nishida Yahoo Japan Corporation anishida@yahoo-corp.jp This is an overview of the NTCIR-13 OpenLiveQ
Improving the Reported Use and Impact of Institutional Repositories
Improving the Reported Use and Impact of Institutional Repositories Kenning Arlitsch, Dean - @kenning_msu Patrick OBrien, Semantic Web Research Director - @sempob Montana State University Library Coalition
The University of Illinois Graduate School of Library and Information Science at TREC 2011
The University of Illinois Graduate School of Library and Information Science at TREC 2011 Miles Efron, Adam Kehoe, Peter Organisciak, Sunah Suh 501 E. Daniel St., Champaign, IL 61820 1 Introduction The
IPv6 at Google. Lorenzo Colitti
IPv6 at Google Lorenzo Colitti lorenzo@google.com Why? IPv4 address space predictions (G. Huston) To put it into perspective... Iljitsch van Beijnum, Ars Technica Why IPv6? Cost Buying addresses will be
Comparing the Sensitivity of Information Retrieval Metrics
Comparing the Sensitivity of Information Retrieval Metrics Filip Radlinski Microsoft Cambridge, UK filiprad@microsoft.com Nick Craswell Microsoft Redmond, WA, USA nickcr@microsoft.com ABSTRACT Information
Search Engine Architecture II
Search Engine Architecture II Primary Goals of Search Engines Effectiveness (quality): to retrieve the most relevant set of documents for a query Process text and store text statistics to improve relevance
arxiv: v1 [cs.ir] 26 Nov 2017
![arxiv: v1 [cs.ir] 26 Nov 2017](/thumbs/76/73108567.jpg "arxiv: v1 [cs.ir] 26 Nov 2017")
Sensitive and Scalable Online Evaluation with Theoretical Guarantees arxiv:1711.09454v1 [cs.ir] 26 Nov 2017 ABSTRACT Harrie Oosterhuis University of Amsterdam Amsterdam, The Netherlands oosterhuis@uva.nl
Retrieval Evaluation
Retrieval Evaluation - Reference Collections Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University References: 1. Modern Information Retrieval, Chapter
for Searching Social Media Posts
Mining the Temporal Statistics of Query Terms for Searching Social Media Posts ICTIR 17 Amsterdam Oct. 1 st 2017 Jinfeng Rao Ferhan Ture Xing Niu Jimmy Lin Task: Ad-hoc Search on Social Media domain Stream
Scholarly Big Data: Leverage for Science
Scholarly Big Data: Leverage for Science C. Lee Giles The Pennsylvania State University University Park, PA, USA giles@ist.psu.edu http://clgiles.ist.psu.edu Funded in part by NSF, Allen Institute for
A Comparative Study of Click Models for Web Search
A Comparative Study of Click Models for Web Search Artem Grotov, Aleksandr Chuklin, Ilya Markov Luka Stout, Finde Xumara, and Maarten de Rijke University of Amsterdam, Amsterdam, The Netherlands {a.grotov,a.chuklin,i.markov}@uva.nl,
FEEDBACK ONLINE PARTICIPANT USER S GUIDE
FEEDBACK ONLINE PARTICIPANT USER S GUIDE March 2003 Accessing Feedback Online To access Feedback Online, type in the LMI website address http://www.lmi-inc.com/feedback/ Logon instructions Type in your
PROMISE Winter School 2013 Bridging between Information Retrieval and Databases
WINTER SCHOOL REPORT PROMISE Winter School 2013 Bridging between Information Retrieval and Databases Maristella Agosti, Nicola Ferro, Gianmaria Silvello University of Padua, Italy {agosti, ferro, silvello}@dei.unipd.it
Learning Ranking Functions with SVMs
Learning Ranking Functions with SVMs CS4780/5780 Machine Learning Fall 2012 Thorsten Joachims Cornell University T. Joachims, Optimizing Search Engines Using Clickthrough Data, Proceedings of the ACM Conference
External Query Reformulation for Text-based Image Retrieval
External Query Reformulation for Text-based Image Retrieval Jinming Min and Gareth J. F. Jones Centre for Next Generation Localisation School of Computing, Dublin City University Dublin 9, Ireland {jmin,gjones}@computing.dcu.ie
RAMP: Repository Analytics and Metrics Portal
RAMP: Repository Analytics and Metrics Portal Accurately measuring use of institutional repositories Kenning Arlitsch, Dean @kenning_msu Patrick OBrien, Semantic Web Research Director @sempob OCLC RLP
Evaluating Search Engines by Clickthrough Data
Evaluating Search Engines by Clickthrough Data Jing He and Xiaoming Li Computer Network and Distributed System Laboratory, Peking University, China {hejing,lxm}@pku.edu.cn Abstract. It is no doubt that
Pilot Testing Information
500 North Bronson Avenue Big Rapids, MI 49307 800.334.6283 www.nocti.org Pilot Testing Information NOCTI is seeking sites to pilot test our newly developed/revised assessments. Pilot testing is a critical
Ranking Web of Repositories Metrics, results and a plea for a change. Isidro F. Aguillo Cybermetrics Lab CCHS - CSIC
Ranking Web of Repositories Metrics, results and a plea for a change Isidro F. Aguillo Cybermetrics Lab CCHS - CSIC Isidro.aguillo@cchs.csic.es 1 Agenda 00:00 A complex scenario: Open Access Initiatives
Adaptive Search Engines Learning Ranking Functions with SVMs
Adaptive Search Engines Learning Ranking Functions with SVMs CS478/578 Machine Learning Fall 24 Thorsten Joachims Cornell University T. Joachims, Optimizing Search Engines Using Clickthrough Data, Proceedings
Implementing Web-Scale Discovery in an Academic Research Library
Implementing Web-Scale Discovery in an Academic Research Library Usability Testing November 2008 Tammy Allgood, Web Services Librarian Evaluation of library web site home page July 2009 Tammy Allgood,
Evaluating the Accuracy of. Implicit feedback. from Clicks and Query Reformulations in Web Search. Learning with Humans in the Loop
Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search Thorsten Joachims, Filip Radlinski, Geri Gay, Laura Granka, Helene Hembrooke, Bing Pang Department of Computer
End-to-End Neural Ad-hoc Ranking with Kernel Pooling
End-to-End Neural Ad-hoc Ranking with Kernel Pooling Chenyan Xiong 1,Zhuyun Dai 1, Jamie Callan 1, Zhiyuan Liu, and Russell Power 3 1 :Language Technologies Institute, Carnegie Mellon University :Tsinghua
Overview of the NTCIR-12 MobileClick-2 Task
Overview of the NTCIR-12 MobileClick-2 Task Makoto P. Kato (Kyoto U.), Tetsuya Sakai (Waseda U.), Takehiro Yamamoto (Kyoto U.), Virgil Pavlu (Northeastern U.), Hajime Morita (Kyoto U.), and Sumio Fujita
Overview of ITU capacity building activities
Europe Centers of Excellence Steering Committee Meeting Copenhagen, Denmark 12 December 2017 Overview of ITU capacity building activities Mike Nxele Senior Human Capacity Building Officer, Human Capacity
Challenges on Combining Open Web and Dataset Evaluation Results: The Case of the Contextual Suggestion Track
Challenges on Combining Open Web and Dataset Evaluation Results: The Case of the Contextual Suggestion Track Alejandro Bellogín 1,2, Thaer Samar 1, Arjen P. de Vries 1, and Alan Said 1 1 Centrum Wiskunde
WEB ACCESSIBILITY. I. Policy Section Information Technology. Policy Subsection Web Accessibility Policy.
WEB ACCESSIBILITY I. Policy Section 15.0 Information Technology II. Policy Subsection 15.2 Web Accessibility Policy III. Policy Statement Grand Rapids Community College is actively committed to providing
Evaluation of Retrieval Systems
Evaluation of Retrieval Systems 1 Performance Criteria 1. Expressiveness of query language Can query language capture information needs? 2. Quality of search results Relevance to users information needs
CSCI 5417 Information Retrieval Systems. Jim Martin!
CSCI 5417 Information Retrieval Systems Jim Martin! Lecture 7 9/13/2011 Today Review Efficient scoring schemes Approximate scoring Evaluating IR systems 1 Normal Cosine Scoring Speedups... Compute the
A Web Service for Scholarly Big Data Information Extraction
A Web Service for Scholarly Big Data Information Extraction Kyle Williams, Lichi Li, Madian Khabsa, Jian Wu, Patrick C. Shih and C. Lee Giles Information Sciences and Technology Computer Science and Engineering
Release Notes May 2017
Release Notes May 2017 About the Release Notes... 3 Release Overview... 3 Other Announcements... 4 SpringCM Login name change... 4 Workflow Step and Workflows Initiated Reports are being deprecated...
What s new in SharePoint Search 2010 for end users. IW109 Mirjam van Olst
What s new in SharePoint Search 2010 for end users IW109 Mirjam van Olst About Mirjam Microsoft Certified Master SharePoint 2007 MVP SharePoint Server SharePoint Architect at Macaw Co-organizer DIWUG and
Learning Ranking Functions with SVMs
Learning Ranking Functions with SVMs CS4780/5780 Machine Learning Fall 2014 Thorsten Joachims Cornell University T. Joachims, Optimizing Search Engines Using Clickthrough Data, Proceedings of the ACM Conference
Relevance and Effort: An Analysis of Document Utility
Relevance and Effort: An Analysis of Document Utility Emine Yilmaz, Manisha Verma Dept. of Computer Science University College London {e.yilmaz, m.verma}@cs.ucl.ac.uk Nick Craswell, Filip Radlinski, Peter
Illinois Tollway Diversity Advisory Council. Gustavo Giraldo, Chief of the Department of Diversity and Strategic Development April 12, 2017
Illinois Tollway Diversity Advisory Council Gustavo Giraldo, Chief of the Department of Diversity and Strategic Development April 12, 2017 Diversity Report recommendations Technical Assistance Program
3/21/2016 AN INTRODUCTION TO SEARCH ENGINE OPTIMIZATION. Search Engine Optimization (SEO) Basics for Attorneys
AN INTRODUCTION TO SEARCH ENGINE OPTIMIZATION DCBA LAW PRACTICE MANAGEMENT & TECHNOLOGY SECTION MARCH 22, 2016 Presenter: Christine P. Miller, OVC Lawyer Marketing Search Engine Optimization (SEO) Basics
University of Delaware at Diversity Task of Web Track 2010
University of Delaware at Diversity Task of Web Track 2010 Wei Zheng 1, Xuanhui Wang 2, and Hui Fang 1 1 Department of ECE, University of Delaware 2 Yahoo! Abstract We report our systems and experiments
INEX REPORT. Report on INEX 2012
INEX REPORT Report on INEX 2012 P. Bellot T. Chappell A. Doucet S. Geva S. Gurajada J. Kamps G. Kazai M. Koolen M. Landoni M. Marx A. Mishra V. Moriceau J. Mothe M. Preminger G. Ramírez M. Sanderson E.
Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs
Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs Filip Radlinski and Thorsten Joachims Department of Computer Science Cornell University, Ithaca, NY {filip,tj}@cs.cornell.edu
Modern Information Retrieval
Modern Information Retrieval Chapter 3 Retrieval Evaluation Retrieval Performance Evaluation Reference Collections CFC: The Cystic Fibrosis Collection Retrieval Evaluation, Modern Information Retrieval,
Library. Summary Report
Library Summary Report 217-218 Prepared by: Library Staff December 218 Table of Contents Introduction..1 New Books.2 Print Circulation.3 Interlibrary Loan 4 Information Literacy 5 Reference Statistics.6
Improving Difficult Queries by Leveraging Clusters in Term Graph
Improving Difficult Queries by Leveraging Clusters in Term Graph Rajul Anand and Alexander Kotov Department of Computer Science, Wayne State University, Detroit MI 48226, USA {rajulanand,kotov}@wayne.edu
Extracting Algorithms by Indexing and Mining Large Data Sets
Extracting Algorithms by Indexing and Mining Large Data Sets Vinod Jadhav 1, Dr.Rekha Rathore 2 P.G. Student, Department of Computer Engineering, RKDF SOE Indore, University of RGPV, Bhopal, India Associate
Robust Relevance-Based Language Models
Robust Relevance-Based Language Models Xiaoyan Li Department of Computer Science, Mount Holyoke College 50 College Street, South Hadley, MA 01075, USA Email: xli@mtholyoke.edu ABSTRACT We propose a new
On Duplicate Results in a Search Session
On Duplicate Results in a Search Session Jiepu Jiang Daqing He Shuguang Han School of Information Sciences University of Pittsburgh jiepu.jiang@gmail.com dah44@pitt.edu shh69@pitt.edu ABSTRACT In this
Project Leader Workshop
Project Leader Workshop SAMANTHA GROVES: OWASP PROJECTS MANAGER November 2013 Workshop Workshop Overview The OWASP PM Role.! Projects Infrastructure: What is it and where can I find it?! OWASP Projects
Multilingual Image Search from a user s perspective
Multilingual Image Search from a user s perspective Julio Gonzalo, Paul Clough, Jussi Karlgren QUAERO-Image CLEF workshop, 16/09/08 Finding is a matter of two fast stupid smart slow great potential for
Passage Retrieval and other XML-Retrieval Tasks. Andrew Trotman (Otago) Shlomo Geva (QUT)
Passage Retrieval and other XML-Retrieval Tasks Andrew Trotman (Otago) Shlomo Geva (QUT) Passage Retrieval Information Retrieval Information retrieval (IR) is the science of searching for information in
Information Retrieval May 15. Web retrieval
Information Retrieval May 15 Web retrieval What s so special about the Web? The Web Large Changing fast Public - No control over editing or contents Spam and Advertisement How big is the Web? Practically