ezdl: An Interactive Search and Evaluation System

Transcription

1 ezdl: An Interactive Search and Evaluation System Workshop on Open Source Information Retrieval SIGIR 2012 Thomas Beckers, Sebastian Dungs, Norbert Fuhr, Matthias Jordan, Sascha Kriewel, Vu Tuan Tran (Demo) Universität Duisburg-Essen, Germany Portland, USA, August 16th 2012 Beckers et al. (UDE) ezdl August 16th / 35

2 Outline 1 Introduction and Concepts 2 Architecture 3 Evaluations 4 Use Cases 5 Conclusion Beckers et al. (UDE) ezdl August 16th / 35

3 Introduction and Concepts What is ezdl? ezdl is a... interactive search tool, development platform for search applications and system for different types of user-oriented evaluations. Beckers et al. (UDE) ezdl August 16th / 35

4 Introduction and Concepts Desktop frontend BC B A E D F Beckers et al. (UDE) ezdl August 16th / 35

5 Introduction and Concepts Major concepts as an interactive search tool Digital Library Life Cycle (Paepcke) Beckers et al. (UDE) ezdl August 16th / 35

6 Introduction and Concepts Major concepts as an interactive search tool Support for tactics and stratagems (Bates) Beckers et al. (UDE) ezdl August 16th / 35

7 Introduction and Concepts Major concepts as an interactive search tool Proactive support Beckers et al. (UDE) ezdl August 16th / 35

8 Introduction and Concepts Major concepts as development platform Easy integration of new data sources (DLs) Extendable user interface Plugin system (OSGi) Beckers et al. (UDE) ezdl August 16th / 35

9 Introduction and Concepts Major concepts as evaluation tool Support for eye-tracking with dynamic Areas of Interest (AOIs) Automated logging of user and system actions Beckers et al. (UDE) ezdl August 16th / 35

10 Architecture 1 Introduction and Concepts 2 Architecture 3 Evaluations 4 Use Cases 5 Conclusion Beckers et al. (UDE) ezdl August 16th / 35

11 Implementation Architecture Licensed under GPLv3 (other licenses also possible) Fully implemented with Java 1.6 (1.7 is planned) no legacy code Unit tests that help maintain stability JavaDoc documentation available Beckers et al. (UDE) ezdl August 16th / 35

12 Architecture Architecture Backend Service-oriented architecture (Agents) Message-driven communication (via JMS-based infrastructure) Frontend Desktop (Swing) Web (Google Web Toolkit) Mobile devices (Android, ios), under development Beckers et al. (UDE) ezdl August 16th / 35

13 Architecture Architecture Client 1 MTA 1 ACM Wrapper Client 2 MTA 2 IEEE Wrapper Client 3 Directory Query History Agent Repository Agent Search Agent DBLP Wrapper Client 4 MTA n Wiley Wrapper Client m Communication bus (e.g. CORBA or JMS) Backend Beckers et al. (UDE) ezdl August 16th / 35

14 Architecture Wrapper for data sources (e.g. DLs) Wrapper toolkit for HTML-based sources currently used for ACM DL, SpringerLink, PubMed... Connectors for common IR systems Apache Lucene/Solr Terrier, Lemur, MG4J (in development) Beckers et al. (UDE) ezdl August 16th / 35

15 Architecture The desktop frontend architecture Beckers et al. (UDE) ezdl August 16th / 35

16 Extending ezdl Architecture New data sources Patent databases: Google Patents, ESPACENET, CLEF-IP Medical databases... New features tool (desktop frontend) backend agent (if required) Beckers et al. (UDE) ezdl August 16th / 35

17 Extending ezdl Architecture New visualisations e.g. Hypergrid visualisation of search results Beckers et al. (UDE) ezdl August 16th / 35

18 Evaluations 1 Introduction and Concepts 2 Architecture 3 Evaluations 4 Use Cases 5 Conclusion Beckers et al. (UDE) ezdl August 16th / 35

19 Evaluations Built-in support for user evaluations Logging user actions Example User searches for information retrieval in 2 connected DLs (DBLP, ACM) event: name: "search" clienttimestamp: 1/4/ :26:32,1234 timestamp: 1/4/ :26:32,3456 sequencenumber: 10 parameters: query: "information retrieval" sources: dblp, acm Beckers et al. (UDE) ezdl August 16th / 35

20 Evaluations Built-in support for user evaluations Tracking AOIs Beckers et al. (UDE) ezdl August 16th / 35

21 Evaluations Built-in support for user evaluations Beckers et al. (UDE) ezdl August 16th / 35

22 Evaluations Built-in support for user evaluations Fixed layout on screen Loading predefined perspectives Splash screen for choosing evaluation settings Beckers et al. (UDE) ezdl August 16th / 35

23 Evaluations Evaluation infrastructure frontend Questionnaire tool Evaluation web app. questionnaire answers Eye tracking tool eye tracking data backend IR engine logging data Experiment database Beckers et al. (UDE) ezdl August 16th / 35

24 Evaluations Evaluation infrastructure (cont d) Evaluation web app. Start Training Task Log Viewer Analysis Tool (Example) Pre-Exp Questionnaire Pre-Task Questionnaire Collected Data Task Post-Task Questionnaire Starts System System Log Data Post-Exp Questionnaire Session Data Questionnaire & Session Data Database End Beckers et al. (UDE) ezdl August 16th / 35

25 Use Cases 1 Introduction and Concepts 2 Architecture 3 Evaluations 4 Use Cases 5 Conclusion Beckers et al. (UDE) ezdl August 16th / 35

26 Khresmoi project Use Cases Multilingual + multimodal search in the medical domain Use cases General public Clinicians and general practitioners Radiologists Beckers et al. (UDE) ezdl August 16th / 35

27 Use Cases Web-based frontend for general public Beckers et al. (UDE) ezdl August 16th / 35

28 Use Cases Desktop frontend for general practitioners Beckers et al. (UDE) ezdl August 16th / 35

29 CAIR project Use Cases User-oriented clustering of search results Clustering was added to ezdl (clustering agent and adapted search tool) Collections: Amazon Book database Wikipedia Web search Beckers et al. (UDE) ezdl August 16th / 35

30 Use Cases CAIR Web Search (Integrated Browser) Beckers et al. (UDE) ezdl August 16th / 35

31 Use Cases iperfedpat: Federated Search for Patents (Mike Salampasis, U. Thessaloniki, Greece) Beckers et al. (UDE) ezdl August 16th / 35

32 Use Cases iperfedpat New Search Tools Result Clustering, Entity Mining Beckers et al. (UDE) ezdl August 16th / 35

33 Conclusion 1 Introduction and Concepts 2 Architecture 3 Evaluations 4 Use Cases 5 Conclusion Beckers et al. (UDE) ezdl August 16th / 35

34 Conclusion Conclusion ezdl is a interactive search tool, development platform and system for user-oriented evaluations Efficient and effective IR systems need an appropriate user interface Framework to avoid reinventing the wheel when developing search UIs Participation by other persons/groups is welcome Beckers et al. (UDE) ezdl August 16th / 35

35 Thank you for your attention. Beckers et al. (UDE) ezdl August 16th / 35