UK Institutional Repository Search Project

Transcription

1 UK Institutional Repository Search Project Vic Lyte 28th October 2009

2 Background Growth in Institutional Repositories supported by JISC RPP; Highly variable quality of deposition re: content and metadata; Lightweight search capability provided by current IR software; Ambiguous aims re: search and discovery outputs from IR movement. Need to showcase what was happening in UK IR s;

3 Project aims Project team supported by JISC, led by University of Manchester, Mimas in partnership with: University of Bath, UKOLN; University of Bristol; ILRT University of Nottingham; SHERPA Herriot Watts; Intute Develop a means to cross-search all content in UK-HE bibliographic IR s; Base subsequent development on a thorough and comprehensive requirements capture;

4 Key Project activities Identify key i) stakeholder communities and ii) search and discovery contexts; Develop an approach to a (SOA) architecture to reflect emerging service needs; UK IR Aggregation Service (UKOLN); Search / discovery service (Mimas NaCTeM) Conduct a systematic review and dialogue with all current and horizon search and discovery technologies: Open Source (Lucene / IBM UIMA); Best of Breed Commercial (Google, Yahoo, Microsoft, Autonomy)

5 Emergent critical requirements /8 Basic Google search is good but not sufficient: Full-text search; New Discovery constructs: Subject browsing; Serendipitous browsing; Qualitative support conceptual and contextual attributes; Domain-specific variances. Metadata overheads: Need to build on and preserve metadata investments; Realisation that overheads required are unsustainable and need help.

6 Continuous user engagement: Repository Support Programme (JISC SHERPA); Academic User Focus Groups (Quantitative / Qualitative); Main development path /9 Stakeholder input (JISC / Librarians / Information Specialists / T&L); Dialogue and application development with technology providers / stakeholders: Commercial: UK focus (Autonomy / IBM); Open Source: Lucene (NaCTeM). Service orientated Architecture Design UK Aggregation Service (UKOLN) Search and Discovery Service (Mimas)

7 Search & Discovery Services UK Institutional Repository Search Cross-searching 103 UK Academic HE Repositories; 560,000 deposited papers;

8 Institutional Repository Search NaCTeM demonstrator Document Clustering & Classification Concept navigation

9 UK ejournals Archive JISC Collections archives of over 382 leading academic journals, containing in excess of 3,746,000 scholarly articles across the sciences, social sciences and humanities. Brill Journal Archive Online Institution of Civil Engineers (ICE) Virtual Library Archive IOP Journal Archive Oxford Journals Archive Periodicals Archive Online JISC Collections Selection RSC Journals Archive Taylor & Francis Geography, Planning, Urban and Environment Online Archive

10 Early English Books On-line / 18 th Century Collections On-line JISC Collections 3 separate collections; EEBO ECCO BL Digitalised books (Microsoft) 65,000 books Develop a POC to prove cross-searchable interface; Develop implementation plan for scalable deployment

11 Information Environment Service Registry Collections level registry Yellow Pages for Academic internet Previous search interface unusable; New interface allows serendipitous browsing of collections.

12 Autonomy IDOL Beyond basic search Conclusions from technology evaluation Scalable, enterprise class discovery layer; Surfacing relationships, conceptual and contextual relevance Extending indexing digital artefact scope to include: Learning Objects Blackboard / OpenJorum; Supplementary Data; Digital Media.

13 NaCTeM - IRS Beyond basic search Conclusions from technology evaluation Agile and focussed deployments; Semantic richness related to professional domains Main message: Natural complement between enterprise (commercial), scalable commodity Knowledge infrastructure and focussed high value (open source) applications built on top.

14 Thank you Vic Lyte Mimas

15 Preparation of metadata for IDOL indexing #1 Heterogeneous metadata from IRAS Originally formatted for import into Lucene search engine; Extract links from a variety of fields identifier; format; relation; Identify URLs for PDF documents Currently just anything with.pdf suffix; Mapping types to common set based on DRIVER Guidelines with additions Manually add to mapping file on each update Also extract pre-print /post-print status info

16 Preparation of metadata for IDOL indexing #2 Use our own script to fetch and parse PDFs where accessible; Generate one XML file per document, with full article text included where available (about 13%); A filesystemfetch connector scans folders for new, edited, or deleted files for indexing; Generate two virtual databases within IDOL: one for fulltext, one for metadata only.