ARROW Update 4 November The ARROW team

Transcription

1 ARROW Update 4 November 2005 The ARROW team

2 Welcome to ARROW What we will cover Background and context Demonstration Issues in populating the repository Putting the pieces together What is yet to come Practicalities what you need to implement it Pricing and contacts 2

3 Why have a repository? Provides a platform for promoting research output in the ARROW context Safeguards digital information Gathers an institution s research output into one place Provides consistent ways of finding similar objects Allows information to be preserved over the long term Allows information from many repositories to be gathered and searched in one step Enables resources to be shared, while respecting access constraints (when software allows access controls) Enables effective communication and collaboration between researchers 3

4 What you need : Policies to populate Mandated thesis deposit Research framework Incentives 4

5 ARROW - Summary of design criteria A generalised institutional repository solution for research information management Initial focus on managing and exposing traditional print equivalent research outputs Expanded to managing other digital research outputs Design decisions accommodate management of other digital objects such as learning objects and research inputs such as large data sets DEST Research reporting and audit, and Research Quality Framework likely to drive deposit of content by academics and research managers in ARROW universities 5

6 ARROW Branded Services Profile Internet National Library of Australia National Library of Australia Aust Digital Theses Program Swinburne UNSW Monash ARROW Repository Digital Object Storage using Fedora & VITAL ARROW Resource Discovery Service Using TeraText to index metadata harvested by OAI PMH Australian Theses Discovery Service Using metadata harvested by OAI PMH Internet Search Engines indexing content specifically exposed by by ARROW Repositories Research Management Systems Sharing descriptive metadata and linking from an RMS to the research publications ARROW Open Access Journal Publishing System Using OJS from Public Knowledge Project ARROW Web Site Project Information Members only area for Meeting Minutes etc 6

7 ARROW Technology Software Selected Flexible Extensible Digital Object Repository Architecture- Fedora Cornell and University of Virginia ARROW a founding member of the Fedora Development Consortium VITAL from VTLS Inc ARROW / VTLS partnership to take the Fedora engine and construct a working repository to meet ARROW s functional requirements using VITAL and open source web services Sustainability through vendor support Open Journal Systems (OJS) from Public Knowledge Project (University of British Columbia) for open access journal publishing 7

8 Repositories : Open Source Software and Sustainability The business case for open source software is not clear cut Red Hat model - manageable open source software for a fee Need for reasonable level of technical expertise Complete self reliance Reliance on a consortium of users of a particular product Total cost of ownership is difficult to calculate Open source software is suited as an environment for preservation no software features are buried in proprietary encodings which compromise the ability to extract content Proprietary software has advantage of support 8

9 Open Source criteria Sustainable as a long-term archive Allows customisation Ensures migration path Supports OAI-PMH Supports a range of metadata standards 9

10 Why Fedora? ARROW needed something as a platform to develop its own application(s) ARROW wanted a flexible object-oriented data model ARROW wanted to be able to have persistent identifiers down to the level of individual datastreams, accommodating its compound content model ARROW wanted to be able to version both content and disseminators (which can be thought of as software behaviours for content) ARROW required clean and open exposure of APIs with welldocumented SOAP/REST web services. Fedora satisfied these requirements 10

11 What we have achieved so far VITAL Manager ingests content and metadata edited externally with XMLSpy, edits existing content/metadata Web submission for theses (expanding to new resource types) Batch Ingest matching metadata and digital objects Institutional search National search on National Discovery Service Ability to specify indexing 11

12 INGEST Vital Client Webform Batch Authorised Self submit e.g. research mgmt tools Bibliographic Sources Metadata only Metadata + docs Metadata only Metadata + docs Metadata only Metadata + docs Validating data, adding evidentiary material Synchronising between ARROW and external resources Export metadata only DEST reporting Inclusion in 3rd party tool eg: Excel, Endnote Academic use for CV Faculty production of webpages7 Export metadata and docs Migration to 3rd party METS compliant tool, eg new repository, research mgmt tool EXPORT WEB ACCESS HARVEST Web access to metadata DEST search Researcher search for topics and research Web access to documents DEST Audit Researcher citing for offprints, or for comment, or for refereeing National Discovery search Research 12

13 ARROW a demonstration Searching The Access Portal offers simple and advanced searching, while the Access Explorer gives indexing options. The National Discovery Service provides consolidated searching across many repositories 13

14 ARROW many different kinds of objects JPEG Images Or Composite weather image of a tropical cyclone Creator NASA Satellite image of Victoria and Northern Tasmania Creator NASA MrSid images with navigation ag Victoria Dock, circa 1910 and 1942 ag Victoria Dock, 1972 and

15 ARROW demonstration Text and supporting images Title: History Australia, Volume2, No.1, Ferals and their muddies: Making a home in the bush Text only Thesis Title: To define the ways in which VITAL will support the creation of a range of branded interfaces, showing either all of the repository or particular subsets 15

16 ARROW demonstration XML plus images Melbourne 2030: Chapter 5 - Residential infill and its threat to Melbourne's liveability MPEG movie Medical computer animation #20 Quicktime movie Medical computer animation #10 mp3 audio Ash Grunwald, Bakelite Radio and Blues Progression 5Mb AVI movie Fantastic Four, movie trailer **WARNING large file 16Mb** DivX codec must be installed first to view. 16

17 ARROW a demonstration Ingest and management The Web ingest tool allows users to submit and describe documents, while the Web Review tool allows a staged review process The Vital Manager : a 3 rd generation repository application allowing ingest and edit The batch ingest tool allows ingest of objects as well as their matching metadata 17

18 Batch Ingest power ingest! A set of command line scripts that uses a series of XML files to define the import/ingest of large batches of like document types and their metadata. Complex functionality to master but rewarded great flexibility in ingest options. Each Batch job requires 3 files. Configuration file. Model Definition file. An XSL Style Sheet to convert the metadata into DC. 18

19 Configuration File Defines the parameters required to configure a batch import Model definition file Location of content files Matching parameter to link up datastreams Fedora connection information Test only status Handles assignment 19

20 Model Definition file Declares object model for the Fedora object created by the batch import Number of datastreams Mime type of datastreams Status of datastream Datastream Label Defines Style sheet 20

21 XSL Style sheet Maps the native metadata tags into Dublin Core DC is used internally in Fedora and is the Metadata payload for the OAI harvest files Currently using specific XSL per batch job Future use OCLC interoperable core tools 21

22 Batch Import Process Shell Script Configuration file ARROW Content directory Location of the ingest target content Model Definition file Defines structure of fedora object to be built for each item ingested. Style Sheet file XSL Style sheet to convert native metadata into DC 22

23 Software tools - XML editor Scripts must be edited for each new batch ingest VITAL delivered with sample scripts for common ingest types theses Images xml Samples can be customised XML editing Con: Complex Pro: Very flexible 23

24 FTP transfer program 24

25 Remote terminal connection service 25

26 Test mode Runs script and does every except assign handles and load into the repository Shows that Batch load has processed all files Object model matches model definition Metadata and content files are matched Log files generated for each batch load 26

27 Output Can process from one to one million records* Have successfully tested with many 100 s of files Can be executed by a cron job overnight run Intend to use Batch Ingest to process approx 1,500 files of Monash content Complex functionality to master but rewarded with great flexibility in ingest options *Theoretical fedora upper limit 27

28 Partner showcase All partners are gathering research content, but they are also considering other collections for ARROW Monash PhD theses. Mandatory collection of all new theses July 2005 Retrospective Digitisation of all post 2000 PhD theses. Indigenous Linguistical Research Resources Portal Related Monash rare books content digitised and stored in ARROW Access to material through portal using XACML access control to manage researcher and community access. Centre for Gippsland Studies Picture Collection historical images in JPEG2000 format. 28

29 Partner showcase School of Biological, Earth & Environmental Sciences (BEES) is trialing the deposit of honours theses by final year students in School of Mining Engineering Trial: academic and research staff (including postgraduate students) are trialing the deposit of research publications until the end of Centre for Health Informatics 18,000 medical x-ray images 29

30 Partner showcase Swinburne University Journal of Applied Psychology (ejap) Australian Journal of Educational Technology in Society Investigating automatic ingest from OJS into ARROW in cooperation with NLA Entire research output from Institute for Social Research since 2001 National Library of Australia Harold White paper - Independent scholars working papers NLA staff papers Archive of research related s 30

31 Lunch break 31

32 ARROW: an Information Management Tool to meet Government reporting requirements Around 30-40% of Australian university research funding comes from government At present an annual statistical return is required and audit evidence of research outputs is compiled as collections on paper ARROW can improve the efficiency of this process as an information management tool In the Monash context this will capture 4000 publications annually 32

33 Research Quality Framework () Integration with Research Management Systems (Currently- RM4) Store Research Objects (with associated Metadata) Using -Open Supported Standards Currently- METS Future- XML based formats (MPEG21 DIDL etc) Provide Persistent Links (HANDLES) Provide Secure Access (XACML) Expose Research Digital Objects (Google, National Discovery Service etc) 33

34 Key Activities Gathering Evidence Internal auditing and analysis Who? What? Information accuracy? Timing? Ethics? Impact? Preparing the report Submission Assessment Commenting 34

35 Working with Research Management Systems (Gathering information) New Research Objects New Research Information Old Research Information Old Research Objects 35

36 Working with Research Management Systems (Recording information) New Research Objects New Research Information Old Research Information Research Management and Analysis Software Old Research Objects 36

37 Working with Research Management Systems (Additional information) New Research Objects Research staff information Research students information Research grants information, etc New Research Information Research Management and Analysis Software Old Research Information Old Research Objects 37

38 Working with Research Management Systems (Depositing into ARROW) New Research Objects New Research Information Research Management and Analysis Software Old Research Information Old Research Objects Research Objects + Subset of Research Information ARROW 38

39 Working with Research Management Systems (Handles returned) New Research Objects Example: New Research Information Research Management and Analysis Software Old Research Information Old Research Objects Handles ARROW 39

40 Working with Research Management Systems (Evidence Portfolios) New Research Objects New Research Information Old Research Information Research Management and Analysis Software Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Old Research Objects ARROW Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidence Submission 40

41 Working with Research Management Systems (Handles link back to stored Research Objects) New Research Objects New Research Information Old Research Information Old Research Objects Research Management and Analysis Software ARROW Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios 41

42 ARROW s role in supporting The If no Research Management System, ARROW will enable you to: Store Research Objects (with associated Metadata) Using- Open Supported Standards Currently- METS Future- XML based formats (MPEG21 DIDL etc) Provide Persistent Links (HANDLES) Provide Secure Access (XACML) Expose Research Digital Objects (Google, National Discovery Service etc) 42

43 (Recording Research Information) New Research Objects New Research Information ARROW Old Research Information Old Research Objects 43

44 (Handles provided and information analysis) New Research Objects New Research Information ARROW Old Research Information Old Research Objects Handles Gather additional information: Research outputs Research staff information Research students information Research grants information, etc Information compilation, auditing, analysis and report preparation 44

45 (Reporting Evidence Portfolios) Collecting New Research Objects Collecting New Research Information Collecting Old Research Information ARROW Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Collecting Old Research Objects Reporting (submission) Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios 45

46 (Linked back to your Research Objects) Collecting New Research Objects Collecting New Research Information Collecting Old Research Information Collecting Old Research Objects ARROW Handles Reporting Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios Evidenc Evidenc e Portfoli e Portfoli os os Evidence Portfolios 46

47 Assessment by DEST (DEST collection ) Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios DEST Collection Evidence Portfolios Evidence Portfolios Evidence Portfolios 47

48 Assessment (DEST informs Assessment Panels) Evidence Portfolios Evidence Portfolios Evidence Portfolios ASSESSMENT Panel Evidence Portfolios Evidence Portfolios Evidence Portfolios DEST Collection ASSESSMENT Panel Evidence Portfolios Evidence Portfolios Evidence Portfolios ASSESSMENT Panel 48

49 Assessment (Panels View Evidence) Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios DEST Collection review review Handles enable viewing of Research Digital Objects in Digital Repositories ASSESSMENT Panel ASSESSMENT Panel Evidence Portfolios Evidence Portfolios Evidence Portfolios review ASSESSMENT Panel 49

50 Assessment (Panels Comment on Evidence) Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios Evidence Portfolios DEST Collection Annotations review Annotations review ASSESSMENT Panel ASSESSMENT Panel Evidence Portfolios Evidence Portfolios Evidence Portfolios Annotations review ASSESSMENT Panel 50

51 Assessment (Viewing annotations) Evidence Portfolios Evidence Portfolios Evidence Portfolios Annotations ASSESSMENT Panel USERS ARROW Evidence Portfolios Evidence Portfolios Evidence Portfolios DEST Collection Annotations ASSESSMENT Panel Evidence Portfolios Evidence Portfolios Evidence Portfolios Annotations ASSESSMENT Panel 51

52 In Summary ARROW supports the Storing Research Exposing Research Allowing commenting and annotations 52

53 To come Soon Interface with Research Master and other research management tools Web ingest for other content types alpha software now being tested at: Enhanced user interface with browse capabilities Early 2006 XACML Access control at Object and datastream levels Support for OAI Sets for metadata harvesting Mid 2006 Generalised content model management Metadata interoperability 53

54 What you need: Infrastructure Recommended Server hardware Operating Platform Processor Speed Memory (Minimum Requirement) Memory (Recommended Requirement) Hard Disk Space For typical 1000 object collection Max # of Users Windows 2 GHz 1 GB RAM 2 GB RAM 20 GB 128 (min config) 256 (max config) Red Hat Linux Enterprise AS 2 GHz 1 GB RAM 2 GB RAM 20 GB 128 (min config) 256 (max config) Solaris 650 MHz 512 MB RAM 1 MB RAM 20 GB 64 (min config) 128 (max config) Recommended Server Software Java Virtual Machine Runtime: JRE Databases: Oracle, MySQL, McKoi Web Server: Apache

55 What you need: Staff, training, knowledge Technical Administrative Marketing and advocacy Training Support- ARROW Users Group and Forum 55

56 Contracts and licensing Monash is the lead institution, and has a Head License Agreement with VTLS Monash arranges a sublicense to (new) Additional Partners Additional Partners enter into a separate maintenance agreement directly with VTLS for support of the Software Monash coordinates software development VTLS installs, trains and supports System and training documentation are made available by VTLS An ARROW users group will be established 56

57 Pricing Vital XMLSpy Others 57

58 Appendices 58

59 How the ARROW components fit together Fedora: the open source storage layer VTLS: commercial and open source application layer Handles: a universal persistent identifier Content models: a consistent way of describing resource types Templates: a method for creating valid MARCXML metadata Standards: MARCXML, Dublin Core, METS Collections: explicit and results sets Exchange: Relationships with RM4, EndNote etc. 59

60 Example of a MARCXML template using XMLSpy 60

61 Compound vs. atomistic object model Atomistic a data object with one or more content datastreams that are all considered primary to the object. Content Content Rels-Ext Content Content Content Content Rels-Int Content Compound a data object consisting of multiple content datastreams that are not all primary to the object. Content 61

62 Arrow has chosen the compound object model Each object in the repository comprises two or more datastreams. One object may contain many different kinds of files. Fedora PID DS1:ExternalUniqueID DC: Dublin Core OMS1:Object metadata CS1:Pub Body 1 CMS1:Body 1 metadata CS2: Pub Body 2 CMS2: Body 2 meta CS3: Images CMS3:Images metadata CS4:WebPages CMS4: Web metadata CS5: Multimedia CMS5: multimedia metad CS6:Bibliography CMS6:Bibliog metadata CS17: Evidence CMS7: Evidence metadata DS18: Native Metadata RELS-INT: RELS-INT SM1: System Meta Compound object 62

63 OCLC Metadata Interoperability Core From: Godby, Smith and Childress Two paths to interoperable metadata p. 3 at 63