Final dissemination and exploitation report

Transcription

1 European Commission Seventh Framework Programme Call: FP7-ICT , Activity: ICT Contract No: Final dissemination and exploitation report D10.5 Version 1.0 Editor: Work Package: Status: Date: Dissemination Level: France Lasfargues WP10 Final M36 R

2 Project Overview Project Name: Call Identifier: Activity Code: Contract No: Partners: LiWA Living Web Archives FP7-ICT ICT Coordinator: Universität Hannover, Learning Lab Lower Saxony (L3S), Germany European Archive Foundation (EA), Netherlands Max-Planck-Institut für Informatik (MPG), Germany Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI), Hungary Stichting Nederlands Instituut voor Beeld en Geluid (BeG), Netherlands Hanzo Archives Limited (HANZO), United Kingdom National Library of the Czech Republic (NLP), CZ Moravian Library (MZK), CZ Document Control Title: Author/Editor: Dissemination and exploitation plan France Lasfargues, Julien Masanes, Nathalie Pischon (IM), Chloé Martin, Shuba Rao (Hanzo) Document History Version Date Author/Editor Description/Comments V0.1 19/1 France Lasfargues First draft based on 10.2 V0.2 25/1 France Lasfargues Integration of Survey results and Hanzo s contribution V0.3 28/1 Jaap Blom Revision V0.4 29/1 Shuba Rao Revision V0.5 30/01 Julien Masanes Revision V1.0 01/02 France Lasfargues Final version Page 2 of 45

3 Legal Notices The information in this document is subject to change without notice. The LiWA partners make no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The LiWa Consortium shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Page 3 of 45

4 Table of Contents 1 INTRODUCTION 6 2 DISSEMINATION Dissemination objectives and strategy Main dissemination action lines First results of dissemination Dissemination results Scientific outreach Online presence Video downloads Professional outreach Coordination with the Web archiving community Platform to share software The community Why a new survey? Characterization of the community Legal Aspect How do they manage their Web archiving program? Web archiving policy Web archiving priority Conclusion 21 3 EXPLOITATION Contribution to open source tools and standards Open source Standardization and API Commercial exploitation Web archiving as a Service (WaaS) Hanzo Archive 25 4 ANNEX Dissemination actions by year Year Year Year Press release: sample of pages about LIWA project List of the scientific publications 32 Page 4 of 45

5 4.3.1 Year Year Year Survey results in details List of the members of the core users group 44 Page 5 of 45

6 1 Introduction This deliverable outline the results of dissemination and exploitation of LiWA, the Living Web archive project, funded by the 7th Framework program. The Dissemination Plan D10.2 (published in 2009) introduced the methods and tools used for the dissemination and exploitation of LIWA activities. This final deliverable uses the work from D10.2 as a basis and updates and extends it with the most recent results. This report provides tables and figures, related to dissemination and exploitation, containing the detailed results that were accumulated during the project. The first part presents the main figures of the dissemination activities in terms of publication, website statistics, press releases etc. The second part presents the results of the different dissemination procedures used to communicate on the project. The third part presents several exploitation strategies and demonstrates how LIWA has contributed to building this strategy. This document has been continuously updated during the entire project. Page 6 of 45

7 2 Dissemination 2.1 Dissemination objectives and strategy Main dissemination action lines The dissemination activities were carried out with the objective of ensuring the exploitation of the LIWA results. The following types of dissemination activities have been carried out: Communication: design guidelines for all communication documents, logo, website Activities and actions: that ensured a wide visibility for the project, discussions with the Web Archiving community, participation in workshops, conferences and meetings with specialists in the field of Internet Archiving. Public relations: presentations in magazines, e-journals, forums, mailing lists, press releases These communication activities provided the partner with the opportunity to understand the needs and expectations of the community and receive feedback from them. WP10 has been dedicated mainly to coordinate and monitor the activities described. Results of this activity are presented in this deliverable First results of dissemination The dissemination plan follows the development of the LIWA project. During the course of the project the archiving community were kept informed of the developments within LIWA thereby enabling feedback that informed the development of appropriate archival tools. The first year was mainly focused on communication activities when the final version of the web site was developed. Within the website, LiWA s research topics were highlighted by providing a specific section for each topic accompanied by a short video. Particular attention was given to the needs of the core user group and reviewers, by preparing specific sections and functionalities designed for each group. Dissemination tools and materials such as presentation templates, project reports and a letterhead with the LIWA logo were made available on the web site. An ad-hoc mailing list was created and each of the research work packages contributed to the writing and the review of the first LiWA newsletter. This newsletter was made available on the web site in January and distributed through the main and the personalized mailing lists created for core user group members. A wide announcement of the project was made during the main event in the domain of web archiving, the International Web Archiving Workshop (IWAW). At IWAW2008, in Aarhus, Denmark (18th-19th Sept. 2008) the partners presented LIWA s main activities and organized a LiWA research session. Particular attention was given to the user community by organizing a half-day session with the IIPC members who had a meeting in conjunction with IWAW The main achievements of the dissemination activities for the second year of the project were related to the promotion of the LiWA project and its results via conferences, workshops, publications and the publication of the 2nd Newsletter. All partners contributed to the Page 7 of 45

8 dissemination activities by presenting publications at different conferences and workshops and through participation in the editorial board of the 2nd Newsletter. With reference to the web archiving platform, the core user group was contacted and version V1 of the platform was prepared. During the third year, the LIWA project and its results were promoted at several conferences. The number of publications and the number of participations in international workshops increased significantly as the results of LiWA s work, matured. The release of the open source LIWA module on google code increased the visibility of the LiWA software deliverables. 2.2 Dissemination results Table 1 below presents the key outcome of the dissemination activities and compares the results of what was forecast and of what has been proven effective. The achievement rate against all objectives is over 134%, meaning indicates that LIWA has either reached or exceeded its objectives in most of the domains. Dissemination Success Indicator Forecasts Effective Objectives % Scientific outreach Presentations at relevant events & conferences % Scientific publications % Online Presence Website: # of visits per months % Video downloads (total) % Professional outreach Professional Magazine & Newsletters % Participants in the Core User Group % Newsletters published % LiWA Demonstration Workshops/Session % Software Downloads % Table 1 Dissemination results Scientific outreach Documentation and scientific publications have been submitted to high-level conferences, workshops and publishers in the field. One of the challenges for LIWA was that Web Archiving is a new field of research and had no established regular scientific publisher. In order to structure this, each year, a dedicated session for the LIWA project was organized at the International Web Archiving Workshop, a workshop with a traditionally strong practical orientation (as opposed to a scientific one). The table below lists LIWA s contributions in the conferences. In the Annex 4.3 a list make the inventory of the scientific publications. Page 8 of 45

9 Name Place Frequency IWAW FIAT, IFFTA ECDL International Web Archiving Workshop The International Federation of Television Archives European Digital Library Conference Yearly Yearly Yearly LIWA Presence 2008, 2009, , 2009, , 2009, 2010 JCDL Joint Conference on Digital Library Yearly 2010 IPRES WWW SIGIR International Conference on Preservation of Digital Objects World Wide Web Conference ACM SIGIR Conference Yearly Yearly Yearly 2008, 2009, , , 2010 SIGMOD ACM SIGMOD Conference Yearly 2009 IIPC IIPC Conference Yearly 2008, 2009, 2010 WICOW Workshop On information Credibility on the Web 2009 CHORUS+ Yearly 2009 VLDB Conference of Very Large Data Bases Yearly 2009 UKWAK UK Web Archiving Consortium Workshop Yearly 2009 SAME Semantic Ambient Media Experience Yearly 2010 Table 2 Conference and workshops Online presence Within the website a dedicated area for each research topic was dedicated in order to highlight each of these topics. In each of these sections a short video was provided to explain the scientific context and to indicate which developments were being planned with respect to the delivery of tools. The analysis of the evolution of usage of the website shows an increasing interest in the LIWA outcomes as the project progresses. The curve shows the evolution of user traffic during the three years of the project. Each peak matches with a publication or a conference. During the third year, the final results are published and in this period we can see a generally higher Page 9 of 45

10 amount of user traffic: an average of 996 visits per month against 422 the first year. The publication of the open source module on Google Code during the spring of 2010 also increased the visibility to the project Series Sep 08 Nov 08 Jan 09 Mar 09 May 09 Jul 09 Sep 09 Nov 09 Jan 10 Mar 10 May 10 Jul 10 Sep 10 Nov 10 Jan 11 Figure 1: website traffic evolution Video downloads More and more communication in the research and technology community is taking place though publication of video. Videos published on the LIWA website have been downloaded 3511 times. Also videos contribute to the project visibility Professional outreach The dissemination activities were carried out with the objective of facilitating the exploitation and publication of the LiWA results. In order to reach the professional community in the web archiving field, several workshops were held, newsletters were posted and access to LiWA deliverables were provided. A community platform was also developed so as to provide a discussion forum for professionals Workshop Workshop 1: 8th International Web Archiving Workshop A session dedicated to LiWA took place during the 8th International Web Archiving Workshop. Over 70 web archivists and researchers in this domain attended the 8th edition of IWAW during which a full session was dedicated to present research objectives and early results from LiWA. During this workshop, the audience expressed great interest and posed several questions about LIWA and the related research outcome. Workshop 2: 9th International Web Archiving Workshop Page 10 of 45

11 IWAW provided a comprehensive overview on active research and practices related to the preservation of the Web. This year s workshop also addressed several new approaches and research (from virtual worlds preservation to the temporal dimensions of Web Archives) as well as practical issues addressed by Archiving institutions, specifically with respect to managing the storage of large volumes of digital material. In this context a special session was devoted to the WARC storage format, which has been accepted as a new ISO standard (ISO 28500:2009), as well as emerging tools to support these container objects. In general, scalability issues and managing large-volume crawls were topics of intensive discussion, because of the increasing body of experience available in numerous institutions that were running a series of Web archiving activities in a range of different configurations. Workshop 3: 1st LiWA Terminology Evolution Evaluation Workshop The 1st LiWA Terminology Evolution Evaluation Workshop was held on March 16, 2010 at L3S Research Center in Hannover, Germany. The workshop spanned half a day and aimed at evaluating the outcome in LiWA WP5 technology. Workshop 4: 10th International Web Archiving Workshop IWAW10 took place on 22nd and 23rd of September 2010 in Vienna at the Austrian National Library with a dedicated session for LIWA. Over 80 participants attended the different presentations on LIWA partners. The public was made of web archivist from European and international heritage institution and researchers on this domain. Linked to IPRES2010 (the seventh international Conference on Preservations of digital objects) and IIPC (International Internet Consortium), the workshop met a large audience and a big interest. The paper presentations were followed by relevant questions, which showed the interest of the audience in those subjects. IWAW10 takes place on 22nd and 23rd of September in Vienna at the Austrian National Library Workshop 5: 2nd LiWA Terminology Evolution Evaluation Workshop On December 15, 2010 the 2nd LiWA Terminology Evolution Evaluation Workshop has been held in Hanover, Germany at L3S Research Center. The workshop aimed at evaluating terminology evolution found inside long-term archives. The workshop attendees also evaluated the performance of the Terminology Evolution Browser, a tool developed within LiWA to better visualize evolution. Workshop 6: The Society of American Archivists In August 2010, Hanzo demonstrated their Web Archiving applications during the Business Archives Section Colloquium at the Annual Conference of The Society of American Archivists. The Business Archives Section (BAS) is a forum for business archivists to be involved in the work of the Society of American Archivists (SAA) and provides a platform for members with a common interest in managing and providing access to business archives, to share information Newsletters LiWA newsletters have been published to provide interested parties with updates on the progress of the project. The newsletters offered readers the latest project information in an Page 11 of 45

12 attractively illustrated overview. Each newsletter was sent through several mailing lists and was also available on the project website. Mailing lists list: Name LIWA-All LIWA Core User group DIGITAL- Target LIWA groups LIWA Core User Group Digital Libraries Research mailing list International Federation of Library Associations mailing list OAI-general -The general OAI list DIGITAL-PRESERVATION Preserving Access to digital information Air-L Association of Internet Researchers Newsletter1: January 15, This first edition of LiWA NEWS was focussed on core research activities. LiWA research partners presented their goals and summarized their achievements for the first year. Newsletter 2: July 02, The second edition of LIWA NEWS presented the first results of a year of research. Newsletter 3: February The third edition presents the results of the LIWA activities at the end of the project Coordination with the Web archiving community Beside what was already described in term of professional outreach, a special attention was given to the Web archiving community. The list of core user group members (list on annex 4.5) was finalized and received regular information on the progress of the project. With several of them, more detailed discussion took place on the progress and applicability of LIWA technology, based on their expressed interest. With IIPC the largest professional group in this domain, close contact have been established through common meetings in conjunction with IWAW 2008 and IWAW 2010, as well as during the IIPC general assemblies in 2009 and As consultation and feedback were gathered with the core user group in public meetings (IWAW, IIPC) it was unnecessary to limit access to reports and software in the duration of the Page 12 of 45

13 project. The main channels for exchange therefore became more open than anticipated before (by having a public website and open-source repository). However, this has certainly been to the benefit of the project s outreach. It was envisioned at the beginning of the project to have a close collaboration on the development of tools with IIPC. A coordinated development roadmap with IIPC was set up and defined through the writing and reviewing of the D10.3: Coordinated development roadmap IIPC. However, in the course of the project some difficulties were encountered with the important change in the Heritrix Roadmap (abandoning version 2 and changing the focus to version 3). It was therefore decided to continue to progress in an independent manner in the meantime. Yet, LIWA open source modules are all compatible with Heritrix Platform to share software LiWA released most of its developed components and tools in open-source. All LiWA software is grouped under the liwa-technologies project on Google code: The Google platform provides: A presentation of the LIWA project A wiki with a specific part for each LIWA area: Temporal-Coherence- quick start, Semantic evolution, web-archive-ui-framework, SpamAssessmentInterface, RichMediaCapture Open source code of LIWA modules: LiwaCurvatureModule, Semantic Analyzer, richmedia-capture-plugin. The two main technologies that have not been open-sourced are: o The Spam detection module (to avoid that spammers will use it). This module is freely available on demand o The Execution-based extractor (Hanzo proprietary software) 2.3 The community Thanks to the dissemination activities, we have been able to reach a large audience from various heritage institutions (mainly from Europe). Archivists, librarians and researchers are the main audience interested in Web preservation. Details of the types of audience can be found in the annexe. At the beginning of the project, the goal was to define the target audience for the LIWA project. In order to investigate this audience a survey was executed in 2008 (see D10.2) At the end of LiWA, the following points of interest were assessed: Is there a noticeable evolution of institutions involved in Web archiving? Who are users of Web archiving in Europe? Are there any new user groups? In what way do they use web archiving? How do they use LIWA technologies? Page 13 of 45

14 Thus, the Internet Memory Foundation carried out a survey on web archiving among European and International institutions. The aim of this survey was to be able to have an inventory of web archiving institutions and to have a clearer understanding of the problems encountered in the field of Internet archiving Why a new survey? The Aims of this survey were to: - Analyse the advancements made in Web archiving projects internationally - Cover a larger group of institutions from Active to Long-Term Engaging ones - Increase the potential LiWA User Community by discovering new institutions engaged in Web archiving. Before this survey, we had a view of Web archiving project through IIPC members, but the aim here was to reach a large number of institutions. - Demonstrate how LiWA developments can be useful for these institutions. The survey has been sent by to 385 persons (within 364 independent institutions) on the 2nd week of December The contained a link to the online survey (made with SurveyMonkey). The survey was spread over 37 European countries and was offered in 5 languages (English, German, French, Italian and Spanish): National and Regional Libraries (19%) University or specialized Libraries (22%) National and Regional Archives (11%) Audiovisual and/or Broadcasting Archives (20%) Institutional Archives, as Parliament Archive, European Institutions (8) and International Institutions (31) (18%) Documentation or Archive Department of Museum (10%) This makes it the largest ever survey in the domain of Web Archiving. A special focus was laid on Germany in order to analyze Web archiving projects at a regional level and because nearly each of the 16 regions has its own Archive, Library, University Library and public TV/Radio Archive. The survey is divided into several sections: Characterization of the institution State of the archiving project Legal restriction Management of Web archiving project: who is involved, how much staff, how much budget etc. Which tools are used in the institution? Research: is the institution involved in specific projects, do they know LIWA? For institutions that are engaged in Web archiving: several specific questions about their future developments Page 14 of 45

15 2.3.2 Characterization of the community The survey has been sent to 385 persons. Some effort was made to find the right department or person to answer. Only 5% of addresses were invalid (17 out of 384 addresses) 21% of the target audience have filled out the questionnaire (74 filled out surveys). This can be considered high enough to be a representative result. In Table 1, the answers are split up per institution, which gives a good idea of which institutions are interested in Web archiving. Audiovisual archives (25,7%) National libraries (23%) and institutional archives (12,2%) are the most representative in Web archiving projects. 4% 3% 3% 1% Broadcasting Archive 8% 8% 26% National Library Institutional Archive 12% 12% 23% National Archive Research Library Figure 2: Breakdown by institution type An interesting comparison is to see the survey answers broken down to determine which institutions are the most interested in Web archiving. Thus, as we can see (Figure 2) 2010 Survey respondents), the most remarkable institutions are: - National and regional libraries: 19% of institutions contacted 25% of all responses - National and regional archives: 11% contacted, 15% of all responses - Audiovisual archives: from 20% contacted to 26% of all responses We have also contacted museum archives, representing 10% of the contacted audience, but we have not received any response from this group at all. Breaking down the answers by department, the majority of answers have been provided by «Archive/Documentation/Library departments» (52%) and «Digital preservation and Web archiving departments» (20,5%). Currently, 55% of institutions answered (Table 3) that they have a web archiving project (fully operational, experimenting, starting), meaning 41 active institutions (including an American University and a company which provides Web archiving services). 23% of the institutions answered that they do not have a current Web archiving program but plan to have one in the next three years. Page 15 of 45

16 Answer Options Response Percent Response Count Yes, it is fully operational 29,7% 22 Yes, it is operational but still experimenting 18,9% 14 Yes, we are just starting a Web Archiving project 6,8% 5 No, we do not but we plan to do it in a short-term (1 or 2 years) 12,2% 9 No, we do not but we plan to do it in a mid-term (from 3 years) 10,8% 8 No, we do not because we do not have funding for this project 5,4% 4 No, we do not because it is not in our mandate 1,4% 1 No, we do not because an other institution is already in charge 6,8% 5 No, other reason 8,1% 6 Answered question 74 Table 3: Status of the archiving program It is clear that heritage institutions that do not have any current Web archiving project would like to start doing so. It is also interesting to notice that among these institutions 42% consider that Web archiving activities are or should be within the mandate of the national library. Moreover, the type of institutions involved or interested in Web archiving has changed. When we compare the results with the first survey in 2008, the most active Institutions were national libraries (58%), three years later however, this group only represents 37%. The other highlight concerns audiovisual archives: in 2008, zero respondents declared to be active; in 2010, the amount is 18% for active respondents and 41% for short-term respondents (see Table 4). Maturity of Web archiving project Active Short-Term Survey Date National & Regional libraries 54 % 37 % 18 % 18 % Other libraries 31 % 3 % 27 % 12 % National & Regional Archives 5 % 18 % Audiovisual Archives 18 % 36 % 41 % University 11 % Other institutions 15 % 26 % 18 % 12 % Table 4: Evolution of respondents according to the maturity of the project Legal Aspect Most of institutions have a law related to Web archiving (Table 5) and don t need to ask permission (60,4%). Yes, law is enacted or passed 50% 34 Yes, law is expected 16,2% 11 Lobbying are in progress 7,4% 5 No, we do not applicable law 26,5% 18 Page 16 of 45

17 answered question 68 Table 5: Legal aspects Depending on legal statutes, institutions do not provide the same access of their collections to users (Table 6). However, it is worth noticing that 41 % of the institutions provide online access to their collection. Table 6: Access restriction Access is online for anyone 41% 25 Access is online with restrictions 28% 17 Access is on site for anyone 18% 11 Access is on site with restrictions 21% 13 We do not have access: contents are in a completely dark archive 21% 13 answered question 61 skipped question How do they manage their Web archiving program? Institutions involved in Web archiving developed their own production process and methods: 64% of them are crawling or would like to crawl in-house while 36% choose or would choose an outsourcing solution. With reference to the staff and budget involved in Web archiving projects, staff numbers are dependent on the phase of the project. 38% of a fully operational program count more than 5 Full-Time Equivalent (FTE), 50% of experimenting program and 67% that just started a project count to 2 to 5 FTE and around 60% of the planned projects count 1 or less FTE. With reference to the budget allocated to the project, the curve follows the same evolution, 41% of fully operational institutions spend more than /year; 50,5% of experimental institutions have a budget of to /year and more than 70% of planned project institutions do not have any budget for a program at all Web archiving policy Most of the institutions conduct selective crawls (71,2%) and 30,8% focus only on their own websites (Table 7). We can also note that only national libraries (33% of them) operate global crawls on their national Top-Level Domain. On the other side of the spectrum, audio-visual archives (75% of them), are focused on their own websites (only 38% collect external material according to Thematic or Selective crawls). Table 7: Archive policy Domain crawls - TLD (.uk,.eu,.com ) 23% 12 Page 17 of 45

18 Thematic/selective crawls 71% 37 Only our websites and associated websites 30,8% 16 answered question 52 Institutions mainly use tools developed by IIPC like Heritrix (80%), Wayback machine (67,5%) or NutchWAX (70%). To improve their methods however, they are interested in new developments, 82,1% answered they would be interested in LIWA tools. With these preliminary results, we notice that there is a real evolution of Web archiving practices. More and more institutions are already involved in and have explicitly expressed an interest in being involved in web archiving. It should also be noted that a high percentage of institutions focus on selective crawls that require archive fidelity and coherence, therefore fitting in well with the results of LIWA Web archiving priority The third part of the survey focuses on the priority of institutions who are already involved in Web archiving. The goal of this part was to define their needs and assess how LIWA tools could meet their requirements. Results of the questionnaire confirm that some issues addressed by LiWA would be useful to many institutions. 47% of respondents (30 institutions) were aware of LIWA project before the survey, indicating that the dissemination exercise was successful.. Through this survey, some topics of improvement can be highlighted (see Figure 3). Figure 3 Improvement Topics 47,37% 45,61% 50,88% 56,14% 63,16% Content policy Archive authenticity and fidelity Preservation Smart access: institutions are interested in improving end-users service. 55% of respondents choose to focus on this in their three main improving activities. This is confirmed by answers about Access and Exploitation (see Figure 4): 82% envisions the improvement of archive access tools as a high priority. Figure 4: Access and exploitation Page 18 of 45

19 82,35% 54,90% 60,78% Access to browse archive: full-text indexing, advanced research Interpretability: data extraction, semantic analysis, text mining This field includes the capacity of full-text indexing and advanced research in order to browse in their archive. The rates increase if we focus on institutions who provide access online or on site to end-users. Several institutions indicated to be interested in Web Archive UI Framework (most answers) and the Semantic Analyser. Moreover, compared to 2008 survey, the question of completeness remains important, especially regarding media content, or dynamic Web. There is still a shared concern that a growing range of material is out of reach to current crawlers and requests are clearly to enable to expand the range and variety of content that can be captured (see Figure 5) Figure 5: Specific Formats 80,00% 70,00% 60,00% 50,00% 40,00% 30,00% 20,00% 10,00% 0,00% This focus is confirmed by institutions, who mention being interested in LiWA tools and specifically in Rich Media Capture, Flash Content, Animation, Streaming. In the LiWA Page 19 of 45

20 project, the Rich Media Capture Module and the Link Extractor addresses exactly these issues. Concerning specific content (See figure 6 and figure 7), the main issue is the size of resources and of websites. The relatively low response on Web Spam are expected as the majority of respondents are not concerned by global crawls. But for those who do large crawls, the main issue is actually webspam detection and traps. Within the framework of the LiWA project, the Spam Detection Module addresses these concerns Interestingly, several respondents have added the social web as a specific content issue. Figure 6: Specific contents for Institutions operating TLD crawls 40% 35,56% 35,56% 48,89% Hidden web: login, forms, subscription forms Web-spams detection Traps Size of resources and websites Figure 7: Specific Contents 40% 35,56% 35,56% 48,89% Hidden web: login, forms, subscription forms Web-spams detection Traps Size of resources and websites Page 20 of 45

21 2.3.7 Conclusion In comparison to the 2008 survey, the types of institutions who express interest in Web Archiving have clearly evolved. Web archiving is considered to be an important part of their mission by more and more institutions. Audiovisual institutions emerge as a new stakeholder, being more and more engaged in Web archiving. The predominance of focus crawls shows the increasing need for higher web archive quality. LIWA is now relatively well known by the stakeholders and it can be said that the technology developed by LIWA fits well with the concerns and expectations of the Web archiving community. Page 21 of 45

22 3 Exploitation 3.1 Contribution to open source tools and standards LiWA made a significant contribution to the international effort to develop open source tools in the domain of Web Archiving Open source Since the outset of the project, it was decided to develop LIWA technology to best fit the architectural framework and a set of tools developed by IIPC. This has been done in the form of contribution to the architecture as well as in contribution in code, where possible, to existing open source modules (LiWA modules). On the project s code repository 1, several modules are available: The Rich Media Capture Module - a plug-in dedicated to the capture of streaming video content: The Temporal Coherence Analyser - a plug-in dedicated to the analysis of the temporal coherence of the archived Web content: The Spam Assessment Interface - a Web service that enables the quality assessment of the archived Web content: The Semantic Analyzer - a component dedicated to the detection of terminology evolution: The Web Archive UI Framework - a client-side framework that helps creating User Interface helpers for Web archive browsing: Some of the modules have been bundeled and have been downloaded 55 times. OpenSource modules : Module to download Downloadcount LiwaCurvatureModule 10 SemanticAnalyser 13 Rich media capture plug-in Page 22 of 45

23 The Rich Media Capture plug-in has been downloaded 32 times. Videos are becoming more prevalent online. Archiving video is not only an issue for audiovisual websites but for all kinds of websites today. Institutions are looking for a solution. The Rich Media module is one of the answers Standardization and API The LiWA exploration in terms of a new field like temporal coherence and webspam detection will have to translate into guidelines on how to capture related meta-information when more operational experiences have been gathered in these fields. This can be done on time for the revision stage of the ISO:28500 WARC format in On a shorter term, the integration work done in LIWA can be used as guideline for future integration work on top of the Heritrix plat-form (see report D6.10 for more details on this topic). Finally, an API for crawl content access has been defined (see D6.7), that can be reused for other projects where crawl content needs to be programmatically accessible. 3.2 Commercial exploitation Part of the objective of LIWA was to reinforce European competiveness in the knowledge economy. LIWA has contributed to establish 2 players in this emerging field. A private company (Hanzo) is now first in this market, including in the US and a non-profit foundation developing services in for cultural institutions in Europe (Internet Memory, formerly European Archive) Web archiving as a Service (WaaS) Definition The Internet Memory Foundation has launched its new service, called archivethe.net adopting the software as a service (SaaS) approach to Web archiving combined with a crawling engineering service as part of the offer. Archivethe.net is an automated software service, which provides an automated process for Web archiving by following the different steps of the Web archiving production from selection to access. A friendly user interface guides the user in the different steps. The quality of the archive is improved significantly by the use of LIWA technology (advanced module for web video, link extractor...). Archivethe.net helps institutions to easily and quickly start collecting websites including dynamic content and rich media, enabling navigation in past versions of a website, as if it were still live. By using a shared platform, institutions retain full control on their collection policy (sites selection, depth, gathering frequency, etc.), while benefiting from a dedicated and optimized crawling infrastructure. Development beyond the State of the Art, archivethe.net offers a real quality in terms of authenticity and fidelity of the archive and in terms of preservation in following the best practices for digital preservation using standards and metadata. They also retain custody of collected material with a regular delivery of complete copies of collected material in the ISO 28500:2009 WARC format. Page 23 of 45

24 ATN is a platform associated at a professional service. Internet Memory team operates in different steps. Web-archivists follow the partner in their Web archiving practice. A team of engineers ensure the quality of each crawl in designing the best strategy to collect websites. A quality assessment team controls the quality of the archive. This service is intended for any institution whose aim it is to collect and preserve web materials for historical, cultural or heritage purposes How does it work? Archivethe.net ( is based on a user-friendly interface specially developed to feed the web-archivist s needs. Several modules guide the user in the different steps of the production: an admin module to create and manage users, a collection module to create web-archive collection, a QA module to control captures and finally a module to generate reports for statistics and metrics. The following tables summarize the key characteristics of the SaaS approach and how it applies to the archivethe.net service: Key Characteristics of SAAS How it applies in ATN Network-based access to, and management of, application functionality Applications in WA are already using networkapplication functionality Activities that are managed from central locations rather than at each user s site A user interface is available online and allows institution to record their order and follow the production process (selection crawl QA Access) One-to-many model (single instance, multitenant architecture) A unique entry point for all the orders, based on multiple servers to crawl, index and to access, managed by an automated process (pipeline), which distributes task on servers. Centralized feature updating, which obviates the need for downloadable patches and upgrades A unique process allow to improved the process in the same time for each partners Workflow and technologies used Archivethe.net is based on a workflow orchestrated by an automated process named pipeline. Users record orders through the ATN interface. Each order combines an URL (a website to collect) and specific parameters (schedule, politeness, respect or ignore robot.txt...) All the orders are recorded in the database. A scheduler sends the order to the pipeline to start the crawl. The pipeline manages crawls by sending the orders to an available crawl instance and by Page 24 of 45

25 managing the queue. At the end of the crawl, data are indexed and made available for QA through a specific access tool. Continuous developments contribute continually to the improvement of its services: Automatic redirection service to redirect users to the archived version of a removed page instead of 404 error message (implemented at large scale on UK governmental websites) Advanced modules for collecting multimedia and social media content Antispam detection in large crawls, which can save up to 20-30% of storage space and time Full text indexing Results Archivethe.net service was launched during the 10th International Web Archiving Workshop (Sept 2010, Vienna, Austria). Since then, several institutions have started to use it. Even if archivethe.net has started since few months, some interesting results can be highlighted: The deployment of the automated service has increased the quality of production in terms of quality and quantity. In one hand, the integration of the LIWA module offers a real gain of quality for the archive in terms of reliability and authenticity of the data collected. And on the other hand, the production process is more reliable and effective. The Sass model has ensured a rapid deployment of an operational service for those who begin to be involved in Web archiving. The production has reached a workflow of more than 6000 captures per year and over 50 Tb of quality controlled archive generated Hanzo Archive Context: Hanzo Archives Ltd specializes in commercial Web archiving solutions for businesses and institutions. Hanzo s Web archiving products are continually developed and adapted to the growing needs of the commercial market and offers a cutting edge solution in Web archiving for records and information management, litigation support and brand heritage management. Hanzo s archived content and user metadata are stored in standards-based web-archives, as outlined in the specifications of the International Internet Preservation Consortium (IIPC). Hanzo also plays an active role in International Standards for Web Archiving, E-discovery Services Hanzo provides Web archiving solution for compliance in legal, financial, media and other industry sectors. Hanzo has the first fully supported, commercial, Web archiving product that enables archiving of any type of web-published information, independent of the underlying Page 25 of 45

26 publishing system and template technology. It utilizes Hanzo's unique archiving crawler technologies that excel in capturing the most complex, media-rich and sophisticated websites. Hanzo Enterprise is our Web archiving tool that provides full control over the Web archiving process and extends the life cycle of the total web presence across all corporate publications, from publication through archival and deletion. This solution enables businesses to: archive any type of web material to a clearly defined and managed archive policy; capture the whole of their web presence in a time-structured archive; retain the archive according to their records management policies; preserve archive content, metadata and other records in an authentic, admissible form; organize, search, review and export to litigation support teams and opposing parties. preserve authentic and the archived content in its native and admissible form Increase interoperability with e-discovery tools by being independent of CMS, web technology or storage solution be organised, searched, reviewed and exported to litigation teams and opposing parties. Hanzo Hanzo Other Other Feature / Benefit Enterprise ECMS Backup Capture and archive in native-form Yes Varies Unlikely Forensically sound archive Yes Varies No Immediate and effective playback in browser Yes Varies No Full-text search for rapid identification Yes Possible No Immediate low-cost discovery and retrieval Yes Possible No De-duplication Yes Possible No Export and remote / 3rd party playback Yes No No Eliminate dependence on CMS and publishing Yes No No technology Tamper-proof storage Yes Possible No Retain every web page and embedded objects Yes No No Application to LIWA: Within the context of LIWA, Hanzo has further developed and improved its crawler technologies making it a technology leader in capturing the most complex, media-rich and sophisticated websites. The improved crawlers can now capture and collect the entirety of the websites, including Flash presentations, RTMP video, websites requiring credentials, and POST based content including forms. Page 26 of 45

27 This development in capture and archiving technology has made it possible for Hanzo to extend the use of web archives within the compliance and litigation market. With financial, legal and other regulatory agencies treating website content as communication, there is a growing need to capture and archive content in a time stamped and native format. As a result of the new product developments Hanzo s client base has extended from Europe to North America, with the USA having the largest market for the product. Large multinational corporations and Fortune 500 companies across the financial, pharmaceutical and the food and beverage sector now use Hanzo s Web archiving products and services for compliance, litigation and brand heritage collection. The success of Hanzo s products was demonstrated at the Annual Conference of the American Archivists Association in Washington D.C in August 2010 where Hanzo received a pubic endorsement of the product by The Coca-Cola Company. Page 27 of 45

28 4 Annex 4.1 Dissemination actions by year Year 1 Institution Presentations at relevant events & conferences Date Scientific publications Audience LUH LUH LUH MTA SZTAKI LUH EA MPG HANZO MPG Presentation at the Institute for Natural Language Processing of the University Stuttgart Presentation at the GI-DL Meeting in Karlsruhe Presentation at the IFIP WG 2.6. Meeting 8th International Web Archiving Workshop Aaarhus, Denmark 8th International Web Archiving Workshop Aarhu, Denmark 8th International Web Archiving Workshop Aarhus, Denmark 8th International Web Archiving Workshop Aarhus, Denmark 8th International Web Archiving Workshop Aarhus, Denmark Lecture Web Archiving held at RWTH Aachen University June- 08 juil-08 Sept- 08 Sept- 08 Sept- 08 Sept- 08 Sept- 08 Sept- 08 Dec-08 "Web spam: a Survey with Vision for the Archivist" "Terminology Evolution in Web Archiving: Open Issues" "LiWA Architecture" "Catch me if you can" "The Challenge of Dynamic Links" Researchers Researchers Practitioners Researchers- Practitioners Archivists Librarians Archivists Librarians Archivists Librarians Archivists Librarians Archivists Librarians Archivists Librarians Year 2 Institution Presentations at relevant events & conferences Date Scientific publications Audience MPG MPG LUH Presentation at the Free University of Bozen-Bolzano, "Bachelor: Minimizing Stochastic Coherence of Web Archives" Lecture on Web Archiving at Stuttgart Media University (HdM), Germany => presentation GI-AK-DL Workshop at the Bayerische Staatsbibliothek München Jan.-09 Mar.-09 Mar.-09 Page 28 of 45 Researchers Archivists Librarians Researchers Students ResearchersLibrarians- Archivists EA Talk "Building on Internet Memory" Mar.09 Researchers-

29 LUH MPG EA LUH at Grenoble Xerox Research Centre Europe (XRCE) => presentation Talk Terminology Evolution in Web Archiving at University Vienna => presentation 3rd Workshop on Information Credibility on the Web (WICOW 2009) Fifth Workshop on Adversarial Information Retrieval on the Web LiWA Poster at the CHORUS Conference in Brussels => presentation Apr.-09 Apr.-09 Apr.-09 May-09 Data Quality in Web Archiving practitioners Researchers-Students Researchers Practitioners Archivists Researchers, Industry practitioners ResearchersLibrarians- Archivists EA Pharos Summer School June-09 Researchers Students 12th International Workshop on Bridging the Terminology Researchers- MPG the Web and Databases (WebDB June-09 Gap in Web Archive practitioners 2009) Search LUH LUH MPG LUH MPG LUH BeG LUH LUH, EA, MZK, MPG JISC, the DPC and the UK Web Archiving Consortium Workshop The Times Archive London => presentation 35th International Conference on Very Large Data Bases (VLDB 2009) => paper 9th International Web Archiving Workshop (Corfu 2009) 9th International Web Archiving Workshop (Corfu 2009) Cultural Heritage On Line 2009 Florence => paper ACM Multimedia, Beijing, October 2009 OnTheMove 2009 Federated Conferences, Workshops and Academy Cultural Heritage on line Conference July-09 July-09 Aug.-09 Sept.-09 / oct.-09 Sept.-09 / oct.-09 Oct.-09 Oct.-09 Nov.-09 Dec.-09 From Web page storages to Living Web Archive SHARC: Framework for Quality-Conscious Web Archiving First Results on Detecting Term Evolutions Catch me if you can : Visual Analysis of Coherence Defects in Web Archiving Automatic Detection of Terminology Evolution Turning pure Web Page Storages into Living Web Archives Librarians- Archivists Researchers Librarians Archivists Researcherspractitioners Researchers Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists Researcherspractitioners Researcherspractitioners Year 3 INSTITUTION PRESENTATIONS AT RELEVANT EVENTS & CONFERENCES DATE MPG Paper presented at ECIR 2010 Mar.-10 LUH Presentation of the LiWA project to the members of the GI Working Group on Digital Apr.-10 Page 29 of 45 SCIENTIFIC PUBLICATIONS "A Language Modeling Approach for Temporal Information Needs" AUDIENCE Researcherspractitioners Researchers Librarians

30 Libraries LUH HANZO EA MPG LUH EA EA LUH 10th ACM/IEEE Joint Conference on Digital Libraries in conjunction with ICADL in Surfers Paradise Annual Conference of The Society of American Archivists, th International Web Archiving Workshop (Vienna 2010), Austrian National Library, Oratorium 10th International Web Archiving Workshop (Vienna 2010), Austrian National Library, Oratorium 10th International Web Archiving Workshop (Vienna 2010), Austrian National Library, Oratorium LiWA toolkit presented to IIPC Harvesting Working Group in conjunction with ipres2010 and IWAW2010 Poster at FIAT 2010 presenting an example of a shared platform dedicated to heritage institutions and archiving web video with its issues and developments Paper presented at SAME 2010 in conjunction with AMI 2010 Jun.-10 Sept.-10 Sept.-10 Sept.-10 Sept.-10 Oct.-10 Nov.-10 "Using Word Sense Discrimination on Historic Document Collections" Demonstration of Web Archiving at the Business Archives Section Colloquium "Archiving Web Video "The SOLAR System for Sharp Web Archiving" "Terminology Evolution Module for Web Archives the LiWA Context" Language Evolution On The Go Researchers Librarians Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists Researchers Librarians Archivists MPG Presentation at the Zuerich University, "Quality-Conscious Web Archiving with SHARCs" Nov.-10 Researchers Librarians Archivists L3S 2nd LiWA Terminology Evolution Evaluation Workshop in Hannover, Germany Dec.-10 Researchers BEG 10th International Web Archiving Workshop (Vienna 2010), Austrian National Library, Oratorium Sept.-10 Internet Archiving in an audiovisual archive Researchers Librarians Archivists Page 30 of 45