DURAFILE. D1.2 DURAFILE Technical Requirements

Transcription

1 Project Title: INNOVATIVE DIGITAL PRESERVATION USING SOCIAL SEARCH IN AGENT ENVIRONMENTS This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement number Project Acronym: DURAFILE Contract Number: Start date of the project: October 1 st, 2013 Duration: 24 months D1.2 DURAFILE Technical Requirements Dissemination Level: Public Submission date: April 1 st, 2014

2 Document revision history Version Date Comments /01/2014 Santiago Macho (ATEKNEA) /02/2014 Miquel Montaner (UDG), Albert Bres (EASY) /04/2014 Santiago Macho (ATEKNEA) 2

3 Document Information This deliverable is for the Research for the SMEs project entitled called DURAFILE This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement number Project Acronym: Project Title: DURAFILE Innovative Digital Preservation Using Social Search in Agent Environments Prepared by: Dissemination level: Project coordinator: UNIVERSITY OF TALLINN Public ATEKNEA 3

4 Table of Contents 1. Introduction Redesign of the DURAFILE Platform Initial Proposal of the DURAFILE Platform Revised DURAFILE Platform Scientific & Technical objectives DURAFILE Specification DURAFILE collected platform Requirements SMEs requirements End Users requirements Conclusions User Roles Personal User Enterprise User Expert User DURAFILE Administrator Harvesting Agent DURAFILE Case Diagram Non-registered user uses cases Registered user uses cases Personal user uses cases Enterprise User uses cases Expert User uses cases DURAFILE Administrator uses cases Harvesting Agent uses cases Sequence diagrams File standard preservation workflow DPP Request DPPs Specification Related Projects Digital Preservation Plans in DURAFILE DURAFILE Implementation Platform Architecture DURAFILE Cloud Server DURAFILE platform users Implemented requirements Digital Preservation Plan Implementation Experts Social Search End-User Interface Desktop app Web application Conclusions Bibliography

5 1. Introduction This document is a complete specification of the proposed DURAFILE platform based on the exposed set of requirements in a previous Draft. For facilitating the understanding how the platform preserves files, a set of examples will be provided. The objective of the DURAFILE project is the development of a platform to facilitate a common infrastructure for researchers, big institutions and end users that will facilitate the interchange of digital preservation plans for multimedia files. The main outcome of the project will be i) the generation of a database of standard preservation plans for multimedia files; ii) the conversion of the out-dated multimedia files into standard formats preserving the quality. The user can choose to only view the preservation advice given by DURAFILE or to have DURAFILE identify and convert out-dated files in specific folders automatically. The user would then have the old file (in out-dated format) as well as the new file (in the new format) stored at their computer. There will be two types of users of the platform: professional users, which are big institutions or SMEs that need to handle a large amount of information that needs to be preserved; and regular home users who need more than a backup for their multimedia files The document is structured as follows. Initially, a detailed specification of the DURAFILE platform was provided and was justified how it was inferred from the collected requirements. The specification included User Roles and detailed Case and Sequence diagrams. The inclusion of the Digital Preservation Plans in the DURAFILE platform was defined. The document continues with a detailed information about the architecture of the DURAFILE platform, which was obtained from the specifications. The deliverable ends with mock-ups of the end-user interface for using the DURAFILE platform. 5

6 2. Redesign of the DURAFILE Platform During the M6 general meeting, the RTDs of the project: University of Girona, University of Tallinn and ATEKNEA presented a new architecture for the DURAFILE Platform. The new design is more oriented to a commercial product and was validated by all the consortium members. We decided to include this section in the deliverable in order to justify the proposed change, comparing the new proposal with the initial one. In this section, first we introduce the initial proposal of the DURAFILE Platform. Next the revised DURAFILE Platform architecture is presented (it is discussed in more detail in this document). We include a comparative of the features of the initial proposal and the new one. 2.1 Initial Proposal of the DURAFILE Platform The initially proposed platform was based on the idea that there is not a single preservation plan for every problem. There are several options and different users might have different useful-preferences among the existing solutions making the adoption of a multi-agent architecture instead of a more conventional centralized repository necessary. The dynamics of these preservation plans (that can be updated during its life) also indicates that the adoption of a distributed architecture is the appropriate solution. USER 1 USER 2 USER 3 USER AGENTS 1 USER AGENTS 2 USER AGENTS 3 COMMUNICATION LAYER DURAFILE PLATFORM USER AGENTS 4 USER AGENTS 5 CLOUD USER 4 USER 5 PLUGIN CLOUD DEVICE Figure 1 DURAFILE initial proposal platform. Figure 1 shows the proposed DURAFILE architecture. Every user is connected to the platform through a set of agents that will work for the digital preservation of his/her multimedia files. This user platform is created one time when the user subscribes to the DURAFILE application and remains proactive in the DURAFILE servers even when the user is not logged in. Similar to the way antiviruses do, the user defines a directory which will be scanned by the platform. In the case of a digital obsolescence sign is found in the user directory, the platform will warn the user. The file obsolescence found can be solved with collaboration among users through Social Search empowered by the intelligent agents. 6

7 2.2 Revised DURAFILE Platform Figure 2 shows the proposed architecture of the platform by the University of Girona, University of Tallinn and ATEKNEA. The main idea of the platform is to automatize the file migration into a longterm file format. We assume a Dropbox model where the user selects a directory in his computer where he will put all the digital files that should be maintained in a long-time preservation format. All the files presented in this directory will be stored in the DURAFILE server and can be accessed from different computers or platforms (similar as Dropbox does). Figure 2 DURAFILE revised platform. The proposed platform has a more commercial focus, including new actors that were not in the initial proposal such as Third-parts providers. An Enterprise Version for companies with large files or with high security issues is included in this new version. FEATURE Proposal DURAFILE Platform Revised DURAFILE Platform INTELLIGENT AGENTS SOCIAL SEARCH DIGITAL PRESERVATION PLANS DIGITAL PRESERVATION PLANS EXECUTION YES. The users of the platform are represented by Intelligent Agents. YES. The social search searches for a suitable Digital Preservation Plan among the users. YES. DURAFILE users interchange Digital Preservation Plans. YES. Digital Preservation Plans were executed at the user computer. NOT YES. Only the experts are represented by Intelligent Agents. Regular/Enterprise users use a client. YES. Social Search is applied only to the Experts Community. YES. Digital Preservation Plans are stored in a central repository which is updated by the Experts and the Harvesting Agent. YES. Digital Preservation Plans are executed in the Recipe Server. 7

8 ENTERPRISE USER THIRD PART PROVIDERS SERVER DATA STORAGE (Dropbox model) possible in case the user has not the right software. NO. This user was not included in the initial proposal. NO. The Third part providers were not included in the initial proposal. NO. The files were stored in the user computer. That makes difficult the automatic execution of Digital Preservation Plans. YES. Enterprise User which includes a local copy of DURAFILE server. YES. Third part provides can connect complementing the services offered by the DURAFILE platform. YES. The files are stored encrypted in the user computer and in a central server. That facilitates the automatic execution of Digital Preservation Plans. Table 1.- Features comparison between the initial proposal and the revised DURAFILE platform Table 1 shows a comparative between the features of the initial proposal and the new DURAFILE platform. The table shows that with the new proposal all the features are maintained. New features were introduced such as the third part providers and the server data storage which makes the new design more commercial focussed. This Deliverable offers a detailed information of the proposed DURAFILE platform. In the next sections, the new platform specification is detailed including also the taken implementation decisions. 2.3 Scientific & Technical objectives Table 2 shows a comparison between the Scientific & Technical objectives of the initially proposed DURAFILE platform and the new proposal. OBJECTIVE Proposal DURAFILE Platform Revised DURAFILE Platform Development of a Software Platform Development of a Security / Trust model Development of Social Search algorithm for digital preservation Development of a GUI for digital preservation Integration, validation and system demonstration YES. A platform made of Intelligent Agents. YES. Security / Trust linked to the agents. YES. Social Search applied to all Digital Preservation Plans. YES. GUI Related to the platform of Intelligent Agents. YES YES. A platform which has the MAIN Server and the Recipe Server YES. Security linked to the data store in the DURAFILE Server, Trust linked to the agent of the Expert Community. YES. Social Search applied to the Digital Preservation Plans generated and recommended by the Expert Community. YES. GUI Related to a server type Dropbox. YES 8

9 Generation of best techniques for multimedia preservation YES. More difficult to collect all the DPP due that they are distributed among the agents. YES. They are centralize in the Recipe Server. Generation of standardization procedures YES. More difficult to collect due that the information is distributed among the agents. YES. The list of standards procedures can be stored at the Recipe Server. Table 2.- Scientific & Technical objectives comparison between the initial proposal and the revised DURAFILE platform 9

10 3. DURAFILE Specification The DURAFILE platform will be a product that will be capable of detecting and migrating obsolete files with as high as possible quality. The scope of the project is: text, video, audio and image files. Ideally the system will consist of installable software that will monitor a certain directory for detecting obsolete or future obsolete formats. The migration could be done by several solutions: Migration at the user machine (like antivirus). Migration at a central server (like Dropbox). Migration at a central server for regular users and local migration for large files or files with privacy issues. The ideal final software product will be platform independent, able to fulfil the mentioned migrations on different hardware configurations. The target groups of the software product are companies with digital preservation problems and endusers which are aware that their multimedia files could become obsolete. Companies will be the main revenue of the platform while the end-users will provide their experience in digital preservation. More experienced users can be validated as experts sharing their Digital Preservation plans. Digital Preservation plans are scripts that could be written in a specific language or we can reuse other languages such as the one used in the Planets project. The DURAFILE platform will be available in a desktop and a server application. User can access to the platform downloading a code that will be installed in their computers or accessing to the platform via web (similar as Dropbox works). 3.1 DURAFILE collected platform Requirements This sections proofs that the previous proposed architecture follows the initial set of specific requirements mentioned in the previous Draft. As mentioned in these Draft, the requirements will be divided into: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. During January 2014 the involved RTDs (University of Tallinn, University of Girona and ATEKNEA) decided to collect information from the consortium about which could be the requirements for the DURAFILE platform. Two types of requirements were collected: SMEs requirements, which are the requirements of the SMEs involved in the project. Their requirements were collected using an EXCEL template. End User requirements, which are the requirements that end users expects the DURAFILE platform implements. The requirements were collected using Google Forms. In the next subsections we study the results obtained from the EXCEL templates and the Google forms. The collected requirements were set as the basis for building the DURAFILE platform. 10

11 3.1.1 SMEs requirements This section explains how we collected the requirements from the SMEs involved in the DURAFILE project, including a comment of the obtained results Gathering SMEs requirements: EXCEL Template At the beginning of January 2014 an Excel file was published on the DURAFILE basecamp for gathering the requirements that the involved SMEs expects from the DURAFILE platform. Among several formats for collecting these requirements the involved RTDs decided to create an EXCEL which was send to all the SMEs participant in the project. The SMEs requirements were divided into 5 different categories (as shown in Figure 3) for facilitating the process of identification and for examining each category separately. This division facilitates that the multitude of identified requirements can be better explored. Functional requirements. Requirements that define those features of the product that will specifically satisfy a consumer need, or with which the Consumer will directly interact. Includes: o Data requirements. Describes data users needs to process or to export. o Interoperability requirements. Ability of the system to share information and services. o o Interface requirements. Specify the interface between the software and the user. Performance requirements. Static requirements: capacity of the system (e.g: number or users) Dynamic requirements: Actions that should be done simultaneously in a certain period of time. Figure 3 DURAFILE SMEs Requirements categories. Operational requirements. Requirements that define those "behind the scenes" functions that are needed to keep the product operational over time ensuring the stability. 11

12 Security requirements. Requirements for protecting the software from accidental or malicious access, use, modification, destruction, or disclosure. Legal requirements. Requirements and constraints forced by legal regulation, e.g. licenses, laws, patents, etc. Documents requirements. Requirements for assuring the software documentation. Figure 4 EXCEL for obtaining the SMEs Requirements Figure 4 shows the first sheet of the EXCEL that were used for obtaining the SMEs requirements. At the second sheet, the SMEs wrote their requirements as shown in Figure 5, which includes a column with the type of the requirement (functional, operational, security, legal and document) and a second column which includes a brief description of the requirement. Figure 5 An example of the Requirements collected from a SME We decided to use an EXCEL for gathering the requirements due to the familiarity of the partners with the tool and the easy of writing a list of requirements. 12

13 Collected SMEs requirements During the first month, we collected all the EXCELs from the involved SMEs in the DURAFILE project and we created a list of requirements that will be commented in the next sections. The study of the requirements is divided into the 5 groups mentioned before. For each group we created the next subdivision: Mandatory requirements. Essential requirements. Desirable requirements Functional Requirements The functional requirements obtained from the EXCELs were classified into three subcategories: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. Platform will interchange Digital Preservation plans (Core). Mandatory Requirements System needs to be updated all time. GUI will not disturb the user with messages and alerts. Automatic updates. Scheduling the run of the tool. Platform must to be able to work up to 1000 users. Essential Requirements Modern GUI which needs to be more than a default java interface. The user will be able to add new solutions to the one provided by the tool. Perhaps it requires a new formal language for the platform. Architecture for storing files in the cloud. Desirable Requirements User able to know his obsolescence level. User can add comments and tags as metadata. User upload Digital Preservation Plans including their reputation. File contains metadata with the preservation action it has suffered. When identified an obsolete or risk file, make a copy of it into a processing folder. Possibility of partial preservation of an archive. Before the execution show to the user the Digital Preservation plan, leaving the choice to the user to execute it automatically or not. Web architecture. Upload obsolete file format, download updated file format. 13

14 Operational Requirements The operational requirements obtained from the EXCELs were classified into three subcategories: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. Mandatory Requirements Sharing successful Digital Preservation plans with other agents. Assuring that the Digital Preservation process creates a readable file. List of obsolete formats with automated process of preservation. The digital preservation plan includes input format, output format, quality of migration, and standard language for scripts. Many levels of administration. Use of metadata. Essential Requirements Different interfaces and functionality for the end user, corporations and advanced users. Multiplatform design. Check the availability of decoders. Desirable Requirements No desirable requirements collected Security Requirements The security requirements obtained from the EXCELs were classified into three subcategories: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. Mandatory Requirements Communication between server and clients must be secured by SSL certificate. Client personal files should be encrypted on the server. System shall provide a robust trust mode. System shall provide security model for assuring privacy of the files. Essential Requirements No essential requirements collected. Desirable Requirements The application should run using minimum privileges on the host computer. 14

15 Legal Requirements The legal requirements obtained from the EXCELs were classified into three subcategories: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. Mandatory Requirements No mandatory requirements collected. Essential Requirements Management of access rights of files y owners. Desirable Requirements The software should be developed using open source. The user can export all his data and erase the account with the related data Documents Requirements The documents requirements obtained from the EXCELs were classified into three subcategories: mandatory, essential and desirable. The DURAFILE platform must be delivered with the entire mandatory and a good portion of essential requirements represented, and with a plan to implement the essential requirements in the subsequent phase. For the desirable requirements some could be planned as an update for future releases of the DURAFILE platform. Mandatory Requirements Language of the platform English. Essential Requirements Include the languages of the involved countries in the project: Spanish, Greek, Romanian, French and Turkish. Desirable Requirements Include Portuguese as a language of the platform. There is not any Portuguese company in the consortium but a partners suggest this language for exploitation in emerging markets End Users requirements This section explains how we collected the requirements from several end users, including comments of the obtained results. The end user were provided by the SMEs participating in the project and from the publicity of the project in Digital Preservation Forums. The goal of collecting end users 15

16 requirements is to have an idea of the functionality that a regular user expects from the DURAFILE platform Gathering End User requirements: Google Form In parallel to the EXCEL template provided for collecting the SMEs requirements, we used a google Form for collecting the user requirements. We decided to use this innovative format, because it is easy to collect answer from partners around Europe. Google Forms it is a very interesting tool due that it provides the collected results in a graphical manner, very suitable for our purpose. Figure 6 Google Form for collecting the End User requirements Figure 6 provides a screenshot of the Google Forms used for collecting the end user requirements. It is still public at the url It is still active, therefore we expect to collect more answers during the next months. 9 questions were submitted in the Google Forms divided into 2 categories. General questions about digital preservation. In this section we proposed 7 questions about the needs and past problems of the users regarding the file obsolescence. The proposed questions are: 1. Which type of digital files usually do you manage? 2. Which is the typical size of the digital files? 3. Are you aware that some file formats will become obsolete? 4. Do you think that preserving digital files is needed? 5. How often in the past did you experience problems when opening a file with an obsolete format? 6. When do you save files, did you use a standard format that will be long time maintained or do you use a proprietary format? 16

17 7. Would you accept automatic suggestions of file format change for long time preservation of your files? Questions about functionality of the DURAFILE platform. In this section we proposed 2 questions about how the DURAFILE platform should work. The proposed questions are: 1. Which type of software solution is more suitable for you? 2. How would you like that the system provides preservation for your files? In the next section, we comment the obtained results from the collected Google Forms Collected End User requirements In this section, we comment the results obtained from the published Google Form document. Type of digital files For the DURAFILE project it is important to know the type of multimedia files used by the users. Figure 7 shows the results of the questionnaire about the type of digital files. Figure 7 Digital file managed The results show that 37% of the answers uses text files that they want to preserve. Initially text files were not included in the scope of the project, but after the obtained results, University of Tallinn, University of Girona and ATEKNEA decided, with the agreement of the SMEs of the consortium, to include the text files into the scope of the DURAFILE project. Typical size of the digital files It is important to know the size of the files the potential users of the DURAFILE project will use. Depending of the size of the files, a local or a cloud solution could be provided. Figure 8 shows the results of the questionnaire about the typical size of the files. Figure 8 Results regarding the size of the file 17

18 The results show that 82% of the answers use file with a size between 1Mb and 10Mb, which is affordable for executing a script (Digital Preservation Plan) and for providing a cloud solution. Aware of file obsolescence File obsolescence is a very important problem. Formats more likely to be accessible in the future are non-proprietary, open, documented standard commonly usage by research community, standard representation (ASCII, Unicode), unencrypted and uncompressed. Figure 9 shows the results of the questionnaire about the awareness of file formats. Figure 9 Results regarding the file format obsolescence The results show that 100% of answers agree that they are aware of the problem of software obsolescence. This is an interesting results because the RTDs of the DURAFILE project though that file obsolescence is a problem that users does not matter about it (due that they think that it is a future problem but now it is not a problem), but the answers shows that people have in mind the problem of file format obsolescence. Aware of the need of digital preservation The problem of file obsolescence mentioned before implies that we need digital preservation plans for preserving the files. Technology is constantly changing and as new formats emerges it is important that accessibility of data can be assured creating digital preservation plans. Figure 10 shows the results of the questionnaire about the needed of digital preservation plans. Figure 10 Results regarding the need of digital preservation The results show that most of the answers agree that digital preservation plans are needed for assuring the access to data in older formats. Combining the previous question of the awareness of file formats and this question of the need of digital preservation plans, we can conclude that the digital preservation is an emerging market, where the DURAFILE project is one of the pioneer projects. Past problems with obsolete formats 18

19 Past experiences with obsolete files is the first step to be aware that several files have a file format which that is obsolete and therefore data cannot be accessed. The goal of the question is to check if this problem is usual or not. Figure 11 shows the results of the questionnaire about the past problems experienced for the users with obsolete formats. Figure 11 Results regarding the experienced problems with digital preservation The results show that 82% of the answers hat experienced problems with obsolete file formats. It is very likely that this number of users that experienced problems is greater than few years ago and that is less than the users that will experience this problem in the future. This implies that digital preservation will be a problem to be solved in the future. File format when saving files Most of the problems experienced by the users with obsolete files are due to the use of proprietary formats that are not maintained. A very important question is to know if users are aware that when they save a file they use a format standard or a proprietary format that could be obsolete in a few years. Figure 12 shows the results of the questionnaire about the use of standard file formats or proprietary formats. Figure 12 Results regarding the use of standard formats The results show that only 29% of the answers use proprietary formats when saving files. This result implies that it is very likely that in the future more problems related with digital preservation will appear. The DURAFILE project will help promoting the use of standard file formats and providing migrations of proprietary format into a long-time maintained format. Acceptance of automatic suggestions for digital preservation The previous questionnaire about the use of file formats, shows that most of the users do not use a standard format when saving files. It is interesting the opinion of the users regarding a tool that will suggest to change their file formats into a standard long-time maintained format. Figure 13 shows the results of the questionnaire about the acceptance of automatic suggestions for changing the file format of the user s files. 19

20 Figure 13 Results regarding automatic suggestion for digital preservation The results show that 53% of the answers probably will change the file format it that leads to a longtime format data. It is important that 47% of the answers will change the file format for long-time data access. 100% of the answers have clear idea that files should be saved using standard formats, therefore DURAFILE can be a tool for helping users in selecting the right file format. Suitable software solution for digital preservation Several questions regarding how the DURAFILE product should act were introduced. There are several approaches for building a new tool for digital preservation: antivirus, self-executable, scripts, etc.. Figure 14 shows the results of the questionnaire about the type of solution that the users want in a software devoted to digital preservation. Figure 14 Results regarding the desired software solution for digital preservation The results show that among all the possibilities, the 76% of the answer prefer a solution like an antivirus. In that case, the software will scan user-defined directories for detecting file formats that are not long-maintained and generating a copy of the original file in a standard format. Suitable manner for DURAFILE for providing digital preservation As shown before, the users prefer a tool for digital preservation that works similar an antivirus system does. When the files with not standard formats are discovered by the tool they are several manners of acting: automatically migrate the file, semiautomatic or manual migration. Figure 15 shows the results of the questionnaire about how the system will act in case a file with an obsolete format is detected. 20

21 Figure 15 Results regarding how users prefer digital preservation The results show that 88% of the answers prefers a tool that automatically or semi-automatically migrate the files into a long-maintained format. Very probably they want to have a semi-automatically tool for deciding when a file will be migrated or not. Both models (automatic or semi-automatic) are consistent with the idea of building a tool that work as an antivirus Conclusions As a conclusion of the collected requirements, we have obtained from the SMEs the functional, operational, security, legal and document requirements captured using an EXCEL. Regarding the end users, we created a questionnaire where most the answers shows that there is a potential market for a tools devoted to the Digital Preservation problem. Most of the users will accept a tool that works as an antivirus, detecting files with obsolete or not standard file formats, and that automatically or semiautomatically migrate the files into a new format for long-time maintenance. The questionnaire detects reticence when migrating files which is likely to be produced by the fear of losing quality in the file. That implies that DURAFILE must migrate the files taking into account the quality of the obtained file. 3.2 User Roles The platform is focussed into 2 main types of users: a user that wants to preserve their personal data and companies that needs digital preservation due to normative or legacy issue (for example, for recording invoices during a certain period of time). A third user role is the Expert user, which is a user that wants to participate in the creation of new digital preservation plans. An external user that extracts automatically information from digital preservation forums is the Harvesting Agent. Finally a user that administrate the DURAFILE platform is needed. As a summary, the consortium has identified 5 user roles in the DURAFILE platform with different functionalities: personal user, an enterprise user, and expert user, the harvesting agent and the DURAFILE administrator. USER ROLE ROLE DESCRIPTION Personal User Regular user that wants to preserve its digital library. 21

22 USER ROLE ROLE DESCRIPTION Enterprise User Expert User User that represents a company or a big institution that wants to preserve large contents of multimedia documents. Very strong security issues applies here. Personal User that wants to participate in the creation of Digital Preservation Plans interchanging them with other expert users. Harvesting Agent DURAFILE Administrator Intelligent agent that monitors certain forums for automatically detect obsolete formats and then automatically create new digital preservation plans. Responsible for the overall platform, administrating users, enterprises and third-part providers Table 3.- DURAFILE User Roles Table 3 shows the list of user s roles and a brief description of their functionalities in the DURAFILE platform. Each user role has a different functionalities, which will be described in section (user cases) Personal User A Personal User is a regular home user who needs more than a backup for their multimedia files. The personal user will install DURAFILE which will scan their selected folders (as an antivirus does) for detecting out-dated files that will be converted into a standard format preserving the quality. The user can choose to only view the preservation advice given by the DURAFILE platform or to let it convert out-dated files in specific folders automatically. During this process, the obsolete files will be upload to the DURAFILE server but using secure SSL communications and encryption. At the end of the process, the user will have the original file with an out-dated format as well as a new version in a longterm format Enterprise User An Enterprise User is a professional user which could be big institutions or SMEs that need to handle a large amount of multimedia information with high security issues. Similar as a personal user does, the enterprise user will install DURAFILE which will scan their selected folders selected folders (as an antivirus does) for detecting out-dated files that will be converted into a standard format preserving the quality. The main difference is that the enterprise user will have a dedicated server for security reasons (files will not be upload to the DURAFILE server). It is very usual that these enterprise server wants to migrate large multimedia collections stored using obsolete or not long-time maintained formats. The enterprise server will communicate with the main DURAFILE server only for updating the digital preservation files. At the end of the process, the user will have the original file with an out-dated format as well as a new version in a long-term format Expert User A challenge of the project is the study of efficiency with social networks for digital preservation tasks. The Expert Users will be experienced personal users that wants to help other users using a social network. In a social network there will always be people who will understand the Digital Preservation 22

23 needs of others and will provide appropriate links to solutions. This claim seems to be rather theoretical but there is a series of Web forums that illustrate the power of social platforms; e.g: a forum about motorbikes, that usually has around 5,000 6,000 users ( Imagine a user who wants to change his motorbike tires for bigger ones. As s/he is not sure about the allowed size and performance of the different tires, s/he reads the forum for good advice by other users that have bought and tested the different tires. Although the forum is an example of collaboration, it is no longer useful for archival reasons. When technology evolves so fast, obsolete solutions need expert advice to be applied to one s own problems, and especially when there can be many unexpected side effects in the old solutions that prevent a user from reusing certain solutions. This is especially true for digital preservation because even the best solutions in the past do not guarantee the desired results in the future DURAFILE Administrator The DURAFILE Administrator is a special user that administrate the DURAFILE platform. It is in charge of controlling the user s accounts in the platform, controlling and maintaining the experts network, maintenance of the digital preservation plans (removing the ones that the users classified as invalid or with a poor performance), authorizing third-part companies (DP providers that can complements the function of the DURAFILE platform) and allowing that enterprises servers which are installed in specific companies can be connected to the DURAFILE server for updates of Digital Preservation Plans Harvesting Agent This user is not a human user, but an intelligent agent software that monitors forums with information about digital preservation. This user detects obsolete formats and then create automatically new digital preservation plans. This agent will incorporate preservation plans outside of the ones interchanged among the Experts users in the platform. It will be able to discover new migration plans and obsolete formats monitoring external sources (web, forums, and blogs), and automatically create an appropriate migration plan. This is an asynchronous agent that will add new discovered preservation plans (from monitoring) to the user platform. The idea of automatically extracting information from forums is a very complex task and it is outside the scope of the project, but we propose a basic keyword based algorithm that automatically discovers the structures of the forum pages and extracts the information inside the discussion using a list of keywords. 3.3 DURAFILE Case Diagram Figure 16 shows the identified user cases of the DURAFILE platform taking into account the presented actors and their functionalities. 23

24 Figure 16 DURAFILE Case Diagram This subsection, describes the uses cases in more detail, describing for every type of user the provided functionality by the DURAFILE Platform Non-registered user uses cases The application will allow non-registered users to register to it as Personal User or Enterprise User. The user will choose his/her role in the application and optionally provide more contacts/personal information. The functionality offered by the DURAFILE platform to a non-registered user is: Register to DURAFILE. 24

25 Figure 17 Non-registered user Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to a nonregistered user (as shown in Figure 17). USE CASE Register to DURAFILE Description User registers to the DURAFILE Platform Actors Personal User, Enterprise User, Expert User, DURAFILE Administrator Assumptions Preconditions The non-registered user asked the application to register to it. Postconditions The user successfully registered to the application Steps 1. The user fills the registration formulary including a personal password. 2. The user chooses if he/she wants to register as Personal Users or Enterprise User 3. The user chooses if he/she wants to take the role of an Expert a. The user selects its field of expertise (audio, video, text) 4. The DURAFILE Administrator accepts the non-registered user. 5. The non-registered user receives a confirmation mail. 6. The non-registered user replies. Trigger The non-registered user clicks the sign in section Registered user uses cases The application will allow registered users to unregister to the platform including the remove of all its personal files and data. The functionality offered by the DURAFILE platform to a non-registered user is: Unregister to DURAFILE. Figure 18 Registered user Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to a registered user (as shown in Figure 18). 25

26 USE CASE Unregister to DURAFILE Description A registered user request to be unregister from the DURAFILE platform. Actors Registered User, DURAFILE Administrator Assumptions Preconditions The registered users request to be unregistered from the DURAFILE Platform Postconditions The DURAFILE Administrator removes all the personal data of the registered user. The DURAFILE Administrator removes all the files stored in the DURAFILE database relating the registered user. Steps 1. The registered user request to be unregistered from the DURAFILE Platform. 2. The DURAFILE Administrator removes all personal data of the registered user obtained when he/she registered to the DURAFILE Platform. 3. The DURAFILE Administrator removes all the files of the registered user stored in the DURAFILE platform. Trigger A registered user clicks the unregister section Personal user uses cases A Personal User is a regular home user who needs more than a backup for their multimedia files. The functionality offered by the DURAFILE platform to a Personal User is: Login to DURAFILE. Set Properties. Add Directory to preserve. Remove Directory to preserve. Add File to preserve. Remove File to preserve. Access Web DURAFILE. Figure 19 Personal user Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to a Personal User (as shown in Figure 19). 26

27 USE CASE Login to DURAFILE Description A Registered User wants to log into the DURAFILE platform. Actors Registered User Assumptions Preconditions The Registered User is not logged into the DURAFILE platform and wants to log in. The Registered User sets correctly its password. Postconditions The User has access to its personal data. The User has access to its personal files. Steps 1. The Registered User request to log into the DURAFILE Platform. 2. The Registered User sets correctly its password. 3. If the password is correct, the Registered User has access to its files. Otherwise the DURAFILE platform ask again for the password. Trigger The user clicks the Login section USE CASE Set Properties Description A Registered User sets the properties such as Desktop notification, upload rate, expert/regular user for interacting with the DURAFILE platform. Actors Registered User Assumptions Preconditions The Registered User is logged into the DURAFILE Platform. The Registered User wants to set new properties. Postconditions The properties set by the Registered User are accepted. Steps 1. The Registered User request to change/modify the properties when interacting with the DURAFILE platform. 2. The Registered User sets the properties for interacting with the DURAFILE platform by checking/unchecking the corresponding box. 3. The Registered User save the changes. Trigger The Logged User clicks the Properties section. USE CASE Add Directory to Preserve Description A Personal User wants to preserve the files include in one of its computer directory by using the DURAFILE platform. Actors Personal User Assumptions Preconditions The Personal User is logged into the DURAFILE platform. The Directory is included in the Personal User computer. Postconditions The files on the selected directory will be preserved by the DURAFILE platform. Steps 1. The Personal User selects the directory included in its computer. 2. The Personal User includes the directory in the list of directories to preserve. Trigger The Personal User clicks the Preserve with DURAFILE section. 27

28 USE CASE Remove Directory to Preserve Description A Personal User wants to remove a directory to be preserved by using the DURAFILE platform. Actors Personal User Assumptions Preconditions The Personal User is logged into the DURAFILE platform. The Directory is included in the Personal User computer. The Directory is included in the set of directories to be preserved. Postconditions The selected directory is not included in the user list of directories to be preserved. Steps 1. The Personal User selects the directory included in its list of preserved directories. 2. The Personal User removes the directory in the list of directories to preserve. Trigger The Personal User clicks the Preserve with DURAFILE section unchecking the option of preserving. USE CASE Add File to Preserve Description A Personal User wants to preserve a files include in one of its computer directory by using the DURAFILE platform. Actors Personal User Assumptions Preconditions The Personal User is logged into the DURAFILE platform. The file is included in the Personal User computer Postconditions The selected file will be preserved by the DURAFILE platform. Steps 1. The Personal User selects a file included in its computer. 2. The Personal User includes the file in the list of files to preserve. Trigger The Personal User clicks the Preserve with DURAFILE section. USE CASE Remove File to Preserve Description A Personal User wants to remove a file to be preserved by using the DURAFILE platform. Actors Personal User Assumptions Preconditions The Personal User is logged into the DURAFILE platform. The file is included in the Personal User computer. The file is included in the set of files to be preserved. Postconditions The selected file is not included in the user list of files to be preserved. Steps 1. The Personal User selects the selected file included in its list of preserved files. 2. The Personal User removes the file in the list of files to preserve. Trigger The Personal User clicks the Preserve with DURAFILE section unchecking the option of preserving. 28

29 USE CASE Access Web DURAFILE Description A Personal Users wants to access to the web version of the DURAFILE platform Actors Personal User Assumptions Preconditions The Personal User is has an account into the DURAFILE platform. Postconditions The Personal User has access to all its directories/files preserved by DURAFILE by using the web interface. Steps 1. The Personal User is logged into the DURAFILE platform. 2. The Personal User enters the URL of the DURAFILE website. Trigger The Personal User enters in its browser the URL of the web version of the DURAFILE platform Enterprise User uses cases An Enterprise User is a professional user which could be big institutions or SMEs that need to handle a large amount of multimedia information with high security issues. The functionality offered by the DURAFILE platform to an Enterprise User is: Login to DURAFILE. Set Properties. Add Directory to preserve. Remove Directory to preserve. Add File to preserve. Remove File to preserve. Access Web DURAFILE. Figure 20 Enterprise user Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to an Enterprise User (as shown in Figure 20). 29

30 USE CASE Login to DURAFILE Description A Registered User wants to log into the DURAFILE platform. Actors Registered User Assumptions Preconditions The Registered User is not logged into the DURAFILE platform and wants to log in. The Registered User sets correctly its password. Postconditions The User has access to its personal data. The User has access to its personal files. Steps 4. The Registered User request to log into the DURAFILE Platform. 5. The Registered User sets correctly its password. 6. If the password is correct, the Registered User has access to its files. Otherwise the DURAFILE platform ask again for the password. Trigger The user clicks the Login section USE CASE Set Properties Description A Registered User sets the properties such as Desktop notification, upload rate, expert/regular user for interacting with the DURAFILE platform. Actors Registered User Assumptions Preconditions The Registered User is logged into the DURAFILE Platform. The Registered User wants to set new properties. Postconditions The properties set by the Registered User are accepted. Steps 4. The Registered User request to change/modify the properties when interacting with the DURAFILE platform. 5. The Registered User sets the properties for interacting with the DURAFILE platform by checking/unchecking the corresponding box. 6. The Registered User save the changes. Trigger The Logged User clicks the Properties section. USE CASE Add Directory to Preserve Description An Enterprise user wants to preserve the files include in one of its computer directory by using the DURAFILE platform. Actors Enterprise user Assumptions Preconditions The Enterprise user is logged into the DURAFILE platform. The Directory is included in the Enterprise user computer. Postconditions The files on the selected directory will be preserved by the DURAFILE platform. Steps 3. The Enterprise user selects the directory included in its computer. 4. The Enterprise user includes the directory in the list of directories to preserve. Trigger The Enterprise user clicks the Preserve with DURAFILE section. 30

31 USE CASE Remove Directory to Preserve Description An Enterprise user wants to remove a directory to be preserved by using the DURAFILE platform. Actors Enterprise user Assumptions Preconditions The Enterprise user is logged into the DURAFILE platform. The Directory is included in the Enterprise user computer. The Directory is included in the set of directories to be preserved. Postconditions The selected directory is not included in the user list of directories to be preserved. Steps 3. The Enterprise user selects the directory included in its list of preserved directories. 4. The Enterprise user removes the directory in the list of directories to preserve. Trigger The Enterprise user clicks the Preserve with DURAFILE section unchecking the option of preserving. USE CASE Add File to Preserve Description An Enterprise user wants to preserve a files include in one of its computer directory by using the DURAFILE platform. Actors Enterprise user Assumptions Preconditions The Enterprise user is logged into the DURAFILE platform. The file is included in the Enterprise user computer Postconditions The selected file will be preserved by the DURAFILE platform. Steps 3. The Enterprise user selects a file included in its computer. 4. The Enterprise user includes the file in the list of files to preserve. Trigger The Enterprise user clicks the Preserve with DURAFILE section. USE CASE Remove File to Preserve Description An Enterprise user wants to remove a file to be preserved by using the DURAFILE platform. Actors Enterprise user Assumptions Preconditions The Enterprise user is logged into the DURAFILE platform. The file is included in the Enterprise user computer. The file is included in the set of files to be preserved. Postconditions The selected file is not included in the user list of files to be preserved. Steps 3. The Enterprise user selects the selected file included in its list of preserved files. 4. The Enterprise user removes the file in the list of files to preserve. Trigger The Enterprise user clicks the Preserve with DURAFILE section unchecking the option of preserving. 31

32 USE CASE Access Web DURAFILE Description A Enterprise users wants to access to the web version of the DURAFILE platform Actors Enterprise user Assumptions Preconditions The Enterprise user is has an account into the DURAFILE platform. Postconditions The Enterprise user has access to all its directories/files preserved by DURAFILE by using the web interface. Steps 3. The Enterprise user is logged into the DURAFILE platform. 4. The Enterprise user enters the URL of the DURAFILE website. Trigger The Enterprise user enters in its browser the URL of the web version of the DURAFILE platform Expert User uses cases An Expert User will be an experienced personal user that wants to help other users using a social network. The functionality offered by the DURAFILE platform to an Expert User is: Generate new Question. Answer Question. Submit new DPP. Figure 21 Expert user Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to an Expert User (as shown in Figure 21). 32

33 USE CASE Generate new Question Description An Expert User generates a new question regarding how to migrate a given format into a long-term maintained format. Actors Expert User. Assumptions Preconditions The Expert User is registered into the Expert Network and wants to make a question regarding a given format migration. Postconditions A question is generated from the Expert User which is distributed among the Experts Users of the Expert Network which are subscribe to the expertise of the question. Steps 1. The Expert User generates a new question in natural language (English) 2. The generated question is distributed among the Expert Network. Only the users with the background expressed by the question will receive it. Trigger The Expert User clicks the Generate new Question button. USE CASE Answer Question Description An Expert User receives a new question regarding how to migrate a given format into a long-term maintained format. Actors Expert User. Assumptions Preconditions The question is related to the expertise of the Expert User Postconditions The Expert User receives a question. Steps 1. The Expert User receives a question regarding to its expertise. 2. The user decides what to do with the received question: a. Answer. The user express in natural language the solution of the question (English) b. Ignores. The user ignores the question. Trigger The Expert User receives a new question related to its expertise into its mailbox. USE CASE Submit new DPP Description The Expert User creates a new DPP that wants to upload into the DURAFILE platform. Actors Expert User Assumptions Preconditions The Expert User wants to create a new DPP Postconditions A new DPP is created and submitted to the DURAFILE platform. Steps 1. The Expert User uses a form for creating a new DPP. 2. The Expert User generates a new XML which represent the DPP. 3. The Expert User uploads the created DPP into the DURAFILE Platform. Trigger The Expert User clicks the Submit new DPP plan. 33

34 3.3.6 DURAFILE Administrator uses cases The DURAFILE Administrator is a special user that administrate the DURAFILE platform. The functionality offered by the DURAFILE platform to a DURAFILE Administrator is: Authorized users. Remove DPP. Deauthorize users. Figure 22 DURAFILE Administrator Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to a DURAFILE Administrator (as shown in Figure 22). USE CASE Authorized users Description The DURAFILE Administrator allows a user to be registered by the DURAFILE platform Actors Non registered-user, DURAFILE Administrator Assumptions Preconditions The user requests for registering into the DURAFILE platform. Postconditions If DURAFILE Administrator accepts the request, the user is registered into the DURAFILE Platform. IF DURAFILE Administrator refuses the request, the user is not registered into the DURAFILE Platform. The unregistered user becomes a registered user. Steps 1. The DURAFILE Administrator receives a new request of an unregistered user is obtained. 2. The DURAFILE Administrator automatically accepts the request or refuse it due to a certain reason. 3. The DURAFILE Administrator sends the activation mail to the user which made the request. Trigger A new request from an unregistered user is received. 34

35 USE CASE Remove DPP Description The DURAFILE Administrator decides to remove a given DPP due to: Low performance or trust of DPP Inappropriate DPP The result of applying a given DPP produces unexpected/unsecure results. Actors DURAFILE Administrator Assumptions Preconditions A DPP is available in the DURAFILE Platform. It must be removed for a given reason. Postconditions The DPP is removed from the DURAFILE Platform. Steps 1. The DURAFILE Administrator searches in the DURAFILE Database the DPP. 2. The DPP is removed from the DURAFILE Database. Trigger The DURAFILE Administrator checks the database of DPP stored in the DURAFILE Platform and detects a DPP with low performance, inappropriate or insecure/unexpected results. The DURAFILE Administrator receives a report from a registered user relating a low performance, inappropriate or insecure/unexpected results of a DPP stored in the DURAFILE database. USE CASE Deauthorize users Description The DURAFILE Administrator decides to deauthorize a certain registered user due: The registered user wants to remove its DURAFILE account. The DURAFILE Administrator reports any abuse or noncomplaint behaviour from a certain registered user. Actors Registered user, DURAFILE Administrator Assumptions Preconditions A registered user will be remove from the DURAFILE platform. Postconditions The DURAFILE Administrator removes the user account including all the data of the registered user. The registered user becomes a registered user. Steps 1. The DURAFILE Administrator receives a request for removing a registered user account or he decides to remove a given registered account. 2. The DURAFILE Administrator removes all the provided information about the registered user. 3. The DURAFILE Administrator removes all the personal data stored in the DURAFILE Platform. Trigger The DURAFILE Administrator receives a request for removing a given registered account. The DURAFILE Administrator decides to deauthorize a given registered user. 35

36 3.3.7 Harvesting Agent uses cases A Harvesting Agent is not a human user, but an intelligent agent software that monitors forums with information about digital preservation. The functionality offered by the DURAFILE platform to a Harvesting agent is: Monitor a provided FORUM. Generate a DPP. Figure 23 Harvesting Agent Case Diagram Below are described in more detail the functionality offered by the DURAFILE platform to a Harvesting Agent (as shown in Figure 23). USE CASE Monitor a provided FORUM Description The harvesting agent monitors a given forum Actors Harvesting Agent Assumptions Preconditions The forum has a defined and known html structure Postconditions A set of keywords are obtained Steps 1. The harvesting Agent connects to a given forum. 2. The harvesting Agent know the html structure of the forum and parses it. 3. The engine extracts several keywords. Trigger The Harvesting agent could be programmed to be executed monthly or weekly. USE CASE Generate a DPP Description The harvesting agent generates a Digital Preservation Plan using the extracted keywords. Actors Harvesting Agent Assumptions Preconditions The harvesting agent has a set of keywords extracted from a given forum. Postconditions A new Digital Preservation Plan is automatically generated. Steps 1. The harvesting agent loads the DPP xml template. 2. The harvesting agent fills the DPP sections with the obtained keys. 3. The harvesting agent generates a new DPP xml file. Trigger The harvesting agent has obtained a set of keywords after parsing a given forum. 36

37 3.4 Sequence diagrams This section shows the preliminary sequence diagrams of how the DURAFILE platform makes the digital preservation of the user files. The sequence diagrams show 2 cases: Workflow for a file preservation, which are the steps for preserving a user file DPP request, which are the steps for asking for a new Digital Preservation Plan for an specific file format which is not present in the server File standard preservation workflow The next figure, tries to describe the normal way to preserve a file, assuming that DURAFILE Server has a preservation plan to preserve the user s file. 1. Client app has a file to preserve, and it sends to DURAFILE server. 2. DURAFILE server tries to preserve the file with some Preservation Plan that it has. 3. If it has some preservation plan for this file, it preserve the file. If it not has any preservation plan, it ask to the recipe server. 4. The Personal Users API alert to the client app that a new file has been preserved. 5. User downloads the migrated file from Users Web Interface Figure 24 File Standard Preservation Workflow DPP Request The next diagram defines which the steps the DURAFILE platform will perform in order to preserve a file that doesn t has any Preservation Plan. 1. The Client app has a file to preserve and send it to the DURAFILE Server. 2. DURAFILE Server realizes it doesn't have any preservation plan associated with that type of file. 37

38 3. DURAFILE Server alert to Recipe server in order to find a solution for this file. 4. Request Handler, evaluate the action to do with that petition, and then it ask to the experts community agents to find a solution. 5. Each agent, decide to ask or not to the human expert in order to find a preservation plan for that file. 6. The experts create a new preservation plan, and introduce it using the Web Experts Platform 7. The new preservation plan is evaluated for the Recipe Server validation processes. If it is tested correctly, the DPP will become one of the plans of the DPP repository. 8. The recipes informer, alert to all DURAFILE servers of the new DPP. Figure 25 DPP Request 3.5 DPPs Specification The task of selecting the optimal choice of action is one of the key responsibilities of the preservation planning function, which is at the heart of the OAIS reference model (ISO 14721:2012) [2, p 5]. According to Christoph Becker the key result of such a preservation planning activity is preservation plan, which has to take into account the designated community of users and their needs, but also technical conditions, options and constraints [2, p 14]. We thus rely on the following definition of what a preservation plan should contain, has been adopted by the Planets project [2]: - A preservation plan defines a series of preservation actions to be taken by a responsible institution due to an identified risk for a given set of digital objects or records (called collection). The Preservation Plan takes into account the preservation policies, legal obligations, organisational and technical constraints, user requirements and preservation goals and describes the preservation 38

39 context, the evaluated preservation strategies and the resulting decision for one strategy, including the reasoning for the decision. It also specifies a series of steps or actions (called preservation action plan) along with responsibilities and rules and conditions for execution on the collection. Provided that the actions and their deployment as well as the technical environment allow it, this action plan is an executable workflow definition. So far, preservation planning is a mainly manual, sometimes ad hoc process with little or no tool support. Publicly available web-based decision support tool Plato has been developed as part of the Planets Project 1 (by the Digital Preservation lab at the Vienna University of Technology 2 ). It implements a solid planning process and integrates a controlled environment for experimental and automated measurements of outcomes which enables trustworthy, evidence-based decision making. The result of using Plato is a complete preservation plan that can be deployed and executed. It also supports import and export of all plans developed. A preservation plan can be exported as an XML file and downloaded as such, and uploaded to a new version of the planning tool [3]. The DPP should contain the elements like Identification; Status and triggers; Description of the institutional setting; Description of the collection; Requirements for preservation; Evidence of decision for a preservation strategy; Costs; Roles and responsibilities and Preservation Action Plan. Preservation Action Plan specifies concrete actions to be undertaken in order to keep the collection of digital objects alive and accessible over time. A preservation action can be just the application of a single tool to a set of objects, but can also be a composite workflow consisting of multiple characterisation and action services. In this sense, the preservation action plan specifies two main aspects: the when and the what, says Becker [2, p 53] Related Projects This section shows an overview of two related projects: The PLANETS and the SCAPE projects. They use a XML format that can be used for the Digital Preservation Plans in the DURAFILE project. Choosing the right treatment for a given set of objects is a crucial decision that needs to be taken based on a profound and well-documented analysis of the requirements and the performance the tool considered [2, p 5]. So far, preservation planning is mainly manual, sometimes ad hoc process with little or no tool support The PLANETS Project Publically available web-based decision support tool Plato has been developed as part of the Planets Project 3 (by the Digital Preservation lab at the Vienna University of Technology 4 ). The Planets project method of systematic preservation planning is based on earlier work done in the DELOS project 5 which has been revised and extended. It implements a solid planning process and integrates a controlled environment for experimental and automated measurements of outcomes which enables trustworthy, evidence-based decision making. Plato provides substantial automation and guidance and it documents all decisions made in the planning process. According to Becker, the main benefits of Plato are guidance, automation and Digital Preservation Lab, Department of Software Technology and Interactive Systems, Vienna University of Te chnology: Digital Preservation Lab, Department of Software Technology and Interactive Systems, Vienna University of Te chnology:

40 documentation [2, p 93]. Preservation action services are discovered and invoked through a flexible layer of adaptors. The time-consuming and inherently subjective process of evaluating the results is being objectified and automated as far as possible by mapping identified requirements such as essential characteristics of objects to properties that can be automatically extracted and compared by characterisation tools. The planning tool implements the entire workflow of 14 steps described previously. The result of using Plato is a complete preservation plan that can be deployed and executed. It also supports import and export of all plans developed. A preservation plan can be exported as an XML file and downloaded as such, and uploaded to a new version of the planning tool. An XML file includes the complete evidence base i.e. the sample objects, the documented scenario, and applying constraints and policies, the candidate components considered the results of experimental evaluation and the analysis documentation [4]. PLATO has basic preservation plan template which can be seen here: Brief overview about different PLATO workflow steps with some examples: Step-by-step presentation, how to make preservation plan in PLATO: Publically available preservation plan example in XML (PLATO) will look like this: S _ xml Basic PLATO execution plan example, which is based on Taverna: S_-_DosBox_Wine_Linux.t2flow So far, PLATO is the only project in digital preservation field that has developed a good set of constrains and policies to automatically create reliable and complete preservation plan. The SCAPE project uses PLATO implementation core for its preservation plans and is developing the PLATO system. The Planets projects method of systematic preservation planning is based on earlier work done in the DELOS project 6 which has been revised and extended. Analysis below is based on two main documents such as: an article of Becker, Kulovits et al, Systematic planning for digital preservation: Evaluating potential strategies and building preservation plans [4] and Planets Preservation Planning Tool: Plato 3.0, User Manual [3] An overview of the four-phase Planets Preservation Planning workflow The DPP workflow according to Planets project consists of four phases: 1. Define requirements 2. Evaluate alternatives 3. Analyse results 4. Build preservation plan. Figure 1 illustrates the preservation planning environment, putting the high-level workflow in the context of the main environment factors to which it relates. The four phases result in a working preservation plan that can be continually executed

41 Figure 26 Preservation Planning Environment An ongoing monitor function is necessary to ensure the ability to adapt to detected changes in either the environment, the technologies used in operations or changing objectives. This results in a continuous circle of revisions to preservation plans and enables the repository to react accordingly to the inevitable changes to be expected. Figure 27 shows the concrete steps within this high-level workflow, which the next sections will discuss in detail. 41

42 Figure 27 Preservation Planning Workflow Short description of preservation planning workflow The 4-phase process starts with defining the preservation scenario, choosing sample records for experiments, and identifying the requirements and goals. The second part of the process consists of the definition and evaluation of potential preservation alternatives. Alternatives are therefore identified, including technical settings and required resources for running the experiments. The Go/No-Go-Decision enforces a review of the work in the previous steps. In the experiments the preservation alternatives are applied to the sample records. The final step of the second phase is the evaluation of the experimental outcomes against the requirements and goals defined in the first phase. In the third phase results of the experiments are aggregated to make them comparable, importance factors are set and the alternatives are ranked. The stability of the final ranking is analysed with respect to minor changes in the weighting and performance of the individual objectives using Sensitivity Analysis. After this consideration a clear and accountable recommendation can be made for one of the alternatives. In the fourth and final phase of the planning workflow a preservation plan based on the decision for a preservation action is created. It specifies a series of steps or actions along with organisational responsibilities, rules and conditions for executing the preservation action on the collection. 42

43 Detailed description of preservation planning workflow The 4-phase process consists of 14 steps: 1. Define basis In this step, the preservation scenario is described in a semi-structured way including the collection to be considered. Information about the collection includes details about the objects, number of objects in the collection, and legal requirements for handling the records. Moreover, the preservation planner documents applying institutional policies, legal regulations, and usage criteria that might affect planning decisions for preservation. This may happen in an unstructured form, but preferably these factors are captured in a more formal way making it easier to derive decisions in the respective workflow steps. 2. Define sample objects A general description of the characteristics of the set of objects, called collection, includes basic properties such as the size of the collection, the class of objects and the object formats they are currently represented in. While this can be done in a manual descriptive way, a formal representation is desirable. Collection profiling tools can provide automated descriptions of the technical characteristics of objects. An example of such a profiling service is described in [10]. Characteristics of interest include not only object formats, file sizes and their variation within the collection, but also aspects such as an assessment of the risks of each object type and each object, thus leading to a risk profile of the collection. As a complete evaluation of the quality of preservation action tools is infeasible on the potentially very large collection of objects, the planner selects representative sample objects that should cover the range of essential characteristics present in the collection at hand. In order to reduce effort to a minimum, this subset should be as small as possible. However, the sample objects are used as a representative set for testing the effects of applying preservation actions to the whole set of objects. A complete and thorough evaluation of the quality of preservation actions relies heavily on the completeness of features present within the test set. Thus it needs to be as large as needed to cover the variety of essential characteristics on both a technical and an intellectual level. Depending on the degree of variance within the collection, typically, between 3 and 10 sample objects are selected and they are later used on for evaluating the preservation alternatives. 3. Identify requirements Requirements definition is the heart of preservation planning and it has proven to be the most critical and complicated stage of the planning procedure. It is the basis for the decisions to be taken, and documents the priorities and preferences of the institution. This step enlists all requirements and goals (objectives) that the optimal digital preservation solution needs to fulfil. Requirements are collected from the wide range of stakeholders and influence factors that have to be considered for a given institutional setting. This may include the involvement of curators and domain experts as well as IT administrators and consumers. The requirements are specified in a quantifiable way, starting at highlevel objectives and breaking them down into measurable criteria, thus creating an objective tree which forms the basis of the evaluation of alternative strategies. The complexity involved in specifying goals and breaking them down to concrete, quantifiable criteria is a considerable challenge. An incomplete requirement specification leads to a skewed evaluation and potentially wrong decisions. On a practical level, two tools have been very useful for the requirements elicitation process: sticky notes and mind-mapping software. While sticky notes and flip charts as classical tool to support brainstorming activities have the benefits of allowing everyone to act at the same time, mind maps 43

44 provide the better overview of the current state of requirements for all participants and allow a moderator to channel the discussion process. Often, a combination of both tools is the most productive approach. Using these tools, the requirements are structured in a hierarchical way, starting from general objectives and refining them via more specific requirements to arrive at measurable criteria that a successful digital preservation solution has to meet. This structure is further referred to as the objective tree, i.e. a tree capturing the objectives to be met. While the resulting objective trees usually differ through changing preservation settings, some general principles can be observed. At the top level, the objectives can usually be organised into four main categories characteristics of the objects, the records, and the process, and requirements on costs. Object characteristics describe the visual and contextual experience a user has when dealing with a digital object. These characteristics are often referred to as significant properties. A common way of describing them is to consider the five aspects Content, Context, Structure, Appearance and Behaviour. Recently, several projects such as INSPECT 7 have presented detailed analysis of the significant properties of different categories of objects including vector images, moving images, e- Learning objects, and software 8 These can provide a very valuable input to this aspect of requirements specification. The automated characterisation of the sample objects defined in the previous step further supports the analysis of their significant technical properties. Record characteristics describe the foundations of digital record, the context, interrelationships and metadata. This may include simple, but often overlooked, linking requirements, such as the fact that filenames need to remain unchanged or consistently renamed across subcollections if they form the basis for cross-referencing or inclusion. Process characteristics describe the preservation process itself, for example the procedure of migrating objects. These characteristics include not just the complexity of applying preservation action tools or their performance, scalability, and usability, but should equally cover aspects such as documentation or the degree of validation. The definition of process characteristics is particularly dependent on the specific context in which the preservation process is taking place. The technical environment may effectuate specific requirements on the interoperability of tools, while institutional policies or legal regulations may enforce specific licensing requirements or require a particular degree of automated documentation. Thus, the institutional and technical context and constraints posed by it have to be considered carefully. Costs have a significant influence on the choice of a preservation solution, but are inherently hard to quantify. Ultimately, the Total Cost of Ownership (TCO) is the guiding figure for deciding whether or not a preservation strategy meets the needs of an institution within the constraints of its budget. Instead of providing a single numeric criterion which is extremely complex to quantify, costs might also be defined as infrastructure characteristics, putting an emphasis on cost factors instead of the resulting figures for cost estimates. An essential step of requirements definition is the assignment of measurable effects to the criteria at the leaf level of the objective tree. Wherever possible, these effects should be objectively measurable (e.g. e per year, frames per second, bits per sample) and thus comparable. However, in some cases, (semi-) subjective scales need to be employed. For example, the quality of documentation that is available for a file format or a tool should not be judged by the number of pages alone; instead, a subjective scale such as excellent, good, average, poor, very poor could be used. Similarly, the openness of documentation of a file format could be one of fully standardised; openly published, but not standardised by a recognised body; and proprietary. Along the same lines, the stability of a format can be measured in revision time intervals and backwards compatibility

45 The assignment of measurable effects to criteria can also align them with characteristics that can be automatically extracted from objects to automate the evaluation procedure. Existing software tools such as JHove 9 allow automated extraction of some of the basic properties of common object formats; the extensible Characterisation Languages provide an in-depth description of the complete informational content of an object in an abstract representation [8]. These descriptions can be used to derive properties to be measured, and support the automated comparison of these properties when migrating the objects to different formats. In the third phase, these measurements are transformed and, thus, made comparable through the definition of transformation rules, which calculate unified utility values based on the knowledge gained in the experiments. In the final step, binary criteria can be used to filter alternatives, while the weighted overall performance across all criteria is then used for the final selection of the best action. A series of case studies have been conducted where objective trees were created for different settings. Examples include electronic publications in a national library [5]; web archives and electronic documents with national archives; interactive multimedia in an electronic art museum [4]; and computer video games. The outcome of the first phase is a complete documentation of the planning context, the collection of objects at question and the specific requirements that form the basis for the evaluation of alternative action paths. 4. Define alternatives Different preservation strategies, using for example migrations tools or emulators, are selected. A detailed description of each preservation alternative is provided. The description includes the software environment and parameters settings of the tool, in order to ensure a clear understanding of the alternative and allow a later re-evaluation of the planning process. For each defined alternative, the amount of work, time, and money required for running experiments is estimated. 5. Take go decision Some experiments need a considerable amount of effort and required resources to run the experiments, for example experiment with great number of alternatives or high cost of hardware and software to run the experiments. Feasibility of the proposed alternatives are determined in this step by considering the defined requirements, the selected preservation alternatives, and estimated resources. The result is a decision for continuing the evaluation process or a justification of the abandonment or postponement of certain alternatives. 6. Develop experiments In the experiments, the preservation alternatives are applied to the previous defined sample records. The results of the experiments are later evaluated against the goals and requirements of the objective tree. In order to run repeatable tests, it is important to document all relevant experiment settings. This stage produces a specific development plan for each experiment, which includes the workflow, the software and hardware systems used for the experiments, and the mechanisms to capture the results. All items needed for the experiment will be developed and/or installed and tested, including copies of the sample objects, software packages and programs, and mechanisms for capturing the results

46 7. Run experiments Experiments are designed to test one or more aspects of a specific preservation alternative when applied to the previously defined sample records. Running an experiment produces results, for example converted computer files, revised metadata, and measured workload of the hardware. The results are evaluated in the next step. 8. Evaluate experiments The results of the experiments are evaluated to determine the degree to which the requirements defined in the objective tree were met. Therefore, the leaf objectives defined in the objective tree are evaluated with the defined measurement unit. For each alternative, the outcomes of this stage are measured performance values for each leaf in the objective tree. 9. Transform measured values The measurements taken in the experiments might all be measured on different scales. In order to make these comparable, they are transformed to a uniform scale using transformation functions. These transformation functions can define thresholds or injective mathematic functions to map the measured values to the uniform scale. The resulting scale ranges from zero to five. A value of zero denotes an unacceptable result and thus serves as a dropout criterion for the whole preservation alternative. 10. Set importance factors Not all of the identified objectives are equally important and different degrees of conformance of a solution are accepted in different objectives. This step assigns importance factors to each objective depending on the specific preferences and requirements in the scenario. 11. Analyse results In this step, the performance measures for the individual objectives are aggregated to one single comparable value for each alternative. Currently the following methods are available: Sum. The measured performance values, as transformed by the transformation functions, are multiplied by the weighting factor. These values are summed up to a single comparable value per alternative. Leaf values that score zero (measured performance under required minimum threshold) have no decisive effect on the final root value. Multiplication. The first step here is to multiply the comparable value per leaf by the weight of that leaf. The results are then multiplied throughout the tree for the whole alternative. The multiplication method highlights alternatives with drop out values, as these alternatives with leaf values zero have a final root value of zero. Sum of Advantages. This aggregation allows the comparison of two alternatives against each other, it is a comparative comparison of two alternatives. The DELOS Testbed (6) has implemented the aggregation function Sum of Priority, which is based on integer numbers. Because we introduced real numbers for the uniform scale in the Planets approach, this aggregation is not feasible. The newly introduced Sum of Advantage function supports the comparison of real numbers. Each alternative is leaf value is compared to the leaf values of the other alternative. The final value for the alternative with the higher leave value is the difference, the final value for other alternative is zero. 46

47 These values are summed up over the tree and form a final value for each alternative. The final values of an alternative show the difference of the performance compared to the other alternative. We thus obtain aggregated performance values for every part of the objective tree for each alternative, including an overall performance value at the root level. A first ranking of the alternatives can be done based on the final root values associated with each alternative. This ranking is based on the specific requirements of the preservation context. It forms the basis for a documented and accountable selection of a specific preservation alternative. Furthermore, an analysis of all parts of the objective tree can identify the strengths and weakness of an alternative. In addition to ranking, Sensitivity Analysis may be performed by analysing, for example, the stability of the ranking with respect to minor changes in the weighting of the individual objectives, or to minor changes in the performance. This will result in a stability value for each alternative and each objective, which may further influence the final decision. 12. Create executable plan This step of the workflow defines the triggers for the execution and under what conditions the preservation action will be carried out. Hard- and software requirements as well as dependencies to other systems are documented. For execution of the preservation plan tool settings and details about the location of the collection on which the action is to be performed are defined. To quality assure the performed actions a subset of the criteria used for evaluating solutions can be selected. These criteria will then be extracted by using characterization services on the objects after the performed action and testing, if defined thresholds of these criteria are met. The necessary documentation that has to be recorded for performing the action is also defined in this step. 13. Define preservation plan While most parts of the preservation planning workflow take care of the technical aspects of the preservation plan, this step mainly defines the organisational details. Cost factors influence the decision on a specific alternative. In this step a more detailed calculation of costs using an approved cost model is performed. Cost models that can be used are for example Life2 or the Total Cost of Ownership (TCO) model. While an estimate of the costs may be fine for evaluating the alternatives, the costs have to be determined as accurate as possible in this step. The assignments of responsibilities is also documented in this step. Monitoring the process of applying the preservation actions has to be done by a different role than executing the preservation plan. It also has to be monitored if an event occurs that makes it necessary to re-evaluate the plan. Possible triggers for this are either a scheduled periodic review, changes in environment such as new available tools detected through technology watch, changed objectives (changed target community requirements) or a changed collection profile (e.g. new objects in the collection). Another possible trigger is that certain thresholds on the evaluation are no longer met by applying the preservation action. 14. Validate plan In the final stage the whole documentation about the preservation plan has to be reviewed. The process of evaluating different alternatives and taking a decision for a recommended tool based on the evaluation and the creation of the preservation action plan, the documentation about the basic framework of the institution and parameters under which the plan is valid are verified again. Tests on a defined set of sample objects are performed in this step to check the validity of the preservation plan and the preservation action plan. Finally the validated plan has to be approved by the person responsible for approval. Once the plan is approved, no more changes on the plan should be done without revising the whole plan. 47

48 The result of the preservation planning process described above is a complete preservation plan. It consists of a concise, objective, and well-documented ranked list of preservation alternatives for a given preservation context, considering institution-specific requirements. By providing both overall and detailed performance measures, based on the standardised and repeatable experiments, it allows the selection of most suitable preservation strategy. Based on this selection a complete preservation plan for performing the preservation action can be defined The SCAPE Project SCAPE has also defined a large number of tools and services that can be used for preservation actions. These tools are evaluated and ranked in SCAPE deliverable 10.1, section 7 [6]. Portuguese National Archive has done research in this field. In SCAPE project they have developed a control policy for different type of preservation classes, variables and tools to be used within preservation plan [5]. SCAPE is an integrated research project to develop solutions for the long-term digital preservation of large-scale and heterogeneous collections of digital-objects. Their aim is to develop scalable services for efficient and automated preservation planning and the execution of preservation actions of large (multi-terabyte) and complex data sets. On abstract architecture, the SCAPE architecture is shown in Figure 28. What could be most important here is that SCAPE uses also PLATO core for automatic preservation planning. Figure 28 SCAPE architecture 48

49 Testbeds are the primary driver for the project in that they determine the use case scenarios, define the preservation workflows, and evaluate the platform and the tools created in other parts of the project. The SCAPE Testbeds define specific preservation issues (described by the scenarios) with a special focus on three application areas: Large-scale digital repositories, Research Datasets, and Web Content. The primary user applications in the SCAPE architecture are Taverna (for the Testbeds) and PLATO (for Preservation Watch). PLATO will be used for preservation planning, and is needs to invoke preservation tools during the planning process. Taverna External Tools & Components are used to build these standardised interfaces. This will be done by defining a shared schema for tool specifications. These simple XML definitions describe individual tools and how to invoke them to perform different types of actions. The specifications are based on those used by the Taverna External Tools plugin which look like this: <program name="bourne_shell_script" description="execute shell script" command="/bin/sh input"> <input name="shell_script"> <file path="input" /> </input> </program> Digital Preservation Plans in DURAFILE DURAFILE digital preservation plan specifications needs to be on high level as possible. For end-user we may not need specification for the business risk analysis for example but when we also aim more corporate version then we will need good specification. Preservation plans in general could be very complex structures of different rules, variables and data sets are needed to select best possible plan for file migration and preservation. Depending on how high level our preservation plan will be, good system architecture will be needed for risk analysis and other important steps which will be done in parallel. For example like requirement analysis (for preservation plan), strategies and polices which are one of the key elements in preservation plan. DURAFILE preservation plan specification contains all the information necessary for safe, secure and trusted file migration and preservation. DPP should be in some format which ensures platform, specific programming language and system dependability. We propose XML format for the preservation plan specification. Basic DDP example Below is very basic demonstration how our DPP for end-user could look like by using XML structure. Elements are definitely not fixed and are expected to be changed. For the end-user we need to ask minimal input as possible. DURAFILE minimal DPP should contain structural parts like: 1) DPP description and general DPP specific metadata. 2) File format metadata. 3) Outsource analyse result data (such as risk analysis and others). 4) System (HW/SW) environment. 49

50 5) Action Plan (List of actions/scripts/software runs to be taken to migrate and preserve file) 6) Simplified rules and policies. DURAFILE Digital preservation plan example: <DPP_for_PDF> <DPP_general> <Name>Preservation Plan for PDF Version 10</Name> <ExpertRating>80</ExpertRating> <AgentRating>70</AgentRating> <CommunityRating>50</CommunityRating> #default 0 </DPP_general> <File_meta> <Format>PDF</Format> <Version>10.0</Version> <Encoding>UTF-8</Encoding> <Undefined> </Undefined> # Like decoder/encoder information, resolution, DPI etc. Depends on file type/format. </File_meta> <Analyze> <RiskFactor>Medium</RiskFactor> # Risk Analysis <Alternatives>2</Alternatives> # Alt-branch search, could contain links to alternative plans as well. <Usage>General</Usage> # Usage analysis <TargetVersion>8.0</TargetVersion> # PRONOM register? </Analyze> <Environment> <OS>Windows 7 SP 1</OS> # Expert/Community tested environment <CPU>Intel Core i7 2700K</CPU> </Environment> <ActionPlan> <Use>Validation Utility 2.0</Use> <Software>PDF10_to_PDF8_converter.exe</Software> <Script>MyPDFRecheck.java</Script> </ActionPlan> <Policy> <FileSizeChangeAllowed>0</FileSizeChangeAllowed> < HaveLosslessCompression>1</ HaveLosslessCompression> </Policy> </DPP_for_PDF> First section describes general DPP information such as DPP name, both expert and agent/software and community rating. There could be more DPP specific metadata and information if needed. Second section describes file format that will undergo preservation action, information and metadata (such as format, version, encoding etc). This information will be automatically extracted by using metadata extractor software (e.g. JHOVE, etc.). Here we can use information about metadata/structure for different file types. Third section describes output data previously collected from different types of analysis steps, like risk analysis, branch analysis, usage analysis etc. This will be done mostly automatically however some user input maybe required. There can also be information about standards, best preservation formats that can be collected from external knowledge bases. In the fourth section the tested environment is described. There will be information about the tested software and hardware environment. It can also include some OS/HW based scripts and tools used in preservation plan description. Idea here is to warn the user and ask input when for example the plan is made on Linux but it is not tested on Windows. The user can accept the risk and use the DPP anyway and afterwards provide 50

51 feedback about its success. There can also be information available about environments where the current plan will definitely not work (based on community feedback and automatic analysis). Then the agent will not apply the DPP if the target system does not match the requirements. Fifth section is about different actions, scripts and software tools that are needed for preserving targeted file. There can be built-in utilities as well as scripts added by experts or third party software tools. We can use microservices as well. These tools are listed in the same order as actions have to take place. Action list can also be described as a separate script that can be used directly in the main code. Here we can use, to an extent, Archivematica s ( roadmap for predefined tools to be used [7] also SCAPE has done great analysis in their deliverable D10.1 [6, section 7] Last section is about policies, rules and variables that are needed to check and validate the output file. We can use the SCAPE policy document as an example [6]. For the end-user we may not need complex policies and rules but for corporate version this needs more analysis. 4 DURAFILE Implementation Based on the specification of the DURAFILE platform and the user requirements mentioned in the previous section, an implementation of the DURAFILE platform is provided. The section provides a detailed documentation about the platform architecture and a justification of the components which made it. The architecture implements as many user requirements as possible and a detailed description of this is provided. Implementing Digital Preservation Plans is a difficult task, which is described in 4.3 explaining how to connect them with the DURAFILE platform. Workflow engines which can be included in the platform are also mentioned. This section also provides several end-user Interface based on the described functionality of the platform. 4.1 Platform Architecture Figure 29 shows the proposed architecture of the platform. The main idea of the platform is to automatize the file migration into a long-term file format. We assume a Dropbox model where the user selects a directory in his computer where he will put all the digital files that should be maintained in a long-time preservation format. All the files presented in this directory will be stored in the DURAFILE server and can be accessed from different computers or platforms (similar as Dropbox does). 51

52 Figure 29.- Components of the architecture for the DURAFILE PLATFORM DURAFILE Cloud Server This is the main platform, which is composed of DURAFILE Server, Recipe Server and DURAFILE Databases. The overview of the DURAFILE Cloud Server is showed in the Figure 30. Where the DURAFILE server and Company server has the same structure, with the only difference that company server is for the enterprise version. The only difference between the Standard version and the Enterprise version is where the Software is placed. In the enterprise version, the server software is installed into company s server. 52

53 Figure 30 - Cloud Server Architecture DURAFILE Server DURAFILE Server has the responsibility to execute the preservation processes over the files uploaded by the final users. The preservation processes analyse the files one by one, identify which DPPs should be applied and execute them. Figure 31 - Conceptual Proposed architecture for the DURAFILE PLATFORM. In order to preserve the files the DURAFILE Server has the following parts: Personal Users API. Those are the public services where the desktop applications will be connected. All the connections will be encrypted to avoid attacks. User Web Interface. Each user will be able to watch all his preserved files using the Web Application. Preservation Recipes. It s a kind of repository, where all DURAFILE Servers will keep the validated Digital Preservation Plans. Preservation Recipes API. What happens if a file has no a DPP associated? Then, a request is generated to the RECIPE Server. The Preservation Recipes API also will be used to update the DURAFILE Server repository of DPPs. 53

54 Figure 31 - Conceptual Proposed architecture for the DURAFILE PLATFORM Recipe Server The RECIPE Server responsibility is to coordinate the experts community to generate new DPPs. In order to do that, the server has several functionalities: Experts directory: all the experts must be registered to the Recipe Server. This directory is used to define the expertise of the experts and which is the trust the system, based on the results of the DPPs that have generated. Agents environment: this is a multi-agents system that supports the experts community to generate DPPs. Social search is one of the functionalities that experts can use to interact with other in order to solve they questions about preservation. Agents try to automatize as much tasks as they can, for example, which are the best contact lists and which questions can be answered automatically without interrupt the expert. Request handler: once a DURAFILE Server detects the need of a DPP sends a request to the Recipe Sever. The request handler receives this requests, and use the expert s directory to ask them for a DPP. Experts Web Interface: this is just a web form that experts use to submit new DPPs. Recipes Pool: All the submitted DPPs are stored in this pool. Validation processes: all the DPPs should pass a set of validation tests automatically executed by the Recipe Server. If any problem/error, the expert that submitted the DPP is contacted in order to improve the DPP. Recipes repository: only the validated DPPs are stored in this repository. Recipes informer: it is the service in charge to broadcast to all the DURAFILE Servers the new DPPs or to update the existing ones. 54

55 Figure 32- The RECIPE Server architecture DURAFILE Databases In a Dropbox model this database is where the user will store his data (if desired). That should be an internal database or located in an external provider (Amazon S3, other ). In this last case, a contract will be signed for assuring the privacy of the data stored in these database providers. The whole Platform DURAFILE consists in three kinds of databases, which has different kind of information depending on the tasks that the database is designed for DURAFILE Server database This Database will be allocated as part of the DURAFILE Server or Enterprise Server. It will be in charge of storing all data related with the DURAFILE clients, his files, the executed preservation actions and the rest of DURAFILE client interactions with the API and the Web User Interface. On the other hand, it has to have a control of what Preservation Plans are validated as Production preservation plans, and are enabled to preserve files Recipes Server Database The DURAFILE platform only will have a unique Recipes Database. This database will contain all the information about the Digital Preservations Plans, and the actions that they are formed for. It also will contain the information about the users of the platform, and the information of the experts of the platform. 55

56 This database will be more complex than the DURAFILE database, because it also will contain all the information about the contact list of the users, the information about each Agent, the trace of the expert s user with the platform. Finally, one of the most important objects that this database will have, is the information about the trust and the reputation of each expert user in the platform Client Database Each client application of the DURAFILE System, will have his own database. In this case, the information that this database will have, it s responsibility of each client application. There could be a client application without database, and there could be another one with a big database. The unique mandatory data that the client has to save is the list of the preserved files in his File System DURAFILE platform users The DURAFILE platform users were extracted directly from the User Roles identified and described in section 3.2. The platform is focussed into 2 main types of users: a user that wants to preserve their personal data and companies that needs digital preservation due to normative or legacy issue (for example, for recording invoices during a certain period of time). Other user that will be implemented are the Harvesting Agent, the administrator user and the third-part actors. Personal Users. Regular user that wants to preserve its multimedia files. Experts Users. Users with specific user requirements (security/large collections) that needs a dedicated server. Enterprise users. Expert users that wants to participate in the creation of Digital Preservation Plans. Harvesting Agent. Automatic software for extracting basic information in certain forums about obsolete formats. Administrator User. Special user which administrates the DURAFILE platform. Third-part Actors. External users that offers complementary services to the DURAFILE platform Personal Users This is a regular user (without any specific issue of privacy) that could be a private user or a company that wants to preserve their digital files. He selects a directory that contains the digital files that must be preserved in a long-time preservation format. It is important to point that the files will be only accessed by the user (similar to Dropbox). The platform includes the FIREWALL for avoiding unauthorized access and the communication with the DURAFILE server will be done through a WS- Security connection Enterprise Users For those users that needs privacy, mostly companies, we propose a local solution, where we will install a local copy of the DURAFILE sever customized for their needs. Also companies that manage large files (such as video files) could use this local solution. Digital Preservation will be done locally; therefore the file will never be uploaded to the central server (except in the case that the user allows it). The local 56

57 server will be connected to the DURAFILE central server by using a SSL connection in order to interchange Digital Preservation Plans that could be not available at the moment in the local server Expert Users Expert users can be personal or companies that want to participate in the generation of new Digital Preservation plans. They form a network of experts that receive digital preservation requests from the DURAFILE Central Server in case a Digital Preservation plan is not stored in the RECEIPE Server DURAFILE Administrator Is/Are the person(s) responsible(s) of the maintenance and monitoring of the DURAFILE platform. It is in charge of controlling the user s accounts in the platform, controlling and maintaining the experts network, maintenance of the digital preservation plans (removing the ones that the users classified as invalid or with a poor performance), authorizing third-part companies (DP providers that can complements the function of the DURAFILE platform) and allowing that enterprises servers which are installed in specific companies can be connected to the DURAFILE server for updates of Digital Preservation Plans Harvesting Agent The Harvesting agent will be programmed for detecting simple string matching in a certain Webpage and using keywords for detecting obsolete formats. Although we want to monitor as many forums as possible, initially, we will monitor a forum about digital preservation from the Open Planets Foundation ( This web has the most up-to-date technical discussion of what happens in the area of preservation planning. This web includes a Digital Preservation Tool Registry ( storing preservation plans that plans can be extracted. Due that we are focusing in a specific site, we need to know the static structure of this webpage. Knowing this static structure, a simple parser can be constructed in order to extract the proposed preservation plans Third part actors The platform also allows that third-part actors can connect to the platform offering services that are out of the scope of the project: THIRD-PART DIGITAL PRESERVATION PROVIDERS. The architecture includes third-part digital preservation providers. In case that a user needs a specific Digital Preservation plan with high quality, several providers can offer their solutions in a pay-per-migration model (or other). THIRD-PART CURATION PROVIDERS. The platform allows that third-part curation providers could offer and complements the services of the DURAFILE platform. Using a pay-per-curation model (or other) they can offer curation services to those corrupted files detected by the platform. THIRD-PART HARDWARE PRESERVATION PROVIDERS. Hardware obsolescence is out of the scope of the DURAFILE platform. The platform offers the possibility that third-parts experts in hardware preservation connect to the platform and for example extract a picture made at the end of the 80 s that was recorder in an old 3.5 floppy disk. Once the third-part provider 57

58 extracts the file, it could be inserted into the platform for provide a suitable migration into an actual format. 4.2 Implemented requirements The proposed DURAFILE platform will implement a list or requirements without which the product may as well not be developed at all; it will be of no use to most Customers without these requirements. We show the list of mandatory requirements obtained and explained in previous section. REQ REQUIREMENT COVERED ARCHITECTURE NUM 1 Automatic migration maintaining the original file. YES The original file is maintained in the selected directory, while a new file in a long term format is obtained. 2 Automatic preservation for files with obsolete formats. 3 The system has to be updated all time. Like an Antivirus, it has to download all the Digital Preservation plans according to the trust. 4 The GUI has to not disturb the user with messages and alerts. 5 The Expert user can upload its Digital Preservation Plans, and share it with the "global" experts community. 6 A user requesting for a Digital Preservation Plan can receive different plans with different trust value. 7 The communication between server and clients should be YES YES YES YES YES YES The platform checks periodically the files in the selected directory, detecting obsolete formats The platform incorporates new Digital Preservation plans through the experts networks and trough the Harvesting agent. Therefore the DURAFILE central server always is up to date. For the local solutions, there is a direct connection between the local server and the central server for the update process. The GUI is similar to the Dropbox. Pop-up messages appear in the tray bar when an obsolete file is detected. The network of experts solves, creates and shares Digital Preservation Plans which share with the other experts by using the Social Search. The network of experts can provide through the Social Search different Digital Preservation Plans to a certain migration. This is a suitable feature due to the dynamic nature of the Digital Preservation Problems. Communication between the server and the clients by using a WS-Security certificate. 58

59 secured by using a SSL certificate. 8 Data stored in the DURAFILE server must be only accessible by the owner. 9 Different interfaces and functionalities for the different roles: users, experts, administrator. 10 The software should be developed using open source technologies, without the need of buying other software licenses. 11 Ability for the user to export all his data and erase the account of the DURAFILE platform with all his related data. 12 Search and analyze the content of pre-defined folders/directories in the file system. 13 Detect an obsolete, incomplete or otherwise at-risk file. 14 Evaluating the success of the applied Digital Preservation Plan both automatically (e.g. has the initially discovered problem disappeared) and through the enduser (e.g. evaluating or ranking of successful outcomes on a scale of 10) sharing the evaluation with other users. YES YES Probably YES YES YES YES YES The files will be stored in an external database provider which will assure that only the owner of the file can access to it. The DURAFILE server will incorporate a firewall for allowing only authorized access. The architecture shows the 3 roles of the DURAFILE Platform: regular user, expert and the administrator. Every user will have a different GUI. Mock-ups will be shown in a document. The idea is to develop all the software using Open Source technologies. They are different issues that should be studied such as the connection with a database with an external provider or the use of SSL connections. These issues should be done with Open Source technologies if possible. In a similar manner as Dropbox does, when a user deletes its account all his data will be erased from the external database provider. Similar to Dropbox the system scans a directory for identifying obsolete formats. The system could be able to detect corrupt files. External providers such as the ones that provide curation will be included at the DURAFILE platform. The DURAFILE platform allows that the user rates the migration. This rating will be the basis for evaluating the different Digital Preservation Plans obtained for a certain migration. 59

60 15 Ability to embed a report for the files that the tool was not able to resolve by itself and the user is not competent to solve them himself. The report will be forward the problem description to expert users in order they find a solution. 16 Automatically creation of an obsolete formats list. 17 Define the cold start for the Social Search (a minimum set of relation between experts and a minimum set of Digital Preservation Plans). 18 The main language of the DURAFILE platform will be the English. 19 Check the availability of decoders for the Digital Preservation Plan execution. YES YES YES YES YES When there is not a suitable Digital Preservation Plan at the DURAFILE database for a certain migration, the system will asks to the user if he wants to share its data with the experts. A report will be generated automatically including the file that will be distributed to the network of experts in order that they find a solution. The platform includes a Database with Digital Preservation Plans. From this database a list of obsolete file formats will be generated. This list will be checked when the platform will scan a user directory for detecting obsolete file formats. We need to define a cold start for the correct working of the Social Search. It will be defined in other documents. English will be the main language of the platform. All the interfaces will be done using configuration files. That will allow that the menus can be displayed in other languages than English. The DURAFILE server needs to have the suitable decoders for the migration of the obsolete files. The script for the Digital Preservation plan will be check that the decoders are present at the machine. 4.3 Digital Preservation Plan Implementation For DURAFILE digital preservation plan specification implementation, project PLATO XML schema can be used and modified: For DURAFILE Execution plan/action plan one possibility could be to use project PLATO schema: or we can develop our own schema. It applies the preservation action to an object, performs QA and provides measures according to the quality level specifications. Most other digital preservation projects are using service-based solutions. But it would be unwise to eliminate all other possibilities. Especially when we aim for a standalone version which also needs to work without internet (i.e. web-services). 60

61 Connecting DDP to DURAFILE recipe server HTTP POST requests can be used. Action plan will be executed via built in scripts (platform specific language) or via workflow execution engine (for example Taverna). 1) Doing it with scripts or some built-in language execution plan can be a separate script file linked within preservation plan (URL) or specified with specific tags in preservation plan then script knows how to execute it. 2) By using workflow execution engine (depending on engine) execution plan needs to have separate xml file with correct schema (for example PLATO schema). Actions will be marked with URL-s to jobs (in DURAFILE server) or scripts (if supported, standalone?). Automatic preservation plan creation can be possible however minimal user input is needed. System will have built-in general selectable questions which user will select and system will make analysis (depending on constraints set behind questions selected) and will generate basic and simplistic DURAFILE preservation plan. DURAFILE DDP creation and execution workflow may look like this: Creating plan: 1) Plan will be created by using XML schema specification (for example PLATO). 2) Plan will be sent to recipe server via HTTP POST requests or API. 3) Plan will be validated a. If correct, status will be changed to valid. b. If not correct, status will be changed to invalid. 4) Execution plan will be created based on information provided, agent search and feedback. Executing Plan: 1) Plan will be requested from recipe server by using HTTP POST requests or API. 2) Preservation file(s) will be determined. 3) Action plan will be executed via workflow execution engine (API or service) or via script language. a. Needed actions (jobs/scripts or services) will be called via provided URL-s and correct order defined in schema will be used. 4) If no errors while processing files, file(s) status will be changed to OK. 4.4 Experts Social Search The social search in DURAFILE is composed by a network of expert users that collaborate with each other in order to create new DPPs in the Recipe Server, and also to create DPPs from a request of nonexistent DPP in the DURAFILE Server. The Social Search Architecture proposed for this project (Figure 33) has four different parts that have to be described: Agent s environment: Each expert user will has his agent. The agent will help expert user make easier the way of create DPPs and collaborate with the other experts in the platform. Contact List: The Agents will have a contact list, where they will look for the best expert to ask questions, or to decide which expert ask for help. 61

62 Q & A Database: The expert s users will have a knowledge database where the Agents will search into in order to find an answer of some question. Experts. They will be the core of the Social Search. If the agents don t find any solution, the experts will be responsible for finding a solution. Figure 33 - Social Search Schema 4.5 End-User Interface This section provides examples with Mock-ups for the Experts Users for the Desktop and Web interfaces Desktop app The desktop application will be similar as Dropbox, making the application work without make noise and showing notifications in the main bar without disturbing the user. An example is shown in Figure 34. Figure 34 - Installed client app The first time User open de desktop application, he will be able to sign up to the platform as a normal user, or as expert user. If he chooses the option of expert user, he can choose in which topics he is expert 62

63 Figure 35 - Forms for register both normal mode and expert mode On the other hand, if the user already has a DURAFILE account, he will be able to Login to the platform directly. Figure 36 - Desktop application login Once the User is logged to the platform, he can change his User preferences; even he can change from normal user to expert User. You also will be able to unbind your DURAFILE client. 63

64 Figure 37 - User settings Personal user account In Figure 38 the main window shown in when the user clicks in the DURAFILE icon would show the preservation level of all the files, a list of last preservation actions that have been done, a properties button (1), and the option to access to a list of local files(2) or to the DURAFILE server (3). Figure 38 - Client app main panel When properties (1) button is clicked, will appear a properties window with different tabs, some of them similar to Dropbox: 64

65 Figure 39 - Client app configuration It can be interesting to add the option of Preservation with the right click over a file or a folder. There will not be a central folder with all preserved files; files will be in different locations. Once a file or a folder is preserved the icon of that file or folder will change as shown in Figure 39 65

66 Figure 40 - Preservation action menu Expert user account The main window when the user clicks in the DURAFILE icon would show also the reputation of the user as an expert, the list of the questions and answers in which the user is implicated, and the requests for new DPPs. There will be a button to generate a new question (5), a button to answer questions (6) and a button to submit a new DPP (7). (5) and (6) will open desktop windows and (7) the web application. Figure 41 - Experts client app When is clicked in order to do a new question (5), it should appear a simple window where the expert has to choose the subject that is referred and an input text to introduce the question. 66

67 Figure 42 - Simple window to make a question When is clicked in order to answer a question (6) it should appear a simple window with the question, two checkbox a text input to answer the question if yes. Figure 43 - Simple window to answer a question 67

68 4.5.2 Web application The DURAFILE platform also includes Web application for the personals and expert users Personal user account Figure 44 show that when the user access to the DURAFILE server (3) the browser will open the following window. Figure 44 - Files web interface When user clicks in +info, will appear a webpage with all the information related to a file (preservation level, metadata, the preservation plans applied to that file, log of actions applied to the file and the option to download the file at any time of the transformations underwent) as shown in Figure

69 Expert user account Figure 45 - Detailed information using the DURAFILE webapp The expert users, could also submit preservation plans using the DURAFILE experts platform. When is clicked in order to and a new DPP plan (Desktop application 7) it should open the web application form to submit the new preservation plan, as Plato tool do. Figure 46: Form to submit a new Digital Preservation Plan 69