Guidelines for Creating and Installing IF Preservation Workflows and Templates

Transcription

1 Prject Number Prject Title Title f Deliverable Deliverable Number Cntributing Sub-prject and Wrk-package Deliverable Disseminatin Level Deliverable Nature IST Planets Guidelines fr Creating and Installing IF Preservatin Wrkflws and Templates IF5-D1 IF5 External Public Reprt Cntractual Delivery Date 31 st May 2009 Actual Delivery Date 19 th June 2009 Authr(s) ARC Abstract The dcument describes the API and functinality f the Planets wrkflw executin engine (WEE) versin 2. We prvide an verview f the main cmpnents, the public interfaces, and describe hw it is utilized thrugh the Wrkflw Cntrl Panel (WCP). The dcument specifically prvides user dcumentatin fr cnfiguring, deplying, and accessing Planets wrkflw templates. Keywrd list: Interperability Framewrk, Wrkflw Executin Engine (WEE), Template Repsitry, Wrkflw Templates, Wrkflw Cntrl Panel (WCP)

2 Cntributrs Persn Rle Partner Cntributin Rss King IF SP Lead ARC Wrkflw Cntrl Panel reprt Andrew Lindley IF/TB Develper ARC Wrkflw Executin Engine reprt Rainer Schmidt IF5, IF7 Lead ARC Editr Dcument Apprval Persn Rle Partner Rss King IF SP Lead ARC Christen Hedegaard PA/4 WP Lead KB-DK Distributin Persn Rle Partner Revisin Histry Issue Authr Date Descriptin 0.1 RS 12/06/2009 Initial draft versin fr internal review 0.2 RK 14/06/2009 Crrectins frm internal review 0.3 CH 19/06/2009 External Review 1.0 RK 19/06/2009 Final versin References Ref. Dcument Date Details and Versin Page 2 f 17

3 TABLE OF CONTENTS 1. Intrductin Main Cmpnents and Services Definitins Wrkflw Executin Engine Utilizing the Wrkflw Envirnment Dwnlading and Installatin The Wrkflw Cntrl Panel (WCP) Editing and Executing Wrkflw Templates Wrkflw Cnfiguratin Schema Implementatin Details and Public Interfaces...14 Appendix A...17 A.1 Service Registry Brwser...17 Page 3 f 17

4 1. Intrductin The dcument prvides an verview f the Planets wrkflw envirnment versin 2, as well as a user guide t creating and utilizing custm wrkflw templates. The emplyment f a suitable wrkflw rchestratin system impses a crucial aspect fr the Planets envirnment. Preservatin experiments are cmprised f a number f well defined executin steps, which have t be executed within a service-riented distributed envirnment. Such preservatin prcesses invlve cmplex cntrl lgic as well as web service interactins and data mdel manipulatins. A preservatin wrkflw cnsists f a sequence f activities, carried ut in a specific rder, in which the utput parameters f ne actin are validly mapped t the input parameters f the fllwing actin. An example f a Planets wrkflw wuld be: fr a given file, first identify a file frmat, then validate the frmat, then characterize it, then migrate it t a new frmat, then characterize the new file, and finally cmpare with the riginal. Initially, we examined the applicatin f the Web Service Business Prcess Executin Language (WS-BPEL) and implemented experimental preservatin wrkflws based n WS- BPEL. This apprach prvided a large degree f flexibility and allwed the specificatin f preservatin prcesses at a lw (i.e. messaging) level using Web service standard languages like BPEL, XML, XPath, and WSDL. Hwever, wrk in this directin was hindered by tw difficulties; first, that the BPEL language is very pwerful but als lw-level and hence very cmplex; and secnd, at the time we cnducted the experiments, BPEL related-tls prved t be nt yet mature. Bth pints turned ut t be a majr hindrance fr implementing preservatin wrkflws by ur targeted user grup (preservatin experts, archivists, data curatrs). Cnsequently, we chse t implement a much simplified, custm wrkflw descriptin language and crrespnding executin engine. It is a crucial requirement f the prgramming language and executin envirnment that they allw the targeted cmmunity t assemble and deply the preservatin wrkflws they require, withut frcing them t understand the underlying system. The WEE versin 2 emplys the cncept f wrkflw templates which are pre-defined wrkflw definitins that can be easily instantiated and executed thrugh a set f Web service interfaces. Page 4 f 17

5 2. Main Cmpnents and Services 2.1 Definitins Wrkflw A Planets preservatin wrkflw defines a sequence f invcatins f Planets Services, in which the utput parameters f a given service are validly mapped t the input parameters f the fllwing service. A Planets service is a sftware cmpnent that implements ne f the specified Planets service interfaces (like Identify, Mdify r Migrate). Wrkflw definitins cntain activities (service invcatins), as well as decisin lgic, lps, cnditins and ther cntrl structures. Wrkflw Template A wrkflw template is a pre-defined wrkflw definitin in which the ndes f the preservatin sequence are service placehlders rather than cncrete service endpint. A service placehlder is a typed service which nly defines the abstract interface while the actual implementatin is undefined. The pint f such a template is that a nrmal end user can select any service placehlder and replace it by a real service that implements the defined interface. This apprach allws nrmal users t create Planets wrkflws withut being frced t deal with the full cmplexity f the underlying system. Wrkflw Cnfiguratin A wrkflw cnfiguratin identifies a wrkflw template, the service endpints assciated with all template placehlders, and the parameters assciated with each service. Hence, a Planets wrkflw is defined by a wrkflw cnfiguratin and its assciated template implementatin. 2.2 Wrkflw Executin Engine The wrkflw executin engine (WEE) currently prvides tw main functinalities t a user: A web service fr brwsing the wrkflw template repsitry and A service fr the executin and mnitring f wrkflw instances. Wrkflw templates are pre-defined wrkflw fragments that must be cmpleted by the user fr example by specifying a particular Web service endpint and/r service parameters. Wrkflw templates are being defined by experts and made available t Planets users thrugh imprt int a Template Registry. The WrkflwTemplateRegistry prvides service interfaces t register, brwse, and retrieve wrkflw template definitins. In rder t schedule a wrkflw executin, a user submits a descriptr dcument as well as a pinter t the data registry t the WrkflwExectuinManager. A descriptr basically cntains the identifier f a template and defines its parameterizatin (sectin 4). The main purpse f the WEE is the prvisin f a cntrlled envirnment fr the specificatin and executin f preservatin prcesses. Planets preservatin wrkflws are built frm Java cmpnents and serialized as XML wrkflw dcuments. Fr this purpse, the wrkflw engine implements an enactr that gverns the rchestratin f the varius preservatin cmpnents, which encapsulate functinalities like cmmunicatin, state management, and preservatin metadata Page 5 f 17

6 handling. Preservatin cmpnents can be implemented as lcal desktp cmpnents (e.g. fr data mdel manipulatins) r can abstract remte applicatins and services. The Wrkflw Cntrl Panel (WCP) cmplements the wrkflw envirnment by prviding a generic prtal client. The WCP prvides a graphical user interface that allws ne t chse frm varius abstract wrkflw scenaris (templates). The wrkflw templates are then rendered and visualized. Selected wrkflw templates are cnfigured using the GUI representatin (e.g. drp-dwn bxes) and can be executed and mnitred using the Web applicatin. Additinally, users can uplad/generate an XML representatin f the actual wrkflw instance. Frm the user s pint f view, the wrkflw executin envirnment (WEE) prvides high-level interfaces fr discvering, cnfiguring, and instantiating preservatin wrkflws. The WEE perfrms wrkflw executin asynchrnusly and may deliver status infrmatin t the user based n inquiry and ntificatin capabilities. It currently prvides a generic web brwser client r can be accessed by end user applicatins using web service r native interfaces. Fr future versins, we plan t incrprate a graphical wrkflw editr that supprts ur Javabased cmpnent mdel. Page 6 f 17

7 3. Utilizing the Wrkflw Envirnment 3.1 Dwnlading and Installatin Cde Repsitry and Installer The Planets wrkflw envirnment is cntained within the interperability framewrk surce prject. The related installatin package and API can be dwnladed frm the Planets IF GFrge prject here: The Planets sftware prttype cde can be brwsed thugh the Planets IF GFrge prject here: Prerequisites This guideline demnstrates the Planets wrkflw executing services based n a simple example. The required template implementatin and xml cnfiguratin is packaged with IF release versin 4. Prerequisites fr installing and running the example are: - The J2SE 1.5 Java Runtime Envirnment (JRE) - A running installatin f the Planets interperability framewrk, versin 4 Please cnsult the INSTALL_IF.txt file if built frm surce r simply use the installer prvided n the GFrge page - A set f installed and registered Planets services (sectin A.1) required service fr the example are the DROID Identificatin Service ImageMagickMigrate SanselanIdentify Service Pure Java services are packaged with the IF installer, fr services that require additinal sftware installed (e.g. ImageMagick) please cnsult the dcumentatin f the Planets services prject (Pserv). Fr a detailed descriptin n hw t install the Planets Interperability Framewrk and Preservatin Services please refer t the latest release dcumentatin. Fr dcumentatin n hw t implement wrkflw templates please cnsult the develper and surce dcumentatin available at the Planets IF cde repsitry. Page 7 f 17

8 3.2 The Wrkflw Cntrl Panel (WCP) The Planets wrkflw envirnment prvides a set Web services (Wrkflw Executin and Mnitring) described later in this dcument. This Web service API allws Planets applicatins t remtely create, execute, and mnitr cmplex preservatin prcesses based n wrkflw dcuments. The Wrkflw Cntrl Panel (WCP) prvides a generic graphical user interface fr the infrmal utilizatin f the wrkflw envirnment. Other applicatins that need t interface with the WEE are the Preservatin Planning and the Testbed Applicatin. Planets Service Architecture Page 8 f 17

9 3.3 Editing and Executing Wrkflw Templates In this sectin we prvide a walkthrugh fr the Planets Wrkflw Cntrl Panel based n a simple example wrkflw. The wrkflw first identifies the filetype f a digital bject, migrates it t the Prtable Netwrk Graphics (PNG) frmat, and again runs anther identificatin using a secnd identificatin tls. The example wrkflw template implementatin as well as the cnfiguratin xml file is prvided with the Planets surce distributin. There are five tabbed panes n the main WCP page, with the fllwing functinalities: 1) Selecting Wrkflw Template This tab prvides a list f registered wrkflw templates, as well as an ptin fr uplading new template surce files (if authrized). Clicking n ne f the registered templates will select that template as the Active Template (Nte: this will erase any wrkflw previusly held in memry) and direct the user t the Edit Wrkflw tab (the active wrkflw will be precnfigured fr the selected template). The cmpnent des carry ut a rugh validatin (it checks whether the prvided template implements the crrect interface) but des nt precmpile the template. 2) Lading a Wrkflw Cnfiguratin This tab allws the user t uplad an XML wrkflw descriptin file (e.g. an executable preservatin plan prduced by the PLATO applicatin). The wrkflw must validate against the planets_wtd.xsd schema in rder t be successfully laded. Furthermre, the template referenced by the XML wrkflw descriptin <class> element must be laded in the Template Registry. If these tw cnditins are met, the wrkflw will be successfully laded, the referenced wrkflw template will be autmatically set as the Active Template (this step will be reprted in the Errr Message bx, althugh it is nt an errr per se), and the laded wrkflw descriptin will be set as the Active Wrkflw. Nte that the Active Wrkflw will be Page 9 f 17

10 tagged with the label (frm File) as lng as the laded descriptin remains unmdified. Mre detailed infrmatin n the wrkflw cnfiguratin schema is prvided in sectin 4. 3) Editing Wrkflw and Exprting the Cnfiguratin File In rder t use this tab, a wrkflw template must be selected, either by selecting a template n the Manage Templates tab, r by lading a valid wrkflw descriptin frm the Lad Wrkflw Tab. The user can then select service endpints fr each Service ID in the wrkflw frm a drp-dwn list that is ppulated by a Service Registry query fr all registered services f the defined Type. The user can als edit the name-value pairs that define the input parameters fr a given service (the default parameters will be laded frm the Service Registry when an endpint is selected) refer t the Wrkflw Parameters Panel descriptin belw. Nte that as sn as a mdificatin t a laded wrkflw is made, the label (in memry) will be attached t the Active Wrkflw name. The user can chse t save the present wrkflw descriptin as an XML file t the lcal file system by clicking n the Exprt XML buttn. The user can als chse t clear the wrkflw (deselect all service endpints, remve all parameters) by clicking n the Reset buttn. Page 10 f 17

11 4) Selecting Digital Objects t Prcess Here the user can brwse a hierarchical display f data surces; presently a file-system data surce and a Data Manager fr the Planets Amazn Simple Strage Service (S3) accunt are pre-cnfigured. The user can chse t add selected Digital Objects t the Included Objects list, r can clear this list. If al required prerequisites are met, the user can submit a wrkflw t the jb queue. 5) Checking the Wrkflw Executin Status A wrkflw can be submitted when an Active Template is defined, when an Active Wrkflw is defined, and when Input Objects Selected is true. When a wrkflw is submitted, the user is autmatically directed t the Wrkflw Status tab. This tab presents a table f infrmatin abut wrkflws that have been submitted. At present, this infrmatin is nt persistent; that is, nly infrmatin abut wrkflws submitted during the current user sessin is displayed. Page 11 f 17

12 Wrkflw Parameters Panel A list f parameters assciated with a given service placehlder are shwn in a ppup windw titled Wrkflw Parameters Panel. The header f the main panel has the title Edit Parameters fr <name>, where <name> is the service placehlder (r Service ID) in questin. If the user wishes t remve an existing parameter, she shuld click the Delete buttn assciated with that parameter. If the user wishes t update an existing parameter, she shuld enter the new parameter value in the New Parameter Value text field, and then click n the Update buttn assciated with the desired parameter. If the user wishes t define a new parameter, she shuld enter a Name and Value in the New Parameter text fields and then click n the Add buttn. When the user is finished mdifying the parameters, she shuld click n the Dne buttn; the windw will clse and the fcus will return t the riginal Edit Wrkflw panel. Page 12 f 17

13 4. Wrkflw Cnfiguratin Schema This sectin describes the basic elements f the wrkflw cnfiguratin schema. The schema defines the XML representatin f the particular wrkflw cnfiguratin which is used t instantiate a wrkflw template. Wrkflws cnfiguratins can be created and serialized (as XML files) using the WCP r by any ther tl that supprts the cnfiguratin schema. The latest release f the schema definitin can be fund at the Planets surce repsitry (planets_wdt.xsd). The figure belw demnstrates a simple wrkflw cnfiguratin specifying a wrkflw template, a set f services, as well as service parameters. Fr mre detailed infrmatin the reader is referred t the schema definitin. Page 13 f 17

14 5. Implementatin Details and Public Interfaces The main elements f the Wrkflw Executin Engine package are Tw web-services / stateless-sessin-beans which prvide the applicatin's interface WrkflwTemplateRegistry: cntains methds fr brwsing, registering and retrieving wrkflw Templates WrkflwExecutinManager: cntains methds fr submitting a wrkflw and plling fr the executin's status and result A message-driven-bean: the WrkflwExecutinEngine - the jb-prcessr WrkflwTemplateRegistry The WrkflwTemplateRegistry prvides an implementatin fr brwsing, retrieving and submitting WrkflwTemplates. The fllwing methds are prvided: getallsupprtedqnames: Returns a list f all QNames f registered WFTemplates getwftemplate(string QWrkflwTemplateName : Returns the surce (Java Wrkflw Template.java file) fr a registered template. This might be necessary t understand a template's behaviur. registerwrkflwtemplate(byte[] javabinary): Registers a new WrkflwTemplate The parameter QWrkflwTemplateName is the fully qualified name f the submitted class. e.g. 'eu.planet_prject.ifr.cre.template1.java'. The parameter javabinary is a byte array f the java class. The template will be added t the template registry and persisted using the Planets data registry. WrkflwExecutinManager The WrkflwExecutinManager is a web service / stateless-sessin bean interface that ffers the fllwing methd: public UUID submitwrkflw(list digitalobjects, String QWrkflwTemplateName, String xmlwrkflwcnfig) DigitalObjects is a list f Planets Digital Objects, which prvide the paylad a wrkflw is invked upn; QWrkflwTemplateName the fully qualified name f the java wrkflwtemplate t use fr this submissin. i.e. specifying the structure. (Please nte: the template must be registered!); xmlwrkflwcnfig is an XML String that defines the wrkflwtemplate's cnfiguratin i.e. specifying hw t ppulate a static template. The return value is a ticket (UUID) fr the submitted jb which can be used fr plling infrmatin abut the wrkflw: querying n a jb's status and its psitin in queue retrieving executin results Page 14 f 17

15 WrkfwFactry The WrkflwFactry class cntains a single public Methd: public static WrkflwInstance create(wrkflwcnf wfcnf, List<DigitalObject> digs) thrws Exceptin The class implements a builder that extracts the infrmatin handed ver in the xml wrkflw cnfiguratin (wrkflwtemplate's QName t use, serviceids, endpints, their parameters, etc.) queries the registry and fetches the requested wrkflwtemplate.java, fllwed by dynamically validating, cmpiling, jar-ing and classlading the required wrkflwtemplate reflects n the template's fields and builds and initializes prxies fr the passed service endpints and stres the service parameter cnfiguratin includes the data t invke the wrkflw upn (in terms f DigitalObjects) and returns a WrkflwInstance Object which can be submitted t the queue and executed upn the WrkflwExecutinEngine. WrkflwTemplate A wrkflwtemplate is a java class that defines the service-interface-types which are used (e.g. Identificatin r Migratin Interface) but nt the actual service-instances (e.g. Drid r ImageMagick) the structure and prcess flw within a wrkflw (e.g. branching, lping, exceptin handling, flw f level-ne-service input and utput data, etc.) results within the data registry and dcuments all actins in the wrkflwresult bject which is returned A wrkflw template may cntain any service that implements the Planets Service Interface. In rder t prvide a valid implementatin ne needs t be aware f the fllwing cnventins: Naming cnventin: The id used within the wrkflwcnfig.xml must crrespnds t the bject's variable name within the template class, e.g. <service id="identify1"> -> private Identify identify1; The WrkflwFactry validates if the ServiceTemplate interface is implemented. The WrkflwFactry checks fr all defined Service interfaces which are used t cnstruct the business lgic f a wrkflw if they are in the range f supprted PlanetsServiceTypes Currently supprted service types are: eu.planets_prject.services.identify.identify eu.planets_prject.services.characterise.characterise eu.planets_prject.services.characterise.determineprperties eu.planets_prject.services.cmpare.basiccmparefrmatprperties eu.planets_prject.services.migrate.migrate eu.planets_prject.services.migrate.migrateasync" Page 15 f 17

16 Planets Preservatin Wrkflw Sequence Diagram A shrt example using a sequence diagram n registering a wrkflwtemplate and n running a Wrkflw Executin is shwn in the figure belw: Wrkflw Executin Sequence Diagram Page 16 f 17

17 A.1 Service Registry Brwser The Service Registry Brwser prvides a user interface fr registering lcal and remte service endpints with the service registry, which generates an instance f a rich service descriptr fr each preservatin service endpint. This infrmatin is generated nce the service is registered with the preservatin system and needs t be supprted by a service descriptin prt type. Page 17 f 17