Deliverable 7.3. Workflow Manager. Poznao Supercomputing and Networking Center

Transcription

1 Deliverable 7.3 Workflow Manager Poznao Supercomputing and Networking Center

2 Table of contents 1 Introduction E-VLBI Correlation Platform Liferay introduction NEXPReS model User Roles and Permission Experiments and Workflow portlets Telescope, Correlator, external tools Portlets Visualization of the experiment Correlation testbed How to conduct an experiment short user manual System log-in Experiments page (and portal navigating) Experiment creation Experiment s workflow Experiment submission Software Table of figures Figure 1. Fabric Workflow Manager Application... 3 Figure 2. NEXPReS module architecture... 5 Figure 3. Example of correlation's graph... 7 Figure 4. WP7 testbed based on partner's resources... 8 Figure 5. Sign in portlet... 9 Figure 6. Welcome screen... 9 Figure 7. Experiments page Figure 8. Creating new experiment Figure 9. Workflows portlet D7.3 Workflow Manager Page2

3 1 Introduction This document describes e-vlbi Platform software developed within WP7 Correlation in a shared infrastructure. It contains a short overview of previous approach prepared in EXPReS project, presents a new web based perspective for data correlation, provides description of the current testbed and finally describes the content of the software components. 2 E-VLBI Correlation Platform The Workflow Manager Application (WFM) has been constructed in EXPReS project within Fabric activity. The idea was to create a prototype system for software correlation. The WFM was a main interface between users and the system and has been constructed as a standalone Java window-based application. Figure 1. Fabric Workflow Manager Application User can create so called observation workflows based on an observation schedule. The application allows also defining correlation parameters which controls the behaviour of the software correlator. The completed observation workflow is then submitted for further processing. The role of WFM application within NEXPReS project has been significantly extended. The WFM has been transformed from a single user application to a multiuser and multipurpose application for all VLBI actors. The application became a central place not only to manage and control a distributed correlation platform, but also to monitor the entire experiment process. PSNC has created an e-vlbi Correlation Platform which will be used within VLBI community to manage and monitor experiments. The WFM application has been rewritten using web based technologies and nested in the Correlation Platform as a portlet. 2.1 Liferay introduction Experiences from the EXPReS Fabric project lead to a new approach for handling e- VLBI experiments. Instead of a single point of entry for Principal Investigator, allowing only submission of the correlation using specified resources, focus was put to facilitate whole process of setting up the observation, execution, monitoring and validating correlation process. A need for a multi-user program arose. It should D7.3 Workflow Manager Page3

4 provide various, pluggable tools and allow easy development, integration and deployment of new applications. A research was done about available solutions (WP7 State of the Art on Web Portals and Web Technologies: and Liferay Portal ( was chosen as a platform for NEXPReS project. Liferay Portal is an enterprise web platform for building business solutions that delivers immediate results and long-term value (from Liferay website). It contains out-of-the-box set of portlets providing basic needs for a web platform user management, login functionality, alerts and announcements, collaboration tools such as chats, documents and so on. It is also a typical portlet container, which allows writing own applications and deploying them inside the portal. Deployed applications can be added to portal pages as portlets modular, pluggable user interface components. Administrators can prepare and set up which sets of portlets will be visible on each page of the portal. A great deal of influence on deciding on portal solution was collaboration and multiuser approach. Using provided user management tools we can model each type of actor as a specific user role, create communities, organizations and user groups. For each such unit a specific page or view can be created, containing only needed applications. This allows to create a dedicated page layouts for different user types. Cooperation between scientists during observation is fulfilled by adaptation of the Liferay portlets, which provide a chat functionality, mail (also connected with external mail services) and message boards. System generated information is also displayed to interested parties using alerts and announcements portlet. Crucial requirement of the Workflow Manager in Liferay portal was intuitive and responsive user interface. From technologies supported by the portal, JSF (Java Server Faces - and ICEFaces ( were chosen. JSF is a Web application framework and simplifies building user interfaces. It supports Model-View-Controller architecture pattern and is written in Java. ICEFaces on the other hand extends JSF framework with Ajax support, which allows development of rich internet applications. Asynchronous updates from server to client is provided by Automatic Ajax and Ajax Push functionalities. This allows to update data in the user s browser without reloading whole page, depending on the state of the model on the server. Objects during e-vlbi observation will be constantly changing, updating the statuses of the components, executing steps of the correlation process. This changes will be immediately displayed to all interested users in their Liferay portal instance. D7.3 Workflow Manager Page4

5 2.2 NEXPReS model Figure 2. NEXPReS module architecture User Roles and Permission During design of the e-vlbi Platform we had to take into account the structure and functionalities of the Liferay Portal. One of such feature is user support. After evaluating NEXPReS use cases we have decided to create 5 NEXPReS roles: PI Principal Investigator SS Support Scientist TO Telescope Operator CO Correlator Operator Admin e-vlbi Platform Administrator Each of the four user types (beside Admin) has an unique perspective in the portal providing default view, customized for this particular role, containing all portlets required by the user. User roles also allow to enforce permissions on whole e-vlbi Platform e.g. general information about NEXPReS displayed for not logged or unregistered visitors, displaying monitoring information of the correlation process to logged, non-correlator operators and finally providing interactive controls for correlator operators Experiments and Workflow portlets In e-vlbi Platform information about experiments (VLBI observations) will be displayed using two portlets: Experiments and Workflow. Experiments portlet displays all existing observations in the system. It also allows to create new experiment, by uploading VEX file describing e-vlbi observation and specifying additional D7.3 Workflow Manager Page5

6 information. General information of the experiment (Status, time to start) is being displayed to the user. By selecting the experiment more information about the experiment is available. Most important is graphical representation of the experiment in Workflow portlet (see Figure 9. Workflows portlet). Interaction with the experiment is possible using this view: before experiment starts it allows modifications of the whole process and during the observation it provides monitoring information about experiment Telescope, Correlator, external tools Portlets Pluggable architecture of e-vlbi Platform allows adding new modules, providing new functionalities to whole application. Some of the distinguished portlets, which will be needed during VLBI observation are: Telescope Portlet - allowing Telescope Operators to operate and communicate with their telescope module, others can monitor progress and status of telescope operations during observation. Correlator Portlet - allowing Correlator Operators set up of the correlation process, specifying parameters of the correlation. It also allows monitoring of the correlation process to all interested parties. Fringe Plots graphical module used to display fringe plots required to update and adjust telescopes and correlation parameters 2.3 Visualization of the experiment Initial experiment upload to Workflow Manager contains only a VEX file and author s information. Basing on this file the correlation representation is created by Workflow Manager both on server side and in user tools, to visualise the flow of data and resources taking part in e-vlbi experiment. On client side the convenient Workflows portlet was developed. It comprises both Java portlet and Adobe Flash technologies, thereby addressing standards support (Java portlet can be deployed in any portlet s container) as well as platforminteroperation (embedding binary Flash content). Portlet embedded in Liferay portal uses JSF (Java Server Pages) GUI libraries Java-based Web application framework intended to simplify development integration of web-based user interfaces (source: Wikipedia). JSF facilitates mapping from/to web application to/from Java model classes on server and extended with AJAX libraries (e.g. IceFaces) results with responsive and intuitive user interfaces. On user selection in web browser, the VLBI experiment s representation is rendered inside Workflows portlet. This portlet is intended as a tool for graphical design of e-vlbi workflow a graph representing data flow within telescopes, translation nodes and correlators during correlation process. e-vlbi workflow visualises resources taking part in experiment (see Figure 3. Example of correlation's graph) and - by connecting them with each other user defines destination/sources of data for services carrying out the correlation. Changes made in correlation s workflow are reflected in model classes and saved on server side. Designed graph can be edited and parameters of experiment can be changed. When correlation time is met D7.3 Workflow Manager Page6

7 workflow s modifications are no longer possible, and the experiment is started on infrastructure. Figure 3. Example of correlation's graph Representation of experiment s workflow is rendered in Workflows portlet using Flex/ActionScript technologies, a web-enabled toolkit supporting visualisation of graph-like data. To enable data synchronisation between server-side model and Flash client application an Adobe s BalzeDS messaging technology was used. BlazeDS allows for RPC-based communication from ActionScript clients to server as well as for PUSH-like transfer of data from server to clients. The data sent is in AMF format a binary protocol enhancing light and fast communication. On the server side experiment is passed to Experiment Manager Unit (EMU) which ensures coherence between correlation definition and infrastructure resources. EMU schedules experiments to start automatically the proper model objects of workflow are created, passed to workflow management module. The correlation is started by invoking web services of Fabric system. D7.3 Workflow Manager Page7

8 3 Correlation testbed Resources assigned by the project partners to the NEXPReS project have been set up to form a testbed for e-vlbi Platform developed by PSNC within work package. The testbed consists of (see also Figure 4. WP7 testbed based on partner's resourcesfigure 4): computational resources which will be used during correlation process, translation. These have been provided by PSNC, Jive and Ventspils. Translation services which serve the telescope data Correlation services which handle correlation process VLBI Broker deployed at PSNC to handle experiments E-VLBI Platform replacement of Fabric Workflow Manager Figure 4. WP7 testbed based on partner's resources A current version of e-vlbi Platform has been deployed on local installation at PSNC. This local installation is available to all project partners. The idea is to keep this software operable till the end of the project duration. This site will be updated whenever a new version of the e-vlbi Platform is available. This will allow us to conduct experiments at any time and also use this installation for demonstration purposes. D7.3 Workflow Manager Page8

9 4 How to conduct an experiment short user manual This section describes actions needed to be taken by user in order to create experiment and submit it for correlation. The process is taken from user point of view and therefore presented in form of short manual illustrated with live e-vlbi Platform system screenshots. The general scenario when running a correlation with e-vlbi Platform is to log into portal, navigate to page containing experiment related-portlets (management and workflows) where new experiment can be created and its workflow designed. The required user interaction ends at this point, as Experiment Manager Unit (EMU) running on server side takes responsibility for running and monitor correlation process. Steps needed to conduct experiment are described in more detail below. 4.1 System log-in In web browser (preferably Mozilla Firefox 9.0+ or Google Chrome 14.0+) navigate to e-vlbi Platform main page Click Log in link in upper right corner Figure 5. Sign in portlet Provide your user credentials in the form that will be displayed (to obtain user and password contact your e-vlbi Platform administrator) On successful log in you will be redirected to portal s welcome site displaying general information. Figure 6. Welcome screen D7.3 Workflow Manager Page9

10 4.2 Experiments page (and portal navigating) Figure 7. Experiments page After successful log in you are located on e-vlbi Portal s main page. To get back there (in case of any navigation problems) click NEXPReS logo located in upper left page corner. Page on which experiment-related portlets are placed is called Experiments. It is listed among other sites on top horizontal bar. To navigate to this page just click on the page name. Other sites can be accessed the same way. The Experiments site contains 2 portlets: o Experiments main entry point to manage all your experiments allows for creation and selection or experiment to be edited in other portlets. For user s convenience view is divided into 2 tabs, grouping experiments by their status: Active the experiments that are to be started in the future (status SCHEDULED) and currently running (status RUNNING). Some of them may still require users interaction (e.g. approving telescopes configuration or correlation workflow customization) Archive on this tab user can access all experiment that are no longer running o Workflows this portal is intended to present correlation in graphical, user-friendly way in form of workflow s graph. The important thing to note is that workflows portal is not independent it is connected with Experiments portal. Content rendered in Workflows comes from experiment selected in Experiments window. D7.3 Workflow Manager Page10

11 4.3 Experiment creation Figure 8. Creating new experiment On Experiments site, in Experiments portlet press Create new button An experiment wizard form is displayed containing: o Name provide name under which experiment will be accessible in e- VLBI Platform, o Description provide a short description of the experiment, o VEX file upload of file containing observation definition, o Start date specify the time at which correlation process should be started. When all information in wizard are provided press OK. On successful experiment creation it will be displayed among other experiments in Active tab of Experiments portlet, with status set to SCHEDULED. D7.3 Workflow Manager Page11

12 4.4 Experiment s workflow Figure 9. Workflows portlet Select one of the experiments from list in Experiments portlet In Workflows portlet a visualization of correlation graph will be presented Detailed information on resources taking part in correlation process (telescopes, translation nodes and correlation nodes) can be obtained by selecting elements of correlation graph. Information will be presented in right panel on portlet, listing resource s name, abbreviation and description. 4.5 Experiment submission When creating an experiment (see 4.3) a start date of correlation has to be specified. Experiments due to start in future are listed on Active tab of Experiments portlet (for more details see 4.2), with status set to SCHEDULED. No user interaction is needed to start correlation process at time specified Experiment Manager Unit (EMU) monitors start times of experiments and takes responsibility for running it at given time. When experiment has been started by EMU, its status changes from SCHEDULED to RUNNING. D7.3 Workflow Manager Page12

13 5 Software The table below presents the content of the attached CD-ROM. This includes all client and server components of new e-vlbi Platform. LP COMPONENT / MODULE /OTHER SHORT DESCRIPTION nexpresworkflowflex kiwicommon misc nexpresmodel Correlation Node expresremote misc vlabcommon VLBI Broker nexpres-portlet Actionscript ActionScript/Flex project that builds graphical (Flash technology) part of Workflows management system in e-vlbi Platform. Displays infrastructure of the experiment (radio telescopes, translation nodes, correlation nodes) and properties of each node that takes a part in work. Common KIWI common library set of general purpose utility classes Main project containing all related libraries and Ant build script for e-vlbi Platform Model for e-vlbi Platform i.e. to describe VLBI experiment and to serve internal persistence for infrastructure resources description Fabric Manages all computations on the underlying high performance computers. It provides an abstraction layer for correlation computations, hence the WfM will be able to work with both different hardware infrastructures and different software correlator. Remote client stubs for communication purpose between modules Project containing all related libraries and Ant build script for Fabric components VLab common library set of general purpose utility classes Handles correlation process on the server side: talks to Translation Nodes and Correlation Nodes decides when to submit new portion of data for the correlation. Portlets Contains GUI portal parts all portlets composing NEXPReS e-vlbi Platform JSF resources and web toolkit libraries. This module covers also Flex resources and configuration files. D7.3 Workflow Manager Page13

14 1. 2. experimentmanager (EMU) Workflow Manager (WfM) Workflows Module responsible for management of experiments. All newly created experiment should be registered with EMU. It monitors for experiment-related events and notifies all interested parts. EMU is responsible for starting experiment at time given. WfM handles all actions related to correlation workflow translates data from experiment description into workflow format and after experiment start monitors workflow execution. The current versions of the software components is always available at workpackage SVN repository hosted by PSNC. The repository location is as follows: kiwi.man.poznan.pl/svn/nexpres. D7.3 Workflow Manager Page14