LEAD Information Model

Transcription

1 LEAD Information Model This document captures the information placement of the LEAD system. The information includes static configurations, input data files, as well as runtime states of a workflow. However this document does not address information collection via Message Bus, which is handled in a separate document. Information model of LEAD system can be categorized as follows. 1. System, Service and Application Configurations - (Service registry and Portal configurations) 2. Workflow descriptions (BPEL document, XBaya Workflow Graphs and images) 3. Data products & Metadata (User's experiments and data products) 4. Runtime information about invocation (Lead Context Header) 5. Use and Group Support 1. System, Service and Application Configurations This category handles the configurations of the workflow system, and there are two types of information under this category. 1. Deployment and resource description documents 2. Service and System configurations 1.1 Deployment and resource description documents These documents reside in Service Registry (Xregistry), and provide XML metadata about different resources of the LEAD system. Generic Service Factory, (Gfac) which create Service from command line applications, defines three deployment descriptors that explain Hosts, Application and Services. 1. Host Description Document - Explains a Host available to use by the workflow system. Host could be used to run services or applications and the document includes host specific information such as environment variables, temp directory, java installation, Gfac installation Application Description Document - Explains an installed application on a host and is always bound to a host. This description includes information like executable location, environment variables Service Description Document - Explains how an application should be mapped in to Web Service operations and how input and output parameters should be resolved. Service Registry includes Abstract and Concrete WSDL documents which describe the available Web services for workflow system. 1. Abstract WSDL is an abstract description of the service, which is a template that can be used to create service instances. (In Web Service terms, it is a WSDL that only define a PortType, which does not define a binding). GFac uses service description document to create Abstract WSDL and register the

2 Abstract WSDL in service registry. Users may use these abstract WSDLs to compose workflow documents. 2. Concrete WSDL is concrete realization of an abstract WSDL, which is bind to a specific host. Relationship between abstract and concrete WSDL is same as the relationship between Java class and a Java Object where Java Object is concrete realization of the class. Concrete WSDLs are also created by Gfac at runtime, when a service instance is created and they have only a soft lifetime. All above documents resides in Xregistry and Xregistry provides register, delete and find method for each document. Supported methods are given below. registerconcretewsdl(), getconcreatewsdl(), removeconcretewsdl(), getabstractwsdl() registerservicedesc(), removeservicedesc(),getservicedesc() registerhostdesc(),gethostdesc(),removehostdesc() registerappdesc(), getappdesc(), removeappdesc() findserviceinstance(), findservicedesc(), findappdesc(), findhosts(), app2hosts() For more details about Xregistry API, please refer to Xregistry Manual. 1.2 Service and System configurations Each service specific configurations are adjusted manually for optimal configurations and they reside in individual service deployments. However portal configuration can be thought as the system configuration because it specifies the location of each service. Lead Context header which is the runtime information model of the system is initiated using service locations in the portal configuration. 2. Workflow descriptions Lead information model consists of Workflows and Services which are abstract descriptions, and for each experiment both workflows and services are instantiated to yield workflow instances and service instances. Following diagram shows logical composition of different entities as seen by the user. User creates an experiment using portal and each experiment could have one or more workflow instances, which are instantiated from a workflow template.

3 Workflow descriptions are generated by XBaya workflow composer using Abstract WSDLs and they can be stored in MyLEAD Metadata catalog or GPEL workflow engine. These workflow descriptions include BPEL documents, workflow graph, images of the graph, and workflow is represented as a workflow template in GPEL engine. A workflow instances can be created using workflow template and those instances are managed by GPEL workflow engine. 3. Data products & Metadata LEAD handles data products, by keeping track of metadata of each data products, and usually actual data are kept in a repository. LEAD data products are categorized as public data, and private data where Public data is stored in Data Catalog and private data is stored in MyLead Metadata catalog. In both cases, Metadata about each file is represented using LEAD Metadata Schema, which is the Common Metadata model for LEAD. The Lead Metadata Schema is a profile of the FGDC schema, and details regarding the schema and an XML Bean implementation of the schema can be found at the following web site maintained by Yogesh Simmhan. Data Items (Files) are represented by physical locations or DataID by LEAD system, and typically one of the two forms (physical locations or DataID) will be passed around the system. If some user needs access to Actual Metadata, he may get them by querying MyLEAD metadata catalog. 3.1 Data Catalog LEAD system includes a Data Catalog, which has input data that can be used with underline meteorology applications. Those data products are of two types. Dynamic Data Products These are collected from sensors (e.g. Radar) deployed across the nation and Data catalog crawl this data every two hour interval (?). Sensors are added to Data catalog by adding a THREDDS catalog to data catalog. This registration enables a Crawler which would be invoked every two hours and Crawler will find Data Items, Generate LMS for data items, and index them. After every crawl, old data are discarded. This

4 dynamic data allows scientists to run simulations using most up-to-date sensor data Static Data Products Usually static data products are data that are generated by workflows. Those data products from workflows are registered with MyLEAD and in turn they get published to data catalog by MyLEAD. LEAD Metadata for those objects get created by GFac and MyLEAD pass those information to Data Catalog as arguments to operations registercollection(), registerdataproduct(). Users may search for data products by providing queries based on region or time. Data Catalog has a method named 'searchdataproduct()' which accepts a query as an argument. It is usually called by users' requests from geogui or by a simple client program for testing. 3.2 MyLead Metadata Catalog MyLEAD Server provides an OGSA-DAI based API for data search and access to Metadata while MyLEAD agent provides a more user friendly API and create Lead Metadata for files registered or discovered though notifications. With Most operations, MyLEAD agent just forwards the requests to MyLEAD Server. LEAD metadata items are created at different points in a workflow invocation. As a metadata catalog, the MyLEAD server stores metadata regarding a user's projects, experiments, and files. The hierarchy within MyLEAD is: Project, Experiment, Collection, and File with Project required to be the top level within a user's workspace. Actual files are represented by File metadata objects, and files can be aggregated into a hierarchy of collections. Each object has one parent, and any object can be a direct child of a higher level in the hierarchy (e.g., a file could be a direct child of an experiment without an intervening collection object). For each of these levels, metadata is stored using the LEAD Metadata Schema (LMS). To access MyLEAD server there is a Java client API. Detailed documentation of this client API can be found in the MyLEAD Developer's Guide. When new objects are added to a user's workspace in MyLEAD using the Create method from the API, one of the parameters is a String with an LMS document representing the metadata for that object. Often the initial addition is only the minimal metadata and there is a utility developed by Marcus Christie that can be used to generate an LMS XML bean with the minimal metadata. The jar file for that utility is available at the LMS web site listed above. Metadata can then be added to existing documents by using the addattribute() method and updated using the updateattribute() method. Both methods take a string parameter that represents one of the nodes in the LMS that has been identified as a metadata attribute. Users can directly generate LEAD metadata and register them or LEAD metadata will be generated by MyLEAD Agent when it detects data files from notifications. New metadata elements are added to MyLEAD via one of following cases. 1. Users can generate Metadata for a document and publish it with MyLEAD; we provide a utility called minimum metadata utility to facilitate creation of Metadata. If users use registerfile() or bulkregisterfile(), the files are first moved to MyLEAD Repository and then Metadata are registered with MyLEAD. On the other hand if registerfilemetadata() or builkregisterfilemetadata() operations are used, files will not moved to MyLEAD repository, rather just Metadata is registered with MyLEAD. 2. MyLEAD agent listen to the notification created by each workflow invocation and at the end of the

5 workflow it create a file for that notification, generate metadata for that file and register it with MyLEAD. 3. When a Service finishes, Gfac registers all created data files, standard output and standard error files created by application invocation with MyLEAD. This registration creates Metadata descriptions the files and registers them with MyLEAD. 4. While creating a new Project or a experiment, they are added to MyLEAD as Lead Metadata Items, this is done via createproject() and createexperiment() methods of MyLEAD Agent 5. When invoking a workflow search for Data files, Portal copies the LEAD metadata items from Data Catalog to MyLEAD. Once the Metadata is stored with MyLEAD, Metadata can be edited via following methods. addattributes() -- adds a new LMS attribute to an existing object (project, experiment, collection, or file) updateattributes() -- updates the value to an LMS attribute that is already in workspace Also MyLEAD agent provide following methods to query Metadata. querylead() -- sends a simple query to mylead server, without context-query capabilities, and returns the results of the query. queryleadwithcontextquery() -- sends a context query to mylead server, and returns the results of the context query queryworkspace() -- sends a query for everything about an object plus its children, and returns the results of the query querybyid() -- sends a query for LMS for an object and returns the LMS querybyids() -- sends a query for LMS of a bunch of objects and returns the LMS queryworkspacestructure() -- sends a query for a skeleton structure of the workspace, only guides and titles are returned. Also there is a service called MyLEAD publisher that allows users to publish their private data as public in Data Catalog. Users can browse those files in there MyLEAD workspace and publish them using a publish link. Once the Publisher is called, it will copy the data from private MyLEAD repository to public Data Catalog repository and copy Metadata from MyLEAD to Data Catalog. For more information about MyLEAD APIs please refer to MyLEAD Agent developer Guide. 4. Runtime information about invocation On this section we discuss information flow took place at a Workflow invocation. We use a SOAP Header called LEAD context header as the container of runtime invocation. LEAD context header is generated by Portal while invoking a workflow and that header is copied to every message with in that workflow invocation. Lead Context Header provides three types of information. 1. It contains correlation IDs, which correlates all the messages of a given Workflow invocation. For an example, Using experiment ID, workflow instance ID, Service ID/Node ID, we can relate messages used in different parts of the invocations and interpret them. It has enough information to correlate particular message with a specific node of the workflow Graph. 2. It has location of all core services (e.g. Registry, DSC, GFac) took part in a given workflow

6 invocation. 3. It contains the workflow level configurations e.g. Location to stage output data directory, which scheduler to use, Urgency parameters act... Lead Context Header together with WS-Addressing Headers creates the information that passed in Soap Headers. Following are parameters defined in either case. 1. WS-Addressing - We use WS-Addressing W3C recommendation (a.k.a final) and these headers changed per each Soap Message. To - Recipient Service Address ReplyTo - Address to send results of the request Action - Soap action for particular message, used to locate the operation\ 2. Lead Context Header Description Created By Used By experiment-id Unique ID for experiment Created by Portal while creating the workflow, who registers it with MyLEAD. workflowinstance-id Unique ID for this workflow invocation. Created by GPEL engine when new workflow instance is instantiate from workflow template. workflowtemplate-id Unique ID for abstract workflow that used to create this workflow invocation. Given by the user when he save the workflow with MyLEAD or GPEL workflownode-id Node id in workflow graph XBaya?

7 workflow-timestep How many time steps have passed in workflow invocation (Is it a integer increment by one). This is set by GPEL engine while invoking a service and each invocation will have a different value. serviceinstance-id Unique ID for this service This is generated by Gfac, adding a time stamp to the Name of the Service Map document of particular service gfac-url Local of Gfac WSDL which will be used to create services for these invocations. Set by portal from portal configuration file. This is used by DSC to create a new service if instance of that service type is not already running dsc-url Location of Dynamic Service Creator Set by portal from portal configuration file. GPEL send all its service requests to DSC, which create an instance of that service if needed and invoke it. resourcecatalog-url or xregistry-url Location of xregistry wsdl, which act as the service registry for this invocation. This is initialized from configuration file of portal. Set by portal from portal configuration file. It is used by GFac to locate Service, Application and Host description documents and register CWSDL for service instances. mylead-agenturl Location of MyLEAD agent WSDL Set by portal from portal configuration file. Used by Gfac to register output of application invocations with MyLEAD event-sink-epr Location to send the events generated by workflow invocation Created by portal. This is used by Gfac and GPEL to find the event channel they should send workflow tracking events.

8 error-sink-epr Location to send errors and warning generated by workflow invocation, this is not in use as of now user-dn Invoker s DN of this experiment, this is set by Portal and not used as of now Set by Portal resourcescheduler Type of resource scheduler used by Gfac, and it is a Gfac specific parameter. It must be one of the values LEAD, VGRADS, or SPRUCE and it is set by portal based on user inputs. Set By Portal Used by Gfac to configure Job Submission URGENCY Spruce Urgency level. Set By Portal based on user input. Used by Gfac to configure urgency levels while submitting jobs to Spruce enabled Job Manager ForceFileStagi ngtoworkdir Gfac specific property that force the output of the application to working directory, if this is not set they are send to $WORKING_DIR/outputData. Set by portal Used by Gfac OUTPUT_DA TA_DIRECTO RY Gfac specific property, location to stage results of the application. It is set by portal and used by Gfac. Set by portal Used by Gfac OPENDAP_DI RECTORY Staged out put file URL usually has a GridFtp URL, this http URL location is used to map that to an http location. Set by portal Used by Gfac while publishing output data. Sample SOAP Header which includes WS-Addressing Headers and LeadContext Header is given below. <S:Header> <wsa:to> <wsa:action> n> <lh:context xmlns:lh=" <lh:experiment-id>urn:uuid:5d08982a-de49-4ecf-8d56-08d </lh:experiment-id> <lh:event-sink-epr> <wsa:address> </lh:event-sink-epr> <lh:user-dn>/o=lead Project/OU=Indiana University Extreme CN=rjunod/ =rajunod@marauder.millersville.edu</lh:user-dn> <lh:resource-catalogurl>

9 <lh:gfac-url> <lh:mylead-agenturl> <lh:workflow-instanceid>tag:gpel.leadproject.org,2006:72r/naminitializedwrfv2/instance307</lh:workflowinstance-id> <lh:resource-broker-url> <lh:workflow-time-step>9</lh:workflow-time-step> <lh:workflow-node-id>gfac:wrf_static_v5_2_7_preprocessor</lh:workflow-node-id> <lh:service-instance-id> </lh:context> </S:Header> 5. User Group Information Currently we have user and group support implemented at portal as well as Xregistry, and there is an ongoing work to use Xregistry group support from the portal. Xregistry support recursive group support (Group may have other groups as children), and Xregistry provide following operations to manipulate users and groups. createuser(), creategroup(), addusertogroup(), removeuserfromgroup(), removegroupfromgroup(), deleteuser(), deletegroup(), listusers(), listgroups(), listgroupsgivenauser(), listsubactorsgivenagroup() Also Xregistry provides capability support that allows users to share there documents with other users. addcapability(), getcapability(), removecapability(),isauthorizedtoacsses() For more details about Xregistry API, please refer to Xregistry Manual.