PROTOCOL DEVELOPMENT Quick Start Guide

Transcription

1 PROTOCOL DEVELOPMENT Quick Start Guide

2 Copyright Notice Copyright 2011 Accelrys Software Inc. All rights reserved. This product (software and/or documentation) is furnished under a License Agreement and may be used only in accordance with the terms of such agreement. Trademarks The registered trademarks or trademarks of Accelrys Software Inc. include but are not limited to: ACCELRYS ACCELRYS & Logo PIPELINE PILOT All other trademarks are the property of their respective owners. Restrictions on Government Use This is a commercial product. Use, release, duplication, or disclosure by United States Government agencies is subject to restrictions set forth in DFARS or FAR , as applicable, and any successor rules and regulations. Acknowledgments and References To print photographs or files of computational results (figures and/or data) obtained using Accelrys software, acknowledge the source in an appropriate format. For example: Imaging results obtained using software programs from Accelrys Software Inc. Data management and analysis performed with the Pipeline Pilot Imaging collection. Graphical displays generated with the Discovery Studio Visualizer. To reference an Accelrys Software Inc. publication in another publication, Accelrys Software Inc. is the author and the publisher. For example: Accelrys Software Inc., Chemistry Collection: Basic Chemistry User Guide, Pipeline Pilot, San Diego: Accelrys Software Inc., Request for Permission to Reprint and Acknowledgment Accelrys may grant permission to republish or reprint its copyrighted materials. Requests should be submitted to Accelrys Scientific and Technical Support, either through to support@accelrys.com or in writing to: Accelrys Scientific and Technical Support Telesis Court Suite 100 San Diego, CA Please include an acknowledgment Reprinted with permission from Accelrys Software Inc., [Document name], [Month Year], Accelrys Software Inc., San Diego. For example: Reprinted with permission from Accelrys Software, Inc., Pipeline Pilot Integration Collection: Server-Side Integration Guide, May, 2011, Accelrys Software, Inc., San Diego.

3 Contents Chapter 1 Introduction Architectural Overview... 4 Extending Pipeline Pilot s Functionality... 5 Application Integration Components... 5 Command-line Integration... 5 Language-based Integration... 6 Web Services Integration (SOAP Components)... 7 Database Integration (with ODBC-compliant and JDBCcompliant Drivers)... 7 Integration with Visualization Tools... 8 Additional Information... 8 Chapter 2 Common Principles Data Records... 9 Input Data... 9 Output Data... 9 Component Lifetime Management States of the Component State Machine Global Data Using Global Data Scope of Global Data Component GUI Component Parameters Component and Parameter Naming Parameter Types Component and Parameter Help Text Client-side Components Chapter 3 Command Line and Web Services Components Run Program Run Program on Remote Host SOAP Chapter 4 Language-Based Components Perl Java NET Windows Script Host Components Python VBScript PilotScript About the PilotScript Language PilotScript vs. Third-Party Scripting Custom Manipulator and Custom Filter Components. 21 Debugging Protocol Development: Quick Start Guide 3

4 Chapter 1 Introduction This guide provides a high-level overview of the components and languages you can use to develop your own Pipeline Pilot protocols. It also presents information about the packages and tools that are available for integrating Pipeline Pilot with third-party applications and services. Pipeline Pilot supports a number of approaches to extending its capabilities by adding new components to an existing installation. To integrate functionality from other software products, you can construct new components using original scripts, compiled code, and simple command lines. Resources are available for incorporating the present capabilities on a local server machine or from across the network. Pipeline Pilot offers many generic and domain-specific (pre-packaged) components that you can use to develop your protocols. Details about how to use these features are available elsewhere (Help Center > Users tab). This guide provides an overview of a number of specialty components, especially useful for customization, integration, scripting, and client-side operations, that allow you to greatly extend the capabilities of the pre-packaged components. Architectural Overview The diagram below represents a high-level overview of the server architecture. It illustrates how you can work with Pipeline Pilot to extend its server-side capabilities with new components or incorporate protocols into other applications using a client-side software development kit (SDK). Client integration and server extension architecture 4 Pipeline Pilot

5 Extending Pipeline Pilot s Functionality You can extend the functionality of Pipeline Pilot by integrating third-party data and computational services on the server. The Integration collection is a set of tools and components designed for this purpose. Integration techniques for extending Pipeline Pilot include: Configuring a program execution component to run a program command line on the server Integrating a web service application Customizing a base language component to encapsulate specific functionality in script or in a compiled language (Java, Perl, VBScript, and Python) Integrating an ODBC-compliant database Incorporating third-party visualization tools running on the client (Excel, Spotfire, and Accelrys Viewer) Constructing a new subprotocol component using encapsulated pipelines of other components For each of these approaches, a number of specific technologies may be available, depending on the goal of the integration project. A collection (described below) may contain components constructed with one or more of these techniques. When you build one or more related components, it is known as a "component collection". You can package the collection along with binaries, scripts, data files, documentation, etc. for consistent, managed deployment into an existing Pipeline Pilot installation. When users install components provided as a package, they can subsequently uninstall or upgrade the component collection following simple, standard procedures. The following information contains guidelines for each of the approaches with reference to the set of components that provide the relevant functionality, with particular emphasis on common techniques and concepts across the different groups of components. Application Integration Components Application integration components include: Command line integration (Run Program, FTP, SCP, SSH, and Telnet) Language-based scripting integration (Java,.NET, Perl, VBScript) Web services integration (SOAP) Command-line Integration You can execute third-party applications and file transfer services as command-line protocol interfaces. The following components are available for this purpose: Run Program Telnet SSH FTP SCP Run Program Components You can extend the functionality of a protocol to include any operation that you can invoke from a command line. Run Program (on Client) and Run Program (on Server) components are available for this purpose. These components provide a way for you to execute an operating system command on the server or on a client. For example, you can write data files, invoke the command line program to work on these files, and then read the results when the protocol is complete. You can also incorporate the STDIN and STDOUT streams into the data pipeline for a smoother flow. Protocol Development: Quick Start Guide 5

6 FTP File Transfer Protocol (FTP) is used to transfer files over a TCP/IP network (Internet, UNIX, etc.). Components that support FTP include Copy File from Remote Host (FTP) and Copy File to Remote Host (FTP). Use them to copy files from a server on your network to (or from) the server. Since these components use the FTP protocol, a valid user name and password are required as parameter values. Typically, FTP components are used in conjunction with Telnet or SOAP components to run programs on a remote machine. SSH Secure Shell (SSH) is a security protocol for logging on to a remote server. SSH provides an encrypted session for transferring files and executing server programs. SSH supports a variety of authentication methods and it provides a secure client/server connection for applications. Components are available that support SSH. The Run Command on Remote Host (SSH) component runs a command on a remote machine running SSH daemon and encrypts communication with the remote host. A password or key file is required as a parameter value for this component. The Manage Host Trust (SSH & SCP) component validates or updates a key signature in a known host file, which allows you to verify the identity of the remote host before sending the encrypted password. SCP Secure Copy Protocol (SCP) is a secure version of the UNIX remote (rcp) command for transmitting files to or from a remote machine. Unlike FTP, SCP encrypts all communication with the remote host. Components that support SCP include Copy File from Remote Host (SCP) and Copy File to Remote Host (SCP). A password or key file is required as a parameter value setting. Telnet Telnet is a terminal emulation method used on the Internet and TCP/IP-based networks. It allows a user to log onto a remote computer and run a program or execute UNIX commands. The Run Command on Remote Host (Telnet) component supports Telnet. Required parameters for the component include username and password. Typically, this component is used with the FTP components to transfer input and result files to and from the remote host. Language-based Integration You can enhance the capabilities of your protocols by including a scripting component that includes a script written in one of the supported languages (Java, Perl, and VBScript). The script writer has access to the contents of each data record passed to the script component and to the relevant protocol properties. This allows you to write components such as data readers, data writers, data filters, and calculators. The script component provides a facility to work within the framework of the program, while having access to the following: The rich code libraries available as Perl modules Client- and server-side software packages that support COM automation Pre-existing script or GUI utilities that you use in your work and integrate with Pipeline Pilot Java Component Development You can write new components in the Java programming languages using the Java (on Server) component. Java is a compiled language, so the component is parameterized with a reference to the class that supports the standard component interface. You write code that handles each data record processed or generated by the component. A full range of component types (readers, writers, filters, calculators, etc.) can be crafted in Java code, allowing you to extend Pipeline Pilot functionality available as third-party Java libraries. Generally, a set of Java classes (or jar files) and component XML are bundled into a package that facilitates deployment onto multiple servers. 6 Pipeline Pilot

7 .NET Component Development You can write new components in any.net language, such as C# or VB.NET. Components are available for dynamic compilation and execution as well as execution of already-compiled code. The components are.net (on Server), Dynamic.NET (on Server), Dynamic C# (on Server), and Dynamic VB.NET (on Server). Perl Component Development You can write new components in Perl using the Perl (on Server) component. Perl scripting is supported on both Windows and Linux, making it a useful language to develop platform-independent components. Perl has full access to the underlying component API, allowing you to write components of all types. Use Perl to design components that launch applications, access network resources, interact with the operating system, and access files. For example, if you are integrating a program that requires parameter files or complex parameter settings, use Perl to develop a more sophisticated program launcher than what is possible with command-line integration. You can save the underlying Perl code as a parameter on the component or as a file for easier reuse and maintenance. VBScript Component Development You can write new components in VB Script using the VBScript (on Client) and VBScript (on Server) components. Scripting with VB Script is supported on Windows using the Windows Scripting Host (WSH). The VBScript components are useful for COM automation such as performing tasks in Microsoft Office applications. The existence of a client version of this component is useful for integrating desktop applications, such as viewers, that users are more likely to install on the client machines than on the server. Python You can write components in Python using the prototype Python (on Server) component. Scripting with Python is supported on Windows only, using the Windows Scripting Host (WSH). This component shares many of the advantages of the Perl component, extending the functionality of Pipeline Pilot within the context of an industry standard programming language. Web Services Integration (SOAP Components) Web services provide a generic way to access remote applications. Once an application is deployed as a web service, clients need not be concerned with implementation details such as programming language or platform. Analogous to a web browser, the client needs only to submit requests to an HTTP address and the data is returned in a standard format. Simple Object Access Protocol (SOAP) is a format for making requests to web service applications. SOAP services consist of methods, each of which has a distinct signature that defines the expected inputs and outputs. The SOAP components convert data records to SOAP requests, send the requests to a SOAP server, collect the results, and add the results to each data record as new properties. In contrast to the Telnet and SSH components, there are no companion SOAP components to transfer files to the remote server. The SOAP request contains both the instructions and the input data, and the output data are returned directly to the SOAP component. Database Integration (with ODBC-compliant and JDBC-compliant Drivers) Relational databases can be accessed through Open Database Connectivity (ODBC) drivers or Java Database Connectivity (JDBC) drivers. These drivers allow connections to a variety of databases such as Oracle, SQL Server, and MS Access that reside anywhere on the network. Connections to these data sources, called Pipeline Pilot Data Sources, are defined through the Pipeline Pilot Administration Portal. All information for the data source can be specified in the portal (including the necessary login and password) or this information can be provided from Data Service Names (DSNs) that are defined on the server using the ODBC Administrator tool. The Data Source is used in the program s SQL components along with the necessary login and password for the database. The database components in the program allow you to select, store, delete, and update data in any accessible database source. Protocol Development: Quick Start Guide 7

8 Integration with Visualization Tools Integration components are available that support standard data visualization tools such as Excel, Spotfire, and Accelrys Viewer. These components are based around the VBScript (on Client) or Run Program (on Client) components that are used to control the client applications to view results data on the server. The common pattern is to export an appropriate data file on the server and execute an operating system command to start up the visualization software and read in the data file. With some client applications, a download step is also required. The scripting components provide a more direct mechanism for the integration of visualization programs that expose a COM automation interface. Following this model, you can extend the deployed client-side visualization tools, or add new ones. Additional Information For more information about the Integration collection and other Accelrys software products, visit 8 Pipeline Pilot

9 Chapter 2 Common Principles Regardless of the approach used to create a new component, you customize the component look and feel by doing the following: Defining new parameters for the component that allow a protocol builder to modify the component s behavior Selecting an icon for the component s graphical appearance Activating a set of input and output ports appropriate to the component behavior Composing a display name Authoring help text for the component and its parameters The final element depends on the type of customization involved. It may involve writing script or code, or configuring a command line or SOAP service. This step defines the behavior of the component associated with the processing of data records, where that behavior is modified by the component s parameter settings. In taking these steps, you create a new, unique component. An end-user can work with your component in a new protocol in Pipeline Pilot. From the end-user s perspective, the component should look like any other, and the implementation language should not be apparent. For example, if you build a file reader component, it should look and feel like other file readers, regardless of its implementation. The major data items that are of interest to the component developer are: Component parameter settings Global protocol properties and their values Data record that flows through the component Data Records It s possible to design a component that resides outside of a data pipeline, representing a single event that does not process data records. However, most of the components that you create are intended to read and write the contents of data records that the framework passes to them. Like the global data, consider a data record as a single flat property collection. In many cases, this is sufficient to access all the data content required by a component. A data record is a container that references a single node that is the root of a node hierarchy. Each node has an associated property collection. The default property collection of a data record is the property collection of the root node. Some components need to access and modify properties deep in the hierarchy to perform their intended task. The Data Record Tree Viewer component provides a useful way to analyze the complete content of a data record. Input Data The input data for a component is either a data record that flows in from an upstream component, or a new, empty data record. The component state dictates the type of records that are passed to it. The component state is under the control of the component script of program, and allows the component to determine how it is treated by the protocol runner, and is adjusted to suit the overall role of the component (reader, calculator, etc.) Output Data Typically, a component performs one or more actions on a data record, and then passes it downstream. A component can act as a filter by directing the data to a specific output port. A component may also suppress the data record so that the component does not output the data at all. If a record does output a data record, it may be identical to the record passed in, or it may be modified, it may be a cached record or it may be a completely new record. Protocol Development: Quick Start Guide 9

10 Component Lifetime Management A component works as a state machine, that is, a component progresses along a set of deterministic states. Code for component behavior must support the following events that occur in language-based components: Initialization: Occurs once before any records are processed. During this event, the component code can access the global data and component parameters. Typically, at initialization, a component reads its parameters and initializes any resources it may require. Process: Occurs once for each data record to process. During this event, the component code can access the data record itself, in addition to the global data and component parameters. Finalization: Occurs once when there are no more data records to process (or none to process in the first place). During this event, the component code can access the global data and component parameters. A component should free up any resources it might have used. The protocol framework guarantees that: A single Initialization event occurs before the first Process event, if any. A single Finalization event occurs when there are no more Process events (or if this no Process event in the first place). In either case, Finalization is called even if a protocol error occurs. Therefore, the overall lifetime sequence is: 1 A single Initialization event 2 Zero, one, or more Process events 3 A single Finalization event This sequence ensures that the component runs correctly, whether the sequence is repeated or not. For example, a Run To Completion subprotocol can run the sequence many times. This type of subprotocol reinitializes its complete environment for each data record that it processes. The implications of this are: In the Initialization event, do not assume that the component variables are already in a properly initialized state. Write your code to deal with the case when multiple lifetime sequences take place. At Initialization, the component might be in an uninitialized state or in whatever state the previous Finalization event left it in. Make sure you handle both with robustness, and ensure that you re-initialize all component variables in the Initialization event In the Finalization event, do not assume that all went well. All you can assume is that the Initialization event was invoked. The Finalization event is called even when the component threw an error, or if the protocol itself terminated prematurely. Always check to verify the state of everything. In general, avoid doing any real work in the Finalization phase. If you must, first confirm that the protocol is not in an error state. States of the Component State Machine With the exception of PilotScript components, language-based components use component state to determine the details of their own lifetime. During the Initialization event, the component determines its initial component state and passes this information back to the Pipeline Pilot framework. The component returns a State value to the Pipeline Pilot framework from each Process event. A simple example is a property calculator component. The framework passes all available input data records to a calculator component, and manipulates one or more property values. The component defines its state as ready to receive any data record available on its input port. Another example is a file reader component. In this case, the component should initially declare its state as expecting to receive new, empty data records whenever control returns to it. To fulfill its function, the reader component adds data to each empty record and passes it to an output port. When the end-of-file is reached, the component indicates that it will no longer receive data records by returning the appropriate state. The framework finalizes the component. 10 Pipeline Pilot

11 With the five following states, you can define a wide range of component behaviors: ReadyForInputData: The component requests that the framework invoke its Process event with any data record that arrives at its input port. ReadyForNewData: The component requests that the framework invoke its Process event with a new, empty data record. A new record is by definition empty, whereas an Input record typically is not empty. ReadyForInputThenNewData: The component requests that the framework invoke its Process event with any data record that arrives at its input port. When there are no more input records to process, the framework will instead pass to the component a new data record. ReadyForInputOrNewData: Similar to ReadyForInputThenNewData except that if there is input data, then no new data record is passed to the component. DoneProcessingData: The component s task is complete. The component requests that the framework does not invoke its Process event again. The required syntax depends on the role of the language-based component. Below are some broad categories of components and how they make use of component state. Component Category Calculator Filter Reader or Generator Writer Integrator Functional grouping Component State Management Component state is typically set to ReadyForInputData. Component state is set to ReadyForNewData. When the operation is complete (for example, end of file for a reader), the state is set to DoneProcessingData. Component state is set to ReadyForInputData. If a maximum output limit is reached, or there are no more data records to process, the state is changed to DoneProcessingData. Component state is set to ReadyForInputThenNewData. For each input data record, the component caches the necessary data. When there is no further data on the input port and a new data record is processed, the component starts to output records of aggregated data. ReadyForInputOrNewData could be used for grouping of similar functionality into a single component of which some functionality requires input, whereas other functionality does not require input. Other variants are possible based on management of the component state in the logic of the component. For example, consider how you would develop a component that outputs one record following each 10 input records, with mean values for the properties, calculated from the 10 input data records. One solution is for the component to flip its state from ReadyForInputData to ReadyForNewData after each 10 records, and then flip it back. Global Data Global data is a set of properties (name-value pairs) available to all components in a protocol. For many purposes, you may consider the global data as a single, flat property collection. A property collection is simply an ordered set of named properties and their values. However, to support scalability and to provide some organization of data, the global data structure is actually a hierarchy of nodes, each with its own property collection. Tip: To see the global hierarchy in its entirety, add the Global Data Tree Viewer component into an empty protocol and run it. Protocol Development: Quick Start Guide 11

12 Using Global Data Global Data Tree Viewer results The default global property collection is the property collection on the root node of this hierarchy. A component writer can access all properties in the hierarchy, for reading and writing by navigating over the node hierarchy. Note that some properties are read-only, since they are defined by a package or by the protocol runner subsystem and act as configuration constants. These are marked with a padlock in the Global Data Tree Viewer. Examples are the user name or the server folders; it makes no sense to edit such properties in a protocol. Component developers are mostly interested in global properties on the root node, so you can handle the global data as a simple property list and ignore anything at a lower level in the hierarchy. However, if you declare global properties in your package those properties appear in the global property hierarchy in a subnode, named after your package, to ensure name uniqueness. (For details, see the Application Packaging Guide.) These properties are defined for any protocol running on the server where your package is installed. All programming environments include APIs and techniques for getting and setting global property values, from either the default property list or from somewhere else in the data hierarchy. Package global properties are an example of deep properties. They can be accessed with a deep property syntax, which means that their name starts with a forward slash and includes the path to the node where they reside. (e.g. /acme/gizmos/bindir ). Alternatively, explicit navigation of the data nodes is also supported by the richer API sets. 12 Pipeline Pilot

13 Scope of Global Data A subprotocol can also define global properties that are scoped only to that subprotocol; it is only visible within that subprotocol and its descendent subprotocols. It is not difficult to see why global property scoping is important. A subprotocol component may be dropped into any other protocol; for a component to pollute the namespace of the top-level protocol could lead to all sorts of problems if a collision were to occur. Even the simple case of using the same subprotocol component twice in a protocol would be error-prone without the possibility of global property scoping. In addition to property scoping, you also can declare any top-level node added as a direct child of the global root node for scoping to a subprotocol. The concept of a hierarchy of global data overlaid by the subprotocol hierarchy of scoped namespaces can be tough to visualize. The main thing to remember is that the global data visible to your component can include properties that are not visible outside the subprotocol where it lives, since they only relate to operations within the subprotocol. Component GUI Component Parameters In the component implementation code, you can access the parameters of its component as a read-only property collection. This allows you to discover the settings specified by the protocol builder or by the enduser, and to modify the component behavior accordingly. During component construction in Pipeline Pilot, you can mark some parameters as required, while others are optional. For example, a file reader component is only useful if the user specifies a parameter value for the source file location. Therefore, the source parameter would be a required parameter. The maximum number of records to read might be a useful, but optional parameter. If not supplied, your code would probably read all the data in the file. Note: Make sure that your component regression tests include a variety of different parameter settings for a component, so that all possible paths through the code are tested. (For more information, see Regression Testing.) The parameters that you define on the component represent the ways in which you allow protocol writers to control or modify the behavior of your component, so your implementation code should honor that. IMPORTANT! Be sure to offer backward compatibility with your component revisions. If you remove or rename a parameter from one product release to the next, ensure that you support the old parameter, since existing protocols require your new implementation to work with the old component interface definition. Component and Parameter Naming Give components and parameters short names that describe their function. Component and parameter names may contain whitespace and are generally more readable if they do. We recommend that you review the existing examples and component and protocol design guidelines. Familiarizing yourself with the conventions case, spacing, and capitalization will give your components a more standard look and feel. Parameter Types Each component parameter has an assigned type which is displayed in the Help tab when that parameter is selected. Some consequences of parameter type: A parameter s type controls what user interface tools are available when users set the value of the parameter. For example, parameters of URL type can be set using a File Browser dialog. Certain parameter types, such as BoolType, predefine the legal values for that parameter and expose a corresponding GUI for setting the value. A list of enumerated legal values can be defined to restrict the value of a parameter and simplify user input. Protocol Development: Quick Start Guide 13

14 Certain parameter types, such as LongType, are subject to validation when the component is initialized. If the current value cannot be converted to that type, the protocol stops running and displays an error. Parameters of ExpressionType are evaluated by the PilotScript interpreter when the component is initialized. The value of the parameter is set to the result of the expression. Parameter types can be set or modified on the Interface tab of the Edit Component dialog. In addition to some intrinsic behaviors, the Edit dialog for a parameter allows you to specify arbitrarily complex validation expressions and enabling/disabling scenarios involving the value of the parameter and the value of other parameters. The set of legal values on a parameter can also be made dynamic in this way. You can even specify the running of a protocol to supply such information, so that the server-side environment can influence the parameter user interface. Component and Parameter Help Text Help text is displayed in the Help window (lower-left corner of program window) when end users select a component or parameter in Pipeline Pilot. Make sure that your components and all of their parameters are completely documented so end users have detailed information on how to use them. Component help text includes a brief purpose statement and a detailed description. The purpose statement summarizes what users can do with the component. It is also used as fly-over help and is exposed as a tooltip when hovering the cursor over a component icon in the Explorer or workspace. The description text provides detailed information about the component. It is displayed below the purpose statement in the Help window. You can use HTML to structure the information in a way that makes it easy to digest. All ports include a comment field that is displayed in the Help window, below the purpose and description text. Use the Ports tab in the Edit dialog to modify the port information that is displayed. Comments are useful to specify requirements for input data, what type of data is in the output, and to define what determines if a data record exits the Pass port or the Fail port. Parameter help text describes how to use the parameter and how it influences the behavior of the component. You can describe the set of legal values for the parameter along with any dependencies on other parameters. The parameter help text is displayed with the parameter type, so the parameter type does not need to be stated in the help text. Again, you can use HTML tags if you need to present more structured information. Client-side Components Components that run on the client machine are an integral part of the Pipeline Pilot platform. Setting up viewers and other interactive functions as client-side components makes it possible to visualize data locally rather than on the remote server. The client-server architecture calls for computation to take place on the server. There may be situations where an application is integrated as a client-side component because the convenience of installing it on the local desktop outweighs the performance advantage of running it on the server. If client-side components are used, the resulting files need to reside on the client machine, instead of on the server. Pipeline Pilot provides components for manipulating files between the server and client. These components include: Create Tempfiles Copy to Client Copy to Server There are also two directories that are useful for this purpose: Temp Directory Client Run Directory 14 Pipeline Pilot

15 The run directories are owned by the running job and can store files temporarily. Their locations are generated at run time, so they are designated using global variables rather than the literal path. The Client Run directory is an ideal place to store files for use by viewers and other interactive components. At run time, the results are generated on the server, written to a file on the server, and copied to a file on the client. After the protocol is finished processing, the data is viewed on the client machine, and any files that reside on the server and client are cleaned up automatically by the server at a later date. Protocol Development: Quick Start Guide 15

16 Chapter 3 Command Line and Web Services Components In many cases, the integration challenge is to construct a component that encapsulates the functionality of an existing legacy program that can execute on a command line. Often the encapsulation process also needs to export the data in a specific format so the program can use it and import of the resulting data. Some programs can take input on the STDIN stream, and most can write data or some sort of report data to STDOUT, STDERR or a named file. This set of components includes support for running a legacy program on the same machine as the server, or on a remote host. In some cases a remote program may expose a SOAP interface, or one may be specially constructed for Pipeline Pilot access. In other cases, more basic means of communication is used. The goal of the components in the command line category is to hide all of this complexity behind a component that looks like any other. The protocol author s concerns should focus on setting appropriate parameter values and not on the underlying mechanisms that make the data processing occur. Run Program The Run Program components execute a command directly on the server or client machine. This is ideal for integrating applications that have a command-line interface that exposes all or most of the functionality. The server version runs on either Windows or Linux platforms. Windows and Linux have different command syntax, so a single instance of this component cannot successfully run on both platforms. A solution is to create two copies of the component and place them on each output port of a filter that checks the server operating system. The convention of Pipeline Pilot s client-server architecture is to perform all computation steps on the server and to launch data viewers on the client. This convention provides a guide when choosing between the client and server version of Run Program, but there are other considerations: The use of a third-party application may dictate either the client or server version, depending on where the application is (or can be) installed. The client runs on Windows, while the server may run on Windows or Linux. A component that integrates with a Windows application is portable to all installations, if written using Run Program (on Client). Pipeline Pilot is not the only way to run protocols. Web Port, SOAP, client SDKs, and some third-party applications provide alternatives. A Run Program (on Client) component is usable only from Pipeline Pilot. To deploy a component, consider creating client and server versions of the component to cover the maximum possible use cases. Unlike COM automation using the VBScript components, the path to the executable file must be specified as part of the Command parameter on the Run Program components. To avoid hard-coding this path, it can be stored in a global variable or exposed as a component parameter. The command-line application can receive data from the current data record directly using STDIN. More typically, all the data records are written to a file and the name of that file is passed to the application as a command-line argument. This strategy often employs Pipeline Pilot s temporary file management, particularly when the program runs on the client. 16 Pipeline Pilot

17 Run Program on Remote Host You can create components that run command-line applications on a remote host using either Telnet or SSH. Telnet sends clear data to the remote machine, while SSH sends encrypted data. These components are available on both Windows and Linux servers. Though typically used to access UNIX or Linux machines, the availability of OpenSSH makes it possible to access remote Windows machines with the SSH component. Both the Telnet and SSH components require supplied credentials. The Telnet component requires a username and password. The SSH component can use a file containing a private encryption key. The SSH component also has features for maintaining a list of trusted hosts and warning if a remote host s identification changes. As with Run Program, the location of the executable must be specified. The same is true for input and output files that are passed to the program as command-line parameters. The SSH component can capture textual output from the command and write it to a property. More typically, the results are collected in a file on the remote machine, and the file is copied back to the server using SCP or FTP. SOAP Simple Object Access Protocol (SOAP) is a method for accessing services on the web. It employs the XML syntax to send text commands across the Internet using HTTP. The Pipeline Pilot SOAP components are available on both Windows and Linux servers. The components access SOAP servers and make requests to services that exist on UNIX, Linux, and other remote machines. Numerous SOAP components are available for sending the necessary data to the SOAP server, collecting the results, and adding the data to the current record in a protocol. The signature of the SOAP service (inputs and outputs) can be read from a Web Service Description Language (WSDL) file or specified using the parameters of the SOAP. The inputs are taken directly from properties on the data records, from global variables, or derived from an expression. The outputs are stored as properties or global variables. In the standard component, one SOAP request is made for each data record and the server waits for a response before sending a request for the next record. Two variations on this order of operations are available. A batched component, which submits multiple records in a single request, can improve performance. However, the SOAP service must be configured to accept batched (array) data. A queued component, which submits multiple requests to the SOAP server without waiting for a response. This component is useful when the SOAP service is running on a cluster or is otherwise capable or parallel operation. Protocol Development: Quick Start Guide 17

18 Chapter 4 Language-Based Components This section focuses on the components that allow you to define functionality with a script or compiled language. These are referred to as language-based components. This chapter provides an overview of the various components available for this purpose and information related to selecting the best component for a specific task. The language-based components include: Perl (on Server) Java (on Server).NET (on Server), Dynamic.NET (on Server), Dynamic C# (on Server), Dynamic VB.NET (on Server) Python (on Server) VBScript (on Server) VBScript (on Client) Custom Manipulator (PilotScript) Custom Filter (PilotScript) All language-based components require script or code to define the behavior of the component in response to certain events, such as receiving a data record to process. For each language, there is an application programming interface (API), which provides access to appropriate data structures and exposes utility functions. Each component has sufficient access to the Pipeline Pilot data structures that are required to plug and play in the protocol environment where it resides. Perl You can build Pipeline Pilot components whose function is coded with Perl. For the Perl (on Server) component, write a Perl script to define the Initialization, Process and Finalization phases of the component life cycle. The Perl script utilizes an object-oriented API to work with the object model of the data records that are passed to it, and to access global properties and the component parameters. The Perl (on Server) component is an empty, but functional component that provides a template for adding your Perl script to the three subroutines (oninitialize, onprocess, and onfinalize), that define the life cycle of the component. Alternatively, you can move this code into a Perl module file that exists in an application package independent of the new component and reference it with the use syntax in the Perl component. This makes it easier to upgrade the function of your component after deploying it in a number of protocols. This approach also hides the implementation of the component from Pipeline Pilot users. The Perl (on Server) component can access package-defined Perl modules by indicating the package it belongs to in the Use Package parameter. This extends path to include any Perl paths that are defined in the package configuration file. Using Perl to code a new component gives you access to the vast number of publicly available Perl libraries, in addition to your own Perl modules. Note: For detailed information on using Perl to write new components, see the Perl Component Development Guide. 18 Pipeline Pilot

19 Java Java is a compiled language, so the process for building new components with Java is more complex, although the principles of component development remain the same. To implement a component, define and compile a Java class that implements a defined Component interface with the three life cycle methods (oninitialize, onprocess, and onfinalize). A full object model API is supported to query and modify the data records and to access component parameter and global settings. Since the Java binary code is a separate entity from the component definition (XML), Java components are always constructed using the package paradigm. (For details, see Application Packaging). An application package is a folder hierarchy containing the component XML, example protocol XML, binary files, example data file, documentation, and other configuration information. All these files are installed and uninstalled together on any deployment server, so the Java class (or JAR) files are always maintained consistently in the same package as the associated component definitions. The Java (on Server) component is parameterized by indicating the package for which it belongs, and the name of the class to load that implements the Component interface. The package defines CLASSPATH values to locate the class within the package file structure. The Java (on Server) component can incorporate package-define CLASSPATH settings to locate the class that implements its Component interface (or any other class or jar file) within the package file structure. This is done by indicating the package for which the component belongs in its Use Package parameter. This extends the Java run time to include any CLASPATH settings that are defined in the package configuration file. Note: For detailed information on using Java to write new components, see Java Component Development Guide..NET.NET languages are compiled into intermediate code that is then executed on the target platform. Pipeline Pilot currently supports.net on Windows. The.NET components are provided in two flavors: precompiled or dynamic..net (on Server) Dynamic.NET (on Server) Dynamic C# (on Server) and Dynamic VB.NET (on Server) Note: For detailed information on using.net languages to write new components, see the.net Component Development Guide. Windows Script Host Components On Windows only, there are a few component based on the Windows Script Host (WSH) technology. All components based on the WSH expose a GUI that allows you to define scripts for the usual component lifetime phases (initialization, record processing and finalization). The script can address the following objects: Data: The data record Globals: The global properties Parameters: The component parameter properties Component: To set the component state and route data Each of these objects has appropriate methods to get and set properties. So the same sort of functionality is exposed as in the Perl and Java component, but in a different way. In addition to the lack of platform portability, one other difference is that the WSH interface does not provide direct access to the node hierarchy. Protocol Development: Quick Start Guide 19

20 Python Starting with version 5.0, Pipeline Pilot supports Python scripting via the Windows Scripting Host (WSH), which restricts the use of such components to the Microsoft Windows platform. For details on the API, see Windows Scripting Integration. There are also some example protocols that include Python components. Python scripting is supported via the Python (on Server) component. Its interface includes separate scripts for the Initialization, Process, and Finalization phases of the component life cycle. This lives in the prototype section of the XML database because of a known, reported, but persistent memory leak in the Python WSH scripting engine distributed by Active State. VBScript The VBScript (on Server) component allows you to deploy (to Windows servers) a component coded in VBScript. This can provide a useful mechanism for hooking into scriptable Windows applications installed on the server or the many scriptable objects that are deployed with the Windows operating system. The interface of the VBScript (on Server) component includes separate scripts for the Initialization, Process and Finalization phases of the component life cycle. For details on the API, see Windows Scripting Integration. There are also some example protocols that include VBScript components. PilotScript About the PilotScript Language PilotScript is an expression language for writing powerful data filters and data manipulator components that work on a streaming data record. The syntax for PilotScript has its origins in a subset of Oracle s PL/SQL language, with overtones of Perl. PilotScript is very fast and has little overhead because it is tied closely to the underlying data structures. It provides a relatively simple way to create components that do direct manipulations and analyses of a data record. PilotScript provides over 150 functions to work on the property lists, properties, and values in data records and global data. Many components include parameters with values that are PilotScript expressions evaluated at run time. In particular, the Custom Manipulator and Custom Filter components (included with Pipeline Pilot) employ PilotScript to define new components that work with the data in the specific ways that you need. PilotScript vs. Third-Party Scripting PilotScript is designed to provide convenient and efficient access to the data structures used by Pipeline Pilot. However, PilotScript does not have any significant facilities for working with data external to Pipeline Pilot. The scripting components are designed for integration tasks. They have access to read and write Pipeline Pilot data, but their strengths are the intrinsic capabilities of the scripting languages, and their ability to work with data and computational services external to Pipeline Pilot. For example (and these are by no means exhaustive descriptions of what is possible with the scripting components): VBScript is well suited to integration tasks that involve services deployed as COM components, such as Windows-based client visualization applications. The Perl (on Server) and Java (on Server) components can directly access functionality from the vast number of Perl modules and Java libraries that are available in the public domain. Python provides an object-oriented environment for working with CORBA; there is much other public domain Python software. The combination of access to Pipeline Pilot data and the integration features of each third-party scripting environment make them useful to consider when you are designing a protocol where you need access to an external service. 20 Pipeline Pilot