EasyFlow - v Application Developer Guide

Transcription

1 EasyFlow - v Application Developer Guide June 23, 2014

2 i 2013 CAMELOT Biomedical Systems Srl. The information in this document is proprietary to CAMELOT. No part of this publication may be reproduced, copied or transmitted in any form or for any purpose without the express prior written permission of CAMELOT. The information contained herein may be changed without prior notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. CAMELOT shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. Any software coding and/or code lines / strings ( Code ) included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended better explain and visualize the syntax and phrasing rules of certain coding. CAMELOT does not warrant the correctness and completeness of the Code given herein, and CAMELOT shall not be liable for errors or damages caused by the usage of the Code, except if such damages were caused by CAMELOT intentionally or grossly negligent.

3 CONTENTS 1 Contents 1 Introduction What is EasyFlow? Getting Started Requirements Installation Test your installation Using CMake Hello World! Introduction Create the project Write the code CMakeLists.txt HelloWorld class main function Compile the project Test your application Complete Functionalities Handling Input Interruption Points Initialise and Release Resources Where to Go from Here Deploying your app Further information Introduction 1.1 What is EasyFlow? EasyFlow is a software framework meant to simplify the integration of image post-processing applications ("apps") into systems meant for the management and storage of images. We refer to the system integrating the apps as the "host". Within the EasyFlow framework, an application can receive data from the Host, execute a processing task, and send the results back to the Host. As an example, in a radiology department a PACS is the system used to archive all image series acquired with the different modalities. Some of the acquired series may require further processing before they can be reviewed by the radiologist, and this task may be delegated to a separate process on a dedicated server. In this scenario the PACS would play the role of Host, integrating the App doing the actual work on the dedicated server. A similar scenario can be seen in the context of digital pathology. Here again we have systems collecting and archiving whole-slide images from the acquisition device, and often these images may require post-processing before being reviewed by the pathologist. Through EasyFlow image management system could play as Host and delegate the post-processing to apps residing on a separate server.

4 1.2 Getting Started 2 Figure 1: EasyFlow architecture The integration of an app and its host occurs by allowing them to communicate over TCP/IP through a well-defined protocol. Since an app and its host communicate over TCP/IP, this allows for a much wider range of set-ups and configurations. The framework defines: a host-application interaction layer, that handles the interaction between the Host and the Application through web service, and it is hidden to third-party developers; a integration layer, that is used for the rapid development of hosted applications and the integration in hosting systems, and it is the layer exposed to third-party developers. This document shall guide you through the deployment of an application as an EasyFlow app. The examples of this tutorial describe how to write a hosted application program using the integration layer offered by the EasyFlow framework, which takes care of all the necessary communication with the host. Using EasyFlow allows the developer to focus on the logic of the post-processing application. 1.2 Getting Started Requirements The EasyFlow framework will work in both Unix-like and Windows operating systems. It has been developed and tested across different combinations of systems, compilers and hardware platforms including Windows, Linux and Mac OSX on Intel-compatible hardware. It is known to work for the following combination of operating systems and compilers: Windows (XP 32bit, 7/8 32/64bit) with Visual Studio 2008/2010 (both Express and Professional edition) Linux (kernel 2.6.3x/3.x 32/64bit) with GCC 4.6.x Mac OSX ( /64bit) with Apple LLVM 5.x In order to build your own EasyFlow application you will also need to install CMake, a cross-platform and opensource family of software tools designed for managing the build, test and package process of software using a compiler-independent method. CMake can be downloaded at no cost from:

5 1.2 Getting Started 3 You can download binary versions for most of the popular platforms including Windows, Mac OSX and Linux. Alternatively you can download the source code and build CMake on your system. Follow the instructions in the CMake Web page for downloading and installing the software. EasyFlow also depends on the Boost ( and Apache Thrift ( apache.org) libraries. The parts of Boost and Thrift that are required for developing an EasyFlow app are already included in the EasyFlow distribution Installation The EasyFlow framework is shipped in two versions: the Application Developer package; the Host Integration package. Both versions are currently shipped as zip archive files, containing the present guide as well as all that is needed to either develop an EasyFlow App, or to integrate the framework in a Host. This guide only deals with the Application Developer package and the subsequent paragraphs will explain how to use it. For the Host Integration package you have to refer to the Host Integration Guide. To install the Application Developer package simply unzip the archive in the folder of your choice. Once unzipped, the installation folder will contain the following directories: Copyright.txt: the copyright notice. bin: the directory holding auxiliary tools, among them a simple Host to test you apps. include: the directory holding all the header files needed compiling an app. lib: the directory holding all the libraries your application will link to. examples: docs: examples/helloworld - contains the files of the "Hello World" example. docs/tutorials/ - contains the tutorial that you are reading. docs/api_reference/ - contains the doxygen documentation which describes in detail every class and method in the framework. The documentation also contains inheritance and collaboration diagrams and listing of data members Test your installation In order to test the installation, we suggest to build the example app in the examples/helloworld directory using CMake, and test it with the simple Host distributed with the packages. You will find more information in section Compile the project Using CMake For detailed instructions on how to use CMake, refer to the website. Here we give a very short summary to introduce you to its use for building an EasyFlow app. CMake expects a CMakeLists.txt file defining the project you want to build. Typically, in this file you will define which source code files must be built, where the header files are placed, which libraries the executable should be linked to and so on. CMake processes the CMakeLists.txt file, possibly requiring input from the user for

6 2 Hello World! 4 information that it cannot gather automatically, and generates the files needed to build the project (e.g. Makefiles, Visual Studio solutions, XCode projects). The CMake program can be used both through textual (only UNIX-like systems) and graphical (on all systems) user interface. In both cases, you will be asked to provide the path to the folder where you have installed the Application Developer package. To use the TUI, open a terminal and change the current working directory to the folder where you want to build your project. The build directory can be the same source directory (what is called an in-source build) or a specially created directory. Then you can type the "ccmake PATH_TO_PROJ" command, where you should replace P- ATH_TO_PROJ with the path to the project directory holding the CMakeLists.txt file. To use the GUI, run the cmake-gui program and provide the build folder and the project folder. 2 Hello World! 2.1 Introduction In this section we will describe how to write your first EasyFlow app. The goal of this example is to guide you in writing a simple EasyFlow app which stores on disk a text file containing the "Hello World!" line. Neither input files nor input parameters will be used in this example. All the sources and files related to this example are present in the installation archive inside the "Examples/Hello- World" folder. 2.2 Create the project First of all, before starting writing the code, you need to create and set up properly the new project that will contain the example. The simplest way to create a new project with EasyFlow is to create a new folder somewhere in your disk and create four files inside it: A CMakeLists.txt file that will be used by CMake to generate a Makefile (if you are using UNIX-like systems) or a Visual Studio solution (if you are using Windows). A source file with the main function that allows the example class to start execution. The header and source files with the implementation of the C++ class doing the actual work. The details of these files are described in the following section. Once all the files are in your directory, you can run CMake in order to configure your project. 2.3 Write the code CMakeLists.txt The file contains the following lines: CMAKE_MINIMUM_REQUIRED(VERSION 2.8) PROJECT(HelloWorld) FIND_PACKAGE(EasyFlowApp) INCLUDE_DIRECTORIES( $EASYFLOW_APP_INCLUDE_DIR ) SET( HDRS HelloWorld.h )

7 2.3 Write the code 5 SET( SRCS HelloWorld.cpp ) ADD_EXECUTABLE(helloWorld main.cpp $SRCS $HDRS) TARGET_LINK_LIBRARIES(helloWorld $EASYFLOW_APP_LIBRARIES) The first line sets the minimum required version of CMake for the current project. If the current version of CMake is lower than that required it will stop processing the project and report an error. The second line defines the name of your project as it will appear in Visual Studio (it will have no effect under UNIX-like systems). The FIND_PACKAGE command deals with setting both the $EASYFLOW_APP_INCLUDE_DIR and the $EASYFLOW_APP_LIBRARIES variables. In order to find the right paths, it will require the path of the Easy- Flow App installation, which should be provided via TUI/GUI. The INCLUDE_DIRECTORIES adds the found directory to those the compiler uses to search for include files. Using the SET command, we specify the header and the source files of the class that will contain the code of our example. The ADD_EXECUTABLE command defines as its first argument the name of the executable that will be produced as result of the building of this project. The remaining arguments of ADD_EXECUTABLE are the names of the source files to be compiled and linked. Finally, in the TARGET_LINK_LIBRARIES command we specify the EasyFlow library which will be linked against this project HelloWorld class In order to create an App, a class must be derived from BaseApp, and some of its methods must be reimplemented. The file HelloWorld.h contains the declaration of the derived class: #ifndef HELLOWORLD_H #define HELLOWORLD_H #include "App/BaseApp.h" #include "Interfaces/DataExchange_types.h" #include <string> #include <vector> class HelloWorld : public BaseApp public: HelloWorld(std::string addr, int port, int http_port, std::string host_addr, int host_port, std::string doc_root, std::string ext_addr); bool inittask(); void exectask(const std::map< std::string, std::vector<apphost::objectlocator> >& input, const std::vector<std::string>& inputparams, apphost::uri outputlocation, std::vector<apphost::uri>& results); bool releasetask(); protected: void setpreferredprotocols(); ; #endif//helloworld_h First we include the BaseApp class header. Then, we declare the HelloWorld class, derived from BaseApp. Its constructor takes as input the same arguments needed by the constructor of the superclass. Furthermore, we declare all the methods that we need to reimplement: exectask is the core method of the class, and it should implement the processing task; inittask is a method called before the execution of the task and used to initialise any structure that does not depend on the input;

8 2.3 Write the code 6 releasetask is a method called after the execution of the task to release any resource allocated during the task execution; setpreferredprotocols is a method used to define which protocols should be preferred for transferring data between app and host. The file helloworld.cpp contains the definition of the class. In order to create the example class, you have to include the helloworld.h header previously described. In this example, you must also include the <fstream> header, required to create and store on disk the text file containing the "Hello World!" line. #include "HelloWorld.h" #include <fstream> Subsequently, you must implement the class constructor calling the superclass constructor in its initialization list. Since this is a simple example, you don t need to initilize and release anything: so both inittask and release- Task methods are implemented to only print (on standard output) a log string. HelloWorld::HelloWorld(std::string addr, int port, int http_port, std::string host_addr, int host_port, std::string doc_root, std::string ext_addr) : BaseApp(addr, port, http_port, host_addr, host_port, doc_root, ext_addr ) bool HelloWorld::initTask() std::cout << "A: Preparing for task" << std::endl; return true; bool HelloWorld::releaseTask() std::cout << "A: Releasing task" << std::endl; return true; The core of the class lies on the exectask. Its arguments are a list of input files, a list of input parameters (both not used in this example), the URI of where the output should be stored and a list of output files. The first three arguments will be provided by the host, while the last will be written in output by the method. In this implementation, we store in the local output folder a text file with the single "Hello World!" string written inside it. Note that to perform this operation, we need to call the auxiliary method buildoutputuris: this method builds the URI to return to the host and helps you to get the local output folder (stored inside the m_outfullpaths field). Before calling this method, we have to clear and populate the m_outfilenames field pushing the filename of the output result; void HelloWorld::execTask(const std::map< std::string, std::vector<objectlocator> >& input, const std::vector<std::string>& inputparams, URI outputlocation, std::vector<uri>& results) std::cout << "A: executing the task" << std::endl; m_outfilenames.clear(); m_outfilenames.push_back("output.txt"); buildoutputuris(outputlocation, results); std::ofstream ofs; ofs.open(m_outfullpaths[0].c_str()); ofs << "Hello World!" << std::endl; ofs.close(); Finally, we find the implementation of the setpreferredprotocols method, which sets the preferred protocols for transferring data between app and host. The protocols are inserted in order of preference into a list. The Host selects the most appropriate protocol and uses it in setting up the resources and generating the URI returned to the app. An EasyFlow app shall at least support both the http: and the file: protocols). void HelloWorld::setPreferredProtocols() m_preferredprotocols.clear(); m_preferredprotocols.push_back("file://"); m_preferredprotocols.push_back("

9 2.4 Compile the project main function This file contains the code of the main function: #include "HelloWorld.h" void print_help_usage() std::cout << "\nhelloworld" << std::endl; std::cout << "usage: app <app_address> <app_thrift_port> <app_http_port> <host_address> <host_thrift_port> <app_doc_root> <app_ext_address>" << std::endl; int main(int argc, char** argv) std::string app_address = ""; int app_thrift_port = 0; int app_http_port = 0; std::string host_address = ""; int host_thrift_port = 0; std::string app_doc_root = ""; std::string app_ext_address = ""; if (argc == 8) app_address = argv[1]; app_thrift_port = atoi(argv[2]); app_http_port = atoi(argv[3]); host_address = argv[4]; host_thrift_port = atoi(argv[5]); app_doc_root = argv[6]; app_ext_address = argv[7]; else print_help_usage(); return 0; HelloWorld hello(app_address, app_thrift_port, app_http_port, host_address, host_thrift_port, app_doc_root, app_ext_address); hello.start(); After including the header with the declaration of the HelloWorld class, we must manage the input arguments: IP address or domain (host name) of the applications; TCP port that the application web service will listen to; TCP port that the application http server will listen to; IP address or domain (host name) of the host; TCP port that the host web service will listen to; Path to the application http server document root. In order to define our hosted application, we declare an instance of the HelloWorld class, passing to its constructor the input arguments. Now we are ready to start our hosted application calling its method start. At this point you have successfully created your first simple EasyFlow app. 2.4 Compile the project After writing the code you need to compile the project in order to use your first simple EasyFlow app. To compile you have to use the file generated by CMake (e.g. Makefiles, Visual Studio solutions, XCode projects) and the appropriate tools/commands. On Visual Studio be sure to select the build configuration (e.g. Debug, Release) properly depending on the configuration of the libraries shipped with the package.

10 2.5 Test your application Test your application In order to test your application you need to write a description file that will be used as interface to the Host. For further details read the Deploying your app section. In this section we will guide you in writing a description file suitable for the "HelloWorld" example. Here we can use a simple description file in which we don t need to specify both the "inputs" and "parameters" fields: "programpath": "<Path to HelloWorld executable>", "programargs": [ "localhost", "localhost", 9090, "<Path to the application http server document root>" ], "address" : "localhost", "name" : "HelloWorld App", "description" : "HelloWorld App Example (file protocol)", "inputs" : [], "outputs" : [ "name": "output", "description" : "text file", "supportedmimetypes" : [ "text/plain" ], "supportedmodalities" : [] ], "parameters" : [] The application can be tested using the simple Host distributed with the package: create a folder to be used as the Host http server document root; open the file bin/hostgui-settings.ini: check the port settings to be sure that there is no conflict with other running applications; set the path to the folder to be used as the Host http server document root; start the Host (bin/hostgui) and load the settings file; load the app JSON file by clicking on the "Load App Description File" button; (optionally) check that the app has been started by looking for a corresponding process; choose the output folder by selecting the "Output" tab and by clicking the "Search" button; start the task by clicking on the button "Start"; check that the file output.txt has been created, and the content is correct. Alternatively the application can be tested using both the Host and the AppStarter RESTful web service distributed with the package: create a folder to be used as the AppStarter http server document root; copy inside it the JSON file of the HelloWorld application; open the file bin/appstarter-settings.ini: check the port settings to be sure that there is no conflict with other running applications; set the path to the folder to be used as the AppStarter http server document root; open a console and start the AppStarter (bin/appstarter.exe) passing the settings file as input argument; create a folder to be used as the Host http server document root; open the file bin/hostgui-settings.ini:

11 3 Complete Functionalities 9 check the port settings to be sure that there is no conflict with other running applications; set the path to the folder to be used as the Host http server document root; start the host (bin/hostgui) and load the settings file; get the applications catalog by clicking the "Get the Apps Catalog" button; (if the operation is completed successfully the "Register an App" button will be enabled); click the "Register an App" button to display a dialog window with the list of all available applications (in this case there will be only one application); select the application from the list; (optionally) check that the app has been started by looking for a corresponding process; choose the output folder by selecting the "Output" tab and by clicking the "Search" button; start the task by clicking on the button "Start"; check that the file output.txt has been created, and the content is correct. The Host program shipped with this package is written using the QT toolkit. The package does not distribute any libraries of the QT toolkit, so in order to use the Host you have to download the QT SDK version 4.8.x from http- ://qt-project.org/downloads. 3 Complete Functionalities 3.1 Handling Input Your application can receive from the host two types of input: files and parameters. Both information are passed to your application as arguments of exectask, first (input) and second (inputparams) respectively. The parameters are all stored as strings, and it is your responsibility to convert the string to the appropriate representation. In the following example, we convert the parameters needed for an image registration task. Some are converted to integer, others to float or boolean: RegParams params; params.numberofhistogrambins = atoi(inputparams[0].c_str()); params.numberofspatialsamples = atoi(inputparams[1].c_str()); params.useexplicitpdfderivatives = inttobool(atoi(inputparams[2].c_str())); params.initialangle = atof(inputparams[3].c_str()); params.backgroundgraylevel = atoi(inputparams[4].c_str()); The input argument to exectask stores map of lists of ObjectLocators objects. Local copies of the input files can be retrieved by accessing the m_infilepaths field after setting the m_indir field and calling either the buildinputfiles or the exectask method of the base class. Both method takes as arguments the input variable. The difference between the two methods is that the latter also manages the output: in fact this method internally calls both the buildinputfiles and the buildoutputuris methods. The m_infilepaths is a map which keys will correspond to the names of the input objects defined inside the.json file. Inside the m_in- Dir field we must specify a path to a local directory (relative to the application http server document root) where the files should be stored if using the http protocol; otherwise if we are using the file protocol the location its already defined in the URI field specified inside the ObjectLocator objects: m_indir = "input"; BaseApp::execTask(input, inputparams, outputlocation, results); //buildinputfiles(input); //buildoutputuris(outputlocation, results);

12 3.2 Interruption Points Interruption Points EasyFlow provides a mechanism for the Host to cancel a task currently executed by an app. This is implemented by adding interruption points to the processing code. We show the use of interruption points in the following example. This code consists in a infinite while loop that calls the boost::this_thread::interruption_point() function inside its body. void interruptionpointstest() int i = 0; while (true) std::cout << i << " "; i++; // cancel task test: put an interuption_point boost::this_thread::interruption_point(); 3.3 Initialise and Release Resources During the execution of a task it is common to allocate new resources (in memory and on disk), and it is important to release all the resources used by a task when it is completed or canceled. To the aim of managing and keeping track of the resources used, the BaseApp class provides three methods: inittask, releasetask and register- UsedFiles. The user may implement the inittask method in order to initialize any resource before execution of the task. The method is called before exectask is called. Any resource (in-memory objects or files) other than the output files that is allocated by inittask or exectask must be deallocated in releasetask, and it is your responsibility to implement releasetask accordingly. EasyFlow will take care of the output files once the Host has retrieved them. As an example, the following implementation of the method deletes the extra local files that the task generates, in addition to the output file. The application kept track of such files and stored them in m_maskimgspath. bool Application::releaseTask() std::cout << "A: Releasing task" << std::endl; std::vector<std::string>::const_iterator cit = m_maskimgspath.begin(); for(; cit!= m_maskimgspath.end(); cit++) boost::filesystem::path filepath = boost::filesystem::path((*cit)); if (boost::filesystem::exists(filepath)) boost::filesystem::remove(filepath); //boost::filesystem::path httpworkitemdir = boost::filesystem::path(m_doc_root) / boost::filesystem::path(m_workitemuuid); //if (!boost::filesystem::exists(httpworkitemdir)) // boost::filesystem::remove_all(httpworkitemdir); return true; Finally, in order to keep track of disk usage, the application should register all the local files is using. To this aim call the BaseApp::registerUsedFiles method, either passing as argument a string or a list of strings; in both cases the strings must represent file paths. In this example we register both the output result and the two mask images. registerusedfiles(m_maskimgspath); 4 Where to Go from Here

13 4.1 Deploying your app Deploying your app In order to deploy your application you need to write a description file that will be used as interface to the Host. Thanks to this file, the application will be registered to the Host, adding it to its pool of available applications and setting up all the parameters needed both to start the communication between application and host and to execute a particular task assigned to the application. In general an application can have more than one description files (i.e. one per task) that may be stored locally, on remote servers providing the services, or on a public "marketplace". The description files and the applications may also be on different locations. The description file must be in JSON format, with the following structure: Path to the application executable. "programpath": String, Input arguments needed by the application, respectively: IP address or domain (host name) of the application web service and http server IP address or domain (host name) of the host web service and http server TCP port of the host web service server Path to the application http server document root Other arguments if needed "programargs": [ String, String, Int, String,... ], IP address or domain (host name) of the application web service and http servers. "address" : String, Name of the task to be executed "name" : String, Description of the task to be executed "description" : String, List of structures providing specific information for each input passed to the application, respectively: Name of the input Description of the input List of MIME types supported by the application for the input List of modalities supported by the application for the input Flag indicating if the input is composed of one or more files "inputs" : [ "name" : String, "description" : String, "supportedmimetypes" : [ MimeType,... ], "supportedmodalities" : [ String,... ], "single" : Bool,... ],

14 4.2 Further information 12 List of structures providing specific information for each output returned by the application, respectively: Name of the output Description of the output List of MIME types supported by the application for the output List of modalities supported by the application for the output "outputs" : [ "name" : String, "description" : String, "supportedmimetypes" : [ MimeType,... ], "supportedmodalities" : [ String,... ],... ], List of structures providing specific information for each parameter required by the application, respectively: Description of the parameter Type of the parameter (possible values: char, string, bool, int, float, double) Minimum value allowed for the parameter Maximum value allowed for the parameter Default value of the parameter "parameters" : [ "description" : String, "type" : String, "min" : String, "max" : String, "defaultvalue" : String,... ] For an example of a compiled description file see the Compile the project section. 4.2 Further information Further information on EasyFlow can gained from the source code reference HTML documentation, generated by Doxygen. It can be found in docs/api_reference.