Thread-based Benchmarking Deployment

Transcription

1 Thread-based Benchmarking Deployment Sébastien Lugan and Benoît Macq Laboratory of Telecommunications and Remote Sensing, Université catholique de Louvain, Place du Levant, 2, B-1348 Louvain-la-Neuve, Belgium ABSTRACT Information and knowledge are actually well and easily distributed thanks to electronic journals, news, mailinglists and forums. It is however more difficult to deploy algorithmic and programming collaboration: the heterogeneity of the programming languages and the operating systems used by researchers constitutes a major problem to the design of a common testing platform. Current solutions exist to develop collaborative work. Generally, these solutions impose specific programming languages and/or operating systems to developers. Some others specific rules have to be respected. These heavy constrains slow down the usage of collaborative programming platforms. The OpenWatermark project proposes a modern architecture for cooperative programming exchanges that takes all these aspects into account. Developers work with their favorite programming languages and operating systems as long as the OpenWatermark platform supports them. In this paper, we will present the OpenWatermark platform ( and its application to the benchmarking of image watermarking algorithms. 1. INTRODUCTION New ways to represent efficiently the characteristics of the watermarking algorithms are under development. Instead of delivering certification for watermarking, the authors are convinced that the benchmarking still needs further research. They promote the use of a cooperative web-based approach because key factors like quality evaluation under geometrical transforms and capacity evaluation still need to be better understood. A first benchmarking approach was developed in Stirmark. New developments of Stirmark are attempting to deploy the suite on the Web 12. The perceptual effects of complex transforms, the capacity evaluation and the resistance against attacks are still open issues for research. This is the reason why the authors are promoting an open-source web-based cooperative working area. 2. OPENWATERMARK OpenWatermark is a distributed application system whose initial purpose is to allow the execution and the comparison (i.e. the benchmarking) of programs uploaded by the user. The user first logs into a web site using her/his preferred web browser, fills a form where she/he is asked to explain some characteristics (such as the programming language used, the syntax of its command-line arguments) of her/his program, and to upload it. Those characteristics, programs as well as the input data sets are stored into an SQL database. The OpenWatermark system determines on which machine the execution should be scheduled and the context of those executions, that is to say mainly which data sets and parameters should be used, and what kind of output should be expected. It then connects to the concerned machines and requests them to download the executable from the database as well as the associated data sets, runs it using the previously specified command-line options, and upload back to the database the results of this execution. Finally, the results could be consulted by the user on the same web site as soon as they become available. The user interface is entirely constituted of web pages written in JSP (Java Server Page), and communicates with the Java application responsible for the execution of the tested program running on each of the benchmarking hosts using RMI (Remote Method Interface) and with the SQL server using JDBC (Java Database Connectivity). This architecture is therefore independent of the operating system used, and hence easily portable on any platform supporting Java, RMI and JDBC.

2 In addition to allowing the upload of the programs and their characteristics, several forms also allow the elaboration of benchmarking templates. Those templates are constituted of blocks and relations composing a block diagram. Each of the blocks represent either a data set, an option (e.g. a compression or quality parameter) or a program. When the user selects templates to be run, the JSP script launches several threads managing the required executions. Browsing the block template, these threads evaluate how many executions of each program (with specific data sets and parameters) are required, and what their dependencies are. They then request the benchmarking hosts through RMI calls to fetch the corresponding data from the database into files which will then be passed as command-line arguments (in addition to the other mandatory or optional parameters) to the program to be executed. As soon as the execution successfully ends, if it did generate an output file, it is then uploaded by the local Java server into the database using JDBC. The state of the execution (finished or not) could be seen from the HTML user interface. The already available results are directly accessible either for download or (in the case of images) view from the user s web browser. This platform-independent architecture thus constitutes a generic framework which could be used in order to comparatively evaluate the efficiency of test programs on various data sets with reusable evaluation schemes. The set of programming rules that OpenWatermark imposes are quite easy to follow: the program must use command-line arguments, e.g.: embedmark -i lena.jpg -m message -f optionsfile.txt and must not rely on any interaction with the user; the command-line arguments may be given in any order; the program must produce a unique result, either a file into the working directory or a message printed on the standard output. The user then builds schemes of executions through web forms. First, she/he selects either her/his own executable or any executable shared by other users, then she/he associates data to the executable inputs, and presses the run button. As soon as the results become avalaible, the user could be able to access them directly from the same web-based interface. These rules allow the OpenWatermark application to execute the user-provided programs following the template (or execution scheme) selected or defined by the user: the OpenWatermark application fetches data from the database and writes it into a temporary directory on a machine running the appropriate operating system (and host application if required), invokes the program to be tested with the appropriate command-line arguments and finally stores the results back into the database. 3. INTERFACE INTUITIVENESS Web-based graphical user interfaces are becoming more and more popular. OpenWatermark users build execution schemes and manage the privacy of their data using web forms, and the results of any execution can be consulted from the same site as soon as they become available, in different formats (depending on their nature). As illustrated on figure 2, the user could upload virtually any kind of input data (pictures, text files, etc.). The user then uploads her/his executables, specifying which programming language, operating system and/or host application are needed to run them, and whether or not they could be used by other users (figure 3). The next step (figure 4) is the definition of the options that must be associated with each of the executables. As illustrated in figure 5, the user builds a block by associating data and/or output blocks to each input of the considered executable (either uploaded by the user herself/himself or shared by another). Step by step, the user builds an execution scheme from the association of the blocks (figure 6). Once the user has decided to run a blocks list, a set of executions is scheduled and sent to the appropriate machines. As soon as they become available, the results could be viewed in different formats.

3 Figure 1. OpenWatermark use case Figure 2. Data form

4 Figure 3. Executable form Figure 4. Executable options form

5 Figure 5. Block form Figure 6. Blocks list form

6 Figure 7. Results forms Figure 8. OpenWatermark component model

7 4. ARCHITECTURE OpenWatermark is a distributed application. It is composed of three main parts (figure 8). The web server and an RMI client constitute the first part. OpenWatermark s web pages are generated with JSP (Java Server Page). The client binds the web site with the benchmarking servers which will execute the programs. The benchmarking servers constitute the second part of the architecture. Each machine is able to run executables developped in various specific programming languages, or that must be run under interpreters (e.g.: matlab, python) and/or specific operating systems. When the OpenWatermark user decides to run a program through the web site, the language and the operating system compliance are checked and the RMI client associates it with an appropriate benchmarking server. This machine fetches all of the needed files (data sets and executable) from the OpenWatermark database. When the execution is finished, the result and any generated message are stored back into the database. The third part is the SQL server. It centralises the OpenWatermark database where information about web activity and execution processes are stored. 5. JAVA IMPLEMENTED APPLICATION OpenWatermark is fully implemented in Java. Each part is therefore compatible with any operating system which is able to run a Java Virtual Machine supporting RMI and JDBC. Java has been chosen for its object-oriented language and its ability to offer highly maintainable code. Java also comes with a complete API, making it easy to implement each part of this framework. The web server is running a free container (Tomcat4 and Apache); JDBC binds the SQL server to the other parts, and RMI (Remote Method Interface) virtually connects the user with the machine that runs her/his programs. The dynamic web pages are implemented in JSP. The rest of the application is done in plain Java. It implements a basic RMI client/server structure. The client (BlockslistManager) is connected to the JSP that manages the blocks list. When a user decides to execute a blocks list, a dedicated java server page sends the identifier of the blocks list to the client. This client checks the blocks list integrity. If it is valid, it generates the associated executions. Each execution is then sent to the associated server in order to be treated. Figure 9. Client side The server side runs the execution request as sent by the client, and stores the result back into the database.

8 Figure 10. Server side 6. CONCLUSION The major part of the dynamic web pages (JSP), the design of the database and the implementation of the client/server application (Java) are done. The current version of OpenWatermark supports C/C++ (Linux and Windows) executables, Python and Matlab scripts. The remaiming work is to improve the design of the web interface, test the benchmarking platform with real users and watermarking executables, propose a secure configuration for the web server and the SQL server. REFERENCES 1. J. Dittmann, A. Lang, and M. Steinebach, Stirmark benchmark: Audio watermarking attacks based on lossy compression, in Proceedings of SPIE Vol. 4675, Photonics West 2002, January 2002, Electronic Imaging 2002 Science and Technology; Multimedia Processing and Applications, Security and Watermarking of Multimedia Contents IV, San Jose, CA, USA, pp , Edward J. Delp III, Ping Wah Wong, (Bellingham, Washington, USA), A. Lang, J. Dittman, and M. Steinebach, Psycho-akustsiche modelle für stirmark benchmark - modelle zur transparenzevaluaierung, in INFORMATIK Mit Sicherheit Informatik, Schwerpunkt Sicherheit - Schutz und Zuverlässigkeit, 29. September - 2. Oktober 2003 in Frankfurt am Main. LNI 36 GI 2003, pp , Rüdiger Grimm, Hubert B. Keller, Kai Rannenberg, M. Kutter and F. Petitcolas, A fair benchmark for image watermarking systems, in Proceedings of SPIE Vol. 3657, Security and Watermarking of Multimedia Contents, pp , F. A. P. Petitcolas, M. Steinebach, J. Dittmann, C. Fontaine, F. Raynal, and N. Fatès, A public automated web-based evaluation service for watermarking schemes: Stirmark benchmark, in Proceedings of SPIE Vol. 4314, Security and Watermarking of Multimedia Contents III, pp , Edward J. Delp III, Ping Wah Wong, 2001.