Do! environment. DoT

Transcription

1 The Do! project: distributed programming using Java Pascale Launay and Jean-Louis Pazat IRISA, Campus de Beaulieu, F35042 RENNES cedex Abstract. The aim of the Do! project is to ease the task of programming distributed applications using Java. In a rst step, the programmer writes his application as a shared-memory parallel program, by dening the components of the program; in a second step, the programmer denes the component locations on distinct hosts; in a third step, our preprocessor transforms the shared-memory parallel program into a distributed-memory program. This paper is an overview of the Do! project. 1 Introduction { overview of the Do! project Writing distributed applications is known to be dicult; many problems have to be taken into account at the same time such as location and access to data and network protocols details. Implementing communications between objects located on distinct hosts is eased in Java by using the Socket class or the Java Remote Method Invocation (rmi) package [8]. However object locations still have to be taken into account and objects are not transparently accessed if they are distant or local. We think that there is a need for higher level tools to ease the programming of distributed applications in Java. The Do! environment (gure 1) provides an easy way to write distributed applications in Java; it comprises a parallel framework 1 [9] used to write parallel programs, a distributed User-defined components Preprocessor Do! environment Parallel framework DoT Distributed framework Do! runtime javac javac rmic JVM JVM JVM JVM Parallel program Distributed program Fig. 1. the Do! environment 1 the framework approach is introduced in section 1.2

2 framework to express distributed programs, a runtime and a preprocessor (DoT) to transform parallel programs into distributed programs. The frameworks and the runtime are composed of standard Java class libraries, without any extension to the language; our system does not make any byte-code transformation. Therefore Do! parallel and distributed programs can run using any standard Java environment. 1.1 Writing a distributed application In a rst step, the programmer writes an application as a shared-memory parallel program without bothering with the component locations. He/she denes the components of the program: tasks and data objects. The Do! parallel framework manages the cooperation scheme between components: the programmer describes a task execution model and accesses to data by choosing a class in the framework library. In a second step, the programmer denes the component locations on distinct hosts by choosing appropriate layout managers. Layout managers are objects included in the parallel framework to describe the object distribution policy. This allow us to automaticaly generate the distributed program from the parallel program. In the distributed framework, layout managers are used to implement the object distribution policy. In a third step, the DoT preprocessor transforms the shared-memory parallel program into a distributed-memory program. Because a program is composed of framework classes and user dened classes, program transformations consist in changing the framework used in the programs (the parallel framework is replaced by the distributed framework), and transforming components (user-dened classes) in order to allow transparent locations of objects (mapping and remote accesses). 1.2 Programming with frameworks New programming models are usually dened through the extension of existing languages or the denition of new languages. We use the framework approach which is an alternative way to dene programming models without extending a language. Frameworks are especially suitable for object-oriented languages; they can be seen as a model of program or a program with \holes", where components are missing and have to be dened by the programmer to t his application in. Unlike in the library approach where the programmer writes the program structure and uses library classes to implement some specic computations, in the framework approach, the program structure is dened by the framework, and the programmer provides the components implementing the program's specic computations (gure 2). framework user-defined program user-defined components library components Fig. 2. the framework and the library approaches

3 2 Parallel framework The Do! parallel framework is used to express parallel programs. In this paper, we call \parallel program" a program consisting of several control ows sharing a single address space (for example running on a shared-memory multiprocessor). Basic constructs for parallelism already exist in Java at the language level (the Thread class) but we think that there is a need for a more structured parallel programming model in Java. The parallel framework is based on the notions of active objects (tasks) (section 2.1) used to introduce parallelism and collections (section 2.2) to structure parallelism. 2.1 Active objects (tasks) An active object is the integration of an object and a process. An active object has its own activity: in a sequential object-oriented program, the control ow runs successively in distinct objects of the program; using active objects, several control ows run concurrently in distinct objects, leading to inter-object concurrency. When active objects access other objects of the program, they can communicate (through shared objects), leading to intra-object concurrency when concurrent control ows run into a single shared object. A task (active object) activation can be synchronous (the control returns to the \activator" after the task termination) or asynchronous (the control returns when the task is activated). Synchronization with a task can be done by waiting the task termination or by stopping the task execution. 2.2 Collections Collections are structured sets of objects such as lists, arrays and trees. Collections are used in the parallel framework to store both passive and active objects and thus to express structured parallelism. The programmer groups parallel tasks into a \task collection" and data into a \data collection". The parallel framework implements a programming model where the tasks grouped in a collection run in parallel, with corresponding data items of a data collection as arguments. We have introduced the class Par as a new class to implement a parallel construct without any extension to the Java language. The class Par implements both the parallel activation of the tasks stored in the task collection with the corresponding data items in the data collection and the synchronization with the parallel task terminations. Nested parallelism is introduced by the fact that all instances of the class Par are active objects (tasks), and can be inserted in a task collection and activated in parallel with other tasks. Using this technique, several parallel programming models can be implemented as framework library classes or user-dened classes. For example, from a data collection and a single task, the DataPar class can be implemented dening a data-parallel programming model: a copy of the task is activated in parallel for each data item of the data collection; from a task collection and a single data item, the SharedPar class can be implemented dening a parallel model where all tasks communicate through the single shared data item. 2.3 Layout managers The parallel framework can be used to write shared-memory parallel programs; but our aim is to generate distributed programs, according to user information about the component locations provided by layout managers. Layout managers are objects included in the parallel framework to describe the collection distribution: the programmer indicates how to distribute the program components (tasks and data) by choosing a layout manager for each collection. Layout managers have no eect in a parallel program, but are used in the distributed framework (section 3.2).

4 2.4 Example: a simple parallel program import do.shared.*; /* parallel framework library */ public class MyTask extends Task { public void run (Object param) { /* task behavior definition */ MyData data = (MyData)param; data.select(criterion); data.print(out); data.add(value); public class SimpleParallel { public static void main (String argv[ ]) { /* task and data collection initialization */ Array tasks = new Array(N); Array data = new Array(N); for (int i=0; i<n; i++) { tasks.add (new MyTask(), i); data.add (new MyData(), i); public class MyData { /* the task parameter */ public void add(...){... public void remove(...){... public void print(...){... public void select(...){... /* Par activation */ Par par = new Par(tasks,data); par.call(); Fig. 3. A simple parallel program Figure 3 shows an toy example of a parallel program written using the parallel framework. This program implements the parallel activation of the tasks stored by tasks (the task collection) with the corresponding data items in data (the data collection) as arguments. This program is composed of three classes: { class MyData is a model of data item, { class MyTask represents a task, { class SimpleParallel contains the main method. To write a parallel program, the programmer rst has to dene the program components: tasks and data. Data are passive objects whereas tasks are active objects. Any object which is not an active object is a passive object: because MyData does not extend Task, instances of MyData are passive objects (data). MyTask represents a model of active object (task) by extending the framework class Task. The task behavior is dened in the run method. The task accesses the data item passed as argument to the run method 2. The parallel activation of tasks is done in two steps. First, the initialization of a task collection and a data collection. In this example, the collections are arrays (the class Array is part of the framework library) initialized with instances of MyTask and MyData. In the second step, an instance of the framework class Par is initialized with the task and data collections as arguments, and activated (par is an active object). 2 as the Java language does not provide us genericity, we have to cast the argument type

5 3 Distributed framework The distributed framework is used to express distributed programs. A distributed program consists of several control ows running on distinct address spaces, for example on distributed-memory machines or on networks of computers. The distributed framework has the same programming interface as the parallel framework. It is based on active objects and distributed collections. 3.1 Distributed collections Distributed collections are structured sets of distributed objects (mapped on distinct hosts). They are used in the distributed framework to store active and passive objects and express distribution: though, parallelism and distribution are integrated through collections. The programmer groups the parallel tasks and data in collections, and species the program distribution by using distributed collections. The distribution policy devolves on a special object called a layout manager (section 3.2). The distributed framework implements a programming model where tasks and data may be mapped on distinct hosts. Like in the parallel framework, the parallel construct exist in the distributed framework, through the Par class. Nested parallelism is possible too. Several distributed programming models can be implemented as framework library classes or user-dened classes, following the same approach. 3.2 Layout managers Distributed collections are responsible for the storage and the accesses to objects mapped on distinct processors. The collection distribution management devolves on layout managers: they are responsible for retrieving the processor owning an element from a key identifying the element (for example for arrays, keys are indexes), and for translating global keys into local keys. A layout manager is associated to each distributed collection. To dene the collection distribution, the programmer has to choose the corresponding layout manager. Specic layout managers are part of the distributed framework library (for example for arrays, cyclic distribution is implemented by the Cyclic class). Several distribution policies can be implemented as several framework classes or user-dened classes. 3.3 Example: simple distributed program Figure 4 shows the previous program transformed to run in a distributed environment. All the code portions in grey are the same as in the parallel program. The classes representing the tasks and data are the same. Instances of MyData are mapped on distinct processors and can be accessed from remote hosts: MyData is an accessible class, implementing the Accessible interface. This class is transformed by our preprocessor to allow transparent remote accesses. The parallel activation of tasks is done in the same way as in the parallel program (collections initialization and Par initialization and activation). The only change is due to the fact that collections are distributed; we have to associate a layout manager to each distributed collection to dene the distribution policy. The classes Cyclic and Block are layout managers describing cyclic and block distribution of arrays; they are part of the framework class library. This program implements the activation of the tasks stored by tasks with the corresponding data items in data as arguments. The tasks and the data are mapped on distinct processors, according to the Cyclic and Block layout managers.

6 import do.distributed.*; /* distributed framework library */ public class MyTask extends Task { public void run (Object param) { /* task behavior definition */ MyData data = (MyData)param; data.select(criterion); data.print(out); data.add(value); public class SimpleDistributed { public static void main (String argv[ ]) { Array tasks = new Array(N); Array data = new Array(N); /* distribution definition */ tasks.attachlayout(new Cyclic()); data.attachlayout (new Block()); for (int i=0; i<n; i++) { tasks.add (new MyTask(), i); data.add (new MyData(), i); public class MyData implements Accessible { /* the task parameter */ public void add(...){... public void remove(...){... public void print(...){... public void select(...){... Par par = new Par(tasks,data); par.call(); Fig. 4. A simple distributed program 4 From parallel to distributed The parallel framework denes the active and passive objects cooperation scheme; additionally, the distributed framework denes the active and passive objects distribution scheme. A program is composed of framework classes and user-dened classes (components). To transform of a parallel program into a distributed program we have to change the framework used in the program and transform components: { framework replacement: the parallel and the distributed frameworks have the same programming interface. The framework used in the program is replaced by only changing the imports in the program (see the imports in the programs in gures 3 and 4); { component transformations: the user-dened classes must be mapped on distinct hosts and communicate with remote objects. Mapping and communications of user-dened components are not managed by the frameworks. We transform components to allow transparent locations of objects: the mapping of objects on remote hosts, and the communications with objects from remote hosts. 4.1 Mapping of objects { remote creations The mapping of an object on a processor is done when the object is created. Collections manage the storage and accesses to elements, but they do not manage element creations. We have implemented runtime classes providing remote creations of objects.

7 In an spmd 3 model, control ows running on distinct hosts execute the same code, so it is possible to insert synchronizations in the program code when needed (between two hosts involved in a remote creation for example). Our model is not spmd, so remote creations rely on servers running on each host. At runtime, two separate threads run on each host: { the thread processing the program computations, { the thread processing remote creations of objects (server). Servers for remote creations have to identify the class of objects that have not been declared locally, and to instanciate those classes at runtime. We must be able to manipulate in the program concepts of the language (such as class, instanciation... ) as rst class objects. For that, we use the reection mechanisms, provided in Java at the language level [7]. 4.2 Communications { remote method invocations The distributed framework manages the cooperation scheme between objects, and the distribution of objects on processors, but does not manage the communications between userdened objects. In object-oriented languages, communications between objects are based on method invocations. We implement communications between objects mapped on distinct hosts through remote method invocations. We use the Java rmi, but this mechanism is not transparent to the user and is dicult to manage for non expert programmers. We transform accessible classes (section 3.3) to allow transparent remote method invocations, using the Java rmi. 4.3 Component transformations Accessible Serializable Unicast Remote Object Remote C accessible class Preprocessor C proxy class C_Impl impl on class RemoteNew C_Static StaticNew static class C_Int C_StInt Fig. 5. Component transformation Figure 5 shows an accessible class transformation. The proxy class has the same programming interface as the source class, but all attributes and method implementations are in the implementation and static classes (the static class contains the static part of the source class). At object creation, a proxy object is created locally, and an implementeation object is created remotely, using our runtime remote creation mechanism (through the remotenew 3 Single Program, Multiple Data

8 method). Thus, the mapping of the object is masked to the user by the proxy object creation, and all accesses to the remote object are catched by the proxy object. An additional argument to the constructor is needed to identify the remote host; this argument is generated by our preprocessor for elements of distributed collections, according to the collection layout manager. The static part of a class is shared by all instances of this class. A single instance of the static object is created, shared by all instances of the proxy class. The staticnew method dened in our runtime implements this specic semantics for object creation in a distributed environment All other classes and interfaces represented on the gure are needed to use the the Java rmi. 5 Related work A classication of ways to introduce parallelism and distribution in object-oriented languages can be found in [3] where three main approaches are identied: { the applicative approach (libraries), { the integrative approach (integration between the notions of objects and concurrency in the core language), { the reective approach (customizing program semantics using reection mechanisms). Our works relies on the three approaches: it is applicative since we use framework libraries; it is integrative because active objects and remote method invocations are part of the Java language; it is reective as we use the Java reection mechanism for remote creations of objects. Many interesting projects using Java are presented in [1, 2]. Here, we highlight some approaches, according to this classication. Integrative approach: in the Javaparty project [10], classes are transformed to allow transparent remote method invocations using the Java rmi (section 4.2), based on an extension to the Java language (remote modier). Unlike us, the mapping of objects is computed through static program analysis and uses a specic runtime. Reective approach: the Java// project [4], is based on active and remote objects, and a high level model of synchronization between objects (wait-by-necessity). This model is based on a meta-object protocol (mop), and can be easily customized. Java// is based on previous works and studies in C++ and Eiel. Applicative approaches: S. Flynn Hummel et al. [5] and the DPJ project [6] introduce data parallelism in Java through class libraries. These projects rely on an spmd model. 6 Conclusion A prototype of the Do! environment is available online 4 ; it implements static collections (arrays) and contains the parallel and distributed frameworks, the runtime and the preprocessor. We are studying the development of a simulation tool for forest growth with our environment. We are currently working on an extension to our model for dynamic collections (such as lists). Our aim is to allow dynamic creation and activation of tasks, by adding new tasks in collections at runtime. This model is already implemented for shared-memory environments; to distribute this model, we have to integrate dynamicity in layout managers (e.g. ability 4

9 to compute element locations at runtime). We plan to dene a dynamic distribution policy that takes into account runtime informations such as ressources availability or load. This technique will allow us to implement and encapsulate dynamic load balancing into layout managers. References 1. Workshop on Java for science and engineering computation; simulation and modelling program. Concurrency: Practice and Experience, 9(6):413{674, June Las Vegas, Nevada. http: // 2. Workshop on Java for high-performance network computing. Concurrency: Practice and Experience, February Palo Alto, California. To appear. java98/program.html. 3. J.-P. Briot, R. Guerraoui, and K.-P. Lohr. Concurrency and distribution in object oriented programming. ACM Computing Surveys, September To appear. 4. D. Caromel and J. Vayssiere. The ProActive Java library for distributed, concurrent and parallel computing V. Getov, S. Flynn-Hummel, and S. Mintchev. High-performance parallel programming in Java: Exploiting native libraries. Workshop on Java for High-Performance Network Computing. Concurrency: Practice and Experience, February To appear. 6. V. Ivannikov, S. Gaissaryan, M. Domrachev, V. Etch, and N. Shtaltovnaya. DPJ: Java class library for development of data-parallel programs. Institute for System Programming, Russian Academy of Sciences, Javasoft. Java core reection { API and specication. ftp://ftp.javasoft.com/docs/jdk1. 1/java-reflection.ps, January Javasoft. Java Remote Method Invocation specication { revision ftp://ftp.javasoft. com/docs/jdk1.2/rmi-spec-jdk1.2.ps, December P. Launay and J.-L. Pazat. A framework for parallel programming in Java. In HPCN'98, LNCS, Springer Verlag, Amsterdam, April To appear. 10. M. Philippsen and M. Zenger. JavaParty { transparent remote objects in Java. In PPoPP, June 1997.