FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software

Transcription

1 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software Vicraj Thomas, Andrew McMullen, and Lee Graba Honeywell Laboratories, Minneapolis MN 55418, USA Vic.Thomas@Honeywell.com Abstract. FT-Java is a Java language based framework for building fault-tolerant distributed software. It is designed to bring to the system programmer much of the flexibility provided by reflective languages and systems without the attendant difficulties of reasoning about correct system structure. FT-Java achieves this by providing the programmer an extremely flexible programming model with sufficient structure to enable reasoning about the system. The FT-Java framework in turn uses the power of reflection to implement the programming model. 1 Introduction An increasing number of mission-critical distributed systems are being developed using the Java programming language. Examples range from aerospace applications such as portions of the software on NASA s Mars Pathfinder and Odyssey missions to financial applications such as the Den norske Bank s Internet banking facility. Its use in critical applications will only increase as implementations of the Real-Time Java Specification [3] become available. FT-Java is a Java language framework for building fault-tolerant distributed software software that must continue to provide service in a multicomputer system despite failures in the underlying computing platform. The framework supports a variety of fault-tolerance design patterns (software structuring techniques) because Java is used to build a variety of different types of systems; different patterns are appropriate for different types of systems or even different parts of the same system. FT-Java recognizes the expressive power of reflection [4] in building systems that support a variety of fault-tolerance design patterns. FT-Java also recognizes reflection can be a double-edged sword: the same flexibility that allows for the implementation of different system structures also allows for the system to be structured in many incorrect ways. This flexibility also makes it very difficult to reason about the runtime behaviour of such systems. FT-Java provides system developers with a framework that reaps the benefits of reflection and yet imposes sufficient structure to permit reasoning about program behaviour. This is achieved by means of a programming model that shields developers from the complexities of reflection while still being flexible R. Meersman and Z. Tari (Eds.): OTM Workshops 2003, LNCS 2889, pp , c Springer-Verlag Berlin Heidelberg 2003

2 900 V. Thomas, A. McMullen, and L. Graba enough to use a variety of fault-tolerance design patterns. The FT-Java framework itself uses reflection to provide this flexible programming model but this reflection is not visible to system developers. FT-Java supports the fail-stop modules programming model [12]. This model supports diff-erent fault-tolerance design patterns including replicated state machines [13] and restartable actions [11]. FT-Java uses Java reflection to implement this programming model. It does not require any extensions to the Java language or runtime and is therefore easily deployed on any system with a standard Java virtual machine. 2 The Fail-Stop Modules Programming Model The fail-stop failure model for processors [11] has proven to be a very convenient abstraction for analyzing the behaviour of distributed systems in the face of processor failures. This model assumes processors produce correct results until they fail and the failure of a processor is detectable by other functioning components. The fail-stop modules programming model extends this abstraction to software modules. A fail-stop module (or FS module) is an abstract unit of encapsulation. It implements a collection of operations that may be invoked by other FS modules. When an operation is invoked it executes to completion as an atomic unit, despite failures and concurrent invocations. The failure resiliency of an FS module is increased either by composing modules to form complex FS modules, or by using recovery techniques within the module itself. Replicating a module N times on separate processors to create a high-level abstract module that can survive N 1 failures is an example of the former, while including a recovery protocol that reads checkpointed state is an example of the latter. A key aspect of FS modules is failure notification. Notification is generated whenever a failure exhausts the redundancy of a FS module, resulting in a failure of the abstraction being implemented. The notification can be fielded by other modules that use the failed module so they can react to the loss of functionality. The failure notification and composability aspects of FS modules are what makes this programming model so useful. Ideally, a fault-tolerant system as a whole behaves as an FS module: commands to the program are executed atomically or a failure notification is generated. This assures users that, unless a failure notification is generated, their commands have been correctly processed and any results produced can be relied upon. Such a program is much easier to design and implement if each of its components are in turn implemented as FS modules. Since the failure of any component is detectable, other components do not have to implement complicated failure detection schemes or deal with the possibility of erroneous results produced by failed components. These components may in turn be implemented by other FS modules, and this process continued until the simplest components are implemented by simple FS modules. At each level, the guarantees made by FS modules simplify the composition process [12]. The fail-stop module programming model does not assume fail-stop semantics for the underlying computation platform. The responsibility of detecting

3 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software 901 platform failures is delegated to failure detectors. Different types of failure detectors are used depending on the failure model assumed for the computation platform. 3 FT-Java Overview In the FT-Java framework FS modules are Java classes that extend an abstract class FSObject. Operations on FS modules are implemented as methods on one or more FSInterface interfaces. The atomicity of execution of operations may be ensured by using Java s synchronized modifier on methods of the FSInterface interfaces. Complex FS modules with greater failure resilience are formed by composing other FS modules. In addition to standard Java composition techniques such as object containment and delegation, FT-Java allows FS modules to be composed by replication. Multiple instances of an FS module may be composed as a group with FT-Java providing the illusion the group is a single FS module. Operations invoked on the group are multicast to all members of the group and failure notifications are generated when all group members have failed. Finally, complex FS modules may also be implemented using the restartable actions pattern [11] where failed modules are restarted from checkpointed state. Two kinds of failure notifications may be generated when an FS module fails: (1) a synchronous notification, generated when a method is invoked on the failed FS module and (2) an asynchronous notification that is generated even if no method is being invoked on the failed module. Synchronous failure notifications are implemented as Java exceptions all methods on FSInterfaces throw an exception that must be caught or propagated by callers of these methods. Asynchronous failure notifications follow the event subscriber model used by Java Swing components. Modules ask to be notified of the failure of an FS module by invoking its addfailurelistener method. This method takes as an argument a FailureListener interface; the failureevent method on this interface is invoked when the FS module fails. 4 Programming with FT-Java 4.1 Defining and Creating FS Modules All FS modules in FT-Java inherit from an abstract class FSObject. An FS module s class is not expected to implement any methods defined by the base class FSObject. However, the constructor of the FS module s class is expected to make a stylized call to the base class constructor. This is illustrated in Fig. 1 that shows a simple FS module. FS modules are created by invoking the newinstance method of the FT- Java class FSObject. The following statement demonstrates the creation of an instance of the BankAccountImpl FS module on a host named Caesar.

4 902 V. Thomas, A. McMullen, and L. Graba class BankAccountImpl extends FSObject implements BankAccount { private int currentbalance = 0; /* class constructor */ public BankAccountImpl(ActivationID actid, MarshalledObject args) throws RemoteException { super(actid, args); /* method from interface BankAccount */ public synchronized int makedeposit(int deposit) throws RemoteException { currentbalance += deposit; return currentbalance; /* implementations of other methods */... interface BankAccount extends FSInterface { public synchronized int makedeposit(int deposit) throws RemoteException;... Fig. 1. A simple FS module BankAccount custaccount = (BankAccount)FSObject.newInstance("BankAccountImpl", new String[] {"Caesar"); The first argument to newinstance is the name of class of the FS module and the second argument is an array with the name of the host on which the FS module is to be created. Of course, the name of the host would typically not be programmed into the application but would instead be obtained from a configuration application, file or database. The newinstance method is also used to create a new FS module by composing other FS modules by replication. The following statement shows the creation of a more failure resilient version of the BankAccountImpl FS module by creating instances on two different hosts. BankAccount custaccount = (BankAccount)FSObject.newInstance("BankAccountImpl", new String[] {"Caesar", "Czar");

5 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software 903 The first argument is the class of the FS module and the second argument is an array of names of hosts on which instances are to be created. Note the reference to the replicated group returned by the newinstance method is indistinguishable from a reference to a single object instance. 4.2 Invoking Methods on FS Modules A method invocation on an FS module is no different from a method invocation on a regular Java object. This is true regardless of whether the FS module is replicated or not. The following code fragment shows the invocation of the method makedeposit on the BankAccount FS module created in the previous section. try { custaccount.makedeposit(100); catch (RemoteException e) { System.err.println("Invocation failed... ); Note that all methods on FS modules can throw a RemoteException. This exception must therefore be caught or propogated. 4.3 Failure Notification Asynchronous notification of the failure of an FS module is obtained by registering a failurelistener for the module by invoking its addfailurelistener method. Declaration and implementation of the addfailurelistener method is transparent to the FS module developer as it is defined on an interface implemented by the FSObject base class. The following code fragment illustrates the creation and registration of a failure listener for the FS module used in previous examples. custaccount.addfailurelistener(new FailureListener { public void failureevent() { System.out.println("BankAccount object failed."); In the case of an FS module composed by replication, failure notifications are generated only when all instances have failed. A synchronous failure notification is generated when a method invocation on an FS module fails or if a method is invoked on a failed FS module. This failure notification is a Java exception that must be caught and handled using the Java try/catch statement, as shown in Sect. 4.2.

6 904 V. Thomas, A. McMullen, and L. Graba 5 FT-Java Implementation 5.1 Overview FT-Java is implemented using version of the Java Platform Standard Edition. It is implemented as a faulttolerance Java package. The implementation of FT-Java makes extensive use of the Java Reflection API [2] and the Java Remote Method Invocation (RMI) package [6]. The Java Reflection API includes a Proxy class that provides methods for creating dynamic proxy classes and instances. A dynamic proxy implements a list of interfaces specified when the proxy instance is created. Associated with the proxy instance is an invocation handler object. Invocations on the dynamic proxy are dispatched to the invoke method of the invocation handler object. This handler processes the method invocation as appropriate and the result it returns is returned in turn as the result of the method invocation on the dynamic proxy instance [14]. As described later in this section, all invocations on FT-Java FS modules go through a dynamic proxy that is co-located (in the same Java virtual machine) with the caller. The Java RMI package is used to support distributed object creation and invocation. The package defines an Activatable class whose subtypes can be created remotely and activated on demand. It also defines a Remote interface; all remotely invocable interfaces must inherit from this interface. As described later in this section, all FS modules are a sub-type of the Activatable class and all FSInterfaces inherit from the Remote interface. The FT-Java architecture is shown in Fig. 2. Three services run on all hosts of a FT-Java system: the Java RMI Registry, the Java RMI Daemon, and a fault-tolerance manager. The RMI Registry and RMI Daemon are part of the standard Java platform; the former is the Java name service and the latter supports the creation of new Java virtual machines (JVM) at runtime and the creation of Java objects in JVMs on the request of objects in other JVMs. The fault-tolerance manager service, called the HostFTManager, is responsible for managing the Java virtual machines on the host. It uses the Java RMI Daemon to create new virtual machines and monitors these virtual machines for failure. Associated with the HostFTManager service is a detector for failure of other hosts in the system. The failure detector to be used is specified as a system parameter when the HostFTManager service is started up. FT-Java allows multiple applications to co-exist. An application is a collection of related Java objects. On a given host an application is a process consisting of a JVM and all the objects on that host that belong to that application. By default, an FS module is created in an application of the same name as the creator of the FS module. This can be overridden by a parameter on the FSObject.newInstance method used to create FS modules. 5.2 FS Module Creation The newinstance method on FSObject used to create FS modules takes as arguments the Java class name for the FS module, a list of hosts on which

7 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software 905 Host 1 Host 2 VM for App 1 VM for App 2 VM for App 1 VM for App 2 FS Modules FS Modules Host Failure Detector RMI Daemon Host Failure Detector RMI Daemon RMI Registry RMI Registry HostFTManager HostFTManager Fig. 2. FT-Java Architecture instances of the FS module are to be created, a list of backup hosts on which new instances are to be created if an active instance fails, and the name of the application that owns the FS module. Not all of these parameter values have to be specified reasonable defaults are used for unspecified parameter values. The following are the major steps executed by FSObject.newInstance. 1. For each host on which an instance of the FS module is to be created: a) Use the Java name service (RMI Registry) to find the HostFTManager on the host. b) Invoke the createobject method on the HostFTManager with the application name and class name of the FS module as parameters. The HostFTManager.createObject() method on the host on which the FS module is to be created executes the following major steps: i. Determine if a JVM already exists for the application on that host. ii. If a JVM does not already exist, create one using the activation services provided by the Java RMI daemon. iii. Activate an instance of the FS module class and return a reference to the instance. (The instance will actually be instantiated on the first invocation on the object.) c) Save reference to instance returned by HostFTManager.createObject 2. Use the Java reflection API to get a list of all FS interfaces (interfaces of type FSInterface) implemented by the FS module. 3. Create a dynamic proxy with the above FS interfaces. 4. Return reference to the proxy.

8 906 V. Thomas, A. McMullen, and L. Graba Client FSObject Client FSObjectProxy Client FSObjectProxy VM 1 VM 2 VM 3 Fig. 3. Method invocations on an FSObject intercepted by FSObjectProxy objects 5.3 Invocations on FS Modules A method invocation on an FS module is essentially a method invocation on a local proxy to the FS module. The proxy class delegates all invocations to a handler called FSObjectProxy. The handler maintains a list of references to all object instances that comprise the FS module. The handler invokes the appropriate method on all of the object instances and returns to the caller the value returned by one of the invocations. Figure 3 shows the invocation of a method on an FSObject instance via FSObjectProxy instances. 5.4 Registering for Failure Notifications Any object can ask to be notified of the failure of an FS module by invoking the addfailurelistener method on the FS module. This method is defined on a FailureNotification interface. The FSObject class implements this interface and since all FS modules inherit from FSObject they automatically implement the FailureNotification interface. The addfailurelistener method takes as its argument a FailureListener interface. The failureevent method on this interface is invoked on the failure of the FS module for which it is a listener. Invocations of addfailurelistener method on an FS module are handled differently from invocations of other methods on the FS module. As with other invocations, this method invocation is delegated by the module s proxy to its FSObjectProxy handler object. The handler notices the addfailurelistener method is being invoked and performs the following steps: 1. For each instance of this FS module: a) Create an InstanceFailureHandler object to handle failures of this instance. b) Ask the local HostFTManager for notification when the remote host on which the instance exists fails by invoking its addhostfailurelistener method with the InstanceFailureHandler object and the remote host name as arguments.

9 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software 907 c) Use the Java name service (RMI Registry) to find the HostFTManager of the remote host on which this instance of the FS module lives. d) Ask the remote HostFTManager for notification of the failure of the application (JVM) of the instance by invoking its addvmfailurelistener method with the InstanceFailureHandler object and application name as arguments. 5.5 Failure Detection and Notification Failure Detection. FT-Java uses two levels of protocols to detect the failure of FS modules: one to detect failures of Java virtual machines and another to detect failures of hosts. Each HostFTManager is responsible for detecting the failure of virtual machines for applications on its host. A simple heartbeat protocol between the HostFTManager and an AppFTManager in each of the virtual machines is used to detect these failures. A second protocol is used to detect the failure of hosts in the system. Associated with each HostFTManager is a HostFailureDetector responsible for monitoring the health of other hosts in the system. FT-Java does not dictate a host failure detector protocol: it is designed to accommodate different failure detectors depending on the failure semantics of the host compute platform. The class name of the specific failure detector to be used is specified as a system parameter; this class must be of type HostFailureDetector. A simple heartbeat based protocol that assumes a crash failure model for hosts is distributed with FT-Java. Failure Notification. An FS module instance fails when the virtual machine on which it is executing fails or when the host on which it is executing fails. When a HostFTManager detects the failure of a virtual machine on its host, it invokes the failureevent method on all of the registered InstanceFailureHandlers for all FS modules on the failed virtual machine. Recall these failure handlers were registered by proxies for these FS modules using the addvmfailurelistener method of the HostFTManager. These proxies may be on any host in the system. When a HostFTManager is informed of the failure of a host in the system, it invokes the failureevent method on all registered InstanceFailureHandlers for all FS modules on the failed host. Recall these failure handlers were registered by proxies local to this HostFTManager by invoking its addhostfailurelistener method. A client that asked to be informed when an FS module fails is notified only when all instances of the FS module have failed. When an instance of an FS module fails, InstanceFailureHandlers for the instance are notified as described above. The InstanceFailureHandlers simply mark the instance as having failed. The failureevent methods of the FailureListeners for the FS module are invoked only when all instances of the FS module are marked as having failed.

10 908 V. Thomas, A. McMullen, and L. Graba A synchronous failure notification (Java exception) is raised when a method in invoked on a failed FS module i.e. all instances of the FS module are marked as having failed. 5.6 Managing State Consistency across Replicated FS-Modules A future version of FT-Java will include better support for managing state consistency across replicated objects. We have designed a scheme that enables the easy insertion of any group message ordering protocol to manage state consistency. In this scheme, shown in Fig. 4, message ordering protocols are implemented as dynamic proxies (GroupProxy objects) that front instances of a replicated FS module. An invocation on a FS module instance is trapped by its GroupProxy. The proxy communicates with other corresponding GroupProxy instances to agree on a consistent message order. All GroupProxy instances pass invocations on to their corresponding FS module instances in the agreed upon order. Note that the message ordering protocol is completely transparent to the FS module being replicated and to the clients that invoke methods on the FS module. 5.7 Performance The overhead associated with invoking a method on an FS module via the dynamic proxy over a direct invocation on a remote object was found to be less than one millisecond, the granularity of Java s standard timing mechanism. The Client GroupProxy FSObject FSObjectProxy VM 1 Ordering Protocol VM 2 Client GroupProxy FSObject FSObjectProxy VM 3 VM 4 Fig. 4. Scheme for implementing message ordering protocols

11 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software 909 time to directly invoke a method on a remote FS module was 3 milliseconds. Going through the proxy did not measurably increase this invocation time. The above times were measured by having a client repeatedly invoke a method on an FS module on a different node. The nodes were 1.2GHz Pentium III class machines running Windows XP and connected by an 11Mbps wireless Ethernet network. 6 Related Work Many research projects have explored the use of reflection for building faulttolerant systems. These projects define meta-objects that reify dependability related properties of the functional objects (or base objects) of the system. They also define a meta-object protocol (MOP) for manipulating these meta-objects to modify the runtime behaviour of corresponding base objects. Maud [1] uses reflection to allow a running system to dynamically switch between different dependability modes of execution by installing and removing dependability protocols. The FRIENDS project [5] showed how MOPs may be used to implement a number of fault-tolerant mechanisms including replication, synchronization, and voting. A MOP for a fault-tolerant CORBA system is described in [8]. In contrast with the above reflection based systems, FT-Java shields the system developer from the complexities of MOPs. In fact, the developer does not have to know anything about reflection and MOPs to use the FT-Java framework. The programming model seen by the developer is the fail-stop modules programming model; reflection allows for a simple and elegant implementation of support for this model within FT-Java. Systems that provide mechanisms for creating highly-available modules by replication include the Object Management Group s Fault-Tolerance Specification for CORBA [7] and the Aroma System [10]. The fault-tolerance specification for CORBA defines a programming language independent mechanism for deploying highly-available applications. It provides mechanisms for replicating CORBA servers but, unlike FT-Java, it does not define a general programming model for building highly-available applications. The fail-stop module programming model of FT-Java is such a model with replication being just one technique for building fault-tolerant software. In addition, the FT-Java framework helps system designers reason about the structure and behaviour of the system and requires them to deal with both anticipated and unanticipated failures of system modules. The Aroma System [10] provides mechanisms for creating and managing replicated Java RMI objects. It differs from FT-Java in two significant ways: (1) it is implemented as a software layer embedded within the transport layer of the Java RMI stack, and (2) it does not provide system designers a framework that supports fault-tolerance techniques other than replication. In contrast, FT- Java requires no extensions to the Java language or virtual machine and supports programming paradigms such as restartable actions in addition to replication.

12 910 V. Thomas, A. McMullen, and L. Graba 7 Summary and Future Work FT-Java provides developers of Java-based fault-tolerant software systems a structured way of harnessing much of the power of reflective systems. Its mechanisms integrate well with existing Java constructs, making them easy to learn and use. FT-Java does not make any extensions to the Java language or runtime. Much of the implementation of FT-Java is complete and is available as a Java package called faulttolerance. Two major FT-Java features that have been designed but have yet to be implemented are the restart of failed FS modules on backup hosts and the detection of the failure of an application (JVM) on a host. Future versions of FT-Java will include better support for managing state consistency across replicated FS module as described in Sect A related direction for this work is support for other replication models such as passive replication. An orthogonal direction is the verification of our hypothesis that the fail-stop failure model seen by the programmer can be maintained even if the host failure model changes from crash failures to Byzantine failures [9]. Naturally, a different host failure detector protocol will be needed for different failure models but this is hidden from the programmer. References 1. G. Agha, S. Frølund, R. Panwar, and D. Sturman. A linguistic framework for dynamic composition of dependability protocols. In Proceedings of the IFIP Conference on Dependable Computing for Critical Applications, Sicily, K. Arnold, J. Gosling, and D. Holmes. The Java TM Programming Language, Third Edition, chapter 11. Addison-Wesley Publishing Company, June G. Bollela, J. Gosling, B. Brosgol, P. Dibble, S. Furr, and M. Turnbull. The Real- Time Specification for Java. Addison-Wesley Publishing Company, January F.-N. Demers and J. Malenfant. Reflection in logic, functional and object-oriented programming: a short comparative study. In Proceedings of the IJCAI 95 Workshop on Reflection and Metalevel Architectures and their Applications in AI, pages 29 38, August J.-C. Fabre and T. Perennou. A metaobject architecture for fault-tolerant distributed systems: The FRIENDS approach. IEEE Transactions on Computers, 47(1): 78 95, W. Grosso. Java RMI. O Reilly and Associates, October Object Management Group. Common Object Request Broker Architecture: Core Specification, chapter 23. Object Management Group (OMG), December M.-O. Killijian and J.-C. Fabre. Implementing a reflective fault-tolerant CORBA system. In Symposium on Reliable Distributed Systems, pages , L. Lamport, R. Shostak, and M. Pease. The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3): , July N. Narasimhan, L. Moser, and P. Melliar-Smith. Transparent consistent replication of Java RMI objects. In 2nd International Symposium on Distributed Objects and Applications (DOA 2000), pages 17 26, September 2000.

13 FT-Java: A Java-Based Framework for Fault-Tolerant Distributed Software R. Schlichting and F. Schneider. Fail-stop processors: An approach to designing fault-tolerant computing systems. IEEE Transactions on Computing Systems, 1(3): , August R. Schlichting and V. Thomas. Programming langauge support for writing faulttolerant distributed systems. IEEE Transactions on Computers, 44(2): , February F. Schneider. Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Computing Surveys, 22(4): , December Sun Microsystems. Java TM 2 Platform, Standard Edition, v 1.4.1, API Specification,