Fault Tolerance in Decentralized and Loosely Coupled Systems

Transcription

1 Fault Tolerance in Decentralized and Loosely Coupled Systems Pekka Nikander Ericsson Research Nomadic Lab* LMF/TT Abstract * Intelligence in networking is gradually shifting from the network nodes to the terminal equipment. As a logical consequence of this development, we will eventually see networks that function without any centralized infrastructure. Personal and interpersonal ad hoc networks will most probably be the first examples of such systems. In this paper, we study fault tolerance from the point of view of these kinds of totally decentralized systems. That is, when a network consists of a number of loosely coupled nodes with intermittent connectivity, the situation is completely different from more traditional, completely connected and centralized systems. More specifically, we focus on ad hoc networks, and the impact of IP version 6 and Jini on the fault tolerance of the network and service layers of ad hoc networks. 1 Introduction One of the fundamental trends in telecommunications is the shift of intelligence from central network nodes to terminal equipment. Eventually, this leads to a situation where the terminal equipment do not require any infrastructure but are able to create networks of their own, based on their situational needs. When short range radio technologies are used, the resulting network consists of an ad hoc mesh of connections. In order to function with a dynamically changing topology and context, the network should be able to automatically and reliably configure itself without any infrastructure support. At the service level, the nodes should be able to find the services provided by each other and the environment. The services also need to survive loss of connectivity without unnecessary disruption of the user experience or leaving behind uncommitted wasted space * Between September 1999 and August 2000, Pekka Nikander was on leave from Ericsson Research and acted as a professor at Helsinki University of Technology (HUT). The main parts of this work were done while at HUT. or other resources. Similar to the network level, the service level must function without on-line support from the infrastructure provided in the fully connected network. In practice, this requires that infrastructure functions, such as service discovery, accounting, authorization, and other management functions, need to be arranged in a decentralized manner. In this paper, we discuss the concept of fault tolerance from the point of view of the network and service layers of ad hoc networks. We especially emphasize requirements that result from using limited range radio technology to create local connections between personal computing and communication devices such as future wireless phones. To illustrate some of our points, we use the current IPv6 [1] and Jini [2] designs as examples. The rest of this paper is organized as follows. First, in Section 2, we present the overall requirements for such an ad hoc network. Next, in Section 3, we briefly describe the most relevant new functionality introduced in IPv6 and Jini. Section 4 outlines some of the architectural aspects involved, and Section 5 discusses their consequences for fault tolerance. Finally, Section 6 contains our conclusions of this research. 2 Requirements In this section, we outline three high level requirements for the networking and service layers. Fulfilling the requirements would allow independent nodes to create networks using limited area radio links in an ad hoc manner, and to use the services provided in such a network, including possible connections to the fixed network infrastructure. Furthermore, the use of the fixed network connections would be possible independent of the location and formation of the ad hoc network, and it would be possible to easily switch over from one connection to another without any unnecessary disruption of services. Thus, the first requirement is that it must be easy to form ad hoc networks locally without help from any ex-

2 ternal infrastructure, and to utilize services provided in such a network. Essentially, this means that the nodes must be able to mutually agree on the network addresses to be used, to dynamically collect topology information for routing purposes, and to include decentralized means for service discovery and security management [3]. As a second requirement, it must be possible to connect the ad hoc networks to the fixed IP infrastructure. There may be several simultaneous connections to the fixed IP world, and should it be possible to use these links concurrently. In any case, it must be possible to switch over from using one link to using another link without any unnecessary disruption of service. As a third high level requirement, it should be possible to use the fixed infrastructure to create a connection with an ad hoc node if the ad hoc node does have active access to the fixed network. In other words, the network architecture should support global roaming and mobility. Technically this requires that, somewhere in the fixed infrastructure, there exists a (possibly distributed) service that allows a permanent identifier of the mobile node to be mapped to its current network address, if such an address is available. However, it is far from clear what the fixed identifiers should be. 3 Base technologies In this section we briefly describe the most relevant features of our example technologies, IPv6 and Jini. 3.1 IP version 6 IP version 6 (IPv6) [1] is the IETF proposed standard for the next generation IP protocol. It is fundamentally different from the current IP, IP version 4 (IPv4), in several respects. The most apparent difference is the address size: IPv6 has 128 bits long addresses while IPv4 addresses are only 32 bits long. Even though there are interesting possibilities created by the longer addresses, e.g., anycast addresses, other aspects are more important when considering fault tolerance. One of the more fundamental features of IPv6 is autoconfiguration [4][5]. Basically, through autoconfiguration any node can create a link local address for itself, and learn the identities of the routers providing connections to elsewhere in the network. Autoconfiguration also allows the node to learn and gradually update its routing prefixes, which may either be globally routable unicast addresses or site local addresses. Another important feature is the mandatory mobility support. In Mobile IP, the basic idea is to use two addresses for mobile hosts: a home address, which is a permanent unicast address assigned to some fixed network, and a care-of-address, which is a temporary address used in the foreign network the host happens to be located at. In a way, the fundamental assumption here is that the mobile host will spend some time, and typically the majority of its time, in its home network. From an ad hoc fault tolerance point of view, this assumption is false. That is, an ad hoc node does not have any home network where it would spend a large amount of its time, and hence it does not have anybody that would naturally support a home address for it. Instead, the node s current address is acquired in an ad hoc manner. Thus, the concept of home address needs to be abandoned. 3.2 Jini Jini [2] is a concept and a corresponding prototype implementation for a Java technology [6] based ad hoc service architecture. From a strictly technical point of view, Jini is little more than a new method and protocol for publishing, discovering, and shipping around information about serialized Java objects. However, from a conceptual point of view, Jini introduces a new and novel way of thinking about how services should be built in highly dynamic network environments [7]. At the basic level, Jini enables devices to automatically create Jini communities. Internally, a community is represented by one or more Jini Lookup Services (LUS) that allow the community members to find out about the services provided in the community. The services themselves, including the LUS, are heavily based on leases and distributed events. From the fault tolerance point of view, the concept of leases is perhaps the most important concept of the ones promoted in Jini. Basically, a lease is an application specific piece of data that represents dynamic reservation of a remote resource. That is, when a node allocates some resource to be used by another node, it creates a lease and sends it to the node that requested the resource. Each lease has an expiration time, and it is the responsibility of the lease holder to renew the lease before it expires. If the lease holder fails to renew the lease, the lease grantor automatically releases the resource reservation. Since both the lease grantor and holder agree on this policy, both of them know that the resource has been released once the lease has expired. Like other service distribution architectures, Jini has the concept of events. However, in Jini a registration for receiving events is considered to reserve resources at the event source. Consequently, acting according to the Jini resource reservation principles, the event source creates a lease that is given to the event receiver. In this way Jini assures that the event source will eventually stop sending events to parties that have crashed or gone away.

3 Yet another important aspect is due to the use of proxies. That is, the Jini architecture decouples services and service protocols by representing a service and the associated protocol in the form of a downloadable proxy. Thus, the client utilizing the service does not need to know how to implement the protocol used to communicate with the service. When utilized correctly, this architecture allows protocol related fault tolerance to be intelligently distributed between the actual application code (the client) and the service code (the proxy). 4 Architectural aspects In this section, we discuss two fundamental features that affect fault tolerance. First, we discuss identifiers and their roles. There seems to be a lot of confusion in this area. Then, as a second issue, we discuss the architectural consequences of the basic functional requirements. 4.1 Identifiers In any modern communications architecture, there are various identifiers on several layers of the protocol stack. However, the functional purpose of the identifiers are different, and should be made distinct. This has not been the case with the current IPv4 addresses, which are today used at least for addressing, auditing, access control, and accounting. We sincerely hope that the same mistake would not be made in the future IPv6 based network architecture. In IPv6, IP addresses are allocated dynamically, and used for two purposes: addressing within the IP routing infrastructure, and identifying the target interface on a multiple access local link. Both of these functions may be considered addressing functions, since the interface identifier part of an IPv6 address may, under some circumstances, dynamically change over time [4][8]. Thus, in general, an IPv6 address cannot serve for long term identification purposes. Thus, to support other functions currently based on IPv4 addresses, other types of identifiers are needed. From an architectural point of view, the AAA functions (authentication, authorization and accounting) should be based on strong identifiers, making them hard to forge. In open networks, this requires that the corresponding identifiers are cryptographic in nature, e.g., public keys in an asymmetric cryptosystem. The service layer requires identifiers, too. In Jini, when a service registers itself to a LUS for the first time, the LUS assigns it a 128 bit universally unique ServiceID. After that, the service is assumed to remember the ServiceID, and register itself with that ID to all other LUSes. The purpose of such an identifier is pure identification; it allows a node to determine what instances of a given service type actually are identical, and to contact any previously met service by its identifier. However, these identifiers are not meant to be fully reliable. That is, a service may occasionally change its identifier, and there is no protection from a malicious node intentionally using an already allocated identifier. From the fault tolerance point of view, identifiers raise a number of considerations. First, in a fully dynamic environment identifiers are not stable. Therefore, the system must be able to automatically reconfigure itself whenever identifiers change. Second, in the presence of maliciously behaving nodes, the uniqueness of identifiers cannot be necessarily relied on. Thus, the system should use lower layer identifiers only as hints, and rely on higher layer cryptographic means whenever strong positive identification is needed. 4.2 Basic functionality The operational environment and functional requirements of any ad hoc networking based system place a heavy burden on reliability and fault tolerance. However, these reliability and fault tolerance requirements are quite different from what is usually expected. That is, there is usually an implicit assumption that a fault tolerant system continues to function perfectly unless some really catastrophic event occurs, in which case it usually fails to continue functioning at all. Thus, in the face of problems the only visible consequence is the possible degradation of performance. In an ad hoc networking environment, on the other hand, there is no such thing as perfect functionality. Consequently, fault tolerance implies graceful degradation of services as the number and severeness of problems grows, the services do not completely stop working but adapt to the changing environment. That is, the external behaviour of the services change in a user perceivable but acceptable way. In practice, for example, in the case of a connection loss with a service, a Jini client might automatically change over to use another similar kind of service. If there is no such service available, the client would continue working but with local functionality only. In any actual implementation, these requirements mean that the applications must be written in a completely new way. Basically, each application must assume that errors and faults are the rule and not an exception. The application shall contain logic that allows it to function when no services are available, and to immediately enhance its functionality when new or better services become available. Similarly, when services go away, the application must be able to again restrict or gracefully degrade its functionality.

4 An important aspect here is the fact that it is application specific how an application should react to the changing situation. Some applications will definitely work quite well with intermittent communications; others require a genuine on-line connection. Thus, the underlying operating system should not try to hide the network and network related problems from the applications, but instead expose the applications to the problems so that they can act and adapt accordingly. 5 Fault tolerance The different nature of the operating environment and functional requirements fundamentally change the content of the term fault tolerance. More importantly, however, it also changes the methods and approaches to achieve fault tolerance. It is no longer the case that replication nodes and links, together with careful planning and hot swapping techniques, is the fundamental basis. Instead, dynamic address assignment, lease based resource management, on demand routing protocols, and dynamic service discovery are examples of the new approaches. Other important approaches include client side adaptation to available services, automatic synchronization of changes, and other types of automatic recovery in the face of eventual faults. We next discuss these in more detail. 5.1 Dynamic address assignment IPv6 autoconfiguration [5] is a prime example of automatic address assignment. It is initiated whenever a host determines that a network interface is available or connected to a new network. In the case of fixed hosts this typically occurs only when the host boots up. However, in the case of mobile nodes we may well expect connectivity to fluctuate. Thus, a mobile node might well be configured to run IPv6 autoconfiguration whenever it detects a new base station, or, alternatively, if the new base station has a different identity from the previous one. Thus, this functionality is quite similar to the careof-addresses assignment in IPv4. A successful run of the autoconfiguration protocol is illustrated in Figure 1. There, the new node first generates a new address candidate for itself, and multicasts an Neighbor Solicitation message containing the address candidate. The scope of the multicast is the local link only. The Neighbor Solicitation message is repeated a few times, and if nobody replies to it, the host assumes that the address it generated is not in use by anybody else, and starts using the address. From the fault tolerance point of view, dynamic address assignment, at least in the form it is performed in Booting host The local network (other nodes) Generate a link local address Neighbor Solicitation Neighbor Solicitation Assign the link local address Router Solicitation Assign routing prefixes Router Advertisement Figure 1: Successful IPv6 autoconfiguration IPv6, seems to be a mixed blessing. On the positive side, it certainly seems to decrease the number of configuration errors, and allows local ad hoc networking without any additional mechanisms. Furthermore, if used correctly, it allows a node to assume a new address in the case of someone accidentally claiming its current address. On the negative side, however, it is quite vulnerable to determined Denial-of-Service attacks. That is, a malicious host may effectively deny all traffic on a local link by claiming to be using all addresses offered by a host, and break havoc by sending packets with the source address identical to the addresses of the other nodes on the link. Fortunately, these kinds of attacks and problems are local in nature and therefore relatively easy to determine. Due to locality, it is not necessarily too hard to pinpoint the offending device and shut it down. Thus, in a word, dynamic address assignment allows a host to initiate local communications automatically, and to recover from many fault situations simply by creating a new address. However, it is not a panacea, but should be combined with other mechanisms. 5.2 Lease based resource management Most distributed computing approaches have aimed at simulating centralized systems by trying to hide faults and errors caused by the underlying network. For example, a typical distributed system, such as ONC RPC or CORBA, attempts to recover from communication errors by automatically resending messages and retrying the requested operation a number of times. Usually this happens below the programming API, and the situation is reported back to the application only after the recovery has failed. Furthermore, in such situations it is not necessarily clear what the state of the communicating

5 public interface Lease { long getexpiration(); void cancel() void renew(long duration)... } Figure 2: The Lease interface (simplified) Radio coverage of Node A Node A public interface ServiceRegistration { ServiceID getserviceid(); Lease getlease(); void addattributes(entry[] attrsets) void modifyattributes(...) void setattributes(entry[] attrsets) } Figure 3: The ServiceRegistration interface Figure 4: An example ad hoc network peer is nor whether it has detected the network disruption at all. The Jini approach [2] attempts to promote a different kind of approach [9]. In Jini, it is assumed that communications as well as client and server errors do occur, and that these errors should be reflected in the API. While the full scope of the mechanisms used in Jini are beyond the scope of this paper, we want to illustrate the lease based resource management mechanisms. For further information about the issue, we refer to Chapter 2 of Core Jini [7]. As we already briefly mentioned in Section 3.2, resource management in Jini is based on the concept of leases. At the API level, Jini provides the Lease interface, illustrated in Figure 2. Now, any API method that allows resources to be allocated is assumed to return a lease as a part of its reply. For example, when a service registers itself at the local Lookup Service (LUS), the returned positive registration contains a lease (see Figure 3). Once the lease has been issued, it is the lease holder s responsibility to renew the lease before it expires by calling the renew() method in the lease. Now, what is important here from the fault tolerance view is what happens when a renewal fails. Basically, the reason for such a failure can be anything; the lease holder may crash, the communications link may be disrupted, or the lease grantor may have crashed. However, from the resource management point of view the actual reason for the failure does not matter, since the lease semantics dictate that whenever a lease expires, for whatever reason, both the client and server assume the resource reservation to be terminated, thereby automatically cleaning up any possibly remaining state. Another important aspect here is to realize that the lease mechanisms can be applied with various lease periods ranging from a few minutes to days or months. Typically, when the resource reservation is temporary in nature and basically requires an on-line connection between the resource and its user, short living leases are most appropriate. On the other hand, if the reserved resource is relatively permanent in nature, such as a user s home directory, and intermittent connectivity is enough to retain the usability of the resource reservation, long lease expiration time would be more appropriate. Thus, in such a situation the owner of the home directory may renew the lease whenever he or she uses the directory. Thus, the applicability of the leasing mechanism seems to be very wide. Once made aware of it, designers seem to spontaneously invent new situations where it is appropriate. 5.3 On demand routing protocols While dynamic address assignment allows instant local connectivity, it does not suffice in larger ad hoc networks. That is, a larger ad hoc network is typically created through a number of nodes acting as routers between a number of local ad hoc media (see Figure 4). In order to create connections through such a network, the network needs to construct routing tables on the fly, allowing nodes to initiate communication with nodes located in other subnets, or to continue communication in the case a previously local node moves beyond the local radio coverage. Due to the highly dynamic nature of ad hoc networks, it seems like it is usually not worth trying to maintain full up to date connectivity data but acquire new routes on demand. This approach is often called on demand routing. There is a number of proposal for such protocols, including AODV [10], DSR [11], and TORA [12]; see also [13] and [14]. However, all of these proposals

6 seem to be targeted to smaller scale situations where a routing table is based on fixed host identifiers instead of dynamic addresses and dynamically created subnets. From a fault tolerance point of view, one of the bigger problems in these approaches may be the amount of control traffic required. That is, an on demand routing protocol, in general, uses a fairly large number of messages in order to determine a route. This, in turn, causes delays and increases the amount of control messages. As a consequence, in the changing situations real time traffic may be hard of even impossible, yielding traditional fully synchronized approaches to fault tolerance impractical or impossible. A possible solution to this problem might be the utilization of so called hybrid protocols, e.g., Zone Routing Protocol (ZRP) [15]. They seem to contain many desired properties; unfortunately, there does not seem to exist enough of practical information about them, yet. Thus, to put this in other words, on demand routing in large scale ad hoc networks that utilize dynamic address allocation seems like an open research problem that should be addressed. As a small step towards solutions in this area, we want to encourage potential authors to consider the possibilities leases provide in this area. To put it bluntly, to us it seems like a good idea to consider active routing table entries as leases, and renew them periodically to acquire updated information. 5.4 Dynamic service discovery In principle, dynamic service discovery is similar to dynamic address assignment, or dynamic route discovery, but acts on a higher protocol layer. That is, from a functional point of view, an address may be considered to represent a local communications service, and a route may be considered to represent a non-local communications service. On the other hand, the services discussed in conjunction with dynamic service discovery are usually higher level services such as naming, database or application services. The Jini approach to dynamic service discovery is founded on the LUS service and the corresponding discovery protocol. The Jini discovery protocol allows a node to locate any LUS servers reachable within the local network. The bounds of the local network, or to be more precise, the scope of the underlying multicast messages used, determines the limits of the local Jini community. In addition to detecting the local LUS servers, a Jini device may be configured to attempt to contact any number of remote LUS servers. These servers typically represent static communities; for example, a mobile device might know a number of communities that provide a Home Location Register (HLR) like functionality for it. From the fault tolerance point of view, the main benefit of dynamic service discovery seems to be the ability to detect the services available, and to utilize any of the potentially redundant services. If the application supports other decentralization mechanisms such as leases or events, it is not too hard to program the application to gracefully hand over to use another service instance if a particular one becomes unavailable. 5.5 Client side adaptation To survive in the highly dynamic environment of the future, client side adaptation seems to be a necessity and not just a convenience. The future mobile devices are likely to support multiple radio networks simultaneously. The available radio bandwidth will vary by several magnitudes of order. Connections need to be carried over from high speed links to slow speed links (and vice versa). However, what is noteworthy is that adoption to the changing QoS situations may not alone be enough. Instead, genuinely more flexible software architectures are needed. That is, the software products of the future should be able to dynamically adapt their behaviour to the changing situations. Since this requirement seems like a common one, to be shared by a multitude of applications, it should not be left to be solved by the applications alone. Instead, generic software engineering mechanisms are needed. One promising approach seems to be the utilization of behavioural object oriented design patterns [16]. In many of these patterns, a certain piece of behaviour is implemented by combining a small number of objects. In the pattern, the objects function in different roles, and the responsibility of the total functionality is divided between the objects. By dynamically changing the objects the software is able to dynamically adapt its behaviour, thereby allowing it to easily react to changing situations. 5.6 Synchronization and conflict resolution To importance of synchronization, and especially conflict resolution, gets new dimensions in the kind of systems considered. That is, due to the intermittent nature of connectivity, it will be practically impossible to achieve fully synchronous updates. Instead, the system will always contain several versions of the same logical piece of data, and the eventual conflicting updates must be resolved in some way. The actual methods for maintaining data synchronity are beyond the scope of this paper.

7 6 Conclusions When taking the shift of intelligence within networks to its extreme, all intelligence is located at the terminal equipment. In such a world, we cannot assume any kind of centralized infrastructure services. Instead, the network nodes must contain enough functionality so that the necessary functions can be created in a decentralized and ad hoc manner. Considering fault tolerance, this extreme fundamentally changes the situation. Due to the ad hoc nature of such networks, we must assume highly dynamic and changing situations. Connectivity will be intermittent, and the actual QoS will have high fluctuations. Services will dynamically appear and disappear. In a word, there will typically be a high level of redundancy in both connectivity and services, but there will also be shorter or longer time periods without any connectivity or services. Thus, replication can be used, and should be used, but it cannot be based on synchronity and hot backups. Instead, the architecture must be built around intermittent synchronizations, intelligent conflict resolution, dynamic matching of demand with available services, and automatic recovery in the advent of eventual failures. In this paper, we briefly discussed how IP version 6 neighbour discovery and autoconfiguration, some of the ad hoc networking routing approaches, and the Jini concepts seem to be important examples of methods to be used in the future. However, it is important to notice that these technologies are just examples of the approaches likely to dominate in the future, and more research is needed. One of the questions left open is how these concepts could be generalized, and how they could be used in the design of products based on alternative technologies. As a tentative result, is seems probable that the traditional approaches to fault tolerance may not necessarily be the most cost efficient ones in the future. Instead, more focus should be directed in defining the desired behaviour from the user experience point of view. Based on such data, it will then be easier to consider whether traditional full scale fault tolerance or something like dynamic behaviour adaptation would be more relevant to the situation at hand. Acknowledgements I d like to use the opportunity to thank Catharina Candolin, Pasi Eronen and Jouni Karvo of HUT for their constructive criticism and suggestions of improvements in the final phases of preparing this paper. References [1] S. Deering and R. Hinden, Internet Protocol, Version 6 (IPv6) Specification, Standards Track Request for Comments (RFC) 2460, IETF, December [2] K. Arnold, B. O Sullivan, R. W. Scheifler, J. Waldo and A. Wollrath, The Jini Specification, , Addison Wesley, July [3] M. Hattig, Zeroconf Requirements, work in progress, IETF Internet draft, July [4] T. Narten, E. Nordmark, and W. Simpson, Neighbor Discovery for IP Version 6 (IPv6), Standards track Request for Comments (RFC) 2461, IETF, December [5] S. Thomson, T. Narten, IPv6 Stateless Address Autoconfiguration, Standards Track Request for Comments (RFC) 2462, IETF, December [6] J. Gosling, B. Joy, G. Steele and G. Bracha, The Java Language Specification, Second Edition, , Addison-Wesley, August [7] W. K. Edwards, Core Jini, Prentice Hall, ISBN X, June [8] T. Narten and R. Draves, Privacy Extensions for Stateless Address Autoconfiguration in IPv6, work in progress, IETF Internet Draft, Oct [9] S. C. Kendall, J. Waldo, A. Wollrath and G. Wyant, A Note on Distributed Computing, Sun Microsystems Technical Report TR-94-29, Sun Microsystems, Palo Alto, CA, November [10] C. E. Perkins, E. M. Royer, S. R. Das, Ad hoc On-Demand Distance Vector (AODV) Routing, work in progress, IETF Internet Draft, July [11] J. Broch, B. Johnson, and D. Maltz, The Dynamic Source Routing Protocol for Mobile Ad Hoc Networks, work in progress, IETF Internet Draft, December [12] V. D. Park, and M. S. Corson, A highly adaptive distributed routing algorithm for mobile wireless networks, in Proceedings of INFOCOM'97, April [13] R. Määttä, Wireless Ad Hoc Routing Protocols, a Taxonomy, in Proceedings of Helsinki University of Technology Internetworking Seminar 2000, [14] E. Royer and C-K. Toh, A Review of Current Routing Protocols for Ad Hoc Mobile Wireless Networks, IEEE Personal Communications, April [15] Z. J. Haas and M. R. Pearlman, The Zone Routing Protocol (ZRP) for Ad Hoc Networks, work in progress, IETF Internet Draft, June [16] E. Gamma et al., Design Patterns Elements of Reusable Object-Oriented Software, Addison- Wesley, Reading, Massachusetts.