Network QoS Assurance in a Multi-Layer Adaptive Resource Management Scheme for Mission-Critical Applications using the CORBA Middleware Framework*

Transcription

1 Network QoS Assurance in a Multi-Layer Adaptive Resource Management Scheme for Mission-Critical Applications using the CORBA Middleware Framework* B. Dasarathy, S. Gadgil, R. Vaidyanathan, K. Parmeswaran, and B. Coan Telcordia Technologies {das, sgadgil, vravi, kirthika, coan}@research.telcordia.com M. Conarty and V. Bhanot PrismTech {murray.conarty, vb}@prismtechnologies.com Abstract We present adaptive network QoS (Quality of Service) technology that provides ongoing, end-to-end assurance that critical traffic belonging to admitted flows has bounded queuing loss, delay, and jitter. Our technology uses a Bandwidth Broker to provide admission control, and leverages Differentiated Services and Class of Service functionality of high-end routers and switches for enforcement. The technology employs an integrated QoS treatment across a hybrid layer- 2/layer-3 network and adapts to changes in mission requirements, work load and configurations; it uses discovery algorithms in these layers to maintain a current view of resource availability. Under the DARPA ARMS (Adaptive and Reflective Middleware Systems) program, our technology is being developed, integrated and validated in a CORBA-based multi-layer resource management framework. 1. Introduction A new generation of distributed real-time and embedded (DRE) middleware technologies is needed to address the performance needs of mission-critical military applications. Our work addresses network performance metrics such as bandwidth, delay, and jitter. In best-effort networks that experience changing topology and offered load, existing network management software or DRE middleware cannot adapt to provide ongoing assurance that critical messages reach their destination within strict performance bounds. The adaptive QoS solution we propose is being integrated and validated as part of a Multi-Layer Resource Management (MLRM) framework being * This work is supported by DARPA Contract NBCH-C Approved for Public Release, Distribution Unlimited. created by the DARPA ARMS (Adaptive and Reflective Middleware Systems) program using CORBA middleware and component technology [1]. In the absence of network performance guarantees, there can be no guarantees of the performance of distributed invocations such as the IIOP protocol or a publish/subscribe system, or, in general, of meeting application task deadlines. The purpose of the MLRM architecture is to push middleware technologies beyond current commercial capabilities. The current capabilities are largely limited to fixed static allocation of resources in support of predefined mission capabilities. This limits the ability of a military application to adapt to conditions that vary from the original system design. It is desirable for resource allocation to be performed dynamically and modified in response to faults, to changes in mission requirements, or to workload distributions that do not match the original mission-planning model. The essence of our adaptive QoS solution consists of a Bandwidth Broker (BB) that provides admission control of applications into each traffic class, thus ensuring that admitted flows of a given class have statistically guaranteed bandwidth and bounded end-toend loss, delay, and jitter. The solution leverages widely available mechanisms that support layer-3 DiffServ (Differentiated Services) and layer-2 CoS (Class of Service) features in commercial routers and switches. Our BB and the associated flow provisioning module are offered as a CORBA object and CORBA component model service for static as well as dynamic network resource allocations for complex layer-2/3 hybrid network topologies; it can be used by either higher level resource managers or by applications themselves. This paper is organized as follows. In Section 2 we describe the MLRM architecture. In Section 3 we begin with a brief tutorial of DiffServ/CoS features of highend commercial routers and switches. We then describe

2 our Bandwidth Broker solution for assuring network QoS. Particular focus in this section is our integrated end-to-end network QoS management across a layer-2/3 hybrid network. Another critical component of network resource control is the accurate determination of paths traversed by network flows in a layer-2/3 hybrid network, so path discovery is also area of focus in this section. In Section 4 we discuss implementation details. In Section 5 we report on our validation plans and discuss preliminary experimental results. In Section 6 we compare and contrast our work with the work reported in the literature. In Section 7 we summarize our results and briefly describe the research and implementation challenges that are ahead of us. 2. MLRM overview As shown in Figure 1, MLRM is a component of the common middleware services. The middleware services themselves are a layer in the software architecture sandwiched above the network/operating system and below the application layer. COMMON MIDDLEWARE SERVICES Multi-Layer Resource Management Resource Status Service Infrastructure Allocator Application String Manager Resource Pool Manager Resource Manager Figure 1: Multi-Level Resource Management (MLRM) Hierarchy The MLRM resource management hierarchy, in turn, consists of: assigns substrings to resource pools taking into account their inter-pool communication needs. Pool Manager (PM): Assigns applications to computing platforms, taking into account their intra-pool communication needs. Resource Manager (RM): Management of individual resources such as storage and CPU is the concern of (individual) Resource Managers. Another service, the Resource Status Service (RSS) provides continuous feedback on the status of resources toward determining how well the QoS concerns are being met by applications and application strings Network resource management service in MLRM The Bandwidth Broker (BB) provides a network resource management service and thus functions as a Resource Manager in the MLRM hierarchy. ASM (in collaboration with IA) and PM components employ the BB to manage network resources across and within pools, respectively, both during static and dynamic allocations. The IA also makes use of the Bandwidth Broker query services (e.g., available bandwidth across subnets or pools of resources) to prune candidate decisions. The IA component is also envisioned to make use of the Bandwidth Broker in dynamic adaptive situations such as mission mode changes (e.g., from alert to battle mode), whereby major configuration changes are made to every router/switch to effect new mission priorities (e.g., less bandwidth for entertainment video and more bandwidth for sensor data transmission). Figure 2 illustrates the Bandwidth Broker s role in the overall MLRM resource management. (In Figure 2, L3 and L2 stand for Layer-3 and Layer-2 switches, respectively.) Infrastructure Allocator (IA): A system-wide layer for coarse-grain global resource allocation. Its primary function is to divide and conquer intractable resource allocation problems. This layer assigns applications to resource pools. A pool is a collection of resources often determined by factors such as physical proximity (e.g., a data center). Application String Manager (ASM): A systemwide layer for resource management with finegrained resource allocation. This layer sets up, monitors, controls, and adapts application strings. (An application string, which is commonly known as a task in real-time computing, is a sequence of applications that interact to provide some service.) The ASM Resource Status Service ORB (JacORB) Resource Pool Manager ORB/CCM (TAO/CIAO) Subnet Application String Manager ORB/CCM (TAO/CIAO) IIOP Flow IIOP Admission Requests IIOP Query: Bandwidth, Observed Latency L3 L2 Bandwidth Broker ORB (JacORB) Figure 2: Bandwidth Broker Functionality in the Overall MLRM Scheme L2 L2 IIOP Query Bandwidth Classification, Marking & Policing L3 Infrastructure Allocator ORB/CCM (TAO/CIAO) IIOP Bandwidth Change Policy Flow Provisioner IIOP ORB/CCM Subnet (JacORB) Subnet Scheduling, Queue Management (for multiple operational modes/policies)

3 A synopsis of the BB Functionality as CORBA IDL (Interface Definition Language) / CIDL (Component Implementation Definition Language) interface operations follows: Flow Admission Functions: Reserve, commit, modify, and delete flows. Queries: Bandwidth availability in different classes among pairs of pools and subnets. Bandwidth Allocation Policy Changes: In support of mission mode changes. Feedback/Monitoring Services: Feedback on flow performance using metrics such as average delay, jitter and packet loss; to be used by RSS. The flow admission functions reserve and commit and the query functions apply in both static and dynamic allocations. The modify and delete functions are useful when adaptation decisions have to be made in changing the priority of a flow, the amount of bandwidth to be allocated to a flow, etc. The Bandwidth Allocation Policy Changes functions, as alluded to earlier, are used to change the amount of bandwidth allocated throughout the network to various traffic classes to support a mission change. This in turn requires on-the-fly changes to configuration parameters such as queue size on each router/switch and the link bandwidth assignment to a traffic class. Finally, in Figure 2, the Flow Provisioner component is depicted. The Flow Provisioner translates technology-independent configuration directives generated by the Bandwidth Broker into vendor-specific router and switch commands to classify, mark and police packets belonging to a flow. 3. Bandwidth Broker: the essentials In the ARMS program, we are primarily concerned with assuring network performance for mission-critical tasks under conditions of varying network load and changing topology. The network of interest is a layer- 2/3 hybrid network with high-end COTS layer-2 Ethernet and layer-3 (IP) switches. The Differentiated Services architecture (DiffServ) [2] provides an umbrella for implementing Quality of Service (QoS) in IP networks. The 6-bit Differentiated Services Code Point (DSCP) [3] in the IP header allows for up to 64 different classes of service. COTS routers implement the basic building blocks of DiffServ, i.e., policing, marking, scheduling, and buffer management. Ethernet standards offer more limited Quality of Service support, with support for QoS features varying among encapsulations and vendor implementations. Typically, layer 2 supports a 3-bit Class of Service (CoS) marking with more limited support mechanisms for scheduling and buffer management. DiffServ features are typically implemented in router software and the CoS features in ASIC (Application Specific Integrated Circuits) hardware. As such, layer-3 solutions have more flexible and varied QoS treatment options (e.g., more queues and a variety of buffer management techniques), but they are relatively slower than layer-2 CoS implementations. Layer-2 solutions are, however, restrictive in their QoS offerings, as alluded to above. The Bandwidth Broker solution leverages DiffServ mechanisms in layer-3 and CoS mechanisms in layer-2 network elements, in a transparent manner, providing end-to-end QoS guarantees in a hybrid, heterogeneous environment. Typical network implementations of QoS consist of the following broad steps: 1. At the ingress of the network, traffic is classified, typically based on the 5-tuple (source IP address and port, destination IP address and port, and protocol number). (In our scheme, we do host-based initial DSCP marking to deal with the absence of port information in a flow request; this is described later in Section 3.5.) The classified traffic is marked (remarked in our scheme) with a DSCP code point as belonging to a particular class and may be policed or shaped to ensure that traffic does not exceed a certain rate or deviate from a certain profile. 2. In the network core, based on the DSCP marking, traffic is placed into different classes and provided differentiated, but consistent perclass treatment. This includes scheduling mechanisms that assign weights or priorities to different traffic classes (such as weighted fair queuing or priority queuing, respectively), and buffer management techniques that include assigning relative buffer sizes for different classes and packet discard algorithms such as Random Early Detection (RED) and Weighed Random Early Detection (WRED). These two features by themselves are insufficient to guarantee end-to-end network QoS, because the traffic presented to the network must be made to match the network capacity. What is also needed is an adaptive admission control entity that ensures there are adequate network resources for a given traffic flow on any given link that the flow may traverse. The admission control entity should be aware of the path being traversed by each flow, track how much bandwidth is being committed on each link for each traffic class, and estimate whether the traffic demands of new flows can

4 be accommodated. In addition it is responsible for assigning the appropriate traffic class to each flow, and provisioning complex parameters for policing, marking, scheduling and buffer management, such that contracted flows obtain end-to-end QoS. In our solution, these functions are done or coordinated by the Bandwidth Broker. It should be stressed that by tracking committed bandwidth correctly, we balance two often contradictory goals: QoS assurance and utilization. We neither overcommit (thus providing the needed QoS guarantee) nor under-commit (and thus providing a higher utilization). The Bandwidth Broker also supports policy-driven functions especially for adaptations such as demoting the QoS guarantees for an existing flow to admit a new flow of higher importance or when a network anomaly occurs and persists Network architecture The network architecture, illustrated in Figure 3, consists of a high-speed core network with layer-2 switches and access networks consisting layer-3 switches. This is a very typical high-speed enterprise architecture. Our Bandwidth Broker design and implementation handles various hybrid combinations of islands of layer-3 and layer-2 switches as shown in Figure 4. While a layer-3 aware edge device allows for policing functions on the IP header, similar functions can be performed in layer-2 devices based on information in the MAC-layer packet encapsulation. SENSORS AND ACTUATORS Figure 3: Illustration of Network Architecture Fabric CONCE N- Access Networ Fabric ing Fabric Access Networ Layer2 Fabric Figure 4: Generalized Network Topology Before we discuss the QoS treatment through this combined layer-3 and layer-2 network, we summarize how a request through the Bandwidth Broker and Flow Provisioner is processed. As stated earlier, a critical function of the Bandwidth Broker is book-keeping through a persistent storage mechanism. The Bandwidth Broker tracks bandwidth (remaining) for each traffic class on each link. When a bandwidth reservation request for a flow is made by other MLRM components or by an application directly, the processing steps are as follows: First, a determination is made of the actual path the flow will take. If there is enough bandwidth in the desired traffic class on every link on the path for the flow, the flow is admitted; otherwise the flow is denied. Once the flow is admitted, to get differential treatment, we instruct the application or the middleware on behalf of the application to mark packets of the flow with the right DiffServ code point. (See Section 3.5.) If the flow is not admitted, the flow packets get only the best-effort treatment. The ingress router is also instructed to police the flow to ensure its rate does not exceed the flow s original requested bandwidth (with a configurable burst size allowance). The same per-hop behavior for all flows belonging to a traffic class is achieved through pre-configuration of scheduling and buffer management attributes for that class, i.e., the necessary configuration steps for the packet forwarding behavior is not part of a flow request Path discovery Knowledge of the network links traversed by flows is a critical component of accurate admission control. If the admission control entity cannot accurately predict the paths taken by individual traffic flows, then it will be unable to estimate the residual network capacity along a network segment. The Bandwidth Broker incorporates a two-pass approach to path discovery in a hybrid layer-2/3 network. In the first pass, the end-to-end layer-3 path is discovered. In the second pass, the layer-2 path between each of the layer-3 segments discovered in the first pass is computed. Broadly speaking, there are two different classes of techniques for discovering the path taken by packets across a network. The first class involves an inference or computation of the network path by the management entity. This class of techniques attempts to duplicate the path computation techniques in the network to arrive at the accurate path. Such techniques can produce results that do not accurately reflect the network path in border cases, where network protocols often resort to random

5 tie-breaking between equal cost paths. The second class of techniques, favored by the Bandwidth Broker solution at layer 3, involves active discovery techniques to determine the actual paths taken by network traffic. In the remainder of this section, we address the details of the path discovery process separately for layer-3 and layer-2 networks. Layer-3 Path Discovery. The Bandwidth Broker primarily uses active discovery techniques based on traceroute for layer 3 path discovery. Traceroute relies on the Time-to-Live (TTL) field in the IP header, and ICMP error messages to track the hop-by-hop IP path between source and destination. Essentially, ICMP packets with TTL values of (1, 2, ) are sent from the source to the destination. The IP TTL field is decremented at each IP hop. When the TTL value reaches 0, IP routers originate an ICMP error message to the source. This enables the traceroute program to track the IP hop-by-hop path. The Bandwidth Broker can initiate traceroute from the first layer-3 access device serving a host to the final destination, or directly from the source to the destination to determine the layer-3 path. Traceroute works well in cases where a single best path is available between source and destination. In cases where there are multiple equal cost paths from a source to a destination, COTS routers support a feature known as Equal Cost Multipath (ECMP). With ECMP, routers load balance traffic on multiple equal cost paths between two points. Typically, this type of load balancing is done in such a way as to keep packets of the same flow together (in order to minimize packet reordering). Depending on the vendor s ECMP implementation, traceroute may or may not be able to discover the specific best cost path used for a source, destination pair. In cases where traceroute is able to predict accurately the specific path used, the Bandwidth Broker makes precise admission control estimations. When traceroute is unable to pinpoint the precise shortest path used from the list of candidates, a conservative approach that accounts for flows along every possible equal cost path is employed to track bandwidth utilization Layer-2 path discovery Layer-2 path discovery presents a more complex problem, due to the limited communication among layer-2 network elements. Layer-2 switches do not run conventional routing protocols, instead they only communicate using the Spanning Tree Protocol (STP). Layer-2 network segments are broadcast domains, i.e. broadcast packets are forwarded to all ports of a layer-2 device. In a layer-2 network topology with redundant connectivity (or a loop), this could lead to broadcast storms, where broadcast packets are continually forwarded around the loop. STP eliminates such redundant loops by marking certain layer-2 ports as not-forwarding. Thus, one of the keys to layer-2 path discovery is to discover the state of the Spanning tree protocol. Virtual LANs (VLANs) further complicate the situation. VLANs offer many benefits, including the ability to group users together in a single virtual layer-2 segment or broadcast domain irrespective of their physical location. VLANs often span many layer-2 switches. When Spanning Tree is used in conjunction with VLANs, there may be one spanning tree per VLAN. Thus, the logical layer-2 forwarding topology also depends on VLAN membership. The Bandwidth Broker uses SNMP MIBs to track VLAN membership and Spanning Tree state, and uses this information to compute layer-2 paths within a single VLAN. The BB functions for layer-2 path discovery are highlighted below: SNMP Discovery: The BB periodically initiates discovery of the VLAN membership state using vendor-specific MIBs such as the CISCO-VTP- MIB [4] that represents VLAN membership information. STP state is then derived from IEEE standard MIBs such as the BRIDGE-MIB. The standard BRIDGE-MIB [5] only supports one spanning tree. In the presence of VLANs, as previously noted, there is typically one Spanning tree instance per VLAN. In order to store and retrieve Spanning tree state for each VLAN, a de facto vendor technique called SNMP community string indexing [6] is used by the Bandwidth Broker. The community string is the password used by SNMP, typically the read-only community string defaults to public. In order to obtain Spanning tree status for VLAN 10 using community string indexing, the BRIDGE- MIB is accessed using a community string of public@10. VLAN Tree Computation: Once SNMP STP discovery is complete, the STP state is overlaid over the physical network topology to arrive at the Spanning tree paths for each VLAN. In the process of computation, a spanning tree structure rooted at an arbitrary switch ( 1) is constructed and maintained for each VLAN. Subsequently, layer-2 path segments between two switches can be determined by rapid lookups of the appropriate VLAN tree structures. We observe our layer-3/layer-2 path discovery techniques can handle highly redundant topologies and multi-homed hosts. In layer-2, the STP protocol blocks

6 redundant links. In layer-3, we discover and account for ECMP conditions, as alluded to earlier Integrated layer-3 and layer-2 QoS treatment In this section, we discuss the Bandwidth Broker approach to layer-2 and layer-3 QoS treatment of traffic flows that are admitted in the network. As discussed previously, the Bandwidth Broker is responsible for provisioning the network elements to provide differentiated service. This is accomplished by enabling the policing and marking of traffic at the network edge, and enabling scheduling and buffer management at the core of the network. Policing and Marking: As the ARMS network architecture calls for access devices to be layer-3 aware, the functions at the edge of the network are typically enabled as follows. The BB solution uses Access Control Lists (ACLs) available on COTS network elements to classify traffic into flows that can be policed. Typically, the classification is based on the TCP/IP 5-tuple. (See the DSCP Marking subsection.) Classified traffic is then policed in accordance with vendor mechanisms such as Committed Access Rate (CAR). In the policing process, an aggregate rate or bandwidth is ensured for the flow through a combination of a rate (kbps), and allowable burst parameters. The BB solution can be configured to either drop packets that exceed the rate profile, or remark such packets to best effort treatment. Scheduling and Buffer Management: Scheduling mechanisms vary significantly across COTS vendor implementations, from simple round-robin mechanisms, to strict priority queuing and sophisticated mechanisms such as weighted fair queuing. The BB solution uses a best-of-breed approach in terms of the utilization of available vendor scheduling mechanisms. High-priority mission-critical traffic is accorded strict priority treatment, within bounds (dictated by admission control policies), while other traffic classes can share link bandwidth in accordance with configurable weights or percentages. Typically at layer 3, isolation is achieved among traffic classes by the use of separate queues where possible. Where sophisticated buffer management schemes such as Weighted Random Early Detection (WRED) are available, they are applied to TCP/SCTP traffic. In the absence of such schemes, tail drop is the only option. Transport of QoS markings: In layer-3 network segments, the DSCP marking is visible in the IP header and scheduling and buffer management decisions can be based on this information. When traffic traverses layer-2 network segments, a translation is effected (by means of configuration) between DSCP markings and their corresponding layer-2 CoS values. In such cases, it should be noted that at most eight distinct classes of service can be identified in the layer-2 segments. In our implementation experience, we have seldom found the need for more than 8 classes of service however, if the deployment requires it, then multiple layer-3 classes may map onto the same layer-2 class. Typically, layer-2 network segments have increased bandwidth and forwarding capacity in contrast to layer-3 network segments, making such a deployment workable. In, multiple classes of traffic may be forced to share the same queue. In such a situation it is difficult to maintain isolation between traffic belonging to different classes. For each queue, one or more configurable drop thresholds may be available. The drop thresholds indicate the percent queue utilization at which frames are discarded from the queue. Multiple drop thresholds can be associated with a single queue. For instance, a drop threshold of 40% can be assigned to CoS 3 in queue 1, and a drop threshold of 90% can be assigned to CoS 4 in queue 1. If traffic marked with CoS 3 arrives for queue 1 when it is say, 50% full, that traffic will be discarded. However, traffic with CoS 4 will be accepted and enqueued DSCP marking When allocation decisions are made by the Pool Manager (PM) and Application String Manager (ASM), these MLRM components may only know the hosts (i.e., IP addresses) to which applications are to be allocated. Typically port numbers for application data transfer are assigned only when applications start, unless the applications use well-known port numbers (e.g., HTTP server). Thus, port number information may not be available to the PM and ASM components. Furthermore, all publish/subscribe communications in the ARMS architecture will involve a single publish/subscribe daemon on each host, rather than being point-to-point between two applications. Thus, we still need to be able to finely distinguish packets either based on the publication topics or based on the publisher or subscriber application. The problem, then, is how to distinguish packets from different application pairs on the same (source, destination) pair, when port number information is either not available or not particularly useful (publish/subscribe daemon). One proposed solution is to classify and mark the packets in the sending application address space (or better yet in the middleware address space on behalf of the sending application). In this scheme, the Bandwidth Broker returns a DSCP marking whenever a flow reservation is made by the PM or ASM component based on traffic type or QoS requested for the flow. The PM or ASM component, in turn, instructs the application or the middleware to set the DSCP field. Whether the instruction is given to the

7 application or middleware, we prefer the DSCP marking to be set by the middleware layer, since this approach requires minimal changes to (legacy) applications. In the case of synchronous IIOP messaging, the underlying ORB and, in the case of publish/subscribe, the publish/subscribe daemon will be setting the code point, respectively. Here we may or may not assume that the same topic published by different applications has the same importance and hence the same DSCP code point. There are two security threats to host-based DSCP marking: covert channel and malicious alterations of the DSCP marking. Other ARMS researchers are investigating counter-measures to minimize these threats. 4. Implementation details The Bandwidth Broker and Flow Provisioner make use of several open source technologies. They are implemented in Java and use the JacORB Object Request Broker. The relational DBMS mysql is the persistence mechanism used to recover from process and processor failures. (The technology foci for the program include fault tolerant middleware.) The persistence information being kept include the network inventory, currently admitted flows, paths taken by data traffic when it flows between various end-points of the network, configuration details such as access list numbers used for each flow, and lastly DSCP value allotment for each flow. Apart from this network data, for each host that uses Bandwidth Broker services, the Bandwidth Broker also maintains a record that indicates where the host is connected to the network and to which gateway IP address its traffic is directed by default. The amount of persistent information we need is small. (We estimate this to be in the order of few kilobytes for the networks and applications under consideration.) In general, ensuring predictable disk access is hard, and requires non-standard disk drivers and interfaces. If the database size gets larger, we plan to consider in-core database products and/or access and update techniques that can bound the disk access time. We will also consider the real-time CORBA object/component replication-based fault tolerance techniques currently under development [7]. The Flow Provisioner is designed as a stateless component, so the Bandwidth Broker can employ more than one instance of the Flow Provisioner for a large network. In case of software failure in the Bandwidth Broker, the recovered instance of the Bandwidth Broker issues delete flow commands for all in-progress requests to the Flow Provisioner. The Bandwidth Broker clients can reissue the reserve flow commands to ensure that routers are properly configured. A failure in an instance of the Flow Provisioner is handled by the Bandwidth Broker reissuing delete flow to in-progress requests and later reissuing add flow commands. To support an arbitrary network deployment configuration, a network topology is input to the Bandwidth Broker through an XML file. The JAXB [8] tools are used to parse the input and build the required Java objects for switches, their interconnections (interfaces), and router-subnet relations. A UML model was developed to represent the network inventory and its state as required by the Bandwidth Broker. We also developed a method to translate a UML model into database tables and the Java code to access those tables. The method uses design patterns that allow one to write minimal SQL code. The SQL code exists only in base classes. As such, the code remains easily extensible to accommodate any changes to the UML model, such as addition of tables, addition of attributes and addition of relationships, and changes in the persistence mechanism employed. Currently, we are in the process of migrating the Bandwidth Broker to a CORBA Component Model (CCM)-based architecture using the OpenCCM [9] implementation for Java integrated with the TAO [10] Naming Service. Most other MLRM component development uses the CIAO [11] CCM implementation for C++ built over TAO. Demonstrating interoperability among the two CCM implementations is a goal. In addition to path discovery, a key area for algorithm development is supporting queries on available bandwidth across several pairs of pools in support of coarse-level decisions by IA. The complexity is because a pool may be served by multiple edge switches and the paths between pairs of pools may overlap. These algorithms need to converge fast, as they may be invoked during dynamic adaptation. 5. Validation plans and preliminary results Currently, the technologies for ARMS Phase I are being integrated in an application integration framework (AIF) testbed at a facility operated by Raytheon, the integration contractor. The following challenge or gate metric problems are being considered to assess how well the resulting technology will perform: (1) enhance the mission-critical system computing environment (through the MLRM middleware being developed) so it automatically and accurately adapts to dynamic changes in mission requirements and environmental conditions; (2) leverage dynamic resource management to increase mission survivability; and (3) increase mission-critical capabilities in the case of failures arising from battle damage or computing system failures. The challenge problems are intended to demonstrate that in comparison with fixed static configurations, dynamic adaptations provide more configuration choices and accomplish

8 more mission-critical tasks (e.g., war fighting capabilities) and provide better QoS handling capability suited to the environment conditions, especially failures. The Bandwidth Broker, in particular, needs to demonstrate that its bandwidth reservation, bandwidth reservation modification functions, dynamic network QoS reconfiguration techniques and query functions are useful and needed when loads are added. Moreover, the Bandwidth Broker, needs to demonstrate that it is tracking paths correctly, reassigning flows when possible to new paths or reporting to PM or ASM that it is not able to so if and when network resources decline. To support this AIF Lab testing and challenge problem demonstrations, we have implemented Flow Provisioners for layer-3 IOS CISCO (e.g., CISCO 3600 routers), layer-2/3 Catalyst switches (e.g., CISCO 6500 switches) and layer-2/3 IOS switches (e.g., CISCO 4507 switches). We have also successfully provisioned these switches at Telcordia and at AIF for desired per-hop behavior. We have also developed a Flow Provisioner for Linux routers for experimentation at emulab, a network emulation testbed that can support very complex layer-3 topologies using (only) Linux routers. Currently, the experimentation and the challenge problem demonstrations are ongoing. The Bandwidth Broker is currently involved in demonstrating its applicability for dynamic resource management to increase mission survivability. The basic steps in this experimentation are: Reserve bandwidth for certain key mission flows; Detect network overload in these key mission tasks (say in view of increased threats); and Re-reserve for increased bandwidth. In addition to the Bandwidth Broker reservation and reservation modification capabilities, we have implemented network overload detection, and overload event generation capabilities using the CORBA component model framework. Since the Bandwidth Broker s admission capability will not allow overload in the network, excess offered loads have to be detected at the ingress of the network. This is easily accomplished as a direct extension to our Flow Provisioner capability that sets up the policing attributes for various flows at the ingress of the network. The monitoring program uses the policing functions provided by an egress network element to determine the rate of packets being dropped or marked down. Such statistics can indicate that the offered load for a flow exceeds its provisioned limits. We are contemplating extending this monitoring function to such functions as deadline violations using techniques such as active probes. We have completed preliminary experimentations using a testbed consisting of CISCO 3600 series routers. These experiments are intended to demonstrate that the Bandwidth Broker improves both performance and predictability. The results of our experiments are summarized in Table 1. Table 1: Illustration of Bandwidth Broker Improving Both Performance and Predictability BB Role in the File Transfer Flow Not admitted by BB Admitted by BB at 2 Mbits/ in an AF Class Contention Traffic None 1 Mps 2 Mps 3 Mps > 5 min In these experiments, we have a simple configuration consisting of two routers each serving a host and connected by a link of 10 Mbits/second capacity. We are transferring a 10-Mbyte (80-Mbit) file over the network using FTP. The first row corresponds to file transfers being done as best effort. The second row corresponds to file transfers being done using a High Reliability traffic class (an Assured Forwarding (AF) class in DiffServ) policed at the rate of 2 Mbits/second. The columns correspond to contention traffic 0, 1, 2, and 3 Mbits/second. As can be seen in Row 1, when there is no QoS treatment (best effort), as the contention traffic increases from 0, 1, 2, and 3 Mbits/second the FTP transfer time gets worse, from 11.4 to 70 to 218, to more than 300 seconds. Basically, the file transfer flow and the contention traffic get the same equal (equally bad) treatment. When the FTP transfer is done using the AF class, as can be seen in Row 2, as the contention traffic increases from 0, 1, 2, and 3 Mbits/second the performance of the FTP transfer stays the same, i.e., the transfer time is about 30.5 seconds. Although the policing rate for the flow is 2 Mbits/second, i.e., it should have taken at least 40 seconds to transfer an 80 Mbits file, it has taken only 30.5 seconds for the transfers. This discrepancy can be explained by the nominal burst size allowance. In our configuration for policing, we have instructed the router to drop the packets instead of marking them down to best effort when the rate exceeds 2 Mbits/second. This is consistent with how the TCP traffic behaves. In the experiments, only 30% of capacity (3Mbits/second) was allocated to this AF class. If there were a request to the Bandwidth Broker to admit another high reliability class (the same AF class) traffic at the rate greater than 1 Mbits/second, the Bandwidth Broker would have rejected this new traffic flow request ensuring the already admitted FTP traffic at the rate of 2

9 Mbits/second in the AF class gets the right QoS treatment (as shown in the second row). 6. Related work The two main technologies for providing differentiated treatment of traffic are DiffServ/CoS and IntServ. The Bandwidth Broker makes use of DiffServ/CoS. In IntServ, every router makes the decision whether or not to admit a flow with a given QoS requirement. The router keeps the status of all admitted flows as well as the remaining available (uncommitted) bandwidth on its links. Some drawbacks with IntServ are that (1) it requires per-flow state at each router, which can limit its scalability; (2) it makes its admission decisions based on local information rather than some adaptive, network-wide policy; and (3) it is applicable only to layer-3 IP networks [12]. At Texas A&M University, utilization-based admission control for DiffServ networks for real-time applications has been an area of focus [13]. We are currently exploring how to incorporate their results and similar results [14] into our admission control decisions for the high-priority class traffic and extend these results for weighted fair queuing, as supported by the DiffServ AF classes. In the Internet 2 community, as part of the Internet 2 QoS Working Committee, there was an effort to standardize an architecture and requirements for a Bandwidth Broker in 2000 [15], but it did not make much progress. In the Grid community, bandwidth brokering concepts are being discussed [16]. However, the focus here is the establishment of a signaling protocol to negotiate bandwidth reservation across administrative domains and not actually guaranteeing network QoS within a domain or across domains. Telcordia has successfully applied Bandwidth Broker technologies to other Government projects and towards commercial offerings [17, 18, 19]. None of these endeavors, however, deals with layer-2 QoS, let alone unified management of QoS across multi-layers. Furthermore, integration into middleware and integration into an end-to-end resource management framework have not been the focus of these efforts. 7. Summary The accomplishments thus far are in assembling a working infrastructure to implement network QoS as part of an end-to-end Multi-layer Resource Management framework. The Bandwidth Broker presents a transparent, unified QoS solution that guarantees network performance for mission-critical applications in complex layer-2/3 network topologies. The initial performance analysis shows that the Bandwidth Broker can improve the predictability of a key network performance measure, latency. One of our immediate tasks is to demonstrate this latency predictability result using more complex topologies. We have made substantial progress toward demonstrating that our bandwidth reservation techniques, bandwidth reservation modification functions, and overload monitoring functions are key to addressing the challenge problems effectively. Some key research areas of immediate focus include the following: Latency guarantees: Although the Bandwidth Broker can be demonstrated to improve latency and can give a measure of guarantee on a flow s performance such as its latency, we need to be able to quantify this. Currently, the input to the Bandwidth Broker is bandwidth needed (bits/second). This may be too blunt of an instrument for certain application needs. Our goal is to be able to specify a latency requirement to the Bandwidth Broker as well as (or instead of) a bandwidth requirement for a flow. An issue, in general, when flows share a class, as in the case of DifFServ or CoS-based implementations is the lack of convergence of deadline or burst calculations, as these calculations have cyclic dependencies. The starting point of our deadline work is that of Charny and Le Boudec and as described in [14]. This work has a closed form expression for the high-priority class traffic for bounding delays when certain bounds can be placed on the utilization or occupancy. Basically, the higher the bound on route hop count, the smaller the occupancy values must be. In the type of applications that are of interest to us such as ship networks, the hop count is expected to be small, say about 5. We are in the process of generalizing deadline bound results to lower priority classes (strict priority or round-robin) when similar occupancy assumptions hold. Level of Abstraction: In general, we need to increase the abstraction level provided by the Bandwidth Broker. The current interface incorporates very network-centric parameters (e.g., IP address, traffic class, etc.). Among our initial steps (already in place) toward this goal is in denoting the traffic class required in a flow reservation. Instead of DiffServ or CoS classes, a BB client can specify abstract classes such as high priority and high reliability. We will be developing guidelines on how to use these classes. We will provide a provisioning layer that implements the translation into scheduling, buffer management, policing, and marking parameters. Predictable performance: The Bandwidth Broker itself needs to operate under predictable performance criteria during adaptation. We plan

10 to measure the latency, reliable execution and convergence of various BB functions and algorithms. Migration of our implementation to real-time Java infrastructures such as real-time Java based on RTSJ [20], and a real-time Java ORB (e.g., RT ZEN [21]) may need to be considered if real-time performance becomes an issue. These real-time Java technologies are maturing. 8. References [1] R. Campbell, R. Daley, B. Dasarathy, P. Lardieri, B. Orner, R. Schantz, R. Coleburn, L. Welch, P. Work, Toward an Approach for Specification of QoS and Resource Information for Dynamic Resource Management, 2 nd RTAS Workshop on Model-Driven Embedded Systems, RTAS 2004, Toronto, Canada, May 25-28, 2004, RM.pdf [2] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, W. Weiss, An Architecture for Differentiated Services, IETF RFC 2474, December [3] K. Nichols, S. Blake, F. Baker, D. Black, Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers, IETF RFC 2474, December [4] Cisco VTP MIB definition, ftp://ftp.cisco.com/pub/mibs/v2/cisco-vtp-mib.my [5] E. Decker, K. McCloghrie, P. Langille, A. Rijsinghani, Definitions of Managed Objects for Source Routing Bridges, IETF RFC 1525, September [6] SNMP Community String Indexing, CISCO Tech Notes, html [7] P. Narasimhan, Presentation: Middleware for Adaptive Embedded Dependability, [8] Article: Java Architecture for XML Binding (JAXB) WebServices/jaxb/index.html es_white_paper09186a00800a3e2f.shtml [13] B-K. Choi, D. Xuan, C. Li, R. Bettati, W. Zhao, Utilization-Based Admission Control for Scalable Real- Time Communication, Real-Time Systems, Vol. 24, No. 2, pp , March [14] J.-Y. Le Boudec, P. Thiran, Network Calculus, A Theory of Deterministic Queuing Systems for the Internet, Chapter 2, Online Version of the Book, Springer Verlag, LNCS 2050, May 2004 [15] QBONE Bandwidth Broker Architecture, Work in Progress, [16] V. Sander, et al., End-to-End Provision of Policy Information for Network QoS, Proc. 10 th IPDC, [17] K. Kim, P. Mouchtaris, S. Samtani, R. Talpade, L. Wong, A Bandwidth Broker Architecture for VoIP QoS, in Proceedings of SPIE s International Symposium on Convergence of IT and Communications (ITCom), Colorado, Aug 01. [18] B. Kim, I.Sebuktekin, An Integrated IP QoS Architecture Performance, Milcom 02, Anaheim, CA, October [19] R. Chadha, S. Gadgil, et al., PECAN: (Policy-Enabled Configuration Across Networks, IEEE 4th International Workshop on Policies for Distributed Systems and Networks, June [20] RTSJ Main Page, [21] About RT ZEN, Acknowledgments The design and architecture of ARMS is a collaborative effort by many institutions. Our thanks to our ARMS colleagues from Raytheon, the integration contractor, BBN, Boeing, Carnegie-Mellon University, Lockheed- Martin, Johns Hopkins University Applied Physics Lab., Ohio University, SRC, Vanderbilt University, and the University of Illinois. [9] OpenCCM Home Page, [10] TAO Home Page, [11] CIAO Overview, [12] White Paper: DiffServ - The Scalable End-to-End QoS Model,