Network Resource Scheduling and Management of Optical Grids. Savera Tanwir

Transcription

1 ABSTRACT TANWIR, SAVERA. Network Resource Scheduling and Managemenet of Optical Grids. (Under the direction of Professor Harry G. Perros). Advance reservation of lightpaths in an optical network has become a popular concept of reserving network resources in support of Grid applications. In this thesis, we have evaluated and compared several algorithms for dynamic scheduling of lightpaths using a flexible advance reservation model. The main aim is to find the best scheduling policy that improves network utilization and minimizes blocking. The scheduling of lightpaths involve both routing and wavelength assignment. Our simulation results show that minimum cost adaptive routing where link costs are determined by the current and future usage of the link provides the minimum blocking. Moreover, searching for k alternate paths within the scheduling window significantly improves the performance. For wavelength assignment, we have used a scheme that reduces fragmentation by minimizing unused leading or trailing gaps. We have also analyzed approaches for failure recovery and lightpath re-optimization. Finally, an advance reservation scheme needs timely information regarding the status of the optical links. To this end, we have surveyed various monitoring tools and techniques and we have proposed a monitoring framework to support fast restoration.

2 Network Resource Scheduling and Management of Optical Grids by Savera Tanwir A thesis submitted to the Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Master of Science Computer Science Raleigh, NC 2007 Approved By: Dr. Michael Devetsikiotis Dr. Harry G. Perros Chair of Advisory Committee Dr. Rudra Dutta

3 ii Dedication To Baba, Mama and Ahmed...

4 iii Biography Savera Tanwir was born on the 26th September, 1982 in Islamabad, Pakistan. All of her schooling was done in Islamabad. She completed her undergraduate in Computer Software Engineering in 2004 at College of Signals, National University of Sciences and Technology (NUST), Rawalpindi, Pakistan. She worked as a Research Assistant at European Organization for Nuclear Research (CERN), Switzerland and NUST Institute of Information Technology (NIIT), Pakistan from June 2004 to July She is currently a Master s student in the Department of Computer Science at North Carolina State University, Raleigh.

5 iv Acknowledgments First of all, I would like to thank my parents Dr. Tanwir Ahmed and Zamored Tanwir for their unconditional love and support through out my life. I would have never been able to accomplish this work without their unwavering faith in my abilities. I want to thank my husband Ahmed Mujtaba Hamdani for encouraging me all these years and for always being there when I needed him. He has always given me strength and hope in the worst of times. I would like to especially thank Dr. Harry Perros for his guidance and encouragement throughout the two years of masters degree. He has always been very supportive and motivating. I am grateful to him for his valuable advice not only on the thesis but also on several other matters of life. I acknowledge Dr. Rudra Dutta and Dr. Michael Devetsikiotis for agreeing to be on my thesis committee and for their valuable advise and suggestions. I would like to thank Dr. Lina Battestilli for sparing her time and helping me so much with the thesis. She has been an inspiration to me and kept me going all this time. I want to thank Gigi-Karmous Edwards for trusting in me and for her enormous help and financial support. I would also like to thank Dr. Yufeng Xin for his valuable discussions and guidance. I want to thank my room mate Ani for bearing me and listening to me all the time. I also want to thank my brothers Mansoor and Sarmad for always being very supportive. I am grateful to my advisor Dr. Arshad Ali from NUST for introducing me to research and encouraging me always. I would not be here without his help and support. Lastly I would like to thank my friends at NC State Archana, Mradula and Xenia in all the ways they helped me.

6 v Contents List of Figures List of Tables vii viii 1 Introduction Background Motivation Contribution Thesis Layout Related Work Advance Reservation of Lightpaths Routing and Wavelength Assignment of Advance Reservation Requests Failure Recovery Network Re-optimization Advance Reservation Scheduling Strategies Problem Description Scheduling Algorithms Switch Path First (SPF) Algorithm Slide Window First (SWF) Algorithm Wavelength Assignment Network Failures Network Re-Optimization Performance Evaluation The Simulation Model Numerical Results Monitoring and Discovery for Grid Middleware Existing Monitoring and Discovery Frameworks for Grid Grid Monitoring Architecture (GMA) MonALISA Globus MDS Network Weather Service

7 vi PerfSONAR EnLIGHtened Computing Monitoring Framework Network Performance Measures Compute Resource Characteristics Monitoring Issues Information Collection Representation Format Update Frequency Non-Intrusiveness Monitoring Strategies Distribution of Monitoring Data Security Fault Tolerance Monitoring Architecture Current Deployment Scenario Conclusions and Future Work 53 Bibliography 55

8 vii List of Figures 1.1 EnLIGHTened Computing Testbed EnLIGHTened Middleware Architecture Scheduling Window Example Topology Reservation table at t Reservation table at t Wavelength Assignment using Min-Gap Network Topologies Discrete Probability Distribution for the Intermediate Time between the Request Arrival Time and Reservation Time Flexibility in Scheduling Window Arrival Rate vs Blocking Probability NSF-NET Link Utilization : SWF vs LB-SWF Node Fully-Connected Network Link Utilization : SWF vs LB-SWF k vs Blocking Probability k vs Blocking Probability d max vs Blocking Probability Reservation Delay vs d max Non-Blocking Scheduler Wavelength Assignment Requests for long and short durations simultaneously IPP Arrival Process Failure Recovery Periodic Reconfiguration Monitoring Architecture MonALISA client GUI : current monitoring information from the EnLIGHTened testbed can be viewed Optical Switch Configuration for EnLIGHTened testbed

9 viii List of Tables 3.1 Reservation Table at t Reservation Table at t

10 1 Chapter 1 Introduction This chapter introduces the concepts and theory related to network resource scheduling and management of Optical Grids. Section 1.1 introduces the background concepts related to resource management in Optical Grids, Section 1.3 gives the motivation for this line of research and Section 1.4 summarizes our contributions to the field. 1.1 Background The Grid is widely regarded as the next stage for the Internet after the World Wide Web. According to IBM [32], Grid computing is an approach to distributed computing that spans not only locations but also organizations, machine architectures and software boundaries to provide unlimited power, collaboration and information access to everyone connected to a grid. The grid allows resource discovery, resource sharing and collaboration in distributed environment, thus it can provide people from different locations and countries to work together to solve a specific problem such as a design collaboration. An Optical Grid uses optical network as a key Grid resource. Optical Grids have emerged around the world to support the bandwidth requirements of many escience and ebusiness applications. Some examples of such applications include Data Intensive applications such as DynaCode [3] for environmental modeling, Collaborative Remote Data Visualization [27] and High Definition Interactive Video Conferencing [30] that requires

11 2 high speed transfer of high definition video streams with multicast capabilities. We have also recently seen great progress in the optical networks technology in terms of transmission capacity and dynamic reconfigurability. With Wavelength Division Multiplexing (WDM), a single wavelength can now reach OC-192 (10 Gbps) and hundreds of wavelengths can be supported on each fiber. So the Optical Grids have emerged to interconnect massive compute clusters and large storage resources through multi-gigabit lightpath connections. Although much progress has been made towards developing Grid technologies, the area that is still underdeveloped is the link between Grid applications and underlying network technologies which make Grids truly effective. The abstraction and encapsulation of these optical network resources into manageable and dynamically provisioned Grid entities is necessary in order to meet the complex demand patterns of Grid applications and to optimize the overall network utilization. Various projects around the world consider the problem of building reconfigurable, dynamic, adaptable Optical Grids [8, 9, 12, 10]. These projects consider the network resources as key Grid resources that can be managed and controlled like any other Grid resource. They have developed Grid middleware that is responsible for the resource managing, coordinating, allocating and scheduling advance resource reservations for all Grid resources: network, cluster, storage, scientific instruments, etc. The middleware establishes the optical connections with the necessary bandwidth and it interfaces with the cluster schedulers (e.g. Moab, OpenPBS/Torgue,etc) to make advance reservations. From a Grid application perspective, all types of resources may be required to meet its computational needs. The EnLIGHTened Computing project [8] is one such effort towards the adaptive and optimized use of network as a virtual coordinated resource along with the other Grid resources. The EnLIGHTened Computing Project has designed an architectural framework that allows Grid applications to dynamically request in advance or on demand any type of Grid resource - not only high-performance computers, storage, scientific instruments but also deterministic, high-bandwidth, network paths, including lightpaths. The proposed framework discovers and monitors the performance and reliability of the Grid resources for adaptive resource meta-scheduling and coordination. The EnLIGHTened testbed is shown is figure 1.1. The EnLIGHTened Computing middleware architecture is shown in detail in figure 1.2. Applications use the Application Launcher Steerer (ALS) to request resources from

12 3 Figure 1.1: EnLIGHTened Computing Testbed the EnLIGHTened Resource Broker (ERB). The request may include compute resources, network, storage, etc. The ERB then queries the underlying Discovery and Monitoring (DM) component for information, and then chooses the most appropriate resources. The Highly-Available Resource Co-allocator (HARC) Acceptors are used to co-allocate the required resources for the selected time range by using the HARC Resource Managers (RM). HARC either co-allocates all the needed resources or none at all by using a phased commit protocol. Its flexible design allows arbitrary resources to be co-allocated with advance reservations. The Compute RM manages the compute resources and wraps existing cluster schedulers ( LSF, OpenPBS, Maui, etc) to provide the advance reservation capability. The HARC Network RM interfaces to the Domain Network RM (DNRM), which manages the network resources and supports advance reservations. The HARC Instrument RM interfaces to scientific instruments.

13 4 Figure 1.2: EnLIGHTened Middleware Architecture 1.2 Motivation The DNRM is the middleware component that controls the optical network. It is responsible for path computation and wavelength assignment of the end-to-end lightpaths. In a shared-lambda environment, the enforcement of policies is crucial for successful operation. Thus, each DNRM has a policy engine that gives Grid administrators control over how networking resources are shared. The current DNRM maintains a reservation table of fixed paths. When a request arrives, it checks if the fixed path between the required source and destination is free or not. The DNRM then reserves the resources by interfacing with the specific control plane, i.e., each network domain may have a dynamic control mechanism such as DRAGON s GMPLS [2], CHEETAH [1], UCLP [13], etc. Currently, the DNRM does not support dynamic path computation based on the most recent network topology. There are several features that can be added to the DNRM to make it more adaptive. These are given below:

14 5 Network Topology Discovery: Determine the latest state of the network resources by interacting with the control plane e.g. OSPF-TE and management plane e.g. SNMP, TL-1 etc. Dynamic Path Computation and Wavelength Assignment: Dynamically compute the paths using the up-to-date network topology. Assign wavelength using algorithms that improve network utilization. Failure Recovery: Employ network monitoring tools to get latest information about the network resources. Implement mechanisms for restoration and recovery of lightpaths in case of failures. Network Re-Optimization: Implement offline re-optimization mechanisms to improve network utilization. 1.3 Contribution In this thesis, we have evaluated and compared various algorithms for advance lightpath scheduling that can be implemented in a DNRM. The main aim is to find the best scheduling policy for a Grid network resource manager that improves network utilization and minimizes blocking. We have also analyzed several approaches for failure recovery and resource optimization. In order to provide fast restoration, there is a need for a robust monitoring architecture. We also need up to date information about the resources to support the resource discovery process. We have identified the appropriate discovery and monitoring parameters which are needed to determine the availability and performance of the network as well as the compute resources in a Grid. We have surveyed a variety of monitoring tools and techniques and have proposed a monitoring framework to collect these parameters for the EnLIGHTENed middleware. 1.4 Thesis Layout The rest of this thesis is organized as follows: Chapter 2 surveys the state-of-art in the area of advance reservation scheduling in optical networks, comparing important

15 6 research efforts with others to place our work in perspective; Chapter 3 describes in detail the advance reservation scheduling strategies and gives the performance evaluation and results. Chapter 4 describes our work for the monitoring and discovery of resources for the EnLIGHTened middleware. Chapter 5 concludes the thesis with a summary of our findings as well as directions for future research.

16 7 Chapter 2 Related Work To place this thesis in perspective, we sketch a broad outline of the field of network scheduling by listing key features of other research efforts in this area. We first discuss advance reservations in networks followed by the routing and wavelength assignment of lightpaths. After that we discuss the existing work in the area of failure recovery of optical networks with advance reservations. We also briefly describe some network optimization techniques. 2.1 Advance Reservation of Lightpaths In an Optical Grid, application requests may be for immediate or for advance reservations. In the immediate reservations case, the application requests resources from the Grid middleware for immediate use. In the advance reservation case, the requests are for a future time, i.e., there is a time period between the request arrival and the time of the resource reservation. In fact, the immediate reservations can be viewed as a special case of the advance reservations, with a zero time period between the request arrival and the time of the resource reservation. We believe that in a shared Optical Grid, advance reservations are a necessity because they guarantee the availability of resources with the required QoS. For the cluster resources, certain schedulers, such as Maui [11], provide some advance scheduling capability. In this thesis, we focus on the scheduling of advance

17 8 reservations of network resources. The concept of advance reservations in an optical network has attracted some attention in recent years. In [18], the author discussed the properties of advance reservation and proposed an architecture for a bandwidth broker to improve the performance of a network based on the additional knowledge of these future reservations. The Globus Architecture for Reservation and Allocation (GARA), presented in [24], supports advance reservations for various types of resources for the Grid. In [23], the authors also discussed advance reservation of heterogeneous network paths in the context of Grid computing and proposed a network resource hierarchy to integrate the path management with Grid information and authentication services. In comparison to the immediate reservations, advance reservations generally degrade the resource utilization and the acceptance rate due to resource fragmentation [19]. This can be improved by introducing some flexibility in defining the advance reservations. In [41] the authors proposed a sliding scheduled traffic model and a demand time conflict resolution algorithm to maximize resource usage in a network. In [26] the authors proposed a Flexible Advance Reservation Model (FARM) and described how to implement this model in the meta-scheduling problem. The results indicate that using this approach the acceptance rate and resource utilization can be improved dramatically. 2.2 Routing and Wavelength Assignment of Advance Reservation Requests The Routing and Wavelength Assignment (RWA) [45, 44] problem deals with finding a route on the network topology and assigning a wavelength on each link of the selected route for every lightpath established over the network. Typically a connection request can be of three types [45] static, incremental, and dynamic. With static traffic, the entire set of connections is known in advance, and the problem is then to set up lightpaths for these connections while minimizing network resources such as the number of wavelengths or the number of fibers in the network. The RWA problem for static traffic is known as the Static Lightpath Establishment (SLE) problem. The SLE problem can be formulated as a mixed-integer linear program [34], which is NP-

18 9 complete [21]. To make the problem more tractable, the SLE problem can be partitioned into two subproblems, namely routing and wavelength assignment, and each subproblem can be solved separately. A review of these approaches is given in [45]. In case of incremental traffic, connection requests arrive sequentially and the lightpath established for each connection remains in the network indefinitely. Lastly for dynamic traffic, a lightpath is set up for each connection request as it arrives, and the lightpath is released after some finite amount of time. The objective in the incremental and dynamic traffic cases is to set up lightpaths and assign wavelengths in a manner that minimizes the blocking probability of a connection, or equivalently maximizes the number of connections that are established in the network at any time. This problem is referred to as the Dynamic Lightpath Establishment (DLE) problem. It is more complex to solve and usually heuristics are used to solve the routing and wavelength assignment subproblems separately [44]. There is another characterization of a traffic model where the setup and teardown times of the demands are known in advance. This is known as the Scheduled Traffic Model [29]. There can be a fixed window or a flexible window with this model. The start and end time of the connection cannot be altered in the fixed window model, but with the flexible approach the start and end times can slide within a larger window. Integer Linear Program formulations and algorithms have been proposed to solve these problems in [28, 29]. In [46] the design of effective RWA algorithms for different types of advance reservations is discussed and algorithms are presented for requests having specific start time and specific duration (STSD), specific start time and unspecified duration (STUD) and Unspecified Start Time and Specified Duration (UTSD). 2.3 Failure Recovery Accepted advance reservations can be affected by network failures. In view of this, a strategy is required in order to restore the existing network connections and re-schedule future reservations. When a failure is observed in the network, all the in-service connections affected by this failure need to be re-scheduled. Also, it is useful to re-route the reservations that are scheduled but not yet in service as they can be affected by the failure at a later time. Since the failure duration is not known in advance, a re-routing interval has to determined

19 10 using different methods. From [20] it is clear that substituting the exact downtime by vague estimations is dangerous, since this leads to a significantly worse performance. Also in [20] two techniques were presented that are independent of the actual downtime. These are load-based and feedback based. In the case of load-based approach the current load and the booking profile during the next time slot should be less than a threshold value η which depends on the network topology. While in the feedback based method, the initial re-routing interval is one slot and it is increased based on the percentage of successfully re-routed reservations during the last slot. The interval duration is increased until the percentage of successfully re-routed reservations is sufficiently high. The results show that the feedback approach gives better results than the load-based approach while avoiding the need to adjust the threshold parameter η. 2.4 Network Re-optimization The network re-optimization can be used to increase the network utilization. The lightpath re-optimization techniques have been discussed in several papers. Most of the authors have proposed solutions for static traffic demand and heuristics for long-term ondemand traffic flows [17, 16]. In [43] the authors have proposed an optimization scheme for advance reservations by re-configuring them without changing their reservation times. Using this scheme, if a request, x, is blocked, all the existing connections that time-overlap with this request are unscheduled and then re-scheduled after sorting them according to their start times. This can result in provisioning of all the requests including the request x which was blocked earlier. If this does not work, all connections are restored to their original state, i.e., no re-configuration takes place.

20 11 Chapter 3 Advance Reservation Scheduling Strategies In this chapter the various advance scheduling algorithms considered in this thesis are described. In Section 3.1 we describe the advance scheduling problem. In Section 3.2 we present the scheduler functions and algorithms. In Section 3.3, some wavelength assignment techniques are discussed. Section 3.4 and 3.5 describe the failure recovery and resource optimization. Section 3.6 gives the simulation model and the numerical results. 3.1 Problem Description Consider a Network Topology Graph G = (N,L,W) where N is the set of nodes, L is the set of links and W is the set of wavelengths supported by each link. A user submits an advance reservation request for a lightpath between any two nodes on G to the DNRM. Each request R is defined by the following parameters: R = [source node, destination node, s, e, d, bandwidth] where d is the reservation duration, and s and e are the starting and ending time of the scheduling window respectively as shown in Figure 3.1. The time is slotted with a slot size equal to t. The scheduling window defines the time period within which the requestor

21 12 would like to make a resource reservation. The scheduling window must be bigger than the reservation duration d. Thus the scheduler must check if a path is available during interval [s + t, s + t + d] where t = 0, 1, 2,..., e s d. Figure 3.1: Scheduling Window This is an online scheduling problem because the requests arrive dynamically and for each request R, the DNRM must compute a path and then check if a wavelength on each link of this path can be reserved for duration d within the scheduling window [s, e]. The DNRM allocates a wavelength on each link along a path from the source to the destination nodes. If a wavelength along the path for the specified period of time is not available, another path has to be determined. In order to do this, the DNRM maintains a schedule of the reservations called the Reservation Table. It contains all current and future reservations and it is used to search for available resources for new advance reservations. Figure 3.2: Example Topology Table 1 shows an example of the reservation table for two lightpath requests for the network shown in figure 3.2. This is an optical network with 2 wavelengths per fiber link and each link has a cost of 1. Let us assume that at time t 0 two lightpath requests arrive,

22 13 Figure 3.3: Reservation table at t 0 R 1 = [n 1,n 7,t 1,t 8,4,1] and R 2 = [n 1,n 8,t 2,t 9,4,1]. We assume that each request requires a bandwidth equal to a wavelength. As there are no other reservations at this time, the links are reserved starting at the beginning of the scheduling window of each request. A pictorial representation is given in figure 3.3 Table 3.1: Reservation Table at t 0 Node1 Node2 Wavelength Start Slot End Slot Req ID n 1 n 3 λ 1 t 1 t 4 R 1 n 1 n 3 λ 2 t 2 t 5 R 2 n 3 n 6 λ 1 t 1 t 4 R 1 n 3 n 6 λ 2 t 2 t 5 R 2 n 6 n 7 λ 1 t 1 t 4 R 1 n 6 n 8 λ 1 t 2 t 5 R 2 Let us assume that a third request arrives at time t 1 for a path between n 1 and n 8 with R 3 = [n 1,n 8,t 3,t 8,3,1]. Since all the wavelengths along links n 1 n 3 and n 3 n 6 are busy till time t 4, the shortest path n 1 n 3 n 6 n 8 is not available for slots t 3 and t 4. But due to the large scheduling window, the request can be still accepted for slots t 5, t 6 and t 7 for the same path. At time t 2 a fourth request arrives with R 4 = [n 1,n 7,t 3,t 5,1,1]. In this case, all the wavelengths along links n 1 n 3 and n 3 n 6 are busy till t 5 and the shortest path

23 14 n 1 n 3 n 6 n 7 is not available for all slots in the scheduling window. So in this case, another path has to be determined. This new path can be a 4-link path i.e. n 1 n 2 n 5 n 6 n 7 and the wavelengths that are reserved are λ 1, λ 1, λ 1 and λ 2. The state of the reservation table at time t 2 is given in Table 2. A pictorial representation is given in figure 3.4. Table 3.2: Reservation Table at t 2 Node1 Node2 Wavelength Start Slot End Slot Req ID n 1 n 2 λ 1 t 3 t 4 R 4 n 1 n 3 λ 1 t 1 t 4 R 1 n 1 n 3 λ 2 t 2 t 5 R 2 n 1 n 3 λ 1 t 5 t 7 R 3 n 2 n 5 λ 1 t 3 t 4 R 4 n 3 n 6 λ 1 t 1 t 4 R 1 n 3 n 6 λ 2 t 2 t 5 R 2 n 3 n 6 λ 1 t 5 t 7 R 3 n 5 n 6 λ 1 t 3 t 4 R 4 n 6 n 7 λ 1 t 1 t 4 R 1 n 6 n 7 λ 2 t 3 t 4 R 4 n 6 n 8 λ 1 t 2 t 5 R 2 n 6 n 8 λ 2 t 5 t 7 R 3 The objective in this thesis is to determine a scheduling policy to route each incoming lightpath connection request dynamically while minimizing the probability that a connection request will be refused due to lack of available lightpaths and maximizing the overall network throughput. 3.2 Scheduling Algorithms In our study, we consider DLE requests that belong to a flexible scheduling window. We have developed several strategies for searching the reservation schedule to determine whether a new request R can be accepted. These strategies use a combination of different scheduling decisions and search techniques. Our objective is to determine a scheduling algorithm that minimizes the blocking probability, i.e. the probability of not scheduling a request within its window, minimizes fragmentation in the usage of each wavelength and maximizes network utilization. We only consider flexible advance reservations as they have lower blocking probability than the fixed advance reservation model [26, 19].

24 15 Figure 3.4: Reservation table at t 2 We consider two categories of advance scheduling. In the first, a call is blocked if it cannot be scheduled within its requested flexible window. In the second, calls are not blocked, but rather delayed and scheduled at the first available time instance which maybe outside the requested window. As will be seen below, we always determine the shortest path from a source to a destination using Dijkstra s algorithm. Different costs associated with the links can be used. The most common link cost is the propagation delay. However, in order to balance the load in the network one can use a link cost that is determined based on the current and future usage. In this case we start with all the links having a cost of 1. As lightpaths are reserved, the link cost is incremented by the number of slots reserved on that link. The cost is decreased after the request is serviced. Hence the weights only reflect the current and future utilization of the link. We propose the following scheduling strategies:

25 16 Switch Path First (SPF): In this scheme we start from the beginning of the scheduling window and check k different paths before sliding the window to the next time slot. Slide Window First (SWF): In this scheme we check one path at a time for all of the scheduling window slots. If a path cannot be reserved during the scheduling window, the next shortest path is checked. We also propose a variant of these strategies by allowing the weights of the links to vary based on current and future utilization of the link. So the weight of a link increases as it gets more and more advance reservations. Opposite, if the link does not have too many advance reservations then its cost will decrease. This provides load-balancing in the network. We call these variant scheduling strategies: Load Balancing - Switch Path First (LB-SPF): In this scheme the weights of the links are based on current and future utilization of the link, but the search algorithm is the same as SPF. Load Balancing - Slide Window First(LB-SWF): In this scheme the weights of the links are based on current and future utilization of the link, but the search algorithm is the same as SWF. All the four strategies can be used with both the wavelength assignment schemes First-Fit and Min-Gap described below in section 3.3. We now proceed to describe in detail the two algorithms SPF and SWF Switch Path First (SPF) Algorithm In this scheme when a request R arrives, we first try to find the shortest available path starting at time s for d slots. This is done by first finding the shortest path, using a delay propagation link cost. Then, we check if all the links on this path have a free wavelength for d slots starting at time s + t, where t=0. If any link is busy along the path, the topology is updated by removing that link and the next shortest path is determined. This step is repeated until either a path is available or a maximum of k different paths have been considered. If a path cannot be determined, we repeat the whole process with a start time equal to s + t, where t=1. t is incremented by one slot each time until an available path is found or t = e s d, whereupon a request is blocked.

26 17 Algorithm 1 The SPF algorithm 1: function FindPath(Request r, topology t) 2: i = 1 3: start time = s 4: end time = s + d 5: while (end time e) do 6: while (i k) do 7: find shortest path with Dijkstra s algorithm with propagation delay link cost 8: if a path is found then 9: if wavelengths are available on all links during start and end time then 10: assign wavelengths, update all tables 11: return 12: else 13: delete busy link from the topology 14: i ++ 15: end if 16: end if 17: end while 18: start time = start time + t 19: end time = start time + d 20: end while 21: end function Slide Window First (SWF) Algorithm In this algorithm, we try to find a free period d starting at s + t, where t = 0, 1, 2,...e s d. If the first shortest path is not free for the required duration during the window, the busiest link defined as the one that uses the maximum number of slots during the scheduling window, is removed from the network topology and the procedure is repeated until either an available path is found or a maximum of k paths is considered. 3.3 Wavelength Assignment It is possible that multiple wavelengths on a particular link along the sourcedestination path, are available for the advance reservation. In this case, the scheduling algorithm must determine which wavelength to allocate from the pool of available wavelengths. We use the following two wavelength allocation schemes: First-Fit : This is a well known method for wavelength assignment as it gives good performance with least computational time. In this scheme, all wavelengths are arbitrarily

27 18 Algorithm 2 The SWF algorithm 1: function FindPath(Request r, topology t) 2: i = 1 3: while (i k) do 4: start time = s 5: end time = s + d 6: find shortest path with Dijkstra s algorithm with propagation delay link cost 7: if A path is found then 8: while (end time e) do 9: if wavelengths are available on all links during start and end time then 10: assign wavelengths, update all tables 11: return 12: else 13: start time = start time + t 14: end time = start time + d 15: end if 16: end while 17: else 18: remove the busiest link during the window from topology 19: end if 20: i ++ 21: end while 22: end function numbered. When searching for available wavelengths, we always start from wavelength 1 and proceed sequentially to the last wavelength until a free wavelength is found. By searching the wavelengths in this manner, connections are packed into a smaller number of wavelengths thus freeing other wavelengths for reservations. First-fit does not require global knowledge of the network and no storage is needed to keep the network state. This scheme performs well in terms of blocking probability and fairness, and is preferred in practice because of its small computational overhead and low complexity [45]. Min-Gap : This scheme aims at reducing the fragmentation in wavelength usage. With advance reservations we have the information of the wavelength usage with the associated times, and hence we can use this information when assigning the wavelength in order to reduce the gaps between reservations. The main drawback of this scheme is the additional computational overhead when searching for the least gap throughout the existing reservations. There can be three different ways to minimize the gaps:

28 19 Figure 3.5: Wavelength Assignment using Min-Gap Min-Leading-Gap: This wavelength assignment scheme minimizes the unused leading gaps on a wavelength. Min-Trailing-Gap: This scheme minimizes the unused trailing gaps on a wavelength. Best-Fit: This scheme minimizes the sum of the leading and trailing gaps. An example of these wavelength allocation schemes is shown in figure 3.5. We can see that the new lightpath (LP) reservation can be accommodated by all the four wavelengths. First-Fit will select λ 1. λ 2 minimizes the leading gap between the new and previous reservations therefore it will be selected by Min-Leading-Gap scheme, while λ 3 minimizes the trailing gaps so it will be selected by Min-Trailing-Gap. The Best-Fit will select λ 4, as it minimizes the sum of leading and trailing gaps. A comparison of these three schemes is presented in section Network Failures Accepted advance reservations can be affected by network failures. In view of this, a strategy is required in order to restore the existing network connections and re-schedule future reservations. We consider network component failures, which could be either link failures or node failures. When a link fails, all its constituent wavelengths will fail. Link failure is more of

29 20 a concern because most nodes have built in redundancy. Because of the high data rate on these networks, it is necessary to develop appropriate path protection and restoration schemes to prevent or reduce data loss. In case of protection, a path or a link is protected against failures by preassigning resources for a backup path, while in restoration schemes an alternate route is discovered dynamically for each failed connection after the failure occurs. Protection schemes have faster recovery time and provide guaranteed recovery but they require more network resources. Therefore due to lower costs involved, we use the restoration mechanism. Given a failure, the question of interest is how far into the future should we reschedule the existing reservations. In order to deal with network failures the DNRM must be aware of the current network state. This information can be obtained from the network devices in the form of a link state database using SNMP or other management protocols. Routing protocols, such as OSPF react slowly to network failures. In view of this, monitoring tools that talk directly to the network devices to get the latest state of interfaces and ports should be used as well. As soon as a link failure is observed, all the in-service connections using this link have to be re-scheduled. Also, it is useful to reroute the requests that are scheduled but not yet in service that use the failed link. These are likely to be affected if the failed link is not restored on time. The straight forward approach is to re-schedule all the future connections that use the failed link. However this may not be good solution because if the link is repaired early, the connections will be using sub-optimal paths. Thus, a re-routing interval has to be determined to re-schedule future reservations so that to minimize the number of dropped reservations. We note that the reservations can be dropped when re-scheduling them, if a free path cannot be found for the start time that a reservation has already been made. We have used an approach in which the re-routing interval is based on the knowledge of the duration of previous failures. Specifically, we maintain a moving average of the historical failure durations and use this as an estimate of the restoration time. Similar to the feedback based approach [20], if the link does not come up at the end of the current re-routing interval, we increase the re-routing interval by an amount equal to the previous re-routing interval.

30 Network Re-Optimization A re-optimization of the network can be performed by re-scheduling the lightpath requests. This can be done in two ways: either by changing paths only or by changing both the path and the start time. The latter is not very desirable as it involves re-negotiating the time with the users, and in view of this we will not be considering it in this study. In [43] the authors have proposed a re-optimization scheme for advance reservations by re-configuring them without changing their reservation times. Using this scheme, if a request is blocked, all the existing connections that time-overlap with this request are unscheduled and then re-scheduled after sorting them according to their start times. This may result in provisioning of all the requests including the one which was blocked earlier. If this does not work, all connections are restored to their original state, i.e., no re-configuration takes place. We use the same technique but instead of doing the re-configuration after every blocked request, we do it periodically by re-scheduling all the requests for a specific number of slots in future. 3.6 Performance Evaluation The Simulation Model We have conducted our simulation experiments on a 14 node NSF-NET topology with 42 uni-directional links and a 33 node GEANT network topology with 94 unidirectional links as shown in figure 3.6. We have also considered a 10 node fully connected network topology. We assume 10 wavelengths on each link and full wavelength conversion at each node. The time is slotted with the duration of each slot being 30 minutes. We assume that requests arrive in a Poisson fashion and all requests need to reserve a lightpath with bandwidth equal to one wavelength. The duration of a reservation is uniformly distributed. The start times of the request are generated within a window of 400 slots. To simulate a more realistic environment, we have generated the intermediate period between the arrival of the request and the start of the reservation using the discrete probability distribution shown in figure 3.7. We assume that more requests will be for the reservation slots in the near future i.e next 24 slots and very few requests will be for reservation slots far into the future e.g. after two or three days. The source and destination

31 22 (a) NSF-NET (b) GEANT (c) 10-node Fully Connected Figure 3.6: Network Topologies

32 23 nodes for the requested connection are selected randomly using a uniform distribution. Figure 3.7: Discrete Probability Distribution for the Intermediate Time between the Request Arrival Time and Reservation Time We assume that the scheduling window is twice the reservation duration, i.e., (e s) = 2d, based on the results in [26] where the authors have shown that using just 1 or 2 units of flexibility improves the performance significantly. Our results agree with this observation. Figure 3.8 gives the blocking probability for different window sizes. The solid curve marked as 1d corresponds to the case e s = d. The dotted line curves correspond to the cases when e s = 2d and e s = 3d. We note that the scheduling window with a width of twice the duration has much lower blocking probability than without any flexibility. Also increasing it further does not improve the performance to a large extent but the delay between the start time of the actual reservation and the start of the window increases. We ran the simulations under different network loads, where network load is determined by the request arrival rate and the reservation durations. The parameter of interest is the blocking probability Bp. We have used a failure model with link failures only. In our simulation, we randomly select any link in the network as a failed link. The mean time to failure is exponentially distributed with a mean of 80 slots. The recovery time is also exponentially

33 24 Figure 3.8: Flexibility in Scheduling Window distributed with a mean of 48 slots. We assume that when a link fails all wavelengths on that link fail. The simulations were run for long time durations with large number of arrivals such that a sufficiently small confidence interval within 1% of the mean with 95% confidence is reached. The confidence intervals are not given in the graphs as they are not discernible. Finally, the simulation model was written in Java Numerical Results Scheduling Algorithms In this section we will present the simulation results and comparison of our scheduling algorithms for advance reservation of lightpaths. Figures 3.9a and 3.9b show the effect of the arrival rate on the blocking probability. The rate is expressed in terms of number of requests/slot. From the graphs, we can see that SWF performs slightly better than SPF because it tends to schedule the connections on shorter paths. We also observe a significant drop in the blocking probability for both SWF and SPF schemes when load-balancing is used. We notice that for the 10 node fully connected network, all the schemes behave the same. Also, we have used the optimum value of k=1 for this experiment as discussed below. At k=1 SWF and SPF algorithms work in the same way so there is no difference between these schemes. The Load Balancing does

34 25 (a) NSF-NET (b) GEANT (c) Fully-Connected Figure 3.9: Arrival Rate vs Blocking Probability

35 26 not improve the blocking probability. The link utilization comparison between SWF and LB-SWF for NSF-NET is shown in figure These results are for an arrival rate of 60 requests/slot. The slope of the curves show how the load is balanced among the links with and without load balancing. The links are sorted in the order of utilization. We also note that for most of the links, LB-SWF achieves lower utilization than SWF. This is because the load is balanced among the links. Figure 3.10: NSF-NET Link Utilization : SWF vs LB-SWF The link utilization comparison between SWF and LB-SWF for 10-node fullyconnected network is shown in figure These results are for an arrival rate of 75 requests/slot. We can see that load-balancing does not improve the network utilization in this case. In our algorithms we try k alternate paths before blocking a request. To find the optimum value of k, we ran the simulation with different values of k. The results are shown in figures 3.12a and 3.12b. It was observed that the blocking probability decreases initially but then it slightly increases as k increases. As we increase the value of k, longer paths are reserved which indirectly affects the overall blocking probability. When k=1, SPF has same blocking probability as SWF because there are no alternative paths. The value of k with the least blocking probability varies with the network topology. It depends on the number of links in the network and is independent of the link costs. We have used the optimum

36 27 Figure 3.11: 10 Node Fully-Connected Network Link Utilization : SWF vs LB-SWF value of k for each scheme for the rest of the experiments e.g. it is 3 for SPF and 4 for SWF for NSF-NET and for GEANT it is 4 for SPF and 5 for SWF. But for the fully connected network it is 1 for all schemes because all nodes have direct shortest path to all the other node and having longer paths increases the overall blocking probability. We have also tested the sensitivity of optimum value of k to the link costs. We observed that optimum value of k depends on the topology and not on the link costs. In figure 3.13a and b, blocking probability is plotted versus different values of k for NSF-NET with reversed link costs and NSF-NET with cost of all links being 1. We can still see that optimal value of k for SWF is 4 and SPF is 3. The graphs in figures 3.14a and 3.14b show the effect of the connection duration d on the blocking probability. The duration is uniformly distributed with a minimum of one slot and a maximum d max. We can see that the blocking probability increases as d max increases. SWF with load balancing still performs the best. We also observe that the blocking probability significantly increases for d max > 5 for the considered topologies. Here again we see that for the fully-connected network all schemes give the same results. Figures 3.15a and 3.15b show the reservation delay, i.e., the time elapsed from the requested start time s to the time s + t where the reservation was actually made, as a function of d max for both SPF and SWF. We see that SPF always tries to schedule as close to the start time s of the scheduling window as possible.

37 28 (a) NSF-NET (b) GEANT (c) Fully-Connected Figure 3.12: k vs Blocking Probability

38 29 (a) NSF-NET Link Cost = 1 (b) NSF-NET Reversed Link Costs Figure 3.13: k vs Blocking Probability

39 30 (a) NSF-NET (b) GEANT (c) Fully-Connected Figure 3.14: d max vs Blocking Probability

40 31 (a) NSF-NET (b) GEANT (c) Fully-Connected Figure 3.15: Reservation Delay vs d max

41 32 We have also implemented a non-blocking version of the scheduler. In this case the requests are never blocked and scheduled at the first available time which can be outside the scheduling window. In this case, both the SPF and SWF algorithms keep on sliding the window until they find an available path. The difference between the two approaches is that SPF checks k different paths before sliding the window by one slot, while SWF slides the window by 2d slots before checking another path. The results are given for NSF-NET in figure In figure 3.16a, the average reservation delay is plotted versus the arrival rate for all four scheduling schemes. In figure 3.16b, the percentage of reservations outside the scheduling window are plotted versus the arrival rate for all the four schemes. We can see that at high arrival rates the network becomes unstable and around 90% [figure 3.16b] of the requests are reserved at a time outside the scheduling window. Also the delay increases tremendously to an average of 500 slots which is not desirable [figure 3.16a]. But at lower rates there is a low percentage of connections scheduled outside the window and also with a low reservation delay. We can see the SPF and LB-SPF schemes give better results because they tend to reserve the request at a time close to the start of the scheduling window. Obviously in case of a non-blocking scheduler, a stability condition needs to be developed in order to protect the network from becoming unstable, i.e., the reservation delay becomes very large. Such a condition is not necessary for a blocking scheduler, where a request will get blocked if it cannot be scheduled within its requested window [s,e]. For wavelength assignment, we have implemented two approaches First-Fit and Min-Gap as discussed earlier. We can see [figure 3.17] that the Min-Gap scheme with minimum leading and trailing gap give the lowest blocking. The First Fit scheme also performs well. The Best-Fit Min-Gap scheme has the worst performance because it leaves small unused gaps before and after a reservation. These small gaps cannot be used and this results in more fragmentation. We have studied how the requests with long and short durations affect each other. For this we generated requests which have a maximum duration d max =2 with 50% probability and d max =20 with 50% probability. We compared the results of blocking probability of both classes of requests with the results when they are not in the presence of each other. We ran the simulation experiment for NSF-NET. The requests arrive in a Poisson fashion. The advance reservation scheme used is LB-SWF with First-Fit. In figure 3.18, the blocking probability curves are plotted against the arrival rate for a) all requests have d max =20. b) all requests have d max =2. c) 50% of the requests have d max =2 and 50% of the requests have

42 33 (a) Average Reservation Delay (b) Percentage of Reservations outside the scheduling window Figure 3.16: Non-Blocking Scheduler

43 34 o (a) NSF-NET with 50 wavelengths/link (b) NSF-NET with 100 wavelengths/link Figure 3.17: Wavelength Assignment

44 35 Figure 3.18: Requests for long and short durations simultaneously d max =20. We have two curves for the last case, one for each class of request. These are labeled as Mixed d max(2) and Mixed d max(20). We see that both classes of requests affect each other and the blocking probability increases for both. The blocking probability of requests with short duration, i.e., d max =2 only was zero before and it increases with the presence of requests with d max =20. Same is the case with the long requests. We have also conducted simulation experiments with requests arriving in a bursty manner using the Interrupted Poisson Process (IPP). The IPP arrival process is an ON/OFF process where both the ON and OFF periods are exponentially distributed. The transition rates for the ON and OFF states are φ and ψ. During the ON state, the process generates Poisson arrivals with rate λ on whereas in the OFF state there are no arrivals. An IPP is bursty because the arrivals are batched together during the ON period and no arrivals occur during the OFF period and it becomes burstier when long OFF periods are followed by short ON periods with a large arrival rateλ on. An IPP is uniquely characterized through the parameters φ, ψ and λ on. We use the squared coefficient of variation c 2 to characterize the burstiness of the arrival process. This is the ratio of the variance to the squared mean of the interarrival time. A small c 2 represents interarrival times that concentrate mostly around the mean. For a Poisson process the c 2 is equal to 1. For an IPP the c 2 is always greater than 1 [15]. Figure 3.19 shows the comparison of the schemes under bursty arrivals. We use

45 36 Figure 3.19: IPP Arrival Process parameters φ, ψ and λ on to characterize and IPP. Given c 2 and λ avg, we can determine these with the following equations [15]. λ on = λ avg r (3.1) φ = 2λ avg(1 r) 2 r(c 2 1) (3.2) where, ψ = 2λ avg(1 r) c 2 1 (3.3) λ avg is the IPP average arrival rate and r is the coefficient of burstiness and is defined by r = ψ φ + ψ (3.4) We have set the value of c 2 = 10 and r = 2 for burstier arrivals. In the graph the blocking probabilities are plotted against various values for λ avg for NSF-NET and we can see that the schemes have the same trend as the Poisson arrivals. But the blocking is much

46 37 higher compared to the Poisson arrivals [see figure 3.9]. Load Balancing schemes still give better results and LB-SWF performs the best. Failure Recovery We used a failure model with link failures only. We assume that when a link goes down all wavelengths on that link fail. Only one link can fail at a time and this link is randomly selected from all the links. The mean time to failure is exponentially distributed with a mean of 80 slots. The recovery time is also exponentially distributed with a mean of 48 slots. The parameters of interest are the percentage termination and the overhead. Percentage termination is the percentage of reserved connections that cannot be re-routed and are dropped. The overhead is defined as the percentage of reserved connections that are unnecessarily re-routed as they were un-affected by the failure because the reservation start time turns out to be after the instant that link is repaired. Also we are interested in how the overall blocking probability is affected by the failure recovery process. We calculate the re-routing interval using the following three policies 1. The Fixed re-routing interval with feedback: The length of the re-routing is fixed. When the link goes down the connections are re-routed for this fixed interval. If the link does not come up at the end of the interval, the re-routing interval is increased by an amount equal to the fixed value. 2. Adaptive re-routing interval with feedback: In this case the re-routing interval is calculated using a moving average of the historical recovery times. If the link does not come up at the end of the interval, the re-routing interval is increased by an amount equal to the last re-routing interval length. 3. All future reservations are re-routed: In this case there is no re-routing interval, all the reservations on the failed link are re-routed. We compare all the three approaches with the ideal scenario where the failure duration is known and only the flows affected are re-routed. In figure 3.20a the results are shown for NSF-NET. We can see that the longer the re-routing interval the lower is the termination percentage, because at an earlier time more alternative paths are free and the reservations can be re-scheduled. If the reservations

47 38 (a) Percentage Termination and Overhead (b) Blocking Probability Figure 3.20: Failure Recovery are re-routed just before they are to start, the remaining alternate routes are more likely to be busy. But the overhead increases with the longer re-routing intervals. Since both the requirements are conflicting, there is a tradeoff between the two and best results are achieved when we use the adaptive method for determining the re-routing intervals. Having very long re-routing intervals also results in an increase in the overall blocking probability

48 39 in the network due to a large number of sub-optimal paths [see figure 3.20b]. Periodic Re-configuration In order to improve the network performance, we have implemented periodic reconfiguration of existing reservations. The network topology considered is NSF-NET. The re-configuration process occurs every 24 slots and re-configures the existing reservations for the next 24 slots without changing the reservation times. This is done by sorting the requests in an ascending order of their start times and then re-scheduling them one at a time starting from the request with the earliest start time. This process can result in terminations, in which case we restore the reservation table to its last state (i.e. no reconfiguration). We observed that the re-configuration process failed most of the time, i.e., the reservation table was restored to its last state due to at least one termination, especially at high rates. We also observed that if we allowed for a small percentage of terminations, the process succeeded most of the time. Figure 3.21a shows the blocking probability as a function of the arrival rate with and without re-configuration process. We can see that the overall blocking probability is slightly improved with the re-configuration. Also re-configuration which allows 2% or 5% blocking of the reserved connections, lying in the re-configuration period, further improved the overall blocking probability in the network. The overall blocking probability also includes the terminations during the re-configuration. In figure 3.21b, the percentage of time the re-configuration process fails is plotted against the arrival rate. The results show that at higher arrival rates the re-configuration fails more frequently, but this improves significantly if some amount of termination is permitted. With 5% allowed termination the re-configuration succeeds all the time even at higher arrival rates. From the results, we can conclude that using this method of periodic re-configuration for offline optimization has very little impact. This may not be the case, should a more efficient re-configuration algorithm be developed.

49 40 (a) Arrival Rate vs Blocking Probability - NSF-NET (b) Failure in Reconfiguration Figure 3.21: Periodic Reconfiguration

50 41 Chapter 4 Monitoring and Discovery for Grid Middleware Resource Monitoring and Discovery of large computational data Grids is essential to guarantee high performance and reliability. In this chapter, we identify the various resource monitoring and discovery issues and challenges that need to be solved in a large adaptive Grid environment. We survey the existing Grid monitoring solutions and discuss the monitoring and discovery framework that we have developed for the EnLIGHTened Computing Project. The EnLIGHTened testbed consists of a variety of devices like ethernet switches, routers, optical switches, clusters and storage devices. The goal is to monitor all these elements and obtain relevant information from each element to help the EnLIGHTened Resource Broker (ERB) in various ways. The network performance monitoring is needed for scheduling decisions, monitoring the current progress of jobs on the Grid, fault management and recovery, verifying QoS and lastly capacity planning and optimization. By discovery, we mean the process of finding a suitable resource to perform a task [25]. For example, a user may want to determine the best platform on which to run an application, a system administrator may want to be notified when changes in the system load occur or free disk space is available or a user having problems with the network may want to know the bottlenecks to fix them. This process is quite complicated in Grids because of their dynamic and distributed nature.

51 Existing Monitoring and Discovery Frameworks for Grid When Grid computing gained popularity, many groups developed Grid monitoring systems [35, 39, 40, 42]. An overview of these different monitoring tools and frameworks is given in this section Grid Monitoring Architecture (GMA) As the Grids are highly distributed and large, the Grid monitoring systems need to be scalable and interoperable. A general purpose information management system can not meet these requirements since the required characteristics of performance information are different. The Global Grid Forum (GGF) [6] defined a Grid Monitoring Architecture (GMA) to meet these requirements [38]. This architecture consists of three components: Consumers, Producers and a Registry/Directory Service. Producers collect information from sensors, register themselves with the Registry and describe the type and structure of information they want to make available to the Grid. Consumers can query the Registry to find out what type of information is available and locate Producers that provide such information. Once this information is known the Consumer can contact the Producer directly to obtain the relevant data. Three kinds of interactions are supported for transferring data between producers and consumers: publish/subscribe, query/response, and notification. In the first kind of interaction the initiator specifies interest in some kind of information and subscribes to it which it later terminates while in case of query/response, all the transfer of information as a result of a query is done in a single response. Notification is a one stage interaction done by the producer to the consumer. Apart from the describing basic architecture the GGF also specified certain guidelines for the implementation like fault tolerance, adaptability, scalability, manageability, non-intrusiveness, distribution, efficiency and security. As a result, many monitoring frameworks were designed to meet the requirements of GMA. These include the R-GMA, Globus MDS, NWS etc MonALISA MonALISA (Monitoring Agents in A Large Integrated Services Architecture) [31] provides a distributed service architecture for collecting and processing monitoring information developed for Grid systems. It was originally developed to support data processing and analysis of global high energy and nuclear physics collaborations but now it is used in

52 43 variety of Grids. This system consists of autonomous multi-threaded, self-describing agentbased subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. MonALISA is designed to easily integrate existing monitoring tools and procedures and to provide this information in a dynamic, self describing way to any other services or clients. These tools and procedures include SNMP, Ganglia, PBS, LSF, Condor, Abing, Iperf etc and modules for monitoring and controlling optical switches. In a Monalisa Service, the received values are stored locally into a relational database. This information can also be used to create WEB repositories components with selected information from specific groups of MonALISA services. The WSDL/SOAP interface is available in both the services and the repositories so that clients can access data from a specific Grid farm or, through a repository, they can access information received from several Grid farms. MonALISA is a robust and flexible monitoring system which can be used to create higher level services to perform scheduling decisions that adapt themselves dynamically to respond to changing load patterns in large Grids Globus MDS Monitoring and Discovery include obtaining, distributing, indexing, archiving and may be processing information about the configuration and state of services and resources. The Globus Alliance [7] has created the Monitoring and Discovery System (MDS) [36, 25, 37] that provides aggregator services to fulfill these requirements. These services collect latest information from the information sources and store it. It also provides interfaces, command line tools and web service interfaces to query and access the information. MDS4 has three types of aggregator services: Index, Trigger and Archiver. The Index Service supports querying the latest information. The Trigger Service performs specified actions as a result of particular events. Lastly, the Archiver Service stores the monitoring data in a persistent database that can be queried for historical information. In this informationaggregator framework all the information sources have to be registered with an aggregator service. These registration have to be renewed after some period to make sure that the information is fresh. Currently the MDS4 information providers include Hawkeye, Ganglia, WS GRAM, RFT and CAS and can accommodate any other WSRF service that publishes resource properties. The MDS-Index service makes the data collected from these sources available as XML documents. This allows users to write their own applications to collect

53 44 this information using Web service interfaces. Furthermore, the index services can register to each other in a hierarchy in order to aggregate data at several levels Network Weather Service The NWS (Network Weather Service) [42, 4] is a distributed system that monitors and forecasts the performance of various network and compute resources that form the Grid. The service consists of a set of performance sensors from which it gathers readings of the instantaneous conditions at regular time intervals. It then uses numerical models to predict near-term of performance of the system. It has been developed to be used by dynamic schedulers and to provide QoS in a networked computational environment. The NWS is made up of four components. The first being the sensors, measure bandwidth, latency CPU, memory and disk usage. Next, a Memory Host is used to store the monitored data temporarily. The sensors and memory host register themselves with the Name Server. Lastly the Forecaster is used to make predictions on the monitored performance information. Presently, mean-based, median-based and autoregressive methods are used for forecasting. NWS identifies the best forecasting technique for a give resource by applying all of them and choosing the one with the lowest cumulative error, thus providing greater accuracy PerfSONAR perfsonar (PERFormance Service Oriented Network monitoring ARchitecture) [22, 5] is an infrastructure for network performance monitoring concentrating on the end-to-end performance. The perfsonar architecture is developed through a collaboration between ESNet, GEANT2 and Internet2. It contains a set of services delivering performance measurements in a multi-domain environment. The project greatly focuses on standardization of interfaces, usability and security. The design goals for this project include flexibility, extensibility, openness, and decentralization. Hence, perfsonar is an effort to automate monitoring data exchange between networks by simplifying troubleshooting performance problems occurring between sites connected through several networks. The services act as an intermediate layer, between the performance measurement tools and the diagnostic or visualization applications. In the perfsonar architecture, there are seven services in all. The Measurement Point Service to create and publish monitoring information, a Measurement Archive Service to stores and publish monitoring information

54 45 retrieved from the Measurement Point Services, a Lookup Service to registers all services, an Authentication Service for managing domain-level access to services via tokens, a Transformation Service that offers custom data manipulation of existing archived measurements. The Resource Protector Service manages granular details regarding system resource consumption and lastly the Topology Service offers topological information on networks. 4.2 EnLIGHtened Computing Monitoring Framework In this section we discuss the various performance measures that are required by the ERB to select and allocate Grid resources. We discuss the monitoring characteristics of the network and the compute resources separately Network Performance Measures Optical Path Characteristics The EnLIGHTened testbed consists of four GMPLS-enabled optical switches, see Figure 1.1. We currently monitor the connections and cross-connects at the optical switches. The presence of light determines the establishment of a connection and thus we monitor the power level at the ports. The link status can be determined by the presence of light between the two ports. The active cross-connects at the optical switches are also monitored in the same way. Bandwidth Bandwidth is defined as the data transferred in a unit time. Bandwidth can be measured at both path and link level. We currently measure the incoming and outgoing traffic at the interfaces of the network elements for each link as well as the end-to-end bandwidth to determine the bottleneck bandwidth for the entire path. There are four characteristics [33] that describe the bandwidth of a path i.e capacity, utilization, available bandwidth and achievable bandwidth. Each of these can be used to describe characteristics of an entire path as well as hops. Delay and Jitter Delay is also a very important parameter especially for real-time applications. It is the time taken by a packet to transfer from one end point to the other. There are many

55 46 existing tools to measure end-to-end and round trip delays line ping, pinger etc. Jitter is the variance in the delay and it is a critical parameter for the voice and video applications. Jitter is used in sizing playout buffers for applications requiring regular delivery of packets. Packet Loss Packet loss can impact the quality of service provided by network application programs. The sensitivity to loss of individual packets, as well as to frequency and patterns of loss among longer packet sequences is strongly dependent on the application itself [33]. For streaming media, packet loss results in reduced quality of sound and images. For data transfers, packet loss can cause severe degradation of achievable bandwidth. Network Topology The Grid middleware needs to have the network topology for scheduling decisions because the resources are located at different sites with different domains and policies. This information is needed by the DNRM for path computation. The ERBs of different domains will exchange the abstracted topology information for path computation across multiple domains. The topology information can also be used for optimization of the network. In the EnLIGHTened computing project we use OSPF-TE for distribution of routing information. We obtain the topology information from the Link State Database of one of the optical switch in the domain. For the ethernet switches this information can be obtained using SNMP MIBs if they support the OSPF MIBs. Traffic Flows Analysis A flow is identified as a unidirectional stream of packets between a given source and destination both defined by a network-layer IP address and transport-layer source and destination port numbers. This detailed traffic flow information can be used for application and user monitoring and profiling, network planning, ensuring and verifying QoS Compute Resource Characteristics Cluster Monitoring The compute resources in a Grid consist mainly of large computer clusters. The clusters have different types of management systems and job schedulers such as LSF, PBS, Torque, Moab, LoadLeveler etc. Using these job management systems, we get information

56 47 like number of jobs running, finished, queued etc. We also monitor the load, cpu usage, memory usage, disk usage of each node that are part of the cluster. Since jobs from several Grids might be running on the same cluster, the monitoring modules need to interact with the Grid middleware to get the information related to only those jobs that are scheduled by the middleware for EnLIGHTened Grid. Application Specific Monitoring and Discovery The ERB also requires application specific monitoring and discovery information. For example, a visualization Grid application may need information about the visualization servers such as hardware information, graphic cards, resolution etc., and the dynamic information like cpu usage, load and bench marking performance data. 4.3 Monitoring Issues The Monitoring system should collect and deliver monitoring information reliably and on-time. But there are several issues that have to be dealt with to ensure this reliability and robustness. These are explained in detail below: Information Collection The monitoring information encompasses a variety of parameters collected from different Grid components using several tools and techniques discussed before. There are many implementations already available for many of the measurements. We need to compare these and figure out the most accurate and suitable and make modifications and enhancements to existing solutions wherever required and possible. The architecture has to be scalable to accommodate more tools without affecting the entire system Representation Format There should be a universal way of publishing and delivering the monitoring data to the Grid middleware. To ensure that, there is a need to use universal technologies like WSDL, XML and SOAP. It is important to use standard interfaces, so that in case of collaboration between two different grids, the two Resource Brokers may obtain the monitoring and discovery information without making any changes to the current implementation. For the EnLIGHTened Computing project, we have chosen XML and WSDL interfaces for the

57 48 time being as they have become universal description languages and are being used by the majority of the Grid community Update Frequency The frequency of update from different monitoring sensors plays a very important role in determining the overall load of monitoring on the Grid. The main challenge is to collect this information accurately at near real-time basis without overloading the network to avoid conflicts and blocking. For example we collect the optical switch information using TL1 currently. A higher frequency of these queries increases the CPU and memory consumptions the switch. Similarly end-to-end bandwidth measured by Iperf introduces heavy traffic on the network, so may be its better to conduct a test when required or after longer time periods. We need to analyze the trade offs to determine the right frequency for different types of tools while not affecting the freshness of information. Because if it is collected after long time intervals it is old and will not be very effective Non-Intrusiveness Different monitoring techniques require different amount of resources themselves. The load incurred by the monitoring tools should be insignificant part of the Grid usage. Similarly we do not want the monitoring applications to require any root privileges or special permissions etc. on the network elements and compute resources Monitoring Strategies Active and Passive Monitoring A monitoring tool is considered as active if its measurements are based on traffic it induces into the network, otherwise its passive [14]. Passive monitoring tools can give a very detailed view of the network while active ones return a response that combines several performance figures. Active monitoring is more effective in measuring network sanity and for application specific measurements, while passive monitoring is mainly used for accounting purposes and measurements like throughput, roundtrip time etc. Both techniques are complementary to each other.

58 49 Emphasis on Standard Protocols As discussed above, it is not a good idea to use special permissions or administrator privileges for the monitoring applications. Instead, we need to choose monitoring techniques that require minimum of these. We must select standard protocols and methods like SNMP for management of network devices. Link vs Path Monitoring End-to-end path monitoring gives a view of a system that is filtered through routing. This type of monitoring is very useful for Grid aware applications as they give a summarized view of the network that helps the ERB in resource selection considering the network path characteristics. The single link monitoring gives a view from a single observation point. These measurements give a very detailed view of the network and can be used in failure detection Distribution of Monitoring Data The monitoring data is stored at some place to be used by the middleware. It is recommended that the repository where it is stored should not be centralized. It is better to have several small repositories with each site to avoid single point of failure. The second problem with centralized data management is that it forms a performance bottleneck. For dynamic data writes often outnumber reads with measurements taken every few seconds or minutes [38] Security Since the monitoring information will include the detailed system information, network configurations and topology, it is important to secure this data from malicious users and attackers. Public key certificates can be used to enforce authentication. There should be an access control list to avoid any intrusions Fault Tolerance In large-scale distributed systems, failures can occur quite often. The monitoring applications can run into errors or the system on which they are installed may go down. For such scenarios, we should have some fault tolerance. The system should be able to recover from these failures and data should be replicated to avoid any losses.

59 50 Figure 4.1: Monitoring Architecture 4.4 Monitoring Architecture A distributed monitoring system is needed to monitor different sites that are part of the Grid. Figure 4.1 shows the Monitoring Architecture we propose to use for the En- LIGHTened Computing project. There is a monitoring service at each site which monitors the clusters and all network elements and records the performance data in a local database. There can be various tools integrated with the service. All the monitoring services report their data to a centralized repository for the entire Grid. This larger repository database maintains history information and can be used for detailed analysis. Web services are used to extract data from all of the databases. Also, all the monitoring services are registered with a Registry. The Grid Resource Broker first queries the Registry for the available monitoring services and then interacts with them directly.

60 51 Figure 4.2: MonALISA client GUI : current monitoring information from the EnLIGHTened testbed can be viewed 4.5 Current Deployment Scenario Currently we have MonALISA services running at four Grid sites in Raleigh, Chicago, Baton Rouge and Los Angeles. These services have their own databases and web services to store and publish the information. We also have a web repository to collect relevant data from all these sites to maintain history and detailed analysis. These services can be seen in the Figures 4.2 and 4.3. We plan to deploy other network monitoring tools for end-to-end measurements, network topology discovery and traffic flow analysis. In future we propose to use standard WSDL interfaces to retrieve data collected from these various tools in the same standard format such that it can be used by any consumer.