Addressing and secure control-plane network design in GMPLS networks

Transcription

1 Addressing and secure control-plane network design in GMPLS networks Abstract 1 : Malathi Veeraraghavan, Xuan Zheng, Xiangfei Zhu {malathi, xuan, xz4p}@virginia.edu April 7, 2006 This document describes a control-plane network design for general scalable GMPLS networks. We use the Circuit-Switched High-speed End-to-End ArcHitecture (CHEETAH) network as an example to illustrate how we implemented our design. The CHEETAH network is a Synchronous Optical NETwork (SONET) based network that uses Sycamore SN16000 switches. While some of the specifics related to this switch may not be applicable to other switches, in general, most of the concepts presented here will apply to any GMPLS network. Our control-plane network design uses the Internet to create out-of-band channels between end hosts and GMPLS switches as well as between neighboring switches. First, we consider the question of what type of IP addresses, static or dynamic, public or private, to assign to control-plane interfaces on switches and end hosts. Our conclusion is that we require static public IP addresses if the goal is to create scalable GMPLS networks. Given the shortage of such IPv4 addresses, we recommend the use of IPv6. Second, we note that the Router ID/Switch IP loopback interface addresses assigned to GMPLS switches should be advertised through routing protocols, allowing them to be reachable through at least one interface on the Internet. Third, to secure the control-plane channels, we describe the use of IPsec tunnels. Using open-source Linux software called Openswan on the end hosts and Juniper NS-5XT devices to protect control ports of switches, we use host based authentication and encryption of RSVP-TE and OSPF-TE messages. Finally, we propose a mechanism to handle IP and MAC addressing on the data-plane in GMPLS networks. When an end-to-end circuit/vc is established, conventional IP networking dictates that the two ends of the connection should be in the same IP subnet. But this leads to an unscalable solution requiring the data-plane interfaces of all hosts on a GMPLS network to be assigned addresses within one subnet. Our solution is to assign IP addresses to these interfaces in different subnets, based on the 1. This work was carried out under the sponsorship of NSF ITR , NSF ANI , NSF ANI , and DOE DE-FG02-04ER25640 grants. 1

2 enterprise within which hosts are located, and to then use IP routing table and ARP table updates to add host-specific entries when circuits/vcs are setup. 1. Introduction Several high-speed optical networking projects, such as CHEETAH [1], DRAGON [2], OMNInet [3], UltraScience Net [4], and CANARIE s CA*net4 [5], have proposed the creation of end-to-end circuits with segments at the ends mapped to SDM/TDM/WDM (Space-/ Time-/Wavelength-Division Multiplexed) circuits across the wide area. Techniques such as Generic Framing Procedure (GFP) [6] and Digital Wrapper [7] are used to map frames onto SONET circuits and WDM lightpaths, respectively, and port-mapped untagged Virtual LANs (VLANs), based on IEEE 802.1q implementations, are used to realize SDM circuits. Such host-to-host dedicated circuits at 1Gbps or 10Gbps are being proposed for fast and predictable file transfers as needed in high-end scientific applications. A few of these projects, such as CHEETAH and DRAGON, are experimenting with GMPLS control-plane protocols to allow for the dynamic setup and release of these end-to-end - SDM/TDM/WDM- circuits. GMPLS control-plane protocols include RSVP-TE for signaling [8], Open Shortest Path First - Traffic Engineering (OSPF-TE) for routing [9], and Link Management Protocol (LMP) [10]. The switches used in the CHEETAH network are SONET switches, specifically Sycamore SN16000 switches, while the switches used in the DRAGON network are WDM switches, specifically Movaz switches. Both these switches implement RSVP- TE and OSPF-TE. For the CHEETAH network [11], we proposed equipping end hosts with two Network Interface Cards (NICs), primary and secondary, as illustrated in Fig. 1. There are many advantages to this architectural solution. First, it would allow an end host access to other hosts on the Internet through its primary NIC while its second NIC is tied up in an -SONET- circuit. Second, this Internet connectivity can be used as a fall-back option if the circuit setup fails for some reason, such as lack of available bandwidth. Third, it allows for a gradual growth of the GMPLS network, by allowing hosts to be upgraded one at a time. If a host with connectivity to a GMPLS network through its second NIC wants to communicate to a host without such capability, it would simply use the primary Internet path. We proposed using the Domain Name Service (DNS) with the TXT resource record to determine if the far-end host also has connectivity to the 2

3 same GMPLS network prior to requesting a circuit setup. Fourth, the Internet connectivity (through the primary NIC) can be used for transporting control-plane messages for the GMPLS network. In other words, control-plane messages (primarily signaling) from an end host to the GMPLS switch to which its secondary NIC is connected, is carried over the Internet. Control-plane traffic Data-plane connection Internet NIC 1 control port control port NIC 1 GbE/ GMPLS OC192 OC192 GMPLS GbE/ RSVP-TE 10GbE engine card card engine 10GbE RSVP-TE Software card card Software Control Control NIC 2 End Host 1 card SN card SN NIC 2 Fig. 1 Use of two NICs at end hosts in the CHEETAH network Similarly, switch-to-switch signaling and routing messages are also carried over the Internet. GMPLS switches, such as the Sycamore SN16000s used in the CHEETAH network, have control cards whose ports (which we refer to as control ports to distinguish them from the data ports on the Gigabit (GbE) and 10GbE interface cards) are connected to the Internet as shown in Fig. 1. For switch-to-switch control-plane connectivity, the SN16000 offers the network administrator two choices: (i) in-band Data Communication Channels (DCC), embedded within SONET data-plane interfaces, and (ii) out-of-band control ports. In the CHEETAH experimental network, we choose the latter for switch-to-switch control-plane communication simply because we needed the ability to capture control-plane messages for experimental reasons. We assume the use of out-of-band signaling in our general treatment of the GMPLS control-plane network design problem addressed in this paper. Under the above assumptions for the architecture and use of GMPLS networks (i.e., the use of second NICs at end hosts and the creation of end-to-end dedicated circuits mapping links to SDM/TDM/WDM circuits), we consider the questions of addressing and the design of a secure control-plane network. In GMPLS control-plane protocols, while different addressing 3

4 schemes are supported in the standards, the expectation is to most commonly use IP addresses. In this paper we consider the type of IP addresses suitable for the data-plane and control-plane interfaces of switches and end hosts. The design of a control-plane network to support a connectionoriented network is an interesting problem. For example, the Signaling System No. 7 (SS7) network that supports the connection-oriented telephony network is not trivial. In this document, we describe the design of the control-plane network for GMPLS networks, and more specifically in the CHEETAH experimental network. The control-plane network in GMPLS networks is referred to as the Data Communication Network (DCN). Since this term is not very descriptive of its functionality, we prefer the term Control-Plane Network. On the face of it, setting up the control-plane network seemed to be as simple as merely connecting end hosts and control ports of switches into the Internet and using the latter for controlplane messages. However, the actual solution turned out to be much more complex. There are three key areas in which we needed to make design decisions. First, GMPLS switches, such as Sycamore SN16000 switches that are used in the CHEETAH network, require the assignment of multiple control-plane IP addresses. Besides needing IP addresses for the two control ports, these switches need to be assigned a Router ID and a Switch IP. We discuss the purpose and usage of these different addresses in Section 2. In Section 3, we discuss the type of IP addresses required for the primary NIC (control-plane) of end hosts and for the various control-plane IP addresses in the GMPLS switches, whether these addresses should be private or public, static or dynamic. We rule out the use of dynamic IP addresses because RSVP-TE and Open Shortest Path First - Traffic Engineering (OSPF-TE) software require Traffic Engineered (TE) links to be configured with local and remote data-plane and control-plane IP addresses at the SN16000s and end hosts. Private IP addresses pose no problem if all end hosts only run the client end of applications; however, additional mechanisms are required for address discovery if end hosts run the server end of applications. This is because a server needs to be reachable. For example, the end-host RSVP-TE software on CHEETAH end hosts is essentially a server module for incoming calls because it needs to be reached [12]. Although we can make private IP addresses reachable with the help of VPN (IP-in-IP) tunnels, the solution will lack scalability because private IP addresses allocated at different geographically distributed campuses have to be unique within the VPN. Given these problems above, we recom- 4

5 mend the use of public static IP addresses for control-plane interfaces of end hosts and for the various control-plane IP addresses on the GMPLS switches. In Section 4, we describe our usage of IP-in-IP tunnels between GMPLS switches for OSPF adjacency. In Section 5, we discuss how control-plane channels between end hosts and switches, and between switches are secured. It is important to secure these control-plane channels to prevent malicious users from tying up bandwidth resources. We selected Juniper s NS-5XT device and Openswan software for Linux, through which we can set up IPsec tunnels to provide security for control-plane channels. These devices provide IPsec capabilities for authentication, integrity and privacy, Network Address Translation (NAT) functionality, and support for tunneling. In Section 6, we describe the actual control-plane setup in experimental CHEETAH network. Finally, in Section 7, we discuss the type of IP addresses required for the secondary NIC (data-plane) of end hosts, and the data-plane interfaces of GMPLS switches. Section 8 presents our conclusions. 2. Assignment of control-plane IP addresses for a GMPLS switch GMPLS switches need to be configured with multiple IP addresses for control-plane operation. For example, the SN16000 in the CHEETAH network needs four IP addresses, which consists of addresses for the two control ports on the control card, a Router ID and a Switch IP. 2.1 control port IP addresses There are two control ports on the control card of the SN Two ports are provided for redundancy. Ideally the two should be assigned IP addresses reachable through two distinct subnets of one Internet Service Provider (ISP) or better still through two different ISPs for high reliability. These ports are used by administrators for switch configuration, as well as by RSVP-TE and OSPF-TE control-plane messages. 2.2 Router ID Before discussing the purpose and usage of Router ID in SN16000s, we review how this parameter is used in Cisco routers because more administrators are familiar with the latter Usage of Router ID in Cisco routers Router ID is used in the OSPF header of all OSPF messages as shown in Fig. 2. It is an IP address but the OSPF specification does not provide any recommendations on how this value should be selected. In the OSPF implementation of Cisco GSR routers, there is an explicit 5

6 command call router-id to set the Router ID [13]. If no explicit router ID is set, reference [13] states OSPF uses the largest IP address configured on the interfaces as its router ID. If the interface associated with this IP address is ever brought down, or if the address is removed, the OSPF process must recalculate a new router ID and resend all its routing information out its interfaces. To avoid such an overhead, Cisco documentation suggests the use of a loopback interface address. Version Type (1-5) Router ID Area ID Packet Length Checksum Authentication Authentication Authentication Type Fig. 2 OSPF Common Header If a loopback interface address is configured, this is automatically chosen by the Cisco IOS as the Router ID. The concept of loopback interface address is to allow the IP layer receiving a packet destined to this address from a higher layer to be looped back at the IP layer and not sent out onto the network. A commonly used value for this loopback address on end hosts is If two processes on the same host use sockets to communicate with each other, they typically use this IP address. The domain name associated with this address is localhost (e.g., ping localhost). An example of a route print command on a Windows host is shown in Fig. 3 (see line 5 for the loopback entry). Note that to reach the public IP address assigned to the NIC card ( ), the routing table points to the loopback address as the next hop (see line 3). Thus packets addressed to this address will also be looped back in the IP layer. Returning to the issue of assigning a router ID, Cisco routers are programmed to use the loopback interface address (if set) as the router ID even if one of the router s interfaces has a larger IP address. Since the loopback interface is not a physical interface, there is a smaller probability of 6

7 Active Routes: Network Destination Netmask Gateway Interface Metric Default Gateway: Fig. 3 Printout of route command on a Windows host whose NIC s IP address is it failing (software failure), and thus changes to the router ID are unlikely. Commands to configure a loopback interface are shown in Fig. 4. To configure a loopback interface on a Cisco router 1. interface loopback 0 2. ip address address-mask Fig. 4 Commands to set the RouterID on a Cisco router In addition to the above two ways for the Cisco router software to select a router ID to use in OSPF messages, in some versions of the Cisco router IOS software, a command is provided to explicitly set a router ID. To verify our understanding of router ID, we conducted an experiment. Fig. 5 shows the experimental setup with two Cisco GSR routers and two zebra-based Linux [14] routers. The router IDs on routers are set as shown in the figure, while the router IDs for the zebra-based Linux routers are set to be equal to the interface IP addresses. These zebra-based Linux routers are Linux computers running the zebra implementation of OSPF [14]. OSPF is enabled on all interfaces of the GSRs and zebra-based Linux routers. After OSPF adjacencies were established between the Cisco GSRs routers and the zebra-based Linux routers, we captured OSPF packets on the GbE interfaces with Ethereal [15]. Fig. 6 shows a snapshot of Ethereal output displaying an OSPF Link State Update packet sent from GSR B to zebra-based Linux router II. As expected, the GSR B uses its loopback IP address ( ) as its router ID (see the Source OSPF Router line in Fig. 6) in the OSPF common header (Fig. 2). The payload part of the OSPF Link State Update shows a Router LSA originated by GSR A (Advertising router: ). See Fig. 7 for the format of a Router LSA. 7

8 Internet Internet control port control port Router ID/ loopback IP Router ID/ loopback IP Router ID/ loopback IP Router ID/ loopback IP GbE interface GbE interface POS interface POS interface GbE interface GbE interface Zebra-based Linux router A GSR A GSR B Zebra-based Linux router B Fig. 5 OSPF experiment setup with Cisco GSR routers The link Type field can be set to one of four values: (a) Point-to-point connection to another router: 1; (b) Connection to a transit network: 2; (c) Connection to a stub network: 3; and (d) Virtual link: 4. For Type 3 (stub networks) links, the Link ID is the IP address and Link Data is the subnet mask. The Link Type field for the loopback interface (Link ID: with a /32 mask) is set to be a stub network (see highlighted line in Fig. 6). In other words, the loopback interface address is advertised as if it were a stub network on the router even though it does not correspond to any physical network. Advertising the loopback interface as such through the physical interfaces of the router, e.g., the GbE and Packet-over-SONET (POS) interface on the GSRs (see Fig. 5) allows the rest of the routers in that OSPF area within which the router is located to note this loopback interface reachability. When an administrator uses the router ID instead of the control port IP address to connect to the router, the IP packets carrying application data from the administrator s host is 8

9 Fig. 6 OSPF LS Update packet sent by GSR B 32 bits LSA header 0 V E B 0 #links per-link fields Type #TOS Metric Link ID Link Data Repeat per-link fields for each link Fig. 7 OSPF Router LSA format routed via the data-plane interfaces (GbE and POS) of the router. Note also in Fig. 6 that the control port IP address ( ) is advertised as a Transit link. This means if this address is used as the Destination IP address to reach the router, the router can also be reached via the data-plane interface. In other words, both the control-plane related IP addresses (the Ether- 9

10 net control port address and loopback interface address) are advertised as reachable addresses by the router. By making the loopback address routable, it will be reachable even if one or more interfaces on the router fails. The loopback address can thus be used not only in the Router ID field of the OSPF message header but also in the Destination IP address of the IP header. If a specific router interface s IP address is used as the Router ID, if that interface fails, then the router needs to advertise its link state database because the Router ID would need to be changed and the LS database in the other routers associates LSAs with the Router ID of the advertising router for each LSA. This explains the choice of the loopback address as the Router ID. Note that the loopback interface of a router should be on a different subnet from the IP addresses allocated to all its physical interfaces. This is because the loopback address is advertised as a separate interface. Two different interfaces of a router cannot be assigned the same subnet number unless it is a multicast IP address. If OSPF is enabled to spread reachability for this loopback interface in a GMPLS switch, the GMPLS service provider would need its collocation service providers (where the GMPLS switches are located) to allocate an IP address for this stub network loopback interface in a subnet distinct from the subnets of the control ports Usage of Router ID in Sycamore SN16000 node The Router ID in a Sycamore SN16000 is used in OSPF message headers just as in Cisco routers. Like with the Cisco IOS, there is an explicit command to set the Router ID through the Command-Line Interface (CLI) [16] as shown in Fig. 8. To assign the router ID: 1. At the IP Config> prompt, enter the Routerid command. For example: IP Config>routerid Set successful 2. Continue to the next section to configure the switch IP address. Fig. 8 Commands to set the RouterID on the SN Switch IP in Sycamore SN16000 nodes We start by understanding the purpose and usage of the Switch IP parameter in the Sycamore SN16000 switch. Reference [16] says the following The Switch IP interface on the SN is equivalent to the loopback interface (or virtual/circuitless IP interface) on routers from manufacturers other than Sycamore. The switch IP address, which must be outside the subnet range of a configured IP interface, is routed using OSPF as a stub link with an automatic prefix 10

11 length of 32 (mask ). To assign the switch IP address, the commands shown in Fig. 9 are executed. 1. At the IP Config> prompt, enter the Switchip command. For example: IP Config>switchip Set successful 2. Continue to the next section to verify the IP address settings. Fig. 9 Commands for set the SwitchIP on the SN16000 Reference [16] shows as an example in which the same IP address is used as both the Router ID and Switch IP address. The purpose of the Switch IP address is for use in RSVP-TE messages in the RSVP_HOP object, just as Router ID is used in the OSPF-TE message headers. The control-plane address corresponding to TE links configured at the switch is set to be the Switch IP address. For example, when a switch sends a Path message to an end host, the switch will use Switch IP as the source IP address in the IP header. Also, it will set the value of the Previous Hop IP address field in the RSVP_Hop object to be the Switch IP. When the end host sends back a Resv message, it uses the value in the Previous Hop IP address as the destination IP address of the Resv message. This means the Switch IP should be a routable address. By allocating the loopback address to be the Switch IP instead of one of the control port IP addresses, even if one of the control ports fail, the RSVP database holding state information about current connections will remain unchanged. This database needs to keep track of previous-hop and next-hop control-plane IP addresses. The reason for recommending the use of Switch IP is that the latter can be configured through OSPF to be reachable via many links (as with the Router ID) and thus even if the control ports fail, the RSVP database can remain unchanged. Note that OSPF is configured per interface on a SN Therefore, if OSPF is configured on the control ports, then the loopback address (Switch IP) is sent in a Router LSA on this interface (telling the neighboring switches that this address is reachable through this router). If inband DCCs are set up, and OSPF is enabled on these DCC links, then again, the Switch IP loopback address will be advertised through those interfaces. Fig. 10 gives an example of Sycamore SN16000 setup. If the control ports fail on an SN16000 (say SNA), then the RSVP-TE software on SNA is still reachable through a neighboring SN16000 (say SNB) via the corresponding inband DCC if the loopback address that 11

12 is assigned as Switch IP is advertised as being reachable via a stub-network at SNA. OSPF is enabled by default on inband DCCs, but needs to enabled manually on the control ports. Internet Internet control port control port Control-plane Software RouterID RouterID /Switch /Switch IP IP SONET interface SNA DCC channel SONET circuit Control-plane Software RouterID RouterID /Switch /Switch IP IP SONET interface SNB Fig. 10 Use of Router ID/Switch IP in Sycamore SN16000 nodes Effectively, the Sycamore software implements IP datagram forwarding functionality (like a small IP router), forwarding IP datagrams destined to the stub network represented by the loopback address (Router ID/Switch IP). Thus RSVP messages with this Switch IP address in the Destination IP address field of the IP header will be routed to the RSVP software module with the SN This solution of using a loopback address as Switch IP and advertising it into the Internet is a flexible robust solution. Even if OSPF is not enabled on the SN16000 control ports to spread reachability information about the switch IP, static routes can be added in the default gateway of the SN16000 control ports (i.e., one of the collocation service provider s routers). This entry sets the Switch IP address as being reachable through the control port address. This provides less reliability in being able to reach the Switch IP via any of the DCC channels or control ports of the SN16000 than if OSPF is enabled. 12

13 Note that the use of OSPF for spreading reachability about control-plane addresses (Router ID/ Switch IP and control port addresses) has nothing to do with the OSPF-TE exchanges between two SN16000s to spread reachability of data-plane addresses (secondary NICs at hosts). The former is an OSPF exchange between the SN16000 and IP routers in the enterprise network of the SN16000 collocation service provider. In summary, we set the Router ID and Switch IP to be the same loopback interface address in our SN16000, and chose this address to be in a distinct subnet from the control-port addresses. Reachability for this loopback interface address should be advertised through OSPF. Since the CHEETAH network is an experimental network, we disabled OSPF from running on interfaces between the SN16000 control ports and production IP routers in the LAN to which they are connected. In a production GMPLS network, this OSPF interaction between IP routers and GMPLS switches should be enabled. Without it, the very purpose of allocating a loopback interface address as the Router ID/Switch IP instead of the control port addresses is lost. 3. Type of IP addresses for use in the control plane In Section 2, we described the need for three control-plane IP addresses (two for the two control ports and one for the Router ID/Switch IP) at each SN16000 switch. End hosts send their control-plane messages via their primary NIC. Therefore, we consider the primary NIC as the control-plane port for end hosts. In this section, we address the question of whether these control-plane addresses at the end hosts and switches need to be public static addresses or whether they can be private addresses or dynamic addresses. We describe four types of addresses and mechanisms used to handle each of these types of addresses in the Appendix 9.1. In this section, we consider the possible use of these four types for the control-plane interfaces of a CHEETAH network. 3.1 Use of static public IP address in the control plane If all end hosts in CHEETAH network have routable static public IP addresses assigned to their control-plane ports (primary NICs) and switches have routable static public IP addresses assigned to their control ports and Router ID/Switch IP parameters, RSVP-TE messages can be sent directly on IP with the destination IP address carried in the IP header corresponding to the control-plane port of a host or Switch IP of a switch. We illustrate an example of a GMPLS net- 13

14 work that uses static public IP addresses on its control-plane in Fig. 11. Switches located at the three Points of Presence (PoP) are assigned public IP addresses. The primary NICs of end hosts, which serve as the control-plane interfaces are also assigned static public IP addresses. Internet Internet control port: Host 3 control port: (routerid/ switchip: ) control port: (routerid/ switchip: ) control port: (routerid/ switchip: ) control port: Host 1 Data-plane interface PoP 3 Data-plane connections Control-plane connections PoP 2 control port: Data-plane address PoP 1 Data-plane interface ID Data-plane address Host 2 Fig. 11 Assigning public IP addresses in a GMPLS network for the control-plane and end host data-plane interface Traffic-Engineered (TE) links need to be configured at the GMPLS switches. Link configuration includes specification of the data-plane addresses (local and remote) and the remote controlplane address. For example, we configure two TE-links at PoP1 of Fig. 11 using the parameters shown in Table 1. 2 Table 1: TE link configuration at the PoP1 GMPLS switch Link Local dataplane interface ID Remote end data-plane interface ID Remote end control-plane address (Router ID/Switch IP) Capacity Link to PoP OC In this example, We show unnumbered data-plane interfaces on both switches and end hosts. We will give the detailed discussion on the data-plane addressing issue in Fig

15 Table 1: TE link configuration at the PoP1 GMPLS switch Link Local dataplane interface ID Remote end data-plane interface ID Remote end control-plane address (Router ID/Switch IP) Capacity Link to end host A Gbps When a call setup request (an RSVP-TE Path message) arrives at a switch, it needs the Switch IP address of the next-hop switch or control-plane address of the destination host toward which it should route the setup procedures. If the TE-link configuration process is static in most switches (as with most GMPLS switches), we need to make these control-plane IP addresses static. If they change dynamically, the RSVP-TE software at a switch will need to be updated dynamically to change this TE-link information. To avoid this overhead, we recommend the use of static public IP addresses for all control-plane ports. 3.2 Use of dynamic public addresses in the control plane If the control-plane addresses of end hosts are dynamically obtained, then the TE-link information at their neighboring switches needed to be updated dynamically. We cannot make the controlplane IP addresses of switches dynamic because then the end hosts could not even contact the switches to update the TE-link information. 3.3 Use of private IP addresses (static and dynamic) in the control plane We cannot use dynamic private IP addresses with NAT because of the problems already cited with dynamic addresses in Section 3.2. As for using static private IP addresses with VPNs (by establishing IP-in-IP tunnels), the problem is a lack of scalability. This is because private IP addresses allocated at different geographically distributed campuses have to be unique within the VPN. Since GMPLS networks (with their distributed control-plane design) are meant to be scalable to a global level, a global scale VPN cannot be built with private IP addresses. As end hosts belonging to different enterprises can connect into the GMPLS network, they cannot all be administered from a static private IP address space by the GMPLS network service provider if largescale network evolution is a goal. Switch control ports and Router ID/Switch IP of the internal switches within a single provider s network can be assigned static private IP addresses but for all border switches, we need static public IP addresses. Otherwise an administrator from one service provider will need to control IP addresses of switches within another domain. This is 15

16 illustrated in Fig. 12, which shows four Autonomous Systems (ASs), Enterprise A, Enterprise B, control port: Service provider A s border GMPLS switch GMPLS switch control port: Service provider A s border GMPLS switch control port: Service provider B s border GMPLS switch GMPLS switch control port: Service provider B s border GMPLS switch Service provider A VPN router VPN router Service provider B VPN router Internet Internet VPN router control port: host host control port: Enterprise A s GMPLS switch Enterprise A s GMPLS switch host host Enterprise A Data-plane connections Control-plane connections VPN tunnels Enterprise B Fig. 12 Scalability problem with the use of static private IP addresses through VPNs Service provider A, and Service provider B. The control ports of the hosts and the switches are connected to VPN routers, one per AS. The VPN routers are in turn interconnected by tunnels through the Internet. Private IP addresses allocated within the four ASs have to be unique as described in Section Summary recommendation Given the problems cited in Sections 3.2 and 3.3 to handle dynamic or private addresses, we recommend the use of public static IP addresses for control-plane interfaces of end hosts and for the control-plane and Router ID/Switch IP addresses on the GMPLS switches. Due to the shortage of public IPv4 addresses, we recommend the use of IPv6 addressing. 16

17 Typical enterprise administrators use firewalls to limit access to their nodes that are assigned static public IP addresses. This is a useful technique to prevent malicious users from accessing hosts with GMPLS connectivity. Appendix 9.2 outlines different firewall settings needed for hosts and switches. 4. Use of IP-in-IP tunnels IP-in-IP tunnels are only required between neighboring GMPLS switches for OSPF adjacency. This is because the OSPF Hello messages are broadcast to all neighbors. GMPLS switches need to establish OSPF-TE adjacency to share data-plane topology, reachability and loading conditions. Given the separation of control-plane and data-plane interfaces in GMPLS networks, an IPin-IP tunnel mechanism is recommended by Sycamore to create OSPF-TE adjacencies between neighboring SN16000s (i.e., neighbors in the data-plane implies they are directly connected by data-plane links). When the OSPF-TE software module at a switch sends a Hello message to the broadcast address, all its adjacent 3 switches routers need to receive, process and respond to the message. Since the SN16000s are not connected directly on the control-plane, we need to configure an IP-in-IP tunnel between them. The tunnel is then automatically recognized as a direct link by the SN16000 software at each end. Using the network example in Fig. 11, we describe how IP-in-IP tunnels are created at the SN16000 switches. Following Sycamore s commands to set up a tunnel, we execute a step similar to the first step shown in Fig. 22 in Appendix 9.3 for setting up a tunnel at a Linux host. Sycamore recommends the use of the control port IP addresses instead of the Router ID/Switch IP for creating the tunnel. This step is executed at both SN16000s. Thus, for example to create an IPin-IP tunnel between the GMPLS switches at PoP1 and PoP2 in Fig. 11, we use the parameters and as the IP addresses in the tunnel configuration command. Following the description of how IP-in-IP tunnels are used in Appendix 9.3, these IP addresses will be in the outer datagram header and allow for packet forwarding through the public Internet. In Sycamore s implementation, all IP-in-IP tunnels have to be unnumbered, which means that no IP address is associated with the tunnel [17]. Therefore the SN16000 software does not have an explicit command to assign a local IP address for the IP tunnel as we demonstrated for the Linux host in line 3 of Fig. 22 in Appendix 9.3. Instead the SN16000 automatically assigns its Router 3. We use the term adjacent switches when there is at least one data-plane interface between two SN16000s. 17

18 ID/Switch IP as the local IP address of all its IP-in-IP tunnels. After a tunnel is configured, when the OSPF Hello message is sent with the broadcast destination IP address, the outer datagram header will carry the control port IP addresses of the two SN16000s as source and destination, but the inner datagram header will carry the local Router ID and the broadcast IP address as source and destination. At this point the two SN16000s determine the remote Router ID/Switch IP addresses of each other and the software automatically adds a routing table entry for these addresses to use the tunnel interface in their corresponding routing tables. Hence it routes all packets destined to the RouterID/SwitchIP address of its neighbor through the IP-in-IP tunnel interface. Both OSPF-TE and RSVP-TE messages exchanged between two switches will carry two IP headers when traversing the Internet, the inner one carrying the RouterID/Switch IP as source and destination IP addresses (except for the OSPF Hello message, whose inner header will carry the broadcast IP address as destination), and the outer one carrying the control port IP addresses of the switches as source and destination IP addresses. As in connectionless IP networks, we do not expect end hosts to run OSPF-TE in GMPLS networks. Therefore, no IP-in-IP tunnels are required between end hosts and switches for routing protocol messages. Furthermore, to transport RSVP messages, as long as the Switch IP is a static public IP address that is reachable through the Internet, no IP-in-IP tunnels are required to transport signaling messages between end hosts and switches (assuming also that end host primary NICs, i.e., control ports, are assigned static public IP addresses). 5. Securing the control plane Both RSVP-TE and OSPF-TE control-plane protocols are used to carry out critical functions. A malicious user could cause significant harm by sending RSVP-TE messages to tie up bandwidth. Similarly a malicious user could send OSPF-TE messages that create routing loops or other such problems. Our solution to securing these control-plane channels is to create IPsec tunnels [18]. Since RSVP-TE runs directly over IP, we cannot use Secure Shell (SSH) [19], Secure Sockets Layer (SSL) [20], or Transport Layer Security (TLS) [21]. Open-source IPsec software such as Linux Openswan [22], or Cisco/Juniper IPsec clients can be used at the end hosts. Openswan supports IPsec in both transport and tunnel modes. If the GMPLS switch software implements IPsec, then transport-mode IPsec tunnels (see Section 9.5) can be used between end hosts and switches and between adjacent switches. If the GMPLS switch 18

19 software does not support IPsec (as is the case with the SN16000) we need an external security device, such as the Juniper NS-5XT [23], through which tunnel-mode IPsec (see Section 9.5) is configured. We use the network example of Fig. 11 to illustrate how through the addition of these NS-5XT devices and Openswan at end hosts, we can create IPsec tunnels. Openswan interoperates with IPsec software in the Juniper NS-5XT. As shown in Fig. 13, the control ports of the three GMPLS switches are connected to the trusted (secured) interface on an NS-5XT device each. Internet Internet Openswan NS NS NS-5 Openswan End host 2 End host 1 control port: (routerid/switchip: ) PoP 3 control port: (routerid/switchip: ) control port: (routerid/switchip: ) PoP 1 Data-plane connections PoP 2 Control-plane connections VPN tunnels Fig. 13 Creating IPsec tunnels on the GMPLS control plane 5.1 IPsec tunnels between two GMPLS switches We establish IPsec tunnels between the NS-5XTs at the two switches. This means the NS-5XT will add an outer IP header and an Encapsulating Security Payload (ESP) or Authentication Header (AH) to IP datagrams received from its trusted interface (the link to the switch). See Section 9.5 for an example of how the NS-5XT can be configured for an IPsec tunnel. The Security Policy Database (SPD) will be programmed to allow only those packets whose destination and 19

20 source IP addresses match the control port addresses and Router ID/Switch IP of the two adjacent switches through this IPsec tunnel. The routing table on the NS-5XT will be configured correspondingly to route outgoing packets destined to the control port addresses and RouterID/SwitchIP of the remote switch on to the IPsec tunnel, and route incoming packets destined to the control port addresses and RouterID/SwitchIP of the local switch to its trusted interface. When an OSPF-TE or RSVP-TE message is sent on an IP-in-IP tunnel set up at two adjacent switches (as described in Section 4, the payload is already encapsulated in two IP headers. The routing table at the PoP1 switch in Fig. 13 will indicate that messages destined to should be sent on the IP-in-IP tunnel whose IP addresses are those of the control ports at the two switches (i.e., on PoP1 and on PoP2 in Fig. 13). When the NS-5XT receives these IP-in-IP datagrams, it determines which security association and routing entry, if any, the packet matches (by consulting the security policy database and the routing table), and then applies the security services indicated in the SA database for that association. For these IP packets (between the control ports of the two SN16000s, the tunnel SA characteristics are set to encrypt the packet, as described in Section 9.5). NS-5XT encrypts the whole IP packet and adds an outer IP header whose destination and source addresses are the NS-5XT untrusted (Internet) interfaces IP addresses. Thus, there are two tunnels in the CHEETAH control-plane network between any two SN16000s, an inner IP-to-IP tunnel described in Section 4, and an outer IPsec tunnel. 5.2 IPsec tunnels between Linux end hosts running Openswan and GMPLS switches Openswan can be configured to operate in tunnel mode or transport mode. Since the switch control port is only reachable through an IPsec tunnel (since the SN16000 software does not implement IPsec), we configure Openswan to operate in tunnel mode. When the RSVP-TE software running at the end host generates an IP datagram to the switch, the destination IP address will be the Switch IP and the source will be that of the primary NIC (control port) of the end host. The SPD corresponding to this association is configured in Openswan to encrypt packets and encapsulate them into an outer IP datagram whose header carries the IP address of the switch s NS-5XT Internet interface as the destination IP address and that of the host primary NIC 4 as the 4. Again, Openswan uses the primary NIC s IP address as its VPN gateway IP address. 20

21 source address. When this packet is received at the switch s NS-5XT, it will decrypt the packet, and send the unencrypted inner datagram to the switch. 5.3 Firewalls Given our proposed use of static public IP addresses for scalability reasons, it becomes important to configure firewall settings to prevent unwanted access to the control ports and Router ID/Switch IP of the switches. The Juniper NS-5XTs have firewall capability, which we use in the CHEETAH network. At the end host, we use the Linux iptables command to configure firewall settings. Section 9.2 shows how to use this command to set firewalls. 5.4 Security analysis With firewalls, unwanted access is strictly limited. With IPsec authentication, each switch authenticates the host from which it receives RSVP-TE messages and its neighboring switches from which it receives RSVP-TE and OSPF-TE. If an intruder is able to gain access to a CHEE- TAH end host, then this host can be made to issue RSVP-TE messages to tie up network bandwidth. We are looking into the use of RSVP Integrity object that would allow for user-level authentication. Finally, DoS attacks are possible to the switch control ports. The NS-5XT device provides some DoS protection but it is not extensive. 6. Current CHEETAH network address configuration Although we recommended the use of public static IP addresses for the control plane, in the CHEETAH experimental network we currently use private static IP address for control-plane interfaces of all the SN16000 switches and for some of the end hosts. This is because some of our enterprises and switch collocation providers, such as ORNL, provided us only one public IP address for all interfaces of the portion of the CHEETAH network within ORNL. We assigned this public IP address to the Internet interface of an NS-5XT and connected all the CHEETAH end hosts and SN16000 control ports behind the NS-5XT with private address assignments. The motivation was to minimize the number of addresses that needed firewall configuration. This is a temporary solution for use during the testing and prove-in period of the CHEETAH experiment. The CHEETAH network control-plane setup is shown in Fig. 14. The control ports of all the SN16000s and Router ID/Switch IP are private IP addresses. We use public IP addresses for most CHEETAH end hosts since public IP addresses were made available to us. The only 21

22 Internet Internet ORNL End host NS x/23 control port: (routerid/ switchip: ) ORNL PoP NS x/23 control port: (routerid/ switchip: ) NS x/23 control port: (routerid/ switchip: ) MCNC PoP Openswan MCNC End host Atlanta PoP Data-plane connections IPsec tunnels Control-plane connections IP-in-IP tunnels Fig. 14 CHEETAH control-plane network setup exception is in ORNL, where we assigned private IP addresses ( x/24) to the primary NICs of the CHEETAH end hosts. 6.1 Case 1: IPsec tunnels and VPN/IP-in-IP tunnels between two SN16000s One NS-5XT device is placed on the control port of each CHEETAH SN16000 switch. The Internet interface of each NS-5XT device is assigned a static public IP address, while the control ports on the switches are assigned private IP addresses as shown in Fig. 14. We establish IPsec/VPN tunnels between each pair of NS-5XTs to secure the control-plane traffic between switches (Section 5.1). Above these IPsec tunnels, IP-in-IP tunnels are set up at the two switches to create OSPF-TE adjacencies. As a result, there are two tunnels in CHEETAH controlplane setup between any two switches, the inner IP-in-IP tunnel and the outer IPsec/VPN tunnel. After an OSPF-TE adjacency is established between two neighboring switches, the OSPF-TE software on the local switch will count the inner IP-in-IP tunnel as a normal network interface and route all packets destined to its neighbor through the inner IP-in-IP tunnel interface. Signaling (OSPF-TE and RSVP-TE) messages exchanged between two switches will carry three IP headers 22

23 when traversing the Internet, the inner one carrying the RouterID/Switch IP as source and destination IP addresses, the middle one carrying the control port IP addresses of the switches as source and destination IP addresses, and the outer one carrying NS-5XT s public IP addresses as source and destination IP addresses. In supporting the IPsec tunnel, the NS-5XT intrinsically handles the VPN solution for handling static private IP addresses. 6.2 Case 2: IPsec tunnels between Linux end hosts with public IP addresses and SN16000s For the end hosts with public IP addresses, we choose Openswan to set up an IPsec tunnel to the NS-5XT device on the control port of each SN IPsec tunnels are established between the Openswan software and the NS-5XT at the switches. RSVP-TE messages sent by a switch to its end hosts will carry the end host public IP address in the destination address field and SwitchIP in the source address field. These datagrams will be encapsulated with an IPsec header carrying the end host public IP address in the destination address field and the NS-5XT s Internet interface address in the source address field. For reply messages from the switch to the end host, while the SwitchIP is a private IP address, it will match the SPD entry of Openswan and hence be sent on the IPsec tunnel, which means it will be encapsulated in an outer header carrying the NS-5XT s Internet interface address as the destination. The NS-5XT will send the inner datagram to its trusted interface to which the SN16000 control port is connected because its routing table will indicate that the SwitchIP address is reachable via that interface. The SN16000 programs its IP layer at the control port to accept datagrams destined to this address and hence the datagram will be accepted. It effectively does packet forwarding for datagrams destined to SwitchIP. Without this capability, an IP-in-IP tunnel would be required with the Switch IP as the address assigned to the tunnel with the control port serving as the physical interface. We do not need an explicit routing table entry for the public address of the end host since it can use the default row in the routing table which will show the NS-5XT as the default gateway. 6.3 Case 3: IPsec tunnels between an SN16000 and the end host with private IP address For the ORNL end hosts with private IP addresses, we use the NS-5XT to handle the VPN IPin-IP tunneling functionality needed to support the private addresses (Section 3.3) since it is already in place. The NS-5XT thus doubles in its role as VPN router for private address handling 23

24 and IPsec tunneling. Since both the end host and its SN16000 switch have NS-5XTs, the configuration is similar to the case between two SN16000s (Section 6.1). One difference though is that between two SN16000s since OSPF-TE is executed, automatic neighbor discovery process is used by the SN16000 to write a routing table entry indicating that the RouterID of its neighbor is reachable via the IP-in-IP tunnel. Since end hosts do not run OSPF-TE in CHEETAH, we rely on a static routing entry (default routing entry) in the routing table to send packets destined to the private address of the end host s primary NIC to the NS-5XT, beyond which the packet will be encapsulated in an outer datagram. For end hosts located in CHEETAH PoPs, we connect the primary NICs to the same NS-5XT as the SN16000 switches (for example, at ORNL in Fig. 14). In this special case, no IPsec tunnels are required between end hosts and the SN16000 since the NS-5XT routes signaling messages on its internal (secure) interfaces. 7. Data-plane addressing As described in Section 1, end hosts have secondary NICs that are connected into GMPLS switches to enable the creation of dedicated end-to-end circuits with frames carried over long-distance SDM, TDM or WDM circuits. The secondary NICs are the data-plane interfaces of the end hosts. The data-plane interfaces of GMPLS switches are the main interfaces on which user data flows. For the latter, we recommend the use of unnumbered interfaces [24], which means these interfaces are not assigned IP addresses; instead an interface identifier is assigned to each interface. As there is no obvious reason for needing to address these interfaces, we choose this option to reduce the total number of IP addresses needed in a GMPLS network. In this section, we therefore focus only on the end host data-plane interfaces and address the question of what type of IP addresses to assign these NICs. A second question deals with MAC addresses. The MAC address is required because once the end-to-end circuit is setup, frames must carry in the Destination MAC address field of the headers the MAC address of the far end host NIC as illustrated in Fig Option 1: a combination of using private IP addresses and ARP In this option, illustrated in Fig. 16, the secondary NICs are assigned private IP addresses. A single administrative entity, e.g., the GMPLS network administrator, assigns IP addresses to the second NICs of hosts that could physically be located in different campuses or enterprises. The 24

25 End host 1 End host 2 SONET/WDM network SONET SONET Fig. 15 End host 1 needs the MAC address of End host 2 s interface to send it frames on the dedicated end-to-end circuit starting point in this option is that a communication-initiating host knows that the correspondent host is on the same GMPLS network as itself and it knows the private IP address assigned to the secondary NIC of the correspondent host. The private addresses assigned to the secondary NICs of all hosts in the GMPLS network should be on the same subnet. The reason for choosing addresses from the same subnet is that once an end-to-end -SONET- or -WDM- circuit is established, it is essentially a direct link, i.e., a Layer-2 link. This would allow the end host to use its automatically created routing table entry for its own NIC s subnet (which indicates that all hosts on that subnet are reachable on the directly connected network) when an IP datagram is passed down with a destination IP address belonging to the same subnet as the NIC. This would cause an ARP query to be issued to find the MAC address of the distant NIC if there is no cached entry in the ARP table. The distant host would respond with its MAC address. There are two disadvantages to this option: (1) the use of private IP addresses makes the network unscalable. The disadvantages of using private IP addresses is described in Sections 3.3. For scalability reasons, there cannot be just one 25

26 The GMPLS network administrator End host 1 End host ARP ( ) ARP reply (MAC) SONET/ WDM network SONET SONET Fig. 16 Option 1: Private IP addresses and ARP GMPLS network administrator who manages the addresses of end hosts within different enterprises. Furthermore, needing these private IP addresses to be on same subnet, makes it even more unsuitable to building large-scale GMPLS networks. (2) the use of private IP addresses makes it harder for the secondary NICs to be used for Internet access when they are not connected on GMPLS circuits. Fig. 17 shows that if a second NIC is idle, for example the interface at end host 3, it should be able to access and be accessible from the Internet. If the GMPLS network administrator is assigning this address, it makes it hard to accommodate this private address behind a NAT router or in VPNs managed by other administrators, e.g., that of the enterprise where the end host is located. Assigning public IP addresses instead of private would allow for this usage of the second NIC when it is idle. (3) there are risks to using ARP, an address resolution mechanism designed for the local-area network, and extending it to wide-area usage. Executing ARP across the wide area causes two problems. First, it incurs a round-trip propagation delay, adding to the overhead of circuit setup delay. In CHEETAH, we strive to reduce call setup overhead [25] to enable the use of circuits for short-duration calls for increased resource sharing. Second, ARP, being a broadcast mechanism, 26

27 Internet End host 1 End host Switch GMPLS network Switch End host 3 Fig Inability to use the secondary NIC for normal Internet access when not engaged in a GMPLS circuit could lead to broadcast storms if there are switches on the end-to-end circuits. Ideally each of these switches should be programmed with port-mapped untagged or tagged VLANs, but in case of configuration errors, a broadcast storm could be triggered across the wide area. 7.2 Option 2: Use of static public IP addresses and DNS TXT resource record This option is one that considers scalability as an important requirement. Unlike in option 1, the starting point assumed here is that the communication-initiating host does not know whether the correspondent host is even on the same GMPLS network nor does it know its IP address. All it has is a domain name for the correspondent host. As described in Section 1, one of the advantages of the architectural assumption of using two NICs at hosts is that it allows for a gradual evolution of the GMPLS network, with end hosts being connected one at a time. In Section 7.2.1, we describe our procedure for how a communication-initiating host determines if the correspondent host is on the same GMPLS network as itself, and if so, how it obtains its secondary NIC s IP and MAC addresses. Our rationale is described in Section Procedure Data-plane interfaces of end hosts are assigned static public IP addresses because end hosts receive incoming calls to these interfaces. In other words, RSVP-TE signaling messages carry IP addresses of these secondary NICs in the destination and source address fields of corresponding message objects. As explained in Section 3.1, for scalability reasons, end hosts in different enter- 27

28 prises should be assigned addresses from each enterprise s address space allocation. Furthermore, if these addresses are dynamic, unless the DNS databases are dynamically updated, a calling end host cannot obtain the called end host s IP address. End hosts in a GMPLS network are comparable to servers on the Internet in that they can be called by other clients, and hence need static public IP addresses. After assigning static public IP addresses to the secondary NICs, these are stored in the local domain name server associated with a domain name. In addition, a TXT resource record is created for this domain name with a text string to indicate the name of the GMPLS network that this secondary NIC is connected to (e.g., GMPLS network: Cheetah ) and the MAC address of the interface. The TXT resource record was designed to return any data related to a domain name. The IP address will be returned on A-type DNS queries and verification of its GMPLS connectivity and its MAC address will be returned on TXT queries. Wide-area address resolutions mechanisms are scalable, such as the hierarchical tree based structure of the Domain Name System. DNS is most commonly used to resolve IP addresses. Extending it to resolve IP addresses to MAC addresses in addition to resolving domain names to IP addresses seems logical. Fig. 18 shows the new entries created for host2 in its local DNS server. When a communication-initiating host wants to start a session with a host whose domain name it has (e.g., the URL of a web page), it issues a DNS query with both A-type and TXT-type requests. The A-type query returns the IP address and the TXT record verifies if the correspondent host is on the same GMPLS network and provides the MAC address. If the domain name lookup is for a host that is not on the same GMPLS network, a blank TXT record is returned causing the communication-initiating host to simply use the Internet. If the host is on the same GMPLS network, the returned IP address is used in the RSVP-TE signaling message. The returned MAC address is used on the data-plane after circuit setup. frame headers use the returned MAC address in the destination MAC field. In order for this step to occur successfully, the IP routing table and the ARP table have to be written with new entries at both end hosts. The signaling engine at each end host adds a host-specific entry (i.e., 32- byte address) into the IP routing table with data indicating that this address is reachable on the directly connected secondary NIC (this routing entry will be selected as a result of the longest prefix match rather than any other entry such as the default gateway through the primary NIC). Fig. 18 shows the update in the IP routing table at end host 1 for the remote end host secondary NIC IP 28

29 address ( ) with a /32 subnet as reachable through its secondary NIC It shows that the default entry is to route packets through the primary NIC whose IP address is Next, the ARP table entry is written mapping the far-end NIC IP address to its MAC address. When the IP software selects this as the outgoing NIC for any datagrams destined to the secondary NIC at the far end of the GMPLS circuit, the lookup in the ARP table will yield the corresponding MAC address and allow the frames to be formatted correctly and sent.fig. 18 shows the ARP update corresponding to host 2 at host 1. Similar Higher-level DNS server host2 IN A host2 IN TXT GMPLS network: CHEETAH f4-cc-dd-6c Local DNS server DNS hierarchy Local DNS server End host 1 End host 1 s IP routing table Destination Interface default / End host Update GMPLS network SONET SONET End host 1 s ARP table f4-cc-dd-6c Update Fig. 18 Demonstration of how the local DNS database, IP and ARP tables are updated operations are executed at end host 2. These two operations, of writing an entry into the IP routing table and into the ARP table, are performed as the last step of connection setup initiated by the RSVP-TE signaling client at the end host. These steps are comparable to the switch fabric configuration executed by RSVP-TE 29

30 engines at switches. That step is essential to prepare the switch fabric to reprogram its crossconnections at every 125 microsecond interval to enable OC1 crossconnections (in a SONET switch). Similarly, setting up these two tables at the end host is required to enable user data to flow through at the end host from the application on to the circuit. In addition to the operation of adding a routing entry for the far end host of the GMPLS circuit, all other routing entries leading to the secondary NIC should be disabled if the NIC is to be used exclusively for the communication session across the GMPLS circuit. Typically, when a NIC is assigned an IP address with a subnet mask and default gateway, two routing entries are created: (1) to indicate other hosts on its subnet are reachable through the directly connected interface and (2) a default entry to indicate that all other subnets are reachable through the default gateway. Any routing entries directing traffic to the secondary NIC should be disabled for the duration of the circuit usage for exclusive use. When the circuit is released the newly added entries for the distant host in the IP routing table and ARP table are deleted, and all other routing entries for the secondary NIC are reenabled. in the absence of a circuit to a far-end secondary NIC, the only path to reach that NIC should be via the IP-routed Internet/Internet2 path Rationale: Internetworking revisited The end-to-end wide-area GMPLS circuit after it is set up can be viewed as a Layer-2 link. Using standard operating procedures in IP networks, the expectation then is that both ends of that link have to be assigned IP addresses from the same subnet. This contradicts our recommendation in which data-plane (secondary) NIC of a GMPLS end host is assigned a static public IP address from its enterprise IP address space. To clarify the above seeming contradiction, we make a case that the end-to-end GMPLS circuit is not a single link; instead it is no different than an internetwork path comparable to paths across the Internet. Consider first that the GMPLS switches run routing protocols such as OSPF- TE. These are protocols typically run at gateways [26], which interconnect networks, i.e., IP routers. The -SONET Sycamore switches shown in most of the figures in this document are effectively circuit-based gateways that interconnect networks comparable to IP routers, which are connectionless packet-based gateways to interconnect networks. On one side there are networks consisting of switches and host interfaces. On the other side is a SONET 30

31 network consisting of SONET switches. In other words, the -SONET- circuit is in fact a connection through an internetwork of networks. We call these circuit-based gateways because they perform their function of forwarding data from one network to another in a circuitswitched mode, i.e., it is preprogrammed with positional switching information such as map data arriving at a port to a set of time slots on a TDM shared link. This is in contrast to a packetbased gateway, which performs its function of forwarding data from one network to another in a packet-switched mode, i.e., by examining packet header information. OSPF-TE at a gateway is used to report address reachability of different addresses (typically in summarized form called subnet addresses) on each of its networks through OSPF LSAs to its adjacent gateways. This is precisely what happens in OSPF-TE exchanges between GMPLS switches. This step is required to then allow GMPLS switches to process RSVP-TE call setup (Path) messages are direct call setup along paths derived from the reachability information propagated through OSPF-TE. Thus, different networks on this GMPLS path need to be assigned different sets of summarized addresses (i.e., different subnet addresses). Therefore, end hosts in different enterprises will have different subnet addresses for their secondary NICs connected to the GMPLS network. This necessarily means that after setting up the circuit, the far-end of the circuit will be in a different subnet. About the procedure of issuing ARPs for hosts on the same subnet, this mechanism is an artifact of our choice of using the IP layer irrespective of whether the session is intra-network or inter-network. Reference [27] explains that this choice was made to simplify application-layer programming, so that application programmers do not have to choose whether or not to include the IP layer. In an intra-network session, the IP layer does not serve any purpose, since the packet will only be routed across switches and packet forwarding is based on the MAC address. Nevertheless the IP header is present due to ease of application programming considerations. Similarly, when an end-to-end dedicated circuit is established, we need neither IP headers nor MAC headers. Consider for example a telephone circuit that simply carries voice on the DS0 circuit established end-to-end. Nevertheless for the same ease-of-programming reason, we retain the IP layer and for reasons of ubiquity and popularity as a cheap NIC, we retain the header. Retaining the header has meant we need the MAC address of the remote end of the GMPLS circuit. The concern then is that if the two ends of the circuit have IP addresses in differ- 31

32 ent subnets, then an ARP request to directly resolve the remote end IP address will not be generated. Instead, a need to route packets through the default gateway (IP router) associated with the second NIC is assumed since the destination address is in a different subnet. This scenario should clearly be avoided once the circuit is setup. A decision on what IP address to resolve using ARP is made by an end host based on the routing table data. The default option is packets destined to a different network (subnet) should be routed through a default gateway (IP router). But there are no restrictions on adding host-specific routes. If a destination is reachable by a direct circuit, an entry can be made into the routing table indicating that the address is reachable directly via the secondary NIC. The presence of this entry will cause an ARP query with the remote end s IP address as the value being resolved if the ARP cache is empty. Again there are no restrictions against making additions to the ARP cache. Resolving a domain name into its IP address and MAC address using the scalable hierarchical DNS solution for address resolution is clearly a better answer for the wide-area than using a broadcast mechanism such as ARP that was designed only for local-area address resolution. At the called end, the routing table is updated by the RSVP-TE signaling client. The ARP table update occurs automatically upon reception of the first IP datagram on the circuit once it is established. In CHEETAH applications [12], we make sure that the communication-initiating end is the one that sends the first data packet on the circuit when it is established. 7.3 Option III: Use IPv6 addresses for the data-plane interfaces of end hosts As suggested in Section 3.4 for control-plane addresses, we recommend the use of IPv6 for data-plane addresses as well. Currently, the SN16000 switches do not support IPv6. But with increasing growth of IPv6, we think this is the best solution to create scalable GMPLS networks. With IPv6, there is a mechanism for the two end hosts to actively discover each other in the data plane after the GMPLS circuit is set up. They will actively do this to discover each other s MAC address instead of waiting for an application request to start the ARP process. However, if the circuits are set up just for duration of file transfers, as suggested in the CHEETAH project [11] for utilization reasons, then this neighbor discovery process will cost the additional round-trip propagation delay (as mentioned for ARP in Section 7.1) because the circuit will be used as soon as it is set up. We thus recommend the same DNS TXT based method for obtaining remote MAC addresses even when IPv6 addressing is used. 32

33 8. Conclusions 1. After considering the use of static and dynamic, public and private IP addresses for the control plane and data plane in a GMPLS network, we conclude that we need to assign static public IP addresses for interfaces in both planes. The only exception is the data-plane interfaces of GMPLS switches, which can be unnumbered, which means they do not need to explicit IP addresses. Given the shortage of IPv4 addresses, we recommend the use of IPv6. 2. The reason for requiring static public IP addresses for the data-plane interfaces of hosts is that these addresses are called, i.e., a host that wants a circuit to a remote host needs to fill the called host s IP address in the RSVP-TE message. This makes all end hosts in a GMPLS network comparable to servers, such as web servers and FTP servers, in the Internet. Given our scalability goals for creating a global-scale connection-oriented internetwork to complement the connectionless services of the Internet, assigning static private addresses to these interfaces with VPN routers performing IP-in-IP tunneling is unsuitable. 3. We described our solution for handling IP and MAC addressing on an end-to-end GMPLS circuit after it is established when the IP addresses assigned to the two ends are on different public IP address subnets. Out solution uses DNS TXT resource records and dynamic updates of IP routing tables and ARP tables at end hosts to properly configure end host IP and software to transport packets on end-to-end GMPLS circuits. We contend that these circuits are internetwork paths since they traverse different networks and SONET networks. 4. The reason for requiring static public IP addresses for the control-plane interfaces of hosts is that these addresses are maintained in TE-databases in switches and used in the RSVP_HOP parameter in Path messages sent from a switch to a called end host. If these were dynamic, the TE-databases would need to be automatically updated. If these addresses are private, scalability goals are thwarted. 5. We recommend the use of the Internet as the out-of-band control-plane network for GMPLS networks. The ubiquity and ease-of-use of the Internet makes it an attractive option. If the GMPLS switches are SONET switches, the other option is to use the inband DCC channels on SONET links. This option is not available if is used to connect hosts to switches, as in the CHEETAH network. Given the add-on nature of the CHEETAH solution, where con- 33

34 nectivity of end hosts to the Internet is preserved through their primary NICs, the Internet serves as an ideal medium to create control-plane channels between hosts and switches. While we could have used inband DCC channels between SONET switches, we used the Internet for this control-plane connectivity as well for the simple reason that we needed to capture controlplane messages for debugging and this was easier to do on the out-of-band control port rather than on inband DCC channels. 6. Each GMPLS switch needs at least two public IP addresses, one for an control port that is connected to the Internet and one for a loopback interface Router ID/Switch IP address. The SN16000 requires these two parameters, RouterID and SwitchIP, but allows for the same address to be used for both parameters. The RouterID is used in OSPF-TE messages and the SwitchIP is used in RSVP-TE messages. Furthermore, some GMPLS switches, such as the SN16000, may have two control ports, in which case, a third address would be needed if this port is also used for redundancy reasons. Assign the two (or three) addresses on different subnets for reliability reasons. 7. In a production network, we recommend enabling OSPF on the control ports and let control-plane addresses (the Router ID/Switch IP loopback address) be propagated to IP routers on the Internet. Since Switch IP is used as the Destination IP address in RSVP-TE messages, these messages can be routed to the destination switch via the DCC channels from a neighboring switch if the control ports of the destination switch are unreachable. For addresses to be routable all three addresses (for the two control ports and the Router ID/ Switch IP) should be public addresses, and for reliability reasons, they should be on different subnets. This OSPF exchange should not be confused with the OSPF-TE exchange between SN16000s for the data-plane addresses of end hosts (secondary NICs). 8. IP-in-IP tunnels are required between the control ports of adjacent GMPLS switches for OSPF-TE adjacencies. With the SN16000s, the remote address assigned to the tunnel is determined from the OSPF-TE exchange and used to update the routing tables setting reachability to be via the tunnel. This makes all signaling and routing messages between two switches travel on the tunnel. 9. We recommend the use of IPsec tunnels between end hosts and SN16000 control ports and between adjacent SN16000s. Since RSVP-TE and OSPF-TE ride directly on IP, SSL cannot be used. In the CHEETAH network, we use Openswan on Linux end hosts and place an IPsec 34

35 router, specifically the Juniper NS-5XT, to create these IPsec tunnels. Protection is at the host level since authentication is IP address based. 10. Given our recommendation for the use of static public IP addresses, firewalls are very important to prevent unwanted access. 9. Appendix 9.1 Background on private vs. public and static vs. dynamic addresses Consider the four options shown in Table 2. Static public IP addresses are required for servers, Table 2: Types of IP address Public Private Static Required for servers VPN using IP-in-IP tunnels, and NAT (ISP migration) Dynamic DHCP NAT/NAPT (pooled addresses) i.e., nodes running the server end of an application, since these servers will need to be accessed by clients. Domain Name Service (DNS) lookups are used by clients to determine the static IP addresses of these servers. Dynamic public IP addresses are handled with Dynamic Host Configuration Protocol (DHCP). DHCP is used to dynamically obtain IP addresses to hosts that only run client ends of applications. As a host powers up, it issues DHCP requests for an IP address and uses it for the duration of its connectivity to the Internet. The same IP address can be allocated to some other host when the first one is done. Although some enterprises use dynamically allocated addresses for administrative ease rather than for address sharing, which means the same IP address will be returned for a host every time its DHCP client requests an address, dynamic address allocation can be used to make up for address shortages. Network Address Translation (NAT) and Network Address Port Translation (NAPT) as used to handle private dynamic addresses for uses such as sharing of pooled IP addresses and having flexibility to change ISPs. A NAT/NAPT router is assigned global public IP addresses for its interfaces to the Internet. End host interfaces connected to the NAT/NAPT router are assigned private IP addresses (local). A NAT router dynamically allocates a public address to a private IP address when it receives a datagram from one of its end hosts that needs to be forwarded to one of its public interfaces. It changes the private source address in the datagram to its own public interface 35

36 address as illustrated in Fig. 19. The datagram sent by the host should carry a public destination IP address. This mechanism is used to share a few public IP addresses amongst a large number of hosts assigned private addresses under the assumption that not all of those hosts are simultaneously engaged in communication sessions with hosts on the Internet. In case of public IP address shortage, NAPT can be used to associate multiple private IP addresses with a single public address with different port assignments. In this case, not only is the source address of an outgoing datagram modified from private to public, but also the TCP or UDP port numbers. For applications that ride directly on IP, such as RSVP-TE, the use of NAPT is thus not possible. The address table, mapping private to public addresses in a NAT router can be static if the purpose of using private addresses is to allow for easy migration of ISPs (with CIDR, address blocks are obtained by an enterprise from its ISP). In this case, the enterprise has a sufficient number of public IP addresses and hence can statically assign private IP addresses to public addresses. If the ISP is changed, only the NAT address translation table will need an update rather than requiring a reconfiguration of each end host. Therefore in Table 2, we list NAT/NAPT in both the static and dynamic rows. Note that there is no IP-in-IP tunneling in NAT. Instead the source (and destination) IP addresses in outgoing (and incoming) datagrams and other header fields (such as checksum) are rewritten payload payload payload End host 1 NAT router A NAT router B End host 2 Fig. 19 Illustration of how NAT routers are used to support private IP addresses 36

37 The last mechanism shown in Table 2 is VPN using IP-in-IP tunnels for the handling of static private addresses. Fig. 20 shows how hosts located in different enterprise campuses are configured to belong to a Virtual Private Network (VPN) created across the public Internet payload payload payload End host 1 VPN router A tunnel1 VPN router B End host 2 End host 1 s IP routing table default router A s IP routing table tunnel1.2.0/24 router B s IP routing table tunnel1 1.0/24 End host 2 s IP routing table default Fig. 20 Illustration of the use of IP-in-IP tunnels to create a VPN and handle static private IP addresses RFC 1631 [28] calls this a backbone-partitioned stub. Private IP addresses allocated to hosts in the different enterprises within one VPN need to be unique. This is because the VPN router needs routing table entries indicating the interfaces via which different private IP addresses can be reached. For example, if the address is assigned to PCs behind both VPN routers in Fig. 20, when end host sends an IP datagram destined to , VPN router A will not know whether to the route the packet out on to the tunnel or to an internal interface. Hence private IP addresses assigned across a VPN need to be unique. We refer to the routers serving in this role as VPN routers rather than NAT boxes as in [28] to draw the distinction between NAT routers, where there is no tunneling and private IP addresses need to be unique only across hosts within an enterprise campus behind any single NAT router, in contrast to these VPN routers, 37

38 where there is tunneling (IP datagrams are embedded in outer IP datagrams) and private IP addresses have to be unique across the whole VPN. Table 3: Contrasting the use of NAT and VPNs Uniqueness of private IP addresses Static vs. dynamic addresses NAT Per campus; across local interfaces connected to the NAT router Dynamic; the purpose of NAT is to reuse a few public addresses across many end hosts VPN using IP-in-IP tunnels Across all end hosts connected to any VPN server on the same VPN; requires VPN administrator to manage IP address allocation within different campuses and across different enterprises if they are within the same VPN Static; since the VPN servers create IP-in-IP tunnels and use their public interface addresses in the outer datagram, the inner datagram carrying private IP addresses is not changed. IP-in-IP tunneling Not used Used; IP datagrams are encapsulated in outer datagrams IP address modification in header Done at NAT routers from public to private and vice versa Not required Table 3 summarizes the differences between NAT and VPN (with IP-in-IP tunnels) to handle private IP addresses. If an enterprise has many geographically distributed campuses, then the use of VPNs is appropriate. On the other hand, if a single geographically collocated campus connected to the Internet as a stub network wants to use private IP addresses, NAT/NAPT is the better choice. 9.2 Setting up firewalls When using static public control-plane IP addresses for the end hosts, the firewall software iptables is used to filter out unwanted traffic. Fig. 21 shows an iptables configuration example on Linux hosts. In this example, only SSH (22), HTTP (80), FTP (21), IKE (500) and ISKAMP packets are allowed. Ports 5900, 5901 are opened for the remote desktop application, VNC. ACCEPT tcp / /0 state NEW tcp dpt:22 ACCEPT tcp / /0 state NEW tcp dpt:80 ACCEPT tcp / /0 state NEW tcp dpt:21 ACCEPT udp / /0 state NEW udp dpt:500 ACCEPT tcp / /0 state NEW tcp dpt:5900 ACCEPT tcp / /0 state NEW tcp dpt:5901 Fig. 21 The example of iptables configuration on Linux hosts 38

39 For public data-plane IP address at end hosts, the same firewall protection should be applied as for the control plane interface before the GMPLS circuit is established. During the period that the circuit is up, all routing entries for the data-plane interface (the secondary NIC) are disabled except the one for the remote communicating host. This is equivalent to a firewall policy in which the only traffic allowed is to/from the remote communicating host. Therefore, we can assume the data plane interface is secure as long as the remote communicating host is trustworthy. For static private control-plane IP addresses of the end hosts/sn16000s behind NS-5XTs, NS- 5XTs are configured to route all traffic only through IPsec/VPN tunnels. A firewall policy can be defined for IPsec tunnels to further enhance the security of IPsec/VPN tunnels. No direct incoming/outgoing Internet traffic to/from the ports of end hosts/sn16000s is allowed by default in such CHEETAH hosts/switches. However, Internet access is possible by configuring the firewall policy if communicating is initiated by end hosts/sn16000s behind the NS-5XT (which serves both as a firewall and a NAT router). Note that firewall policies for the IPsec tunnels and the ports of end hosts/sn16000s can be distinct. 9.3 An example of IP-in-IP tunnel configuration at a Linux host In this section, we illustrate how to set an IP-in-IP tunnel at a Linux host. In Fig. 22, a tunnel named ipip1 is created by specifying the IP addresses of the two ends ( and in the example of Fig. 22). The tunnel is regarded as a link at each end. Therefore it needs to be assigned an IP address, which could be a private IP or public IP address. In Fig. 22, we assign it a private IP address, A route add could be executed for the address assigned to the remote end of the tunnel, as shown in step 3 in the example of Fig. 22. The IP address assigned to the remote end of the tunnel is When the user of the IP layer passes it a datagram payload to be sent to remote address of the tunnel ( in the example of Fig. 22), it will place this remote address in the inner IP datagram header s destination IP address and the local address of the tunnel in the source address field ( in the example of Fig. 22), but because this is a tunnel, it will add an outer dataip tunnel add ipip1 mode ipip remote local ttl 255 ip link set ipip1 up ip addr add dev ipip1 ip route add /32 dev ipip1 Fig. 22 Commands for setup a IP-in-IP tunnel on a Linux host 39

40 gram header whose destination and source addresses are the IP addresses of the two interfaces on which of the tunnel was created, and , respectively, in the example of Fig. 22. This allows the IP datagram to be carried to the far-end of the tunnel. 9.4 Spreading of reachability of private IP addresses in OSPF Does the OSPF-TE implementation in the SN16000 or in Cisco routers propagate reachability data for private IP addresses? The answer is yes, both Cisco routers and Sycamore switches propagate this data. In our experiments described in Section 2.2, all interfaces on the Cisco GSR routers except the control ports are assigned private IP addresses as shown in Fig. 5. The GSR routers do propagate LSAs with private IP addresses and maintain correct OSPF databases. The GSR routers do not block LSAs with private IP addresses. The Cisco IOS software allows the administrator to choose whether or not to enable OSPF on individual interfaces. For example, to enable OSPF on the GbE interface of GSR A in Fig. 5, we execute the commands shown in Fig. 23. IETF RFC 1918 states that Routers in networks not using private address space, especially those of Internet service providers, are expected to be configured to reject (filter out) routing information about private networks. This mean explicit filtering commands need to be set to prevent the handling of IP packets with private addresses while the default option is to handle them a(config)# router ospf a(config-router)# network area a(config-router)# end Fig. 23 Cisco GSR IOS commands to enable OSPF on an interface The same behavior was seen in the OSPF interpretability testing between SN16000s. We saw the information about a TE-link with the private IP address being propagated correctly by the OSPF-TE implementation at the SN Fig. 24 shows an OSPF LS Update packet sent by one 40

41 of SN16000s in CHEETAH network. The highlighted line shows a stub network with a private IP address. Fig. 24 OSPF LS Update packet sent by MCNC SN Commands to configure IPsec tunnels Security associations in IPsec can be established in one of two modes: transport mode or tunnel mode. In transport mode, an ESP or AH header is appended to the IP datagram payload before it is encapsulated in an IP datagram for transmission. The destination and source IP addresses carried in the IP header are those of the interfaces at the ends of the security association. In tunnel mode, the ESP or AH header is added by an external device. Thus the original IP datagram issued by an end host to a destination host is encapsulated in another IP datagram. The destination and source IP addresses of the outer header correspond to those of the external device interfaces at the two ends On Openswan-equipped Linux hosts In this section, we describe how IPsec tunnels can be configured using Openswan software in a Linux host. We use the MCNC end host in Fig. 14 as an example. As shown in Fig. 25, after installing Openswan, two files, /etc/ipsec.secrets and /etc/ipsec.conf, are configured to specify the authentication/encryption methods for the IPsec tunnel. In this example, the preshared key (PSK) 41