Internet2 Network Service Descriptions DRAFT December 4, 2006

Transcription

1 DRAFT December 4, 2006 Table of Contents 1. Network Infrastructure and Services Introduction Architectural Overview Circuit Service Descriptions IP Services Details of Multiservice Switching Infrastructure and Dynamic Circuit Services Multiservice Switching Infrastructure Architecture Dynamic Circuit Service Connecting Internet2 Dynamic Circuit Services to Users Use Cases Appendix A. Dynamic Circuit Service Detailed Description A.1 Connecting to the Dynamic Circuit Service A.2 Transport Service Definition A.3 Encapsulating Ethernet in SONET A.4 Control Plane Definition Appendix B. Bandwidth Used by Physical Interface Appendix C. Phase 1 Control Plane Rollout Appendix D. One Example of How to Connect a Campus to the Dynamic Services Network... 29

2 1. Network Infrastructure and Services 1.1 Introduction This document describes the infrastructure and services of the Internet2 hybrid network. In order to do so, it first gives an overview of the Internet2 network architecture showing how the different services and infrastructures fit into the total picture. One important goal in showing the architecture is to show the distinction between the infrastructure being rolled out, and the services offered on the infrastructure. Internet2 has standardized the new network connection model around a 2 x 10Gbps connection. Slower connections from 1Gbps will be supported during a migration period. (More information is available on the Fee Structure page at One of the connections is used for the IP services; the other is available for dynamic circuit services. The speed of the physical connection and the speed of the network connection do not need to match. Most connectors will connect at 10Gbps, with their traffic rate-limited, if necessary, based on the terms of their connection agreement Part 1 of this document presents an overview of the network architecture and services. Part 2 describes in detail the new circuit services and infrastructures that are offered. The circuit service description includes details of physical connections to the network, the bandwidth of the circuits across the network, and other details of circuit connections. The Internet2 physical implementation is made up of several related infrastructures. Each of these infrastructures has its own architecture. The infrastructures are: 1) the core infrastructure 2) the IP infrastructure 3) the multiservice switching infrastructure 4) the HOPI testbed infrastructure In the initial rollout, these infrastructures will be kept as independent as possible, both to limit the amount of interaction between technologies while we gain experience with new technology, and to make transition from the previous network infrastructure as straightforward as possible. The mapping of circuit services to infrastructure for the first rollout is: 1) static circuits provided on the core infrastructure or on the multiservice switching infrastructure 2) dynamic circuits provided on the multiservice switching infrastructure 3) HOPI test service provided on the HOPI testbed infrastructure Note that infrastructures can and do use services provided by other infrastructures. For example, the IP network, the multiservice switching infrastructure and the HOPI testbed use static circuits provided by the core infrastructure to interconnect their boxes. In the future, the IP network and the HOPI testbed may use static circuits from the multiservice switching infrastructure as well as from the core DRAFT December 4,

3 infrastructure. The HOPI testbed might use MPLS paths over the IP network. In addition, services provided may use multiple infrastructures in interesting combinations. For example, the dynamic circuit service initially is provided over the multiservice switching infrastructure, but in the future the service may be enhanced so that the core and multiservice switching infrastructures are used in an integrated way, such that when the multiservice switching infrastructure does not have enough bandwidth to satisfy a request, it will attempt to get more bandwidth from the core infrastructure. Other combinations are possible as we gain experience with the network and the ways people want to use it. The ability to combine infrastructures and services to create new capabilities and services is what makes the hybrid network unique and powerful. The form of this future integration in services and infrastructure is a subject for a later document. This document focuses on describing the initial basic infrastructure and services. In Part 1 of this document all infrastructures and services are described at a functional level. In addition to the overall architecture, Part 1 includes a section that describes the IP network and the IP services provided on it, and a section that describes circuit infrastructures and circuit services, including the dynamic circuit service and multiservice switching infrastructure. In Part 2, the dynamic circuit services and the multiservice switching infrastructure are described in more detail. This is because the dynamic circuit service is new and complex. The IP service, while complex, is an enhancement of the Abilene network, with some additional services added. The static circuit service is simply providing a point to point circuit across the network. The HOPI test service is a continuation of the earlier HOPI network, using some new infrastructure. Part 2 has four sections: 2.1) Multiservice Switching Infrastructure This section includes an architectural diagram of the multiservice switch, and a description of how multiservice switches physically interconnect with each other. The multiservice switches are controlled by independent control plane software, and a brief overview is given, showing the interaction between the data plane that creates the actual circuits, and the control plane that sets them up and tears them down. 2.2) Dynamic Circuit Service The dynamic circuit service section describes how dynamic circuits are set up between users and Internet2. Two subsections describe the connectivity at the circuit level and at the control level. 2.3) Connecting Internet2 Dynamic Circuit Services to Users To show how dynamic circuits may connect to users and other networks, a section showing possible connection scenarios is included. Users of dynamic circuit services include regional networks, end users, and other provider networks. By connecting to Internet2 dynamic services, users are able to connect to others who are also connected to Internet2 dynamic services. If Internet2 is connected to other provider networks, then a user on Internet2 may create circuits to a user on a different provider network. Several connection scenarios show how Internet2 may be used in providing point to point circuits across several domains. This also shows the coordination of control planes between domains that is needed to create such circuits. DRAFT December 4,

4 2.4) Use Cases This section presents a number of use cases showing applications where dynamic circuits can be used. This section has been written before most of the applications have been deployed. Therefore, the use cases include some that are in a planning stage as well as some that are presented as possible cases for consideration. This section includes models of how network resource allocation may be managed and the contrasting benefits of different models for different applications. The model part of the section is to provide ideas about what might be done, and to help shape thinking on how resource sharing in the dynamic circuit environment can be most effective. The expectation is that this will be a living document that will be modified over time as the network, and our understanding of how to best describe it, evolve. For example, the document could describe the Infinera, Ciena and HOPI hardware in more detail. It could also add the need for testing and debugging circuit services, the need to support performance testing between segments of circuits, the need to support perfsonar on cross-domain circuits, and other debugging and setup issues. 1.2 Architectural Overview This section describes the architecture of the Internet2 network. The network consists of multiple interconnected infrastructures, each with independent architectures. The infrastructures are used by Internet2 to provide network services to users. The Internet2 network provides the following broad classes of services: 1) IP service 2) circuit service The IP service is an IP network which replaces and enhances the Abilene network. The details of the services provided by the IP network are described in Section 1.4. The circuit service described in Section 1.3 provides point to point circuits between ports on the Internet2 network. These circuits provide guaranteed bandwidth and deterministic behavior in terms of bounds on jitter, latency and packet loss. Three different circuit services are provided: 1) static circuit service 2) dynamic circuit service 3) HOPI test service There are four architecturally distinct infrastructures in the Internet2 network: 1) core infrastructure, consisting of fiber segments interconnected with wave equipment 2) IP network 3) multiservice switching infrastructure 4) HOPI testbed infrastructure DRAFT December 4,

5 Figure 1 is a diagram of the overall architecture of the Internet2 network. In the initial deployment, each service will be supplied by a particular infrastructure. Note that this diagram does not show all the nodes and connectors, but shows how the different services interact architecturally. It also shows some of the similarities and differences between the services. Figure 1 DRAFT December 4,

6 Architecturally, the base of the Internet2 network is a core infrastructure, which consists of fiber segments carrying waves that are connected with wave equipment. Each of the fibers carries a number of waves; the Infinera wave equipment connects a wave on one fiber to a wave on another fiber. By connecting waves at sequential wave equipment nodes, a circuit from a port on local wave equipment to a port on remote wave equipment may be instantiated. The other Internet2 infrastructures use static circuits created on the core infrastructure to interconnect their devices. The IP network uses the core infrastructure to create links between routers. The routers form an IP network functionally identical to the previous Internet2 IP network (Abilene). The multiservice switching infrastructure and HOPI testbed also use static circuits created on the core infrastructure to interconnect their nodes. As shown in Figure 1, the IP, dynamic circuit and HOPI infrastructures use the core infrastructure in the same way. All infrastructures are architecturally independent of each other, although they may use services from one another to implement their architectures. Future documents will describe how the infrastructures and services may integrate more extensively in the future. 1.3 Circuit Service Descriptions For the initial deployment of Internet2, each circuit service will be deployed in a way that limits interaction between infrastructure elements providing different services. As the network matures, the level of integration of both the services and the infrastructure will increase. This integration is a subject for a future document. However, it should be noted that separation of the architecture of the services is likely to remain even as the services begin to interoperate. This architectural independence is valuable for the maintenance of the integrity of each service, independently of the others. Two major aspects of the circuit service are: 1) the physical connection between the Internet2 device and the user device 2) the bandwidth of the circuit across Internet2 The physical connection is defined by the type and speed of the device connecting to the service, while the bandwidth of the circuit across Internet2 may be any multiple of STS-1 (approximately 50Mbps) from STS-1 to STS-192. DRAFT December 4,

7 Figure 2 Figure 2 is an illustration of a user A connecting to the circuit service with a 10 Gigabit Ethernet interface, user B connecting with a 1Gigabit Ethernet interface, and a circuit between users A and B of 100Mbps. This shows that the speed of the interfaces and the bandwidth of the circuit do not have to match The Static Circuit Service The static circuit service allows users to create point to point circuits across the Internet2 network. Users connect at an Internet2 Point of Presence (PoP) to either the Infinera core infrastructure wave equipment or the Ciena multiservice switching infrastructure. The capabilities and requirements of each differ slightly. For static circuits provided by the Infinera infrastructure, the physical interface to the static circuit service is at either 10GE or OC192 SONET. Connection across Internet2 is Ethernet circuit or SONET circuit and matches the type of the physical interface. Bandwidth of static circuits provided by Infinera is 10Gbps. If the connection to the static service is provided by the Ciena multiservice switching infrastructure, the physical connection may be 1GE, 10GE, or SONET at OC-48 or OC-192. In the Ciena infrastructure, connection across Internet2 is always SONET. If the physical device connecting to the multiservice switch is Ethernet, the Ethernet frames are encapsulated in SONET using GFP. When using the Ciena multiservice switch, the bandwidth across the network can be any multiple of OC-1. Note that the bandwidth of the physical interface and the bandwidth of dynamic circuits carried over these interfaces do not have to be the same. For example a user could connect using a 1GE interface and request a 100 Mbps (2 x OC-1) circuit across Internet2. In all cases of users connecting to the static circuit service using SONET, the incoming SONET may be channelized or not; for example, the connection may be OC-48 or OC-48c. DRAFT December 4,

8 Static circuits are provisioned by the Internet2 NOC. Connections between Internet2 and a user are coordinated with the user's NOC. Static circuits are long-lasting connections (e.g., a year) that are set up between two specific users that bring a connection specific to that circuit to an Internet2 PoP The Dynamic Circuit Service The dynamic circuit service allows users to create point to point circuits across Internet2 for shorter periods of time than with static circuits. The dynamic circuit service uses control plane software to set up the circuits. A user connects physically to the dynamic infrastructure either as a single circuit to be switched, or as a point to point circuit that multiplexes multiple circuits over the physical connection. The current expectation is that most users will connect in multiplexing mode, but either mode is possible. On request, a dynamic circuit is created across Internet2 from one user to the other. The Internet2 circuit connects at either end to a user circuit. The user circuit is, as described above, either a single circuit or a multiplexed circuit. Dynamic circuits are expected to be set up for periods of time ranging from minutes to months. The dynamic circuit service is essentially a switching service that creates circuits between users. As such, its value depends significantly on having a set of users physically connected to it that want to make connections between each other periodically. Topologically, it is similar to a telephone exchange. The dynamic circuit service is provided by the Ciena multiservice switching infrastructure. Physical connection to the multiservice switch is provided at an Internet2 PoP. The physical interface to the multiservice switch may be Ethernet with or without VLANs or SONET. The dynamic service uses control plane software to create and tear down circuits. The ability to set up circuits is permitted to a limited set of people. Initially, permissions will be granted only to certain Internet2 staff, but in the longer term, others will be given permissions to make at least some connections. In all cases the circuits will be made using control plane software. Most circuits that cross Internet2 are expected to be a segments of longer point to point circuits, including segments in regional networks and perhaps in other provider networks as well as in Internet2. When setting up such cross-domain circuits, it is necessary to coordinate the setup of all the segments with each of the organizations. Coordination may be done in two ways: 1) staff at different organizations using , phone or other manual methods, or 2) using software at each participating organization to set up cross-domain connections automatically. To make cross-domain connections automatically, users will need appropriate control plane software. Internet2 will provide experimental software for regional networks that will allow such interconnection. Support is also planned for circuits that connect with other networks automatically, as well as connecting with other networks using staff to staff communications. DRAFT December 4,

9 1.3.3 The HOPI Test Service The HOPI test service is a set of equipment that allows experimentation in implementing and using dynamic point to point circuits. The test service is carried over from the original experimental HOPI network that ran on a different set of waves. At the start, the new HOPI service is expected to be functionally identical with the previous service. It is expected that the HOPI test service will expand, though perhaps in unexpected ways. HOPI is experimental in a number of ways: 1) Its footprint is limited relative to the total Internet2 footprint. 2) It uses experimental control plane software to create circuits. 3) It uses its control software to interconnect with other domains using their control software. 4) It allows other networks to interconnect with it to test their interdomain circuit capabilities. As with the dynamic circuit service, users connect physically to an Internet2 PoP which contains a HOPI access point. As with the dynamic circuit service, a physical connection carries one or more subcircuits. Each sub-circuit may be connected over a circuit across the HOPI infrastructure to another circuit connected to a HOPI access point. The kinds of physical connections available on this service will change over time as devices are added. For the initial service, the connections will consist of the existing HOPI Force10 Ethernet switches and DRAGON software. The physical interfaces are 1GE or 10GE. Connections across HOPI are point to point Ethernet VLAN based circuits. Circuit bandwidth is in increments of 100Mbps. Requests for guaranteed bandwidth circuits may be made by using GMPLS-style Peer Mode, GMPLS-style UNI Mode, Web Service API, or /phone to the NOC. User input devices must support standard Ethernet 802.1q VLAN capabilities. The control plane software used in HOPI is developed by a set of collaborators including Internet2, DRAGON, University of Southern California/Information Sciences Institute East (USC/ISI-East), Mid-Atlantic Crossroads (MAX), and others. In addition to using the waves of the core infrastructure, HOPI will connect to the dynamic services of Internet2 as a separate domain. The network control software on HOPI will interoperate with dynamic services as a separate domain, setting up circuits automatically that use segments in both HOPI and the dynamic circuit domain, as well as perhaps other domains. In this way HOPI provides a way to demonstrate control-plane interoperability. HOPI may also be used to exercise and test new control plane software being developed by Internet2 and others. It may also be used for testing on dynamic services being developed by other domains. This will likely include HOPI interconnects to test labs and with other organizations, such as regional networks, that are planning to deploy or develop dynamic circuit provisioning control plane software and infrastructure. DRAFT December 4,

10 Finally, HOPI may be used to allow testing of applications prior to using them on the dynamic or static circuit services. For example, performance measuring software for dynamic services networks may be tested on HOPI prior to rolling it out on the dynamic circuit infrastructure. 1.4 IP Services Internet2 provides IP services that are a continuation and enhancement of services provided by the Abilene Network. IP services are provided by the IP infrastructure which uses circuit service from the previously described core infrastructure to form a new Internet2 IP backbone network. The IP services provided consist of a standard IP backbone network service that can be viewed as a replacement for Internet2 s existing Abilene backbone network, and new services that are being tested and are available upon request on the IP backbone network. These services, described below, are the commercial peering service and the commodity transit service. Lastly, a brief description of the use of MPLS tunnels on the Internet2 IP network backbone is included IP Backbone Network The new Internet2 IP backbone network consists of routers at nine locations, connected with multiple physically diverse 10Gbps backbone links. Each connection is provided on a dedicated 10Gbps wavelength between the two interconnecting locations. As traffic needs increase beyond 10Gbps, Internet2 will allocate additional waves on the core backbone to augment the capacity for the IP backbone network. The routers are located at: New York Washington Chicago Atlanta Houston Kansas City Los Angeles Salt Lake City Seattle All of the services provided by Abilene will be provided on the new Internet2 network. Figure 3 shows the Internet2 IP network overlaid on the core infrastructure. DRAFT December 4,

11 Figure 3 Connecting to the Internet2 IP Network The connector must build out the capabilities to reach the Internet2 suite in one of the Level(3) PoPs where Internet2 has equipment. Internet2 will carry the connection to one of the IP network backbone routers for IP connectivity. Generally, 10Gbps connections will be carried over the core infrastructure. Connections of less than 10Gbps may be carried over the multiservice switching infrastructure. In anticipation of the potential addition of the other services mentioned below, which will require separate BGP peerings, Internet2 is encouraging connectors to connect initially with VLANs enabled (in the case of Ethernet connections) or a frame relay Data Link Connection Identifier (DLCI) set up (in the case of SONET connections). However, if research IP access is the only service desired, these need not be enabled. A dedicated backup circuit to another router may be purchased, so long as the aggregate usage across the two links remains below the connection fee allowance. The redundant network costs shown in the price schedule include the cost of the router interface and transport to the alternate node. Other redundancy options will be considered upon request. (More information is available on the Fee Structure page at DRAFT December 4,

12 1.4.2 Commercial Peering Service Internet2 is creating a content peering exchange fabric as a benefit to its members. This service will be available to connectors that choose to participate. A beta trial is currently underway with several Internet2 connectors to help understand the implications before deploying this across a national infrastructure. Early results with a few peers show a 20-30% traffic offload to settlement-free peering (no additional charge beyond Internet2 connector fees). As configured for the trial, the service is a separate peering between Internet2 and the participating connectors, allowing the connectors to control how the peering service is distributed to their members. This service is made available over the connector s access to the IP backbone network. However, to allow connectors to better differentiate this service, the connection will multiplex via the use of VLANs (in the case of Ethernet) or Frame Relay DLCI (in the case of Packet Over SONET). The Network Technical Advisory Committee (NTAC) and Internet2 staff will be seeking community input on how this service should be expanded to additional connectors and rolled out to the entire Internet2 community. (The NTAC is an advisory body similar to the Abilene TAC. The current NTAC chair is Paul Schopis of OARnet.) The Internet2 IP backbone network will connect at 10Gbps to commercial colocation facilities in Chicago, IL and Ashburn, VA, with additional locations contemplated in the near future Commodity Transit Service Internet2 will offer commodity transit service to connectors through their IP connection. Internet2 has negotiated with Level(3) the ability to provide commodity Internet services to its connectors and members at a reduced rate. To ensure that this agreement does not undermine the significant progress the Quilt has made in this area, Level(3) has agreed that purchases by Internet2 for its members will count towards the Quilt s aggregate purchase levels. Level(3) will also allow some existing 1GE ports that do not connect to the Abilene backbone, but provide commodity transit directly to Internet2 connectors or members, to be transitioned to the new rate as direct connections from the Level(3) backbone to the members or connectors. So as not to be dependent on only one supplier, Internet2 will engage other commodity transit providers in the near future, and add their services to the overall service offering. (For details on current commodity transit pricing, see the Fee Structure page at Commodity transit is an optional service to Internet2 connectors, respecting the preference that regional connectors are the primary access points to the network for community participants. Internet2 will consider providing commodity transit directly to a member institution whose connector declines the service, or where there otherwise is agreement between a member and connector. Internet2 will continue to be sensitive to the wide variety of business models among the RONs, and will support the option to use this direct connection service in coordination with and in support of the connectors and regional network operators desire to serve their members. DRAFT December 4,

13 Internet2 provides the optional commodity transit service as a separate BGP peering, allowing the connectors to control distribution to their members. As with the commercial peering service, we anticipate that in general this service will be made available over the IP backbone connection, and that in such cases it will require the use of VLANs (for an Ethernet connection) or frame relay DLCI (for a SONET connection) MPLS Services Internet2 network staff will work with connectors to implement MPLS tunnels through the Internet2 Network on a case-by-case basis. DRAFT December 4,

14 2. Details of Multiservice Switching Infrastructure and Dynamic Circuit Services 2.1 Multiservice Switching Infrastructure Architecture The multiservice switching infrastructure provides fixed-bandwidth circuits across the Internet2 backbone, between multiservice switches at the edge of the network. The following section describes the multiservice switch functionality as deployed in Internet2, then shows the architecture of the boxes interconnections with each other and with users Multiservice Switch Description Each multiservice switch has two or more core infrastructure circuits that connect the box to other multiservice switches. The circuits that connect to other infrastructure boxes are all SONET, and each SONET circuit consists of multiple OC-1 channels. Each box also has one or more user interfaces. The user interfaces may be Ethernet or SONET. If the connections are Ethernet, the box encapsulates the Ethernet frames into SONET streams using Generic Framing Protocol (GFP). The user SONET streams are divided into one or more OC-1 channels at the interface, using the Virtual Concatenation (VCAT) capabilities of the box. DRAFT December 4,

15 Figure 4 The Add/Drop Multiplexer (ADM) in the multiservice switch breaks out each OC-1 stream in a SONET connection and inserts it into a different SONET connection. Subrate circuits are carried in one or more OC-1 streams within each SONET connection. VCAT is used to define which sets of OC-1 streams are part of the same subrate circuit. To combine multiple OC-1 streams into a subrate circuit, VCAT signaling is carried within the SONET stream. Figure 4 shows a multiservice switch Multiservice Switching Infrastructure Interconnections between Boxes and Users The multiservice switching infrastructure is used to provide circuits for both the static and dynamic circuit services. The two types of user circuits use different core circuits to carry the user circuits internally. This is to prevent any interference between static and dynamic services on the internal circuits. Section 2.3 contains further discussion of how this service is used to interconnect with other networks and provide connections between users on connected networks. DRAFT December 4,

16 Figure 5 shows the architecture of the multiservice switch data plane infrastructure. Figure 5 Figure 5 shows a set of multiservice switches interconnected with SONET circuits to provide dynamic circuits. User multiplexers are attached to each multiservice switch (note that multiple users could be connected; to simplify the picture, only one is shown). Each user may multiplex multiple circuits to the multiservice switch. Each user circuit is connected to a circuit which is made up of one or more segments between multiservice switches. Note that the bandwidth on a segment between any two multiservice switches is shared by all users of that segment. Thus the connection between boxes A and B is used for circuits between A and B, but also for circuits between A and C where the circuit between A and C is made up of segments A-B and B-C. Note that the use of multiservice switches for static circuits is very similar to the dynamic circuit setup, except that static connections between multiservice switches are only made when there are static user circuits that need them to make a circuit. DRAFT December 4,

17 2.1.3 Multiservice Switch Control Plane The setting up of dynamic circuits will be managed by control plane software that keeps track of bandwidth, allows reservation of future bandwidth, authenticates users requesting bandwidth, and reports on the status of the infrastructure as a whole, as well as on individual multiservice switches. Figure 6 shows the separation of control plane and data plane capabilities. In the multiservice switching infrastructure, control of the data plane is done using control infrastructure that is different than the infrastructure that supports the circuits. The control plane is used to communicate between control elements, and the data plane is the infrastructure that provides circuits between multiservice switches. Control elements communicate with multiservice switches to effect control. Figure 6 In this model, a connection request is sent to the control plane, which then determines if and how the circuit can be implemented. The control plane elements signal the multiservice switches to find the state of each box and to set the boxes to make the circuit. The control plane for the multiservice switching infrastructure is the Ciena software that manages the Ciena Core Director devices. Additional software will be integrated with the Ciena software to provide additional capabilities for dynamic circuit services, such as interoperation with other networks offering dynamic circuits. DRAFT December 4,

18 2.2 Dynamic Circuit Service Two major differences between static circuits and dynamic circuits are: 1) dynamic circuits assume a fabric of interconnected devices that want to make periodic connections to each other, and 2) dynamic circuits give users the ability to create services directly rather than having them initiated by network engineers. These concepts were introduced in section 1.3.2, and are expanded in this section Dynamic Circuit Details To use the dynamic circuit service, a user must make a physical connection to the Ciena multiservice switching infrastructure using an Ethernet or SONET client interface. The Ethernet interfaces may be 1GE or 10GE. SONET interfaces may be OC-48 or OC-192. Over the physical interface, the user may carry a single circuit or multiplex multiple sub circuits. An Ethernet interface multiplexes VLANs. A SONET interface multiplexes streams, where each stream is made up of multiples of OC-1 streams. The bandwidth of the physical interface and the bandwidth of the circuit provided across the network do not need to be the same. Each incoming, possibly multiplexed, circuit is connected to a dynamic circuit across Internet2. The physical interface to the Internet2 circuit has a specific bandwidth, and this bandwidth sets the upper limit for the bandwidth of the circuit (or circuits) carried by this interface. The dynamic circuit may be configured in increments of OC-1. For Ethernet connections, the circuit provided to a remote Ethernet device is a point to point Ethernet circuit. The circuit may be a tagged VLAN at one end of the circuit and untagged Ethernet at the other end, or be the same at both ends. Ethernet circuits are carried on Internet2 dynamic circuit services as GFP encapsulated SONET. GFP encapsulation and decapsulation of Ethernet frames is done at network ingress and egress. When the Internet2 segment of the dynamic circuit connects with a segment from another network using a SONET interface, the data carried between Internet2 and the other network is GFP encapsulated SONET. This means that the GFP encapsulation must be done at the termination of the SONET part of the circuit. A good example occurs when the circuit includes a segment from Internet2 and a segment from GÉANT, where the users at each end connect using Ethernet, but Internet2 and GÉANT connect using SONET. In this case the circuit between Internet2 and GÉANT is GFP encapsulated SONET. Physical connections that use Ethernet must support 802.3x (flow control). Physical connections using Ethernet VLANs must support 802.1q (VLAN). DRAFT December 4,

19 For SONET connections, the circuits provided are point to point SONET circuits at multiples of OC-1 bandwidth. Incoming physical SONET circuits may be channelized or unchannelized. As noted, the Ciena infrastructure connection across Internet2 is always SONET. Dynamic circuits' SONET connections are always channelized. User SONET connections may be channelized or unchannelized. If unchannelized they are converted to channelized at the edges Dynamic Circuit Control The dynamic circuit service creates circuits using control plane software. In the initial deployment, dynamic circuits are provisioned only by the control plane, and are requested by users interacting with the Internet2 NOC. The NOC uses control plane software to implement the circuit. In future deployments, software will be added to allow circuits to be created at user request, if the request meets the policy requirements of the networks providing the circuit segments. We expect that several incremental deployments will create capabilities to make automated requests using a web form, then to make requests directly from applications, then to check policy but leave final approval to the NOC. Other enhancements will be made to support the ability to interact with dynamic circuit capabilities from other domains. We expect that research on and initial deployment of much of the control plane software will take place on the HOPI test service, before the software is moved to the dynamic circuit service. The ability to create dynamic circuits across domains using shared software is a topic of research in many areas. Internet2 expects to bring its experience in deploying such networks to IETF and perhaps other standards bodies. 2.3 Connecting Internet2 Dynamic Circuit Services to Users This section examines a number of ways that dynamic circuits may be connected to other users and networks. The methods of interconnecting are undergoing intense investigation. Presented here are best guesses as to how this will happen, in order to provide insight for actual implementations Connecting Two Regional Networks and Their Users The dynamic circuit service provides the ability for a RON to connect to Internet2 and provide dynamic circuit services across Intenet2 to a user on another regional network. In this case, the regional network makes a physical connection to Internet2. In the most common case this will be an Ethernet connection that supports VLANs. The regional network on each end creates a VLAN circuit to its user and then makes a VLAN connection to Internet2. The VLAN segments are joined to create an end to end circuit between the users. DRAFT December 4,

20 Figure 7 shows such a connection. The regional network provides its own circuit multiplexing capability that takes circuits from its users and multiplexes them over its connection to Internet2. Each multiplexed connection is sent by Internet2 on a separate SONET Virtual Container connection to a multiplexed connection on a different RON. Figure 7 Some notes about this setup. The physical connection between the RON and Internet2 is permanent. The dynamic circuits carried over the physical connection change over time, depending on the requirements of the users. The dynamic circuits are set up and managed by control plane software, and if the RON and Internet2 have compatible control plane software, the circuits may be set up automatically across domains. Cross-domain control plane software is a heavily investigated research topic at this time. Internet2 is working with others to develop software that will allow RONs to control Ethernet switches and interface with the Internet2 control plane software. Internet2 is also participating in research and standardization efforts to create control plane designs and specifications that will allow interoperability of control plane software developed by different organizations. DRAFT December 4,

21 Figure Connecting Internet2 with other National and International R&E Networks Internet2 dynamic circuit services connect with similar services provided by other national and international R&E networks such as ESnet, GÉANT, and CANARIE. Each interconnection is a physically permanent connection over which dynamic circuits are multiplexed. These interconnections take place either directly with other networks on a one-to-one basis, or as proposed by the Global Lambda Integrated Facility (GLIF) at exchange points where multiple networks come together and circuits may be switched among any of the connected networks. Following GLIF, we refer to these exchange points as Global Optical Lambda Exchanges (GOLEs). Using these interconnections and appropriate manual and automatic control of circuit switching, Internet2 will be able to be a partner in creating circuits from users on regional networks in the United States to users on networks connected to other core networks. The data interface between core networks may be done using either Ethernet or SONET/SONET interfaces. The control plane interface is logically very similar to the interface between Internet2 and regional networks. At the present time, the biggest differences in control plane software interfaces appear to be in defining how to authorize users requesting services, and how to share information about reachability between networks. Internet2 is involved with groups working on ways to provide and standardize these. DRAFT December 4,

22 Two models of how networks could interconnect are described below. A core provider dynamic circuit network architecture Figure 9 shows a possible architecture of a set of networks that allow interoperation among themselves. In this case, there is a set of core networks that are interconnected such that they can create dynamic circuits among themselves, just as in the previous example. Figure 9 Connected to each core network is a set of regional or provider networks. These regional networks provide dynamic circuit service to organizations that have connections to them. Interconnecting networks at GOLEs Figure 10 shows the addition of exchange points, or GOLEs, to which networks connect with dynamic circuits. By connecting to a GOLE, a network can create a circuit to any other network that also connects to a GOLE. DRAFT December 4,

23 Figure 10 TeraGrid 2.4 Use Cases Applications TeraGrid provides an integrated set of computing resources to users for a scheduled period of time. Users connect to these resources during their scheduled time. Dynamic circuit services can be used to create a high-performance circuit from the user to the TeraGrid infrastructure during each user s scheduled time. High-performance videoconferencing Dynamic circuits can be used to set up high-definition video conferences. This would be valuable for scientists and researchers working on a joint project from widely separated locations. An application currently being developed will permit groups in Michigan, Indiana, Maryland, and Virginia to create dynamic video conferences. This application will use 2 x 1GE circuits to transmit data. DRAFT December 4,

24 evlbi (electronic Very Long Baseline Interferometry) This is an application in which multiple radio telescopes in widely separated locations are used simultaneously to provide sensitivity equal to that of a single gigantic (but impractical to build) radio telescope. Dynamic circuits will be used to carry signals from many sources to a central processing site. The radio telescopes may also be used sequentially by multiple cooperating processing sites. Remote medicine Dynamic circuits can be used to create temporary high-performance connections between hospitals at various locations. A specialist at one location could create and use a connection to another location to view real-time images from remote diagnostic equipment (e.g. MRI) or perform telesurgery. Highperformance videoconferencing over dynamic circuits will be useful for medical diagnosis and consultation. IP load shifting Two sites that normally connect over IP networks may occasionally have the need to transfer very high volumes of traffic, such that this would be disruptive to other users. In this case the IP routers at each site may be configured to create a direct path between the sites, using a temporary circuit directly between the local routers. File transfer Dynamic high-bandwidth circuits can be used for file transfer. The protocols and timing for doing these transfers are being investigated in a number of places, mostly using the IP network for transport. The dynamic circuit service will eliminate congestion problems with high-speed networks, but not other problems caused by high-speed long-distance transfer. Internet2 projects such as Phoebus and VFER may use dynamic circuits as one way of improving performance Models The following are a few models describing how the dynamic circuit service may choose to allocate resources. The actual methods for allocating resources are being worked out in the network as applications come on line. These models are presented to give an idea of the different methods that are possible when circuits can be allocated relatively quickly (in minutes, seconds, or less). Note that where circuits consist of segments from multiple networks, all the networks must have an interoperable method of allocating resources. This is a subject of current research. Telephone model The telephone model allocates resources at the time of request, on a first-come-first-served basis. If a resource is not available then the request is refused and the user tries again later. Hopefully the DRAFT December 4,

25 resource will be available shortly, and within a small number of tries. This method depends on the resource being used for short periods of time, and the rate of requests not being such that the requests cannot be satisfied by existing resources. We will use this model initially as we bring applications on line. We will supplement or replace this model with other models as requirements become clear. Doctor's office scheduling In doctor's office scheduling, a user requests a resource at a specific time. A scheduling program checks to see if the resource is available at this time, schedules an appointment if time is available, or suggests an alternative time. When the appointment time arrives, the resource is likely to be available, but there is a chance that the user may have to wait some (usually small) amount of time. This type of scheduling is relatively simple and does not require aborting existing applications in order to meet a timetable. Scheduling is done such that the waiting time for the resource is less than a specified time for a specified percentage of the time. Classroom scheduling In this type of scheduling, a resource is allocated for a specific amount of time. Like a classroom, it is used for a period and must be vacated after that period is complete, so that a new user may take over the resource. This type of scheduling is needed when other resources (e.g. supercomputing clusters) are also scheduled for fixed amounts of time, or when some other resources need to be scheduled simultaneously. Virtual organization In this method of resource allocation, a set of resources is assigned to a virtual organization and the resources are allocated by that organization. This lets an organization use a set of communication resources as it sees fit, given the total set of resources, including computation and storage, that it has available to solve a particular problem. Multiple simultaneous circuits This is a method of scheduling multiple circuits to be used simultaneously in an application. In some cases, exactly when a circuit is available may not be as important as having a number of circuits available at the same time. The evlbi application, where multiple sources must be correlated at the same time, is a good example of such a scheduling need. DRAFT December 4,

26 Appendix A. Dynamic Circuit Service Detailed Description A.1 Connecting to the Dynamic Circuit Service Connections to the dynamic circuit service are either SONET-based or Ethernet-based. Type Ethernet SONET Interface 1GE 10GE OC-48 OC-192 Frequency 1310ns 1310ns 1310ns 1310ns 1550ns 1550ns 1550ns 1550ns Other LAN/PHY Channelized Channelized Tagging VLAN or VLAN or Untagged Untagged These interface types are supported at all locations where Ciena multiservice switch equipment is located. A.2 Transport Service Definition The service infrastructure of the dynamic circuit service is transported using SONET framing and incorporating GFP encapsulation for carrying Ethernet. Both VCAT and LCAS capabilities will be implemented. The initial deployment of the service will use a set of waves on the infrastructure dedicated to the dynamic circuit service, and separate from the set of waves used for the static circuit service. Note that 10GE connections to the Ciena equipment will be carried by SONET and will be restricted to SONET bandwidth limitations. Service Requirements: All SONET connectors must support VCAT and LCAS. All SONET connectors providing Ethernet services must support GFP. All Ethernet connectors must be capable of supporting 9K (MTU) payload frames. Ethernet participants may be tagged with VLANs or untagged, and VLANs may be switched internally on the transport. That is, a VLAN tag on one end need not be the same as a VLAN tag on the far end. Transport bandwidth will be allocated in increments of OC-1 channels in SONET or OC-3 channels as needed. Path setup must specify bandwidth. It is strongly recommended that all Ethernet connectors support IEEE 802.1p (flow control). DRAFT December 4,

27 A.3 Encapsulating Ethernet in SONET When a circuit attaching to the dynamic circuit service is carried over an Ethernet physical interface, the Ethernet frames are encapsulated using GFP and carried over SONET. When the circuit is converted back from SONET to Ethernet, Ethernet frames are decapsulated from SONET back to Ethernet. When connecting to Internet2, signaling on a SONET circuit will indicate whether it carries GFPencapsulated Ethernet. A.4 Control Plane Definition Initial services will be manually configured. Connectors will contact the Internet2 NOC for circuit setup, and a NOC engineer will be assigned to coordinate with other networks to create the required paths across the network. Automated services are expected in the near future. Automated services will use DRAGON-style control plane capabilities. Control plane channels will be transported via IP, and connectors will be offered several options for implementation and/or interaction with the dynamic control plane. DRAFT December 4,

28 Appendix B. Bandwidth Used by Physical Interface The bandwidth used by each circuit on a physical connection is assigned by the control plane. Each physical connection may be set up to carry a single circuit on the backbone or may be multiplexed to carry multiple connections. When it carries multiple connections, each connection to the network has a different path in the network. Each path may go to a different egress multiservice switch on the network. The control plane allocates bandwidth to each path. The maximum that can be allocated to all circuits coming in on a physical circuit is limited only by the speed of the physical interface. The minimum bandwidth of a circuit is OC-1. Each physical interface may multiplex multiple circuits over the interface. 1GE and 10GE physical connections may multiplex VLANs. SONET interfaces may multiplex SONET sub-channels. In both cases, the bandwidth of the path allocated to the multiplexed circuits is allocated in increments of OC-1 channels (approximately 50Mbps). For example, a 1GE connection may multiplex 4 VLANs: one with a bandwidth of 1 channel, one with a bandwidth of 3 channels, and two with a bandwidth of 6 channels each, with the rest (5 channels) available for additional VLANs. Note that the bandwidth of the circuit across the backbone and the bandwidth of the interface to the user may be different. It is up to applications to be able to handle this difference in bandwidth. For example, a RON may have a physical connection to Internet2 over which it multiplexes a number of circuits for its users. One of these users connects to the RON with a GE. The RON creates a VLAN to Internet2 to carry that circuit to a remote location. The VLAN is allocated 250 Mbps of bandwidth. In this case, an application that sends data at a regular rate less than 250 Mbps will work well, but an application that sends data at a bursty rate, even if less than 250 Mbps, may not work well. DRAFT December 4,

29 Appendix C. Phase 1 Control Plane Rollout The following describes current thinking about how control plane software for dynamic circuit services will roll out. This is provided as information about current thinking (December 2006) and will be updated as we gain a better understanding of how deployment will happen. This is not a commitment that any of the following will happen. In Phase 1 of the rollout, the control plane software provided by Ciena will be used to create circuits. In December the connections will be created by the NOC based on requests from users. This is for the multiservice switches that are deployed at the time. Around February 2007 a web form will be provided to allow users to request a connection. These requests will initially be filled by the NOC, but in about the same timeframe, the web form will be able to interface with the control plane software to make connections without intervention by the NOC. Also in February or March, Internet2 will provide software to regional networks that will allow them to provide switched circuits to their users. The software will also interact with Internet2 software to allow circuits in a RON to attach to Internet2 circuits. The details of the future control plane software are still being worked out. We expect to have a plan for long term development by December or early January. The future control plane software will be based on the DRAGON software, which is currently deployed on the HOPI network for control of HOPI Ethernet switches, as well as on the DRAGON network. Internet2 is increasing the resources devoted to developing this software, and will collaborate with the DRAGON project and others to continue its development. In addition to collaborating with DRAGON, Internet2 is working with a group of core networks (GÉANT, ESnet, and CANARIE) to define how to set up connections between the networks. The expectation is that initially we will set up connections using NOC interactions between networks. We expect to define and implement mechanisms to allow users to request connections using web forms, and then to create software that will allow the core networks to request connections with each other on behalf of their users. We will present a more detailed plan for development of this inter-core software in December. DRAFT December 4,

30 Appendix D. One Example of How to Connect a Campus to the Dynamic Services Network This appendix outlines a simplified process for connecting a campus infrastructure to the Dynamic Services portion of the Internet2 Network. It describes a step-by-step procedure for first connecting to the Hybrid Optical and Packet Infrastructure (HOPI) and then evolving that connection to the Dynamic Services network. Clearly, one could connect directly to the Dynamic Services network, right from the beginning. See the Dynamic Services Description ( specifying these types of connections. Alternate connections are possible, but Ethernet is the simplest way to begin. Phase 1. Connect the campus infrastructure through the RON to the HOPI switch using a fixed set of VLANs. The campus can deploy an inexpensive 1GE or 10GE switch that can connect to campus applications expected to use the dynamic services. The campus also can deploy a Virtual Label Switch Router (VLSR) on a PC and connect it to the switch. The VLSR will participate in a Generalized Multi-Protocol Layer Switching (GMPLS) control plane through the IP network that parallels the data plane. The DRAGON software first can be deployed on the VLSR; then it can begin participating in the HOPI control plane. At this point, the campus is connected to HOPI and participating in the HOPI control plane administrative domain. Note: the RON does not have to set up the control plane capabilities to begin this process. The following illustration describes the setup: Phase 2. Add a Network Aware Resource Broker (NARB). This can be done on the same PC as above, or it can be deployed on a separate PC. The function of the NARB is to separate the campus domain from the HOPI domain. Once this is accomplished, the campus is running its own administrative GMPLS control plane and it is peering with the HOPI administrative GMPLS control plane. The picture below describes the setup: DRAFT December 4,