Plexxi Theory of Operations White Paper

Transcription

1 White Paper Introduction Cloud computing, virtualization, and distributed application architectures are reshaping data center traffic flows, escalating bandwidth and performance demands, and introducing network engineering and capacity planning challenges for IT organizations. Plexxi s network fabric solutions are specifically designed to address the ever-increasing performance, availability, and agility demands of the cloud. At the heart of the solution lies the Plexxi optical fabric an interconnected configuration of Plexxi Switches using LightRail connections that utilize a combination of wave-division multiplexing (WDM), optical switching and packet switching technology to create a highly scalable, adaptable, and resilient network fabric. Orchestrating the entire optical fabric is Plexxi s Control, a software controller that actively manages and provisions Plexxi Switches, discovers the L1 and L2 topography created, and calculates optimized forwarding topologies for fabric workloads through it s unique Dynamic Fitting Engine. This white paper introduces the Plexxi optical fabric and explains the advantages and benefits of Plexxi s networks for today s highly virtualized and dynamic data center environments. Plexxi Solution Overview Legacy data center networks, originally engineered to support conventional client-server applications, are ill-equipped for the dynamic east-west traffic flows that dominate the contemporary data center. They employ complex networking protocols and inefficient hierarchical topologies to achieve statically engineered any-to-any server connectivity. Legacy networks rely on heavily over-subscribed network designs that put untenable constraints on workload placement and provide no flexibility for changing workload needs, or over-built networks that are underutilized that impact capital expenses. Page 1 of 9

2 The Plexxi network fabric is designed from the ground up for today s highly virtualized and fluid cloud computing environments. The solution combines intelligent control software with state-of-the-art Ethernet switching hardware, delivering a unique multi dimensional fabric that enables orchestrated, direct server-to-server network connectivity across the data center, based on workload affinities. Simply put, a Plexxi fabric is built to support a collection of resources such as instances of an application workload or members of a virtual network that have a relationship to each other. In Plexxi s new networking model, application requirements direct network interconnectivity and capacity in real-time, providing direct L1, L2 or L3 pathways for machine-to-machine communications, directed by workload demands. The Plexxi solution includes intelligent Plexxi Figure 2 Plexxi highly meshed network fabric Control network orchestration and control software, and advanced Plexxi Switch hardware, which work in concert to create the dynamic network fabric. Plexxi Switches are interconnected across a highly scalable, adaptable, and resilient Plexxi optical mesh network that is ideally suited for the dynamic machine-tomachine traffic flows that dominate the contemporary data center. Plexxi Control Software Figure 1 Multi tier static spine and leaf network The Plexxi solution delivers on the promise of software-defined networking (SDN) by decoupling control and forwarding functions, and providing centralized network intelligence. Plexxi Control provides centralized discovery, configuration, and control functions in a Plexxi network. The software builds an understanding of the affinity relationships between compute, storage, and network resources across the entire data center or cloud; defines policies that optimally fit the network to each workload; and automatically orchestrates the appropriate physical L2 and packet layer L2 and L3 topology across the Plexxi optical fabric. Page 2 of 9

3 Plexxi Switch Hardware Plexxi Switches are innovative switching systems specifically designed to meet the strict performance, agility, and resiliency needs of today s virtualized data centers. The switch s advanced architecture integrates packet and optical switching with a unique LightRail optical connection that enables a network with direct switch-to-switch connectivity. Figure 3 Plexxi Switch 2s As shown in Figure 2, the Plexxi Switches includes multiple 1/10/40 GbE server access interfaces used to connect servers and other data resources to the Plexxi network. Also included are two MTP /MPO type fiber interfaces that provide LightRail connectivity to other Plexxi Switches to form the Plexxi fabric. The Plexxi Switch 3 family is based on 25GbE Ethernet technology. It provides 10, 25, 40,50 and 100GbE connectivity, and its fabric is based on 25GbE point-to-point connections. Throughout this document, Switch 2 technology is used as a reference, with 10GbE fabric link capacity (LightRail). The exact same architecture applies to the Switch 3 product family, but each fabric link runs at 25GbE (LightRail3 ), for a total of 600GbE per LightRail3, of 1.2Tbit/sec for a dual LightRail3 switches. Plexxi Optical Fabric with Lightrail Connectivity Plexxi Switches are physically connected to each other using the LightRail optical interconnect. Each switch is directly connected to neighboring switches using an inexpensive high-density single-mode fiber cable. Using the LightRail optical interconnections, the fabric can transport traffic using multiple wavelengths of light (channels) so that each switch can connect to any other switch in any number of combinations. Additional channels can be added, depending on workload requirements, to increase the bandwidth needed to support traffic between switches, and channels can be redirected to create connectivity between any two switches in a network. Each LightRail interface initially supports twenty-four 10 Gbps wavelengths allowing for the creation of highly resilient, high capacity, low hop-count meshed networks that can be dynamically controlled and optimized based on changing workload needs. LightRail3 interfaces in Plexxi s Switch 3 product family support twenty-four 25Gbps wavelengths for a combined fabric capacity of 600 Gbit/sec per Switch, 1.2Tbit/sec for dual LightRail3 Plexxi Switch configurations. Plexxi optical fabrics offer numerous advantages over conventional hierarchical networks, including: Better performance Plexxi networks provide high capacity, low-latency, direct server-to-server connectivity for latency-sensitive, bandwidth-intensive applications. Greater agility Network topologies and capacity can be re-configured in response to changing traffic demands based on workload requirements. High availability Plexxi networks are resilient to equipment or interface failures. Page 3 of 9

4 Better economics Plexxi networks and Control software contain CAPEX and OPEX by improving network utilization, requiring fewer devices, network layers, and cabling. Greater flexibility Plexxi networks combined with optical and packet switching support a variety of physical and packet network topologies, enabling diverse applications including multicast or broadcast applications. Plexxi Optical Fabric Theory of Operation Plexxi Performance and Performance plus family switches contain an optical cross-connects that enables any-to-any connectivity. These programmable optical cross-connects can connect traffic entering the switch locally or pass it through to another Plexxi Switch. Optical wavelengths carrying network traffic are connected to the optical cross-connect as they enter the switch from the LightRail connection. The optical cross-connect can connect traffic to another Plexxi Switch over a LightRail interface or connect traffic locally to the internal packet switch or access port side interface for further processing. Plexxi Switches are interconnected though the optical cross-connect to form an any-to-any physical layer Chordal Mesh. Since all the wavelengths in a switch are connected through optical cross-connects, the Plexxi network topology can be easily reconfigured to address changing bandwidth demands or interconnectivity requirements. Initial Topology When a switch is first initialized, it establishes various optical paths to other switches in the same fabric. Four parallel 10 GbE paths (40 Gbps), known as base lanes, are established to each of its immediate neighbors. Base lanes provide dedicated optical paths between adjacent switches on the LightRail. Two 10 GbE (20 Gbps), connections, known as express lanes, are established to the next four east and west neighbors. Express lanes are dedicated optical pathways that use wavelength channels on the LightRail to connect to non-adjacent switches. Figure 4 depicts a Plexxi optical fabric with base lanes and express lanes (for a single swich): Plexxi s Performance Plus switches feature multiple LightRail connections, allowing for multidimensional interconnections between switches in a Plexxi network, with each LightRail contributing 240Gbit/sec to the fabric s capacity. Figure 4 Plexxi Switch Fabric Base and Bypass connections Page 4 of 9

5 In addition to the initial path connections, the Plexxi optical fabric also supports configurable point-to-point and point-to-multipoint L1 optical paths - which can be used to efficiently carry unicast, multicast and broadcast traffic at latency s of only a few 10s of nanoseconds per switch. These L1 paths can be configured to support the connection of a single optical channel originating at one switch to multiple switches in the network. Figure 5 depicts a segment of a Plexxi optical fabric with base lanes (in red), express lanes (dotted), multi-hop paths (blue) and L1 pointto-point (yellow) and point-to-multipoint paths (red). Initial Traffic Flow Once the base topology has been established, traffic begins to flow across the fabric. For L2 traffic, the first packet arriving on an access port yields a destination lookup failure and the packet is flooded across the base lanes in both the east and west directions as well as across any other access ports in the same VLAN. Each subsequent eastward switch continues to flood in the east direction and across any local ports for that VLAN. Each westward switch does the same in the west direction, until a logical flooding break has been reached. Figure 5 Plexxi fabric Paths Simultaneously, the packet header is copied to the control processor of the switch where the source MAC address is learned. Once learned, the source MAC address is passed on to all other switches in the fabric leveraging the supervisor census protocol: an internal status communication mechanism between Plexxi Switches. Plexxi Switches do not learn MAC addresses on inter switch links; access port learned MAC addresses are exchanged between switches using the supervisor census protocol. When a switch receives a MAC address advertisement, it instantiates this MAC address in its co-resident controller L2 forwarding table. When a return packet destined to this MAC address is first received, the forwarding hardware will not have this address in its hardware forwarding tables and will send the packet up to the control processor. The control processor will program the forwarding hardware with an entry for this specific MAC address, using one of the base lanes 10 GbE paths as its egress port, in the direction that provides the shortest path towards the destination switch, or directly on an express lane connected to the destination switch if such a lane exists. The selection of the actual lane is the result of a hash calculation on the destination MAC address, ensuring traffic is distributed equitably across the available express or base lanes. L3 forwarding is done very similar to L2 forwarding. When the first to-be-routed IP packet arrives, the switch detects its own MAC address as the destination, and performs a lookup on the destination IP address. In a Plexxi network, all ingress switches perform the L3 forwarding function for all VLANs, so the next hop destination is always a local L2 destination. After this next hop destination lookup, the packet is forwarded within the destination VLAN the same way L2 packets are forwarded. Page 5 of 9

6 Plexxi Control Optimization After the base topology has been established, each Plexxi Switch communicates with Plexxi Control running in a central, IP-reachable location, using a secure TCP/IP connection. Each switch conveys its base lane connectivity and express lane connectivity information to Plexxi Control and supplies the MAC addresses of its locally-attached devices. Plexxi Control uses the individual topology information to create a network-wide topology reflecting all switches and available paths between them. The combination of all base lanes and express lanes results in a highly diverse fabric that offers a vast array of possible paths between any two switches. Paths may be established directly using the base lanes or express lanes, or indirectly via intermediate switches at L1, L2 or L3. Through the Plexxi Control APIs, administrators define affinity groups collections of applications, workloads, and tenants that have specifically defined network sensitivities between them. Plexxi Control takes these affinities, and based on the location of the affinity members, the network topology and existing traffic load creates the optimal placement of affinitized and non-affinitized traffic on the network topology. Network Topology Calculation Based on the location of the affinity group resources, their expressed needs, the topology of the network, and network statistics, Plexxi Control fits affinities onto the topology, creating a fully specified affinity topology (FSAT) for each affinity group. An FSAT is a set of end-to-end paths specified for specific sources and destinations for which an affinity has been defined. Each path in an FSAT represents the best possible path based on the affinity sensitivities and the state of the network. Once all affinities have been fit, the remaining paths between switches are used for residual traffic, and they are used for partially specified affinity topologies (PSATs). Each PSAT articulates a set of paths from each switch to any destination behind another switch. Forwarding paths in PSATs are assigned weights, which are used to determine how a switch balances traffic across PSATs to a destination. An FSAT is used when the system (Plexxi Control) is taught a well defined affinity between applications, whereas PSATs are used for all other traffic. In both cases the fitting algorithm uses all known paths between destination and load balances flows appropriately. FSAT traffic allows more fine-grained optimization. Once all FSATs and PSATs have been calculated, Plexxi Control pushes the results to the Plexxi Switches using a two-phase commit protocol to ensure reliable delivery and acceptance. Traffic Flow in an Optimized Plexxi Network Fully Specified Affinity Topologies In a Plexxi Switch, FSATs are programmed as a specific set of sources and destinations, with an egress port programmed for each. Every switch in the path between these sources and destinations is programmed with the exact same path for these sources and destinations. Traffic arriving on an access port that matches an FSAT is directed to an egress port (possibly arriving at an intermediate switch on a LightRail port) where the same match Page 6 of 9

7 and forward action is performed. Ultimately the packet is received by the switch that connects to the destination, and that switch employs conventional L2 forwarding to switch the packet towards its final destination onto an access port. Partially Specified Affinity Topologies PSATs represent a set of weighted directed graphs 1 from a switch to a destination behind another switch in a Plexxi network. When the switch needs to forward traffic to a destination MAC address that is attached to another switch, the set of PSATs for that destination switch are used to determine the link over which traffic is forwarded. Multiple weighted directed graphs are identified within a PSAT. A hash on the destination MAC address (and associated broadcast domain) determines which is used. The egress port for that path is then instantiated in the L2 forwarding hardware and used for all packets forwarded to that destination. Since the PSATs and hashing calculations are consistent across all switches, all switches place the same traffic on the same path toward the destination switch to guarantee loop free forwarding. Weighted balancing of the traffic is performed based on the destination MAC address, not actual traffic load. User-Defined Paths An operator of a Plexxi fabric can manually create paths through a Plexxi network. The path, created by starting at an ingress switch to the fabric, traveling through base and express lanes to other switches in the fabric, to arrive at the desired egress switch, can be provisioned through the Plexxi Control UI and APIs. Once provisioned, this path is monitored end-to-end by the Plexxi switches. Through the use of flow entries, user can now direct traffic matching the flow definition onto this path. If the path is not fully operational (a link or switch along the path has failed), the path will be marked invalid, and flows that used to be redirected along this path will fall back to a lower priority User Defined Path, an FSAT, or a PSAT. Topology Reconfiguration Example The Plexxi Control contains Plexxi s Fitting Engine, software that runs fitting algorithms that creates optimal network paths based on physical connectivity and application and workload defined affinities - specifically defined sensitivities between a collection of applications, workloads, and tenants. Figure 6, depicting a five-node Plexxi network is used to illustrate this process. Assume the network is configured in its default topology. Say an application at Node A has an affinity relationship and needs to communicate with a storage device attached to Node D. A connection satisfying the workload affinity attributes is required between the two nodes. Also assume that traffic associated with other affinitized applications is already flowing over the default direct optical layer Express Lane connection from Node A to Node D. 1 In mathematics, a directed graph is a set of nodes connected in a manner that avoids loops. A PSAT provides a loop-free path between Plexxi Switches. Page 7 of 9

8 In this example, the affinity relationship between the two resources dictates an isolated 10 Gbps bandwidth path between them. There are a number of potential paths from Node A to Node D, both in the East and West direction. The Plexxi network can use the default direct optical layer connection from Node A to D (shown in aqua), both East and West directions, or reconfigure the optical cross connects to set up connections through node A, E, D (shown in purple) or A, B, D (shown in blue) or other alternatives. Obviously the direct connection from Node A to D offers the most direct path for the traffic. However, since the workload requirement is to provide an isolated 10 Gbps bandwidth path and the direct path is already taken from Node A to D, an alternative path is orchestrated. The Plexxi Switches with path information calculated by the Fitting Engine in Plexxi Control establishes primary path A, E, D with the bandwidth required to support the resource workload. Node A configures a traffic connection from its packet switch to its optical cross-connect. The optical cross-connect connects the traffic to a wavelength on the west LightRail interface heading to node E. At Node E, the wavelength enters the node from the east LightRail and passes through the optical cross connect on to the west LightRail. It does not get processed by the Ethernet packet switch at node E. At Node D, the wavelength containing the traffic enters the node from the east LightRail and connects to the cross-connect that is configured to drop the traffic to the Ethernet packet switch for further processing and delivery to the storage device attached to Node D. Alternative paths, for example, path A-B-D is available and can be used to satisfy the resource workload requirement. The primary paths and alternative paths are calculated by Plexxi Control based on all network resource requirements using an advanced algorithm and distributed to all the switches in the fabric. The switches within the Plexxi network establish the connectivity without any real time involvement from the Plexxi Control; backup paths are pre-calculated by the Fitting Engine and sent to the Plexxi switches. In case of network failures, Plexxi Switches will independently react by invalidating impacted paths and replacing them with backup paths. Forwarding Priorities Plexxi Control s Dynamic Fitting Engine will calculate forwarding topologies that optimize the use of available network resources based on the workload requirements as well as a traffic matrix that describes the residual traffic requirements between Plexxi Switches. As a result, traffic will be distributed across available links, but as any network, there is still a chance that temporary congestion occurs on specific links. Traffic on links between Plexxi switches is treated with a forwarding priority, varying from very low, low, medium, high to very high. These priorities map to physical queues on Plexxi Switches for these links. Prioritization between these priorities in done in a strict fashion. Traffic that is considered very high priority will always receive priority over all other priorities. By default, residual traffic being forwarded using PSAT topologies will be considered low. Workload traffic that is being forwarded as the result of an FSAT will be considered high, and traffic forwarded using a User-Defined Path will be forwarded as very high. Forwarding Priorities can be modified from their default behavior using Affinity definitions. Page 8 of 9

9 Failure Handling Switch and link failures can happen in any network. In a Plexxi network, failure recovery is handled automatically by the Plexxi Switches without intervention by Plexxi Control. If a Plexxi Switch fails, the failure is detected within tens of milliseconds due to probe packets failing. Once the failure is detected, all PSAT directed graphs containing the failed switch as an intermediate switch are invalidated. Each switch deterministically selects new PSATs from the backup set of PSATs, and starts using those PSATs. Each switch is provided with hundreds of backup PSATs for this purpose. When the failed switch recovers and is fully functional, PSATs that include that switch are reinstated. Similarly when links between switches fail, PSATs that use these links are invalidated and backup PSATs that do not use this link are put into place. When the failed link recovers, the original PSATs are reinstated. FSATs are treated slightly differently. No backup FSATs are currently calculated by Plexxi Control. If a switch or link fails, all FSATs that use that switch or link are invalidated, and traffic associated with that FSAT falls back to the set of PSATs that direct traffic to that specific destination. Consequently the affinity sensitivities for which this FSAT was created may be lost during failure conditions but traffic forwarding will not be interrupted. Upon recovery of a failed link or switch, the FSAT is reinstated and affinitized traffic is again forwarded as defined by the FSAT. A user could run another fit when this switch is out of service, and all paths calculated will avoid this out of service switch. Conclusion Cloud computing, workload mobility, and virtualization are transforming the data center, introducing complex traffic engineering and capacity planning challenges for IT organizations. Plexxi offers the industry s first and only scaleout networking solution, built from the ground up for today s highly virtualized, on-demand data centers. Using Plexxi Control and Plexxi Switches with LightRail connectivity, enterprises and cloud providers can build flat, low latency, high performance Plexxi optical fabrics that readily accommodate the delay sensitive, bandwidth intensive, east-west traffic flows that dominate today s data centers. To learn how Plexxi can help you build a more scalable, agile and cost-effective data center network please visit us on the web at or contact a Plexxi sales representative at The information contained herein is subject to change without notice. Plexxi, the Plexxi logo, LightRail, and Flexx Ports are trademarks of Plexxi Inc.. Other company, product or service names may be trademarks or service marks of their respective owners Innovative Way, Suite 3322 Nashua, NH PLEX (7539) info@plexxi.com Page 9 of 9