Layer 1 Replication. Leveraging the Layer 1 Crossbar for Low Latency Local and Remote Replication

Transcription

1 Leveraging the Layer 1 Crossbar for Low Latency Local and Remote Replication Introduction Plexxi has created the industry s first data center switching solution that includes user, computational, and programmatic-based agility including layer 1. This type of architecture and solution provides many advantages to data center operators and those responsible for delivering innovative and fast evolving applications that rely on the data center infrastructure. While much of the industry has focused on providing manual, centralized override to the traditional legacy control plane (i.e. SDN), very little to no attention has been paid to the tremendous advantages that can be gained by additionally offering Layer 1 manual, computational, and programmatic controls. Much of the power of Plexxi s solution comes from this ability to dynamically re-compute physical topologies based on actual workload demand. An additional, oft-overlooked, capability that results from this is also the ability to leverage the layer 1 infrastructure for tasks that are normally very complex and inefficient at Layers 2-4, namely packet replication. Replication at Layer 1 does not require complicated protocols or latency intensive packet duplication (copying) in silicon. Instead, layer 1 replication takes the entire physical stream and replicates it at the wire. If such a capability could be combined with centralized, user-defined, and computational controls, it could provide immense benefits for customers attempting to do very high-speed, high-bandwidth, low-latency distribution of content within and across data centers. This paper will examine exactly these aspects of Plexxi s industry leading solution. Page 1 of 11

2 Control Plane Plexxi Legacy Competitors Legacy Competitors + SDN Layer 2-4 (Logical Topologies) Computational Distributed protocols, shortest-path defined Manual Centralized Override Flow-defined (i.e. SDN) Computational Centralized Override policy, workload, or traffic defined (i.e. application-centric) Yes Yes Yes Yes No Yes Yes No No Control Plane (distributed + centralized) Cooperation Model Fully cooperative N/A Ships in the Night Programmatic / Automated Yes Some Yes Layer 1 (Physical Topologies) Manual Override Link-by-link remapping Computational Centralized Override policy, workload, or traffic defined (i.e. application-centric) Yes No No Yes No No Control Plane (distributed + centralized) Cooperation Model Fully cooperative N/A N/A Programmatic / Automated Yes N/A N/A Figure 1. Plexxi's Innovations with focus on Layer 1 Figure 1. Plexxi s Innovations with focus on Layer 1 Plexxi Switch Port Overview Access and Ports Page 2 of 11

3 Plexxi Switch Port Overview Access and Ports All 4 models of the Switch 2 (excluding the Switch 2e), are based on the same base design with different port combinations. The total number of equivalent 10 GbE ports on all of these systems is 96, which represents the total non-blocking capacity of the internal L2/L3 switching ASIC (Broadcom Trident 2). The ports are laid out differently on different models. The Switch 2 and the Switch 2p (shown in Figure 3. Switch 2 and 2p external ports) are both nonoversubscribed configurations, so that 48 of the ports (presented as 12 QSFP+ ports) are used as access ports (for external host or switch connectivity). The other 48 ports are used for the switch to switch interconnect system. For these ports, internal optical transceivers are used to generate 12 x 10 GbE wavelengths which are muxed onto each individual port. In the Switch 2 there are 4 of these ports, providing the 48 Waves, which correspond to 48 internal ports on the L2/L3 ASIC. 4 x MTP) Figure 2. Switch 2 and 2p external ports 12 x QSFP+ Dedicated FlexxPorts 4 x QSFP+ & 12 x SFP+ The Switch 2s and 2sp (show in Figure 2. Switch 2s and 2sp external ports) are both oversubscribed configurations so that 72 of the 96 ports (presented as 16 QSFP+ and 8 SFP+) are presented as access ports, while the remaining 24 are mapped to waves on the ports. Since the 2/2p and the 2s/2sp are based on the same design, it should be noticed that the additional access ports on the 2s/2sp are actually FlexxPorts that are initially mapped to Broadcom Trident ports internally in the switch, which we call Flexx Access Ports. These ports can actually be re-purposed as FlexxPorts (discussed below) which enable the L1 replication capabilities that will be discussed in this paper. 2 x MTP Figure 3. Switch 2s and 2sp external ports 12 x QSFP+ dedicated Re-purposable Access Ports (Flexx Access) 4 x QSFP+ 8 x SFP+ Dedicated FlexxPorts 4 x SFP+ 2 x SFP+ Monitor Page 12 of 11

4 2 x MTP 12 x QSFP+ dedicated Re-purposable Access Ports (Flexx Access) 4 x QSFP+ 8 x SFP+ Dedicated FlexxPorts 4 x SFP+ 2 x SFP+ Monitor The table below shows the basic configuration variations between the 4 models of the Switch 2. Model Equiv. 10 GbE Ports Dedicated Access Ports Flexx Access Ports Waves Dedicated FlexxPorts Switch 2s access x QSFP+ 8 x SFP+ 4 x QSFP 24 x 10 GbE 4 x SFP+ Switch 2 96 (48/48) - 48 access x QSFP+ 48 x 10 GbE 4 QSFP+ & 12 SFP+ Switch 2sp 96 (72/24) - 72 access x QSFP+ 8 x SFP+ 4 x QSFP 24 x 10 GbE 4 x SFP+ Switch 2p Figure 4. Switch Model Table 96 (48/48) - 48 access x QSFP+ 48 x 10 GbE 4 QSFP+ & 12 SFP+ FlexxPorts The Switch 2 family also introduces an industry-unique notion of a FlexxPort. FlexxPorts are physical ports, either in SFP+ or QSFP+ form factor, that can be used for a number of purposes. On the Switch 2 series, some number of ports are dedicated as FlexxPorts, meaning they are additional ports not counted in the total access or ports, and not mapped to either the Broadcom Trident or the in the initial configuration. Figure 4 also shows the number of dedicated FlexxPorts in every switch model. It is also important to understand that some number of access ports can also be repurposed as FlexxPorts and we call these Flexx Access Ports. These are ports that are initially configured and presented as access ports, but can be redesignated by the user as FlexxPorts. This is important as we will later see since these ports, when used for L1 replication, can simultaneously provide L1 replication and traditional access port capabilities that a source may expect (VLANs, MLAG, etc). Page of 11

5 Internal Switch Connectivity (introducing the Cross-connect) In order to understand what a FlexxPort is, it is important to understand the switch internal connectivity, shown below (Switch 2 variant). 48 CWDM Transceivers 8 colors Feeding LR ports 2x East, 2x West Each LR in each direction Carries 12x10GbE Mux/Demux + Filters Cross-connect (1:N capable) Trident 2 Configured to connect 48 Trident ports to 48 CWDM Transceivers Intel CPU Ports 4 x MTP (4 x 12 x 10 GbE) Access Ports 12 x QSFP FlexxPorts 4 x QSFP 12 x SFP+ 2 x Monitor SFP+ FlexxPorts not connected To Trident, ONLY to Cross-connect Figure 5. Switch 2 and 2p internal mapping Starting in the middle at the Broadcom Trident 2 L2/L3 switching ASIC there are a total of 96 ports connected to this device. 48 of these ports are presented as 12 x QSFP+ access ports. The other 48 ports connect to an internal Layer 1 cross-connect. This cross-connect acts as an internal patch panel of sorts, which based on its software configuration, can map any incoming port to any outgoing port. In its initial configuration, 48 of these ports (called fabric ports) are mapped to internal WDM-based transceivers that generate the waves and are muxed onto the 4 external ports seen on the left. However, the cross-connect is also mapped to the FlexxPorts, which exist in both in QSFP and SFP flavor. Using software control, a user could decide to leverage these FlexxPorts as additional access ports by changing the cross-connect to re-map some number of those 48 fabric ports to the FlexxPorts instead of to the. Or, FlexxPorts could be used to allow the user to connect an external source directly onto the by disconnecting one of the existing fabric ports from the muxing complex and remapping it to a FlexxPort. This could be used to provide long distance fabric ports, for example. 48 CWDM Transceivers 8 colors Feeding LR ports 2x East, 2x West Each LR in each direction Carries 12x10GbE Mux/Demux + Filters Cross-connect (1:N capable) Trident 2 Configured to connect 48 Trident ports to 48 CWDM Transceivers Intel CPU Ports 4 x MTP (4 x 12 x 10 GbE) Access Ports 12 x QSFP FlexxPorts 4 x QSFPs & 8 x SFPs Initially mapped as Access Ports 4 x SFP+ dedicated 2 x Monitor SFP+ dedicated 4 FlexxPorts not connected To Trident, ONLY to Cross-connect Figure 6. Switch 2s and 2sp internal mapping Page 5 of 11

6 Understanding Cross-Connects Cross-connects are devices that allow any input port to be mapped to any output port. Depending on the type of cross-connect, they may also include the ability to take an input port and map it to N output ports (called replication). This is the type of device that resides in a Plexxi Switch 2. Input ports Output ports Output signals Input signal Cross-connect 1:1 and 1:N mappings Figure 7. Simple depiction of a cross-connect Cross-connects in the Plexxi Switch 2 In the Plexxi Switch 2, a cross-connect like the one depicted in Figure 7 is implemented to provide any-to-any mapping of the various types of ports. As discussed earlier, there are 3 main types of ports in the Plexxi Switch: Access Ports (Dedicated), FlexxPorts (both dedicated and Flexx Access), and Ports. Access Ports and FlexxPorts typically have a 1:1 correspondence between the port and the internal signal mapped to the crossconnect, while the Ports leverage Wavelength Division Multiplexing, so the signals on the crossconnect are actually the individually Waves. As was shown earlier, all 3 types of these ports connect into the internal cross-connect as depicted in Figure 8. Mux/Demux + Filters 48 Waves Cross Connect (1:N capable) Trident 2 Access Ports FlexxPorts Figure 8. Cross-connect configuration on Switch 2/2p Page of 11

7 Usage of the Cross-connect in Plexxi Switch 2 There are a number of uses for the cross-connect in the Plexxi Switch 2 that are completely unique in the industry: 1. Reconfiguring Access Port to Wave mappings to create workload specific L1 topologies. 2. Mapping Access Ports to FlexxPorts for metro or long-distance connectivity (DCI) 3. Mapping FlexxPorts to Access Ports to create more access port connectivity 4. Leveraging FlexxPorts or Flexx Access Ports as L1 Replication ports (focus of this paper) Depending on the specific use case, there control model can be centrally computed via the controller s fitting functionality, or manually specified by the user, or in some cases both. Uses 1-3 are covered in detail in separate papers. L1 Replication Functionality The internal cross-connect in the Switch 2 family not only has the ability to be reprogrammed to dynamically to change input/output port mappings as discussed above, but it also has the ability to map any input port to N output ports, providing layer 1-based replication. This type of replication does not perform packet copies, so it can be performed at very low latency (normally nanoseconds (independent of packet size), versus nanoseconds latency of a typical switching ASIC doing packet copying). By combining the 1:1 port mapping and the 1:N port mapping capabilities on different switches, the solution can provide both local and remote replication, and can do both simultaneously. This has immense implications for operators looking to easily distribute some number of sources to many receivers within the same rack, across the data center, or even across many data centers. Here are some examples of the many combinations that are possible due to this capability: Local Replication (same switch) Source (1) -> FlexxPort (1) -> FlexxPort (N) -> Receivers (N) Source (1) -> FlexxPort (1) -> Access Port* (1) -> Receivers (N) Remote Replication (across Plexxi fabric) Source (1) -> FlexxPort (1) -> Port (1) -> Port (1) -> FlexxPort (N) -> Receivers Source (1) -> FlexxPort (1) -> Port (1) -> Port (1) -> Access Port* (1) -> Receivers (N) *it s important to note that access port replication is done using typical packet copying on the switch silicon (Broadcom Trident), which will incur additional latency, depending on packet size. Page of 11

8 Market Data Customers Customer Access Switches Plexxi Fabric Market Figure 9. L1 Replication for Market Data Distribution Up to 80km without additional optical transport equipment 1000s of km with traditional optical transport Local Replication In the local replication cases, the source is connected to a FlexxPort, which can either be a dedicated FlexxPort, or Flexx Access Port. The FlexxPort can then be configured to replicate the traffic stream onto N output FlexxPorts, which would be locally connected receivers or remaining FlexxPorts (either dedicated or Flexx Access). Additionally, the input FlexxPort can be configured to output onto an Access Port by mapping the cross-connect to the appropriate access port. The access port is an input into the switching ASIC, which has its own ability to replicate traffic, leveraging traditional Ethernet/IP multicasting capabilities. 2 x MTP N receivers connected to receive stream at ns latency (depending on packet size) Receivers 12 x QSFP+ dedicated Multicast Source Multicast source connects to Flexx Access or Dedicated FlexxPort Flexx Access and Dedicated FlexxPorts N receivers connected to Flexx Access or Dedicated FlexxPorts. Receive stream at 20-30ns latency Receivers Figure 10. Local Replication Example Remote Replication There are 2 types of Remote Replication use cases supported by Plexxi: Direct Drop and Multi-drop. Page of 11

9 Direct Drop Remote Replication The direct drop remote replication cases are similar to the local cases, except that the input FlexxPort on the cross-connect is mapped to an output port connected to the. This means that the traffic is directly placed onto an outgoing wave that terminates onto a remote switch. On ingress into the remote switch, that wave hits the remote switch s cross-connect, from where it gets mapped (and replicated) to N FlexxPort receivers and/ or to a single access port that is connected to N Ethernet/IP multicast receivers. Delivery out customer access ports can be Done packet switched (through Trident) Or through optical bypass Multicast server attached to a Flexx Access port Trident provides regular access port functions Like VLAN, IP, etc Cross connect in Switch replicates signal towards Customer access switches directly using Optical bypass Figure 11. Direct Drop Remote Replication with A/B Feeds (blue/gray) Multi Drop Remote Replication The multi drop remote replication use case leverages the ability of each switch to both replicate the packet stream locally and remotely at the same time. Therefore, the Plexxi fabric can leverage tremendous fiber savings by essentially passing the replicated traffic around the fabric while at each switch replicating it to the locally connected receivers. This use case shows a little more latency in that each optical replication will incur an additional 20-30ns replication latency, plus the propagation delay (~1ns per foot) if the switches are geographically dispersed. However, the resulting performance is still orders of magnitude better in latency and efficiency than IP-based multicasting. Page of 11

10 Each Switch introduces approximately ns latency when replicating signal to next switch Replication is done using optical bypass Cross connect in Switch replicates signal onto Wave towards East or West switch through optical bypass Same for all customer facing switches: replicates into east direction, Delivery out customer access ports can be Done packet switched (through Trident) Or through optical bypass Receiving Switch replicates into east direction Through optical bypass Summary Figure 12. Multi-drop Remote Replication Adding Layer 1 programmability and flexibility into a data center switching solution is a completely novel capability, only supported by Plexxi. The ability to compute logical and physical topologies with a centralized controller and user-based manual override is also critical to support the extreme flexibility and agility needed in today s data centers. No other data center networking vendor offers such powerful capabilities. Additionally, for customers that rely heavily on content replication, Plexxi s unique ability to flexibly replicate traffic at Layer 1 for low-latency copies and simultaneously provide the ability to give users to control of where and how these replicated feeds are sent means that content replication no longer has to be costly, inefficient, or difficult to control. These capabilities are immensely powerful, yet with Plexxi s computational and user-defined manual overrides can be very simple to implement. Glossary of Terms Multicast server attached to a Flexx Access port Trident provides regular access port functions Like VLAN, IP, etc Ports are external ports (for pluggable SFP/QSFP Ethernet transceivers or DACs) that are directly connected to the internal switching ASIC and provide traditional L2/L3 functionality for connected hosts or networking devices. FlexxPorts Dedicated FlexxPorts Dedicated FlexxPorts are external ports (for pluggable SFP/QSFP transceivers) that connect to the internal cross-connect. These ports are not initially wired to either the switching ASIC or the, but are available to be used for either additional access ports or additional capacity, based on the user need. Flexx Access Ports Flexx Access Ports are external ports (for pluggable SFP/QSFP transceivers) that connect to the internal cross-connect but are also initially mapped (via the cross-connect) to terminate onto the internal switching ASIC, therefore functioning as Access Ports. They are known as Flexx Access ports because they can also be repurposed as needed, and they have the unique capability that they can simultaneous behave as traditional access ports and FlexxPorts in the L1 replication case. Page 10 of 11

11 Port A Port is an external port that leverages an MTP style optical connector. MTP provides 12 fibers (6 pairs) per connector. Wave A Wave is an internal 10 Gbps connection that is mapped to a specific wavelength (lambda) and multiplexed onto a fiber that egresses to a Port. Internal Switch Components L2/L3 Switching ASIC The L2/L3 Switching ASIC (also referred to as the Broadcom, the Broadcom Trident, or the Trident) is an off-the-shelf ASIC that provides traditional L2/L3 switching and routing functionality. Cross-connect The cross-connect is an internal component that provides 1:N mapping of signals and can be reconfigured dynamically via software. Mux/Demux Complex The Mux/Demux Complex is a series of passive optical muxes and filters that takes multiple ingress Waves and muxes them onto a single or multiple fibers, or in the reverse direction takes multiple fibers and demultiplexes the carried Waves. To Learn More Contact your Plexxi sales representative, or contact us using the information below. The information contained herein is subject to change without notice. Plexxi, the Plexxi logo,, and Flexx Ports are trademarks of Plexxi Inc.. Other company, product or service names may be trademarks or service marks of their respective owners Innova ve Way, Suite 3322 Nashua, NH PLEX (7539) info@plexxi.com Page 11 of 11