InfiniBand a natural SDN foundation

Transcription

1 InfiniBand a natural SDN foundation October 2nd - Interop 2013 New York Dr. David Southwell, CVO

2 Scope What are supercomputers doing these days? Thesis: data centers and supercomputers are converging About InfiniBand InfiniBand and SDN ( because while it may be software defined, it's not turtles all the way down! ) 2

3 Supercomputers A device for converting a compute-bound problem into an I/O bound problem A computer costing > $1M 3

4 Supercomputers of old Small numbers of tightly-coupled processors Exotic (GaAs, ECL, liquid cooled) export controlled technologies Seats! Interconnect considerations dominated early SMP architectures 3,600 PCBs interlinked with twisted pair copper wires...80 MHz clock speed. 4

5 Supercomputer architecture transitions "If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray,

6 Clustering makes sense Economics of scale Cost reduction Higher volume (can) mean higher quality Delay or avoid resorting to exotic technologies Better understood by more users Scale out, not scale up 6

7 So clustering has dominated for a decade... Largest machines: > 3 Million cores, 20 Megawatts, 30 PFLOPS HPC workloads are very demanding on the interprocessor interconnect: High bandwidth Low latency Low jitter Low power Low CPU loading High reliability High scalability Support accelerators (MIC, GPU, FPGA) 7

8 Supercomputers are now Power Constrained Not really power, but energy (meaning $) 2007 was the crossover year; going forward server lifetime OpEx > server CapEx An ExaScale machine today would consume multiple GigaWatts (target is < 20MW by 2018) ( figure $1 Million per MegaWatt per year ) 8

9 Will CMOS scaling save us from power issues? In some respects, but the scaling physics helps data movement much less than it does processing or storage: ( wire capacitance decreases linearly with feature size, but resistance increases much faster with the inverse square of the feature size... ) 9

10 Data Centers and Supercomputer on converging paths Already many shared core components: CPUs, memory + support infrastructure (power, cooling) Many shared design metrics: power efficiency, density, scale-out, CapEx/OpEx and serviceability Internet Data Centers currently consume > 3% of global energy burn, exceeding that of all aviation. (Unlike the airline industry, the Internet is still growing exponentially!) 10

11 Old School Data Centers contrasted with Supercomputers Less Homogeneous servers and storage Lower capacity networking (typically 1 Gigabit Ethernet LAN and perhaps FC) Copper UTP LAN cabling Lack of centralized orchestration and management Many independent applications, running on a mosaic of operating systems across fragmented resources that are hard to deploy and maintain. Lack of automation results in slow, error-prone and labor intensive operations. Overall efficiency is generally very low. However, server, storage and networking virtualization is radically changing this picture......and the pipes will need to be fatter! 11

12 Fat Pipe Driver #1 - Hardware Virtualization Processors, storage and even network connections are virtualized from underlying hardware Allows fragmented OS environments to be overlaid and consolidated Ultimately, all elements may become dis-aggregated and dynamically composed on the fly Composited servers may even be elastically scaled (core count/ memory/ storage/ IO) Once many VMs have been multiplexed onto a single node, there are no gaps and the CPUs become 100% loaded. This has the effect of massively increasing the network utilization! Supercomputer-class interconnects may be required to support this extra load. The data center is morphing towards a supercomputer architecture. 12

13 Fat Pipe Driver #2 - Non-Rotating Storage Rotating disk access latencies are measured in milliseconds SSD access latencies are measured in microseconds Interconnect to non-rotating storage cannot hide its own latency in the media latency Aggregate bandwidths can also be significantly higher, especially if the SAN is a unified fabric carrying LAN traffic too. Supercomputer-class interconnects may be required to support this extra load. The data center is morphing towards a supercomputer architecture. 13

14 Fat Pipe Driver #3 Multicore and MIC CPU core clock frequency ramps have stalled Server processors are handling more throughput by integrating more cores Memory demands will be met by caches + emerging tech (edram, HMC, 3D-die stacking) Multicore and MIC have huge off-socket bandwidth demands, outstripping conventional data center networks. Supercomputer-class interconnects may be required to support this extra load. The data center is morphing towards a supercomputer architecture. 14

15 Fat Pipe Driver #4 Trends towards more East West Traffic Scale-out architecture at the rack level Unified LAN/ SAN connectivity with DAS Distributed workloads e.g. Hadoop Memcached Clustered Enterprise applications, e.g. DB East West demands lower latency switches and high bandwidths in multihop connections. Supercomputer-class interconnects may be required to support this extra load. The data center is morphing towards a supercomputer architecture. 15

16 At this point I have hopefully illustrated that: 1) Supercomputer industry has sustained exponential growth for more than two decades 2) Clusters of non-exotic processors have dominated for the last decade, and dominate today 3) Specialized interconnects are the enabling secret sauce for scalable clustering 4) Supercomputers are currently energy efficiency limited new technology evolution focus 5) Moving data is more energy intensive than processing or storing it 6) Modern highly virtualized data centers are trending towards supercomputer network demands Next, InfiniBand... 16

17 InfiniBand a proudly 'Ethernot' Technology By the late 1990s, several proprietary networking technologies had sprung up to enable more efficient server clustering (TCP/IP Ethernet was and still is - too slow). (Future IO) (ServerNet) (NGIO) The Founding Fathers (above) decided to pool their ideas and create an open standard for high performance networking the InfiniBand v1.0 standard was ratified late The InfiniBand Trade Association (IBTA) is responsible for maintaining the specification. The Open Fabrics Alliance oversees an open-sourced software stack for Linux and Windows. Notice the lack of networking companies among the founding seven... 17

18 A Green Field! 18

19 InfiniBand Some important features Links operate at 2.5, 5.0, 10.0, 14.0, 25.0 Gigabits per second per channel Links have 1, 4, 8 or 12 channels (InfiniBand delivers from Gigabits per second, full duplex on each link) InfiniBand natively operates in userland virtual address spaces (OS bypass) Host Channel Adapters (HCA) provide hardware offload for most functions (CPU bypass) InfiniBand is a memory-to-memory technology using Remote Direct Memory Access (RDMA) Links can carry RDMA traffic, low-latency send/receive primitives or encapsulated forms such as IPoIB (Internet Protocol over InfiniBand), SRP (SCSI RDMA Protocol), FCoIB InfiniBand can serve as a unified fabric, carrying IPC, LAN, management and storage over a single wire 19

20 InfiniBand Some important features QoS is implemented using Virtual Lanes (VLs) multiplexed over the physical links InfiniBand can be used to intelligently prioritize different traffic types InfiniBand links implement buffer credit based flow control, and is perfectly lossless InfiniBand is poised to displace FC as a storage interconnect (already has for supercomputers) Virtual memory, Hypervisor and SRIOV (Single Root IO Virtualization) support InfiniBand interfaces cleanly in virtualized environments with just a few percent of performance penalty! 20

21 InfiniBand Some important features The Global Unique ID (GUIDs) are IPv6-header based For speed, switches key on 16-bit software-assigned Local IDS (LIDs) InfiniBand balances scalability and performance Switches are large, fast and simple, using pre-configured Linear Forwarding Tables (LFTs) Switches do not learn; low latency and low jitter InfiniBand was designed as an SDN from the start a software entity called the Subnet Manager (SM) is hosted within each subnet (switch or server) and pre-calculates all local routing. SMA also handles runtime conditions not met by the LFTs. The SM is InfiniBand's control plane, the simple switches comprise the data plane. The protocol used to interface the control plane to the data plane is baked into the freely available InfiniBand specification. 21

22 InfiniBand why should you care? InfiniBand leads the supercomputing interconnect market (June 2013 numbers top500.org)......but it already connects more compute nodes at 10G+ than any other interconnect! The volume argument comes from an Ethernot! 22

23 InfiniBand why isn't it everywhere? (we're working on it!) InfiniBand first found traction in the supercomputer world where performance is paramount and other features are de-emphasised: Important for supercomputing: Bandwidth Latency Scalability CPU offload Lossless Flow Control Power Efficiency Less Important for supercomputing: Security Inter Subnet Routing Long Distance Connectivity Friendly + Fault Tolerant Control Plane To be deployable in mainstream Enterprise data centers, the capabilities from the right hand column must be added in to the core performance oriented InfiniBand value propositions. This has been Obsidian's focus since inception...take the training wheels off the rocket-bike! 23

24 Enterprise Grade InfiniBand Military Application Obsidian pioneered and commercialised global reach native InfiniBand range extension through its Longbow TM product series. The first customer was the Military: Large Data. InfiniBand over standard global WANs Integrated InfiniBand routing Integrated InfiniBand cryptography High Availability 24

25 Enterprise Grade InfiniBand Military Application 25

26 Enterprise Grade InfiniBand Military Application 26

27 Enterprise Grade InfiniBand Data Migration 27

28 Enterprise Grade InfiniBand Data Migration 28

29 InfiniBand + BGFC an Ideal SDN Foundation In addition to hardware programs, Obsidian has focussed on software development to round out the solution platform, starting with a complete re-imagining of the InfiniBand subnet manager / control plane BGFC with the encouragement of NASA and the DOE. BGFC is aimed at providing Enterprise Grade features and capabilities from the start, necessary for both commercial data center deployments, but also for ExaScale supercomputing. Essential BGFC functionality has already been demonstrated, and it is being developed with a view to extending it with SDN interfaces and mechanisms. Mathematically perfect LFT generation (100% deadlock free) N-way active:active cluster SM implementation InfiniBand router support (complex topologies) Performance at scale ( C++11 runtime implementation, using Python for scripting and interfaces) 29

30 InfiniBand + BGFC an Ideal SDN Foundation 30

31 InfiniBand + BGFC an Ideal SDN Foundation When fully realized: Flexible, scalable, fast, cost-effective standards based SDN platform Evolves at supercomputer industry speeds easier to upgrade metal once fully abstracted Geared for energy efficiency driving both supercomputing and data center evolution Unified fabric with QoS Security, partitioning, dynamic provisioning Highly compatible with future silicon (MIC, SSD, flat core GHz) just scale at memory speed! Compatible with existing protocols using virtual devices (IP, FC) Able to accommodate newer protocols and primitives Due to lossless extension, able to break out of data centers for metro, regional or global WANs 31

32 Connect Without CompromiseTM Thank you for your attention! 32