Cisco - Enabling High Performance Grids and Utility Computing

Transcription

1 Cisco - Enabling High Performance Grids and Utility Computing Shankar Subramanian Technical Director Storage & Server Networking Cisco Systems 1

2 Agenda InfiniBand Hardware & System Overview RDMA and Upper Layer Protocols High Performance Computing Architectures HPC Building-blocks I/O Virtualization Server Virtualization 2

3 InfiniBand Overview 3

4 What Is InfiniBand? InfiniBand is a high speed low latency technology used to interconnect servers, storage and networks within the datacenter Standards Based InfiniBand Trade Association Scalable Interconnect: 1X = 2.5Gb/s 4X = 10Gb/s 12X = 30Gb/s 4

5 InfiniBand Physics Copper and Fiber interfaces are specified Copper Up to 15m* for 4x connections Up to 10m for 12x connections Optical Initial availability via dongle solution Up to 300m with current silicon Long Haul possible, but not with current silicon * 20m in certain circumstances 5

6 InfiniBand Physics Link is bonded 2.5Gbps (1x) links Fiber is a ribbon cable Copper is a multi-conductor cable Each Link is 8b/10b encoded 4x Link is 4 2.5Gbps Physical Connections Each connection is 2Gbps data SAR provides a single 8Gbps data connection (4x) 24 Gbps (12x) 6

7 Pluggable Optics Module Transforms Powered Copper Ports to Optical Ports Coverts a copper port to an optical port on a port by port basis Extends port to port reach to 150m - 300m with fibre ribbon cables Topspin Optical Module 7

8 InfiniBand Nomenclature CPU CPU Host Interconnect Mem Cntlr HCA System Memory Server Server Host Host Host Host Host Host Host Host Host Host Host Host Host Host HCA Host Host Host Channel Adaptor Host SM - Subnet Host manager Host Host Server Server TCA Target Channel Adaptor IB Link SM IB Link IB Link TCA Ethernet link Switch IB Link TCA FC link 8

9 InfiniBand Switch Hardware Hardware switch devices is a cut-through memory switch Full-duplex, non-blocking 24 port tag forwarding switch Tags are system Local ID, provided to all network endpoints by the Master Subnet Manager on system startup 9

10 InfiniBand Host Channel Adapter Network interface for IB attached Servers Provides hardware Virtual/Physical memory mapping, Direct Memory Access (DMA), and memory protection Provides RDMA (Remote DMA) data transfer engine and reliable packet forwarding capabilities 10

11 InfiniBand Gateway Technically a Target Channel Adapter Similar to an HCA attached to an embedded device Usually doesn t require virtual memory manipulation and mapping Simplified HCA on a specialized device Examples, Ethernet to InfiniBand or Fibre Channel to InfiniBand packet forwarding engines 11

12 RDMA and Upper Layer Protocols DC _06_2005_X 12

13 Current NIC Architecture CPU CPU Host Interconnect Mem Cntlr NIC Server (Host) System Memory App Buffer OS Buffer interconnect Data traverses bus 3 times Multiple context switches robs CPU cycles from actual work Memory bandwidth and per packet interrupts limit max throughput OS manages end-to-end communications path 13

14 With RDMA and OS Bypass CPU CPU Host Interconnect Mem Cntlr HCA Server (Host) System Memory App Buffer OS Buffer interconnect Data traverses bus once, saving CPU and memory cycles Secure Memory Memory transfers with no CPU overhead PCI-X/PCI-e becomes the bottleneck for network data transmission HCA manages remote data transmission 14

15 Kernel Bypass Traditional Model Kernel Bypass Model Application Application User Kernel Sockets Layer User Kernel Sockets Layer RDMA ULP TCP/IP Transport TCP/IP Transport Driver Driver Hardware Hardware 15

16 Upper Layer Protocols Variety of software protocols to handle high speed communication over RDMA Protocols include IP-over-InfiniBand IETF SDP (Socket Direct Protocol) InfiniBand Trade Association SRP (SCSI RDMA Protocol) ANSI T10 DAPL (Direct Access Provider Library) DAT Collaborative MPI (Message Passing Interface) MPI Forum 16

17 InfiniBand Performance Measured Results Application BSD Sockets TCP IP IPoIB 1GE Async I/O extension SDP 10G IB udapl SRP MPI Direct Access Throughput 1 Gb/s 4.1Gb/s 4.5 Gb/s 7.9Gb/s 8Gb/s 8Gb/s Latency usec 30 usec 18 usec 18 usec 8 usec 3.5 usec 17

18 High Performance Computing DC _06_2005_X 18

19 High Performance Computing Applications Parallel processing applications Closely coupled Finite Element Analysis (Crash Simulation) Fluid Dynamics (Injection Molding) Loosely coupled Dataset searches (Terabyte->Petabyte datasets) Monte-Carlo simulation (10,000s of repetitions) 19

20 High Performance Computing Networks Two Standards Based Technologies Gigabit Ethernet/10 Gigabit Ethernet InfiniBand Multiple Uses HPC interconnect Storage traffic Load/Unload data movement Application/Systems management 20

21 Network Types Network style is guided by application Closely coupled applications Latency is a problem Throughput is key to resolving latency issues Loosely coupled applications Load/unload of datasets can be a key bottleneck Low latency for rapid response is critical 21

22 IB Routing for HPC Subnet Manager provides global route engine for entire IB Fabric Shortest Path First routing Round Robin load balancing Static routes 22

23 IB Routing for HPC 23

24 HPC Storage Problem As the job grows so does the storage Clusters growing to 1000 nodes or more Storage growing into the PetaByte range The storage problem Load/Unload a large dataset Get shared access to large datasets on the fly Performance is an issue: Need multi-gigabyte/s throughput 24

25 Current Solutions NFS Old standby: Doesn t scale well (single server) Current commercial cluster file systems Designed for multiple reader/writer situations Don t scale beyond 10s of nodes Don t necessarily manage the throughput problem Need to build out separate Fibre Channel fabric 25

26 HPC Storage Solutions Next Generation Cluster File-systems Based on new file-systems or modification of old (nfs or iscsi) Split the data across multiple file service hosts Either act as a RAIF (Redundant Array of Inexpensive Fileservers), or allow clients to access any one file server, while everyone shares the same back-end storage devices 26

27 InfiniBand Storage Acceleration Further accelerate storage access IB based forwarding over SDP or DAPL can provide CPU offload and increased throughput Low latency improves overall throughput and performance High bandwidth improves load/unload times dramatically 27

28 HPC Glossary Fat-tree Non-blocking switch architecture (aka CLOS) Bisectional bandwidth the total system bandwidth across the middle of the network Non-blocking full host bandwidth all-2-all communication 28

29 A New Infrastructure Category DC _06_2005_X 29

30 A New Category of Data Center Infrastructure- The Server Fabric Switch Network Switch Storage Switch Server Switch Clients Storage (SAN) Servers Network Resources (Internet, Printer, Server) Server Network Storage 30

31 What Makes The Server Fabric Switch Different? High Performance Server-to-Server Interconnect Virtualization (I/O, Storage, and CPU) RDMA High Bandwidth Low Latency InfiniBand today; PCI-Express and /or 10GigE when ready Policy-Based Dynamic Resource Mapping Shared Resources Across Entire Cluster Routing, Aggregation, Load Balancing App/OS to CPU provisioning Performance and Control 31

32 Server Fabric Switch Applications Why Performance and Control? Server Clustering High Performance Computing (HPC) Enterprise-Class HPC Database Scalability I/O Virtualization I/O Consolidation I/O Aggregation Server Consolidation Utility or Grid Computing Applications Application Provisioning Server Re-purposing Server Migration 32

33 Cisco Building Blocks (Server Fabric Switching) Switches Gateway Modules - InfiniBand to Ethernet - InfiniBand to Fibre Channel Host Channel Adapter (HCA) With upper layer protocols SRP SDP udapl MPI IPoIB Linux and Windows driver support Integrated System and Fabric management 33

34 The Cisco SFS Product Line Software Server Fabric Switch InfiniBand Multifabric VFrame Server Fabric Virtualization Software R3.0 SFS 3001 (TS90) (12) 4XIB + 1 Gw SFS 7000 (TS120) (24) 4XIB SFS 3012 (TS360) (24) 4XIB + 12 Gws SFS 7008 (TS270) (96) 4XIB (2) 2G FC GW (6) GE GW Blade Server HCA IBM BladeCenter HCA (2) 1XIB PCI-X Embedded switch (14) 1XIB (Internal) + (1) 4XIB and (1) 12XIB (External) (2) 4XIB PCI-X (2) 4XIB PCI-ex *plus InfiniBand Cables Dell 1855 HCA (2) 4XIB PCI-ex Passthru Module (10) 4XIB Remote Boot Linux Host Driver Windows Host Driver 34

35 I/O Virtualization DC _06_2005_X 35

36 The Evolution of I/O Virtualization SMP Dis-aggregation Virtualization Fibre Channel Ethernet Network Pro: Single managed entity, fast backplane Con: Expensive, Proprietary server + backplane Pro: Standard servers, inexpensive Con: Lots of managed components, lowperforming interconnect Pro: Reduced # of managed components, virtual I/O, fast standards backplane 36

37 Evolution of the Data Center Network and Storage Virtualization 37

38 Evolution of the Data Center Server Virtualization - The Server Switch 38

39 Virtual I/O for Network and Storage Unified wire-once fabric Server Cluster Single InfiniBand link for: -Storage -Network SAN Cisco SFS 3012 LAN/WAN Fibre Channel to InfiniBand gateway for storage access Two 2-Gbps Fibre Channel ports per gateway Create 10-Gbps virtual storage pipe to each server Ethernet to InfiniBand gateway for LAN access Six Gigabit Ethernet ports per gateway Create virtual GigE pipe to each server 39

40 Virtual I/O for Blade Servers Eliminating I/O Bottlenecks Server Blades Integrated 10Gbps InfiniBand switch or pass-thru module SAN Server Switch LAN/WAN Shared FC I/O across blades and racks Shared Ethernet I/O across blades and racks 40

41 Integrated InfiniBand for Blade Servers Create wire-once fabric Integrated 10Gbps InfiniBand switches provide unified wireonce fabric Optimize density, cooling, space, and cable management. Virtual I/O provides shared Ethernet and Fibre Channel ports across blades and racks Option of integrated InfiniBand switch (ex: IBM BC) or passthru module (ex: Dell 1855) IB Switch 10Gbps IB Switch 30Gbps Blade Chassis with InfiniBand Switches HCA 41

42 Virtual I/O: How it Works DC _06_2005_X 42

43 Transparent Topology Architecture IP Communications Inside Fabric: IP over InfiniBand (IPoIB) enables transparent communications for any IP-based applications. Outside Fabric: InfiniBand-to-Ethernet Gateways provide transparent access to existing IP Ethernet infrastructure. Fibre Channel Storage Inside Fabric: SCSI RDMA Protocol (SRP) enables SCSI over InfiniBand. Outside Fabric: InfiniBand-to-Fibre Channel Gateways provide transparent access to existing SAN infrastructure. Cluster Communications Inside Fabric: Remote DMA protocols for transparent performance enhancements. 43

44 The InfiniBand Driver Architecture NETWORK APPLICATION INFINIBAND SAN BSD Sockets BSD Sockets UDAPL NFS-RDMA FS API User TCP IP SDP TS SDP TS DAT FILE SYSTEM SCSI Kernel Drivers IPoIB TS API VAPI SRP FCP ETHER INFINIBAND HCA FC ETHER SWITCH ETH GW E INFINIBAND SWITCH FC GW FC SWITCH LAN/WAN UNIFIED FABRIC SAN 44

45 InfiniBand-to-Ethernet Gateway Overview Ensures seamless integration with IP-based applications. Act like L2 bridge between IB and Ethernet Bridge group is the main forwarding entity Bridge group has two bridge ports Ethernet and IPoIB Bridge group bridges one VLAN to one IB partition Ethernet bridge port can be tagged or untagged Ethernet bridge port can aggregate up to 6 ports 45

46 InfiniBand-to-Ethernet Gateway Features IP-Only protocols 802.1Q VLAN support Link aggregation IPv4 multicast support Loop protection Ethernet jumbo frames up to 9k IP fragmentation High availability 46

47 InfiniBand-to-Fibre Channel Gateway Ensures seamless integration with important SAN tools. Fabric-based Zoning LUN-based access controls Storage and host-based HA and load balancing tools Creates SAN network addresses on InfiniBand. SAN Management Tools must see each node. Creates talk-through mode with virtual WWNNs per server. Enables SAN Interoperability with InfiniBand. Appears as virtual NL-Port, N-Port, E-Port. Proven interoperability with Cisco MDS, Brocade, McData, Qlogic, EMC, IBM, Hitachi, and more. 47

48 monitoring/analysis monitoring/analysis monitoring/analysis IDC Typical SAN Today WWNN_a Sniffer Server Sniffer Server Sniffer Server Assumptions Made by SAN Mgmt Tools: Each Physical Connection is a Logical Connection Fibre Channel Fabric Each Physical Connection Has a Unique Address WWNN_b WWNN_c LUN 0 LUN 0 LUN 0 FC RAID Complex LUN n LUN n LUN n Logical Connectivity Access Between Nodes Can be Controlled by Physical Address WWNN_d WWNN_e LUN 0 LUN n LUN 0 LUN n WWNN_f LUN 0 FC RAID Complex LUN n 48

49 monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis monitoring/analysis Physical vs. Logical View Sniffer Server Sniffer Server Sniffer Server Physical View Sniffer Server Sniffer Server Sniffer Server Logical View Sniffer Server Sniffer Server Sniffer Server Sniffer Server Each server assigned unique WWNN. Appears as direct-attached N_Port. 49

50 Server Virtualization DC _06_2005_X 50

51 Three Categories of Server Virtualization Virtual Machine: Splits a servers into independent virtual servers. VMWare, XEN, MSFT Main value is higher server utilization. Virtual SMP: Combines servers together into a single managed powered entity. Virtual Iron, Qlusters Main value is scaling mission critical apps on commodity HW. Physical Server Virtualization: Makes servers stateless by moving server identity into the network, including storage and I/O subsystem. Cisco VFrame, Egenera Main value is making infrastructure change easier in heterogeneous environment. 51

52 VFrame Server Virtualization Framework Building Blocks Policy and Provisioning Services Virtualization and Boot Services Topology Transparency Ethernet (I/O) InfiniBand (RDMA) High Performance Server-Server Connectivity Fibre Channel (Storage) 52

53 VFrame Software suite that makes the Server Switch programmable Three main components VFrame Embedded System Logic Policy ingestion, interpretation, and enforcement at the server switch VFrame APIs (and SDK) Allows 3 rd party (End-user Customers, Software Partners, System Vendor OEMs) management and provision tools to program and manage the server switch fabric Software Partners VFrame Director Software package disseminates policies to server switch fabric Central policy enforcement provides better system wide decision making and conflict arbitration Can be installed on any server in the network 53

54 Programmability VFrame 1) Server Switch receives policy from VFrame Director or 3 rd party software. Policy Virtual Server 2) Based on policy, Server Switch assembles the virtual server vip vip vhba vhba vhba Selects server(s) that meet minimum criteria (e.g. CPU, memory) Cisco SFS 3012 Boot server(s) over the network with appropriate app/os image Creates virtual IPs in servers and maps to VLANs for client access. Creates virtual HBAs in servers and maps to Zones, LUNs, and WWNNs for storage access CPUs LAN SAN 54

55 Case Study: Large Wall Street Bank Enterprise Grid Computing Application: Replace proprietary platforms with standards-based components Build scalable on-demand compute grid for financial applications Benefits: 20X Price/Performance Improvement over four years 30-50% Application Performance Improvement Standards-based solution for ondemand computing Environment that scales using 500- node building blocks Centralized shared I/O pool for flexibly allocating SAN/IP bandwidth Existing N/W Grid I/O Core Fabric Edge Storage LAN 44x 24-port 512 Nodes 12 hosts 55

56 Additional Content Cisco Press Cisco Press 56

57 Q and A DC _06_2005_X 57

58 DC _06_2005_X 58