Unification of Data Center Fabrics with Ethernet

Transcription

1 Unification of Data Center Fabrics with Ethernet A High Speed Peek into the Future! Matthew Brisse, Dell Inc.

2 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee Storage Networking Industry Association. All Rights Reserved. 2

3 Abstract Unification of Data Center Fabrics with Ethernet: A High Speed Peek into the Future! A key challenge that the network and storage industries are now facing is the abundance of new high speed interconnect protocols proposed for future data center applications. In this presentation we take a peek into what the future may hold for high speed fabrics and investigate the potential for their unification. We will provide a market and technical overview of the competitive landscape for next generation 10Gb technologies with particular focus on the operational characteristics and implementation aspects of Ethernet Storage Networking Industry Association. All Rights Reserved. 3

4 10GE Technology 10GE is 10 Gbps Ethernet It has four different types of physical layer interfaces 1m: Backplane (802.3ap), XAUI (KX4) 2.65GB/lane, 4 lane = 10GB KR (Serial) 10 GB 1 lane 100m to 10km: Optical (SR, LR, LRM, ER) 15m: CX-4 (Infiniband Type Cable-XAUI) m 10GBase-T Copper (802.3an), over CAT6, CAT6a or CAT7 Offload MUST be a part of any 10GE solution A dumb-only 10GE NIC is of limited value 10GE readily consumes compute and memory resources (unless offloaded) Less latency and greater bandwidth (both are important) 10GE is more than just another, interchangeable physical interface Offload and system characteristics are key to plumbing 10GE properly 2007 Storage Networking Industry Association. All Rights Reserved. 4

5 Copper Cable Options Category Type Spectral B/W Length Applicatio ns Notes Cat5 UTP 100MHz 100m 100Base-T Common Cat5e UTP 100MHz 100m 1000Base-T Common Cat6 UTP 250MHz 55m 10GBase-T Emerging Cat6a UTP 400MHz 100m 10GBase-T Emerging Cat7 ScTP 600MHz 100m 10GBase-T Emerging Alien Crosstalk? RJ45 RJ45 RJ45 RJ45 $1.25/ft $0.65/ft $0.15/ft $0.10/ft $0.05/ft CAT7 CAT6a CAT6 CAT5e CAT5 Longitudinal spacer Longitudinal spiral spacer Longitudinal spacer 4 twisted pairs 2 twisted pairs Wrapped pairs, Shielded SHIELDED UNSHIELDED (100m) 1GE (100m) (100m) (55m) 10GE RJ45 Challenge! 2007 Storage Networking Industry Association. All Rights Reserved. 5

6 PCIe and 10GE PCI Express Generation 2 will be required to support full 10GE with x8 (Current 10GE NICs can t realize full throughput due to PCIe Gen 1 bus speed limitations but PCIe Gen 2 will realize the maximum throughput) Server Server PCIe, Gen1 x8 PCIe, Gen2 x8 40 Gbps, bi-directional datarate 16 Gbps, effective bi-directional throughput 80 Gbps, bi-directional datarate 32 Gbps, effective bi-directional throughput 10GE 10GE 20Gbps bi-directional 20Gbps bi-directional 2007 Storage Networking Industry Association. All Rights Reserved. Datarate: Bits on the wire 6 Throughput: Information on the wire

7 10 GE Requires Multi-Function Offload Capabilities Simple NICs offload nothing Stateless offload accelerates packet processing (RSS, LSO, TSO, Checksum) Stateful offload provides protocol and packet processing (TOE, Chimney, RDMA, iwarp) RDMA abstracts the underlying transport and achieves direct memory to memory data placement Virtualization mediates access to multiple instantiations of limited hardware resources VIRTUALIZATION Lower CPU usage Greater I/O efficiencies Higher network throughput TOE RDMA (iwarp) RDMA over TCP/IP Protocol Acceleration 10 GE Viability No Off Load ACCELERATION Check Sum Off Load 1 GE Viability NIC Now Storage Networking Industry Association. All Rights Reserved. 7

8 10GE Overall Technology Comparison 10GE is an acceptable technology for fabric convergence IB and 10GE bandwidth both have visibility to 100GE FC will continue to have a place for a considerable time GE latency will settle at < 10 microseconds end-to-end Cost will come down Economy of scale UNIFIED FABRIC SWEET SPOT 400 EN Fast, some latency good enough FC IB Faster, less latency, storage Fastest, less latency, cluster + storage was the fabric convergence in the past SPEED (Gbps) LATENCY (End-to-End) (µsec) COST (NIC/HBA) 2007 Storage Networking Industry Association. All Rights Reserved. 8 ($) POWER (SR MAC/PHY) (W)

9 10GE non-technology Comparison FC and IB will both perform better than 10GE in cluster and storage fabrics, respectively 10GE will, however, be good enough to potentially gain market share from IB and FC Non technical factors will drive 10 GE adoption and continued evolution Ethernet trust, ubiquity and robust investment climate are important driving factors Ethernet always incorporates the best from competing technologies EtherNOT becomes EtherNET HI HI HI HI EN FC LO LO MED LO MED IB TRUST UBIQUITY INVESTMENT Well-known protocols Ethernet everywhere More $ to Ethernet Well-known management More Ethernet R&D Staying power 2007 Storage Networking Industry Association. All Rights Reserved. 9

10 Unified Fabric A peek in the future Reduce the number of disparate networks and Data Center complexity Provide a seamless model for networking, clustering and storage applications Not to mention thin client, KVM and other usages Consolidate Management protocols over a common IP framework Leverage economies of scale in terms of silicon integration, interoperability and product development Provide support for legacy network architectures in the transition to a unified fabric architecture 2007 Storage Networking Industry Association. All Rights Reserved. 10

11 Unified Fabric Vision Making Many Fabrics LAN HPC COMPATIBLE EN LOW CPU UTILIZATION IB SCALABLE NAS HIGH THROUGHPUT BLOCK LOW TRANSFERS LATENCY FC HIGH TRANSACTION RATE SAN 2007 Storage Networking Industry Association. All Rights Reserved. 11

12 Unified Fabric Vision Making Many Fabrics One LAN COMPATIBLE HIGH THROUGHPUT NAS LOW CPU UTILIZATION 10GE BLOCK TRANSFERS SCALABLE LOW LATENCY HPC HIGH TRANSACTION RATE SAN Simplifying 2007 Storage Networking Industry Association. All Rights Reserved. 12

13 Emerging Trends Clustering Improves cost-performance over large scale cache coherent systems Brings flexibility to the data center Provides better fault tolerance Grid Academic processing Grids tried and tested Commercial deployment in niche areas only Storage Grids now in discussion Object Storage Completing standardisation New paradigm of intelligent storage Requires converged compute & storage platform Systems Management Keyboard Video & Mouse (KVM) already available and other tools are coming System Level Virtualization Ethernet is rapidly becoming the key to enterprise systems convergence 2007 Storage Networking Industry Association. All Rights Reserved. 13

14 Industry Initiatives Blurring of compute and storage is now happening Traditional view of totally discrete functions is getting old New applications require dynamic configurations Lightweight blades clustered with shared access to I/O resources are enabling convergence Systems Virtualization Protected virtual memory architectures return to give scale up virtualisation Scale out virtualization still requires major developments in the industry but everyone expects it to happen Server re-purposing paradigm maturing The 3 tier Data center model is now passing its sell by date Evolving Ethernet enables system wide convergence 2007 Storage Networking Industry Association. All Rights Reserved. 14

15 Target 10GE Applications 10GE provides the basis for a unified datacenter fabric Servers HPC server-to-server interconnect Virtualization increased bandwidth High-Performance Transactions financial applications IPTV streaming media applications Digital Cinema web farms Switches Edge Layer server-to-switch uplink iscsi storage array to switch Aggregation & Core Layer switch-to-switch interconnect iscsi Storage Arrays General Purpose array-to-switch connectivity HPC 10GE UNIFIED FABRIC SAN LAN Single Adapter Multiple Offloads Common Management Simplicity 2007 Storage Networking Industry Association. All Rights Reserved. 15

16 Enterprise Network Enterprise Network Today Today Clients Web Servers Application Servers Database Cluster Storage Array Units DAS SAN Load Balancers Switches RAID Storage DAS IP/Ethernet Storage Cluster Management RAID Storage 2007 Storage Networking Industry Association. All Rights Reserved. 16

17 Converged Network Converged Network Clients Web Servers Application Servers Database Cluster Storage Array Units DAS SAN Load Balancers Switches RAID Storage DAS IP/Ethernet Storage Cluster Management RAID Storage 2007 Storage Networking Industry Association. All Rights Reserved. 17

18 Unified Fabric Example Server Blades Server Blade Network Switch Storage Switch Cluster Switch Management Switch Ethernet Ethernet Backup Networking Switch Compute Blades Backup Storage Switch Backup Clustering Switch Management Switch Clustering Switch MGT CPU FC Ethernet Ethernet Storage Switch E-net IB Networking Switch Local Hard Disk Mid-plane 2007 Storage Networking Industry Association. All Rights Reserved. 18

19 Unified Fabric Example Server Blades Server Blade Network Switch Storage Switch Cluster Switch Management Switch New Ethernet Technologies Compute Blades Ethernet Ethernet iscsi Remote Direct Memory Access (RDMA) MGT CPU FC Ethernet Ethernet Integrated Management E-net IB Accelerated Networking (TOE) Local Hard Disk Mid-plane 2007 Storage Networking Industry Association. All Rights Reserved. 19

20 10GE Switch Market Growth 10GE growth ( ) driven by adoption of low-cost 10GBASE-T copper 1GE will be pushed to the edge of LANs and must be aggregated with 10GE 10GE will be initially positioned for inter-switch connections and clusters Vendors drive low-cost 10GE, accelerating adoption resulting from customer demand 20 Compound Annual Growth Rate CY03 CY Mbps 1 Gbps Revenue ($B) % 10% 10 Gbps 0 CY03 CY04 CY05 CY06 CY07 CY08 CY09 Source: IDC Storage Networking Industry Association. All Rights Reserved %

21 10GE Market 5x port growth from 2005 to 2006; 10m ports, $4B market in Fiber Copper Available market data represents growth in the 10GE switch port market, not the NIC or other 10GE markets. The growth in 10GE infrastructure, however, implies growth in other 10GE markets, including server port shipments Year of copper 10W today, 5W power W down on the Motherboard 000 s switch ports Copper Fiber Total % Copper % 4% 8% 14% 29% 39% 42% 2007 Storage Networking Industry Association. All Rights Reserved. Source: Dell Oro Group January ,616 1,871 1,013 2,511 3,524 2,509 4,008 6,517 4,209 5,748 9,958

22 Cluster Fabric Trends INFINIBAND MYRINET OTHER I0GE ETHERNET 1 GE 1 GE Top500.org, Nov 2005 Projection, Storage Networking Industry Association. All Rights Reserved. 22

23 NIC 10GE Cost Trends 1GE vs. 10GE RELATIVE PRICE TARGET 10x Relative Cost (1GE Cy04 = 1.0) 8x 6x 4x 2x ~$100/port 10GE NICs 10 GE 3-4x 1 GE for wide adoption 10GE 0 1 GE Storage Networking Industry Association. All Rights Reserved. 23

24 10GbE Standards 802.3ae Media Access Controller (MAC) Full-Duplex 10 Gigabit Media Independent Interface (XGMII) or 10 Gigabit Attachment Unit Interface (XAUI) LAN PHY (8B/10B) Serial LAN PHY (64B/66B) Serial WAN PHY (64B/66B + WIS) 10G BaseT 10G Backplane Various Optical Technologies Cat6/7 Cable 802.3an 802.3ap 802.3an and 802.3ap make 10GbE even more compelling for data center applications Layer Storage Networking Industry Association. All Rights Reserved. 24

25 10GE Copper/Backplane Standards and Opportunity Timeline BARRIERS TO OVERCOME LOM CONCEPT and DESIGN BLADE EMERGING SOLUTIONS NIC MATURE PRODUCTS SWITCH CY05 CY06 CY07 CY08 CY09 CY an ratified 802.3ap ratified Unification of Data Center (10GBASE-T) Fabrics with Ethernet: A High Speed (Backplane) Peek into the Future! 2007 Storage Networking Industry Association. All Rights Reserved. 25

26 Changes with 10GbE 802.3ae defines 10GbE 10Gbps Data Rates Full-duplex only; no need for carrier-sensing multiaccess / collision detection (CSMA/CD) Optical Physical Layer LAN PHY and WAN PHY options WAN PHY compatible with SONET 802.3an adds twisted pair cabling Cat 6 and Cat ap adds backplane specifications Blade servers and communications equipment 2007 Storage Networking Industry Association. All Rights Reserved. 26

27 IEEE 10GE Standards IEEE P802.3ae, 10GE Working Group Definition of the 10GE MAC architecture and components IEEE P802.3an, 10GBASE-T Task Force. Definition of 10GE encoding over CAT 6/7 (Copper) cable IEEE P802.3ap, Backplane Ethernet Task Force. Definition of 1/10GE encoding over backplanes (KX, KX4, KR) IEEE P802.3aq, 10GBASE-LRM Task Force. Definition of short haul MMF/1310nm LR encoding IEEE P802.3ar, Congestion Management Task Force. Definition of L2 congestion discovery and avoidance protocols IEEE P802.3as, Frame Expansion Task Force. Not Jumbo frames; expansion of frame envelope, not data size IEEE P802.3at, DTE Power Enhancements Task Force. Definition of how to deliver power at 40-60W over 1 and 10GE 2007 Storage Networking Industry Association. All Rights Reserved. 27

28 Unified Fabric End Goal Innovation SIMPLE EFFICIENT SECURE Hardware NIC/HBA PHY SWITCH MAC VIO UNIFIED FABRIC 10-Gigabit Ethernet Software SECURITY OS MGMT APP VOS Standards IEEE DMTF IETF SNIA STANDARDS SIMPLICITY PARTNERS COORDINATION CONSOLIDATION SNIA Standards LEADERSHIP 2007 Storage Networking Industry Association. All Rights Reserved. 28

29 Call to Action Get involved!! Participate in Standards Drive your vendors Leverage the power of the user Ethernet vendors are developing new protocol acceleration features and the speed is increasing from 1G to 10G, Understand the future impact 10Gb Ethernet with protocol acceleration could be used a 'Unified fabric' to handle SAN, HPC, management and client network needs in addition to traditional data networking requirements. Be prepared for the transition! Plan today! 2007 Storage Networking Industry Association. All Rights Reserved. 29

30 Q&A / Feedback Please send any questions or comments on this presentation to SNIA: trackapplications@snia.org Many thanks to the following individuals for their contributions to this tutorial. SNIA Education Committee Allen Light Matt Brisse Robert Winter Robert Peglar 2007 Storage Networking Industry Association. All Rights Reserved. 30

31 Questions?