Optimization of Virtualized, Converged Data Center Networks

Transcription

1 Renato Recio, DE, IBM Data Center Networking CTO DYNAMIC INFRASTRUCTURE Helping build a smarter planet Optimization of Virtualized, Converged Data Center Networks

2 Legal Slide Trademarks and disclaimers Dynamic Infrastructure: Helping build a smarter planet 8 IBM Corporation All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography. Photographs shown may be engineering prototypes. Changes may be incorporated in production models.

3 Today s Physical Data Center Infrastructure Analogy $ $ $ Traversing today s Data Center Infrastructure is analogous to going through a long toll road, using only the Cash lanes... Each toll plaza requires a stop where a queue forms, as each car pays the toll The end result... High latency caused by long queues per stop & many stops. High staffing cost associated with manning the toll plaza. We are simplifying DCNs, by working with network partners to: automate the process (analogous to an E-ZPass ), flatten the path (analogous to short-cuts).

4 Today s Non-Virtualized DataCenter Network High OpEx due to: sprawl of demilitarized Zone appliances >10 μsec >20 μsec >30 μsec High CapEx & OpEx to support each server, due to many networking layers. High latency due to many layers, which repeat processing steps per layer. >2 μsec High OpEx due to many network, storage & server managers High CapEx & OpEx due to lightly utilized servers moving VMs (to increase utilization) is limited by low bandwidth links & manual management issues. High CapEx & OpEx due to dual fabrics (Ethernet and FC), using lower bandwidth links

5 The Data Center is Undergoing Transition Application Web Database 1 Gbps Ethernet 4 Gbps Fibre Channel 10 Gbps Ethernet or Converged Ethernet 8 Gbps Fibre Channel Virtual Storage From a physical compute model that has: VMs held captive by low bandwidth links & manual network configuration Few Virtual Machines (VMs) per server Using lower bandwidth links Sprawl of manually intensive, expert based IT management tools Poor image life-cycle management tools Physical appliance sprawl Expense issues with this model: High capital expenses due to under utilized servers and multiple fabrics High operational expenses due to manual administration of many management tools. To a virtualized, Dynamic Infrastructure model that: Has simpler, integrated & virtualized appliances and systems Has many highly utilized servers, each with many VMs; using virtual storage, networks & appliances Is enabled by high bandwidth links and migrating to virtual & converged fabrics Includes new, faster time to value Cloud Building Block (CBB) Multi-Rack acquisitions The value of this new model is: Lower capital expenses through higher server, storage and network utilization. Lower operational expenses through automated & integrated management that optimizes and automates Data Center infrastructure and delivers self configured services to the business.

6 It s time to start thinking differently about Data Center infrastructure. Scale, cloud based server integration, integrate the management, automate virtualization, converge the network, and flatten the infrastructure.

7 Integration Trends Integrated management One interface for managing all physical and logical resources idataplex Cloud Building Block (CBB) based Acquisition (Software, server, storage & network) Manager Switch Switch Manager Blade Servers (with network) Chassis Switches Storage CBB based integration Integrated, multi-rack server, switch, storage and virtualization solution. Ships with single CBB manager. BladeCenter Rack based integration Integrates servers, switch, manager appliance, storage High density rack (100 servers) ships complete, ready to deploy Blade Chassis based integration Integrates servers, switch, storage Higher density than modular Over 3x cable reduction Improved RAS (less cables to bump)

8 Cloud Building Block Solution Example IBM CloudBurst A service delivery platform that is pre-integrated at the factory. Prepackaged, pre-configured servers, storage, networking, software and installation services needed to provide a private cloud Built-for-purpose based on the architectural requirement of specific workloads Delivered and supported as a single product GTS Quickstart Services: CloudBurst provides rapid implementation of a private cloud environment consisting of the cloud management infrastructure, including the cloud resources to be provisioned Initial install, configuration and training services Pre-engagement workshop for initial requirements Post-engagement workshop for growth Base Hardware Configuration*: 1 42U rack M2 Mgmt Server, 8 cores, 32GB Ram 1 HS22 Blades, 8 cores, 48GB RAM Base configuration: 1x BladeCenter chassis 3 managed H22 blades, 8 cores, 48GB RAM DS3400 FC attached storage, 5.4TB raw capacity *Base HW configuration can be scaled out per blade server with up to two chassis and 28 blades. Cloud Software Configuration: BladeCenter Foundation stack runs on 3650M2 management server Windows 2003R2 Enterprise Systems Director with BOFM, AEM; ToolsCenter 1.0; DS Storage Mgr for DS4000 v10.36; VMware VirtualCenter 2.5 U4; LSI SMI-S provider for DS3400 VMware ESXi 3.5 U4 hypervisor on all blades Thundercloud software stack will ship as VMware images to run on the HS22 management blade BlueCloud v1.6.1 including appliance wizards Tivoli Provisioning Manager v7.1 DB2 ESE 9.1; WAS ND ; TDS Tivoli Monitoring v6.2.1 OS pack SUSE Enterprise Linux 10 sp2

9 It s time to start thinking differently about Data Center infrastructure. Scale, cloud based server integration, integrate the management, automate virtualization, converge the network, and flatten the infrastructure.

10 Today s Non-Virtualized DataCenter Network Multiple Platform Managers Server VMs Ethernet Fibre Channel Integrated Platform Manager Server VMs Ethernet, CEE, Fibre Channel, FCoE

11 Data Center Management Components Service management Integrated visibility, control & automation across heterogeneous business and technology assets Align IT operations with the business Govern and control the business Optimize the business Detailed platform management Consolidated management across systems Tell me what I have and if it s working Let me install, configure and update Integrated physical and virtual management Servers, storage & network Automated physical and virtual provisioning

12 IBM Data Center Management Automated, Integrated, Flexible, Provisioned by Request Compute Node Compute Node Compute Node Compute Node Access Switch Hypervisor SMP Hardware System Compute Node Compute Node Compute Node Compute Node Access Switch Datacenter Fabric Virtual I/O Server (VIOS) Shared I/O Mgt Node Storage Storage Storage Access Switch Tivoli Self Service Application Tivoli Provisioning Manager Tivoli Storage Productivity Center Tivoli Monitoring VMControl Network Manager IBM Systems Director Application Workflow Control Image libraries Config files User Request Control Consoles End user requests Service center Storage & Network Management Network Integration Storage Integration Automated Relocation Virtual Systems Management VM Lifecycle (Create, deploy, relocate & maintain VMs) Image Lifecycle (Capture, import, create, edit and delete virtual images) Platform Management Discovery, Inventory and Monitoring of System Resources Active Energy Controls

13 It s time to start thinking differently about Data Center infrastructure. Scale, cloud based server integration, integrate the management, automate virtualization, converge the network, and flatten the infrastructure.

14 Today s Virtual Server Integration Virtual IO Server (VIOS) Driver PCIe Port F Driver PCIe Adapter Enet Port Linux Drv Virtual Switch AIX Drv Hypervisor Hypervisor Bypass, eliminates overhead Virtual Switch Bandwidth Latency Drv AIX Drv Port Host Ethernet Adapter VEB Enet Port In External In Adapter Switch 4 8 GB/s 1 GB/s < 1 us 2 us System virtualization technology offers: Many OSs / server up to 254 for PowerVM Integrated virtualization manager Application & image mobility High resource utilization Virtual and dedicated IO Image quality of service Dynamic resizing High availability Today s virtual IO technology (e.g. Power) Native IO Virtualization (IOV) IO directly shared by adapter hardware. Adds Virtual Ethernet Bridge to adapter. IO shared through Virtualization Intermediary (e.g. VIOS on Power). All IO is performed through VI. Adds overhead to every IO operation.

15 Server Virtual Switching Trends VMs System virtualization, with integrated virtual switches, has been around for decades. However, managing virtual switches is too manual today and, as covered earlier, scale-out sprawl magnifies this complexity. Virtual switching is evolving into two use cases: 1. Virtualized systems or clusters that exploit up & VM trends: more cores more VMs more VM-VM networking. Attributes: Hypervisor Internal virtual switching in hypervisor or PCIe adapter (with Single-Root IOV). Highest throughput and lowest latency. Virtual switch management integrated with server management. Virtual NIC s port profile (e.g. VLAN) automatically migrates with VM. Server based access, QoS controls and virtual security appliances. 2. Multi-tier, enterprise DC environments where state of the art network access and traffic controls are more important than VM-VM performance. External network performs virtual switching, using advanced controls & network security. Less switches (i.e. no server switches) to manage under common management tool. Emerging automated port profile migration mechanism PCIe PCIe External Switch

16 Automating Port Profile Management for Network based VS VM manager VM BPV IO B. VM Hypervisor IO BPV VM We are working with partners and the industry on a port profile automation mechanism that: Selects where Virtual Switching is to be done (i.e. server or network). Automates port profile migration for both cases. The approach allows automated migration of a VM s external network profile on activation and during VM migration, by: 1. Using management plane protocols to: A. create/destroy virtual switch port profiles; and B. communicate port profile to VM associations to the Hypervisor. 2. Using a control plane protocol to: C. associate a port profile with a specific VM s MAC Address; and D. de-associate & migrate that profile when the VM migrates. C. Hyp. Switch / Port Pr. Mgr Associate VM MAC Address Port Profile ID Hyp. Switch / Port Pr. Mgr Associated VM MAC Address Port Profile ID Switch Port Profile Manager A. Hypervisor & PCI SR-IOV adapters implement BPV (Bridge Port Virtualizer) protocols to associate virtual port profiles. Simplifies VM mobility by mobilizing network profile thru control plane Routes VM-VM traffic through external switch

17 It s time to start thinking differently about Data Center infrastructure. Scale, cloud based server integration, integrate the management, automate virtualization, converge the network, and flatten the infrastructure.

18 Fabric Convergence Options iscsi/nas storage for middle tier servers IB FC High performance clusters Enet IB: GB/s and 150 <100 ns Lack native storage & use slow gateways Scale-out File System (SoFS) Clustered File System with Global Namespace Centrally managed Scales horizontally Integrated Information Lifecycle Mgt, HA & disaster recovery SMB storage SMB and Mid-tier Storage Low $/Gbps (GigE) Growing and maturing, considering 10 GigE Fibre Channel over Convergence Enhanced Ethernet Ethernet LAN Fibre Channel SAN Enet TORS TOR Converged switch Enables 50% CapEx reduction within the chassis (rack or BC). Lower power Improved RAS Plugs into existing DC infrastructure

19 Fibre Channel over Ethernet Dynamic Infrastructure: Helping build a smarter planet FC over Ethernet Layers FC frames directly over Ethernet (requires jumbo/mini-jumbo) Ethernet Frame FCoE Encapsulation FC Packet Replacing lowest level of FC with Ethernet T11 FC-BB-5 standard (Fibre Channel standards group). FC over Ethernet requires convergence enhancements to Ethernet: To carry maximum sized FC packets Requires Ethernet Jumbo (or mini-jumbo) frames To provide FC like no-drop behavior in face of congestion Requires new Priority-based Flow Control protocol To control traffic interferences: Requires new Enhanced Transmission Selection protocol To detect both ends of the wire support the above Requires new Data Center Bridging exchange protocol CEE Authors was formed to facilitate/accelerate definition & adoption of above. CEE Authors submitted proposals for above protocols to IEEE

20 Fabric Convergence Value Proposition vs Barriers Servers Multiple Fabrics One Fabric Value Proposition Barriers Lower Cost, Lower Power Less adapters, cables and switches Improved RAS Reduced failure points, time, misconnections, bumping, Simpler Management Single physical fabric to manage. Customer organizational Server, Network, Storage silos Operational management Quality of service control Security and access control Maturity Standards, technology, management Resiliency and robustness

21 Example DCN Use Case Steps Case 1: Upgrade Case 2: Dual fabric CBB Case 3: Single fabric CBB Ethernet LAN Rack Blade Ethernet LAN Row of Modular Servers Fibre Channel (FC) SAN FC SAN Row of Modular Servers Ethernet LAN Large SMPs FC SAN Converged access switch attaches to FC and Ethernet Enables 50% CapEx reduction within the chassis (Power, modular or BC): Adapters, Cables and Access Switches. Improved RAS (fewer elements) Lower power ~ 2570 ~1256 W/hr per rack Simplified management Director with integrated network management Converged access switch attaches to FC and Ethernet Enables 50% CapEx reduction in networking per row of servers Adapters, Cables and Access Switches. Improved RAS (fewer elements) Lower power ~36 ~19 KW/hr per row Simplified management Director with integrated network management Note: Power estimates are subject to change. Fully converged fabric, with FC attached storage Enables 50% CapEx reduction in networking at data center level Eliminates all FC switches Lower power ~42 ~21 KW/hr per row Improved RAS Single DCN Manager

22 IBM FCoCEE Networking Solution Description 6/09 6/09 Brocade CNA x86 Systems 5/09 Base QLogic CNA Mid-09: FCoCEE use case 1 & 2 on x86 with BladeCenter enablement Converged Network Adapters (CNAs) for x86 & BladeCenter Converged TOR Switch (Brocade & Cisco) BladeCenter CEE Pass-thru IBM System Director with base network manager x86 Systems BladeCenter TOR Switch Ethernet Fabric Brocade 5/09 Brocade Cisco 2008 Fibre Channel Fabric

23 Fabric Convergence Options iscsi/nas storage for middle tier servers IB FC High performance clusters Enet IB: GB/s and 150 <100 ns Lack native storage & use slow gateways Scale-out File System (SoFS) Clustered File System with Global Namespace Centrally managed Scales horizontally Integrated Information Lifecycle Mgt, HA & disaster recovery SMB storage SMB and Mid-tier Storage Low $/Gbps (GigE) Growing and maturing, considering 10 GigE Fibre Channel over Convergence Enhanced Ethernet Ethernet LAN Fibre Channel SAN Enet TORS TOR Converged switch Enables 50% CapEx reduction within the chassis (rack or BC). Lower power Improved RAS Plugs into existing DC infrastructure

24 How does FCoCEE Compare? Key Market Segments Targeted Performance HPC & Analytics SMB and Mid-tier servers Data Center upgrades InfiniBand iscsi / NAS = SoFS SoNAS = FCoCEE = = Maturity - Standard hardware available when? - Technology, management s Storage attachment - Enterprise data tier functions* - Bridges to FC fabrics - Native storage attachment Management simplification Security Investment (HW, tools, skills) protection Price/Performance (compared to FC) = = *For example: synchronous remote-site switch-over; etc????

25 Projected FCoCEE Volume Adoption Factors for faster adoption FCoCEE uses FC Layers & Services Value of capital expense reduction Ability to plug-into existing DC Late Majority (FICON) Use Case 3 (Single fabric Cloud Cell) Use Case 2 (Dual fabric Cloud Cell) Use Case 1 (FCoCEE server upgrade) Early Adopters HPC and Analytics Factors for slower adoption Customer organizational issues Management robustness FCoCEE maturity Current economic realities Note: These are early estimates and subject to change.

26 Flattening Today s DCN Internet Edge Routers Access Convergence - Eliminates FC switches - Eliminates FC adapters Core Layer-3 Aggregation DMZ Appliances Layer-2 Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

27 Flattening Today s DCN Internet Edge Routers Access Convergence - Eliminates FC switches - Eliminates FC adapters Core Layer-3 Aggregation DMZ Appliances Layer-2 Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

28 Fabric Convergence Conclusion No single solution satisfies all requirements. IB fits when performance is critical. NAS and iscsi fit well in SMB and middle-tier servers. SoFS & SoNAS fit well in scale-out environments. As FCoCEE matures it will play well in FC based enterprises seeking convergence.

29 It s time to start thinking differently about Data Center infrastructure. Scale, cloud based server integration, integrate the management, automate virtualization, converge the network, and flatten the infrastructure.

30 Flattening Today s DCN Internet Core Edge Routers Core & Edge Integration - Integrates Core switches with Routers - Consolidate Appliances Aggregation Layer-2 Layer-3 DMZ Appliances Access TOR Switches.... x86 servers x86 servers Web Tier Application Tier Database Tier Storage fabric

31 Flattening Today s DCN Internet Core Edge Core & Edge Integration - Integrates Core switches with Routers - Consolidate Appliances Aggregation Layer-2 Layer-3 DMZ Appliances Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

32 2009 Ethernet Product Content One Manager, One OS Deliver a complete DCN switching solution, that begins to flatten the network. NI MLX, CES FI SX, GS BI RX One Manager, One OS, One Release Brocade networking: 4/28 announced two switch families: NetIron with layer-3 functions (e.g. MPLS/VPLS) that enable Data Center core and router integration FastIron with Layer-2 functions for small/medium clusters. Working with Juniper on OEM products IBM announced an original equipment manufacturing (OEM) agreement with Juniper Networks. IBM will rebrand and sell selected Juniper EX and MX switches and routers. Availability is expected later this year. This expands upon IBM and Juniper's long-term relationship that includes a reseller agreement with IBM Global Technology Services, as well as collaboration on Juniper's Stratus Project, & IBM's 10 worldwide Cloud Labs.

33 Flattening Today s DCN Internet Aggregation Layer-2 Layer-3 Core Edge DMZ Appliances Layer-2 Consolidation - Consolidates lower performing Layer-2 Switches & eliminates Spanning Tree (e.g. Using TRILL and/or Virtual Switching) Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

34 Flattening Today s DCN Internet Aggregation Layer-2 Layer-3 Core Edge DMZ Appliances Layer-2 Consolidation - Consolidates lower performing Layer-2 Switches & eliminates Spanning Tree (e.g. Using TRILL and/or Virtual Switching) Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

35 Flattening Today s DCN Internet Edge SAN Convergence - Converges SAN & LAN core switches Core Layer-3 Aggregation DMZ Appliances Layer-2 Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier Storage fabric

36 Flattening Today s DCN Internet Edge SAN Convergence - Converges SAN & LAN core switches Core Layer-3 Aggregation DMZ Appliances Layer-2 Access TOR Switches.... Web Tier x86 servers Application Tier x86 servers Database Tier

37 Flattening Today s DCN Internet Edge Virtualize the Network - Simplifies the management - Further cost reduction Layer-3 Core DMZ Appliances Virtual Core Virtual Storage Aggregation Layer-2 Access.... Web Tier Application Tier Database Tier

38 Data Center Network Flattening Summary Router N layer 2 layer 1 layer Core Aggregation Access.. Virtual Core Virtual Storage Storage fabric Virtual Servers Technology evolution allows for creation of single layer Data Center Network. Convergence Enhanced Ethernet enables elimination of a dedicated storage fabric New networking technologies (higher bandwidth links, switch stacking, switch virtualization and layer-2 multi-pathing) enable flatter networks. Integrated appliances reduce the number of stand-alone boxes that must be managed. Convergence drives (IP/Ethernet & FC) network management integration and virtualization enhancements drive server and network management integration.

39 Data Center Networking Simplification Summary Scalable, cloud based system integration Enables lower CapEx & OpEx Cloud based, integrated solutions: servers, network, storage, virtualization, software, platform & service management for: scale-up (large SMPs) & scale-out servers. Automated virtualization Enables lower CapEx & OpEx Automated virtual system life cycle management, includes automated network and storage identity orchestration. Integrated management Enables lower OpEx Simplified, integrated management suite that covers servers, network, storage, virtualization and service management; and thereby enables lower OpEx. Converged network Enables increasingly lower CapEx & OpEx with each step Evolutionary network convergence approach. Flattened infrastructure Enables lower CapEx & OpEx Leverage advancements in layer-2 switching and virtualization technologies to collapse DC networking layers.

40 IBM DC Products & Partners in Support of Dynamic Infrastructure Dynamic Infrastructure: Helping build a smarter planet IBM Provides Extensive, Optimized, Robust DC Infrastructure Extreme virtualization with massive scalability and advanced security IBM System z IBM Power Systems Dynamic resource allocation with advanced mobility for AIX, Linux & IBM i OS Highly scalable storage virtualization for SAN and NAS environments IBM System Storage IBM Systems Director IBM System x More workloads consolidated on fewer servers with exceptional reliability High performance links connecting to virtual and converged networks IBM Data Center Switching IBM BladeCenter Highly integrated, fully redundant, energy efficient with advanced I/O Manage your entire datacenter, physical and virtual, from one place Additionally, Tivoli manages multi-vendor elements.