NetApp s Global Engineering Cloud: The Journey to Nexus 9k and ACI

Transcription

1

2 NetApp s Global Engineering Cloud: The Journey to Nexus 9k and ACI Michael McKee - Senior Network Architect Michael Torelli Senior Network Architect CCSACI-2552

3 Agenda Introduction FlexPod with Nexus 9K Network Design Nexus 9000 NX-OS Design Nexus 9000 ACI Design Conclusion Questions

4 Introduction

5 NetApp is a leading vendor of innovative storage and data management solutions that help organizations around the world store, manage, protect, and retain one of their most precious assets: their data. Year founded: 1992 Number of employees: 12,000 Revenues: $6.3 billion (FY2014) Member of S&P 500 Member of Nasdaq 100 FORTUNE 500 Company Stock symbol: NTAP Worldwide offices: 150

6 NetApp Products FAS-Fabric Attached Storage FlashRay EF-Series EF560 - HA Pair, All Flash 2U FlashRay 6U DATA AltaVault SS3030 Controller 2U MetroCluster OnCommand StorageGrid OnCommand Insight StorageGrid

7 NetApp Engineering Engineering Data Center Services 9 R&D labs user base We are Customer Zero Drive Innovation: Leveraging newly emerging capability from our products and partners

8 NetApp Engineering By the numbers Users Workloads Engineering DC Engineering Labs 8 32M K 36M Mins Self-Service VM Lines of Product Code Hypervisors ESeries & ONTAP Controllers VM s Served Lines Test Automation Code 4,100 NetApp Dev & QA Engineers 7 Mins Incremental Build K Monthly Bisect Runs 6K 33PB Datacenter Storage 75% ~75K Active Self-Service VMs 10K 300PB Testbed Storage 15K 10K IPv4/v6 1G-40G Networking 220K Products Monthly Presub Smokes cdot Adoption Weekly VM turnover Storage Controllers (Virtual/Physical) Test cases per month Tech Stack

9 NetApp Engineering Global Engineering Cloud Global Engineering Cloud Self-Service Portal & API NetApp + ACI + OpenStack integrations Hybrid cloud service Cloud based test-beds Converge resources into improved services Hypervisor agnostic HYPERVISOR VM VM VM HYPERVISOR VM VM VM HYPERVISOR VM VM VM SVM Clustered Data ONTAP SVM Clustered Data ONTAP SVM Clustered Data ONTAP Massively scalable shared virtual data center infrastructure Cisco UCS NetApp cdot, E-Series, NPS, Cloud ONTAP ACI - Nexus 9K, 7K (Nexus 9508, 93128, 9396, 7018, 7010)

10 FlexPod with Nexus 9000

11 FlexPod Data Center Cisco Nexus 9000 NX-OS Design Cisco UCS C220 M3 C-Series Server(s) Cisco UCS 5108 B-Series Blade Chassis 2208XP Chassis FEX Modules B200 M3 B-Series Blade(s) Cisco UCS 6248UP Fabric Interconnects Po Po Po Po Cisco Nexus 9396PX Switches Po NetApp FAS8040 Storage Controllers w/ HA Backplane 10Gbe Only SAS Only Controller 1 Controller 2 ifgrp ifgrp Cisco Nexus 5596 Cluster Interconnects Po NetApp DS2246 Disk Shelves

12 FlexPod Data Center Cisco Nexus 9000 ACI Design Unified Computing System UCS 6248 Fabric Interconnects, Nexus 2232 Fabric Extender & UCS C and B Series Servers Legend 10Gbe only Converged Interconnects Standby Link SFO Interconnect 40Gbe only Spine Switches Nexus 9508 APIC 1-3 Nexus 9396 FAS Controller 8000 Nexus 9508 Nexus 5596 Cluster Interconnects x8

13 NetApp Engineering Facilities - RTP

14 Global Dynamic Lab Operational Since June, racks with 12kW/rack Cold room containment / hot aisle First Energy Star certified Data Center PUE 1.2 PUE 1.14 Operational as of May, racks capable 12kW/rack 36 rows with 63 racks per row Built on GDL design with many improvements Al Lawlis Sr. Director, NetApp

15 Nexus 7000/5000/2000 NX-OS Design GDL1

16 NetApp Engineering Design Nexus 7000/5000/2000 Architecture based on nexus 7K/5K/2K RTP Data Center Each VLAN is isolated to a specific distribution block 10G to the Top-of-Rack (ToR) Traditional Ethernet, unified I/O and Fibre Channel GDL1 External Connectivity CS01 Core CS02 Nexus 7K VDC and nexus 5K leveraged for FCoE/FC DS01 DS03 DS04 DS05 DS07

17 GDL2 Network Requirements R & D Data Center Design Principles Design that will last the next ~10 years Based on FlexPod Architecture Programmability, Automation, Orchestration Reduce the failure domain size from ~500 racks to Top-of-Rack (ToR) Scale and density seen at the distribution layer before are now being required at the Top-of-Rack Adding 40G/100G capability User/Group isolation on shared infrastructure

18 Nexus 9000 NX-OS Design GDL2

19 NetApp Engineering Design Nexus 9000 GDL2 and OTV RTP Data Center Benefits: Based on same NX-OS as Nexus 7000/5000/2000 OTV on Nexus 7004s to provide layer 2 connectivity 40G to Top-of-Rack (ToR) Migration path to ACI External Connectivity GDL1 CS01 CS02 GDL2 Core DI01 DS01 DS03 DS04 DS05 DS07 DI02 DI03 DS11 DS12 DS13 Future DS14 NX- OS/ACI DI04 40G L3 40G L2 Layer 3 Layer 2

20 Nexus 9000 NX-OS Design GDL1

21 GDL1 Network Refresh Planning for the Migrations Used inventory script copper vs. fiber used/total Created a detailed step by step procedure for the migration and tested the procedures in Cisco Solution Validation Services lab Converted configurations from the existing 7Ks to the new 9Ks, accounting for updated best practices and port conversions Staged Nexus 9000 switches with the appropriate configurations and software Pre-labeled the fiber for the new ports to speed up the swap out Goal was zero downtime

22 GDL1 Network Refresh Core/Distribution Steps Core shutdown interfaces on CS02 Verify traffic converges to CS01 Remove CS02 from rack Rack new CS02 no shut interfaces on CS02 Verify traffic convergences shutdown interfaces on CS01 Verify traffic converges to CS02 Remove CS01 from rack Rack new CS01 no shut interfaces on CS01 Verify traffic convergences Distribution shutdown interfaces on DS04 Verify traffic converges to DS03 Remove DS04 from rack Break Rack new DS04 no shut northbound/southbound interfaces on DS04 Verify DS04 is root bridge and HSRP active shutdown interfaces on DS03 Verify traffic converges to DS04 Remove DS03 from rack Rack new DS03 no shut northbound/southbound/east-west L3 interfaces on DS03 Configure no shut peer-link added members to the portchannels External Connectivity CS01 CS01 DS03 Pair CS02 CS02 DS04

23 GDL1 Network Refresh Lessons learned Core Swap out went well Routing Protocols converged as expected External Connectivity CS01 CS02 Challenges first Distribution Pair Bridge Assurance lacp suspend-individual DS03 DS04 spanning-tree guard root Pair OSPF summaries

24 GDL1 Network Refresh Top-of-Rack Total of 1100 racks to be refreshed Leveraging Power on Auto Provisioning (PoAP) and python scripting to get the configurations from the old ToRs to the new 9Ks Logistical challenge to work with the users in the half-rows to schedule the down time DS01 Pair DS02 Nexus 9372PX Nexus 9332PQ Nexus 9372TX Nexus 2248TP-E

25 NetApp Engineering Design Nexus 9000 GDL1 RTP Data Center Benefits Based on same NX-OS as Nexus 7000/5000/ G Core which allows 40G to GDL2 40G to Top-of-Rack (ToR) Migration path to ACI External Connectivity GDL1 CS01 Core CS02 GDL2 DI01 DS01 DS03 DS04 DS05 DS07 DS08 DI02 DI03 DS11 DS12 DS13 Future DS14 NX- OS/ACI DI04 40G L3 40G L2 Layer 3 Layer 2

26 NetApp Engineering Design GDL to 9516 Benefits Consistent infrastructure 9516s required for the 500 ToRs per DB RTP Data Center Lessons Learned auto-recovery timer Nexus 9000 BU and Cisco Advanced Services has created a forklift upgrade procedure External Connectivity GDL1 CS01 Core CS02 GDL2 DI01 DS01 DS03 DS04 DS05 DS07 DS08 DI02 DI03 DS11 DS12 DS13 DS14 Future NX- OS/ACI DI04 40G L3 40G L2 Layer 3 Layer 2

27 NetApp Engineering Design GDL2 Core Benefits Inter-DB traffic local to GDL2 Minimize 40G LR links between GDL1/GDL2 External Connectivity RTP Data Center Lessons Learned Went according to plan GDL1 CS01 Core CS02 CS03 Core CS04 GDL2 DI01 DS01 DS03 DS04 DS05 DS07 DS08 DI02 DI03 DS11 DS12 DS13 DS14 Future NX- OS/ACI DI04 40G L3 40G L2 Layer 3 Layer 2

28 Nexus 9000 NX-OS Benefits NetApp custom PoAP portal Leveraging Power on Auto Provisioning (PoAP) has improved the time to deploy ToRs 800% (4hrs to 30 min) Hot patching combined with python scripts is allowing us to deploy fixes Bi-di optics leveraged for 40G links Built-in programmability of the platform Easy transition for Ops staff

29 Nexus 9000 ACI Design

30 NetApp Engineering Design Current Deployment GDL2 Fabric Leaf Nodes Current Version is1.03f Fabric Spine Nodes Legend Fabric Layer 2 Layer 3 Border Leaf Nodes APIC APIC APIC External Routed Access for IPv4 Test Filers NFS/CIFS DS11 DS12 L3 Boundary for IPv6 11,000 Virtual Machines Infrastructure The Topology consist FlexPod Architecture with two spines and 9 leafs running in production since Feb 15 Existing GDL2 Infrastructure NetApp Release 8.3 cdot is being used to provide the NFS/CIFS data stores for the 11,000 virtual machines

31 ACI Defining Terms Private Network A Bridge Domain Subnet A EPG1 Tenant A App Profile A Private Network B Bridge Domain Subnet A Subnet B EPG2 Bridge Domain Subnet A EPG3 Tenant B Private Network A App Profile B Bridge Domain Subnet B EPG4 Tenant - Logical separator for: Customer, TG, BU, group etc. separates traffic, admin, visibility, etc. Private Network - Equivalent to a VRF, separates routing instances, can be used as an admin separation Bridge Domain - Not quite a VLAN, simply a container for subnets, can be used to define L2 boundary Application Network Profile - logical representation of an application and its interdependencies in the network fabric. End-Point Group - (EPG) Container for objects requiring the same policy treatment, i.e. app tiers, or services

32 ACI Policy Defining Terms EPG1 EPG2 Contract Contract Subject Subject Subject Subject Filter Action Label Contract - Definition of policy. Defines how an EPG communicates with other EPGs. Subject - Used to build definitions of communication between EPGs. Contains: filter, action, and optional label. Filter - Identifier for a subject, i.e. the traffic do you want to take action on. Required within a subject. Action - Action to be taken on the filtered traffic with a subject. Required within a subject. Label - Optional advanced identifier, when used labels allow for more complex definition of relationships within the policy model In/out port, etc. Drop, mark, redirect, etc. Optional label

33 NetApp Engineering Design ACI Specifics Tenant GEC1 Routed Outside Global Private Shared Network From default Common Tenant Bridge Domain GEC_UCS08_ Guest_1 Subnet A EPG Guest_1 Bridge Domain GEC_UCS08_ Guest_2 Subnet A App Profile UCS08 EPG Guest_2 DHCP Server External Connectivity via OSPF Tenant Common Routed Outside Global Private Network default Bridge Domain GEC1_UCS08_ Guest_1 Subnet A Bridge Domain GEC1_UCS08_ Guest_2 Subnet B Bridge Domain GEC2_UCS59_ Guest_1 Subnet C Bridge Domain GEC2_UCS59_ Guest_2 Subnet D Tenants Sharing OSPF Process Running in Common Tenant DHCP relay to an external DHCP server Contracts Providing at the private network and consuming at the EPG Tenant GEC2 Routed Outside Global Private Shared Network From default Common Tenant Bridge Domain GEC2_UCS59_ Guest_1 Subnet C EPG Guest_1 Bridge Domain GEC2_UCS59_ Guest_2 Subnet D App Profile UCS59 EPG Guest_2

34 Virtual Private Test Beds Vision LDAP Tenant - Common ANP-Common Services NIS DHCP DNS EPG - LDAP EPG - NIS EPG - DHCP EPG - DNS Filers Filers Filers EPG - Filers EPG - Filers EPG - Filers Test Tools Clients Test Tools Clients Test Tools Clients EPG - TestTools EPG - Clients ANP-Test Bed 1 EPG - TestTools EPG - Clients EPG - TestTools EPG - Clients ANP-Test Bed 2 ANP-Test Bed 3 Tenant - Virtual Private Test Beds

35 NetApp Engineering Design ACI Zone Based Design Fabric Building Spine Nodes Legend Fabric Layer 2 Layer 3 Fabric Zone Spine Nodes Fabric Leaf Nodes APIC APIC PODs PODs APIC APIC APIC APIC PODs PODs PODs WR IT

36 NetApp Engineering Design Long Term Plan - Site Benefits: Any Layer 2 segment anywhere Legend Fabric Layer 2 Layer 3 Single Fabric for the Campus Centralized Policy Management GDL1 GDL2 APIC APIC PODs PODs APIC APIC PODs PODs APIC APIC APIC APIC PODs PODs PODs WR APIC APIC APIC APIC PODs PODs PODs WR IT IT

37 NetApp Engineering Design ACI Challenges Fabric Leaf Nodes Scale endpoints per leaf and total number of leafs FC/FCoE support on the Leaf UCS08 Native IPv6 support in the ACI Fabric, shipping in the 1.1 release In-service software upgrade for the Leaf Spine support for 9516, shipping in the 1.1 release Transit routing for route peering with service appliances, shipping in the 1.1 release.

38 Conclusion

39 Conclusion Key Takeaways Programmability and automation enables removing ourselves from the user s work flows We leverage and influence the FlexPod reference Architecture Migrations from 7K/5K/2K to 9K can be done with 0 down time ACI in its current offering is missing a few key features that prevents us from wide scale deployment We still have to tackle the NX-OS to ACI migrations NetApp Engineering is committed to making ACI successful in our environment

40 References auto Recovery -> x/interfaces/configuration/guide/b_cisco_nexus_9000_series_nx- OS_Interfaces_Configuration_Guide_7x/configuring_vpcs.html#concept_09474B 25A6434B6792CEE6322B5EF7F7

41 Questions?

42 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

43 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings Related sessions

44 Thank you

45

46 R&S Related Cisco Education Offerings Course Description Cisco Certification CCIE R&S Advanced Workshops (CIERS-1 & CIERS-2) plus Self Assessments, Workbooks & Labs Implementing Cisco IP Routing v2.0 Implementing Cisco IP Switched Networks V2.0 Troubleshooting and Maintaining Cisco IP Networks v2.0 Interconnecting Cisco Networking Devices: Part 2 (or combined) Interconnecting Cisco Networking Devices: Part 1 Expert level trainings including: instructor led workshops, self assessments, practice labs and CCIE Lab Builder to prepare candidates for the CCIE R&S practical exam. Professional level instructor led trainings to prepare candidates for the CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in self study elearning formats with Cisco Learning Labs. Configure, implement and troubleshoot local and wide-area IPv4 and IPv6 networks. Also available in self study elearning format with Cisco Learning Lab. Installation, configuration, and basic support of a branch network. Also available in self study elearning format with Cisco Learning Lab. CCIE Routing & Switching CCNP Routing & Switching CCNA Routing & Switching CCENT Routing & Switching For more details, please visit: Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com

47 Design Cisco Education Offerings Course Description Cisco Certification Designing Cisco Network Service Architectures (ARCH) Provides learner with the ability to perform conceptual, intermediate, and detailed design of a network infrastructure that supports desired capacity, performance, availability required for converged Enterprise network services and applications. CCDP (Design Professional) Designing for Cisco Internetwork Solutions (DESGN) Instructor led training focused on fundamental design methodologies used to determine requirements for network performance, security, voice, and wireless solutions. Prepares candidates for the CCDA certification exam. CCDA (Design Associate) For more details, please visit: Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com

48 Data Center / Virtualization Cisco Education Offerings Course Description Cisco Certification Cisco Data Center CCIE Unified Fabric Workshop (DCXUF); Cisco Data Center CCIE Unified Computing Workshop (DCXUC) Implementing Cisco Data Center Unified Fabric (DCUFI); Implementing Cisco Data Center Unified Computing (DCUCI) Introducing Cisco Data Center Networking (DCICN); Introducing Cisco Data Center Technologies (DCICT) Prepare for your CCIE Data Center practical exam with hands on lab exercises running on a dedicated comprehensive topology Obtain the skills to deploy complex virtualized Data Center Fabric and Computing environments with Nexus and Cisco UCS. Learn basic data center technologies and how to build a data center infrastructure. CCIE Data Center CCNP Data Center CCNA Data Center Product Training Portfolio: DCAC9k, DCINX9k, DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K Get a deep understanding of the Cisco data center product line including the Cisco Nexus9K in ACI and NexusOS modes For more details, please visit: Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com

49 Network Programmability Cisco Education Offerings Course Description Cisco Certification Integrating Business Applications with Network Programmability (NIPBA); Integrating Business Applications with Network Programmability for Cisco ACI (NPIBAACI) Developing with Cisco Network Programmability (NPDEV); Developing with Cisco Network Programmability for Cisco ACI (NPDEVACI) Designing with Cisco Network Programmability (NPDES); Designing with Cisco Network Programmability for Cisco ACI (NPDESACI) Implementing Cisco Network Programmability (NPENG); Implementing Cisco Network Programmability for Cisco ACI (NPENGACI) Learn networking concepts, and how to deploy and troubleshoot programmable network architectures with these self-paced courses. Learn how to build applications for network environments and effectively bridge the gap between IT professionals and software developers. Learn how to expand your skill set from traditional IT infrastructure to application integration through programmability. Learn how to implement and troubleshoot open IT infrastructure technologies. Cisco Business Application Engineer Specialist Certification Cisco Network Programmability Developer Specialist Certification Cisco Network Programmability Design Specialist Certification Cisco Network Programmability Engineer Specialist Certification For more details, please visit: Questions? Visit the Booth or contact

50 Cloud Cisco Education Offerings Course Description Cisco Certification Designing the FlexPod Solution (FPDESIGN); Implementing and Administering the FlexPod Solution (FPIMPADM) UCS Director (UCSDF) Learn how to design, implement and administer FlexPod solutions Learn how to manage physical and virtual infrastructure using orchestration and automation functions of UCS Director. FlexPod Design Specialist; FlexPod Implementation & Administration Specialist Cisco Prime Service Catalog Learn how to deliver data center, workplace, and application services in an on-demand, automated, and repeatable method. Cisco Intercloud Fabric Learn how to implement end-to-end hybrid clouds with Intercloud Fabric for Business and Intercloud Fabric for Providers. Cisco Intelligent Automation for Cloud Learn how to implement and manage cloud deployments with Cisco Intelligent Automation for Cloud For more details, please visit: Questions? Visit the Booth or contact