with ACI Any workload anywhere.

Cisco IT: Scalable Enterprise UCS with ACI Any workload anywhere. Hugh Flanagan, Senior IT Engineer Jason Stevens, IT Engineer BRKCOC-0

Agenda Introduction Challenges of Large Scale UCS Deployments in the Datacenter How ACI and UCS Central Allow us to Enhance UCS Infrastructure Operational Gains Conclusion

Introduction

Cisco IT s Internal Cloud Journey Reduce Cost Deliver Services Faster Virtualization Home Grown Automation Private Cloud (Iaas + PaaS) Application Centric Cloud Hybrid Cloud We Are HERE UCS APIC VM VM VM ESX VMware Virtualization Cisco Intelligent Automation for Cloud (CIAC) Virtual Security Gateway Prime Network Services Controller Application Centric Infrastructure Cisco Intercloud Open Source BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Challenges of Large Scale UCS Deployments in the Datacenter

Cisco IT s Legacy Design Up to Chassis per UCS Domain Fault / Migration Domain Limited to Single UCS Domain Hardware Maintenance Set Aside = % BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Domain Upgrade Maintenance Summary Cisco IT has 0 UCS Domains Upgrades Complicated by: Change Freeze Windows Maintenance Windows Critical Client Workloads This Results in UCS Upgrades Taking up to Years to Complete BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

How ACI and UCS Central Allow us to Enhance UCS Infrastructure

End to End Programmable Infrastructure Resources Controllers Resource Managers Orchestration (Cloud) Client Application Code PaaS Compute CIAC Portal ASA Network Integrated Security eaclm Security Admin IP File/ IP Block/ IP Object Storage Block Storage Compute Network/Storage Admins BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Fabric Infrastructure Requirements Flexible Topology Multi-Tenant Virtual Boundaries Physical and Virtual Services Easier to Manage Components ACI: 0, PX, APIC UCS: Bm, B0m, B0m ESXi + AVS Openstack + KVM + OVS NetApp CDoT (.) Citrix SLB ASA CIAC (estore & automation) BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Next Generation Virtual Compute Design vcenter Domain- vcenter Domain- vcenter Domain- vcenter `Domain- Logical Pod- Logical Pod- Logical Pod- NKV Domain- NKV Domain- NKV Domain- NKV Domain- Logical Pod- Physical Leaf pair- Physical Leaf pair- Physical Leaf pair- Physical Leaf pair- Legacy Virtual Compute Design Virtual Compute Design with ACI vmotion vmotion vmotion vmotion vmotion Across VMM Domain BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Challenges with Next Generation Design Increased Capacity. Terahertz of CPU Capacity (not including hyper-threading) 0 Terabytes of Memory Per UCS Domain (0 Blade UCS Domain) VLANs are not Able to Support Number of Possible Tenants Keeping UCS Domains Configuration in Sync BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

UCS Central Architecture XML API Web Client SNMP UCS Central Virtual Appliance DME Services Framework PostgreSQL DB UCS Manager # UCS Manager #n Hardened Guest OS BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

VXLAN Extends Layer networks across Layer infrastructure MAC-to-UDP encapsulation VXLAN Tunnel Endpoint (VTEP) 0-byte overhead to original frame Can support up to million segments Cisco IT is using AVS (Application Virtual Switch) for VXLAN connectivity VXLAN traffic uses the Infrastructure VLAN configured during initial fabric turn-up BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

OpFlex: An Open Source Framework OpFlex changes have been upstreamed as part of OpenDaylight. OpFlex library and MODB are reusable across any device Rendering agent must be rewritten to specific device OpFlex Agent OpFlex (Policy Manager) Managed Objects Store (MODB) OVS Render Plugin (Policy Enforcement) Open vswitch changes are all upstream and included in OVS. Open vswitch OVSDB OpenFlow Flow Table Datapath and Linux kernel (.) include support for VXLAN-GBP Datapath SW/HW Datapath BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

OpFlex Hypervisor Switch Integration OpFlex Proxy runs as part of ACI leaf managing interaction with APIC OpFlex Agent runs on hypervisor in userspace and will be packaged as a separate component OpFlex Proxy OpFlex Proxy OpFlex Proxy OpFlex Proxy OpFlex Proxy OpFlex Proxy It speaks to OVS through OpenFlow / OSDB interfaces Available as part of FCS+ (BMR release of ACI) OpFlex Agent OPEN VSWITCH OpFlex Agent CISCO AVS OpFlex Agent MICROSOFT HYPER-V OpFlex Agent THIRD PARTY VSWITCH OpFlex Agent OPEN VSWITCH OpFlex Agent CISCO AVS BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

ID 0 0 0 0 0 0 0 0 0 Cisco Nexus T GE Fabric Extender STAT 0 0 0 0 0 0 0 0 0 ID 0 0 0 0 0 0 0 0 0 Cisco Nexus T GE Fabric Extender STAT 0 0 0 0 0 0 0 0 0 ID 0 0 0 0 0 0 0 0 0 Cisco Nexus T GE Fabric Extender STAT 0 0 0 0 0 0 0 0 0 ID 0 0 0 0 0 0 0 0 0 Cisco Nexus T GE Fabric Extender STAT 0 0 0 0 0 0 0 0 0 Fabric Design Bare Metal Border Storage Virtual Compute DMZ Core Internal Core Clustered Data on Tap ESXi Openstack BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Operational Gains

Hardware Specifications Legacy Deployment - B0 M X0 CPU x ( Cores per CPU) GB RAM VIC CNA Modern Deployment - B0 M E- v CPU x ( Cores per CPU) GB RAM VIC 0 CNA (Supports VXLAN Offload) BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Blade Consolidation Legacy DC Environment has,00 Hypervisor Blades,00 of which are B0 M ACI, Leveraging B0 M, Consolidates this to 00 Blades : Consolidation on Average, Achieved Through Higher Density Blades and Larger Migration Domains : Consolidation Seen when Consolidating B0 M Environments BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Domain Upgrade Maintenance Time Expended for Legacy hours Determining Maintenance Windows hours negotiating with clients minutes running health checks and verifications hour fixing issues to ensure no downtime. hours executing upgrade Time Expended for ACI minutes evacuating. hours executing upgrade BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Capacity Additions Under ACI Each UCS Domain is built with a minimal amount of chassis and blades for Day workloads Chassis additions are performed when additional capacity is needed for quick turnaround Automation built leveraging CIAC, UCS Central, and APICs configures the blades for use BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

UCS Domain Evacuation Demo

Best Practices / Lessons Learned

Cisco and/or its affiliates. All rights reserved. Cisco Public BRKCOC-0 Hitless Upgrades and Failover Testing All failovers and upgrades: max - sec connectivity loss ACI Switch Application Virtual Switch UCS Compute (B-Series) ESXi/Vmware multiple IP Storage Leaf Pairs Border Leafs Leaf Pairs Multiple Compute & Services Leaf Pairs NAS DMZ Internal Services (physical or virtual) IP Network ACI Fabric Services (physical or virtual) Compute Compute Internet Cisco+Corp APIC APIC APIC vcenter NAS

Best Practices and Lessons Learned Test/Certify new Features and Code Prior to Production Deployment Utilize lab environment for testing Create certification process with standard must have capabilities and verification Document/track issues found Start off with basic functionality and add new features as you go Build with Automation in Mind Create standard and reusable constructs Document naming conventions for various objects to make readability and troubleshooting easier Scripting skills will help you on your journey BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Best Practices and Lessons Learned AVS for vswitch Use AVS and VXLAN where possible L-L services required to be on VLAN VMM Reduced configuration overhead (No need to trunk down huge amounts of VLANs) Better management and visibility when troubleshooting connectivity issues BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Best Practices and Lessons Learned Upgrades Test upgrade process in lab prior to production rollout and check release notes for any important changes Maintenance Groups Border Leaf Even / Border Leaf Odd Separate Compute Leaf Pairs as Capacity Allows (Still into Evens / Odds) Config backups Schedule secure configuration backups/archives on daily basis BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Conclusion

Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Pod Pod Pod Pod Resource Stripe - ACI Greater Resiliency Easier Maintenance Resource pools and applications striped across multiple pods BRKCOC-0

Flexible Assets Storage or Compute Unit Unit Unit Vertical Lock-in Initial placement of applications results in storage and compute resources physically locked in Reserve (future & failover) capacity can t be flexibly used Stranded Storage and Compute Capacity Today Horizontal Scaling Applications can flexibly tap into any available compute or storage resources ACI Unit Unit Unit Initially: ACI + VMware + NetApp CDOT ACI Policy Based Control Infrastructure Maintenance BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Any Application, Anywhere ACI allows any EPG to be leveraged on any compute resource within fabric Using VXLAN allows any compute resource to be immediately capable of running any VM with no further network configuration Stretched Fabric allows for logically combining multiple datacenter halls into a single fabric. UCS Central allows massively scaling compute capacity while keeping policies and configuration consistent throughout BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public 0

Additional Reading Cisco IT ACI Storage Deployment http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers /Cisco_IT_ACI_Storage_Deployment.html Cisco IT ACI Design http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers /Cisco_IT_ACI_Design.html Application Virtual Switch http://www.cisco.com/c/en/us/products/collateral/switches/application-virtualswitch/datasheet-c-0.html AVS vs DVS Convergence http://www.cisco.com/c/en/us/products/collateral/switches/application-virtualswitch/white-paper-c-.html BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Additional Reading Cisco APIC Controller with Openstack Integration Demo https://dcloud-rtp-web-.cisco.com/dcloud/demoinfo.jsp?d=0 UCS Central Demo https://dcloud-cms.cisco.com/demo/cisco-ucs-central---v VXLAN Whitepaper http://www.cisco.com/c/en/us/products/collateral/switches/nexus-000-seriesswitches/white-paper-c-.html Design Guide for L-L Services on AVS http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus00/avs/refere nce-guide/l-l-services/cisco_avs_l-l_white_paper.html BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $0 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer : meetings Related sessions BRKCOC-0 Cisco and/or its affiliates. All rights reserved. Cisco Public

Please join us for the Service Provider Innovation Talk featuring: Yvette Kanouff Senior Vice President and General Manager, SP Business Joe Cozzolino Senior Vice President, Cisco Services Thursday, July th, : am - : pm, In the Oceanside A room What to expect from this innovation talk Insights on market trends and forecasts Preview of key technologies and capabilities Innovative demonstrations of the latest and greatest products Better understanding of how Cisco can help you succeed Register to attend the session live now or watch the broadcast on cisco.com

Thank you