Cisco Ultra Automation Services

Transcription

1 Solution Overview Cisco Ultra Automation Services Solution for automating and orchestrating the Cisco Ultra Services Platform 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 23

2 Disclaimer The products and features described in this document are in varying stages of development and will be offered on a when-and-if-available basis. Such products and features and all pricing related to future features and products are subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in any roadmaps. Customer should not rely on the availability of any future product or feature in executing any agreements or placing any orders related to specified projects. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 23

3 Contents Scope and Objectives... 4 Network Function Virtualization and Management and Orchestration (NFV-MANO)... 4 NFVI and VIM installation... 6 VNFM and VNF orchestration... 6 Cisco Ultra Services Platform (USP) and Ultra Automation Services (UAS)... 7 Key considerations... 8 Cisco Ultra Services Platform VNF-EM... 8 Cisco Ultra M UAS building blocks AutoIT: automation of NFVI and VIM installation AutoVNF: automation of VNFM and VNF orchestration Cisco Ultra Services Platform deployment workflow summary Cisco Ultra Services Platform operational monitoring Examples of day-n automation Customer deployment examples Evolving Ultra Automation Services (UAS) Helpful reference documents Frequently asked questions Failure effects of Cisco Ultra Services Platform software component Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 23

4 Scope and Objectives This white paper is meant to outline the general automation and orchestration needs in virtualized network operation environments and explain at a high-level on how Cisco Ultra Automation Services work within typical Network Function Virtualization Infrastructure/Virtualized Infrastructure Manager (NFVI/VIM) installation and VNF/Virtual Network Function Manager (VNFM) deployment workflows. The goal is to help readers to conceptualize where each Ultra Automation Services (UAS) module fits within the end-to-end workflows. This white paper does not intend to provide in-depth explanation of Network Function Virtualization Management and Orchestration (NFV-MANO) architecture and details about vendor-specific components in any layer of the NFV- MANO architecture. Readers are expected to know the fundamentals of the OpenStack, European Telecommunication Standards Institute (ETSI) defined management and orchestration architecture, and VNF lifecycle management. After reading this document, readers should be able to understand minimally the following four topics: Day-0 Ultra VNF orchestration and deployment workflow Role of each Ultra Automation Services component in day-0 orchestration and deployment workflow Role of the Virtual Network Function Element Manager (VNF-EM) and how VNF-EM benefits the lifecycle management use cases and integration with the rest of the MANO environment Ultra M solution stack Network Function Virtualization and Management and Orchestration (NFV-MANO) Network Function Virtualization (NFV) has been an extremely popular topic within the mobile operator community for the past three to five years. The concept is about decoupling network functions for example, evolved packet core gateway, network address translation, and network firewall from proprietary hardware so that they can run in virtualized computing environments with virtual servers, storage, and networks. One well-known, and commonly accepted in many cases, benefit is the reduction of CapEx/OpEx through the use of more resource-efficient cloud infrastructures. However, the larger promise of cost savings is from the ongoing operation of services by increasing service velocity, service agility, and service flexibility through software management and orchestration, driving the operational costs involved in constant manual changes, which often take weeks or months. Although there has been very little debate about the enormous potential for cost saving from the automation and orchestration of highly repetitive and time-consuming tasks required to rapidly bring software into life and serviceready state, innovations in these areas have been relatively sparse for the telecommunications industry. A relevant standards body in the space European Telecommunications Standards Institute (ETSI) has been leading the effort of defining the NFV architectural framework and reference points, including the Management and Orchestration (MANO) specifications, but a full-featured NFV-MANO solution is yet to be seen in large production environments, particularly the Network Function Virtualization Orchestrator (NFVO), where onboarding various Virtualized Network Functions (VNFs) would be done along with providing full-service lifecycle management functions from provisioning to decommissioning. (See Figure 1.) 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 23

5 Figure 1. ETSI NFV-MANO architectural framework and reference points Note that, unlike IETF, where specifications often prescribe messaging protocols and protocol data units, ETSI NFV-MANO remains guidelines. Therefore, there is very little interoperability out of the box in terms of descriptors and protocols to be used. Consequently, multivendor integration remains challenging and often requires significant effort. Based on the ETSI NFV-MANO specification, NFVO is expected to provide the following services (nonexhaustive): Cataloging of descriptors and Application Programming Interfaces (APIs) to define service topology Onboarding of artifacts onto a Virtualized Infrastructure Manager (VIM) required to construct network services, either directly or in conjunction with a VNF Manager (VNFM) Network service instantiation and lifecycle management: start, stop, update, monitor, scale, SLA enforcement in terms of service availability and performance and event collection/correlation Management of the instantiation of VNFM Management of the instantiation of VNFs, in coordination with VNFM At the absence of an NFVO, which can provide the preceding services for VNFs, manual scripting and/or human intervention is required at every phase of each VNF s lifecycle, which would inevitably prevent operators from reaping the operational and cost benefits of NFV because of extreme lack of scalability Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 23

6 NFVI and VIM installation The very first step of the VNF deployment workflow, of course, is the hardware rack and stack. This step involves physically mounting servers, switches, and other hardware on the racks and physically connecting cables and configuring them to establish network connectivity. Subsequently, low-level software updates for example, firmware upgrades might be required, as well as running various tests to detect and remedy any faulty hardware components or incorrect networking configuration. Often these steps are manual, just as in traditional hardware appliance deployments. Not surprisingly, the need for automating some of these steps efficiently and repeatably with less errors is quite evident. Before VNF can be deployed onto the virtualized infrastructure, NFVI resources must be installed/configured, and VIM must be fully operational. Installation and configuration tasks of NFVI and VIM are often complicated, extremely time-consuming, and error-prone if attempted manually, because there are many software updates to perform, dependencies to check, additional packages to download, and series of Command Line Interface (CLI) commands and API calls to make. It is not uncommon for users with adequate proficiency to spend days and weeks performing these required tasks. In the case of RedHat OpenStack Platform (OSP), the OpenStack Platform Director (OSPD) is a toolset for installing and managing a complete OpenStack environment. It is primarily on the OpenStack project Triple-O, which is an abbreviation for OpenStack on OpenStack. This project takes advantage of OpenStack components to install fully operational OpenStack environments. This includes new OpenStack components that provision and control bare-metal nodes to use as OpenStack nodes. RedHat OpenStack director uses two main concepts: an undercloud and an overcloud. The undercloud installs and configures the overcloud. The undercloud is the main director node. It is a single-node OpenStack installation that includes components for provisioning and managing the OpenStack nodes such as compute and storage nodes. The undercloud installation results in OpenStack environmental planning, bare-metal system control, and manifest files to create an OpenStack environment. The overcloud is the result of what the undercloud creates, which includes the OpenStack controllers (1:2 redundant by default) running various OpenStack cloud services, compute nodes, and storage nodes (CEPH, block, and/or object). After completion of the overcloud deployment, the Virtualized Infrastructure Manager (VIM) becomes operational and ready for provisioning. VNFs often have stringent requirements on NFVI to meet certain Key Performance Indicators (KPIs), which, in turn, necessitates VIM-level inventory management and resource provisioning. VNFM and VNF orchestration VNFM is responsible for instantiating VNFs using virtualized resources created by NFVO. However, before the VNFM can instantiate a VNF, VNFM itself must be orchestrated and be in an operational state. This can be accomplished through either a manual installation or an automated instantiation using NFVO. The manual installation typically involves manually onboarding a VNFM binary image, creating virtual machine instances, creating/configuring networks, and configuring the VNFM through its CLI. The automated instantiation, even if NFVO is capable of onboarding the Virtual Network Function Descriptor (VNFD) of the VNFM, typically requires some level of bootstrapping to start the VNFM instances using the OpenStack CLI or APIs Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 23

7 After the VNFM becomes operational, the user might be able to use a series of CLI commands or make API calls to NFVO to start the process of deploying the VNF. In the case of the Cisco Ultra Services Platform, the very first component that must be deployed is the VNF Element Manager (VNF-EM), which is one of several VNF Components (VNFCs) that make up the complex Cisco Ultra Services Platform VNF. VNF-specific element managers are almost always a component of a given VNF, containing VNF-specific lifecycle logics, policies, and executable scripts. It often serves as the API endpoint for VNFM and northbound Operations Support System/Billing Support System (OSS/BSS) entities, providing an abstraction layer for external systems. As the VNF-EM is being deployed, the complete VNFD supporting specific use cases is passed on to the VNF-EM as a day-0 VNF configuration file, enabling the VNF-EM to construct the complete VNF deployment by onboarding required VNFCs and configuring them. (See Figure 2.) Figure 2. Generic ETSI NFV-MANO VNF deployment workflow Cisco Ultra Services Platform (USP) and Ultra Automation Services (UAS) The Cisco Ultra Services Platform is a software-defined mobility solution based on the core principles of NFV, as well as Cisco s market-leading technologies in evolved packet core, cloud, and software virtualization. Cisco delivers on the promise of empowering mobile operators to create and deliver customized private networks for enterprise, IoT, Mobile Virtual Network Operator (MVNO), and consumer services not only drastically faster, but also more cost-effectively for initial deployments and ongoing operations. One of ways in which Cisco provides a huge advantage is through software installation automation and deployment orchestration. The Cisco Ultra Services Platform enables multiple mobility use cases based on different provisioning parameters and deployment configurations. At the time of this writing, the Cisco Ultra Services Platform supports both OpenStack and VMware Virtualized Infrastructure Managers (VIMs), although most mobile operators have adopted RedHat or another mainstream distribution of OpenStack. Virtualized environments based on either of these VIMs allow deployment of the Cisco Ultra Services Platform on virtualized compute hosts, storage hosts, and networks using core OpenStack services for example, Nova, Cinder, Neutron,, and so on in a fully automated manner without having to have a full-featured NFVO in place as a prerequisite. Going much further, UAS provides automated installation of the NFVI (future) and VIM based on user-provided manifests, allowing operators to have their virtualized environments ready for VNF deployment in a fraction of the time that manual installations would normally require. At the time of this writing, UAS supports automated RedHat OSP installation only Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 23

8 Key considerations To determine if one or more of the UAS modules are needed to deploy the USP, one must ask several critical questions, including: Do I have tools to generate the full USP VNFD, based on specified VNF attributes and VNF configuration parameters, that is fully compatible with my NFVO? Is my NFVO capable of onboarding the full USP VNFD? Do I have a fully functional NFVO that is capable of providing: Logical models and service topologies Virtualized resource management VNF inventory management VNF day-n configuration management Is my NFVO capable of onboarding an Elastic Services Controller (ESC) VNFD and instantiate it? Do I have tools to automate NFVI and OpenStack installation? Do I have tools to automate VIM provisioning to meet VNF prerequisites? Do I have tools to automate installation validation steps? If the answers to the preceding questions include at least one no, then one or more UAS modules can drastically help to reduce human errors and the time required to provision, orchestrate, and deploy USP. Cisco Ultra Services Platform VNF-EM The ETSI GS NFV-MAN 001 document states that Most of the VNF Manager functions are assumed to be generic common functions applicable to any type of VNF. However, the NFV-MANO architectural framework needs to also support cases where VNF instances need specific functionality for their Lifecycle Management (LCM), and such functionality may be specified in the VNF Package. Sophisticated telecommunications VNFs for example, Cisco Ultra Services Platform require VNF-specific LCM logics beyond what a generic VNFM can typically provide. This means that there are two likely options when it comes to the placement of these VNF-specific logics: in VNFM or within VNF itself. To ease the onboarding of the USP without complex integration with VNFM and/or NFVO, all LCM logics specific to the USP are contained inside the USP itself, more specifically inside the VNF-EM. Descriptors are written per ETSI NFV-MANO guidelines, and interfaces are defined as dynamically loadable plug-ins using industry-standard specifications (NETCONF/YANG). This enables seemless integration of the Cisco Ultra Services Platform with the other components of the MANO ecosystem without imposing a mandatory requirement of a specific VNFM by using a set of primitive LCM verbs and nouns. During the -0 USP deployment workflow, the VNF-EM is deployed first using VNFM. It is important to note here that VNF-EM is one of several possible VNF-Cs, implying that VNF-EM is an internal part of the VNF. At this stage, only the management plane virtual machines are up and running, and it is not a fully constructed VNF yet. VNF-EM requires a full USP VNFD as day-0 configuration to construct the rest of the VNF and apply relevant VNFC configurations. As the VNF-EM parses through the complete VNFD, the VNF-EM makes simple API calls to an external VNFM (using the ve-vnfm reference point) to bring up other VNFCs control function and session function instances and properly network them together. VNF-EM interacts closely with the control function, which is a StarOS-based subsystem that contains necessary application-level monitoring and healing logics relevant to all other internal VNFCs and works with VNFM to apply appropriate policies when action is required. For example, when there is a session function instance failure, the control function detects the failure in the application level well before VNFM would detect it in the virtual machine level. Upon the detection, the control function updates the Virtual Network Function Record (VNFR), and VNF-EM detects the session function instance failure. VNF-EM then 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 23

9 triggers appropriate recovery events through the VNFM based on VNF-specific policy. In short, VNF-EM, together with control function, provides the VNF-specific lifecycle management functionalities, whereas VNFM provides generic virtual machine level lifecycle management functionalities because one virtual machine simply appears the same as another virtual machine in the eyes of the VNFM. Consequently, primitive virtual machine level heartbeat monitoring cannot adequately detect service-level health. After the Cisco Ultra Services Platform VNF is fully constructed, VNF-EM configures the VNF with day-n configuration parameters using the internal Service Configuration Manager (SCM). Alternatively, day-n configuration management service can be provided by an external network service configurator, which can be either an OSS or MANO entity. VNF-EM naturally serves as the NETCONF/REST API endpoint for OSS/BSS components, serving aggregated VNF operational data and providing a day-n configurable interface. For example, if there is a need to change the VNF during its operation which would result in change of VNF running state a northbound OSS entity can make API calls to push down the desired configuration changes. Similarly, a BSS can make API calls to collect VNF realtime statistics through VNF-EM rather than polling each individual VNFC. (See Figure 3.) Figure 3. Cisco Ultra Services Platform VNF-EM Other VNF-EM functionalities include: VNFM proxy: VNF-EM contains a modularized plug-in for a specific VNFM. At the time of this writing, the plug-in VNF-EM uses is Cisco Elastic Services Controller (ESC). The plug-in allows third-party VNFM provider to develop back-end APIs for its own VNFM. KPI collection: VNF-EM serves as the KPI aggregation point for internal VNFCs. Third-party VNF onboarding: VNF-EM provides an onboarding framework for third-party VNFs through the use of Network Element Drivers (NEDs) or VNF-specific plug-ins. After being onboarded, VNFs become USP VNF-Cs and are managed as native components of the Cisco Ultra Services Platform Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 23

10 Cisco Ultra M Cisco Ultra M is a prepackaged and fully validated virtualized mobile packet core solution designed to simplify the deployment of USP VNFs. The solution combines the USP VNF, RedHat OSP validated by Cisco, and Cisco networking and computing hardware platforms into a fully integrated and scalable stack delivered as a turnkey solution. As such, UAS is fully integrated with Ultra M, providing the essential installation and deployment capabilities to instantiate and provide lifecycle management for the USP using OpenStack virtual infrastructure manager. Hardware and software components included in Ultra M can be found in separate release documentation provided to customers. Ultra M is offered in multiple sizes with varying hardware models and software versions. More detailed description of each model can be found in the Ultra M in the Hassle-Free Full Stack Mobility Solution document. UAS building blocks UAS is composed of several functional components, each providing unique services at different parts of the end-toend installation and orchestration workflows. There are two main categories of UAS modules: VIM installation and VNF deployment. The former provides automated installation of VIM-Orch and VIM OSP undercloud and overcloud, respectively, in the case of RedHat OpenStack whereas the latter provides many functionalities of what ETSI-defined NFVO would provide in the NFV-MANO reference architecture framework. In essence, UAS serves as: A VNF-specific NFVO dedicated to orchestrating and managing the lifecycle of VNF in a hosted, private, or public cloud Workflow automation to instantiate, provision, and monitor NFVI/VIM for USP hosted on hardware supplied by Cisco All UAS modules provide standard NETCONF APIs and configuration YANG models for maximum level of flexibility in programmability and software integration. (See Table 1.) Table 1. Ultra Automation Services building blocks (current) AutoDeploy AutoIT AutoVNF VIM/NFVI Inventory Management VNF Inventory Management Full Day-N Service Configuration Automated software installation and patching Automated OpenStack VIM installation VIM Provisioning for VNFs Health monitoring Onboarding of VNF artifacts onto VIM Automated composition of VNFD Automated deployment of VNFM and VNF Monitoring and healing of VNF-EM and ESC AutoDeploy: Top-level interface where users can define the underlying NFVI/VIM domain boundaries (often mapped to geographic sites) for NFVI/VIM installation and operational purposes, create service catalogs for logical service topologies, and manage multiple VNF software images. Service topologies are defined per ETSI and can be constructed using APIs or CLI. Currently, one instance of AutoDeploy is required per site (or per VIM domain), but a single instance is capable of managing multiple VNFs. AutoDeploy is designed to eventually become multisite capable, which means a single instance can act as a central deployment control center for a region or country. AutoDeploy serves as the endpoint where all user workflows for example, VIM installation, VNF deployment, VNF upgrade,, and so on would be initiated. This allows abstraction of multiple software layers through an AutoDeploy configuration model. As such, AutoDeploy is also a logical entity to aggregate and correlate stats, events, and alarms to represent end-toend service-level views, as opposed to limited virtual machine or VNF views Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 23

11 AutoIT: Performs bootstrapping of NFVI nodes and configures hardware profiles for provisioning and installation. AutoIT installs and configures VIM undercloud and overcloud (including OpenStack controllers, compute nodes, and storage nodes) and performs VIM sanity checks to determine VNF deployment readiness. As a part of the VIM configuration, AutoIT provisions required virtualized resources through the VIM for VNF onboarding such as tenants, availability zones, storage, and networking. AutoVNF: Orchestrates the VNFM (ESC) deployment, which handles the VNF lifecycle process in conjunction with VNF-EM. AutoVNF also composes an ETSI-compliant VNF Descriptor (VNFD) for full USP VNF based on user-provided configuration parameters. This VNFD is used by VNF-EM to further construct the full VNF for specific mobile use cases such as SAEGW, epdg, PGW, and so on. AutoVNF can run in two different modes: standalone or NFVO mode. The standalone mode is for environments where there are existing NFVOs to handle all VNF onboarding and provisioning processes (loading images into OpenStack Glance service, provisioning varieties, networks, subnets, and other required OpenStack artifacts), whereas the NFVO mode is used to mimic AutoVNF as an NFVO to handle these processes when there are no NFVOs. (See Table 2.) Table 2. Recommended virtual machine resource allocation Virtual Machine vcpu RAM (GB) Root Disk (GB) AutoIT AutoDeploy AutoVNF VNF-EM All UAS and VNF-EM virtual machines can be deployed on the same physical hosts where VNFM virtual machine instances are deployed. For obvious reasons, cluster VM instances, if applicable, should be deployed with antiaffinity policy so that redundant VM instances are deployed on different physical hosts to maximize the integrity of the high-availability design. Furthermore, these management-plane virtual machine instances should not be placed on the same physical hosts where control-plane and/or data-plane VNF-Cs are deployed. AutoIT: automation of NFVI and VIM installation As highlighted earlier, before USP can be deployed, the virtualized infrastructure must be prepared and operational. AutoIT runs as a single virtual machine instance on the same node where the AutoDeploy virtual machine instance would be running. These virtual machines are created using a script that is provided in the USP ISO file on bare metal running RHEL 7.3. This node is referred to as the Ultra M Manager and used to automate the installation of the following: Underlying NFVI elements (compute and storage, L2/L3 elements, and so on): H1CY2018 RedHat OSP undercloud and overcloud At the time of this writing, the automated installation of the NFVI is under development for future release in H1CY2018. As for the automation of VIM installation, AutoIT performs automated RedHat OSP undercloud and overcloud installations. As mentioned in a previous section, RedHat uses OSPD to manage the lifecycle of the OpenStack cloud, including the predeployment planning, deployment, and environmental changes (for example, node replacement). Because there is no OpenStack cloud at the time of the planning and deployment stages, this is referred to as the undercloud, and it runs inside a virtual machine created by AutoDeploy based on the userprovided configuration file. In this workflow, AutoDeploy serves as the abstraction layer of the RedHat OSP, providing a programmable configuration model into RedHat OSP, which serves a critical role in deploying multiple VIMs through an abstracted set of APIs and descriptors Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 23

12 During overcloud deployment, bare-metal hardware is installed with the RedHat Enterprise Linux OpenStack platform, and roles are assigned to specific nodes based on the overcloud installation manifest files. This vastly simplifies the administrative overhead, especially with large-scale deployments. As a result of the overcloud installation, the OSPD virtual machine creates three OSC controller nodes (for HA), along with a specified number of compute and storage nodes. After the VIM is fully operational, AutoIT can proceed with provisioning the virtualized infrastructure using the VIM in preparation for USP VNF onboarding. Provisioning includes uploading VNF software packages and defining required OpenStack tenants, availability zone, security parameters, storage types, and networking types (for example, SR-IOV, physnet). This provisioning step can alternatively be done out of band by a VIM administrator. Other AutoIT functionalities include: Providing VNF health monitoring as described in the USP Operational Monitoring section later Managing versions of software packages using YUM repo Storing public key information in ISO database for RPM signature verification Deploying AutoVNF virtual machine cluster (1+2 redundancy) Note: In UAS releases prior to 6.0, AutoIT is deployed in two discreet types AutoIT-NFVI and AutoIT-VNF, where the former is responsible for the instantiation of RedHat OSPD virtual machine instance and the latter is responsible for the provisioning of the overcloud as described above. Starting in UAS release 6.0 and later, all AutoIT-NFVI and AutoIT-VNF features are consolidated into AutoIT. AutoVNF: automation of VNFM and VNF orchestration Based on its given roles, AutoVNF (in NFVO mode) can be viewed as a VNF-specific NFVO for USP because AutoVNF performs six main actions within the USP deployment workflow: 1. Composes USP VNFD based on specified configuration parameters for VNF-EM. 2. Deploys VNFM (ESC) based on VNFM descriptor. 3. Invokes RPC to onboard and deploy VNF-EM using VNFM northbound API. 4. Provisions all required OpenStack resources required by USP for example, images, flavors, networks, and so on if not precreated. 5. Bootstraps and configures Virtual Deployment Units (VDUs) with day-0 configuration as they are deployed. 6. Manages and allocates IP addresses for VNFCs, including Virtual IP (VIP) addresses for control function and VNF-EM clusters (for high availability) and for non-dhcp based deployments. AutoVNF can alternatively run in standalone mode in environments where there are existing NFVOs that can adequately perform USP and ESC onboarding and OpenStack resource creation. In this mode, AutoVNF is used to compose ETSI-compliant USP VNFDs for NFVO (step 1 earlier) by referencing precreated OpenStack resources that are provisioned out of band by NFVO and bootstrap VDUs with day-0 configuration (step 5 earlier). Minimally, AutoVNF standalone mode is required for all USP deployments. AutoVNF NFVO mode runs in three virtual machine clusters (1:2 redundancy) to provide high availability of its services, whereas the standalone mode only requires single instance running in non-ha configuration. AutoVNF supports multiple VNF deployments by managing different set of catalogs required for each VNF deployment, including VDU, networks, volumes, secure tokens, and so on. (See Figure 4.) 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 23

13 Figure 4. AutoVNF NFVO mode Other AutoVNF functionalities include: Monitors and automatically heals VNF-EM and VNFM. Facilitates automated workflow of upgrading VNF and maintenance (for example, graceful shutdown). Provides VNF security, credentials, and SSH keys through the use of secure tokens. Cisco Ultra Services Platform deployment workflow summary Generally speaking, there are set of prerequisites which must be met before a VNF can be deployed, in both manual and automated workflows: General Prerequisites Rack-and-Stack of hardware Fully configured & operational NFVI Fully installed & operational VIM VNF package & descriptor on-boarded VNFM package & descriptor on-boarded VIM provisioned for VNF OpenStack resources provisioned for VNF VNFM deployment VNF deployment Responsible Provider N/A; manual N/A; manual N/A; manual NFVO NFVO N/A; manual NFVO NFVO NFVO Using Ultra M as a reference deployment model, the USP deployment workflow using UAS can be summarized as shown in Figure 5. Figure 5. Simplified end-to-end Cisco Ultra Services Platform day-0 deployment workflow 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 23

14 Table 3 summarizes the benefits of each UAS module. Table 3. Simplified end-to-end Cisco Ultra Services Platform day-0 deployment workflow Main Functionalities Values AutoDeploy Site/POD/VIM inventory management Allows customers to centralize the management of multiple VIM domains or sites VNF cataloging Simplifies the management of multiple VNF based on use cases VNF inventory management Provides centralized view of available VNFs for deployment and tracking active deployments AutoIT OSP VIM installation automation Automates the installation of RedHat under-cloud and over-cloud VIM provisioning for VNF deployment VNFD composition Automates the VIM provisioning to reduce errors and ensures smooth VNF deployment Drastically reduces errors in composing complex VNFDs in fraction of time AutoVNF VNFM (ESC) orchestration Automates the deployment and configuration of VNFM through simple workflow while reducing potential errors UWS VNF orchestration GUI-based automation, orchestration, & monitoring workflows Automates the deployment and configuration of VNF through simple workflow while reducing potential errors Provides intuitive GUI for various Day-0 & Day-N end-user workflows It should be fairly straight-forward to imagine what the deployment workflow would look like without the cloud installation automation and/or VNF deployment automation. Each and every step would be much more timeconsuming with much higher probability for human errors. Any misstep during the installation, provisioning, or deployment would result in non-functional VNF in error state and would require complete un-deployment and redeployment of the VNF. Cisco Ultra Services Platform operational monitoring Monitoring USP as a mobility application for example, SAEGW, epdg,, and so on mostly does not change compared to the classic ASR 5500 monitoring of the StarOS as described in the ASR 5500 System Administration Guide using Simple Network Management Protocol (SNMP) notification and bulk statistics. Operators can continue to use the same monitoring method to monitor and measure gateway faults/events and performance statistics as they already do. However, monitoring USP as a VNF within a vertically preintegrated system (for example, Ultra M) requires additional visibility from different management domains. Because USP is a VNF running on commercial off-the-shelf servers, instead of monitoring the ASR 5500 hardware, operators can monitor the servers in a much similar way using the Cisco UCS SNMP MIB and SYSLOG. (See Figure 6.) Figure 6. EPC solution stack comparison 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 23

15 If operators want to monitor the complete system starting from servers up to the application, UAS provides a consolidated view of the system, aggregating and proxying faults/events from each layer in the solution stack. This is straightforward to set up and configure if the entire solution stack is deployed in a fully integrated manner such as Ultra M. Using Ultra M as a reference deployment model, the health monitoring service running in an AutoIT virtual machine has different plug-ins to communicate with different layers in the solution stack based on each layer s supported protocol, while exposing the data as NETCONF events, SNMP traps, and SYSLOG. In this model, UAS becomes the single point of data source for all southbound entities. (See Figure 7.) Figure 7. Ultra M system monitoring This model is especially helpful because each layer of the solution stack supports different ways to collect and expose the data. For example, because OpenStack is specifically intended for a modern cloud-based software and ubiquitous infrastructure, OpenStack only provides REST APIs to poll and receive the operational state information such as KPIs and health status. Similarly, other cloud deployment automation and orchestration entities such as VNF-EM, AutoVNF, and ESC only support NETCONF and/or REST APIs to expose VM, VNF, and VNFM operational data. There are several NFV-appropriate monitoring solutions in the industry for example, ZenOSS and Splunk that can extract data from all stacks and serve as an aggregation point for visualization and monitoring. However, it is not uncommon to find operators whose Network Management Service (NMS) systems are still based on legacy SNMP and want to maintain the continuity of their monitoring practices. In such cases, UAS or more specifically, the Ultra M manager running the AutoIT service can abstract faults/events and KPIs from southbound entities by exposing them as SNMP traps based on the new CISCO-ULTRAM-MIB. The Ultra M manager periodically communicates with its monitored entities (Cisco UCS servers, RedHat OSP, ESC, VNF-EM, and AutoVNF) to check their health states and sends an event trap to the external NMS nodes with the following information: Fault domain: hardware (Cisco UCS, L2 switches, L3 routers), VIM orchestrator (RedHat OSPD), VIM (RedHat OSP overcloud), UAS, VNFM (ESC), VNF-EM, VNF (USP) Fault severity: informational, alert, major, critical, emergency Fault class: service failure, configuration, security violation, hardware failure, resource threshold, resource usage, network connectivity, other NFVI identity: unique ID of the Ultra M POD Fault source: unique ID of the failed resource within the Ultra M POD Fault creation time: Date/Time of when the fault occurred 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 23

16 Using this method, fault/event notifications are generated for monitored running service failures in OpenStack controllers, compute nodes, storage nodes, ESC instances, AutoVNF instances, and VNF-EM instances. The Ultra M manager also uses the OpenStack Ceilometer telemetry service for creating and generating Threshold-Crossing Alarms (TCAs) based on user-level usage data. Details on event/alarm severity and monitored services can be found in UltraM Solutions Guide, Release 5.5. Examples of day-n automation Automated manual processes such as cloud installation, provisioning, and VNF deployment are all critical day-0 capabilities. Equally critical are day-n capabilities for common manual processes for the solution, which are naturally executed more frequently. UAS provides several day-n process automations including: VNF software upgrade: AutoDeploy provides automated workflows for upgrading USP VNF software in a stateless manner. The upgrade involves undeployment of the VNF and deployment of the VNF with newer software while preserving critical operational data stored in OpenStack volumes such as Charging Data Records (CDRs). As mentioned in a previous section, AutoDeploy can onboard multiple USP VNF ISO images, and its configuration file contains the VNF package information with specifics of an ISO image. VNF software patching: AutoDeploy provides automated workflows for patching UAS and ESC software without stopping the service or restarting the VM instances. VNF graceful shutdown and restart: To free up the resources or to scale down, it s important to orchestrate the operation gracefully such that there is no effect on subscriber traffic. VNF-EM along with control functions provides an API for mobile operators to gracefully shut down the VNF or restart, where VNF-EM triggers this action using VNFM only when all the subscriber sessions are drained out. VNF maintenance mode: This is required to perform certain maintenance operations such as replacement of failed hardware. VNF-EM provides an API to suspend a VDU in a coordinated way to make sure the event is not treated as failure instead of a maintenance operation. These services can be brought back into service by invoking resume operations on suspended VDUs. This allows VNF-EM to bootstrap the newly created VDUs and reinsert them as part of service instead of bringing up a VM. Customer deployment examples This section describes multiple modes of USP VNF deployment using UAS in different environments. Although customer names cannot be revealed to protect confidentiality, all deployment modes are based on real customer requirements. Full-Stack Ultra M This deployment mode involves deploying the full-stack Ultra M B1.0 system, which consists of Cisco UCS C240 servers, Cisco Nexus leaf and spine switches, RedHat OSP10 hyperconveged, ESC 2.3.2, and USP In this deployment model, the customer would use all the UAS modules for bringing up the system into a service-ready state: 1. AutoDeploy is deployed by the installation script and is first used to define the VIM domain and onboard one or more USP ISO images. AutoDeploy then triggers the OSPD (undercloud) VM instantiation on the same node and uses it to install the VIM (overcloud). Three OSCs are installed, and all servers are configured to run various OpenStack services serving as compute or storage nodes. 2. AutoIT is deployed by the installation script and provisions various resources through the VIM for VNF onboarding. 3. AutoDeploy starts AutoVNF cluster. 4. AutoVNF provisions required OpenStack resources to deploy ESC and VNF Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 23

17 5. AutoVNF deploys the ESC cluster. 6. ESC deploys the VNF-EM cluster based on dep.xml file. 7. ESC passes the full VNFD to VNF-EM. 8. VNF-EM constructs rest of the VNF through ESC using resources provisioned by AutoVNF in step AutoVNF bootstraps newly created VM instances and applies relevant configurations. 10. If multiple VNF deployments are needed, AutoDeploy repeats step 3 through 8. Standalone AutoVNF as OSS subcomponent This deployment mode involves deploying a cluster of AutoVNF only in standalone mode. AutoVNF interfaces with an external OSS system through the NETCONF or REST APIs. This mode is typically for telecommunications private cloud environments where third-party NFVO already exists. 1. Northbound OSS invokes API exposed by AutoVNF to deploy a new VNF instance. 2. AutoVNF deploys the ESC cluster using precreated and preallocated OpenStack resources. 3. ESC deploys the VNF-EM cluster based on dep.xml file using precreated and preallocated OpenStack resources. 4. ESC passes the full VNFD to VNF-EM. 5. VNF-EM constructs rest of the VNF through VNFD. 6. AutoVNF bootstraps newly created VNF-Cs and applies relevant configurations. 7. Northbound OSS invokes API exposed by VNF-EM to make any changes to the VNF after deployment. AutoVNF in this deployment mode does not directly interface with the VIM; therefore, it is not involved in the provisioning of OpenStack resources for ESC and VNF deployments. These resources would be precreated and preallocated by the existing NFVO. AutoVNF running in this mode is acting as a subcomponent of telecommunications OSS system specialized in USP VNF day-0 deployment by facilitating the VNF deployment with a relevant VNFD and applying relevant configurations to VNF-Cs. AutoVNF would subsequently remain in the OSS layer for any day-n VNF changes. Standalone AutoVNF as a VNFD composer This deployment mode involves deploying a single instance of AutoVNF only in standalone mode. The single instance is only used to compose the VNFD offline and therefore does not require high availability. This mode is typically for telecommunications private cloud environments where a third-party NFVO or another type of deployment automation tool already exists (for example, Ansible). Because AutoVNF in this deployment mode is strictly for offline use, it can be used over and over again to generate different VNFDs based on the use case and configuration. Because AutoVNF would serve no other functions in this mode, the instance can be destroyed after VNFD generation is completed. Evolving Ultra Automation Services (UAS) Looking beyond the day-0 and day-n automation capabilities described in this white paper, there are additional high-valued automation services in the roadmap. These include, but are not limited to: AutoQA: Provides a wide selection of automated test cases grouped into relevant use cases, particularly in the functional area. The goal is to drastically reduce the time required to validate the quality of a new release before deploying into a production environment Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 23

18 AutoSLA: Measures KPIs from application and NFVI to make predefined LCM actions. One of many potential use cases is automatic scale-in and scale-out. Predefined LCM actions are based on userconfigured policy involving near-real-time reading of relevant KPIs. AutoIV: Provides virtualized testing instruments such as call generation tools and packet analyzer tools for troubleshooting and software quality verification purposes. Multilevel event/fault correlation: Provides meaningful correlated view of major events and faults rather than reporting multiple events and faults individually from various parts of the solution stack. The goal is to reduce the time required to precisely identify the cause of the event/fault quickly by intelligently deducing events from directly monitored events. Stateful VNF software upgrade: Provides automated VNF software upgrade workflow using Interchassis Session Recovery (ICSR) and Service Recovery Protocol (SRP) for stateful software upgrades. Using this upgrade method allows nondisruptive operation of service using active and backup VNF pairs. Complete NFVI monitoring: Expansion of the current health monitoring framework to include non-vnf components such as L2 switches and L3 routers, providing complete visibility of the service-bearing components ranging from NFVI to application domains. Support of additional VIMs: Expansion of the current portfolio of supported VIMs to include additional OpenStack distributions for example, Mirantis and VMware vsphere. Integration with open-source cloud infrastructures: Provides cloud infrastructure plug-ins to allow transparent integration with open-source cloud management projects such as Open Network Automation Platform (ONAP). NFVI software upgrades and updates: Abstracts NFVI software management layers of standalone Cisco UCS and Cisco Nexus switches to provide automated software upgrades and updates, including firmware, through AutoDeploy on per-site and/or per virtual rack basis. Helpful reference documents The following documents contain more details about the automation and orchestration capabilities described in this white paper, including configuration steps: Ultra M Solutions Guide Cisco Ultra Services Platform Deployment Automation Guide Network Function Virtualization (NFV): Management and Orchestration Network Function Virtualization (NFV): Terminology for Main Concepts in NFV Frequently asked questions General VNF orchestration Q. Do you require Network Services Orchestrator (NSO) to deploy Ultra VNF? A. No. VNF-EM uses Network Element Drivers (NEDs) to dynamically program VNF day-n configuration. Q. Can my customer s NFVO deploy Ultra VNF? A. Yes. In the absence of an NFVO in the environment, UAS orchestration modules (AutoDeploy, AutoIT, and AutoVNF) can be used to deploy Ultra VNF Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 23

19 Q. Can USP be orchestrated using OpenStack Heat template? A. While it is technically possible to orchestrate the deployment of USP using an OpenStack Heat template, it is important to remember that doing so would result in losing the Day-N lifecycle management capabilities altogether. OpenStack Heat is a simple YAML-based template for deployment of one or more virtual machine instances, but completely lacks the lifecycle management capabilities such as monitoring and healing. Q. Why is AutoVNF mandatory when deploying USP? A. Minimally, AutoVNF running in the stand-alone mode is mandatory in deploying the USP because composing a full USP VNFD is highly error-prone if attempted to do so manually. Furthermore, as VM instances are deployed during the bring-up of the USP, VM instances need to be bootstrapped for service-level configuration in order to realize the end-to-end VNF deployment automation. VNF-EM Q. Is Ultra VNF-EM compliant to the ETSI NFV-MANO specification? A. Yes. VNF-EM uses the Ve-Vnfm interface to communicate with the VNFM. From the VNFM point of view, it is interfacing with a VNF. Q. Can you deploy Ultra VNF without the VNF-EM? A. No, Ultra VNF deployment requires VNF-EM whether UAS is used in conjunction or not. Q. Is VNF-EM equivalent to an Element Management System (EMS)? A. No. VNF-EM is a VNF-specific lifecycle manager that only manages the VNF, whereas an EMS is often an OSS entity that manages a variety of network elements in physical and virtualized forms from multiple vendors. The Cisco EMS solution is Cisco Prime Mobility and would interface with VNFM over Ve-Vnfm-Em interfaces. Q. Is VNF-EM considered part of the UAS or VNF? A. VNF-EM is a part of the USP VNF, meaning it is a VNF-C, just like CF and SF are. The interaction between VNF-EM and VNFM uses the Ve-Vnfm-Vnf interface. Q. Where do VNF-EM virtual machine instances get deployed? A. Three (1:2 redundancy) VNF-EM virtual machine instances get deployed on the same physical hosts where OpenStack controllers and other management entities for example, ESC clusters reside. Q. Does VNF-EM support day-n configuration changes to VNF? A. Yes, VNF-EM uses NEDs to dynamically program VNF day-n configuration. In case of Ultra Gateway Platform use cases, the StarOS CLI NED can be used if the NED has adequate configuration modeling that suits your customer s needs. Q. How many VNFs can a single-cluster instance of VNF-EM support? A. At the time of this writing, as of the 5.7 release, VNF-EM only supports one VNF. In the near future, in 2018, VNF-EM will be capable of supporting multiple VNFs, enabling a single cluster of VNF-EM instances to be shared by multiple VNFs. Q. Does VNF-EM support automatic scaling in and/or scaling out? A. At the time of this writing, as of the 5.7 release, VNF-EM does not support autoscaling. In the near future, in 2018, VNF-EM will support user-configurable automatic scaling based on relevant application-level KPIs. Q. Does VNF-EM have visibilities into NFVI resource utilization information? A. No. Because VNF-EM is a part of the VNF, it does not and should not interface directly with the VIM to maintain and respect management autonomy. The only way to make this happen is through the VNFM if VNFM reexposes VIM-level APIs to the VNF-EM Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 23

20 Q. Which VNFMs does VNF-EM support as of today? A. At the time of this writing, as of the 5.7 release, VNF-EM supports the VNFM proxy plug-in for Cisco Elastic Services Controller (ESC). Additional plug-ins can be easily developed with the library and configuration modeling information released to the developer. Q. Does VNF-EM support Cisco Policy Server (CPS)? A. At the time of this writing, as of the 5.7 release, VNF-EM does not support CPS lifecycle management. Development is currently under way to integrate VNF-EM with CPS so that the same VNF-EM instance can manage both Ultra Gateway Platform (UGP) and CPS. This work is projected for completion in early Q. Why does a single cluster of VNF-EM instance require three VM instances? A. The cluster is composed of three VM instances running in 1:2 redundancy mode using ZooKeeper. This is so that VNF-EM is not exposed to a Single Point Of Failure (SPOF) any time during its lifecycle. Because VNF- EM manages the VNF lifecycle, its criticality of the availability warrants this design. Ultra Automation Services Q. What are the purposes of the UAS? A. Two main purposes of the UAS are (1) automate the NFVI and VIM installations and (2) automate the deployment of the Ultra VNF without the dependency of an external NFVO. UAS empowers Ultra VNF to be self-sufficient from day-0 deployment to day-n operation. Q. Can UAS adequately function as an NFVO? A. Yes, but only specifically for Ultra VNF. All services NFVO would provide in deployment workflows are provided by UAS. Q. Is UAS only compatible with Ultra M? A. At the time of this writing, as of the 5.7 release, the full stack of the UAS is only validated for the Ultra M deployment model.no This does not mean that UAS does not work in other environments. Check with the MCBU product management team to see if your customer s environment would be compatible with UAS. Q. Is UAS mandatory in deploying Ultra VNF? A. At least one instance of AutoVNF running in standalone mode is required for Ultra VNF deployment for VNFD composition and VNF-C configuration bootstrapping. Q. What is Ultra web services (UWS)? A. UWS is a graphical user interface of the UAS and VNF-EM. UWS uses APIs exposed by UAS and VNF-EM to graphically render deployment workflows and operational monitoring of the VNF. Q. Can UAS be used for Cisco Policy Server (CPS) orchestration and automation use cases? A. At the time of this writing, as of the 5.7 release, UAS does not support CPS automation and orchestration. Development is currently under way to integrate UAS with CPS so that the same automation and orchestration framework can be used to orchestrate both UGP and CPS deployments. This work is projected for completion in early Q. Can UAS be patched without affecting the service? A. Yes, patching UAS RPMs does not affect the service availability in any way. Q. Can UAS be upgraded without having to upgrade the VNF software? A. Yes, but the version of the UAS must be compatible with other software components for example, ESC with which it interfaces Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 23

21 VNF structure and hierarchy VNF can contain one or more VM virtual machine instances. In case of USP, which is an example of a complex VNF, there are multiple VNFCs serving different functions. Each VNFC is based on a VDU with a specific set of properties such as image, connection points, virtual links, and so on. After VNFC is deployed, it is referred to as VNFCi. (See Figure 8.) Figure 8. VDU and VNFC VDU, in a practical sense, can be viewed as a VM template. Single VDU can be used to create multiple VNFCs, as long as VNFCs share the same day-0 artifacts, but require different initialization variables such as name, virtual slot number, and so on. VNFCs based on the same VDU are referred to as constituent VDUs (cvdus). (See Figure 9.) Figure 9. Constituent VDUs One or more VNFCs can be grouped together for the purpose of deployment, scalability, or redundancy. A group of one or more VNFCs is referred to as an element group. An example of an element group is the di-chassis, which consists of a pair of control functions and a number of session functions. (See Figure 10.) 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 23

22 Figure 10. Element group Failure effects of Cisco Ultra Services Platform software component As one can expect, many solution components run in redundant mode. Failures in the management plane are not expected to affect the control and data planes of the VNF, although subsequent LCM activities and user workflows might be limited. (See Table 4.) Table 4.USP Component Failure Impact Component Redundancy Failure Detection Failure Action(s) Impact to Control-Plane Element Manager (EM) Control Function (CF) Session Function (SF) Elastic Services Manager (ESC) 1:2 ESC VM keep-alive & Internal HA keepalive (Zookeeper) 1:1 ESC keep-alive StarOS HAT 1 N:1 ESC keep-alive StarOS HAT 1 1:1 Internal HA keepalive between two instances Zookeeper: Switchover from Standby to Active ESC: Recover failed instance HAT: Switchover from Standby to Active ESC: Recover failed instance HAT: Switchover from Standby to Active ESC: Recover failed instance Switchover from Standby to Active Manual recovery Impact to Data- Plane Impact to Mgmt- Plane None None None in case of single instance failure; VNF LCM in case of more than one failure None in case of single instance failure; VNF nonoperational if more than one failures None in case of single instance failure; reduction of capacity in case of more than one failures None in case of single instance failure; VNF nonoperational if more than one failures None in case of single instance failure; reduction of capacity in case of more than one failures None in case of single instance failure; VNF nonoperational if more than one failure None None None VM LCM in case of more than one failure AutoDeploy 2 None None Manual recovery None None All user workflows AutoIT-NFVI 2 None None Manual recovery None None Health monitoring service AutoIT-VNF 2 None None Manual recovery None None New VIM provisioning and subsequent VNF (re)deployments 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 23