FlexPod Datacenter with Apprenda 8.2

Transcription

1 FlexPod Datacenter with Apprenda 8.2 FlexPod Datacenter for Platform as a Service with Apprenda 8.2, Cisco ACI 3.1, and Kubernetes Last Updated: May 30, Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 24

2 Contents Introduction... 3 Solution summary... 3 Solution design... 5 Design objectives... 5 Design validation... 5 Design overview... 5 Cisco ACI design... 5 Cisco ACI integration with Kubernetes... 6 NetApp Jenkins Framework... 6 NetApp SnapCenter plug-in integration... 6 Data protection setup... 6 Apprenda platform design... 6 Apprenda platform overview... 7 Apprenda components... 7 Apprenda platform server roles... 7 Apprenda logical design... 8 Applications on Apprenda... 9 Implementation topology... 9 Infrastructure design Infrastructure components and versions Infrastructure setup for the Apprenda platform Infrastructure virtual machine sizing Infrastructure component deployment Apprenda setup details Apprenda application requirements Integration of Apprenda with Cisco ACI NetApp SnapCenter and NetApp Service Level Manager installation Database cloning with the Apprenda SnapCenter plug-in Continuous integration with Jenkins on NetApp ONTAP Accelerating the continuous integration process Continuous delivery of databases using the Apprenda SnapCenter plug-in Conclusion For more information Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 24

3 Introduction The FlexPod solution is a predesigned, integrated, and validated data center architecture that combines Cisco Unified Computing System (Cisco UCS ) servers, Cisco Nexus Family switches, and NetApp All Flash Fabric-Attached Storage (FAS) arrays in a single, flexible design. FlexPod supports several different hypervisor options and bare-metal servers and can be sized and optimized based on customer workload requirements. The FlexPod Datacenter with Apprenda Release 8.2 provides customers and business owners with a ready-to-use continuous integration and continuous delivery (CI/CD) solution with a shortened lead time for application development and deployment on the Apprenda container-management platform. NetApp ONTAP is an integral part of the solution, delivering a standard data management platform that provides persistent data stores to the container environment for data generated during the software development lifecycle and release into production. The solution provides a secure, scalable, and seamless way to consume infrastructure on demand with zero-touch storage and more control over data for developers. Solution summary FlexPod Datacenter with the Cisco Application Centric Infrastructure (Cisco ACI ) platform introduces the application-centric policy model. In this model, connectivity is defined by consolidating endpoints (physical devices, virtual machines, and containers) into endpoint groups (EPGs), allowing control of communication between EPGs based on application requirements. Apprenda is a container management platform that empowers any organization to operate a self-service cloud environment for building, deploying, and running custom applications. Apprenda integrates with the Cisco ACI platform to provide automation of network policies and application segmentation. Apprenda enables developers to answer a few simple questions about the network requirements for their applications and then automates the configuration of Cisco ACI on their behalf. Cisco ACI and Apprenda together create a policy-based platform-as-a-service (PaaS) solution to provide a very secure and advanced enterprise-class application development platform. Apprenda integrates with Cisco ACI open policy interfaces to free developers and IT staff from the manual constraints of network configuration and to achieve complete isolation of application tiers and data within shared infrastructure. Apprenda provides plug-ins to integrate with development tools such as the GitLab and Jenkins open-source tools. The underlying NetApp ONTAP data platform on NetApp All Flash FAS (AFF) provides persistent on-demand data volumes for stateful Docker containers. These containers are managed by Apprenda as source code and binary repositories and as image registries and repositories for builds during the continuous integration process using NetApp Service Level Manager (NSLM), also known as ONTAP APIs. Figure 1 shows the stack for the Apprenda Cloud Platform (ACP) on FlexPod Datacenter and integration with NetApp ONTAP at the data layer Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 24

4 Figure 1. FlexPod Datacenter with Apprenda for DevOps Apprenda manages the applications developed on the cloud platform using the NetApp Jenkins Framework to accelerate the continuous integration process. The framework uses NSLM representational state transfer (REST) APIs to perform all the data management and protection functions at the data layer. The NetApp Jenkins Framework on Apprenda is designed to allow DevOps administrators and infrastructure engineers to set up the development environment with GitLab, Jenkins, and NetApp ONTAP at the start of the project. Developers use the custom user interface that is part of the framework to create instant user workspaces at scale with improved data control and high infrastructure efficiency, allowing the business owners to provision more workloads at lower infrastructure costs. Apprenda and NetApp SnapCenter integration allows developers to instantly provision a copy of the production database during user acceptance testing (UAT) to test the changes before releasing the application to the production environment. Apprenda integrates with SnapCenter REST APIs to perform data management and protection functions to provide developers with more agility, enabling them to create a production-like copy of the database from the cloud platform with zero-touch storage during the application deployment process. After the successful build is deployed on the Apprenda Cloud Platform, the Apprenda plug-in allows developers to quickly provision a production-like database for testing using SnapCenter REST APIs. This integration mitigates risk for the production database and allows developers to test their changes against real data and find any problems before going to production. Developers can instantly revert back to a clean copy of the database after it becomes polluted during the test process. The Apprenda plug-in provides several options from the developer portal to allow the developer to change states during the database application testing process. The NetApp Jenkins Framework using a Kubernetes cluster on the Apprenda platform together with native ONTAP features improves developer productivity by up to 30 times and reduces build time by up to 50 percent, therefore providing up to 40 times greater infrastructure efficiency in the continuous integration process Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 24

5 Solution design This section describes the design of the FlexPod Datacenter PaaS solution with Apprenda 8.2. Design objectives The FlexPod Datacenter with Apprenda 8.2 solution provides secure automated deployment of custom-built applications in containers for preproduction and production purposes. The solution addresses important challenges that technology organizations with large-scale custom application development face every day: Long negotiations among IT staff, network operators, database administrators (DBAs), and developers about application deployment are reduced through automation. Network segmentation is automated at deployment time based on the business requirements and the application servicelevel agreements (SLAs) defined in the application configuration. Network segmentation automation removes human error from the deployment process and achieves true secure isolation of applications as required by the business. Preproduction testing of applications is automated through a comprehensive CI/CD pipeline, which includes: Build automation, version management, and binary repository management (or artifactory) Automated policy-based deployment of applications from the build server to the container platform Build server optimization and increased availability through containerization Rapid database cloning at application deployment time to facilitate preproduction validation and improve the quality of testing Design validation To validate the solution, a mixed-architecture application was used to demonstrate secure containerization across different container clusters. The test application consists of a Microsoft.NET Windows Communication Foundation (WCF) service, which relies on a Microsoft SQL database, and a Node.js microservice, which relies on a MongoDB cluster deployed on the Amazon Web Services (AWS) platform. During the development process, the Whatif application uses stateful containers or pods in a Kubernetes cluster managed by Apprenda. These containers mount persistent data volumes on ONTAP Network File System (NFS) Version 3 in an automated continuous integration process using development tools such as GitLab, Jenkins, and Docker Registry. In this deployment model, network segmentation in Cisco ACI is automatically created based on the application SLAs configured in Apprenda. When a successful build of the Whatif application is deployed on the Apprenda Cloud Platform as a traditional.net application, it uses a MySQL database to demonstrate rapid database cloning of the production database to support application testing. The deployment is conducted as a part of the continuous deployment automation process. Design overview To deliver a successful CI/CD solution on the FlexPod Datacenter infrastructure, the design described in the following sections was implemented and validated. Cisco ACI design To deploy the Apprenda application, a dedicated tenant called FPV-Lassen is configured in the Cisco ACI infrastructure. The Cisco ACI infrastructure is used to map components of the solution to various application profiles and EPGs. The core applications, storage access, and common services such as Microsoft Active Directory (AD) and Domain Name System (DNS) are secured by using appropriate contracts between various EPGs. Communication from the Apprenda application to various platform components and to deployed containers and applications is enabled and verified during the various validation steps. The details of 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 24

6 the Cisco ACI layout, including tenants, application profiles, EPGs, and contracts, are discussed in the Infrastructure design section later in this document. Cisco ACI integration with Kubernetes To integrate Kubernetes with the Cisco ACI platform, a series of tasks are performed on the Kubernetes cluster. After the Kubernetes integration is complete, the Cisco Application Policy Infrastructure Controller (APIC) can be used to view the Kubernetes infrastructure layout in the Cisco ACI fabric. To integrate Kubernetes with Cisco ACI using the Cisco Network Interface plug-in, a second tenant, called fpv-kubernetes-vx, is configured. This tenant is used by the Kubernetes cluster to deploy container-based applications. Various network resources are required to provide capabilities to the Kubernetes cluster, including several subnets and VLANs. These details are discussed in the Infrastructure design section later in this document. NetApp Jenkins Framework To deliver CI/CD capabilities in the solution, the NetApp Jenkins Framework is set up and configured in the Kubernetes cluster managed by Apprenda. The framework uses common development tools such as GitLab Community Edition (CE) Version 10.5 for source-code management (SCM), Jenkins Version for private and integrated builds in the continuous integration process, and an internal Docker Registry to store all the image and binary files that mount persistent data volumes from ONTAP over NFS. These data volumes are provided on demand for various Jenkins jobs in the pipeline. Continuous builds are automated and managed using Jenkins pipelines from the Flask-based user interface for the Whatif application. Developer workspaces are created instantly, at scale, using ONTAP features such as NetApp FlexVol volumes, Snapshot copies, and FlexClone volumes, by using NSLM for Whatif source code, artifacts, and binaries. The NetApp Jenkins Framework installs and runs GitLab CE, Jenkins, and the Docker Registry as a pod in the Kubernetes cluster. NetApp SnapCenter plug-in integration In this deployment, the Apprenda SnapCenter plug-in is configured with appropriate settings to enable the use of SnapCenter REST APIs to manage the data layer from the Apprenda Cloud Platform. SnapCenter discovers the MySQL database for appropriate resource-group settings to create an application-consistent snapshot of the production database. The database is created, cloned, and restored for testing and development using this plug-in. The Whatif application is deployed on the Apprenda Cloud Platform, and the three custom properties configured on the Whatif application in the Apprenda developer portal clone original, restore clone, and keep existing are verified. Data protection setup The Apprenda SnapCenter plug-in creates new application-consistent snapshots on demand from a production database and clones the database instantly. The plug-in also creates a snapshot of the database clone to protect and restore the database to a clean state after it becomes polluted from the testing process. The plug-in supports automatic creation of checkpoints or snapshots by the NetApp Jenkins Framework for every check-in of code changes in GitLab and for every successful merge operation from the continuous integration build process by the Jenkins pipeline. The old checkpoints or snapshots for GitLab and continuous integration builds data volumes are deleted as a background process. The number of snapshots can be customized for each data volume in the NetApp Jenkins Framework. Apprenda platform design Apprenda is a comprehensive, policy-based, application deployment platform with capabilities to containerize both cloud-native and traditional application workloads. It uses multiple container orchestrators, including Kubernetes, to support deployment of a wide variety of applications Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 24

7 Apprenda platform overview Apprenda is an application runtime engine that supports traditional and cloud-native workloads for.net and Java applications in which application components and tenants are both first-class citizens of the architecture. Apprenda aggregates Microsoft Windows and Linux infrastructure into a single logical layer that is a self-service computing platform for one or more development teams. Apprenda offers various APIs and frameworks that provide capabilities such as message brokering, distributed caching, and application metering. More specifically, Apprenda provides: A multiple-machine application server and runtime engine for hosting and managing.net and Java applications Instrumentation and transformation engines that virtualize application components A user and tenant management layer that applications can use so that user and account information can be shared across applications An API and set of associated services that provide fundamental building blocks for user and account management, provisioning, caching, and message brokering A set of web portals and tools to manage applications and tenants The capability to attach and manage multiple Kubernetes clusters Apprenda components The Apprenda platform is composed of the following: Apprenda fabric: The server fabric provides the platform's core capabilities. For details, visit Software developer kit (SDK): The SDK consists of APIs, documentation, samples, and local development and deployment tools and integrated development environment (IDE) hooks. These components are designed to aid in the development process and allow a developer to emulate the running of an application on a live Apprenda instance. System operations center (SOC): The SOC is a web portal for back-end management of network infrastructure, global settings, application components, and the server fabric. Account portal: The account portal is a web portal that provides tenants with account management functions. This portal can be branded by the organization. When a new organizational unit signs up on the platform, it and its users are provisioned at the Apprenda level, which gives the users access so they can manage their Apprenda account. Developer portal: The developer portal is a web portal for product management. It allows the organization s technical assets to manage application uploading, deployment, and configuration. This portal can be branded by the organization. Apprenda platform server roles When the Apprenda platform is installed, the platform executables are positioned on a collection of servers according to the OS and the hosting capabilities of the server. These executables interact with one another in a peer-based fashion to deliver a single logical deployment and execution runtime engine. The following components make up the Apprenda platform: Microsoft Windows OS: Application server Web server Load manager Platform coordination Cache Storage control services host Linux OS: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 24

8 Linux server (optional) Storage: Microsoft SQL instance Oracle Relational Database Management System (RDBMS; optional) Platform repository Kubernetes cluster (optional for the platform, but required for this deployment) Note: Roles that are not essential to the functioning of the platform are marked as optional. Figure 2 provides a high-level overview of the platform server and storage roles. For an in-depth discussion of the various components, visit Figure 2. Apprenda and NetApp components in the FlexPod Datacenter setup For this validation, all the Apprenda platform repositories and Microsoft SQL databases are configured on NetApp ONTAP Common Internet File System (CIFS) shares Apprenda, Binds, SOC, Application, and MSSQL, as shown in Figure 2.. NSLM Release 1.0 and SnapCenter Release 4.0 are configured on the Red Hat Enterprise Linux (RHEL) 7.2 and Windows nodes respectively. NSLM is an off-device REST API implementation for performing various data management functions on the file systems created during the continuous integration process. SnapCenter is a windows-based NetApp tool that provides application-consistent protection and cloning capabilities for database testing and development during the continuous deployment process. Apprenda logical design The Apprenda platform in the current implementation manages four container runtime processes: Microsoft Windows containers (deployed on the multirole coordination node, application, and cache node virtual machines) Java containers (deployed on the Linux application servers) Docker containers (deployed on the Linux application servers) Kubernetes (deployed on the Kubernetes nodes) Depending on the architecture of the application component, all.net components are routed to Windows containers, Java applications go to Java containers, Docker images are orchestrated in the Docker runtime engine, and pod specifications trigger orchestration on the Kubernetes cluster. In the case of hybrid multicomponent applications, the components of the same application can end up on different container runtime engines. The routing of traffic to the application and across the components 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 24

9 is facilitated through the Layer 7 load manager. The load manager is the main entry point into the platform. When a request comes in, the load manager uses its internal up-to-date information about the location and the number of the application containers to route the request to the appropriate server and port. The load manager also serves as the TLS termination point. The internal traffic can be encrypted by the platform, and the decision to do that needs to be made at the time of platform installation. See the Apprenda setup details section later in this document. Apprenda runs a distributed cache cluster for guest applications, deployed on coordination nodes 1 and 2 in this design. Applications on Apprenda An application running on Apprenda is composed of a combination of the following:.net Microsoft Internet Information Services (IIS) hosted ASP.NET web application (Model-View-Controller [MVC], WebForms, etc.) Self-hosted Microsoft.NET WCF web services Standalone Microsoft Windows services Java Java web applications (Web Archive [WAR] or Java Archive [JAR] deployment model) Java web applications (Java 2 Platform, Enterprise Edition [J2EE]) Containers Docker images Kubernetes Deployment specification (YAML) Databases Microsoft SQL Server database Oracle database For additional information, see the Apprenda standard reference architecture at Implementation topology The lab used for the validation of the solution meets the minimal requirements for this solution to provide uptime and highavailability, with the exception of the load manager. For production environments, the use of at least two load manager nodes is recommended. For the purposes of the current implementation, the Apprenda platform consists of the following components: Microsoft Windows nodes Active Directory 1 Active Directory 2 Load manager Coordination node 1 (platform coordination node, cache node, and application server) Coordination node 2 (platform coordination node, cache node, and application server) Coordination node 3 (platform coordination node) NetApp SnapCenter Microsoft SQL Server 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 24

10 Linux nodes Linux application server 1 Linux application server 2 Kubernetes master Kubernetes etcd Kubernetes worker Note: All servers used in this implementation are virtualized. The solution can also run on bare-metal or mixed infrastructure. If the Kubernetes cluster is deployed on physical machines, the Cisco Network Interface plug-in must be installed in VLAN mode. Repository CIFS share on a NetApp storage virtual machine (SVM) Infrastructure design The infrastructure to host Apprenda is built on the FlexPod configurations and best practices identified by previous Cisco Validated Designs. These include hardware and software compatibility between all components and configuration best practices for each of these components. The base FlexPod infrastructure used to deliver the FlexPod Datacenter with Apprenda 8.2 solution is discussed in the following document: The FlexPod with Cisco ACI design consists of the following data center hardware and software components: Cisco Nexus 9000 Series Switches deployed in Cisco ACI mode Cisco APIC Release 3.1 and Cisco UCS Release 3.2 VMware vsphere Release 6.5u1 hypervisor Cisco UCS B-Series and C-Series M4 and M5 blade and rack servers NetApp AFF 300 NetApp ONTAP Release 9.3 NFS, CIFS, and Small Computer System Interface over IP (iscsi; for SAN boot only) based storage access Figure 3 shows the physical connectivity of the hardware components listed here Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 24

11 Figure 3. FlexPod Datacenter with Cisco ACI topology The solution is configured with the following design choices to deliver a flexible architecture: Computing and storage designs validated in FlexPod with the Cisco ACI solution Ethernet connectivity through the Cisco ACI fabric for both virtual machine traffic and NFS, CIFS, and iscsi storage access iscsi-based stateless boot of computing nodes (VMware ESXi) Deployment environment set up as a combination of the following: VMware infrastructure and Microsoft Windows and Linux virtual machines for hosting application components Microsoft Windows 2012 and RHEL 7.4 operating systems Docker Enterprise Infrastructure components and versions Table 1lists the versions of the components used in the solution. Table 1. Infrastructure component versions Component Cisco APIC M2 appliance Cisco Nexus 93180YC and 9336PQ Cisco UCS Software Release 3.1(1i) Release 13.1(1i) Release 3.2(2d) 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 24

12 VMware vsphere Release 6.5 Update 1 Apprenda Application Platform Release 8.2 NetApp AFF A300 NetApp ONTAP Release 9.3 NSLM Release 1.0 NetApp SnapCenter Release 3.0 Jenkins framework for NetApp ONTAP Release 2.0 Trident Release Microsoft Windows virtual machines Microsoft Windows Server 2012 R2 Linux virtual machines Red Hat Enterprise Linux Release 7.4 Infrastructure setup for the Apprenda platform To deploy the Apprenda platform in FlexPod with Cisco ACI infrastructure, the following configurations were implemented in the existing architecture: VLAN and IP address ranges were allocated and contracts were defined to enable communication between components. A new SVM, called Lassen-SVM, was deployed. A new VMware cluster, called FPV-Lassen, was set up. Two new tenants were configured in Cisco APIC: FPV-Lassen and fpv-kuberbetes-vx. Note: The cluster and tenant names used in this document follow lab naming conventions and do not need to be used in the customer environment. For example, the name FPV-Lassen follows an internal lab naming convention used during validation. Customers are encouraged to use easy-to-remember descriptive names for the various elements. VLAN and IP address ranges Table 2 lists the VLAN and IP address ranges configured and used to deploy the solution components. Table 2. VLAN and IP address ranges Use SVM VLAN Cisco UCS VLAN IP network NFS /24 CIFS /24 Tenant FPV-Lassen 223 APIC-assigned VLAN /24 In-band management Defined in FlexPod with Cisco ACI Defined in FlexPod with Cisco ACI /24 Out-of-band management /24 Kubernetes nodes /24 Kubernetes pods /24 Storage virtual machine A storage virtual machine, FPV-Lassen, is created to meet the storage requirements of the solution, including the use of NFSbased VMware data stores to host all the virtual machines. Management, CIFS, and NFS logical interfaces (LIFs) are configured as shown in Figure 4. and are made available to the rest of the infrastructure. The LIFs used by the ESXi hosts to mount the NFS data stores are configured with a maximum transmission unit (MTU) value of 9000 for improved storage performance. The LIFs used by the application virtual machines to mount the NFS volumes are configured with the default MTU value of 1500 to avoid any issues related to the MTU when implementing storage to the virtual machine. Two 1-terabyte (TB) NFS data stores, lassen_datastore_1 and lassen_datastore_2, are configured and presented to the VMware infrastructure for virtual machine hosting Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 24

13 Figure 4. Storage virtual machine: Lassen-SVM VMware cluster For this deployment, a 4-node VMware cluster is configured using Cisco UCS B200 M5 Blade Servers. The ESXi hosts are configured to boot from SAN using iscsi boot and then mount the two NFS data stores defined on the new SVM: lassen_datastore_1 and lassen_datastore_2. Figure 5. shows the relationship between the VMware cluster, virtual machines, and storage system data store. Figure 5. VMware ESXi cluster and data stores Note: Figure 5 shows only a single hard-disk drive (HDD) while depicting multiple virtual machines (for example, Linux 1 to 6) for ease of illustration. In an actual deployment, each virtual machine has its own dedicated HDD. Cisco ACI tenants To connect various elements of the computing, storage, and application resources, a Cisco ACI tenant called FPV-Lassen is created. Figure 6. shows VLAN-to-EPG mappings for the tenant Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 24

14 Figure 6. FPV-Lassen EPGs As shown in the figure, the NFS and CIFS LIFs from the storage system are mapped in the appropriate EPGs using VLANs assigned to the LIFs. This connectivity is then extended to the virtual machines by creating new EPGs and contracts and then attaching these new EPGs to the virtual machine manager domain as shown in the figure. Virtual machines requiring access to CIFS and NFS storage will be attached to the appropriate port groups NFS-VM and NFS-CIFS using dedicated network interface cards (NICs). A dedicated tenant called fpv-kubernetes-vx is defined in the APIC to define Kubernetes integration with Cisco ACI, and the configuration is derived at the time of Cisco Network Interface setup. Infrastructure virtual machine sizing Table 3 provides sizing details for the virtual machines used in the solution. Connectivity details for these virtual machines are discussed in the Infrastructure component deployment section. Table 3. Infrastructure virtual machine sizing Quantity Role Configuration Platform 2 Active Directory 2 vcpus, 4 GB of RAM, 80-GB HDD, and 1 virtual NIC (vnic) Microsoft Windows Server 2012 R2 Standard 1 Load manager 2 vcpus, 4 GB of RAM, 80-GB HDD, and 2 vnics Microsoft Windows Server 2012 R2 Standard 3 Platform coordination node, cache node, and application server 4 vcpus, 8GB RAM, 80GB HDD, and 2 vnics Microsoft Windows Server 2012 R2 Standard 3 Linux application server 4 vcpus, 8 GB of RAM, 80-GB HDD, and 3 vnics RHEL NSLM 4 vcpus, 8 GB of RAM, 80-GB HDD, and 4 vnics RHEL NetApp SnapCenter 2 vcpus, 4 GB of RAM, 80-GB HDD, and 2 vnics Microsoft Windows Server 2012 R2 Standard 1 Microsoft SQL 8 vcpus, 32 GB of RAM, 150-GB HDD, and 3 vnics Microsoft Windows Server 2012 R2 Standard 3 Kubernetes cluster 4 vcpus, 16 GB of RAM, 100-GB HDD, and 4 vnics RHEL Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 24

15 Infrastructure component deployment Depending on their individual role, the virtual machines listed in Table 3require access to particular resources such as storage systems, CIFS and NFS shares, and DNS. To enable this access, separate NICs and Cisco ACI contracts are set up. Figure 7 shows the NIC configuration for all the virtual machines. Figure 7. Virtual machine NIC configuration Port group and EPG configuration Figure 7 shows a number of port groups configured in the environment to provide access to application and storage networks. The port groups shown in the figure are derived from the EPGs defined in the Cisco ACI fabric on which the EPG policies are pushed to the APIC-controlled virtual distributed switch (VDS) as port groups in the VMware environment. The EPG configuration is as follows: IB-MGMT: This EPG provides access to the in-band management network. This EPG is configured in the tenant common and provides a contract that is consumed by all EPGs across multiple tenants that requires access to Active Directory and DNS services. Apprenda: This EPG is defined in the tenant FPV-Lassen to host the Apprenda application components. This EPG provides and consumes separate contracts to give virtual machines access to the: IB-MGMT network for core services Internet using a shared Layer 3 outside (L3Out) interface Kubernetes cluster virtual machines SVM management and application NFS network CIFS-VM: This EPG is defined in the tenant FPV-Lassen to provide direct access to the CIFS LIFs in the NetApp SVM using static VLAN mapping. NFS-VM: This EPG is defined in the tenant FPV-Lassen to provide direct access to the NFS LIFs in the NetApp SVM using static VLAN mapping Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 24

16 fpv-kubernetes-vx: This port group is configured as part of Cisco Network Interface integration to provide connectivity for the Kubernetes deployment. Note: For more information about the IP addressing using by the virtual machines in the following sections, refer to Table 2 Microsoft Active Directory virtual machines Active Directory virtual machines are part of the in-band management port group and are allocated an IP address from the /24 network. All the virtual machines can access the Active Directory virtual machines by consuming the Cisco ACI contract provided by the IB-MGMT EPG. Load manager and platform coordination node virtual machines The load manager and platform coordination node virtual machines are connected to both the Apprenda EPG and the CIFS EPG. The virtual machines are allocated IP addresses from the /24 and /24 subnets, and the gateway is set to a switch virtual interface (SVI) in the /24 network. These virtual machines have access to the Internet, Active Directory, and CIFS shares. In this deployment example, the platform coordination nodes support additional roles as cache nodes and application servers. Linux application virtual machines Linux virtual machines are part of the Apprenda, CIFS, and NFS EPGs and are allocated an IP address from the /24, /24, and /24 networks. These virtual machines have access to the Internet, Active Directory, NFS, and CIFS shares and SVM management interfaces. NSLM virtual machine The NSLM virtual machine is part of the IB-MGMT, Apprenda, CIFS, and NFS EPGs. It is allocated an IP address from the /24, /24, /24, and /24 networks. This virtual machine has access to the Internet, Active Directory, NFS, and CIFS shares and SVM management interfaces. The virtual machine is connected to the IB- MGMT network to enable routing to the cluster IP address of the NetApp A300. NetApp SnapCenter virtual machine The SnapCenter virtual machine is part of the Apprenda and IB-MGMT EPGs and is allocated an IP address from the /24 and /24 networks. The virtual machine has access to the Internet and Active Directory servers. The virtual machine is connected to the IB-MGMT network to enable routing to the cluster IP address of the NetApp A300. Microsoft SQL virtual machine The Microsoft SQL virtual machine is part of the Apprenda, NFS, and CIFS EPGs and is allocated an IP address from the /24, /24, and /24 networks. The virtual machine has access to the Internet, Active Directory, NFS, and CIFS networks. Kubernetes cluster virtual machines The three Kubernetes cluster Linux virtual machines are part of the IB-MGMT, CIFS, and NFS EPGs and are allocated an IP address from the /24, /24, and /24 networks. Additionally, the virtual machines are configured with an interface in the fpv-kubernetes-vx port group used in the Kubernetes deployment. The interface connected to this port group is set up with subinterfaces as follows: A subinterface is created and assigned an IP address by the Cisco ACI infrastructure using Dynamic Host Control Protocol (DHCP). This subinterface uses the Cisco ACI infrastructure VLAN (VLAN 4093). A subinterface is created and assigned an IP address in the Kubernetes node network VLAN (VLAN 1492) in the /24 subnet. The Kubernetes cluster virtual machines have access to the Internet, Active Directory, NFS, and CIFS shares and IB-MGMT network as well as the Kubernetes node and pod networks defined in the fpv-kubernetes-vx tenant Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 24

17 Apprenda setup details When the infrastructure configuration is complete and various component virtual machines are deployed with appropriate resources and connectivity, the Apprenda platform is ready to be set up. This section discusses the requirements for the installation of the Apprenda platform. Apprenda application requirements All Apprenda nodes need to be in the same domain and resolvable by the host name. All hosts must have access to the shared repository. The load manager component is configured with two NICs: one for Apprenda internal and external traffic and one for CIFS access. These are the requirements: Apprenda 8.2 or later Docker 1.13 Community Edition (CE) installed on two Linux application servers Kubernetes Cisco ACI with Cisco Network Interface Plug-in 1.7 or later Apprenda with Cisco ACI Plug-in Apprenda with NetApp SnapCenter Plug-in 1.0 For a detailed list of hardware and software requirements, refer to the following documentation: Integration of Apprenda with Cisco ACI The integration of Apprenda with the Cisco ACI platform is powered by two Apprenda plug-ins for Cisco ACI. The extension plugin uses the application metadata configured by the developers to process calls to the APIC to create application profiles, EPGs, and contracts for the application being deployed. The bootstrap plug-in modifies the YAML specifications of the application components to associate them with the newly created application profiles and EPGs. Figure 8 shows the integration components. Figure 8. Apprenda and Cisco ACI integration components For detailed information about the solution and demonstration videos, refer to the following resources: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 24

18 Apprenda platform installation and integration with the Cisco ACI platform In this deployment, the Apprenda components are set up in the following order: 1. Prepare the infrastructure. For details, refer to 2. Install the core Apprenda platform. For step-by step instructions, refer to 3. Attach the Linux application servers. For installation instructions, refer to 4. Install Kubernetes on the server cluster using the Kismatic Enterprise Toolkit ( with the Custom Cisco Network Interface (CNI) plug-in option. 5. Run the Cisco ACI provisioner (acc-provisioner) using python-pip. For details, refer to j3_srz_n1b. 6. Attach the Kubernetes cluster to Apprenda through the Apprenda operator portal. Note: Before the cluster can be attached, the cluster client certificate must be placed in the trusted certificate store of every coordinator node. 7. Deploy the Apprenda plug-ins for Cisco ACI. The plug-ins run as guest applications on Apprenda. The deployment package is located here: Apprenda integration with Cisco ACI installation steps Follow these steps to integrate Apprenda and the Cisco ACI platform: 1. Run generate-certificate.ps1 to generate the credentials-encrypting certificate. 2. Copy the certificate created by step 1 and ProvisionNode.ps1 to each Microsoft Windows application and web node. 3. Run provision-node.ps1 on each Windows application and web node to import the certificate, grant the Active Directory account read access to the private key, and grant "Log on as service" permission to the Active Directory account. 4. Pass the required Active Directory account credentials to the script. 5. Run customize-archive.ps1 to embed the Active Directory account credentials in the archive. 6. Run install-integration.ps1 to create and promote the APIC Proxy application, create all the necessary custom properties, and create the APIC Proxy bootstrapper. The script accepts the following parameters: CloudUrl: Credential: Platform credentials; the PSCredential object will be created behind the scenes ApicProxyArchive: Path to apicproxy.zip (can be left blank if the archive is in the same directory as the script) BootstrapPolicyArchive: Path to bootstrapper.zip (can be left blank if the bootstrapper is in the same directory as the script) ApicTenant: The tenant used by the Kubernetes cluster ConsumedContracts: Additional contracts to add as options to the consumed services custom property; they must reside in either the Kubernetes or common tenants ProvidedContracts: Additional contracts to add as options to the provided services custom property; they must reside in either the Kubernetes or common tenants This script requires the PSApprendaApi module for Microsoft PowerShell. It can be installed with the following command: Install-Module PSApprendaApi For step-by-step instructions, refer to this documentation: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 24

19 NetApp SnapCenter and NetApp Service Level Manager installation The SnapCenter plug-in consists of an extension, which runs as a guest application, and a boostrapper. Both components can be downloaded here: Detailed installation instructions for the SnapCenter plug-in can be found at the following location: NSLM can be downloaded and installed by following the instructions at the following location: Database cloning with the Apprenda SnapCenter plug-in The automated database provisioning is performed in two steps: 1. The Apprenda extension for SnapCenter is invoked by the Apprenda deployment pipeline. This extension communicates with the SnapCenter REST API to perform the desired type of cloning and mounting as specified by the metadata that accompanies the application. 2. The Apprenda bootstrapper for SnapCenter is called at a later stage of the pipeline to update the application configuration file with the new connection information, so that the newly containerized application can connect to the cloned database. The section Continuous delivery of databases using the Apprenda SnapCenter plug-in later in this document discusses SnapCenter and its components in detail. For a full demonstration of the solution, refer to the following URL: Continuous integration with Jenkins on NetApp ONTAP NetApp has developed a Jenkins framework that accelerates the continuous integration process on the Apprenda Cloud Platform. The NetApp Jenkins Framework consists of these components: Open-source GitLab CE Jenkins and Docker Registry along with an Adobe Flash microframework user interface that that functions as a web server and interfaces with the Jenkins jobs from the pipeline Kubernetes to map the persistent volume claims (PVCs) to persistent volumes for data volume and clone creation NSLM to perform all the data management functions on ONTAP as shown in the highlighted box in Figure 9. Flask running as a broker Jenkins does not have a database and uses a master-slave configuration. The Flash web service stores all the configuration information pertaining to the data layer using Apache CouchDB as a persistent data store. The entire NetApp Jenkins Framework is packaged in a Helm Chart that can be installed in the Kubernetes cluster managed by Apprenda. GitLab and the Docker Registry can be replaced and configured with any other SCM or binary repository tool as long as a Docker image is available Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 24

20 Figure 9. NetApp Jenkins Framework with Kubernetes The NetApp Jenkins Framework installs Jenkins and its home directory on a Kubernetes pod that mounts a persistent data volume from ONTAP over NFS for scalability and resiliency. Flask communicates with NSLM to make REST API calls to ONTAP to create data volumes, snapshots, and clones during the continuous integration process, on demand. The framework then runs the Jenkins jobs in the pipeline by communicating with Flask to create PVCs, and with NSLM to create persistent volumes. Accelerating the continuous integration process Reducing build time and improving developer productivity are always high priorities for most business owners in midsize to large organizations who practice agile development in data center and private cloud environments. During the build process, the developer group is checked out from the source code repository by the developer, prebuild artifacts are pulled from the artifact repositories, and the source code is compiled to generate the binary or executable file. The builds are of two types: user builds that the developer runs in the workspace, and continuous integration builds in which changes are merged in the main code base and compiled. Continuous integration is an automated and iterative build-and-test process used to identify bugs at early stages of the development cycle. This process enables better code quality through repetitive testing, leading to innovation and faster time to market. Git or any other Git implementation, such as GitLab, Atlassian Bitbucket, or managed services from GitHub, is used as the SCM tool during the application development process. Ideally, developers perform a git clone operation to get a copy of the source code along with the history of all the code changes to manage the source files in isolation. This process may be useful for teams with fewer than 10 developers, but it is challenging for organizations with hundreds of developers. When a high number of git clone operations are performed in parallel, the Git server resources for scaling and load balancing are constrained, resulting in long wait times for developers to create and clone their workspaces. After the git clone operation, the prebuild artifacts must be downloaded and a full build created before the developer can start writing or modifying code. This takes time, depending on the code size. For example, if 5 minutes are needed to perform a git clone operation and 20 minutes are needed to build the code every time a developer creates a workspace, at scale the organization loses hundreds of person hours before all the developers can start writing code. This loss impairs developer productivity immensely. Similarly, this distributed approach to code development using Git has its own challenges: Moving the source code in developer workspaces hosted in the developers portable computers poses a security challenge. Delayed code submittals from developers can lead to long wait times before changes to the main code base can be merged during the continuous integration process. Data recovery from failures is a challenge, especially when code changes are not submitted in a timely manner. Apprenda and the NetApp Jenkins Framework can accelerate this process (Figure 10) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 24

21 Figure 10. Accelerating the continuous integration process with the NetApp Jenkins Framework In a NetApp integrated continuous integration process, a git clone operation is performed one time from the source repository to a data volume on ONTAP, at the start of the project. After the full build is complete, the build number is tagged to a snapshot at the data layer. This operation is completely transparent to the developer and is an automated process in the framework as illustrated in Figure 10. Apart from the baseline build, all the subsequent continuous integration builds are incremental, thus saving a lot of time in the overall development process. The snapshots are pointer-based read-only data blocks that do not consume any additional space except for the changed blocks. As the iterative continuous integration process progresses after each code-change submittal, these snapshots are created and tagged for every successful build. Developers select the last successful build-and-create a workspace. As mentioned earlier, every build is tagged with a snapshot. The NetApp FlexClone technology instantly creates a user workspace with a minimum data space. The framework automates the process of creating instant developer workspaces on demand from the Jenkins pipeline. The NetApp clones are primarily sand boxes that contain the source code, artifacts, and compiled code. The developers can start writing code immediately, hence improving their productivity. All the stress that the git clone operation places on the Git server at scale is now offloaded to ONTAP, which provides better scalability with very little resource constraint. Developers can use IDE tools such as Cloud9 to connect to their workspaces from their desktops for distributed development while gaining more data control for security and recovery from failures. In many situations, developers need to perform a git merge operation to merge the changes in their workspaces with the latest continuous integration build. The git merge operation traditionally triggers a full build for the user that consumes considerable resources and time. However, with NetApp FlexClone technology, git merge performs an incremental build that saves time and induces more efficiency in the continuous integration process for the developer by eliminating merge conflicts. Continuous delivery of databases using the Apprenda SnapCenter plug-in Databases are integral to the software development lifecycle (SDLC). Applications need a production-like database to use during staging or during the UAT process before the application is released to production. Developers engage with database administrators, or DBAs, to perform a risk assessment before getting a copy of the production database to test the application. Sometimes DBAs obfuscate sensitive data, limiting the developers ability to test the application freely. One of the most common ways to provision a database for testing during application development is to use migration scripts to re-create the database; however, re-creating the database is a very time-consuming process Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 24

22 NetApp and Apprenda jointly created a plug-in using SnapCenter 4.0 to help enable developers to provision a production-like database instantly for application testing during the development cycle. SnapCenter 4.0 is a Microsoft Windows based tool from NetApp that provides an application-consistent snapshot that protects a data volume that was used in the production environment. The tool also provides other data management capabilities, such as database cloning, restoration, and replication. SnapCenter supports many common databases, including Oracle, Microsoft SQL, MySQL, MongoDB, and DB2. In this solution, the Apprenda SnapCenter plug-in uses a MySQL database to perform data protection and management using REST APIs from the Apprenda Cloud Platform. SnapCenter 4.0 must be configured as documented in the data protection guide from NetApp, at SnapCenter discovers the data management platform (ONTAP in this case) before proceeding to install the custom agent for the MySQL database in the production instance. This MySQL custom agent communicates with the SnapCenter engine to perform all the developer-related data protection and management functions from Apprenda. Apprenda allows the organization to set policies and permissions for DBAs and developers governing the use of the plug-in for provisioning and implementing a copy of the production database. Figure 11 shows the components of the Apprenda SnapCenter plug-in that communicate with the application and the database using SnapCenter REST APIs. After the application is deployed in the Apprenda sandbox phase, a copy of the database is requested using custom properties. Figure 11. Apprenda SnapCenter plug-in layers for database testing and development The Apprenda SnapCenter plug-in shown in Figure 11 has two components: the bootstrapper and the extension. The extension communicates with SnapCenter from the Apprenda Cloud Platform, and the bootstrapper connects to the application to perform various activities as discussed here. Apprenda SnapCenter plug-in extension: Follow these steps to use the Apprenda SnapCenter plug-in extension: 1. Check whether a snapshot is available for the production MySQL data volume. If no snapshot is available, create one. 2. Clone a copy of the production database based on the snapshot created. 3. Mount the clone on one of the Linux nodes managed by the Apprenda Cloud Platform. Apprenda chooses to mount the clone based on the CPU and the memory available on the Linux node. The node using the production database is not part of this selection, to enable load balancing and mitigate risk. 4. Start the database clone when the NFS mount is complete on the Linux node. 5. Take a snapshot of the cloned database Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 24

23 Note: The database can be instantly restored to its clean state if it becomes polluted from testing by using the Snap Restore feature in SnapCenter. Apprenda SnapCenter plug-in bootstrapper: The Apprenda SnapCenter plug-in bootstrapper functions as follows: 1. The application connects to the clone when the clone is fully mounted and database startup is complete. 2. Every time the application is deployed and redeployed during the test process, the bootstrapper gets the clone information as well as the changes that the application connection made to the database. Figure 12 shows the plug-in workflow. Figure 12. Apprenda SnapCenter plug-in workflow Developers can perform these functions through three custom property settings in the Apprenda Cloud Platform: CloneOriginal, RestoreClone, and KeepExisiting. CloneOriginal: This option checks or creates a snapshot of the production database, clones it, mounts the MySQL database on a Linux node over NFS, starts the database, connects the application to the database, and creates another snapshot of the cloned database. RestoreClone: This option instantly restores the system to a clean database using the snapshot created for the clone database if the tests start to fail. KeepExisting: This option allows the developer to keep using the database for the application even though there may be changes to the database and the tests are still being performed. The connect string to the application stays intact. The Apprenda SnapCenter plug-in provides data consistency and more data controls for the MySQL database, thereby allowing developers to release applications faster with zero-touch storage and mitigating risk to the production database. Conclusion FlexPod solution is a predesigned, integrated, and validated architecture for the data center that combines Cisco UCS servers, Cisco Nexus Family switches, and NetApp All Flash FAS (NetApp AFF) into a single, flexible architecture. FlexPod can support a variety of hypervisor options and bare-metal servers and can be sized and optimized based on customer workload requirements. FlexPod Datacenter with Apprenda 8.2 integrates the Cisco ACI platform with the Apprenda Cloud Platform and uses plug-ins such as SnapCenter and NSLM along with development tools such as Jenkins and Kubernetes to provide a complete, ready-to-use CI/CD solution Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 24