NATIVE HYBRID CLOUD, OPENSTACK EDITION WITH VXRACK SYSTEM WITH NEUTRINO

Transcription

1 Solution Guide NATIVE HYBRID CLOUD, OPENSTACK EDITION WITH VXRACK SYSTEM WITH NEUTRINO Provides a cloud-based application production environment Enables faster time to market for software applications Improves application quality and update schedules EMC Solutions Abstract This solution guide describes Native Hybrid Cloud, a production environment for cloud native applications. It is integrated with VxRack System with Neutrino, a cloud computing environment based on OpenStack. This document introduces the main features and functionality of the solution and the solution architecture. September 2016

2 Copyright Copyright 2016 EMC Corporation. All rights reserved. Published in the USA. Published September 2016 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, EMC2, Native Hybrid Cloud, ScaleIO, ViPR, VxRack, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Part Number H

3 Contents Contents Chapter 1 Introduction 5 Business case... 6 Solution overview... 7 Key benefits... 8 Document purpose... 9 Audience... 9 We value your feedback!... 9 Chapter 2 Native Hybrid Cloud Architecture 10 Solution architecture Key components Hardware resources Software resources Chapter 3 Before You Begin 17 Introduction Customer pre-deployment tasks EMC Professional Services pre-deployment tasks Integration with existing IT infrastructure Accessing Pivotal Operations Manager Accessing Native Hybrid Cloud Monitoring and Reporting Chapter 4 Scaling Hardware and Software 24 Scaling compute and storage resources Requesting additional resources for Native Hybrid Cloud Modifying OpenStack project quotas for Native Hybrid Cloud Scaling Pivotal Cloud Foundry Chapter 5 Native Hybrid Cloud Monitoring and Reporting 31 Introduction Monitoring and reporting architecture SolutionPacks and Collectors Dashboards Reports Thresholds Alerts

4 Contents Active Directory and LDAP server integration for reporting Chapter 6 Logging 46 Introduction Loggregator Native Hybrid Cloud logging Chapter 7 Data Protection 49 Introduction Pivotal Cloud Foundry data protection Native Hybrid Cloud data protection Native Hybrid Cloud Installer virtual machine restoration Native Hybrid Cloud Blobstore restoration Native Hybrid Cloud Monitoring and Reporting virtual machine restoration Chapter 8 User administration 62 Introduction VxRack Neutrino accounts and users Pivotal Cloud Foundry identity providers Native Hybrid Cloud Monitoring and Reporting users Chapter 9 Support and troubleshooting 66 Customer support Chapter 10 Conclusion 68 Conclusion Chapter 11 Reference documentation 70 Native Hybrid Cloud documentation VxRack Neutrino documentation Pivotal Cloud Foundry documentation Native Hybrid Cloud Monitoring and Reporting documentation Appendix A Open source license and copyright information 72 Software packages GNU General Public License version

5 Chapter 1: Introduction Chapter 1 Introduction This chapter presents the following topics: Business case... 6 Solution overview... 7 Key benefits... 8 Document purpose... 9 Audience... 9 We value your feedback!

6 Chapter 1: Introduction Business case Businesses are embracing hyper-converged infrastructure and cloud-native frameworks to disrupt conventional business models, differentiate themselves from competitors, and explore new business opportunities. A requirement for growing a business is the ability to analyze consumer data and rapidly and continuously deliver new, innovative, and unique content to consumers. Consumers have come to expect applications to always be available, secure, and updated. This trend is driving a shift in application-development platforms and processes. Application developers are moving to new practices and technologies like development and operations (DevOps), continuous delivery, agile methodology, twelve-factor application frameworks, microservices, and API-based collaboration. These technologies and practices speed application development and quickly deploy features into production. However, most businesses encounter challenges when transitioning to new practices and technologies. Application development groups wrestle with decisions about the best system to manage growing application development needs. Short-term revenue goals can displace proper planning for an integrated application development environment. Consequently, disparate applications are installed at different times in various functional areas, resulting in process inefficiencies and software and hardware integration challenges. Many challenges arise when high-growth application development groups have several applications in silos. These include wasted effort, lack of realtime visibility, and increased integration complexity and cost. IT departments are also under pressure to provide enterprise-quality development environments with a shrinking budget. The performance and convenience offered by an integrated development environment based on cloud computing and storage addresses these challenges. An integrated development environment can also help application developers decrease the time to market, increase operational agility, and facilitate automated deployment of applications. The variety and complexity of proprietary and public cloud offerings increases the difficulty of choosing an integrated cloud solution. Several new open source technologies are also available to help create a cloud solution for application development, testing, and deployment. However, IT departments must know how to best use these technologies to drive standardization, integrate open source and proprietary systems, minimize cost, and support service-level agreements. The availability of the public cloud and the ease with which it can be accessed creates the perception that it is easy for IT to duplicate the same level of service inhouse. Planning, designing, and building a private cloud to support cloud native applications can be a complex project that takes too long if you need to roll out a solution to address immediate business needs. IT also needs to maintain and manage the infrastructure supporting these new applications to ensure the environment is reliable, secure, and upgradeable. 6

7 Solution overview Chapter 1: Introduction As a result, IT departments must make difficult decisions when implementing or creating a cost-effective and cloud-based application development environment. EMC has solved these challenges with Native Hybrid Cloud. Native Hybrid Cloud is an engineered solution that uses an elastic infrastructure and monitoring tools to support an application production environment. IT professionals can easily monitor, manage, and scale the environment to ensure its fitness for current and future application production needs. Native Hybrid Cloud provides a predictable, accurate, and protected deployment that is sized correctly and installed on a VxRack System with Neutrino, a hyperconverged infrastructure that is powered by OpenStack cloud compute and ScaleIO storage software. VxRack Neutrino hardware sizing configurations are based on expected Pivotal Cloud Foundry usage and system testing. The virtual infrastructure is sized based on Pivotal Cloud Foundry usage. The Native Hybrid Cloud Monitoring and Reporting dashboard monitors and reports consumption and billing information transparently. Data protection, including backup and restoration, is provided by OpenStack, Pivotal Cloud Foundry, and EMC ScaleIO, which provides data replication and redundancy beyond normal OpenStack and Pivotal Cloud Foundry capabilities. Pivotal Cloud Foundry Pivotal Cloud Foundry is a cloud native platform (CNP) on which developers can create, test, deploy, update, and scale applications on public and private clouds. Pivotal Cloud Foundry enables developers to code in multiple languages and frameworks. Pivotal Cloud Foundry provides developers with a ready-to-use cloud computing environment and application development services, all hosted by virtualized servers on your VxRack Neutrino infrastructure. The services include: Pivotal Operations Manager is a web application used to deploy and manage Pivotal Cloud Foundry and associated services such as SSO and Pivotal Cloud Foundry Metrics. It is the industry s first turnkey enterprise CNP management platform with infrastructure as a service (IaaS) integration. Pivotal Cloud Foundry Metrics is a suite of products designed to help operators and developers better understand the health and performance of the Pivotal Cloud Foundry platform and all applications running on it. It includes Java Management Extensions (JMX) bridge for Elastic Runtime. Pivotal Cloud Foundry Metrics collects and exposes system data from Pivotal Cloud Foundry components with a JMX endpoint to help you monitor your installation and assist in troubleshooting. Elastic Runtime is the framework that hosts running applications, manages system health, and provides client access to the runtime environment, including running applications through the cloud controller API endpoint. Buildpacks provide framework and runtime support for your applications. Buildpacks typically examine user-provided artifacts to determine what 7

8 Chapter 1: Introduction dependencies to download and provide instructions on how to configure applications to communicate with bound services. Pivotal Marketplace provides developers with a robust self-service user experience that boosts productivity with on-demand access to a large marketplace of mobile and data services, an intuitive console, and open APIs. The Marketplace's managed and user-provided services can be used without the need for IT action. Native Hybrid Cloud Monitoring and Reporting Native Hybrid Cloud Monitoring and Reporting is resource management software that provides detailed relationship and topology views for virtual or physical hosts, including the underlying infrastructure, to help you visualize infrastructure relationships, analyze and report on its capacity and health, and optimize resources in traditional and software-defined environments. This software also provides performance and usage views across the layers of the physical and virtual infrastructure to help you understand the impact that infrastructure has on enterprise applications. Native Hybrid Cloud automated deployment EMC has developed a set of manual and automated tools to simplify the deployment of Native Hybrid Cloud on VxRack Neutrino. EMC Professional Services deploys the Native Hybrid Cloud installer as an OpenStack virtual machine instance and it is used to run automated tasks that validate previously collected customer site-related data (network, DNS, SSO, and so on) and run automated scripts that deploy Native Hybrid Cloud components. Key benefits Native Hybrid Cloud is an engineered solution from EMC that provides an optimized and pre-tuned application production platform. The solution stack is based on Pivotal Cloud Foundry and integrated with VxRack Neutrino, an OpenStack-based private cloud. Native Hybrid Cloud accelerates the development cycle of cloud-native applications, and allows businesses to deliver new and unique capabilities to market as quickly as possible. Native Hybrid Cloud enables developers to create cloud native applications in multiple programming languages and track application quality. The platform is based on an integrated infrastructure, which enables IT operators to easily monitor, manage, and scale the development environment. Integration with Microsoft Active Directory and Lightweight Directory Access Protocol (LDAP) servers makes it easier to manage users, roles, and permissions. OpenStack Horizon dashboard, Native Hybrid Cloud Monitoring and Reporting, Pivotal Operations Manager, and Pivotal Cloud Foundry Metrics give you the tools you need to monitor Native Hybrid Cloud. With Native Hybrid Cloud, customers will benefit from: An engineered solution that integrates the industry s top open source technologies, professional services, and single call support into a cloud-native application development platform that is easy to deploy, use, and grow 8

9 Document purpose Audience We value your feedback! Chapter 1: Introduction A turn-key, private cloud and infrastructure service that eliminates the time IT spends on complex deployments and updates A fully tested and integrated application development, testing, deployment, updating, and management solution A virtual infrastructure that can be deployed quickly to reduce the time from planning to coding A single platform to write and store code in multiple languages This document describes an engineered solution for managing a cloud native application production environment with Native Hybrid Cloud. The solution uses VxRack Neutrino, a private cloud-hosting rack-scale hyperconverged appliance that uses OpenStack cloud computing software. This document introduces the main features and functionality of the engineered solution, describes the individual components, and lists the validated hardware and software environments. This document also describes how the components and functionality of Native Hybrid Cloud and VxRack Neutrino are integrated. This document is for DevOps cloud architects, DevOps cloud operators, or application development managers who want to design, manage, and deploy cloud native applications with Native Hybrid Cloud on VxRack Neutrino. Readers should be aware of OpenStack, Pivotal Cloud Foundry, virtualization, networking, and storage technologies, and general IT functions. EMC and the authors of this document welcome your feedback on the solution and the solution documentation. Contact EMC.Solution.Feedback@emc.com with your comments. Authors: Sergey Schindler, Eoghan Kelleher, Brian Coe. 9

10 Chapter 2: Native Hybrid Cloud Architecture Chapter 2 Native Hybrid Cloud Architecture This chapter presents the following topics: Solution architecture Key components Hardware resources Software resources

11 Chapter 2: Native Hybrid Cloud Architecture Solution architecture This chapter provides an overview of the architectural components and concepts of Native Hybrid Cloud. The diagrams in this section depict the layout of major components that comprise Native Hybrid Cloud. Figure 1 depicts a high-level overview of the components in the Native Hybrid Cloud architecture, which includes: Pivotal Cloud Foundry, an industry standard, open source-based, and cloud native application platform, which developers can use to build, deploy, run, and scale applications on public and private clouds. Native Hybrid Cloud Monitoring and Reporting and resource management software. IT can use this software to visualize usage relationships; analyze and report on capacity, performance, and health; and optimize resource consumption. VxRack Neutrino can concurrently host multiple services with its elastic, scaleout, and hyper-converged design. 11

12 Chapter 2: Native Hybrid Cloud Architecture Figure 1. Native Hybrid Cloud architecture Key components Native Hybrid Cloud uses the following key components in addition to Pivotal Cloud Foundry and VxRack Neutrino. 12

13 Chapter 2: Native Hybrid Cloud Architecture Native Hybrid Cloud Monitoring and Reporting SolutionPacks and Collectors for Native Hybrid Cloud Monitoring and Reporting Blobstore for Pivotal Cloud Foundry Native Hybrid Cloud Monitoring and Reporting and Pivotal Cloud Foundry Metrics analyze health, configurations, and capacity growth. You can quickly spot service level agreement (SLA) problems through custom dashboards and reports that meet the needs of a wide range of users and roles. With Native Hybrid Cloud Monitoring and Reporting you can also see capacity consumption across your Pivotal Cloud Foundry deployment with built-in views to help you understand who is using capacity, how much they are using, and when more will be required. Organizations can take advantage of Native Hybrid Cloud Monitoring and Reporting SolutionPacks and Collectors, which are plug-ins, extensions, and management packs that add functionality to management tools already deployed in data centers. For example, the Generic Host SolutionPack provides custom component monitoring. The JMX Collector delivered with Native Hybrid Cloud enables administrators to get a complete and in-depth picture of Pivotal Cloud Foundry s health, capacity, and resource availability. Pivotal recommends using an external Blobstore for Pivotal Cloud Foundry production deployments. The external Blobstore must be compatible with the Amazon Simple Storage Service (Amazon S3) API, which provides developers and IT teams with secure, durable, and highly-scalable object storage. The Blobstore holds application code, Buildpacks, and applications. In Native Hybrid Cloud, the Blobstore is deployed on the VxRack Neutrino compute service as a cluster of OpenStack virtual machine instances in a high availability configuration. OpenStack Cinder volumes created in VxRack Neutrino provide the backend storage for this Blobstore cluster. A built-in HAProxy load balancer is combined with multiple storage and proxy nodes to ensure the service s high availability. Figure 2 shows the data storage architecture of the solution. 13

14 Chapter 2: Native Hybrid Cloud Architecture Figure 2. Data storage for Pivotal Cloud Foundry Data protection Pivotal Cloud Foundry, OpenStack, and ScaleIO provide data protection capabilities for Native Hybrid Cloud. Pivotal Operations Manager offers the ability to export Pivotal Cloud Foundry installation settings and assets. When you export an installation, the exported file contains the base virtual machine images, necessary packages, and references to the installation IP addresses. ScaleIO distributed storage protects Native Hybrid Cloud data by creating multiple redundant copies. An S3-compatible Blobstore uses Cinder volumes protected by ScaleIO replication, and it is used to protect all Pivotal Cloud Foundry applications and metadata. The Blobstore is backed up at a customer-specified interval, which is typically once daily. Integration with external identity providers Native Hybrid Cloud integration with external identity providers (IDP) such as Active Directory or LDAP servers provides the separate built-in capabilities of VxRack Neutrino, Pivotal Cloud Foundry, and Native Hybrid Cloud Monitoring and Reporting. EMC Professional Services performs IDP integration services for customers. Logging 14

15 Chapter 2: Native Hybrid Cloud Architecture Loggregator, the Pivotal Cloud Foundry component responsible for logging, provides a stream of log output from your applications and from Pivotal Cloud Foundry system components that interact with your applications during updates and execution. Hardware resources VxRack Neutrino comes in two brick types: high performance and high capacity. Table 1 lists the maximum hardware resources for both brick types. The high performance brick has a 2U chassis with four nodes. The high capacity brick has a 1U chassis with one node. Table 1. Hardware details Hardware High performance, each node High capacity, one node CPU Dual Intel Xeon E v3 sockets Dual Intel Xeon E v3 sockets Memory 512 GB 512 GB Network Dual Port GbE Dual Port GbE Storage 800 GB SSD One 400 GB SSD (OS) One 800 GB SSD (caching) Twenty-two 1.8 TB HDD (data) Software resources Table 2 lists the software used in Native Hybrid Cloud. Table 2. Solution software Software Version Notes VxRack Neutrino 1.1 Provides the physical and virtual infrastructure to use Native Hybrid Cloud Pivotal Cloud Foundry Pivotal Operations Manager Pivotal JMX Bridge (Operations Metrics) Pivotal Elastic Runtime 1.7 Facilitates application production Provides a graphical interface to manage Pivotal Cloud Foundry components Used to help you monitor your installation and assist in troubleshooting build 5 Provides a scalable runtime environment to deploy applications Ansible 2.3 Used to deploy Native Hybrid Cloud Native Hybrid Cloud Monitoring and Reporting 1.1 Used to monitor Native Hybrid Cloud 15

16 Chapter 2: Native Hybrid Cloud Architecture Software Version Notes EMC ViPR SRM 3.7 SP2 Used to monitor Native Hybrid Cloud Elasticsearch Used for log collection and analysis Logstash Used for log collection and analysis Kibana Used for log collection and analysis 16

17 Chapter 3: Before You Begin Chapter 3 Before You Begin This chapter presents the following topics: Introduction Customer pre-deployment tasks EMC Professional Services pre-deployment tasks Integration with existing IT infrastructure Accessing Pivotal Operations Manager Accessing Native Hybrid Cloud Monitoring and Reporting

18 Chapter 3: Before You Begin Introduction This chapter provides an overview of the Native Hybrid Cloud installation process, including the work EMC Professional Services performs to install Native Hybrid Cloud and information you will provide to EMC Professional Services. EMC Professional Services performs the site survey, sizes your solution, allocates resources, and integrates existing IT infrastructure. Customer pre-deployment tasks The customer must perform these tasks before EMC Professional Services arrive onsite. Network isolation considerations Pivotal Cloud Foundry best practices for network isolation require separate virtual networks: a Management network for management functions and floating IPs, and a Developer network for development functions and access to the Pivotal Cloud Foundry Router virtual machines. Native Hybrid Cloud and Pivotal Cloud Foundry components are deployed by EMC Professional Services in a single private virtual network. The Developer network may be isolated, and access from the Developer network to the Pivotal Cloud Foundry Router virtual machines is enabled by routing from an external customer router or Layer 3 switch. If you want to isolate traffic, you can create a separate virtual network apart from the OpenStack virtual network. Figure 3. Native Hybrid Cloud and Pivotal Cloud Foundry virtual networks Configure your firewall for Pivotal Cloud Foundry Pivotal Cloud Foundry supports a deployment environment that is protected by a customer-provided firewall. For instructions on how to configure your firewall for Pivotal Cloud Foundry, and how to verify that Pivotal Cloud Foundry can resolve DNS 18

19 Chapter 3: Before You Begin entries behind your firewall, refer to Preparing Your Firewall for Deploying Pivotal Cloud Foundry. Note: EMC recommends that customers prepare their firewall before EMC Professional Services arrive on-site. Firewall and Internet Proxy considerations Pivotal Cloud Foundry has specific access requirements for deployments that reside behind a firewall. For more information on how to configure your firewall to allow the required access, refer to Preparing Your Firewall for Deploying Pivotal Cloud Foundry. Pivotal Cloud Foundry relies on accessing external websites to download components required in Buildpacks, which are used to compile applications that are pushed to Pivotal Cloud Foundry. In cases where external access is not possible from the Pivotal Cloud Foundry network to external websites, use offline or custom Buildpacks. For more information, refer to Custom Buildpacks. DNS considerations Native Hybrid Cloud relies on an external DNS service for its application name resolution. The DNS service must be provided through an existing customer DNS. An existing customer DNS server must be configured with at least one wildcard DNS domain, such as *.nhc.example.com, to facilitate the publishing to and access of applications hosted by Native Hybrid Cloud. For more information, refer to the General Requirements section of Prerequisites to Deploying Operations Manager and Elastic Runtime. Note: For production deployments, EMC recommends using two wildcard domains if the system and applications need to be separated. Configure your load balancer The typical Pivotal Cloud Foundry deployment for lab and test environments uses a single internal instance of HAProxy. Production environments should use a highlyavailable and customer-provided load balancing solution for forwarding application traffic to the Pivotal Cloud Foundry Elastic Runtime router IP addresses. The customer-provided external load balancer must be capable of: Providing load balancing to each of the Pivotal Cloud Foundry router IP addresses Supporting SSL termination with a wildcard DNS location Inserting appropriate X-Forwarded-For and X-Forwarded-Proto HTTP headers to incoming requests Optionally supporting WebSockets for application logging with Pivotal Cloud Foundry Loggregator Configure source network address translation SNAT is enabled for the VxRack Neutrino software-defined networking service. However, all Pivotal virtual machines must be assigned floating IP addresses to communicate with external targets. The impact on Pivotal Cloud Foundry deployment on VxRack Neutrino is minimal. Designated Pivotal Cloud Foundry virtual machine instances, such as Pivotal 19

20 Chapter 3: Before You Begin Operations Manager, HAProxy, or Router require a floating IP assignment. For more information, refer to Network address translation. EMC Professional Services pre-deployment tasks Pre-deployment tasks include procedures that do not directly relate to installation and configuration, however, you do need the results from these tasks at the time of installation. Examples of pre-deployment tasks are collection of host names, IP addresses, VLAN IDs, license keys, and installation media. These tasks are performed before the Native Hybrid Cloud is installed and configured. EMC Online Support can provide more information on these prerequisites. See Customer support for customer support contact information. Required customer data Native Hybrid Cloud deployment readiness list EMC Professional Services collects customer-specific configuration data for networking, arrays, accounts, and so on. This information must be entered into the Customer Configuration Worksheet during the deployment process. The following tasks must be performed and information must be obtained prior to deploying the Native Hybrid Cloud: Pivotal Operations Manager Director administrator username and password Pivotal Cloud Foundry DNS A record DNS wildcard entry or entries for Pivotal Cloud Foundry HAProxy SSL certificate and SSL ciphers (if used) for Pivotal Cloud Foundry System and application domain names for Pivotal Elastic Runtime Desired Pivotal Cloud Foundry Application Instance (AI) size SSO and LDAP servers and SMTP endpoints and credentials Customer DNS server and NTP server IP addresses Prior to deploying Native Hybrid Cloud on VxRack Neutrino, EMC Professional Services ensure that the following components are available in the VxRack Neutrino infrastructure: Root password for SSH access to VxRack Neutrino infrastructure servers Access to a dedicated OpenStack project in the default domain Native Hybrid Cloud user in the dedicated project with full OpenStack administrative rights OpenStack private and public networks for the Native Hybrid Cloud tenant, including DNS name server addresses for external virtual machine access to customer DNS and NTP The Native Hybrid Cloud private network must only contain a single subnet The Native Hybrid Cloud private network must be a full /24 subnet with no existing IP addresses in use 20

21 Chapter 3: Before You Begin Ability to assign a minimum of 75 floating IP addresses in the Native Hybrid Cloud OpenStack tenant namespace Sufficient OpenStack resource quotas are assigned for Native Hybrid Cloud deployment in the OpenStack tenant namespace Sizing prerequisites Table 3 provides a list of the minimum OpenStack resource quotas needed for a Native Hybrid Cloud installation on VxRack Neutrino. This table assumes a deployment of three Pivotal Cloud Foundry Diego Cell instances of size m1.xlarge. Additional OpenStack resources are required for larger deployments. Table 3. Minimum OpenStack resource quotas OpenStack resource name vcpu 120 Instances 70 Volumes 30 Minimum quota requirement RAM (MB) Total size of volumes and snapshots (GiB) 6656 Volume snapshots 50 Floating IP addresses 75 Neutron ports 100 Use the PCF Sizing & Cost Estimation Tool to estimate the necessary resources to deploy Native Hybrid Cloud on OpenStack. Integration with existing IT infrastructure EMC Professional Services performs the integration with both VxRack Neutrino and existing IT infrastructure. This includes integrating or installing the following services: SSO and LDAP servers DNS and NTP servers Customer s load balancer Customer s SSL certificates and SSL ciphers MySQL database, if a customer is only using external databases for Pivotal Cloud Foundry Customer s log aggregators SMTP endpoints Accessing Pivotal Operations Manager 21

22 Chapter 3: Before You Begin Use a web browser to access Pivotal Operations Manager when it is successfully deployed. Steps 1. In your web browser, type the IP address assigned to the Pivotal Operations Manager virtual machine: where opsmgr-address is the Pivotal Operations Manager IP address. 2. On the Pivotal Operations Manager login page, type your Pivotal Operations Manager administrator username and password, as shown in Figure 4. Figure 4. Pivotal Operations Manager login page Accessing Native Hybrid Cloud Monitoring and Reporting By default, the Native Hybrid Cloud Monitoring and Reporting server address is specified as an IP address. Use a web browser to access Native Hybrid Cloud Monitoring and Reporting. Steps 1. In your web browser, type the Native Hybrid Cloud Monitoring and Reporting URL: where nhcmnr-address is the Native Hybrid Cloud Monitoring and Reporting virtual machine instance floating IP address. 2. On the Native Hybrid Cloud home page, type a user name and password, as shown in Figure 5. 22

23 Chapter 3: Before You Begin Figure 5. Native Hybrid Cloud Monitoring and Reporting login page 23

24 Chapter 4: Scaling Hardware and Software Chapter 4 Scaling Hardware and Software This chapter presents the following topics: Scaling compute and storage resources Requesting additional resources for Native Hybrid Cloud Modifying OpenStack project quotas for Native Hybrid Cloud Scaling Pivotal Cloud Foundry

25 Chapter 4: Scaling Hardware and Software Scaling compute and storage resources Native Hybrid Cloud does not have direct control over scaling the VxRack Neutrino infrastructure and is dependent on the VxRack Neutrino scale-out capabilities. Native Hybrid Cloud gains access to additional resources when compute and storage resources are scaled out for VxRack Neutrino. When additional nodes are added to the VxRack Neutrino hardware rack, these nodes must be configured by VxRack Neutrino and provisioned for the compute service. Compute and storage capacity are added as part of every node added to VxRack Neutrino, and disks immediately become part of the ScaleIO disk pool. After new disks are added to ScaleIO, they can be used by Native Hybrid Cloud. Native Hybrid Cloud has no native discovery mechanism for additional resources. Native Hybrid Cloud is scaled by requesting additional resources. The request is denied if there are no additional resources and fulfilled if the resources are present in VxRack Neutrino. OpenStack project quotas can be increased so users can gain access to additional resources. Adding hardware capacity in Native Hybrid Cloud does not change OpenStack project quotas. When the additional compute and storage nodes are added to VxRack Neutrino, Pivotal Operations Manager Director uses an API call to add additional virtual machines. When the request is granted, the Nova scheduler service, which determines how to dispatch compute requests, places the virtual machines on new or existing nodes. Note: VxRack Neutrino 1.1 and Native Hybrid Cloud 1.1 do not support external storage. Requesting additional resources for Native Hybrid Cloud A Cloud Administrator, which is a user with the administrator role in the default VxRack Neutrino account, can add nodes to the Cloud Compute Service. Adding nodes to the Cloud Compute Service provides the virtual compute and storage capacity required for the creation and management of virtual machines and volumes in OpenStack. Before you begin: Only a Cloud Administrator can perform this operation. There must be three platform service nodes in the VxRack Neutrino system. These nodes are different from the nodes that run the Cloud Compute Service. For more information, refer to the VxRack System 1000 with Neutrino 1.1 Administrator Guide. Steps 1. Select Infrastructure > Nodes. 2. Select Manage > Add to Service. 3. Select Select a Service > Cloud Compute. 25

26 Chapter 4: Scaling Hardware and Software 4. Click Select next to each node in the Actions column. These are the nodes that you want to add to the Cloud Compute Service. You must select a minimum of three nodes. 5. Click Deploy Service. A progress banner at the top of the page shows the deployment progress of the various VxRack Neutrino containers and processes that comprise the Cloud Compute Service. The Cloud Compute Service deployment takes approximately 30 minutes. The time depends on the number of nodes selected. Results When the Cloud Compute Service deployment completes on the nodes, Cloud Compute displays in the Service column next to each node in the Node Management page, verifying that these are now Cloud Compute nodes. Native Hybrid Cloud gains access to the node when the VxRack Neutrino node is added. Modifying OpenStack project quotas for Native Hybrid Cloud You can increase OpenStack project quotas after additional nodes are added. The minimum OpenStack project quotas for Native Hybrid Cloud represent a full Native Hybrid Cloud installation with a minimal Pivotal Cloud Foundry installation of Pivotal Operations Manager, Pivotal Elastic Runtime, Pivotal Operations Metrics, and three Pivotal Cloud Foundry Diego cells. The project quotas must be adjusted if you need to scale out or add additional Marketplace services to your Pivotal Cloud Foundry deployment. Before you begin: Only a Cloud Administrator can perform this operation. Steps Follow these steps to modify the OpenStack project quotas: 1. Log in to the OpenStack dashboard as the admin user in the default domain. 2. Select Identity > Projects in the OpenStack Dashboard. 3. Select the Native Hybrid Cloud project name in the list of projects. 4. Select Modify Quotas Project in the Actions list box. 5. On the Quota tab, shown in Figure 6, modify quotas for vcpus and memory to match your desired Pivotal Cloud Foundry sizing increase. 26

27 Chapter 4: Scaling Hardware and Software Figure 6. Modifying OpenStack project quotas 6. Click Save when the quotas are modified. 27

28 Chapter 4: Scaling Hardware and Software Scaling Pivotal Cloud Foundry Pivotal Cloud Foundry offers several scaling strategies. These strategies increase the capacity and availability of the Pivotal Cloud Foundry platform and the applications deployed on the platform, and decrease the chances of downtime. The strategies include: Scaling out or scaling up the Pivotal Cloud Foundry platform and its components Scaling out applications Scaling out services For more information, refer to Zero Downtime Deployment and Scaling in CF. Scaling Pivotal Cloud Foundry platform components You can scale out the Pivotal Cloud Foundry platform to deploy multiple instances of platform components to achieve redundancy. For more information, refer to Scaling Instances in Elastic Runtime. Scaling Pivotal Cloud Foundry applications Factors such as user workload can change the amount of disk space and memory an application uses. For many applications, increasing the available disk space or memory can improve overall performance. Similarly, running additional instances of an application can allow the application to handle increased user workload and concurrent requests. Applications deployed on Pivotal Cloud Foundry can be scaled up or vertically in size by increasing CPU, memory, and storage. Pivotal Cloud Foundry also allows you to horizontally scale an application by creating multiple instances of an application. This is called scaling out. When an application is scaled out, incoming requests to an application are automatically load balanced across all instances, and each instance handles tasks in parallel with every other instance. Adding more instances allows an application to handle increased usage and network traffic. You can scale out applications deployed on Pivotal Cloud Foundry by using the Pivotal Applications Manager Console, as shown in Figure 7, or the Pivotal Cloud Foundry command line interface by using the cf scale command. 28

29 Chapter 4: Scaling Hardware and Software Figure 7. Scaling an application For more information, refer to Scaling an Application Using cf scale. Scale Pivotal Cloud Foundry managed services Managed services are integrated with Cloud Foundry using APIs, and enable users to provision reserved resources and credentials on demand. Pivotal Cloud Foundry offers managed services that enable users to provision on-demand reserved resources, as shown in Figure 8. Examples of reserved resources include databases on a shared or dedicated server, or accounts on a software as a service (SaaS) application. These resources are called Service Instances and the systems that deliver and operate these resources are called Services. Service Instances are bound to applications using service brokers. A service broker can provide additional instances of a service to serve multiple application instances. For more information, refer to Custom Services. 29

30 Chapter 4: Scaling Hardware and Software Figure 8. Pivotal Cloud Foundry managed services 30

31 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Chapter 5 Native Hybrid Cloud Monitoring and Reporting This chapter presents the following topics: Introduction Monitoring and reporting architecture SolutionPacks and Collectors Dashboards Reports Thresholds Alerts Active Directory and LDAP server integration for reporting

32 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Introduction Native Hybrid Cloud Monitoring and Reporting resource management software provides comprehensive monitoring, reporting, and analysis for heterogeneous physical and virtual environments. Native Hybrid Cloud Monitoring and Reporting enables IT personnel to visualize relationships, analyze and report on capacity, performance, and health, and optimize resources in traditional and software-defined environments. The Native Hybrid Cloud Monitoring and Reporting platform provides the core of an out-of-the-box monitoring and reporting solution that you can customize and scale based on your environment and performance requirements. The platform is enhanced and customized with SolutionPacks that support a wide variety of EMC and third-party devices, hosts, and networks. Native Hybrid Cloud Monitoring and Reporting is an embedded software technology that is common across EMC ViPR, ViPR SRM, and EMC Service Assurance Suite. By using common technology, EMC provides customers with a consistent look, feel, and experience as they use these products together to manage their data center. The common technology also simplifies deployment, ongoing maintenance, and resource utilization. Monitoring and reporting services As shown in Figure 9, Native Hybrid Cloud Monitoring and Reporting and Pivotal Cloud Foundry Metrics provide monitoring and reporting services and help you analyze health, configurations and capacity growth. You can quickly spot SLA problems through custom dashboards and reports that meet the needs of a wide range of users and roles. With Native Hybrid Cloud Monitoring and Reporting, you can also view capacity consumption across your Pivotal Cloud Foundry deployment with built-in views to help you understand who is using capacity, how much they are using, and when more will be required. 32

33 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Figure 9. Pivotal Cloud Foundry metrics collection for Native Hybrid Cloud Monitoring and Reporting Monitoring and reporting architecture All Native Hybrid Cloud Monitoring and Reporting components for Native Hybrid Cloud, including the frontend, primary backend, and collector, are located on a single virtual machine. The components use built-in SolutionPacks and Collectors, including: Generic Host SolutionPack for monitoring all Native Hybrid Cloud virtual machines JMX Collector for data collection from Pivotal Cloud Foundry Metrics Stream Collector for data collection from Pivotal Operations Manager REST API Native Hybrid Cloud Monitoring and Reporting Health Collector Figure 10 shows the Native Hybrid Cloud Monitoring and Reporting architecture. 33

34 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Figure 10. Native Hybrid Cloud Monitoring and Reporting architecture SolutionPacks and Collectors A SolutionPack is an installable application that provides data collection and reporting capabilities for specific entities in your infrastructure. SolutionPacks provide a list of reports, pre-configured alerts, and automation required for Native Hybrid Cloud Monitoring and Reporting dependencies to monitor the managed infrastructure s performance. The SolutionPack Center makes updating a SolutionPack or changing a configuration, such as device credentials, as simple as picking applications from an application store. SolutionPacks support server, application, and virtualization systems along with many common third-party infrastructure components. Collectors retrieve raw data from multiple sources. Collectors can easily be scaled and deployed as business needs grow. 34

35 Generic Host SolutionPack Chapter 5: Native Hybrid Cloud Monitoring and Reporting This SolutionPack discovers and monitors physical and virtual servers and provides a global view of hosts in a data center. The Generic Host SolutionPack provides the ability to report, alert, and examine host, capacity usage, and performance measurements. The Host Enterprise Dashboard provides the following Generic Host reports: Host information and details Performance metrics such as CPU, memory, and disks for host devices File systems summary report Local and remote disk capacity reports Native Hybrid Cloud Monitoring and Reporting Health Monitor SolutionPack Native Hybrid Cloud Monitoring and Reporting Health Monitor SolutionPack monitors the health of your Native Hybrid Cloud Monitoring and Reporting infrastructure to keep the Native Hybrid Cloud Monitoring and Reporting services at optimal performance levels. It provides instant access to performance data, reports, and alerts. Native Hybrid Cloud Monitoring and Reporting Health main reports include: Modules performance Collecting level performance Backend and database events and utilization Server summary Native Hybrid Cloud Monitoring and Reporting Health metrics include: Collectors Event processing manager Backends Tomcat and frontend Databases JMX Collector The JMX Collector is designed to collect raw values from JMX enabled applications, such as Pivotal Cloud Foundry Metrics. The JMX Collector generates raw values and properties from MBean attribute values and uses regular expressions to transform values obtained from attributes and operations. You can apply multiple transformations on the same value. Stream Collector The Stream Collector is a collector module that imports data from various text-based sources using text streaming APIs. Unlike other text-based collectors such as Text Collector, XML Collector, and Remote Shell Collector, this collector parses data as it is received, instead of storing all the data in memory prior to parsing it. 35

36 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Dashboards The Stream Collector retrieves absolute values for Pivotal Cloud Foundry virtual machine memory, virtual CPUs, and ephemeral and persistent disks. It also retrieves the Pivotal Cloud Foundry application chargeback information The Native Hybrid Cloud Monitoring and Reporting user interface provides end-to-end visibility into the performance, health, availability, and configuration of infrastructure; this visibility includes hosts, storage, adapters, and switches, as shown in Figure 11. You can also view virtual storage, storage chargeback, and capacity for object and file storage. Figure 11. Native Hybrid Cloud Monitoring and Reporting dashboard Figure 12 shows the Native Hybrid Cloud Monitoring and Reporting Health dashboard, which displays a summary of component health. Green, yellow, and red are used to indicate good health, potential problems, and components that need immediate attention. 36

37 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Figure 12. Native Hybrid Cloud Monitoring and Reporting Health dashboard Reports The Native Hybrid Cloud report library includes reports for logs, reports for Pivotal Cloud Foundry runtime resources and operations, chargeback reports, and Pivotal Cloud Foundry health reports. Table 4, Table 5, Table 6, Table 7, and Table 8 show the available Native Hybrid Cloud Monitoring and Reporting reports. Table 4. Alerts, chargeback, and log reports Alerts Chargeback Logs (Kibana) Alerts Summary Chargeback by Organization Dashboard All Alerts Chargeback by Space Filters Top Components with Critical Alerts Chargeback by Service Plan 37

38 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Alerts Chargeback Logs (Kibana) Unacknowledged Alerts Unassigned Alerts Alerts for Watched Components High Impact Alerts Chargeback by Application Table 5. Hosts and Native Hybrid Cloud Monitoring and Reporting health reports Native Hybrid Cloud hosts Host Inventory Host Information File System Usage Host Situations to Watch Native Hybrid Cloud Monitoring and Reporting health Native Hybrid Cloud Monitoring and Reporting server stress CPU Usage Memory Usage File System Usage Collection Health Database Health Server Critical Logs Table 6. Blobstore and situations to watch reports S3-compatible Blobstore Blobstore Health Reports Blobstore Health Alerts Situations to watch Application Staging Issues CloudController Health Issues Logging Health Issues etcd Key-Value Store Health Issues Table 7. Applications and dashboard overview reports Applications Organizations Org Summary with Spaces, Services, Apps, and Users Org Summary with Spaces and Quotas Dashboard overview Alerts Alerts Summary (Link to summary) Alert Summary History (Link to history) 38

39 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Applications Spaces Spaces Summary with Services, Service Plans, and Apps Spaces Details Applications Apps Summary with Users, Services, and Resources Consumption Applications Details Dashboard overview Capacity Pivotal Cloud Foundry Total Resource Consumption vs. Free (Bar Chart) Pivotal Cloud Foundry Spaces by Quota Consumption (Bar Chart) Pivotal Cloud Foundry Orgs by Total Used Capacity (Stacked Chart with Trend) S3-compatible Blobstore Capacity Utilization (Bar Chart) Performance Top-5 Apps by Resource Consumption (vcpu, RAM, Disk Hours) (Bar Chart) Top-5 Orgs by Resources Consumption (vcpu, RAM, Disk Hours) (Bar Chart) Services Service Summary with Service Plans and Apps Service Plans Buildpacks Droplets Users Table 8. Pivotal Cloud Foundry operations and runtime resources reports Pivotal Cloud Foundry operations Global Status Stress Heatmap of Pivotal Cloud Foundry Components by Health and Alerts Pivotal Cloud Foundry runtime resources Pivotal Cloud Foundry virtual machine inventory runtime resources (vcpu, RAM, Disk Table) CloudController CC Health (Table) DEA (Diego) DEA (Diego) Health (Table) DEA (Diego) Performance (Table) DEA (Diego) Droplet Status (Table) DopplerServer DopplerServer Health (Table) 39

40 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Pivotal Cloud Foundry operations HealthManager HealthManager Health (Table) Applications Health (Table) Pivotal Cloud Foundry runtime resources LoggregatorTrafficController Loggregator Health (Table) Router Router Health (Table) Router Performance (Table) Collector Collector Performance (Table) etcd Key-Value Store etcd Key-Value Store Health (Table) UAA UAA Health (Table) Figure 13 shows the Native Hybrid Cloud Reports dashboard. 40

41 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Figure 13. Native Hybrid Cloud Reports dashboard Custom reports Native Hybrid Cloud Monitoring and Reporting reports enable operators to combine information from various existing reports into a customized report specific to your enterprise. You can edit report definitions, including timespans and sampling types, and make customizations visible to other users. The pinned reports feature lets you create new reports that suit your needs by reusing existing tables and graphs and combining them onto a single customized report. Each pinned report becomes an element on the customized report. You can rearrange those elements in the My Reports node. This node is a workspace for creating customized reports and testing changes before making the reports available to other users. In a fresh installation, this node might not be available until you create a new report. When the customized report is ready, you can make it visible to other users by placing it in a new location in the report tree. Additional report customization features are available in Edit mode. This mode lets you change items such as the style of the report graphs, the frequency of samples shown, and the range of data to include. Changes that you make are saved in the 41

42 Chapter 5: Native Hybrid Cloud Monitoring and Reporting original report and are visible to all users immediately after you apply them. The changes apply to all views of the report, whether the report is viewed from its original location in the report tree or pinned as an element in a customized report. Edit mode also permits more advanced operations, such as applying filters and expansions, creating formulas for new statistics, and defining new reports. You can export and schedule customized reports the same as you would for existing reports. Create custom Native Hybrid Cloud Monitoring and Reporting reports Administrators can develop and deploy custom reports in the Native Hybrid Cloud Monitoring and Reporting server, which can then be viewed in the Native Hybrid Cloud Monitoring and Reporting front-end server. Steps 1. Log in to an instance of Native Hybrid Cloud Monitoring and Reporting where the JMX collector is configured. For example, Native Hybrid Cloud Monitoring and Reporting URL>:58080/APG/admin/, using the credentials admin/changeme. 2. Click Report Pack and create a new report pack. 3. Type the required fields and upload the XML file to add the new template. The XML is also available under Reports. 4. Browse to Native Hybrid Cloud Monitoring and Reporting URL>:58080/APG/ and view the reports under PCF Operations. Scheduled Reports Scheduled Reports are like the Stored Reports feature but with more options and features. Scheduled Reports are generated based on a schedule. Using Scheduled Reports allows you execute or run a report on a regular basis, according to your specification (for example, off-load, off-hours, sent by mail, stored, and so on). You can choose and adjust the report so it displays the information you need and set various parameters of a scheduled report. Exporting reports From the Export tab, you have access to a full set of options for exporting the current report. You can export a report in these formats: PDF format generates the report in a PDF format you can save or print CSV format exports the report in a comma separated values file XLS format exports the report to Microsoft Excel XLS 1 column exports the report to Microsoft Excel, but all values are in one column PNG image exports the graphic to an image JPEG image exports the graphic to an image SVG image exports the graphic to an image 42

43 XML format dependent upon current mode Chapter 5: Native Hybrid Cloud Monitoring and Reporting Browse Mode exports the report s data to an XML file, which can then be used, for example, by a third party tool Edit Mode exports the report s template, which can be re-imported in Watch4net Sharing Native Hybrid Cloud Monitoring and Reporting reports You can share reports using: if a report is sent by , the fields in this tab can be used to configure the report format and destination. This allows the report to be sent by . The report format can be specified, as well as a subject and a short message. HTML tags are allowed in report s. Web additionally, Native Hybrid Cloud Monitoring and Reporting reports can be shared through the web by embedding the report URL into a custom portal. Thresholds Native Hybrid Cloud Monitoring and Reporting uses threshold settings to apply visual cues to values that fall within defined ranges. When a value s threshold is breached or it falls outside of the defined range, an alert is generated. The visual cues are for informational purposes only. They do not have any other effects in the reporting system. The visual cues can enhance your interpretation of the metrics. For example, they can: Help you classify the data Draw attention to high or low values Provide reassurance that connections or availability are working Provide additional information useful in your working environment Some reports come with predefined thresholds. You can edit existing threshold boundaries and threshold types, add new thresholds, and delete thresholds. Alerts Native Hybrid Cloud Monitoring and Reporting generates threshold alerts based on metric data that is consolidated from external sources. You must enable the threshold alert definitions for each source. You can use the default alert definitions available or create a custom definition and then enable it. Alert reports support these features that allow you to examine data in greater detail: From the summary-level bar and pie charts, you can click a bar or a section in a pie chart to open a tabular report that lists the alerts in the summarized category. From most of the tabular reports, you can click a row in the table to open a detailed report for a single alert. 43

44 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Administrators can mark alerts as acknowledged and take ownership of alerts. These actions are convenient, optional ways to track and organize alerts. When there is a reoccurrence of an alert, the latest alert is shown on a report and the state of the existing alert changes from ACTIVE to INACTIVE. When an alert becomes INACTIVE, it is automatically closed. When a MOMENTARY alert is acknowledged, the state of the alert changes from ACTIVE to INACTIVE and the alert is closed. Closed alerts are not available on the All Alerts Report. When a DURABLE alert is acknowledged, it remains ACTIVE and remains on the alerts reports with Yes in the ACK column. Figure 14 shows the Native Hybrid Cloud Alerts dashboard. Figure 14. Native Hybrid Cloud Alerts dashboard Active Directory and LDAP server integration for reporting 44

45 Chapter 5: Native Hybrid Cloud Monitoring and Reporting Native Hybrid Cloud Monitoring and Reporting uses a Java interface contained within Apache Tomcat to connect to an external managed security server such as Active Directory or LDAP servers. This interface consists of multiple plug-ins that support connections to different types of user access databases. JNDIRealm is the plug-in used for LDAP server integration. Realm is a database that Tomcat uses to identify authorized web application users. The default realm in ViPR SRM is a local realm. In a local realm, the user accounts and roles are stored in the local master database. LDAP authentication mechanisms rely on the JNDI Directory Realm. Java Naming and Directory Interface (JNDI) is a set of APIs used to interact with naming services and directory services. When connecting with a LDAP user, Native Hybrid Cloud Monitoring and Reporting creates a user entry in the local database and classifies the user as an external user so that the administrator can add access rights. 45

46 Chapter 6: Logging Chapter 6 Logging This chapter presents the following topics: Introduction Loggregator Native Hybrid Cloud logging

47 Chapter 6: Logging Introduction Native Hybrid Cloud logging capabilities are an integral part of the existing customer logging technologies and process. In addition to the existing Pivotal Cloud Foundry log aggregation features, Native Hybrid Cloud also makes use of the open source tools, such as Elasticsearch, Logstash and Kibana. Native Hybrid Cloud also includes preconfigured log filters and dashboards to simplify the analysis of event logs and log-based troubleshooting. Loggregator Native Hybrid Cloud logging Loggregator, the Pivotal Cloud Foundry component responsible for logging, provides a stream of log output from your application and from Pivotal Cloud Foundry system components that interact with your application during updates and execution. By default, Loggregator streams logs to your terminal. To retain more than the limited amount of logging information that Loggregator can buffer, this solution uses the ELK Stack to collect, analyze and display logs for management and operational analysis. The logs visualized in Kibana are presented to the user by a Native Hybrid Cloud Monitoring and Reporting report configured as an external report type. Native Hybrid Cloud logging uses the Elastic platform of open source projects designed to search, analyze, and visualize your data, allowing you to get actionable insight in real time. Elasticsearch Elasticsearch is a distributed open source search and analytics engine, designed for horizontal scalability, reliability, and easy management. It combines search capabilities and powerful analytics with a sophisticated, developer-friendly query language covering structured, unstructured, and time-series data. Elasticsearch is based on Apache Lucene. It provides a distributed, multitenantcapable full-text search engine with a RESTful web interface and schema-free JSON documents. Based on a Performance and Scale (P&S) evaluation of Elasticsearch, EMC Professional Services may require fewer or more Elasticsearch server nodes. These server nodes house the search service and its indexes and databases. Logstash Logstash is a flexible, open source data collection, enrichment, and transportation pipeline. With connectors to common infrastructure for easy integration, Logstash receives, processes, and outputs logs, and presents a powerful pipeline for log storage, query and analysis. Logstash is designed to efficiently process a growing list of log, event, and unstructured data sources for distribution into a variety of outputs, including Elasticsearch. Logstash runs a Native Hybrid Cloud Monitoring and Reporting node inside a container. It must be dynamically configured to communicate to a corresponding elastic search service and Native Hybrid Cloud Monitoring and Reporting service. 47

48 Chapter 6: Logging Kibana Kibana is an open source data visualization platform that allows you to graphically interact with your data. It is a data visualization plug-in for Elasticsearch that provides visualization capabilities in addition to the log content indexed on an Elasticsearch cluster. Users can create bar charts, line and scatter plots, or pie charts and maps from large volumes of logs. Kibana s browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time. You can construct search result visualizations in the Visualization page. Each visualization is associated with a search. For more information on configuring log visualization with Kibana, refer to the Kibana User Guide. 48

49 Chapter 7: Data Protection Chapter 7 Data Protection This chapter presents the following topics: Introduction Pivotal Cloud Foundry data protection Native Hybrid Cloud data protection Native Hybrid Cloud Installer virtual machine restoration Native Hybrid Cloud Blobstore restoration Native Hybrid Cloud Monitoring and Reporting virtual machine restoration

50 Chapter 7: Data Protection Introduction The built-in capabilities of Pivotal Cloud Foundry, Native Hybrid Cloud, and VxRack Neutrino provide the data protection and recovery capabilities for this solution. Pivotal Cloud Foundry data protection Pivotal Operations Manager lets you export the current Pivotal Cloud Foundry installation settings and assets. When you export an installation, the exported file contains the base virtual machine images, necessary packages, and references to the installation IP addresses. The Pivotal data protection procedure includes steps to create copies of critical databases, such as the Cloud Controller, User Account and Authentication (UAA) database, MySQL databases, and Applications Manager databases. EMC Professional Services or an expert administrator can perform the data protection procedure. For more information, refer to Backing Up Pivotal Cloud Foundry. The complete state of a Pivotal Elastic Runtime deployment is protected by capturing the following: Pivotal Cloud Foundry installation settings Critical databases: Pivotal MySQL server database Cloud Controller database UAA database Applications Manager database S3-compatible Blobstore Note: Native Hybrid cloud replaces the Network File System (NFS) server storage used by Pivotal Cloud Foundry with a S3-compatible Blobstore. To create a backup copy of an Elastic Runtime deployment, you must temporarily stop the Cloud Controller virtual machines (cloud_controller_partition and cloud_controller_worker) and save all of the above and your database encryption credentials to restoration files. To restore a deployment, you must temporarily stop the Cloud Controller virtual machines and restore the state of each component from the restoration files. For instruction on protecting Elastic Runtime, refer to Backing Up Pivotal Cloud Foundry. For instructions on restoring Elastic Runtime, refer to Restoring Pivotal Cloud Foundry from Backup. Export installation settings Pivotal recommends that you frequently export your installation settings. Always export an installation before importing a new one. Import an installation to restore your settings or to share your settings with another user. 50

51 Chapter 7: Data Protection Pivotal Operations Manager enables you to export the current Pivotal Cloud Foundry installation settings and assets. When you export an installation, the exported file contains the base virtual machine images, necessary packages, and references to the installation IP addresses. Native Hybrid Cloud data protection Native Hybrid Cloud Installer virtual machine protection The Native Hybrid Cloud Installer virtual machine is protected by taking snapshots at a customer-specified interval, which is typically once daily. These virtual machine snapshot images are stored in the Glance image repository. Native Hybrid Cloud virtual Blobstore protection The S3-compatible Blobstore is backed up at a customer-specified interval, which is typically once daily. The automated process involves quiescing I/O, flushing all data down to disk, taking a snapshot of the storage volumes, and then un-quiescing the I/O. A golden copy volume is then created from each snapshot, and can be used immediately. The snapshots and old backup volumes are deleted on successful creation of the new backup volumes. Native Hybrid Cloud Monitoring and Reporting virtual machine protection The Native Hybrid Cloud Monitoring and Reporting solution is an OpenStack virtual machine and an associated OpenStack Cinder volume. Daily backups of Native Hybrid Cloud Monitoring and Reporting include backups of the virtual machine and its associated volume. Native Hybrid Cloud creates a backup schedule for the Native Hybrid Cloud Monitoring and Reporting virtual machine. The typical backup interval, which an administrator can modify, is 24 hours. This preserves the historical data collected by Native Hybrid Cloud Monitoring and Reporting. Note: Only the last Native Hybrid Cloud Installer, Native Hybrid Cloud Monitoring and Reporting, and Blobstore backups are retained. Native Hybrid Cloud Installer virtual machine restoration The Native Hybrid Cloud Installer virtual machine can be restored in case of failure. Before you restore it you must verify that a valid backup of the Native Hybrid Cloud Installer virtual machine exists in OpenStack. Then you can delete the failed Native Hybrid Cloud Installer virtual machine. Steps 1. In the OpenStack dashboard, go to Project > Compute > Images. 2. Verify that the OpenStack images contain a snapshot image of the Native Hybrid Cloud Installer virtual machine. Valid snapshots are named in the following format: NHC-InstallerDD-MM-YYYY 51

52 Chapter 7: Data Protection 3. In the OpenStack dashboard, go to Project > Compute > Instances to remove the Native Hybrid Cloud Installer virtual machine. 4. Type nhc-installer in the Instance Name field and click Filter to filter on the Native Hybrid Cloud Installer virtual machine instance, as shown in Figure 15. Figure 15. Native Hybrid Cloud Installer virtual machine instance 5. If the Native Hybrid Cloud Installer virtual machine is not deleted and is in an Error state, click the Action list box and select Terminate Instance, as shown in Figure

53 Chapter 7: Data Protection Figure 16. Select terminate instance Follow the steps below to restore the Native Hybrid Cloud Installer virtual machine when the instance is terminated. Note: Contact EMC Online Support for assistance if you cannot delete the Native Hybrid Cloud Installer virtual machine. Steps 1. Retrieve the private IP address that was assigned to the Native Hybrid Cloud Installer virtual machine. This was recorded by EMC Professional Services in the Native Hybrid Cloud Configuration Workbook. 53

54 Chapter 7: Data Protection 2. In a web browser, open a connection to the Neutrino OpenStack dashboard. 3. Log in to the OpenStack dashboard as the Native Hybrid Cloud tenant. 4. Go to Project > Compute > Images. 5. Locate the Native Hybrid Cloud Installer virtual machine snapshot image, as shown in Figure 17. Figure 17. Native Hybrid Cloud snapshot images 6. Select the image name to view the image details. 7. Record the Image ID in the Image Overview screen, as shown in Figure 18. Figure 18. Snapshot image details 8. Go to Project > Compute > Instances. 9. Locate a running Native Hybrid Cloud virtual machine in the Instances screen, as shown in Figure

55 Chapter 7: Data Protection Figure 19. Instances screen 10. Select one of the running instance names to view the instance details. 11. Scroll down to the Metadata section and record the key name, as shown in Figure 20. Figure 20. Key name 12. Go to Project > Network > Networks in the OpenStack dashboard. 13. Select the network name for Native Hybrid Cloud, as shown in Figure 21. Figure 21. Networks screen 14. Record the ID of the private network in the Network Details screen, as shown in Figure

56 Chapter 7: Data Protection Figure 22. Network details 15. Select one of the running Native Hybrid Cloud virtual machines. 16. Select Associate Floating IP in the list box, as shown in Figure 23. Figure 23. Associate a floating IP address 17. Select a floating IP address in the IP Address menu, as shown in Figure 24. Figure 24. Manage floating IP address associations a. If there are no floating IP addresses available, click the + icon to associate a new floating IP to the Native Hybrid Cloud tenant, as shown in Figure

57 Chapter 7: Data Protection Figure 25. Associate a new floating IP address b. In the Allocate Floating IP screen, click Allocate IP to allocate a new floating IP address, as shown in Figure 26. Figure 26. Allocate a floating IP address 18. When the floating IP address is selected, click Associate to associate the floating IP address with the selected virtual machine, as shown in Figure

58 Chapter 7: Data Protection Figure 27. Manage floating address associations 19. Open an SSH session to the floating IP address you have just assigned, using your Native Hybrid Cloud key pair file and the root account. 20. Download the OpenStack RC file to authenticate against the OpenStack command line from the following location in Horizon: Project > Compute > Access & Security-> API Access. 21. Click Download OpenStack RC File v In the SSH session create a new file using a text editor and paste the contents of the RC file into the new file. 23. Change the OS_AUTH_URL version from v3 to v2.0 as shown below and save your changes. export OS_AUTH_URL= Figure 28. OS_AUTH_URL version 24. Source the RC file using the following command: source <rc_filename> 25. Run the command below to restore the Native Hybrid Cloud Installer virtual machine from the snapshot image, using the retrieved image ID, NHCSecurityGroup, key pair name, network ID, and private IP address from the Native Hybrid Cloud Configuration Workbook: nova boot NHC-Installer --image IMAGE --flavor 2 \ --security-groups SECURITYGROUP --key-name KEYNAME \ -nic net-id=neutron_network_id,v4-fixed-ip=ip_address 26. Return to the OpenStack Dashboard when the Nova boot command is complete. 58

59 Chapter 7: Data Protection 27. When the Native Hybrid Cloud Installer virtual machine has successfully booted, follow the steps outlined above to re-assign a floating IP address to the virtual machine. 28. Remove the floating IP from the Native Hybrid Cloud Installer virtual machine that was temporarily used to issue the Nova command to create the Native Hybrid Cloud Installer virtual machine. When the Native Hybrid Cloud virtual machine is restored, subsequent backup cycles create backup copies of the restored running instance. Native Hybrid Cloud Blobstore restoration If a proxy or storage node is lost, there is no impact due to the high availability design of the Blobstore. Customers can create a new node. For a storage node, customers can create a new storage volume and synchronize data to it from the remaining nodes. It is not restored from a copy. Customers must initiate all restores. Native Hybrid Cloud provides support for three restore scenarios: Restore from complete failure Restore a single proxy node Restore a single storage node and associated volume The sections below outline the processes for restoring from each of these failure scenarios. Note: Contact EMC Online Support if you cannot restore the storage node or proxy node to their original private IP addresses. All Blobstore restores must be initiated from the Native Hybrid Cloud Installer virtual machine. Access the Native Hybrid Cloud Installer virtual machine using these steps. Steps 1. Open the Neutrino OpenStack user interface in a web browser. 2. Log in using Native Hybrid Cloud tenant credentials. 3. Go to Project > Compute > Instances. 4. Use your OpenStack Native Hybrid Cloud SSH key pair to open an SSH session to the Native Hybrid Cloud Installer virtual machine s floating IP address. ssh -i emcnhc.pem root@ xxx.xxx Note: The OpenStack Native Hybrid Cloud key pair is provided by EMC Professional Services on completion of the installation and configuration of this solution. 59

60 Chapter 7: Data Protection Restore from complete failure Follow this restore from a complete failure of the Native Hybrid Cloud Blobstore, where all storage nodes, proxy nodes, and storage volumes have failed or have been deleted. Steps 1. Verify that all three storage nodes, both proxy nodes, and the three storage volumes have failed. 2. Terminate the storage node and proxy node instances if they are not already deleted. Storage nodes are named nhc-swift-storage-# and proxy nodes are named nhc-swift-proxy-#. 3. Delete the storage node volumes if they are not already deleted. Storage volumes are named swift-<storage_node_ip>. 4. Run the nhc-blobstore redeploy command to create new storage node and proxy node instances and attach the backup volumes. Refer to the /opt/emc/nhc/config/nhc-fixed-ip.yml file for the proxy node and storage node IP addresses. nhc-blobstore redeploy --proxy-nodes <proxy_node_1_ip>,<proxy_node_2_ip> --storage-nodes <storage_node_1_ip>,<storage_node_2_ip>,<storage_node_3_ip> 5. Verify that the storage nodes, proxy nodes, and storage volumes are created. 6. Verify that applications can be successfully pushed. 7. Perform a fresh backup using the swift-backup command. Restore a single proxy node Follow the steps outlined below to restore a single proxy node in the Native Hybrid Cloud Blobstore. Steps 1. Verify that the proxy node has failed. 2. Terminate the proxy node instances if they are not already deleted. Proxy nodes are named nhc-swift-proxy-#. 3. Run the nhc-blobstore redeploy command to create new proxy node instances. Refer to the /opt/emc/nhc/config/nhc-fixed-ip.yml file for proxy node IP addresses. nhc-blobstore redeploy --proxy-nodes <proxy_node_ip> 4. Verify that the proxy node is created. 5. Verify that applications can be successfully pushed. 6. Perform a fresh backup using the swift-backup command. Restore a single storage node and associated volume 60

61 Chapter 7: Data Protection Follow the steps outlined below to restore a single Storage node and its volume in the Native Hybrid Cloud Blobstore. Steps 1. Verify that the storage node and its volume have failed. 2. Terminate the storage node instance if it is not already deleted. Storage nodes are names nhc-swift-storage-#. 3. Delete the associated storage node volume if it is not already deleted. Storage volumes are named swift-<storage_node_ip>. 4. From an SSH session to NHC Installer-VM, run the nhc-blobstore redeploy command to create new storage node instances and attach the backup volume. Refer to the /opt/emc/nhc/config/nhc-fixed-ip.yml file for proxy node and storage node IP addresses. nhc-blobstore redeploy --storage-nodes <storage_node_ip> -- from-backup 5. Verify that the storage node and storage volume are created. 6. Verify that applications can be successfully pushed. 7. Perform a fresh backup using the swift-backup command. Note: During redeployment of the Swift node, if you see one of the following messages, rerun the redeploy command: "mount /dev/vdb on /srv/node/vdb failed: Structure needs cleaning" or "mkfs.xfs: /dev/vdb appears to contain a partition table (DOS)." Native Hybrid Cloud Monitoring and Reporting virtual machine restoration Contact EMC Online Support if you need to restore the Native Hybrid Cloud Monitoring and Reporting virtual machine. 61

62 Chapter 8: User administration Chapter 8 User administration This chapter presents the following topics: Introduction VxRack Neutrino accounts and users Pivotal Cloud Foundry identity providers Native Hybrid Cloud Monitoring and Reporting users

63 Chapter 8: User administration Introduction This section provides information about configuring Native Hybrid Cloud users so authenticated users can log in and use Native Hybrid Cloud services and applications associated with their projects. VxRack Neutrino accounts and users In addition to built-in account services, VxRack Neutrino supports the addition of external identity providers, such as Active Directory or LDAP servers. New accounts, projects, and users can be created in the VxRack Neutrino local database using Accounts menu, as shown in Figure 29. Figure 29. VxRack Neutrino Account Management page Refer to the VxRack System 1000 with Neutrino 1.1 Administrator Guide for instructions on adding identity providers, accounts, projects, and users. Pivotal Cloud Foundry identity providers Pivotal Cloud Foundry enables you to create local users, and integrate them with external identity providers, as shown Figure 30. Local users are able to create organizations and spaces, as well as push new applications to the Pivotal Cloud Foundry platform. 63