Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide

Transcription

1 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide For Red Hat Mobile Application Platform 4.1 Red Hat Customer Content Services

2

3 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide For Red Hat Mobile Application Platform 4.1

4 Legal Notice Copyright 2017 Red Hat, Inc. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Java is a registered trademark of Oracle and/or its affiliates. XFS is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project. The OpenStack Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community. All other trademarks are the property of their respective owners. Abstract This document provides guides related to installation and administration of the RHMAP 4.x MBaaS on OpenShift 3.

5 Table of Contents Table of Contents. CHAPTER RHMAP X... MBAAS OVERVIEW ARCHITECTURE OF THE MBAAS SECURITY CONSIDERATIONS 4. CHAPTER INSTALLING THE.... MBAAS INSTALLATION STEPS 6. CHAPTER PREPARING NODES FOR.... MBAAS INSTALLATION OVERVIEW PREREQUISITES PROCEDURE NEXT STEPS 10. CHAPTER PROVISIONING AN.... RHMAP X... MBAAS IN.. OPENSHIFT OVERVIEW PREREQUISITES AUTOMATIC INSTALLATION MANUAL INSTALLATION CREATING AN MBAAS TARGET NEXT STEPS 22. CHAPTER ADJUSTING SYSTEM RESOURCE USAGE OF... THE.... MBAAS AND..... CLOUD APPS OVERVIEW PREREQUISITES ADJUSTING RESOURCE USAGE OF THE MBAAS ADJUSTING RESOURCE USAGE OF CLOUD APPS SETTING RESOURCE REQUESTS AND LIMITS USING CLUSTER METRICS TO VISUALIZE RESOURCE CONSUMPTION 26. CHAPTER SETTING UP... SMTP FOR.... CLOUD APP.... ALERTS OVERVIEW PREREQUISITES CONFIGURING SMTP SETTINGS IN FH-MBAAS VERIFYING SMTP SETTINGS TROUBLESHOOTING 27. CHAPTER USING MONGODB IN.. AN... RHMAP X... MBAAS OVERVIEW ACCESSING DATA IN THE MONGODB IN THE MBAAS 29. CHAPTER CENTRALIZED LOGGING FOR.... MBAAS COMPONENTS OVERVIEW ACCESSING LOGS THROUGH KIBANA WEB CONSOLE IDENTIFYING ISSUES IN AN MBAAS 31. CHAPTER MONITORING THE.... MBAAS WITH..... COCKPIT OVERVIEW INSTALLATION VIEWING THE CONTAINERS ON AN OPENSHIFT NODE VIEWING MULTIPLE HOSTS SIMULTANEOUSLY 34. CHAPTER MONITORING THE.... MBAAS WITH..... NAGIOS OVERVIEW 35 1

6 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide PREREQUISITES DEPLOYING NAGIOS TO OPENSHIFT USING NAGIOS CHAPTER TROUBLESHOOTING THE.... RHMAP X... MBAAS OVERVIEW CHECK THE HEALTH ENDPOINT OF THE MBAAS ANALYZE LOGS COMMON PROBLEMS. CHAPTER KNOWN ISSUES IN... THE.... RHMAP MBAAS OVERVIEW DELETING OPENSHIFT SSH KEY BREAKS CLOUD APP DEPLOYMENTS CLOUD APP ANALYTICS DATA DOES NOT UPDATE INCORRECT DEPLOYMENT STATUS INDICATED BY PROGRESS BAR FOR SUCCESSFUL DEPLOYMENTS MONGODB DOESN T WORK AFTER A RESTART MONGODB POD STARTING IN REMOVED STATE

7 Table of Contents 3

8 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide CHAPTER 1. RHMAP 4.X MBAAS 1.1. OVERVIEW Red Hat Mobile Application (RHMAP) 4.0 has a hybrid deployment model the Core, the MBaaS, and the Build Farm are deployed in different locations. Development and management of apps occurs in the multi-tenant cloud instance of the RHMAP Core hosted by Red Hat. Application data, runtime, and integrations are deployed to the RHMAP MBaaS installed in a private or public instance of OpenShift Enterprise 3. The Build Farm is deployed separately from the Core and the MBaaS and is shared between all instances of RHMAP. Third-party Linux, Windows, and Apple server hosting providers are used to support building client app binaries for all platforms. The Mobile Backend-as-a-Service (MBaaS) is a core component of RHMAP the back-end platform hosting containerized cloud applications in conjunction with database storage (MongoDB). The cloud applications deployed in an MBaaS can make use of RHMAP APIs, such as data synchronization, caching, or push notifications, and integrate with enterprise systems or other cloud services ARCHITECTURE OF THE MBAAS The RHMAP MBaaS 4.0 is built on top of several technologies, including OpenShift Enterprise 3, Kubernetes, Docker, and Red Hat Software Collections. The MBaaS consists of several components, each running in its own Docker container. Similarly, every cloud app deployed to the MBaaS runs in a Docker container. Those containers are deployed and orchestrated by Kubernetes. In the MBaaS, the users can configure multiple isolated runtime and storage environments to support software development life-cycle stages, such as development, testing, and production. Each environment can host multiple cloud apps SECURITY CONSIDERATIONS 4

9 CHAPTER 1. RHMAP 4.X MBAAS Since the MBaaS is not hosted in Red Hat s public multi-tenant cloud, the data transmitted between the mobile device and the cloud app does not pass through any servers operated by Red Hat or any other third party. Private data from back-end systems is transmitted directly between mobile devices and the MBaaS. The following data still resides in the RHMAP Core: User names and passwords of RHMAP accounts Master database of the Core, with entries for projects, apps, and their IDs Git repositories hosting the source code of client and cloud apps App store containing the built binaries of client apps 5

10 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide CHAPTER 2. INSTALLING THE MBAAS This guide provides a detailed overview of the steps required to get from purchasing a Red Hat Mobile Application Platform (RHMAP) 4.x subscription to a working installation of an MBaaS connected to your RHMAP domain INSTALLATION STEPS After Red Hat receives the purchase order for a Red Hat Mobile Application Platform (RHMAP) 4.x subscription, a member of the sales team internally requests a new RHMAP domain for access to an instance of the RHMAP Core hosted by Red Hat. Once the domain is created, a representative of the Red Hat Customer Enablement team will instruct you to install the MBaaS, which involves installation of Red Hat Enterprise Linux (RHEL), installation of OpenShift Enterprise, provisioning of the MBaaS in the OpenShift cluster, and installation of other optional components Getting Access to RHMAP Core The following steps for getting access to RHMAP Core are performed by a representative of the Red Hat Customer Enablement team: 1. Create a domain. The domain, such as customername.redhatmobile.com, hosts the RHMAP Core for a single customer. 2. Create an administrator account. An RHMAP administrator account is created in the domain, and the customer s technical contact receives an activation which allows access to the domain using the new account Preparing Nodes for MBaaS Installation The following steps are covered in the Preparing Nodes for MBaaS Installation guide. 1. Install RHEL on each cluster node as per the RHEL Installation Guide. You can Download RHEL from the Red Hat Customer Portal. 2. Register each node with Red Hat Subscription Manager (RHSM). Follow step 2 in the Preparing Nodes for MBaaS Installation guide. 3. Install OpenShift as per the OpenShift Installation and Configuration guide. Follow the considerations for infrastructure sizing to determine how many nodes to configure in your OpenShift cluster. 4. Enable Docker on each node to access container images of RHMAP components hosted in the Red Hat Docker registry. Follow step 4 in the Preparing Nodes for MBaaS Installation guide. 6

11 CHAPTER 2. INSTALLING THE MBAAS Installing the MBaaS After the nodes in the cluster have RHEL installed, OpenShift installed, and are registered with RHSM, you can proceed to the main installation step. 1. Follow the Provisioning an RHMAP 4.x MBaaS in OpenShift 3 guide. As part of the provisioning process, OpenShift automatically downloads RHMAP container images from the Red Hat Docker registry. 2. Adjust system resource usage of MBaaS components. If the MBaaS components are deployed on dedicated nodes in your cluster (separate from cloud apps), we strongly recommend that you adjust the resource limits of MBaaS components to take full advantage of the available system resources. Follow the Adjusting System Resource Usage of the MBaaS and Cloud Apps guide for detailed steps Installing Additional Features 1. Set up cloud app alerts. Setting Up SMTP for Cloud App Alerts 2. Set up centralized logging. Centralized Logging for MBaaS Components 3. Set up monitoring. Monitoring the MBaaS with Cockpit Monitoring the MBaaS with Nagios 7

12 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide CHAPTER 3. PREPARING NODES FOR MBAAS INSTALLATION 3.1. OVERVIEW Before installation of the MBaaS, you must first register each node in the cluster with the Red Hat Subscription Manager (RHSM), and install OpenShift 3. The registration enables OpenShift to access the Docker container images of RHMAP components hosted in the Red Hat Docker registry PREREQUISITES Access to an RHMAP domain and an RHMAP administrator account. See Installing the MBaaS for more information PROCEDURE 1. Install RHEL. Install RHEL on each machine that will serve as a node in the OpenShift cluster backing the MBaaS. Follow the RHEL Installation Guide. 2. Register all cluster nodes using RHSM and attach the nodes to the RHMAP subscription. Perform the following procedure for each node in the cluster. a. Register the node with RHSM. Replace <username> and <password> with the user name and password for your Red Hat account. sudo subscription-manager register --username=<username> -- password=<password> Registering to: subscription.rhn.redhat.com:443/subscription The system has been registered with ID: abcdef abcd b. List the available subscriptions. sudo subscription-manager list --available c. Find the pool ID for an RHMAP subscription and attach it. sudo subscription-manager attach --pool=<pool_id> You will see output similar to the following: Successfully attached a subscription for: {ProductName} 8

13 CHAPTER 3. PREPARING NODES FOR MBAAS INSTALLATION 3. Install OpenShift. See the Installation and Configuration guide in the OpenShift documentation for detailed installation procedure. Follow the considerations for infrastructure sizing to determine how many nodes to configure in your OpenShift cluster. Note In the OpenShift Installation and Configuration guide: Skip steps 1, 2 and 3 in section Software Prerequisites Registering the Hosts, which describe the same registration process that is already covered in this guide in step 1. Choose the default, RPM-based installation method. See section 2.3. RPM vs. Containerized for more details. 4. Enable Docker to access container images of RHMAP components. Perform the following procedure for each node in the cluster. a. Enable the rhel-7-server-optional-rpms repository. sudo subscription-manager repos --enable=rhel-7-serveroptional-rpms Repository 'rhel-7-server-optional-rpms' is enabled for this system. b. Install the RHSM plugin subscription-manager-plugin-container. sudo yum install subscription-manager-plugin-container c. Run rhsmcertd-worker to refresh the local certificate store. rhsmcertd-worker must be run as the superuser, otherwise it may fail to work without a warning. sudo /usr/libexec/rhsmcertd-worker To verify that the certificates were downloaded, check the contents of the /etc/docker/certs.d/ directory. ls -l /etc/docker/certs.d/ grep access.redhat.com /etc/docker/certs.d/ now contains directories access.redhat.com and registry.access.redhat.com. 9

14 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide drwxr-xr-x. 2 root root 67 Jun 01 10:30 access.redhat.com drwxr-xr-x. 2 root root 67 Jun 01 10:30 registry.access.redhat.com After registering each node with RHSM, downloading the entitlement certificates, and installing OpenShift, you can proceed to installation of the MBaaS NEXT STEPS Provisioning an RHMAP 4.x MBaaS in OpenShift 3 10

15 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT OVERVIEW An OpenShift 3 cluster can serve as an MBaaS target and host your Cloud Apps and Cloud Services. This guide provides detailed steps to deploy the RHMAP 4.x MBaaS on an OpenShift 3 cluster. You can choose a simple automated installation to preview and test the MBaaS, or follow the manual installation steps for a fully supported production-ready MBaaS: Automatic Installation You can quickly try the RHMAP 4.x MBaaS by choosing the automatic installation. The following limitations apply to the automatically installed MBaaS: not suitable for production use single replica for each MBaaS component single MongoDB replica with no persistent storage Manual Installation For production use, follow the manual installation procedure, which results in an MBaaS with the following characteristics: suitable for production use three replicas defined for each MBaaS component (with the exception of fh-statsd) three MongoDB replicas with a 50GB persistent storage requirement each nodeselectors of mbaas_id=mbaas1, mbaas_id=mbaas2, and mbaas_id=mbaas3 for the MongoDB replicas 4.2. PREREQUISITES This guide assumes several prerequisites are met before the installation: All nodes in the cluster must be registered with the Red Hat Subscription Manager and have RHMAP entitlement certificates downloaded. See Preparing Nodes for MBaaS Installation for detailed steps. An existing OpenShift Enterprise installation, version 3.2 The OpenShift master and router must be accessible from the RHMAP Core. A wildcard DNS entry must be configured for the OpenShift router IP address. A dedicated OpenShift user for MBaaS administration. This guide refers to this user as the MBaaS administrator. The user must have the authorization to: create projects; 11

16 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide create any objects associated with a project (for example, Services, DeploymentConfigs, Routes). An OpenShift user with the admin role is needed for the manual installation. For example, the default system:admin user. This guide refers to this user as the OpenShift administrator. For information on installation and management of an OpenShift cluster and its users, see the official OpenShift documentation AUTOMATIC INSTALLATION The automatic installation of an MBaaS in OpenShift 3 results in the MBaaS components being installed on nodes chosen by the OpenShift scheduler. Only a single instance of each component runs at any time and thus makes the MBaaS susceptible to downtime in case of failure of a single node. The data of the MongoDB database is not backed by any permanent storage and is therefore transient. Warning The automatic installation procedure must not be used in production environments. You should only use this procedure for evaluation purposes, since the provided template is not optimized for resiliency and stability required in production environments. Follow the manual installation steps for production use. There are no setup steps required before the automatic installation. Refer to Creating an MBaaS Target to continue the installation. Note In order for automatic MBaaS installation to work, the OpenShift SDN must be configured to use the ovs-subnet SDN plugin (this is the default). If it is not set to this, refer to Network Configuration MANUAL INSTALLATION The manual installation of an MBaaS in OpenShift 3 results in a resilient three-node cluster with MBaaS components spread across all three nodes, MongoDB replica set spread over three nodes, and the MongoDB data backed by persistent volumes. The installation consists of several phases. Before the installation, you must prepare your OpenShift cluster: Set up persistent storage - you need to create Persistent Volumes with specific parameters in OpenShift. Label the nodes - nodes need to be labeled in a specific way, to match the node selectors expected by the OpenShift template of the MBaaS. Network Configuration - configuring the SDN network plugin used in OpenShift so that Cloud Apps can communicate with MongoDB in the MBaaS. 12

17 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 After the OpenShift cluster is properly configured: Install the MBaaS from a template Verify the installation Before The Installation The manual installation procedure poses certain requirements on your OpenShift cluster in order to guarantee fault tolerance and stability Network Configuration Cloud Apps in an MBaaS communicate directly with a MongoDB replica set. In order for this to work, the OpenShift SDN must be configured to use the ovs-subnet SDN plugin. For more detailed information on configuring this, see Migrating Between SDN Plug-ins in the OpenShift Enterprise documentation Making Project Networks Global Alternatively, if you cannot use the ovs-subnet SDN plugin, you will need to make the network of the MBaaS project global after installation. For details on how to do this with your MBaaS project, see Making Project Networks Global in the OpenShift Enterprise documentation Persistent Storage Setup An MBaaS running on OpenShift 3 contains a MongoDB replica set. The replica set members use persistent storage for the directory where the database data is stored. The OpenShift template that is used to create the project and resources for the MBaaS requires: At least one PersistentVolume (PV) resource in an Available state for each of the three MongoDB replica members. Each PersistentVolume has at least 50GB of space. For detailed information on PersistentVolumes and how to create them, see Persistent Storage in the OpenShift Enterprise documentation Apply Node Labels By applying labels to OpenShift nodes, you can control which nodes the MBaaS components, MongoDB replicas, and cloud apps will be deployed to. This section describes the considerations for: Section , Labelling for MBaaS components Section , Labelling for MongoDB replicas Cloud apps get deployed to nodes labeled with the default nodeselector, which is usually set to type=compute (defined in the OpenShift master configuration). You can skip this entire labeling section if your OpenShift cluster only has a single schedulable node. In such case, all MBaaS components, MongoDB replicas, and cloud apps will necessarily run on that single node. 13

18 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide Labelling for MBaaS components It is recommended, but not required, to deploy the MBaaS components to dedicated nodes, separate from other applications (such as RHMAP cloud apps). Refer to Infrastructure Sizing Considerations for Installation of RHMAP MBaaS for the recommended number of MBaaS nodes and cloud app nodes for your configuration. For example, if you have 12 nodes, the recommendation is: Dedicate three nodes to MBaaS and MongoDB. Dedicate three nodes to cloud apps. To achieve this, apply a label, such as type=mbaas to the three dedicated MBaaS nodes. oc label node mbaas-1 type=mbaas oc label node mbaas-2 type=mbaas oc label node mbaas-3 type=mbaas Then, when creating the MBaaS project, as described later in Section 4.4.2, Installing the MBaaS, set this label as the nodeselector. You can check what type labels are applied to all nodes with the following command: oc get nodes -L type NAME STATUS AGE TYPE ose-master Ready,SchedulingDisabled 27d master infra-1 Ready 27d infra infra-2 Ready 27d infra app-1 Ready 27d compute app-2 Ready 27d compute app-3 Ready 27d compute mbaas-1 Ready 27d mbaas mbaas-2 Ready 27d mbaas mbaas-3 Ready 27d mbaas In this example, the deployment would be as follows: Cloud apps get deployed to the three dedicated cloud app nodes app-1, app-2, and app-3. The MBaaS components get deployed to the three dedicated MBaaS nodes mbaas-1, mbaas- 2, and mbaas-3 (if the nodeselector is also set on the MBaaS Project) Labelling for MongoDB replicas In the production MBaaS template, the MongoDB replicas are spread over three MBaaS nodes. If you have more than three MBaaS nodes, any three of them can host the MongoDB replicas. To apply the required labels (assuming the three nodes are named mbaas-1, mbaas-2, and mbaas-3): oc label node mbaas-1 mbaas_id=mbaas1 oc label node mbaas-2 mbaas_id=mbaas2 oc label node mbaas-3 mbaas_id=mbaas3 14

19 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 You can verify the labels were applied correctly by running this command: oc get nodes -L mbaas_id NAME STATUS AGE MBAAS_ID Ready 27d <none> Ready 27d <none> Ready 27d <none> Ready 27d mbaas Ready,SchedulingDisabled 27d <none> Ready 27d mbaas Ready 27d <none> Ready 27d <none> Ready 27d mbaas2 See Updating Labels on Nodes in the OpenShift documentation for more information on how to apply labels to nodes Why are MongoDB replicas spread over multiple nodes? Each MongoDB replica is scheduled to a different node to support failover. For example, if an OpenShift node failed, data would be completely inaccessible if all three MongoDB replicas were scheduled on this failing node. Setting a different nodeselector for each MongoDB DeploymentConfig, and having a corresponding OpenShift node in the cluster matching this label will ensure the MongoDB pods get scheduled to different nodes. In the production MBaaS template, there is a different nodeselector for each MongoDB DeploymentConfig: mbaas_id=mbaas1 for mongodb-1 mbaas_id=mbaas2 for mongodb-2 mbaas_id=mbaas3 for mongodb-3 Excerpt of DeploymentConfig of mongodb-1 { "kind": "DeploymentConfig", "apiversion": "v1", "metadata": { "name": "mongodb-1", "labels": { "name": "mongodb" } }, "spec": {... "template": {... "spec": { "nodeselector": { "mbaas_id": "mbaas1" } 15

20 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide Installing the MBaaS In this step, you will provision the MBaaS to the OpenShift cluster from the command line, based on the MBaaS OpenShift template. First, download the latest version of the MBaaS OpenShift template. 1. In the Studio, navigate to the Admin > MBaaS Targets section. Click New MBaaS Target. 2. Choose OpenShift 3 as Type. 3. At the bottom of the page, click Download Template and save the template file fh-mbaastemplate-3node.json. You may now close the New MBaaS Target screen. Using the downloaded template, provision the MBaaS in the OpenShift cluster from the command line. For general information about the OpenShift CLI, see CLI Operations in the OpenShift Enterprise documentation. 1. Create a new project. Log in as the MBaaS administrator. You will be prompted for credentials. oc login <public URL of the OpenShift master> Create the project: Warning The name of the OpenShift project chosen here must have the suffix -mbaas. The part of the name before -mbaas is used later in this guide as the ID of the MBaaS target associated with this OpenShift project. For example, if the ID of the MBaaS target is live, the OpenShift project name set here must be livembaas. oc new-project live-mbaas 2. Set the node selector of the project to target MBaaS nodes. This ensures that all MBaaS components are deployed to the dedicated MBaaS nodes. Note If you ve chosen not to have dedicated MBaaS nodes in Section , Labelling for MBaaS components, skip this step. Log in as the OpenShift administrator. You will be prompted for credentials. oc login <public URL of the OpenShift master> 16

21 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 Set the openshift.io/node-selector annotation to type=mbaas in the project s namespace: Note You may need to add this annotation if it is missing. oc edit ns live-mbaas apiversion: v1 kind: Namespace metadata: annotations: openshift.io/node-selector: type=mbaas... Log back in as the MBaaS administrator if you had to switch users for the above steps. You will be prompted for credentials. oc login <public URL of the OpenShift master> 3. Start the installation. Create all the MBaaS resources from the template. oc new-app -f fh-mbaas-template-3node.json After all the resources are created, you should see output similar to the following: --> Deploying template fh-mbaas for "fh-mbaas" With parameters: MONGODB_FHMBAAS_USER=u-mbaas... --> Creating resources... Service "fh-mbaas-service" created Service "fh-messaging-service" created Service "fh-metrics-service" created Service "fh-statsd-service" created Service "mongodb-1" created Service "mongodb-2" created Service "mongodb-3" created DeploymentConfig "fh-mbaas" created DeploymentConfig "fh-messaging" created DeploymentConfig "fh-metrics" created DeploymentConfig "fh-statsd" created PersistentVolumeClaim "mongodb-claim-1" created PersistentVolumeClaim "mongodb-claim-2" created PersistentVolumeClaim "mongodb-claim-3" created DeploymentConfig "mongodb-1" created DeploymentConfig "mongodb-2" created DeploymentConfig "mongodb-3" created 17

22 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide Pod "mongodb-initiator" created Route "mbaas" created --> Success Run 'oc status' to view your app. It may take a minute for all the resources to get created and up to 10 minutes for all the components to get to a Running status Verifying The Installation 1. Verify the Services. Each MBaaS component defines a Service, which load-balances and proxies traffic to the underlying pods. To verify that all the services of the MBaaS have been created, enter the following command: oc get svc The output should look similar to the following: NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE fh-mbaas-service <none> 8080/TCP name=fh-mbaas 1m fh-messaging-service <none> 8080/TCP name=fh-messaging 1m fh-metrics-service <none> 8080/TCP name=fh-metrics 1m fh-statsd-service <none> 8080/TCP,8081/UDP name=fh-statsd 1m mongodb-1 None <none> 27017/TCP name=mongodb-replica-1 1m mongodb-2 None <none> 27017/TCP name=mongodb-replica-2 1m mongodb-3 None <none> 27017/TCP name=mongodb-replica-3 1m Verify that the output contains a service for all MBaaS components and a service for each MongoDB replica. Each service forwards traffic to one or more pods. During the MBaaS creation, some intermediate pods get created, which are responsible for deploying or setting up other pods. These intermediate pods have a -deploy or -build suffix. If these intermediate pods fail, the installation can not proceed. You can try viewing logs of the intermediate pods to identify the cause of the failure. 2. Verify that the MongoDB replica set is configured correctly. a. Verify that the status of the mongodb-initiator pod is Completed. oc get pod mongodb-initiator 18

23 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 NAME READY STATUS RESTARTS AGE mongodb-initiator 0/1 Completed 0 5d b. Verify that each MongoDB replica has all replica set members configured. Enter the following command to list the replica set members of each MongoDB replica: for j in `(for i in 1 2 3;do oc get po -l deploymentconfig=mongodb-$i -o name;done) sed -e 's pod/ '`; do echo "## $j ##" && echo mongo admin -u admin -p \${MONGODB_ADMIN_PASSWORD} --eval "printjson\(rs.conf\ (\).members\)" oc rsh --shell='/bin/bash' $j; done 3. Verify the pods. Each replica should have the same three members listed in its array of replica set members mongodb-1, mongodb-2, and mongodb-3. If there are not exactly three members in the array of replica set members of each node, refer to the Common Problems section of the troubleshooting guide for help. To verify that all the pods are running, enter the following command: oc get pods The output should look similar to the following: NAME READY STATUS RESTARTS AGE fh-mbaas-1-h511r 1/1 Running 0 53m fh-mbaas-1-hg5ub 1/1 Running 0 54m fh-mbaas-1-uwpl6 1/1 Running 0 53m fh-messaging-1-3wxap 1/1 Running 0 53m fh-messaging-1-j5asf 1/1 Running 0 53m fh-messaging-1-yh8hn 1/1 Running 0 54m fh-metrics-1-f5ems 1/1 Running 0 53m fh-metrics-1-faihq 1/1 Running 0 53m fh-metrics-1-vleqs 1/1 Running 0 54m fh-statsd-1-36hw0 1/1 Running 0 54m mongodb l8b 1/1 Running 0 54m mongodb-2-1-hwmzx 1/1 Running 0 54m mongodb-3-1-bl12r 1/1 Running 0 54m mongodb-initiator 0/1 Completed 0 54m Verify that all Pods are in a Running state, with the exception of the mongodb-initator, which should be in a Completed state. If any Pod is in a different state, they may require more time, or may have an issue starting up. A stream of events for the namespace, including any issues with scheduling and creating pods, pulling images and any other potential issues, can be viewed using oc get events -w. 4. Verify the Route To verify the MBaaS route is exposed, enter the following command: oc get routes mbaas 19

24 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide The output should look similar to the following: NAME HOST/PORT PATH SERVICE LABELS INSECURE POLICY TLS TERMINATION mbaas live-mbaas.example.com fh-mbaas-service Allow edge 5. Ping the health endpoint. If all services are created, all pods are running, and the route is exposed, the MBaaS health endpoint can be queried as follows: curl `oc get route mbaas --template " {{.spec.host}}"`/sys/info/health The endpoint responds with health information about the various MBaaS components and their dependencies. If there are no errors reported, the MBaaS is ready to be configured for use in the Studio. Successful output will resemble the following: { "status": "ok", "summary": "No issues to report. All tests passed without error", "details": [ { "description": "Check Mongodb connection", "test_status": "ok", "result": { "id": "mongodb", "status": "OK", "error": null }, "runtime": 33 }, { "description": "Check fh-messaging running", "test_status": "ok", "result": { "id": "fh-messaging", "status": "OK", "error": null }, "runtime": 64 }, { "description": "Check fh-metrics running", "test_status": "ok", "result": { "id": "fh-metrics", "status": "OK", "error": null }, "runtime": 201 }, { "description": "Check fh-statsd running", "test_status": "ok", 20

25 CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3 } ] } "result": { "id": "fh-statsd", "status": "OK", "error": null }, "runtime": 7020 After verifying that the MBaaS is installed correctly, you must create an MBaaS target for the new MBaaS in the Studio CREATING AN MBAAS TARGET 1. In the Studio, navigate to the Admin > MBaaS Targets section. Click New MBaaS Target. 2. As Type, choose OpenShift Fill in the following information MBaaS Id - a unique ID for the MBaaS, for example, live. The ID must be equal to the OpenShift project name chosen in the Installing the MBaaS section, without the -mbaas suffix. OpenShift Master URL - the URL of the OpenShift master, for example, OpenShift Username, OpenShift Password - username and password of an OpenShift user which can create projects and any objects associated with a project (for example, services, DeploymentConfigs, Routes). For the manual installation, enter the username and password of the dedicated MBaaS admin user. OpenShift Router DNS - a wildcard DNS entry of the OpenShift router, for example, *.cloudapps.example.com If you ve chosen the manual installation procedure, uncheck Automatic MBaas Installation and fill in these two additional fields: MBaaS Service Key Equivalent to the value of the FHMBAAS_KEY environment variable, which is automatically generated during installation. To find out this value, enter the following command: oc env dc/fh-mbaas --list grep FHMBAAS_KEY Alternatively, you can find the value in the OpenShift Console, in the Details tab of the fh-mbaas deployment, in the Env Vars section. MBaaS URL A URL of the route exposed for the fh-mbaas-service, including the https protocol prefix. This can be retrieved from the OpenShift web console, or by running the following command: 21

26 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide echo " get route/mbaas -o template --template {{.spec.host}}) 4. Click Save MBaaS and you will be directed to the MBaaS Status screen. If you chose an automatic installation, it can take several minutes before the status is reported back. For a manual installation, the status should be reported back in less than a minute. Once the process of creating the MBaaS has succesfully completed, you can see the new MBaaS in the list of MBaaS targets. In your OpenShift account, you can see the MBaaS represented by a project NEXT STEPS Create an Environment - you must create at least one environment for the MBaaS to be usable by Cloud Apps and Cloud Services Adjusting System Resource Usage of the MBaaS and Cloud Apps - we strongly recommend that you adjust the system resource usage of MBaaS components as appropriate for your production environment Optional: Set up centralized logging - deploy a centralized logging solution based on ElasticSearch, Fluentd, and Kibana to debug issues with the MBaaS 22

27 CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS 5.1. OVERVIEW In the RHMAP 4.x MBaaS based on OpenShift 3, each cloud app and and each MBaaS component runs in its own container. This architecture allows for granular control of CPU and memory consumption. A fine level of control of system resources helps to ensure efficient use of nodes, and to guarantee uninterrupted operation of critical services. An application can be prepared for various situations, such as high peak load or sustained load, by making decisions about the resource limits of individual components. For example, you could decide that MongoDB must keep working at all times, and assign it high, guaranteed amount of resources. At the same time, if the availability of a front-end Node.js server is less critical, the server can be assigned less initial resources, with the possibility to use more resources when available PREREQUISITES The system resources of MBaaS components and cloud apps in the MBaaS can be regulated using the mechanisms available in OpenShift resource requests, limits, and quota. Before proceeding with the instructions in this guide, we advise you to read the Quotas and Limit Ranges section in the OpenShift documentation ADJUSTING RESOURCE USAGE OF THE MBAAS The RHMAP MBaaS is composed of several components, each represented by a single container running in its own pod. Each container has default resource requests and limits assigned in the MBaaS OpenShift template. See the section Overview of Resource Usage of MBaaS Components for a complete reference of the default values. Depending on the deployment model of the MBaaS, you may have to adjust the resource limits and requests to fit your environment. If the MBaaS components are deployed on the same nodes as the cloud apps, there is no adjustment required. However, when the MBaaS components are deployed on nodes dedicated to running the MBaaS only, it is strongly recommended to adjust the resource limits to take full advantage of the available resources on the dedicated nodes Calculating the Appropriate Resource Requests and Limits Note This section refers to CPU resources in two different terms the commonly used term vcpu (virtual CPU), and the term millicores used in OpenShift documentation. The unit of 1 vcpu is equal to 1000 m (millicores), which is equivalent to 100% of the time of one CPU core. The resource limits must be set accordingly for your environment and depend on the characteristics of load on your cloud apps. However, the following rules can be used as a starting point for adjustments of resource limits: 23

28 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide Allow 2 GiB of RAM and 1 vcpu for the underlying operating system. Split the remainder of resources in equal parts amongst the MBaaS Components Example Given a virtual machine with 16 GiB of RAM and 4 vcpus, we allow 2 GiB of RAM and 1 vcpu for the operating system. This leaves 14GB RAM and 3 vcpus (equal to 3000 m) to distribute amongst the 5 MBaaS components. 14 GiB / 5 = 2.8 GiB of RAM per component 3000 m / 5 = 600 m per component In this example, the resource limit for each MBaaS component would be 2.8 GiB of RAM and 600 millicores of CPU. Depending on the desired level of quality of service of each component, set the resource request values as described in the section Quality of service tiers in the OpenShift documentation Overview of Resource Usage of MBaaS Components The following table lists the components of the MBaaS, their idle resource usage, default resource request, and default resource limit. MBaaS component Idle RAM usage RAM request RAM limit Idle CPU usage CPU request CPU limit fh-mbaas 160 MiB 200 MiB 800 MiB <1% 200 m 800 m fh-messaging 160 MiB 200 MiB 400 MiB <1% 200 m 400 m fh-metrics 120 MiB 200 MiB 400 MiB <1% 200 m 400 m fh-statsd 75 MiB 200 MiB 400 MiB <1% 200 m 400 m mongodb 185 MiB 200 MiB 1000 MiB <1% 200 m 1000 m 5.4. ADJUSTING RESOURCE USAGE OF CLOUD APPS The resource requests and limits of cloud apps can be set the same way as for MBaaS components. There is no particular guideline for doing the adjustment in cloud apps Overview of Resource Usage of Cloud App Components 24

29 CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS Cloud app component Idle RAM usage RAM request RAM limit Idle CPU usage CPU request CPU limit nodejs-frontend 125 MiB 500 MiB 1 GiB <1% 100 m 500 m redis 8 MiB 100 MiB 500 MiB <1% 100 m 500 m 5.5. SETTING RESOURCE REQUESTS AND LIMITS The procedure for setting the resource requests and limits is the same for both MBaaS components and cloud app components. Open the DeploymentConfig of a component, for example fh-mbaas: oc edit dc fh-mbaas The DeploymentConfig contains two resources sections with equivalent values: one in the spec.strategy section, and another in the spec.template.spec.containers section. Set the cpu and memory values of requests and limits as necessary, making sure the values stay equivalent between the two sections, and save the file. apiversion: v1 kind: DeploymentConfig metadata: name: fh-mbaas... spec:... strategy: resources: limits: cpu: 800m memory: 800Mi requests: cpu: 200m memory: 200Mi... spec: containers:... resources: limits: cpu: 800m memory: 800Mi requests: cpu: 200m memory: 200Mi 25

30 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide 5.6. USING CLUSTER METRICS TO VISUALIZE RESOURCE CONSUMPTION It is possible to view the immediate and historical resource usage of pods and containers in the form of donut charts and line charts using the Cluster Metrics deployment in OpenShift. Refer to Enabling Cluster Metrics in the OpenShift documentation for steps to enable cluster metrics. Once cluster metrics are enabled, in the OpenShift web console, navigate to Browse > Pods and click on the component of interest. Click on the Metrics tab to see the visualizations. 26

31 CHAPTER 6. SETTING UP SMTP FOR CLOUD APP ALERTS CHAPTER 6. SETTING UP SMTP FOR CLOUD APP ALERTS 6.1. OVERVIEW Each cloud app can automatically send alerts by when specified events occur, such as when the cloud app gets deployed, undeployed, or logs an error. See Alerts & Notifications for more information. For the RHMAP 4.x MBaaS based on OpenShift 3, the function is not available immediately after installation. You must configure an SMTP server to enable support PREREQUISITES An RHMAP 4.x MBaaS running in OpenShift Enterprise 3 An account on an SMTP server through which notification alerts can be sent An address where alerts should be sent A deployed Cloud App 6.3. CONFIGURING SMTP SETTINGS IN FH-MBAAS The FH_ _SMTP and FH_ _ALERT_FROM environment variables in the fh-mbaas DeploymentConfig need to be set, using the below commands: oc project <mbaas-project-id> oc env dc/fh-mbaas FH_ _SMTP="smtps://username:password@localhost" FH_ _ALERT_FROM="admin@example.com" After modifying the DeploymentConfig, a redeploy of the fh-mbaas pod should be triggered automatically. Once the pod is running again, you can verify the changes VERIFYING SMTP SETTINGS 1. In the Studio, navigate to a deployed Cloud App. 2. Go to the Notifications > Alerts section. 3. Click Create An Alert. 4. In the s field, enter your address. 5. Click Test s. You should receive an from the address set as FH_ _ALERT_FROM TROUBLESHOOTING If the test fails to send, verify the SMTP settings in the running fh-mbaas Pod. oc env pod -l name=fh-mbaas --list grep 27

32 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide It may help to view the fh-mbaas logs while attempting to send an , looking for any errors related to SMTP or . oc logs -f fh-mbaas-<deploy-uuid> Ensure the Cloud App you are using to send a test mail with is running. If the test sends OK, but fails to arrive, check it hasn t been placed in your spam or junk folder. 28

33 CHAPTER 7. USING MONGODB IN AN RHMAP 4.X MBAAS CHAPTER 7. USING MONGODB IN AN RHMAP 4.X MBAAS 7.1. OVERVIEW In an RHMAP 4.x MBaaS based on OpenShift 3, all components of the MBaaS run within a single OpenShift project, together with a shared MongoDB replica set. Depending on how the MBaaS was installed, the replica set runs either on a single node, or on multiple nodes, and may be backed by persistent storage. The recommended production-grade MongoDB setup for an MBaaS has 3 replicas, each backed by persistent storage. Each cloud app deployed to the MBaaS has its own OpenShift project. However, the database of a cloud app is created in the shared MongoDB instance. Therefore, all management operations on the persistent data of cloud apps and the MBaaS, such as backup, or replication can be centralized. At the same time, the data of individual cloud apps is isolated in separate databases ACCESSING DATA IN THE MONGODB IN THE MBAAS A simple way to store data is using the $fh.db API, which provides methods for create, read, update, delete, and list operations. See the $fh.db API documentation for more information. If you need the full capability of a native MongoDB driver, or want to use another library to access the data, such as Mongoose, you can use the connectionstring method of the $fh.db API to retrieve the connection string to the MongoDB instance: $fh.db({ "act" : "connectionstring" }, function(err, connectionstring){ console.log('connectionstring=', connectionstring); }); Note To avoid concurrency issues, we recommend using either the $fh.db API or a direct connection to the database, but not both at the same time. 29

34 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide CHAPTER 8. CENTRALIZED LOGGING FOR MBAAS COMPONENTS 8.1. OVERVIEW Logging output from RHMAP MBaaS components can be aggregated and accessed through a web console when using an MBaaS backed by OpenShift Enterprise 3 (OSEv3). Aggregated logging is enabled by deploying an EFK logging stack to your OSEv3 instance, which consists of the following components: Elasticsearch indexes log output collected by Fluentd and makes it searchable. Fluentd collects standard output of all containers. Kibana is a web console for querying and visualizing data from Elasticsearch. To enable this functionality, follow the official OpenShift guide Aggregating Container Logs ACCESSING LOGS THROUGH KIBANA WEB CONSOLE The Kibana web console is where logs gathered by Fluentd and indexed by Elasticsearch can be viewed and queried. You can access the Kibana web console via the OpenShift web console, or directly by its URL configured through the KIBANA_HOSTNAME in the deployment procedure Viewing Logs of a Single Pod If you have configured loggingpublicurl in step 8 of the deployment procedure, the OpenShift web console allows you to view the log archive of a particular pod. 1. In the OpenShift web console, select a project, and look for the deployment named fhmbaas. 2. Click on the Pods circle. 3. Choose one of the pods to inspect. 4. Click on the Logs tab. 5. Click on the View Archive button at the top right corner to access the logs of the chosen pod in the Kibana web console. Note By default, Kibana s time filter shows the last 15 minutes of data. If you don t see any values, adjust the Time filter setting to a broader time interval Accessing Kibana Directly You can access the Kibana web console directly at where KIBANA_HOSTNAME is the host name you set in step 4 of the deployment procedure. 30

35 CHAPTER 8. CENTRALIZED LOGGING FOR MBAAS COMPONENTS Configuring an Index Pattern When accessing the Kibana web console directly for the first time, you are presented with the option to configure an index pattern. You can also access this configuration screen in the Settings tab. By default, there is an index pattern in the format <MBaaS ID>-mbaas.*, matching the ID of the deployed MBaaS target. To make queries more efficient, you can restrict the index pattern by date and time. 1. Select the Use event times to create index names 2. Enter the following pattern in the Index name or pattern input text field. For example: [onprem-mbaas.]yyyy.mm.dd 3. You will see output similar to the following below the input field Pattern matches 100% of existing indices and aliases onprem-mbaas onprem-mbaas Click Create to create the index based on this pattern. 5. You can now select this newly created index in the Discover tab when doing searches, as well as in other parts, such as the Visualizations tab IDENTIFYING ISSUES IN AN MBAAS If you suspect that an error of an MBaaS component may be the cause of an issue, you can use Kibana s Discover tab to find the root of the problem. The following steps describe the general procedure you can follow to identify issues. 1. Select the index for the MBaaS target you are interested in Use the dropdown just below the input bar in the Discover view to list all available indices. An index is similar to a database in relational database systems. Select which index your searches will be performed against. 2. Select a time interval for your search Click the Time Filter (clock icon) and adjust the time interval. Initially, try a broader search. 3. Perform a simple search To search for all error events, perform a simple search for error in the Discovery field. This will return the number of hits within the chosen time interval. 4. Select the msg or message field to be displayed On the left hand side of the Discover view is a list of fields. From this list you can select fields to display in the document data section. Selecting a field replaces the _source field in the document data view. This enables you to see any error messages and might help you refine your original search if needed. You can also select more fields to help you locate the issue. 5. Narrow down the time interval 31

36 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide The histogram shows search hits returned in the chosen time interval. To narrow down the search in time you have the following options: Click on a bar in the histogram to narrow down the search to that bar s time interval. Select a time window in the date histogram by clicking and dragging between the start/end time you are interested in. 6. Inspect the document data Once you narrow down the search, you can inspect the document data items. Apart from the msg and message fields, you might be interested in kubernetes_pod_name to see the pod a message originates from Viewing All Debug Logs for an MBaaS Component If searching for error messages doesn t help, you can try looking into debug logs of individual MBaaS components. 1. Select the index for the MBaaS target that you are interested in 2. Start a new search Click on the New Search button to the left of the search input bar, which looks like a document with a plus sign. 3. Search an MBaaS component for all debug messages For example, to search for all debug messages of the fh-messaging component, enter the following query: type: bunyan && level: 20 && kubernetes_container_name: "fhmessaging" If you know some part of the error message, you can specify that as part of the search: type: bunyan && level: 20 && kubernetes_container_name: "fhmessaging" && "Finished processing" You can narrow down your search further by time, as described in step 5 above. As a reference, the following are the Bunyan log levels: TRACE = 10; DEBUG = 20; INFO = 30; WARN = 40; ERROR = 50; FATAL = 60; 32

37 CHAPTER 9. MONITORING THE MBAAS WITH COCKPIT CHAPTER 9. MONITORING THE MBAAS WITH COCKPIT 9.1. OVERVIEW System resources of nodes and containers in the MBaaS on OpenShift 3 can be monitored and managed using Cockpit. Cockpit is a system administration tool, that provides insights into how nodes and containers are performing. It lets you monitor current values and adjust limits on system resources, control lifecycle of container instances, and manipulate container images. For more information about Cockpit, refer to the official web site of the Cockpit Project and its Documentation INSTALLATION For most OpenShift 3 instances, Cockpit is most likely already installed on all nodes. This is not the case if your nodes use the RHEL Atomic Host, where Cockpit needs to be installed manually. To check whether Cockpit is installed in your OpenShift cluster, try visiting the URL of the Cockpit web interface: node host>:9090 If there s no response to the request, Cockpit is most likely not installed Installing Cockpit Manually 1. Install Cockpit on nodes. The following three steps must be repeated for each node you wish to monitor in your OpenShift cluster. 2. Log in to the node. ssh <node host> 3. Install Cockpit packages. yum install cockpit cockpit-docker 4. Enable and start the Cockpit service. systemctl enable cockpit.socket systemctl start cockpit.socket 5. Create a Cockpit system user on master. To log in to the Cockpit web interface, you will have to provide the username and password of an operating system user existing on the OpenShift master node. This guide refers to this user as the Cockpit system user. To allow Cockpit to access system resources, perform operations on Docker containers and Kubernetes resources, the Cockpit system user must: be in the docker group; 33

38 Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide be able to log in to other nodes using ssh; be able to perform Kubernetes operations. Create the Cockpit system user on the master node, or modify an existing user to have the necessary privileges VIEWING THE CONTAINERS ON AN OPENSHIFT NODE Navigate to the Cockpit dashboard for a node in a web browser (port 9090 by default) and log in as the Cockpit system user. To see all containers deployed on that node, click Containers in the lefthand side menu. You can filter the list to only display running containers, using the dropdown menu above the list of containers. This view lets you see the RAM and CPU usage of all running containers. If you select an MBaaS node, you will see the containers for all MBaaS components. Clicking on a container will show the current logs, CPU shares, and RAM usage. In the Tools menu on the left hand side, you can get terminal access into the node for further investigation VIEWING MULTIPLE HOSTS SIMULTANEOUSLY Cockpit can connect to multiple hosts from a single Cockpit session. This can be useful to compare resource usage of two or more machines in the same dashboard. See Multiple Machines in the Cockpit documentation for more information. 34