Performance Monitoring and Management of Microservices on Docker Ecosystem

Transcription

1 Performance Monitoring and Management of Microservices on Docker Ecosystem Sushanta Mahapatra Sr.Software Specialist Performance Engineering SAS R&D India Pvt. Ltd. Pune Richa Sharma Sr.Software Specialist Performance Engineering SAS R&D India Pvt. Ltd. Pune Application packaging and deployment using Docker is trending and fast catching up in the infrastructure world. The big players of the industry are either deploying on Docker, integrating with Docker or developing for Docker. The reason being it is flexible, light weight, easy to manage and highly scalable. Docker containers wrap applications and their dependencies, use a shared kernel and run on the same host along with other containers. Essentially they are isolated processes in the user space on the host operating system. Similarly Microservice architecture enables decomposition of applications into small services which improves faulttolerance and manageability. These services are designed to run on their own, and can communicate with the outside world via lightweight protocols like HTTP. These small services are designed to be flexible in deployment and work independently. One such deployment mechanism is containerization of these micro services via Docker. So Docker containers are one of the preferred options to build, ship and run these micro services. Containerization of these micro services via Docker is highly convenient as Docker provides automation for the deployment of applications inside the containers. Docker provides an additional layer of abstraction and automation of operating system level virtualization on Linux. The Docker containers are built on top of the Linux containers mechanism and are very light weight. However the trade-off is the performance monitoring and management overhead that poses the challenge to measure system level utilization and performance characteristics of all the services. This paper talks about various options that a performance engineer can leverage to monitor and manage the Docker based micro services. The options include live monitoring of containers, persistent storage options for offline monitoring and analysis along with integration possibilities with few of the industry standard performance testing tools (commercial). Additionally, this paper also delves into few container management tools like Docker UI and Consul Web UI.

2 Introduction: Microservices architecture is the latest buzz amongst leading software architects and design guru s. Over the last few years a sharp surge has been evident towards Microservice way of designing software services, leaving behind the monolithic approach. There is a very consistent pattern of migration to this architecture using the container technology, and primarily the choice of the container being Docker. Microservice adoption is been driven by de-composition of applications to granular level enabling a single service to be self-sufficient in itself. Cloud based infrastructure and elastic scalability are also drivers behind microservice adoption. Figure 1: Decomposition of Service into microservices From performance standpoint there is a need of adopting different approaches to monitor such Docker deployed micro services. Typically, there will be various micro services communicating with each other on a production system and each of the services will ideally be hosted on a Docker container on top of the Linux kernel or on a cloud infrastructure. As these containers consume resources in isolation, the monitoring approaches should aim at gauging the individual containers in isolation similar to monitoring a single host. There are various ways in which a performance test engineer/tester can monitor and manage such containers and many platforms and frameworks are developed or are under development to monitor such micro services. Docker Containers and Resource Isolation: Docker is written in Go Language and makes use of several Linux kernel features like namespaces, control groups, union file systems and container format, to deliver the container isolation functionalities that are discussed below. Docker takes advantage of a technology called namespaces to provide the isolated workspace called the container. When you run a container, Docker creates a set of namespaces for that container which provides a layer of isolation: each aspect of a container runs in its own namespace and does not have access outside it as shown is Figure 2.

3 Figure 2: Docker Container resource isolation Monitoring Approaches for Docker Containers: As discussed in the above section each Docker container will have its own subsystem and the resources can be monitored in isolation as the container is a separate host. There are several ways to achieve this. Listed below are few of the approaches that are most helpful. Approach 1: Control Groups Linux provides a kernel feature: control groups or commonly called as cgroups that allow us to allocate resources such as CPU time, memory, network bandwidth, or combinations of these resources among user-defined groups of tasks or processes running on a system. We can configure and monitor the cgroups, control user access to cgroups. By using cgroups, system administrators gain fine-grained control over allocating, prioritizing, denying, managing, and monitoring system resources. Figure 3 below shows how the cgroups can be used to control the resources. Figure 3 : Control Groups and an Analogy with real world The Docker Containers, are based on control groups to isolate the resource usage (CPU, memory, disk I/O, and network) for a collection of processes. This is quite similar to the way the resource isolation happens in a multi-

4 stored building where the resource consumption details (i.e. water or electricity) are measured commonly for the whole building where as it is measured individually for each unit. Usage Example: The memory metrics for a running container can be retrieved by using the container ID with cgroups by specifying /sys/fs/cgroup/memory/docker/(id) at command prompt, which will show the following output of the specified container s memory usage in bytes: total_cache total_rss total_rss_huge Similarly we can find any of the performance matrices (i.e. CPU, IO, Network) for any of the containers in isolation. Approach 2 : Using Google s Cadvisor Google created Cadvisor initially for their internal use but later on they added Docker container support and released it for open source use. Cadvisor is short for Container Advisor which is easy to use and gives a detailed look into the resource usage and the various performance characteristics of all the running containers. Cadvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB. Cadvisor has native support for Docker containers out of the box. It provides details of resource usage and performance characteristics of the running containers. It is a running daemon that gathers, aggregates and presents information about all the running containers. For each containers it keeps resource isolation parameters, historical resource usage and network statistics container-wide and machine-wide. Pulling and Running Cadvisor Using Cadvisor is quite easy, from your host just pull Cadvisor Docker container with the following command and you are ready to go sudo Docker run volume=/:/rootfs: ro volume=/var/run:/var/run:rw volume=/sys:/sys:ro volume=/var/lib/docker/:/var/lib/docker:ro publish=8080:8080 detach=true name=cadvisor google/cadvisor:latest Once the Cadvisor container starts running you can bring up the UI from: (preferably using Firefox, Chrome) and drill down to any container you want to monitor from the Docker Containers link.

5 Figure 4 : Cadvisor UI and typical graphs it shows for a container (Image customised for representation). Additionally Cadvisor provides REST API end points which can be used to get all stats in JSON format which can be further consumed. Few of them are as follows: (your container Instance) Limitation: The data captured is live and there is no internal way of storing it for later analysis. For persistent storage of data, Cadvisor needs to be interfaced with some external time-series databases like InfluxDB and OpenTSDB. This will be discussed in the later part of this paper. Approach 3 : Integration with Leading Performance Testing Tools This section touches upon two approaches of integrating Docker container metrics with Leading Performance testing tool (commercial) for monitoring of Microservices. Approach 3.1: Integrating Cadvisor API end points with Performance testing tools: As mentioned in the previous section, Google s Cadvisor is good to get performance counters from Docker containers as well as from the services or processes running within those containers. The Cadvisor rest API end points can be called and processed programmatically to extract the different performance metrics and then store them as part of the analysis report (output from the tool).

6 Performance test tool has support for consuming those data points and generate graphs as part of the performance test report which can be stored permanently. Limitation(s): A single virtual user should run another script at a specified interval (in background) to collect and process the metrics during the course of the test run (this will not add much overhead) The parser to process those Cadvisor JSON responses has to be implemented (this can be custom made using any simple program) The actual data values cannot be stored as it is. For Further analysis in case of any problem the test has to be re-run. Usage Example: Extract each value and store it as a key/value pair as follows: HashMap results=new HashMap(); results.put ( Counter1,"Value"); Feed the data to the tools storage system or store it on any storage medium for offline analysis as depicted in the following diagram. Figure 5 : Cadvisor API endpoints and their integration with Industry standard tools. Advantages: The data received via the API end points will be stored permanently and can be used for live or offline analysis.

7 Approach 3.2: Registering custom scripts to leverage cgroups and integrate with industry standard tools. Red Hat Enterprise Linux kernel has a feature called control groups or cgroups that was discussed earlier in this paper. Cgroups can be used to collect the system resources for a specific group in complete isolation. Based on cgroups, one or more custom monitoring scripts can be developed to be registered with Leading Performance monitoring tools and those scripts can be used from within Performance test tool to collect metrics for any of the containers. The figure below displays a simple process of writing a custom shell script in Linux leveraging cgroups to retrieve container specific matrices and how tools like JMeter and Gatling can consume those script s output. Latest versions of JMeter as well as Gatling have support for host monitoring via direct plugins or collectd plugins which can be configured to use the custom script to pull matrices during the course of the test run and show them as part of the test result. Figure 6 : Integrating custom cgroups script with tools like Jmeter and Gatling Limitation: Proper understanding of cgroups and how to define the hierarchy and the subsystems is required. Approach 4 : A persistent Monitoring Infrastructure using Cadvisor, InfluxDB and Grafana As discussed in the previous section, Cadvisor can be used to monitor containers but the limitation in that approach is: the monitoring counters and their values that we are seeing are live and not really getting stored anywhere. Additionally, another limitation is customizing the performance statistics into meaningful graphs. Further section describes a quick approach to store the performance metrics persistently in a time-series database and use a charting solution to build more meaningful graphs based on the stored metrics.

8 Components in the Framework: Cadvisor ( to read performance matrices) InfluxDB or OpenTSDB (time-series databases for persistent storage) Grafana (charting solution that supports InfluxDB/OpenTSDB) How the above components are integrated? Figure 7 : A persistent Docker monitoring Infrastructure using Cadvisor, InfluxDB and Grafana The integration is quite simple as shown in Figure 4. Let s see in detail how each of the components integrate with each other. The objective is to monitor the Docker containers that sits on top of the Linux kernel. Run the Cadvisor Docker container which will provide all the resource usage statistics and performance measures for all the running containers. In order to store the metrics retrieved by Cadvisor (in some database) use either InfluxDB or OpenTSDB. For ease of use, InfluxDB is recommended since Cadvisor can be auto configured to push data to InfluxDB. Once the data is persisted, a charting solution such as Grafana can be used which allows to connect to these databases and helps in building rich and meaningful graphs to assist in performing better analysis. Once the performance dashboards are ready, end-users can simply view and monitor the performance characteristics live via these Dashboards This infrastructure can be setup quickly as all these component are available as Docker containers, so if you have a Docker enabled Linux box, it will take just few minutes to build this dashboard and start gathering performance metrics. Limitation: As already mentioned it s a quick way of building a monitoring infrastructure. So once you stop the InfluxDB or Grafana Docker instances you may lose all the data. If you are interested to build a robust monitoring infrastructure, it will be good to host the InfluxDB/OpenTSDB and Grafana configuration permanently or find a way to store the data that these Docker instances are using permanently somewhere so that they can be reusable.

9 Managing Docker Containers: DockerUI: In addition to the resource monitoring it is equally important to manage the containers. Docker UI is a web interface which helps visualizing running containers and allows to manage them easily. It assists users in various container lifecycle actions like starting, stopping, pausing, removing and killing a container, making it pretty easy to manage containers and images with simple clicks without needing to execute lines of commands to do small jobs. Consul Web UI: Figure 8 : Docker UI (Image Source : Internet) Consul is a tool for discovering and configuring services in your infrastructure. It comes with a user-friendly web UI that can be used for viewing all services and nodes, for viewing all health checks and their current status, and for reading and setting key/value data. It can act as a one-shot dashboard for managing your services.

10 Figure 9 : Consul web UI (Image Source : References: Definitions of Microservices: RedHat guide on Control Groups: US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html Google s Cadvisor: Grafana Information: InfluxDB Information: Docker Information: Consul Information: