Top 5 Key Capacity Management Concerns for UNIX / Linux

Top 5 Key Capacity Management Concerns for UNIX / Linux

UNIX and the Datacenter Since UNIX systems were developed back in the 1970 s, things as you d expect have moved on a long way. Single CPU systems, the size of washing machines, have been replaced with many racks of multi-core (hyper-threaded) blade servers. In more recent years, the re-introduction of virtualization allowed for multiple virtual systems to be hosted on a single piece of hardware via a hypervisor. Modern data centers will typically host many thousands of both physical and virtual servers. Physical and Virtual hosts Web Servers Database hosts The mix of physical and virtual tends to depend on what applications are being hosted. It is likely that physical servers will be hosting large RDBMS instances and virtual servers hosting web applications. Licensing concerns vcpus, pcpus As most applications are licensed by the numbers of host CPUs, there is a concern on the cost of hosting applications on fixed physical core systems. Virtual systems now provide the flexibility to change the number of vcpus on a running system or in most cases after a reboot KPIs Due to this additional complexity, effective Capacity Management is even more important, more specifically where services are sharing both physical and virtualized resources. With the numbers of systems being virtualized on the increase year on year, there are a number of Key Performance Indicators (KPI) we should be closely monitoring: Reduction in physical estate (servers) Reduction in physical space Reduction of power usage, cooling, associated costs The 1990 s and early part of the millennium witnessed a rise in the number of physical distributed systems (UNIX/Windows) to a point where physical data center space and energy demand was becoming an issue. The introduction of virtualization has helped address these issues, by allowing multiple low usage systems to be virtualized on to a 2 Top 5 Key Capacity Management Concerns

single piece of hardware. This in-turn had initially started to reduce the number of physical systems hosted within a data center, which removed the need for additional space and/or allowed for that space to be reclaimed. The reduction of physical components and servers also allowed for a reduction in power usage and if reclaiming space and/or moving systems to a small data center were feasible then the amount of power dedicated to cooling could be reduced. Use of Internal / External / Hybrid clouds Cloud technology, whether the resources are internally or externally provided, is underpinned by virtualization. Virtualization provides the 3 following minimums for cloud services. On demand self-service Resource Pooling Rapid Elasticity Virtualization Why virtualize? It s an important question and one that shouldn t be taken lightly. There are many benefits as I have already mentioned about virtualization, but it shouldn t be taken as a broad brush approach. In my experience it is better to perform some kind of pre-virtualization performance analysis before actually committing to virtualizing an application, even though the vendor tells you it is ok! I am a big advocate of virtualization as it provides great flexibility, reduces hardware and associated costs, and allows you to manage your estate from a single pane of glass. Commercial Vendors IBM,HP, Oracle, VMware, Microsoft Specifically looking from a Capacity and Performance viewpoint, virtualizing applications in some instances can prove to be detrimental to performance. Even though virtualization allows for multiple virtual systems to be hosted, those hosts are still physical servers which have finite capacity and I ll cover this later on. Top 5 Key Capacity Management Concerns 3

Open source (Linux) KVM Xen Underpins cloud technology - As mentioned previously, virtualization underpins Cloud technology. Cloud services can be provided through demand resourcing, flexibility and quick provisioning. There are many more cloud services now available than the traditional Infrastructure, Platform and Software. IaaS, PaaS, SaaS Oracle Enterprise Linux - Oracle s offering in the UNIX/Linux space is Oracle Enterprise Linux, which is a clone of Red Hat Enterprise Linux and Solaris after their acquisition of SUN. Both operating systems provide virtualization offerings, however the Linux coverage is growing quite substantially with its support for tablets and phones and database coverage. The topic of cost differences for Linux over Solaris or AIX is a whole paper in itself. Both RHEL and Solaris are free and open source and are supported on x86 architecture, however entry point x86 hardware is cheaper than ultrasparc hardware, unless you have the capacity requirements for ultrasparc entry offerings. VM Principals The following principles of virtual machines hold true across all virtualized environments provided within the IT industry. Isolation Own root (/), processes, files, security Virtualization Instance of Solaris O/S Granularity Resource allocation, Pools Transparency Standard Solaris interfaces Security No global reboots, isolated within each zone 4 Top 5 Key Capacity Management Concerns

Performance Concerns Just think about this for a moment it s just you and your workstation. No contention. Introduce a server which hosts one or many applications and can be accessed by one or many users simultaneously and you are likely to get contention. Where contention exists you get queuing and inevitably response times will suffer. With UNIX/Linux virtualization we have two main concerns: 1. Which virtual systems hosted on shared hardware (via Hypervisor) will impact one another? 2. What additional overhead and impact on performance does requesting hardware resources through a hypervisor cost? To answer question number 1, we have to understand what resources each virtual machine is requesting at the same time. So it is prudent to produce performance reports for each hosted VM. Do they conflict with each other or complement each other? There are virtualization tools that have been developed to keep VMs apart if they conflict and together if they complement to improve performance. To answer question number 2, initial software virtualization techniques such as Binary Translation allowed for existing x86 operating systems to be virtualized. The hardware call request had to go via the hypervisor thus adding on an additional overhead, somewhere between 10-20% on average. As time progressed a combination of hardware virtualization and paravirtualized operating systems have reduced overhead and improved virtualized application response times. Paravirtualization is a virtualization technique that presents a software interface to VMs that is similar, but not identical to that of the underlying hardware. The intention is to reduce the guest s execution time spent performing operations which are substantially more difficult to run in a virtual environment compared to a non-virtualized environment. It provides specially defined hooks to allow the VMs and underlying host to request and acknowledge these tasks, which would otherwise be executed in the virtual domain (where execution performance is worse). A successful paravirtualized platform may allow the virtual machine monitor (VMM) to be simpler (by relocating execution of critical tasks from the virtual domain to the host domain), and/or reduce the overall performance degradation of machine-execution inside the virtual-guest. Data To produce and analyze the recommended performance reports we need to capture and store performance data. This process is normally performed by installing an agent responsible for running UNIX/Linux system tools such as SAR, VMSTAT and IOSTAT to capture information or running (potentially intrusive) kernel commands. As with any data capture whether it is local or remote you d expect to incur some overhead. Typically agents should incur no more than 1% CPU usage when capturing data, however as mentioned some agents may incur more. Top 5 Key Capacity Management Concerns 5

In addition when capturing data, can you rely on what it is reporting? Remember this is software and software can contain bugs. But you say we have to rely on what the operating system gives us, and this is true to some extent. From my experience there are several tools to provide this information within the UNIX operating system some are accurate and some are not. For example: Does your Linux system have the sysstat package installed and is it an accurate and reliable version? Or in Solaris Containers are the Resident Set Size (RSS) values being incorrectly reported due to a double counting of memory pages? An example of this is shown on the next slide. Zone Memory Reporting This report is an interesting one. It shows the amount of RSS memory per zone against the Total Memory Capacity of the underlying server (Red). Because RSS values are double counted the sum of RSS for each zone far exceeds the actual Physical Memory Capacity. Linux Differences Following on from the development and introduction of UNIX and also the creation of the GNU General Public License (GNU GPL), in order to spread software freely, many of the programs required in an OS (such as libraries, compilers, text editors, a UNIX shell, and a windowing system) were completed by the early 1990s, but few elements such as device drivers, daemons, and the kernel were incomplete. In 1991, Linus Torvalds began to work on MINIX, a Unix-like OS originally developed for academics, whose code was freely available under GNU GPL project. After which the first LINUX kernel was released on 17 September 1991, for the Intel x86 PC systems. This kernel included various system utilities and libraries from the GNU project to create a usable operating system. All underlying source code can be freely modified and used. 6 Top 5 Key Capacity Management Concerns

Linux is great for small-to medium-sized operations, and today it is also used in large enterprises where UNIX was considered previously as the only option. A few years ago, most big enterprises where networking and multiple user computing is the main concern, didn t consider Linux as an option. But today, with major software vendors porting their applications to Linux, being freely distributed and being installed pretty much on anything from mobile cell phones to supercomputers, the OS has entered the mainstream as a viable option for Web serving and office applications. Both the main commercial virtualization offerings in VMware s vsphere and Microsoft s Hyper-V both support Linux as an operating system and along with Windows is the most popular virtualized operating system within organizations. Its popularity has increased to include a user base estimated to be in excess of 25 million because of its application in embedded technologies and because it is free and easily available. Cloud Concerns To understand our Capacity and Performance concerns around cloud, we need to fully understand what the cloud is and its relation to UNIX/Linux systems. As mentioned previously cloud is underpinned by virtualization due to the ability to provide virtualized systems rapidly, providing services such as Capacity on Demand and then charging accordingly. Cloud (Internal, External, Hybrid) Rapid Provisioning Metering (Chargeback) Capacity on Demand Resource Pooling and Elasticity Hybrid popularity is growing. At present less secure tasks are being handled in external clouds - The initial use of Hybrid clouds (a mixture of Public and Private) was traditionally to run tasks that were not classed as data sensitive. The use of Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) in a Public cloud was very much favored by Test and Development departments to get quick and easy access to resources. Increasing confidence in security - As security has improved and individuals and businesses confidence in Public clouds increases, the popularity of the Hybrid cloud is growing, providing an extension to business systems as and when required. How easy is access to data? - How can we apply effective Capacity Management to a Hybrid cloud? We can and should have access to data from within an Internal cloud and also know what applications are running on what systems sharing what resources. Getting the same information from an External cloud provider however is not easy if you can get it at all. Would we want to and do we care? - Does the cloud provider give guarantees for availability, capacity and performance? Top 5 Key Capacity Management Concerns 7

SLA s and appropriate penalties if breaches occur - Do you have or need to have any Service Level Agreements in place? If so, what penalties are in place in terms of a breach or number of breaches? Big Data Concerns So what is Big Data? Big Data - Data sets whose size grows beyond the management capabilities of traditional software that has been used in the past. Vast amounts of information - are now stored, specifically when dealing with social media applications, such as Facebook and Twitter. Therefore the Big Data solution needs to provide support for Hexa- and Peta-bytes of data. Hadoop (HDFS) - One such solution is Apache s Hadoop offering. Hadoop is an open source software library project administered by the Apache Software Foundation. Apache defines Hadoop as a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. Using HDFS, data in a Hadoop cluster is broken down into smaller pieces (called blocks) and distributed throughout the cluster by auto-replication. In this way, the map and reduce functions can be executed on smaller subsets of your larger data sets, and this provides the scalability that is needed for big data processing. Map/Reduce - A general term that refers to the process of breaking up a problem into pieces that are then distributed across multiple computers on the same network or cluster, or across a grid of disparate and possibly geographically separated systems (map), and then collecting all the results and combining them into a report (reduce). Google s branded framework to perform this function is called MapReduce. Not recommended for SAN/NAS - Because the data is distributed across multiple computers whether they are on the same network or disparately separated, it is not recommended to use SAN or NAS storage but local system block storage. So what should we be monitoring? This is relatively new technology and knowledge around performance tuning is immature. Our instinct tells us that we monitor the systems as a Cluster, how much CPU and Memory is being used with the local storage being monitored both individually and as one aggregated piece of storage. Metrics such as I/O response times, file system capacity and usage are important, to name a few. 8 Top 5 Key Capacity Management Concerns

Big Data Capacity Challenges So with Big Data technology being relatively new with limited knowledge, our challenges are: Working with the business to predict usage - so we can produce accurate representations of future system and storage usage. This is normally quite a challenge for more established systems and applications so we have to bear in mind that getting this information and validating it will not be easy. New technology - limited knowledge around performance tuning Very dynamic environment - which provides the challenge to be able to configure, monitor, and track any service changes in order to provide effective Capacity Management for Big Data. Multiple tuning options - that can greatly affect the utilization/performance of systems. Key Capacity Metrics This is not an exhaustive list, but does provide information on the key capacity metrics you should be monitoring for your Big Data environment. It lists some key CPU, Memory, File System and I/O metrics which can give you a good understanding of how well your systems are performing and whether any potential capacity issues can be identified. Standard CPU metrics utilization, system/user breakdown Memory Usage, Paging, Swapping User/Process breakdown define workloads File System I/O Size Number of Files Number of Blocks Ratio s User breakdown Response time Read/Writes Service times Utilization Top 5 Key Capacity Management Concerns 9

By capturing the user/process breakdown on your UNIX/Linux systems, we can start to define workloads and couple that with the predicted business usage to produce both baseline and predictive analytical models. Some of the following key questions can then be answered: What is the business usage/growth forecast for next 3, 6, 12 months? Will our existing infrastructure be able to cope? If not what will be required? Are any storage devices likely to experience a capacity issue within the next 3,6,12 months? Are any storage devices likely to experience a capacity issue within the next 3, 6, 12 months? CPU Breakdown Below is an athene example chart displaying the kind of CPU Usage breakdown (System + User = Total) reporting you should be doing for your systems. 10 Top 5 Key Capacity Management Concerns

Some more questions: What is the breakdown? Is user CPU higher? Should it be? If it s not, what is happening on the system to generate much greater CPU usage? I/O Response Time Another example report is shown below, this time showing I/O response times on UNIX/Linux disks. This example is from a SUN Fire 280, because of the large numbers of disks we will want to filter on a Top N (5) basis, to identify the key disks which could be experiencing performance problems. The remainder of the disks are then aggregated as shown in Pink. This in effect produces a cumulative picture of disk performance. Top 5 Key Capacity Management Concerns 11

Conclusion UNIX/Linux systems - are a well-established part of any data center, hosting applications and database alike. UNIX and Linux virtualization - is commonplace within organizations providing the flexibility to host many virtual machines and also to underpin Cloud Computing services such as IaaS, PaaS and SaaS. Monitor and analyze the data - but be aware of what it is telling you. If it doesn t add up then it doesn t add up. Have you got the right packages installed, e.g. sysstat? Linux popularity is ever increasing - GUI driven, a similar model to Windows, supported on x86 virtualization, gone from SMBs to multi-national organizations and installed on supercomputers. Also found on a range of hardware from mobile/cell phones to aircraft entertainment systems. UNIX/Linux systems support Big Data implementations - by using HDFS (Hadoop) software and its Map/Reduce functionality. Many individual systems with local block storage storing the data via Map and reported on via Reduce. Understand the technology we need to have a good understanding of the technology to be able to perform effective Capacity Management. Identify what you need to monitor and analyze - get the business information, and predict future usage. 12 Top 5 Key Capacity Management Concerns

References Unix and Linux differences http://www.diffen.com/difference/linux_vs_unix Cloud Computing An explanation http://www.cloudstacking.com/?page_id=318 Big Data http://metron-athene.com/_downloads/on-demand-webinars/index_2.asp Top 5 Key Capacity Management Concerns 13

Metron Metron, Metron-Athene and the Metron logo as well as athene and other names of products referred to herein are trade marks or registered trade marks of Metron Technology Limited. Other products and company names mentioned herein may be trade marks of the respective owners. Any rights not expressly granted herein are reserved. www.metron-athene.com