BSA Sizing Guide v. 1.0

Size: px

Start display at page:

Download "BSA Sizing Guide v. 1.0"

Barbara Banks
5 years ago
Views:

1 Best Practices & Architecture BSA Sizing Guide v. 1.0 For versions Nitin Maini, Sean Berry 03 May 2016

2 Table of Contents Purpose & Audience 3 Scope 3 Capacity & Workload Basics 3 BSA Basics... 3 Terminology... 3 Environment Basics... 4 Use Case Basics... 4 Capacity Basics... 4 Basic Assumptions... 6 Workload Estimation Example... 6 Use Case Estimation 7 Disclaimers... 7 Patching: Analysis & Deploy... 7 Patching Basics... 7 Typical Calculations... 7 Performance Lab Configuration... 8 Patch Analysis: Constant Time... 8 Patch Analysis: Constant Resources... 9 Patch Deploy: Constant Time... 9 Patch Deploy: Constant Resources Compliance Compliance Basics Typical Calculations Performance Lab Configuration Compliance - CIS RHEL 6 Policy: Constant Time Compliance CIS RHEL 6 Policy: Constant Resources Compliance - CIS Windows 2008 Policy: Constant Time Compliance - CIS Windows 2008 Policy: Constant Resources Script Execution: NSH Script Job Script Execution Basics Typical Calculations Performance Lab Configuration Script Execution - NSH Script Job (Type 1): Constant Time Script Execution - NSH Script Job (Type 1): Constant Resources BSA Component Sizing 15 Typical deployment environment diagram Job Server Configuration Server Database & Repository Sizing Patch Repository Sizing Additional BSA Components Classification based on managed targets Network Bandwidth Summary 22

3 Document Information Version: 1.0 Created by: Nitin Maini, Sean Berry Last Modified on: 3 May 2016 Modified by: Nitin Maini/Sean Berry: final review.

4 Purpose & Audience The purpose of this document is to help customers & consultants appropriately size BSA infrastructure to support the required use cases, be performant and stable, at a reasonable cost. The audiences for this document include new and existing customers, and BMC and Partner consultants engaged in sizing exercises or evaluating existing customer infrastructure. We make the assumption that readers of this material have basic familiarity with BSA and common data center automation (DCA) topics. Scope This document evaluates typical single-use case scenarios for small, medium, and large environments, with the additional consideration of geographically-distributed environments, and some common Managed Services or Large Enterprise use cases. It does not consider sizing for very large environments, or those with special use cases. If you have these requirements (geographically distributed, managed services, very large enterprise), please don t hesitate to contact BMC Customer Support, who will put you in touch with someone who can review your needs, and provide feedback. Capacity & Workload Basics BSA Basics There s a good introduction page on communities, that covers many of the basics of BSA. Since it s commonly found using the search phrase getting started with BladeLogic, it s usually referred to as the Getting Started page: Terminology Target Job Server Config Server Repeater Console A managed server or OS instance running the RSCD agent. An instance of the application server that does work in the form of Jobs, communicating with the RSCD agents. An instance of the application server that handles all the interaction from the RCP console. One or more systems located in remote data centres for efficient oneto-many replication of deployment payloads. An instance of the graphical client, used to interact with the Application Servers and managed Targets.

5 Environment Basics Target Server Count How many servers/os instances will be managed? Use Cases Patching, Compliance, Script Execution, Provisioning, etc. Timing Constraints Maintenance Window Frequency Users Maximum Concurrent Users using either RCP Console, BLCLI clients or API calls. Use Case Basics Automation use cases require resources such as CPU and working memory at different levels, according to their function. The usage will also vary based on whether the main work of the use case is executed on a BSA component such as Application Server (centrally) or a managed target (at the edge). Patching Analysis Processing (and most workload) takes place on the targets based on the size/scope of the policy and patch metadata. Deploy Transfers payloads over network and installs them on targets Compliance Rules based assessment of each target in application server by pulling the necessary information from the target Remediation Applies remediation scripts to all the non-compliant targets Script Execution (NSH Script Job) NSH Script Jobs invoke the actual NSH scripts in a separate process that can be created and managed either by the Application Server or by a separately-running application known as the process spawner. Refer for more details about resource consumption for a given use case or process Capacity Basics The most common unit of capacity for an application server instance is the Work Item Thread. (WIT).

6 The pre 8.2 default was 50 WIT per appserver. From 8.2 and above, taking into account 64-bit memory limits and overall application server performance the new default is now 100 WIT per appserver. As of BSA 8.5, in most customer environments, each application server instance will have 100 WIT Work Item Units of work to be performed for a use case. For a typical scripting use case running against several servers, copying and executing the script will consist of one work item per server. In that instance, the script is separately copied to each target and executed, and the results are consolidated at the application server. This allows for faster parallel execution. For the simplest use case, a script is sometimes executed once, and passed a list of targets. For this use case, there is only one work item. Some tasks, like software deployment, can have several workitems, including the simulate, stage, and commit phases, each of which is one work item. Work Item Threads (WIT) Units of work to be performed for a use case. Typically 1 WIT will process 1 Target WIT Minutes (WIT-Min) (Targets) X (Typical time taken by a single WIT to process a single target for a specific use case) Example: Available Capacity of 1 Job Server for 60 min: 1 Job Server (100 WIT) Capacity will be 100 X 60 = 6000 WIT-Min

7 Basic Assumptions Database should have necessary basic resources Connections/sessions Memory (PGA/SGA for Oracle) Disk performance Horizontal scaling is the recommended option to scale capacity Adding an additional Job Server (with the recommended settings) is the most common solution to scale capacity Configure and Schedule your jobs/tasks to distribute load Minimum overlap between critical jobs. For example try to schedule Patching Job and Compliance Job in such a way so that these two jobs are not waiting for application server resources at the same time to run. Jobs should always have JOB_TIMEOUT and JOB_PART_TIMEOUT properties set such that the job runs complete in a timely and controlled manner, even in environments with uneven target or agent availability. Timeout settings do not apply to asynchronous tasks Workload Estimation Example Compliance Use Case Compliance is a rules based assessment of the targets and below are the actions performed: Collect all required information from the target Transfer this data back to the Job Server Apply the rules to the returned data to assess compliance CIS Win2k8 Compliance Policy Contains ~250 Rules Sizing Calculation 1 Typical WIT runtime (in lab) = 5 WIT-Min 2 Target Count = Workload (WIT runtime X Target count) = 5 X 1000 = 5000 WIT-Min 4 Available WIT in 1 Job Server = 100 WIT 5 Typical time taken to finish the Workload by 1 Job Server = Workload / Available WIT = 5000 / 100 = 50 Min 6 Typical WIT runtime (in lab) = 5 WIT-Min 7 Target Count = If you wish to do this in half that time, than we will need 2 Job Servers

8 Use Case Estimation Disclaimers Example runtime figures are based on data captured in a lab environment. These conditions that may not exactly match your environment. Mitigating factors may include: Network Bandwidth or Latency Database Performance Incorrect Configuration Oversubscribed Virtual Clusters, Target Performance etc. For best results, follow Best Practices for Deployment Architecture, as detailed in product documentation. Patching: Analysis & Deploy Patching Basics Typically done in two phases - Analysis and Deploy Analysis 1. Identifies patches needed for each host 2. Usually, can be done over a few hours across many hosts (5k-25k), but sometimes must run quickly. 3. Relatively simple to calculate compared to Deployment s other moving parts Deploy 1. Transfers missing patch payloads over network and installs them on targets based on the analysis results Factors that Impact Performance 1. Typical execution time for one target (varies by patching policy & typical target performance) 2. Managed server (target) count in scope for task 3. Required completion time (maintenance window) Typical Calculations Time vs Infrastructure Below are the typical scenarios used to size an environment: Time - Number of targets required to complete in 1 hour (maintenance window) Infrastructure - Time taken to complete with only 2 available Job servers

Per-Target Runtime: We have evaluated these in a lab environment, and found that typical execution for two scenarios execute in similar time windows, Analysis ~5 minutes / target for a typical

9 Per-Target Runtime: We have evaluated these in a lab environment, and found that typical execution for two scenarios execute in similar time windows, Analysis ~5 minutes / target for a typical Windows 2008R2 target analysis ~6 minutes / target for a typical RHEL 5 64-bit target analysis Deploy ~25 minutes / target to deploy 25 patches (~85MB size) on the target Performance Lab Configuration Test Setup Details Windows Patch Analysis Conditions Linux Patch Analysis Conditions Patch Deploy Environment has NSH Proxy configured Multiple Application Server configurations with ~15K targets Jobs did not overlap or compete with other jobs for resources Standard Patch catalog and analysis jobs were created Filters: Win2k8 & Win2k12 Platforms Parallel Target Processing: 100 Filters: RHEL-5 & RHEL-6 Platforms Parallel Target Processing: Patches with a total size if 85MB Parallel Target Processing: 100 Patch Analysis: Constant Time Given the constraints and typical configurations above, we see that in an environment where Patching Analysis must complete in about an hour: 1 Job Server can process about 1000 targets in that hour 10,000 targets in an hour will require about 8 Job Servers If you need to process many hosts quickly, add more Job Servers

Patch Analysis: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 30 minutes to

10 Patch Analysis: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 30 minutes to analyze 1000 targets, 4 hours to analyze 10,000 targets. If you have limited resources, analysis will take more time to complete. Patch Deploy: Constant Time Given the constraints and typical configurations above, we see that in an environment where Patching Analysis must complete in about an hour: 3 Job Servers can process about 500 targets in that hour 3000 targets in an hour will require about 18 Job Servers If you need to process many hosts quickly, add more Job Servers

Patch Deploy: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 65 minutes to

11 Patch Deploy: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 65 minutes to deploy on 500 targets, 6 hours 20 minutes to deploy on 3000 targets. If you have limited resources, analysis will take more time to complete. Compliance Compliance Basics Runs in two parts - Assessment and Remediation Compliance Phases Assessment - Rules based assessment of each target in application server by pulling the necessary information from the target Remediation Applies remediation scripts to all the non-compliant targets Factors that Impact Performance Typical execution time for one target (varies by Compliance Policy & typical target performance) Managed server (target) count in scope for task Required completion time (maintenance window) Typical Calculations Time vs Infrastructure Below are the typical scenarios used to size an environment: Time - Number of targets required to complete in 1 hour (maintenance window) Infrastructure - Time taken to complete with only 2 available Job servers Per-Target Runtime:

12 We have evaluated these in a lab environment, and found that typical execution for two scenarios execute in similar time windows, ~6 minutes / target for a typical CIS RHEL 6 Policy (250 Rules 70% Native & 30% EO) ~7 minutes / target for a typical CIS Windows 2008 Policy (250 Rules 90% Native & 10% EO) Performance Lab Configuration Test Setup Details CIS RHEL 6 Policy CIS Windows 2008 Policy Environment has NSH Proxy configured Multiple Application Server configurations with ~15K targets Jobs did not overlap or compete with other jobs for resources Standard CIS Policy Template Typical RHEL 6 Target - 2GB RAM, 1vCPU, 20GB Disk 250 Rules 70% Native & 30% EO (8.7 BSA and higher version) Typical Windows 2008 Target - 2GB RAM, 1vCPU, 20GB Disk 250 Rules 90% Native & 10% EO (8.7 BSA and higher version) To create an extended object (EO) rule, you must identify a script that generates output according to a prescribed format. Using one of the many grammar files BMC Server Automation supports (see Grammar files supported by BMC Server Automation), the system can parse that output and present it in a format consistent with other information you view when browsing server objects or viewing the results of snapshots or audits. When you access the extended object (using browse, snap, audit, or compliance), the script executes and information for the object is refreshed. You can associate an extended object with a custom icon, making it easy to identify the extended object when browsing. Compliance - CIS RHEL 6 Policy: Constant Time Given the constraints and typical configurations above, we see that in an environment where Compliance must complete in about an hour: 1 Job Server can process about 1000 targets in that hour 10,000 targets in an hour will require about 10 Job Servers If you need to process many hosts quickly, add more Job Servers

Compliance CIS RHEL 6 Policy: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 40

13 Compliance CIS RHEL 6 Policy: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 40 minutes to analyze 1000 targets,. More than 4 hours to analyze 10,000 targets. If you have limited resources, analysis will take more time to complete. Compliance - CIS Windows 2008 Policy: Constant Time Given the constraints and typical configurations above, we see that in an environment where Compliance must complete in about an hour: 1 Job Server can process about 1000 targets in that hour 10,000 targets in an hour will require about 12 Job Servers If you need to process many hosts quickly, add more Job Servers

14 Compliance - CIS Windows 2008 Policy: Constant Resources Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 45 minutes to analyze 1000 targets, Nearly 5 hours to analyze 10,000 targets. If you have limited resources, analysis will take more time to complete. Script Execution: NSH Script Job Script Execution Basics Network Shell (NSH) Script Jobs are categorized into 4 types: Execute the script separately against each host (Type 1) Execute the script once, passing the host list as a parameter to the script (Type 2) Copy and execute the script against each host separately (Type 3) Execute the script using the PERL interpreter, passing the host list as a parameter (Type 4) Factors that Impact Performance Typical execution time for one target (varies by commands inside the script & typical target performance: a find script looking across an entire filesystem will take much longer to execute than the uname command) Managed server (target) count in scope for task Available system resources on appservers and targets. When time is constrained, as when doing changes within maintenance windows, it s important to consider these factors. It s common to either parallelize tasks using Type 1 or Type 3 tasks, break tasks into parts or server lists into groups, or manage other workloads for performance in critical maintenance windows.

Typical Calculations Time vs Infrastructure Below are the typical scenarios used to size an environment: Time - Number of targets required to complete in 1 hour (maintenance window) Infrastructure -

15 Typical Calculations Time vs Infrastructure Below are the typical scenarios used to size an environment: Time - Number of targets required to complete in 1 hour (maintenance window) Infrastructure - Time taken to complete with only 2 available Job servers Per-Target Runtime: We have evaluated these in a lab environment, and found that typical execution for two scenarios execute in similar time windows ~30 seconds / target for NSH Script Type 1 Job Performance Lab Configuration Test Setup Details Environment has NSH Proxy configured Multiple Appserver configurations with ~15K targets Jobs did not overlap or compete with other jobs for resources NSH script was used to gather below information from the target Running Process Info nsstats information Network Share information Account Settings NSH Script Job types tested Type 1 Execute the script separately against each host Typical Windows 2008 Target - 2GB RAM, 1vCPU, 20GB Disk Script Execution - NSH Script Job (Type 1): Constant Time Given the constraints and typical configurations above, we see that in an environment where NSH Script Job must complete in about an hour: 1 Job Server can process about targets in that hour If you need to process many hosts quickly, add more Job Servers Script Execution - NSH Script Job (Type 1): Constant Resources

Given the constraints and typical configurations above, we see that in an environment where we have

limited resources, analysis will take more time to complete.

Graphical Interface NSH & BLCLI Authentication BDSSA Data Warehouse Web Portal Standard and Custom

- Job Partitioning - RBAC JOB + NSH PROXY N O D E S RSCD Windows, Linux, Unix, ESX Lightweight

Oracle, SQL Server BDSSA Server Job Server Minimum Hardware Requirements for the Job Type

16 Given the constraints and typical configurations above, we see that in an environment where we have only two (2) Job Servers available we will take 30 minutes to analyze targets If you have limited resources, analysis will take more time to complete. BSA Component Sizing Typical deployment environment diagram C O N S O L E Management Console Graphical Interface NSH & BLCLI Authentication BDSSA Data Warehouse Web Portal Standard and Custom Reporting M I D - T I E R CONFIG + NSH Application Server PROXY - Automation Engine - Load Balanced - Job Partitioning - RBAC JOB + NSH PROXY N O D E S RSCD Windows, Linux, Unix, ESX Lightweight Service Encrypted communication Single TCP port DEPOT File Server Local Disk, SAN NAS Database Oracle, SQL Server BDSSA Server Job Server Minimum Hardware Requirements for the Job Type Application Server Supported Platforms Architecture Processor Disk Storage RAM RHEL (6.x and above) 64-bit 2vCPU (Dedicated) 50GB 12-16GB Windows (2008R2 and above) (min)

17 Notes: 1. Solaris is still available as an application server type, and OS sizing is similar. 2. On an application server host keep at least 4GB for the OS and associated applications running before configuring BladeLogic. Standard Settings (recommended) Settings Standalone Deployment Deployment with NSH Proxy** Work Item Threads Database Connections Required JVM Heap Size Default Job (default) Job + NSH Proxy 100 (default) 200 (default) 8GB Note: ** NSH Proxy with Job Server is the recommended option for: 1. Connectivity to managed targets (solves some firewall challenges) 2. Authentication for NSH 3. Centralized audit point 4. Adds an extra layer of security protecting NSH agents Configuration Server Minimum Hardware Requirements for the Configuration (CONFIG) Type Application Server Supported Platforms Architecture Processor Disk Storage RAM RHEL (6.x and above) 64-bit 2vCPU (Dedicated) 50GB Windows (2008R2 and above) (min) 12-16GB Standard Settings (recommended) Settings Standalone Deployment Deployment with NSH Proxy* Default Config (default) Config + NSH Proxy

18 Database Connections Required** 50 JVM Max Heap 4GB (Config) / 8GB (Config + NSH Proxy) Note: *Recommended deployment to manage interactive traffic from Network Shell clients. **Database specific connections needed by Config Server to handle client side requiests.

19 Database & Repository Sizing Database Server Database Type Architecture Processor Disk Storage Oracle (11g R2 / 12c) 64-bit 4vCPU 500GB Oracle RAC (11g R2 /12c) (Dedicated) SQL Server (2012 / 2014) SQL Server Cluster (2012 / 2014) RAM 32GB Storage Types Processor Disk RAM Storage File Server 2vCPU (Dedicated) 350 GB 8 GB PXE Data store 1vCPU (Dedicated) 100 GB 4 GB Repeater 1vCPU (Dedicated) 25 GB 4 GB Patch Payload Repository (See below for specifics) 1vCPU (Dedicated) See next slide 4 GB

20 Patch Repository Sizing Offline Downloader Support Offline Downloader Lists the OS platforms where the Offline Downloader can be run and patch payloads can be downloaded on to. Repository for Catalog Lists the OS platforms where the catalog / metadata is prepared for the downloaded patches OS Platform Offline Downloader Platform Repository Creation Platform for Catalog AIX (SUMA) AIX AIX AIX (FIXGET) AIX / Windows / Any RPM AIX based Linux SOLARIS All OS Platforms All OS Platforms RHEL RHEL / Windows Any RPM based Linux RHEL 7 RHEL Any RPM based Linux WINDOWS All OS Platforms All OS Platforms OEL Any RPM based Linux Any RPM based Linux SUSE Linux Any RPM based Linux SUSE SUSE 12 SUSE with SMT installed SUSE DEBIAN All OS Platforms All OS Platforms Typical Patch Catalog Storage Sizing (may change over time) OS Filter Application Platforms that can store the payload RHEL RHEL 7 WINDOWS - RHEL 5 x86_64 - RHEL 7 (30GB) x86_64 (10GB) - RHEL 5 x86 (30GB) - RHEL 6 x86_64 (35GB) - RHEL 6 x86 (35GB) - OEL - RHEL - SUSE Linux - RHEL 6 - RHEL 7 - Windows 2012 (8GB) - Windows 2008 (12GB) - Java (5GB) - Office 2007 (5GB) - Office (10GB) - Office 2013 (30GB) - Notepad ++ (1GB) - Winzip (1GB) - SQLServer 2008 (15GB) - RLAP (0.5GB) All

21 AIX SOLARIS OEL SUSE LINUX OS Filter - AIX 6.1 TL 01 (20GB) - AIX 7.1 TL 01 (20GB) Platforms that can store the payload - Solaris 10 x86 (5GB) - Solaris 10 SPARC (7GB) - Solaris 9 x86 (5GB) - Solaris 9 SPARC (8GB) - OEL 6 x86 (43GB) - OEL 6 x86_64 (47GB) - OEL 5 x86 (40GB) - OEL 5 x86_64 (45GB) AIX All - OEL - RHEL - SUSE Linux - SUSE 10 x86 (10GB) - SUSE 10 x86_64 (12GB) - SUSE 11 x86 (10GB) - SUSE 11 x86_64 (12GB) SUSE Linux (recommended) Additional BSA Components Classification based on managed targets Component Small Scale Medium Scale Large Scale Geographically Distributed Targets >10000 (multiple) Targets < Active Client Connections (RCP + BLCLI Clients) Configuration Server SOCKS Proxy ** 1 SOCKS Proxy per targets, multiple proxy servers are common for very large environments Repeater*** - 1 Repeater per remotely managed targets, multiple repeaters are common for large remote data centers Citrix Servers for Remote Users - Yes Yes Yes (1/Location) Note: ** : SOCKS Proxy should manage not more than 1500 targets per instance *** : Repeaters needed only if there are subnets or remote data centres

22 Component Small Scale Medium Scale Large Scale Geographically Distributed Targets Patch Repository & Downloader 1 Provisioning Data store 1 1 PXE data store for 1500 remotely managed targets File Server 200GB 350GB 500GB 500GB* Database Storage 100GB 500GB 1TB 1.5TB-2TB Database Type Network Bandwidth Appserver-Database: Typically, we see network bandwidth between the App server and database components in the 1-10GBps range, and latency is ideally as low as possible, typically sub-1ms. Typical customer deployments are within the same building. Appserver-Repeater: Typical utilization for the repeater components are in the 10Mbps range (a bit more than for an individual agent), as these are used to scale payload deployment in remote data centers. Appserver-Agent: Varies by use case, but 1.5Mbps is pretty typical for timely deployments of software payloads. Bandwidth from appserver to agents can be relatively low due to the staging part of the deployment process, but may impact timeliness of deployments (maintenance windows) if staging ahead is not used. Latency tolerance is relatively high, but can cause problems if very high (>2000 msec typ.) Network Bandwidth (Scope) Oracle / SQL Server Small Scale SQL Cluster / Oracle RAC Medium Scale Oracle RAC Large Scale Geographically Distributed Targets Appserver --- Database 1-10Gbps 1-10Gbps 1-10Gbps 1-10Gbps Appserver --- Agent 1.5Mbps 1.5Mbps 1.5Mbps 45Mbps Appserver --- Appserver 2Mbps 2Mbps 2Mbps 6Mbps Appserver --- Repeater 10Mbps 10Mbps 10Mbps 45Mbps Oracle RAC

23 Summary There are many ways to trade off the performance and sizing resource requirements in automation environments: like any sizing exercise, there will be some balance between the requirements of speed, cost, and quality. Appropriately balancing these requirements will help conserve resources for staffing, enablement/education, and other infrastructure projects. It is important to revisit these requirements regularly (most customers do so every 1-2 years, some do quarterly reviews) to ensure customers are satisfied with performance, work is being executed in a timely fashion, that there is enough slack in the system to handle unplanned workloads, and that budgets are reasonable for the workload.

24 BMC delivers software solutions that help IT transform digital enterprises for the ultimate competitive business advantage. From mainframe to cloud to mobile, we pair high-speed digital innovation with robust IT industrialization allowing our customers to provide amazing user experiences with optimized IT performance, cost, compliance, and productivity. We believe: Technology is the heart of every business IT drives business to the digital age

BSA Best Practices Webinars Distributed Installations Sean Berry Customer Engineering

BSA Best Practices Webinars Distributed Installations Sean Berry Customer Engineering Agenda Overview Use Cases Repeaters Repositories Proxies Copyright 5/11/2013 BMC Software, Inc 2 Overview Use Cases