Scalability Engine Guidelines for SolarWinds Orion Products

Scalability Engine Guidelines for SolarWinds Orion Products Last Updated: March 7, 2017 For a PDF of this article, click the PDF icon under the Search bar at the top right of this page. Your Orion Platform product installation consists of a Main Polling Server (Orion Web Console and the Main Polling Engine), and the Orion Database Server. Polling engines gather device statistics and store the information on the Orion Database Server. The Orion Platform server reads the stored information from the Orion Database Server. Your Main Polling Engine polls a definite number of elements depending on the Orion Platform product. Too many elements on a single polling engine can have a negative impact on your SolarWinds server. When the maximum polling throughput on a single polling engine is reached, the polling intervals are automatically increased to handle the higher load. To keep default polling intervals, you need to add polling capacity. How do I improve the monitoring capacity? Install an Additional Polling Engine to disperse the load between multiple servers. Install an Additional Web Server to balance the load on the Main Polling server, for example when more users are logged in at the same time, or in secured environments where you have your Orion Platform products behind a firewall. For NPM and SAM, poll more elements by "stacking" multiple Additional Polling Engine licenses on a polling engine. Additional Polling Engines can cope with database outages, handle bad or slow connections, and work over WAN. Additional Polling Engines use Microsoft Message Queuing (MSMQ) to enable the local storage for up to 1 GB of polled data per polling engine to prevent data loss when the connection between the polling engine and the database is temporarily lost. How do I know that I need to scale my Orion Platform product? When you exceed a polling engine capacity, Orion Platform products notify you about it. See the Notifications in the Orion Web Console. 1

Review your alerts. If the Polling rate limit exceeded out-of-the-box alert is enabled, the alert sends an email and adds an entry to All Active Alerts. Available Scalability Engine deployment options Centralized Deployment with Additional Polling Engines Distributed Deployment with Main and Additional Polling Engines in regions Centralized Deployment with Remote Office Pollers Requirements Before you begin, be sure your Additional Polling Engines and Additional Web Servers meet the following requirements. The Main Polling Engine should be upgraded or installed before upgrading/installing the Additional Polling Engine. The latency (RTT) between each Additional Polling Engine and the database server should be below 300 ms. Degradation may begin around 200 ms, depending on your utilization. Ping the Orion SQL Server to find the current latency. A reliable static connection between the server and the regions. Additional Polling Engine Ports Additional Polling Engines have the same port requirements as Main Polling Engine. The following ports are the minimum required for an Additional Polling Engine to ensure them most basic functions. Port Protocol Service/ Process Direction Description 1433 TCP SolarWinds Collector Service Outbound The port used for communication between the APE and the Orion database. 1801 TCP Message Queuing WCF Inbound The port used for MSMQ messaging from the Orion Web Console to the Additional Polling Engine. 2

Port Protocol Service/ Process Direction Description 5671 TCP RabbitMQ Bidirectional The port used for SSL-encrypted RabbitMQ messaging from the Orion Web Console to the Additional Polling Engine. 17777 TCP SolarWinds Information Service Bidirectional The port used for communication between the Additional Polling Engine and the Orion Web Console. Additional Web Server Ports Port Protocol Service/ Process Direction Description Default additional web server port. Open the port to enable 80 TCP World Wide Web Publishing Service Inbound communication from your computers to the Orion Web Console. If you specify any port other than 80, you must include that port in the URL used to access the web console. For example, if you specify an IP address of 192.168.0.3 and port 8080, the URL used to access the web console is http://192.168.0.3:8080. 1433 TCP SolarWinds Collector Service Outbound The port used for communication between the SolarWinds server and the SQL Server. Open the port from your Orion Web Console to the SQL Server. 1801 TCP Message queuing Outbound The port used for MSMQ messaging from the Additional Web Server to the Main Polling Engine. 5671 TCP RabbitMQ Outbound The port used for SSL-encrypted RabbitMQ messaging from the Additional Web Server to the Additional Polling Engine. 17777 TCP SolarWinds Collector Service Outbound Orion module traffic. Open the port to enable communication from your polling engine to the web server, and from the web server to your polling engine. 3

Centralized Deployment with Additional Polling Engines Centralized Deployment with Additional Polling Engines polls data locally in each region and the polled data is stored centrally on the database server in the primary region. All licenses are shared in a Centralized Deployment. Use this deployment if your organization requires centralized IT management and localized collection of monitoring data. Users can view all network data from the Orion Web Console in the Primary Region where the main SolarWinds Orion server is installed. Users can log in to a local Web Console if an Additional Web Server is installed in a secondary region. With Centralized Deployment, you can: Add, delete, and modify nodes, users, alerts and reports centrally, on the Main Orion Server. Scale all installed Orion Platform products. Scaling one Orion Platform product increases the capacity of the other Orion Platform products. For example, installing an Additional Polling Engine for NPM also increases the polling capacity for SAM. Specify the polling engine that collects data for monitored nodes and reassign nodes between polling engines. All Key Performance Indicators (KPIs), such as Node Response Times, are calculated from the perspective of the polling engine. For example, the response time for a monitored node in Region 2 is equal to the round trip time from the Additional Polling Engine in Region 2 to that node. 4

Distributed Deployment with Main and Additional Polling Engines in regions In a Distributed Deployment each region is licensed independently, and data is polled and stored locally in each region. Scale each region independently by adding Additional Polling Engines. You can access monitoring data from each region in a central location with the Enterprise Operations Console (EOC). SolarWinds Enterprise Operations Console must be installed and licensed for a Distributed Deployment. With Distributed Deployment you can: Use local administration to manage, administer, and upgrade each region independently. Create, modify, or delete nodes, users, alerts, and reports separately in each region. Export and import objects, such as alert definitions, Universal Device Pollers, and SAM templates between instances. Mix and match modules and license sizes as needed. For example: Region 1 has deployed NPM SL500, NTA for NPM SL500, UDT 2500, and 3 additional polling engines Region 2 has deployed NPM SLX, SAM AL1500, UDT 50,000, and 3 additional polling engines Region 3 has deployed NPM SL100 only and 3 additional polling engines Bandwidth used by EOC is not significant. EOC polls only status and statistics incrementally for each SolarWinds Orion server and does not store historical data. Centralized Deployment with Remote Office Pollers To deploy your Orion Platform product in numerous, remote locations when you do not need to scale up your installation, use a Remote Office Poller for Additional Polling Engine (ROP, mini-poller). See Scalability Engine Guidelines by product to verify if your product supports Remote Office Pollers. 5

Select a Remote Office Pollers by the number of elements you need to poll: ROP250 polls up to 250 elements. ROP1000 polls up to 1000 elements. Follow the steps for installing and activating Additional Polling Engines in order to deploy Remote Office Pollers. Deploy Scalability Engines 1. Review the Scalability Engine deployment options. 2. Review the requirements and pre-flight checklist. 3. Use the Orion Scalability Engine Installer to deploy Additional Polling Engines and Additional Web Servers. Pre-flight checklist Before you install an Additional Polling Engine in your environment, be sure you complete the following actions: Be sure your product uses Orion Platform 2016.2 and later. To find out the Orion Platform version, log in to the Orion Web Console and see the Orion Platform version in the footer. If the version is 2016.1 and earlier, see Orion Bundle for additional servers. Install or upgrade the Main Polling Engine. Ensure product versions match between the Primary Polling Engine, all Additional Polling Engines, and Additional Web Servers. This includes the version of.net. Find a version number listed in the footer of the Web Console. If your product versions do not match, you must upgrade before you can install Additional Polling Engines. Verify port requirements for your SolarWinds product. Acquire a user name and password with administrative privileges to the Orion Web Console on your Main Polling Engine. Be sure the Additional Polling Engine uses the same SQL database as the Main Polling Engine. If you configured an alert with a Send Email action to trigger on a node monitored by an Additional Polling Engine, confirm that the Additional Polling Engine can access your SMTP server. Add the IP address of your Additional Polling Engine to Windows Servers on the Security tab. Make sure that the following options are set: 6

Ensure that a case-sensitive community name has been specified. Ensure that Accept SNMP packets from any host is selected OR ensure that the ipmonitor system is listed within the Accept SNMP packets from these hosts list. Ensure that your network devices allow SNMP access from the new polling engine. On Cisco devices, you can for example modify the Access Control List. Deploy Additional Polling Engines and Additional Web Servers with the Orion Scalability Engine Installer Stack licenses (NPM and SAM) If your NPM or SAM polling engine have enough resources available, you can stack the licenses. Stacking licenses enhances the polling capacity of your Main Polling Engine or Additional Polling Engine. A stack requires only one IP address, regardless of the number of additional polling engines. If the resources on your polling engine are already constrained and you cannot allocate additional resources, consider installing an Additional Polling Engine. Assign multiple licenses to a polling engine with the web-based License Manager. The maximum number of licenses you can apply to a single server depends on the Orion Platform product. 1. In the Orion Web Console, click Settings > All Settings > License Manager. 2. Click Add/Upgrade License, enter the activation key and registration details, and click Activate. The activated license with activation details displays in the License Manager. 3. In the License Manager, select the license, and click Assign. 4. Select a polling engine, and click Assign. The license is stacked on the selected polling engine, and its polling capacity is extended. Install Additional Polling Engines or Additional Web Servers The installer checks your Main Polling Engine for products and versions, and then it downloads them directly from your Main Polling Engine. Keep the Orion services running on the Main Polling Engine when installing Additional Polling Engine and Additional Web Server. 1. In the Orion Web Console, click Settings > All Settings > Polling Engines. 2. Click Download Installer Now. 3. Run the installer on the computer you want to use as your additional polling engine or website. 4. Enter the main Orion server credentials. 5. Choose either Additional Polling Engine or Additional Website. If you want to install a backup server for High Availability, view SolarWinds High Availability requirements first. The packages are downloaded from the main Orion server. 7

6. Follow the onscreen instructions to complete your installation and configuration. The Additional Polling Engine or Additional Web Server is installed, together with all hotfixes for products released later than September 2016. For example, when installing or upgrading to SAM 6.3 and IPAM 4.3.2, any hotfixes for SAM 6.3 are installed. You must install the hotfix for IPAM manually. Repeat installing on all Additional Polling Engines and then on all Additional Web Servers in your environment. Activate the Additional Polling Engine licenses. 1. In the Orion Web Console, click Settings > All Settings > License Manager. Activate licenses Activate licenses in the web-based License Manager in the Orion Web Console. 2. Locate the license in the License Manager, click Activate, and complete the activation. The Additional Polling Engine license is activated. Review the polling engine it is assigned to, and reassign the license if necessary. When finished, specify nodes to be polled by the Additional Polling Engine. Scalability Engine Guidelines by Product The following sections provide guidance for using scalability engines to expand the capacity of your SolarWinds installation. DameWare in Centralized Mode Database Performance Analyzer (DPA) Engineer's Toolset on the Web Enterprise Operations Console (EOC) IP Address Manager (IPAM) Log and Event Manager (LEM) NetFlow Traffic Analyzer (NTA) 8

Network Configuration Manager (NCM) Network Performance Monitor (NPM) Patch Manager (SPM) Quality of Experience (QoE) Server & Application Monitor (SAM) Serv-U FTP Server and MFT Server Storage Resource Monitor (SRM) User Device Tracker (UDT) Virtualization Manager (VMAN) VoIP & Network Quality Manager (VNQM) Web Performance Monitor (WPM) DameWare in Centralized Mode DameWare Scalability Engine Guidelines 150 concurrent Internet Sessions per Internet Proxy 5,000 Centralized users per Centralized Server 10,000 Hosts in Centralized Global Host list 5 MRC sessions per Console Database Performance Analyzer (DPA) DPA Scalability Engine Guidelines Less than 20 database instances monitored on a system with 1 CPU and 1 GB RAM 21-50 database instances monitored on a system with 2 CPU and 2 GB RAM 51-100 database instances monitored on a system with 4 CPU and 4 GB RAM 101-250 database instances monitored on a system with 4 CPU and 8 GB RAM More than 250 database instances monitored through Central Server mode 9

DPA Scalability Engine Guidelines See Link together separate DPA servers in the DPA Administrator Guide Engineer's Toolset on the Web Engineer's Toolset on the Web Scalability Engine Guidelines 45 active tools per Engineer's Toolset on the Web instance 3 tools per user session 1 active tool per mobile session 10 nodes monitored at the same time per tool 48 interfaces monitored at the same time per tool 12 metrics rendered at same time per tool Enterprise Operations Console (EOC) EOC Scalability Engine Guidelines Maximum 1 million elements WAN and/or Bandwidth Minimal monitoring traffic is sent between the EOC server and any remote Orion servers or Additional Polling Engines 10

IP Address Manager (IPAM) IPAM Scalability Engine Guidelines 3 million IPs per SolarWinds IPAM instance Log and Event Manager (LEM) LEM Scalability Engine Guidelines Maximum 120 million events per day 10,000 rule hits per day NetFlow Traffic Analyzer (NTA) NTA Scalability Engine Guidelines Remote Office Poller No Main Polling Engine Limits 50k FPS per polling engine For more information, see Network Performance Monitor (NPM) 5 APEs up to 300k FPS For more information, see Network Performance Monitor (NPM) WAN and/or Bandwidth 1.5% - 3% of total traffic seen by exporter 11

NTA Scalability Engine Guidelines Other See "Section 4 Deployment Strategies" of "NetFlow Basics and Deployment Strategies" Network Configuration Manager (NCM) NCM Scalability Engine Guidelines Remote Office Poller No Main Polling Engine Limits ~10k devices 1 Additional Polling Engine for every 10k devices, for NCM 7.1 and later Maximum of 30k devices per primary SolarWinds NCM instance (NCM server + 2 NCM Additional Polling Engines) Integrated standalone mode Network Performance Monitor (NPM) NPM Scalability Engine Guidelines Stackable Polling Engines NPM 12.0 and later: up to four total polling engines may be installed on a single server, for example one Primary Polling Engine with one to three Additional Polling Engines, or four Additional Polling Engines on the same server NPM 11.5.3 and earlier: up to three polling engines on a single server A stack requires only 1 IP address, regardless of the number of APEs. 12

NPM Scalability Engine Guidelines NPM 10.4 and later Remote Office Poller ROP250 supports 250 elements ROP1000 supports 1000 elements ~48k elements at standard polling frequencies (12k per polling engine): Node and interface up/down: 2 minutes/poll Node statistics: 10 minutes/poll Interface statistics: 9 minutes/poll Main Polling Engine Limits 25-50 concurrent Orion Web Console users SNMP Traps: ~500 messages per second (~1.8 million messages/hr) Syslog: 700-1,000 messages/second (2.5-3.6 million messages/hr) To monitor more than ~100,000 elements, consider using SolarWinds Enterprise Operations Console. One polling engine for every ~12k elements Maximum of 100k elements per primary SolarWinds NPM server (1 NPM server + 9 Additional Polling Engines). See How is SolarWinds NPM licensed? WAN and/or Bandwidth Minimal monitoring traffic is sent between the primary SolarWinds NPM server and any Additional Polling Engines that are connected over a WAN. Most traffic related to monitoring is between an Additional Polling Engine and the SolarWinds Orion database. NetPath TM Scalability With the default polling interval of 10 minutes, you can poll up to 100 paths per probing computer (polling engine or agent). The number of paths per probing computer depends on the hardware specification of the computer. See NetPath requirements for more information. Other How much bandwidth does SolarWinds require for monitoring? 13

NPM Scalability Engine Guidelines See "Orion Server Hardware Requirements" in the SolarWinds NPM Administrator Guide Patch Manager Patch Manager Scalability Engine Guidelines 1,000 nodes per automation server 1,000 nodes per SQL Server Express instance (SQL Server does not have this limitation) SQL Express is limited to 10 GB storage. For large deployments, SolarWinds recommends using remote SQL. Quality of Experience (QoE) QoE Scalability Engine Guidelines 1,000 QoE sensors 50 application per sensor Server & Application Monitor (SAM) SAM Scalability Engine Guidelines Stackable Polling Engines SAM 6.2 and later 2 polling engines can be installed on a single server. 14

SAM Scalability Engine Guidelines Remote Office Poller SAM 5.5 and later Main Polling Engine Limits ~8-10k component monitors per polling engine 25-50 concurrent Orion Web Console users 1 Additional Polling Engine for every 8-10k component monitors. Maximum of 150k component monitors per primary SolarWinds SAM installation (1 SAM server + 14 Additional Polling Engines). For more information about licensing, see the SAM Licensing Guide. WAN and/or Bandwidth Minimal monitoring traffic is sent between the primary SAM server and any Additional Polling Engines that are connected over a WAN. Most traffic related to monitoring is between an Additional Polling Engine and the SolarWinds database. Bandwidth requirements depend on the size of the relevant component monitor. Based on 67.5 kb / WMI poll and a 5 minute polling frequency, the estimate is 1.2 Mbps for 700 component monitors. For more information, see How do SNMP and WMI polling compare? WMI is best suited for environments where latency is < 100ms. Other WMI Security Blog Serv-U FTP Server and MFT Server Serv-U FTP Server and MFT Server Scalability Engine Guidelines 500 simultaneous FTP and HTTP transfers per Serv-U instance 50 simultaneous SFTP and HTTPS transfers per Serv-U instance 15

Serv-U FTP Server and MFT Server Scalability Engine Guidelines For more information, see the Serv-U Distributed Architecture Guide. Storage Resource Monitor (SRM) SRM Scalability Engine Guidelines Stackable Polling Engines No, one APE instance can be deployed on a single host Yes Remote Office Poller Poller remotability is a feature enabling the local storage, using MSMQ, of up to ~1 GB of polled data per poller in case the connection between the polling engine and the database is temporarily lost. Main Polling Engine Limits Maximum of 40K LUNs per polling engine (primary or additional) 25-50 concurrent Orion Web Console users Use Additional Polling Engines for horizontal scaling The upper limit that can be handled by a single SRM instance is 160K LUNs. For larger environments, please contact SolarWinds for further assistance. WAN and/or Bandwidth Minimal monitoring traffic is sent between the primary SRM server and any Additional Polling Engines that are connected over a WAN. Most traffic related to monitoring is between an Additional Polling Engine and the SolarWinds database. 16

User Device Tracker (UDT) UDT Scalability Engine Guidelines Remote Office Poller No Main Polling Engine Limits 100k ports 1 Additional Polling Engine per 100k additional ports Maximum of 500k port per instance (1 Primary Polling Engine and 4 Additional Polling Engines) WAN and/or Bandwidth None Other UDT version 3.1 supports the ability to schedule port discovery In UDT version 3.1 the Max Discovery Size is 2,500 nodes/150,000 ports Virtualization Manager (VMAN) VMAN Scalability Engine Guidelines 3000 VMs* 700 Hosts 75 Clusters 1800 Datastores *By using federated collectors, you can monitor 10,000 or more VMs. For information about federated collectors, see the Virtualization Manager documentation. 17

VoIP & Network Quality Manager (VNQM) VNQM Scalability Engine Guidelines Remote Office Poller No Primary Polling Engine Limits ~5,000 IP SLA operations ~200k calls/day with 20k calls/hour spike capacity 1 Additional Polling Engine per 5,000 IP SLA operations and 200,000 calls per day Maximum of 15,000 IP SLA operations and 200,000 calls per day per SolarWinds VNQM instance (SolarWinds VNQM + 2 VNQM Additional Polling Engines) WAN and/or Bandwidth Between Call Manager and VNQM: 34 Kbps per call, based on estimates of ~256 bytes per CDR and CMR and based on 20k calls per hour Web Performance Monitor (WPM) WPM Scalability Engine Guidelines Remote Office Poller No, but recordings may be made from multiple locations Main Polling Engine Limits 12 recordings per player One Player Location per 12 monitored transactions, with the complexity of transactions determining the limits per player. 18

Frequently Asked Questions Does each module have its own polling engine? No, an Additional Polling Engine may have all relevant modules installed on it, and it performs polling for all installed modules. An Additional Polling Engine works the same way as your Main Polling Engine on your main server. For example, if you have NPM and SAM installed, install one Additional Polling Engine and it performs polling for both NPM and SAM. Are polling limits cumulative or independent? For example, can a single polling engine poll 12k NPM elements AND 10k SAM monitors together? Yes, a single polling engine can poll up to the limits of each module installed, if sufficient hardware resources are available. Are there different size license available for the Additional Polling Engine? No, the Additional Polling Engine is available with an unlimited license. Can you add an Additional Polling Engine to any size module license? Yes, you can add an Additional Polling Engine to any size license. Adding an Additional Polling Engine does not increase your license size. For example, if you are licensed for an NPM SL100, adding an Additional Polling Engine does not increase the licensed limit of 100 nodes/interfaces/ volumes, but the polling load is spread across two polling engines. Will an Additional Polling Engine allow me to monitor overlapping IPs? Yes, you can add nodes with the same IP Address to separate polling engines to monitor overlapping IP Addresses. What happens if the connection from a polling engine to the Orion Database Server is lost? If there is a connection outage to the Orion Database Server, polling engines use Microsoft Message Queuing (MSMQ) to cache the polled data on the Additional Polling Engine servers. The amount of data that can be cached depends on the disk space available on the polling engine server. The default storage space is 1 GB. Up to one hour of data can be cached. When the connection to the database is restored, the Orion Database Server is updated with the locally cached data, the oldest data is processed first. 19

If the database connection is broken for a longer than an hour, the collector queue becomes full, the newest data is discarded until a connection to the database is re-established. 20