1
You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about this documentation, submit your feedback to docfeedback@vmware.com. 3401 Hillview Ave Palo Alto, CA 94304 www.vmware.com Copyright 2018. All rights reserved. Copyright and trademark information. 2
Contents 1. Introduction...4 2. Basis for Sizing...6 3. Recommended Pace of IoT Infrastructure Enrollment...7 4. Recommended Load of Operations in Peak Scale...8 5. Sizing for Medium up to 15,000 Managed Objects...9 6. Sizing for Small up to 5,000 Managed Objects... 10 7. Recommended Server Tuning/Configuration Parameters... 11 8. Recommended Configuration Parameters for Various Sizes... 13 3
1. Introduction This document captures the recommended sizing guidelines of the Pulse IoT Center 1.1 components in customer deployments with two different sizing levels. A small deployment represents customers having up to 5000 devices. A medium deployment represents customers having up to 15000 devices. This document also includes some important application level or component level parameters and their recommended settings based on the design and tests that were conducted. VMware has scale tested the Pulse IoT Center system in a vcenter infrastructure and simulated 15000 managed objects using virtual gateways. At this state, the system and its components has been monitored for stability for 3-4 weeks, while performing user operations or workflows. The details are mentioned in Sections 3 and 4 below. VMware Pulse IoT Center provides a control plane solution to monitor and manage the Enterprise IoT infrastructure and drive its operational efficiency. Pulse IoT Center 1.1 is available as an on-premise solution. A collection of edge services including a combination of device management, security, identity, ability to upgrade, and networking capabilities are offered to customers. VMware Pulse IoT Center includes the following server-side components: VMware Pulse Management Suite (Backend and Console) vrealize Operations Manager with Helix Adapter Support EMQTT Broker VMware Pulse IoT Center Console (UI) VMware Pulse IoT Center API Server The following schematic depicts the deployment view of the various components and how they talk to each other. 4
Internet DMZ Intranet Windows Server Windows Server Airwatch Services Airwatch Cloud Messaging Airwatch MSSQL Database Sensor/ Actuator Sensor/ Actuator IoTC Agent --------------- Liota 2001 User Sync Windows Server Airwatch Admin Console Airwatch API Service Outbound only Windows Server Airwatch Cloud Connector Active Directory/ Other Directory Services IOT Gateway SUSE Linux (Optional Component) Sensor/ Actuator 8883 VMware Identity Management Server Photon OS Pulse Console Service Photon OS SUSE Linux Helix Adapter EMQTT Broker 8883 vrealize Operations Manager Photon OS Pulse API Service Postgres Database Note: For a small deployment, you can combine the AirWatch Admin Console and the AirWatch Services into a single machine. 5
2. Basis for Sizing The sizing guidelines for the Pulse IoT Center has been arrived at, after testing the system with a scale of 15000 simulated gateways and by monitoring the system for over three weeks. The test scenarios performed on the gateways and the sequence of operations is represented below. A gateway, as soon as it is enrolled, publishes its properties to IoTC from the registration package, publishes metrics for itself using a Liota package which also registers five IoT devices. Subsequently, properties and metrics for the devices are loaded in a sequence for each device. 1. Gateway Enrolment & 2. Gateway Metrics 3.a IoT -1 4.a IOT -2 5.a IoT -3 6.a IoT -4 7.a IoT -5 3.b 4.b 5.b 6.b 7.b 3c. Metrics 4.c Metrics 5c. Metrics 6.c Metrics 7.c Metrics 6
3. Recommended Pace of IoT Infrastructure Enrollment If you plan to build the system up to 15000 managed objects in quick succession, VMware recommends a phased approach to enroll them. Avoid large number of enrollments at the same time. Following is the pace/concurrency at which enrollments were tested. VMware recommends that you contain the concurrency and space the activities as per the guidelines for effective performance of the system. You can start a maximum of six gateways at a time. After enrollment, each gateway can simultaneously load packages for connected devices at an average of 5 devices/gateway, load properties at an average of 10/managed object, and metrics at an average of 2/ managed object. s were scaled up in three different phases as follows, to reach a steady state with 15000 managed objects, with entities as listed below: Concurrency Rate Batches Gateways IoT s 1st Phase 2nd Phase 3 rd Phase Metrics /5 Mins Cumulative Managed Objects Cumulative Cumulative Metrics Flowing/5 mins 5 25 300 60 5,000 50,000 10,000/5 mins 5 25 300 60 10,000 100,000 20,000/5 mins 5 25 300 60 15,000 150,000 30,000/5 mins 7
4. Recommended Load of Operations in Peak Scale It is recommended that you do not exceed 15000 managed objects for the VMware Pulse IoT Center 1.1 with a medium sized instance. However, if there is a need to scale beyond 15000, there are some recommendations provided at the end of this document for different types of scale from 5000 to 20000 managed objects. However, at a scale of 15000 managed objects, the system was tested for the following rate of usual operations and it was observed that the system responded as expected. Transaction in Steady State Registrations Property Updates Product Provisioning Agent Upgrade Frequency/Rate of Transactions 1 GW Registrations in 3 hours 15000 properties updated every day 1 OTA update/day involving 3000 managed objects 150 Gateways upgraded at a time Remarks Spaced at 1000 property updates/hour 8
5. Sizing for Medium up to 15,000 Managed Objects The following table captures various components of the Pulse IoT Center and their required hardware capacities, targeting a medium sized instance of up to 15000 managed objects. Medium up to 15,000 Managed Objects Module vcpus RAM (GB) Storage (GB) DB IOPS (Total) Prerequisite Software Management Suite Console (+Api) Management Suite - Services Management Suite - Database Management Suite - AWCM Server Operations Management suite backend 4 8 50 Windows Server 2012/R2 8 16 100 Windows Server 2012/R2 12 32 450 1500-2000 Windows Server 2012/R2, SQL Server 2012 8 8 16 Windows Server 2012/R2, 24 128 500 The required software is packaged Storage must be customized as per the need. This normally works as per the standard recommendations. Pulse Console 8 32 40 The required software is packaged Pulse API Server 12 32 40 The required software is packaged MQTT Broker 8 32 32 The required Software is packaged Total 84 288 1228 9
6. Sizing for Small up to 5,000 Managed Objects The following table captures various components of the Pulse IoT Center and their required hardware capacities targeting a small sized instance of up to 5000 managed objects. Small up to 5,000 Managed Objects Module vcpus (Total) RAM (GB) (Total) Storage (GB) (Total) DB IOPS (Total) Prerequisite Software Management Suite (Console+Api+ Services) Management Suite - Database 4 6 80 Windows Server 2012/R2 4 16 100 500-1000 Windows Server 2012/R2, SQL Server 2012 Management Suite AWCM Server Operations Management suite backend 4 8 10 Windows Server 2012/R2, 8 16 280 The required software is packaged Pulse Console 4 8 40 The required software is packaged Pulse API Server 8 16 32 The required software is packaged MQTT Broker 4 8 16 The required software is packaged Total 36 78 558 10
7. Recommended Server Tuning/Configuration Parameters During testing, some important application/component parameters of various systems were identified, that can play a vital role in the responsiveness of the system. The following are the recommended settings for these important parameters of the Pulse IoT Center components. Component Parameter or Configuration Name MQTT Broker mqtt.session.max_inflight = 10 mqtt.session.retry_interval = 3600s mqtt.mqueue.max_length = 1000000 Details The release binaries (OVAs) come pre-tuned with these settings. These parameters are located in the MQTT Broker at the following path: /etc/emqttd/emq.comf Pulse API Server scheduler.fixed.delay: 3600000 By default, Pulse API DB synchronizes managed objects data every one-hour. During this operation, the system will be at a higher utilization of resources. Hence, it is recommended that you set the parameter to smaller intervals that are less than one-hour. You can increase it if the new registrations are not that frequent. The example value can be set in the following path on the Pulse API server: Management Suite Management Suite Database Connection Pool Heartbeat Interval = 60 mins /opt/iot-api/config/application.yml It is recommended to keep this parameter at a default of 100. However, in cases where the activities lead to timeouts on the server, it is recommended to change the connection pool to 200 (maximum pool size below), and not more than that. To change the connection pool settings, add the following line in the path specified below by updating the size and credentials: <add name="airwatchdevelopment" connectionstring="data Source=localhost;Initial Catalog=AirWatchDev;User ID=<<DB Admin>>;Password=<<Admin Pwd>>; Max Pool Size=<<Size>>;" providername="system.data.sqlclient "/> <installation folder>\airwatch<version>\websites\ WanderingWiFi.AirWatch.Servic es\web.config It has been observed that increased frequency of beaconing to be synched with the server, leads to request timeouts on the client side. This is because 11
Management Suite Operations Management Suite Data Sample Interval = 120 mins Data Transmit Interval = 120 mins Action History = 10 days Deleted Objects = 24 Hours Deletion Scheduling Interval = 24 Hours Object History = 30 Days Session Timeout = 30 Minutes Symptom/Alerts = 15 Days Time Series Data Retention = 6 Months Additional Time Series Retention = 36 Months of the result of exhausted database connection pools on the server. It is recommended that you contain their frequency as recommended here. Groups & Settings -> All Settings ->s & Users -> Linux -> Agent Settings It has been observed that increased frequency of data samples to be synched with the server, leads to request timeouts on the client side. This is because of the result of exhausted database connection pools on the server. It is recommended that you contain their frequency as recommended here. Groups & Settings -> All Settings ->s & Users -> Linux -> Agent Settings It is recommended that you configure these parameters for a maximum of six months of data retention. Sizing is computed on that basis. These settings can be updated under Administration > Management > Global Settings. It is also recommended that customers keep the symptom definitions appropriately, and do not generate a high volume of symptoms or alerts. Ideally, symptoms are defined for abnormal events which must not be more than 20-30% of the generated metrics. If the symptoms occur frequently, you may need to clean up cancelled alerts or alarms manually, to keep the storage utilization in control. 12
8. Recommended Configuration Parameters for Various Sizes Even though it is recommended that the system be contained at 15000 managed objects, there may be a need to go beyond these numbers. VMware does not claim that the system will support higher loads, however, it is recommended that the following settings can yield good results. management User Package Parameters Suite Size Beacon Data Interval Sampler no_of_edge_system_in_thousands (mins) Interval (mins) 5000 30 60 1 (for 1000 GWs) 10000 45 90 2 (for 2000 GWs) 15000 60 120 3 (for 3000 GWs) 20000 120 240 4 (for 4000 GWs) You are expected to write packages with some guidelines, for considering the number of retries, delays between retries for an MQTT connection, and registration of entities. VMware recommends that you refer to the packages in the following location while designing packages for registration of the gateway, IoT devices, and the data flow of properties and metrics. The path on the Liota GitHub where these packages exist, is given below: Liota GitHub Repository https://github.com/vmware/liota/tree/master/packages/user_packages Gateway Registration Package o iotcc_mqtt.py IoT s and Metrics Package o iotcc_mqtt_device.py Gateway Metrics Package o iotcc_mqtt_edge_system_stats.py 13