Performance and Scalability

Transcription

1 Measuring Critical Manufacturing cmnavigo: Affordable MES Performance and Scalability for Time-Critical Industry Environments Critical Manufacturing, 2015

2 Summary Proving Performance and Scalability Time-Critical Environments The Hardware Configuration The test material flow scenario Results How Scaling Impacts CPU utilization of database servers Behind the results cmnavigo Scales for time-critical needs

3 COURTESY OF NANIUM S.A. Before selecting a manufacturing execution system (MES) for a complex process, the company must take time to carefully describe their data flow needs both in current terms and in future scale-up scenarios. This is a matter of understanding the number of materials and sub-materials processed that need to be tracked, the number of process steps involved, the number of pieces of equipment used, how many parameters are tracked, and speed required to keep production rolling. Those desiring an enterprise standard for MES should calculate this for the most demanding of their facilities. Critical Manufacturing s cmnavigo was designed for the most time-critical and complex of industries. New performance benchmark tests show that even on relatively inexpensive standard hardware, cmnavigo can manage well over 400 transactions per second. These tests also show that adding application servers delivers linear performance improvements, indicating that and further increases in performance are available by adding application servers and by scaling up the database server. Many MES are not capable of that level of performance. If your company may have these needs, you need to know the system you select will support you as your needs grow and change. 2

4 Time-Critical Environments The ability of an MES system to perform transactions below a 1/100 of a second may seem beyond any real requirement for some industries, but for others, it could offer new heights of performance and scalability. Manufacturing environments with intrinsic material, process and flow complexity can render many MES systems helpless in high volume production applications. Equipment state change monitoring and material flow tracking are two functions that demonstrate this fact. In equipment state change monitoring, automated data or equipment state change information collected from production or measurement equipment might result in the need for such high resolution MES capability. If a given piece of production equipment changes state every second -- changing for example from productive/standby to standby/productive -- and there are 200 pieces of equipment in the production line, transaction volume would be 200 state changes per second. And if every half-second that equipment also sends engineering parameters for recording and associating with the material in process, the number of transactions per second increases even further. But at some point the process is going to hit the wall. If we can anticipate when the volume might begin to take its toll, we can manage that scale-up more effectively. Similarly, a material flow-related scenario involving a hierarchy of sub-materials could benefit from ability to process high resolution transactions. Moving processed material through a high volume production line already results in high transaction volume, but when those materials also contain sub-materials requiring individual traceability, the pressure on the MES system intensifies. A semiconductor wafer composed of thousands of dies or an electronic board composed of hundreds of subcomponents that must all be tracked, are examples of processing tasks that could bring a conventional MES system to its knees. Here too is the need to anticipate and manage thresholds. Both the equipment and material flow examples mentioned above require the ability of the system to scale to accommodate the growth of the underlying production volume, automation, data collection or traceability requirements. Another important scalability consideration is the number of pieces of equipment and number of parameters collected per process step. Project teams must review any MES purchase carefully, seeking demonstrable proof that the systems they are considering have significant processing granularity and can scale easily. When selecting a system for applications that can potentially expand, manufacturers need assurance that their system can cope with all flow changes, automation levels, complexity or volume growth. They need to know that they can add hardware nodes as needed, without risking production downtime or other negative impacts to an urgent scale-up. They also need assurance that the scale-up will not require incurring exorbitant additional expenses. 3

5 Proving Performance and Scalability Many studies attempting to prove MES performance and scalability have been conducted in extremely highperformance laboratory environments. While these do prove the attainability of achieved values, the hardware sets used are not typically characteristic of the more modest technology used in most plants. To prove the real-world scalability and performance of its software, Critical Manufacturing conducted studies using hardware more typical of that found in a working factory. Each transaction is not a simple write or read from the database, but a WIP (material movement) transaction, which minimum includes a database write action and usually several database reads. 4

6 The Hardware Configuration While Critical Manufacturing s MES can be deployed in several variations depending on the load and scalability requirements, starting from single server, this test deployed one database server for the online database, one for Operation Data Storage (ODS), two physical and two virtual applications servers, and several client machines to perform the requests, all across a one gigabit network (Figure 1). CLIENT APPLICATION SERVER DATABASE SERVER STORAGE HBA1 HBA2 SP1 HBA3 HBA4 SP2 Internet Browser Network Load Balancing Microsoft Clustering Services FIBRE CHANNEL SWITCHES STORAGE AREA NETWORK Figure 1. HBA - Host Bus Adapter SP - Storage Processor. 5

7 The configuration shown in Figure 2 can be scaled to manufacturing needs easily, at any time. The application server farm can scale up with more powerful hardware (CPU or memory) or scale out with the addition of servers via Network Load Balancing. The database can be configured in highavailability mode either through failover clustering or mirroring. For scalability, the database server can be scaled-up (e.g. by upgrading the CPU, memory or storage) and the various databases, including online, operational data storage and data warehouses, can be spread across different servers. None of the hardware used in the test was state of the art and the total estimated cost for the entire hardware configuration was below $50,000 USD. Figure 2. 6

8 The test material flow scenario To demonstrate the scalability of this solution multiple test clients were started concurrently and programmed to make consecutive calls, performing the following loop: Create a material Create and attach 50 submaterials Track the material with 50 sub-materials through a flow with 1000 steps, 95 percent of which are regular steps with dispatch, track-in, track-out and move-next, and five percent are pass through, requiring only move-next Sub-material track-in and track-out in 50 percent of the steps Collect data before track-in in 30 percent of the steps Collect data before track-out in 60 percent of the steps; each data collection contains one parameter with five readings So by performing the above mentioned loop, each test client performs a combination of the following transaction set: Create a material Attach material Dispatch material Track-in material (including resource state change and in some cases data collection) Track-out material (including resource state change and in some cases data collection) Move-next material Split and merge in 70% of the steps Record loss in 30% of the steps Hold and release in 30% of the steps Storage and retrieve in 30% of the steps Then, to capture the number of sustained transactions testers varied the number of application servers from one to four and measured the API transactional writes to each. Some of the API transactions had more than one sub-api transaction and the more complex transactions made up to 10 database calls. 7

9 Results Figure 3 shows how the addition of successive servers impacted the maximum sustained transactions per second, growing from transactions per second with a single server, to transactions per second when the fourth server was added TRANSACTIONS PER SECOND Number of Application Servers Figure 3. Table One shows the API latency results, in seconds, for the top 10 most called services, obtained with a two minute time slice taken #4. Note that call times vary because calls are occurring simultaneously and that there is known warm-up effect (caused by initial caching and the normal behavior of the.net framework), which affects the maximum durations of the initial transactions as observed below, but diminishes to almost nothing as the number of consecutive calls increases. The fact that the increase in Transactions per Second is essentially linear with the addition of servers indicates that there are no inefficiencies or bottlenecks. A flat zone would mean that something is limiting scalability and a cause must be determined. SERVICE # CALLS MIN. DURATION (S) MAX. DURATION (S) AVG. DURATION (S) STDEV. DURATION (S) Dispatch Material Complex Track In Material Complex Track Out Material Move Materials To Next Step Perform Immediate Data Collection Split Material Merge Materials Store Material Retrieve Material Hold Material Table 1. 8

10 How Scaling Impacts CPU utilization of database servers Figures 4 through 8 show another extremely important aspect of scalability: the CPU utilization of both the application servers and the online database server. If the CPUs are being used properly, we would expect utilization to be nearly 100 percent. As shown in the charts, CPU utilization of the database server increases to an average of around 60% with the addition of four servers, leaving room for continued scaling and enabling us to predict where the next bottleneck will occur. Figure 4. CPU utilization - Performance Test Run With 1 Application Server. Figure 5. CPU utilization - Performance Test Run with 2 Application Servers. Figure 6. CPU utilization - Performance Test Run with 3 Application Servers. Because all application servers will use the same database, after addition of more application servers, in this case two or three, the database server will eventually become the bottleneck and utilization charts will help determine whether it is better to accommodate the increased load by adding more application servers, more memory, more CPU power or other techniques. Figure 7. CPU utilization - Performance Test Run with 4 Application Servers. 9

11 Behind the results Achieving such results from budget-friendly hardware is a combination of unique intervention at both the serialization and database levels. Serialization is the formatting of data structures for storage or transmission across a network. Serialization and the related deserialization are CPU intensive, especially when verbose XML structures are involved. So the strategy is to reduce the message footprint, while still carrying the necessary information. The cmnavigo architecture enables this by locating all system functionality in a single foundation layer, which allows all functional modules to benefit automatically from any optimization. At the database level, tradeoffs are between concurrency and consistency. These must be evaluated in the context of each company s business logic and performance testing under high stress load. This is done essentially at the tempdb system database, where the database SQL Server stores global or local temporary tables, temporary stored procedures, table variables, or cursors. Optimization options include: reducing the contention time by separating tempdb into a separated disk; and splitting it in different files (1 per core of the CPU). It is worth noting that the online and operational data storage databases are also split and therefore some other optimizations must be done at the replication level, to cope with the volume of data in these stress tests. And as dramatic as these results are, this is by no means the upper limit. Additional performance can be reached by continually reducing message footprint and contention times, for example, by strategically portioning the most frequently used data tables. 10

12 cmnavigo Scales for time-critical needs These results lead us to the following three conclusions: 1. That Critical Manufacturing s MES systems can cope with an unprecedented number of concurrent transactions, reaching a value of much more than 400 transactions/sec, with an average hardware set consisting of two physical and two virtual applications servers. 2. The almost linear growth of the number of transactions per second suggests the diminishing returns area has not yet been hit. 3. The marked increase in CPU utilization with the addition of servers seems to confirm that diminishing returns have not yet set in and suggests that the capability can be still significantly increased through scale-up the database server. So before selecting an MES system for a complex process, manufacturers should gauge to their best ability the potential of their process to grow in volume and their potential need to scale up quickly. They should consider expected increase in number of materials processed and steps involved. If they can predict how these will evolve, than can achieve an almost linear increase in required transactions. About Critical Manufacturing Critical Manufacturing provides manufacturers in highlycomplex environments with a modular, scalable manufacturing execution and intelligence system that enables users to flexibly address market demands, increase efficiency and bolster reliability across the supply chain while lowering TCO. The company is part of the Critical Group, a private group of companies founded in 1998 to provide solutions for mission and business critical information systems. HQ: Porto, Portugal t: Austin, TX t: Dresden, Germany t: +49 (0) Suzhou, China t: