Enterprise Strategy Group Getting to the bigger truth. ESG Lab Validation MiTek Sapphire Build Scalable Software for Home Building Management By Brian Garrett, Vice President, ESG Lab April 2017 This ESG Lab Report was commissioned by MiTek Industries and is distributed under license from ESG.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 2 Contents Abstract... 3 The Challenges... 3 The Solution: Sapphire Build... 4 ESG Lab Validation... 5 Getting Started... 5 Performance and Scalability... 6 Scaling Performance with Multiple Databases... 7 Scaling Performance with More Server Resources... 10 Price/Performance Analysis... 11 The Bigger Truth... 13 Appendix... 14 ESG Lab Reports The goal of ESG Lab reports is to educate IT professionals about data center technology products for companies of all types and sizes. ESG Lab reports are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objective is to go over some of the more valuable feature/functions of products, show how they can be used to solve real customer problems and identify any areas needing improvement. ESG Lab's expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 3 Abstract ESG Lab validated testing of MiTek Sapphire Build, a software package that manages home building projects for developers. The goal of the testing was to prove that the recently enhanced scale-out architecture of the back-end transaction database can be used to support the sale and construction of more than 30,000 new homes per year. The Challenges A growing number of organizations are undergoing digital transformation initiatives with a goal of increasing employee productivity, customer satisfaction, and profitability. The real estate and home construction industries are no different. Emerging technology trends including mobility, cloud, and data analytics are being used to reduce the cost and complexity associated with coordinating the many people and processes involved in selling, configuring, building, and servicing new home construction. ESG s annual survey of IT purchasing decision makers indicates that the two most-cited desired business outcomes for IT investments are reducing costs and increasing productivity (see Figure 1). 1 A modernized building management IT infrastructure can be used to address those challenges and more, including increasing revenue and acquiring new customers. Figure 1. Highest Priority Business Outcomes over the Next 12 Months Which of the following business outcomes would you consider to be your organization s highest priorities over the next 12 months? (Percent of respondents, N=641, five responses accepted) Reducing costs Increasing productivity Improving information security 40% 39% 43% Increasing revenue Acquiring new customers Ensuring regulatory compliance 29% 31% 33% Retaining existing customers 24% Attracting, developing, and retaining employees Developing new products Entering new markets 20% 20% 19% Don't know 2% 1 Source: ESG Research Report, 2017 IT Spending Intentions Survey, March 2017.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 4 The Solution: Sapphire Build Sapphire Build Enterprise Management Suite is an Internet-enabled software solution that provides collaborative management of the real estate development and home building process. Sapphire Build enables home builders to integrate their business operations into a single, unified system while providing a mobile device-enabled web portal for marketing, sales, construction operations, product development, and customer support. Sapphire Build uses a workflowbased approach, which enables companies to implement business processes for a structured approach to managing product information, product changes, vendor management, scheduling, finances, and customer information, all while maintaining flexibility and customization. An overview of the Sapphire Build architecture is depicted in Figure 2. Users including customers, sales professionals, subcontractors, and mangers access the application from a web browser or mobile device. A RESTful API is supported for integration with third-party software packages (e.g., accounting and CRM). Job requests are load balanced across the network and forwarded to one or more application servers. The front-end application servers are connected to one or more back-end Microsoft SQL Server instances including a transaction database, a temporary database, and a data warehouse. Support for multiple SQL transaction servers, making its debut in the latest release of Sapphire Build, splits the transaction database load across up to eight SQL Server instances. Figure 2. Sapphire Build Overview The Sapphire Build architecture was developed using industry-standard technologies and programming languages including ASP.NET, C#, and SQL. Sapphire Build servers (app and database) can be deployed on-premises on bare metal or virtual servers (e.g., VMware) and in the public cloud (e.g., AWS EC2). As summarized toward the bottom of Figure 2, Sapphire Build centralizes and simplifies the end-to-end workflow associated with new home construction: from marketing and sales through design, orders, production, vendor management, field build, accounting, and customer service.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 5 ESG Lab Validation ESG Lab validated the recently enhanced back-end performance scalability of Sapphire Build with a real-world customer database deployed in a public cloud. Performance scalability was measured as up to eight back-end Microsoft SQL Server instances were deployed in parallel. The performance and price/performance of a single SQL Server instance running in different sizes of Amazon AWS EC2 virtual machine instances were also evaluated. Getting Started ESG Lab testing began with a quick tour of the Sapphire Build user interface with a real-world database that was harvested from an organization that is building new homes at a rate of approximately 1,500 per year. The web and mobile user interfaces were used to get a feel for the value of a centralized system of record that automates and simplifies the process of selling, configuring, building, and servicing the construction of new homes (see Figure 3). Starting in the role of a sales representative, we explored lot and home model options. Prepopulated rules and dependencies were exercised as an optional bedroom upgrade was explored. Longer running jobs with more dependencies were executed as well, including creating the production data for a new home start. The mobile user interface was used to examine a project timeline and subcontractor billing status. Figure 3. The Sapphire Build User Interface Why This Matters The traditional approach of customizing existing accounting, ERP, or CRM software to meet the complex needs of home builders is complicated and time-consuming compared with deploying purpose-built building management software. ESG Lab has validated that Sapphire Build was purpose built with a goal of simplifying and centralizing the role-based management of the home building process. Web and mobile access to a singular and consistent system of record increases employee productivity, which saves time and money.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 6 Performance and Scalability The Sapphire Build architecture can be deployed in a scale-out topology to support an increasing number of concurrent users and home building processes. Multiple front-end application servers are configured behind a load balancer to scale the number of concurrent web connections and application servers running in parallel. Front-end application servers communicate with a back-end transactional instance through a virtual private network connection. Previously, one SQL Server instance was used for all data. The latest release of Sapphire Build has increased the number of supported back-end transaction SQL server instances from one to eight with a goal of supporting more concurrent users and building processes. Sapphire Build scales back-end database performance with an industry-standard technique that is generally known as datadependent routing. With data-dependent routing, the data is partitioned among multiple database instances, and either the application or a middleware service routes queries to the correct database. As shown in Figure 4, the business logic layer within a Sapphire Build application server routes database requests to the correct database. In this case, the transaction data associated with 4,500 homes is partitioned across three database servers. Each server contains only its portion of the transaction data as well as a complete copy of the master database to ensure speedy access. When a transaction request comes in, Sapphire Build s Business Logic Layer (BLL) determines which SQL server it should be routed to, and once the Data Access Layer has verified that access is permitted, the BLL directs the transaction to that server. When a transaction requires a change to the master data (e.g., a pricing update), the master data update is replicated to each of the other database servers. When transactions updates are committed, they are posted to the data warehouse in the background. Note that with this architecture, a home builder can deploy separate database instances for different regions or subsidiaries. Figure 4. Scaling Sapphire Build Throughput with Data-dependent Routing
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 7 Scaling Performance with Multiple Databases Sapphire Build back-end database scalability was tested with a real-world database from a customer who is building approximately 1,500 homes a year. As shown in Figure 5, the open source Apache Jmeter utility was used as a load generator and performance statistics collector. 2 The load generator forwarded job requests to one through eight Amazon AWS c4.xlarge application servers and Microsoft SQL Server database instances. Note that the web and mobile client access layers were not included in the scope of this test with a goal of isolating back-end performance and quantifying the performance improvements that can be achieved with additional database instances. More specifics about the test bed are listed in the Appendix. Figure 5. The ESG Lab Test Bed A series of transactions that simulates the process of initiating a home start was tested beginning with the acquisition of a new customer and ending with the sale and creation of production data for a new home start: Load the homepage. Create a new customer. Create three sales worksheets. Create a sales order from one of the three worksheets. Create a job associated with the design and sale of the home. Create a complete set of production data and purchase orders for the new home. It should be noted the jobs that were tested included a mix of response time-sensitive workloads (e.g., creating a new customer) and long running batch updates (e.g., creating production data) that place the most stress on the back-end database. The jmeter load generator executed the jobs in a loop with a goal of testing 60 jobs per DB Server as the number of DB Servers scaled from one to eight. The results are summarized in Figure 6 and Table. 2 http://jmeter.apache.org/
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 8 Figure 6. Scaling Sapphire Build Throughput with Multiple Back-end Database Servers 500 Scaling Sapphire Build Throughput with Multiple Back-end Database Servers (1 to 8 SQL Server DBs, AWS c4.xlarge VMs, Sapphire Build v2017.1.27 ) 10 Throughput (Jobs per hour) 400 300 200 100 8 6 4 2 Response Time (Seconds per new customer transaction) 0 1 2 4 7 8 Number of Database Servers 0 Table 1. Scaling Sapphire Build with Multiple Back-end Database Servers Concurrent SQL Server Instances One Two Scalability Four Scalability Eight Scalability Jobs per hour 56 111 99.11% 218 97.32% 443 98.90% Create customer (msec) 831 839 98.16% 836 96.76% 899 91.45% Create home (msec) 782 763 100.61% 742 101.98% 771 92.75% Create production data 175,322 174,986 100.80% 176,098 101.53% 173,203 93.89% Job (msec) CPU utilization (avg.) 68.53% 67.92% 101.72% 67.51% 103.07% 66.79% 96.34% Avg. SQL Server transactions per 230 440 95.69% 864 94.02% 1,786 97.17% second Max SQL Server transactions per 673 1,261 93.71% 2,753 102.30% 4,801 89.18% second Overall scalability (avg.) 98.54% 99.57% 94.24%
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 9 What the Numbers Mean The blue bars in Figure 6 depict the aggregate performance of Sapphire Build in jobs per hour as additional database servers were added and additional jobs were executed in parallel. Aggregate performance scaled in a nearly perfect linear fashion as the number of database servers scaled from one to eight. As you can see in Table, two servers scaled to within 2% of perfect linearity (98.54%) and eight scaled to 94.24%. The green dotted line depicts the job runtime in milliseconds for a response time-sensitive create customer job. As you can see in the table, create customer jobs completed in less than a second during each of the tests (0.831 to 0.899 seconds). Note that the most database resource-intensive create production data job took the longest to run (175,322 milliseconds, which is about 2.9 minutes). The database servers were instrumented to capture CPU utilization and SQL Server transactions per minute during each of the test runs. Those results are listed in Table and depicted in Figure 7 and Figure 8. Figure 7 shows the real-time CPU utilization that was captured during four test runs as the number of database servers was scaled from one to eight. A consistent range of CPU utilization between about 60% and 75% was measured during the single database test. Other than a couple of dips below 50%, the CPU utilization was nearly identical during the two, four, and eight server tests. These results clearly indicate how Sapphire Build uses data-dependent routing to leverage the additional CPU horsepower provided by each of the database servers running in parallel. Figure 7. CPU Utilization 100 CPU Utilization (1 to 8 SQL Server DBs, AWS C4.xlarge VMs, Saphire Build v2017.1.27) 90 80 CPU Utilization (%) 70 60 50 40 30 20 10 0 One DB Two DBs Four DBs Eight DBs DB1 DB2 DB3 DB4 DB5 DB6 DB7 DB8 Having proved that job performance scales in a nearly perfect linear fashion and that CPU utilization remains balanced as back-end database servers are added, ESG Lab examined SQL Server statistics to better understand what s happening
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 10 down at the database level. As shown below in Figure 8, the aggregate number of SQL Server transactions per second scaled in a nearly linear fashion as database servers were added. SQL Server transactions per second is a measure of the number of database transactions that are being processed by each database server. This counter, which is well known and tracked by most SQL Server database administrators, is a good indicator of how much activity is being processed by each SQL Server instance. A higher number indicates that the more work is being done by SQL Server. An impressive peak rate of 4,801 transactions per second (280,060 transactions per minute) was measured during the eight-server test. Figure 8. SQL Server Transactions per Second 6,000 SQL Server Transactions per Second (1 to 8 SQL Server DBs, AWS C4.xlarge VMs, Saphire Build v2017.1.27) SQL Transactions Per Second 5,000 4,000 3,000 2,000 1,000 0 1 2 4 7 8 Number of Database Servers Max Average Scaling Performance with More Server Resources Next, ESG Lab examined the performance boost that can be achieved with more server resources. The same series of jobs was run against the same customer database that was used for the multi-database scalability test. A single application server and single database server were tested with AWS EC2 instances of different sizes. The first test was run with an AWS virtual machine that was the same size as the one tested during the multi-database test (c4.xlarge). Virtual machine resources were increased during subsequent tests (c4.2xlarge and c4.4xlarge). The number of jobs that were run serially was increased from 60 to 480 and the elapsed runtime was used to calculate the number of jobs per hour that a single database server can support. The results are summarized in Figure 9.
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 11 Figure 9. Scaling Sapphire Build Throughput with More Server Resources 140 120 100 80 60 40 20 Scaling Sapphire Build Throughput with More Server Resources (One SQL Server DB, AWS c4.xl vs c4.2xl and c4.4xl VMs, Sapphire Build version 2017.1.27 ) 0 60 120 240 480 c4.xlarge c4.2xlarge c4.4xlarge What the Numbers Mean This graph, which is generally known in the industry as a response-time throughput curve, provides a picture of how well an IT solution performs under duress. The best performing configuration is represented by the lowest curve (executes fastest) that extends the farthest to the right (does more work). As traffic increases along the X-axis, runtimes increase on the Y-axis. When the server becomes overloaded, the application has a hard time keeping up as queues start to fill up, runtimes elongate, and performance begins to feel sluggish to end-users. As an example, note how runtimes increased dramatically for the smallest resource-constrained c4.xlarge configuration and hit a resource limit at around 120 jobs. Upgrading from a c4.xlarge to a c4.2xlarge VM delivered a performance boost of nearly 400% at lower levels of stress (60 jobs), and nearly 600% at 120 jobs. The largest and most expensive c4.4xlarge virtual machine performed the best (480 jobs in 41 minutes). Price/Performance Analysis Finally, ESG Lab analyzed database server price/performance using publicly available AWS EC2 pricing. 3 The AWS c4.2xlarge virtual machine has double the CPU and RAM and costs twice as much as a c4.xlarge virtual machine. The c4.4xlarge virtual machine doubles resources again (16 vcpu and 30 GB of RAM) and costs twice as much as the c4.2xlarge per hour 3 AWS EC2 pricing as of March 13, 2017, as published at https://aws.amazon.com/ec2/pricing/on-demand/
Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 12 ($0.862/hour). An upgrade from c4.xlarge to c4.2xlarge virtual machines increased single server throughput by nearly four times (394%) during the 60 job test and nearly eight times during the 480 job test (7.6x). Taken together, those results indicate that an upgrade from xlarge to 2xlarge delivers roughly two to four times better price/performance. As you can see in Figure 10, upgrading again to a c4.4xlarge virtual machine doesn t provide the same bang for the buck. Note that for each of the tests, the c4.2xlarge configuration depicted in orange had the lowest cost per jobs per hour. Figure 10. Price Performance as SQL VMs Increase in Size 0.005 Sapphire Build on AWS Price Performance Analysis (c4.xlarge, c4.2xlarge, c4.4xlarge AWS VMs; Sapphire Build v2017.1.27) Price Performance (cost per jobs per hour) 0.004 0.003 0.002 0.001 0 60 120 240 480 Jobs c4.xlarge c4.2xlarge c4.4xlarge Why This Matters A digital transformation initiative that leverages a single system of record for home builders must be able to scale to meet the needs of the business. If the solution can t scale to meet the number of homes being built annually, then multiple solutions must be deployed and managed, which adds complexity, risk, and cost. The legacy Sapphire Build architecture with a single back-end transaction database has scaled to meet the needs of organizations supporting up to 1,500 new home starts a year. ESG Lab has validated that recently introduced support for up to eight back-end database servers has increased that number to 12,000 when deployed on Amazon ECS c4.xlarge servers. Testing on a larger c4.2xlarge virtual server provided a cost-effective performance boost of four to six times. Based on these results, and the continuously improving horsepower of industry-standard server technology, ESG Lab is confident that Sapphire Build can support up to 48,000 new home starts per year. Lifting the current limit of eight SQL Servers, which is architecturally possible but not yet implemented or qualified, could increase that number to 100,000 and beyond.
The Bigger Truth Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 13 A growing number of organizations are embracing digital transformation initiatives that leverage powerful new technologies including data analytics, cloud computing, and mobility. ESG s 2017 IT spending intentions survey indicates that the top five most important IT initiatives over the course of 2017 include the use of data analytics for real-time business and customer insight, the use of public cloud for applications and infrastructure, and providing employees and customers with mobile access to applications and IT resources. 4 Traditionally, home building management software solutions are either highly customized ERP, CRM, or accounting software packages or a complex multi-vendor basket of applications that require customization to make the different pieces to work together. Extensive customization and integration adds complexity, time, and risk. Advanced capabilities including mobile device access and public cloud support are often missing. ESG Lab has confirmed that Sapphire Build is a purpose-built software solution for home builders that can be deployed onpremises or in a public cloud. It delivers deep analytic insight into all phases of the building process. Accessed via a web browser or mobile device, Sapphire Build provides a single system of record that can be shared by customers, employees, subcontractors, vendors, managers, and suppliers. It is important to note that the benchmark results that were presented in this report were collected under controlled conditions with a focus on back-end database performance scalability. While a real-world customer database and a prototypical job workflow that included one of the most back-end database-intensive jobs was tested, a more real-world test would include a wider mix of jobs as seen in production. Client access via a web browser or mobile device was not tested, which eliminated network congestion and latency as a potential performance scalability issue. That said, ESG Lab is pleased to report that the new scale-out database architecture of Sapphire Build can scale to manage tens of thousands of new home projects annually being accessed by hundreds of thousands of customers, employees, contractors, suppliers, and vendors. ESG Lab believes that home-building companies that wish to run their businesses, large or small, on a purpose-built application that is sold and supported by a single vendor, and that offers consistent, predictable performance should take a close look at MiTek Sapphire Build. 4 Source: ESG Research Report, 2017 IT Spending Intentions Survey, March 2017.
Appendix Lab Validation: MiTek Sapphire Build: Scalable Software for Home Building Management 14 Table 2. ESG Lab Test Bed Software Version Windows Server 2012 R2, Build 9600 MS SQL SQL Server Developer 64-bit, 2014 SP1 (12.0.4100.1) IIS 8 JMeter 2.13 Sapphire Build 2017.1.27 Virtual Machine Amazon Web Services ECS: c4.xlarge Amazon Web Services ECS: c4.2xlarge Amazon Web Services ECS: c4.4xlarge Specification 4 vcpus, 7.5GB RAM, EBS storage, 750 Mbps 8 vcpus, 15GB RAM, EBS storage, 1,000 Mbps 16 vcpus, 30GB RAM, EBS Storage, 2,000 Mbps All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at 508.482.0188. Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community. www.esg-global.com 2017 by The Enterprise contact@esg-global.com Strategy Group, Inc. All Rights Reserved. P. 508.482.0188