Five Hard-Won Lessons In Performance, Load, and Reliability Testing

Transcription

1 Five Hard-Won Lessons In Performance, Load, and Reliability Testing Ferrari 360 race from Cosmo

2 Why Does PLR Testing Matter? Why do performance, load, and reliability testing? Inadequate performance can lead to user dissatisfaction, user errors, and lost business System failure under load often means loss of system stem functionality when the need is highest Reliability problems damage your reputation and your business Perhaps the more important question PLR testing is, Are you sure that there are not critical risks to your business related to performance, load, or reliability? Experience is a dear teacher, but some will learn from no other. Ben Franklin, 1700s-era wit and US Revolutionary Copyright (c) RBCS Page 2

3 Great tmoments in PLR Risks If 10% of an e-commerce site s s customers abandon their shopping cart due to sluggish performance, how long will that company stay in business? One bank estimated its average cost of field failure at over 100,000 often related to PLR The Victoria s Secret 1999 Internet holiday lingerie show good concept; terrible execution mistake: insufficient performance, load, and reliability testing On introduction, a new 999 (emergency) phone service in the UK suffered from reliability (and other) problems that resulted in late ambulance arrivals and consequent fatalities Copyright (c) RBCS Page 3

4 I See the Risk So What Do I Do? I can t tell you everything you need to know no one can but I can give you some ideas The following pages will illustrate five lessons I ve learned in PLR testing Case studies show both success and failure I ll indicate tips for success with I ll indicate warnings for common mistakes with All figures and case studies are real (some details omitted) thanks to the kind permission i of my clients and my associates on the projects Copyright (c) RBCS Page 4

5 Configure performance, load, and reliability test environments to resemble production Generate loads and transactions that model various real-world scenarios Test the tests with models and simulations and vice versa Invest in the right tools but don t waste money Start modeling, simulation, and testing during design and continue throughout the lifecycle! These are hard-won lessons, learned through personal experience with successful projects and expensive failures Copyright (c) RBCS Page 5

6 Lesson 1: Realistic Environments Software engineering is not yet like other engineering fields Civil engineering g example: Bridges Aeronautical engineering: Scale models Software engineering counterexample: Binary trees, normalized relational databases It can be difficult to extrapolate results from smaller environments under scaled-down d loads into larger environments Copyright (c) RBCS Page 6

7 Unrealistic Environment One client s systems: Support large, diverse networks Gather, integrate, and control complex data Tested in a minimal environment For a large customer with a complex network and data, one query that was to run overnight, each night, took 25 hours! For systems that deal with large data-sets, don t forget that the data is part of the test environment Copyright (c) RBCS Page 7

8 Realistic Environment For a banking application, we built a test environment that mimicked the production environment Differences were limited to the bandwidth between the call center (where the clients were) and the data center (where the servers were) We used testing and modeling to ensure that these differences would not affect the PLR results in the production environment Understand all differences between actual and test environments and how they might affect your results Copyright (c) RBCS Page 8

9 Test and Production Environments Tests were run in steps where each step added 20 users to the load, looking for knees in the performance The only differences between the test and production environments were in the bandwidth and throughput limits on this wide-area network that tied the call center to the data center. Realistic loads put into the system from the call center side of the interface Copyright (c) RBCS Page 9

10 Lesson 2: Real-World ld Loads A realistic test environment makes possible the next level of realistic PLR testing: realistic transactions and loads Load scenarios should include Regular events (like backups) Seasonal events (e.g., holiday shopping, year-end end closing, etc.) Different classes of users (power, novice, etc.) Growth for the future External factors (such as WAN or Internet load) Try tip over tests : Increase load until the system fails to check for graceful degradation and unexpected bottlenecks Copyright (c) RBCS Page 10

11 Light, Unrepresentative ti Loads For an interactive voice response server, the developers tested subsystem performance by generating load using half of the telephony cards The system load imposed by the load generators was below that of the telephony subsystems under test Warnings from test professionals about the meaninglessness of such tests were ignored The tests passed, until PLR testing is a specialized, complex task that requires expertise and study: Careless OJT can kill your project Copyright (c) RBCS Page 11

12 Rp Representative tti Loads We built a load generator that ran on an identical but separate host We loaded all telephony ports The tests failed, revealing project-threatening design problems Crash!!! And when we cranked up the load here Prefer non-intrusive load generators for most PLR testing PLR testing of subsystems in unrealistic system settings yields misleading results Copyright (c) RBCS Page 12

13 When Self-Generated tdload dis Okay MaxConfig A cluster of 31 systems A mix of mainframes and PCs running Unix Load generators Simple Unix/C programs Used files, interprocess communication, process migration, and other cross-network activities Created high, random loads for up to 48 hours Intrusive (self-generated) load works for load, reliability tests Copyright (c) RBCS Page 13

14 Lesson 3: Test the Tests and dthe Mdl Models If realism of environment and load is important, how can we be sure we got it right before we go live or ship? Models can help us test our tests The tests also test the models A model which is validated through testing is useful for predicting future performance Spreadsheets are good for initial PLR modeling and design Dynamic simulations allow for more what-if scenarios and are useful during design, implementation, and testing Copyright (c) RBCS Page 14

15 Test, but tverify For an Internet project, we tested at 25K, 40K, and 50K We used a production environment and realistic load generator Functional testing of the devices during PLR tests gave us a sense of user experience under load We compared our test results against the simulation results Copyright (c) RBCS Page 15

16 Double-Checking An extract from our PLR results presentation Simulation data for IMAP servers 45% server utilization at 25K users 75% server utilization (saturation) at 40K users Worst-case snapshots (25K users) Server CPU Idle MTA1 68% MTA2 79% Maildb1 67% Maildb2 89% IMAP1 59% IMAP2 55% Test data supports simulation 25K users Copyright (c) RBCS Page 16

17 AdM And More Double-Checking We did a detailed analysis of performance, too Simulation Observed Transaction Server Frequency Time (ms) Time (ms) Connect IMAP 1 per connect Banner IMAP 1 per connect Authorize IMAP 1 per connect Select IMAP 1 per connect Fetch IMAP 1 per connect , Be careful to design your tests and models so that you can compare the results. Copyright (c) RBCS Page 17

18 Lesson 4: Invest in Tools Don t Splurge Tools are necessary for PLR testing There are many types of suitable tools Commercial Open-source Custom-developed For complex test situations, you may well need a combination of the above Check and sourceforge.org for open-source tools Don t overload on free tools; even free tools have costs Copyright (c) RBCS Page 18

19 Money Wasted on Tools One client s all-too-often-repeated often test tool tale They selected a commercial tool based solely on a salesperson s representations that it would work for them They hired a consultant to automate all their tests for them, without any transition plan to train staff, maintain tests They tried to automate at an unstable interface (in this case, the GUI) The consultant left, and the day after the tool and the tests were found to be unusable by the staff Total cost? About $500,000! First, design your PLR test system (cases, data, automation strategy, etc.), then identify the specific tool requirements for your system, and only then go shopping for tools Copyright (c) RBCS Page 19

20 Money and dtime Well-Spent For a complex system with a wide-area network of interactive voice response servers connected to a large call center, we used the following tools QA Partner (now Silk) to drive the call center applications through the graphical user interface Custom-built load generators to drive the interactive voice response server applications Custom-built middleware to tie the two sides together and coordinate the testing end-to-end Free tools such as TCL as building blocks for our test components Total cost? Under $500,000 Let s see how Copyright (c) RBCS Page 20

21 Architecture t of the System Under Test ACD PBX Telephone N x T1 CSA Workstation ACD IVR City IVR Telephone CT-Connect Server CM Workstation CSA-Server WAN CM Server Pub/Su Telephone CSA Workstation Voice Repository Copyright (c) RBCS CityPage 21 IVR

22 Test tsystem Architecture t ACD PBX N x T1 QA Partner drove the call center GUI application CSA Workstation CM Workstation CSA Workstation IVR CT-Connect Server CSA-Server CM Server Voice Repository Pub/Su ACD WAN City IVR Custombuilt load generator drove the IVR application City IVR Custom-built testing middleware coordinated the tests end-to-end Copyright (c) RBCS Page 22

23 Lesson 5: Start tearly and dstay on Course Performance, load, and reliability problems often arise during initial system design A bad unit or subsystem can affect the entire system Interface problems between subsystems can create PLR problems Slow, brittle, unreliable system architectures can t be patched into perfection Use models to identify and fix PLR problems during design Include PLR in unit, subsystem, integration, and system test Throwing more hardware at the problems might not work Copyright (c) RBCS Page 23

24 Failing to Stick with the Program For the interactive voice response server testing mentioned on slides 11 and 12, one senior technical staff member had built a spreadsheet model that predicted many of the problems we found in later testing However, there was no follow-up during design, implementation, or subsystem testing Late discovery of problems during integration and system test lead to patching of the IVR server Copyright (c) RBCS Page 24

25 Problems with Patching PLR Bugs Patching of a problem would improve performance However, the next bottleneck would be discovered during testing Peeling the onion was a slow process which h didn t resolve the performance problems Lines on the axes indicate desired response time and supported load level l Copyright (c) RBCS Page 25

26 PLR: Test tearly, Test toften For the Internet project discussed on slides , we started with a spreadsheet during initial server system design That initial server design was turned into a simulation, which was fine-tuned as detailed design work continued Performance testing of units and subsystems was compared against the simulation Few surprises were found during system testing Iteratively improve the models and the tests to resolve discrepancies between the predicted and observed results Copyright (c) RBCS Page 26

27 Where Does Your PLR Testing Stand? Assess your PLR testing: How well or poorly does your company do in applying each of these lessons? Which common mistakes are you making? Which tips for success could you apply? Avoid expensive, unforeseen field failures by assessing and improving your PLR risk mitigation strategy t throughout h t the lifecyclel Don t let PLR field failures be your first sign you re at risk Direct questions to info@rbcs-us.com Copyright (c) RBCS Page 27

28 Contact trbcs For over a dozen years, RBCS has delivered services in consulting, outsourcing and training for software and hardware testing. ti Employing the industry s most experienced and recognized consultants, RBCS conducts product testing, builds and improves testing groups and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit. Address: RBCS, Inc Beck Road Bulverde, TX USA Phone: +1 (830) Fax: +1 (830) info@rbcs-us.com Web: Five Testing Best Practices Copyright (c) RBCS Page 28