Hochperformante Softwarearchitekturen Planung, Zufall oder Erfahrung? Unser Geheimrezept!

Transcription

1 Hochperformante Softwarearchitekturen Planung, Zufall oder Erfahrung? Unser Geheimrezept! Alexander Buchmann, Wolfgang Strunk Siemens Enterprise Communications SeaCon 2011 Hamburg, June 28th Page 1

2 Welcome / Overview The business domain Load modeling Load testing and performance engineering as integral part of the development process The reality check (why and what to measure) Developing your own load test infrastructure Conclusion Page 2

3 Our Business Domain: Unified Communications Main OpenScape Features: Conferencing (Voice/Web) Instant Messaging Presence (User/Media) One Number Service Voice Routing (Preferred Device, Rules) Telephony client (Contacts, Journal, etc.) Page 3

4 Architectural Challenges The ilities matter (aka NFRs, quality attributes) Scalability: up to 30,000 users in one large deployment up to 500 users in one single-server deployment Soft real time constraints / usability: deliver feedback to users within 500ms Availability, etc. not in scope of this talk Event driven, asynchronous communication Multiple parties/users involved in interactions, distributed state Dimensioning of system (which hardware?) - not in scope of this talk Page 4

5 Load Model Example: Call Control Busy hour call attempts (BHCA) 5 BHCA = 2.5 outbound inbound calls per user per hour 30,000 users * 2.5 calls/hour = 75,000 calls / hour = 20.8 calls / sec How many events do we have to distribute per call? Telephony client (signaling, journal, etc. for call parties) = approx. 20 Media presence (busy, idle) = 200 Total flow of events: 4500 events / sec 16.2 Mio events / hour Where do the numbers come from? domain expertise (telephony switch development) Buest guesses for new features Page 5

6 What does 4500 Events per Second Mean? 4500 events / sec Photo taken with 2 ms exposure time Need to deliver 9 events in this time span Page 6

7 Performance Engineering Activities & Development Cycle Page 7

8 Performance Test Feedback Cycle Product Definition Experience Load Model Test Client SUT Test & Measure Verify Analysis Re-Design & Optimization Product Release Customer is happy! Page 8

9 Thoughts on Performance Engineering & Architecture Performance engineering (PE) is NOT just a quality assurance / testing effort, but a vital architecture task. Make PE part of the core architecture team PE is involved from project day one on, during the whole development cycle Close cooperation between PE and development required Dedicated performance test team to run load tests PE staff must be accepted in development organization Avoid QA vs. development quarrel Warning: don t be too agile! You ll need to do a lot of BUFD the get the architecture right, product coding starts late ilities are nothing you can add on in later phases Page 9

10 Development Process, Back to Reality Theory: Use-case driven, iterative and incremental approach avoids the big integration bang. Practice: But what if the product evolves during development? Shift of client technology (SOAP -> Http/Ajax, long poll ) Integration of legacy parts Late integration of components Test Driver Call Control Call Routing Presence Consumer JMS Presence Service Test Driver Web Frontend JMS / MoM Call Journal DBMS Presence Contacts What we tested to reach LCA How the shipped product looks like Page 10

11 The Big Integration Bang Page 11

12 Lessons Learned Sum of well performing tested components is not a well performing product Competition for resources, e.g. DB connections, cache size, memory, etc. Complex side-effects and interactions may occur Need to test real end-to-end slices with result verification on a business level Performance test client must evolve with the product Double ditch: The more obvious / stupid a bug or bottle neck, the faster it is discovered Hard architecture problems are usually well hidden behind them Redesign at a late stage is extremely painful and costly Page 12

13 Feedback From The Real World The product changes the way the users communicate. E.g., use of voice/web conference even in a 1-to-1 calls because it s more convenient! much more load on the conferencing part, less on traditional calling Customers use the product in an unforeseen way Contacts per user It s not the average 50 contacts per user, some users have 10,000 contacts, some almost 0 Distribution of samples matters, must be part of the load model! DB cache, result set size, message size, paging Feature use depends a lot on the customer (company culture? Phone, , IM, or better not talk to the colleagues?) May invalidate up-front assumptions you made in the original load model Need to feed back the experience to adjust the original load model. May result in partial re-design and new releases! Page 13

14 Performance Test Feedback Cycle, Revisited Product Definition Experience Load Model Real World Feedback Customer Installation Test Client SUT Test & Measure Verify Customer Problem Analysis Re-Design & Optimization Product Release Page 14

15 How And What to Measure? Generic metrics (memory, CPU, I/O, garbage collection, etc.) are a good starting point. But you need to also have domain specific values, e.g. to determine the speaker change ratio of voice conferences. Number of measurement points and metrics will grows over time. Each hot spot or customer issue will add additional metrics. Security / privacy issues MUST NOT expose person-related information in performance counter Calling home is a no-go! Page 15

16 Why we modeled real user behavior Example: 30k users, 5 BHCA, 75k calls/h, 2 TCP/IP connections per user TPS-approach: extrapolation of a sample 250 users with 300 calls / h Short call duration (5 calls / min per user) 500 TCP/IP connections Possible optimization points are CPU, I/O, network traffic, DB performance. Wrong implications: Bottlenecks, conflicting DB transactions, etc. High cache hit rate Use case approach 30k users with 2.5 calls / h to randomly picked other users Realistic call duration, e.g. 3 min TCP/IP connections Dimensioning problems found: Connections (TCP/IP, DB) Queue and buffer sizes Cache hit, index selectivity, etc. Higher memory consumption for session objects (standing calls) Page 16

17 Where / When to Run Load Tests Load tests should be cheap to run. Costs for licenses and tests systems is crucial Amount of required hardware must be optimized give every team a chance to run load tests Run load tests for every change of a component Use a background load generator while executing functional tests Include load tests in continuous integration System tuning and evaluation of test results requires expert knowledge -> establish a dedicated team Establish long running tests (only stopped when new official delivery is available) Page 17

18 Why to develop your own Performance Test Infrastructure? Off-the-shelve tools based on capture-reply or scripting did not fulfill our requirements: It s not request-reply, but asynchronous, event-driven communication involving multiple parties (test users) that participate in a use case. Use cases are composed of several message exchanges and may last for several minutes, so we need to keep session state. Testing correctness is a key point for us we need to verify that soft real times constrains are met with all involved parties in a use case. We also expect feature interaction / concurrency bugs (with low potential probability e.g. 1 to 1 Mio) to occur. How can we detect and help in fixing them? Page 18

19 The Skunk Test Infrastructure Replaces the existing Load Runner / Grinder based load test based on TPS/sample extrapolation. Use-case based, user centered approach. Developed in Java by a small 3 person team within 1 year. Incorporates the lesions learned, early prototype, load test for the load test, continuous integration with load test to measure impact of new features. Non-blocking, asynchronous programing model. Provides library as base for automated feature testing. Performance test builds on top as independent client. Page 19

20 How Well Does Skunk Scale? 25 x SOAP / BD test 2 x PhoneSimulator 2 x Voice portal test Total test machines: x SUT Total SUT machines: 14 4 x Phone Simulator 1 x Perftest 14 x SUT Total test machines: 5 Total SUT machines: 14 Page 20

21 Conclusion Performance engineering is a core architectural activity involved in all phases. Load tests need to be tailored to the problem domain. Feedback cycles are central and should be applied as soon as possible and often. Discovered problems may force partial re-design / implementation. The product changes the way users use it, so you need to validate if the original assumptions still hold. The process works demonstrated by the development of the performance test client. Page 21

22 Contact the Authors Alexander Buchmann Software Architect Wolfgang Strunk Director Test Engineering Page 22