Performance Testing of a Road Tolling System

Transcription

1 Performance Testing of a Road Tolling System Oliver Baendl Siegfried Goeschl

2 Gauteng Open Road Tolling South Africa

3 42 Tolling Points

4 registered road-user

5 11 Satellite Centers

6 41 E-Toll Shops

7 20 Mobile Payment Units

8 Central Operation Center

9 750 Concurrent CRM Client Users

10 162 vehicle passges per second

11 400 concurrent web users

12 Two Data Centers

13 Master Control Program

14 Project Overview Vehicles are passing through a tolling point The vehicle is identified either using the license plate number (via OCR) or a tag Vehicle passage are transmitted to Open Road Tolling Back Office (ORTBO) 90% video passages with three images 10% tag passages

15 Project Overview Vehicle passages are collected and assigned to road-user accounts in the Transaction Clearing House (TCH) TCH responsible for billing the customer Passages of unregistered or black-listed cars are processed by the Violation Processing Center (VPC)

16 Project Overview Overall system contains multiple Oracle RACs and MSSQL clusters Interfaces between the subsystems mostly web services Backup data center replicates the Central Operation Center over dark fiber

17 Schedule Phase One Performance testing of CRM Server simulating 750 concurrent & active CRM client sessions Performance testing of public-facing web portal simulating 400 concurrent & active web user sessions On success the CRM subsystem goes live and has real user accounts

18 Schedule Phase Two Simulating up to 162 vehicle passage per second Validating the end-to-end processing under load (passage processing, customer billing and violation processing) On success the overall system goes live

19 Performance Test Scope End to end performance acceptance test Mandatory and client-witnessed test Strict acceptance criteria regarding throughput number and severity of errors validation of test runs

20 Performance Test Types Baseline Test One hour average load Stress Test One hour peak load Endurance Test 24 hours maximum load

21 Performance Test Hardware Six dedicated load injectors Windows 2008 Server 3 physical and 3 virtual boxes Dual quad-cores with 8 GB RAM No direct access outside the data center Citrix over RDP over RDP

22 Performance Test Tools All performance test tools have issues Make sure that they work for you We prefer JVM-based test tools Easy to migrate between different OS We are Java developers We prefer command-line invocation RDP over slow network is really slow

23 Performance Test Tools We use a non-distributed installation Complex network and routing Firewall not under our control Check the license and price tags Virtual users can be expensive Connectivity to license server?

24 The Test Tools We Used Subsystem Interface Planned Delivered CRM Server WebService Public Web Portal HTML Vehicle Passage Processing WebService

25 Apache JMeter Open Source and free Implemented a new reporting backend to scope with huge result files Rock solid - only minor issues Extensible using scripting and Java libraries Rough GUI compared to commercial tools Unsuitable for complex ASP.NET web sites

26 WAPT Pro Excellent value for money Good support ASP.NET module saved my day Extensible through Javascript One major issue with generation of test values under high load (requires restart)

27 soapui Pro Excellent for prototyping and functional testing of web services WebService mocking saved my day Extensible using Groovy and Java libraries Various issues during load testing soapui was replaced with JMeter for performance tests

28 Some More Thoughts No software installation Take two (or more) Automation is your friend Performance test for everyone Performance tests are a valuable asset

29 No Software Installation

30 No Software Installation Portable Apps on USB Stick Run tests directly from USB Stick Clone USB Stick to hard disk Have all your tools on the USB stick Java, JMeter, Editor, Git,...

31 Take Two (or more) A minimum of two share-nothing load injectors are required to prove that the production servers is causing the performance bottleneck (and not your test tool) We used a maximum of five sharenothing load injectors simultaneously

32 Automation Is Your Friend Performance test were run mostly at night and over the weekend Test execution managed by Hudson Test failures triggers notification Test protocols are copied to FTP server

33 Performance Tests for Everyone All test scenarios are configured as Hudson jobs Baseline, stress & endurance test Some other internal tests Everyone can start and monitor a performance test scenario over the web browser Even developers can run performance tests

34

35 Performance Tests as Asset Performance tests can be used in creative and un-planned ways Smoke test for new deployment Testing database failover Testing data center replication & failover Testing different database setups

36 The few things you should take home

37 Lessons Learned

38 Lessons Learned Tests tools are buggy

39 Lessons Learned Tests tools are buggy Performance tests are a valuable asset

40 Lessons Learned Tests tools are buggy Performance tests are a valuable asset Creation and maintenance of complex performance test suite is a project on its own

41 Questions & Answers

42 Resources JMeter SLA Report JAMon API JMeter ANT Tasks WAPT Pro soapui projects/jmeter-ant-task.php

43 The Software Stack JMeter WAPT Pro soapui Pro Hudson Portable Apps Cacti Git JSPWiki Scripting

44 The Challenge Ahead

45 The Challenge Ahead A 100+ million EUR project

46 The Challenge Ahead A 100+ million EUR project High complexity

47 The Challenge Ahead A 100+ million EUR project High complexity Prohibitive contractual penalties

48 The Challenge Ahead A 100+ million EUR project High complexity Prohibitive contractual penalties Multi-site Development

49 The Challenge Ahead A 100+ million EUR project High complexity Prohibitive contractual penalties Multi-site Development Teams located in Austria, Sweden, Argentina and South Africa

50 The Challenge Ahead A 100+ million EUR project High complexity Prohibitive contractual penalties Multi-site Development Teams located in Austria, Sweden, Argentina and South Africa Development & QA overloaded

51 The Challenge Ahead A 100+ million EUR project High complexity Prohibitive contractual penalties Multi-site Development Teams located in Austria, Sweden, Argentina and South Africa Development & QA overloaded

52 The Challenge Ahead

53 The Challenge Ahead Performance testing on critical path

54 The Challenge Ahead Performance testing on critical path Lots of politics and finger-pointing

55 The Challenge Ahead Performance testing on critical path Lots of politics and finger-pointing Your performance tests are broken

56 The Challenge Ahead Performance testing on critical path Lots of politics and finger-pointing Your performance tests are broken Complex test scenarios

57 The Challenge Ahead Performance testing on critical path Lots of politics and finger-pointing Your performance tests are broken Complex test scenarios Often broken by new deployment

58 Plans are invitation to disappointment.

59 Two Data Centers, altogether 300 servers, MSSQL clusters, Oracle RACs, large SAN storage, 1.5 million registered road-users, 750 CRM users, backup data center over dark fiber

60 Things We Did To JMeter

61 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request

62 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request Sending up to 400 SOAP requests / sec

63 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request Sending up to 400 SOAP requests / sec Sending 400 KB SOAP requests

64 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request Sending up to 400 SOAP requests / sec Sending 400 KB SOAP requests 70 MByte/sec over the wire

65 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request Sending up to 400 SOAP requests / sec Sending 400 KB SOAP requests 70 MByte/sec over the wire Using up to 750 worker threads per JVM

66 Things We Did To JMeter Three million WS requests over HTTPS requests where the server closes the socket after each request Sending up to 400 SOAP requests / sec Sending 400 KB SOAP requests 70 MByte/sec over the wire Using up to 750 worker threads per JVM 64-bit JVM with -Xmx=2048M