Performance Testing WITH

Transcription

1 1 Performance Testing WITH GATLING

2 Workshop mission 2 Theory Practice What? Why? How? Introduction to Gatling Test Design Structure Load Modelling (Diagnostics) The Juice Shop Story Record Edit & Execute Load Tests with Gatling

3 By the end of this Workshop 3 You should have a local test setup with An example Application A Gatling Test Script Basic Monitoring So you could experiment for yourself afterwards You should know How to create a Simulation using the Recorder (Record & Replay) Waht the elements of a Gatling Load Test are Know the Basics for Designing a Load Test

4 4 How to do Performance Testing?

5 Type of Load Test 5 Batch Get the most done in the least time and resources Data Driven Focus on Maximum Troughput Online Process the most events in the least time Event Driven Focus on Low Response Times

6 Input to Load Test Scenarios 6 Performance Requirements Target Users (Concurrent, per Duration, Total) Response Time Targets (90%, 95%, 99%) Throughput Historical Data Number of Total Users per Duration Number of Concurrent Users Peak Loads (Peak Month/Day/Hour/Minute) Request Logs Educated Guesses / Gut Feeling

7 Example 7 Requirements The system is capable of Serving 1000 concurrent users with an average Response Time of 1.5s Source: The Project Manager What are the most relevant information? This Photo by Unknown Author is licensed under CC BY-NC-SA

8 Numbers need a Context 8 Concurrent Users (fit into CPU) vs. Concurrent Sessions (fit into Memory) vs. Users per Period (fit into System) Average vs. perceived Average (90 %) vs. Percentiles (95%,99%,99.9%) 1000 Conc.Users, avg < 1.5s vs 1000 Users/h, 90% < 1.5s This Photo by Unknown Author is licensed under CC BY-NC-SA

9 Load Testing Practices 9 Soak Testing Discover Leaks SLA Regressions Stress Testing Testing Stability Overload / Recovery Benchmarking Discover Regressions between different Versions or Configurations

10 A typical load test: Constant Load 10 Constant Load + Ramp up / down Load % Good for: Allowing the System to adapt to Load (warm up) Distributes Load (virtual users) Finding latent bugs, i.e.memory or Resource Leaks Precise Measurements Finding Regressions t Finding Stability Issues Statistical Response Times Known Load Distributions

11 Why do a ramp up? 11 System Warm-up Allow JIT to optimize code Allow Caches to be populated Fetch or Initiate Resources (i.e. Database Connections) Allow Queues and Buffers to fill to a stable level Distribute Load evenly System s Performance Characteristics are non-linear during ramup This Photo by Unknown Author is licensed under CC BY-NC-ND

12 What do we test for? 12 Capacity Oversubscribed resources CPU Memory Bandwith Storage Queues Response times under load Stability Resources leaks Memory Network connections Filehandle System crashes System recovery

13 13 Any Heuristics to sum it up?

14 CCD IS EARI 14 Context: Project context is central to successful performance testing. Criteria: Business, project, system, & user success criteria. Design: Identify system usage, and key metrics; plan and design tests. Install: Install and prepare environment, tools, & resource monitors. Script: Implement test design using tools. Execute: Run and monitor tests. Validate tests, test data, and results. Analyze: Analyze the data individually and as a cross-functional team. Report: Consolidate and share results, customized by audience. Iterate: "Lather, rinse, repeat" as necessary.

15 IVECTRAS 15 INVESTIGATION or VALIDATION of END-TO-END or COMPONENT response TIMES and/or RESOURCE consumption under ANTICIPATED or STRESSFUL conditions

16 16 Gatling AN INTRODUCTION

17 Gatling Basics 17 Written in Scala And so are the Load Tests Asynchronous, Non-Blocking, Event-Driven Concurrency Model Virtual Users are «Messages», not threads Less Resource Consumption & Higher Loads Supports Open and Closes Workload Modells Standalone or Integration with Maven / Gradle Useful in Continuous Integration Pipelines Visual Reports

18 Pros & Cons 18 Pro Testing Framework Small, lightweight reduced to max No installation: Maven / Gradle Plugin Download Standalone Highly Scalable Event Based Load Generation Documentation & Community Free & OpenSource Cons No UI only for Recorder Requires programming skills Scala Limited Protocol Support HTTP & JMS Basic reporting

19 19 Threads vs Events LOAD GENERATION BASICS

20 Thread based Load Generation 20 Thread 1 Virtual User

21 Thread based Load Generation 21 Request Thread 1 Virtual User

22 Thread based Load Generation 22 Thread 1 Virtual User Thread WAIT

23 Thread based Load Generation 23 Response Thread 1 Virtual User

24 Thread based Load Generation 24 Thread 1 Virtual User Think Time

25 Thread based Load Generation 25 Thread 1 Virtual User User Scenario

26 Thread based Load Generation 26 Thread 1 Virtual User Pace time

27 Thread based Load Generation 27 Real User 1 Real User 2 Thread 1 Virtual User

28 Thread based Load Generation 28 Thread 1 Thread 2 Thread 3

29 Thread based Load Generation 29 Characteristics Each Simulated Real User depends on it s previous user Each consecutive Request depends on Response for previous Request Requires lot of Resource Suffers from Coordinated Omission (SUT throttles Load Generator) Rampup ensures that generated load is evenly distributed Pacing ensures that load distribution remains stable Good for Closed User Groups (i.e. Employees, Named Users), Dependant Users Determine Capacity Tool Example: Apache JMeter

30 Event Based Load Generation 30 Event Thread Async Request Async Response Timer Event Response Event

31 Event Based Load Generation 31 More Users Event Thread

32 Event Based Load Generation 32 Characteristics Simulated Users are Independent Sent Requests are independent from previous Requests Requests/Responses are handled asynchronously Requires less resources for same load as thread based systems No Coordinated Omission Limited by the Processing Capacity of the Event-Thread Ramp up / down is defined by change in User Rate Good for Independent Users Open User Groups (i.e. for public web sites) User Experience Rating Tool-Example: Gatling

33 33 Three elements of a Load Test Page Scripts User Flows Load Function How groups of users navigate through your application How to fetch a typical page and it s resources How the users arrive at your page

34 34 Three elements of a Load Test Page Scripts User Flows Load Function GET /index.html GET /favicon.ico GET /somescript.js GET /img/background.jpg

35 Test Development with Gatling 35 Pages Scenarios Load Models

36 Test Development with Gatling 36 val page1 = exec( http("request_0").get("/").headers(accepthtml).resources(...)) val scn = scenario("simple").exec(page1) scn.inject( constantuserspersec(500) during (1 minute) ) Pages Scenarios Load Models

37 37 Exercise RECORD & REPLAY SCRIPT WITH GATLING

38 The Boutique de Jus story 38 We want to revolutionize the way people consume healthy food. We re going to launch several service offerings, starting with our ultimate Juice-Shop. Business goal: Our investors expect profit at peak of 1.5Mio$ / month (30days) Assuming a revenue 5$/order and profit margin of 10% we need 100k orders per day We expect 70k-100k visits on our site per hour. We assume that all visits occur within 10 hours per day- Conversion rate of 10% (submitted orders) Response time per page should be 1.5s 4s

39 CCD IS EARI 39 Context: Startup, Webshop: boutique de jus Criteria: 100k user/h, 10k orders/h, Response times: 1.5s-4s Design: see excercises Install: see excercise 1 Script: see excercises Execute: execute scripts & monitor with VisualVM Analyze: see gatling reports & VisualVM Report: create a brief report in exercise 3 using PROOF Iterate: in three excercises

40 Test Mission 40 Criteria: Set the technical baseline Are you able to test with Gatling? Design: Record a very simple flow for load testing Install: setup the test lab package on your machines

41 41 Designing Performance Tests

42 FIBLOTS 42 Frequent: Common application usage. Intensive: i.e. Resource hogging activities. Business Critical: Even if these activities are both rare and not risky Legal: Stuff that will get you sued or not paid. Obvious: Stuff that is likely to earn you bad press Technically Risky: New technologies, old technologies, places where it s failed before, previously under-tested areas Stakeholder Mandated: Don t argue with the boss (too much).

43 What is a Load Model? 43 How Load is distributed over time Defined by a rate of events in a period Users arriving at the application Requests Actions / Clicks Bytes

44 Equi-Distribution 44 Completely Synthetic Suitable for long-running stability Tests Benchmarking Load should aim at 80% system capacity Good for tracking long-term effects Resource / Memory Leaks Gargbage Collection Singularities Native support in Gatling

45 Ramp-up 45 Slowly increasing Load on System System can Warm-up Populate Caches Compile & Optimize Code (Java -> JIT) Users & Requests get distributed Native support in Gatling

46 Open vs Closed Workload 46 Open Load Generator and System Under Test have no Feedback Loop SUT can be overloaded Closed Load Generator and System under Test in a Feedback Loop SUT throttles LG Load System establishes a balance Modelled by Arrival Rate Modelled by Concurrent Users Suitable for Unknown User Group and Statistical Distributions Suitable for Closed User Groups and Known Loads Supported by Gatling Since Gatling 3.0

47 48 Analysis & Reporting Photo: Calvinius

48 What do we need for a deeper analysis? 49 Precise reporting of your observations Baseline configuration of the system Model of the system (architecture) to visualize the flow The right people available

49 External Observations 50 Response Times Averages, Min, Max Percentile (90% is perceived average!) Outliers Histogram Error Responses Which page fails most often? Live Uses (Users that have not finished their scenario) Piling up? Constant

50 Internal Observations 51 CPU Consumption System I/O Activity Disc Garbage Collection Activity GC Logs Allocation Rates GC Generational Behavior Network Context Switching User Threads & Contention GC Activity Algorithms Memory Consumption Heap Non-Heap Thread Stacks Direct Allocations Code Cache Metaspace Hardware Counters Leaks

51 Java Performance: Diagnostic Model* 52 Usage Patterns / Actors Dynamics Application JVM No Dynamics

52 Java Performance: 53 Diagnostic Process System CPU: sys > 10% user? IO, Disk, Network, system profiling: netstat, mpstat, iostat, sar, strace,... Threads User <100%? Thread Dump Thread Starvation JVM Memory efficient? GC Logs GC Tuning, Collectors, Pool Sizes, Memory Profiling, Frequency, Lifespan Application App/CPU Profiling CPU Profiling Algorithms & Data

53 Reporting: PROOF 54 Past. What happened during the session? Results. What was achieved during the session? Obstacles. What got in the way of good testing? Outlook. What still needs to be done? Feelings. How does the tester feel about all this?

54 55 Summary

55 56 Warning! THE NEXT SLIDE CONTAINS A FLASHY PICTURE! VIEWER DISCRETION IS ADVISED.

56 57 Forget everything what you have heard so far, now comes the real important stuff!

57 Test Development with Gatling 58 Pages Scenarios Load Models

58 Test Development with Gatling 59 val page1 = exec( http("request_0").get("/").headers(accepthtml).resources(...)) val scn = scenario("simple").exec(page1) scn.inject( constantuserspersec(500) during (1 minute) ) Pages Scenarios Load Models

59 # BaselOne18