Quantitative System Evaluation with Java Modelling Tools

Transcription

1 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy Quantitative System Evaluation with Java Modelling Tools Giuliano Casale Imperial College London g.casale@imperial.ac.uk Giuseppe Serazzi Politecnico di Milano giuseppe.serazzi@polimi.it.net Working Group - BCAM G.Casale G.Serazzi 1

2 tutorial outline overview of Java Modelling Tools ( case study 1 (CS1): bottlenecks identification, performance evaluation, optimal load case study 2 (CS2): model with multiple exit paths case study 3 (CS3): resource contention case study 4 (CS4): multi-tier applications, web services G.Casale G.Serazzi 2

3 Java Modelling Tools ( CS2 CS3 CS4 CS1 CS1 CS4 G.Casale G.Serazzi 3

4 architecture Model Views JAVA/JWAT/JMVA JSIMwiz JSIMgraph XML XSLT XSLT XML Status Update JMT framework jsimengine Controller G.Casale G.Serazzi 4

5 software development JMT is open source, Java code and ANT build scripts at size: ~4,000 classes; 21MB code; 174,805 lines subversion svn co jmt source tree trunk (root also for help, examples, license information,...) src jmt analytical (jmva algorithms) commandline (command line wrappers) common (shared utilities) engine (main algorithms & data structures) framework (misc utilities) gui (graphical user interfaces) jmarkov (JMCH) test (application testing) G.Casale G.Serazzi 5

6 core algorithms - jmva Mean Value Analysis (MVA) algorithm (e.g., [Lazowska et al., 1984]) fast solution of product-form queueing networks open models: efficient solution in all cases closed models: efficient for models with up to 4-5 classes Product-form queueing networks solvable by MVA PS/FCFS/LCFS/IS scheduling Identical mean service times for multiclass FCFS Mixed models (open + closed), load-dependent Service at a queue does not depend on state of other queues No blocking, finite buffers, priorities Some theoretical extensions exist, not implemented in jmva G.Casale G.Serazzi 6

7 core algorithms jsimengine: simulation components in the simulation are defined by 3 sections component sections external arrivals (open class) discrete-event simulation engine queueing station serve admit route complete G.Casale G.Serazzi 7

8 core algorithms jsimengine: statistical analysis transient filtering flowchart [Spratt, M.S. Thesis, 1998] Transient [Pawlikowski, CSUR, 1990] (Steady State) [Heidelberger&Welch, CACM, 1981] G.Casale G.Serazzi 8

9 core algorithms jsimengine: simulation stop simulation stops automatically confidence level maximum relative error 9 traditional control parameters G.Casale G.Serazzi 9

10 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy CASE STUDY 1: Bottlenecks identification Performance evaluation Optimal load closed model multiclass workload JABA + JMVA G.Casale G.Serazzi 10

11 Outline objectives system topology bottlenecks detection and common saturation sectors performance evaluation optimal loading G.Casale G.Serazzi 11

12 characteristics of the system e-business services: a variety of activities, among them information retrieval and display, data processing and updating (mainly data intensive) are the most important ones two classes of requests with different resource loads and performance requirements presentation tier: light load (less demanding than that of the other two tiers) application tier: business logic computations data tier: store and fetch DB data (search, upload, download) to reduce the number of parameters (and to simplify obtaining their values) we have choosen to parameterize the model in term of global loads L i, i.e., service demands D i G.Casale G.Serazzi 12

13 topology of a 3-tier enterprise system clients 3-tier e-business system Internet workload 1 Web Server Application Servers Storage Servers workload 2... N customers 2 classes workload 1 Web Server Application Servers Storage Servers closed model workload 2 presentation tier business tier data tier G.Casale G.Serazzi 13

14 workload parameters resource Loadings matrix: Service Demands, i resources, r classes D ir = V ir * S ir global number of customers: N=100 system population: N={N 1,N 2 } {1,99} {99,1} population mix: β={β 1,β 2 }, fraction of jobs per class, β variable: study of the optimal load (optimal mix) asymptotic behavior: β constant, N increasing G.Casale G.Serazzi 14

15 Service Demands (resource Loadings) name of the model natural bottleneck of class 1 (Storage 2) Storage 3: potential system bottleneck natural bottleneck of class 2 (Storage 1) G.Casale G.Serazzi 15

16 What-if analysis (JMVA with multiple executions) parameter that changes among different executions fraction of class 1 requests number of models requested (may be not all not executed) G.Casale G.Serazzi 16

17 Bottlenecks switching (JABA asymptotic analysis) bottlenecks global loadings of class 2 bottlenecks fraction of class 2 jobs that saturate two resources concurrently (Common Saturation Sector) G.Casale G.Serazzi global loadings of class 1 17

18 throughput and Response time {N=1,99}-{99,1}, JMVA r/ms system Common Saturation Sector system class 1 class ms equiload Common Saturation Sector 0.48 class 1 class 2 G.Casale G.Serazzi throughput X Response times 18

19 Utilizations and Power {N=1,99} {99,1} system Storage 2 Storage 1 best QoS to class 1 Storage 3 best QoS to class 2 class 1 Common Saturation Sector class 2 G.Casale G.Serazzi Utilizations Power (X/R) 19

20 optimized load: service demands and bottlenecks multiple bottlenecks equi-utilization line Class 1 G.Casale G.Serazzi 20

21 optimized load: U and X Storage 3 system r/ms Storage 2 Storage 1 class 1 equi-utilization mix 0.48 class 2 G.Casale G.Serazzi Utilizations throughput X 21

22 optimized load: Response times and Residence times class 2 system Common Saturation Sector 4.78 ms 4.78 ms system Storage 1 Storage 2 class 1 Storage G.Casale G.Serazzi Response times Residence times 22

23 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy CASE STUDY 2: model with multiple exit paths open model single class workload different routing policies JSIMgraph G.Casale G.Serazzi 23

24 Outline objectives system topology what-if analysis performance with probabilistic routing performance with least utilization routing performance with Joint the Shortest Queue routing G.Casale G.Serazzi 24

25 objectives fallacies in using the index system response time also in single class models open model with multiple exit paths (sinks), e.g., drops, alternative processing, multi-core, load balancing, clouds,... differencies between response time per sink and system res ponse time impact on performance of different routing policies G.Casale G.Serazzi 25

26 system topology source of requests 0.5 S = 0.3 sec exponential distributions path 1 λ = 1 req/s S = 0.2 sec utilizations S = 1 sec 0.5 path 2 selection of the routing policy Casale - Serazzi 26

27 What-if analysis settings control parameter enable the what-if analysis initial arrival rate final arrival rate number of models requested G.Casale G.Serazzi 27

28 n. of customers N in the two paths (prob. routing) path 1 path 2 mean N = 0.37 j mean N = 9.13 j G.Casale G.Serazzi 28

29 Utilizations (per path) with prob. routing path 1 path 2 U = 0.27 U = 0.89 G.Casale G.Serazzi 29

30 system Response time (prob. routing) perf. indices collected mean R = 5.51 s number of models executed in this run (What-if) no requested precision 30

31 Response time per path (prob. routing) path 1 path 2 mean R = 0.72 s mean R = s system response time R = 5.5 sec G.Casale G.Serazzi 31

32 Utilizations with least utilization routing path 1 path 2 U = 0.41 U = 0.41 utilizations well balanced G.Casale G.Serazzi 32

33 Response times with least utilization routing path 1 path 2 R = 0.88 sec R = 3.55 sec system response time R = 1.5 sec G.Casale G.Serazzi 33

34 Utilizations with Joint the Shortest Queue routing path 1 path 2 U = 0.35 U = 0.61 G.Casale G.Serazzi 34

35 N of customers with JSQ routing path 1 path 2 N = 0.88 N = 0.47 G.Casale G.Serazzi 35

36 Response times with JSQ routing path 1 path 2 R = 0.70 sec R = 1.72 sec system response time R = 1.05 sec G.Casale G.Serazzi 36

37 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy CASE STUDY 3 Resource Contention (use of Finite Capacity Regions - FCR) contention of components hardware: I/O devices, memory, servers,... software: threads, locks, semaphores,... bandwidth open model single class workload JSIMgraph G.Casale G.Serazzi 37

38 modeling contention fixed number of hw/sw components (threads, db locks, semaphores,...) clients compete for the available component free request execution time: wait time for the next free component + wait time for the hardware resources (CPU, I/O,...) + execution time request interarrival times exponentially distributed payload of different sizes (exponentially distributed) evaluate the execution time of requests when the number of clients ranges from 1 to 20 and the number of components ranges from 1 to 10 ( ), evaluate the drop rate and the wait time in queue for the next available component implement several models with different level of completeness G.Casale G.Serazzi 38

39 threads (resource hw/sw) contention (simple model) λ=1 20 r/s server D I/O =0.047s D CPU =0.010s clients CPU I/O sink threads = 1 thread requests queue (inside the server) G.Casale G.Serazzi 39

40 model definition (unlimited threads and queue size) selection of perf.indices simulation results name of the model source of requests queue resource fraction of capacity used sink λ = 1 20 req/sec fraction of n.o of requests G.Casale G.Serazzi 40

41 input parameters (service demands) mean service time = s mean service time = s G.Casale G.Serazzi 41

42 system Response time (λ=20 req/sec) perf.indexes selected confidence interval transient duration the number of samples analyzed is greater than the max defined here actual sim. parameters default values of parameters G.Casale G.Serazzi 42

43 λ=1 20 req/s, unlimited threads & queue size (JSIMgraph) U I/O = λd I/O = 20*0.047 = 0.94 (exact) (sim) R = s (sim) system Response time R = s (exact) Utilization of I/O X = r/s throughput same as λ no limitations system Power G.Casale G.Serazzi 43

44 Number of requests (unlimited threads & queue size) req 0.25 req. N = req (sim) N = XR = req (exact) G.Casale G.Serazzi 44

45 set of a Finite Capacity Region FCR step 1 select the components of the FCR step 2 set the FCR queue region with constrained number of customers drop G.Casale G.Serazzi 45

46 FCR parameters global capacity of the FCR max number of requests per class in the FCR drop the requests when the region capacity is reached (for both the constraints) G.Casale G.Serazzi 46

47 system Number of requests (limited n. threads and drop) unlimited 15 threads 10 threads 5 threads G.Casale G.Serazzi 47

48 Utilization of I/O server (limited n. threads and drop) unlimited 15 threads 10 threads 5 threads G.Casale G.Serazzi 48

49 system Response time (limited n. threads and drop) unlimited 15 threads 10 threads 5 threads G.Casale G.Serazzi 49

50 ... external finite queue for limited threads λ=20 r/s Blocking After Service policy server queue D server =0.047s clients server sink drop policy threads = 5 queue for threads with finite capacity (outside the server) the queue for threads is limited (e.g., to limit the number of connections in case of denial of service attack, to guarantee a negotiated response time for the accepted requests,...) the requests arriving when the queue is full are rejected (drop policy) the number of threads is limited and the requests are queued in a resource different from the server (load balancer, firewall,...) evaluate the combination of different admission policies G.Casale G.Serazzi 50

51 set Block After Service (BAS) blocking policy station with finite capacity max number of requests in the station selection of the BAS policy BAS policy: requests are blocked in the sender station when the max capacity of the receiver is reached G.Casale G.Serazzi 51

52 different admission policies for Queue and Server λ=20 req/s N R U X Drop Queue and Server stations Qsize= Q Ser=5, queue S Queue 5 Server Qsize= Ser=5, BAS Q S Queue BAS 5 Server Qsize=5 drop Q Ser=5, BAS S drop Queue BAS 5 5 Server Qsize= Q Ser=5, drop S Queue drop Server 5 G.Casale G.Serazzi 52

53 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy CASE STUDY 4 Multi-Tier Applications and Web Services (Worker Threads, Workflows, Logging, Distributions) closed models single class and multiclass workloads fork-join JSIMgraph+JWAT G.Casale G.Serazzi 53

54 performance evaluation of a multi-tier application multi-tier application serves a transactional workload which requires processing by an application server (AS) and by a database (DB) the AS serves requests using a fixed set of worker threads requests waiting for a worker thread are queued by the admission control system utilization measurements available for the AS and for the DB know both for AS and DB the average service time S e.g., linear regression estimate U=SX+Y, U = utilization, X = throughput, Y =noise evaluate response time for increasing worker threads G.Casale G.Serazzi 54

55 transaction lifecycle Client-Side Application Server DB Server Network latency (1) Request arrives Queueing time Worker Thread Admission control Worker thread admission time Load context in memory Request Response time Server Response time Simultaneous Service time (1) Resource Possession DB query time (1) CPU Data access Service time (2) CPU DB query time (2) Data access Service time (3) CPU Network latency (2) Response arrives G.Casale G.Serazzi 55

56 modelling abstraction (easier to define and study) Client-Side Server-Side Network latency (1) Request arrives Queueing time Worker Thread Admission control Request Response time Server Response time Server admission time Service time (1) Service time (2) Service time (...) Application Server Steps Load context in memory CPU Data access CPU+I/O DB Server Steps DB query time (1) DB query time (2) Data access CPU+I/O Network latency (2) Response arrives G.Casale G.Serazzi 56

57 modelling multi-tier applications send to jmva simulate N=300 app users Exponential Distributions Scpu = 0.072s Sdb = 0.032s FCR Admission Queue is Hidden! Zload = 0.015s 4 Servers (Cores) PS scheduling FCR FCR Capacity FCR Admission Policy G.Casale G.Serazzi 57

58 simulation vs jmva model FCR not included in product-form model G.Casale G.Serazzi 58

59 SAP Business Suite [Li, Casale, Ellahi; ICPE 2010] Response Time REAL SIM R S Quad-Core Server N=300 users R S MVA M M R S M G.Casale G.Serazzi 59

60 what-if analysis adding a web service class some requests now access the service composition engine of the multi-tier application to create a business travel plan services are composed on the fly from external providers (travel agencies, flight booking service) according to a workflow worker thread remains busy for the entire duration of the web service workflow evaluate end-to-end response time for each class G.Casale G.Serazzi 60

61 business trip planning (BTP) web service N=300 app users Nbtp=50 BTP users Sbtp =?, Exp? pbtp=1.0 FCR Class-Based Admission G.Casale G.Serazzi 61

62 BTP web service sub-model Logger Zsce=0.025s, Exp S2=?, Exp? S0=?, Exp? N=1 WS instance S1=?, Exp? G.Casale G.Serazzi 62

63 jwat Workload Analysis Tool Column-Oriented Log File Specify Format Data Format Templates Load Data G.Casale G.Serazzi 63

64 jwat data filtering Ignore Negative Samples G.Casale G.Serazzi 64

65 jwat descriptive statistics Scatter plots c=std. dev. /mean Histogram Hyper-Exp (c >1) G.Casale G.Serazzi 65

66 jwat scatter plot Scatter plot Outliers? G.Casale G.Serazzi 66

67 BTP web service sub-model N=1 WS instance log inter-arrival times Zsce=0.025s, Exp S2=0.911 HyperExp c= S0=0.967 HyperExp c= S1=2.151, HyperExp c=1.689 G.Casale G.Serazzi 67

68 BTP response times e.g., Weibull, Lognormal. Gamma logarithmic transformation G.Casale G.Serazzi 68

69 response time distribution logger components timestamp, class id, job id Sbtp = 3.611s Gamma c=1.44 timestamp, class id, job id global.csv job id (same throughout simulation) logger id job class G.Casale G.Serazzi 69

70 response time distribution analysis (matlab) cumulative distribution 95 th percentile cdf [seconds] G.Casale G.Serazzi 70

71 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy CONCLUSION 71

72 Final remarks Analysis with Java Modelling Tools ( Queueing network simulation Bottlenecks identification Workload analysis Mean value analysis... JMT-Based examples and exercises ( Topics not covered by this tutorial jmch Burstiness analysis Trace-driven simulation... JMT discussion forum: G.Casale G.Serazzi 72

73 References G.Casale, G.Serazzi. Quantitative System Evaluation with Java Modelling Tools (Tutorial). in Proc. of ACM/SPEC ICPE 2011 (companion paper). M.Bertoli, G.Casale, G.Serazzi. User-Friendly Approach to Capacity Planning Studies with Java Modelling Tools, in Proc. of SIMUTOOLS M.Bertoli, G.Casale, G.Serazzi. JMT - Performance Engineering Tools for System Modeling. ACM Perf. Eval. Rev., 36(4), 2009 M.Bertoli, G.Casale, G.Serazzi. The JMT Simulator for Performance Evaluation of Non Product-Form Queueing Networks, in Proc. of SCS Annual Simulation Symposium 2007, 3-10, Norfolk, VA, Mar M.Bertoli, G.Casale, G.Serazzi. Java Modelling Tools: an Open Source Suite for Queueing Network Modelling and Workload Analysis, in Proc. of QEST 2006, , Sep E.Lazowska, J.Zahorjan, G.S.Graham, K.C.Sevcik, Quantitative System Performance: Computer System Analysis Using Queueing Network Models, Prentice-Hall, K.Pawlikowski: Steady-State Simulation of Queuing Processes: A Survey of Problems and Solutions. ACM Comput. Surv. 22(2): , P.Heidelberger and P.D.Welch. A spectral method for confidence interval generation and run length control in simulations. Comm. ACM. 24, , S.C.Spratt. Heuristics for the startup problem. M.S. Thesis, Department of Systems Engineering, University of Virginia, G.Casale G.Serazzi 73

74 Politecnico di Milano Dip. Elettronica e Informazione Milan, Italy Contact us! g.casale@imperial.ac.uk giuseppe.serazzi@polimi.it 74