An Experimental Study of Rapidly Alternating Bottleneck in n-tier Applications

Transcription

1 An Experimental Study of Rapidly Alternating Bottleneck in n-tier Applications Qingyang Wang, Yasuhiko Kanemasa, Jack Li, Deepal Jayasinghe, Toshihiro Shimizu, Masazumi Matsubara, Motoyuki Kawaba, Calton Pu

2 Scaling Web Applications On-Demand in Cloud 2 Good performance + Cost efficiency High throughput + low response time High resource utilization Bottleneck Bottleneck

3 3 What If No Bottleneck Was Detected? How to scale a web application while no bottleneck is identified? Bad performance CPU util. 34.6% Disk I/O 0.5% Memory 50% CPU util. 79.2% Disk I/O 0.5% Memory 30% CPU util. 26.7% Disk I/O 0.5% Memory 50% CPU util. 78.1% Disk I/O 0.5% Memory 50%

4 4 Rapidly Alternating Bottlenecks workload Web server App server Bottleneck Bottleneck DB server Time Bottleneck Bottleneck t0 t1 t2 t3 t 1. Throughput is limited with no saturated resources 2. Duration of each bottleneck is short (e.g., < 100ms)

5 5 Experimental Setup RUBBoS benchmark: a bulletin board system like Slashdot 24 web interactions CPU intensive Workload consists of emulated clients Intel Xeon E quad-core 2.26 GHz 16 GB memory

6 Throughput [req/s] 6 Response time [s] Motivational Example Response time & throughput of a 3-minute benchmark on the 4-tier application with increasing workloads. WL 14,000 WL 14,000 Workload [# users x1000] Workload [# users x1000]

7 No Obvious Bottleneck is Detected at WL 14,000 Workload is CPU intensive Disk I/O utilization (<5%), network I/O utilization (< 20%), Memory usage (<40%); CPU util. 79.2% CPU util. 78.1% CPU util. 34.6% CPU util. 26.7% CJDBC 7

8 Rapidly Alternating Bottleneck: Sources and Detection Sources: We find that other than bursty workload, system environmental conditions: JVM garbage collection VM collocation Detection and Visualization: We implement a fine-grained monitoring method based on passive network tracing. Negligible monitoring overhead for running applications 8

9 9 Outline Introduction & Motivation Detection and Visualization Fine-grained load/throughput analysis Two Observations of Rapidly Alternating Bottlenecks JVM garbage collection (JVM GC) VM collocation Conclusion & Future Works

10 Two Steps for Detecting Rapidly Alternating bottlenecks Find the participating servers that present transient bottlenecks(e.g., 50ms) 2. Check whether the transient bottlenecks of each participating server occur in an alternating pattern

11 11 Passive Network Tracing Infrastructure Collect interaction messages in the system using SysViz to measure fine-grained active load and throughput on each server. Active load: The # of concurrent requests in a server Throughput: The # of completed requests of a server Web AP DB SysViz box 01:58: Web server AP server DB server 01:58: :58: :58: :58: Network switch 01:58: :58: :58: :58: :58: :58: :58:20.182

12 Fine-Grained Active Load Calculation in a Server 12 SysViz monitoring for MySQL time arrival timestamp departure timestamp 50ms 50ms

13 Server throughput Active-Load/Throughput Correlation Analysis 13 TPmax Non- Saturation area Saturation area Saturation point N* Active load in a server

14 MySQL throughput [req/s] Active load [#] Throughput [req/s] Active-Load/Throughput Analysis for MySQL at WL 14,000 MySQL active load (every 50ms) MySQL throughput (every 50ms) Time [s] Time [s] N* MySQL active load [#] 14

15 15 Outline Introduction & Motivation Detection and Visualization Fine-grained load/throughput analysis Two Observations of Rapidly Alternating Bottlenecks JVM garbage collection (JVM GC) VM collocation Conclusion & Future Works

16 Active-Load/Throughput Analysis at Workload 7,000 System Throughput [req/s] 16 Tomcat throughput [req/s] Point of Interest (POI) Tomcat active load [#] WL 7,000 MySQL throughput [req/s] Workload [# users x1000] System is far from saturation at WL 7,000 MySQL active load [#]

17 Active-Load/Throughput Analysis at Workload 14,000 System Throughput [req/s] 17 WL 14,000 Tomcat throughput [req/s] Tomcat active load [#] POI MySQL throughput [req/s] Workload [# users x1000] System achieves the maximum throughput at WL14,000 MySQL active load [#]

18 Active load [#] Active load [#] 18 Tomcat GC running ratio [%] Throughput [req/s] Timeline Analysis at Workload 14,000 POI 4 4 Tomcat scatter graph Time (s) Time (s)

19 Active load [#] Throughput [req/s] Active load [#] Throughput [req/s] Timeline Analysis at Workload 14,000 (Cont.) POI 4 4 Tomcat scatter graph Time (s) 4 MySQL scatter graph Time (s) 4 19

20 Tomcat active load [#] Correlation Analysis of Rapidly Alternating Bottlenecks 20 MySQL active load [#] Time (s) Correlation coefficient: -0.42, negative correlation suggests rapidly alternating bottleneck.

21 21 Outline Introduction & Motivation Detection and Visualization Fine-grained load/throughput analysis Two Observations of Rapidly Alternating Bottlenecks JVM garbage collection (JVM GC) VM collocation Conclusion & Future Works

22 22 Conclusion & Future Work Rapidly alternating bottlenecks can cause non-trivial performance loss in an n-tier system. We proposed a rapidly alternating bottleneck detection and visualization method through fine-grained active-load/throughput analysis Ongoing work: more analysis of different types of workloads and more system factors that cause rapidly alternating bottlenecks.

23 23 Thank You. Any Questions? Qingyang Wang

24 Backup slides 24

25 Tomcat throughput [req/s] MySQL throughput [req/s] 25 Tomcat throughput [req/s] MySQL throughput [req/s] Resolving Rapidly Alternating Bottlenecks Tomcat active load [#] Tomcat active load [#] MySQL active load [#] MySQL active load [#]

26 Response time[s] Performance Gain After Resolving Rapidly Alternating Bottlenecks 26 Throughput [req/s] Workload [# users x1000] Workload [# users x1000]

27 Active-Load/Throughput Analysis at Workload 14,000 System Throughput [req/s] 27 WL 14,000 System achieves the maximum throughput at WL14,000 Workload [# users x1000]