Performance Prediction and Optimization using Linux/cgroups

Transcription

1 Linux Con JAPAN 0(Yokohama) June st 0 Performance Prediction and Optimization using Linux/cgroups Yuzuru Maya Hitachi, Ltd., Yokohama Research Laboratory

2 Agenda Background Outline of Linux/Cgroups Performance prediction with RA Evaluation Conclusion RA: Regression Analysis

3 Background - Grow up server consolidation with the progress of the virtualization technology. - guarantee real time and performance at the same time and in the same server. - need stable operation. - Important to optimize computing resources with performance prediction. - Cgroups will be essential for process computing and cloud computing because of computing resources and access control.

4 Outline of Linux/Cgroups () Bandwidth control - Bandwidth control of CPU, Memory, Network and Disk. - Account the CPU usage. () Access control - Access control of device. - Constrain CPU and memory placement of tasks. - name space control. () Group control - Freeze task. - checkpoint and restart. 4

5 Target we research and develop on Linux/Cgroups AP-A Priority:High AP-B Priority:Low Require IO Throughput and Latency. Statistics data Resource Control/Access Control IO Control CPU/MemoryControl Monitor () Dev target Develop Cgroups itself. Prediction performance Feedback Compare Data measured and required data Disk Server Resource assignment () Dev target Develop using Cgroups (This presentation) 5

6 Outline of Block IO control () About blkio.weight bandwidth of IO. range of weights is from 00 to 000. IO-Weight is blkio.weight/00 (from to 0) to simplify in only this presentation. () Construct Measurement environment. mount -t cgroup none /cgroup/ -o blkio mkdir -p /cgroup/test /cgroup/test echo 900 > /cgroup/test/bklio.weight echo 00 > /cgroup/test/bklio.weight./fio-test.sh 6

7 Measurement Environment Measurement Machine # Item Content Dell PowerEdge840 Machine OS CPU:Intel Xeon 060.4GHz, Core(), Memory:4 GB RHEL6.0β x86_64 Benchmark FIO (Flexible IO) version.4 FIO condition # Item Content Measurement IO Throughput (IO issue per second) item & Latency (IO Complete clat) Access type Random read/write Measure time 60 sec 4 IO type Asynchronous I/O 5 I/O type 4kB,64kB 7

8 Measurement item and result querya: (g=0): rw=randrw, bs=k-k/k-k, ioengine=libaio, iodepth=50 Starting process querya: (groupid=0, jobs=): err= 0: pid=656 read : io=60mb, bw=,70kb/s, iops=5, runt=60548msec slat (usec): min=6, max=k, avg=60.46, stdev=690. clat (usec): min=, max=,8k, avg= , stdev= bw (KB/s) : min= 0, max= 487, per=0.88%, avg=776.9, stdev= write: io=590mb, bw=,677kb/s, iops=5, runt=60548msec slat (usec): min=6, max=,9k, avg=459., stdev=8.7 clat (usec): min=6, max=,00k, avg= , stdev= bw (KB/s) : min= 0, max= 596, per=96.80%, avg=6.9, stdev=947.7 Measurement item IO Throughput (IO issue per sec) Latency (clat) clat:complete latency cpu : usr=0.04%, sys=0.%, ctx=6654, majf=0, minf=4 IO depths : =0.%, =0.%, 4=0.%, 8=0.%, 6=0.%, =99.6%, >=64=0.0% submit : 0=0.0%, 4=00.0%, 8=0.0%, 6=0.0%, =0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=00.0%, 8=0.0%, 6=0.0%, =0.0%, 64=0.%, >=64=0.0% issued r/w: total=968/889, short=0/0 lat (usec): 50=0.0%, 00=0.0%, 50=0.0%, 500=0.65%, 750=0.09% lat (usec): 000=0.% lat (msec): =0.5%, 4=0.4%, 0=.78%, 0=.%, 50=7.80% lat (msec): 00=8.9%, 50=6.79%, 500=.5%, 750=5.86%, 000=9.9% lat (msec): 000=.%, >=000=0.% Run status group 0 (all jobs): READ: io=60mb, aggrb=,70kb/s, minb=,75kb/s, maxb=,75kb/s, mint=60548msec, maxt=60548msec WRITE: io=590mb, aggrb=,676kb/s, minb=,77kb/s, maxb=,77kb/s, mint=60548msec, maxt=60548msec Disk stats (read/write): dm-0: ios=50/4760, merge=0/0, ticks=54097/6650, in_queue=8477, util=00.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan% 8

9 Measurement Data,000 IO Throughput Latency IO Size: 4kB 6 IO Throughput (times/sec),500,000,500, Latency (msec) IO-Weight: blkio.weight/00 IO-Weight 9

10 What is Regression Analysis (RA)? Why the straight line? To reduce computational complexity to be able to set the moderate value in online in the future. () Regression Analysis? To be approximated by the straight line that is the nearest to data. y = a + b x b = S xy /S S x xy = (x i -x) ~ (y i -y) ~ S a = y ~ b x ~ x = (x i -x) ~ ~:average () Least Squares Method To minimize the distance between each point and the straight line. 0

11 Adapt Regression analysis () Purpose Predict performance with Regression analysis Procedure () Regression analysis with all range of IO-Weight - IO-Weight : from to 9 Result - IO throughput May be good(easy to predict). -Latency Not so good(hard to predict). point to rise suddenly when I make IO-Weight small(-). IO-Weight is an evaluation to -9, and the predictive range of the regression analysis is wide, and a prediction is difficult. An evaluation to considering a characteristic of IO throughput the latency is necessary.

12 Regression Analysis () IO Throughput (times/sec),000 IO Throughput Latency,500,000,500, IO Size: 4kB Latency (msec) IO-Weight: blkio.weight/00 IO-Weight

13 Purpose Adapt Regression Analysis () Extract characteristic of IO Throughput and Latency by limited IO-Weight range. Procedure () Regression analysis with partitioned range of IO-Weight - Divide IO-Weight as limited range as-, 4-6, 7-9 etc. - Do Regression analysis. () Characteristic of IO Throughput and Latency - IO throughput Improve linearly according to the mount of resources. -Latency Be constant (cannot shorten) even if resources is increased.

14 Regression Analysis () IO Throughput (times/sec),000 IO Throughput Latency,500,000,500, IO Size: 4kB Latency (msec) IO-Weight: blkio.weight/00 IO-Weight 4

15 Optimized procedure () Collect statistics data Optimized procedure Normalize data. (Sum of IO-Weight is 0 and decide performance of each IO-Weight ( to 9).) Collect statistics data (Normalize) () Predict performance Predict performance by statistics data. Set IO-Weight. () Compare with measured and required data Within allowable range, do nothing to modify IO-Weight. Without allowable range, do feedback and set IO-Weight again. Predict Performance Set IO-Weight. Feedback Set IO-Weight again Compare data measured and required data 5

16 Procedure Evaluation () Predict Latency and IO Throughput of 6kB by data of 4kB and 64kB. () Predict most suitable IO-Weight and perform performance measurement and check whether AP requirement is satisfied. () Feed back again if I do not meet it. Case AP Required value Latency IO Throughput AP-A.0,500 AP-B AP-A.0,500 AP-B AP-A.0,500 AP-B.5,750 6

17 Example of case(ok without feedback),000 IO Throughput (times/sec),500,000,500, kB 6kB 4kB 4kB 64kB AP-B AP-A Latency (msec) IO-Weight 7

18 Example of case (OK with feedback),000 IO Throughput (times/sec),500,000,500, kB 6kB 4kB 4kB 64kB AP-B AP-A Latency (msec) IO-Weight 8

19 Example of case (NG: shortage of resource),000 IO Throughput (times/sec),500,000,500, kB 0 64kB 6kB 4kB 64kB AP-B AP-A Latency (msec) IO-Weight 9

20 Evaluation result Case AP IO- Weight Before feedback Latency IO Throughput IO- Weight After feedback Latency IO Throughput Evalu ation AP-A 5.0 (.0),789 (,500) AP-B.9 (4.0) 985 (900) Reserved AP-A 7.5 (.0),4 (,500) 8.0 (.0),58 (,500) AP-B Reserved AP-A AP-B 9 -> 6 6 -> 4-5 -> (6.0) (.0).89 (.5) 670 (500) 7,879 (,500), (,500) Reserved - - Upper:Measured value,(lower):required value (.0) - 75 (500) - Resource shortage by reserved value after performance prediction. 0

21 Conclusion. Summary () Normalize throughput and latency with fio. - Evaluate Linux/Cgroups () Adapt Regression Analysis - RA: Adapt RA in all field (from to 9). - RA: Divide parts and adapt RA. () Propose Feedback function - to enable to be within an allowable error.. Future Works () Predict performance in online and set. () Cooperate other resource assignment - CPU and memory () and cooperate access control.

22 -Linux is a trademark of Linus Torvalds in the United States, other countries, or both. -Other company, product, or service names may be trademarks or service marks of others.