Perkit Benchmarker: A User Perspective. Feb 27th, Liqun (Legion) Cheng Staff Performance Engineer, Google Platforms

Transcription

1 Perkit Benchmarker: Feb 27th, 2015 A User Perspective Liqun (Legion) Cheng Staff Performance Engineer, Google Platforms liquncheng@google.com

2 Goal To create an OpenSource Living Benchmark Framework that represents how Cloud developers are building applications, evaluating Cloud alternatives, and exploring architecture. Living because it will change and morph quickly as developers change.

3 Shared on GitHub PerfKitBenchmarker: The framework for running benchmarks PerfKitExplorer: The Explorer for dashboarding and analysis vm spec provisioning workflow installer topology provisioning upload reporting publish prepare run cleanup parse & store execution analysis & scoring scoring

4 Partners Academia Stanford MIT EPFL Berkeley Cloud Providers RedHat Rackspace Qualcomm Microsoft Mellanox Google Intel Canonical Cisco CenturyLink Broadcom ARM Industry Tradeworx CloudSpectator CloudHarmony

5 Used in Production for Performance Regression Side-by-side reports...

6 PerfKit Benchmarker: Benchmarks Workloads Disk CPU Network Varied - Aerospike - Bonnie++ - Coremark - Iperf - Cluster Boot - Cassandra - Copy - Spec CPU Mesh - Object Storage - Hadoop - Fio - Network - Sysbench OLTP - HPCC - Synthetic - Netperf - Unixbench - MongoDB Storage - Ping - Oldisim - Redis

7 A User Perspective

8 A New Definition of Performance Measurement Step 1: Setup Step 2: Warm up Step 3: Pre-execute Step 4: Execute Step 5: Post-Execute Step 6: Cleanup Step 7: Publish results Report this as the result

9 A New Definition of Performance Measurement Cloud is *** ELASTIC *** Report these. Step 1: Setup Step 2: Warm up Step 3: Pre-execute Step 4: Execute Step 5: Post-Execute Step 6: Cleanup Step 7: Publish results This is OLD thinking! Report this as the result

10 VM has high Machine to Machine Variation Slow leaf effect is exaggerated in cloud! 20% Variation

11 OLDIsim: The Scaleout Benchmark in PKB

12 Workloads are becoming more Diverse Google fleet by courtesy of Svilen Kanev

13 Datacenter Tax and Virtualization Tax? Virtualization by courtesy of Svilen Kanev

14 Maintenance and Governance

15 Framework and Benchmarks Official benchmarks, kept on public github Benchmarker Framework Default Set Benchmarks Trash Benchmarks are promoted and demoted Private named benchmark set Partner 1 Set Benchmarks Partner 2 Set Benchmarks Partner n Set Benchmarks Example implementation: $ pkb --benchmarks=googleset --cloud=gcp --zone=us-central1-a...

16 Roles and responsibilities Framework engineering and maintenance (committers) Owner: Google until Jan 2016 Issue triage and fixes, improve existing benchmarks, better reporting, etc Coordinate pull requests, code style, and minimum test coverage New maintainer is elected by contributors in Dec 2015 Official Benchmarks and settings (management committee) Owner: Stanford, MIT Quarterly definition of what benchmarks run by default and their settings. Manage named benchmark sets (might also own one named set each) Decision by enlightened dictatorship every quarter Extended Benchmarks and settings (contributors) Owner: Contributors willing to sign up to the project in public Define a named super/sub set of benchmarks and optimized settings Independent decision making

17 How you can get involved? Join Work with us on the tools (via GitHub) Help us gain adoption with industry (+you) Suggest/add benchmarks for representative complex workloads Evolve (include/exclude) benchmarks as the market evolves

18 Thank you! Questions? Liqun Cheng Performance Engineer Google Images by Connie Zhou

19 Appendix

20 Important Design Point The framework runs like a developer. It leverages the CLI tools rather than Cloud API s. All steps are logged, so execution is fully auditable.

21 PerfKit Benchmarker: Example Output $./pkb.py --project=google.com:voellm --benchmarks=iperf --cloud=gcp --zones=us-central1-a,us-central1-b :43:37,105 INFO Verbose logging to: /tmp/perfkitbenchmarker/run_8f31ef02/pkb.log :43:37,234 INFO Preparing benchmark iperf :43:37,236 INFO Running: gcloud compute disks create perfkit-8f31ef size 10 --type pd-standard --project google.com:voellm --format json --quiet --zone us-central1-b --image debian-7-backports :43:37,237 INFO Running: gcloud compute disks create perfkit-8f31ef size 10 --type pd-standard --project google.com:voellm --format json --quiet --zone us-central1-a --image debian-7-backports :43:56,847 INFO Running: gcloud compute disks describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-a :43:57,273 INFO Running: gcloud compute disks describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-b :43:58,293 INFO Running: gcloud compute instances create perfkit-8f31ef disk name=perfkit-8f31ef02-0 boot=yes mode=rw --machine-type n1-standard-1 --tags=perfkitbenchmarker --maintenance-policy TERMINATE --metadata-from-file sshkeys=/tmp/perfkitbenchmarker/run_8f31ef02/key-metadatagexpwg --metadata owner=voellm --project google.com:voellm --format json --quiet --zone us-central1-a :43:58,738 INFO Running: gcloud compute instances create perfkit-8f31ef disk name=perfkit-8f31ef02-1 boot=yes mode=rw --machine-type n1-standard-1 --tags=perfkitbenchmarker --maintenance-policy TERMINATE --metadata-from-file sshkeys=/tmp/perfkitbenchmarker/run_8f31ef02/key-metadatam8d7f8 --metadata owner=voellm --project google.com:voellm --format json --quiet --zone us-central1-b :44:12,694 INFO Running: gcloud compute instances describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-b :44:13,895 INFO Running: gcloud compute instances describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-b :44:15,075 INFO VM: :44:15,075 INFO Waiting for boot completion :44:15,075 INFO Running: gcloud compute firewall-rules create perfkit-firewall-8f31ef allow tcp:22 udp:22 --project google.com:voellm --format json --quiet :44:17,664 INFO Running: gcloud compute instances describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-a :44:19,223 INFO Running: gcloud compute instances describe perfkit-8f31ef project google.com:voellm --format json --quiet --zone uscentral1-a :44:20,775 INFO VM: :44:20,776 INFO Waiting for boot completion....

22 PerfKit Explorer: Analytics Dashboards Controls for the Widget Data and Transform Controls for the Dashboard and Widget Display Query Editor for the power user!

23 PerfKit Benchmarker: Stages Prepare Run Cleanup This stage is where all resources like machines are provisioned for measurement. Allows a benchmark to be run multiple times. Great for A/B testing before/after flag flip --metadata= testa is used for tagging a run This stage is used to teardown all resources.

24 PerfKit Benchmarker: Example Runs Example: Measuring boot time./pkb.py --project=google.com:voellm --cloud=gcp -- benchmarks='cluster boot' Example: Measuring VM to VM performance cross zone./pkb.py --project=google.com:voellm --cloud=gcp --zones=uscentral1-a,us-central1-b --benchmarks='iperf' Example: Running in stages Flags for stages: [--run_stage=prepare run publish cleanup -- run_uri=<id>]

25 How to Propose a Change This can optionally done within a virtualenv. 1. Clone the repository: git clone git@github.com: GoogleCloudPlatform/PerfKitBenchmarker.git -b dev && cd PerfKitBenchmarker 2. Install Python Install pip: sudo apt-get install -y python-pip 4. Install PerfKitBenchmarker python dependencies: [sudo] pip install -r requirements.txt 5. Install PerfKitBenchmarker test dependencies: [sudo] pip install -r test-requirements.txt 6. Install git pre-commit hooks, to enable linting and copyright checks: hooks/install.sh

26 How to Propose a Change 1. Start from dev (git checkout dev). 2. Create a branch for your changes (git checkout -b <your-branch-name>). 3. Make your modifications to the code in one or more commits. 4. Push your changes to GitHub (git push origin) 5. Create a pull-request to integrate your change into dev. This can be done on github.com, or by installing and using the command line tool gh a. gh pull-request -h <your-branch-name> -b dev