scc: Cluster Storage Provisioning Informed by Application Characteristics and SLAs

Transcription

1 scc: Cluster Storage Provisioning Informed by Application Characteristics and SLAs Harsha V. Madhyastha*, John C. McCullough, George Porter, Rishi Kapoor, Stefan Savage, Alex C. Snoeren, and Amin Vahdat UC Riverside* and UC San Diego Bourns College of Engineering 1

2 Provisioning Hardware for Cluster Applications 2

3 Provisioning Hardware for Cluster Applications Need 100 Queries/s 2

4 Provisioning Hardware for Cluster Applications Need 100 Queries/s 2

5 Provisioning Hardware for Cluster Applications Need 100 Queries/s Need 1000 Views/s 2

6 Provisioning Hardware for Cluster Applications Need 100 Queries/s Need 1000 Views/s 2

7 Goals for Provisioning at Low-cost High performance Redundancy Supporting multi-tenancy High availability... Our focus: reach performance goal and minimize cost for a single application, emphasizing storage 3

8 Challenge: Large Configuration Space Diverse server enclosures/architectures Diverse storage options 7.2k-rpm 15k-rpm SSD DRAM Size MB/s (r/w) IOPS Cost 500GB 90/90 125/125 $ GB 150/ /285 $296 32GB 250/ /1000 $496 1GB 13k/13k 1.6B/1.6B $36 4

9 Challenge: Large Configuration Space Diverse server enclosures/architectures Diverse storage options 7.2k-rpm 15k-rpm SSD DRAM Current state-of-the-art: Size MB/s (r/w) IOPS Cost 500GB 90/90 125/125 $ GB 150/ /285 $296 32GB 250/ /1000 $496 1GB 13k/13k 1.6B/1.6B $36 4

10 Challenge: Large Configuration Space Diverse server enclosures/architectures Diverse storage options 7.2k-rpm 15k-rpm SSD DRAM Size MB/s (r/w) IOPS Cost 500GB 90/90 125/125 $ GB 150/ /285 $296 32GB 250/ /1000 $496 1GB 13k/13k 1.6B/1.6B $36 Current state-of-the-art: Apply rules-of-thumb from experience Trial-and-error with various configurations Configuration duplicated to scale-out 4

11 Scale-out Shortcomings for Photo-sharing Application 5

12 Scale-out Shortcomings for Photo-sharing Application 4x 5

13 Scale-out Shortcomings for Photo-sharing Application 4x Insight: Match storage to workload, leverage heterogeneity 5

14 Goal: Understand Configuration Space What is a low-cost configuration now? What will low-cost configurations look like in the future? Measure in-the-small Model Application Performance Predict in-the-large 6

15 scc: Storage Configuration Compiler scc 7

16 scc: Storage Configuration Compiler Cluster Building-Blocks scc 7

17 scc: Storage Configuration Compiler Cluster Building-Blocks Application Model scc 7

18 scc: Storage Configuration Compiler Cluster Building-Blocks Application Model scc SLA Specification 7

19 scc: Storage Configuration Compiler Cluster Building-Blocks Application Model scc SLA Cost Specification SLA 7

20 scc: Storage Configuration Compiler Cluster Building-Blocks Application Model scc SLA Cost Specification SLA 7

21 Outline Modeling Applications and Hardware Computing low-cost configurations Example Validation Applications of scc 8

22 Cluster Building Blocks Many types of servers 9

23 Cluster Building Blocks CPU Many types of servers Cores 9

24 Cluster Building Blocks CPU RAM Many types of servers Cores RAM 9

25 Cluster Building Blocks CPU HDD RAM HDD Many types of servers Cores RAM Storage 9

26 Cluster Building Blocks CPU HDD RAM HDD Many types of servers Cores RAM Storage I/O & Network 9

27 Application Model Breakdown application into: Tasks (Computation) Datasets (Storage) Edges between Tasks and Datasets (I/O) Edges among Tasks (dependencies) 10

28 Example Model: Photo-Sharing Tasks Datasets 11

29 Example Model: Photo-Sharing Tasks Datasets Photos Thumbnails (Tag, Photo) 11

30 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails (Tag, Photo) 11

31 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails (Tag, Photo) 11

32 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails (Tag, Photo) 11

33 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo (Tag, Photo) 11

34 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo (Tag, Photo) 11

35 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo View Tag (Tag, Photo) 11

36 Example Model: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo View Tag (Tag, Photo) 11

37 Example Model: Photo-Sharing Tasks Datasets Photo Upload Thumbnail 5ms 100ms Photos Write Tags View Photo View Tag 1ms 10ms 10ms Thumbnails (Tag, Photo) 11

38 Example Model: Photo-Sharing Tasks Datasets Photo Upload 5ms 1x200kB Thumbnail 100ms 1x4kB Photos Write Tags 10ms Thumbnails 1ms 10x1kB View Photo 10ms (Tag, Photo) View Tag 11

39 Example Model: Photo-Sharing Tasks Datasets Photo Upload 5ms 1x200kB Thumbnail Write Tags 100ms 10ms 1x4kB 1x200kB Photos Thumbnails 1ms 10x1kB 10x4kB View Photo View Tag 10ms 10x1kB 10x1kB (Tag, Photo) 11

40 Example Model: Photo-Sharing Tasks Datasets Photo Upload 5ms 1x200kB Thumbnail Write Tags 1ms 100ms 10ms 1x4kB 10x1kB 1x200kB 10x4kB Photos 1TB Thumbnails 20GB View Photo View Tag 10ms 10x1kB 10x1kB (Tag, Photo) 2GB 11

41 Photo-Sharing SLA Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo View Tag (Tag, Photo) 12

42 Photo-Sharing SLA Tasks Datasets Photo Upload Photos Tasks/s Thumbnail Write Tags View Photo View Tag Thumbnails (Tag, Photo) 12

43 Outline Modeling Applications and Hardware Computing low-cost configurations Example Validation Applications of scc 13

44 Navigating the Configuration Space Cost SLA 14

45 Navigating the Configuration Space Cost SLA 14

46 Navigating the Configuration Space Cost SLA 14

47 Navigating the Configuration Space Cost SLA 14

48 Navigating the Configuration Space Cost SLA 14

49 Guiding Principle to Meet SLA Complex interaction across storage-type and dataset assignments Need to consider costs of meeting SLA for each permutation Our configuration space is: D datasets, S storage-types S D configs 15

50 Meeting the SLA at Low-cost Photos Thumbs Tags 16

51 Meeting the SLA at Low-cost Photos Thumbs Tags 7.2krpm 7.2krpm 7.2krpm 16

52 Meeting the SLA at Low-cost Photos Thumbs Tags 7.2krpm 7.2krpm 7.2krpm # storage units per-dataset 16

53 Meeting the SLA at Low-cost Photos Thumbs Tags 7.2krpm 7.2krpm 7.2krpm # storage units per-dataset Assign CPUs for computation 16

54 Meeting the SLA at Low-cost Photos Thumbs Tags 7.2krpm 7.2krpm 7.2krpm # storage units per-dataset Assign CPUs for computation Pack into servers 16

55 Meeting the SLA at Low-cost Photos Thumbs Tags 7.2krpm 7.2krpm 7.2krpm # storage units per-dataset Assign CPUs for computation Pack into servers Compute cost 16

56 Meeting the SLA at Low-cost Photos 7.2krpm 7.2krpm 7.2krpm 7.2krpm 7.2krpm Thumbs 7.2krpm 7.2krpm 7.2krpm 7.2krpm 7.2krpm... Tags 7.2krpm 15krpm SSD 7.2k+DRAM 15k+DRAM # storage units per-dataset Assign CPUs for computation Pack into servers Compute cost 16

57 Meeting the SLA at Low-cost Photos 7.2krpm 7.2krpm 7.2krpm 7.2krpm 7.2krpm Thumbs 7.2krpm 7.2krpm 7.2krpm 7.2krpm 7.2krpm... Tags 7.2krpm 15krpm SSD 7.2k+DRAM 15k+DRAM Find minimum cost # storage units per-dataset Assign CPUs for computation Pack into servers Compute cost 16

58 Detail: How Many Storage Units? # of storage units per-dataset Assign CPUs for computation Pack into servers Compute cost 17

59 Detail: How Many Storage Units? # of storage units per-dataset SLA+Model=Requirements Need to match requirements to storage units Assign CPUs for computation Pack into servers Compute cost 17

60 Detail: How Many Storage Units? # of storage units per-dataset SLA+Model=Requirements Need to match requirements to storage units Assign CPUs for computation Pack into servers Compute cost Capacity IOPS Write-heavy: Short-stroke disks Read-heavy: Consider RAM for caching 17

61 Detail: How Many CPUs? # of storage units per-dataset Assign CPUs for computation Enough CPUs to satisfy computation requirements Pack into servers Compute cost Linearly extrapolate computation time Minimum one core/server (Details in paper) 18

62 Detail: Fits in how many servers? # of storage units per-dataset Assign CPUs for computation Pack into servers Compute cost 19

63 Detail: Fits in how many servers? # of storage units per-dataset Integer Linear Programming Multiple server types, each with different constraints: I/O Bus Bandwidth I/O Slots CPU Cores Network Bandwidth Assign CPUs for computation Pack into servers Compute cost 19

64 Detail: Fits in how many servers? # of storage units per-dataset Integer Linear Programming Multiple server types, each with different constraints: I/O Bus Bandwidth I/O Slots CPU Cores Network Bandwidth Assign CPUs for computation Pack into servers Compute cost Sequential-workload Bus/Network constrained IOPS-workload Slot capacity constrained 19

65 Detail: How much does it cost? # of storage units per-dataset ILP minimizes cost: Capital expenses Assign CPUs for computation Pack into servers Compute cost Each component Operation expenses Power & cooling 20

66 Outline Modeling Applications and Hardware Computing low-cost configurations Example Validation Applications of scc 21

67 Example Application: Photo-Sharing Tasks Datasets Photo Upload Photos Thumbnail Write Tags Thumbnails View Photo View Tag (Tag, Photo) 22

68 Example: Building Blocks Building blocks 3GHZ Intel Xeon (max 4 cores) 146GB 15k-rpm Disks } 500GB 7.2k-rpm Disks (max 32GB SSD 1 GB RAM (max 15 per machine) Gigabit Ethernet 4 per machine) 23

69 Example: SLA SLA Requirement 100 image uploads/s 300 photo views/s 100 tags views/s 24

70 Example: Final Stages for Photo-sharing Photos Thumbnails Tags 15k-rpm SSD 7200rpm w/dram 15k-rpm 15k-rpm WWW 15k-rpm CPU CPU RAM CPU CPU RAM 25

71 Example: Final Stages for Photo-sharing Photos Thumbnails Tags 15k-rpm SSD 7200rpm w/dram 15k-rpm 15k-rpm WWW 15k-rpm CPU CPU RAM CPU CPU RAM 25

72 Example: Final Stages for Photo-sharing Photos Thumbnails Tags 15k-rpm SSD 7200rpm w/dram 15k-rpm 15k-rpm WWW 15k-rpm CPU CPU RAM CPU CPU RAM 25

73 Example: Final Stages for Photo-sharing Photos Thumbnails Tags CPU 15k-rpm 15k-rpm SSD 7200rpm w/dram RAM 15k-rpm WWW 15k-rpm CPU CPU RAM CPU RAM CPU CPU RAM 25

74 Example: Final Stages for Photo-sharing Photos Thumbnails Tags CPU 15k-rpm 15k-rpm SSD 7200rpm w/dram RAM 15k-rpm WWW 15k-rpm CPU CPU RAM CPU RAM CPU CPU RAM 25

75 Outline Modeling Applications and Hardware Computing low-cost configurations Example Validation Applications of scc 26

76 Experimental Methodology Built three applications Photo-sharing Product Search Terasort Micro-benchmarked each to create model Deployed scc output to cluster of machines 27

77 Validation: scc meets SLA at lower cost for Photo-sharing 28

78 Validation: scc meets SLA at lower cost for Photo-sharing 28

79 Validation: scc meets SLA at lower cost for Photo-sharing 28

80 Validation: scc meets SLA at lower cost for Photo-sharing 28

81 Validation: scc meets SLA at lower cost for Photo-sharing 28

82 Validation: scc meets SLA at lower cost for Photo-sharing 28

83 Validation: scc meets SLA at lower cost for Product Search 29

84 Validation: scc meets SLA at lower cost for Terasort 30

85 Validation: scc meets SLA at lower cost for Terasort Best Practice 30

86 Validation: scc meets SLA at lower cost for Terasort Best Practice 30

87 Outline Modeling Applications and Hardware Computing low-cost configurations Experimental methodology Validation Applications of scc 31

88 Storage Type Regimes Cost SLA Uploads/s Photos Thumbnails Tags <5 Disk Disk Disk 5-25 Disk Disk Disk SSD SSD Disk w/ DRAM Disk w/ DRAM Disk w/ DRAM Disk w/ DRAM k Disk w/ DRAM Disk w/ DRAM Disk w/ DRAM 32

89 Output Sensitivity to Model Parameters Attribute Range with same architecture Photo size 50kB 200kB 850kB Thumbnail size 1kB 4kB 30kB SSD unit price $200 $450 $900 33

90 Output Sensitivity to Model Parameters Attribute Range with same architecture Photo size 50kB 200kB 850kB Thumbnail size 1kB 4kB 30kB SSD unit price $200 $450 $900 Dataset Photos Thumbnails Tags Sensitive to what hardware cost? 20% drop in 7.2k-rpm drive price 92% drop in DRAM price 31% drop in 15k-rpm drive price 33

91 Conclusion 34

92 Conclusion Our scc tool finds low-cost cluster configurations Saves 2-4.5x over simple scale-out for Photo-share and Product Search 16% lower cost for Terasort vs. Map Reduce best-practice configuration 34

93 Conclusion Our scc tool finds low-cost cluster configurations Saves 2-4.5x over simple scale-out for Photo-share and Product Search 16% lower cost for Terasort vs. Map Reduce best-practice configuration Better matches hardware to application requirements 34

94 Conclusion Our scc tool finds low-cost cluster configurations Saves 2-4.5x over simple scale-out for Photo-share and Product Search 16% lower cost for Terasort vs. Map Reduce best-practice configuration Better matches hardware to application requirements Useful for predicting output sensitivity and future needs 34

95 Conclusion Our scc tool finds low-cost cluster configurations Saves 2-4.5x over simple scale-out for Photo-share and Product Search 16% lower cost for Terasort vs. Map Reduce best-practice configuration Better matches hardware to application requirements Useful for predicting output sensitivity and future needs Future work More precise network models Cloud deployment 34

96 Questions? Cluster Building-Blocks Application Model SLA Specification scc Cost SLA Bourns College of Engineering 35