A Case for Packing and Indexing in Cloud File Systems Saurabh Kadekodi, Bin Fan*, Adit Madan*, Garth Gibson and Greg Ganger University, *Alluxio Inc.
In a nutshell Cloud object stores have a per-operation pricing structure Small object workloads have poor performance and high cost Packing Performance Price http://www.pdl.cmu.edu/!2
Cloud file systems Client S3, Azure, GCS AWS, GCE, etc. Object store Natural approach is 1:1 file to object mapping; bad for small files Packing (batching or coalescing) small files improves local FS performance How much does packing help in cloud file systems? http://www.pdl.cmu.edu/!3
Motivation Performance Price http://www.pdl.cmu.edu/!4
Performance Motivation Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 r4.4xlarge r4.4xlarge r4.4xlarge r4.4xlarge 4KiB objects 1 object per 1 file to S3 3.2M objects of 4KiB each = 12.5GiB http://www.pdl.cmu.edu/!5
Performance Motivation Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 r4.4xlarge r4.4xlarge r4.4xlarge r4.4xlarge 4MiB objects 1 object per 1K files to S3 3.2K objects of 4MiB each = 12.5GiB http://www.pdl.cmu.edu/!5
Performance Motivation Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 r4.4xlarge r4.4xlarge r4.4xlarge r4.4xlarge 40MiB objects 1 object per 10K files to S3 320 objects of 40MiB each = 12.5GiB http://www.pdl.cmu.edu/!5
Performance Motivation Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 r4.4xlarge r4.4xlarge r4.4xlarge r4.4xlarge 400MiB objects 1 object per 100K files to S3 32 objects of 400MiB each = 12.5GiB http://www.pdl.cmu.edu/!5
Throughput By Object Size Throughput (MB / s) 500 375 250 125 0.42 0 4KB (1:1) 4MB (1K:1) 40MB (10K:1) 400MB (100K:1) 355.5 Object Size 457.1 492.3 http://www.pdl.cmu.edu/!6
S3 Throttling in Action PUT req / sec PUT Req / min 166 10000 1.66 100 333 83 20000 5000 S3 Exceptions / min S3 Exceptions / sec 02:45 02:50 02:55 03:00 03:05 03:10 03:15 03:20 03:25 03:30 03:35 03:40 03:45 03:50 03:55 04:00 04:05 04:10 04:15 04:20 Time (hh:mm) Request rate throttling by S3 for 4KiB writes S3 warns routinely requesting >100 PUT req / s subject to throttling NW bandwidth of r4.4xlarge instances ~ 10 Gb / sec minimum 13MB writes to avoid being throttled packing http://www.pdl.cmu.edu/!7
Motivation Performance Price http://www.pdl.cmu.edu/!8
Price Motivation S3 PUT, COPY, POST GET Data Retrieval Data Storage Standard $0.05 / 10K req $0.04 / 100K req Free (for certain data center locations) $0.023 / GB-month 1400 Operation cost Storage cost Say we want to store 1TB per month as 4KiB objects, approx 100 req/s for a month Price ($) 1050 700 350 Percentage of operation costs 98% Nil 16% <0.001% 0 S3 EFS DynamoDB Packing Data ingest methodology http://www.pdl.cmu.edu/
Price Motivation S3 PUT, COPY, POST GET Data Retrieval Data Storage Standard $0.05 / 10K req $0.04 / 100K req Free (for certain data center locations) $0.023 / GB-month 1400 Operation cost Storage cost Say we want to store 1TB per month as 4KiB objects, approx 100 req/s for a month Price ($) 1050 700 350 Percentage of data storage costs 2% 100% 84% >99.999% 0 S3 EFS DynamoDB Packing Data ingest methodology http://www.pdl.cmu.edu/
Price Motivation S3 PUT, COPY, POST GET Data Retrieval Data Storage Standard $0.05 / 10K req $0.04 / 100K req Free (for certain data center locations) $0.023 / GB-month 1400 Operation cost Storage cost Say we want to store 1TB per month as 4KiB objects, approx 100 req/s for a month Price ($) 1050 700 350 Percentage of data storage costs Packing cost split almost equal to EFS, but EFS storage cost >10x S3 storage costs 2% 100% 84% >99.999% 0 S3 EFS DynamoDB Packing Data ingest methodology http://www.pdl.cmu.edu/
Packing in Alluxio Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 to S3 http://www.pdl.cmu.edu/!10
Packing in Alluxio AWS Master Packed blob Alluxio Worker 1 Worker 2 Worker 3 Buffer for packing Worker 4 Index to S3 http://www.pdl.cmu.edu/!10
Packing in Alluxio Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 to S3 http://www.pdl.cmu.edu/!10
Packing in Alluxio Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 to S3 http://www.pdl.cmu.edu/!10
Packing in Alluxio Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 to S3 http://www.pdl.cmu.edu/!10
A Packed Blob Packing policy determines what to pack Triggered by dirty bytes & timeout Embedded Index blob-extent 1 blob-extent 2 blob-extent k Blob Extent: alluxio-path:logical-offset:physical-offset:length http://www.pdl.cmu.edu/!11
Evaluation http://www.pdl.cmu.edu/!12
Configuration Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 http://www.pdl.cmu.edu/!13
Configuration Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 http://www.pdl.cmu.edu/!13
Configuration AWS Master r4.4xlarge EC2 instance Alluxio Worker 1 Worker 2 Worker 3 Worker 4 http://www.pdl.cmu.edu/!13
Configuration AWS Master r4.4xlarge EC2 instance Alluxio 8 workload generators Worker 1 Worker 2 Worker 3 Worker 4 http://www.pdl.cmu.edu/!13
Configuration AWS Master r4.4xlarge EC2 instance Alluxio 8 workload generators Worker 1 Worker 2 Worker 3 Worker 4 100K files; 8KB each http://www.pdl.cmu.edu/!13
Configuration Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 http://www.pdl.cmu.edu/!13
Configuration Alluxio AWS Master Worker 1 Worker 2 Worker 3 Worker 4 800K files 800K files 800K files 800K files 3.2M files of 8KiB = 24.4GB http://www.pdl.cmu.edu/!13
Packing Configuration Max blob size: 1 GB Packing interval: 5 sec # Packing threads: 16 Index # Master threads: 16 Backup interval: 1 min http://www.pdl.cmu.edu/!14
Motivation Revisited Performance Price http://www.pdl.cmu.edu/!15
Write Performance Comparison 100 71.5 Direct 10000 Packed 21780 60x 30 22.5 Data Metadata 0.9 MB / s 10 60x Seconds 100 355 GB 15 24.4 24.4 7.5 1 1.2 Throughput Comparison 1 Runtime Comparison http://www.pdl.cmu.edu/!16 0 Direct Packing
Motivation Revisited Performance Price http://www.pdl.cmu.edu/!17
S3 Data Ingest Price 10000 1729 1000000 257820 25000x Direct Packed 1 1000 0 0.00067 Data Ingest Cost 1 Rate-Limited Retries Request rate is throttled much more than data rate http://www.pdl.cmu.edu/!18
Research questions (feedback) 1 Price with packing = price without packing 25000 Will fine-tuning help? By how much? Are there workload specific tuning opportunities / challenges? Garbage collection of packed blobs Different from LSM Trees, LFS, TableFS, etc. Cost-driven GC policies? Interference with foreground workload? http://www.pdl.cmu.edu/!19
Conclusion Packing Performance Price Price reduction because of: Matching application write sizes to storage system write sizes Elimination of retries due to aggressive rate-limiting imposed by S3-like services saukad@cs.cmu.edu http://www.pdl.cmu.edu/!20