D3N: A multi-layer cache for data centers with imbalanced networks

Transcription

1 D3N: A multi-layer cache for data centers with imbalanced networks Emine Ugur Kaynar *, Mohammad Hossein Hajkazemi, Mania Abdi, Ata Turk *, Raja R. Sambasivan *, Larry Rudolph, Peter Desnoyers, Orran Krieger *, Matt Benjamin+, Ali Maredia+ *Boston University, Northeastern University, Two Sigma, +Redhat

2 Problem A typical private datacenter with a datalake has network congestion & high latency fetching data from the datalake. Solution Multi-layer cache architecture. Strategically placed shared object cache. Main idea: Cache data on the access side of bottlenecks.

3 D3N s Architecture Features:

4 D3N s Architecture Features: Rack-local cache servers

5 D3N s Architecture Features: Rack-local cache servers Cache services

6 D3N s Architecture Features: Rack-local cache servers Cache services

7 D3N s Architecture Features: Rack-local cache servers Cache services Multiple cache layers across the network hierarchy Cooperative caching

8 D3N s Architecture Features: Rack-local cache servers Cache services Multiple cache layers across the network hierarchy Cooperative caching

9 D3N s Architecture Features: Rack-local cache servers Cache services Multiple cache layers across the network hierarchy Cooperative caching

10 D3N s Architecture Features: Rack-local cache servers Cache services Multiple cache layers across the network hierarchy Cooperative caching

11 Implementation Modification to Ceph s RADOS gateway. Upstream the code. S3/Swift-compatible object interface. Incorporates L1 and L2 cache. Cached data is striped across the NVMe-SSDs.

12 Earlier Results Realistic Workloads Value of Multi-level Micro Benchmarks

13 Earlier Results Realistic Workloads Performance improved by 3x compared to Vanilla Rados Gateway. Value of Multi-level Multi-layer provides higher throughput than single layer cache. Micro Benchmarks D3N saturates NVMe SSDs and 40 GbE NICs Read throughput is increased by 5x. Write cache saturates the write bandwidth of the dual NVMe SSDs.

14 How to partition cache space across L1 and L2? Layer 2 The algorithm partitions the cache space based on: Access Pattern Network Congestion Layer 1

15 How to partition cache space across L1 and L2? Layer 2 Layer 1 High cluster locality Congestion to storage network

16 How to partition cache space across L1 and L2? Layer 2 Layer 1 High rack locality Congestion within the cluster network

17 Dynamic Cache Size Management The algorithm calculates the reuse distance histogram mean miss latency D3N periodically runs the algorithm.

18 Evaluation Implemented a trace-based simulator. Use the Facebook s hadoop traces. Simulated a datacenter with 10 racks and a fat tree topology. Compared the performance of the dynamic algorithm with the static allocation (50/50).

19 Adaptability to different access patterns

20 Adaptability to network load changes Dynamic Static

21 Performance under different locality levels

22 Future Works Eviction / admission and prefetching policies for a multi-level cache. Characterization of the access patterns to object stores. Fair sharing and cache prioritization. Caching intermediate datasets using write-back cache. Project Webpage

23 Collaboration with RedHat Red Hat s collaboration in D3N is one aspect of a larger collaboration around Ceph data architectures for scientific computing and collaborations in cloud, in collaboration with Mass Open Cloud (MOC). Portions of the work (e.g., RGW write-back caching) extend prior collaborative work between the experimenters and Red Hat in 2017.

24 Collaboration with RedHat Red Hat Ceph engineering sees significant potential in the D3N architecture to accelerate analytics workloads using the S3A protocol. In particular: Impact of workload-guided prefetching and adaptation of S3A clients to guide prefetching General impact of intermediate caching and write-back caching on analytics workload performance using S3A

25 Collaboration with RedHat Red Hat Ceph engineering are working with the experimenters in several areas, concentrating on application to S3A data analytics workloads and data lake, and are working with selected customers as well as Red Hat s internal big data centers of excellence to develop relevant workload profiles and traces for evaluation.