VIDAS: OBJECT-BASED VIRTUALIZED D A T A S H A R I N G F O R H I G H PERFORMANCE STORAGE I/O

Size: px

Start display at page:

Download "VIDAS: OBJECT-BASED VIRTUALIZED D A T A S H A R I N G F O R H I G H PERFORMANCE STORAGE I/O"

Arthur Bryant
6 years ago
Views:

1 Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas VIDAS: OBJECT-BASED VIRTUALIZED D A T A S H A R I N G F O R H I G H PERFORMANCE STORAGE I/O Pablo Llopis, Javier Garcia Blas, Florin Isaila, Jesus Carretero Computer Architecture and Technology Group (ARCOS) University Carlos III of Madrid, Spain ScienceCloud New York City, June 2013

2 2 Agenda 1. Motivation 2. Goals and common problems 3. Common storage I/O virtualization solutions 4. VIDAS Design and Implementation 5. Evaluation 6. Conclusion

3 3 Motivation HPC in virtualized clouds gaining popularity Increasing amount of data stresses importance of I/O performance I/O overhead in virtualized environments is still significant Agreement that POSIX not suitable for high performance Virtualized environments trade off protection for performance Lack of I/O coordination mechanisms across domains

4 4 Goals and contributions Propose new node-level abstractions and mechanisms in virtualized environments which: Enable building efficient virtualized data sharing Coordinate I/O across domains Provide shared access spaces across node-local domains Relax POSIX consistency Allow flexible data write and data read policies Expose data locality

5 5 Agenda 1. Motivation 2. Goals and common problems 3.Common storage I/O virtualization solutions 3.1. Block-level solution 3.2. Filesystem-level solution 4. VIDAS Design and Implementation 5. Evaluation 6. Conclusion

6 6 Virtualized block device drivers A1 A2 A3 Device Driver Hypervisor Device Driver Hypervisor Buffer Cache Scheduler Device Driver Front-end Hypervisor Device Driver Back-end Device Driver Device Driver Device Driver Host Host Host A1 virtualized block-level device on top of a physical device driver A2 virtualized block-level device on top of filesystem A3 paravirtualized device drivers

7 7 Virtualized filesystems B1 B2 B3 Buffer Cache Scheduler Device Driver Hypervisor Buffer Cache Scheduler Device Driver Host Buffer Cache Scheduler Device Driver Front-end Hypervisor Device Driver Back-end Device Driver Host Paravirtualized filesystem front-end Hypervisor Paravirtualized filesystem back-end Scheduler Device Driver Host B1 virtualized filesystem within a file (stacked filesystems) B2 paravirtualized filesystem B3 sophisticated paravirtualized filesystem, inter-domain coordination

8 8 Virtualized filesystems B1 B2 B3 Buffer Cache Scheduler Device Driver Hypervisor Buffer Cache Scheduler Device Driver Host Buffer Cache Scheduler Device Driver Front-end Hypervisor Device Driver Back-end Device Driver Host Paravirtualized filesystem front-end Hypervisor Paravirtualized filesystem back-end Scheduler Device Driver Host B1 virtualized filesystem within a file (stacked filesystems) B2 paravirtualized filesystem B3 sophisticated paravirtualized filesystem, inter-domain coordination

9 9 Agenda 1. Motivation 2. Goals and common problems 3.Common storage I/O virtualization solutions 4. VIDAS Design and Implementation 4.1. Abstractions 4.2. Design 4.3. Implementation 5. Evaluation

10 10 VIDAS Guest 0 Guest 1 Guest N-1 Storage

11 10 VIDAS Guest 0 Guest 1 Guest N-1 Object 0 Object 1 Object 2 Object M-1 Storage

12 11 Abstractions Objects Containers Guest 0 Guest 1 Guest N-1 Object 0 Object 1 Object 2 Object M-1 write update offset0, sz0 offset0, size0

13 12 Containers Containers are access control domains which allow to share objects among a set of guests. Guest 0 Guest 1 Guest N-1 Object 0 Object 1 Object 2 Object M-1 write update offset0, sz0 offset0, size0

14 13 Objects: POSIX differences Strong consistency not enforced, but optional. Data writes and updates guided by configurable policy: write-through or write-back Provide locality awareness Guest 0 Guest 1 Guest N-1 Object 0 Object 1 Object 2 Object M-1 Objects are uniquely associated to an external storage resource through its name offset0, sz0 write update offset0, size0

15 14 API Overview Container operations int container create(char *name, int domain ids[]) int container destroy(char* name) int container attach(char *name) int container leave(char* name) Object data operations int obj write(obj handle t o, char *buf, size t o set, size t sz) int obj read(obj handle t o, char *buf, size t o set, size t sz) int obj flush(obj handle t o) int obj update(obj handle t o) Object metadata operations obj handle t object create(char* ext storage rsc, size t o set, size t size, char* cname) obj handle t object join(char* ext storage rsc, size t o set, size t size, char* cname) int object get locality(char* ext storage rsc, obj handle t *objects[]) int object leave(obj handle t o) int object destroy(obj handle t o) int object getattr(obj handle t o, char *name, void *value, size t size) int object setattr(obj handle t o, char *name, void *value, size t size) Object synchronization operations int object wait(obj handle t o, char **bufp) int object notify(obj handle t o, char *buf)

16 15 Agenda 1. Motivation 2. Goals and common problems 3.Common storage I/O virtualization solutions 4. VIDAS Design 5. VIDAS Implementation 5.1. Inter-domain communication mechanisms 5.2. Implementation 6. Using VIDAS 7. Evaluation

17 16 Inter-domain communication Inter-domain data communication techniques Ring buffers Shared memory Domain A Domain 1 Domain 2 Ringbuffer Shared data Domain B Domain N

18 17 Inter-domain communication A high-efficient inter-domain data transferring system for virtual machines. (2009) pp Presented at the Proceedings of the 3rd International Conference on Ubiquitous Information Management and Communication, New York, NY, USA: ACM. A study of Inter-domain Communication Mechanisms on Xen-based Hosting Platforms. (2009). Diakthaté F., Perache, M., Namyst, R., & Jourdren, H. (2009). Efficient Shared Memory Message Passing for Inter-VM Communications. Euro-Par 2008 Workshop Huang, W., Koop, M. J., Gao, Q., & Panda, D. K. (2007). Virtual machine aware communication libraries for high performance computing (p. 9). Presented at the SC '07 Inter-domain socket communications supporting high performance and full binary compatibility on Xen (pp ). Presented at the Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, New York, NY, USA: ACM Wei Huang, Koop, M. J., & Panda, D. K. (2008). Efficient one-copy MPI shared memory communication in Virtual Machines (pp ). Presented at the 2008 IEEE International Conference on Cluster Computing (CLUSTER), IEEE. All of these optimize inter-vm data sharing and communication, but none of them work around the limitation of being able to share pages with only 2 domains.

19 18 Xen/Linux grant reference extension Memory page Hypervisors usually provide a mechanism to share a page with only 2 domains domains at once. Domain 1 Domain 2 Memory page In our work we extended Linux/Xen to provide user-space applications with the ability to share a page with any number of domains. Domain 2 Domain 2 Domain 1

20 19 Paravirtualized data sharing While most solutions use mainly ringbuffers for paravirtualization, VIDAS combines ringbuffers and multi-domain shared memory Point-to-point Collective VM A VM B VM A VM B I/O Stack I/O Stack PV device driver Ringbuffer PV device driver Ringbuffer Shared data object Buffer cache I/O stack, device drivers I/O stack, device drivers

20 Implementation object_create object_join object_destroy object_leave object_get_locality object_flush object_update container_create container_destroy Front-end object_getattr

21 20 Implementation object_create object_join object_destroy object_leave object_get_locality object_flush object_update container_create container_destroy Front-end object_getattr object_setattr object_write object_read object_notify object_wait Object data Object attributes Hypervisor Front-end Ring buffer transport Container index Object index Back-end Host Ring buffer transport

22 21 Implementation Front-end container_create object_create Front-end A container creator specifies which domains are allowed to access the objects within the container Hypervisor Container index Object index Ring buffer transport Back-end Host Ring buffer transport

23 22 Implementation container_attach object_join Another domain can then access the objects in a container by joining in Front-end Front-end Hypervisor Container index Object index Ring buffer transport Back-end Host Ring buffer transport

24 23 object_wait object_read Front-end Object data Object attributes Hypervisor Front-end Ring buffer transport Container index Object index Back-end Host Ring buffer transport

25 24 object_getattr object_setatt object_write object_notify Front-end Object data Object attributes Hypervisor Front-end Ring buffer transport Container index Object index Back-end Host Ring buffer transport

26 25 Agenda 1. Motivation 2. Goals and common problems 3.Common storage I/O virtualization solutions 4. VIDAS Design and Implementation 5. Evaluation 6. Conclusion

27 26 Evaluation CPU Intel Xeon 12 core RAM 64GB DDR3 synchronous 1333Mhz HDD 1TB Toshiba MK1002TS Linux LVM ext4 Hypervisor Xen 4.2 Privileged domain Linux Guest domains Linux Synthetic Benchmarks: Broadcast, Independent read, Collective r/w

28 27 Bandwidth: In-memory communication 600" Bandwidth)(MB/s)) 500" 400" 300" 200" 100" 0" 2" 4" 8" 16" Number)of)Virtual)Machines) MPI-Bcast" VIDAS-Bcast" ringbuffer- Bcast" MPI-Bcast: share 128MB of data using MPI_Bcast VIDAS-Bcast: share 128MB of data using VIDAS ringbuffer-bcast: transfer 128MB of data to each VM using Xen ringbuffer transports

29 28 Independent file read (512MB) Throughput)(MB/s)) 160" 140" 120" 100" 80" 60" 40" 20" 0" 1" 2" 4" 8" 16" Number)of)Virtual)Machines) MPI+IO"(NFS)" VIDAS"I/O+Bcast" MPI+IO"(PVFS2)" MPI-IO (NFS) Concurrent file read on a node-local NFS mount MPI-IO (PVFS2) Concurrent file read on a node-local PVFS2 mount VIDAS I/O+Bcast Load data into object, broadcast to all VMs

30 29 Collective I/O (Two-phase I/O) Throughput)(MB/s)) 160" 140" 120" 100" 80" 60" 40" 20" 0" Read))))))))))))))))))))))))))))))))))))))))))))))Write) 1" 2" 4" 8" 16" 1" 2" 4" 8" 16" Number)of)Virtual)Machines) MPI+IO"collec1ve"(NFS)" VIDAS"collec1ve" MPI+IO"collec1ve"(PVFS2)" Non-overlappingly interleaved strided vectors of 2MB blocks

31 30 Agenda 1. Motivation 2. Goals and common problems 3.Common storage I/O virtualization solutions 4. VIDAS Design and Implementation 5. Evaluation 6. Conclusion

32 31 Conclusion and Contributions I/O overhead in virtualized environments is still significant Reduce memory copy operations Reduce domain context switches Agreement that POSIX not suitable for high performance Relax POSIX consistency Control data write and update policies Control data locality Virtualized environments trade off protection for performance Created shared access spaces Reduced memory copy operations and context switches Lack of I/O coordination mechanisms across domains Coordinate storage I/O across domains Proposed new multi-domain data sharing mechanisms

33 32 Questions? Q?

Exploring I/O Virtualization Data paths for MPI Applications in a Cluster of VMs: A Networking Perspective

Exploring I/O Virtualization Data paths for MPI Applications in a Cluster of VMs: A Networking Perspective Anastassios Nanos, Georgios Goumas, and Nectarios Koziris Computing Systems Laboratory, National