Stergios Anastasiadis Department of Computer Science & Engineering University of Ioannina, Greece

Transcription

1 Stergios Anastasiadis Department of Computer Science & Engineering University of Ioannina, Greece City University London Wednesday, January 27 th, 2016

2 Motivation for storage research Two interesting cloud problems Host-side durable storage (Arion) Design, Implementation, Evaluation, Summary Multitenant access control (Dike) Design, Implementation, Evaluation, Summary About University of Ioannina Stergios Anastasiadis University of Ioannina 2

3 Amount of used data grows 40% annually 44 zettabyte (10 21 /2 70 B) total by 2020 (4.4ZB in 2013) 65% of sold storage is file-based [IDC 2012] 80% of shipped capacity (133EB) unstructured by 2017 [IDC 2014] Common data-intensive applications (2,456,359 messages sent/s) Text search (52,292 queries/s at Google) Streaming video (114,540 views/s at YouTube) Social networks (7,071 posts/s on Twitter) Digital TV (4K/4096x2160, UHDTV/3840x2160) 3

4 Different types of needs Root images for virtual machines (e.g., NAS/SAN, Azure BLOB) Large files for big data and structured data (e.g., Colossus, HDFS) Many files for high-performance computing (e.g., Ceph, Lustre) Heterogeneous workloads Potentially from multiple tenants Scale out Multiple tiers Horizontal partitioning Typical tiers client-side frontend caching tier backend storage Apps Client Apps Client Network Caching Tier Backend Servers Apps Client 4

5 Virtualization flexibility No sharing Translation overhead Semantic gap File sharing Improved performance Semantics awareness Compatibility limitations VM VM block Host VM VM Host VM VM file Host VM VM Host block or file block file file Storage Servers Storage Servers Block-based File-based Stergios Anastasiadis University of Ioannina 5

6 Performance Improved throughput Reduced latency Efficiency Lower server & network load Functionality No file sharing Overhead Translation overhead Metadata persistence Crash Consistency Semantic gap Ordering of requests Grouping of operations Strengths Weaknesses Stergios Anastasiadis University of Ioannina 6

7 Loss or corruption of updates to critical data Highly relevant in large-scale, multi-tier environments Mean time between failures inversely proportional to # machines Causes Mostly software bugs and operator or maintenance tasks Hardware contributes much less to service-level failures Solution Data replication at mid-tier/backend across multiple components Frontend Stateless for reduced cross-layer communication May lose recent data at client crash or reboot Recovery Point Objective (RPO) at several tens of seconds! 7

8 Amount (MB) Ceph (Red Hat) object-based scale-out file system Client-side, memory-based caching typical for performance Volatile client memory may cause data loss in case of failure! Filebench fileserver Writeback every 5s dirty data older than 30s (default) Unflushed data On average, 24.3MB of dirty data only in volatile memory Unflushed Dirty Data Ceph Time (s) 8

9 Key insights Strengthen data durability at the client of shared storage systems Improve application performance and resource efficiency Assume quality of host hardware on par with that of server Goals Interface: Sharing: Durability: Performance: Scalability: POSIX-like, file-based Native file-sharing within and across hosts Recent updates survive client reboots Sequential disk throughput for writes Scale-out backend servers 9

10 Distributed filesystem as backend Object-based Multiple data & metadata servers Virtualized or bare-metal client Multiple backend replicas Guest HOST Guest Local journal at the client side Separate journal per client Redo log for both data and metadata Redundant directly-attached storage Heterogeneous replication Local, log-based, limited amount of time journal device Hypervisor Object Storage Servers 10

11 Local journal device Attached to client at mount-time Commit Synchronously transfer data updates from memory to journal Periodically or by explicit flush request Writeback Copy data blocks from memory to filesystem servers Periodically, or under space pressure from memory or journal Written-back & invalidated pages Removed from journal RAM 1. write 3. writeback Client 2. commit Object Storage Servers HOST Journal 11

12 Normal operation: shared file access With lease tokens Normal operation: conflicting writes from different clients Checkpoint pending writes and invalidate related journal entries Revoke write token and disable local caching Failure handling: client network reconnection or reboot Acquire needed tokens Replay file updates only if journaled metadata newer than metadata fetched from the server 12

13 Atomicity Granularity of individual filesystem operations Who assigns timestamps to operations? The client in order to allow delayed transfer to servers Assuming time synchronization with servers (e.g., NTP, PTP) If a client that modified a file gets disconnected Assume the file modification has not made it yet to the servers Discard the file modification of the disconnected client if, during the disconnection, a different (connected) client Reads the original (rather than the modified) file at the server, OR Writes the original file at the server ( last writer wins ) 13

14 Prototype CephFS kernel-level client (Linux kernel v3.6.6) with Linux JBD2 Commit Include metadata attributes in the journal tags of the descriptor block Recovery Compare journaled metadata with that newly fetched from MDS Replay writes only for files not accessed after the journal commit CLIENT inode info block count start offset end offset checksum flags Descriptor block header tag 1 tag 2 Data blocks Commit block D D D C 14

15 Backend Servers Ceph v (3 OSDs, 1 MON, 1 MDS) 3GB RAM, 2x 300GB 15KRPM SAS disks, 2x quad-core x GHz Separate OSD journal device, replication factor 3 Virtualized client 2GB RAM, 2x VCPUs Journal size 2GB, 1s commit interval Host Xen v x 300GB 15KRPM SAS disks (RAID0), 2x quad-core x GHz Default flush timeouts 5s writeback interval 30s expiration interval 15

16 Amount (MB) Filebench fileserver workload Arion flushes dirty data to local journal every second Amount of vulnerable data in memory On average reduced from 24.3MB down to 5.4MB RPO (Recovery Point Objective) down to 1s Unflushed Dirty Data Ceph Arion Time (s) 16

17 Average Latency (ms) Cumulative Received Data (MB) Zipfian random writes 2-16KB w/ overwrites Arion-60 (writeback = expiration = 60s) reduces Write latency of Ceph (default) by 40% Network traffic received at OSD by 41% Duration of experiment by 36% Random Writes (Zipfian) Ceph Ceph-1 Ceph-sync Arion-60 Arion-inf 2,500 2,000 1,500 1, Network Load (OSD) Ceph Ceph-1 Arion-60 Arion-inf Ceph-sync Time (s) 17

18 Journal Device (%) Filesystem Device (%) Arion-60 reduces disk utilization at the servers Filesystem device utilization reduced by 76.5% wrt Ceph-1 (writeback = expiration = 1s) Disk Utilization (OSD) Disk Utilization (OSD) Ceph-1 Arion Time (s) Time (s) 18

19 Backend Servers Ceph v Separate OSD journal device 5GB Replication factor 3 Clients Journal size 5GB (RAID0) Commit interval 1s Writeback & expiration intervals extended to 120s AMI vcpu RAM (GiB) Storage Network m1.large (HDD) x420GB Moderate c3.large(ssd) x16GB Moderate 19

20 Throughput (ops/s) Throughput (ops/s) MySQL database and log files stored over shared storage Synchronous I/O HDD Setup Up to 92% improved operation throughput with 12 OSDs & 16 clients SSD Setup Up to 59% improved operation throughput with 12 OSDs & 8 clients OLTP (HDD) - 12 OSDs OLTP (SSD) - 12 OSDs Arion Ceph Arion Ceph Number of Clients Number of Clients 20

21 Higher client statefulness through Local journal added separately to each client at the host Benefits Improve host-side data durability Increase throughput and reduce latency at client Lower network & disk bandwidth consumption at servers Tunable control over Amount of dirty pages staged at the host Time period for dirty pages to reach the backend servers 21

22 Goal Storage infrastructure shared among different tenants Requirements Scalability: Isolation: Sharing: Compatibility: Manageability: Support enormous number of end users Isolate user identities and access control of tenants Flexible data sharing within or between tenants Compatibility with existing applications Flexible resource management Research focus Efficient and secure multitenancy in VM filesystems 22

23 Sharing co-located data resources Inflexible due to lack of appropriate sharing software architecture Shared filesystem namespace User identity crosstalk between tenants Complicated security enforcement UID: 1100 GID: 1000 UID: 1000 TENANT 1 Shared File System GID: 1000 UID: 2000 UID: 1000 UID: 1050 GID: 1000 UID: 1000 TENANT N Native multitenancy at the filesystem level Clean way to isolate multiple tenants Shared storage hardware and filesystem software Flexible isolation, sharing, performance, manageability FS NATIVE USERS 23

24 Centralized (e.g., Grid CAS) The principals identities of all tenants centrally maintained Poor scalability, isolation and manageability Peer-to-peer (e.g., SFS) Tenants communicate to publicize their principals identities Overhead to periodically synchronize the tenants Mapping (e.g., HekaFS) Local principal IDs mapped to global unique IDs Mapping overhead, sharing complications, security violations Network-level isolation (e.g., EFS of Amazon) Typical NFS-like identities limit scalability and security Lacking native support for multitenant user management 24

25 Hierarchical identification and authentication The provider manages the tenants The tenants manage their own users Native multitenant authorization Separate ACLs per tenant and provider Namespace isolation through filesystem views Efficient permission management and storage Shared common permissions across files within a folder Hierarchical inheritance of permissions in filesystem tree 25

26 Principals Tenant principals: Use/manage tenant resources Native filesystem principals: Manage the filesystem Tenant Authentication Server (TAS) Certifies local clients and principals Filesystem Authentication Server (FAS) Certifies filesystem services, tenants, native principals Clients Tenant Authentication Server Users 1 TENANT 1 Filesystem Filesystem Authentication Authentication Service Server MDS OSD OSD FILESYSTEM SERVERS Clients Tenant Authentication Server Users N TENANT N 26

27 TENANT 1 Access control isolation Separate ACLs per tenant, provider Metadata accessible through views Filesystem view Used by native filesystem principals to manage tenants Authorization Decision Authorization Request Tenant view Used by tenants to access or manage tenant resources File access Private to a principal, or Shared across principals of one or more tenants Client MDS Metadata Ticket Tenant Tenant 1 Policy 1 Policy Tenant Tenant 2 Policy 2 Policy Tenant Tenant N Policy N Policy Per file access policy Data Ticket 27

28 Goal Reduce filesystem load by reducing ACLs Per tenant permission inheritance Permissions can be inherited to child files & folders Per tenant common permissions Files share ACL w/ common parent Access Control Lists Tree Folder ACLs Tenant i Folder Tenant i Folder Tree Folder ACLs Tree File ACLs Tenant i Tree File ACLs Tenant i Tree folder ACLs:Folder permissions Tree file ACLs: Shared permissions of child files Private file ACLs:Permissions of child file configured by user Private File ACL Tenant k 28

29 Captured credential Fresh tamperproof credentials cannot be forged Compromised tenant principal account Compromised tenant view is isolated Attack limited to principal s private or shared files Cross-tenant policy violation is prohibited Attack by revoked tenant Restricted through deleted tenant view Tenant cannot access other views Compromised provider administrator account Detect through external attestation Handle via good practices (e.g., restricted remote access) 29

30 Session Tenant identified by client Session limited to one tenant Permissions Tenant view: Extended Attributes Filesystem view: Regular fields Capabilities Include principal and tenant identifiers Sent to clients with tenant file access 30

31 Configuration: AWS EC2 Instances m1.xlarge: x3 servers, 4 VCPU, 15 GB RAM, Linux t1.micro: x32 clients, 1 VCPU, 615 MB RAM, Linux Filesystem configuration Ceph/Dike: m1.xlarge, 1xOSD+MON, 1xOSD+MDS, 1xOSD GlusterFS/HekaFS: m1.xlarge, 3 fileservers Replication factor 3 Microbenchmark mdtest created files and folders 31

32 Client 1-32 clients: similar to Ceph Tenant 1k-5k tenants: 2% overhead 32

33 Dike limited overhead 1k tenants overhead: 14% 5k tenants overhead: 16% HekaFS mapping costly 1k tenants overhead: 49% 5k tenants overhead: 84% 33

34 Native filesystem multitenancy with sharing support Hierarchical identification scheme Namespace isolation: separate ACLs per tenant and provider Per tenant common permissions and inheritance Performance and security analysis Limited multitenancy overhead up to 16% Dike scalable to several thousand tenants Tenant principals not able to violate cross-tenant policy 34

35 This has been joint work with Andromachi Hatzieleftheriou Giorgos Kappes 35

36 University of Ioannina Operates since 1964 (established as independent institution since 1970) Enrollment ugrad & 3500 grad students 15 departments including: Medicine, Biology, Architecture, Math, Physics, Chemistry, Economics, Philosophy, Linguistics, Education Ioannina Located in Northwestern Greece Population 111,740 (census of 2011) Exists since 6 th century A.D. Dodoni theater since 300 B.C. (c.~19000) 36

37 Thank you! Questions? 37