Distributed Storage with GlusterFS

Transcription

1 Distributed Storage with GlusterFS Dr. Udo Seidel Amadeus OSDC

2 Agenda Introduction High level overview Storage inside Use cases Summary OSDC

3 Introduction OSDC

4 Me ;-) Teacher of mathematics & physics PhD in experimental physics Started with Linux in 1996 Linux/UNIX trainer Solution engineer in HPC and CAx environment Head of the Linux Strategy LinuxCon Europe 05-07/

5 Storage: History Reviewing storage task responsibilities Block allocation Space management Extension of SCSI standard Object based storage Meta-Data handling separated from data management OSDC

6 Object based storage Storage objects quite general Partition, file,... Unique identifier OSD (Object based Storage Device) Hardware -> original trigger Software -> common implementation Main component of distributed file systems OSDC

7 Block -> Object Distributed storage: Paradigm changes Central -> Distributed Few -> Many Big -> Small Server <-> Storage OSDC

8 Distributed File Systems 'Recent' attention on distributed storage Cloud hype Big Data See also CEPH talk the other day OSDC

9 Distributed storage Now what?!? Several implementations Different functions Support models Storage vendors initiatives Relation to Linux distributions Here and now ==> GlusterFS OSDC

10 High level overview OSDC

11 History Gluster founded in 2005 Gluster = GNU + cluster Acquisition by Red Hat in 2011 Community project 3.2 in in 2012 Commercial product: RedHat Storage Server OSDC

12 The Client Native 'speaks' GLUSTERFS Not part of the Linux Kernel FUSE-based NFS Normal NFS client stack S3/Swift compatible Proxy needed OSDC

13 The Server Data Bricks Translators Volumes -> exported/served to the client Meta-Data No dedicated instance Distributed hashing approach OSDC

14 The picture OSDC

15 Storage inside OSDC

16 The Brick Trust each other Interconnect via TCP/IP or RDMA/Infiniband Dedicated file systems on GlusterFS server XFS recommended, EXT4 works too Extended attributes a must Two main processes/daemons glusterd glusterfsd OSDC

17 The Translator One per purpose Replication POSIX Quota I/O behaviour Chained -> brick graph Technically: configuration OSDC

18 The Volume Service unit Layer of configuration 'RAID level' -> distributed, replicated, striped, combinations NFS Cache Permissions... OSDC

19 Meta Data 2 kinds More of local file system style Related to distributed nature Some stored in backend file system Permissions Time stamps Distribution/replication Some calculated on the fly Brick location OSDC

20 Elastic Hash Algorithm Based on file names Name space divided Full brick handled via relinking Stored in extended attributes Client needs to know topology OSDC

21 Self-Healing On demand vs. Scheduled File based Based on extended attributes Split-brain Quorum function Sometimes: manual intervention OSDC

22 Geo replication Asynchronous Based on rsync/ssh Master-Slave If needed: cascading One way street Clocks in sync! OSDC

23 From files to objects Introduced with version 3.3 Hard links with some hierarchy Re-uses GFID (inode number) UFO Unified File and Object Combination with RESTful API S3 and swift compatible OSDC

24 Operations: Growth, shrinkage.. failures A Must! Easy Rebalance! Order of servers important OSDC

25 What else...? Encryption :- Compression :-( Snapshots :-( Hadoop connector :-) Locking granularity :- File system statistics :-) OSDC

26 Use cases OSDC

27 NAS replacement NFS as 1:1 Server: GlusterFS Client: NFS NFS as such Server: GlusterFS Client: GlusterFS OSDC

28 SAN replacement Not quite advanced (yet) New translator needed Development started Presenting GlusterFS as block device Additional items needed Locking... OSDC

29 Storage back-end for KVM and Co Stacked (indirect) Not smart Workable for main hypervisors Direct QEMU libvirt ovirt/rhev OSDC

30 Summary OSDC

31 Take aways Thin distributed file system layer Modular architecture Operationally ready Still some surprises Active development and community OSDC

32 References (pictures) OSDC

33 Thank you! OSDC

34 Distributed Storage with GlusterFS Dr. Udo Seidel Amadeus OSDC