Distributed Storage with GlusterFS Dr. Udo Seidel Linux-Strategy @ Amadeus OSDC 2013 1
Agenda Introduction High level overview Storage inside Use cases Summary OSDC 2013 2
Introduction OSDC 2013 3
Me ;-) Teacher of mathematics & physics PhD in experimental physics Started with Linux in 1996 Linux/UNIX trainer Solution engineer in HPC and CAx environment Head of the Linux Strategy team @Amadeus LinuxCon Europe 05-07/11 2012 4
Storage: History Reviewing storage task responsibilities Block allocation Space management Extension of SCSI standard Object based storage Meta-Data handling separated from data management OSDC 2013 5
Object based storage Storage objects quite general Partition, file,... Unique identifier OSD (Object based Storage Device) Hardware -> original trigger Software -> common implementation Main component of distributed file systems OSDC 2013 6
Block -> Object Distributed storage: Paradigm changes Central -> Distributed Few -> Many Big -> Small Server <-> Storage OSDC 2013 7
Distributed File Systems 'Recent' attention on distributed storage Cloud hype Big Data See also CEPH talk the other day OSDC 2013 8
Distributed storage Now what?!? Several implementations Different functions Support models Storage vendors initiatives Relation to Linux distributions Here and now ==> GlusterFS OSDC 2013 9
High level overview OSDC 2013 10
History Gluster founded in 2005 Gluster = GNU + cluster Acquisition by Red Hat in 2011 Community project 3.2 in 2011 3.3 in 2012 Commercial product: RedHat Storage Server OSDC 2013 11
The Client Native 'speaks' GLUSTERFS Not part of the Linux Kernel FUSE-based NFS Normal NFS client stack S3/Swift compatible Proxy needed OSDC 2013 12
The Server Data Bricks Translators Volumes -> exported/served to the client Meta-Data No dedicated instance Distributed hashing approach OSDC 2013 13
The picture OSDC 2013 14
Storage inside OSDC 2013 15
The Brick Trust each other Interconnect via TCP/IP or RDMA/Infiniband Dedicated file systems on GlusterFS server XFS recommended, EXT4 works too Extended attributes a must Two main processes/daemons glusterd glusterfsd OSDC 2013 16
The Translator One per purpose Replication POSIX Quota I/O behaviour Chained -> brick graph Technically: configuration OSDC 2013 17
The Volume Service unit Layer of configuration 'RAID level' -> distributed, replicated, striped, combinations NFS Cache Permissions... OSDC 2013 18
Meta Data 2 kinds More of local file system style Related to distributed nature Some stored in backend file system Permissions Time stamps Distribution/replication Some calculated on the fly Brick location OSDC 2013 19
Elastic Hash Algorithm Based on file names Name space divided Full brick handled via relinking Stored in extended attributes Client needs to know topology OSDC 2013 20
Self-Healing On demand vs. Scheduled File based Based on extended attributes Split-brain Quorum function Sometimes: manual intervention OSDC 2013 21
Geo replication Asynchronous Based on rsync/ssh Master-Slave If needed: cascading One way street Clocks in sync! OSDC 2013 22
From files to objects Introduced with version 3.3 Hard links with some hierarchy Re-uses GFID (inode number) UFO Unified File and Object Combination with RESTful API S3 and swift compatible OSDC 2013 23
Operations: Growth, shrinkage.. failures A Must! Easy Rebalance! Order of servers important OSDC 2013 24
What else...? Encryption :- Compression :-( Snapshots :-( Hadoop connector :-) Locking granularity :- File system statistics :-) OSDC 2013 25
Use cases OSDC 2013 26
NAS replacement NFS as 1:1 Server: GlusterFS Client: NFS NFS as such Server: GlusterFS Client: GlusterFS OSDC 2013 27
SAN replacement Not quite advanced (yet) New translator needed Development started Presenting GlusterFS as block device Additional items needed Locking... OSDC 2013 28
Storage back-end for KVM and Co Stacked (indirect) Not smart Workable for main hypervisors Direct QEMU libvirt ovirt/rhev OSDC 2013 29
Summary OSDC 2013 30
Take aways Thin distributed file system layer Modular architecture Operationally ready Still some surprises Active development and community OSDC 2013 31
References http://www.gluster.org http://www.sxc.hu (pictures) OSDC 2013 32
Thank you! OSDC 2013 33
Distributed Storage with GlusterFS Dr. Udo Seidel Linux-Strategy @ Amadeus OSDC 2013 34