Ceph Intro & Architectural Overview. Abbas Bangash Intercloud Systems

Transcription

1 Ceph Intro & Architectural Overview Abbas Bangash Intercloud Systems

2 About Me Abbas Bangash Systems Team Lead, Intercloud Systems intercloudsys.com 2

3 CLOUD SERVICES COMPUTE NETWORK STORAGE the future of storage 3

4 COMPUTER TAPE 4

5 YOU TECHNOLOGY YOUR DATA 5

6 gaaaaaaaaahhhh!!!!!! cloud computing distributed storage computers carving writing paper How Much Store Things All Human History?! 6

7 COMPUTER 7

8 COMPUTER 8

9 GIANT SPENDY COMPUTER 9

10 COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER 10

11 COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER 11

12 STORAGE APPLIANCE COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER COMPUTER 12

13 Storage Appliance 13

14 SUPPORT AND MAINTENANCE 34% of 2012 revenue (5.2 billion dollars) PROPRIETARY SOFTWARE 1.1 billion in R&D spent in 2012 PROPRIETARY HARDWARE COMPUTE R COMPUTE R COMPUTE R COMPUTE R 1.6 million square feet of manufacturing space 14

15 15

16 philosophy design OPEN SOURCE COMMUNITY- FOCUSED SCALABLE NO SINGLE POINT OF FAILURE SOFTWARE BASED SELF-MANAGING 16

17 OBJECTS VIRTUAL S FILES & DIRECTORIES CEPH OBJECT GATEWAY A powerful S3- and Swiftcompatible gateway that brings the power of the Ceph Object Store to modern applications CEPH BLOCK DEVICE A distributed virtual block device that delivers highperformance, costeffective storage for virtual machines and legacy applications CEPH FILE SYSTEM A distributed, scale-out filesystem with POSIX semantics that provides storage for a legacy and modern applications CEPH STORAGE CLUSTER A reliable, easy to manage, next-generation distributed object store that provides storage of unstructured data for applications 17

18 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 18

19 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 19

20 OSD OSD OSD OSD OSD FS FS FS FS FS btrfs xfs ext4 M M M 20

21 M M M 21

22 OSDs: 10s to 10000s in a cluster One per disk (or one per SSD, RAID group ) Serve stored objects to clients Intelligently peer to perform replication and recovery tasks M Monitors: Maintain cluster membership and state Provide consensus for distributed decision-making Small, odd number These do not serve stored objects to clients 22

23 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 23

24 APP LIBRADOS socket M M M 24

25 L LIBRADOS Provides direct access to RADOS for applications C, C++, Python, PHP, Java, Erlang Direct access to storage nodes No HTTP overhead

26 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 26

27 APP REST RADOSGW LIBRADOS socket M M M 27

28 RADOS Gateway: REST-based object storage proxy Uses RADOS to store objects API supports buckets, accounts Usage accounting for billing Compatible with S3 and Swift applications 28

29 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 29

30 VM VIRTUALIZATION CONTAINER LIBRBD LIBRADOS M M M 30

31 CONTAINER LIBRBD LIBRADOS VM CONTAINER LIBRBD LIBRADOS M M M 31

32 HOST KRBD (KERNEL MODULE) LIBRADOS M M M 32

33 RADOS Block Device: Storage of disk images in RADOS Decouples VMs from host Images are striped across the cluster (pool) Snapshots Copy-on-write clones Support in: Mainline Linux Kernel ( ) Qemu/KVM, native Xen coming soon OpenStack, CloudStack, Nebula, Proxmox 33

34 APP APP HOST/VM CLIENT LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RADOSGW A bucket-based REST gateway, compatible with S3 and Swift RBD A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 34

35 CLIENT metadata data M M M 35

36 Metadata Server Manages metadata for a POSIX-compliant shared filesystem Directory hierarchy File metadata (owner, timestamps, mode, etc.) Stores metadata in RADOS Does not serve file data to clients Only required for shared filesystem 36

37 What Makes Ceph Unique? Part one: CRUSH 37

38 C D C D C D C D APP?? C C D D C D C D C D C D C D C D 38

39 How Long Did It Take You To Find Your Keys This Morning? azmeen, Flickr / CC BY

40 APP C C C C C C C C C C C C D D D D D D D D D D D D 40

41 Dear Diary: Today I Put My Keys on the Kitchen Counter Barnaby, Flickr / CC BY

42 C D C D A-G C D C D APP F* C C D D H-N C D C D O-T C D C D C D U-Z C D 42

43 HOW DO YOU FIND YOUR KEYS WHEN YOUR HOUSE IS INFINITELY BIG AND ALWAYS CHANGING? 43

44 The Answer: CRUSH!!!!! pasukaru76, Flickr / CC SA

45 hash(object name) % num pg CRUSH(pg, cluster state, rule set) 45

46

47 CRUSH Pseudo-random placement algorithm Fast calculation, no lookup Repeatable, deterministic Statistically uniform distribution Stable mapping Limited data migration on change Rule-based configuration Infrastructure topology aware Adjustable replication Weighting 47

48 CLIENT?? 48

49 NAME: "f oo" POOL: "bar" "bar" = 3 OBJECT hash("f oo") % 256 = 0x PLACEMENT GROUP PLACEMENT GROUP CRUSH TARGET OSDs 49

50 50

51 51

52 CLIENT?? 52

53 What Makes Ceph Unique Part two: thin provisioning 53

54 VM VIRTUALIZATION CONTAINER LIBRBD LIBRADOS M M M 54

55 HOW DO YOU SPIN UP THOUSANDS OF VMs INSTANTLY AND EFFICIENTLY? 55

56 instant copy =

57 write CLIENT write write write =

58 read read CLIENT read =

59 What Makes Ceph Unique? Part three: clustered metadata 59

60 POSIX Filesystem Metadata Barnaby, Flickr / CC BY

61 CLIENT M M M 61

62 M M M 62

63 one tree three metadata servers?? 63

64 64

65 65

66 66

67 67

68 DYNAMIC SUBTREE PARTITIONING 68

69 OpenStack + Ceph Used for Glance Images, Cinder Volumes and Nova ephemeral disk (coming soon) Ceph + OpenStack offers compelling features: CoW clones, layered volumes, snapshots, boot from volume, live migration Cost effective with Thin Provisioning ~110TB used, ~45TB * replicas on disk Ceph is the most popular network block storage backend for OpenStack 69

70 Deployment Automated deployment using Cephdeploy Automated machine commissioning and maintenance Add a server to the hostgroup (osd, mon, radosgw) OSD disks are detected, formatted, prepared, auth d Also after disk replacement Auto-generated ceph.conf Last step is manual/controlled: service ceph start Cephdeploy for bulk operations on the servers Ceph rpm upgrades daemon restarts 70

71 Getting Started With Ceph Have a working cluster up quickly. Read about the latest version of Ceph. The latest stuff is always at Deploy a test cluster using ceph-deploy. Read the quick-start guide at Deploy a test cluster on the AWS free-tier using Juju. Read the guide at Read the rest of the docs! Find docs for the latest release at 71

72 Getting Involved With Ceph Help build the best storage system around! Most project discussion happens on the mailing list. Join or view archives at IRC is a great place to get help (or help others!) Find details and historical logs at The tracker manages our bugs and feature requests. Register and start looking around at Doc updates and suggestions are always welcome. Learn how to contribute docs at 72