Ceph vs Swift Performance Evaluation on a Small Cluster. edupert monthly call Jul 24, 2014

Transcription

1 Ceph vs Swift Performance Evaluation on a Small Cluster edupert monthly call July, 24th 2014

2 About me Vincenzo Pii Leading research initiative on Cloud Storage Under the theme IaaS More on ICCLab:

3 About this work Performance evaluation study on cloud storage Small installations Hardware resources hosted at the ZHAW ICCLab data center in Winterthur Two OpenStack clouds (stable and experimental) One cluster dedicated to storage research

4 INTRODUCTION

5 Cloud storage Cloud storage Based on distributed, parallel, fault-tolerant file systems Distributed resources exposed through a single homogeneous interface Typical requirements Highly scalable Replication management Redundancy (no single point of failure) Data distribution Object storage A way to manage/access data in a storage system Typical alternatives Block storage File storage

6 Ceph and Swift Ceph (ceph.com) Supported by Inktank Recently purchased by RedHat (owners of GlusterFS) Mostly developed in C++ Started as PhD thesis project in 2006 Block, file and object storage Swift (launchpad.net/swift) OpenStack object storage Completely written in Python RESTful HTTP APIs

7 Objectives of the study 1. Performance evaluation of Ceph and Swift on a small cluster Private storage Storage backend for own-apps with limited requirements in size Experimental environments 2. Evaluate Ceph maturity and stability Swift already widely deployed and industry-proven 3. Hands-on experience Configuration Tooling

8 CONFIGURATION AND PERFORMANCE OF SINGLE COMPONENTS

9 Network configuration Three servers on a dedicated VLAN 1 Gbps NICs 100BaseT cabling Node 1 (.2) Node 2 (.3) Node 3 (.4) x/24

10 Servers configuration Hardware Lynx CALLEO Application Server x Intel Xeon E5620 (4 core) 8x 8 GB DDR3 SDRAM, 1333 MHz, registered, ECC 4x 1 TB Enterprise SATA-3 Hard Disk, 7200 RPM, 6 Gb/s (Seagate ST1000NM0011) 2x Gigabit Ethernet network interfaces Operating system Ubuntu Server Edition with Kernel generic

11 Disks performance READ: $ sudo hdparm -t --direct /dev/sdb1 /dev/sdb1: Timing O_DIRECT disk reads: 430 MB in 3.00 seconds = MB/sec WRITE: $ dd if=/dev/zero of=anof bs=1g count=1 oflag=direct 1+0 records in 1+0 records out bytes (1.1 GB) copied, s, 123 MB/s

12 Network performance $ iperf -c ceph-osd Client connecting to ceph-osd0, TCP port 5001 TCP window size: 85.0 KByte (default) [ 3] local port connected with port 5001 [ ID] Interval Transfer Bandwidth [ 3] sec 1.10 GBytes 942 Mbits/sec 942 Mbits/s MB/s

13 CLOUD STORAGE CONFIGURATION

14 Ceph OSDs ceph status cluster ff0baf2c-922c-4afc-8867-dee72b9325bb health HEALTH_OK monmap e1: 1 mons at {ceph-mon0= :6789/0}, election epoch 1, quorum 0 ceph-mon0 osdmap e139: 4 osds: 4 up, 4 in pgmap v17348: 632 pgs, 13 pools, 1834 bytes data, 52 objects 199 MB used, 3724 GB / 3724 GB avail 632 active+clean cluster-admin@ceph-mon0:~$ ceph osd tree # id weight type name up/down reweight root default host ceph-osd osd.0 up osd.1 up host ceph-osd osd.2 up osd.3 up 1 Monitor (mon0) HDD1 (OS) Not used Not used Not used St. node 0 HDD1 (OS) osd0 XFS osd1 XFS Journal St. node 1 HDD1 (OS) osd2 XFS osd3 XFS Journal

15 Swift devices Building rings on storage devices (No separation of Accounts, Containers and Objects) export ZONE= # set the zone number for that storage device export STORAGE_LOCAL_NET_IP= # and the IP address export WEIGHT=100 # relative weight (higher for bigger/faster disks) export DEVICE= swift-ring-builder account.builder add z$zone-$storage_local_net_ip:6002/$device $WEIGHT swift-ring-builder container.builder add z$zone-$storage_local_net_ip:6001/$device $WEIGHT swift-ring-builder object.builder add z$zone-$storage_local_net_ip:6000/$device $WEIGHT Swift Proxy HDD1 (OS) Not used Not used Not used St. node 0 HDD1 (OS) dev1 XFS dev2 XFS Not used St. node 1 HDD1 (OS) dev3 XFS dev4 XFS Not used

16 Highlighting a difference LibRados used to access Ceph Plain installation of a Ceph storage cluster Non ReST-ful interface This is the fundamental access layer in Ceph RadosGW (Swift/S3 APIs) is an additional component on top of LibRados (as block and file storage clients) ReST-ful APIs over HTTP used to access Swift Extra overhead in the communication Out-of-the box access method for Swift This is part of the differences to be benchmarked, even if... HTTP APIs for object-storage are interesting for many use cases This use case: Unconstrained self-managed storage infrastructure for, e.g., own apps Control over infrastructure and applications

17 WORKLOADS

18 Tools COSBench (v b2) - Developed by Intel Benchmarking for Cloud Object Storage Supports both Swift and Ceph Cool web interface to submit workloads and monitor current status Workloads defined as XML files Very good level of abstractions applying to object storage Supported metrics Op-Count (number of operations) Byte-Count (number of bytes) Response-Time (average response time for each successful request) Processing-Time (average processing time for each successful request) Throughput (operations per seconds) Bandwidth (bytes per seconds) Success-Ratio (ratio of successful operations) Outputs CSV data Graphs generated with cosbench-plot - Describe inter-workload charts in Python

19 Workloads gist

20 COSBench web interface

21 Workload matrix Containers Objects size R/W/D Distr. (%) Workers 1 4 kb 80/15/ kb 100/0/ kb 0/100/ kb MB MB 512

22 Workloads 216 workstages (all the combinations of the values of the workload matrix) 12 minutes per workstage 2 minutes warmup 10 minutes running time 1000 objects per container (pools in Ceph) Uniformly distributed operations over the available objects (1000 or 20000)

23 Performance Results READING

24 Read tpt Workstage AVGs

25 Read tpt 1 cont 4 KB

26 Read tpt 1 cont 128 KB

27 Read ResponseTime 1 cont 128 KB

28 Read bdw Workstage AVGs

29 Read tpt 20 cont 1024 KB

30 Response time 20 cont 1024 KB

31 Performance Results WRITING

32 Write tpt Workstage AVGs

33 Write tpt 1cont 128 KB

34 Ceph write tpt 1 cont 128 KB replicas

35 Ceph write RT 1 cont 128 KB replicas

36 Write bdw Workstage AVGs

37 Write Response Time 20 cont 512 KB

38 Performance Results READ/WRITE/DELETE

39 R/W/D tpt Workstage AVGs

40 Read R/W/D Response Time

41 General considerations and future works CONCLUSIONS

42 Performance Analysis Recap Ceph performs better when reading, Swift when writing Ceph librados Swift ReST APIs over HTTP More remarkable difference with small objects Less overhead for Ceph Librados CRUSH algorithm Comparable performance with bigger objects Network bottleneck at 120 MB/s for read operations Response time Swift: greedy behavior Ceph: fairness

43 General considerations: challenges Equivalency Comparing two similar systems that are not exactly overlapping Creating fair setups (e.g., Journals on additional disks for Ceph) Transposing corresponding concepts Configuration Choosing the right/best settings for the context (e.g., number of Swift workers) Identifying bottlenecks To be done in advance to create meaningful workloads Workloads Run many tests to identify saturating conditions Huge decision space Keep up the pace Lot of developments going on (new versions, new features)

44 General considerations: lessons learnt Publication medium (blog post) Excellent feedback (e.g., Rackspace developer) Immediate right of reply and real comments Most important principles Openness Share every bit of information Clear intents, clear justifications Neutrality When analyzing the results When drawing conclusions Very good suggestions coming from you could, you should, you didn t comments

45 Future works Performance evaluation necessary for cloud storage More object storage evaluations Interesting because it s very close to the application level Block storage evaluations Very appropriate for IaaS Provide storage resources to VMs Seagate Kinetic Possible opportunity to work on a Kinetic setup

46 Vincenzo Pii: THANKS! QUESTIONS?