Open vstorage RedHat Ceph Architectural Comparison

Transcription

1 Open vstorage RedHat Ceph Architectural Comparison Open vstorage is the World s fastest Distributed Block Store that spans across different Datacenter. It combines ultrahigh performance and low latency connections with a data integrity that has no comparison. Data is distributed across datacenters using both Replication and Erasure Coding. Joining Performance and Integrity is not a simple bolt-on solution and requires a from-the-ground-up approach. Disk Failures, Node Failures and even Datacenter Failures do not present data loss and hence do not threaten any of your Data Integrity. You have been lead to believe that in order to have a 100% Data Loss Protection you have to compromise on Performance. While this might sound logical and acceptable, in is time to step out of the box and demand a noncompromise Storage Platform. With Open vstorage you can have your cake and eat it too! This document provides an overview of the Nexenta architecture and highlights the difference with the Open vstorage architecture. The comparison is not intended to be exhaustive, but covers the most relevant items where both solutions differ as seen from a customer perspective. Antwerpse Steenweg 19, 9080 Lochristi Belgium Phone: Mail: Info@openvstorage.com

2 Introduction Ceph, now part of RedHat, is an open-source, distributed object store designed to provide excellent performance, reliability and scalability. Next to scalable object storage interface Ceph offers a file system and block storage interface. All these interfaces make Ceph a popular storage solution. It has even been labeled the de facto storage backend for OpenStack 1. This should not come as a surprise as 57% of the OpenStack users and 45% of the cores deployments are using Ceph RBD as block storage solution 2. But although it is the most used OpenStack block storage solution, there are other solutions on the market which might be able to push Ceph from its OpenStack throne. Open vstorage takes a different approach which allows it to beat the performance of Ceph and while offering more functionality. Architectural Design RedHat Ceph At the core of Ceph is the RADOS object-based storage system. It provides the foundation for some of Ceph s advanced features such as RADOS Block Device (RBD), the RADOS S3 Gateway and the Ceph File System. The RADOS architecture consists out of a cluster map, a diagram explaining how nodes and disks are distributed, a set of OSDs (Object Storage Daemons) and monitors. The cluster map is built upon the CRUSH (Controlled Replication Under Scalable Hashing) algorithm to efficiently compute information about data location, instead of having to depend on a central lookup table. This is a critical part as the CRUSH algorithm utilizes a hash function that makes the mapping completely deterministic and calculable. The OSDs are responsible for storing the actual data on disks. The monitors contain the cluster map and clients can retrieve the map from the monitors. Ceph clients such as RBD run on the client side and write objects to the OSDs based upon the cluster map received from the monitors

3 In many cases RBD, the block interface, is used to provide storage for Virtual Machines by offering a virtual block device to the hypervisor. Underlying RBD splits the virtual block device into 4MB chunks that are stored as objects on the RADOS object store. This means a 4K write ends up as a 4MB write on the backend. Data on a Ceph backend is typically stored 3 times as this is the default policy. But let s have a look at the write flow in detail. The client writes to the primary OSD, a disk of the Ceph cluster. As soon as the data is written to the first OSD, it is written to 2 additional OSDs. Once both the secondary and tertiary OSD acknowledge the write to the primary OSD, the primary OSD will acknowledge to the client that the write is safely stored on the backend. Now let s have a closer look as how data is stored on the OSDs. As soon as IO goes into the OSD, it gets written twice as the first IO is done to the Ceph journal and later it is written a second time to the backend file system for safeguarding. If the data is safely stored on the backend file system, the journal gets trimmed to reclaim space. The same dual write also applies to the secondary and tertiary copy so a single write IO is written 6 times, twice per OSD multiplied by 3 different OSDs, before it can be acknowledged to the application.

4 Open vstorage The basic components of Open vstorage are the Open vstorage Edge, the Open vstorage Volume Driver and ALBA. The Open vstorage Edge exposes block devices to applications that need a volume. The Edge component communicates across RDMA, a low-latency, high-throughput networking protocol, directly with the memory of the server running the Volume Driver. The Volume Driver is the technology that converts block storage into objects (Storage Container Objects) that can be stored on the ALBA backend. This ALBA backend is a special purpose object storage solution and is made up out of storage nodes that run ALBA daemons. These daemons manage the local storage devices (HDDs, SSDs, PCIe flash cards...) and contribute these devices to the storage pools. The Volume Driver is a combination of a locationbased approach (performance) and a log structured approach (unlimited history, unlimited snapshots, thin cloning). Each incoming 4K write is appended to the write buffer. This write buffer can actually be seen as a Transaction Log storing one or more Storage Container Objects, a consecutive group of incoming writes. The incoming write is also dispatched to a Transaction Log onto a second node in the cluster to prevent data loss. This principle of dispatching incoming writes to an additional write buffer is referred to as the Open vstorage Distributed Transaction Log (DTL). It is important to note that both the write buffer and the DTL are very small in size, as they only need to hold data that is not yet protected by the ALBA backend. They are typically limited to 256MB per volume.

5 Once a SCO is full, it gets split into chunks, compressed, encrypted and spread across the ALBA backend for redundancy. This ALBA backend is typically built on top of a pool of large capacity drives with an allflash performance tier as acceleration layer. On reads the Volume Driver fetches the correct block directly from the right chunk on the ALBA backend across RDMA. Similarities Software-defined storage Both Ceph and Open vstorage are software-defined storage solutions. They are infrastructure agnostic and can run on any x86 hardware. The actual specifications of the hardware have of course a huge impact on the performance numbers you can achieve. Both solutions use SSDs to speed up the performance and use SATA drives to store the data in the backend. Scalability Both solutions are built to scale capacity wise from a couple of TB up to a few petabyte. Each node that gets added to the cluster puts its capacity at the disposal of the storage pool. For example CERN is running a single production cluster of 3PB based upon Ceph and has tested a cluster of 30PB3. The same applies to Open vstorage. A cluster can grow as big as 1024 nodes and up to 30 PB. Performance of both solutions also scales with every node added as more drives are added to distribute the IO load. 3

6 Key Differences Deterministic vs Explicit Meta Data With Ceph the placement of data on the backend is computation based (CRUSH), this makes the whole process repeatable and deterministic. So no lookup on a hash table, only formulas and calculations are used to determine which disks contain the object. Open vstorage on the other hand uses explicit metadata as it stores the exact location of the objects in the backend in a distributed database. Note that based upon the data in the backend, the distributed database can always be reconstructed in case there would be an issue with the database. Open vstorage has optimized the footprint of the metadata set and manages to keep the whole set in RAM for performance reasons. While having explicit metadata consumes more storage space, the benefits of explicit metadata are substantial: In case of a broken disk Open vstorage can easily regenerate the data from the broken disks, as it knows exactly what was on the disk. It even knows which bits of data are most at risk due to many missing chunks and can repair the most vulnerable data first. As Ceph doesn t keep track of what is on the disks, it needs to wait until scrubbing picks up that objects are missing. As Open vstorage stores backend metadata it can offer APE (Asynchronous Policy Enforcement) which allows to spread data according to the best available policy. In case data can t be written optimally, it will be first stored sub-optimal and later, when for example nodes are replaced or the network link between the datacenters is restored, data will be re-written with the most optimal policy. Ceph doesn t have this type of features and will just stop all writes. Huge Performance Difference A lot has been said about the performance of Ceph or better the lack thereof. Execute a Google Search for Ceph Performance Tuning and you will find 25,000 suggestions to improve the performance. The truth is that Ceph was designed for scalability and wasn t developed to be the fastest distributed block storage solution. Just to give you some numbers; running a cluster of around k SATA HDD drives will sadly only give you roughly IOPs. Even with all-flash hardware, the random write performance isn t impressive. On an InfiniFlash Sandisk server with 64 SSDs, Ceph can only reach 180,000 IOPS for 4K random writes4. On top, since Ceph only acknowledges the write after 3 disks have acknowledged the write, a single slow disk will impact all VMs as clients spread data across all disks in the cluster. 4 ance_optimization.pptx

7 Open vstorage performance of a single node. Internal benchmarks show that Open vstorage can be 10x faster than Ceph on the same hardware. This huge performance difference can be explained by the fact that both solutions have a radical different design: Ceph uses a location based approach where each volume is divided into 4MB fragments and writes are location based hence happen randomly. Open vstorage uses a log structured approach using immutable objects that creates a lock less design and leverages multiple cores to get very high performance. This approach leads to lower latency and hence better performance. Erasure Coding Since the Hammer release, April 2015, Ceph supports erasure coding on the backend. But, as Ceph works location based and uses 4MB extents, the update of a 4k block means that the whole 4MB extent to which the block belongs needs to be erasure coded again. This is a very high penalty. One could of course lower the size of an extent but that will make the metadata grow out of control. Basically erasure coding isn t usable with RBD so replication must be used. Open vstorage on the other hand uses a log-structured approach where consecutive writes are appended to an immutable Storage Container Object (SCO). This SCO can be erasure coded by means of forward error correction techniques and the resulting fragments can be stored across multiple hosts or even datacenters. Space Efficiency As discussed earlier, Ceph can only work with an X-way replication strategy in case RBD is used. By default Ceph proposes a 3-way replication strategy. This isn t very space efficient, as every 4MB will lead to 12MB being stored on the physical disks. This means three times the amount of nodes, three times the amount of networking equipment, three times the amount of power and cooling, basically tripling the TCO. Open vstorage is much more space efficient as it uses forward error correction which provides 100x better reliability as compared to 3 copies although storing only 1.25x more data. This means that for every 1MB, 1.25MB will be saved on the backend. Secondly, Open vstorage allows for the creation of different error correction policies for flash and HDD and hence the customer can select a lower redundancy factor on flash and a higher redundancy factor on HDD. In addition, contrary to Ceph, Open vstorage can further reduce its storage footprint by compressing the fragments before storing them on

8 SSDs and HDDs. Ceph has compression functionality but since updating a 4K block means the whole 4MB extent needs to be compressed again, it makes no sense to enable it for RBD. Multi datacenter Ceph is designed to be used on local networks, as it requires low latency links between hosts to be performant. Since the writes need to be acknowledged by 3 storage nodes, the latency for writes would be too high in case the secondary or tertiary write would have to go to other datacenters. Open vstorage on the other hand is designed to store data safely and efficiently across multiple datacenters. To ensure local performance it uses an all-flash tier per datacenter while for disaster recovery reasons data can be stored on a capacity tier across multiple datacenters. The fact that Open vstorage is designed to be used in a multi datacenter approach is also visible in that Open vstorage supports 2 levels of erasure coding: a global and a local policy. The local policy makes sure that a single disk or host failure doesn t require data from other datacenters to be sent over the network in order to rebuild the missing chunks of data. As Ceph relies on replication the chunks will need to come from a different datacenter to replace the missing replicas. This puts additional stress on the network. Encryption of the backend Data security is an essential aspect of a large-scale storage implementation. Encryption, encoding the data with a secret key, is the most effective way to achieve data security. Open vstorage encrypts the data before the SCOs are transferred to the backend. This means that data being transferred over the network to the backend is already encrypted. Ceph on the other hand only encrypts data when it is being stored on the OSD. This means data is in a clear format when it is transferred over the network. This approach poses a higher risk as data might get wiretapped while in transit to the backend. Snapshots and Clones For Open vstorage taking a snapshot is as simple as placing a marker in the metadata. Therefore it takes less than a millisecond to complete a snapshot. Due to the design, taking a snapshot in Ceph involves a lot more work as Ceph works location based. When a snapshot is created and data is being overwritten, the old data needs to be copied, a copy-on-write process, to safeguard the overwritten data. This adds additional latency to the write path. Both solutions offer thin cloning but both implement this feature quite differently. Ceph uses an approach where there is no tracking of which objects exist in a clone. A read that hits a non-existent object will attempt to read from the parent snapshot, and this will continue recursively until an object exists or an image with no parent is found. It goes without saying that each lookup attempt adds additional latency to the read path. Open vstorage on the other hand keeps track of the metadata of the volume. Finding the data only requires to look up the right SCO in the metadata. Even if the block being read is in the root snapshot and the volume has been cloned multiple times, the block can be fetched immediately from the backend with a single read operation. This single-hop-read allows Open vstorage to minimize the read latency even for the most complex clone trees.

9 Flash Friendly As Ceph is location based it requires high endurance flash technology as random writes, generate small updates across the device. Many updates of the same location within the volume will also cause the flash chips to wear out faster than normal as you are constantly updating the same cells of the flash memory. Open vstorage is much more flash friendly as the write buffer sequentializes the random writes in large fragments that can be written sequentially. Due to random write amplification, sequential write endurance for SSDs and PCIe flash cards is typically higher so the flash drives are less likely to fail with Open vstorage compared to Ceph. As every update to a volume also gets appended to a new SCO, overwriting the same LBA of a volume many times will hammer the same cells of the flash memory over and over. Since Open vstorage is less intensive on flash as compared to Ceph, cheaper, lower endurance SSDs can be used for the performance tier. The bottom line Open vstorage and Ceph are software based distributed block storage solutions and they both scale very well. Although both can be installed on the same hardware, the different design approaches result in a remarkable performance difference. Based upon tests it has been proven that Open vstorage can be 10 times faster than Ceph on the same hardware. Next to the difference in performance, Open vstorage is more space efficient. Ceph uses replication to protect the data while Open vstorage can use both erasure coding and replication. This allows Open vstorage to store data across datacenters and survive even a complete datacenter going offline.