Benchmarking Ceph's BlueStore. Niklaus Hofer Open Cloud Day

Transcription

1 Benchmarking Ceph's BlueStore Niklaus Hofer Open Cloud Day

2 stoney cloud Benchmarking Ceph's BlueStore 3/25

3 Where we are coming from Small cloud company Mostly PaaS FOSS-Cloud.org based cloud libvirt based No longer actively maintained GlusterFS storage backend Benchmarking Ceph's BlueStore 4/25

4 State of the (OpenStack) cloud OpenStack based Live: next month A lot of expensive hardware is currently idly Benchmarks! Benchmarking Ceph's BlueStore 5/25

5 Early benchmarks Direct SSD vs Ceph vs GlusterFS Later: GlusterFS is NOT an option Variations within Ceph BlueStore vs Filestore Benchmarking Ceph's BlueStore 6/25

6 Ceph hardware Dedicated nodes 3 nodes SATA SSDs Micron m5100 max NVME SSDs Test Benchmarking Ceph's BlueStore 7/25

7 Ceph setup Ceph Luminous BlueStore Replica Benchmarking Ceph's BlueStore 8/25

8 Ceph Benchmarking Ceph's BlueStore 9/25

9 Ceph journal Limitations of POSIX filesystem All data gets written twice Optimization: Journal on separate disk # Start End Size Type Name T Ceph OSD ceph data G Ceph Journal ceph journal /dev/sdd : /dev/sdd1 ceph data, active, cluster ceph, osd.0, journal /dev/sdd2 /dev/sdd2 ceph journal, for /dev/sdd Benchmarking Ceph's BlueStore 10/25

10 Ceph journal optimization Benchmarking Ceph's BlueStore 11/25

11 Ceph BlueStore Benchmarking Ceph's BlueStore 12/25

12 Healing performance --dsk/sdc-- --dsk/sdd-- --dsk/sde-- read writ: read writ: read writ 3905k 4397k:3469k 3666k:3380k 5890k 0 193M: 0 0 : 111M M: 0 0 : 123M M: 0 0 : 114M M: 0 0 : 114M M: 0 0 : 121M M: 0 0 : 116M M: 0 0 : 120M M: 0 0 : 122M M: 0 0 : 122M M: 0 0 : 110M M: 0 0 : 118M M: 0 0 : 124M M: 0 0 : 126M M: 0 0 : 122M M: 0 0 : 114M 0 --dsk/sdc-- --dsk/sdd-- --dsk/sde-- read writ: read writ: read writ 0 132M: 128M 0 : M: 132M 0 : M: 128M 0 : M: 108M 0 : M: 128M 0 : M: 128M 0 : M: 117M 0 : M: 112M 0 : M: 128M 0 : M: 127M 652k: M: 129M 0 : M: 132M 0 : M: 132M 0 : M: 128M 0 : M: 132M 0 : M: 128M 0 : Benchmarking Ceph's BlueStore 13/25

13 Benchmarking Benchmarking Ceph's BlueStore 14/25

14 Objectives Single VM VM located on separate storage Phoronix Test Suite pts/disk Benchmarking Ceph's BlueStore 15/25

15 Benchmarking results Benchmarking Ceph's BlueStore 16/25

16 AIO-Stress AIO-Stress v0.21 Random Write MB/s, More Is Better PHORONIX-TEST-SUITE.COM local filesystem SE +/ CEPH Jewel 3 OSDs replica 1 SE +/ Direct SSD io=native cache=none SE +/ CEPH Jewel 1 OSD w/ external Journal SE +/ CEPH Jewel 1 OSD SE +/ CEPH Jewel 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 1 SE +/ CEPH luminous bluestore 1 OSD SE +/ (CC) gcc options: -pthread -laio Phoronix Test Suite Benchmarking Ceph's BlueStore 17/25

17 SQLite SQLite v3.22 Timed SQLite Insertions Seconds, Less Is Better PHORONIX-TEST-SUITE.COM local filesystem SE +/ CEPH Jewel 3 OSDs replica 1 SE +/ Direct SSD io=native cache=none SE +/ CEPH Jewel 1 OSD w/ external Journal SE +/ CEPH Jewel 1 OSD SE +/ CEPH Jewel 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 1 SE +/ CEPH luminous bluestore 1 OSD SE +/ (CC) gcc options: -O2 -ldl -lpthread Phoronix Test Suite Benchmarking Ceph's BlueStore 18/25

18 FS-Mark FS-Mark v Files, 1MB Size Files/s, More Is Better PHORONIX-TEST-SUITE.COM local filesystem SE +/ CEPH Jewel 3 OSDs replica 1 SE +/ Direct SSD io=native cache=none SE +/ CEPH Jewel 1 OSD w/ external Journal SE +/ CEPH Jewel 1 OSD SE +/ CEPH Jewel 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 1 SE +/ CEPH luminous bluestore 1 OSD SE +/ (CC) gcc options: -static Phoronix Test Suite Benchmarking Ceph's BlueStore 19/25

19 Dbench Dbench v Clients MB/s, More Is Better PHORONIX-TEST-SUITE.COM local filesystem SE +/ CEPH Jewel 3 OSDs replica 1 SE +/ Direct SSD io=native cache=none SE +/ CEPH Jewel 1 OSD w/ external Journal SE +/ CEPH Jewel 1 OSD SE +/ CEPH Jewel 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 1 SE +/ CEPH luminous bluestore 1 OSD SE +/ (CC) gcc options: -lpopt -O2 Phoronix Test Suite Benchmarking Ceph's BlueStore 20/25

20 Threaded I/O tester Threaded I/O Tester v MB Random Write - 32 Threads MB/s, More Is Better PHORONIX-TEST-SUITE.COM local filesystem SE +/ CEPH Jewel 3 OSDs replica 1 SE +/ Direct SSD io=native cache=none SE +/ CEPH Jewel 1 OSD w/ external Journal SE +/ CEPH Jewel 1 OSD SE +/ CEPH Jewel 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 3 SE +/ CEPH luminous bluestore 3 OSDs replica 1 SE +/ CEPH luminous bluestore 1 OSD SE +/ (CC) gcc options: -O2 Phoronix Test Suite Benchmarking Ceph's BlueStore 21/25

21 Analysis and conclusion Benchmarking Ceph's BlueStore 22/25

22 What went wrong? Largely guesswork for now Spectre patches? Regression? Configuration problem? Benchmarking Ceph's BlueStore 23/25

23 Future work Optimization NVME vs SATA Especially for database setups More precise IOPS measurements fio, highly workload dependant Benchmarking Ceph's BlueStore 24/25

24 Questions? Benchmarking Ceph's BlueStore 25/25

25 stepping stone GmbH Wasserwerkgasse 7 CH-3011 Bern Telefon: info@stepping-stone.ch Benchmarking Ceph's BlueStore 26/25