MongoDB Revs You Up: What Storage Engine is Right for You?

Transcription

1 MongoDB Revs You Up: What Storage Engine is Right for You? Jon Tobin, Director of Solution Eng. Linkedin.com/in/jonathanetobin

2 Agenda How did we get here? What storage engines are available? Why does the data structure ma8er? What makes them unique? Where should I start my evalua<on? How can I evaluate the engines?

3 First: Background

4 Let s Level Set { _id : ObjectId( 507f1f77bcf86cd ), studentid : 100, firstname : Jonathan, middlename : Eli, lastname : Tobin, classes : [ { courseid : PHY101, grade : B, coursename : Physics 101 credits : 3

5 MongoDB History MMAP rules the universe; concurrency suffers Per mongod lock v1.2.0 December 10 th 2009 v2.0.7 August 9 th 2012 Per database lock v2.2.0 August 29 th 2012 v2.6.8 August 25 th 2015 Concurrency!! Per document lock MongoDB, Inc acquires wiredtiger December 16 th, 2014 v3.0 March 3 rd 2015 First implementa<on of a storage engine API

6 Storage Engines & Data Structures

7 MongoDB Storage Engines MongoDB, Inc. & Percona Server for MongoDB MMAP wiredtiger MongoDB Enterprise Advanced Only In Memory Encrypted (wiredtiger) Percona Server for MongoDB PerconaFT RocksDB

8 B- tree Overview

9 B- tree Insert Pivot Rule >=

10 B- tree Search

11 B- tree - Importance of I/O AWS Insert 200M Rows Predictable I/O Response VS Not 15 hours VS 91 hours 6x

12 What s the Problem? Performance is I/O limited when data is > RAM Each insert/update requires at least 1 I/O plus an I/O for every extra index

13 What s Up With: MMAP Overview Very basic storage engine Collec<on level lock Highly reliant on the OS for caching Uses b- tree indexes to point to disk offset At the offset is the record In the record is the document Best Use In place updates Record migra<on should be minimized $inc, $set, etc Read only*

14 What s Up With: MMAP Problems Record alloca<on is fixed size Space inefficient (powerof2) What if document grows bigger than record? Probably not for you. Going the way of the dodo

15 What s Up With: wiredtiger Overview Concurrency: Document level Supports mul<ple data structures B- tree (v3.0 +) LSM tree (v3.2 +) Controls cache Best Use Depends on data structure B- tree: reads (point or small range) / dataset close to cache LSM: random updates Promising but s9ll a bit of a black box

16 What s Up With: RocksDB Overview Wri8en & maintained by Facebook Cut it s Parse Data Structure = LSM Trees Uses Google s LevelDB API Space efficient + compression Excellent core scaling Best Use Point queries Updates Easy incremental backups Has very advanced func9onality. Lots of poten9al

17 What s Up With: LSMs Writes go to memtable + journal Memtable fills up and overflows (flush) to file(s) Files are read only Acts like layers of logs Files are eventually merged and old files are marked for deletion Files are like small structured trees memtbl Level 0 Level 1 Level 2 Level 3 Level 4

18 What s Up With: LSMs Range Ops Range scans are tough Each file is it s own tree No good way to tell if data lies in any file Read amplification is H-I-G-H memtbl RANGE SCAN Level 0 Level 1 Level 2 Level 3 Level 4

19 What s Up With: LSMs Point Ops Point operations are tough too However, Bloom filters work well Filter determines if the required info exists in a set Can have false positives memtbl Level 0 Level 1 Level 2 Level 3 Level 4

20 Fractal Tree Indexes

21 Fractal - Insert

22 Fractal Message InjecOon

23 What s Up With: PerconaFT Overview Developed by MIT, SUNY Stony Brook & Rutgers Concurrency: Document level Unique data structure Fractal Tree Controls cache Compresses well (quicklz, zlib, lzma) Best Uses Best compression CPU efficient (rela<vely) Sequen<al workloads S9ll developing as a pluggable engine. Needs to learn API

24 Benchmarks Disclaimer: They re just benchmarks. It s all made up. (like economics & meteorology)

25 Insert Workload collec<ons = 8 database name = sbtest writer threads = 16 documents per collec<on = 10,000,000 feedback seconds = 20 auto commit = N run seconds = 1200 oltp range size = 100 oltp point selects = 0 oltp simple ranges = 0 oltp sum ranges = 0 oltp order ranges = 0 oltp disonct ranges = 0 oltp index updates = 0 oltp non index updates = 0 oltp inserts = 20 Applies to all benchmarks in this presentation

26 What s Up With: Writes 800 Mongo Engines - Write TPS TPS PerconaFT wiredtiger RocksDB Elapsed Seconds

27 Read Workload run seconds = 1200 oltp range size = 100 oltp point selects = 10 oltp simple ranges = 1 oltp sum ranges = 1 oltp order ranges = 1 oltp disonct ranges = 1 oltp index updates = 0 oltp non index updates = 0 oltp inserts = 0

28 What s Up With: Reads 1600 Mongo Engines - Read TPS Axis Title PerconaFT wiredtiger RocksDB Axis Title

29 Update Workload run seconds = 1200 oltp range size = 100 oltp point selects = 0 oltp simple ranges = 0 oltp sum ranges = 0 oltp order ranges = 0 oltp disonct ranges = 0 oltp index updates = 50 oltp non index updates = 5 oltp inserts = 0

30 What s Up With: Updates Mongo Engines - Updates TPS PerconaFT wiredtiger RocksDB Elapsed Seconds

31 Mixed Workload run seconds = 1200 oltp range size = 100 oltp point selects = 10 oltp simple ranges = 1 oltp sum ranges = 1 oltp order ranges = 1 oltp disonct ranges = 1 oltp index updates = 50 oltp non index updates = 5 oltp inserts = 10

32 What s Up With: Mixed Workloads 250 Mongo Engines - Mixed TPS PerconaFT 100 wiredtiger RocksDB Elapsed Seconds

33 EvaluaOon Resources Flashback replay Mongo opera<ons in real <me or as fast as possible with your workload Benchrun javascript benchmark harness in MongoDB. Cut out driver problems Sysbench & iibench for Mongo Yahoo Cloud Services Benchmark Mongo- perf *Whenever possible, run with YOUR workload, or a workload that accurately simulates yours.

34 Percona Live Data Performance Conference April in Santa Clara, CA at the Santa Clara Conven<on Center Register with code WebinarPL to receive 15% off at registra<on MongoDB, MySQL, NoSQL, Data in the Cloud