High Performance NoSQL with MongoDB

Transcription

1 High Performance NoSQL with MongoDB

2 History of NoSQL June 11th, 2009, San Francisco, USA Johan Oskarsson (from organized a meetup to discuss advances in data storage which were all using distributed databases leveraging clusters. He asked the group for a short term they could use as a hashtag. [1] Eric Evans (not of DDD fame) proposed #NoSQL and it stuck.

3 Michael's NoSQL Definition Database systems which are cluster-friendly and which trade inter-entity relationships for both simplicity and performance.

4 Four types of "NoSQL" DBs Key Value Stores Amazon DynamoDB Redis Column-Oriented databases Hbase Cassandra Google BigQuery Graph Databases Neo4J OrientDB Document Databases MongoDB CouchDB DocumentDB (on Azure)

5 Key-value data storage

6 Column Oriented DBs

7 Graph DBs

8 Document DBs

9 Not so different

10 How much do you need perf? Image credit: nerovivo

11 Relational 3NF models are complex

12 Document DBs for simplicity Document db style

13 Document DBs for simplicity Document db style

14 Single server performance Single biggest performance problem (and fix)? Incorrect indexes (too few or too many)

15 Adding indexes Be data-driven: profile and then add indexes

16 Adding indexes Indexes are more important than for RDBMSes

17 Demo time

18 Step 1: Enable profiling

19 Step 2: Run common queries

20 Step 3: Analyze system.profile

21 Step 4: Add indexes for slow

22 Step 5: GOTO 1

23 Scaling out Image credit: credit: johnantoni Torkild Retvedt Image

24 Scaling out Scale-out is the great promise of NoSQL MongoDB has two modes of scale out Sharding Replication Real-word statistics from one company 120,000 DB operations / second 2GB of app-to-db I/O / second

25 Replication vs. scalability Sharding is the primary way to improve single query speed Replication is not the primary way to scale even though you may get better read performance, not much better write performance unless very read heavy Replication Server 1 A-B-C-D-E Sharding Server 1 A Server 2 A-B-C-D-E Server 3 A-B-C-D-E Server 2 B Server 3 C Server 4 A-B-C-D-E Server 5 A-B-C-D-E Server 4 D Server 5 E

26 Sharding...

27 Scaling via Sharding an example Weather data from the entire 20 th century in MongoDB Case study by MongoDB Inc:

28 Data size and quantity 2.5 billion data points 4 Terabyte (1.6k per document)

29 Sample record (JSON) { } "st" : "u725053", "ts" : ISODate(" T22:51:00Z"), "airtemperature" : { "value" : 21.1, "quality" : "5" }, "atmosphericpressure" : { "value" : , "quality" : "5" }

30 Sample record in C# class WeatherRecord { public string st {get; set;} public DateTime ts {get; set;} public Temp airtemperature {get; set;} public Pressure atmosphericpressure {get; set;} } class Temp { public int value {get; set;} public string quality {get; set;} } class Pressure { public int value {get; set;} public string quality {get; set;} }

31 Scale Up A single server with a really big disk Application mongod c3.8xlarge i2.8xlarge 251 GB RAM 6 TB SSD

32 Scale out configuration A really big cluster where everything is in RAM mongod Application / mongos c3.8xlarge x 61 GB RAM 100 GB disk

33 Can scale even more A really big cluster where everything is in RAM Application / mongos mongod x 61 GB RAM 100 GB disk

34 Cost per year in AWS? $60,000 / yr $700,000 / yr...

35 Performance: single time and place db.data.find({"st" : "u747940", "ts" : ISODate(" T12:00:00Z")}) 2 ms avg 95th 99th max. throughput: 0 single server 40,000/s cluster 610,000/s (10 mongos)

36 Performance: 1 year's weather db.data.find({"st" : "u747940", "ts" : {"$gte": ISODate(" "), "$lt" : ISODate(" ")}}) ms single server cluster max. throughput: 20/s 430/s (10 mongos) targeted query avg 95th 99th

37 Analytics db.data.aggregate([ { "$match" : { "airtemperature.quality" : { "$in" : [ "1", "5" ] } } }, { "$group" : { "_id" : null, "maxtemp" : { "$max" : "$airtemperature.value" } } } ]) 61.8 C = 143 F 4 h 45 min Single Server 2 min Cluster 142x faster

38 Get the code and data

39 Want to go deeper? talkpython.fm training.talkpython.fm