Run your own Open source. (MMS) to avoid vendor lock-in. David Murphy MongoDB Practice Manager, Percona

Transcription

1 Run your own Open source Click alternative to edit to Master Ops-Manager title style (MMS) to avoid vendor lock-in David Murphy MongoDB Practice Manager, Percona

2 Who is this Person and What Does He Know? Former MongoDB Master Former Lead DBA for ObjectRocket, original high-performance DBaaS for Mongo RDBMS background: 15+ years in MySQL NoSQL/MySQL Electronic Arts Contributor to Mongo Core and tool builder 2

3 MongoDB Replication and Sharding What is really in Ops Manager Monitoring Visualizing your schema AWS provisioning Backups Monitoring Options Schema Analysis Provisioning why stop at AWS? What is needed for backups? Bearing Fruit 3 Stand-alone & Replica sets Sharded

4 What is Ops / Cloud Manager Also know as MongoDB Inc s MMS as its older name Monitoring and alerting on 100+ metrics Provisioning new nodes on AWS Scheduled Backups Point in time replica set recovery Sharded snapshot recovery Visualizing your queries and schema (compass) 4

5 Monitoring Options Hosted ( Typically more closed source) New Relic Scout VividCortex SeveralNines Self-Run Nagios Prometheus+Grafana Munin/Ganglia 5

6 Monitoring Options What types of things should you monitor? CPU/MEM Flush Times Intra node latency Lock% or IO Queued Works Write Conflicts Connections Up/Down Replication Lag HA Events 6

7 Provisioning why stop at AWS? Many many tools out there for build nodes Personally I am looking at mist.io - github.com/mistio/mist.io Support many clouds EC2, Amazon, OpenStack, Docker, KVM, Softlayer, VMware, Digital Ocean, Linode, & more API for listing, adding, removing, managing machines Easy to implement wit Ansible or Chef to manage node based on tags Foreman is another great one - theforeman.org Chef/Docker/Puppet all have plugins for this also Eucalyptus and RightScale also are things to consider 7

8 Provisioning why stop at AWS? Key things to consider: Do I already have a cloud management tool? Can I use api s to write scripts to easily build new nodes? How often do I need to do scaling? Is it worth the invest in this overhead? ( If you're managing a handful of machines maybe not) 8

9 What is needed for solid backups? Everyone here has likely heard that MongoDB backups are not consistent. However this is rated as FALSE MongoDB in everything it does could be called optionally supporting. This leads to two questions 1) What is needed to take a consistent backup 2) What is needed to restore to a specific point in time. Unfortunately we need to break the answer into 2 parts Un-sharded and Sharded 9

10 Consistent Backups - Standalone & Replica sets Standalone Node Most basic LVM snapshot or dump works No point in time recovery due to no oplog Can affect running system more directly than other methods. Not consistent really Replica Set Out of the box you need dump Snapshots possible but scripts needed to get oplog Point in time recovery if oplog covers now to last backup time Backup on secondary avoids impact Consistent with dump & --oplog flag 10

11 Consistent Backups - Standalone & Replica sets Easily the most complicated area All tools are per node not cluster aware. You need to additionally backup a config server Generally Point in time recovery is very hard due to metadata changes Sharded Mongo If you're using 3.2 & and oplog trailer you can have consistent sharded backups. Before 3.2 only snapshot backups are suggested, and point in time can be done if your have you do a lot of scripting. 11

12 Schema Analysis tools Variety JS based table scan and sorting Pretty slow ASCII and JSON output Schema-analyzer ( soon to be on github.com/percona-lab) Supports 3.2+ (for now soon to be 3.0) Uses $lookup for good random sampling Gnuplot to build stacked bar or types used per field In 3.0 and previous will try to randomly pick some id values Faster and Avoids table scans 12

13 13 Schema Analysis tools - Schema-analyzer(alpha)

14 Percona Management & Monitoring (PMM) Much of the beta announcement around PMM is MySQL, however Mongo is also going to be a star player very soon. You can expect it to grow in Monitoring, Management, and Provisioning in time. Helping to centralize all of the topics I mentioned today. Please reach out to us and let us know what you want in those area, and why it s valuable to your groups. 14

15 Percona-Lab Github Keep watching for tools and snippets: NOT officially supported tools yet, think of like an incubator. My group plans to publish some backup scripts and an oplog tailer We already have monitoring and schema visualizers what other tools you want These are all at your own risk until they are added to Percona Tools or Percona Management & Monitoring officially. 15

16 Thanks and Enjoy the Conference! Follow me on Twitter Percona is hiring, are you ready to be challenged?