Working with MongoDB as MySQL DBA. Date: Oct

Transcription

1 Working with as DBA Date: Oct

2 About us Lead Database Consultant since Lead Database OSDB managed services since 2014

3 Overview What is, what Replication in compared to Resyncing nodes after crash Changing variables and configuration files Status check and processlist Security and authentication Schema design, DDL, DML Storage engine and concurrency Error and Slow query logs Backups and recovery

4 What is Open-source relational database management system (RDBMS) Stores data in tables and uses Structured Query Language (SQL) for database access Predefined database schema based requirements and rules to govern the relationships between fields in your tables Related information may be stored in separate tables, but associated through the use of joins Data duplication is minimized

5 What is Open-source database, stores data in JSON-like documents that can vary in structure Related information is stored together for fast query access through the query language Uses dynamic schemas, creating records without first defining the structure Changing the structure simply by adding new fields or deleting existing ones Ability to represent hierarchical relationships, to store arrays, and other more complex structures easily Documents in a collection need not have an identical set of fields and denormalization of data is common

6 replication Group of mysqld processes that maintain the same data set Write operations go to the Master node, reads can go to Slave All changes are recorded into binary log on the master Asynchronous replication, Semi-Synchronous replication, virtually synchronous replication (Galera) Slave node replicates the events by copying binary logs as relay logs IO Thread, SQL Thread Binary Logs format - Statement, Row, Mixed Expire logs days dynamic Replication filters

7 replication Master Process Slave Process IO Thread Data SQL Thread Data Binary Logs Relay logs

8 Replication Group of mongod processes that maintain the same data set Write operations go to the Primary node All changes are recorded on all nodes into operations log - oplog Asynchronous replication, Statement based Secondaries replicate the Primary oplog Secondary can use sync source other Secondary Automatic failover Oplog size change Adding new node

9 Replication

10 Crash recovery InnoDB does crash recovery MyISAM can repair tables Restore from backup - reseed replication No option for automated initial sync Does not handle the quorum Primary node crash. Election of new Primary from majority Replay events from Oplog - idempotent Clear data dir and restart node. Initial Sync Repair database

11 Failover There is no build-in automatic failover Alternatives can be MHA, HAProxy, ProxySQL, etc. No option for automated initial sync Primary nodes goes down for more than 10 seconds "heartbeattimeoutsecs" : 10 Secondary node holds election Secondary with majority of votes becomes new Primary

12 Variables and configuration file /etc/my.cnf [client] port = 3306 socket = /var/run/mysqld/mysql.sock [mysqld] port = 3306 user = mysql socket = /var/run/mysqld/mysql.sock basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp log_error = hostname_error.log slow_query_log_file = hostname_slow.log innodb_buffer_pool_size = 128M innodb_file_per_table = 1 [mysqldump] max_allowed_packet = 64M /etc/mongod.conf systemlog: destination: file path: /var/log/mongodb/mongod.log logappend: true storage: dbpath: /mongodb/data journal: enabled: true wiredtiger: engineconfig: cachesizegb: 2 net: bindip: port: operationprofiling: slowopthresholdms: 100 mode: off

13 Server status mysql> SHOW GLOBAL STATUS; Variable_name Value Aborted_clients 0 Aborted_connects 0 Connections Created_tmp_disk_tables 0 Created_tmp_tables 8340 Created_tmp_files Threads_connected 1 Threads_running 1 Uptime PRIMARY> db.serverstatus() { "host" : "mongodb1",... "version" : "3.0.11", "process" : "mongod", "pid" : NumberLong(15805), "uptime" : , "uptimeestimate" : , "localtime" : ISODate(" T14:39:43.769Z",... "connections" : { "current" : 802, "available" : 50398, "totalcreated" : NumberLong(644684) }

14 Processlist mysql> SHOW FULL PROCESSLIST\G ************* 1. row *************** Id: 3123 User: igor Host: localhost db: test Command: Query Time: 0 State: NULL Info: SHOW FULL PROCESSLIST PRIMARY> db.currentop() { "inprog" : [ { "desc" : "conn531228", "threadid" : "0xe856680", "connectionid" : , "opid" : , "active" : true, "secs_running" : 0, "microsecs_running" : NumberLong(995), "op" : "query", "ns" : "hbrg.user.cart", "query" : { "$query" : { "user" : "C2ABBA1F D8-7A544F15EA94" }},... "plansummary" : "IXSCAN { user: 1 }", } ] }

15 Security and authentication CREATE USER IDENTIFIED BY 'changeme1'; GRANT ALL ON app_db.* TO There are not Roles in :( 8 will support roles. CREATE ROLE 'app_developer'; GRANT ALL ON app_db.* TO 'app_developer'; security: keyfile: <path-to-keyfile> use admin db.createuser( { user: "admin", pwd: "changeme1", roles: [ { role: "useradminanydatabase", db: "admin" } ] }) GRANT 'app_developer' TO 'dev1'@'localhost'; db.getsiblingdb("admin").auth("admin", "changeme1" )

16 Security and authentication Encrypt data in transit with SSL /etc/my.cnf [mysqld] ssl-ca=ca.pem ssl-cert=server-cert.pem ssl-key=server-key.pem mysql> GRANT USAGE ON *.* TO REQUIRE SSL; Encrypt data in transit with SSL /etc/mongod.conf net: ssl: mode: requiressl PEMKeyFile: /etc/ssl/mongodb.pem CAFile: /etc/ssl/ca.pem shell> mysql --ssl-ca=ca.pem \ --ssl-cert=client-cert.pem \ --ssl-key=client-key.pem shell> mongo --ssl --host hostname.example.com --sslcafile /etc/ssl/ca.pem

17 Schema design, DDL, DML mysql> CREATE DATABASE test; mysql> USE test; PRIMARY> use test switched to db test CREATE TABLE `users` ( `id` INT(10) NOT NULL AUTO_INCREMENT, `age` CHAR(3) NULL DEFAULT NULL, `status` ENUM('A','P') NOT NULL DEFAULT 'P', `name` VARCHAR(50) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB; PRIMARY> db.createcollection( "users", ) { validator: { $and: } } [ { name: { $type: "string" } }, { status: { $in: [ "A", "P" ] } } ] * Available from 3.2

18 Schema design, DDL, DML mysql> SHOW TABLES; Tables_in_test users mysql> INSERT INTO `users` (`age`, `status`, `name`) VALUES ( '45', 'A', 'John'); mysql> SELECT * FROM users; id age status name A John PRIMARY> show collections users PRIMARY> db.users.insert({"age": "45", "status": 'A', "name": "John"}) WriteResult({ "ninserted" : 1 }) PRIMARY> db.users.find().pretty() { "_id" : ObjectId("57c5d9ab0b044cb923fffaf9"), } "age" : "45", "status" : "A", "name" : "John"

19 Schema design, DDL, DML mysql> INSERT INTO `users` (`age`, `status`, `name`) VALUES ( '25', 'R', 'Tom'); ERROR 1265 (01000): Data truncated for column 'status' at row 1 mysql> SELECT * FROM users; id age status name A John PRIMARY> db.users.insert({"age": "25", "status": 'R', "name": "Tom"}) WriteResult({ "ninserted" : 0, "writeerror" : { "code" : 121, "errmsg" : "Document failed validation" }}) PRIMARY> db.users.find().pretty() { "_id" : ObjectId("57c5dbd67fd59bca3fb314ac"), } "age" : "45", "status" : "A", "name" : "John"

20 Schema design, DDL, DML mysql> UPDATE users SET status='p' WHERE id=1; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 PRIMARY> db.users.update({ "_id" : ObjectId("57c5dbd67fd59bca3fb314ac")}, {$set : {"status":'p'} } ) WriteResult({ "nmatched" : 1, "nupserted" : 0, "nmodified" : 1 }) mysql> truncate users; delete from users; PRIMARY> db.users.remove( {} ) mysql> ALTER TABLE `users` ADD COLUMN `position` VARCHAR(50) NULL AFTER `name`; mysql> UPDATE `users` SET `position`='dba' WHERE `id`=1; PRIMARY> ALTER collection users? PRIMARY> db.users.update({ "_id" : ObjectId("57c5dbd67fd59bca3fb314ac")}, {$set : {"position":"dba"} } )

21 Schema design, DDL, DML mysql> ALTER TABLE `users` ADD INDEX `name_idx` (`name`); mysql> explain select * from users where name = 'John'\G *********** 1. row ************** id: 1 select_type: SIMPLE table: users type: ref possible_keys: name_idx key: name_idx.. rows: 1 Extra: Using index condition 1 row in set (0.00 sec) mysql> DROP TABLE users; PRIMARY> db.users.createindex({"name":1}) PRIMARY> db.users.find({"name":"john"}).explain() { } "winningplan" : { "stage" : "FETCH", "inputstage" : { "stage" : "IXSCAN", "keypattern" : { "name" : 1 }, "indexname" : "name_1", "indexbounds" : {"name" : [ PRIMARY> db.users.drop( ) "[\"John\", \"John\"]" ] }

22 Storage engines MyISAM InnoDB MyRocks Memory CSV Archive Federated Blackhole MMAPv1 WiredTiger RocksDB In-memory PerconaFT

23 Log files Error log Slow query log Error log Slow query log /var/log/mysql/error.log /var/log/mysql/slow_query.log /var/log/mongodb/mongod.log

24 Backups and recovery mysqldump Logical backup Slow to create and worse for restore mydumper Logical backup Still slow but faster than mysqldump xtrabackup Binary backup Fast to create and restore mysqlbackup NOT Open Source Binary backup mongodump Logical backup Slow to create and restore File system snapshot LVM, EBS snapshot cp or rsync Copy files from freezed node Cloud manager Monthly subscription Point in time backup Ops manager (Enterprise) Yearly subscription Point in time backup

25 Questions?

26 We re Hiring!