MYSQL DOCUMENT STORE

Transcription

1 MYSQL DOCUMENT STORE UNDER THE HOOD Jan Kneschke, Architect, MySQL Middleware and Clients Percona Live, April 2016

2 WHAT TO EXPECT What is MySQL Document Store Foundations of the X Protocol How X Plugin maps CRUD to SQL

3 { '_id': 1, 'name': 'Milk', 'properties': { 'volume_l': 1, 'amount': 32 } } DOCUMENT STORE A Document is typed structured as a tree no schema enforced

4 DOCUMENT STORE Store provides fast retrieval optimizations for updates operations Find, Update, Modify, Delete Projections Filters Aggregation

5 MYSQL DOCUMENT STORE part of MySQL X DevAPI MySQL Shell, Connector/J, C/.net and C/node.js X Protocol implemented by X Plugin using stable interfaces from MySQL 5.7 GA SQL service JSON support

6 MYSQL 5.7'S JSON SUPPORT MySQL added: JSON datatype JSON functions like JSON_EXTRACT(), JSON_OBJECT(),... Generated Virtual/Stored Columns Function Indexes based on JSON datatype and functions

7 MYSQL 5.7'S JSON SUPPORT EXAMPLES CREATE TABLE product ( doc JSON, _id CHAR(16) AS (JSON_EXTRACT(doc, "$._id") STORED), PRIMARY KEY(_id)); INSERT INTO product (doc) VALUES ( '{ "_id": 1, "name": "Milk", "properties": { "volume_l": 1, "amount": 32 } }'); SELECT JSON_OBJECT("amount", JSON_EXTRACT(doc, "$.properties.amount")) FROM product WHERE JSON_EXTRACT(doc, '$.name') == "Milk"

8 X DEVAPI working with Documents and Tables fluent API prod = sess.getschema("prod") res = prod.users. find("$.name = 'Milk'"). fields(["name", "properties"])

9 A NEW PROTOCOL The common set of queries: CRUD == many small PK reads queries are independent plenty of data sharding

10 X PROTOCOL optimize for common operations no manatory handshake reduced message sizes pipelining expectations notifications

11 X PROTOCOL Client Server Conn.GetCap() Negotiation Conn.Caps({"tls":...}) Authentication Sess.AuthStart(mech="MySQL41") Sess.AuthContinue Sess.AuthContinue Sess.AuthOk Statements Client Server

12 X PROTOCOL Client Server Authentication Sess.AuthStart(mech="MySQL41") Sess.AuthContinue Sess.AuthContinue Sess.AuthOk Statements Client Server

13 REDUCED MESSAGE SIZES message ColumnMeta { optional name string = 1; optional orig_name string = 2; optional catalog string = 3; //... } Note don t send orig_nameif it is equal to name don t send catalog if it is def

14 QUERY TIME Client Network Server stage network path latency exectime time 1ms 0.1ms

15 CLASSIC REQUEST/RESPONSE Client Server Sql.StmtExecute("INSERT INTO... (1)") exec Sql.StmtExecuteOk() Sql.StmtExecute("INSERT INTO... (2)") exec Sql.StmtExecuteOk() Client Server total: 4x path + 2x exectime = 4.2ms

16 PIPELINING Client Server packet Sql.StmtExecute("INSERT INTO... (1)") Sql.StmtExecute("INSERT INTO... (2)") exec, exec packet Sql.StmtExecuteOk() Sql.StmtExecuteOk() Client Server total: 2x path + 2x exectime = 2.2ms

17 MANAGING EXPECTATIONS Pipelining makes error handling harder Assumptions are usually wrong Fail early? Fail late? Statement Replication Lag? Max Execution Time

18 MANAGING EXPECTATIONS FAIL EARLY Client Server Expect.Open([+no_error]) Sql.StmtExecute("INSERT INTO... (1)") Sql.StmtExecute("INSERT INTO... (1)") Sql.StmtExecute("INSERT INTO... (2)") Expect.Close() Ok() Sql.StmtExecuteOk() Error("Dup Key") Error("Expectation Failed") Ok() Client Server

19 MANAGING EXPECTATIONS III NESTING Expect.Open([+no_error]) PrepStmt.Prepare(id=1, "INSERT INTO... (?)") Expect.Open([+no_error]) PrepStmt.Execute(id=1, values=[1]) PrepStmt.Execute(id=1, values=[2]) Expect.Close() PrepStmt.Close(id=1) Expect.Close()

20 MANAGING EXPECTATIONS IV try: prep = prepare(..) try: prep.execute([1]) prep.execute([1]) finally: prep.close() except Exception: pass NESTING, PYTHON STYLE

21 NOTIFICATIONS server to client can appear at any time local events belong to current statement warnings status variable changes

22 NOTIFICATIONS GLOBAL EVENTS async notification channel global events server going down connection timed out membership changed: node le group foundation for message bus, pub/sub,...

23 MINIMAL CLIENT BUILDING BLOCKS Protobuffor serialization SASLmethods for authentication Simple Authentication Security Layer PLAINover TLS MYSQL41 TLSfor encryption

24 MINIMAL CLIENT PROTOBUF message Find { required Collection collection = 2; optional DataModel data_model = 3; repeated Projection projection = 4; optional Mysqlx.Expr.Expr criteria = 5; repeated Mysqlx.Datatypes.Scalar args = 11; optional Limit limit = 6; repeated Order order = 7; repeated Mysqlx.Expr.Expr grouping = 8; optional Mysqlx.Expr.Expr grouping_criteria = 9; };

25 MINIMAL CLIENT PREPARATIONS Specification Message Definition Protobuf $ protoc I python_out=....../mysql.proto

26 MINIMAL CLIENT # connect c = socket.socket() c.connect((config.hostname, Config.port)) # start auth msg = Mysqlx_Session.AuthenticateStart( mech_name="mysql41") send_msg(c, msg) # recv challenge auth_challenge_msg = recv_msg(c)

27 MINIMAL CLIENT # send auth response msg = Mysqlx_Session.AuthenticateContinue( auth_data="%s\0%s\0*%s\0" % ( "", Config.auth_username, sasl_mysql41_client( Config.auth_password, auth_challenge_msg.auth_data))) send_msg(c, msg)

28 MINIMAL CLIENT # read all messages until Ok or Error while True: msg = recv_msg(c) if msg. class in [Mysqlx.Error, Mysqlx_Session.AuthenticateOk]: break

29 Q&A Questions, please.