MySQL Document Store and Node.js

Transcription

1 MySQL Document Store and Node.js Johannes Schlüter Rui Quelhas MySQL Clients and Middleware September 2017 Copyright 2017, Oracle and/or its affiliates. All rights reserved. 1

2 Follow along h ps://ruiquelhas.github.io/percona-live-europe-2017/ 2

3 (Rough) Outline 1. Unstructured Data 2. MySQL 3. Document Store 4. X Protocol 5. Node.js 6. MySQL Connector/Node.js 7. Pu ng it all together 8. Demo 3

4 Unstructured Data Not only rela onal data O en data of unstructured nature Data from external sources 4. 1

5 Unstructured Data { } "type": "node", "id": , "lat": , "lon": , "tags": { "amenity": "pub", "name": "The Viscount" } 4. 2

6 Unstructured Data { } "type": "node", "id": , "lat": , "lon": , "tags": { "amenity": "pub", "name": "The Viscount" } 4. 3

7 Unstructured Data { } "type": "node", "id": , "lat": , "lon": , "tags": { "addr:city": "Dublin", "addr:housenumber": "21", "addr:street": "Store Street", "amenity": "pub", "name": "Le Monde", "wheelchair": "no" } 4. 4

8 Unstructured Data { } "type": "node", "id": , "lat": , "lon": , "tags": { "amenity": "pub", "name": "The Ballinteer House", "wheelchair": "limited" } 4. 5

9 Unstructured Data { } "type": "node", "id": , "lat": , "lon": , "tags": { "accommodation": "yes", "addr:street": "Lombard Street East", "amenity": "pub", "opening_hours": "Mo-Th 09:00-23:30;Fr-Sa 09:30-01:30;Su 10:00-23:00", "name": "The Lombard",... } 4. 6

10 Unstructured Data JSON data type Introduced in MySQL 5.7 Efficient storage Set of func ons to create, modify and access elements Generated columns allow indexing parts of a document 4. 7

11 Unstructured Data Access using SQL CREATE TABLE pubs (doc JSON); INSERT INTO pubs VALUES ('{ "type":"node"... }'); ALTER TABLE pubs ADD COLUMN wheelchair VARCHAR(255) GENERATED ALWAYS AS (json_unquote(json_extract(doc, '$.tags.wheelchair'))) STORED KEY; SELECT * FROM pubs WHERE wheelchair!= 'no'; SELECT * FROM pubs WHERE json_unquote(doc->'$.tags.wheelchair')!= 'no'; 4. 8

12 MySQL is a Great Database Powerful Features InnoDB Replica on, Group Commit,... Proven and Stable 5. 1

13 Document Databases Simple to setup Higher level APIs 5. 2

14 So, why not combine both!? 5. 3

15 MySQL Document Store An X-Over between a rela onal database and a document database 6. 1

16 MySQL Document Store Built upon a new protocol, the X Protocol Implemen ng a common API over mul ple languages and environments C++, Java,.Net, Node.JS, Python. PHP New MySQL Shell Easy CRUD opera ons on rela onal tables and document database style collec ons Based on proven MySQL founda on 6. 2

17 MySQL Document Store Collec ons An Collec on is a set of JSON Documents Each document has it's unique ID in an _id property Easy to search within documents Implemented as a table with a JSON column and indexes on generated columns 6. 3

18 X Protocol Protocol Insights New Protocol implemented in X Plugin Based on Google Protocol Buffers More flexibility for future extensions Be er understanding of content CRUD Expression trees in protocol Expecta ons to a ach condi ons on pipelined statements Bound parameters 7. 1

19 X Protocol Protocol opera ons message ClientMessages { enum Type { CON_CAPABILITIES_GET = 1; CON_CAPABILITIES_SET = 2; CON_CLOSE = 3; SESS_AUTHENTICATE_START = 4; SESS_AUTHENTICATE_CONTINUE = 5; SESS_RESET = 6; SESS_CLOSE = 7; SQL_STMT_EXECUTE = 12; CRUD_FIND = 17; CRUD_INSERT = 18; 7. 2

20 X Protocol CRUD Example Mysqlx.Crud.Find { collection { name: "collection_name", schema: "test" } data_model: DOCUMENT criteria { type: OPERATOR operator { name: "==" param { type: IDENT, identifier { name:"_id" } } param { type: LITERAL, literal { type: V_STRING, v_string: { value: "some_string" } } } } } } 7. 3

21 Node.js (straight) from the horse's mouth Node.js is a JavaScript run me built on Chrome's V8 JavaScript engine. Node.js uses an event-driven, nonblocking I/O model that makes it lightweight and efficient. Node.js' package ecosystem, npm, is the largest ecosystem of open source libraries in the world. 8. 1

22 Node.js Overview JavaScript server run me on top of V8 HTTP as a first-class ci zen Filesystem, OS, network (TCP/UDP) and TLS Out-of-the box streaming APIs Non-blocking IO through the event loop Bindings for C/C++ na ve addons Unified module system ( CommonJS) and package manager ( npm) 8. 2

23 Node.js Programming model Single-threaded environment (it's JavaScript a er all) Asynchronous callback-based core Non-blocking opera ons run immediatelly and sequen ally Blocking opera ons queued outside execu on scope Result of a blocking opera on is handled "eventually" Concurrency with C++ APIs (threads) or mul ple processes 8. 3

24 Node.js Dealing with asynchronicity Push-based abstrac ons for asynchronous callbacks: Event emi ers Streams (special kind of event emi ers) Promises Pull-based abstrac ons for asynchronous callbacks: Generators Async/await Many other userland experiments of both kinds 8. 4

25 Node.js Callback pa ern const asyncfn = (success, callback) => { settimeout(() => { if (!success) return callback(new Error('bar')) callback(null, 'foo') }, 1000); } asyncfn(true, (err, data) => { }) console.log(data) // foo asyncfn(false, (err) => { }) console.log(err.message) // bar 8. 5

26 Node.js Event emi er const emitter = new EventEmitter() emitter.on('someevent', message => { }) console.log(message) // foo emitter.emit('someevent', 'foo') 8. 6

27 Node.js Streams const readable = new Stream.Readable() const writable = new Stream.Writable() readable.on('data', data => { }) console.log(data) // foo readable.pipe(writable) writable.write('foo') 8. 7

28 Node.js Promises const promise1 = Promise.resolve('foo') const promise2 = Promise.reject(new Error('bar')) promise1.then(data => { }) console.log(data) // foo promise2.catch(err => { }) console.log(err.message) // bar 8. 8

29 Node.js Generators const gen = function * () { yield 'foo' return 'bar' } console.log(gen.next()) // { value: 'foo', done: false } console.log(gen.next()) // { value: 'bar', done: true } 8. 9

30 Node.js Async/await const asyncfn1 = () => Promise.resolve('foo') const asyncfn2 = () => Promise.reject(new Error('bar')) (async function () { try { const value = await asyncfn1(); console.log(value) // 'foo' await asyncfn2() } catch (err) { console.log(err.message) // 'bar' } }()) 8. 10

31 MySQL Connector/Node.js Overview Official implementa on Based on the brand new X protocol Asynchronous and Promise-based Supports async/await Fluent/chainable API (out-of-the-box query builder) Flexible parameter methods 9. 1

32 MySQL Connector/Node.js Some features Document store CRUD API Rela onal CRUD API Fallback to plain old SQL Transac ons and row locking Secure sessions (SSL/TLS) Authen ca on Configura on handling interface 9. 2

33 MySQL Connector/Node.js Fluent API (Document Store) collection.add({ name: 'foo' }).add({ name: 'bar' }).execute() collection.find('$.name == :name').bind(name, 'foo').execute() collection.modify('$.name == "foo"').set('$.name', 'qux').execute() collection.remove('$.name == "bar"').execute() 9. 3

34 MySQL Connector/Node.js Database opera ons // // CollectionAdd const addoperation = collection.add() // CollectionFind const findoperation = collection.find() // CollectionModify const modifyoperation = collection.modify() // CollectionRemove const removeoperation = collection.remove() 9. 4

35 MySQL Connector/Node.js CollectionAdd 9. 5

36 MySQL Connector/Node.js CollectionFind 9. 6

37 MySQL Connector/Node.js CollectionModify 9. 7

38 MySQL Connector/Node.js CollectionRemove 9. 8

39 MySQL Connector/Node.js Fallback to plain-old SQL // insert document into a collection session.sql(`insert INTO pubs (doc) VALUES ('{"_id": ,"tags":{"name":"foo"}}')`).execute() // project "name" property values session.sql(`select JSON_EXTRACT(doc, '$.tags.name') FROM pubs`).execute() // create a virtual column and use it as an index session.sql(`alter TABLE collection ADD COLUMN name TEXT(50) GENERATED ALWAYS AS (JSON_EXTRACT(doc, '$.tags.name')) VIRTUAL, ADD INDEX i_name (name)`).execute() 9. 9

40 MySQL Connector/Node.js Asynchronous API <=> Promise getsession() createschema() dropschema() createcollection() dropcollection() execute() 9. 10

41 MySQL Connector/Node.js Simple push-based "cursors" // the cursor gets executed for each element of the result set function cursor (p) { // p: result set element currently being processed } collection.find().execute(cursor) 9. 11

42 MySQL Connector/Node.js Opera on comple on collection.find().execute(function (d) { // d: document instance }).then(function (r) { // r: instance of Result (coming up next) }).catch(function (e) { // e: instance of Error }) 9. 12

43 MySQL Connector/Node.js Result getdocumentid() getdocumentids() getaffectedrowscount() getwarnings() getwarningscount() 9. 13

44 Pu ng everything together Boilerplate, sessions and schemas $ npm const mysqlx = require('@mysql/xdevapi') const session = await mysqlx.getsession('mysqlx://demo@localhost:33060') // re-create the schema await session.dropschema('ple2017') const schema = await session.createschema('ple2017') 10. 1

45 Pu ng everything together Working with collec ons and documents (CRUD) const schema = session.getschema('ple2017') // re-create a collection await schema.dropcollection('pubs') const collection = await schema.createcollection('pubs') // or re-use an existing collection const collection = await schema.createcollection('pubs', { ReuseExistingObject: true }) 10. 2

46 Pu ng everything together CREATE const collection = schema.getcollection('pubs') // pipelining parallel calls await Promise.all([ collection.add({ type: 'node', _id: ,... }).execute(), collection.add([ { type: 'node', _id: ,... }, { type: 'node', _id: ,... } ]).execute() ]) 10. 3

47 Pu ng everything together READ const collection = schema.getcollection('pubs') const pubs = [] await collection.find().execute(pub => pub && pubs.push(pub)) console.log(pubs) // [{ _id: ,... }, { id: ,... },... ] 10. 4

48 Pu ng everything together UPDATE const collection = schema.getcollection('pubs') await collection.modify('$.id == ').set('type', 'foo').set('oldtype', 'node').execute() const pubs = [] await collection.find().execute(pub => pub && pubs.push(pub)) console.log(pubs) // [{ type: 'foo', _id: , oldtype: 'node' },... ] 10. 5

49 Pu ng everything together DELETE const collection = schema.getcollection('pubs') await collection.remove('$.type == "foo"').execute() const pubs = [] await collection.find().execute(pub => pub && pubs.push(pub)) console.log(pubs) // [{ type: 'node', _id: ,... }, { type: 'node', _id: ,... }] 10. 6

50 Pu ng everything together Transac on Support try { await session.starttransaction() const result = await schema.getcollection('beers').add({ name: 'Guinness', pubs: [ ] }).execute() await schema.getcollection('pubs').modify('$._id == ').arrayAppend('beers', result.getdocumentid()).execute() await session.commit() } catch (err) { await session.rollback() } 10. 7

51 DEMO h ps://gitlab.com/ruiquelhas/pub-crawler 11

52 Links h ps://dev.mysql.com/doc/refman/8.0/en/document-store.html h ps://dev.mysql.com/doc/x-devapi-userguide/en/ h ps:// h ps://github.com/mysql/mysql-connector-nodejs h ps://dev.mysql.com/doc/dev/connector-nodejs/ 12

53 Some related Sessions Tomorrow 9:15AM State of the Dolphin (Keynote by Geir Høydalsvik) 11:20AM Introduc on to MySQL InnoDB Cluster (Frederic Descamps, Jan Kneschke) 4:55PM MySQL Document Store (Frederic Descamps, Jan Kneschke) 13. 1

54 Thank You 13. 2

55 14

56 Safe Harbor Statement The preceding is intended to outline our general product direc on. It is intended for informa on purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or func onality, and should not be relied upon in making purchasing decisions. The development, release, and ming of any features or func onality described for Oracle s products remains at the sole discre on of Oracle. 15