Introduction to the MySQL Document Store Alfredo Kojima, Rui Quelhas, Mike Zinner MySQL Middleware and Clients Team October 22, 2018

Transcription

1 Introduction to the MySQL Document Store Alfredo Kojima, Rui Quelhas, Mike Zinner MySQL Middleware and Clients Team October 22, 2018

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle s products may change and remains at the sole discretion of Oracle Corporation.

3 Program Agenda Why a Document Store? Introducing The MySQL Document Store Classic Database Application Development Improved Development With the MySQL Document Store Demo

4 Why a Document Store?

5 Relational Databases vs. Document Store Systems Relational Model customer id name city_id 3412 John Smith 45 city id city country_id 45 San Francisco US shop_order id id_customer date total Document Model JSON Document Example { } _id: 3412, name: "John Smith", john@oracle.com, city: "San Francisco", country: "US", shop_orders: [ {date: " ", total: }, {date: " ", total: 24.95} ]

6 Relational Databases vs. Document Store Systems Which system is better? Relational Model Document Model Steeper learning curve of SQL Easier to get started with Higher upfront effort during design stage (data modeling/normalization/physical schema definition) Query language to express complex queries with multi-table JOINs Schema changes during deployment (esp. for upgrades) can be problematic Lower upfront effort, data structure defined in application code Limited or no JOINs, complexity offloaded to application Simpler deployment. Documents created in different application versions can coexist. Database ensures integrity (invalid data rejected) via data types, constraints There are advantages on both sides. Depending on the use case, either can be a good choice. Application responsible for data integrity, possibility for bad data to creep into the system

7 Why not Have both schema-less and schema in the same technology stack? One that checks all the boxes of all stakeholders: Developers: Schema & Schemaless Rapid Prototyping & Simple APIs & SQL Document Model & Relational Model Transactions & Consistency Operations: Performance Management & Visibility Robust Replication & Backup & Restore Comprehensive Tooling Ecosystem Simpler application schema upgrades Business Owner: Don t lose my data = ACID transactions Capture all my data = Extensible/Schemaless Products On Schedule/Time to Market = Rapid Development Security

8 Introducing The MySQL Document Store Cross(X)over between Relational- and Document Model

9 The MySQL Document Store is now GA Available on every MySQL 8 installation by default Adds document oriented JSON data storage for MySQL Makes MySQL suited for both, structured and unstructured data Store JSON Documents in MySQL through classic SQL JSON type columns or new X DevAPI interface New X DevAPI interface for SQL + NoSQL via new X Protocol Support for simple CRUD + SQL in single cross(x)over interface Much easier to work with documents in Collections Freely mix and match document and relational data according to specific needs Full Support in all MySQL 8 Components MySQL Server 8.0, MySQL Shell 8.0, MySQL Router 8.0, MySQL Connectors 8.0

10 The MySQL Document Store Key Take-Away Available on every MySQL 8 installation by default Combines MySQL / InnoDB strengths with NoSQL Easy to use via X DevAPI combining CRUD interface with classic SQL RDBMS features like Transactions & Locking & Efficient query optimization Referential integrity / foreign keys (optional) Traditional MySQL Performance Integrates with upcoming scale-out solutions

11 Classic Database Application Development

12 Database Development The basic problem Application Objects Objects Objects Database How to get these persisted here

13 Database Development The ideal solution Application Objects Objects Database Objects Objects Objects Objects Objects Objects db.store(product); product = db.load(productid);

14 Database Development although there's more to it Concurrent access Consistency Integrity Performance Complex queries and reporting Analytics High Availability Scalability

15 Database Development Relational Databases Application Objects SQL Database Schema Objects Objects rows Tables Tables

16 Database Development Storing an Object as Rows in Relational Tables r = Restaurant() r.id = 1234 r.name = "100 Menu" r.rating = 5 r.types_of_food = ["Chinese", "Fusion"] r.address = " " r.url = " "

17 Database Development Storing an Object as Rows in Relational Tables r = Restaurant() r.id = 1234 r.name = "100 Menu" r.rating = 5 r.types_of_food = ["Chinese", "Fusion"] r.address = " " r.url = " " restaurants id name rating address URL Menu City R restaurant_food_types restaurant_id food_types id type 14 Chinese 15 Fusion food_type_id

18 Database Development Storing an Object as Rows in Relational Tables r = Restaurant() r.id = 1234 r.name = "100 Menu" r.rating = 5 r.types_of_food = ["Chinese", "Fusion"] r.address = " " r.url = " " INSERT INTO food_types (14, 'Chinese'), (15, 'Fusion'); INSERT INTO restaurant ('100 Menu', 5, ' ', ' '); INSERT INTO restaurant_food_types (25, 14), (25, 15); SELECT FROM restaurant r JOIN food_types t JOIN restaurant_food_types rt ON WHERE r.id = 1234 ; (1234, '100 Menu', 5, 'Chinese', ' ', ' '), (1234, '100 Menu', 5, 'Fusion', ' ', ' ') restaurant food_types restaurant_ food_types

19 Database Development Storing an Object as Rows in Relational Tables Schema creation Normalize data Design schema Deploy schema in physical DB Application development Serialize application objects as table rows Code/generate SQL for inserts and updates SQL for queries and fetching rows Unserialize application objects from query results

20 Database Development Storing an Object as Rows in Relational Tables Schema Adding new fields later requires modifying the schema Deploying schema changes is often complicated Limitations of the relational model Mapping application objects to table rows not always straightforward Arrays, hierarchies, nested structures etc Many-to-many relationships Normalization into multiple tables and JOINing often overkill

21 Database Development Document Databases Application Objects Document Database Objects Objects Document Collection

22 Database Development Storing Objects as (JSON) Documents r = Restaurant() r.id = 1234 r.name = "100 Menu" r.rating = 5 r.types_of_food = ["Chinese", "Fusion"] r.address = " " r.url = " "

23 Database Development Storing Objects as (JSON) Documents r = Restaurant() r.id = 1234 r.name = "100 Menu" r.rating = 5 r.types_of_food = ["Chinese", "Fusion"] r.address = " " r.url = " " { } "id" : 1234, "name": "100 Menu", "rating": 5, "types_of_food": ["Chinese", "Fusion"], "address": " ", "URL": " "

24 Database Development Storing Objects as Documents No explicit schema (optional?) Simpler development But application responsible for typical RDBMS tasks constraints, type checks, data integrity and consistency etc Simpler mapping to/from application objects and data structures Arrays, Dictionaries etc Simple to represent nested data

25 Improved Development With the MySQL Document Store

26 MySQL Document Store X DevAPI SQL + JSON JSON Documents with a Schemaless, NoSQL API JSON for relational tables and SQL

27 MySQL Document Store How it works Classic MySQL Protocol 3306 MySQL Connection handling MySQL Connector/J SQL Traditional Application Optimizer Storage MySQL Server

28 MySQL Document Store How it works X DevAPI X Protocol Classic MySQL Protocol X Plugin 3306 MySQL Connection handling MySQL Connector/J SQL Traditional Application MySQL C/Node.js X DevAPI Node.js Application X DevAPI SQL Result Optimizer Storage MySQL Server 8.0

29 MySQL Document Store How it works X Protocol Classic MySQL Protocol JavaScript / Python X DevAPI MySQL Shell X Plugin 3306 MySQL Connection handling MySQL Connector/J SQL Traditional Application MySQL C/Node.js X DevAPI Node.js Application MySQL Connector/Net X DevAPI Windows Application... SQL Result Optimizer Storage MySQL Server 8.0 Existing applications are not touched Developers adopt DocStore only if they want to X Protocol allows SQL and CRUD SQL: SELECT * FROM mytable WHERE id = 12; X DevAPI: db.mytable.find("id=12"); Directly hooked into the optimizer Bypass MySQL connection handling

30 MySQL Document Store Components MySQL Server 8.0 JSON Datatype MySQL X Plugin Introduces X Protocol for relational- and document operations Maps CRUD operations to standard SQL (relational tables, JSON datatype and functions) X Protocol New MySQL client protocol based on top of industry standard (Protobuf) Works for both, CRUD and SQL operations X DevAPI New, modern, async developer API for CRUD and SQL operations on top of X Protocol Introduces Collections as new Schema obj. MySQL Shell 8.0 Offers interactive X DevAPI mode for app prototyping MySQL Connectors 8.0 Support for X DevAPI and X Session

31 JSON with SQL (Powerful but complicated )

32 JSON with SQL Combine JSON data with relational tables JSON column type Path syntax Functions: extraction, modification, aggregation, relational mapping Indexing Virtual columns

33 JSON Datatype Store JSON data in table columns Efficient internal binary format: optimized for lookups and partial updates Max size = 4GB Contents validated

34 JSON Path Syntax Extract values from JSON data Address through nested objects, arrays etc. SELECT * FROM product WHERE attributes->'$.weight'

35 JSON Functions JSON_SEARCH() Search inside JSON value JSON_CONTAINS() Test whether a JSON value contains another JSON_SET() Set value in a JSON object or array JSON_INSERT() Insert values into a JSON value JSON_ARRAY_APPEND() Append to a JSON array JSON_MERGE_PATCH() Patch a JSON object using another JSON_EXTRACT() Extract a value from the etc

36 Constructing JSON from SQL JSON_OBJECT() Construct a JSON object JSON_OBJECT('name', user.name, ' ', user. ) JSON_ARRAY() Construct a JSON array JSON_ARRAY(user. 1, user. 2, user. 3)

37 Aggregating Rows as JSON JSON_ARRAYAGG() Aggregate rows into an array JSON_OBJECTAGG() Aggregate rows into an object SELECT user, JSON_ARRAYAGG(host) FROM mysql.user GROUP BY user;

38 JSON_TABLE() Makes complex queries inside JSON objects possible Maps a JSON value to a relational table Extract columns out of object attributes Extract one or more rows out of array elements Filter, re-aggregate, apply SQL functions etc Manipulate JSON as relational data

39 JSON_TABLE() id name addresses 23 Jack Doe [{"zip": "94103", "city": "San Francisco", "state": "CA", "address": "1350 Harrison St"}, {"zip": "94115", "city": "San Francisco", "state": "CA", "address": "1394 Steiner St"}, {"zip": "91505", "city": "Burbank", "state": "CA", "address": "823 N Maple St"}]

40 JSON_TABLE() id name addresses 23 Jack Doe [{"zip": "94103", "city": "San Francisco", "state": "CA", "address": "1350 Harrison St"}, {"zip": "94115", "city": "San Francisco", "state": "CA", "address": "1394 Steiner St"}, {"zip": "91505", "city": "Burbank", "state": "CA", "address": "823 N Maple St"}] JSON_TABLE(addresses, '$[*]' COLUMNS ( address VARCHAR(100) PATH '$.address', city VARCHAR(40) PATH '$.city', state VARCHAR(2) PATH '$.state', zip VARCHAR(9) PATH '$.zip')) id name address city state zip 23 Jack Doe 1350 Harrison St San Francisco CA Jack Doe 1394 Steiner St San Francisco CA Jack Doe 823 N Maple St Burbank CA 91505

41 X DevAPI (The easier way!)

42 X DevAPI NoSQL Document Store API CRUD API for Document Store Schema not required: Directly store JSON documents in MySQL SQL not required: Program in the same language as the application Available for Node/JS, Python, Java,.NET, C++

43 Documents A free-form JSON object { "_id": "55f14312c7447c3da7051b27", "name": "@ Thai", "rating": 5.5, "types_of_food": ["Thai"],! _id field is pre-defined to be the Primary Key string defaults to an auto-generated globally unique ID at insert optimized for sequential inserts

44 ! Document Collections Holds JSON documents MySQL table with pre-defined schema for storing documents only restaurants = db.createcollection("restaurants")! Regular InnoDB MySQL tables under the hood

45 CRUD add() Add document to the collection doc = { "_id": "55f14312c7447c3da7051b27", "name": "@ Thai", "rating": 5.5, "types_of_food": ["Thai"], }; restaurant.add(doc).execute();

46 CRUD getone() Simple method to return a specific document by its _id value doc = restaurant.getone(id)

47 CRUD find() More complex queries for collections docs = restaurant.find(":type in types_of_food"). bind("type", "Thai").execute().fetchAll(); Binding of placeholder variables

48 CRUD find() (cont.) Supports filtering, projection, aggregation, row locking, limiting result size etc. In the Result object: fetchone() Fetch and return one document from the result or None fetchall() Fetch and return an array with all documents from the result

49 CRUD modify() Modify matching documents Set field values, modify arrays, replace values etc. products.modify( _id = 123"). set("dimensions.weight", 50).execute();

50 CRUD remove() and removeone() removeone() Delete one document from the collection by its _id remove() Delete all matching documents from the collection restaurant.removeone("55f14312c7447c3da7051b29") restaurant.modify("_id = :id"). bind("id", "55f14312c7447c3da7051b29"). set("rating", 6.0).execute();

51 CRUD Transactions and Locking All transaction isolation levels supported by InnoDB session.starttransaction() session.commit() Row level locking Maintain consistency during concurrent updates Acquire READ or UPDATE locks during a find() matches = restaurants.find("zip = '95124'"). lockshared().execute().fetchall()

52 CRUD replaceone() and addorreplaceone() replaceone() Replace document with matching _id with the given one restaurant.replaceone("55f14312c7447c3da7051b29", newdoc) addorreplaceone() Replace document with matching _id, insert if it doesn't exist (upsert) restaurant.addorreplaceone(docid, newdoc)

53 Demo

54 MySQL Document Store Summary and Take Away New, modern way to develop database applications Combine best of relational and document oriented models Blogs: mysqlserverteam.com/category/docstore/

55 Getting Ready The Basics MySQL Server Binary downloads: MySQL Repos MySQL Shell

56 MySQL Document Store - X DevAPI Supported Languages JavaScript / Node.js Connector/Node.js Get it from npm or API Reference: C#/.NET Connector/Net Get it from NuGet or API Reference: Java Connector/J Get it from API Reference:

57 MySQL Document Store - X DevAPI Supported Languages Python Connector/Python Get it from API Reference: C++ Connector/C Get it from API Reference: PHP 7.1+ Get it from API Reference:

58 MySQL Document Store - X DevAPI More information X DevAPI Users Guide Using MySQL as a Document Store

59 Related Sessions [DEV5986] Node.js and the MySQL Document Store (*) [DEV5959] Python and the MySQL Document Store [DEV5981] Develop PHP Applications with MySQL X DevAPI [DEV5985] MySQL Connector/Node.js and the X DevAPI (*) [DEV6233] Connector/J Beyond JDBC: The X DevAPI for Java and the MySQL Document Store [HOL1703] A Practical Introduction to the MySQL Document Store [HOL1706] Developing Modern Applications with the MySQL Document Store and Node.js (*) shameless plug

60 Thank you!

61