Combining SQL and NoSQL with MySQL

Transcription

1 Combining SQL and NoSQL with MySQL Manyi Lu Director MySQL Optimizer Team, Oracle April, 2018 Copyright 2016, 2014, Oracle and/or its affiliates. All rights reserved.

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Agenda JSON data type JSON functions Indexing JSON data Partial update of JSON 3

4 The New JSON Datatype CREATE TABLE employees (data JSON); INSERT INTO employees VALUES ('{"id": 1, "name": "Jane"}'), ('{"id": 2, "name": "Joe"}'); SELECT * FROM employees; data {"id": 1, "name": "Jane"} {"id": 2, "name": "Joe"} Validation on INSERT No reparsing on SELECT Optimized for read Dictionary of sorted keys Can compare JSON/SQL Can convert JSON/SQL Supports all native JSON datatypes Also supports date, time, timestamp etc. 4

5 Agenda JSON data type JSON functions Indexing JSON data Partial update of JSON 5

6 JSON Functions MySQL 5.7 and 8.0 JSON_ARRAY_APPEND() JSON_ARRAY_INSERT() JSON_ARRAY() JSON_CONTAINS_PATH() JSON_CONTAINS() JSON_DEPTH() JSON_EXTRACT() JSON_INSERT() JSON_KEYS() JSON_LENGTH() JSON_MERGE[_PRESERVE]() JSON_OBJECT() JSON_QUOTE() JSON_REMOVE() JSON_REPLACE() JSON_SEARCH() JSON_SET() JSON_TYPE() JSON_UNQUOTE() JSON_VALID() JSON_PRETTY() JSON_STORAGE_SIZE() JSON_STORAGE_FREE() JSON_ARRAYAGG() JSON_OBJECTAGG() JSON_MERGE_PATCH() JSON_TABLE() 6

7 Inlined JSON Path Expressions [[database.]table.]column-> $<path spec> SELECT * FROM employees WHERE data-> $.name = Jane"; Is a short hand for SELECT * FROM employees WHERE JSON_EXTRACT(data, $.name ) = Jane ; SELECT * FROM employees WHERE data->'$.id'= 2; ALTER ADD COLUMN id INT AS (data->'$.id') CREATE VIEW.. AS SELECT data->'$.id', data->'$.name' FROM 7

8 JSON_TABLE JSON is not limited to CRUD, can also be used in complex queries JSON_TABLE creates a relational view of JSON data Each object in a JSON array as a row JSON values within an JSON object as column values Query the result of JSON_TABLE() as a relational table using SQL Leverage existing framework for aggregation 8

9 JSON_TABLE CREATE TABLE t1(json_col JSON); INSERT INTO t1 VALUES ( '{ "people": [ { "name":"john Smith", "address":"780 Mission St, San Francisco, CA 94103"}, { "name":"sally Brown", "address":"75 37th Ave S, St Cloud, MN 94103"}, { "name":"john Johnson", "address":"1262 Roosevelt Trail, Raymond, ME 04071"} ); ] }' 9

10 JSON_TABLE Convert JSON documents to relational tables SELECT people.* FROM t1, JSON_TABLE(json_col, '$.people[*]' COLUMNS ( name VARCHAR(40) PATH '$.name', address VARCHAR(100) PATH '$.address')) people; name address John Smith 780 Mission St, San Francisco, CA Sally Brown 75 37th Ave S, St Cloud, MN 9410 John Johnson 1262 Roosevelt Trail, Raymond, ME

11 JSON_TABLE Filter JSON data SELECT people.* FROM t1, JSON_TABLE(json_col, '$.people[*]' COLUMNS ( name VARCHAR(40) PATH '$.name', address VARCHAR(100) PATH '$.address')) people; WHERE people.name LIKE 'John%'; name address John Smith 780 Mission St, San Francisco, CA John Johnson 1262 Roosevelt Trail, Raymond, ME

12 JSON_TABLE Nested Arrays [ ] { "father":"john", "mother":"mary", "marriage_date":" ", "children": [ { "name":"eric", "age":12 }, { "name":"beth", "age":10 } ] }, { "father":"paul", "mother":"laura", "children": [ { "name":"sarah", "age":9}, { "name":"noah", "age":3}, { "name":"peter", "age":1} ] } id father married child_id child age 1 John 1 1 Eric 12 1 John 1 2 Beth 10 2 Paul 0 1 Sarah 9 2 Paul 0 2 Noah 3 2 Paul 0 3 Peter 1 12

13 JSON_TABLE Nested Arrays JSON_TABLE (families, '$[*]' COLUMNS ( id FOR ORDINALITY, father VARCHAR(30) PATH '$.father', married INTEGER EXISTS PATH '$.marriage_date', NESTED PATH '$.children[*]' COLUMNS ( child_id FOR ORDINALITY, child VARCHAR(30) PATH '$.name', age INTEGER PATH '$.age ) ) id father married child_id child age 1 John 1 1 Eric 12 1 John 1 2 Beth 10 2 Paul 0 1 Sarah 9 2 Paul 0 2 Noah 3 2 Paul 0 3 Peter 1 13

14 JSON_TABLE SQL aggregation on JSON data SELECT father, COUNT(*) "#children", AVG(age) "age average" FROM t, JSON_TABLE (families, '$[*]' COLUMNS ( id FOR ORDINALITY, father VARCHAR(30) PATH '$.father', NESTED PATH '$.children[*]' COLUMNS (age INTEGER PATH '$.age ) ) AS fam GROUP BY id, father; father #children age average John Paul

15 JSON Aggregation Combine JSON documents in multiple rows into a JSON array CREATE TABLE t1 (id INT, grp INT, jsoncol JSON); INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}'); INSERT INTO t1 VALUES (2, 1, '{"keya":"valuea","keyb":"valueb"}'); SELECT JSON_ARRAYAGG(jsoncol) FROM t1; [{"key1":"value1","key2":"value2"}, {"keya":"valuea","keyb":"valueb"}, {"keyx":"valuex","keyy":"valuey"}] INSERT INTO t1 VALUES (3, 2, '{"keyx":"valuex","keyy":"valuey"}'); 15

16 JSON Aggregation Combine JSON documents in multiple rows into a JSON object CREATE TABLE t1 (id INT, grp INT, jsoncol JSON); INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}'); INSERT INTO t1 VALUES (2, 1, '{"keya":"valuea","keyb":"valueb"}'); SELECT grp, JSON_OBJECTAGG(id, jsoncol) FROM t1 GROUP BY grp; 1 {"1":{"key1":"value1","key2":"value2"}, "2":{"keyA":"valueA","keyB":"valueB"}} 2 {"3":{"keyX":"valueX","keyY":"valueY"}} INSERT INTO t1 VALUES (3, 2, '{"keyx":"valuex","keyy":"valuey"}'); 16

17 Agenda JSON data type JSON functions Indexing JSON data Partial update of JSON 17

18 Indexing JSON data CREATE TABLE employees (data JSON); Use Functional Indexes, ALTER TABLE employees ADD COLUMN name VARCHAR(30) AS (data->> $.name ) VIRTUAL, ADD INDEX name_idx (name); STORED and VIRTUAL types are supported Functional index approach Use inlined JSON path or JSON_EXTRACT to specify field to be indexed Support both virtual and stored generated columns data->> $.name is a shorthand for JSON_UNQUOTE(data-> $.name ) 18

19 Simplified Functional Index Syntax Upcoming! In 5.7: Create a virtual column, then add an index on this column In 8.0: Add a functional index directly, hide creation of virtual column ALTER TABLE employees ADD COLUMN name VARCHAR(30) AS (data->> $.name ) VIRTUAL, ADD INDEX name_idx (name); ALTER TABLE employees ADD INDEX name_idx ((CAST(data->>"$.name" AS VARCHAR(30)))); 19

20 Indexing JSON Array Upcoming! 5.7: Single value index, one index record <-> one data record 8.0: Multi value index, multiple index records <-> one data record Multi-value index for efficient queries against array fields Use CAST( <expr> AS <type> ARRAY) to create multi-valued index Expr is any expression that returns JSON array Type is any SQL type supported by CAST() ALTER TABLE employees ADD INDEX name_idx ((CAST(data->>"$.address" AS VARCHAR(30) ARRAY)); 20

21 Indexing JSON Array id Families 1 {"father":"john", "mother":"mary", "marriage_date":" ", "children": [ {"name":"eric","age":12,"eyes":"blue"}, {"name":"beth","age":10, "eyes": "green"}]} 2 {"father":"paul", "mother": "Laura", "children": [ {"name":"sarah","age":9,"eyes":"brown"}, {"name":"noah","age":3,"eyes": "green"}, {"name":"peter","age":1,"eyes": "green"}]} Upcoming! CREATE TABLE t (id int primary key, families JSON); ALTER TABLE t ADD INDEX eye_color ((CAST(families->>"$.children[*].eyes" AS CHAR(10) ARRAY))) Index eye_color id blue 1 brown 2 green 1 green 2 21

22 Indexing JSON Array Find parents with children having brown eyes: Upcoming! SELECT id, families->>"$.father", families->>"$.mother" FROM t WHERE "brown" IN (families->>"$.children[*].eyes") id father mother 2 Paul Laura Index eye_color id blue 1 brown 2 green 1 green 2 22

23 Indexing JSON Array Upcoming! Find parents with children having either blue or brown eyes. SELECT id, families->>"$.father", families->>"$.mother" FROM t WHERE JSON_OVERLAPS(families->>"$.children[*].eyes", '["blue", "brown"]') Index id father mother 1 John Mary 2 Paul Laura eye_color id blue 1 brown 2 green 1 green 2 23

24 Agenda JSON data type JSON functions Indexing JSON data Partial update of JSON 24

25 Partial Update of JSON In 5.7: Updating a JSON document always causes the entire document to be written In 8.0: Automatically detect potential in-place updates JSON_SET, JSON_REPLACE, JSON_REMOVE Input and target columns are the same Updating existing values, not adding new ones New value is equal to or smaller than the original value, or previous partial update has left sufficient space In 8.0: Only diff is shipped through binlog (default full image) In 8.0: InnoDB supports partial update of BLOB 25

26 26

27 Agenda JSON data type JSON functions Indexing JSON data Partial update of JSON JSON or column? 27

28 Storing as JSON More flexible way to represent data that is hard to model in schema No painful schema changes Easier prototyping Fewer types to consider No enforced schema 28

29 Storing as Column Easier to apply schema to your application Schema may make applications easier to maintain over time as change is controlled; Do not have to expect as many permutations Allows some constraints over data 29

30 Hybrid: Schema + Schemaless CREATE TABLE pc_components ( id INT NOT NULL PRIMARY KEY, description VARCHAR (60) NOT NULL, vendor VARCHAR(30) NOT NULL, serial_number VARCHAR(30) NOT NULL, attributes JSON NOT NULL ); 30