MySQL Document Store. How to replace a NoSQL database by MySQL without effort but with a lot of gains?

Transcription

1 1 / 71

2 2 / 71

3 3 / 71 MySQL Document Store How to replace a NoSQL database by MySQL without effort but with a lot of gains? Percona University, Ghent, Belgium June 2017 Frédéric Descamps - MySQL Community Manager - Oracle

4 4 / 71 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purpose only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in making purchasing decisions. The development, release and timing of any features or functionality described for Oracle s product remains at the sole discretion of Oracle.

5 5 / 71 about.me/lefred Who am I?

6 6 / 71 Frédéric MySQL Evangelist Hacking MySQL since 3.23 devops believer MySQL Community Manager since May 2016 living in Belgium

7 7 / 71

8 8 / 71 RDBMS & MySQL Relational Databases

9 9 / 71 Relational Databases Data Integrity

10 10 / 71 Relational Databases Data Integrity normalization

11 11 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...)

12 12 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...) Atomicity, Consistency, Isolation, Durability

13 13 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...) Atomicity, Consistency, Isolation, Durability ACID compliant

14 14 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...) Atomicity, Consistency, Isolation, Durability ACID compliant transactions

15 15 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...) Atomicity, Consistency, Isolation, Durability ACID compliant transactions SQL

16 16 / 71 Relational Databases Data Integrity normalization constraints (foreign keys,...) Atomicity, Consistency, Isolation, Durability ACID compliant transactions SQL powerfull query language

17 17 / 71 NoSQL & MySQL NoSQL Databases

18 18 / 71 NoSQL or Document Store Schemaless

19 19 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,...

20 20 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development

21 21 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure

22 22 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects

23 23 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model

24 24 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model objects persistance without the use of any ORM - mapping oobject-oriented

25 25 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model objects persistance without the use of any ORM - mapping oobject-oriented JSON

26 26 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model objects persistance without the use of any ORM - mapping oobject-oriented JSON close to frontend

27 27 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model objects persistance without the use of any ORM - mapping oobject-oriented JSON close to frontend native in JS

28 28 / 71 NoSQL or Document Store Schemaless no schema design, no normalization, no foreign keys, no data types,... very quick initial development Flexible data structure embedded arrays or objects valid solution when natural data can't be modelized optimaly into a relational model objects persistance without the use of any ORM - mapping oobject-oriented JSON close to frontend native in JS easy to learn

29 29 / 71 How DBAs see data How Developers see data { } "GNP" : , "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : , "LifeExpectancy" : }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" }

30 30 / 71 DMBS or NoSQL?

31 31 / 71 DMBS or NoSQL? Why not both?

32 32 / 71 MySQL as Document Store Requirements

33 33 / 71 MySQL as Document Store: requirements

34 34 / 71 MySQL as Document Store: requirements 1. support of JSON datatype

35 35 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations

36 36 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin

37 37 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin 4. X-Protocol

38 38 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin 4. X-Protocol extends and replaces the traditional MySQL protocol

39 39 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin 4. X-Protocol extends and replaces the traditional MySQL protocol supports asynchronous communication to the server

40 40 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin 4. X-Protocol extends and replaces the traditional MySQL protocol supports asynchronous communication to the server supports multiple API calls like Javascript, Python, SQL...

41 41 / 71 MySQL as Document Store: requirements 1. support of JSON datatype 2. CRUD Operations 3. X-Plugin 4. X-Protocol extends and replaces the traditional MySQL protocol supports asynchronous communication to the server supports multiple API calls like Javascript, Python, SQL MySQL Shell

42 42 / 71 Starting using MySQL DS in few minutes

43 43 / 71 Starting using MySQL DS in few minutes This is the new MySQL shell as it will be in 8 ;-)

44 44 / 71 Migration from MongoDB to MySQL DS For this example, I will use the well known restaurants collection: We have two options: 1. We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data. 1. We retrieve the data from MongoDB and load it to MySQL via the Shell in javascript

45 45 / 71 Dump and load using MySQL Shell & Python this example is the courtesy $ mongo --quiet --eval 'DBQuery.shellBatchSize=300; var all=db.restaurants. nd() ; all' perl -pe \ 's/(?:objectid ISODate)\(("[^"]+")\)/ $1/g' > all_recs.json

46 46 / 71 Dump and load using bash and MySQL Shell $ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; var all=db.restaurants. nd() ; all' \ perl -pe 's/(?:objectid ISODate)\(("[^"]+")\)/ $1/g'\ head -n -1 while read line do mysqlsh root@ password='' -e "rec=$line; session.getschema('docstore').getcollection('restaurants').add(rec).execute()" done

47 47 / 71 Let's query

48 48 / 71 Let's query That's too much records to show it here... let's limit it

49 49 / 71

50 50 / 71 Some more examples

51 51 / 71 Some more examples Let's add a selection criteria:

52 52 / 71 Syntax slightly different than MongoDB > db.restaurants. nd({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2) { "borough" : "Queens", "cuisine" : "French", "name" : "La Baraka Restaurant" } { "borough" : "Queens", "cuisine" : "French", "name" : "Air France Lounge" }

53 53 / 71 CRUD operations

54 54 / 71 what about my old SQL? The hidden part of the iceberg

55 55 / 71 JSON datatype is behind the scene native datatype (since 5.7.8)

56 56 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4

57 57 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values

58 58 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH,...)

59 59 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH,...) Generated/virtual columns

60 60 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH,...) Generated/virtual columns Indexing JSON data

61 61 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH,...) Generated/virtual columns Indexing JSON data Foreign Keys to JSON data

62 62 / 71 JSON datatype is behind the scene native datatype (since 5.7.8) JSON values are stored in MySQL tables using UTF8MB4 Conversion from "native" SQL types to JSON values JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH,...) Generated/virtual columns Indexing JSON data Foreign Keys to JSON data SQL Views to JSON data

63 63 / 71 Example

64 64 / 71 Example

65 65 / 71 Example (2)

66 66 / 71 Example (2)

67 67 / 71 Example (3): explain

68 68 / 71 Example (4): add index

69 69 / 71 what's do I gain? Conclusion

70 70 / 71 Conclusion This is the best of the two worlds in one product! Data integrity ACID Compliant SQL schemaless flexible data structure easy to start (CRUD)

71 71 / 71 Thank you! Any Questions?