MySQL 5.7 News. Abel Flórez Technical Account Manager Copyright 2016 Oracle and/or its affiliates. All rights reserved.

Transcription

1 MySQL 5.7 News Abel Flórez Technical Account Manager Copyright 2016 Oracle and/or its affiliates. All rights reserved.

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Agenda 1 News in MySQL Upgrading to MySQL MySQL Document Store 3

4 MySQL 5.7 is GA! Performance & Scalability Manageability 3 X Faster than MySQL 5.6 Native JSON Support Enhanced InnoDB: faster online & bulk load operations Improved Security: safer initialization, setup & management Replication Improvements (incl. multisource, multi-threaded slaves...) Performance Schema Improvements New Optimizer Cost Model: greater user control & better query performance MySQL SYS Schema And many more new features and enhancements. Learn more at: dev.mysql.com 4

5 MySQL 5.7 Sysbench Benchmark: SQL Point Selects 3x Faster than MySQL 5.6 4x Faster than MySQL 5.5 1,600,000 QPS Queries per Second MySQL 5.7:Sysbench OLTP Read Only (SQL Point Selects) 1,800,000 1,600,000 1,400,000 1,200,000 1,000, , , , , MySQL 5.7 MySQL 5.6 MySQL 5.5 1,024 Connections Intel(R) Xeon(R) CPU E v3 4 sockets x 18 cores-ht (144 CPU threads) 2.5 Ghz, 512GB RAM Linux kernel

6 MySQL 5.7 Sysbench Benchmark: Connection Requests 82% Faster than MySQL K Connect / Sec MySQL 5.7:Sysbench OLTP Read Only 120,000 MySQL 5.7 MySQL 5.6 MySQL 5.5 Connect / sec 100,000 80,000 60,000 40,000 20, ,024 Connections Intel(R) Xeon(R) CPU E v3 4 sockets x 18 cores-ht (144 CPU threads) 2.5 Ghz, 512GB RAM Linux kernel

7 MySQL 5.7 Sysbench Benchmark: OLTP Read Only 3x Faster than MySQL 5.6 6x Faster than MySQL 5.5 ~ 1,000,000 QPS MySQL 5.7:Sysbench OLTP Read Only 1,200,000 MySQL 5.7 MySQL 5.6 MySQL 5.5 Queries per Second 1,000, , , , , ,024 Connections Intel(R) Xeon(R) CPU E v3 4 sockets x 18 cores-ht (144 CPU threads) 2.5 Ghz, 512GB RAM Linux kernel

8 MySQL 5.7 Sysbench Benchmark: OLTP Read Write 1.5x Faster than MySQL 5.6 3x Faster than MySQL 5.5 MySQL 5.7:Sysbench OLTP Read Write 700,000 MySQL 5.7 MySQL 5.6 MySQL 5.5 Queries per Second 600, , , , , , Connections 1,024 Intel(R) Xeon(R) CPU E v3 4 sockets x 18 cores-ht (144 CPU threads) 2.5 Ghz, 512GB RAM Linux kernel

9 MySQL 5.7: Optimizer Improvements Queries execute faster, while using less CPU and disk space! New Cost based Optimizer Optimizer and Parser refactoring Readability, maintainability and stability Separate parsing, optimizing, execution stages Easier feature additions, with lessened risk New hint framework Easier to manage With support for additional new hints Improved JSON EXPLAIN EXPLAIN for running thread Easier to extend Configurable and tunable mysql.server_cost and mysql.engine_cost tables API for where data resides: on disk or in cache InnoDB for internal temp tables Better ONLY_FULL_GROUP_BY mode Many specific new optimizations Generated Columns 9

10 Optimizer Cost Model: Performance Improvements DBT-3 (Size Factor 10, CPU bound) Execution time relative to 5.6 (%) MySQL 5.6 MySQL Q3 Q7 Q8 Q9 Q12 5 out of 22 queries get a much improved query plan (others remain the same) 10

11 Optimizer Cost Model: Performance Improvements DBT-3 (Size Factor 10) Disk bound Q2 Q18 Execution time relative to 5.6 (%) Execution time relative to 5.6 (%) CPU bound Q2 Q18 2 out of 22 queries get a significantly improved query plan (others remain the same) 11

12 MySQL 5.7: Query Rewrite Plugin New pre and post parse query rewrite APIs Users can write their own plug-ins Provides a post-parse query plugin Rewrite problematic queries without the need to make application changes Add hints Modify join order Many more Improve problematic queries from ORMs, third party apps, etc Eliminates many legacy use cases for proxies 12

13 MySQL 5.7: Optimizer - Cost Info in JSON EXPLAIN {{ Expanded JSON EXPLAIN Now includes all available cost info Used for Visual Explain In MySQL Workbench "query_block": "query_block": {{ "select_id": "select_id": 1, 1, "cost_info": "cost_info": {{ "query_cost": "query_cost": "200.40" "200.40" }, }, "table": "table": {{ "table_name": "table_name": "nicer_but_slower_film_list", "nicer_but_slower_film_list", "access_type": "access_type": "ALL", "ALL", "rows_examined_per_scan": "rows_examined_per_scan": 992, 992, "rows_produced_per_join": "rows_produced_per_join": 992, 992, "filtered": "filtered": 100, 100, "cost_info": "cost_info": {{ "read_cost": "read_cost": "2.00", "2.00", "eval_cost": "eval_cost": "198.40", "198.40", "prefix_cost": "prefix_cost": "200.40", "200.40", "data_read_per_join": "data_read_per_join": "852K" "852K" }, }, "used_columns": "used_columns": [[ "FID", "FID", "title", "title", "description", "description", "category", "category", "price", "price", "length", "length", "rating", "rating", "actors" "actors" ], ],

14 MySQL 5.7: JSON Native JSON data type Native internal binary format for efficient processing & storage Built-in JSON functions Allowing you to efficiently store, search, update, and manipulate Documents JSON Comparator Allows for easy integration of Document data within your SQL queries Indexing of Documents using Generated Columns InnoDB supports indexes on both stored and virtual Generated Columns New expression analyzer automatically uses the best functional index available New inline syntax for easy SQL integration 14

15 MySQL 5.7: JSON Functions 5.7 supports functions to CREATE, SEARCH, MODIFY and RETURN JSON values: JSON_APPEND() JSON_INSERT() JSON_REPLACE() JSON_ARRAY_INSERT() JSON_KEYS() JSON_SEARCH() JSON_ARRAY() JSON_LENGTH() JSON_SET() JSON_CONTAINS_PATH() JSON_MERGE() JSON_TYPE() JSON_CONTAINS() JSON_OBJECT() JSON_UNQUOTE() JSON_DEPTH() JSON_QUOTE() JSON_VALID() JSON_EXTRACT() JSON_REMOVE() 15

16 MySQL 5.7: JSON and Text Datatype Comparison Unindexed traversal of 206K documents # With feature column as JSON type SELECT DISTINCT feature->"$.type" as json_extract FROM features; json_extract "Feature" row in set (1.25 sec) # With feature column as TEXT type SELECT DISTINCT feature->"$.type" as json_extract FROM features; json_extract "Feature" row in set (12.85 sec) Explanation: Binary format of JSON type is very efficient at searching. Storing as TEXT performs over 10x worse at traversal. 16

17 MySQL 5.7: Functional Indexes with JSON From table scan on 206K documents to index scan on 206K materialized values ALTER TABLE features ADD feature_type VARCHAR(30) AS (JSON_UNQUOTE(feature>'$.type')); Meta data change only (FAST). Query OK, 0 rows affected (0.01 sec) Does not need to touch table.. Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE features ADD INDEX (feature_type); Query OK, 0 rows affected (0.73 sec) Records: 0 Duplicates: 0 Warnings: 0 SELECT DISTINCT feature_type FROM features; feature_type Down from 1.25 sec to sec "Feature" row in set (0.06 sec) Creates index only, does not touch row data. Create table features ( id int primary key, feature json ) 17

18 MySQL 5.7: Performance Schema Memory Instrumentation Aggregates statistics by Type of memory used (caches, internal buffers, ) Thread/account/user/host indirectly performing the memory operation Attributes include Memory used (bytes) Operation counts High/Low Water Marks Statement Instrumentation Additional Information Replication slave status Stored Procedures MDL lock instrumentation Stored Functions Status and variables per thread Prepared Statements Transactions Server stage tracking Track long running SQL Improved configuration and ease-of-use All while reducing total footprint and overhead 18

19 MySQL 5.7: SYS Schema Helper objects for DBAs, Developers and Operations staf Helps simplify DBA / Ops tasks - Monitor server health, user, host statistics - Spot, diagnose, and tune performance issues Easy to understand views with insights into - IO hot spots, Locking, Costly SQL statements - Schema, table and index statistics SYS is similar to - Oracle V$ catalog views - Microsoft SQL DMVs (Dynamic Mgmnt Views) 19

20 MySQL 5.7: GIS Improvements Replaced custom code with Boost.Geometry For spatial calculations For spatial analysis Enabling full OGC compliance We re also Boost.Geometry contributors! InnoDB R-tree based spatial indexes Full ACID, MVCC, & transactional support Index records contain minimum bounding box GeoHash GeoJSON Helper functions such as ST_Distance_Sphere() and ST_MakeEnvelope() 20

21 MySQL 5.7: InnoDB Improvements Native Partitioning General TABLESPACE support Eliminates previous limitations Eliminates resource usage problems Transportable tablespace support Store multiple tables in user defined shared tablespaces Support for MySQL Group Replication High priority transactions Native Full-Text Search Improved support for cache preloading Including full CJK support! Load your hottest data loaded at startup Native Spatial Indexes Configurable fill-factor Transparent page compression Allows for improvements in storage footprint Support for 32K and 64K pages Use with transparent page compression for very high compression ratios Improved bulk-data load performance Resize the InnoDB Buffer Pool online 21

22 MySQL 5.7: InnoDB Compression Thank you, SanDisk Fusion-io Transparent Page Level Compression Happens transparently in background threads Managed entirely within the IO layer Uses sparse file and "hole punching" support in OS kernels and File Systems Reduces IO Improves MySQL performance Improves storage efficiency Reduces write cycles, thus increasing SSD lifespan Applies to all InnoDB data, including the system tablespace and UNDO logs 22

23 MySQL 5.7: Security Improvements AES 256 Encryption now the default SSL Password rotation policies Can be set globally, and at the user level Deployment: enable secure unattended install by default Random password set on install Remove anonymous accounts Deployment without test account, schema, demo files Easier instance initialization and setup: mysqld initialize New detection and support for systemd Enabled by default Auto-detection of existing keys and certs Auto generation of keys and certs when needed New helper utility: mysql_ssl_rsa_setup New --require_secure_transport option to prevent insecure communications Added SSL support to binary log clients Extended Proxy User Support Added Built-in Authentication Plugins support for Proxy Users Allows multiple users to share a single set of managed privileges 23

24 MySQL 5.7: Server-Side Statement Timeouts Thank you Davi Arnaut! Server side statement timeouts Global for server, per session, or for individual SELECT statements SELECT /*+ MAX_EXECUTION_TIME(1000) */ FROM my_table; * Expanded to Windows and Solaris, restricted by removing USER option 24

25 MySQL 5.7: Replication Improvements 8-10x Faster slave throughput GTID enhancements On-line, phased deployment of GTIDs Binary logging on slave now optional Enhanced Semi-synchronous replication Write guaranteed to be received by slave before being observed by clients of the master Option to wait on Acks from multiple slaves Multi-Source Replication 250% Slavethroughputvs. 96ThreadMaster 200% 150% 100% Consolidate updates from multiple Masters into one Slave Dynamic slave filters Often removes slave as a bottleneck; keep pace with master with 8+ slave threads Option to preserve Commit order Automatic slave transaction retries 50% 0% SlaveThreads 25

26 MySQL 5.7: Replication Improvements Multi-Source Replication Consolidate updates from multiple Masters into one Slave Consolidated view of all shards More flexible topologies Centralized point for backups Compatible with Semi-Sync Replication & enhanced MTS Performance Schema tables for monitoring slave Online Operations: Dynamic Replication Filters, switch master Master 1 Master 2 Master N Binlog Binlog Binlog IO 1 IO 2 IO N Relay 1 Relay 2 Relay N Coordinator Coordinator Coordinator Coordinator W1 W2 WX W1 W2 WX W1 W2 WX W1 W2 Slave 26 WX

27 MySQL Router Connection and Transaction routing Transparently improve your MySQL apps Transparent support for MySQL Group Replication clusters Transparent support for custom clusters and HA setups Easily extendable using plugin APIs Many new plugins to come Aggregation, Binary Log, Load Balancing, 27

28 MySQL Router More plugins to come Extensible Architecture Agg reg atio n? App Servers with MySQL Router Plugins Fabric cache, B og? L y inar Group Replication Core Connection nce???? routing, transaction routing Harness Logging, thread handling ; config, service, and plugin management Load Ba la MySQL HA cluster Copyright 2016, Oracle and/or its affiliates. All rights reserved 28

29 MySQL Workbench 6.3 GA Performance Dashboard Performance Schema Reports & Graphs Visual Explain GIS Viewer Migration New: Microsoft Access Microsoft SQL Server, Sybase, PostgreSQL, SQLite MySQL EE features New Easy to Use Wizards for Fast Data Migration Table<->File Data Import/Export (like Excel) SSL Certificate Creation 29

30 MySQL Group Replication labs.mysql.com Active/Active Update Anywhere Conflict detection and resolution (transaction rollback) Optimistic State Machine Replication Application MySQL Nodes MySQL Server API Replication Plugin Group Comms Automatic group membership management and failure detection No need for server fail-over Elastic scale out/in No single point of failure Automatic reconfiguration Well integrated InnoDB GTID-based replication PERFORMANCE_SCHEMA 30

31 The Future of MySQL Scaling (HA + Sharding) MySQL Router Server/Shard State & Mapping SQL Queries Global Data Coordination and Control MySQL Fabric controller cluster Shard 1 Shard 2 Group Replication cluster Group Replication cluster Group Replication cluster Global Group HA Group HA Group 31

32 MySQL Repos MySQL on GitHub Distributions Git for MySQL Engineering Fast, flexible and great for a distributed team Great tooling Large and vibrant community Oracle, Red Hat, CentOS Fedora Ubuntu, Debian SUSE GitHub for MySQL Community Official MySQL Docker Image from Oracle Coming Soon Preconfigured Containers Improved support for popular DevOps deployment tools Easy and fast code availability to the community and to downstream projects Pull Requests

33 Agenda 1 News in MySQL Upgrading to MySQL MySQL Document Store 33

34 MySQL Upgrade Supported Paths All minor upgrades like to or doing a jump like to is supported. Upgrading one major ( ) release level is supported. It s recommended upgrading to the latest 5.6 release before moving to 5.7. Doing long-jumps ( ) is supported if you upgrade one release level at a time. Direct upgrades that skip a release level (for example, upgrading directly from MySQL 5.5 to 5.7) is not recommended. 34

35 MySQL Upgrade Before you start! Review the Release Notes for target MySQL version Review the manual page on upgrade for this MySQL version, take special care of sections covering: Server and Status Variables and Options Added, Deprecated, or Removed Checking Whether Tables or Indexes Must Be Rebuilt Changes Affecting Upgrades to MySQL 5.N Create a plan for your upgrade/downgrade path Create a backup of your database 35

36 MySQL Upgrade In-place Upgrade 1. Do all necessary preparations/plans as explained earlier 2. Do a graceful shutdown of MySQL (innodb_fast_shutdown=0) 3. Do upgrade, replace the old binaries with the new ones 4. Start new MySQL Server using new binaries 5. Run mysql_upgrade binary to upgrade internal mysql repository 6. Done!

37 MySQL Upgrade Logical Upgrade 1. Do all necessary preparations/plans as explained earlier 2. Do a dump of only the user databases/schemas (skipping the mysql system schema, remember stored procedures, views and functions) 3. Do a clean installation of new MySQL version (remember to migrate important my.cnf options) 4. Load the dump file into the new MySQL server instance 5. Done!

38 MySQL Upgrade Using Replication simplified 1. Do all necessary preparations/plans as explained earlier 2. Restore backup from production (5.6) database to new slave server (5.7) 3. Start replication from master to slave 4. Once slave server has caught up with master Application 5. Change application to connect to slave (5.7) 6. Done! MySQL 5.6 MySQL 5.7 Replication 38

39 MySQL Upgrade Regression Problems Explain is you friend! Optimizer_switch can help get back old query plans for slow queries Monitor resources/mysql before/after to spot changes Without proper monitoring you have little chance to find root cause of problem Did you change more than just MySQL version? Normally we see people refresh HW/OS at the same time with creates many possible regression scenarios Use PERFORMANCE_SCHEMA and SYS schema to spot problems. 39

40 Upgrading from MySQL 5.1 or 5.5 to MySQL 5.7? News in MySQL 5.6 Need version of Connector/J (due to retire of SET OPTION ) InnoDB Files per table now default The Query Cache is disabled by default 8 new reserved key words have been defined ( get and partition ) New data format of TIMESTAMP and DATETIME (microseconds) Password on command line causes warnings (problematic for scripts) New optimizer features may cause different query plans Old password are not allowed by default 40

41 New Servers Default s in MySQL 5.7 innodb_strict_mode is ON InnoDB file format is now Barracuda sql_mode option NO_AUTO_CREATE_USER, STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, NO_ZERO_IN_DATE is now default More changes to optimizer_switch, may cause changed query plans! Use explain to identify problem and set optimizer_switch to solve problem MySQL accounts expire after 360 days. The default-password-lifetime is 360 and many more, see link below for more details 41

42 Removed Features in MySQL 5.7 Old password are not allowed in MySQL 5.7 Remove IGNORE for ALTER TABLE in 5.7, WL#7395 Remove of INSERT DELAYED, WL#6073 Remove YEAR(2) support, WL#6263 Deprecate Unique Option Prefixes, WL#6978 Deprecate skip-innodb in 5.6 and remove it in 5.7, WL#7976 For complete list: 42

43 MySQL 5.7 Installation changes mysql_install_db is removed Replaced by mysqld --initialize MySQL is more Secure by default: TLS/SSL enabled by default and for some packages keys are created by default No creation of test database No creation of anonymous users Only one root user and password is set at installation time All user accounts will automatically have their passwords rotated after 360 days New mysql_ssl_rsa_setup tool Use mysqld --initialize-insecure to get old behavior 43

44 MySQL 5.7: Additional Info

45 Agenda 1 News in MySQL Upgrading to MySQL MySQL Document Store 45

46 Today s Challenges Developers want to move faster Time to market has a premium value Rapid prototyping, iterate fast 46

47 Today s Challenges (cont.) Relational databases ask for schema up front Potentially saving you time in the future Less variations; less code to handle edge cases Added cost with each schema change NoSQL databases do not ask for schema Saving you time up front But potentially adding operational costs over time No cost per schema change 47

48 Why can you not Have both schema-less and schema in the same technology stack? One that checks all the boxes of all stakeholders: Developers: [ x ] Schemaless [ x ] Rapid Prototyping/Simpler APIs [ x ] Document Model Operations: [ x ] Performance Management/Visibility [ x ] Robust Replication, Backup, Restore [ x ] Comprehensive Tooling Ecosystem Business Owner: [ x ] Don t lose my data = ACID transactions [ x ] Capture all my data = Extensible/Schemaless [ x ] Products On Schedule/Time to Market = Rapid Development 48

49 NEW! MySQL Document Store Native JSON Documents in MySQL 5.7 Schema-less Document Storage X Protocol (MySQL DMR) Implemented by X Plugin to Extend MySQL Server as a Document Store X Dev API SQL and Document CRUD Operations Implemented in Connector/Node.js, Connector/J, Connector/Net MySQL Shell Javascript, Python, SQL modes

50 Architecture SQL API CRUD API Std Protocol X Protocol Std Protocol 3306 MySQL Shell X and Std Protocols X Protocol MySQL Plugins Core X Protocol Plugin Memcached Plugin 50

51 New! X Protocol Asynchronous API support Supporting Parallelism and Batching Pipelining - Send multiple requests, Reduce Round Trips Push Notifications - Beyond the Request/Response model Middleware Friendly Routing, Sharding, Read Write Splitting Uses open standards: TLS, SASL, Protobuf etc Proven, community friendly 51

52 New! MySQL X DevAPI Modern: fluent API, method chaining Stateless sessions enable transparent scaling to multi-server environments SQL support CRUD for Collections of Documents and Tables Documents as simple basic domain objects Search expressions match SQL SELECT expressions Implemented in MySQL Shell & MySQL Connectors NEW! MySQL Connector/node.js MySQL Connector/J MySQL Connector/Net

53 New! MySQL Shell Integrated Development and Administration Shell Exposes New X DevAPI Multi-Language scripting JavaScript, Python, and SQL Configurable results formats Traditional Table, JSON, Tab Separated 53

54 $ mysqlsh uri root@localhost/test Creating an X Session to root@localhost:33060/test Enter password: Default schema `test` accessible through db. Currently in JavaScript mode. Use \sql to switch to SQL mode and execute queries. mysql js> db.createcollection("posts"); <Collection:posts> mysql js> db.posts.add({"title":"hello World", "text":"first post!"}) Query OK, 1 item affected (0.03 sec) mysql js> db.posts.find("title = 'Hello World'").sort(["title"]); [ { "_id": "8202bda28206e611140b b6526", "text": "First post!", "title": "Hello World" } ] 1 document in set (0.01 sec) 54

55 CRUD Operations NoSQL/Document and SQL/Relational 55

56 CRUD Operations NoSQL/Document Javascript Java NodeJS C#

57 CRUD Operations SQL/Relational Javascript Java NodeJS C# 57

58 MySQL Document Store

59 Thanks! Copyright 2016 Oracle and/or its affiliates. All rights reserved.

60