<Insert Picture Here> Boosting performance with MySQL partitions

Similar documents
<Insert Picture Here> MySQL 5.1 and 5.5 partitions

<Insert Picture Here> Introduction to MySQL

MySQL Query Tuning 101. Sveta Smirnova, Alexander Rubin April, 16, 2015

MySQL 5.1 Past, Present and Future MySQL UC 2006 Santa Clara, CA

Oracle 11g Partitioning new features and ILM

Covering indexes. Stéphane Combaudon - SQLI

MySQL Schema Review 101

<Insert Picture Here> Upcoming Changes in MySQL 5.7 Morgan Tocker, MySQL Community Manager

1Z MySQL 5 Database Administrator Certified Professional Exam, Part II Exam.

Tired of MySQL Making You Wait? Alexander Rubin, Principal Consultant, Percona Janis Griffin, Database Evangelist, SolarWinds

Chapter 8: Working With Databases & Tables

Use Cases for Partitioning. Bill Karwin Percona, Inc

DATABASE SYSTEMS. Database programming in a web environment. Database System Course, 2016

IT Certification Exams Provider! Weofferfreeupdateserviceforoneyear! h ps://

Oracle 1Z MySQL 5.6 Database Administrator. Download Full Version :

When and How to Take Advantage of New Optimizer Features in MySQL 5.6. Øystein Grøvlen Senior Principal Software Engineer, MySQL Oracle

MySQL Schema Best Practices

MySQL 101. Designing effective schema for InnoDB. Yves Trudeau April 2015

Upgrading to MySQL 8.0+: a More Automated Upgrade Experience. Dmitry Lenev, Software Developer Oracle/MySQL, November 2018

Sheeri Cabral. Are You Getting the Best Out of Your MySQL Indexes? Slides

Data Warehousing & Big Data at OpenWorld for your smartphone

The Blackhole and Federated Storage Engines: The Coolest Kids on the Block

Partitioning Tables and Indexing Them

A quick tour of MySQL 8.0 roles

Load Testing Tools. for Troubleshooting MySQL Concurrency Issues. May, 23, 2018 Sveta Smirnova

Database Management Systems,

What's New in MySQL 5.7?

How to Use JSON in MySQL Wrong

MySQL Group Replication in a nutshell

Advanced MySQL Query Tuning

Oracle Advanced Compression: Reduce Storage, Reduce Costs, Increase Performance Bill Hodak Principal Product Manager

CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen

Switching to Innodb from MyISAM. Matt Yonkovit Percona

Delegates must have a working knowledge of MariaDB or MySQL Database Administration.

Lecture 19 Query Processing Part 1

<Insert Picture Here> New MySQL Enterprise Backup 4.1: Better Very Large Database Backup & Recovery and More!

MySQL Indexing. Best Practices for MySQL 5.6. Peter Zaitsev CEO, Percona MySQL Connect Sep 22, 2013 San Francisco,CA

MySQL JSON. Morgan Tocker MySQL Product Manager. Copyright 2015 Oracle and/or its affiliates. All rights reserved.

The Top 20 Design Tips

Downloaded from

Writing High Performance SQL Statements. Tim Sharp July 14, 2014

MySQL 8.0 What s New in the Optimizer

Practical MySQL indexing guidelines

The Hazards of Multi-writing in a Dual-Master Setup

<Insert Picture Here> DBA s New Best Friend: Advanced SQL Tuning Features of Oracle Database 11g

DATABASE SYSTEMS. Database programming in a web environment. Database System Course,

MongoDB and Mysql: Which one is a better fit for me? Room 204-2:20PM-3:10PM

Advanced MySQL Query Tuning

SQL (and MySQL) Useful things I have learnt, borrowed and stolen

Mastering the art of indexing

MySQL Indexing. Best Practices. Peter Zaitsev, CEO Percona Inc August 15, 2012

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

dbdeployer the future of MySQL sandboxes Giuseppe Maxia Software explorer

Automating Information Lifecycle Management with

Upgrading MySQL Best Practices. Apr 11-14, 2011 MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc

Introduction to troubleshooting Basic techniques. Sveta Smirnova Principal Support Engineer March, 10, 2016

Partitioning. Sheeri K. Cabral Database Administrator The Pythian Group, January 12, 2009

Jens Bollmann. Welcome! Performance 101 for Small Web Apps. Principal consultant and trainer within the Professional Services group at SkySQL Ab.

SQL Data Definition Language: Create and Change the Database Ray Lockwood

What s New in MySQL 5.7 Geir Høydalsvik, Sr. Director, MySQL Engineering. Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Getting started with MySQL Proxy

Oracle Database 10g Express

Using DDL Statements to Create and Manage Tables. Copyright 2004, Oracle. All rights reserved.

Oracle Flashback Data Archive (FDA) O R A C L E W H I T E P A P E R M A R C H

Effective Software Development with MySQL

Testing and Verifying your MySQL Backup Strategy

MySQL Replication : advanced features in all flavours. Giuseppe Maxia Quality Assurance Architect at

Mysql Manually Set Auto Increment To 1000

So you think you know everything about Partitioning?

Chapter 2. DB2 concepts

PostgreSQL to MySQL A DBA's Perspective. Patrick

Oracle Exam 1z0-883 MySQL 5.6 Database Administrator Version: 8.0 [ Total Questions: 100 ]

OKC MySQL Users Group

Copyright 2017, Oracle and/or its aff iliates. All rights reserved.

Tips from the Trenches Preventing downtime for the over extended DBA. Andrew Moore Senior Remote DBA Percona Managed Services

CMPS 181, Database Systems II, Final Exam, Spring 2016 Instructor: Shel Finkelstein. Student ID: UCSC

ITS. MySQL for Database Administrators (40 Hours) (Exam code 1z0-883) (OCP My SQL DBA)

Dealing with schema changes on large data volumes. Danil Zburivsky MySQL DBA, Team Lead

CS 327E Lecture 2. Shirley Cohen. January 27, 2016

SQL Gone Wild: Taming Bad SQL the Easy Way (or the Hard Way) Sergey Koltakov Product Manager, Database Manageability

ALTER TABLE Improvements in MARIADB Server. Marko Mäkelä Lead Developer InnoDB MariaDB Corporation

'information_schema' When Using Lock Tables

Update Table Schema Sql Server 2008 Add Column Default Value

InnoDB: Status, Architecture, and Latest Enhancements

Mysql Information Schema Update Time Null >>>CLICK HERE<<< doctrine:schema:update --dump-sql ALTER TABLE categorie

Basic SQL. Basic SQL. Basic SQL

MySQL Cluster An Introduction

Physical Design of Relational Databases

MySQL for Database Administrators Ed 3.1

MySQL Performance Tuning

Oracle Exam 1z0-882 Oracle Certified Professional, MySQL 5.6 Developer Version: 7.0 [ Total Questions: 100 ]

ASSIGNMENT NO 2. Objectives: To understand and demonstrate DDL statements on various SQL objects

T-sql Check If Index Exists Information_schema

INDEX. 1 Basic SQL Statements. 2 Restricting and Sorting Data. 3 Single Row Functions. 4 Displaying data from multiple tables

The appendix contains information about the Classic Models database. Place your answers on the examination paper and any additional paper used.

Relational Database Line-Up

Indexes - What You Need to Know

Database Programming with SQL

MySQL 8.0 Performance: InnoDB Re-Design

Transcription:

<Insert Picture Here> Boosting performance with MySQL partitions Giuseppe Maxia MySQL Community Team Lead at Oracle 1

about me -Giuseppe Maxia a.k.a. The Data Charmer MySQL Community Team Lead Long time hacking with MySQL features Formerly, database consultant, designer, coder. A passion for QA An even greater passion for open source... and community Passionate blogger http://datacharmer.blogspot.com 2

Slides for this presentation http://tinyurl.com/mysql-partitions This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. 3

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 4

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 5

The problem(s) Too much data Not enough RAM Historical data Growing data Rotating data 6 6

Too much data data 7 7

Not enough RAM RAM data INDEXES INDEXES 8 8

Not enough RAM data INDEXES INDEXES RAM 9 9

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 10

What exactly is this "partitions" thing? Logical splitting of tables Transparent to user 11

Remember the MERGE tables? MERGE TABLE separate tables risk of duplicates insert in each table no constraints 12

It isn't a merge table! PARTITIONED TABLE One table No risk of duplicates insert in one table constraints enforced 13

Wait a minute... WHAT THE HELL DOES "LOGICAL SPLIT" REALLY MEANS? LET ME EXPLAIN... 14 14

Physical partitioning (1) Take a map 15 15

Physical partitioning (2) cut it into pieces 16 16

Physical partitioning (3) What you have, is several different pieces 17 17

Physical partitioning (4) If you want the map back, you need some application (adhesive tape) and you may get it wrong 18 18

Logical partitioning (1) Take a map 19 19

Logical partitioning (2) fold it to show the piece you need 20 20

Logical partitioning (3) what you have is still a map, even if you see only one part. 21 21

Logical partitioning (4) if you unfold the map, you still have the whole thing 22 22

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 23

Partition pruning 1a - unpartitioned table - SINGLE RECORD select * from table_name where colx = 120 24

Partition pruning 1a - unpartitioned table - SINGLE RECORD D A T A I N D E X select * from table_name where colx = 120 25

Partition pruning 1b - unpartitioned table - SINGLE RECORD select * from table_name where colx = 350 26

Partition pruning 1c - unpartitioned table - RANGE select * from table_name where colx between 120 and 230 27

Partition pruning 2a - table partitioned by colx - SINGLE REC 1-99 100-199 200-299 300-399 select * from table_name where colx = 120 400-499 500-599 28

Partition pruning 2a - table partitioned by colx - SINGLE REC 1-99 D A T A 100-199 200-299 300-399 400-499 I N D E X select * from table_name where colx = 120 500-599 29

Partition pruning 2b - table partitioned by colx - SINGLE REC 1-99 100-199 200-299 300-399 select * from table_name where colx = 350 400-499 500-599 30

Partition pruning 2c - table partitioned by colx - RANGE 1-99 100-199 200-299 300-399 400-499 select * from table_name where colx between 120 and 230 500-599 31

Partition pruning before EXPLAIN select * from table_name where colx = 120 in 5.1 EXPLAIN PARTITIONS select * from table_name where colx = 120 32

Partition pruning - unpartitioned table explain partitions select count(*) from table_name where colx=120\g ***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: NULL type: index... 33

Partition pruning - unpartitioned table explain partitions select count(*) from table_name where colx between 120 and 230\G ***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: NULL type: index... 34

Partition pruning - table partitioned by colx explain partitions select count(*) from table_name where colx between 120 and 230\G ***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: p02,p03 type: index... 35

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 36

Partitioning Types Partition a table using CREATE TABLE or ALTER TABLE CREATE TABLE <table_name> (<columns>) ENGINE=<engine_name> PARTITION BY <type> ( <partition_expression> ); <type> can be RANGE LIST HASH KEY 37

RANGE CREATE TABLE Employee ( emp_id INT AUTO_INCREMENT, name VARCHAR(50), store_id TINYINT, PRIMARY KEY (emp_id) ) ENGINE=MyISAM PARTITION BY RANGE (emp_id) ( PARTITION p0 VALUES LESS THAN (10000), PARTITION p1 VALUES LESS THAN (20000), PARTITION p2 VALUES LESS THAN (30000), PARTITION p3 VALUES LESS THAN (40000), PARTITION p4 VALUES LESS THAN MAXVALUE) MAXVALUE is optional Partition ranges must be listed smallest to greatest and must be integers 38

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 39

Benchmarking partitions Compare results Unpartitioned vs partitioned ISOLATION Repeatability Check your resources! 40

Benchmarking partitions - Compare results Execute query Record execution time CHECK RETRIEVED RECORDS! 41

Benchmarking partitions - Unpartitioned vs partitioned Make it simple. Do not change structure If needed, remove PK from partitioned table 42

Benchmarking partitions - ISOLATION Try to reproduce working conditions No other servers running while benchmarking Restart the server before each test Do NOT mix partitioned and unpartitioned tables in the same server Use MySQL Sandbox 43

Benchmarking partitions - Repeatability Measure more than once Make sure you have the same conditions Make sure your results are consistent between runs 44

Benchmarking partitions - Check resources InnoDB check disk space (uses more than MyISAM) check CPU usage Partitioned MyISAM tables use 2 file handles per partition If you use more than one partitioned table, count total file handles If you use Archive partitioned tables check CPU usage check memory usage 45

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 46

Partitioning by date - limits of partitioning Can partition only by INTEGER columns OR you can partition by an expression, which must return an integer 47

Partitioning by date - your (GOOD) options GOOD ideas YEAR(date_column) TO_DAYS(date_column) WHY? Both functions optimized for partitions Partition pruning does not kick with other functions 48

Partitioning by date - HOW TO - YEAR CREATE TABLE t1 ( d date ) PARTITION BY RANGE (YEAR(d)) ( PARTITION p01 VALUES LESS THAN (1999), PARTITION p02 VALUES LESS THAN (2000), PARTITION p03 VALUES LESS THAN (2001), PARTITION p04 VALUES LESS THAN (MAXVALUE) ); 49

Partitioning by date - HOW TO - TO_DAYS CREATE TABLE t1 ( d date) PARTITION BY RANGE (TO_DAYS(d)) ( PARTITION p01 VALUES LESS THAN (TO_DAYS('2008-01-01')), PARTITION p02 VALUES LESS THAN (TO_DAYS('2008-02-01')), PARTITION p03 VALUES LESS THAN (TO_DAYS('2008-03-01')), PARTITION p04 VALUES LESS THAN (MAXVALUE) ); 50

Partitioning by date - How TO How it works partition BY FUNCTION query BY COLUMN 51

Partitioning by date - WRONG!!! PARTITION BY RANGE(year(from_date)) select count(*) from salaries where year (from_date) = 1998; +----------+ count(*) +----------+ 247489 +----------+ 1 row in set (2.25 sec) 52

Partitioning by date - RIGHT PARTITION BY RANGE(year(from_date)) select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ count(*) +----------+ 247489 +----------+ 1 row in set (0.46 sec) 53

Partitioning by date - EXPLAIN explain partitions select count(*) from salaries where year(from_date) = 1998\G ***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p01,p02,p03,p04,p05,p06,p07,p08,p09,p10,p 11,p12,p13,p14,p15,p16,p17,p18,p19... type: index 54

Partitioning by date - EXPLAIN explain partitions select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'\G ***** 1. row ****... id: 1 select_type: SIMPLE table: salaries partitions: p14,p15 55

Partitioning by date with different sizes Mixing partitions by year, month, day in the same table HOW TO: Use the "TO_DAYS" function set appropriate intervals 56

Partitioning by date with different sizes ALTER TABLE salaries partition by range (to_days(from_date)) ( # 5 years # 1 year partition p01 values less than (to_days('1985-01-01')), partition p06 values less than (to_days('1990-01-01')), partition p11 values less than (to_days('1995-01-01')), partition p12 values less than (to_days('1996-01-01')), partition p13 values less than (to_days('1997-01-01')), partition p14 values less than (to_days('1998-01-01')), partition p15 values less than (to_days('1999-01-01')), partition p16 values less than (to_days('2000-01-01')), 57

Partitioning by date with different sizes ALTER TABLE salaries partition by range (to_days(from_date)) ( # 1 month partition p17 values less than (to_days('2001-01-01')), partition p18 values less than (to_days('2001-02-01')), partition p19 values less than (to_days('2001-03-01')), partition p20 values less than (to_days('2001-04-01')), partition p21 values less than (to_days('2001-05-01')), partition p22 values less than (to_days('2001-06-01')), partition p23 values less than (to_days('2001-07-01')), partition p24 values less than (to_days('2001-08-01')), partition p25 values less than (to_days('2001-09-01')), partition p26 values less than (to_days('2001-10-01')), partition p27 values less than (to_days('2001-11-01')), partition p28 values less than (to_days('2001-12-01')), 58

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 59

Hands on - Partitioning with MyISAM Primary key matters - not always needed Size of indexes matters 60

components used for testing MySQL Sandbox created MySQL instances in seconds http://launchpad.net/mysql-sandbox MySQL Employees Test Database has ~ 4 mil records in 6 tables http://launchpad.net/test-db Command line ability fingers ingenuity 61

employees test database 62

How many partitions from information_schema.partitions +-------+-----------------+----------+ pname expr descr +-------+-----------------+----------+ p01 year(from_date) 1985 p02 year(from_date) 1986... p13 year(from_date) 1997 p14 year(from_date) 1998 p15 year(from_date) 1999 p16 year(from_date) 2000... p19 year(from_date) MAXVALUE +-------+-----------------+----------+ 63

How many records select count(*) from salaries; +----------+ count(*) +----------+ 2844047 +----------+ 64

How many records in 1998 not partitioned select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ count(*) +----------+ 247489 +----------+ 1 row in set (1.52 sec) # NOT PARTITIONED 65

How many records in 1998 partitioned select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ count(*) +----------+ 247489 +----------+ 1 row in set (0.41 sec) # partition p15 66

Deleting records in 1998 NOT partitioned delete from salaries where from_date between '1998-01-01' and '1998-12-31'; Query OK, 247489 rows affected (19.13 sec) # NOT PARTITIONED 67

Deleting records in 1998 partitioned alter table salaries drop partition p15; Query OK, 0 rows affected (1.35 sec) 68

Partitions with InnoDB Slower than MyISAM But more robust Requires more storage 69

Partitioning with InnoDB - File per table CREATE TABLE table_name (...... ) ENGINE = INNODB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8 PARTITION BY XXXX 70

Partitions with InnoDB (laptop) Key points: Takes much more storage than other engines engine storage (MB) innodb 221 myisam 181 archive 74 innodb partitioned (whole) 289 innodb partitioned (file per table) 676 myisam partitioned 182 archive partitioned 72 71

Benchmarking results (laptop) engine query year query year InnoDB 2000 1.25 2002 1.25 MyISAM 1.72 1.73 Archive 2.47 2.45 InnoDB partitioned 0.24 0.10 whole InnoDB Partitioned 0.45 0.10 (file per table) MyISAM partitioned 0.18 0.12 Archive partitioned 0.22 0.12 72

Partitions with InnoDB (huge server) Key points: Takes much more storage than other engines engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13 73

Benchmarking results (huge server) engine 6 month range InnoDB 4 min 30s MyISAM 25.03s Archive 22 min 25s InnoDB partitioned by month 13.19 MyISAM partitioned by year 6.31 MyISAM partitioned by month 4.45 Archive partitioned by year 16.67 Archive partitioned by month 8.97 74

Partitions with Archive Key points: REMOVE INDEXES (not supported by archive) For pure statistics applications, ARCHIVE can be ALMOST AS FAST AS MyISAM! 75

Partitions with ARCHIVE - in practice CREATE TABLE titles ( emp_no INT NOT NULL, title VARCHAR(50) NOT NULL, from_date DATE NOT NULL, to_date DATE, KEY (emp_no), FOREIGN KEY (emp_no) REFERENCES employees (emp_no) ON DELETE CASCADE, PRIMARY KEY (emp_no,title, from_date) )ENGINE = InnoDB; 76

Partitions with ARCHIVE - in practice CREATE TABLE titles ( emp_no INT NOT NULL, title VARCHAR(50) NOT NULL, from_date DATE NOT NULL, to_date DATE #, # KEY (emp_nono), # FOREIGN KEY (emp_no) # REFERENCES employees # (emp_no) ON DELETE CASCADE, # PRIMARY KEY (emp_no,title, # from_date) ) ENGINE = ARCHIVE; 77

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 78

TOOLS The INFORMATION_SCHEMA.PARTITIONS table The partition helper http://datacharmer.blogspot.com/2008/12/partitionhelper-improving-usability.html A Perl script that creates partitioning statements The mysqldump_partition_backup... anything you are creating right now :) 79

INFORMATION SCHEMA PARTITIONS table TABLE_NAME PARTITION_NAME SUBPARTITION_NAME PARTITION_ORDINAL_POSITION SUBPARTITION_ORDINAL_POSITION PARTITION_METHOD SUBPARTITION_METHOD PARTITION_EXPRESSION SUBPARTITION_EXPRESSION PARTITION_DESCRIPTION TABLE_ROWS PARTITION_COMMENT 80

INFORMATION SCHEMA PARTITIONS table select * from partitions where table_name='salaries' and table_schema='employees' limit 1\G TABLE_SCHEMA: employees TABLE_NAME: salaries PARTITION_NAME: p01 SUBPARTITION_NAME: NULL PARTITION_ORDINAL_POSITION: 1 SUBPARTITION_ORDINAL_POSITION: NULL PARTITION_METHOD: RANGE COLUMNS SUBPARTITION_METHOD: NULL PARTITION_EXPRESSION: from_date SUBPARTITION_EXPRESSION: NULL PARTITION_DESCRIPTION: '1985-12-31' TABLE_ROWS: 18238 81

INFORMATION SCHEMA PARTITIONS table select partition_name,partition_description, table_rows from partitions where table_name='salaries' and table_schema='employees'; +----------------+-----------------------+------------+ partition_name partition_description table_rows +----------------+-----------------------+------------+ p01 '1985-12-31' 18238 p02 '1986-12-31' 37915 p03 '1987-12-31' 57395 p04 '1988-12-31' 76840 p05 '1989-12-31' 95890 p06 '1990-12-31' 114520 p07 '1991-12-31' 132578 p08 '1992-12-31' 151019 p09 '1993-12-31' 168103 p10 '1994-12-31' 185121 p11 '1995-12-31' 201576 p12 '1996-12-31' 218244 p13 '1997-12-31' 233144 p14 '1998-12-31' 247458 82

The Partition helper Introduction, syntax, and examples http://forge.mysql.com/tools/tool.php?id=173 83

The partition helper $ partition_helper --help The Partition Helper, version 1.0.1 This program creates a ALTER TABLE statement to add or reorganize partitions for MySQL 5.1 or later (C) 2008 Giuseppe Maxia syntax: partitions_helper [options] -t --table = name -c --column = name -i --interval = name "year", "month", or a number -p --partitions = number... 84

The partition helper... --first_partition = number --reorganize = name -s --start = name -e --end = name -f --function = name -l --list --prefix = name --explain 85

The partition helper example partitions_helper --table=t1 \ --column=mydate \ --interval=year \ --start=1998-01-01 --end=2002-12-01 # partitions: 5 ALTER TABLE t1 PARTITION by range (to_days(mydate)) ( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1999-01-01')), partition p003 VALUES LESS THAN (to_days('2000-01-01')), partition p004 VALUES LESS THAN (to_days('2001-01-01')), partition p005 VALUES LESS THAN (to_days('2002-01-01')) ); 86

The partition helper example partitions_helper --table=t1 \ --column=mydate \ --interval=month \ --start=1998-01-01 --end=2002-12-01 # partitions: 60 ALTER TABLE t1 PARTITION by range (to_days(mydate)) ( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1998-02-01')), partition p003 VALUES LESS THAN (to_days('1998-03-01')) #[...], partition p058 VALUES LESS THAN (to_days('2002-10-01')), partition p059 VALUES LESS THAN (to_days('2002-11-01')), partition p060 VALUES LESS THAN (to_days('2002-12-01')) ); 87

The partition helper example partitions_helper --table=t1 \ --column=mydate \ --interval=week \ --start=1998-01-01 --end=2002-12-01 # partitions: 255 ALTER TABLE t1 PARTITION by range (to_days(mydate)) ( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1998-01-08')), partition p003 VALUES LESS THAN (to_days('1998-01-15')) #[...], partition p253 VALUES LESS THAN (to_days('2002-10-31')), partition p254 VALUES LESS THAN (to_days('2002-11-07')), partition p255 VALUES LESS THAN (to_days('2002-11-14')) ); 88

The partition helper example partitions_helper --table=t1 \ --column=mynum \ --interval=250 \ --start=1000 --end=3000 # partitions: 8 ALTER TABLE t1 PARTITION by range (mydate) ( partition p001 VALUES LESS THAN (1250), partition p002 VALUES LESS THAN (1500), partition p003 VALUES LESS THAN (1750), partition p004 VALUES LESS THAN (2000), partition p005 VALUES LESS THAN (2250), partition p006 VALUES LESS THAN (2500), partition p007 VALUES LESS THAN (2750), partition p008 VALUES LESS THAN (3000) ); 89

Partitions maintenance through events An article by Greg Haase about this topic http://dev.mysql.com/tech-resources/articles/ partitioning-event_scheduler.html 90

mysqldump_partition_backup a script created by Roland Bouman http://forge.mysql.com/tools/tool.php?id=258 detects the partitions for a table, and generates mysqldump statements for each one 91

Partitions maintenance through events HOW TO 1. Gather information about partitions (from INFORMATION_SCHEMA) 2. dynamically create the partitions info 3. fire the event 92

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 93

the NULL partition when partitioning by date using TO_DAYS the first partition matters partition rows p1 100,000 p2 150,000 p3 120,000 p4 100,000 94 94

partition condition rows p1 to_days('1990-01-01') 1,000,000 p2 to_days('1990-02-01') 1,000,000 p3 to_days('1990-03-01') 1,000,000 p4 to_days('1990-04-01') 1,000,000 p5 to_days('1990-05-01') 1,000,000 p6 to_days('1990-06-01') 1,000,000 95

date_field between '1990-04-10' and '1990-04-15' partition condition rows p1 to_days('1990-01-01') 1,000,000 p2 to_days('1990-02-01') 1,000,000 p3 to_days('1990-03-01') 1,000,000 p4 to_days('1990-04-01') 1,000,000 p5 to_days('1990-05-01') 1,000,000 p6 to_days('1990-06-01') 1,000,000 = 2,000,000 rows 96

alter table t1 reorganize partition p1 into ( partition p0 values less than (0), partition p1 values less than (to_days('1990-01-01')) ); partition condition rows p0 0 0 p1 to_days('1990-01-01') 1,000,000 p2 to_days('1990-02-01') 1,000,000 p3 to_days('1990-03-01') 1,000,000 p4 to_days('1990-04-01') 1,000,000 p5 to_days('1990-05-01') 1,000,000 p6 to_days('1990-06-01') 1,000,000 97

date_field between '1990-04-10' and '1990-04-15' partition condition rows p0 0 0 p1 to_days('1990-01-01') 1,000,000 p2 to_days('1990-02-01') 1,000,000 p3 to_days('1990-03-01') 1,000,000 p4 to_days('1990-04-01') 1,000,000 p5 to_days('1990-05-01') 1,000,000 p6 to_days('1990-06-01') 1,000,000 = 1,000,000 rows 98

Lock before inserting single records If you send MANY SINGLE INSERT at once LOCK before inserting No need to do it for bulk inserts 99 99

Your needs What Partition pruning Table of contents Syntax Benchmarking Partitioning by date hands on tools tips leveraging replication 100

Replication schemes MASTER concurrent insert INNODB NOT PARTITIONED INNODB NOT PARTITIONED SLAVE concurrent read INNODB PARTITIONED BY RANGE MyISAM PARTITIONED BY RANGE SLAVE SLAVE concurrent batch processing large batch processing 101 101

Replication schemes MASTER INNODB PARTITIONED BY HASH concurrent insert SLAVE MyISAM PARTITIONED BY RANGE INNODB NON PARTITIONED SLAVE batch processing 102 concurrent reads 102

Replication schemes - dimensions MASTER INNODB PARTITIONED BY HASH SLAVE ARCHIVE PARTITIONED BY RANGE (locations) concurrent insert dimensional processing ARCHIVE PARTITIONED BY RANGE (date) SLAVE SLAVE ARCHIVE PARTITIONED BY RANGE (product) dimensional processing 103 dimensional processing 103

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 104 104

Slides for this presentation http://tinyurl.com/mysql-partitions This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. 105

THANKS! Q&A Comment on Twitter: @datacharmer 106

BONUS SLIDES NEW FEATURES 107 107

MySQL 5.5 enhancements PARTITION BY RANGE COLUMNS PARTITION BY LIST COLUMNS TO_SECONDS 108 108

MySQL 5.5 enhancements CREATE TABLE t ( dt date ) PARTITION BY RANGE (TO_DAYS(dt)) ( PARTITION p01 VALUES LESS THAN (TO_DAYS('2007-01-01')), PARTITION p02 VALUES LESS THAN (TO_DAYS('2008-01-01')), PARTITION p03 VALUES LESS THAN (TO_DAYS('2009-01-01')), PARTITION p04 VALUES LESS THAN (MAXVALUE)); BEFORE 109 5.1 109

MySQL 5.5 enhancements SHOW CREATE TABLE t \G Table: t Create Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE (TO_DAYS (dt)) (PARTITION p01 VALUES LESS THAN (733042) ENGINE = MyISAM, [ ] BEFORE 5.1 110 110

MySQL 5.5 enhancements CREATE TABLE t ( dt date ) PARTITION BY RANGE COLUMNS (dt) ( PARTITION p01 VALUES LESS THAN ('2007-01-01'), PARTITION p02 VALUES LESS THAN ('2008-01-01'), PARTITION p03 VALUES LESS THAN ('2009-01-01'), PARTITION p04 VALUES LESS THAN (MAXVALUE)); AFTER 111 5.5 111

MySQL 5.5 enhancements SHOW CREATE TABLE t Table: t Create Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 /*!50500 PARTITION BY RANGE COLUMNS (dt) (PARTITION p01 VALUES LESS THAN ('2007-01-01') ENGINE = MyISAM, [ ] AFTER 5.5 112 112

MySQL 5.5 - Multiple columns CREATE TABLE t ( a int, b int )PARTITION BY RANGE COLUMNS (a,b) ( PARTITION p01 VALUES LESS THAN (10,1), PARTITION p02 VALUES LESS THAN (10,10), PARTITION p03 VALUES LESS THAN (10,20), PARTITION p04 VALUES LESS THAN (MAXVALUE, MAXVALUE)); 113 113

records a b 1 10 10 9 10 10 10 11 partition definition by range (a,b) partition LESS THAN p01 10 10 p02 10 20 p03 10 30 p04 10 MAXVALUE p05 MAXVALUE MAXVALUE 114

records a b 1 10 10 9 10 10 10 11 partition definition by range (a,b) partition LESS THAN p01 10 10 p02 10 20 p03 10 30 p04 10 MAXVALUE p05 MAXVALUE MAXVALUE (1,10) < (10,10)? (a < 10) OR ((a = 10) AND (b < 10)) (1 < 10) OR ((1 = 10) AND (10 < 10)) TRUE 115

records a b 1 10 10 9 10 10 10 11 partition definition by range (a,b) partition LESS THAN p01 10 10 p02 10 20 p03 10 30 p04 10 MAXVALUE p05 MAXVALUE MAXVALUE (10,9) < (10,10)? (a < 10) OR ((a = 10) AND (b < 10)) (10 < 10) OR ((10 = 10) AND (9 < 10)) TRUE 116

records a b 1 10 10 9 10 10 10 11 partition definition by range (a,b) partition LESS THAN p01 10 10 p02 10 20 p03 10 30 p04 10 MAXVALUE p05 MAXVALUE MAXVALUE (10,10) < (10,10)? (a < 10) OR ((a = 10) AND (b < 10)) (10 < 10) OR ((10 = 10) AND (10 < 10)) FALSE 117

records a b 1 10 10 9 10 10 10 11 partition definition by range (a,b) partition LESS THAN p01 10 10 p02 10 20 p03 10 30 p04 10 MAXVALUE p05 MAXVALUE MAXVALUE (10,10) < (10,20)? (a < 10) OR ((a = 10) AND (b < 20)) (10 < 10) OR ((10 = 10) AND (10 < 20)) TRUE 118

CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) DEFAULT NULL, hire_date date NOT NULL ) ENGINE=MyISAM PARTITION BY RANGE COLUMNS(gender,hire_date) (PARTITION p01 VALUES LESS THAN ('F','1990-01-01'), PARTITION p02 VALUES LESS THAN ('F','2000-01-01'), PARTITION p03 VALUES LESS THAN ('F',MAXVALUE), PARTITION p04 VALUES LESS THAN ('M','1990-01-01'), PARTITION p05 VALUES LESS THAN ('M','2000-01-01'), PARTITION p06 VALUES LESS THAN ('M',MAXVALUE), PARTITION p07 VALUES LESS THAN (MAXVALUE,MAXVALUE) 119 119

MySQL 5.5 enhancements TRUNCATE PARTITION TO_SECONDS() 120 120

MySQL 5.6 enhancements ALTER TABLE EXCHANGE PARTITION 121 121

ALTER TABLE t1 EXCHANGE PARTITION p2 WITH TABLE t2 122

p1 p2 p3 t1 f1 f2 f3 1 a xxx 2 b xxx 3 c yyy 4 d xxx 5 e zzz 6 f www 7 g ccc 8 h sss 9 i www 10 j xxx 11 k yyy 12 l zzz t2 f1 f2 f3 (empty) 123

p1 p2 p3 t1 f1 f2 f3 1 a xxx 2 b xxx 3 c yyy 4 d xxx 5 e zzz 6 f www 7 g ccc 8 h sss 9 i www 10 j xxx 11 k yyy 12 l zzz t2 f1 f2 f3 (empty) 124

p1 p2 p3 t1 f1 f2 f3 1 a xxx 2 b xxx 3 c yyy 4 d xxx (empty) 9 i www 10 j xxx 11 k yyy 12 l zzz t2 f1 f2 f3 5 e zzz 6 f www 7 g ccc 8 h sss 125