Load Data Fast! BILL KARWIN PERCONA LIVE OPEN SOURCE DATABASE CONFERENCE 2017

Transcription

1 Load Data Fast! BILL KARWIN PERCONA LIVE OPEN SOURCE DATABASE CONFERENCE 2017

2 Bill Karwin Software developer, consultant, trainer Using MySQL since 2000 Senior Database Architect at SchoolMessenger SQL Antipatterns: Avoiding the Pitfalls of Database Programming Oracle ACE Director

3 Load Data Fast! Common chores Dump and restore Import third-party data Extract, Transfer, Load (ETL) Test data that needs to be reloaded repeatedly Is it done yet?

4 How to Speed This Up? 1. Query Solutions 2. Schema Solutions 3. Configuration Solutions 4. Parallel Execution Solutions

5 Example Table CREATE TABLE TestTable ( id INT UNSIGNED NOT NULL PRIMARY KEY, intcol INT UNSIGNED DEFAULT NULL, stringcol VARCHAR(100) DEFAULT NULL, textcol TEXT ) ENGINE=InnoDB; Let s load 1 million rows!

6 Best Case Performance Running a test script to loop over 1 million rows, without inserting to a database. $ php test-bulk-insert.php --total-rows noop This should have a speed that is the upper bound for any subsequent test. Time: 2 seconds (00:00:02) rows = rows/sec stmt = stmt/sec txns = txns/sec conn = conn/sec

7 Worst Case Performance INSERT INTO TestTable (id, intcol, stringcol, textcol) VALUES (?,?,?,?); Run a test script that executes one INSERT, commits, reconnects. $ php test-bulk-insert.php --total-rows Time: 34 seconds (00:00:34) rows = rows/sec stmt = stmt/sec txns = txns/sec conn = conn/sec

8 Inserting One Row: Overhead Connecting Sending query Parsing Inserting row Closing query

9 Query Solutions

10 Inserting One Row at a Time INSERT INTO TestTable (id, intcol, stringcol, textcol) VALUES (?,?,?,?); Run a test script that executes one INSERT, commits using a single connection. $ php test-bulk-insert.php --total-rows \ --txns-per-conn Time: 527 seconds (00:08:47) rows = rows/sec stmt = stmt/sec txns = txns/sec 1 conn = 0.00 conn/sec

11 Inserting One Row: Overhead Sending query Parsing Inserting row Closing query

12 Inserting Multiple Rows INSERT INTO TestTable (id, intcol, stringcol, textcol) VALUES (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?), (?,?,?,?); Q: How many rows can you insert in one statement? A: As many as fit in max_allowed_packet bytes.

13 Inserting Multiple Rows: Overhead Sending query Parsing Inserting row Closing query

14 Inserting Multiple Rows: Results $ php Test-bulk-insert.php --total-rows \ --rows-per-stmt txns-per-conn Time: 85 seconds (00:01:25) rows = rows/sec stmt = stmt/sec txns = txns/sec 1 conn = 0.01 conn/sec

15 Transactions BEGIN TRANSACTION; INSERT INTO TestTable INSERT INTO TestTable INSERT INTO TestTable INSERT INTO TestTable INSERT INTO TestTable INSERT INTO TestTable COMMIT; Q: How many statements can you do in one transaction? A: In theory this is constrained by undo log segments, but it's a lot.

16 Transactions: Results $ php test-bulk-insert.php --total-rows \ --rows-per-stmt stmts-per-txn txns-per-conn 100 Time: 63 seconds (00:01:03) rows = rows/sec stmt = stmt/sec 100 txns = 1.57 txns/sec 1 conn = 0.02 conn/sec

17 Inserting with Prepared Queries BEGIN TRANSACTION; PREPARE INSERT INTO TestTable EXECUTE EXECUTE EXECUTE EXECUTE COMMIT; Q: How many times can you execute a given prepared statement? A: There is no limit, as far as I can tell.

18 Prepared Queries: Overhead Sending query Parsing Inserting row Inserting row Inserting row Inserting row Closing query

19 Prepared Queries: Results $ php test-bulk-insert.php --total-rows \ --rows-per-stmt stmts-per-txn txns-per-conn 100 Time: 63 seconds (00:01:03) rows = rows/sec $ php test-bulk-insert.php --total-rows \ --rows-per-stmt stmts-per-txn txns-per-conn 100 \ --emulate-prepares Time: 95 seconds (00:01:35) rows = rows/sec

20 Load Data in File: Results mysql> LOAD DATA LOCAL INFILE 'TestTable.csv' INTO TABLE TestTable; Flat-file data load in a single transaction. Works with replication.

21 Overhead: Load Data Infile Sending query Parsing LOAD DATA INFILE Closing query

22 Load Data in File: Results $ php test-bulk-insert.php --total-rows load-data Time: 25 seconds (00:00:25) rows = rows/sec 1 stmt = 0.04 stmt/sec 1 txns = 0.04 txns/sec 1 conn = 0.04 conn/sec

23 Load XML in File: Results LOAD XML LOCAL INFILE 'TestTable.xml' INTO TABLE TestTable; $ php test-bulk-insert.php --total-rows load-xml Time: 77 seconds (00:01:17) rows = rows/sec 1 stmt = 0.01 stmt/sec 1 txns = 0.01 txns/sec 1 conn = 0.01 conn/sec

24 What about Load JSON in File? Sorry, the hypothetical LOAD JSON INFILE is not supported by MySQL yet. But it has been proposed as a feature request: Go vote for it! Or better yet, implement it and contribute a patch!

25 Schema Solutions

26 Indexes How much overhead for one index? Two indexes? 1. mysql> ALTER TABLE TestTable ADD INDEX (intcol); 2. mysql> ALTER TABLE TextTable ADD INDEX (stringcol);

27 Indexes: Overhead Sending query Parsing Inserting row Inserting indexes Closing query

28 Indexes: Results $ php test-bulk-insert.php --total-rows rows-per-stmt 100 \ --stmts-per-txn txns-per-conn 100 Time: 63 seconds (00:01:03) rows = rows/sec $ php test-bulk-insert.php --total-rows rows-per-stmt 100 \ --stmts-per-txn txns-per-conn indexes 1 Time: 71 seconds (00:01:11) rows = rows/sec $ php test-bulk-insert.php --total-rows rows-per-stmt 100 \ --stmts-per-txn txns-per-conn indexes 2 Time: 95 seconds (00:01:35) rows = rows/sec

29 Index Deferral What if we insert with no indexes, and build indexes at the end? Thi is what Percona s mysqldump --innodb-optimize-keys does. Load time is like when you have no indexes: Time: 63 seconds (00:01:03) rows = rows/sec Then create indexes after data load. This reduces the effective rate of rows/second: mysql> ALTER TABLE TestTable ADD INDEX (intcol); Query OK, 0 rows affected (7.02 sec) mysql> ALTER TABLE TestTable ADD INDEX (stringcol); Query OK, 0 rows affected (8.54 sec) Time: seconds (00:01:35) rows = rows/sec effective data load rate

30 Triggers How much overhead for a trigger? mysql> CREATE TRIGGER TestTrigger BEFORE INSERT ON TestTable FOR EACH ROW SET NEW.stringCol = UPPER(NEW.stringCol); This is a very simple trigger. If you have more complex code, like subordinate INSERT statements, the cost will be higher.

31 Triggers: Results $ php test-bulk-insert.php --total-rows \ --rows-per-stmt stmts-per-txn txns-per-conn 100 \ --trigger Time: 69 seconds (00:01:09) rows = rows/sec stmt = stmt/sec 100 txns = 1.43 txns/sec 1 conn = 0.01 conn/sec

32 CSV Storage Engine mysql> CREATE TABLE TestTable ( id INT UNSIGNED NOT NULL, intcol INT UNSIGNED NOT NULL, stringcol VARCHAR(100) NOT NULL, textcol TEXT NOT NULL ) ENGINE=CSV; # ls -l /usr/local/mysql/data/test total 24 -rw-r _mysql _mysql 5824 Apr 22 20:10 TestTable_429.SDI -rw-r _mysql _mysql 35 Apr 22 20:10 testtable.csm -rw-r _mysql _mysql 0 Apr 22 20:10 testtable.csv

33 CSV Storage Engine Move CSV file into datadir: # time cp data.csv /usr/local/mysql/data/test/testtable.csv real 0m8.359s # ls -l /usr/local/mysql/data/test/ total rw-r _mysql _mysql 5824 Apr 22 20:18 TestTable_431.SDI -rw-r _mysql _mysql 35 Apr 22 20:18 testtable.csm -rw-r _mysql _mysql Apr 22 20:19 testtable.csv Time: (00:00:08) rows = rows/sec

34 CSV into InnoDB Storage Engine Use CSV storage engine, then alter to InnoDB table (and add a primary key): ALTER TABLE TestTable ADD PRIMARY KEY (id), ENGINE=InnoDB; Query OK, rows affected (1 min sec) Time: seconds (00:01:46) rows = rows/sec effective data load rate

35 Partitioning

36 Transportable Tablespaces

37 Configuration Solutions

38 Increase Buffering, Decrease Durability innodb_buffer_pool_size = 4G (default 128M) innodb_log_buffer_size = 1G (default 16M) innodb_log_file_size = 4G (default 48M) innodb_flush_log_at_trx_commit = 0 (default 1) # log-bin = mysql-bin Time: 56 seconds (00:00:56) rows = rows/sec

39 Increase Buffering, Decrease Durability Same, but at least flush the log buffer: innodb_flush_log_at_trx_commit = 2 (default 1) Time: 60 seconds (00:01:00) rows = rows/sec

40 Tuning + Load Data $ php test-bulk-insert.php --total-rows load-data Time: 22 seconds (00:00:22) rows = rows/sec

41 Config for More Buffering Innodb_buffer_pool_size=4G (default 128M) Innodb_change_buffering=none (default all) Innodb_log_buffer_size=1G (default 16M) Binlog_cache_size=256K) (default 32K) Time: 82 seconds (00:01:22) rows = rows/sec Time: 81 seconds (00:01:21) rows = rows/sec

42 Config for Greater Throughput Innodb_log_file_size=4G (default 48M) Innodb_io_capacity=2000 (default 200) Innodb_lru_scan_depth=8192 (default 1024) Time: 80 seconds (00:01:20) rows = rows/sec Time: 80 seconds (00:01:20) rows = rows/sec Time: 81 seconds (00:01:21) rows = rows/sec

43 Config for Lower Durability Innodb_doublewrite=OFF (default ON) Innodb_flush_log_at_trx_commit=0 (default 1) Time: 85 seconds (00:01:25) rows = rows/sec Time: 84 seconds (00:01:24) rows = rows/sec # Log_bin Time: 82 seconds (00:01:22) rows = rows/sec Sync_binlog=0 (default 1) Time: 83 seconds (00:01:23) rows = rows/sec

44 Config for Fewer Checks Innodb_checksum_algorithm=none (default crc32) Innodb_log_checksums=OFF (default ON) Foreign_key_checks=0 (default 1) Unique_checks=0 (default 1) Time: 84 seconds (00:01:24) rows = rows/sec Time: 84 seconds (00:01:24) rows = rows/sec

45 Parallel Execution Solutions

46 Parallel Import Like LOAD DATA INFILE but supports multi-threaded import: $ mysqlimport --local --use-threads 4 \ dbname table1 table2 table3 table4 Runs a fixed number of threads, imports one table per thread. If an import finishes and there are more tables, first available thread does it.

47 Parallel Import Connecting to localhost Connecting to localhost Connecting to localhost Connecting to localhost Selecting database test Selecting database test Selecting database test Selecting database test Loading data from LOCAL file: TestTable2.csv into TestTable2 Loading data from LOCAL file: TestTable3.csv into TestTable3 Loading data from LOCAL file: TestTable1.csv into TestTable1 Loading data from LOCAL file: TestTable4.csv into TestTable4 test.testtable3: Records: Deleted: 0 Skipped: 0 Warnings: 0 Disconnecting from localhost test.testtable1: Records: Deleted: 0 Skipped: 0 Warnings: 0 Disconnecting from localhost test.testtable2: Records: Deleted: 0 Skipped: 0 Warnings: 0 Disconnecting from localhost test.testtable4: Records: Deleted: 0 Skipped: 0 Warnings: 0 Disconnecting from localhost

48 MysqlImport: Results $ php test-bulk-insert.php --total-rows load-data \ --use-threads 4 Time: 31 seconds (00:00:31) rows = rows/sec 4 stmt = 0.13 stmt/sec 4 txns = 0.13 txns/sec 4 conn = 0.13 conn/sec

49 Conclusions

50 50000 Rows per Second why are you still doing this?

51 Want to Try The Tests Yourself? The test-bulk-insert.php script is available here:

52 One Last Thing What Was Our Solution? We cheated: Load database once. Take a filesystem snapshot. Run tests. Restore from snapshot. Re-run tests. etc. This is not a good solution for everyone. It worked for one specific use case.

53 License and Copyright Copyright 2017 Bill Karwin Released under a Creative Commons 3.0 License: You are free to share to copy, distribute, and transmit this work, under the following conditions: Attribution. You must attribute this work to Bill Karwin. Noncommercial. You may not use this work for commercial purposes. No Derivative Works. You may not alter, transform, or build upon this work.