The Blackhole and Federated Storage Engines: The Coolest Kids on the Block

Transcription

1 The Blackhole and Federated Storage Engines: The Coolest Kids on the Block Kai Voigt, Senior Instructor, MySQL AB Giuseppe Maxia, QA Developer, MySQL AB

2 Kai Voigt Mister B MySQL Instructor and Consultant Based in Kiel, Germany Will Trade Advice for Beer

3 Giuseppe Maxia Mister F MySQL QA Developer Based in Sardinia, Italy Plays Chess, but cheats!

4 Kai Voigt & Giuseppe Maxia according to Google Maps, they are both excellent swimmers

5 Kai Voigt & Giuseppe Maxia according to Google Maps, they are both excellent swimmers

6 Kai Voigt & Giuseppe Maxia according to Google Maps, they are both excellent swimmers

7 Kai Voigt & Giuseppe Maxia according to Google Maps, they are both excellent swimmers

8 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

9 MySQL Storage Engine Concept Table Level Different Media, different Features Most popular: InnoDB, MyISAM, Memory, Cluster SQL Layer Disk Memory Cluster Nowhere Remote Database

10 Blackhole and Federated Storage Engines Storage Engines... without Storage Blackhole: Every data you put into it disappears Federated: Cockoo table that takes storage from a remote database

11 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

12 Blackhole Overview Storage Media: Trashbin, aka /dev/null Simple implementation, bugfree INSERT, UPDATE and DELETE statements are very fast! SELECT statements too Backup is easy, so is Recovery Most common question: Why the heck do I need that Storage Engine?

13 Blackhole Example [test]> CREATE TABLE bh (a INT) ENGINE=BLACKHOLE; Query OK, 0 rows affected (0.00 sec) root@localhost [test]> INSERT INTO bh (a) VALUES (42); Query OK, 1 row affected (0.00 sec) root@localhost [test]> SELECT * FROM bh; Empty set (0.00 sec)

14 Storage Engines Code Size $ ls -l */ha_*.cc -rw-r--r-- 1 k k Jan 30 08:34 archive/ha_archive.cc -rw-r--r-- 1 k k 5770 Jan 30 08:34 blackhole/ha_blackhole.cc -rw-r--r-- 1 k k Jan 30 08:34 csv/ha_tina.cc -rw-r--r-- 1 k k Jan 30 08:34 example/ha_example.cc -rw-r--r-- 1 k k Jan 30 08:34 federated/ha_federated.cc -rw-r--r-- 1 k k Jan 30 08:34 heap/ha_heap.cc -rw-r--r-- 1 k k Jan 30 08:34 myisam/ha_myisam.cc -rw-r--r-- 1 k k Jan 30 08:34 myisammrg/ha_myisammrg.cc

15 Implementation Example int ha_blackhole::write_row(byte * buf) { DBUG_ENTER("ha_blackhole::write_row"); DBUG_RETURN(0); }

16 Features Available ACID-Transactions... A+C+I, no D Full Character Set Support Not Available Table Level Locks AUTO INCREMENT

17 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

18 What s a Federated table?

19 Creating a Federated table # REMOTE SERVER CREATE TABLE City ( ID int(11) NOT NULL auto_increment, Name char(35) NOT NULL, CountryCode char(3) NOT NULL, District char(20) NOT NULL, Population int(11) NOT NULL, PRIMARY KEY (ID) ) ENGINE = MyISAM;

20 Creating a Federated table # LOCAL SERVER CREATE TABLE City ( ID int(11) NOT NULL auto_increment, Name char(35) NOT NULL, CountryCode char(3) NOT NULL, District char(20) NOT NULL, Population int(11) NOT NULL, PRIMARY KEY (ID) ) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/world/City';

21 Creating a Federated table ENGINE = FEDERATED CONNECTION= 'mysql:// user:pass@remote.com:3306/world/city';

22 Creating a Federated table ENGINE = FEDERATED CONNECTION= 'mysql:// user:pass@remote.com:3306/world/city'; Username and password host port schema table name

23 Creating a Federated table (>= ) # LOCAL SERVER (1) CREATE SERVER my_server OPTIONS ( user = 'user', password= 'password', host='remote.com', port = 3306, database = 'world', ) ;

24 Creating a Federated table (>= ) # LOCAL SERVER (2) CREATE TABLE City ( ID int(11) NOT NULL auto_increment, Name char(35) NOT NULL, CountryCode char(3) NOT NULL, District char(20) NOT NULL, Population int(11) NOT NULL, PRIMARY KEY (ID) ) ENGINE = FEDERATED CONNECTION='my_server';

25 What happens when you create a Federated table # REMOTE SERVER general log 7 Connect remote_user@your_host on test 7 Query SELECT * FROM City WHERE 1=0 7 Quit

26 FMM (Federated Missing Manual) FMM Rule # 1 The remote server does not know it s being federated

27 FMM (Federated Missing Manual) FMM Rule # 2 Creating a federated table is an inexpensive operation

28 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

29 Blackhole Scenarios Good Laughter in Classes Submitting Talk for the User Conference Benchmarking and Testing SQL Layer Multi Level Replication Real Time Data Warehousing!

30 Benchmarking and Testing SQL Layer Stored Procedure or Query takes a long time. Is it the SQL Layer taking the time? Or is it the Storage Layer? Use Blackhole Storage Engine Storage Time goes down to Zero Time for SQL Layer remains Testing SQL Layer Will the Dump File execute on the new MySQL version?

31 Multi Level Replication Master Slave/ Master Slave/ Master Slave/ Master Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave

32 Multi Level Replication Transactions on the Master InnoDB Read Requests on the Slaves MyISAM No Data needed on Relay Systems Blackhole Binary Log will still be created

33 Real Time Data Warehousing Triggers work on Blackhole Tables Automatically fill Summary Tables Avoiding expensive Joins Only work with INSERT Statements No DELETE or UPDATE

34 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

35 First operation with a Federated table (local) # LOCAL SERVER SHOW TABLES; Tables_in_test City

36 First operation with a Federated table (remote) # REMOTE SERVER general log 8 Connect server2usr@localhost on test # REMOTE SERVER show processlist; Id User Host db remote_user your_host:46443 test 8 root localhost test

37 Creating a second Federated table # LOCAL SERVER CREATE TABLE City2 ( ID int(11) NOT NULL auto_increment, Name char(35) NOT NULL, CountryCode char(3) NOT NULL, District char(20) NOT NULL, Population int(11) NOT NULL, PRIMARY KEY (ID) ) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/world/City';

38 After creating the second Federated table (remote) # REMOTE SERVER general log 8 Connect server2usr@localhost on test 10 Connect server2usr@localhost on test # REMOTE SERVER show processlist; Id User Host db remote_user your_host:46443 test 10 remote_user your_host:46456 test 8 root localhost test

39 FMM (Federated Missing Manual) FMM Rule # 3 Each federated table creates a permanent connection to the remote server (even if using the same table)

40 TROUBLE with Federated Let s the trouble begin

41 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE ID = 1; ID Name CountryCode District Population Kabul AFG Kabol

42 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE ID = 1; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City WHERE ID = 1

43 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE ID = 1; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City WHERE ID = 1 SO FAR, SO GOOD

44 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE Name = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol

45 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE Name = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City

46 Naively using a Federated table # LOCAL SERVER SELECT * FROM City WHERE Name = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City DISASTER! (no WHERE clause)

47 FMM (Federated Missing Manual) FMM Rule # 4 The WHERE clause is honored only when using an index

48 TROUBLE with Federated Cheating the system

49 Re-creating the Federated table (alter table not supported) # LOCAL SERVER CREATE TABLE City ( ID int(11) NOT NULL auto_increment, Name char(35) NOT NULL, CountryCode char(3) NOT NULL, District char(20) NOT NULL, Population int(11) NOT NULL, PRIMARY KEY (ID), KEY (Name) # Let's cheat it ) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/world/City';

50 Smartly using a Federated table # LOCAL SERVER SELECT * FROM City WHERE City = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol

51 Smartly using a Federated table # LOCAL SERVER SELECT * FROM City WHERE City = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City WHERE Name = 'Kabul'

52 Smartly using a Federated table # LOCAL SERVER SELECT * FROM City WHERE City = 'Kabul'; ID Name CountryCode District Population Kabul AFG Kabol # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City WHERE Name = 'Kabul' Now you're talking!

53 TROUBLE with Federated more trouble...

54 Naively using a Federated table # LOCAL SERVER SELECT COUNT(*) FROM City; COUNT(*)

55 Naively using a Federated table # LOCAL SERVER SELECT COUNT(*) FROM City; COUNT(*) # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City

56 Naively using a Federated table # LOCAL SERVER SELECT COUNT(*) FROM City; COUNT(*) # REMOTE SERVER general log 8 Query SELECT ID,Name,CountryCode,District,Population FROM City Disaster again!

57 FMM (Federated Missing Manual) FMM Rule # 5 Aggregate functions in federated tables are EXTREMELY EXPENSIVE

58 TROUBLE with Federated Getting smarter

59 Getting smarter with Federated tables

60 Getting smarter with Federated tables # REMOTE SERVER CREATE VIEW CityCount AS SELECT COUNT(*) AS c FROM City;

61 Getting smarter with Federated tables # REMOTE SERVER CREATE VIEW CityCount AS SELECT COUNT(*) AS c FROM City; # LOCAL SERVER CREATE TABLE CityCount ( c INT(11) ) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/world/CityCount';

62 Smarter usage of a Federated table # LOCAL SERVER SELECT * FROM CityCount c

63 Smarter usage of a Federated table # LOCAL SERVER SELECT * FROM CityCount c # REMOTE SERVER general log 8 Query SELECT c FROM CityCount

64 Smarter usage of a Federated table # LOCAL SERVER SELECT * FROM CityCount c # REMOTE SERVER general log 8 Query SELECT c FROM CityCount Cheated at its own game!

65 FMM (Federated Missing Manual) FMM Rule # 6 You can federate a VIEW instead of a table

66 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

67 Federating different things

68 Using nested queries in MySQL 3.23! # LOCAL SERVER (5.0.37) mysql> SELECT * -> FROM City -> WHERE CountryCode = ( -> SELECT Code FROM Country WHERE Name = 'Afghanistan'); ID Name CountryCode District Population Kabul AFG Kabol Qandahar AFG Qandahar Herat AFG Herat Mazar-e-Sharif AFG Balkh

69 Using nested queries in MySQL 3.23! # REMOTE SERVER ( ) general log 2 Query SELECT `Code`, `Name`,... FROM `Country` WHERE (`Name` = 'Afghanistan') 1 Query SELECT `ID`, `Name`,... FROM `City` WHERE (`CountryCode` = 'AFG')

70 FMM (Federated Missing Manual) FMM Rule # 7 You can federate any MySQL table, regardless of the remote server version

71 Cool usage of Federated tables (replication) help in replication failover (1)

72 Cool usage of Federated tables (replication) help in replication failover (2)

73 Cool usage of Federated tables (replication) help in replication failover (3)

74 Cool usage of Federated tables (replication) help in replication failover (4)

75 Cool usage of Federated tables (run remotely)

76 Remote execution with Federated tables

77 Remote execution with Federated tables # REMOTE SERVER (1) CREATE TABLE params_table (y int)

78 Remote execution with Federated tables # REMOTE SERVER (1) CREATE TABLE params_table (y int) # LOCAL SERVER (1) CREATE TABLE params_table (y int) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/test/params_table'

79 Remote execution with Federated tables

80 Remote execution with Federated tables # REMOTE SERVER (2) CREATE FUNCTION my_function() RETURNS INT BEGIN INSERT INTO some_table (x) VALUES ((SELECT y FROM params_table)); RETURN 1; END;

81 Remote execution with Federated tables # REMOTE SERVER (2) CREATE FUNCTION my_function() RETURNS INT BEGIN INSERT INTO some_table (x) VALUES ((SELECT y FROM params_table)); RETURN 1; END; # LOCAL SERVER (2) no action required

82 Remote execution with Federated tables

83 Remote execution with Federated tables # REMOTE SERVER (3) CREATE VIEW run_it AS SELECT 1 FROM Dummy WHERE my_function();

84 Remote execution with Federated tables # REMOTE SERVER (3) CREATE VIEW run_it AS SELECT 1 FROM Dummy WHERE my_function(); # LOCAL SERVER (3) CREATE TABLE run_it (x int) ENGINE = FEDERATED CONNECTION='mysql://user:pass@remote.com: 3306/test/run_it'

85 Remote execution with Federated tables

86 Remote execution with Federated tables # LOCAL SERVER (4) DELETE FROM params_table; INSERT INTO params_table VALUES (1); SELECT * FROM run_it; # This command will execute my_function() on the remote server

87 Agenda Storage Engines Blackhole Overview Federated Overview Blackhole Scenarios Federated gotchas Federated Scenarios Questions & Answers

88 Questions & Answers You know what happens if you ask a Blackhole table... I don t know anything. You know what happens if you ask a Federated table... I have to ask someone else.

89 Thank you! See you next year!