MySQL Proxy. Making MySQL more flexible Jan Kneschke

Size: px

Start display at page:

Download "MySQL Proxy. Making MySQL more flexible Jan Kneschke"

Morgan McCormick
6 years ago
Views:

1 MySQL Proxy Making MySQL more flexible Jan Kneschke

2 MySQL Proxy proxy servers forward requests to backends and can transform, handle or block them released under the GPL see developed as part of the Enterprise Tools since February 2007

3 Design Decisions goal is to be transparent to the application layer supports all platforms and languages designed to handle thousands of parallel connections (c10k) uses a embedded scripting language for customizations

4 Transparency SHOW WARNINGS can be worked around with Query Injection SELECT USER() shows the connected user (the proxy, not the client) which can be corrected with result set rewriting host auth against the MySQL server

5 Latency early tests via localhost same script run directly and through the proxy latency per mysql packet: 0.4ms ping RTT on 1Gbit: 0.1ms

6 Load Balancing load balancing distributes the load across several slaves Shortest Queue First is default send new connections to the server with the least number of open connections

7 Fail Over dead host are detected taking out of load balancing for 2min uses custom load balancers to decide how to handle a dead host hot + standby uses load balancing

8 Removing SPoF one Proxy == Single Point of Failure use external Heartbeat (linuxha.org) or 2 LB proxies + 1 Host Selecting Proxy per application server

9 Failsafe Load Balancing

10 Flexibility proxy embeds LUA allows analyzing and manipulating packets Inspection Rewriting Blocking Injection

11 LUA PiL embedded, simple, efficient can do OO like programming has scalars, tables, metatables and anonymous functions

12 Query Rewriting Macro Packages (ls, cd, who,...) tagging queries with SQL_CACHE migrating table names and SQL dialects turn EXPLAIN UPDATE DELETE into equivalent EXPLAIN SELECT

13 Query Profiling SHOW SESSION STATUS around a Query Exec_time: 6749 us.. Handler_read_rnd_next = Handler_write = Select_scan = 1

14 Query Statistics Normalize Queries to track query usage Count Table and Index usage Optimize Query Cache Usage by injecting SQL_CACHE in cachable queries see lib/analyze queries.lua

15 Auditing Diagnostic Auditing track which user+ip run which query or accessed which objects when assign query costs log gathered information in a central place see lib/auditing.lua

16 Global Transaction ID Inject a counter in all transactions Answers questions like which slave is most current can I read from this slave, or do I have to read from master you name it

17 Connection Pooling reusing open connections between proxy and server reduces concurrency on the MySQL Server external connection pool for PHP

18 Statement Routing split the query stream into reading and writing READs go to the slaves WRITEs and transactions to the master automatic scale out sharding

19 Tokenizer turns a SQL query into a token stream not a full parser, just a tokenizer for speed reasons understands KEYWORDS, /*comments*/, strings, 123 and `literals` later we'll add support for SQL modes

20 normalizing Queries 1: { TK_SQL_SELECT, select } 2: { TK_STAR, * } 3: { TK_SQL_FROM, from } 4: { TK_LITERAL, t1 } 5: { TK_SQL_WHERE, where } 6: { TK_LITERAL, id } 7: { TK_EQ, = } 8: { TK_INTEGER, 1 } normalized query: SELECT * FROM `t1` WHERE `id` =?

21 Libraries auto config SET GLOBAL... balance load balancers commands parser extract tablenames tokenizer normalize() cleanup queries parse MySQL Command Packets

22 Internals LUA scripting proxy.* is the namespace proxy.connection.* is the current connection proxy.backends[...] are the backends proxy.global.* is the global table proxy.global.config.* is used for the config

23 Internals Scope Each connection has its own script scope proxy.global.* to share data between connections use local to make variables local to the function use package.seeall() to export functions from modules

24 Internals Threading the global scope and threading don't play nice by default patches lua to apply mutexes around variable access

25 Internals Script Cache we reload the script on each connection start adding a script cache with mtime check lua_pushvalue(l, 1) does the trick

26 Roadmap to be released tokenizer read write splitting Query Statistics later parallel Queries proxy initiates connections

27 LUA Gotchas only false and nil are!true, 0 is true to say not equal you use ~= there are no shortcuts no a++, no a *= 4,... no a > b? a : b (there is (a > b) and a or b)

Getting started with MySQL Proxy

Getting started with MySQL Proxy Giuseppe Maxia QA Developer - MySQL AB Sofia - OpenFest 2007 Agenda Overview Some fun Injecting queries Filtering and rewriting queries Working with results Proxy for logging