Skytools3. PostgreSQL trigger-based replication. Dimitri Fontaine July,

Transcription

1 Skytools3 PostgreSQL trigger-based replication Dimitri Fontaine July, Dimitri Fontaine Skytools3 July, / 27

2 Dimitri Fontaine 2ndQuadrant France PostgreSQL Major Contributor X pgloader, prefix, skytools,... X apt.postgresql.org X CREATE EXTENSION X CREATE EVENT TRIGGER X MySQL migration tool, new pgloader version Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

3 Dimitri Fontaine 2ndQuadrant France PostgreSQL Major Contributor X pgloader, prefix, skytools,... X apt.postgresql.org X CREATE EXTENSION X CREATE EVENT TRIGGER X MySQL migration tool, new pgloader version Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

4 Dimitri Fontaine 2ndQuadrant France PostgreSQL Major Contributor X pgloader, prefix, skytools,... X apt.postgresql.org X CREATE EXTENSION X CREATE EVENT TRIGGER X MySQL migration tool, new pgloader version Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

5 Skytools Skype Tools for Replication X PGQ X londiste X walmgr Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

6 Skytools Architecture PGQ is organized into 3 components Producer, Consumer, Ticker Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

7 PGQ3 Some things did change in PGQ version 3 X The ticker is now pgqd X The network topology is now known pgq nodes X And we have Cooperative Consumers pgq coop Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

8 Londiste3 concepts Londiste relies on PGQ nodes X Root X Branch X Leaf X Leaf merge= qname Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

9 Operating londiste Basic commands X status X members X change-provider X takeover all dead Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

10 Londiste Add Table Dimitri Fontaine Skytools3 July, / 27

11 londiste add-table Plenty new options in add-table X --wait-sync X --dest-table X --skip-truncate X --create, --create-full X --trigger-flags X --trigger-arg X --no-triggers X --copy-node, --copy-condition X --merge-all, --no-merge X --max-parallel-copy Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

12 DDL Dimitri Fontaine Skytools3 July, / 27

13 DDL Handling X londiste conf.ini execute X EXECUTE queue events X System Table londiste.applied execute X SQL meta-data attributes Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

14 execute Meta-Data attributes --*-- --*-- Local-Table: mytable, othertable, --*-- thirdtable --*-- Local-Sequence: thisseq --*-- X Local-Table Table must be added to local node with add-table. X Need-Table Physical table must exist in database. It does not matter if it is replicated or not. Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

15 DDL and table renaming --*-- Local-Table: mytable ALTER TABLE Dimitri Fontaine Skytools3 July, / 27

16 Londiste Handlers It s possible to register handlers to deal with specific needs X add --handler [ --handler-arg... ] X show-handlers X pl/proxy sharding, re-sharding Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

17 Sharding pluging Dimitri Fontaine Skytools3 July, / 27

18 Handler: shard X Event filtering by hash, for partitioned databases. X key=column column name to use for hashing X hashfunc=funcname function to use for hashing X (default: partconf.get hash raw) X ev extra3= hash= partconf.get hash raw(key column) Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

19 Preparing for part: hashlib >... add-table pgbench_accounts \ > --handler=part --handler-arg=key=aid psql rootdb < /usr/share/postgresql/8.4/contrib/hashlib.sql psql sharddb_0 < /usr/share/postgresql/8.4/contrib/hashlib.sql psql sharddb_1 < /usr/share/postgresql/8.4/contrib/hashlib.sql > psql rootdb -c create extension hashlib; > psql sharddb_0 -c create extension hashlib; > psql sharddb_1 -c create extension hashlib; Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

20 Preparing for part: setup CREATE SCHEMA partconf; CREATE TABLE partconf.conf ( part_nr integer, max_part integer, db_code bigint, is_primary boolean, max_slot integer, cluster_name text ); Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

21 Preparing for part: get hash raw CREATE FUNCTION partconf.get_hash_raw ( i_input integer) RETURNS integer LANGUAGE sql AS $$ -- used to wrap hashtext so that we can replace it in with older implementation to keep compatibility select hash_string(\$1::text, lookup2 ); $$; Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

22 Preparing for part: local part > psql rootdb < partconf.sql > psql sharddb_0 < partconf.sql > psql sharddb_1 < partconf.sql > psql sharddb_0 => insert into partconf.conf(part_nr, max_part) values (0,1); > psql sharddb_1 => insert into partconf.conf(part_nr, max_part) values (1,1); Dimitri Fontaine Skytools3 July, / 27

23 Preparing for part: add-table > londiste3 st3partsplit/st3_rootdb.ini add-table pgbench_accounts --handler=part --handler-arg=key=aid > londiste3 st3partsplit/st3_sharddb_0.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid > londiste3 st3partsplit/st3_sharddb_1.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid Dimitri Fontaine Skytools3 July, / 27

24 Other Handlers X applyfn X bulk X dispatch X multimaster X part X qtable X vtable Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

25 Handler: bulk bulk loading with 3 options X correct: COPY, COPY temp + UPDATE, COPY temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

26 Handler: dispatch bulk monthly batch X bulk X hourly daily monthly yearly X event batch field time Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

27 Handler: dispatch X table mode part, direct, ignore X part mode batch time, event, time, date field, current time X part field date field X period hour, day, month, year X row mode plain, keep latest, keep all X event types I,U,D X load mode direct, bulk X method correct, delete, merged, insert X fields field name mapping, no COPY support X skip fields X table X pre part, post part X encoding X analyze Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

28 Extras X check X fkeys X compare X repair X wait-sync X wait-provider X wait-root Dimitri Fontaine dimitri@2ndquadrant.fr Skytools3 July, / 27

29 Questions? Now is the time to ask! Dimitri Fontaine Skytools3 July, / 27