Schema Management with Flyway

Transcription

1 Schema Management with Flyway Scott Mead Sr. Architect OpenSCG & BigSQL PGConf US

2 Strategies Single File Application Managed (ORM) Migration 2

3 Single File 3

4 Single File Easiest to implement Single SQL file holds full schema Single import creates entire database 4

5 Single File - Pros Easy to apply Easy to track (git, hg, etc..) Centrally managed (devops know exactly where to look) 5

6 Single File - Cons Large, Monolithic files 1000 s of lines of SQL Zabbix Monitoring 127 tables 308 indexes pg_dump -s zabbix wc -l = 6027 lines 6

7 Single File CREATE TABLE people ( id BIGSERIAL PRIMARY KEY, fname VARCHAR(25), lname VARCHAR(25) ); 7

8 Deploy $ psql -f schema.sql CREATE TABLE 8

9 Single File - Cons What about changes? Modify one table One developer updates the SQL How to apply this change? 9

10 Single File - Add column CREATE TABLE people ( id BIGSERIAL PRIMARY KEY, fname VARCHAR(25), lname VARCHAR(25), VARCHAR(25) ); 10

11 Deploy $ psql -f schema.sql psql:schema.sql:6: ERROR: relation "people" already exists 11

12 Deploy Changes Need a delta script to apply changes Could be done manually Reproducability? Write & Save a Delta script Simplicity lost Update DEV system Which version is it? What s the target version? What scripts do I need to update? 12

13 Application Managed (ORM) 13

14 Manual Application deploys schema at boot Custom coded by developers Harder to cluster Custom parser 14

15 ORM Object Relational Mapper Map application objects < > database code Typically generates DDL and DML Examples Hibernate SQLAlchemy Django (includes ORM) Active Record gorm etc... 15

16 ORM Example (SQLAlchemy) See Example (simple_orm.py) 16

17 ORM Example (SQLAlchemy) See Example (insert_orm.py) 17

18 ORM Run $ python insert_orm.py INFO sqlalchemy.engine.base.engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1 INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1 INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine PRAGMA table_info("people") INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine CREATE TABLE people ( id INTEGER NOT NULL, fname VARCHAR, lname VARCHAR, PRIMARY KEY (id) ) 18

19 ORM Run (cont)... INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine COMMIT INFO sqlalchemy.engine.base.engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1 INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1 INFO sqlalchemy.engine.base.engine () INFO sqlalchemy.engine.base.engine BEGIN (implicit) INFO sqlalchemy.engine.base.engine INSERT INTO people (fname, lname) VALUES (?,?) INFO sqlalchemy.engine.base.engine ('Scott', 'Mead') INFO sqlalchemy.engine.base.engine COMMIT 19

20 Migration 20

21 Migration Simple scripts Upgrade from one version to the next Upgrading 2 versions means running two scripts 21

22 Example Version 0 to Version 1 V1 example.sql CREATE TABLE people ( id BIGSERIAL PRIMARY KEY, fname VARCHAR(25), lname VARCHAR(25) ); 22

23 Version 2 - Desired State CREATE TABLE people ( id BIGSERIAL PRIMARY KEY, fname VARCHAR(25), lname VARCHAR(25), VARCHAR(25) ); 23

24 Version 2 - Migration Version 1 to Version 2 V2 example.sql ALTER TABLE people ADD COLUMN VARCHAR(25); 24

25 Version 3 - Desired State CREATE TABLE people ( id BIGSERIAL PRIMARY KEY, fullname TEXT ); 25

26 Version 3 - Migration V3 example.sql CREATE TABLE t_people ( id BIGSERIAL PRIMARY KEY, fullname TEXT ); INSERT INTO t_people SELECT id, fname ' ' lname from people; ALTER TABLE people RENAME TO people_old; ALTER TABLE t_people RENAME TO people; 26

27 Applying migrations Upgrades are simplified Flexible, run any command Must be run in order Over time, many scripts 27

28 Flyway 28

29 Flyway Schema management tool Designed to execute schema migrations Java-Based Apache V2 Licensed 29

30 Flyway Adds a schema_version table Applies migrations in order Tracks metadata in schema_version 30

31 Philosophy Examine current database version Examine availble migrations Apply migrations, in order 31

32 Commands Migrate Clean Info Validate Baseline Repair 32

33 Run Migrations Migrate Clean Apply necessary migrations Applies all available Applies to a specific version Drop / Re-create 33

34 Commands Info Give data from schema_version metadata Validate Migrations are sane 34

35 Migration 35

36 Rollback? Rollback scripts are rare Run them manually 36

37 Baseline Convert existing DB to flyway migration 1 Take a backup 2 flyway baseline 37

38 Other features Schema migrations can be java classes Allows for super-complex migration Serialize / Deserialize, web-service etc... Flyway in your app 38

39 Real Validation? flyway validate only ensures that *.sql files have not changed Gold Master Run migration Run pg_dump -s Load pg_dump into master DB diff after application pg_dump pg_quarrel apgdiff 39

40 Questions? 40