t w readify.net Managing database schemas in a Continuous Delivery world

Transcription

1 t w readify.net Managing database schemas in a Cntinuus Delivery wrld January 2015

2 Executive Summary One f the trickier technical prblems t address when mving t a cntinuus delivery develpment mdel is managing database schema changes. Unlike sftware changes, it s hard t rll back r rll frward when database changes have been applied. Typically, rganisatins address this prblem by having DBAs manually apply changes s they can manually crrect any prblems, but this has the dwnside f prviding a bttleneck t deplying changes t prductin and als intrduces human errr as a factr. A large part f cntinuus delivery invlves the setup f a largely autmated deplyment pipeline that increases cnfidence and reduces risk by ensuring that sftware changes are deplyed cnsistently t each envirnment (e.g. dev, test, prd). T fit in with that mdel it s imprtant t autmate database changes s that they are applied autmatically and cnsistently t each envirnment thus increasing the likelihd f prblems being fund early and reducing the risk f database changes. This reprt utlines an apprach t managing database schema changes that is cmpatible with a cntinuus delivery develpment mdel, the reasns why the apprach is imprtant regardless f develpment mdel and sme f the cnsideratins that need t be made when taking this apprach. The appraches discussed in this dcument aren t specific t cntinuus delivery and in fact shuld be cnsidered regardless f yur develpment mdel. 2 / Cpyright 2014 by Readify Pty Ltd ABN

3 The prblem with manual schema management Managing database schemas t be in sync with sftware develpment is traditinally a cmplex task fraught with prblems such as: Tedius manual prcesses Human errr Applying the wrng scripts Applying scripts in the wrng rder Frgetting t apply scripts Applying scripts multiple times Cnfusin ver the current versin f the database in particular with regards t different envirnments (e.g. dev vs test vs prductin) Inefficiencies fr the develpment f sftware when using lcal databases due t develpers being required t cnstantly mdify their database after pulling in changes frm ther develpers while the database schema is actively in develpment Smetimes develpers will instead use a shared develpment database t vercme this, but this is nt recmmended since there is additinal pain assciated with breaking changes and instability DBAs becming a bttleneck fr releasing changes t prductin (and wrse pre-prductin) and reducing the agility f teams and increasing deplyment risk (thrugh batching f changes) 3 / Cpyright 2014 by Readify Pty Ltd ABN

4 Database Migratins The recmmended apprach f this reprt t address the prblems f manual schema management and that is cmpatible with a Cntinuus Delivery develpment mdel is t use the database migratins technique. This technique invlves all schema changes being applied as a migratin, which is a file that describes that schema change (pssibly as SQL, pssibly in a Dmain Specific Language (DSL) via a prgramming language) and is cmmitted in surce cntrl alngside the sftware that uses the versin f the database that results frm applying that migratin. Migratins will typically have a timestamp r incrementing integer number assciated with them t designate the rder in which they shuld be applied. These migratins can then be autmatically applied t the database in any envirnment by a migratin tl/library. These tls will autmatically determine which migratins that are defined have and haven t been applied t the current database and nly apply the nes that haven t already been applied in the crrect rder. The migratins will typically be applied within a transactin s if the migratins fail then they will be rlled back. The advantages f this apprach are: Schema changes are made autmatically withut human interventin thus reducing the chance f human errr. Schema changes are cnsistently applied in each envirnment, s by the time the migratins are applied t prductin they wuld have already been successfully applied t dev, test and any ther pre-prductin envirnments. This significantly reduces the risk f schema changes Schema changes are kept in surce cntrl next t the surce cde that uses the schema that is defined at that versin. This means that any versin f the sftware can be checked ut and run and, assuming a fresh database was als used (befre any migratins, r at least befre the database versin f the cde that was checked ut), the state f the applicatin at that time can be tested against this can assist with finding bugs in specific deplyments. Yur database is nw versined (just like the sftware applicatin) and yu can inspect the versin f yur database in any envirnment and knw what changes have and haven t been applied. This is invaluable when lking fr envirnment-specific bugs. There is n cnfusin abut what scripts d/dn t need t be applied t a database at any time and scripts are never frgtten t be run they always get autmatically applied. Develpers can easily use separate lcal databases since any database schema changes they pull in frm ther develpers will autmatically be applied. Any schema change cnflicts will be expsed and thus reslved earlier (develpment time, r at least n the cntinuus integratin server if they are run as a test), which significantly imprves develpment efficiency and reduces risk. Schema changes will always be autmatically applied in the crrect rder. 4 / Cpyright 2014 by Readify Pty Ltd ABN

5 Database Migratins and Cntinuus Delivery Database Migratins can be integrated t a Cntinuus Delivery pipeline in ne f tw ways: Autmatically run n app start Requires the applicatin t have DDL permissins, which culd be a security risk Means it s pssible t d a deplyment that will be brken due t database migratins failing Simpler t set up Run as a deplyment step befre the deplyment f the applicatin sftware If the running f the database migratins fails, then fail the deplyment The ther cnsideratin is whether r nt the database migratins will result in a schema that is incmpatible with the applicatin cde. T avid this prblem, it s a gd idea t run autmated tests against the applicatin cde that talk t a migrated schema. This requires an islated database n the cntinuus integratin server that the tests can talk t, which can be achieved by: Run the database migratins nce nly fr the whle test run and use a transactin arund each test run that is then rlled back afterwards: This can be dne as a crss-cutting cncern withut mdifying test r applicatin cde by using smething like.net s TransactinScpe (in cmbinatin with the MSDTC service) This can als be dne by sharing a database cnnectin with an pen transactin between the different cmpnents (seeding, ding the wrk and verificatin) If yu are using an ORM then yu may want t create new sessin/cntext instances fr each f seeding/wrk/verificatin s that their internal cache desn t interfere with the test run and just ensure they each share the same database cnnectin Drp and recreate the entire database each run; pssibly by using in-memry versin f the database fr speed Delete any data frm the test run after/befre each run This can be dne manually in each test (but it s likely this will get missed ccasinally) This can als be dne using the technique described here (but yu need t be careful t nt delete seed data) 5 / Cpyright 2014 by Readify Pty Ltd ABN

6 Rllbacks Depending n the migratin tl being used, database migratins can include an up definitin (apply the migratin) and a dwn definitin (rllback the migratin). This then allws yu t mve a database back (r frward) t any specific versin. On first thught, this can be used in cmbinatin with sftware rllbacks if an errr is deplyed t prductin. Hwever, this is fraught with danger: If the crrespnding migratin that caused the break is mdified then every single envirnment that migratin had previusly been applied t als needs t be rlled back (that als includes all develper machines with that migratin applied). Dwn migratins ften result in data lss (by necessity, if the up migratin added a table the dwn migratin has t drp it) and that might be undesirable. If the dwn migratin fails as well then yu are in truble. Given dwn migratins are nt ften applied this is actually quite likely. It s cmplex t implement a prper rllback in mst Cntinuus Delivery pipeline tling because yu need t rllback the database versin (which has t be wrked ut smehw) in the cde that was just deplyed (since it will have the definitin f the migratins that need t be deplyed) and then rllback t the previus versin f the sftware. Yu may need t make this a manual prcess, r prvide a tl f sme srt t allw smene t g in and select a versin t rllback t (and then manually trigger a deplyment f the previus cde versin). Anther way t address the prblem is by dealing with breaking changes carefully (e.g. clumn renames, drpping tables, etc.) if yu can ensure that the schema changes are generally nt breaking changes (r the breaking change is deferred fr a deplyment r tw e.g. have the ld and new clumn side-by-side fr a while) then yu can safely perfrm rllbacks. It s imprtant t keep in mind that this als intrduces sme cmplexity arund keeping track f thse in-prgress changes, what has been deplyed where and managing technical debt (i.e. nt leaving the in-prgress changes behind). T avid a lt f these issues and take advantage f the fact that the Cntinuus Delivery pipeline allws the sftware team t quickly and safely get changes int prductin a strategy that is ften used is t never rll-back (sftware r database changes), but t always rll-frward (i.e. fix the bug, put that change thrugh the Cntinuus Delivery pipeline and apply t all envirnments). Regardless f whether a rllback r rll-frward is used, it s imprtant that nce the issue is fixed the team stps wrk and determines hw that bad change was able t get t prductin withut being picked up first. Then as part f the cntinuus imprvement fcus the team shuld make a change t the Cntinuus Delivery pipeline t ensure that particular prblem can never make its way t prductin again. This might invlve adding better seed data t a pre-prductin envirnment, adding autmated tests t pick up prblems with the migratins, r smething else. 6 / Cpyright 2014 by Readify Pty Ltd ABN

7 Sme migratin tls dn t bther with dwn migratins because f this cmplexity (DbUp is an example) and even if the tl yu are using des supprt them yu can chse nt t implement them. Dwn migratins can be handy fr debugging a specific versin in a pre-prductin envirnment r useful in develpment when a develper is making changes t migratins they are develping. If yu decide yu need dwn migratins then the recmmendatin wuld be t add an autmated test that takes a fully migrated database all the way back dwn and then back up that gives sme cnfidence that the dwn migratins are likely t wrk (but isn t perfect since smetimes prductin data vilates integrity cnstraints). 7 / Cpyright 2014 by Readify Pty Ltd ABN

8 Database Migratins are immutable Fr the reasns described in the Rllbacks sectin abve it s imprtant that in a Cntinuus Delivery envirnment Database Migratins are cnsidered immutable. This is because the master/trunk/mainline branch can be deplyed at any mment and as sn as yu change a migratin thse changes will nt get applied in any envirnment that already has that migratin versin nted as being applied. This ties int the granularity f migratin changes: Cntinuus Delivery requires that every separate set f schema changes that each develper makes are cnsidered a separate migratin If a prject invlves a lt f database changes then this can get ptentially unwieldy as the number f migratins grws and yu may decide t allw migratins t be changed while they are islated t develpment envirnments This is directly incmpatible with Cntinuus Delivery and can be hard t manage (i.e. ensuring that the migratins are dne befre mving past the develpment envirnments) If migratins are allwed t be changed while in develpment then the develpers need an easy t way t get their database t a clean slate and re-apply all migratins (e.g. a.bat file r cnsle applicatin r deleting the database versins and ensuring all migratin scripts can be safely re-run) This cmplicates the develpment prcess and can cause sme merging pain n the migratins (if there is a lt f churn n specific migratins), but it als results in less migratins fr database-heavy applicatins Smetimes it makes sense t try and rll-up migratins t a single file fr a release at sme defined pint alng the develpment cycle (e.g. each sprint), but this further cmplicates the release prcess (hw d yu crrelate the rlled up file with the previusly deplyed migratin numbers that are cntained in it) and shuld be avided unless abslutely necessary 8 / Cpyright 2014 by Readify Pty Ltd ABN

9 Other cnsideratins when using Database Migratins There are a number f things t watch ut fr when using Database Migratins: While database migratins mean that a DBA isn t needed t manually apply the changes that desn t mean there isn t a need fr DBA skillsets DBAs can still prvide advice t develpment teams arund the best way t mdel the schema fr their applicatin and can cllabrate with the develpers t create the migratins DBAs can be n hand t help with any prblems that d ccur when a deplyment is made the need fr this shuld reduce ver time as mre cnfidence is gained by the deplyment pipeline, but if this cncept is new then there is likely t be mre invlvement needed DBAs can be kept up t date with schema changes t prvide advice r raise any ptential issues by sending an autmated diff f the schema differences cntained in the migratins fr a particular deplyment as part f the deplyment pipeline (this is a relatively simple thing t write cde fr) Ideally, a new develper r a tester testing against an ad hc sftware revisin shuld be able t quickly and easily spin up a fresh cpy f a database befre any migratins have been applied This is easy fr an applicatin that has used migratins frm the start, but slightly mre difficult fr a legacy applicatin t which migratins have been added Seed data shuld be als be autmatically added; depending n the Migratin Tl being used, this can be achieved by a set f SQL statements, r smething mre prgrammatic like lading a CSV f data Testing data is slightly mre tricky since yu want t apply it t sme envirnments and nt thers If yu want t apply the testing data nce nly, n all envirnments apart frm a set f specific envirnments (e.g. prductin) then it makes sense t apply the data in a migratin that has guard clauses t prevent executin in thse envirnments If yu want the ability t apply the testing data in a mre ad hc fashin then yu shuld use sme ther means t apply the data e.g. a custm cnsle applicatin, r a set f SQL scripts that can be manually applied Ideally, all schema changes shuld be applied using migratins therwise the fllwing disadvantages apply: Nt all schema changes are autmatic leaving human errr as a prblem 9 / Cpyright 2014 by Readify Pty Ltd ABN

10 Nt all schema changes are cnsistently applied in all envirnments leaving a risk that sme f the manual changes r indeed the autmatic changes might nt wrk when applied t prductin There is n clear way t tell the cmplete versin f the database There is cnfusin abut which scripts have / haven t been run Lcal develpment databases are trickier t facilitate Nt all schema change cnflicts will be expsed early Schema changes might nt be applied in the crrect rder (e.g. if the autmatic migratins rely n the schema changes frm manual changes r vice versa) Stred prcedures and functins are sftware rather than schema changes and ften it is thus easier t manage them differently One way t d this is t keep them in surce cntrl and autmatically apply the latest versin t the database every time the schema migratins have been applied This ensures that the versin in surce cntrl is always the versin applied t the database Sme ORMs prvide the ability t autmatically change the schema t match the bject mdel defined in cde This is really handy at develpment time, particularly when spiking changes r wrking n a prf-f-cncept applicatin This is dangerus and risky fr a prductin applicatin and shuld be avided it can result in unpredictable schemas, pr perfrmance schemas and data lss If the database engine yu are targeting desn t allw transactins arund DDL statements then applying migratins is mre risky because a prblem that ccurs can leave the database in a semi-mdified state an example f this is Oracle database Techniques like blue-green deplyments are tricky t perfrm unless there is a way t have a ht cpy f the database that can als be switched at the same time (nt easy t achieve) Smetimes, migratins can fail in prductin because they time ut and are t slw t be applied (an example might be adding a new clumn with a default value t a table with millins f recrds) Getting prductin-like data earlier in the pipeline is imprtant in these scenaris t ensure that the prblem is picked up earlier Increasing the timeut f the migratins might be necessary, but if yu d that it s imprtant t ensure migratins are applied separately t the deplyment f the applicatin rather than alngside 10 / Cpyright 2014 by Readify Pty Ltd ABN

11 Separating ut slw changes t be applied when there is lwer lad n the database may help, but it cmplicates the release prcess Identifying the migratin mdificatins that are slw t run is imprtant s they can be identified early and sme srt f apprpriate actin taken (e.g. in the abve example, making the new clumn nullable and adding the default value in the applicatin cde instead may be an ptin) 11 / Cpyright 2014 by Readify Pty Ltd ABN

12 Database Migratins and.net applicatins There are three main migratin tls that we use at Readify fr applying migratins t.net applicatins: DbUp An Open Surce library that runs SQL scripts (up migratin nly) against SQL Server (including SQL CE and Azure SQL Database). FluentMigratr An Open Surce library that includes a DSL fr defining migratins (up and dwn) in cde against mst database engines. EntityFramewrk Migratins Part f the EntityFramewrk 6 library that includes a DSL fr defining migratins (up and dwn) in cde which can be autmatically generated frm yur mdel against mst database engines / Cpyright 2014 by Readify Pty Ltd ABN