database reliability engineering what. why. how. Percona Live, Dublin, 2017 Laine Campbell, Sr. Dir, Production Engineering, Fastly

Transcription

1 database reliability engineering what. why. how. Percona Live, Dublin, 2017 Laine Campbell, Sr. Dir, Production Engineering, Fastly

2 we made a book!

3 yesterday s DBA gatekeeper master builder superhero siloed specialized 3

4 today s DBRE educator and mentor platform builder force multiplier operations expert t-shaped 4

5 database reliability engineering develop reliable and resilient datastores and shared components for provisioning Provide patterns and knowledge to support other team s processes to facilitate their work understand and teach data access and storage nuances to ensure all service level objectives can be met anchor teams with expertise for troubleshooting, recovery and other tasks requiring depth, not breadth 5

6 why DBRE? reliability is only a smidgen of the operational mandate reliability isn t always required in operations the closer to the data, the more reliability is necessary data requires paranoia, not chaos - 6

7 the path to DBRE Who's your DBA? If you don t know the answer, it s probably you. the DBA, continuing to evolve their skillset the ops engineer, taking data ownership software engineers developing data driven applications 7

8 paradigm shifts 8

9 polyglot persistence relational is not the end of the line new and shiny brings risk One engine is less likely to suit all cases we vet, find edge cases, learn patterns Integration between engines is crucial The data must flow 9

10 virtualization and cloud commodity hardware for DBs forces designing for resilience new storage paradigms the power of rapid provisioning DBaaS availability reduces toil 10

11 infrastructure as code forces componentization no special snowflakes requires coding chops changes become deployments Infrastructure can be versioned 11

12 continuous delivery we must be teachers, not gatekeepers testing and compliance become top priorities we build patterns to replace reviews build guardrails and tools 12

13 database engineers: O.G. devops shared goals, tools and processes DEV OPS DBE 13

14 dbre guiding principles protect the data self-service for scale elimination of toil databases are not special snowflakes reduce the barriers between software and operations 14

15 protect the data responsibility of data protection shared by cross-functional teams. standardization and automation with simplicity over expensive/complicated infrastructures. durability and integrity baked into every part of the architecture and software development lifecycle 15

16 self service for scale metrics knowledge base and automated discovery/collection backup/recovery utilities and APIs auto-deployed for new builds reference architectures and configs for data stores deployment. security standards for data store deployments. safe deployment patterns and tests for database changesets 16

17 dbs are not special snowflakes moving databases into the cattle, not pets paradigm a la Bill Baker databases are the last holdouts of commoditization keep it simple stupid 17

18 reduce barriers between teams Use the same code review and deploy processes Use the same provisioning and config management Data-land should feel familiar and intuitive, not alienating or special 18

19 dbre core competencies 19

20 datastore decision making choose boring tech if possible testing consistency guarantees benchmarking performance building gold configurations testing operability 20

21 Spend innovation tokens only on key differentiators If your company is very young, optimize for velocity and developer productivity. The more mature your company is, the more operational impact over the long term trumps all. 21

22 recoverability build an effective tiered backup solutions - the right backup for the right dataset work with SWE to build data validation pipelines integrate recovery into daily activities build recovery testing automation 22

23 23

24 data distribution determine effective number of copies distribute broadly based on requirements replication, partitioning and sharding 24

25 failover and availability understand availability strategies for each datastore Evaluate limitations and consequences of failover (cold caches, data loss etc ) practice and document failover in controlled circumstances continued training of all staff on the process to get it in muscle memory 25

26 scaling paths plan for and test up to 10x growth keep an eye for constraints that will impact the future (aka don t screw your future self) 26

27 build guard rails backups, restores. verified. unit tests migration and fallback testing DDL migration patterns migration heuristic analysis boring failovers shared on call rotations network isolation between prod & other don t get pissy when people mess up, help them fix it don t swoop in and do it all yourself 27

28 automation rolling DDL scaling up or down failovers and traffic shifts detecting or killing bad queries 28

29 build for services observability instrumentation graceful degradation 29

30 build for people tools for debugging checklists, documentation tractable levels of graphs and alerting 30

31 release management develop patterns for migrations heuristics to surface risky changes migration and fallback testing tiered dataset sizes for testing (development, integration, full) continued developer training and education 31

32 32

33 self-actualized databases your data is safe and recoverable resilient to common errors understandable, debuggable shares processes and tooling with the rest of the stack empowers you to achieve your mission. 33

34 In conclusion... Don t be scared. Mind your damn backups. Everything else will probably be ok. 34