Charity

Transcription

1 Charity

2 Charity

3 @mipsytipsy Production Engineering FB Operations Engineer Accidental DBA Hates software (all of it)

4 Mobile Backend Powers >500k apps MongoDB, Cassandra, MySQL, Redis Acquired by Facebook in 2012 Ruby & Rails => golang

5 How to upgrade your databases

6 without losing your data, your perf or your mind data integrity performance your mind

7

8 1. Why upgrade? Why not? 2. Risk assessment 3. How to perform hard upgrades

9 Why upgrade? New features, bugfixes Better performance (?) Better support from the vendor Avoid code rot, don t get too far behind current All your cool friends have done it

10

11 35% perf reduction. *60%* perf reduction A Very Short List Of Terrible Things Database Upgrades Have Done To Me data corruption (so many flavors) db files deleted on startup. DELETED!! indexing race conditions invalid indexes bug causes collections to be unwritable undocumented change in geoquery behavior election deleted ALL THE DATA default storage format has 60% more bloat backwards-incompatible mysql replication storage format changes all geo indexes block global lock until the first document found when row-based replication was launched and it doubled the size of the bin log password hash format changed

12 all geo indexes seeks take the global lock until the first document found undocumented query syntax changes I m Not Done Yet changed the definition of scan limits, doesn t cache query plans that exceed scan limit unindexable writes suddenly refused internally-assigned data type changes secondaries crash instead of pausing replication query planner fails to cache plans when race phase interrupted query planner caches plans for least data not representative data accepted a bad op in the primary which bricked secondaries preventing quorum

13 The cowboy continuum whoa there yee haw!

14 Risk assessment: organizational How mature is the db? How critical is the data? How mature is your company?

15 Risk assessment: technical Can you roll back? How hard will it be? How unusual is your workload? How much changed between releases?

16 yolo # apt-get upgrade nothing can ever change let s just use oracle

17 Redis Cassandra MySQL MongoDB

18 MongoDB 2.6 risk assessment for Parse: How mature is the db? NOT How critical is the data? TERRIBLY How mature is your company? FAIRLY Can you roll back? How hard will it be? DEPENDS How much does your workload push the boundaries of the db? EXTREMELY Are other people doing similar workloads? LOLNO How much changed between releases? A LOT

19 Paranoid upgrades

20 The Minimal Set: Read the release notes Run unit tests Assess your appetite for risk

21 Real production traffic YOUR query set YOUR data set with YOUR hardware and YOUR concurrency

22 Correctness Performance

23 Correctness unit tests replay queries and compare results against two primaries (e.g. pt-upgrade) production traffic splitters to shadow dbs

24 A few small correctness questions: are you writing and returning the same data from version to version? are there ordering differences? are the client versions serializing and deserializing the same way? are the error/success codes the same? are the syntaxes identical?

25 Splitters & shadows

26 Performance run your own benchmarks.

27 How predictable is your workload?

28 How predictable is your workload? VERY! sysbench linkbench generate artificial data & use tools

29 How predictable is your workload? not very, or i don t know

30 Capture/Replay 1. take a consistent snapshot 2. sniff ~24 hours of requests 3. reconstitute requests into ops or transactions 4. replay reqs against offline snapshot 5. reset snapshot, flush caches, tweak & repeat

31 Replaying n concurrent workers pulling off a queue beware the GIL as fast as possible, or respecting timestamps both are valuable

32 Resetting evict working set between runs LVM snapshot reset echo 3 >/proc/sys/vm/drop_caches)

33 Generating results break down results by op type and percentile possibly also by query family look for outliers

34 Pro tip: ** Make sure you re stressing the performance of your db, not your replay apparatus.**

35 p.s. don t forget your clients

36 Confidence

37 I upgraded and got 35% worse performance I upgraded and 30% of my writes started getting rejected bc mongo started enforcing index key lengths I upgraded and I m getting corrupt data due to indexing race conditions BAD PROBLEMS

38 BETTER PROBLEMS I upgraded and.01% of my apps started ordering slightly differently for certain find queries I upgraded and one of my offline DW jobs had an incorrect implicit data type I upgraded and had to adjust to a slightly different administrative workflow

39 We re not going for perfection here. This is data, there will Always Be Something Wrong

40 data integrity performance your sanity

41 Capture/Replay MongoDB: MongoDB flashback tools: Mysql: Apiary (deprecated): Percona Playback:

42 Charity