Upgrading Databases. without losing your data, your performance or your mind. Charity

Transcription

1 Upgrading Databases without losing your data, your performance or your mind Charity

2 Upgrading Databases without losing your data, your performance or your mind Charity

3 @mipsytipsy CTO at Hound Technology Formerly Facebook, Parse, Linden, etc Operations engineer. Accidental DBA.

4 Baby startup: (Has nothing whatsoever to do with this talk, because our data is mostly disposable)

5 Case study: Linden Lab * Mysql shop * Second Life, virtual worlds & gaming * Attempted upgrade to 5.0 in 2008, massive trauma * Stuck on mysql 4.1 until Embarrassing. Developed distributed capture/replay framework (Apiary)

6 Case study: Parse * MongoDB shop * Mobile backend as a service, over 1M apps * Needed to upgrade from 2.4 -> 2.6, 2.6 -> 3.0 * Outlier workloads & data sets * First customer for MongoDB + RocksDB storage engine Developed distributed capture/replay framework (Flashback)

7 How to upgrade your databases

8 without losing your data, your perf or your mind data integrity performance your mind

9

10 1. Should you upgrade? Why, or why not? 2. Risk assessment 3. How to perform hard upgrades

11 Why upgrade? New features Bugfixes Better performance (?) Better support from the vendor Avoid code rot, don t get too far behind current

12

13 A Very Short List Of Terrible Things Database Upgrades Have Done To Me 35% perf reduction. *60%* perf reduction data corruption (so many flavors) db files deleted on startup. DELETED!! indexing race conditions invalid indexes bug causes collections to be unwritable undocumented change in geoquery behavior election deleted ALL THE DATA default storage format has 60% more bloat backwards-incompatible mysql replication storage format changes all geo indexes block global lock until the first document found when row-based replication was launched and it doubled the size of the bin log password hash format changed

14 I m not done yet all geo indexes seeks take the global lock until the first document found undocumented query syntax changes changed the definition of scan limits, doesn t cache query plans that exceed scan limit unindexable writes suddenly refused internally-assigned data type changes secondaries crash instead of pausing replication query planner fails to cache plans when race phase interrupted query planner caches plans for least data not representative data accepted a bad op in the primary which bricked secondaries preventing quorum

15 The cowboy continuum whoa there yee haw!

16 Risk assessment: organizational How mature is the db (or engine)? How critical is the data? How mature is your company?

17 Risk assessment: technical Can you roll back? How hard will it be? How unusual is your workload? How much changed between releases?

18 Why not upgrade? It s very green, and you don t have the cycles to help harden it You don t need the new bug fixes or features Client interaction problems (You prob shouldn t fall more than 2 versions behind tho)

19 yolo # apt-get upgrade nothing can ever change let s just use oracle

20 Redis Cassandra MySQL MongoDB

21 MongoDB 2.6 risk assessment for Parse: How mature is the db? NOT How critical is the data? TERRIBLY How mature is your company? FAIRLY Can you roll back? How hard will it be? DEPENDS How much does your workload push the boundaries of the db? EXTREMELY Are other people doing similar workloads? LOLNO How much changed between releases? A LOT

22 Paranoid upgrades

23 The Minimal Set: Read the release notes Run unit tests Assess your appetite for risk

24 Test real production workloads.

25 YOUR query set YOUR data set with YOUR hardware and YOUR concurrency

26 Golden rule of benchmarking: Change ONE thing at a time. hardware db version data set concurrency client version etc

27 Correctness Performance

28 Correctness Unit tests Replay queries and compare results against two primaries (e.g. pt-upgrade) Production traffic splitters to shadow dbs

29 A few correctness questions: Are you writing and returning the same data from version to version? Are there ordering differences? Are the client versions serializing and deserializing the same way? Are the error/success codes the same? Are the syntaxes identical?

30 Splitters & shadows

31 Performance run your own benchmarks.

32 How predictable is your workload?

33 VERY Predictable! sysbench linkbench artificial data these things may be enough for you!

34 How predictable is your workload? not very, or i don t know == NO

35 Capture/Replay: the basics 1. Take a consistent snapshot 2. Capture raw requests 3. Reconstitute requests into ops or transactions 4. Replay against offline snapshot 5. Reset snapshot, flush caches, tweak & repeat

36 Snapshot your data EBS snapshots are fine, but prewarm the blocks LVM snapshots are better Bring it up offline somewhere

37 Start capturing requests Grab at least a day s worth Tcpdump/pcap, or full query logs Reconstitute raw logs into transactions or ops

38 Replaying Adjust # of workers to simulate concurrency As fast as possible (measures throughput) Respecting original timestamps (measures latency)

39 Reset the caches Evict working set between runs LVM snapshot reset echo 3 >/proc/sys/vm/drop_caches)

40 Generating results Break down results by op type and percentile and by query family Look for outliers and MAX

41 Pro tip: ** Make sure you re stressing the performance of your db, not your replay apparatus.**

42 Monitoring Monitor your test dbs *just* like your production dbs Run a control test that mirrors prod Monitor & alert on system metrics for your replay clients

43 P.S. don t forget your clients

44 Confidence

45 I upgraded and got 35% worse performance I upgraded and 30% of my writes started getting rejected bc mongo started enforcing index key lengths I upgraded and I m getting corrupt data due to indexing race conditions Really Bad Problems

46 Much Better Problems I upgraded and.01% of my apps started ordering slightly differently for certain find queries I upgraded and one of my offline DW jobs had an incorrect implicit data type I upgraded and had to adjust to a slightly different administrative workflow

47 We re not going for perfection here. This is data, there will Always Be Something Wrong

48 data integrity performance your sanity

49 Capture/Replay Resources MongoDB: MongoDB flashback tools: Mysql: Apiary: Blog post: Paranoid-Mysql-Upgrade/ba-p/ Percona Playback:

50 Charity