Data Integrity in Stateful Services. Percona Live, Santa Clara, 2017

Transcription

1 Data Integrity in Stateful Services Percona Live, Santa Clara, 2017

2 Data Integrity Bringing Sexy Back

3 Protect the Data. -Every DBA who doesn t want to be fired

4 Breaking Integrity Down Physical Integrity - Help, my data files are gone! Logical Integrity - Help, my s disappeared!

5 Data Integrity is the mission of the entire organization.

6 When to plan for integrity?

7 If you are planning for recovery after your application and infrastructure is built You re guarding a henhouse that already has a fox in it.

8 Goals for Integrity Elimination Empowerment Detection Flexibility

9 Elimination Where possible, eliminate the potential for corruption and data loss. Optimize for durability based on your user needs Understand Isolation Levels Consistency vs Availability Velocity Levers

10 Elimination Case Study: Jepsen and Galera Impacts and Ambiguities of Isolation Levels Dirty Read : Prohibited by RC, RR, 1SR Fuzzy (non-repeatable) Read: Prohibited by RR, 1SR Phantom: Prohibited by 1SR

11 Elimination Snapshot Isolation in Galera Prevents Dirty Writes, Dirty Reads, and Lost Updates, and some types of Phantoms As of 2015, Galera Documentation claimed snapshot isolation. Jepsen tests proved this was not true, First Committer Wins was disabled due to performance reasons. Galera s recommendation of using leader only does not take into account data loss during leader transitions

12 Empowerment Help people and systems to recover rapidly from their own mistakes. don t trust destructive requests soft deletes w/recovery API data versioning

13 Detection Early detection of corruption is as important as the ability to recover from it. unit and regression testing data validation pipelines tools for investigation

14 Detection Case Study: Data Validation Pipelines Each DB Constraint should have validation tests running constantly on sample datasets to ensure that they are being enforced. Constraints can be disabled. Applications should have expected isolation levels. Running transactions regularly to ensure expected behaviors should be done. Databases without constraints, particularly federated ones require more rigorous tests to be run.

15 Flexibility You cannot predict all of the ways you can lose data. Focus on flexibility in your toolbox. Tiered Storage Replication and Data Portability

16 What could go wrong? Famous Last Words

17 Planned Recovery Production deployments Environment duplication Downstream services (analytics, compliance) Operational tests

18 Unplanned Recovery Google estimates 24 combinations of data integrity failures possible Scope Category Impact

19 Scenario Scope Small: Localized or single instance in redundant scenarios. Small subset of data (1000 customers) Medium: Cluster-Wide or a full Zone A full dataset (all customers in a shard) Large: Multiple clusters, or a full DC Multiple datasets (full data loss, all customers across shards)

20 Scenario Impact Small: Some features impacted, non-slo threatening. Small subset of users impacted. Medium: SLO threatening. Moderate subset of users impacted. Large: SLO impacting, application down. Majority of users impacted.

21 Operator Error Data Deletion Data Corruption Relaxed Constraints Storage removal

22 Application Errors Removing pointers to assets in external storage Character set mutilation Duplication of data

23 Infrastructure Services Orchestration got frisky? Configuration management change some durability parameters? Proxies or DNS points to the wrong node?

24 OS and Hardware Errors Silent corruption due to failed ECC error checks? Filesystem corruption Data loss during a power down

25 Hardware Failures Disk Failures Memory Failures Controller Failures

26 Datacenter Failures Catastrophic power loss Wiped out storage Fires, catastrophic events

27 Anatomy of a Recovery Strategy

28 Building Block 1 Early Detection, Bad Data Propagates A culture of unit and regression testing Data validation test suite Example: Storing external media Tools and analytics to investigate errors

29 Building Block 2 Tiered Storage Fast, expensive storage for dataset portability Slow, inexpensive storage for long-term backups Long-term storage (tape, offsite) Object storage for versioning Distributed logs (i.e. Kafka) for versioning and events

30 Building Block 3 Toolbox Full and incremental online backups Full and incremental offline/long-term backups APIs for soft deletion/undeletion APIs for version rollback/play forward Producers for event streams to recreate objects

31 Building Block 4 Testing Daily use as testing - incorporate recovery into daily work Continuous testing of less-used recovery methods Regular game days - team scenario testing

32 Final Considerations

33 Data Integrity is cultural Design, Build, Test, Deploy - each stage is an opportunity to think about data integrity Checks and balances between teams keeps us honest and focused on the goal Data becomes too complex for one person or team to understand it. We must help each other.

34 Data Integrity Must be Continuous These processes are crucial and cannot be allowed to gather dust. Humans will not do this on their own, integration and automation is required. This must be put into project functional requirements.

35 You cannot plan for everything New and interesting things will occur that will challenge your plans. Flexibility and multiple options must be made available. Early detection is crucial for ensuring problems do not propagate out of control.

36 Good

37 Check out our book Laine Campbell and Charity Majors