Consistency Without Transactions Global Family Tree

Transcription

1 Consistency Without Transactions Global Family Tree NoSQL Matters Cologne Spring by Intellectual Reserve, Inc. All rights reserved. 1

2 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra 2

3 What is FamilySearch? Familysearch.org website Largest single pedigree (Family Tree) Largest collection of free genealogical records Largest genealogical library Family History Department of Church of Jesus Christ of Latter-day Saints (known as Mormons) 3

4 Why does FamilySearch exist? Visit 4

5 Record Preservation Neglect Time Disasters (e.g. WWII) 5

6 Record Preservation (continued) 222 cameras in 48 countries Each camera captures images per hour 6

7 Record Preservation (continued) 150 million images published online 7

8 Indexing 1 billion indexed records about 1M per day! Turns this into this! 8

9 Memories 9

10 Community 10

11 Family Tree & Records Community Records Indexing Family Tree Memories 11

12 Family Tree Data Family Tree: 900M+ person records, open-edit 500M+ relationships, open-edit 8.4B change log entries, 100M+ per quarter Dynamic OLTP system Data-dependent performance issues 12

13 Family Tree: Example 9 Gen Pedigree up to 511 person slots Dynamic content! 13

14 Family Tree: Example Pedigree App 31+ persons per section Dynamic content! 14

15 Family Tree: Example Ancestor Page 10+ persons in families changes Dynamic content! 15

16 Family Tree: Example Change History changes Dynamic content! 16

17 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra 17

18 Motivations Performance (worse as data has grown) Scale (wider audience) Cost (license & hardware) Cloud (infrastructure moving to cloud) 18

19 Performance & Scale Slow page views pedigree ( ms for 3 generations) change history (2000+ms for first page of changes) large family view Query problems relationships connect persons, range scan by person id every person => person traversal is M btree scan (global index) change history queries travers 8+B btree scan (global index) 19

20 Performance & Scale Query performance problems Wide range scan Pedigree Person Relati onship Person Wide range scan Change History Change History 20

21 Performance & Scale Performance degradation over time more data means wider queries, even for simple records no ability to scale data tier further without prohibitive cost data has resisted sharding, M connected graph Caching wasn t enough cache invalidation was almost as expensive as request immediate read-after-write response required 21

22 Scale & Cost Database bottlenecks even after tuning queries even after getting relationship queries down ~5-10ms database CPU headroom ramping traffic Huge DB box already need to be able to scale data tier directly without huge additional hardware or DB license cost 22

23 Cloud Transition Switching app & database to cloud wide load variation in a week ramp app on-demand db needs to be in the cloud too tolerance for zone / region outages 23

24 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra 24

25 Cassandra Reimplementation selected Cassandra after extensive testing full data scale proof-of-concept & tests required: new data model (performance) required: new consistency model (critical!) 25

26 Cassandra Reimplementation event-sourced data model journal / views new data model no indexes new consistency model satisfies consistency P1 JE #8 P1 Views A B P2 JE #6 P2 Views A B 26

27 Cassandra Reimplementation denormalized relationships R1 R4 P1 R2 R3 P2 R5 27

28 Cassandra Reimplementation denormalized relationships R1 R4 P1 R2 R3 R2 R3 P2 R5 28

29 Cassandra Reimplementation denormalized relationships R1 R4 P1 R2 R3 R2 R3 P2 R5 29

30 Cassandra Reimplementation denormalized relationships exact duplication allows biderectional traversal P1 R1 R2 R3 R2 R3 R4 P2 Person Relatio nship Wide query Person R5 Person /Rels Person /Rels 30

31 Cassandra Reimplementation change history is a core feature denormalized change history optimizes for displaying recent changes P1 JE #8 P1 Change History View Last changes (local to a single Cassandra cell) 1000s of changes (spread over multiple Cassandra cells) 31

32 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra 32

33 Journal-based Consistency Model Rough Data Flow Command Journal View View View captures edits safely stores edits canonically view-optimized summations 33

34 Journal-based Consistency Model Command Journal View View View Command write-once with LOCAL_QUORUM application to journal requires 3 tables: pending / completed / aborted idempotent application to journal 34

35 Journal-based Consistency Model Command Journal View View View Command Schema key: command v1 uuid (as text) value: blob (binary json) 35

36 Journal-based Consistency Model Command Journal View View View Journal write-once with LOCAL_QUORUM & C* batch denormalized byte-exact across affected entities each entry stored in separate cell (compaction required for fast journal reads) 36

37 Journal-based Consistency Model Command Journal View View View Journal CmRDT (commutative replicated type) partitions converge without conflict because of unique uuid 37

38 Journal-based Consistency Model Command Journal View View View Partition Key Command UUID Content (blob) KWZ3-P71 KWZ3-P71 eda6f af8d90c-8f3a { "attribution": {}, } (binary json) { "attribution": {}, } (binary json) KCDT-J59 fd35ac61-7def { "attribution": {}, } (binary json) KCDT-J59 b2db2fa5-da5f { "attribution": {}, } (binary json) 38

39 Journal-based Consistency Model Command Journal View View View View multiple views for multiple uses (person, person card, change history) populated by applying journal entries incrementally updated in steady state not canonical data, can be recalculated 39

40 Journal-based Consistency Model Command Journal View View View View CvRDT (convergent replicated type) partitions converge with conflict; resolved by full view refresh from canonical journal steady state: one view of a given type per entity 40

41 Journal-based Consistency Model Command Journal View View View P1 P1 Views A B 41

42 Journal-based Consistency Model Command Journal View View View P1 JE #8 P1 Views JE #8 JE #8 A B 42

43 Journal-based Consistency Model Command Journal View View View P1 JE #8 P1 Views A (new) B (new) JE #8 JE #8 A B 43

44 Journal-based Consistency Model Command Journal View View View View same schema as journal enables journal entries to be written to view for incremental refresh core of the consistency model 44

45 Journal-based Consistency Model Constraints needs to meet scale & performance requirements needs to enable strong consistency needs to support change history natively needs to be easily implementable on key-value stores needs to be tolerant of outages (node, zone, region) 45

46 Journal-based Consistency Model Performance & Scale lookup by partition key only, no indexes any cross-entity change happens in duplicate on all stored current-state views cheapest possible read custom views tunable to different use cases disposable views able to tweak view over time 46

47 Journal-based Consistency Model Strong consistency command store atomic capture of a single command command handling idempotent writes to journal, picked up later even if interrupted no global lock needed for optimistic concurrency 47

48 Journal-based Consistency Model Business Rule Enforcement pre-command checks with LOCAL_QUORUM reads prevent invalid changes write with LOCAL_QUORUM ensures consistent write post-command checks with LOCAL_QUORUM reads prevent business-rules conflicts administrative revert marks command as not applicable and thereby causes full refresh which ignores changes 48

49 Journal-based Consistency Model Journal / View Concerns native support for change history no journal tombstones in steady state write-once simple blob schema able to be implemented on any db engine that supports two-level keys (partition, composite) view tombstones occur on every write, but not likely to live long for any single person, because of write patterns 49

50 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra 50

51 Experience with Cassandra tested Community 1.2 and 2.0 fantastic performance easy cloud setup great developer response easy to bulk load through CQL3 harder to get running inside AWS VPC 51

52 Experience with Cassandra Bulk import experience 8.4B change log records => 5.8B journal entries (2.5TB lzo) 5 node hi1.4xlarge cluster (2TB SSDs) 11 hours to import through CQL (5h on 30-node cluster) 140k writes / sec, fed from 128 writer threads 20 records / unlogged batch write, 1-2k record size minimal post-import compaction (size-tiered) ended up with 3.5-4TB on C* disk after import OpsCenter great visibility for tuning Community harder to automate repairs, etc. 52

53 Experience with Cassandra Full-scale load test experience got to 25x our production peak load on node cluster production peak load included significant write load working-set size was about 2M persons in a month enabled row cache, ran almost entirely without disk access bottlenecked on interconnect socket round robin client still net bandwidth to spare, trying token-aware next OpsCenter great visibility for tuning Large SSD cluster able to handle repair during scale tests 53

54 Experience with Cassandra current system cassandra impl (1x, 10x, 20x) 54

55 Experience with Cassandra LOG SCALE! current system cassandra impl (1x, 10x, 20x) 55

56 Current Status still working on implementation & rollout migration, reconciliation, integration consistency model code 56

57 Contents Introduction to FamilySearch Family Tree Motivation for reimplementation Outline of Cassandra reimplementation Journal-based Consistency Model Experience with Cassandra Questions? 57

58 Contact Info John Sumsion Sr. Software Engineer Thanks to team at FamilySearch: - RandyB, James, Arn, Jason, Michael, RandallJ, JonM, Louis, JohnK, Tyler, Atsuko, Spencer, Dan Thanks to the awesome presenters & organizers here! 58