SATURN FamilySearch s Family Tree Web Application. Replacing Relational Database Technology and Transitioning to Cloud-Hosted Computing

Transcription

1 SATURN th Annual SEI Architecture Technology User Network Conference MAY 7 10, 2018 PLANO, TEXAS FamilySearch s Family Tree Web Application Replacing Relational Database Technology and Transitioning to Cloud-Hosted Computing Randy A. Ynchausti Software Architect ynchaustira@familysearch.org 1

2 Outline FamilySearch Web Site Family Tree Web Application Replacing RDBMS and Cloud-Hosted Computing Project - Scope and Motivation - Three Architecture and Design Challenges - Results Summary 2

3 FamilySearch Web Site 3

4 FamilySearch Web Site 4

5 FamilySearch Web Site Notable Statistics Statistic Value (Billion) Searchable Names in Historical Records 6.2 Historical Record Images Online 1.25 Published Indexed Records Per Year Web Page Views Per Day 11.2 This Photo by Unknown Author is licensed under CC BY-NC-ND 5

6 Family Tree Web Application 6

7 FamilySearch Web Site 7

8 FamilySearch Web Site 8

9 FamilySearch Web Site 9

10 FamilySearch Web Site 10

11 FamilySearch Web Site Notable Statistics Statistic Value (Billion) Total Tree Persons 1.18 Tree Persons Added Per Month 0.38 Total Sources Sources Added Per Month Sources Attached to Tree Persons 1.28 This Photo by Unknown Author is licensed under CC BY-NC-ND 11

12 Replacing Relational DBMS and Cloud-Hosted Computing Project 12

13 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation Given: Virtualized private (on premise) data center Tens of services and hundreds of nodes Hundreds of database server licenses Many development teams Continuous delivery Budget events 13

14 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation Affordability Consumption-Based (L, L) Monthly total cost is the operating cost of the resources used by the application (M, L) The resources the system needs to operate efficiently can change daily based on load projections and other factors We want: 1. Security 2. Availability 3. Scalability 4. Performance 5. Affordability Utility Performance Availability Capital Ownership Cost Transaction Throughput Data Latency Hardware Failure Service Failure (L, L) The operating budget is nine times or more higher than the capital budget of the system (L, M) The Family Tree database does not constrain the total number of transactions the system can process per hour (M, H) The data the system uses to draw a nine-generation pedigree is retrieved in one second or less (H, L) Power outage in an availability zone causes the system to redirect users to the services running in another availability zone within one minute or less from the outage event (H, M) Redundant services in an availability zone provide %99.99 service availability Data Confidentiality (H, H) Access-controlled data is secure % of the time Security Data Integrity (H, H) % of all data is accurate and consistent over its entire life cycle in the system Horizontal Approach (M, M) Adding additional resources allows the system to service a corresponding percentage increase in users and workload Scalability Elastic (M, L) Additional resources can be provisioned and deployed within 10 minutes or less (L, H) The system can automatically expand and contract its resources to accommodate fluctuating workloads. 14

15 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation 1 hr peak Max Capacity Linear (1 hr peak) 15

16 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation 16

17 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation 17

18 Replacing Relational DBMS and Cloud-Hosted Computing Project Project Scope and Motivation Project Scope and Motivation (Revision 2) Throttling Public API Web Client DMC Throttling Platform API Web Client Diagram 1 of Tree Data Tree Data Conclusion Tree Tree Foundation Publish-Subscribe Amazon SQS Key Process Communication / Data Flow Queue User / Consumer Service Application Context Batch Data Transfer Conclusion Tree Change Message Queue Message Queue Publish- Subscribe Admin Labels Postgres RDS Admin Labels Responsibility Division Relational Database N-Node NoSQL Database Extract, Transform, Load Private Data Center Amazon Cloud 18

19 Three Architecture and Design Challenges 19

20 Three Architecture and Design Challenges 1) Performance differences between the existing and target technology RDBMS Parallel query execution Parallel index build Replication aborting longer running queries NoSQL Operation and maintenance 20

21 Three Architecture and Design Challenges 2) Weak transactional semantics in the target NoSQL technology Patterns that helped us form a solution Event Source Commutative Replicated Data Type Command Query Resource Segregation This Photo by Unknown Author is licensed under CC BY 21

22 Three Architecture and Design Challenges 2) Weak transactional semantics in the target NoSQL technology (continued) 22

23 Three Architecture and Design Challenges 3) Deploying and operating application services using a cloud service provider platform Java Blueprint JavaScript End Users Domain Traffic Manager Developers System GitHub Elastic Load Balancer APICA Beanstalk Provisioning Service CloudFormation Service Janitor Blueprint Simple Systems Manager AWS Console Splunk xmatters AppDynamics Change Tracking JIRA Developers / Administrators 23

24 Results 24

25 Results Production cutover was about 16 months after project launch We achieved our schedule objective Complete RDBMS transition took about six additional months 25

26 Results Convert from R3-2XL to R4-4XL Instance Type 33 to 24 Nodes SSD to EBS Storage 36 to 27 Nodes DSE to DSE We achieved our scalability objective for the Tree database 27 to 33 Nodes Peak DB Transactions Capacity Linear (Peak DB Transactions) 26

27 Results 5000 Web Client-Facing Service Latency We achieved our performance objective for the Tree web client Latency (Milliseconds) /13/2016 3/27/2016 4/10/2016 4/24/2016 5/8/2016 5/22/2016 6/5/2016 6/19/2016 7/3/2016 7/17/2016 7/31/2016 8/14/2016 8/28/2016 9/11/2016 9/25/ /9/ /23/2016 7:00:00 PM 2:30:00 PM 12:00:00 PM 8:00:00 AM Sunday 27

28 Results We achieved our affordability, scalability, and security objectives via our cloud platform approach Statistic Average Code Check-Ins / Builds Per Day 250 Deploys to Production Per Days 161 Deploy Time in Minutes 10 Auto-scale Events Per Day

29 Summary 29

30 Summary The miracle occurred and it s working! This Photo by Unknown Author is licensed under CC BY-NC-SA 30

31 Thank you 31

32 SATURN th Annual SEI Architecture Technology User Network Conference MAY 7 10, 2018 PLANO, TEXAS FamilySearch s Family Tree Web Application Replacing Relational Database Technology and Transitioning to Cloud-Hosted Computing Randy A. Ynchausti Software Architect ynchaustira@familysearch.org 32