Cloud Architecture Patterns. Running PostgreSQL at Scale (when RDS will not do what you need) Corey Huinker Corlogic Consulting December 2018

Transcription

1 Cloud Architecture Patterns Running PostgreSQL at Scale (when RDS will not do what you need) Corey Huinker Corlogic Consulting December 2018

2 First, we need a problem to solve.

3 This is You

4 You Get An Idea For a Product

5 You make a product!...now you have to sell it.

6 To advertise the product, you need an ad......so you talk to an ad agency.

7 But placing ads has challenges Need to find websites with visitors who: Would want to buy your product Are able to buy your product Would like the style of your advertisement

8 A Websites' Claims about their Visitors...

9 ...are not always accurate.

10 Buying ad-space on websites directly is usually not possible. You must use an auction service.

11 How Modern Ad Tracking Is Done Each advertisement is wrapped in a JavaScript program The program starts program starts when the web page loads The program sends a message every ~5 seconds until the page closes The program also sends messages when important events happen Is the advertisement in a space that fits the size of the image? Is the advertisement in a part of the screen that is visible to the user? Did the mouse pass over the ad? Did the video begin to play? Is the audio muted? Did the video finish? These messages are collected by a "pixel server" and combined to construct a timeline of the life of the advertisement on that web page

12 Now your monitored ad reports events.

13 Focal points of ad monitoring We want to know the number of times: How many times did the ad land on a page? ("Impressions") How many times the ad landed in a favorable spot on the page? How many times the ad fit into the space allotted? How many times the ad was visible for 10 seconds? 20? 30? Did the viewer interact with the ad in any way?

14 We want to know: The Real Purpose of ad monitoring How many of our ads were seen by actual humans? ("Engagement") How many of our ads were seen by NHT - Non-Human Traffic? ("bots") How do these numbers compare with the claims of the website? How do these numbers compare with the claims of the auction service? Ultimately, we want to know how much of our money was wasted, so we can change where we spend money in the future.

15 Ad Tracking creates a lot of data Not all impressions report their events Default rate is about 3% Full reporting would require 30x the infrastructure for only a small gain in accuracy Customers can pay for a higher sampling rate Approximate sampling events recorded per day: 50,000,000,000 Full reporting would be 1.5T events per day. Sampling events are chained together to tell the story of that impression. Impression data is then aggregated by date, ad campaign, browser type After aggregation, we have about 500M rows per day. Each row has > 125 measures of viewability metrics

16 Tagged Ads On Web Browsers The Original Architecture (2013) Viewable Events (Billions/day) Pixel Servers Log shipping Stats Cache Aggregators MySQL OLTP ( ) MySQL DW DBs ( ) Redshift ( ) Vertica ( ) Daily ETLs Daily Summary Files 100s of files (~100 GB total size) csv.gz files on S3 One per customer per day

17 User Query (2013) Website Request Stats-Cache API Partial Query Partial Query Vertica ( ) Stats Cache Aggregators Searches MySQL OLTP DB Partial Queries MySQL DBs - Shard By Date Redshift ( ) CSV.gz files not accessed directly by queries Partial Query Results combined at the application level 3 dialects of SQL 1 custom API

18 Capturing Ad Activity Events "Pixel" server: Website that only serves up one 1x1 pixel image Captures data about visiting browsers in web logs Needs to be fast to not delay user experience or risk losing event data Data must be read quickly to give customers real time results. Tagged Ads On Web Browsers Viewable Events (Billions/day) Number of Servers: ~500 Probably too many servers Over-provisioned for reliability EC2 Type: t3.xlarge or similar Low CPU workload Low disk I/O workload High network bandwidth low latency Pixel Servers

19 Real-time Event Accumulation and Aggregation Stats Cache machines consume syslogs from Pixel Servers Log Events from the same browser are combined to form the ad outcome. Outcomes are aggregated by ad campaign, product brand, etc. (10 different aggregations) Each Stats Cache is an incomplete shard of today's data At end of day, all shard data is combined into one CSV per customer Pixel Servers Log shipping Number of Servers: ~450 Custom in-memory DB No disk storage of data CPU load: nearly 100% Cannot use swap EC2 type: r5.2xlarge or similar Stats Cache Daily Summary Files 100s of files (~100 GB total size) S3 - CSVs

20 Real-time Stats Reporting (2013) API Request Stats Cache Shards Aggregation in Application Code Aggregating in application code high memory usage high network usage high potential for error API Response

21 Non-Real-time Data Reporting (2013) API Request MySQL Shards (by date) Aggregation in Application Code Redshift (for very large clients) API Response MySQL OLTP DB

22 First Steps to Fix MySQL OLTP DB Converted to PostgreSQL Logical Replication not yet available Conversion took 2-4 weeks using 2 programmers Added Triggers on MySQL tables to identify modified rows Used mysql_fdw to create migration tables on PostgreSQL Created each new PostgreSQL table as SELECT * FROM foreign table Scheduled tasks update PostgreSQL by reading new records in trigger tables Moved read-only workloads to postgres instance Migrated read-write apps in stages Only downtime was in final cut-over Final system: Single 32 core EC2 master with 1-2 physical read replicas

23 Next To Fix: MySQL Data Warehouse shards Performed adequately when daily volume was < 1% of current volume Impossible to add new columns to tables Easier to create a new shard than to modify an existing one. New metrics being added every few weeks or days (over 100 metrics) Dozens of shards, some cover a month of data, others only a few days Each new shard adds workload to application level aggregator

24 Understanding User Interest In Data Yesterday User Interest In Data Today's Real-time Data Age Of Data 8-30 Days Ago Older 2-7 Days Ago 85% of API requests are for data <= 7 days old This follows Zipf's Law: Conclusions: put newest data on fastest servers move older data onto fewer, slower servers

25 Postgres For The Most Needed Data Vanilla PostgreSQL instance 9.4 i3.8xlarge or similar: 32 cores, 240GB RAM, 5TB disk Data partitioned by day Drop any partitions > 10 days old. All data is copy of data in S3. No need for backups. Focus on loading the data as quickly as possible (< 2 hours) Smaller customer's data available earlier. Adjust application logic to make this data visible earlier. Codename: L7 CSV files on S3 One Per Customer+Date Daily ETL L7 DB

26 What Didn't Work: Redshift Intended to compliment MySQL Performed adequately when daily volume was < 1% of current volume Needed sub-second response, was getting 30s+ response Was the only machine that had a copy of data across all time HDD was slow, tried SSD instances, but had limited space Eventually grew to a 26 node cluster with 32 cores per node. Could not distinguish a large query from a small one Had no insight into how the data was partitioned Reorganizing data according to AWS suggestions would have resulted in vacuums taking several days.

27 What Didn't Work: Vertica Intended to compliment MySQL Good response times over larger data volumes Needed local disk to perform adequately, which limited disk size each cluster could only hold a few months of data 5 node clusters, 32 cores each. Could only have K-safety of 1, or else load took too long (2 hrs vs 10) Nodes failed daily, until glibc bug was fixed Expensive

28 Storing More History with PostgreSQL Goal: Increase storage of L7 to replace Vertica and/or Redshift Combining 30 small EBS drives via RAID-0 to make 1 30-TB drive This method had more IOPS than a single provisioned EBS drive of the same size Same hardware as an L7 could now store ~40 days of data As number of customers increased, 40 days would shrink to 25 Same strategy as L7, just keep the data longer Codename: Elmo - It stores "mo" (more) data CSV files on S3 One Per Customer+Date Daily ETL Elmo Clusters

29 Typeahead Search What we need: "Type-ahead" search queries, like Google search autocomplete query must finish in < 100ms queries can be across any time range, so all customer data must be covered Not all statistics are needed Only show best 10 matches

30 Typeahead Search What we did: Re-structure data to only store each searchable text string once Combine All data for a Customer's Day into one row using arrays PostgreSQL will compress those arrays via TOAST When compressed, all data can fit in 40TB Use btree_gin indexes for full text search All search ETL handled by 1 32 core machine (i3.8xlarge) All search requests handled by 2 replicas (i3.8xlarge) CSV files on S3 One Per Customer+Date Daily ETL Search DB

31 Applying TOAST to Regular Data Combined All of a customer's data for one day into one row with arrays TOAST Compression shifts workload from scarce IOPS to abundant CPU Some customer's data too large for a single row Split the customer's data into several "chunk" rows Used same hardware as other instances (i3.8xlarge) Same RAID-0 as used in Elmo instance could now hold all customer data

32 Applying TOAST to Regular Data ETL too slow to be handled by just once machine (compression takes time) 5 32-core machines with an ETL-load sharing feature such that each one processes a client/day then shares it with other nodes Replaced all Redshift (1 5-node cluster) and Vertica instances (9 5-node clusters)! Big cost savings Codename: Marjory (Elmo and Marjory are Muppets from Jim Henson TV shows) Marjory DB CSV files on S3 One Per Customer+Date Daily ETL Compressed Data Sharing Marjory DB Marjory DB

33 APIs to Foreign Data Wrappers The Stats-Cache API data must be added to any data which is fetched from PostgreSQL Existing in-memory database written in Python The re-aggregation of this data was handled in regular code, not SQL This is slow and error-prone We created a Foreign Data Wrapper using the multicorn Python API The FDW takes the SQL query an makes an API call, then puts results in a result set The API now looks like a set of PostgreSQL tables Aggregation in SQL much faster Code much simpler

34 Complex Foreign Data Wrappers Codename: Frackles Store csv.gz in compressed SQLite files on S3 Each query starts a web server Start one AWS Lambda per customer/day Each lamba fetches and queries on SQLite file Report results back to web server Web server aggregates results and returns as result set Queries are slow, but data is available sooner Very Short ETL, but slower than dedicated servers Very good for queries across long date ranges AWS now offers Athena, a similar (but costly) service SQL Query Frackles FDW AWS Lambda AWS Lambda AWS Lambda AWS Lambda SQLite S3 SQLite File S3 SQLite File S3 SQLite File S3 File

35 Other Tools: PMPP Poor Man's Parallel Processing Written by me First written in PL/PgSQL, but re-coded in C for performance reasons Set returning function that takes db names + queries as input. Allows an application to send multiple queries in parallel to multiple servers all the queries have the same shape (columns, types) User can re-aggregate data returned from the set returning function. Any machine that talks libpq could be queried (PgSQL, Vertica, Redshift) Allows for partial aggregation on DW boxes Secondary aggregation can occur on local machine

36 Other Tools: Decanters Large queries can exhaust memory on an application machine A decanter lets wine "breathe" These machines let the data "breathe" Abundant CPUs, abundant memory per CPU, minimal disk Some very small lookup tables replicated for performance reasons All other local tables are FDWs to OLTP database (postgres_fdw) Common use: Big PMPP query to Stats-Cache, Elmo, Marjory, Frackles, each one doing a local aggregation Final aggregation happens on decanter Can occasionally experience OOM (rather than on an important machine) New decanter can spin up and enter load balancer in 5 minutes No engineering time to be spent rescuing failed decanters

37 ETL Process (2017) Tagged Ads Elmo Clusters Viewable Events Pixel Servers Log shipping Stats Aggregators Marjory Clusters Daily ETLs S3 - CSVs Daily Summaries Search Clusters S3 - SQLite

38 User Queries (2017) User Search Clusters Searches Stats Requests OLTP DB Third Party DW Pg FDW Pg FDW Decanters Frackles FDW S3 - SQLite Stats-Cache FDW PMPP Requests Live Stats Aggregators Elmo Clusters Marjory Clusters

39 Why Not RDS? No ability to install custom extensions (PMPP, pg_partman, etc) No place to do local \copy operations Reduced insight into the server load (this is better now with RDS Performance Insights) Reduced ability to tune pg server No ability to try beta versions Expense

40 Why Not Aurora? Had early adopter access AWS Devs said that it was not geared for DW workloads I/O sometimes good, sometimes bad Wasn't ready yet Data volumes necessitate advanced partitioning Advanced partitioning was not available until v10 Expense

41 Why Not Athena? Athena had no concept of constraint exclusion to avoid reading irrelevant files Costs $5/TB of data read Most queries would cost > $100 each Running thousands of queries per hour

42 Questions?