Fast and Efficient A/B Testing Analysis with Shiny and SQL. Charlie Thompson Storyblocks

Transcription

1 Fast and Efficient A/B Testing Analysis with Shiny and SQL Charlie Thompson Storyblocks

2 A/B Testing at Storyblocks

3 Our search page for stock video

4 Related Search cards test

5 Related Search cards test Test Control

6 We store results for our tests in Shiny

7 We have > 100 metrics to analyze per test

8 A/B testing generates big data We have thousands of A/B tests with millions of users Multiple ways to measure users Lots of metrics per user

9 Shiny and SQL together

10 A brief history Automated online dashboard in SQL Outsourced to 3rd party 2014 Scaling within Shiny Adhoc SQL queries 2017 To Shiny!

11 Loading big data into Shiny Overnight preprocessing on shiny server R script queries the SQL database and saves off an.rdata file for each test that contains the raw data test_1.rdata test_2.rdata Raw A/B testing data (SQL) load_data.r test_3.rdata test_4.rdata

12 Loading big data into Shiny Overnight preprocessing on shiny server Live in dashboard R script queries the SQL database and saves off an.rdata file for each test that contains the raw data As tests are selected in the dashboard, Shiny pulls the raw data file and computes all the metrics needed, including hypothesis tests test_1.rdata test_2.rdata Raw A/B testing data (SQL) load_data.r test_3.rdata test_4.rdata server.r Shiny Dashboard

13 Constraints with Shiny at scale Overnight preprocessing on shiny server Live in dashboard R script queries the SQL database and saves off an.rdata file for each test that contains the raw data As tests are selected in the dashboard, Shiny pulls the raw data file and computes all the metrics needed, including hypothesis tests test_1.rdata test_2.rdata Raw A/B testing data (SQL) load_data.r test_3.rdata test_4.rdata Bottleneck #3: Users queue Bottleneck #1: Reading in large tests server.r Shiny Dashboard Bottleneck #2: Calculating hypothesis tests for 50+ metrics

14 Overcoming Shiny constraints Overnight preprocessing on shiny server R script queries the SQL database and calculates hypothesis tests and saves off an.rdata file for each test that contains the aggregated data Raw A/B testing data (SQL) load_data.r Live in dashboard As tests are selected in the dashboard, Shiny pulls the aggregated file for each test, which now contains historical values instead of daily snapshots FUHGETTABOUTIT! Aggregated data is wicked small Bottleneck #3: test_1.rdata Bottleneck #1: Users queue Reading in large tests test_2.rdata server.r test_3.rdata test_4.rdata NO WORRIES! The dashboard is so fast we won t notice Shiny Dashboard NOT ANYMORE! Bottleneck #2: This is done in the Calculating hypothesis tests for morning 50+ metrics

15 Making the most of your data

16 When is a test done?

17 Aggregated data gives a time series view Test begins

18 Time series helps prevent premature reads P Value Test looks 95% significant here! Date

19 P-value should stabilize over time P Value Win or lose, the P-value should stabilize before a test is finished Date

20 When to think about scaling

21 Shiny: prototype vs production Prototype Production Hosting Local Shiny server, shinyapps.io, etc Number of concurrent users One Multiple Page load time Easy to overlook Instant, UX is important Data storage Often pull in unused rows or columns Loads only necessary data Stability and maintenance Only needs to be working when demoing Minimal downtime

22 Measuring Shiny usage Make sure you know how many users you have!

23 What we learned

24 Let SQL be SQL and R be R R SQL Big data aggregation Possible, but slow Made for exactly this Hypothesis tests and charts Made for exactly this Painful, need tools

25 Data tips for Shiny in production 1. Subset your input data before reading it in 2. Use.RData files 3. Consider ETL process - do you really need real-time data? 4. Monitor usage

26 Additional resources A/B Testing in the Wild [Etsy] - Emily Robinson A/B Testing at Stack Overflow - Julia Silge Experiments at Airbnb - Jan Overgoor Shiny server system performance monitoring - Huidong Tian

27 Questions? We re hiring! Contact me