Business Analytics: Asking the Right Questions. Ben Porterfield Founder, VP Engineering

Transcription

1 Business Analytics: Asking the Right Questions Ben Porterfield Founder, VP Engineering

2 BUSINESS INTELLIGENCE Operational Control How many sales did I do today? Understand & Improve Experience Are users engaging? Do they like the new features? Make business decisions Should we start delivering in a new city?

3 Data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher ANDREW LEONARD Salon

4 1 Tracking Data 1 The Analytical Process 2 Storing Data 3 Merging Data (ETL) 4 Retrieving Data 5 Analysis & Decision Making

5 Tracking Data

6 What To Track? Transactional Event Users Orders Inventory Views Clicks In-app actions

7 Tracking - Event Data Embed in product process Serverside too Taxonomy matters Every new feature should come with events Lots of nontransactional events happen on server Big flat event space becomes unwieldy

8 Storing Data

9 Storing - Transactional Data Go with SQL Store all states Keep it clean NoSQL could be a burden long-term Even offline processes Messy schema = complicated analytics

10 We were relying on MongoDB while it was easier for developers to play with the data, it became a hurdle for other team members. MICHAEL ERASMUS Back-end Engineer, Buffer

11 Storing - Event Data Own it Use ecosystem too Store all the IDs Or, at a minimum, be able to get it Lots of great SaaS event platforms Need to be able to correlate events to transactions

12 Merging Data

13 You should combine transaction and event data, +more Transactional Data Application Other Data? Event Data Use an analytical database Redshift is current leader Difficult - data is heavy Raw Queries WITH user_order_activity AS ( SELECT user_id, age FROM ORDERS GROUP BY user_id) SELECT AVG(users.age) as average_age_of_purchaser FROM user_order_activity LEFT JOIN users ON user_order_activity.user_id = users.user_id Biz-User Tools

14 Traditional Approach DIFFICULT & COMBERSOME ETL - Heavy Transformation SILOED OLAP / Data Summaries LIMITED Restricted Q&A EVENT DATA TRANSACTIONAL DATA A B X E F A L Q D P B I M G S N Q R SUMMARY A C B? ETL TEAM EDW TEAM BI TEAM END USER WANT TO ASK NEW QUESTIONS?

15 Modern Approach CONSOLIDATED Simple Extract & Load AGILE Data Modeling Layer FLEXIBLE Transformation at Query ACCESSIBLE Anywhere for Anyone EVENT DATA TRANSACTIONAL DATA A Z B Q G A Q G D Q MPP REDSHIFT IMPALA U P I M Z S U Q N T R S A W X - name: first_purchasers type: single_value base_view: orders measures: [orders.first_purchase_count] listen: Data Model - name: orders_by_day_and_category title: "Orders by Day and Category" type: looker_area base_view: order_items A Z ANY DEVICE 3 RD PARTY APP API DATA TEAM INNOVATION END USERS

16 Asana s Data Infrastructure

17 Retrieving Data

18 We have a developer name Juan and any reports we needed would flow through him. TODD LEHR SVP Engineering, Dollar Shave Club

19 When he got backlogged, our team didn t have access to the data immediately. TODD LEHR SVP Engineering, Dollar Shave Club

20 Initially whenever we were asked for data, we would write a custom script ANNIE CORBETT Business Intelligence Analyst, Venmo

21 ..and then repeat this process whenever the product team wanted to extend the timeframe. ANNIE CORBETT Business Intelligence Analyst, Venmo

22 What s selling? What colors and sizes is it selling in? Questions from a retail buyer at e-commerce store: What s getting returned? Is there a particular size/color? Is there a product people buy first that increases their likelihood of becoming a repeat customer?

23 Self-Service is Key Get them the tool Decisions vs. data science Gamechanging insights People with questions are running the businesses. Should we open a new market in Maine? Don t only come from analyst group

24 Analysis and Decision Making

25 1 1 Clearly define success metrics Analysis and Decision Making 2 Look for low-hanging fruit 3 Go one level deeper

26 Analysis and Decision Making: Success Metrics

27 SUCCESS METRICS Focus on desired outcome What do you want users to experience? Measure Engagement In most cases this is first-line business analytics Measure Retention Are people coming back?

28 HOW TO TRACK ENGAGEMENT? Not with page views Usually not even with time on page Upworthy s attention minutes Lots of indicators (mouse, video, etc) Looker s approximate usage Any event in 2 minute window

29 Deriving Approximate Usage created_date user_id count approximate_usage SELECT event.created_at AS created_date, event.user_id as user_id, COUNT(*) AS count, COUNT(DISTINCT CONCAT( 1/10 1/10 1/ minutes 50 minutes 80 minutes CONCAT(event.user_id,' ',event.user_browser_id), FLOOR(UNIX_TIMESTAMP(event.created_at)/(60*2)) ) )*2 AS approximate_usage_in_minutes FROM event GROUP BY created_date, user_id

30 Derived Tables Transactional Derived Table Analytical SELECT orders.user_id as user_id COUNT(*) as lifetime_orders MIN(orders.created_at) as first_order MAX(orders.created_at) as latest_order COUNT(DISTINCT DATE_TRUNC('month )) as distinct_months_with_orders FROM orders GROUP BY user_id Event Insights

31 Derived Tables Start simple Most useful at row level Great for cohorts and sessionization Subselects until slow, SQL on cron works surprisingly well Don t roll up data, precompute facts Tiered derived dimension vs. some other metric

32 Derived Table - User Order Facts SELECT orders.user_id as user_id COUNT(*) as lifetime_orders MIN(orders.created_at) as first_order MAX(orders.created_at) as latest_order COUNT(DISTINCT DATE_TRUNC('month )) as distinct_months_with_orders FROM orders GROUP BY user_id user_id lifetime_orders first_order latest_order distinct_months_with_orders /10/15 2/14/15 2

33 Derived Table + Sourcing

34 Derived Table + Sourcing

35 Usage How likely to purchase if they do X Retention Are users coming back Repeat buyers What s different about them Churn Users that will likely never do X again Time to transaction How long till first X??? Invent a metric

36 Pay/Charge Mistake.

37 It was clear some users were accidentally paying instead of charging, but it wasn't clear how widespread the problem was and whether it was worth prioritizing a fix Inventing Metrics

38 Inventing Metrics Identify behavior Measure % of population Experiment Can be good or bad just something possibly significant Who is doing this thing? Ability to play with numbers is crucial

39 Analysis and Decision Making: Low-hanging Fruit

40 Low-hanging Fruit This is the kind of very visual, very data driven piece of analysis that helps us think, "Is opening the sale at noon the right decision????

41 Out of stocks are huge detractors from the customer experience - it sucks ordering something and then not getting it - as well as revenue we failed to capture Low-hanging Fruit

42 Analysis and Decision Making: One Level Deeper

43 One Level Deeper While this immediate insight might have led us to focus on small groups, this didn t match our expectations of people planning an outing on a Friday night, prompting us to look further. Group Size Time To Book 2 3 4

44 We analyze all the platform data available - When someone attempts to sign, completes the signup, pushes an app, has spend, etc One Level Deeper

45 Even though it looks like we were having nice incremental growth, looking into the details we see some things to look into further One Level Deeper

46 Don t confuse an increase in a metric with success.

47 Takeaways Put data in analytical database Give business users tool Define success metrics Make sure it s fast and speaks SQL Empower them to answer their own questions Focus on engagement and retention

48 Ben Porterfield Founder, VP Engineering