Counting is Hard: Probabilistically Counting Views at Reddit. Krishnan Chandra, Data Engineer

Transcription

1 Counting is Hard: Probabilistically Counting Views at Reddit Krishnan Chandra, Data Engineer

2 What is probabilistic counting? Overview How did probabilistic counting help us scale? What issues did we face along the way?

3 What is Reddit? Reddit is the frontpage of the internet A social network where there are tens of thousands of communities around whatever passions or interests you might have It s where people converse about the things that are most important to them

4 Reddit by the numbers 4th/7th Alexa Rank (US/World) 330M+ MAU 138K+ Active Communities 10.7M Posts per month 14B Screenviews per month

5 Counting Views

6 Why Count Views? Includes logged-out users Better measure of reach than votes Currently exposed to moderators and content creators

7 Cat Fist Bumping Cat Walking a Human

8 Why is Counting Hard?

9 Product Requirements Counts are over the life of a post The same user should not count multiple times within a short time frame Should build in some protections against spamming/cheating (similar to votes) Should provide (near) real-time feedback

10 Exact counting: Requires storing state per user per post Exact vs. Approximate Counting Approximate counting: Requires much less state and storage Provides an estimate of reach within a few percentage points of the exact number

11 HyperLogLog (HLL) Hash-based probabilistic algorithm published in 2007 Approximates set cardinality Works well for large cardinalities, but not for small ones HyperLogLog (And Friends) HyperLogLog++ Introduced by Google in 2013 Uses sparse and dense HLL representations Switches over to HLL once needed

12

13 Hash table consisting of m registers or buckets, each of width k bits Hash the input value, and split the hash value into 2 portions How does HLL work? First portion (log2m bits) used to index to a register Second portion used to count the number of leading zeros and set the register value

14 Assume: m=8 registers, k=3 bits input hash 1 1 r0 1 Register# leading zeroes Record 3+1=4 into Register# 7 r1 r2 r3 r4 r5 r6 r7 Adapted from HyperLogLog - A Layman s Overview 1 0 0

15 Estimate of cardinality is computed by taking the harmonic mean of the registers and raising 2 to that power Computing Cardinality Intuition: HLL is like flipping a coin! Largest run of heads gives an estimate of total number of flips

16 Counting Error HLL standard error Number of registers/hash buckets m Standard error = 1.04/sqrt(m) Using Redis s HLL implementation, standard error is 0.81%!

17 Using HLL to Count Views 1 HLL per post HLL inserts are idempotent! Allows reprocessing data if needed How to manage de-duping over short time window? Store user + truncated timestamp as the value

18

19 Exact counting: User id = 8 byte long ~1.5m users * 8 bytes = 12 MB Space Usage HLL (Redis implementation) Max size = 12 KB 0.1% of the exact counting storage

20 Counting Architecture

21 Architecture Goals 1. Consume a stream of view events and filter out spam/bad events 2. For good events, insert into an HLL in real time 3. Allow clients to consume views values in real time

22 Server Side Events Anti-Spam App Servers Client Side Events Counting

23 Kafka Main message bus for view events Stream Processing Infrastructure Redis Used for storing state + HLLs Intended as short term storage Functions as a cache for Cassandra Cassandra Used to store the final counts and HLLs in separate column families Intended as long term storage

24 Counting Application (Part 1) Anti-Spam Consumer Consumes the stream of views from Kafka Basic rules engine backed by Redis Consumer outputs a decision to a Kafka topic

25 Counting Application (Part 2) Counting Consumer Consumes the decisions topic output by the anti-spam consumer Creates/updates the HLL for the post in Redis. Stores both the count and the HLL filter out to Cassandra.

26 Scaling Challenges

27 Redis Problems Rules engine is very memory heavy HLL counting is very CPU-heavy Rules engine data is generally time-bound with expiry HLL data should be kept in Redis as long as possible to avoid reading from Cassandra

28 Solutions Separate Redis instances for the 2 parts of the application Different instance types to reflect the different workloads Allkeys-lru expiration on HLLs, volatile-ttl expiration on the rules engine

29

30 Cassandra Problems 1 row per post - overwritten frequently Read rate on page loads overwhelming the cluster Issues with load when catching up Storage grows forever with the number of posts!

31 Solutions Updates to the same row in Cassandra throttled to every 10 seconds Read caching Slow the update rate when catching up More disk!

32

33 Observations Views on Reddit skew towards newer posts Allows most views to be served by Redis Keeps read rate on Cassandra very low

34

35 Thanks to HLLs, counting views became much more efficient Current storage usage is ~1TB for a full year of posts! Takeaways Delivery was possible in a quarter with an engineering team of 3 (not always full time)

36 /u/gooeyblob - Cassandra + Backend /u/d3fect - Backend + API Thanks to our team! /u/powerlanguage - Product Management

37 Thanks! Krishnan Chandra u/shrink_and_an_arch PS: We re hiring!

38 View Counting at Reddit (Blog Post from 2017) References Original HyperLogLog paper Redis blog announcing HLL support Google paper announcing HLL++ algorithm HyperLogLog - A Layman s Overview