real-time delivery architecture

Transcription

1 real-time delivery uc berkeley - 27 august 2012

2 designing twitter

3 what are the goals? evolve from being solely a web stack

4 ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s)

5 what are the goals? evolve from being solely a web stack isolate responsibilities and concerns site speed and reliability developer innovation speed

6

7 Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams

8

9

10

11

12 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Cache Fanout

13 Write API pipelined 4k destinations at a time Blender replicated Timeline Service HTTP Push Mobile Push Hadoop Batch Compute keyed off recipient Timeline Cache Search Cache insert Fanout Push Compute Ingester Social Graph Service

14 Write API Ingester RPUSHX to only add to cached timelines Blender Timeline Service Tweet IDPush User ID HTTP 8 bytes 8 bytes Mobile Push Bits Hadoop 4 bytes Batch Compute native list structure Push Compute Search Cache using redis Timeline Cache Fanout

15 Write API Ingester RPUSHX to only add to cached timelines Blender Timeline Service Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop Tweet ID User ID Mobile Tweet ID User ID Push Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID Tweet ID Batch Compute native list structure Push Compute Search Cache using redis Timeline Cache Fanout

16 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Cache Fanout

17 Write API Fanout Timeline Cache Gizmoduck Timeline Service TweetyPie

18 Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams

19 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Cache Fanout

20 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Cache Fanout

21 Write API Ingester Blender Timeline Service HTTP Push Hadoop queries one replica of all indexes Mobile Push merges & ranks results Batch Compute blender Push Compute Timeline Cache Search Index Fanout

22 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

23 Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams

24 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

25 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

26 http push / hosebird maintains persistent connections with end clients processes tweet & social graph events event-based router

27 Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow event propagation write API sends all events into hosebird; sees content creation events, social graph changes, etc. different queues for public tweets, protected tweets, social events, etc.

28 Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow event cascading bandwidth management simultaneous connection management (~1m long lived & open connections to this cluster)

29 Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow firehose edge machine simply outputs the public tweet queue only allow a limited number of firehoses per hosebird box for bandwidth management

30 Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streams track / follow simple query based on tweet content keeps list of terms / users of interest parses public tweets at the edge, and if term matches a token, or user is of interest, then route

31 Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streams user streams replicate home timeline experience upon login, obtain following list keep cached following list coherent by seeing social graph updates route tweet if from a followed user

32 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

33 Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams

34 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

35 Write API Ingester Blender Timeline Service Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

36 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

37 Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams

38 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

39 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

40 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

41 Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout Query Path

42 Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout Query Path

43 Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout Query Path

44 Blender Timeline Service Write Path HTTP Push Mobile Push Hadoop Batch Compute Timeline Cache Fanout Push Compute Ingester Search Index Read Path Write API

45 Blender Timeline Service Write Path HTTP Push Mobile Push Hadoop Batch Compute Timeline Cache Fanout Push Compute Ingester Search Index Read Path Write API

46 things we re trying...

47

48 Write API Ingester Fanout Search Index Timeline Cache search index [ hello, world ] fanout index [@danadanger,...]

49 User Intent Query Expansion Hello, world Hello AND s home timeline user_timeline:nelson OR user_timeline:danadanger

50 Write API fan-in fan-out O(1) write Ingester Fanout O(n) write O(n) read Search Index Timeline Cache O(1) read

51 User Intent Hello, world Query Expansion Hello AND s home timeline home_timeline:raffi

52 User Intent Query Expansion Hello, world Hello AND s home timeline home_timeline:raffi OR user_timeline:taylorswift13

53 streaming compute continuous computation driven by the events that come into twitter generalizing the push mechanism

54 Write API Ingester Blender Timeline Service Mobile Push Hadoop Batch Compute HTTP Push Push Compute Timeline Cache Search Index Fanout

55 timeline query statistics >150m active users worldwide 300k qps poll-based 1ms p50 / 4ms p99 30k qps search-based timelines

56 tweet input ~340m tweets per day ~4K/sec daily average ~6K/sec daily peak >10K/sec during large events

57

58 followed by following

59 timeline delivery statistics 26b deliveries / day (~18m / min) 3.5 p50 to deliver to 1m ~300k deliveries / sec

60 thanks!