Triple R Riak, Redis and RabbitMQ at XING

Transcription

1 Triple R Riak, Redis and RabbitMQ at XING Dr. Stefan Kaes, Sebastian Röbke NoSQL matters Cologne, April 27, 2013

2 ActivityStream Intro

3

4

5

6 3 Types of Feeds

7 News Feed

8 Me Feed

9 Company Feed

10 Activity Creation

11

12 ActivityStream POST /activitystream/activities

13

14 ActivityStream events.participation.changed

15 Events App events.event.created events.participation.changed... Groups App groups.article.created groups.member.joined... User App users.profile.updated users.contact.created... etc ActivityStream

16 Old Approach

17 Activity INSERT INTO `activities`... Comment INSERT INTO `comments`... several hundred millions Like INSERT INTO `likes`...

18 Efficient Activity Creation

19 But... slow reads

20

21 Groups App Companies App User App GET group memberships Settings GET privacy settings GET followed companies etc.... GET contacts ActivityStream activities? likes comments

22 Groups App Companies App User App GET group memberships Settings GET privacy settings GET followed companies etc.... GET contacts ActivityStream activities likes comments

23 Activities immediately visible

24 Consistency

25 SQL Databases are well understood

26 Database Master is Single Point of Failure

27 Sharding

28 Unsatisfactory Read Performance

29 New Approach

30 Materialized Feeds

31 Activity create User App GET contacts... etc. ActivityStream Storage activities news feeds me feeds company feeds likes comments?

32 Requirements

33 Better Read Performance

34 Activities created by me must be visible to myself immediately

35 Activities created by others should appear within a reasonable time frame in my stream

36 Storage layer must tolerate high read and write loads

37 Storage layer must provide easy capacity scaling

38 Low maintenance

39 Option 1: Do it yourself SQL database design

40 Option 2: Off the shelf NoSQL database

41 We chose

42 We tend to view it as a highly available distributed hash table

43 Eventual consistency/conflict resolution is the hard part

44 Bounded size feeds are easy

45 Unbounded feeds are much harder

46 Object Model

47 JSON Documents

48 Activities

49 Activities

50 Activities

51 Activities 2P-Set

52 JSON Documents

53 Feeds

54 Feeds bounded list of chunk references

55 Feeds chunk sequence number

56 Feeds youngest activity ref

57 Feeds oldest activity ref

58 Feeds size of referenced chunk

59 JSON Documents

60 FeedChunk

61 FeedChunk 2P-Set

62 The Migration

63 Incremental rollout

64 Part 1: From old to new

65 Let s start simple!

66 Replicating some data

67 Old ActivityStream activity.created activity.deleted comment.created comment.deleted like.created like.deleted data migration processors New ActivityStream

68 Measuring performance

69 Old ActivityStream newsfeed.viewed mefeed.viewed companyfeed.viewed data migration processors shadow query processors New ActivityStream

70

71 Part 2: From new to old

72 Old ActivityStream POST /activitystream/activities DELETE /activitystream/activities/{id} POST /activitystream/activities/{activity_id}/comments DELETE /activitystream/activities/{activity_id}/comments/{id} PUT /activitystream/activities/{activity_id}/likes/{user_id} DELETE /activitystream/activities/{activity_id}/likes/{user_id} Beta User B Beta User A Beta User C New ActivityStream

73 Old ActivityStream POST /activitystream/activities DELETE /activitystream/activities/{id} POST /activitystream/activities/{activity_id}/comments DELETE /activitystream/activities/{activity_id}/comments/{id} PUT /activitystream/activities/{activity_id}/likes/{user_id} DELETE /activitystream/activities/{activity_id}/likes/{user_id} Beta User A Beta User B Beta User C activity id New ActivityStream

74 Part 3: What about the old data?

75 Bulk Data Migration: Failed Version 1

76 Bulk Data Migration: Failed Version 1 1. Reset data in the new system 2. Query the old system REST API for the feeds 3. Store them in the new system 4. Switch to the new system

77 This was naive

78 The old system was way too slow to return the millions of feeds in their full length

79 Bulk Data Migration: Failed Version 2

80 Bulk Data Migration: Failed Version 2 1. Reset data in the new system 2. Read all activities based on a dump of the old system 3. Publish created messages to RabbitMQ for each activity/comment/like 4. Let the new system build its data structures 5. Switch to the new system

81 This was naive

82 You can t replay the history of 2.5 years this way

83 Bulk Data Migration: Successful Version

84 Bulk Data Migration: Successful Version 1. Reset data in the new system 2. Obtain data dump from old system 3. Extract data from the dumps and compute a representation of the feeds in Redis with a massive amount of batch processors 4. Use this data to build up the structures in the new system 5. Switch to the new system

85 It worked!

86 But...

87 A lot of additional code

88 Fragile, manual steps involved

89 A lot of technology: RabbitMQ, Riak, Redis, Varnish

90 One run took 5 days

91 But it worked!

92 Current Status

93 New system is live for all users since 12/12/12

94 Old and new system were kept in sync till April 2013

95 In case of serious trouble, we could have switched to the old system within seconds

96 Performance goals have been met

97 Performance goals have been met Old system New system happy t < 0.1s 0.17% 62.01% satisfied t < 0.5s 41.36% 99.71% tolerating 0.5s t < 2s 58.20% 0.28% frustrated t 2s 0.44% 0.00% Apdex Score

98 Production setup 10 Riak Servers as Primary Cluster 10 Riak Servers as Backup Cluster (Multi-DC Replication) SSDs, Raid 0 and a proper Linux file I/O scheduler setting (noop) Bitcask storage backend 4 REST API Servers 4 Background Worker Servers Monitoring using Ganglia and Logjam (App Performance)

99 Lessons learned

100 Eventual consistency sounds easy, but is hard to implement correctly in practice There s a steep learning curve at the beginning High update rates and large objects don t go together well, if your storage system offers just get, put and delete operations Achieving high performance requires careful thought about data structures, algorithms and access patterns Building a new system from scratch is lot easier than migrating an existing system

101 Protobuffs API faster than HTTP Use the best performing JSON library you can find (Ruby: Oj gem) Avoid a full-blown ORM for Riak if you care about performance (Ruby: Ripple gem)

102 At one point we saturated the Gigabit network cards on the Riak cluster This lead to compressing all data before storing it on the cluster and breaking news feeds into chunks

103 Thank you for your attention! Dr. Stefan Kaes Sebastian Röbke We re hiring: The professional network