O Reilly RailsConf,

Transcription

1 O Reilly RailsConf,

2 Who is that guy? Jesper Richter- Reichhelm Berlin, Germany Head of wooga

3 Wooga does social games

4 Wooga has dedicated game teams Cooming soon PHP Cloud Ruby Cloud Ruby Bare metal Erlang Cloud

5 Flash client sends state changes to backend Flash client Ruby backend

6 Social games need to scale quite a bit 1 million daily users 200 million daily HTTP requests

7 Social games need to scale quite a bit 1 million daily users 200 million daily HTTP requests 100,000 DB operagons per second 40,000 DB updates per second

8 Social games need to scale quite a bit 1 million daily users 200 million daily HTTP requests 100,000 DB operagons per second 40,000 DB updates per second but no cache

9 A journey to 1,000,000 daily users Start of the journey 6 weeks of pain Paradise Looking back

10 We used two MySQL shards right from the start data_fabric gem Easy to use Sharding by Facebook s user id Sharding on controller level

11 Scaling applicalon servers was very easy Running at Amazon EC2 Scalarium for automagon and deployment Role based Chef recipes for customiza5on Scaling app servers up and out was simple

12 Master- replicalon for DBs worked fine lb app app app

13 We added a few applicalon servers over Lme lb app app app app app app app app app

14 Basic setup worked well for 3 months May Jun Jul Aug Sep Oct Nov Dec

15 A journey to 1,000,000 daily users Start of the journey 6 weeks of pain Paradise Looking back

16 SQL queries generated by Rubyamf gem Rubyamf serializes returned objects Wrong config => it also loads associagons

17 SQL queries generated by Rubyamf gem Rubyamf serializes returned objects Wrong config => it also loads associagons Really bad but very easy to fix

18 New Relic shows what happens during an API call

19 More traffic using the same cluster lb app app app app app app app app app

20 Our DB problems began May Jun Jul Aug Sep Oct Nov Dec

21 MySQL hiccups were becoming more frequent Everything was running fine and then DB throughput went down to 10% A]er a few minutes everything stabilizes again just to repeat the cycle 20 minutes later!

22 AcLveRecord s checks caused 20% extra DB load Check connecgon state before each SQL query MySQL process list full of status calls

23 AcLveRecord s status checks caused 20% extra DB Check connecgon state before each SQL query MySQL process list full of status calls

24 I/O on MySQL masters slll was the boyleneck New Relic shows DB table usage 60% of all UPDATEs were done on the Gles table

25 New Relic shows most used tables

26 Tiles are part of the core game loop Core game loop: 1) plant 2) wait 3) harvest All are operagons on Tiles table.

27 We started to shard on model, too We put this table in 2 extra shards old master old

28 We started to shard on model, too We put this table in 2 extra shards 1) Setup new masters as s of old ones old master old new master

29 We started to shard on model, too We put this table in 2 extra shards 1) Setup new masters as s of old ones old master old new master new

30 We started to shard on model, too We put this table in 2 extra shards 1) Setup new masters as s of old ones 2) App servers start using new masters, too old master old new master new

31 We started to shard on model, too We put this table in 2 extra shards 1) Setup new masters as s of old ones 2) App servers start using new masters, too 3) Cut replica5on old master old new master new

32 We started to shard on model, too We put this table in 2 extra shards 1) Setup new masters as of old ones 2) App servers start using new masters, too 3) Cut replica5on 4) Truncate not- used tables old master old new master new

33 8 DBs and a few more servers lb app app app app app app app app app app app app app app app app 5les 5les 5les 5les

34 Doubling the amount of DBs didn t fix it May Jun Jul Aug Sep Oct Nov Dec

35 We improved our MySQL setup RAID- 0 of EBS volumes Using XtraDB instead of vanilla MySQL Tweaking my.cnf inno_flush_log_at_trx_commit inno_flush_method

36 Data- fabric gem circumvented AR s internal cache 2x Tile.find_by_id(id) => 1x SELECT

37 Data- fabric gem circumvented AR s internal cache 2x Tile.find_by_id(id) => 1x SELECT

38 2 + 2 masters and slll I/O was not fast enough We were geing desperate: If is not enough,... perhaps masters will do?

39 It s no fun to handle 16 MySQL DBs lb app app app app app app app app app app app app app app app app app app 5les 5les 5les 5les

40 It s no fun to handle 16 MySQL DBs lb app app app app app app app app app app app app app app app app app app 5les 5les 5les 5les 5les 5les 5les 5les

41 We were at a dead end with MySQL May Jun Jul Aug Sep Oct Nov Dec

42 I/O remained the boyleneck for MySQL UPDATEs Peak throughput overall: 5,000 writes/s Peak throughput single master: 850 writes/s We could get to ~1,000 writes/s but then what?

43 Pick the right tool for the job!

44 Redis is fast but goes beyond simple key/value Redis is a key- value store Hashes, Sets, Sorted Sets, Lists Atomic opera5ons like set, get, increment 50,000 transacgons/s on EC2 Writes are as fast as reads

45 Shelf Lles : An ideal candidate for using Redis Shelf Lles: { plant1 => 184, plant2 => 141, plant3 => 130, plant4 => 112, }

46 Shelf Lles : An ideal candidate for using Redis Redis Hash HGETALL: load whole hash HGET: load a single key HSET: set a single key HINCRBY: increase value of single key

47 Migrate on the fly when accessing new model

48 Migrate on the fly - but only once true if id could be added else false

49 Migrate on the fly - and clean up later 1. Let this running everything cools down 2. Then migrate the rest manually 3. Remove the migragon code 4. Wait ungl no fallback necessary 5. Remove the SQL table

50 MigraLons on the fly all look the same Typical migragon throughput over 3 days Ini5al peak at 100 migra5ons / second

51 A journey to 1,000,000 daily users Start of the journey 6 weeks of pain Paredise (or not?) Looking back

52 Again: Tiles are part of the core game loop Core game loop: 1) plant 2) wait 3) harvest All are operagons on Tiles table.

53 Size mayers for migralons MigraGon check overloaded Redis Migra5on only on startup We overlooked an edge case Next 5me only migrate 1% of users Migrate the rest when everything worked out

54 In- memory DBs don t like to saving to disk You sgll need to write data to disk eventually SAVE is blocking BGSAVE needs free RAM Dumping on master increased latency by 100%

55 In- memory DBs don t like to dump to disk You sgll need to write data to disk eventually SAVE is blocking BGSAVE needs free RAM Dumping on master increased latency by 100% Running BGSAVE s every 15 minutes

56 ReplicaLon puts Redis master under load ReplicaGon on master starts with a BGSAVE Not enough free RAM => big problem Master queue requests that cannot process Restoring dump on s is blocking

57 Redis has a memory fragmenlon problem 44 GB 24 GB in 8 days

58 Redis has a memory fragmenlon problem 38 GB 24 GB in 3 days

59 If MySQL is a truck... Fast enough for reads Can store on disk Robust replicagon h\p://

60 If MySQL is a truck, Redis is a Ferrari Fast enough for reads Super fast reads/writes Can store on disk Out of memory => dead Robust replicagon Fragile replicagon h\p://

61 Big and stalc data in MySQL, rest goes to Redis Fast enough for reads Super fast reads/writes Can store on disk Out of memory => dead Robust replicagon Fragile replicagon 256 GB data 60 GB data 10% writes 50% writes h\p://

62 95% of operalon effort is handling DBs! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app redis redis redis redis redis redis redis redis redis redis

63 We fixed all our problem - we were in Paradise! May Jun Jul Aug Sep Oct Nov Dec

64 A journey to 1,000,000 daily users Start of the journey 6 weeks of pain Paredise (or not?) Looking back

65 Data handling is most important How much data will you have? How o]en will you read/update that data? What data is in the client, what in the server? It s hard to refactor data later on

66 Monitor and measure your applicalon closely On applicagon level / on OS level Learn your app s normal behavior Store historical data so you can compare later React if necessary!

67 Listen closely and answer truthfully Q & A Jesper Richter- slideshare.net/wooga

68 If you are in Berlin just drop by Thank you Jesper Richter- slideshare.net/wooga