Cloud Gaming Architectures

Transcription

1 Cloud Gaming Architectures From Social to Mobile to MMO Mark Bate, AWS Solutions Architect 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2 Gratuitous logo slide

3 Scale to what you need, pay for what you use Unmet Demand Upset Players Missed Revenue :( Servers Demand Capacity Demand Excess Capacity Wasted $$ Traditional: Rigid AWS: Elastic

4 Global is good 11 regions 53 edge locations Continuous expansion

5 Common game back-end concepts Think in terms of APIs HTTP + JSON Get friends, leaderboard Binary asset data Multiplayer servers High availability Scalability

6 Core (HA) game back end Choose region >=2 Availability Zones Amazon EC2 for app Elastic Load Balancing Amazon RDS database Multi-AZ ELB Region

7 Scale it way out Amazon S3 for game data Assets UGC Analytics ELB Region

8 Scale it way out Amazon S3 for game data Assets UGC Analytics... With Amazon CloudFront! ELB CloudFront CDN Region

9 Scale it way out Amazon S3 for game data Assets UGC Analytics... with CloudFront! Auto Scaling group Capacity on demand Respond to users Automatic healing ELB CloudFront CDN Region

10 Scale it way out Amazon S3 for game data Assets UGC Analytics... with CloudFront! Auto Scaling group Capacity on demand Respond to users Automatic healing Amazon ElastiCache Memcached Redis ELB Region CloudFront CDN

11 Writing is painful Games are write heavy Caching of limited use Key value Binary structures Database = bottleneck ELB CloudFront CDN Region

12 Sharding (not fun)

13 Amazon DynamoDB Fully managed NoSQL data store Provisioned throughput Secondary indexes PUT/GET keys Document support! ELB CloudFront CDN Region

14 Example: Leaderboard in DynamoDB UserID (hash key) BoardName (range key) TopScore TopScoreDate "101" "Galaxy Invaders" 5842 " T17:24:31" "101" "Meteor Blasters" 1000 " T23:18:01" "101" "Starship X" 24 " T13:14:21" "102" "Alien Adventure" 192 " T11:07:56" "102" "Galaxy Invaders" 0 " T07:33:42" "103" "Attack Ships" 3 " T01:13:24" "103" "Galaxy Invaders" 2317 " T06:53:00" "103" "Meteor Blasters" 723 " T01:14:24" "103" "Starship X" 42 " T06:53:03" Hash key = Primary key Range key = Sub key Range key = Sort key Others attributes are undefined So How to sort based on top score?

15 Leaderboard with secondary indexes UserID (hash key) BoardName (range key) TopScore TopScoreDate "101" "Galaxy Invaders" 5842 " T17:24:31" BoardName (hash key) TopScore (range key) UserID "Alien Adventure" 192 "101" "Attack Ships" 3 "103" "Galaxy Invaders" 0 "102" "Galaxy Invaders" 2317 "103" "Galaxy Invaders" 5842 "101" "Meteor Blasters" 723 "103" "Meteor Blasters" 1000 "101" "Starship X" 24 "101" "Starship X" 42 "103" Create a secondary index! Set hash key to BoardName Set range key to TopScore Project extra attributes as needed Can now query by BoardName, sorted by TopScore Handles many common gaming use cases

16 Documents in DynamoDB Scalar types: String, Number, Binary, Boolean, Null Multivalue types: String Set, Number Set, Binary Set Document types: List, Map Document content addressing "name": Mark", "games": ["Megablast", "Spacerace"], "score": { "Megablast" : 123, "Spacerace" : 41 } document.score.megablast "name": { "S": Mark" } "games": { "L": [ { "S": "Megablast" }, { "S": "Spacerace" } ] }, "score": { "M": { "Megablast": { "N": "123" }, "Spacerace": { "N": "41" } } }

17 Related sessions DAT204 DAT401 NoSQL? No Worries: Building Scalable Applications on AWS NoSQL Services Amazon DynamoDB Deep Dive: Schema Design, Indexing, JSON, Search, and More GAM401 Serverless Mobile Game Development with Amazon Cognito, AWS Lambda, and Amazon DynamoDB

18 Customer Story: Devsisters 18

19 What to expect from the session How we started How we improved our design Tips and tricks Retrospect 19

20 Cookie Run 20

21 Cookie Run video 21

22 About Cookie Run Cloud Gaming Architectures from Mobile to Social to MMO 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved M~ downloads 10M DAU Top free 1st in 10 countries Top free 10th in 38 countries

23 More about Devsisters and Cookie Run Cloud Gaming Architectures from Mobile to Social to MMO 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 23

24 How We Started 24

25 In early 2013 Lack of infrastructure, lack of developer, no hope (1 server developer / 0 system engineers) Only 1 game in service Ovenbreak 2 - AWS US East Cookie Run - Only 1 person, 1 month left 25

26 Goal Highly reliable Quality assured Scalable design Auto configuring and scaling Real-time monitoring system Log system 26

27 First design Game server Java, Spring MVC, MySQL 5.5 Operation tool Python, Django, Boto Monitoring Amazon CloudWatch, Zabbix, Statsd, Graphite 27

28 First design 28

29 After 11 days 29

30 Design Improvements 30

31 Design improvements Redesigning the back end Improving the logging system Improving the game patch system Adding global user ranking system 31

32 Redesigning the back end Situation Players send game hearts to each other. Back ends do the bookkeeping - Plan A: Used MySQL for storing data Trouble: MySQL can t keep up;; too many rows (100M ~) - Plan B: Gave unlimited hearts to users! Disabled the feature Trouble: Not so bad, but need to come up with a better solution 32

33 Redesigning the back end Solution MySQL NoSQL (Couchbase) Use MySQL for game data (shop data, stage data, ) Use NoSQL for user data (user items, level, coin, ) 33

34 Before 34

35 After 35

36 Improving the logging system Situation We need real-time log querying capability Solution Real-time log viewing system based on ELK 36

37 Before 37

38 After 38

39 Real-time log viewing system / 39

40 Improving the game patch system Situation App Store binary size limit Some resources need to be downloaded on demand Wanted to distribute patches without App Store update Solution Constructed a decent patch system Based on Amazon S3 and Amazon CloudFront 40

41 Before 41

42 After 42

43 Improving the logging system Situation Total log size >10 TB;; want to analyze all logs Situation forced us to look for big data solutions Solution Adopted big data platforms using Amazon EMR or Amazon EC2 Eventually migrated to Spark and Spark SQL 43

44 Before 44

45 After 45

46 Spark 46

47 Log dashboard 47

48 Adding global user ranking system Situation Want to introduce global user ranking system Solution Use ordered set based on skip list using with ElastiCache with custom caching and a lot of optimization techniques 48

49 Before 49

50 After 50

51 Tips and Tricks 51

52 WARNING THIS CAN HAPPEN TO YOU BASED ON THE TRUE STORY OF OUR TEAM 52

53 Auto Scaling gotchas Frequency: More than 10 times during 2 years Many users connect to the game simultaneously During holiday seasons Start of in-game events When bulk push notifications are sent Or reasons unknown Booting instances takes several minutes, which isn t quick enough to handle spikey loads We have to predict traffic surges and prepare beforehand 53

54 Our bulk push system 54

55 Auto Scaling gotchas Don t set minimum instance of 1 or 2 If one machine dies, service fails Use multiple Availability Zones Sometimes instance availability of a single AZ can run out Use multiple AZ with ELB cross-zone balancing 55

56 Auto Scaling gotchas Set scale-out(scale-in) policy meticulously scale-out: +4 when Latency >= 0.1 for 2 minutes scale-in: -2 when CPUUtilization < 10 for 2 minutes Sometimes scale-up can be a useful option 56

57 Chef server failure Auto Scaling group relying on Chef server is dangerous Chef server is a single point of failure (SPOF) May become unresponsive when too many servers start simultaneously Errors happen in unexpected places! 57

58 Couchbase storage failure Hardware problems can occur in EC2 instances The worst, the most hopeless system failure Front end API server can crash;; that s OK But if you are maintaining a database on EC2, this can be a tragedy It really happens 58

59 Couchbase storage failure June 2015 A monumental hell gate in our company history Server down for 12 consecutive hours because of a disk error in Couchbase Also, our daily backup script had not worked for 1 week prior to the shutdown Some data were restored via replication The other data were restored through adding the lost week s logs to previously backed up data Lesson learned: Replica is necessary. Confirm backups. 59

60 Overseas network failure Frequency: More than 5 times over 2 years This situation has really happened ISPs cut costs leading to overseas packet loss Just Call AWS 60

61 Final Design Review 61

62 First design 62

63 Final 63

64 Future Plans 64

65 Future plans Transactional log system High latency / packet loss networks : QUIC Entertain the world! 65

66 Thank you!

67 Amazon Cognito

68 Amazon Cognito Joe Anna Bob Unique Identities Support multiple login providers Easily integrate with major login providers for authentication, or use your own authentication system Identity Providers Your own Auth Unique users vs. devices Manage unique identities; automatically recognize unique user across devices and platforms Any Device Any Platform Helps implement security best practices Securely access any AWS service from mobile device; it simplifies the interaction with AWS Identity and Access Management Any AWS Service Mobile Analytics S3 DynamoDB Kinesis

69 Synchronize data across devices with Amazon Cognito Sync user profiles across OS, devices, web State transition (link multiple accounts) Sync game state across OS, devices

70 Related sessions GAM401 MBL402 Serverless Mobile Game Development with Amazon Cognito, AWS Lambda, and Amazon DynamoDB Mobile Identity Management and Data Synchronization Using Amazon Cognito WRK202 Build a Scalable Mobile App on Serverless, EventTriggered, BackEnd Logic

71 Player Two Press Start

72 Multiplayer game servers API back-end app Core session Matchmaking S3 + CloudFront DLC, assets Game saves UGC Public server tier Direct client socket Scale on players Region

73 Multiplayer game servers Login via API Request matchmaking Get game server IP Region

74 Multiplayer game servers Login via API Request matchmaking Get game server IP Connect to server Pull down assets Other players join Region

75 Multiregion game servers Region B Region A

76 Related sessions GAM403 GAM404 GAM407 From 0 to 60 Million Player Hours in 400 Billion Star Systems Evolve: Hunting Monsters in a Low Latency Multiplayer Game on Amazon EC2 Quiplash: The Multiscreen, Multidevice, Multiplayer Game for 10,000

77 Wrap it up already Use Auto Scaling to save money Amazon CloudFront + Amazon S3 for download and upload Painful DIYDB? No! Use Amazon DynamoDB Dynamically manage game servers using the APIs Even multiregion!

78 Thank you! Mark Bate Steam: thevegemiteavenger PSN: nuclears0ck-de