Spotify. Scaling storage to million of users world wide. Jimmy Mårdell October 14, 2014

Transcription

1 Spotify Scaling storage to million of users world wide! Jimmy Mårdell <yarin@spotify.com> October 14, 2014

2 2 About me Jimmy Mårdell Tech Product Owner in the Cassandra team 4 years at Spotify

3 Section name 3 Contents Understanding the problem Cassandra Future challenges

4 Understanding the problem 4

5 Understanding the problem 5 Data at Spotify Binary data files MP3 files, cover art Static structured data Updated once a day Search indexes, personal music suggestions Ephemeral data Caches, short-lived data What s happening now Read-write persistent structured data User accounts, playlists, notifications, social network, radio stations etc

6 Understanding the problem 6 Relational databases Master-slave setups Pros: more read capacity Cons: same write capacity, replication delays, still SPoF Sharding Pros: more read and write capacity Cons: more complexity, data integrity issues, many more SPoF s Both All pros and cons of the above

7 Understanding the problem 7 Distributed K/V-stores More simple data model, but often good enough Easier to scale out Many options today: Cassandra HBase MongoDB Redis Riak

8 Understanding the problem 8 Why is this a hard problem? Hardware failures Slow machines Network partitions Data consistency CAP-theorem

9 Understanding the problem 9 The Dynamo paper DynamoDB - Amazon s K/V store Paper released in 2007 Simple but powerful distribution mechanism

10 Cassandra 10

11 Cassandra 11 What is Cassandra? Open source distributed database (Apache project) More than just a K/V-store Built-in sharding Asynchronous masterless replication No Single Point of Failure! Extremely robust Other prominent users: Apple, Netflix, Instagram

12 Cassandra 12 Cassandra at Spotify Began using it in 2010 Now 500+ nodes across 50+ clusters First database choice when developing a new system

13 Cassandra 13 Cassandra architecture Combining two technologies:! Amazon Dynamo : How the data is replicated among nodes Google BigTable : How data is stored locally on node

14 Cassandra 14 Data distribution Nodes are assigned integer tokens All nodes know about each other Gossip hash(key) determines location of data Replication to succeeding nodes

15 Cassandra 15 Write requests Client oblivious about tokens Coordinator node sends write to replica nodes Customizable consistency level

16 Cassandra 16 Read requests Coordinator sends read to some replica nodes Consistency level Latest data wins Asynchronous synchronization

17 Cassandra 17 Consistency levels Determines how many replicas to read/write to ONE - fast, fault tolerant, least consistency guarantee ALL - slow, no fault tolerance, best consistency guarantee QUORUM - majority, good compromise Specified per request

18 Cassandra 18 Consistency levels Read ONE, Write ALL Read ALL, Write ONE Read QUORUM, Write QUORUM Read ONE, Write ONE Fault tolerant on write Fault tolerant on read Read always gets latest data NO YES YES YES YES NO YES YES YES YES YES NO

19 Cassandra 19 Fault tolerance Cluster typically fully operational with one node down Depends on consistency levels Eventual consistency Hinted Handoffs Read-repair Anti-Entropy repair

20 Cassandra 20 Multiple data centers One cluster - multiple rings All DC s contain same data

21 Cassandra 21 Multi-DC consistency Network partitioning is a problem LOCAL_QUORUM and EACH_QUORUM Consistency within DC, not to other DC s Usually good enough

22 Future challenges 22

23 Future challenges 23 Cassandra automation tools Every feature uses it s own Cassandra cluster Decoupling is the key to scalability Tools to automate common operations Setup new clusters Upgrade clusters Replace broken hardware Schedule repairs and compactions Backup/restore

24 Future challenges 24 Recovery after network split Important to ensure eventual consistency Old data may resurface! Anti-entropy repair slow and potentially risky Merkle-tree calculation Low granularity causes overstreaming Causes more problems than one would think

25 Future challenges 25 Improving Cassandra Performance Code Configuration GC tuning New features vnodes CQL Light-weight transactions Incremental repairs

26 Future challenges 26 Many data centers Cassandra scales linearly within a data center Not so well to many datacenters Waste of machines Repair an increasing problem Do we really need to store all data everywhere? Geolocated data

27 Thanks for listening!! Questions?! Jimmy Mårdell, October 14, 2014