Big Data Infrastructure at Spotify

Transcription

1 Big Data Infrastructure at Spotify Wouter de Bie Team Lead Data Infrastructure September 26, 2013

2 2 Who am I? According to ZDNet: "The work they have done to improve the Apache Hive data warehouse system also aligns well with our needs, as we use Hive extensively for ad-hoc queries and for the analysis of large datasets," de Brie said in a statement.

3 3 Who am I?

4 4 Why data? We play music, right?

5 5 Why data? We were the first company to do free music streaming. But now everybody can do it, so we need to learn quickly to iterate faster!

6 6 Agenda Let s talk about Data Infrastructure, how we did it, what we learned and how we ve failed Some Context Use Cases Our Infrastructure Lesson learned

7 7 Some Context Spotify started in 2006 (in Sweden) Now employees, 450+ engineers 26 million monthly active users 20+ million tracks available 4 data centers across the globe 17 data engineers building a platform for easy access to data

8 8 Reporting Business Analytics Operational Analytics Product features Use Cases We re a data-driven company, so data is used almost everywhere

9 Reporting Reporting to labels, licensors, partners and advertisers We support our partners

10 Business Analytics Analyzing growth, user behavior, sign-up funnels, etc Company KPIs A/B testing NPS analysis Segmentation analysis Ad performance

11 11 Listening behavior in Sweden 0.00%$ 0.20%$ 0.40%$ 0.60%$ 0.80%$ 1.00%$ 1.20%$ 1.40%$ 1.60%$ Mon$-$00$ Mon$-$02$ Mon$-$04$ Mon$-$06$ Mon$-$08$ Mon$-$10$ Mon$-$12$ Mon$-$14$ Mon$-$16$ Mon$-$18$ Mon$-$20$ Mon$-$22$ Tue$-$00$ Tue$-$02$ Tue$-$04$ Tue$-$06$ Tue$-$08$ Tue$-$10$ Tue$-$12$ Tue$-$14$ Tue$-$16$ Tue$-$18$ Tue$-$20$ Tue$-$22$ Wed$-$00$ Wed$-$02$ Wed$-$04$ Wed$-$06$ Wed$-$08$ Wed$-$10$ Wed$-$12$ Wed$-$14$ Wed$-$16$ Wed$-$18$ Wed$-$20$ Wed$-$22$ Thu$-$00$ Thu$-$02$ Thu$-$04$ Thu$-$06$ Thu$-$08$ Thu$-$10$ Thu$-$12$ Thu$-$14$ Thu$-$16$ Thu$-$18$ Thu$-$20$ Thu$-$22$ Fri$-$00$ Fri$-$02$ Fri$-$04$ Fri$-$06$ Fri$-$08$ Fri$-$10$ Fri$-$12$ Fri$-$14$ Fri$-$16$ Fri$-$18$ Fri$-$20$ Fri$-$22$ Sat$-$00$ Sat$-$02$ Sat$-$04$ Sat$-$06$ Sat$-$08$ Sat$-$10$ Sat$-$12$ Sat$-$14$ Sat$-$16$ Sat$-$18$ Sat$-$20$ Sat$-$22$ Sun$-$00$ Sun$-$02$ Sun$-$04$ Sun$-$06$ Sun$-$08$ Sun$-$10$ Sun$-$12$ Sun$-$14$ Sun$-$16$ Sun$-$18$ Sun$-$20$ Sun$-$22$ SE$23-27$ SE$45-59$ SE$$0-17$ SE$35-44$ SE$60-150$ SE$28-34$ SE$18-22$

12 12 Listening behavior in Spain 0.00%$ 0.20%$ 0.40%$ 0.60%$ 0.80%$ 1.00%$ 1.20%$ 1.40%$ Mon$-$00$ Mon$-$02$ Mon$-$04$ Mon$-$06$ Mon$-$08$ Mon$-$10$ Mon$-$12$ Mon$-$14$ Mon$-$16$ Mon$-$18$ Mon$-$20$ Mon$-$22$ Tue$-$00$ Tue$-$02$ Tue$-$04$ Tue$-$06$ Tue$-$08$ Tue$-$10$ Tue$-$12$ Tue$-$14$ Tue$-$16$ Tue$-$18$ Tue$-$20$ Tue$-$22$ Wed$-$00$ Wed$-$02$ Wed$-$04$ Wed$-$06$ Wed$-$08$ Wed$-$10$ Wed$-$12$ Wed$-$14$ Wed$-$16$ Wed$-$18$ Wed$-$20$ Wed$-$22$ Thu$-$00$ Thu$-$02$ Thu$-$04$ Thu$-$06$ Thu$-$08$ Thu$-$10$ Thu$-$12$ Thu$-$14$ Thu$-$16$ Thu$-$18$ Thu$-$20$ Thu$-$22$ Fri$-$00$ Fri$-$02$ Fri$-$04$ Fri$-$06$ Fri$-$08$ Fri$-$10$ Fri$-$12$ Fri$-$14$ Fri$-$16$ Fri$-$18$ Fri$-$20$ Fri$-$22$ Sat$-$00$ Sat$-$02$ Sat$-$04$ Sat$-$06$ Sat$-$08$ Sat$-$10$ Sat$-$12$ Sat$-$14$ Sat$-$16$ Sat$-$18$ Sat$-$20$ Sat$-$22$ Sun$-$00$ Sun$-$02$ Sun$-$04$ Sun$-$06$ Sun$-$08$ Sun$-$10$ Sun$-$12$ Sun$-$14$ Sun$-$16$ Sun$-$18$ Sun$-$20$ Sun$-$22$ ES$$0-17$ ES$35-44$ ES$23-27$ ES$45-59$ ES$60-150$ ES$28-34$ ES$18-22$

13 13 Impact of hurricane Sandy 29 October 2012

14 14 Impact of hurricane Sandy 30 October 2012

15 Operational metrics Root cause analysis Latency analysis Better capacity planning (servers, people, bandwidth)

16 Product features Radio Top lists Recommendations (better than external parties, because of the amount of data)

17 Everybody should be able to use data! September 26, 2013

18 18 So why Data Infrastructure? View Backend Data pipe Analysis

19 19 Our data infrastructure

20 20 Some geeky numbers 1 TB of compressed data from users per day 400 GB of data from services per day 61 TB of data generated in Hadoop each day 328 node Hadoop cluster 6500 jobs/day ( /month) Soon 690 nodes (11040 cores) 10 PB of storage capacity (soon 28 PB)

21 21 Spotify s data infrastructure Backend services Product features HDFS Map/Reduce Reporting Operational Databases Luigi Scheduler Hive Analytical databases Map/Reduce Jobs Dashboards

22 22 The thee pillars of our Data Infrastructure Kafka Collection Hadoop Processing Databases Analytics/Visualization

23 Data collection September 26, 2013

24 24 18 Data collection Kafka: High volume pub-sub system Started with a store-and-forward system Evaluated Apache Flume Currently from Backend-to-HDFS, but in the future Backend-to-Backend It was almost a good fit, but...

25 25 Guaranteed delivery Kafka doesn t provide message acknowledgements..... at least, not in 0.7 (stable) 0.8 has ACKs, but no end-to-end acknowledgements A track streamed =~ monetary transaction

26 26 Hacking Kafka acknowledgements HDFS Backend service Kafka broker Kafka client S3 Tape

27 27 Dumping databases Not only do we have log files, but also production databases Using Sqoop for dumping PostgreSQL Map/Reduce job (table scans) for dumping Cassandra hdfs2cass for uploading SSTables into Cassandra For large DB s we parse application logs and only dump deltas

28 28 Failures (or the hard stuff) We started with a store-and-forward system that didn t scale Kafka has multiple components (client, broker, ZooKeeper) Internet weather As with many large Java systems: Garbage collection

29 Hadoop: our trusted elephant September 26, 2013

30 30 Scheduling We wrote and open-sourced our own scheduler: Luigi Nothing suitable out there.. (unless you really, really like the XML hell of Oozie) Written in Python Generic scheduler and dependency system that supports Python and Java M/R, Pig, Hive and Sqoop

31 31 Map/Reduce languages Python with Hadoop Streaming Pros: fast development, many Spotify libraries available Cons: slower then Java, no access to Hadoop API Java Pros: fast, access to Hadoop API Cons: verbose language, not many Spotify libraries available PIG Pros: very small scripts, faster then streaming Cons: yet another language to learn, not many Spotify libs available Hive Pros: SQL like syntax (easy for non-programmers) and relational data model Cons: more moving parts (not well suited for a whole pipe line)

32 32 Scaling Hadoop at Spotify Our Journey Started with a small (scrap metal) cluster of 37 servers Moved to Amazon Elastic Map/Reduce (EMR) and S3 to quickly scale Built an in-house cluster of 60 nodes because of EMR costs Capacity planning every 6 months, grown to 327 nodes today 363 more waiting to be provisioned Put in place data-retention policy and data archive

33 33 Hadoop failures We just need good developers - No, we need Hadoop experts We underestimated the complexity of Hadoop You can throw money at the problem of scaling, but at our scale, it pays off to optimize Give people easy tools early on

34 34 Lessons learned 4+ years of Hadoop taught us Hadoop has brought us very far. We would never be able to handle the current volume with a cheap RDBMS Commodity hardware doesn t mean cheap hardware Hadoop isn t a silver bullet Hadoop is a complex system that needs love and care You will have to extend Hadoop (and eco-system components) to tailor it to your needs

35 Databases and visualization September 26, 2013

36 36 Databases: used for aggregates Aggregates from Hadoop are put into PostgreSQL or Cassandra PostgreSQL powering dashboards and empowering analysts Cassandra for columnar sets (Spotify Analytics for labels) Databases are used for low-latency access by systems or analysts

37 37 Spotify Data Warehouse

38 38 Spotify Data Warehouse

39 39 Spotify Analytics Dashboards

40 40 Spotify Analytics

41 41 Spotify Analytics: Daft Punk

42 42 Spotify Analytics: Whitney Houston

43 43 Open Source at Spotify Luigi ( Snakebite ( hdfs2cass (

44 Questions? September 26, 2013

45 45 Lessons learned Data is crucial for our business Hadoop and other cutting-edge technology work pretty well for us Hiring technical analysts was a good idea! There is no one-size-fits-all data product Spotify has the build it ourselves mindset. Sometimes it s better to buy then build

46 Want to join the band? Check out for more information. Or mail: Or September 26, 2013