From 1 to 10K with Ganglia and Nagios. Spike Morelli aka Space Linden

Transcription

1 From 1 to 10K with Ganglia and Nagios Spike Morelli aka Space Linden

2 About Second Life 3D Virtual World Not a game

3 About Second Life Built by Residents Textured Scripted Animated Owned

4 About Second Life Business Education Art & Design

5 About Second Life ~1M unique users/60 days ~77+ million concurrent scripts ~30K simulators ~2K square Km = ~10x Nuremberg

6 Our Infrastructure 6K nodes simulation grid (simulator, dataservice, memcache, s3asset_proxy, region presence) 600 infrastructure nodes (databases, im<-> , region presence, groupchat, messaging system, logging system, websites, caches, data warehousing...) 3 DataCentres running with a 4th being set up and another planned for Europe Integration of EC2 instances for development grids Size matters but services' complexity matters much more a 3D world is more complicated to run than a web farm

7 Monitoring? What do you think when people say 'Monitoring'?

8 Monitoring?

9 Monitoring?

10 Monitoring?

11 Monitoring?

12 Outage Big enough to get its own name tag...

13 Outage Big enough to get its own name tag... and change the game

14 Outage How long to find the root cause? Are you really understanding what's going on? Is that a symptom or a cause? Is something different from yesterday? From one hour ago? From five minutes ago? The bigger your infrastructure is, the harder is to answer those questions, the higher your chances are that something small will pass undetected, but have a devastating ripple effect which will be caught

15 Statistical Data They've done studies, you know. 60% of the time it works every time.

16 Statistical Data INSTRUMENT EVERYTHING^W A LOT Too much data can be confusing and misleading Monitoring is not just about detecting failures Trends Capacity planning Bottlenecks identification Identify bugs in your code (since the last commit mem usage doubled) When we say monitoring we think about systems, but software engineering has a LOT to benefit from it

17 Ganglia Ganglia to the rescue Almost zero configuration required for new nodes Design to scale from the ground up Agent based Can leverage multicast or multiple unicast targets for data redundancy Lightweight and smart use of network resources (could be better!) Plug in Nagios (performance data) into ganglia via gmetric Can support resolutions down to the second with good performance Support custom RRDs for higher data resolution

18 Ganglia

19 Ganglia

20 Ganglia

21 Ganglia Infrastructure 7K nodes * ~40 metrics = 280K metrics collected and stored (and plenty capacity left)

22 RRD Defaults are insufficient (hour, day, week, month, year) RRRcached (in trunk, more work needed, some security issues) Tmpfs + cron for backup + sync at boot Application Buffer-Cache Management for Performance: Running the World's Largest MRTG _html/index.html

23 Monitoring If you are thinking that there is a lot of duplication between Nagios and Ganglia you're right, but you should...

24 Monitoring If you are thinking that there is a lot of duplication between Nagios and Ganglia you're right, but you should......change the way you're thinking about monitoring! Don't think checks, think metrics Don't think if a service up, but rather how it's doing

25 Fault Detection with Nagios

26 WYSINWTS WYSINWTS What You See Is Not What They See

27 WYSINWTS External Nagios External generic http (or tcp) proxy CDN based 3rd party monitoring service

28 Needed Improvements Dashboards and data analysis tools TCO Monitoring tools for developers CEP

29 Code!

30 Thank You! Questions? Ideas? Contact me at Thanks for listening! Please feel free to say hi and chat with me about these topics during the conference!