Monitoring with Ganglia

Size: px

Start display at page:

Download "Monitoring with Ganglia"

Felicia Laura Moody
5 years ago
Views:

1 Monitoring with Ganglia Matt Massie, Bernard Li, Brad Nicholes, and Vladimir Vuksan Beijing Cambridge O'REILLY* Farnham Koln Sebastopol Tokyo

2 Table of Contents Preface ix 1. Introducing Ganglia 1 It's a Problem of Scale 1 Hosts ARE the Monitoring System 2 Redundancy Breeds Organization 3 Is Ganglia Right for You? 4 gmond: Big Bang in a Few Bytes 4 gmetad: Bringing It All Together 7 gweb: Next-Generation Data Analysis 8 But Wait! That's Not All! 9 2. Installing and Configuring Ganglia 11 Installing Ganglia 11 gmond 11 gmetad 14 gweb 16 Configuring Ganglia 20 gmond 20 gmetad 33 gweb 38 Postinstallation 41 Starting Up the Processes 41 Testing Your Installation 41 Firewalls Scalability 43 Who Should Be Concerned About Scalability? 43 gmond and Ganglia Cluster Scalability 43 gmetad Storage Planning and Scalability 44 RRD File Structure and Scalability 44 Hi

3 Acute IO Demand During gmetad Startup 46 gmetad IO Demand During Normal Operation 46 Forecasting IO Workload 47 Testing the IO Subsystem 48 Dealing with High IO Demand from gmetad The Ganglia Web Interface 53 Navigating the Ganglia Web Interface 53 The gweb Main Tab 53 Grid View 53 Cluster View 54 Host View 58 Graphing All Time Periods 58 The gweb Search Tab 60 The gweb Views Tab 60 The gweb Aggregated Graphs Tab 63 Decompose Graphs 64 The gweb Compare Hosts Tab 64 The gweb Events Tab 64 Events API 66 The gweb Automatic Rotation Tab 67 The gweb Mobile Tab 67 Custom Composite Graphs 67 Other Features 69 Authentication and Authorization 70 Configuration 70 Enabling Authentication 70 Access Controls 71 Actions 72 Configuration Examples Managing and Extending Metrics 73 gmond: Metric Gathering Agent 73 Base Metrics 75 Extended Metrics 77 Extending gmond with Modules 78 C/C++ Modules 79 Mod_Python 89 Spoofing with Modules 96 Extending gmond with gmetric 97 Running gmetric from the Command Line 97 Spoofing with gmetric 99 How to Choose Between C/C++, Python, and gmetric 100 iv Table ofcontents

4 XDR Protocol 101 Packets 102 Implementations 103 Java and gmetric4j 103 Real World: GPU Monitoring with the NVML Module 104 Installation 104 Metrics 105 Configuration Troubleshooting Ganglia 107 Overview 107 Known Bugs and Other Limitations 107 Useful Resources 108 Release Notes 108 Manpages 108 Wiki 108 IRC 108 Mailing Lists 108 Bug Tracker 109 Monitoring the Monitoring System 109 General Troubleshooting Mechanisms and Tools 110 netcat and telnet 110 Logs 114 Running in Foreground/Debug Mode 114 strace and truss 115 valgrind: Memory Leaks and Memory Corruption 116 iostat: Checking IOPS Demands of gmetad 116 Restarting Daemons 117 gstat 117 Common Deployment Issues 119 Reverse DNS Lookups 119 Time Synchronization 119 Mixing Ganglia Versions Older than 3.1 with Current Versions 119 SELinux and Firewall 120 Typical Problems and Troubleshooting Procedures 120 Web Issues 120 gmetad Issues 125 rrdcached Issues 126 gmond Issues Ganglia and Nagios 129 Sending Nagios Data to Ganglia 130 Monitoring Ganglia Metrics with Nagios 133 Tableof Contents v

5 Principle of Operation 134 Check Heartbeat 135 Check a Single Metric on a Specific Host 135 Check Multiple Metrics on a Specific Host 136 Check Multiple Metrics on a Range of Hosts 136 Verify that a Metric Value Is the Same Across a Set of Hosts 137 Displaying Ganglia Data in the Nagios UI 138 Monitoring Ganglia with Nagios 139 Monitoring Processes 139 Monitoring Connectivity 140 Monitoring cron Collection Jobs 140 Collecting rrdcached Metrics Ganglia and sflow 143 Architecture 145 Standard sflow Metrics 147 Server Metrics 147 Hypervisor Metrics 149 Java Virtual Machine Metrics 150 HTTP Metrics 151 memcache Metrics 153 Configuring gmond to Receive sflow 155 Host sflow Agent 157 Host sflow Subagents 158 Custom Metrics Using gmetric 160 Troubleshooting 161 Are the Measurements Arriving at gmond? 161 Are the Measurements Being Sent? 165 Using Ganglia with Other sflow Tools Ganglia Case Studies 171 Tagged,Inc. 172 Site Architecture 172 Monitoring Configuration 173 Examples 175 SARA 180 Overview 180 Advantages 181 Customizations 183 Challenges 184 Conclusion 186 Reuters Financial Software 186 Ganglia in the QA Environment 186 vi Table of Contents

6 Ganglia in a Major Client Project 188 Lumicall (Mobile VoIP on Android) 190 Monitoring Mobile VoIP for the Enterprise 191 Ganglia Monitoring Within Lumicall 191 Implementing gmetric4j Within Lumicall 192 Lumicall: Conclusion 194 Wait, How Many Metrics? Monitoring at Quantcast 195 Reporting, Analysis, and Alerting 196 Ganglia as an Application Platform 198 Best Practices 199 Tools 199 Drawbacks 200 Conclusions 202 Many Tools in the Toolbox: Monitoring at Etsy 202 Monitoring Is Mandatory 202 A Spectrum of Tools 203 Embrace Diversity 204 Conclusion 204 A. Advanced Metric Configuration and Debugging 205 B. Ganglia and Hadoop/HBase 215 Index 221 Tableof Contents vii

Programming Windows Azure

Programming Windows Azure Sriram Krishnan O'REILLY* Beijing Cambridge Farnham Koln Sebastopol Taipei Tokyo Table of Contents Preface xiii 1. Cloud Computing 1 Understanding Cloud Computing 1 History of