System for Large Computing Clusters

Transcription

1 (Towards) A Scalable Monitoring System for Large Computing Clusters Moreno Marzolla marzolla@dsi.unive.it Web: Dip. Informatica, Università di Venezia

2 Talk Outline System Monitoring Definition Tassonomy Distributed Systems Monitoring Case Study: the BaBar Computing Farm ASC: A prototype of an SNMP Based monitoring application Conclusions Moreno Marzolla 2

3 Monitoring A monitor is a tool used to observe the activities on a system Collect performance statistics Analyze the data Display the results. Why? Identify frequently used portions of a program; Measure resource utilization to find performance bottlenecks; Characterize the Workload; Find model parameters, validate models, or develop inputs for a model. Moreno Marzolla 3

4 Monitor Classification Based on the level at which it is implemented Software Hardware Firmware Hybrid Based on the triggering mechanism Event driven Sampling based Based on the display ability On line Batch Moreno Marzolla 4

5 Hw versus Sw Monitors Input Rate Much higher on Hw monitors; Depends on CPU power for Sw ones. Time Resolution Higher on Hw monitors; Overhead None for Hw monitors. Can be an issue for Sw monitors. Portability Architecture and/or OS dependent? Moreno Marzolla 5

6 Event Driven vs Sampling Based Event Driven monitors collect data when triggered by some condition Pros: No polling required. Only "interesting" notifications are sent Cons: What happens when the triggering mechanism crashes? Sampling Based require periodic queries Pros: Can detect crashes of the monitored entity. Sampling rate can be changed "outside" the monitored system. Cons: Can overload the medium used to transport the requests Moreno Marzolla 6

7 Distributed Systems Monitoring To monitor a distributed system, the monitor needs itself to be (at least partly) distributed. Monitor Observer Observer Observer Observer Host 1 Host 2 Host 3 Host 4 Moreno Marzolla 7

8 Layered View of DS monitors Management Console Interpretation Presentation Analysis Collection Observation The entity deciding to change the parameters using the console Provides an interface to change the parameters and states The user who interprets the results User interface: provides reports, displays, alarms... Statistical routines to summarize the characteristics of the data Collects data from various observers Gathers raw data on individual components of the system Moreno Marzolla 8

9 Issues of DS monitoring Scalability Can it monitor 10 hosts? 100 hosts? 1000 hosts?... Latency Long delays make real time, on line monitoring a challenge Consistency Are you getting a consistent view of the system s status, despite notification delays? Moreno Marzolla 9

10 BaBar Case Study BaBar is a High Energy Physics experiment, studying matter antimatter asimmetry. Only a small fraction of events has real information ( ) Large amounts of data needed Large computing facilities are required Moreno Marzolla 10

11 BaBar INFN Padova 150 2xPIII 1.26GHz Machines, 1GB Ram, RH Linux 7.2 Tape Library with a capacity of ~70TB not compressed; Network switch, UPSes, Environmental conditioning system,... Moreno Marzolla 11

12 Monitoring Requirements Hardware Status Machine Crashes, disk crashes, CPU temperatures, disk/partitions overflows... Processes status Environmental conditions Humidity, Temperature, UPS status... System administrators should be notified as soon as a problem occurs Some automatic action should be taken when possibile (eg, shut down the machines if overheated). Moreno Marzolla 12

13 Other requirements The monitoring system should also be: Scalable Efficient (little resources requirement) Flexible and customizable Easy to configure Should be able to operate in batch mode (as a regular UNIX dæmon, no GUI) Should be able to observe different quantities with different granularities eg, CPU utilization sampled every 5 seconds, CPU temperature sampled every minute,... Moreno Marzolla 13

14 Some existing monitoring tools There are many of them: MRTG Ngop isd.fnal.gov/ngop/ Netsaint/Nagios Ganglia RemStats Cricket GxSNMP (add your own here) Moreno Marzolla 14

15 So, what s wrong? We examined some publicly available monitoring tools. Unfortunately, many of them were not suited to us: Not scalable Require their own dæmons running on the monitored hosts Can t install a dæmon on a network switch, or on a tape library Hard to configure In many cases we gave up without trying the program. Poorly implemented "...My God, it s full of scripting languages..." Moreno Marzolla 15

16 Architectural sketch User Interface User Interface User Interface Monitor Collector Collector Observer Observer Observer Observer Observer Hardware Hardware Hardware Hardware Hardware Moreno Marzolla 16

17 The Observer The Observer must be able to collect statistics on any networked equipment We decided to use SNMP (Simple Network Management Protocol) It is a well known protocol It is implemented by virtually every vendor It is reasonably simple yet powerful A very good open source implementation is available on Unix/Linux platforms snmp.sourceforge.net/ Moreno Marzolla 17

18 More on SNMP GetRequest Management Application GetNextRequest SetRequest SNMP UDP IP GetResponse Trap Network Protocol SNMP Messages LAN/WAN Managed Resources SNMP Managed Objects GetRequest GetNextRequest SetRequest GetResponse Trap SNMP UDP IP Network Protocol W. Stallings, SNMP, SNMPv2, SNMPv3 and RMON 1 and 2, 3rd edition, p. 81 Moreno Marzolla 18

19 The Collector/Monitor Is being developed Features: Asynchronous (nonblocking) parallelized SNMP Polling; XML based configuration file; The RRDTool package is used to store data and produce graphs Old data have lower resolution than recent ones. Round Robin Databases have known maximum size. Graphing capabilities are provided by the library. Dynamic generation of HTML pages using XSLT stylesheets. Moreno Marzolla 19

20 Architecture XML Configuration File <?xml version="1.0" Status Stylesheets XSLT Stylesheet 1 HTML Pages HTML Page 1 standalone="no"?> <!DOCTYPE monitor SYSTEM "monitor.dtd"> <monitor>... </monitor> ASC HTTPD XSLT Stylesheet 2 HTML Page 2 Host 1 Host 2 Host n Monitored Hosts Moreno Marzolla 20

21 Example of XML Configuration File <?xml version="1.0" standalone="no"?> <!DOCTYPE monitor SYSTEM "monitor.dtd"> <monitor numconnections="20" asclogfile="/monitor/asc.log" httpdlogfile="/dev/null" rrddir="/monitor" htmldir="/monitor/html" ascverbosity="3" > <host name="localhost"> <description>this machine</description> <miblist> <mib id="cpuuser" name=" " type="counter"> <archives> <rra cf="average" granularity="60" expire="604800"/> </archives> </mib> </miblist> <graphs> <! Graph definitions here > </graphs> </host> </monitor> Moreno Marzolla 21

22 Example of XML Status Dump <?xml version="1.0"?> <hosts> <host name="localhost" status="nr"> <mibs> <mib id="availswap" lastupdated=" "> </mib> <mib id="totalswap" lastupdated=" "> </mib> <mib id="totalmem" lastupdated=" "> </mib> <mib id="cachedmem" lastupdated=" "> </mib> <mib id="buffermem" lastupdated=" "> </mib> <mib id="sharedmem" lastupdated=" "> </mib> <mib id="freemem" lastupdated=" "> </mib> <mib id="cpusystem" lastupdated=" "> </mib> <mib id="cpuuser" lastupdated=" "> </mib> <mib id="tempcpu2" lastupdated=" "> </mib> <mib id="tempcpu1" lastupdated=" "> </mib> <mib id="tempmb" lastupdated=" "> </mib> </mibs> <graphs> <graph id="hourly.png" title="hourly data"/> </graphs> <notifications> <msg ts=" :11: " severity="critical"> Timeout </msg> </notifications> </host> </hosts> Moreno Marzolla 22

23 Sample HTML Output Moreno Marzolla 23

24 What happens if it does not scale? "Hierarchize" ASC? HTTPD Proxy HTTPD ASC HTTPD ASC SNMP Agent SNMP Agent SNMP Agent SNMP Agent Host 1 Host 2 Host 3 Host 4 Moreno Marzolla 24

25 Hierarchical Monitoring A hierarchical configuration of SNMP agents has already been shown to scale Rajesh Subramanyan, José Miguel Alonso and José A. B. Fortes, "A Scalable SNMP Based Distributed Monitoring System for Heterogeneous Network Computing", Proc. SC2000. Dallas, Texas, Nov What about fault tolerance? "Well written applications do not crash" is right, but machines running them crash anyway... Well known leader election techniques could be applied Redundant monitoring? Moreno Marzolla 25

26 Redundancy for fault tolerance Additional collectors could be used to ensure that at least K out of N of them are sufficient to cover the whole system Collector 1 Collector 1 Host 1 Host 2 Host 3 Host 4 Host 5 Host 6 Host 7 Host 8 Host 9 Collector 1 Any 2 collectors monitor all the hosts Moreno Marzolla 26

27 Conclusions Monitoring a large computing cluster is a highly nontrivial task. Many available monitoring tools exist, but many of them are not adequate for large distributed systems. We are trying to build a general purpose SNMP and XML based monitoring tool. A prototype exists and is working well...but see next slide Moreno Marzolla 27

28 Future work Alarms are not currently implemented, but are at the top position of the to do list ASC is running on a small ( 20 machines) test farm, while the bigger farm is being installed. ASC is expected to scale; however, serious scalability experiments will be done as soon as all the machines are up and running. Could it be extended for monitoring Grids? Moreno Marzolla 28

29 Bibliography W. Stallings, SNMP, SNMPv2, SNMPv3 and RMON 1 and 2, third edition, Addison Wesley, 1999 R. Jain, The art of computer systems performance analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley and Sons, 1991 Rajesh Subramanyan, José Miguel Alonso and José A. B. Fortes, A Scalable SNMP Based Distributed Monitoring System for Heterogeneous Network Computing, Proc. SC2000. Dallas, Texas, Nov Grid Performance Working Group, didc.lbl.gov/gridperf/ M. Bearden and R. Bianchini Jr., Efficient and Fault Tolerant Distributed Host Monitoring using System Level Diagnosis, Proceedings of the IFIP/IEEE International Conference on Distributed Platforms, Dresden, Germany, pp , February Flaviu Cristian, Understanding fault tolerant distributed systems, Comm. ACM 34(2), 1991 Moreno Marzolla 29