Hadoop JMX Monitoring and Alerting

Transcription

1 Hadoop JMX Monitoring and Alerting Introduction High-Level Monitoring/Alert Flow Metrics Collector Agent Metrics Storage NameNode Metrics DataNode Metrics HBase Master Metrics RegionServer Metrics Data Retention Monitoring Dashboard & Alerting Metrics Dashboard Overview Dashboard Chart Introduction Hadoop slave node will regularly emit some metrics information to reflect the service healthy, service team will look the metrics to understand if the service is in healthy state, and trace back to understand the history behavior. Some typical use cases are: Pre-caution for un-healthy HBase RegionServer (the heap usage), RPC handling metrics and region aliveness, etc. Troubleshooting through the metrics history dashboard NameNode RPC traffic from client is very high, identify the source of the client, grep the user from audit log as well User can flexibly set threshold for each monitored metric and get alert notification w/o re-writing or create policies from scratch Notification about the HDFS clients which generating abnormal RPC traffic Extract the DN/RS list with abnormal RPC processing time High-Level Monitoring/Alert Flow Metrics Collector Operation team is always struggling with the metrics monitoring for HBase cluster, e.g. HBase RegionServer heap usage, the RPC handling metrics for RegionServer, and region aliveness in regionserver. So we need solutions to get all those metrics. One option is to deploy a standalone JMX client in each node; another is to add JMX sink in Hadoop s metrics system. A JMX client is expected to be deployed at each RegionServer slave node, or deployed at a collection of nodes to retrieve the JMX information. Since we have thousands of slave nodes, it s not suppose to deploy those clients in a single server, which may lead to heavy load on that machine.

2 JMX sink should be developed according to Hadoop s metrics interface and plug into the Hadoop runtime environment We tend to write the data into Kafka as a distributed caching layer to decouple the JMX client and the back-end storage system, and to avoid the storing latency of JMX data as well. Agent If we tend to build the JMX client to collect JMX metrics, we d better have an agent to monitor whether the JMX client is working well, otherwise there may be some JMX data loss if we have some JMX clients stop working. If we use JMX sink to collect data, no agent is required for JMX client, the data collection lifecycle is the same as the daemon lifecycle. Metrics Storage A scalable backend for large scale metrics information storage, as well as the query engine for time-series data, along with the min/max/average aggregation semantics. NameNode Metrics Bean Bean Name Property Description Metric Name Memory java.lang:type=memory HeapMemoryUsage - used hadoop.memory.heapmemoryusa NonHealMemoryUsage - used hadoop.memory.nonheapmemory Name System State Hadoop:service=NameNode,name=FSNamesystem CapacityTotal hadoop.namenode.fsnamesystem CapacityUsed hadoop.namenode.dfs.capacityus CapacityRemaining hadoop.namenode.dfs.capacityre BlocksTotal hadoop.namenode.dfs.blockstota FilesTotal hadoop.namenode.dfs.filestotal UnderReplicatedBlocks hadoop.namenode.dfs.underrepli MissingBlocks hadoop.namenode.dfs.missingblo CorruptBlocks hadoop.namenode.dfs.corruptblo LastCheckpointTime hadoop.namenode.dfs.lastcheckp TransactionsSinceLastCheckpoint hadoop.namenode.dfs.transaction LastWrittenTransactionId hadoop.namenode.dfs.lastwritten SnapshottableDirectories hadoop.namenode.dfs.snapshotta Snapshots hadoop.namenode.dfs.snapshots RPC Hadoop:service=NameNode,name=RpcActivityForPort8020 RpcQueueTimeAvgTime hadoop.namenode.rpc.rpcqueuet RpcProcessingTimeAvgTime hadoop.namenode.rpc.rpcproces NumOpenConnections hadoop.namenode.rpc.numopenc CallQueueLength hadoop.namenode.rpc.callqueuel DataNode Metrics

3 Bean Bean Name Property Description Memory java.lang:type=memory NonHeapMemoryUsage - used HeapMemoryUsage = used hadoop.memory.nonh hadoop.memory.heap General Hadoop:service=DataNode,name=FSDatasetState-bb8ac17a-d75b-4aab-9f9e-0ec1ef2d58f4 Capacity hadoop.datanode.fsda DfsUsed hadoop.datanode.fsda Hadoop:service=DataNode,name=DataNodeInfo XceiverCount hadoop.datanode.dat RPC Hadoop:service=DataNode,name=RpcActivityForPort50020 RpcQueueTimeAvgTime hadoop.datanode.rpc RpcProcessingTimeAvgTime NumOpenConnections CallQueueLength hadoop.datanode.rpc hadoop.datanode.rpc hadoop.datanode.rpc HBase Master Metrics Bean Bean Name Property Description Memory java.lang:type=memory NonHeapMemoryUsage - used General Hadoop:service=HBase,name=Master,sub=Server averageload HeapMemoryUsage - used Hadoop:service=HBase,name=Master,sub=AssignmentManger ritcount Counts the number of regions in transition Hadoop:service=HBase,name=Master,sub=AssignmentManger ritcountoverthreshold Counts the number of regions in transition that exceed the threshold as defined by the property rit.metrics.threshold.time Region Assignment Hadoop:service=HBase,name=Master,sub=AssignmentManger Assign_num_ops Assign_min Assign_max Assign_75th/95th/99th/99.9th_percentile BulkAssign_num_ops BulkAssign_min BulkAssign_max BulkAssign_75th/95th/99th/99.9th_percentile Balancer Hadoop:service=HBase,name=Master,sub=Balancer BalancerCluster_num_ops BalancerCluster_min BalancerCluster_max BalancerCluster_75th/95th/99th/99.9th_percentile

4 Split Hadoop:service=HBase,name=Master,sub=FileSystem HlogSplitTime_min HlogSplitTime_max HlogSplitTime_75th/95th/99th/99.9th_percentile HlogSplitSize_min/max MetaHlogSplitTime_min/max MetaHlogSplitTime_75th/95th/99th/99.9th_percentile MetaHlogSplitSize_min/max RegionServer Metrics Bean Bean Name Property Descriptio n Metric Name Memory java.lang:type=memory NonHeapMemoryUsage - used hadoop.memory.nonheapmemoryusag HeapMemoryUsage - used hadoop.memory.heapmemoryusage.us Java Direct Memory java.nio:type=bufferpool,name=direct MemoryUsed Java Direct Memory Used hadoop.bufferpool.direct.memoryused JVM Metrics Hadoop:service=HBase,name=JvmMetrics GcCount hadoop.hbase.jvm.gccount GcTimeMillis hadoop.hbase.jvm.gctimemillis IPC Hadoop:service=HBase,name=RegionServer,sub=IPC queuesize hadoop.hbase.regionserver.ipc.queues NumCallsInGeneralQueue hadoop.hbase.regionserver.ipc.numca NumActiveHandler hadoop.hbase.regionserver.ipc.numac QueueCallTime_99th_percentile ProcessCallTime_99th_percentile IPC Queue Time (99th) IPC Process Time (99th) hadoop.hbase.regionserver.ipc.queuec hadoop.hbase.regionserver.ipc.proces QueueCallTime_num_ops hadoop.hbase.regionserver.ipc.queuec ProcessCallTime_num_ops hadoop.hbase.regionserver.ipc.proces Regions Hadoop:service=HBase,name=RegionServer,sub=Server regioncount hadoop.hbase.regionserver.server.reg storecount hadoop.hbase.regionserver.server.stor memstoresize hadoop.hbase.regionserver.server.mem storefilesize hadoop.hbase.regionserver.server.stor totalrequestcount hadoop.hbase.regionserver.server.tota

5 ReadRequestCount hadoop.hbase.regionserver.server.rea WriteRequestCount hadoop.hbase.regionserver.server.writ splitqueuelength hadoop.hbase.regionserver.server.s compactionqueuelength hadoop.hbase.regionserver.server.com flushqueuelength hadoop.hbase.regionserver.server.flus blockcachesize hadoop.hbase.regionserver.server.bloc blockcachehitcount hadoop.hbase.regionserver.server.bloc blockcachecounthitpercent hadoop.hbase.regionserver.server.bloc Data Retention Metrics should be collected at least 1 minute interval (Hadoop emits the metrics at 10 secs interval). Aggregate to 5 minute level for data older than 30 days and keep half year. Monitoring Dashboard & Alerting Metrics Dashboard Overview

6 Dashboard Chart

7 Generally, we will follow the UI layout in Ambari, within that, the service health check application will also be included in service status and summary information. Metrics Query Pattern: 1. Flexibly change the time range from 1 hour to