Cassandra Integration User Guide

Transcription

1 Cassandra Integration User Guide Functional Area: Cassandra Integration Geneos Release: v4.7 Document Version: v1.0.0 Date Published: 24 May 2018

2 Copyright ITRS Group Ltd. All rights reserved. Information in this document is subject to change without notice. The software described in this document is furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in accordance with the terms of those agreements. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of ITRS Group Ltd. ITRS Group Ltd 6th Floor, The Bonhill Building, 15 Bonhill Street, London, EC2A 4DN, UK t: +44 (0) f: +44 (0)

3 Table of Contents Cassandra Monitoring 4 General Information 4 Overview 4 User Requirements 4 System Requirements 5 Audience 5 Install And Set Up 6 Set Up The Samplers 9 Set Up The Variables 9 Set Up The Rules 11 Technical Reference 13 Cassandra Monitoring Dataviews 13 Cassandra Disk Usage 13 MBeans For Cassandra-DiskUsage 13 Cassandra Errors 14 MBeans For Cassandra-Errors 14 Cassandra GC 15 MBeans For Cassandra-GC 15 Cassandra Latency 16 MBeans For Cassandra-Latency 16 Cassandra Tasks 17 MBeans For Cassandra-Tasks 17 Cassandra Throughput 18 MBeans For Cassandra-Throughput 18

4 Cassandra Monitoring General information Overview Cassandra monitoring is a Gateway configuration file that enables monitoring of Cassandra through a set of samplers with customised JMX plug-in settings. Apache Cassandra is a free and open-source distributed NoSQL database management system that provides scalability and high-availability. Some of Cassandra's key attributes are: Fault tolerant - Data is automatically replicated to multiple nodes for fault-tolerance. Decentralized - There are no single points of failure. Elastic - Read and write throughput increase linearly as new machines are added, with no downtime or interruption to applications. It is important to monitor Cassandra performance to identify database slowdowns, interruptions, or pressing resource limitations - and take quick and appropriate actions to correct them. User requirements This monitoring template is a Gateway configuration set-up that can be included in the Gateway Setup Editor. To use this template, your configuration must meet the following requirements: A machine running the Netprobe must have access to the Cassandra published HTTP address and port. An Active Console 2 that is connected to the Gateway. Cassandra configuration files extracted to their Gateway set-up. Specify a Managed entity for each monitored node in the Cassandra Cluster. A JMX-enabled Cassandra Cluster. Page 4 of 18

5 System requirements The following requirements must be met prior to the installation and set-up of the template: Template package: geneos-integration-cassandra-<version>.zip. Managed entities utilising the samplers defined in the include/cassandramonitoring.xml. Netprobe version 4.5 or higher. Note: The template is verified to be working with Cassandra version Audience This document is a reference guide to introduce templates and scripts built using the Geneos development toolkit plug-ins. The template allows you to integrate specific applications and services to collect metrics without having to create a new plug-in. Page 5 of 18

6 Install and set up Ensure that you have read and can follow the system requirements prior to installation and set-up of this integration template. 1. Download the config file (geneos-integration-cassandra-<version>.zip) from the ITRS Downloads site. 2. Open Active Console Extract the Cassandra monitoring configuration file into the Gateway Setup directory. 4. Click Includes to create a new file in the Navigation panel. 5. Enter the location of the file to include in the Location field. In this example, use the include/cassandramonitoring.xml: Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file. 6. Expand the file location in the Includes section. 7. Select Click to load Click Yes to load the new Cassandra include file. 9. Click Managed entities in the Navigation panel. 10. Add the Cassandra type to the Managed Entity section that you will use to monitor Cassandra. Page 6 of 18

7 11. Click the Validate button to check your configuration. The Validate button allows you to check if there are any errors or warnings in your configuration set-up. Page 7 of 18

8 Once the Gateway configuration appears in the Includes section, you can add the samplers and other variables. Page 8 of 18

9 Set up the samplers These are the pre-configured samplers available to use in CassandraMonitoring.xml. Configure the required fields by referring to the table below: Samplers Cassandra-GC Cassandra-Throughput Cassandra-Latency Cassandra-DiskUsage Cassandra-Errors Cassandra-Tasks Set up the variables The CassandraMonitoring.xml template provides the variables that are set in the Environments section: Variable CASSANDRA_JMX_PORT Description Cassandra host name. Default: localhost CASSANDRA_JMX_HOST Cassandra JMX port. Default: 7199 CASSANDRA_MONITORING_GROUP_NAME Sampler group name. Default: Cassandra Page 9 of 18

10 After checking and saving the changes, the samplers you have set in the Gateway configuration display in Active Console 2. Page 10 of 18

11 Set up the rules The CassandraMonitoring-SampleRules.xml template also provides a separate sample rules that you can use to configure the Gateway Setup Editor. Your configuration rules must be set in the Includes section. 1. Enter the location of the file to include in the Location field. In this example, use the include/cassandramonitoring-samplerules.xml: Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file. 2. Expand the file location in the Includes section. 3. Select Click to load Click Yes to load the new Cassandra include rules file. 5. Click Rules in the Navigation panel to create new rules. Sample Rules Errors - Read Unavailable Description Sets the severity to critical if the number of read unavailable exceptions exceeds CASSANDRA_ERRORS_READ_UNAVAILABLE_THERESHOLD. Default: Threshold is set to 0 Page 11 of 18

12 Sample Rules Errors - Write Unavailable Description Sets the severity to critical if the number of write unavailable exceptions exceeds CASSANDRA_ERRORS_WRITE_UNAVAILABLE_THERESHOL. Default: Threshold is set to 0 Disk Usage - High Load Throughput - High Read Throughput Throughput - High Write Throughput GC - ParNew - High Last GC Duration GC - CMS - High Last GC Duration Sets the severity to critical if the storage load exceeds CASSANDRA_DISK_USAGE_HIGH_ LOAD_THRESHOLD. The threshold must be set for the rule to take effect. There is no default value provided. Sets the severity to critical if the read throughput of the last minute exceeds CASSANDRA_ THROUGHPUT_HIGH_READ_THRESHOLD. The threshold must be set for the rule to take effect. There is no default value provided. Sets the severity to critical if the write throughput of the last minute exceeds CASSANDRA_ THROUGHPUT_HIGH_WRITE_THRESHOLD. The threshold must be set for the rule to take effect. There is no default value provided. Sets the severity to critical is the last ParNew GC duration exceeds CASSANDRA_GC_ LONG_PARNEW_GC_DURATION_THRESHOLD. Default: Threshold is set to 300 milliseconds. Sets the severity to critical if the last CMS GC duration exceeds CASSANDRA_GC_LONG_ CMS_GC_DURATION_THRESHOLD. Default: Threshold is set to 300 milliseconds Page 12 of 18

13 Technical reference Cassandra monitoring dataviews Cassandra disk usage This dataview displays the disk usage-related metrics. Monitoring these node-level metrics are critical to determine if additional nodes are needed: Row Compaction CompletedTasks Compation PendingTasks Storage Load Description Number of completed compactions since the server (re)start. Estimated number of compactions remaining to perform. The size, in bytes, of the on disk data size this node manages. MBeans for Cassandra-DiskUsage org.apache.cassandra.metrics:type=compaction,name=completedtasks org.apache.cassandra.metrics:type=compaction,name=pendingtasks org.apache.cassandra.metrics:type=storage,name=load Page 13 of 18

14 Cassandra errors This dataview displays the count of specific errors and exceptions encountered by a Cassandra node. These metrics are helpful in identifying problematic nodes: Column StorageExceptions ReadTimeouts WriteTimeouts ReadUnavailables WriteUnavailables Description Number of internal exceptions caught. Under normal exceptions, this should be zero. Number of read timeouts encountered. Number of write timeouts encountered. Number of read unavailable exceptions encountered. Number of write unavailable exceptions encountered. MBeans for Cassandra-Errors org.apache.cassandra.metrics:type=storage,name=exceptions org.apache.cassandra.metrics:type=clientrequest,scope=read,name=timeouts org.apache.cassandra.metrics:type=clientrequest,scope=write,name=timeouts org.apache.cassandra.metrics:type=clientrequest,scope=read,name=unavailables org.apache.cassandra.metrics:type=clientrequest,scope=write,name=unavailables Page 14 of 18

15 Cassandra GC This dataview displays the selected JVM Garbage Collector metrics. Cassandra is a Java-based system so it relies on Java garbage collection (GC) processes to free up memory. Any significant increase in GC latency will impact Cassandra s performance: Column ConcurrentMarkSweep CollectionCount ConcurrentMarkSweep CollectionTime ConcurrentMarkSweep LastGCDuration ParNew CollectionCount ParNew CollectionTime ParNew LastGCDuration Description Total number of CMS collections that have occurred. Approximate accumulated CMS collection elapsed time in milliseconds. Elapsed time of the last CMS GC in milliseconds. Total number of ParNew collections that have occurred. Approximate accumulated ParNew collection elapsed time in milliseconds. Elapsed time of the last ParNew GC in milliseconds. MBeans for Cassandra-GC java.lang:type=garbagecollector,name=concurrentmarksweep,* java.lang:type=garbagecollector,name=parnew,* Page 15 of 18

16 Cassandra latency This dataview displays the node-level latency metrics. It gives a view on Cassandra's performance and can identify potential, network issues, or bottlenecks: Column Operation Events TotalLatency AverageLatency Description Type of operation (Read or Write). Number of operation events. Accumulated latency in microseconds. Average latency in microseconds (TotalLatency divided by Events). MBeans for Cassandra-Latency org.apache.cassandra.metrics:type=clientrequest,scope=read,name=latency org.apache.cassandra.metrics:type=clientrequest,scope=read,name=totallatency org.apache.cassandra.metrics:type=clientrequest,scope=write,name=latency org.apache.cassandra.metrics:type=clientrequest,scope=write,name=totallatency Page 16 of 18

17 Cassandra tasks This dataview displays the count of pending and blocked tasks in various stages. It identifies bottlenecks and potential problems: Column Status CounterMutationStage MutationStage ReadRepairStage ReadStage RequestResponseStage Description Task status (Blocked or Pending). Number of tasks in the Counter Mutation stage. Number of tasks in the Mutation stage. Number of tasks in the Read Repair stage. Number of tasks in the Read stage. Number of tasks in the Request Response stage. MBeans for Cassandra-Tasks org.apache.cassandra.metrics:type=threadpools,path=request,scope=countermutationstage,na me=<currentlyblockedtasks PendingTasks> org.apache.cassandra.metrics:type=threadpools,path=request,scope=mutationstage,name=<c urrentlyblockedtasks PendingTasks> PendingTasks>org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadRep airstage,name==<currentlyblockedtasks PendingTasks> org.apache.cassandra.metrics:type=threadpools,path=request,scope=readstage,name==<curr entlyblockedtasks PendingTasks> org.apache.cassandra.metrics:type=threadpools,path=request,scope=requestresponsestage, name==<currentlyblockedtasks PendingTasks> Page 17 of 18

18 Cassandra throughput This dataview displays the node-level throughput metrics. It gives a high-level view on the node s activity levels and is important in understanding how and how much the node is being used: Column TimePeriod ReadThroughput WriteThroughput Description Time period (1 or 5 minutes). Read events per second during the last time period. Write events per second during the last time period. MBeans for Cassandra-Throughput org.apache.cassandra.metrics:type=clientrequest,scope=read,name=latency org.apache.cassandra.metrics:type=clientrequest,scope=write,name=latency Page 18 of 18