Kafka Integration User Guide. Functional Area: Kafka Integration. Geneos Release: v4.7. Document Version: v1.0.0

Transcription

1 Kafka Integration User Guide Functional Area: Kafka Integration Geneos Release: v4.7 Document Version: v1.0.0 Date Published: 23 July 2018

2 Copyright ITRS Group Ltd. All rights reserved. Information in this document is subject to change without notice. The software described in this document is furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in accordance with the terms of those agreements. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of ITRS Group Ltd. ITRS Group Ltd 6th Floor, The Bonhill Building, 15 Bonhill Street, London, EC2A 4DN, UK t: +44 (0) f: +44 (0)

3 Table of Contents Kafka Monitoring 4 General Information 4 Overview 4 User Requirements 4 System Requirements 4 Audience 4 Install And Set Up 5 Set Up The Samplers 7 Set Up The Variables 7 Set Up The Rules 8 Technical Reference 9 Kafka Monitoring Dataviews 9 Kafka Broker (per Broker Metrics) 10 MBeans For Kafka Broker 11 Kafka Topics (per Topic Metrics) 12 MBeans For Kafka-Topics 12 Kafka Cluster MBeans For Kafka-Cluster Metrics 13 Kafka Heap Memory Usage MBeans For Kafka-HeapMemoryUsage 14

4 Kafka Monitoring General information Overview Kafka monitoring is a Gateway configuration file that enables monitoring of Kafka Brokers through a set of samplers with customised JMX plug-in settings. Kafka is a distributed streaming platform that allows you to: Publish and subscribe to stream of records. Store streams of records in a fault-tolerant way. Process streams of records as they occur. It is important to monitor Kafka because it carries crucial data that many applications rely on. Geneos provides a JMX server sampler configuration to monitor Kafka. User requirements This monitoring template is a Gateway configuration set-up that can be included in the Gateway Setup Editor. To use this template, your configuration must meet the following requirements: A machine running the Netprobe must have access to the Kafka published HTTP address and port. An Active Console 2 that is connected to the Gateway. Kafka configuration files extracted to their Gateway set-up. Specify two (2) Managed entities variables: KafkaJmxHost and KafkaJmxPort. A JMX-enabled Kafka Cluster. System requirements The following requirements must be met prior to the installation and set-up of the template: Template package: geneos-integration-kafka-<version>.zip. Managed entities utilising the samplers defined in the include/kafkamonitoring.xml. Netprobe version 4.5 or higher. Audience This document is a reference guide to introduce templates and scripts built using the Geneos development toolkit plug-ins. Page 4 of 14

5 The template allows you to integrate specific applications and services to collect metrics without having to create a new plug-in. Install and set up Ensure that you have read and can follow the system requirements prior to installation and set-up of this integration template. 1. Download the config file (geneos-integration-kafka-<version>.zip) from the ITRS Downloads site. 2. Open Active Console Extract the Kafka monitoring configuration file into the Gateway Setup directory. 4. Click Includes to create a new file in the Navigation panel. 5. Enter the location of the file to include in the Location field. In this example, use the include/kafkamonitoring.xml: Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file. 6. Expand the file location in the Includes section. 7. Select Click to load Click Yes to load the new Kafka include file. 9. Click Managed entities in the Navigation panel. 10. Add the Kafka-Broker and Kafka-Cluster types to the Managed Entity section that you will use to monitor Kafka. Page 5 of 14

6 11. Click the Validate button to check your configuration. The Validate button allows you to check if there are any errors or warnings in your configuration set-up. Once the Gateway configuration appears in the Includes section, you can add the samplers and other variables. Page 6 of 14

7 Set up the samplers These are the pre-configured samplers available to use in KafkaMonitoring.xml. Configure the required fields by referring to the table below: Samplers Kafka-HeapMemoryUsage Kafka-Broker Kafka-Topics Kafka-Cluster Set up the variables The KafkaMonitoring.xml template provides the variables that are set in the Environments section: Variable KAFKA_MONITORING_GROUP_NAME Sampler group name. Page 7 of 14

8 Variable KAFKA_JMX_HOST KAFKA_JMX_PORT Host name (or IP address) of the machine hosting the broker. JMX port that has been configured for that broker. After checking and saving the changes, the samplers you have set in the Gateway configuration display in Active Console 2. Set up the rules The KafkaMonitoring-SampleRules.xml template also provides a separate sample rules that you can use to configure the Gateway Setup Editor. Your configuration rules must be set in the Includes section. 1. Enter the location of the file to include in the Location field. In this example, use the include/kafkamonitoring-samplerules.xml: Page 8 of 14

9 Note: The priority controls the importance of a file when merging. Sections in a higher priority file will take precedence over sections in a lower priority file. This priority setting affects the priority of configuration in the main setup file. 2. Expand the file location in the Includes section. 3. Select Click to load Click Yes to load the new Kafka include rules file. 5. Click Rules in the Navigation panel to create new rules. Sample Rules Broker - Kafka Status Generates a human readable version of the numeric State value of the broker and has the following conditions: Red - the broker State=0: Not Running Amber - the broker State=2, 5, or 6 Cluster - Partition Underreplicated Broker - Active Controller Broker - Offline Partitions Applies to all cells in a Kafka-Cluster. The rule turns red if the value is > 0. Applies to all Active Controller Count Cells in the Kafka-Broker. The Active Controller count shows the number of active controllers across the cluster. A value of greater than 1 in this column indicates that there are more than one controllers with alerts. Applies to all offline partitions count in the Kafka-Broker. The offline partitions count shows the total number of under replicated partitions across the cluster. A value of greater than 0 indicates that there are under replicated partitions with alerts. Technical reference Kafka monitoring dataviews The JMX Server sampler configurations are used to monitor Kafka. Page 9 of 14

10 Kafka broker (per broker metrics) This provides the state of the Kafka broker: Column Name Kafka Version State Kafka Status Row name. Kafka binary version. State of the Kafka broker. Manipulated base of the Kafka state value. The following are available: Broker State = 0: Not Running extends Broker States Broker State = 1: Starting extends Broker States Broker State = 2: Recovering from Unclean Shutdown extends Broker States Broker State = 3: Running as Broker extends Broker States Broker State = 4: Running as Controller extends Broker States Broker State = 5: Pending Controlled Shutdown extends Broker States Broker State = 6: Broker Shutting Down extends Broker States PartitionCount LeaderCount UnderReplicatedPartitions Total number of partitions for all topics in the broker which is is usually even across all brokers. Leader Replica Count. The Leader is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions. Number of partitions under replicated per broker. Replicas are the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently active. Page 10 of 14

11 Column Name ActiveControllerCount The number of active controllers in the cluster. One of the brokers is elected as the controller for the whole cluster. It will be responsible for: leadership change of a partition (each leader can independently update ISR). new topics. deleted topics. replica reassignment. OfflinePartitionsCount PreferredReplicaImbalanceCount IsrExpand IsrShrink The number of partitions that do not have an active leader and are hence not writable or readable. The imbalance count in the preferred replica. If a broker goes down, the ISR for some partitions will shrink. When that broker is up again, the ISR will be expanded once the replicas are fully caught up. Other than that, the expected value for both the ISR shrink and expansion rates is 0. When a broker is brought up after a failure, it starts syncing by reading from the leader. Once synced, it gets added back to the ISR. ISR is the set of in-sync replicas. This is the subset of the replicas list that is currently alive and synced with the leader. MBeans for Kafka Broker kafka.server:type=kafkaserver,name=brokerstate kafka.server:type=app-info,id=0 kafka.server:type=replicamanager,name=partitioncount kafka.server:type=replicamanager,name=leadercount kafka.controller:type=kafkacontroller,name=activecontrollercount kafka.controller:type=kafkacontroller,name=offlinepartitionscount kafka.controller:type=kafkacontroller,name=preferredreplicaimbalancecount kafka.server:type=replicamanager,name=isrshrinkspersec kafka.server:type=replicamanager,name=isrexpandspersec Page 11 of 14

12 Kafka topics (per topic metrics) This provides all the metrics available for a topic in the broker: Column ID Topic Name Count / EventType / MeanRate / RateUnit Topic and matrix names. Topic name. Matrix name. Attribute values. MBeans for Kafka-Topics kafka.server:type=brokertopicmetrics,name=bytesinpersec,topic=* kafka.server:type=brokertopicmetrics,name=bytesoutpersec,topic=* kafka.server:type=brokertopicmetrics,name=bytesrejectedpersec,topic=* kafka.server:type=brokertopicmetrics,name=failedfetchrequestspersec,topic=* kafka.server:type=brokertopicmetrics,name=messagesinpersec,topic=* kafka.server:type=brokertopicmetrics,name=totalfetchrequestspersec,topic=* Kafka cluster This shows the number of partitions that do not have an active leader and are hence not writable or readable per topic for the entire cluster. Page 12 of 14

13 Column Name Topic Partition UnderReplicatedPartition Topic name and partition number. Topic name. Partition number. Number of under replicated partitions. MBeans for Kafka-Cluster Metrics kafka.cluster:type=partition,name=underreplicated,topic=*,partition=* Page 13 of 14

14 Kafka heap memory usage Column Committed UsageInit UsageMax UsageUsed PercentageUsed Amount of memory in bytes that is committed for the Java virtual machine to use. Amount of memory in bytes that the Java virtual machine initially requests from the operating system for memory management. Maximum amount of memory in bytes that can be used for memory management. Amount of used memory in bytes. Percentage of maximum usable memory currently used. MBeans for Kafka-HeapMemoryUsage java.lang:type=memory Page 14 of 14