Ingesting Data from Kafka Queues Deployed On-Prem into jsonar Cloud Systems

Transcription

1 Ingesting Data from Kafka Queues Deployed On-Prem into jsonar Cloud Systems Most jsonar systems are deployed on the Cloud yet consume data generated within enterprise data centers. Since Kafka has emerged as one of the most popular integration platforms and enterprise service bus architectures, this tech note describes how to integrate and consume data produced and placed on an on-prem Kafka queue with a jsonar Cloud service. As shown in Figure 1, the setup is based on rsyslog. Rsyslog is available as part of any RHEL system. Figure 1: Data flow between on-prem Kafka and the jsonar Cloud service Rsyslog is configured to be a Kafka consumer and subscribes to any number of Kafka topics. Rsyslog is configured to compress the data and encrypt the data as it is passing on the public Internet. It is also normally configured to batch the sending of the data rather than stream the data. All these are standard rsyslog configs. Data arrives in the jsonar Cloud through the rsyslog layer and is processed by SonarGateway. Any number of enrichment, mapping and analytic rules can be defined on the Gateway and once the JSON documents have been processed they are inserted into the SonarW column-store.

2 1. Create Secure SonarGateway Connections This section describes how to set up Transport Layer Security (TLS) connections between the onprem Kafka and jsonar Cloud. The authentication here is two-way. The sender authenticates with the receiver, and vice-versa. Communication is encrypted and compressed. We ll call the machines: 1. kafka -- the on-prem machine forwarding kafka events to sonargateway 2. sonargateway the jsonar cloud receiving events. In the following assume that sonargateway, and kafka are RedHat machines. Note that the instructions work for multiple kafka machines. You can copy the same kafka certificates (if that suits your security needs) to multiple machines and they will authenticate in the same fashion. 1. Configuration On-Prem (called the agent host or the Kafka source host): 1.1 Certificate creation $ sudo cat <<'EOF' sudo tee /etc/yum.repos.d/rsyslog.repo [rsyslog_v8] name=adiscon CentOS-$releasever - local packages for $basearch baseurl= enabled=1 gpgcheck=0 gpgkey= protect=1 EOF $ sudo yum update rsyslog $ sudo yum install -y gnutls-utils rsyslog-gnutls Generate a private key and a self-signed CA certificate. You will be asked some questions during the certificate generation. For this example choose some simple values and used mostly defaults. Configure the certificate generation to suit your organization: $ certtool --generate-privkey --outfile ca-key.pem Generating a 2048 bit RSA private key... $ certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem Generating a self signed certificate... Details of the certificate's distinguished name. Press enter to ignore a field. Common name: SonarGateway

3 State or province name: BC Country name (2 chars): CA The certificate will expire in (days): 365 Extensions. Does the certificate belong to an authority? (y/n): y Enter the of the subject of the certificate: sonargateway@jsonar.com Will the certificate be used to sign other certificates? (y/n): y Other Information: Public Key ID: cf caeda4d2ea28f873ed924e74a76208c Is the above information ok? (y/n): y Signing certificate... $ chmod 400 ca-key.pem $ ls -l total 12 -r root root 5823 Oct 20 13:31 ca-key.pem -rw-r--r--. 1 root root 1216 Oct 20 13:37 ca.pem The security of the TLS connections depends on the security of ca-key.pem. Keep this file safe. Now generate certificates for the sonargateway and kafka machines, similar to the CA certificate. The key here is the Common name setting. You will have to refer to these names later when you configure rsyslog. Repeat this section twice -- once for the sonargateway machine and once for the kafka machine. $ certtool --generate-privkey --outfile key.pem Generating a 2048 bit RSA private key... $ certtool --generate-request --load-privkey key.pem --outfile request.pem Common name: sonargateway ###### < nd time use 'kafka' ###### State or province name: BC Country name (2 chars): CA Is this a TLS web client certificate? (y/n): y Is this a TLS web server certificate? (y/n): y Self signature: verified

4 $ certtool --generate-certificate --load-request request.pem --outfile cert.pem -- load-ca-certificate ca.pem --load-ca-privkey ca-key.pem Generating a signed certificate... The certificate will expire in (days): 365 Is this a TLS web client certificate? (y/n): y Is this a TLS web server certificate? (y/n): y Other Information: Public Key ID: 12af4f2b06018fc1f114a030f763385f1eeed Is the above information ok? (y/n): y Signing certificate... Now rename the two generated files: $ mv cert.pem sonargateway-cert.pem ## <---- 2nd time, to kafka-cert.pem $ mv key.pem sonargateway-key.pem ## <---- 2nd time, to kafka-key.pem Repeat these steps to generate a private key for kafka. Once you ve created both private keys copy the sonargateway*.pem and ca.pem files to the sonargateway. The command can look something like this: ## if sonargateway machine IP is $ scp -i ~/.ssh/aws.pem {sonargateway*,ca}.pem ec2-user@ : On the kafka machine move the files, restrict permissions, and set a selinux type for them. By using semanage you can keep selinux 'Enforcing': $ sudo bash $ mkdir -p /etc/rsyslog.d/sonar/gateway/tls $ mv *pem /etc/rsyslog.d/sonar/gateway/tls $ chmod 400 /etc/rsyslog.d/sonar/gateway/tls/*.pem $ chown root:root /etc/rsyslog.d/sonar/gateway/tls $ chown root:root /etc/rsyslog.d/sonar/gateway/tls/* $ semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/sonar/gateway/tls(/.*)?" $ restorecon /etc/rsyslog.d/sonar/gateway/tls/*

5 1.2 Configure Rsyslog to communicate with the jsonar Master / Gateway Host Install the imkafka input module for rsyslog: $ sudo yum install rsyslog-kafka In what follows assume that the kafka machine is running the broker. If your broker is a separate on-prem machine you can also configure a TLS connection between that broker and the kafka machine. See the rsyslog imkafka documentation and apache kafka documentation for TLS. imkafka: stable/configuration/modules/imkafka.html Apache Kafka: $ cat >> /etc/rsyslog.d/kafka_forward.conf <<'EOF' global(defaultnetstreamdrivercafile="/etc/rsyslog.d/sonar/gateway/tls/ca.pem" defaultnetstreamdrivercertfile="/etc/rsyslog.d/sonar/gateway/tls/agent-cert.pem" defaultnetstreamdriverkeyfile="/etc/rsyslog.d/sonar/gateway/tls/agent-key.pem") module(load="imkafka") input(type="imkafka" broker=[" :9092"] # the ip/port of your kafka broker topic="topic1" consumergroup="default" ruleset="10546_kafka_forward") ruleset(name="10546_kafka_forward") { action(type="omfwd" keepalive="on" streamdriver="gtls" streamdrivermode="1" streamdriverauthmode="x509/name" streamdriverpermittedpeers="sonargateway" # must match the certificate common name ziplevel="9" # for compression compression.mode="single" # for compression protocol="tcp" queue.type="disk" # for message queuing queue.spooldirectory="/var/lib/rsyslog"

6 queue.filename="rsyslog_kafka_queue" action.resumeretrycount="-1" target=" " # the ip address of the jsonar Cloud port="10546") stop } EOF Restart the rsyslog service and check that you are listening for kafka input on port 9092 using lsof. $ sudo systemctl restart rsyslog $ lsof -i :9092 ## connection from the kafka broker COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd root 11u IPv t0 TCP kafka: >kafka_broker:xmlipcregsvc (ESTABLISHED) rsyslogd root 16u IPv t0 TCP kafka: >kafka_broker:xmlipcregsvc (ESTABLISHED) 2. Configurations On jsonar Master Host (Gateway / Master): 2.1 Configure SonarGateway Create a file that references the certificates: $ cat >> /etc/rsyslog.d/01-global-tls.conf <<'EOF' global(defaultnetstreamdrivercafile="/etc/rsyslog.d/sonar/gateway/tls/ca.pem" defaultnetstreamdrivercertfile="/etc/rsyslog.d/sonar/gateway/tls/sonargateway -cert.pem" defaultnetstreamdriverkeyfile="/etc/rsyslog.d/sonar/gateway/tls/sonargatewaykey.pem") EOF Then modify the input modules file: $ sed -i '/imtcp/d' /etc/rsyslog.d/sonar/gateway/include/modules.conf $ cat >> /etc/rsyslog.d/sonar/gateway/include/modules.conf <<'EOF' module(load="imtcp"

7 streamdriver.name="gtls" streamdriver.mode="1" streamdriver.authmode="x509/name" permittedpeer=["kafka"] keepalive="on" ) EOF From the directory where the certificate and pem files were copied to, run the following commands: $ sudo bash $ mkdir -p /etc/rsyslog.d/sonar/gateway/tls $ mv *pem /etc/rsyslog.d/sonar/gateway/tls $ chmod 400 /etc/rsyslog.d/sonar/gateway/tls/*.pem $ chown root:root /etc/rsyslog.d/sonar/gateway/tls $ chown root:root /etc/rsyslog.d/sonar/gateway/tls/* 2.2 Configure the SonarGateway Kafka Service: Edit /etc/sonar/gateway/kafka_consumer.json. In the second output_connection (around line 45) add two attributes to specify the database into which the data should be inserted and the collection into which it should be inserted. For example: { "output_connection": { "event_format": { "standard": "JSON" }, "target_db": "my_database, "default_collection": "my_kafka_data", "unique_label": "json", "group_label": "json" } } Uncomment the kafka listener in the sonargateway rsyslog configuration: $ vi /etc/rsyslog.d/sonargateway.conf #$IncludeConfig /etc/rsyslog.d/sonar/gateway/rulesets/ingest.conf $IncludeConfig /etc/rsyslog.d/sonar/gateway/rulesets/kafka_consumer.conf #$IncludeConfig /etc/rsyslog.d/sonar/gateway/rulesets/mapr.conf

8 Restart the rsyslog service and check that you are listening on port using lsof. $ sudo systemctl restart rsyslog $ lsof -i :10546 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd root 5u IPv t0 TCP *:10546 (LISTEN) rsyslogd root 6u IPv t0 TCP *:10546 (LISTEN)