boom AWS-agentless-MPI Documentation Version 2.0

Transcription

1 Documentation Version 2.0

2 Legal Notice The information contained in this document is subject to change without notice. Warranty Worldline Germany GmbH makes no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Worldline Germany GmbH shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material whether based on warranty, contract, or other legal theory. Copyright Notices Copyright 2018 Worldline Germany GmbH. This document contains proprietary information and is protected by copyright. All rights are reserved. No part of this document may be copied, reproduced, translated to another language, transferred to electronic media, or transferred into machine readable format without the prior written consent of Worldline Germany GmbH. Trademark Notices Java is a U.S. trademark of Sun Microsystems, Inc. Linux is a U.S. registered trademark of Linus Torvalds. Microsoft, Windows, Windows 2000, Windows Server 2003, Windows Server 2008 Windows Vista and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the Unites states and/or other countries. UNIX is a registered trademark of the Open Group. Sun and Solaris are trademarks of Sun Microsystems, Inc. Written and developed by Worldline Germany GmbH. Contact Worldline Germany GmbH Wegener Str. 5/ Sindelfingen Germany support@bes-worldline.com Phone: Worldline Germany GmbH Page 2

3 Table of Contents 1. Overview AWS EC2 Instance, S3 and Lambda AWS CloudWatch Logs (additional/autarkic feature) 8 2. Requirements Java Secure Socket Layer Phyton (optional needed for AWS-CLI only) AW-CLI (optional needed for accessing AWS CloudWatch Logs only) AWS Account Passing S3 Request Metrics (optional) AWS User Login AWS-agentless-MPI Installation AWS-agentless-MPI Package AWS EC2 Instance, S3, Lambda AWSLOGS-agentless-MPI Package (additional/autarkic) AWS CloudWatch Logs AWS-agentless-MPI Configuration boom agent Linux system for AWS EC2 Instance, S3, Lambda Add Session Enable Monitoring boom agent Linux system for AWS CloudWatch Logs (additional/autarkic) Add Session Enable Logstream Scanning Configuration Steps Summary Monitoring AWS EC2 Instance, S3 and Lambda Scanning AWS CloudWatch Logs (additional/autarkic) AWS-agentless-MPI Actions Monitoring AWS EC2 Instance, S3 and Lambda Scanning AWS CloudWatch Logs (additional/autarkic) AWS-EC2 agentless Policies (Local_Remote) AWS-EC2 Instance State (Monitor Policy) Worldline Germany GmbH Page 3

4 8.2 AWS-EC2 EBS Volume State (Monitor Policy) Quantity of all AWS-EC2 Instances per AccountID (Logfile Policy) AWS-EC2 Instance specific Quantity (Logfile Policy) AWS-EC2 Instance Metric CPUUtilization (Monitor Policy) AWS-EC2 Instance Metric DiskReadBytes (Monitor Policy) AWS-EC2 Instance Metric DiskWriteBytes (Monitor Policy) AWS-EC2 Instance Metric NetworkIn (Monitor Policy) AWS-EC2 Instance Metric NetworkOut (Monitor Policy) AWS-Lambda agentless Policies (Local_Remote) AWS-Lambda Function Errors (Monitor Policy) AWS-S3 agentless Policies (Local_Remote) AWS-S3 Bucket 4xxErrors (Monitor Policy) AWS-S3 Bucket 5xxErrors (Monitor Policy) AWS-Logs agentless Policies (additional/autarkic) AWS-Logs Get LogStream (Logfile Policy) AWS-agentless-MPI De-Installation AWS EC2 Instancw, S3, Lambda AWS CloudWatch Logs Appendix URLs Amazon EC2 CloudWatch Built-in Metrics for Instances Amazon CloudWatch Custom Metrics Publish Custom Metric for EC2 Linux Instance Publish Custom Metric for EC2 Windows Instance Amazon Lambda CloudWatch Built-in Metrics for Functions Amazon S3 CloudWatch Metrics S3 CloudWatch Daily Storage Metrics for Buckets S3 CloudWatch Request Metrics for Buckets Monitoring Script Session-Environment-Files Monitoring Script Instance-/IP-/Volume-Description-Files Monitoring Script Constants Setting Monitoring Script Log-Files AWSLOGS Script Session-Environment-Files AWSLOGS Script Constants Setting AWSLOGS Script Log-File Worldline Germany GmbH Page 4

5 13.13 AWSLOGS Out-Files Worldline Germany GmbH Page 5

6 1. Overview Amazon provides amongst others Cloud Computer,Lambda, S3 acting in a public cloud environment which can be managed by Amazon Web Services (AWS). Such a Cloud Computer corresponds to an Amazon EC2 (Elastic Compute Cloud) Instance which can be equipped with different Operating Systems (i.e. Linux, Windows). AWS Lambda is a compute service that lets run customizable code (Lambda Function) without provisioning or managing servers. Amazon S3 (Simple Storage Service) is storage for the Internet. S3 Buckets are the fundamental container in Amazon S3 for data storage. Amazon provides cloud environment separated in 9 regions worldwide. An AWS user is referenced by an AWS Account-ID and can handle its instances, Lambda Functions, S3 Buckets within 1 region. What is Amazon CloudWatch Logs? Amazon CloudWatch Logs can be used by an AWS user to monitor, store, and access log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, Route 53, and other sources (e.g. Lambda Function). The associated log data can be retrieved from Amazon CloudWatch Logs and hence the original log data can be viewed to see the source of the problem if needed. An AWS user is referenced by an AWS Account-ID and can handle its CloudWatch Logs (log stream) data within 1 region. 1.1 AWS EC2 Instance, S3 and Lambda Amazon s CloudWatch will monitor by default each EC2 instance, Lambda Function, S3 Bucket. The Basic Monitoring collects 10 metrics (built-in metrics) every 5 minutes and stores these data with retention of 63 days. Actual information about the instance state and the metric data (time-ordered set of data points) can be retrieved remotely either using the AWS Web Console or by command execution via different Command Line Interfaces. In the following the monitoring of an EC2 Instance, Lambda Function, S3 Bucket will be described using AWS CLI exclusively via endpoint access meaning that no boom agent is acting on the instance/aws locally which is an agent-less proceeding. The boom AWS-MPI provides scripts and policies. The scripts connect to AWS using Account-ID specific access credentials and calls for instance state and metric data of all the Account-ID related instances. The boom policies schedule these scripts and generate alarms in case of threshold over-/undershooting. How does it work: one Linux system with a running boom agent must have an installed AWS EC2-CLI (providing instance state data) and AWS CloudWatch-CLI (providing metric data) which are parts of AWSagentless-MPI an AWS Account-ID specific AWS Session configuration consuming AWS access key, AWS secret key and AWS region must be added an Account-ID related AWS connection test using the specified security credentials (via EC2-CLI and CloudWatch-CLI) must be completed successfully this enables by setting policy s global variable automatically retrieving state information and metric data about all Account-ID related EC2 Instances, Lambda Functions, S3 Buckets 2018 Worldline Germany GmbH Page 6

7 AWS-EC2 policy executes AWS EC2-Instance monitoring script periodically monitoring script queries CloudWatch for EC2 Instance description data per each available AWS session; EC2 Instance description data contain information about all available EC2 Instances per AWS AccountID/Session EC2 Instance state can have one of these possibilities: pending running shutting-down terminated stopping stopped any instance state transition will be alarmed any instance quantity change will be alarmed AWS-EC2 policy executes AWS EBS monitoring script periodically monitoring script queries CloudWatch for EBS Volume description data per each available AWS session; EBS Volume description data contain information about all available EBS Volumes per AWS AccountID/Session EBS (Elastic Block Storage) Volume (def. root-volume for EC2 Instance) state can have one of these possibilities: ok impaired insufficient-data AWS-EC2 policy executes AWS EC2-Metric monitoring script periodically monitoring script queries CloudWatch per each available EC2 instance name for EC2 Instance metric per each available AWS session (an active EC2 Instance state data collection is required for the alignment of instance related metric data with instance specific information like IP address) EC2 Instance metric can have one of these facilities: CPUUtilization NetworkIn NetworkOut DiskReadBytes DiskWriteBytes <custom-metric> AWS-Lambda policy executes AWS Lambda-Function monitoring script periodically scanning CloudWatch namespace AWS/Lambda for available Lambda function names monitoring script queries CloudWatch per each available Lambda function name for Lambda Function metric per each available AWS session Lambda Function metric can have one of these facilities: Invocations Errors Dead Letter Error Duration Throttles (these metrics are generated by CloudWatch only if Lambda Function has been executed) AWS-S3 policy executes AWS S3-Bucket monitoring script periodically scanning CloudWatch namespace AWS/S3 for available S3 bucket names monitoring script queries CloudWatch per each available S3 bucket name for S3 Bucket metric per each available AWS session S3 Bucket metric can have one of these facilities: BucketSizeBytes NumberOfObjects AllRequests GetRequests PutRequests DeleteRequests HeadRequests PostRequests ListRequests BytesDownloaded BytesUploaded 4xxErrors 5xxErrors FirstByteLatency TotalRequestLatency S3 storage metrics BucketSizeBytes, NumberOfObjects are available per default; the remaining request metrics have to be activated (see Requirements ) and are generated by CloudWatch only if S3 Bucket has been requested any over-/undershooting of the received metric values against the specified thresholds will be alarmed Note: AWS CloudWatch stores metric data with the Universal Time Coordinated (UTC/GMT) time stamp. Because of the CloudWatch s default sample interval of 5 minutes the monitoring scripts calculate an actual time frame (UTC) from now back to the elapsed 13 (~2,5 x 5) minutes. Within this time frame at least 1 metric value will be expected. Note: an AWS Session is described by an AccountID related environment file. Each monitoring script searches for these files (cfg/<account_id>_*.env) and repeats (per default) querying AWS as long as an environment file 2018 Worldline Germany GmbH Page 7

8 is found. When retrieving instance related data the given AccountID will be compared with the defined one and if these IDs are different the script processing will be skipped. This behavior can be ignored with the -f (force) option. Checking validity of AccountID can only be provided for EC2 Instance and not for S3/Lambda data. 1.2 AWS CloudWatch Logs (additional/autarkic feature) AWS user can retrieve the associated log data (e.g. provided by Lambda Function) from CloudWatch Logs using the Amazon CloudWatch console, the CloudWatch Logs commands in the AWS CLI, the CloudWatch Logs API or the CloudWatch Logs SDK. boom s AWS-agentless MPI is using the AWS CLI exclusively. Log data can be stored and accessed for as long as user needed using highly durable, low-cost storage so user don t have to worry about filling up hard drives. So for archiving Log Data user can use CloudWatch Logs to store log data in highly durable storage. An AWS user can change the log retention setting so that any log events older than this setting are automatically deleted. The CloudWatch Logs agent makes it easy to quickly send both rotated and non-rotated log data off of a host and into the log service. boom s AWS-agentless MPI is accessing the raw log data exclusively. In the following the scanning of CloudWatch Logs (log stream) data will be described using AWS CLI exclusively via endpoint access meaning that no boom agent is acting on the instance/aws locally which is an agent-less proceeding. The boom AWS-MPI provides scripts and policies. The scripts connect to AWS using Account-ID specific access credentials reads newly-discovered log data of all specified (Account-ID related) log streams and appends these retrieved raw log data to a dedicated output file locally on boom agent side. A boom Logfile Policy schedules the script providing the output log-file and analyzes it regarding abnormal entries. Note: Amazon CloudWatch Logs can be used independently or in combination with monitoring EC2 instance, Lambda Function and S3 Bucket. The used AWS CLI is different to the used one of monitoring EC2 instance, Lambda Function and S3 Bucket. Therefore boom agent uses a dedicated (and separated) working environment to sample CloudWatch Logs. How does it work: one Linux system with a running boom agent must have an installed AWS-CLI (providing an overall set of AWS commands) which is not part of AWS-agentless-MPI an AWS Account-ID specific AWS Session configuration consuming AWS access key, AWS secret key, AWS region, LogGroup and LogStream must be added ( LogGroup and LogStream are CloudWatch Logs specific classification items) if arguments LogGroup or LogStream are unknown Add Session command can be invoked without these parameters returning all found LogGroup/LogStream belonging to AWS Account-ID related user credentials if LogGroup belongs to a specific AWS Lambda Function the assignment to LogStream name is done automatically by using always the latest log stream internally because each processing of a Lambda Function provides a dedicated (and separated) log stream in contrast to a not Lambda- Function-LogGroup a successful return of Add Session enables automatically retrieving AWS log stream data by setting policy s global variable 2018 Worldline Germany GmbH Page 8

9 AWS-Logs policy executes get-aws-logstream-data script periodically this script reads specified LogStream for each available AWS Session in case of a Lambda Function ( LogGroup is set to name of Lambda Function) the complete content of latest available LogStream(s) since last script call will be retrieved and unchanged written (appended) to the LogStream related output file in the other case ( LogGroup is set to any name but not a Lambda Function) the newly-discovered content of the specified LogStream since last script call will be retrieved and unchanged written (appended) to the LogStream related output file the output file (1 file per LogStream) is located locally on boom agent side and comes with the properties maximum size and number of instances which are configurable all the records appended to the output file which are newly-discovered within 1 scan interval are marked with an unique time stamp at the beginning ( LogStream <logstream_name> <ddmmyyyy>-<hhmm> ) AWS-Logs policy analyzes every available output file concerning new entries and generates an indication for records not containing phrases like Loading function, START RequestId, END RequestId, REPORT RequestId (these are typical Lambda Function related entries) Note: an AWS Session is described by an AccountID related environment file. get-aws-logstream-data script searches for these files (cfg/<account-id>_awscli_<lambda-fct>.env_latest or cfg/<accountid>_awscli_<loggroup>.env_<logstream>) and repeats (per default) sampling AWS CloudWatch Logs as long as an environment file is found Worldline Germany GmbH Page 9

10 2. Requirements The following is required for the operating of AWS-agentless-MPI. 2.1 Java Java 7 Standard Edition: boom agent system must have an installed Java 7 (or higher) standard edition 2.2 Secure Socket Layer openssl: boom agent system must have an installed openssl edition 2.3 Phyton (optional needed for AWS-CLI only) python: boom agent system must have an installed python 2 (or higher) edition 2.4 AW-CLI (optional needed for accessing AWS CloudWatch Logs only) AWS-CLI (see boom agent system must have an installed AWS-CLI 1 (or higher) edition 2.5 AWS Account AWS Account Identification AWS User must be aware of his own AWS Account ID see yellow marked section between the most right colons (example for user pso ) 2018 Worldline Germany GmbH Page 10

11 AWS IAM Management Console AWS Security Credentials Access to AWS via command line is permitted by using security credentials. These Credentials can be generated for the AWS User which is owner of the instances. This is not recommended because typically the AWS User has administration power. Therefore a dedicated monitor-user with restricted access qualification (read-only) should be used: Create IAM-User (e.g. mon ) via Amazon IAM Console and assign IAM Policy AmazonEC2ReadOnlyAccess Download Credential-CSV-File for this user Credential-File contains: o Access key o Secret key 2018 Worldline Germany GmbH Page 11

12 AWS IAM Management Console AWS Region AWS User must be aware of the AWS region where the EC2 instance, Lambda Function, S3 Bucket is located possible region notations are: ap-south-1 eu-west-1 ap-northeast-2 ap-northeast-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west Passing S3 Request Metrics (optional) There are two types of CloudWatch metrics for Amazon S3: storage metrics and request metrics. Storage metrics (BucketSizeBytes, NumberOfObjects - see appendix) are reported once per day and are provided to all customers at no additional costs. Request metrics (see appendix) are available at 1 minute intervals after some latency to process, and metrics are billed at the standard CloudWatch rate. To get request metrics, you must opt into them by configuring them in the console or with the Amazon S3 API. To configure request metrics on a bucket: 1. Sign in to the AWS Management Console and open the Amazon S3 console at console.aws.amazon.com/s3/ Opt In (new Amazon S3 Console) 2. In the Bucket name list, choose the name of the bucket that has the objects you want to get request metrics for Worldline Germany GmbH Page 12

13 3. Choose the Management tab at the top of the page. 4. Choose Metrics. 5. Choose Requests. 6. From the name of your bucket in the left-side pane, choose the edit icon. 7. Choose the Request metrics check box. 8. Choose Save. Note: About 15 minutes after CloudWatch begins tracking these request metrics, you can see graphs for the metrics in both the Amazon S3 or CloudWatch consoles Worldline Germany GmbH Page 13

14 3. AWS User Login In order to access the EC2 Instance, Lambda Function, S3 Bucket you will require a valid AWS user with the administration right managing the instances and IAM users. AWS EC2 Management Console (Region e.g. eu-central-1 ) AWS EC2 Lambda Console (Region e.g. eu-central-1 ) 2018 Worldline Germany GmbH Page 14

15 AWS EC2 S3 Console (Region e.g. eu-central-1 ) 2018 Worldline Germany GmbH Page 15

16 4. AWS-agentless-MPI Installation The AWS MPI consists of 2 parts. One is used for threshold monitoring of EC2 instances, S3 Buckets and Lambda functions. The other part ( AWSLOGS ) is used for fetching log data (stored by CloudWatch) and analyzing it by boom of log sources like Lambda functions or other resources. Extract the AWS-agentless-MPI boomaws-agtl-mpi_vx.xx.zip to a temporary directory and import all necessary components to the boom server using the Import MPI View of the boom GUI. 4.1 AWS-agentless-MPI Package Composition: 2018 Worldline Germany GmbH Page 16

17 4.1.1 AWS EC2 Instance, S3, Lambda Deploy boom package AWS and Policy Group AWS/Local_Remote to that boom agent Linux system which should handle AWS EC2 instance, S3, and Lambda function data connection. Deployment of the package automatically extracts AWS EC2-CLI and CloudWatch-CLI as executable Java software to boom agent s destination folder <boom-agent-home>/spi/aws/. Note: Local_Remote marks Policies which also can be used by a boom agent acting locally at AWS. Here we run a remote processing situation. 4.2 AWSLOGS-agentless-MPI Package (additional/autarkic) Composition: 2018 Worldline Germany GmbH Page 17

18 4.2.1 AWS CloudWatch Logs Deploy boom package AWSLOGS and Policy Group AWSLOGS to that boom agent Linux system which should handle AWS CloudWatch Logstream connection. Note: boom Package AWSLOGS contains Shell scripts accessing the AWS CloudWatch Logs by usage of the AWS-CLI. The AWS-CLI has to be installed manually by yourself on the boom agent system. Note: AWSLOGS Policies can also be used by a boom agent acting locally at AWS. Here we run a remote processing situation Worldline Germany GmbH Page 18

19 5. AWS-agentless-MPI Configuration 5.1 boom agent Linux system for AWS EC2 Instance, S3, Lambda Add Session Make sure, that the boom agent system is able to establish a connection to AWS. This can be done by adding an AWS Account-ID specific setting for a dedicated AWS Session configuration. To add an AWS Session configuration you have to open the Action View in the boom GUI and double-click on the Action agtl_add Session: adjust parameters select the appropriate boom agent click on execute Note: Parameter AWS_ACCESS_KEY=<access-key> and AWS_SECRET_KEY=<secret-key are defined in the Credential-CSV-File downloaded by Amazon IAM Console. All required information will be stored in 2 text files on the boom agent side. These Environment-Files are located in the boom agent directory: /<boom-agent-home>/spi/aws/cfg/<account-id>_aws_ec2cli.env /<boom-agent-home>/spi/aws/cfg/<account-id>_aws_cwcli.env PLEASE NOTE: AWS_ACCESS_KEY and AWS_SECRET_KEY will be encrypted! Note: Replaying this action with an identical AWS Account-ID will fail because the Environment-Files already exist. You have to delete them by calling action agtl_delete Session first Worldline Germany GmbH Page 19

20 5.1.2 Enable Monitoring The next step is to enable the monitoring. This can be done by testing successfully an AWS Account-ID specific AWS Session configuration. To test an AWS Session configuration you have to open the Action View in the boom GUI and double-click on the Action agtl_test Session: adjust parameter select the appropriate boom agent click on execute This action tries to connect to AWS first using EC2-CLI and afterwards CloudWatch-CLI with Account ID related security credentials. Only if both attempts have been successful the result for the complete test action will be set to success described by the result message passed: OK. In case of a failed result agtl_add Session has to be repeated with corrected values for the corresponding script arguments. You have to delete the session by calling agtl_delete Session first. 5.2 boom agent Linux system for AWS CloudWatch Logs (additional/autarkic) Add Session Make sure, that the boom agent system is able to establish a connection to AWS CloudWatch Logs. This can be done by adding an AWS Account-ID specific setting for a dedicated AWSLOGS Session configuration. To add an AWSLOGS Session configuration you have to open the Action View in the boom GUI and doubleclick on the Action agtl_awslogs_add Session: 2018 Worldline Germany GmbH Page 20

21 adjust parameters select the appropriate boom agent click on execute Note: Parameter AWS_ACCESS_KEY=<access-key> and AWS_SECRET_KEY=<secret-key are defined in the Credential-CSV-File downloaded by Amazon IAM Console. All required information will be stored in 1 text file on the boom agent side. These Environment-Files are located in the boom agent directory (in case of Lambda function): Or in case of other log-stream resource: /<boom-agent-home>/spi/awslogs/cfg/<accountid>_awscli_<lambda_function>.env_latest /<boom-agent-home>/spi/aws/cfg/<accountid>_awscli_<other_loggroup>.env_<other_logstream> PLEASE NOTE: AWS_ACCESS_KEY and AWS_SECRET_KEY will be encrypted! Note: Replaying this action with an identical AWS Account-ID will fail because the Environment-Files already exist. You have to delete them by calling action agtl_awslogs_delete Session first. Note: A successful Add Session execution starts the Logstream scanning automatically Enable Logstream Scanning In contrast to the AWS EC2 Instance, S3, Lambda monitoring AWS CloudWatch Logstream scanning is enabled automatically when agtl_awslogs_add Session was performed successfully because an AWSLOGS Add Session queries always the Account-ID related LogGroup which implies a working connection to AWS. So AWSLOGS Session is tested implicitly Worldline Germany GmbH Page 21

22 6. Configuration Steps Summary 6.1 Monitoring AWS EC2 Instance, S3 and Lambda The best and easiest configuration sequence is: 1. Deploy the AWS-agentless-MPI Assignment (contains AWS EC2-CLI, AWS CloudWatch-CLI). Make sure the boom agent is still running correctly. 2. Deploy the AWS-agentless-MPI Actions. 3. Run action agtl_add Session to provide the AWS AccountID related configuration parameters. 4. Run action agtl_test Session to test AWS-agentless-MPI is working at all. Here you can see if the boom agent is able to connect correctly to AccountID specific AWS-EC2 instance. A successful connect starts automatically the monitoring because of Policy s 2 Global Variables (1 per each working CLI-connection) will be enabled. If there are some problems here, please re-check you security credentials. Note: agtl_test Session has to be performed mandatory otherwise monitoring will not start. 6.2 Scanning AWS CloudWatch Logs (additional/autarkic) The best and easiest configuration sequence is: 1. Deploy the AWSLOGS-agentless-MPI Assignment (contains not AWS-CLI). Make sure the boom agent is still running correctly. 2. Deploy the AWSLOGS-agentless-MPI Actions. 3. Run action agtl_awslogs_add Session to provide the AWS AccountID related configuration parameters. If there are some problems here, please re-check you security credentials. Note: Scanning will be started (because of Policy s Global Variable) automatically as soon as any agtl_awslogs_add Session was performed successfully Worldline Germany GmbH Page 22

23 7. AWS-agentless-MPI Actions 7.1 Monitoring AWS EC2 Instance, S3 and Lambda Name Description Command Parameters Name agtl_add Session Sets AWS AccountID specific environment for AWS HTTP-Session (agentless) using EC2-CLI and CloudWatch-CLI. Region must be one of the following: ap-south-1 eu-west-1 ap-northeast-2 ap-northeast-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2 aws/aws_add_session.sh AWS_ACCOUNT_ID=<account-id> REGION=<region> AWS_ACCESS_KEY=<key> AWS_SECRET_KEY=<secret-key> agtl_delete Session 2018 Worldline Germany GmbH Page 23

24 Description Command Parameters Name Description Command Parameters Name Description Command Parameters Name Description Deletes AWS AccountID specific environment for AWS HTTP-Session (agentless). In case of last environment Policy s 2 Global Variables will be removed meaning that monitoring will be stopped automatically. aws/aws_del_session.sh <account-id> agtl_test Session Tests connection to AWS HTTP-Session (agentless) via EC2-CLI and CloudWatch-CLI using AccountID specific security credentials given by the AccountID related environment file. A successful connect starts automatically the monitoring (Policy s 2 Global Variables will be set). /aws/aws_tst_session.sh <account-id> agtl_list Sessions Lists all active AWS Sessions. Shows number and AccountID per AWS Session. aws/aws_list_session.sh Get Instance State Checks status using AWS EC2-CLI of all available AWS EC2 instances per given AccountID or for all discovered AccountIDs (occurrence by all files./cnf/<account-id>_aws_ec2cli.env) and compares it with historical data. Shows actual state and state transition per instance. Possible state values: pending running shutting-down terminated stopping stopped Script started with no argument <account-id> consumes any available AWS AccountID related environment data specified by Add Session and queries AWS instance description data for each AccountID. In case of Add Session related AccountID does not match ID returned by instance description query, further AccountID specific processing regarding instance state/quantity will be skipped. Argument -f forces processing to continue in case of diversity for discovered AccountID with AccountID returned by internal instance description query. Argument <account-id> enables querying and processing instance description data only for given AccountID instead of multi-processing for all found AccountIDs. Command Optional Parameters Name Description aws/ec2/aws_mchk_inst.sh <account-id> <instance-id> -f Get Instance Metric Returns actual (Universal Time Coordinated UTC/GMT) metric data using AWS CloudWatch-CLI of one/all available AWS EC2 instances for all discovered 2018 Worldline Germany GmbH Page 24

25 AccountIDs (occurrence by all files./cfg/<account-id>_aws_cwcli.env) or per given Account and InstanceID. CloudWatch namespace for all built-in metrics (standard metric e.g. "CPUUtilization") is "AWS/EC2" (default). By default CloudWatch collects metrics per instance every 5 minutes (no detailed AWS monitoring). Script calculates an actual time frame (UTC) from now back to elapsed 12 minutes. Within this time frame at least 1 metric value will be expected by querying CloudWatch. Possible Metrics: CPUUtilization NetworkIn NetworkOut DiskReadBytes DiskWr itebytes - cm=<custom-metric> -cns=<custom-namespace> Requires an active execution of aws_mchk_instance.sh which provides InstanceID and IP to an Account-ID related Summary-File (./tmp/<accountid>_isum_all.hist).. Argument -f forces processing to continue in case of diversity for discovered AccountID with AccountID returned by internal instance description query. Arguments <account-id>, <instance> and <metric> enables querying and processing instance description data only for given AccountID specific instance instead of multi-processing for all found AccountIDs. Example for standard metric arguments: -f 1234 i-0dad7b7546ff403f4 CPUUtilization Example for custom metric arguments: -f 1234 i-0dad7b7546ff403f4 - cm=memory -cns=system/windows Command Parameters Optional Parameters Name Description aws/ec2/aws_mget_instance_metric.sh <standard-metric> -cm=<custom-metric> -cns=<cloudwatch-customnamespace> <account-id> <instance> -f Get Lambda Metric Returns actual (Universal Time Coordinated UTC/GMT) metric data using AWS CloudWatch-CLI of one/all available AWS Lambda Functions for all discovered AccountIDs (occurrence by all files./cfg/<account-id>_awscwcli.env) or per given AccountID and Functionname. CloudWatch namespace for all specified Lambda Functions is "AWS/Lambda" (default). CloudWatch generates metric data per Lambda Function on demand (as soon as Function has been executed). Script calculates an actual time frame (UTC) from now back to elapsed 12 minutes. Within this time frame at least 1 metric value will be expected by querying CloudWatch. Possible Metrics: Invocations Errors Dead Letter Error Duration Throttles Argument -f forces processing to continue in case of diversity for discovered AccountID with AccountID returned by internal CloudWatch namespace query. Arguments <account-id>, <function-name> and <metric> enables querying and processing metric data only for given AccountID specific Lambda Function instead of multi-processing for all found AccountIDs Worldline Germany GmbH Page 25

26 Command Parameters Optional Parameters Name Description aws/lambda/aws_mget_lambda_metric.sh Invocations Errors Dead Letter Error Duration Throttle <account-id> <function-name> -f Get S3 Metric Returns actual (Universal Time Coordinated UTC/GMT) metric data using AWS CloudWatch-CLI of one/all available AWS S3 Buckets for all discovered AccountIDs (occurrence by all files./cfg/<account-id>_awscwcli.env) or per given AccountID and Bucketname. CloudWatch namespace for all specified S3 Buckets is "AWS/S3" (default). CloudWatch generates metric data per S3 Bucket on demand (as soon as an request has been executed). Script calculates an actual time frame (UTC) from now back to elapsed 12 minutes. Within this time frame at least 1 metric value will be expected by querying CloudWatch. Possible Metrics: BucketSizeBytes NumberOfObjects AllRequests GetRequests PutRequests DeleteRequests HeadRequests PostRequests ListRequests BytesDownloaded BytesUploaded 4xxErrors 5xxErrors FirstByteLatency TotalRequestLatency Argument -f forces processing to continue in case of diversity for discovered AccountID with AccountID returned by internal CloudWatch namespace query. Arguments <account-id>, <bucket-name> and <metric> enables querying and processing metric data only for given AccountID specific S3 Bucket instead of multi-processing for all found AccountIDs. Command aws/s3/aws_mget_s3bucket_metric.sh Parameters BucketSizeBytes NumberOfObjects AllRequests GetRequests PutRequests DeleteRequests HeadRequests PostRequests ListRequests BytesDownloaded BytesUploaded 4xxErrors 5xxErrors FirstByteLatency TotalRequestLatency Optional Parameters <account-id> <function-name> -f Name Description Get Volume State Checks status using AWS EC2-CLI of one/all available AWS EBS volume(s). Shows actual state. Possible state values: ok impaired insufficient-data. Script started with no argument <account-id> consumes any available AWS AccountID related environment data specified by Add Session and queries AWS instance description data for each AccountID. In case of Add Session related AccountID does not match ID returned by instance description query, further AccountID specific processing regarding instance state/quantity will be skipped. Argument -f forces processing to continue in case of diversity for discovered AccountID with AccountID returned by internal instance description query. Argument <account-id> enables querying and processing instance description data only for given AccountID instead of multi-processing for all 2018 Worldline Germany GmbH Page 26

27 found AccountIDs. Command Optional Parameters aws/ec2/aws_mget_volstat.sh <account-id> <volume-id> -f 7.2 Scanning AWS CloudWatch Logs (additional/autarkic) Name Description Command Parameters Optional Parameters Name Description Command agtl_awslogs_add Session Sets AWS AccountID specific environment for AWS CloudWatch-Logs HTTP- Session (agentless) using AWS-CLI. Region must be one of the following: ap-south-1 eu-west-1 ap-northeast-2 ap-northeast-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2 awslogs/awslogs_add_session.sh AWS_ACCOUNT_ID=<account-id> REGION=<region> AWS_ACCESS_KEY=<key> AWS_SECRET_KEY=<secret-key> LOG_GROUP=<log-group> LOG_STREAM=<log-stream> agtl_awslogs_delete Session Deletes AWS AccountID specific environment for AWS CloudWatch-Logs HTTP-Session (agentless). In case of last environment Policy s Global Variable will be removed meaning that logstream scanning will be stopped automatically. awslogs/awslogs_del_session.sh 2018 Worldline Germany GmbH Page 27

28 Parameters Optional Parameters Name Description Command Parameters Name Description Command Optional Parameters AWS_ACCOUNT_ID=<account-id> LOG_GROUP=<log-group> LOG_STREAM=<log-stream> agtl_awslogs_list Sessions Lists all active AWSLOGS Sessions. Shows number and AccountID per AWS CloudWatch-Logs Session. awslogs/awslogs_list_session.sh Get LogStream Data Picks up Logstream content within calculated time frame per AWS Account ID using AWS-CLI and put it to output file. If LogGroup is a Lambda Function Logstream is set internally to LATEST because each function processing provides a separate dedicated Logstream (marked LATEST). If no actual Logstream (ref. to act. scan-time) exists output file is not appended with Logstream content. Output file handling: Maximum file size ($OUTF_MAX_SIZ) and number of files ($OUTF_MAX_CNT) is configurable. Time frame calculation: end time = actual time (sec.); start time = end-time - <interval> (sec.) awslogs/awslogs_mget_logdata.sh <account-id> <logstream-name> INTERVAL=<interval_sec> (def. 300) 2018 Worldline Germany GmbH Page 28

29 8. AWS-EC2 agentless Policies (Local_Remote) When defining a Monitor Policy you have to make sure that: 1. Policy with Reset is set to YES 2. Monitor Type is set to OPM 3. Detect Duplicates is activated for all conditions. 8.1 AWS-EC2 Instance State (Monitor Policy) Possible states (pending, running, shutting-down, terminated, stopping, stopped) per AWS-EC2 instance will be evaluated Worldline Germany GmbH Page 29

30 8.2 AWS-EC2 EBS Volume State (Monitor Policy) Possible states (ok, impaired, insufficient-data) per AWS-EC2 EBS volume will be evaluated. EC2 Instance has a dedicated EBS (Elastic Block Store) Volume per default Worldline Germany GmbH Page 30

31 8.3 Quantity of all AWS-EC2 Instances per AccountID (Logfile Policy) The number of all AWS-EC2 instances per AWS AccountID will be evaluated (added/removed/unchanged). AWS Account-IDs will be discovered automatically Worldline Germany GmbH Page 31

32 8.4 AWS-EC2 Instance specific Quantity (Logfile Policy) Detects instance has been added or removed per AccountID and report instance related characterization (AWS Account-IDs will be discovered automatically) Worldline Germany GmbH Page 32

33 8.5 AWS-EC2 Instance Metric CPUUtilization (Monitor Policy) CPU-Utilization (percent) per AWS-EC2 instance will be evaluated Worldline Germany GmbH Page 33

34 8.6 AWS-EC2 Instance Metric DiskReadBytes (Monitor Policy) Data input capacity (Bytes per second) application reads from Disk per AWS-EC2 instance will be evaluated (should be higher than ~ 2MB/sec) Worldline Germany GmbH Page 34

35 8.7 AWS-EC2 Instance Metric DiskWriteBytes (Monitor Policy) Data output capacity (Bytes per second) application writes to Disk per AWS-EC2 instance will be evaluated (should be higher than ~ 2MB/sec) Worldline Germany GmbH Page 35

36 8.8 AWS-EC2 Instance Metric NetworkIn (Monitor Policy) Network input capacity (Bytes per second) including traffic from EBS-Volumes per AWS-EC2 instance will be evaluated (should be always higher than 10B/sec and lower than ~ 100MB/sec) Worldline Germany GmbH Page 36

37 8.9 AWS-EC2 Instance Metric NetworkOut (Monitor Policy) Network output capacity (Bytes per second) including traffic to EBS-Volumes per AWS-EC2 instance will be evaluated (should be always higher than 10B/sec and lower than ~ 100MB/sec) Worldline Germany GmbH Page 37

38 9. AWS-Lambda agentless Policies (Local_Remote) When defining a Monitor Policy you have to make sure that: 1. Policy with Reset is set to YES 2. Monitor Type is set to OPM 3. De-Duplicate KeyOnly is activated for all conditions (because script output contains a varying time stamp for message text duplicates ). 9.1 AWS-Lambda Function Errors (Monitor Policy) Errors (count) per AWS-Lambda Function will be evaluated. Name of Lambda Function will be searched automatically by script using "mon-list-metrics --namespace AWS/Lambda". Note: This metric (Errors) will be provided by AWS CloudWatch only (not periodically) if Lambda Function has been executed Worldline Germany GmbH Page 38

39 10. AWS-S3 agentless Policies (Local_Remote) When defining a Monitor Policy you have to make sure that: 1. Policy with Reset is set to YES 2. Monitor Type is set to OPM 3. De-Duplicate KeyOnly is activated for all conditions (because script output contains a varying time stamp for message text duplicates ) AWS-S3 Bucket 4xxErrors (Monitor Policy) 4xxErrors (number of HTTP 4xx client errors) per AWS-S3 Bucket will be evaluated. Requires enabled Request Metric property for S3 Bucket. This metric can only be queried if "Request Metric" feature has been enabled for S3 Bucket (e.g. via AWS S3 Management "new" Console). Metric will be provided by AWS CloudWatch only (not periodically) on occurrence Worldline Germany GmbH Page 39

40 10.2 AWS-S3 Bucket 5xxErrors (Monitor Policy) 5xxErrors (number of HTTP 5xx client errors) per AWS-S3 Bucket will be evaluated. Requires enabled Request Metric property for S3 Bucket. This metric can only be queried if "Request Metric" feature has been enabled for S3 Bucket (e.g. via AWS S3 Management "new" Console). Metric will be provided by AWS CloudWatch only (not periodically) on occurrence Worldline Germany GmbH Page 40

41 11. AWS-Logs agentless Policies (additional/autarkic) When defining a Logfile Policy you have to make sure that: 1. Type is set to Logfile Monitor 2. Detect Duplicates is activated for all conditions (because script output contains a varying time stamp for message text duplicates ). 3. Search Text matches everything except Loading function START RequestId END RequestId REPORT RequestId 11.1 AWS-Logs Get LogStream (Logfile Policy) Worldline Germany GmbH Page 41

42 12. AWS-agentless-MPI De-Installation 12.1 AWS EC2 Instancw, S3, Lambda Please process the following steps to correctly remove AWS-MPI: 1. Un-Deploy all deployed AWS-MPI related elements like: Assignments Packages single deployed Policies 2. Delete all AWS-MPI related elements from the boom GUI: Policies Assignments Packages Actions 3. Delete manually from the /<boom-agent-home>/spi/ the complete AWS directory: rm r aws/ 12.2 AWS CloudWatch Logs Please process the following steps to correctly remove AWS_Logs-MPI: 1. Un-Deploy all deployed AWSLOGS-MPI related elements like: Assignments Packages single deployed Policies 2. Delete all AWSLOGS-MPI related elements from the boom GUI: Policies Assignments Packages Actions 3. Delete manually from the /<boom-agent-home>/spi/ the complete AWSLOGS directory: rm r awslogs/ 2018 Worldline Germany GmbH Page 42

43 13. Appendix 13.1 URLs AWS Management Console: Services: - EC2 ( - Lambda ( - S3 ( o Opt In (new console for S3 management e.g. metric handling) - CloudWatch ( - IAM AWS Billing Management Console: AWS Documentation: AWS Forum: Amazon EC2 CloudWatch Built-in Metrics for Instances Amazon Elastic Compute Cloud (Amazon EC2) sends metrics to CloudWatch for your EC2 instances. Basic (five-minute) monitoring is enabled by default. You can enable detailed (one-minute) monitoring. The AWS/EC2 namespace includes the following metrics: Metric Description CPUCreditUsage (Only valid for T2 instances) The number of CPU credits consumed during the specified period. This metric identifies the amount of time during which physical CPUs were used for processing instructions by virtual CPUs allocated to the instance. Note +CPU Credit metrics are available at a 5 minute frequency. Units: Count 2018 Worldline Germany GmbH Page 43

44 Metric Description CPUCreditBalance (Only valid for T2 instances) The number of CPU credits that an instance has accumulated. This metric is used to determine how long an instance can burst beyond its baseline performance level at a given rate. Note CPU Credit metrics are available at a 5 minute frequency. Units: Count CPUUtilization The percentage of allocated EC2 compute units that are currently in use on the instance. This metric identifies the processing power required to run an application upon a selected instance. Note Depending on the instance type, tools in your operating system may show a lower percentage than CloudWatch when the instance is not allocated a full processor core. Units: Percent DiskReadOps Completed read operations from all instance store volumes available to the instance in a specified period of time. Note To calculate the average I/O operations per second (IOPS) for the period, divide the total operations in the period by the number of seconds in that period. Units: Count DiskWriteOps Completed write operations to all instance store volumes available to the instance in a specified period of time. Note To calculate the average I/O operations per second (IOPS) for the period, divide the total operations in the period by the number of seconds in that period. Units: Count 2018 Worldline Germany GmbH Page 44

45 Metric Description DiskReadBytes Bytes read from all instance store volumes available to the instance. This metric is used to determine the volume of the data the application reads from the hard disk of the instance. This can be used to determine the speed of the application. Units: Bytes DiskWriteBytes Bytes written to all instance store volumes available to the instance. This metric is used to determine the volume of the data the application writes onto the hard disk of the instance. This can be used to determine the speed of the application. Units: Bytes NetworkIn The number of bytes received on all network interfaces by the instance. This metric identifies the volume of incoming network traffic to an application on a single instance. Units: Bytes NetworkOut The number of bytes sent out on all network interfaces by the instance. This metric identifies the volume of outgoing network traffic to an application on a single instance. Units: Bytes NetworkPacketsIn The number of packets received on all network interfaces by the instance. This metric identifies the volume of incoming traffic in terms of the number of packets on a single instance. This metric is available for basic monitoring only. Units: Count Statistics: Minimum, Maximum, Average NetworkPacketsOut The number of packets sent out on all network interfaces by the instance. This metric identifies the volume of outgoing traffic in terms of the number of packets on a single instance. This metric is available for basic monitoring only Worldline Germany GmbH Page 45

46 Metric Description Units: Count Statistics: Minimum, Maximum, Average StatusCheckFailed A combination of StatusCheckFailed_Instance and StatusCheckFailed_System that reports if either of the status checks has failed. Values for this metric are either 0 (zero) or 1 (one.) A zero indicates that the status check passed. A one indicates a status check failure. Note Status check metrics are available at 1 minute frequency. For a newly-launched instance, status check metric data is only available after the instance has completed the initialization state. Status check metrics become available within a few minutes of the instance being in the running state. Units: Count StatusCheckFailed_Instance Reports whether the instance has passed the Amazon EC2 instance status check in the last minute. Values for this metric are either 0 (zero) or 1 (one.) A zero indicates that the status check passed. A one indicates a status check failure. Note Status check metrics are available at 1 minute frequency. For a newly-launched instance, status check metric data is only available after the instance has completed the initialization state. Status check metrics become available within a few minutes of the instance being in the running state. Units: Count StatusCheckFailed_System Reports whether the instance has passed the EC2 system status check in the last minute. Values for this metric are either 0 (zero) or 1 (one.) A zero indicates that the status check passed. A one indicates a status check failure. Note Status check metrics are available at a 1 minute frequency. For a newly-launched instance, status check metric data is only available after the instance has completed the initialization state. Status check metrics become available within a few minutes of the instance being in the running state Worldline Germany GmbH Page 46

47 Metric Description Units: Count AWS CloudWatch User Guide The following dimensions are used to filter Amazon EC2 metrics. Dimension AutoScalingGroupName Description This dimension filters the data you request for all instances in a specified capacity group. An Auto Scaling group is a collection of instances you define if you're using Auto Scaling. This dimension is available only for Amazon EC2 metrics when the instances are in such an Auto Scaling group. Available for instances with Detailed or Basic Monitoring enabled. ImageId This dimension filters the data you request for all instances running this Amazon EC2 Amazon Machine Image (AMI). Available for instances with Detailed Monitoring enabled. InstanceId This dimension filters the data you request for the identified instance only. This helps you pinpoint an exact instance from which to monitor data. InstanceType This dimension filters the data you request for all instances running with this specified instance type. This helps you categorize your data by the type of instance running. For example, you might compare data from an m1.small instance and an m1.large instance to determine which has the better business value for your application. Available for instances with Detailed Monitoring enabled. AWS CloudWatch User Guide 13.3 Amazon CloudWatch Custom Metrics Custom metrics are values which are not measured by CloudWatch by default. Values of custom metrics must be collected by yourself locally on the AWS EC2 instance and send to Amazon CloudWatch (API cloudwatch:putmetricdata ). Afterwards the receipt of custom metric from CloudWatch can be performed as for standard (built-in) metric with the exception of specifying a custom namespace. The CloudWatch namespace for standard metrics is preassigned to AWS/EC2 by default Worldline Germany GmbH Page 47