Log Analytics with Amazon Elasticsearch Service. Christoph Schmitter

Transcription

1 Log Analytics with Amazon Elasticsearch Service Christoph Schmitter

2 What we'll cover Understanding Elasticsearch capabilities Elasticsearch, the technology Aggregations; ad-hoc analysis Amazon Elasticsearch Service is a drop-in replacement for self-managed Elasticsearch Q&A

3 Understanding Elasticsearch capabilities

4 Scenario: Log data analytics Application monitoring and event diagnosis You need to monitor the performance of your application, web servers, and hardware You need easy to use, yet powerful data visualization tools to detect issues in near real-time You want the ability to dig into your logs in an intuitive, fine-grained way Kibana provides fast, easy visualization

5 Scenario: Batch data analytics Reporting and Analysis You are a mobile app developer You have to monitor/manage users across multiple app versions You want to analyze and report on usage and migration between app versions Use Kibana for dashboarding. Use the query API for deeper analysis

6 Scenario: Full-text search Traditional search Your application or website provides search capabilities over diverse documents You are tasked with making searchable this knowledge base and accessible You need key search features including text matching, faceting, filtering, fuzzy search, auto complete, and highlighting Use the query API to support application search

7 CloudTrail delivers API calls to you AWS API call monitoring You need to understand the changing landscape of your AWS resources You need to do security analysis and compliance auditing You want the ability to dig into your logs in an intuitive, fine-grained way

8

9 How Elasticsearch can help Combined with Kibana, Elasticsearch provides a tool for search, real-time analytics, and data visualization

10 Demo Architecture AWS Resources CloudTrail Logs Amazon CloudWatch Logs Amazon Elasticsearch Service

11 Log lines

12 Demo: Log Analytics

13 Elasticsearch the technology

14 Elasticsearch is like a database Search Value Field Document Index Cluster Database Value Column Row Table Database Queries SQL

15 Documents are the core entity { "eventversion": "1.03", "eventtime": " T00:16:19Z", "eventsource": "dynamodb.amazonaws.com", "eventname": "DescribeStream", "awsregion": "eu-west-1", "sourceipaddress": " XX", "useragent": "leb-kcl a6-5f94-4ce0-ac69-cdeb609ba16a,amazonkinesis-client-library-java-lambda_1.2.1, aws-internal/3", "requestparameters": { "streamarn": "arn:aws:dynamodb:eu-west- 1: XXXX:table/restaurant/stream/ T18:07:53.837" }, "responseelements": null, "requestid": "KC608PH8POAF2I184E2SL1PS2FVV4KQNSO5AEMVJF66Q9ASUAAJG", "eventid": "49b b-4f04-8ce5-d21bbfcf8ab3", "eventtype": "AwsApiCall", "apiversion": " ", "recipientaccountid": " XXXX", "useridentity": { "type": "AssumedRole", "principalid": "AROAJBQVRM7LN25CAHX7Y:awslambda_338_ ", "arn": "arn:aws:sts:: :assumed-role/geospatial-rec- engine-applicationexecutionrole- 9LPKB77QMR97/awslambda_338_ ",... ID F1 F2 Value Value

16

17 Lucene provides text analysis and indexing Segment Index Writer 0 quick 1,3,5 1 brown 2,3,4,6 2 fox 1,7,9 3 lazy 2,8 4 dog 24 Term ID Term Postings Index Searcher

18 Elsaticsearch query processing quick brown Query fox lazy lorem ipsum dolor sit Query logic and postfiltering id: 216 id: 305 id: 486 id: 713 Matches Scoring, aggs id: 713 id: 305 id: 486 id: 216 Sorted matches (results) Index Lookup

19 Aggregations; ad-hoc analysis

20 Faceting: basic aggregation Query: shirt ID F1 Value F2 Value Facets Carhartt (1092) Russell Athletic (1087) Dickies (954) RALPH LAUREN (823) Wrangler (701) Doublju (259) Levi's (12)

21 Elasticsearch Aggregations Buckets a collection of documents meeting some criterion Metrics calculations on the content of buckets. Metric: count Bucket: time

22 A more complicated aggregation Bucket: ARN Bucket: Region Bucket: eventname Metric: Count

23 More kinds of aggregations Buckets Date histogram Histogram Range Terms Filters Significant terms Metrics Count Average Sum Min Max Std. Dev Unique Count Percentiles

24 Setting up your cluster

25 Shards: independent collections of documents {Index/Type Shard 1 Shard 2 Shard 3 Shard 4 { { { { Id Id Id... Documents

26 Deployment of indices to a cluster Index 1 Primary Replica Amazon ES cluster Shard Shard 2 Shard Index 2 Shard 1 Shard Instance 1, Master Instance 2 Shard Instance 3

27 Determining storage Data:Index ratio is typically close to 1:1 Add a replica, double the storage Figure out data node count based on storage Current limits; 10T EBS, 32T instance store

28 Determining instance type Instance type is workload-dependent T2; dev, test, QA M3; solid performance R3; heavier queries, aggs I2; largest storage option

29 Best practices Take the minimum number of shards for 50G max data per shard Number of replicas = 1 For all prod workloads: use 3 dedicated masters Use the _bulk API. Some ingest mechanisms do this automatically Increase index.refresh_interval for higher throughput

30 Indexing strategy

31 Indexing strategy for streaming data Use an index per time period, typically indexper-day, high volume can go to index-per-hour Shard the index according to data size; use 50GB as a soft limit per shard Master nodes increase cluster stability

32 Index settings control sharding and more curl -XPUT <endpoint>/<index>/_settings -d '{ "number_of_shards" : 5, "number_of_replicas" : 1, "refresh_interval": "5s" }'

33 Mappings control how data is indexed curl -XPUT <endpoint>/<index> -d '{ "mappings" : { <type> : { "properties" : { "eventname" : { "type" : "string", "index" : "not_analyzed" } } } } }'

34 Index templates simplify mapping creation curl -XPUT <endpoint>/_template/<name> -d '{ }' "template" : "<wildcard e.g. cwl-*>", "settings" : { "number_of_shards" : 2 }, "mappings" : { <type, e.g. _default_> : { "dynamic_templates" : [ { <name> : { "index" : "not_analyzed" } } ] "properties" : { "@timestamp" : { "type" : "date" } } }

35 Don't forget the query API!

36 Direct access to the Elasticsearch API $ curl -XPUT -d '{ "settings" : { "number_of_shards" : 3, "number_of_replicas" : 1 } }' $ curl -XPOST -d '{ "author":"jon handler", "title":"amazon ES Launch" }' $ curl -XPOST -d ' { "index" : { "_index" : "blog", "_type" : "post", "_id" : "2"}} {"title":"amazon ES for search", "author": "carl meadows"}, { "index" : { "_index":"blog", "_type":"post", "_id":"3" } } { "title":"analytics too", "author": "vivek sriram"}' $ curl -XGET {"took":16,"timed_out":false,"_shards":{"total":3,"successful":3,"failed":0 },"hits":{"total":2,"max_score": ,"hits":[{"_index":"blog","_type": "post","_id":"1","_score": ,"_source":{"author":"jon handler", "title":"amazon ES Launch" }},{"_index":"blog","_type":"post","_id":"2","_score": ,"_source":{ "title":"amazon ES for search", "author": "carl meadows"},}]}}

37 Elasticsearch is a full-featured search engine Built on Lucene, the popular, open-source library Search structured and unstructured data with complex, boolean queries Supports common search features: geo search, aggregations, highlighting, search suggestions, and more

38 Challenges with self-managed Elasticsearch Easy to get started, challenging to scale Scaling ingest pipelines is difficult Undifferentiated heavy lifting

39 Amazon Elasticsearch Service

40 Amazon ES overview CloudTrail CloudWatch Elasticsearch API IAM Elastic Load Balancing Amazon Route 53

41 Easy cluster configuration and reconfiguration AWS Elasticsearch Version Data nodes, count and type Master nodes, count and type Storage option EBS/instance HA option Advanced options

42 High availability with Zone Awareness Amazon ES cluster Instance 1 Instance 2 Instance 3 Instance 4 Availability Zone 1 Availability Zone 2

43 Monitor with CloudWatch metrics FreeStorageSpace monitor and alarm before the cluster runs out of space CPUUtilization alarm at 80% CPU to signal the need to scale up ClusterStatus.yellow check whether replication requires additional nodes JVMMemoryPressure check instance type and count for sufficient resources MasterCPUUtilization monitoring for master nodes is separated from data nodes

44 Amazon Kinesis Amazon RDS Amazon DynamoDB Amazon SQS Queue Integration with the AWS ecosystem Logstash Logstash Cluster REST CWL Agent EC2 Instances Amazon CloudWatch AWS Lambda Amazon Elasticsearch Service Amazon ECS AWS CloudTrail Access Logs Amazon VPC Flow Logs Amazon S3 bucket AWS IoT Amazon Kinesis Firehose

45 Security with IAM { "Version": " ", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam: :user/susan" }, "Action": [ "es:eshttpget", "es:eshttpput", "es:eshttppost", "es:createelasticsearchdomain", "es:listdomainnames" ], "Resource": "arn:aws:es:us-east-1:###:domain/logs-domain/<index>/*" } ] }

46 Pay for compute and storage you use With Amazon Elasticsearch Service, you pay only for the compute and storage resources you use. AWS Free Tier for qualifying customers.

47 Wrap up Combined with Kibana, Elasticsearch provides search and visualization for streaming data and full-text use cases. Elasticsearch is based on Lucene, which reads and writes search indices Aggregations allow you to analyze your data, splitting into Buckets and computing Metrics Amazon Elasticsearch Service makes it easy to set up and manage your Elasticsearch cluster on AWS Amazon ES is a great way to get started with Elasticsearch!

48 Q&A Christoph Schmitter: Solutions Architect

49 Demo Screenshots

50

51

52

53

54