Splunk & Amazon Web Services

Transcription

1 Splunk & Amazon Web Services June 2016 Tony Bolander Daniel Lew 1

2 Industry Leading Platform For Machine Data Machine Data: Any Location, Type, Volume Answer Any Question On- Premises Servers Online Services Security Web Services Networks GPS Location Packaged Applications Ad hoc search Monitor and alert Report and analyze Custom dashboards Developer Platform Private Cloud Storage Desktops Messaging Custom Applications Public Cloud Online Shopping Cart Smartphones and Devices Telecoms Web Clickstreams RFID Call Detail Records Databases Energy Meters Platform Support (Apps / API / SDKs) Enterprise Scalability Universal Indexing

3 Splunk Insights for AWS Machine Data Why Splunk for AWS? Security Intelligence (Cloudtrail, Config Cloudwatch, Inspector, VPC) Operational Intelligence (Cloudwatch, Config, RDS, ELB, EC2, S3, Cloudfront) DevOps Intelligence (Cloudwatch, Lambda) Big Data Insights (Kinesis, EMR, IoT, S3) Service Billing & Usage Explore Analyze Dashboard Alert Splunk App for AWS Add-on for AWS CloudTrail CloudWatch Config Inspector Billing & Other Services EC2 ECS S3 Cloudfront VPC ELB EMR Kinesis Lambda RDS IoT

4 Ingests Data From Heterogeneous Data Sources syslog TCP/UDP Local File Monitoring Splunk Forwarder syslog hosts and network devices *nix Windows Mainframes Mounted File Systems \\hostname\mount Active Directory Wire Data Splunk App for Stream Scripted or Modular Inputs shell scripts API subscriptions Unix, Linux and Windows hosts HTTP HTTP Event Collector shell perf API virtual host

5 Splunk App for AWS: The Data AWS Cloudtrail Service that records AWS API calls for your account and delivers activity logs Provides data to enable security analysis, resource change tracking, compliance auditing, and troubleshooting AWS Config & Config Rules Service that provides resource inventory, configuration history and configuration change notifications Config Rules enables creation of rules to auto-check AWS configurations Provides data to enable resource discovery, service relationships, change tracking & troubleshooting Amazon Cloudwatch Service that collects AWS system metrics and log files Offers ability to stream logs via Amazon Kinesis Provides data to enable utilization & health reporting for services such as EC2, EBS, & RDS Amazon Cloudwatch VPC Flow Logs Service that enables capture of IP traffic information to/from VPC network interfaces Data stored and accessible from AWS Cloudwatch Logs Provides data used to troubleshoot undesired traffic behavior for both operational and security use cases Amazon Inspector Automated security assessment service to help improve security and compliance of apps on AWS Provides data from knowledge base and security findings based on security best practices AWS Access Logs Elastic Load Balancing (ELB) Provides data on load balancer requests to anlayze traffic patterns Cloudfront CDN Provides data about every user request received from Cloudfront S3 Provides data about a single access request and can be used for security and access audits AWS Billing Current Month via Cloudwatch metrics Monthly Detailed Billing for Capacity Management 5

6 Getting Started! Create a Splunk account: Access Splunk AMIs on AWS Marketplace and then set-up Splunk App for AWS & AWS Technology Add-On *or* Access Splunk Cloudformation template by following these directions. This environment will include the Splunk App for AWS and Splunk TA for AWS Be sure to take self-paced Using Splunk tutorial + Review Splunk>Docs and Splunk>Apps Automate your deployment: Puppet: Chef: 6

7 Splunk & Amazon Web Services June

8 Why is Splunk Important For AWS Customers? You can t protect what you can t see. Security monitoring will make or break a technology risk management program. Security requires visibility. Best Practices for Securing Workloads in Amazon Web Services Gartner, April 2015 Neil MacDonald, Greg Young Assessing the Risk: Yes, the Cloud Can Be More Secure Than Your On-Premises Environment IDC, July 2015 Pete Lindstrom Amazon Web Services Intro to AWS Security 2015 AWS Summit Series 8

9 Extrapolating for Other AWS Use Cases You can t operate what you can t see. You can t manage cost for what you can t see. You can t gain business analytics for what you cant see. 9

10 Splunk s AWS Credentials AWS Advanced Technology Partner AWS Big Data Competency AWS Security Competency AWS Government Competency AWS SaaS Sales Alignment Program (Internal Program) AWS MSP Technology Provider AWS Marketplace Partner AWS IoT Launch partner for IoT analytics AWS Security by Design Program Partner 1 st partner with published Blueprints for AWS Lambda 1 st partner to pass SaaS extension for Well Architected framework

11 Splunk Portfolio of Cloud/AWS Solutions Cloud Apps & Solutions Splunk App for AWS, ServiceNow, SFDC, Box, more AMIs & Cloudformation AMI for Splunk Enterprise AMI for Splunk Light AMI for Hunk Cloudformation Templates Enterprise on AWS Splunk Core + Enterprise Security & ITSI available 100% Uptime SLA SOC2 Type II Certified Available on AWS & Cloud Starts at $90 / Month App for AWS Hunk for AWS EMR Splunk Analytics for AWS Elastic MapReduce (Hadoop/HDFS) AWS Specific Integrations AWS Lambda: First partner blueprint AWS IoT: Featured analytics platform AWS Kinesis: TA & Mod Input AWS EC2 Container Service: Splunk Driver 11

12 Splunk Insights for AWS Machine Data Why Splunk for AWS? Security Intelligence (Cloudtrail, Config Cloudwatch, Inspector, VPC) Operational Intelligence (Cloudwatch, Config, RDS, ELB, EC2, S3, Cloudfront) DevOps Intelligence (Cloudwatch, Lambda) Big Data Insights (Kinesis, EMR, IoT, S3) Service Billing & Usage Explore Analyze Dashboard Alert Splunk App for AWS Add-on for AWS CloudTrail CloudWatch Config Inspector Billing & Other Services EC2 ECS S3 Cloudfront VPC ELB EMR Kinesis Lambda RDS IoT

13 How FINRA Uses Splunk Cloud for Security Transforms third-party threat intelligence information into security alerts Leverages the Splunk App for AWS Extends solution to report on AWS Cost Optimization Splunk Cloud gives you applications which let you get huge amounts of value from your data. Sr. Director of Information Security 13

14 Reduced error rates by 2 orders of magnitude in a couple of weeks Better Code, Faster Development Rapidly found and fixed one line of code responsible for 30,000+ errors Real-time dashboards on error rates and production impact In-depth visibility as they strategically migrate apps to AWS Cloud and Migration to Cloud 14

15 Supporting Global Websites Real-time insight ensures an optimum customer experience, even during peak sales periods Proactive troubleshooting results in faster resolution of issues Real-time monitoring ensures confidence in the cloud When I look at the e-commerce chain from customer service, through to the warehouse and even in the physical stores there s opportunity to drive value with Splunk everywhere. E-Commerce Systems Architect, Kurt Geiger

16 Splunk App for AWS v4.2

17 Splunk App for AWS: The Data AWS Cloudtrail Service that records AWS API calls for your account and delivers activity logs Provides data to enable security analysis, resource change tracking, compliance auditing, and troubleshooting AWS Config & Config Rules Service that provides resource inventory, configuration history and configuration change notifications Config Rules enables creation of rules to auto-check AWS configurations Provides data to enable resource discovery, service relationships, change tracking & troubleshooting Amazon Cloudwatch Service that collects AWS system metrics and log files Offers ability to stream logs via Amazon Kinesis Provides data to enable utilization & health reporting for services such as EC2, EBS, & RDS Amazon Cloudwatch VPC Flow Logs Service that enables capture of IP traffic information to/from VPC network interfaces Data stored and accessible from AWS Cloudwatch Logs Provides data used to troubleshoot undesired traffic behavior for both operational and security use cases Amazon Inspector Automated security assessment service to help improve security and compliance of apps on AWS Provides data from knowledge base and security findings based on security best practices AWS Access Logs Elastic Load Balancing (ELB) Provides data on load balancer requests to anlayze traffic patterns Cloudfront CDN Provides data about every user request received from Cloudfront S3 Provides data about a single access request and can be used for security and access audits AWS Billing Current Month via Cloudwatch metrics Monthly Detailed Billing for Capacity Management 17

18 Splunk App for AWS: The Value Increase visibility into AWS resource utilization & user activity across all accounts Ensure adherence to security and compliance standards with a audit reporting Understand AWS environmental dependencies via interactive topology visualization Monitor VPC traffic utilization for additional patterns & security insights Cost Optimization through Monthly and Detailed Billing Dashboards 18

19 Overview for Splunk App for AWS The overview page shows you on one screen information about: Configuration changes Compute Storage Billing ELB Cloudfront Security Notable CloudTrail Activity is highlighted on the map. Drill down on any event and gain detailed information.

20 AWS Topology Topology view gives you a holistic view of your current or historical AWS deployment using AWS Config Maps out relationships between all the components, giving you a clear view into the environment Clickable layers adds additional visual queues for high CPU or network traffic Snapshot feature allows for topology to be saved for future use Config

21 AWS Topology - Expanded Visuals Config CloudWatch CloudTrail

22 AWS Topology - IAM IAM Topology view uses AWS Config to provide a comprehensive view of Identity and Access Management Information Provides visual way to manage IAM Users, Groups and Policies Select entity of interest to see IAM relationships

23 AWS Usage Overview In one glance, instantly see your EC2 usage and EBS Volume data info via Cloudwatch metrics Click through dashboards for details on individual EC2 instances and EBS Volumes Drill down into raw search for even more detailed views on your instances.

24 VPC Flow Data - Traffic Utilizes VPC Flow Logs from Cloudwatch for Traffic Analysis Visualize VPC traffic by interface, time, and location

25 VPC Flow Data - Security Utilizes VPC Flow Logs from Cloudwatch for Security Analysis Drill down into rejected vs. accepted traffic View top Source Country and City information See top source / destination and IP Addresses and ports

26 AWS Billing & Capacity Planning Utilizes Billing Logs from Cloudwatch for Month-to-Date billing and End-of-Month projections Detailed Historical Billing Dashboard available using Monthly AWS Detailed billing reports Capacity Planner gives additional clarity on AWS On-Demand instance spending

27 AWS S3 Access S3 Access logs provide visibility on the health, requests, and traffic volume handled by your S3 bucket objects across all accounts. Aggregations by requester, useragent, and error codes give insights for troubleshooting, security and general product/business analytics.

28 AWS Elastic Load Balancer ELB dashboards provide visibility on the health, latency and request volume of your load balancers Client and server side errors are surfaced (HTTP 4XX-5XX errors) by account and region

29 AWS Cloudfront CDN Cloudfront Dashboards displays visitor information per edge location, referrers, cache hits/misses and traffic volume Provides operational utility by adding visibility to errors, latency, distribution Provides business insights such as geo location of visitors, user agents and referrers.

30 AWS User & IAM Activity Utilizes Cloudtrail data to quickly see the number of active users logged into the system Get alerted on Unauthorized user activities and create additional alerts for any user action See what ARN s are being used to access services and the correlated functions

31 AWS Key Pairs Activity Utilizes Cloudtrail data to quickly see number of In-Use Key Pairs, Error events and actions Reports on Key Pair usage by Region and activity over time

32 Getting Started! Create a Splunk account: Access Splunk AMIs on AWS Marketplace: archterms=splunk and then set-up the Splunk App for AWS & AWS Technology Add-On *or* Access Splunk Cloudformation template by following these directions. This environment will include the Splunk App for AWS and Splunk TA for AWS Be sure to take self-paced Using Splunk tutorial + Review Splunk>Docs and Splunk>Apps 32

33 AWS Extras

34 Utilizes new Splunk HTTP Event Collector Enabling Developers by Monitoring Lambda functions Use Lambda to pipe events from services like Kinesis to Splunk Configure in the AWS Console or use our JavaScript and Java logging libraries Splunk & AWS Lambda 34

35 Splunk & AWS IoT Visibility into Connected Devices communicating with Cloud Apps Enables advanced analytics & insights for IoT deployments

36 Hunk & AWS Elastic Map Reduce (EMR) Gain insights - Explore, analyze, and visualize Amazon EMR and Amazon S3 data at massive scale Unlock the business value of stored data Preview search results before MapReduce jobs finish Quickly conduct sophisticated analytics Easily provision Hunk from AWS EMR Console Use for only as long as you need it Charged by the hour 36

37 Splunk Enterprise on AWS Deployment Guidelines 37 Workload = Searching + Indexing Storage - Ephemeral or EBS - Data Retention Dependent Compute - Best Available Archiving - S3 Best Practices for Sizing Splunk on AWS Tech Brief Splunk Cloudformation Templates Splunk Admin Docs Search Heads (8+ users) c4.4xlarge 16 vcpu, 30 GB RAM c4.8xlarge 36 vcpu, 60 GB RAM Indexers (50-250GB/day/indexer) c4.4xlarge d2.4xlarge c4.8xlarge 16 vcpu, 30 GB RAM 16 vcpu, 122 GB RAM 36 vcpu, 60 GB RAM CloudFormation Templates Consistent, repeatable deployments for Splunk on AWS Abstract away details of configuring distributed Splunk Extensible and customizable to fit any need CF Templates On GitHub