Splunk for Akamai Cloud Monitor

Transcription

1 Copyright 2015 Splunk Inc. Splunk for Akamai Cloud Monitor Pierre Pellissier Leela Kesireddy Performance Management PayPal, Inc.

2 Disclaimer During the course of this presentaeon, we may make forward looking statements regarding future events or the expected performance of the company. We caueon you that such statements reflect our current expectaeons and esemates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentaeon are being made as of the Eme and date of its live presentaeon. If reviewed arer its live presentaeon, this presentaeon may not contain current or accurate informaeon. We do not assume any obligaeon to update any forward looking statements we may make. In addieon, any informaeon about our roadmap outlines our general product direceon and is subject to change at any Eme without noece. It is for informaeonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaeon either to develop the features or funceonality described or to include any such feature or funceonality in a future release. 2

3 ! Why we need Log Analysis! What we had was too slow! Real Time logging OpEon! GeXng to Real Time! Benefits & Challenges! ConfiguraEon Details! Akamai Splunk ApplicaEon! Final Analysis and Q & A Agenda 3

4 Why we need log analysis Customer interaceons on Akamai exceed 1 BN/day It is business criecal that we : Know the state of the site Track the impact of changes Analyze the customer experience 4

5 What we had was too slow! Akamai for content delivery, web acceleraeon and security! Akamai log colleceon takes hours to get logs into NetStorage.! Downloading logs hourly and indexing them in Splunk makes them available for analysis.! But this is not a real Eme solueon. Akamai Network WAF Log Gathering DC #1 DC #2 NetStorage Hourly download 5

6 Real Eme logging opeon! Akamai Cloud Monitor delivers logs in < 60 sec! Faster delivery enables real Eme operaeonal monitoring and easier analyecs by internal users.! Lots of configuraeon decisions.! The results are worth the effort.! UlEmately easier than expected. Akamai Network WAF CM Log Streaming Aggregation Real Time Delivery DC #1 DC #2 6

7 GeXng to real Eme! OperaEonal goals required: Real Eme visibility into site Improved usability of logs! Lots of design decisions needed to be made How will we receive the logs? How do we configure CM on Akamai? How will this change the indexed volume? How do we ensure we get the logs? How will the logs be used?! The inieal benefits were achieved quickly MulEple bonus benefits were also realized Some challenges needed to be dealt with Which Sites? What aggregaeon? Redundancy? Dashboards views? Which data? CM Log Streaming DC #2 Aggregation Real Time Delivery How to receive? What volume? 7

8 IniEal benefits site control! Receiving the logs in <60 sec. makes them usable for real Eme analysis The NOC can immediately see the effects of site changes The CDN team can see the effects of Akamai configuraeon changes in real Eme Monitoring of site ramp acevity. 8

9 IniEal benefits log usability! NetStorage log parsing needed improving! Akamai CM logs use JSON formaxng! Splunk TA for CM logs put them into CIM format for ES consumpeon! Internal customers can work with the JSON formaned logs more intuievely 9

10 AddiEonal benefits opeonal CM data! OpEonal data fields enable rich analyecs for: OperaEonal Teams Performance Security Fraud! Adding more fields results in higher indexed volume 10

11 AddiEonal benefits custom fields! Data from the conneceon, HTTP header or payload can be inserted into CM! Enables powerful insight for muleple teams: Crypto use analysis Customer Experience Referer AnalyEcs TroubleshooEng 11

12 AddiEonal benefits more complete data! For real Eme analyecs we chose to receive 100% of the CM logs Some use cases may only need 10%! NetStorage log colleceon does not capture 100% of the logs Akamai Network WAF Sample percentage = 100%! The CM log count is ~5% greater than the NetStorage log count This increases our confidence that we are seeing the enere picture Real Time Delivery DC #1 DC #2 12

13 Challenges! Greater log volume with CM ~50% more data indexed Partly due to JSON formaxng Mostly due to addieonal CM data! IniEal configuraeon of log receiver did not provide redundancy CM Logs were lost when it was unavailable! Log forwarding logic took several iteraeons to get it right A POC got us close and then we adjusted arer we were in produceon Akamai Network WAF DC #1 DC #2 Delivery Failure 13

14 ConfiguraEon Details

15 CM & splunk checklist Ini$al configura$ons: q Add Cloud Monitor to the Akamai contract q Set up receiver VIP, DNS, SSL Cert and redundant servers q Choose the property and CM data sets to log q Build the inieal CM and Splunk configuraeons Trial Run: q Configure for a short duraeon of logging at 10% to verify receiver is working q Verify the CM data is being mapped to Splunk CIM q Verify that the receiver and Splunk can support the expected load q Verify the data is what you need and expect Plan on doing several rounds of configura$ons, tes$ng and tuning q It is easy to iterate and change the configuraeons as needed 15

16 CM data delivery!!!! Select Data Sets to Include Configure Delivery Endpoint Configure the AggregaEon opeons: Time: 60 seconds max Line Count: Max 3000 records Message Size: Max 900 KB of data Configure DistribuEon & Failover OpEons: Primary receiver gets 100% Secondary receiver gets 100% if Primary receiver is unavailable NetStorage gets 100% if Primary & Secondary are unavailable ê With scheduled FTP download hourly 16

17 CM configuraeons! Default data sets: CP, Format, Message, Type, Version! OpEonal data sets: Akadebug, network, netperf, Geo, WAF, PPCustomData ê Can not opt out of data elements within an opeonal data set ê Results in duplicates of some data or unwanted data! Custom Data field: Selected HTTP Header data is included Other header info is excluded- like large cookies 17

18 CM receiver! Receiver build Akamai Posts CM data to receiver VIP SSL cert required for secure conneceon by Akamai MulEple Linux servers in pool running nginx & node.js Writes logs to a local data file Splunk Universal Forwarder monitors logs and forwards them to Indexers 18

19 CM receiver (future state)! Plan for receiver redundancy in the inieal design CM VIP1 CM VIP2 CM VIP1 Splunk VIP NetStorage CM VIP2 19

20 Splunk configuraeon! Index configured with 100 day reteneon! Built Add- on s with Field renames & EvaluaEons! Normalized data to be CIM Complaint! LDAP group configured for accessing CM data! Apps set up for internal teams to build and share searches! Common Akamai CM Dashboard for all teams Performance & Availability Dashboard for each property StarEng point for internal teams to view data but with click through access to granular data and search bar 20

21 New- Splunk 6.3 HTTP event collector! HTTP endpoint can securely receive high- volume JSON- based applicaeon and IOT data! Create and mange receiver configuraeons using the HTTP event collector configuraeon! Token based authenecaeon Model! Supports to both hnp & hnps! Replaces the nginx & node.js receiver solueon 21

22 Akamai ApplicaEon

23 Akamai App! Monitor the Real Time Traffic! Insights by Property! Monitor Health & Performance of Origin! Monitor Heath & Performance of Akamai Edge! Observe User s Pain Points! Inspect App / URL level Issues & Performance! InvesEgate Issues 23

24 ! Performance Issue Searches Were Slow Dashboards taking longer to load Greater than 25K events/sec! Summary Index Loses the Rich informaeon in data Hurdles! Report AcceleraEon AcceleraEon is suspended as the summary reaches 10% of its total size No Control on Summary Schedule 24

25 Akamai ApplicaEon Data Models & AcceleraEon Came to Rescue 25

26 Data Models Data Model s facilitated to format, evaluate fields and work with necessary event data for each data set and also accelerate the model! Data models are made up of object types like lookups, transaceons, fields extraceons & Calculated fields.! Data models encodes only necessary knowledge objects.! Data Model for each Property or group of ProperEes.! Objects by Success or Failure (HTTP Status)! Add addieonal fields using regular expressions, eval expressions and lookups. 26

27 Dashboards! Real Time Availability & Performance! Origin Insights! Edge Overview! Security! Release Monitoring 27

28 Availability Availability & Performance Volume of Hits. Success & Failure Volume by HTTP Status. Success & Failure by GEO Performance Track total Eme for transaceons. Origin Latency Last Mile Round Trip Time 28

29 Origin & Edge Origin Monitor traffic distribueon. Error Monitoring. Monitor Latency. Edge Edge Errors Edge Performance stats Last Mile RTT 29

30 Monitor the WAF denies and warnings. Monitor Top Deny Rules. Report on the WAF warning triggered. Top deny & warning URL s. Deny & warn tracking by Geo Top Denied Client IP s Security 30

31 Monitor Traffic by Origin / DC Monitor Issue s & Performance by Property Performance by Origin Last Mile Time by Edge Performance by GEO Release Monitoring DC 01 DC 02 DC 03 DC 04 DC 05 DC 01 DC 02 DC 03 DC 04 DC 05 31

32 Final Analysis and Q & A

33 Final analysis! The inieal business needs for Real Time logging have been met: Real Eme monitoring enables us to know the current state of the site Dashboards allow us to track the effect of site changes JSON data formaxng makes it easier to do analyecs on the customer experience! The addieonal benefits are extremely valuable: Custom fields to help troubleshoot site and code issues Header informaeon providing customer experience analyecs data Rich data set about customer crypto to help with SHA256 migraeon Performance data on a regional and network level! ConEnuing to find new ways to leverage the CM data in Splunk! Q & A 33

34 THANK YOU