Event Driven network automation

Transcription

1 Event Driven network automation Openlab Technical Workshop 2018 Salvador Ferrer January

2 Agenda StackStorm event driven automation What can be automated? 2

3 3

4 StackStorm is like f or DevOps Rule Trigger Action 4

5 IFTTT, for DevOps cat /opt/stackstorm/packs/st2-demos/rules/demo.yaml --- name: "sensu_crit_to_slack" pack: "st2-demos" description: "Post all critical alerts to the demo enabled: true trigger: type: "sensu.event_handler" criteria: trigger.check.status: pattern: 2 type: "equals" trigger.check.name: pattern: "demo_.*" type: "matchregex" action: ref: "slack.post_message" parameters: message: > [ALERT]{{trigger.client.name}} {{trigger.check.output}} channel: "#demos" 5 5

6 Event-driven, Cross-domain Automation Platform Workflows Rules Sensors Actions Inbound integration plugins that watch for events from external systems Outbound integration plugins that request changes to external systems Multi-vendor Network Domain Compute Domain Storage Domain Ops Support Services Cloud Infrastructure IT Domains 6

7 On-call, Without Automation 2:00 AM 2:02 AM 2:07 AM 2:10 AM 2:15 AM 2:20 AM 2:30 AM PagerDuty Alert Engineer Wakes up Logs in and ACK Studies the alert Checks runbook Runs diagnostics Fixes the problem Source: Winston: Helping Netflix Engineers Sleep at Qcon 16 SF 7

8 On-call With Winston 2:00 AM False Positive Fixed the problem 2:05 AM 2:05 AM Winston Assisted Diagnostics 2:15 AM Source: Winston: Helping Netflix Engineers Sleep at Qcon 16 SF 8

9 What can be automated? 9 9

10 What is being automated with StackStorm Security checks On malware detection in a VM, isolate network port on a switch App blue-green deployment On Jenkins tests passed, bring new vm claster, deploy and configure app, set loadbalancer to send % of traffic to new app, monitor, roll forward, or back out Networking On BGP peer goes down: collect troubleshooting data, post on slack & create JIRA ticket On Link aggregation member error, check load, if capacity of rest of LAG bundle enough, disable link with error OpenStack orphan VM clean-up: On orphans detected, shut down, owner, keep for few days, delete VM evacuation on HW failures: On host RAID failure, get list of impacted VMs, VM owners, evacuate VMs, create JIRA ticket for hardware replacement. Service remediation: Cassandra node down recovery: On ring node dying, deploy new node, configure, add to the ring. Remediating RabbitMQ, Galera cluster, MySQL, and more 10

11 Troubleshooting and Remediation Example: Turning 2:00 a.m. calls into 10:00 a.m. follow-ups ebgp Switch Failure (BGP) Spine Private ASN Elapsed Time Event Switch link goes down. Leaf Seconds Switch sends SYSLOG message. Seconds EWC SYSLOG sensor matches error message (e.g., REGEX). Seconds EWC sensor triggers troubleshooting workflow. Seconds EWC action extracts information: Switch IP, switch peer IP address, egress interface Help Desk Slack <1 Minute <1 Minute EWC action logs into switch, performs the following actions: Execute workflow to determine service and interface state If interface state is down, attempt to restart interface EWC action creates help desk ticket with outputs from above. SYSLOG Extreme Workflow Composer Seconds EWC initiates an alert workflow if interface cannot be restarted: Include help desk ID and URL Post message to Slack for network ops team visibility Launch incident to PagerDuty to notify operator PagerDuty Total elapsed time: <5 minutes 11

12 Infrastructure Service Provisioning 1. ZTP process registers the switch to the inventory service through the registration sensor. Data center fabrics 2. Registration triggers the IP Fabric Leaf workflow. 3. Workflow engine models the IP fabric. 4.Extreme Workflow Composer walks through the steps in the workflow. 5. Configuration is executed through the appropriate action. 6. IP fabric provision is complete. 7. Validation workflow begins. DAD (ZTP) Credential / Authorizatio n 1 Visual Flow VDX (NOS) 3 4 Workflo w Engine Registration Sensor Web GUI VDX (NOS) 7 Device Inventor y CL I Data Store Message Bus VDX (NOS) Leaf/Spine Validation 2 Rules Engine 6 IP fabric Chat Ops VDX (NOS) REST API Configuration Action 5 Audit 12

13 Agility through Automation and Visibility Accelerating mean-time-to-innocence through automation Extreme SLX Family Devices Extreme SLX Insight Architecture My application is slow! Extreme SLX Visibility Services ACTION: Start traffic analysis RESULT: Not a network problem Extreme Workflow Composer Distributed App 13

14 Device Abstraction through NAPALM NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different router vendor devices using a unified API. EOS JunOS IOS-XR NXOS IOS Module Name napalm-eos napalm-junos napalm-iosxr napalm-nxos napalm-ios Driver Name eos junos iosxr nxos ios Structured data Yes Yes No Yes No Minimum version F [1] 12.4(20)T Backend library pyeapi junos-eznc pyiosxr pynxos netmiko Caveats EOS NXOS IOS 14

15 Published Demos Cassandra Auto-Remediation Splunk + EWC + VDX + Jira + Slack Elastic + EWC + VDX + Jira + Slack EWC + VDX + Docker StackStorm + NAPALM Autoscaling Swarm on AWS 15

16 Event Driven network automation Questions? Salvador Ferrer January