Scripting the Catalyst: Beyond the Basics

Scripting the Catalyst: Beyond the Basics Jeff McLaughlin, Principal TME

Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Agenda Why programmability? Planning a script Python and Tools Finding and using Models NCClient and YDK-Py REST APIs Completing the script On-Box Python

Why Programmability?

Why automation and programmability? Administrator Needs to configure hostname switch1 int g0/0 ip address 10.1.1.11/24 vlan 100,200,300... hostname switch6 int g0/0 ip address 10.1.1.16/24 vlan 100,200,300 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

Notepad is the most common automation tool. It s just a very bad automation tool.... Programmability Reason #1 Do repetitive and tedious tasks more easily 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9

52037606 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored if error counters too high: then shutdown interface* * pseudo-code Programmability Reason #2 Programmatic Control of network devices 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

party apps Cisco ISE NETCONF REST API Catalyst switches APIC-EM Programmability Reason #3 Interaction between network devices and other systems 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

Transactionality int g0/0 ip address 10.1.1.0/24 no shutdown router bgp 65001 router-id 172.17.1.99 bgp log-neighbor-changes neighbor 192.168.1.2 remote-as 40000 neighbor 192.168.3.2 remote-as 50000 address-family ipv4 unicast neighbor 192.168.1.2 activate network 172.17.1.0 mask 255.255.255.0 exit-address-family Programmability Reason #4 Stop bad configuration being committed to devices 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

Operational Simplification # ping 172.16.100.101 # show arp i 172.16.100.101 # show mac address-table address 001a.a24d.5141 # show cdp neighbor g0/1 detail How to find the red user's switch/port? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

Operational Simplification # show mac address-table address 001a.a24d.5141 # show cdp neighbor g0/10 detail How to find the red user's switch/port? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

Operational Simplification How to find the red user's switch/port? # show mac address-table address 001a.a24d.5141 # show cdp neighbor g0/10 detail Vlan Mac Address Type Ports ---- ----------- -------- ----- 244 001a.a24d.5141 DYNAMIC Gi0/15 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

1 User types command into Spark 2 Command pulled down by script REST 5 Data posted back to Spark room 3 Script sends NETCONF request 4 Switch replies via NETCONF with data NETCONF Catalyst 3850 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

Planning your script

What tool to use? CLI CFG MGMT TOOLS PYTHON SCRIPT Advantages: Easy to use Well documented Immediate result Advantages: Device roles Powerful templating Easy to learn Advantages: Numerous libraries Programmable intfs Well supported Disadvantages: Time consuming Tedious Error-prone Disadvantages: Less functionality Disadvantages: Development time 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

Python: Why not C, Ruby, Perl, TCL, Go, etc.? Easy to Learn Interactive shell Easy to begin with simple scripts Libraries! Extensive libraries, like an app store for developers Read Excel, output PDF, draw graphs, etc. And it s well supported in the networking community! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

How do I build it? Catalyst 3850 Read Spark messages Post Spark messages Post PNG image Poll Spark room Parse Spark command Format data from switch Graph data from switch Read the routing table 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

Setting up Python environment

Python 2 vs 3 Python 2.7.10 >>> print "Hello World! Hello World! Python 2 Python 3.6.0a4 >>> print "Hello World!" SyntaxError: Missing parentheses >>> print ("Hello World") Hello World Python 3 Most common version Default installation on Linux Most libraries Losing popularity Several language enhancements Often separate installation Not all libraries supported Commonly used in books/courses Python 2 and 3 are not mutually compatible! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

PIP Install $ pip install netaddr Downloading/unpacking netaddr Downloading netaddr-0.7.19-py2.py3-none-any.whl (1.6MB): 1.6MB downloaded Installing collected packages: netaddr Successfully installed netaddr Cleaning up... PDFrw YDK-Py OpenPy XL Python NCClient 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

Virtual Environments V-Env1 V-Env2 V-Env3 Python 2.5 Python 2.7 Python 2.7 Requests = 1.2.0 NCClient = 0.4.5 Jinja2 = 2.8.1 Jinja2 = 2.7 NCClient = 0.4.5 paramiko = 2.1.1 Requests = 1.2.0 NCClient = 0.5.3 openpyxl = 2.4.1 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

Installing Virtual Environment $ pip install virtualenv Collecting virtualenv Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB) 100% 1.8MB 658kB/s Installing collected packages: virtualenv Successfully installed virtualenv-15.1.0 Install Virtualenv with PIP like any Python library. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

Using VirtualEnvironments $ mkdir spark_app $ cd spark_app $ virtualenv env New python executable in /Users/jemclaug/Documents/Projects/CL LV 2017//env/bin/python Installing setuptools, pip, wheel...done. $ source env/bin/activate (env) $... (env) $ (env) $ deactivate $ Make a directory for your project Create a virtual environment Activate the virtual environment Do Pythonic stuff like PIP installs Deactivate 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

Initial Commit BGP over NC Added oper data Fixed nc_get Initial Commit BGP over NC Added oper data Fixed nc_get Version control scenario 1: You make changes and need to track them Version control scenario 2: You work with collaborator(s) and need to track and merge changes to code. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

Git is a version control system. GitHub is an online source code repository. GitHub provides an easy way to share code and collaborate. It works together with Git. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

Installing git Install Git using the standard installers on Mac/Windows Use the usual methods on Linux (apt-get, yum, rpm, etc.) Highly Recommended: Install GitHub Desktop! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

Tracking your project with Git 1 Create.gitignore file $ cat.gitignore.gitignore *.py[co] env/ (Sometimes) Tells git, don't track these files! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

Tracking your project with Git 2 Initialize the repo JEMCLAUG-M-34E1:spark_app jemclaug$ git init Initialized empty Git repository in /Users/jemclaug/Documents/Projects/CL LV 2017//spark_app/.git/ 3 Add file(s) to be tracked $ echo "My First File" > file.txt $ git add file.txt 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

Tracking your project with Git 4 Commit your changes $ git commit Aborting commit due to empty commit message. $ git commit -m "Added my first file" [master (root-commit) cba52b5] Added my first file 1 file changed, 1 insertion(+) create mode 100644 file.txt 5 View changes in GH Desktop! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

Cloning repo's from GitHub $ git clone https://github.com/ccie14023/pyfabric Cloning into 'pyfabric'... remote: Counting objects: 89, done. remote: Total 89 (delta 0), reused 0 (delta 0), pack-reused 89 Unpacking objects: 100% (89/89), done. $ cd pyfabric/ $ virtualenv env New python executable in /Users/jemclaug/pyfabric/env/bin/python Installing setuptools, pip, wheel...done. $ source env/bin/activate (env) $ pip install -r requirements.txt Collecting cffi==1.9.1 (from -r requirements.txt (line 1)) Using cached cffi-1.9.1-cp27-cp27m-macosx_10_10_intel.whl Collecting cryptography==1.7.1 (from -r requirements.txt (line 2)) Using cached cryptography-1.7.1-cp27-cp27m-macosx_10_10_intel.whl 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

Data Models

Structured vs Unstructured Data Un-structured Structured John Smith 42 14155551212 What is this? His age? The year he graduated college? Meaning of life, the universe & everything? Name: Age: Phone: Keys John Smith 42 +1-415-555-1212 Values 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

Hierarchical Structured Data (XML-like) First User Second User { { <user1> <name>john Smith</name> <age>42</age> <phone>+1-415-555-1212</phone> </user1> <user2> <name>sarah Kim</name> <age>27</age> <phone>+1-718-555-1212</phone> </user2> 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

Ancient unstructured data COULDYOUREADITIFWEWROTELIKETHIS WITHNOPUNCTUATIONITISHARDTOFIGUR EOUTWHEREONEWORDORSENTENCEBE GINSORENDSANDITISNOTEASYTOSEPAR ATEOUTALLTHEELEMENTSOFTHETEXTTH ANKGOODNESSSOMEONECAMEUPWITHA BETTERWAY 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57

Note inconsistent key format! switch1# sh int e1/10 Ethernet1/10 is up Hardware: 1000/10000 Ethernet, address: 0005.73d0.9331 (bia 0005.73d0.9331) Description: To UCS-11 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Switchport monitor is off EtherType is 0x8100 Last link flapped 8week(s) 2day(s) Last clearing of "show interface" counters 1d02h 30 seconds input rate 944 bits/sec, 118 bytes/sec, 0 packets/sec 30 seconds output rate 3110376 bits/sec, 388797 bytes/sec, 5221 packets/sec CLI = Unstructured Data 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

What we need: Standard, structured way to represent configuration and operational data. <ipv4 xmlns="http://openconfig.net/yang/interfaces/ip"> <addresses> <address> <ip>172.26.194.212</ip> <config> <ip>172.26.194.212</ip> <prefix-length>24</prefix-length> </config> </address> </addresses> </ipv4> 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

XML vs JSON <interfaces xmlns:= [ ]yang:ietf-interfaces > <interface> <name>eth0</name> <type>ethernetcsmacd</type> <location>0</location> <enabled>true</enabled> <if-index>2</if-index> </interface> </interfaces> NETCONF/RESTCONF { } "ietf-interfaces:interfaces": { "interface": [ { "name": "eth0, "type": "ethernetcsmacd, "location": "0, "enabled": true, "if-index": 2 } ] } RESTCONF 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

Error! Sends <interface>gigabit 1/0</interface> <ifaddr>10.0.0.1/24</ifaddr> Expecting Expecting: <interface> <name>gigabit 1/0</name> <address>10.0.0.1/24</address> </interface> 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62

So why do we need YANG? <interface>ethernet 0/0</interface> <name>switch1 to UCS1</name> <ipaddr>1.1.1.1/24</ipaddr> <name>ethernet 0/0</name> <descr>switch1 to UCS1</descr> <ip>1.1.1.1/24</ip> Question: Which of these is correct? Answer: They all are! <ifname>ethernet 0/0</ifname> <ifalias>switch1 to UCS1</ifalias> <ifaddr>1.1.1.1/24</ifaddr> 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

YANG Data Models container ip { list vrf { leaf rd } } vrf red rd 1:1 <vrf>red</vrf> <rd>1:1</rd> YANG Model Data XML Data YANG models do not contain data or XML. YANG models are like templates used to generate consistent XML. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

YANG Data Models <vrf>red</vrf> <rd>1:1</rd> container ip { list vrf { leaf rd } } vrf red rd 1:1 XML YANG Model Data { vrf : red rd : 1:1 } JSON YANG models can be used as a template for generating structured data in many different formats. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

YANG Configuration Model Example* container ip { YANG list vrf { description "Configure an IP VPN Routing/Forwarding instance"; } } leaf name { type string; } leaf rd { description "Specify Route Distinguisher"; type rd-type; } <ip> <vrf> <name>vrf_red</name> <rd>65000:1</rd> </vrf> <vrf> <name>vrf_green</name> <rd>65000:2</rd> </vrf> </ip> ip vrf vrf_red rd 65001:1! ip vrf vrf_green rd 65001:2! XML CLI * Note: YANG model simplified for clarity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

So why is this: <ip> <vrf> <name>vrf_red</name> <rd>65000:1</rd> </vrf> <vrf> <name>vrf_green</name> <rd>65000:2</rd> </vrf> </ip>...better than this? ip vrf vrf_red rd 65001:1! ip vrf vrf_green rd 65001:2! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

YANG-structured data <ip> <vrf> <name>vrf_red</name> <rd>65000:1</rd> </vrf> <vrf> <name>vrf_green</name> <rd>65000:2</rd> </vrf> </ip> Designed for machine consumption Directly convertible to/from Python dicts! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

Where are YANG models? Models installed on device automatically with IOS-XE. On some devices/versions, can be updated independently https://github.com/yangmodels/yang/tree/master/vendor/cisco Also can be downloaded from GitHub. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

Who defines the YANG models? Vendors Standards Bodies Only work on specific vendor devices Greater feature coverage Can be OS-unique (IOS-XE, XR, etc.) Multi-vendor support More limited feature coverage Allow vendor-specific extensions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

Important Point! Cisco s data models and IETF/OpenConfig data models are just two ways of doing the same thing. <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces"> <interface> <name>gigabitethernet 1/0/24</name> <description>configured by NETCONF!</description> </interface> </interfaces> IETF-defined model Both of these do exactly the same thing! <native xmlns="http://cisco.com/ns/yang/cisco-ios-xe-native"> <interface> <GigabitEthernet> <name>1/0/24</name> <description>configured by NETCONF!</description> </GigabitEthernet> </interface> </native> Cisco-defined native model 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

Important Point! Cisco s data models and IETF/OpenConfig data models are just two ways of doing the same thing. <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces"> <interface> <name>gigabitethernet 1/0/24</name> <description>configured by NETCONF!</description> </interface> </interfaces> switch# show run interface g1/0/24 interface Both of these GigabitEthernet do exactly the 1/0/24 same thing! description Configured by NETCONF! <native xmlns="http://cisco.com/ns/yang/cisco-ios-xe-native"> <interface> <GigabitEthernet> <name>1/0/24</name> <description>configured by NETCONF!</description> </GigabitEthernet> </interface> </native> IETF-defined model Cisco-defined native model 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

Configuration vs. Operational data Configuration data tells the device what to do. It is data that you see in a show run. Operational data tells us how a device is operating, from show commands other than show run. # sh run int g0/0 interface GigabitEthernet0/0 description Management Interface vrf forwarding Mgmt-vrf ip address 172.26.244.49 255.255.255.0 # sh int g0/0 GigabitEthernet0/0 is up, line protocol up Hardware is RP management port Description: Management Interface We can write configuration data (think conf t ), and we can read configuration data (think show run ). Operational data is read-only. Some data can be read either as config data or operational data! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process 1 3 31 96 0.00% 0.00% 0.00% 0 Chunk Manager 2 3687 4786 770 0.07% 0.01% 0.00% 0 Load Meter Challenge: Write a Python script to go through the list of nearly 500 running processes and print the names of only those with runtime of 10 seconds or greater. Regex hard to understand Tied directly to table layout 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process 1 3 31 96 0.00% 0.00% 0.00% 0 Chunk Manager 2 3687 4786 770 0.07% 0.01% 0.00% 0 Load Meter Challenge: Write a Python script to go through the list of nearly 500 running processes and print the names of only those with runtime of 10 seconds or greater. XML easily rendered as Python dict Uses YANG data models Intuitive nomenclature 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

Enabling NETCONF: 3 Steps C3850-1#conf t Enter configuration commands, one per line. End with CNTL/Z. C3850-1(config)#aaa new-model C3850-1(config)#aaa authentication login default local C3850-1(config)#aaa authorization exec default local C3850-1(config)#username admin password cisco C3850-1(config)#line vty 0 15 C3850-1(config-line)#transport input all C3850-1(config)#netconf-yang C3850-1(config)# Enable AAA Enable SSH Enable NETCONF 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

Finding and using YANG models

What are we looking for? Extract route and next hop interface or IP from routing table NO regular expressions!!! Open model if possible 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

YangExplorer: A Cisco tool for exploring models Available at: https://github.com/ciscodevnet/yang -explorer Or just Google: "cisco yangexplorer" 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

Finding a model, method 2 Pull the config from an existing device. Select get-config for the hierarchy you need. Click Run RPC. The XML is displayed. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

Python NETCONF Libraries

NCClient: YDK: <interface> <GigabitEthernet> <name>1/0/14</name> <description>to_core_switch</description> <ip> <address> <primary> <address>15.10.1.1</address> <mask>255.255.255.0</mask> </primary> </address> </ip> </GigabitEthernet> </interface> gigabitethernet= interface.gigabitethernet() gigabitethernet.name = "1/0/14" gigabitethernet.description = "To_Core_Switch" gigabitethernet.ip.address.primary.address = "15.10.1.1" gigabitethernet.ip.address.primary.mask = "255.255.255.0" ip_add.gigabitethernet.append(gigabitethernet) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94

REST APIs

REST API URLs determine what resource is being accessed. POST https://api.ciscospark.com/v1/rooms POST https://api.ciscospark.com/v1/messages POST https://api.ciscospark.com/v1/team/memberships 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 102

The headers of the request contain basic information such as an authentication token, while the body of the request contains the data to send. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103

Pulling it all together

On-Box Scripting

Off-box Python Scripting NETCONF* Off-box scripts run on an external server and communicate with the switch over the network using NETCONF or other protocols. * or other protocol 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 116

On-box Python Scripting CLI NETCONF Syslog On-box Python scripts run in a container on the device itself. They can communicate with the network or the device itself. Guestshell Container 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 117

Advantages On-Box Access CLI directly on device Trigger syslog messages Interact with Embedded Event Manager Access device bootflash Zero Touch Provisioning Use interactive Python shell 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 118

Embedded Event Manager EEM takes certain actions based on triggering events. Events: cli temperature IPSLA Etc Trigger Actions: cli python reload syslog Etc 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 119

3 Spark posts diff to room 4 Python script diffs configs and sends diff to Spark Catalyst 3850 1 User changes device config EEM 2 Change detected by EEM 3 EEM Triggers on-box Python script 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 120

Wrapping Up

How do I learn Python? Automate the Boring Stuff with Python, Al Sweigart Great introduction to Python focused on automation. (Not specifically network automation.) Covers Python 3.0 only. Assumes zero knowledge. Read Excel docs, generate PDFs, etc. Highly recommended. Real Python. http://realpython.com Three-part course. Begins with basics assuming no knowledge. Covers Python 2.7 and 3.0. Parts II and III focus on web development with Python. Covers flask, Django, jinja2 templates. Many resources on the web site for free. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 122

Work in the USB lab Identify one problem you can solve with a script Start small Copy and mod scripts from DevNet (developer.cisco.com) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 125

Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 128

Thank you