U-M Network Background Information Network Automation Strategy Network Automation Execution

Transcription

1

2 U- Network Background Information Network Automation Strategy Network Automation Execution

3 Organic Growth of U- Network Tools: A broad range of tools Commercial, open source, dead-end forks of open source, in-house developed Each tool fulfills a particular niche Tool fragmentation resulted in technical debt: Legacy and in-house tools support lapsed Open source tool releases lag Overlaps in tool versions and functionality Source of authority fragmented across tools 1990 T T TR 2000 TR T C C C C C C T 2007 C I T T S 2010 C 2014 S 2016 I TR TR

4 200+ Buildings 4 Data Centers Thousands devices 4 core, 3 border, 227 distribution layer routers 3,083 access-layer switches ~ 14,000 wifi APs ajor device vendors: Cisco and Juniper Dozens of device models, dozens of OS versions: 125+ combinations Traditional Cisco three-tier architecture Elements of enterprise and service-provider networks: spanned VLANs + VRF

5 TROUBLESHOOTING perfsonar Testing throughput & latency Cisco Prime Client connectivity metrics aggregation and reporting CONFIGURATION ANAGEENT Rancid - U- 24 hour pull and analysis (diffs) of all router/switch configs Cisco Prime Template most of AP configuration changes OmniPeek Distributed packet capture TOPOLOGY Interapper Easily accessible web view, large data volumes, manual data entry Spectrum Autodiscovery, topology mapping; not as accessible web view TopologyWalker Network device discovery, machine readable data source for other tools NetFlow Analysis with flowtools LOGGING Splunk Syslog from all network devices INVENTORY DATABASE CONFIGURATION TOOLS REEDY Device type, role and location for all network hardware - EOL date REEDY Building info, time tracking, SA data, feeds CDB... NetDisco Gathers real-time inventory information, best for firmware audit NetInfo Primary source for all logical network data, tracks address space... Cisco Prime Single source for all AP and controller data NetDisco atches AC/IP to switchports, used by admins and IA for device location DCT Homegrown, used by campus admins for port changes support limited CoolTools Homegrown, config and query, web front, broad range, limited support Cisco Prime Template most of AP configuration changes Rancid U- Highly customized, captures hardware information ServiceNow CDB, WiFi engineering DB Significant overlaps in functionality, tool interdependencies, and staff turnover have made our current tool chain difficult to maintain. ONITORING / ALERTING Interapper alerts, used by admins for checks of campus devices Spectrum Customizable, integrates with other monitoring systems / tools Cacti Tracks bandwidth, pps ucast/non-ucast, errors Netflow Used for billing purposes Cisco Prime Central monitoring of APs and controllers Splunk Simple triggered alerts based on syslog HOSTASTER TOOLS BLUECAT DNS/DHCP Administration of campus DHCP and DNS service DNS stats collector - resides on nettools server Host lookup tool same tool/different function tool functional dependencies

6 Transition to FEWER, ORE INTELLIGENT TOOLS to: Proactively alert, even react to problems in the network Harden the network Reduce staff time needed to detect and respond to incidents Improve the quality and consistency of network management and configuration Deploy AUTOATION FRAEWORK built out to become, for most basic tasks, as trustworthy as the switch itself

7 Networks are currently managed with the existing collection of tools and processes.

8 Our approach is to build a second software stack in parallel, implement the minimum synchronization between the two stacks, build new networks with the new stack, and then migrate existing networks to it. Existing system is left in place and the new system is built in parallel: ETL process loads data from the old system into the new New system is validated and tested DB Salt / Ansible [New systems]

9 Our approach is to build a second software stack in parallel, implement the minimum synchronization between the two stacks, build new networks with the new stack, and then migrate existing networks to it. Existing system is left in place and the new system is built in parallel: ETL process loads data from the old system into the new New system is validated and tested Subsets of networks are gradually moved to the new system DB Salt / Ansible [New systems]

10 DB Salt / Ansible [New systems] [New systems] [New systems] Our approach is to build a second software stack in parallel, implement the minimum synchronization between the two stacks, build new networks with the new stack, and then migrate existing networks to it. Existing system is left in place and the new system is built in parallel: ETL process loads data from the old system into the new New system is validated and tested Subsets of networks are gradually moved to the new system Existing system runs in parallel while confidence is built in the new system Continue to add functionality and networks to the new system.

11 DB Salt / Ansible [New systems] [New systems] [New systems] New system All functionality replicated from the old Sufficient trust built in the new service Old system shut down

12 Establish central source of configuration authority for all devices ake network changes in the SoA rather than the device(s). Software reads configuration data and implements changes programmatically. Network industry seems to be settling on Python, so we naturally gravitated towards Salt and/or Ansible. Utilize open-source tools conventionally Refactor existing processes rather than implement U--specific versions. If changes are necessary, contribute them upstream so U- does not run special software.

13

14

15 Goal: provision a small leaf-spine data center fabric entirely using automation Step 1: Datacenter inventory, IPA, and topology defined in Netbox: Network device roles, physical connections between devices. IP addressing/interface assignments... Step 2: Templates written to map data from Netbox into device-specific configuration: Use some data directly, e.g. interface IP addresses, VLAN IDs... Other configuration implicit: e.g. for a device with the spine role: for each interface connected to a device with the leaf role configure IS-IS on that interface (underlay), configure a route-reflector client for the IP address on the remote end of that connected interface (overlay). Step 3: Salt renders complete device configuration, pushed to device via NAPAL load-replace Load-replace ensures that the automation process generates and applies all device configurations.

16 Use several existing sources of operations data, including: NetInfo (home-grown tool) Custom tables in a Remedy database Developed a suite of python scripts to extract data from sources (NetInfo and Remedy), and load them into NetBox. Automatically transferring thousands of records and relationships. Work to do: validate accuracy, and source data quality.

17 Source of authority With this flow, the older systems are the source of authority. But NetBox will be where automation tools (Salt / Ansible) will pull data from. Automation

18 Source of authority We have many existing scripts that pull data from sources like NetInfo and Remedy into other tools like BlueCat. BlueCat Source of authority Automation

19 Option: Rewrite import script Create a new import script that uses the NetBox APIs to pull data and import into systems like BlueCat BlueCat Source of authority Automation

20 Option: Reverse the flow Implement an export script to move data from NetBox to the existing databases. BlueCat Source of authority Automation

21 We will strongly prefer to rewrite the import script, but will consider reversing the flow. This will depend partly on the complexity of the systems involved and how long we expect them to continue to exist. BlueCat 1 Automation 2 Source of authority

22 Created a Salt pillar (data source middleware) to pull data from the NetBox API. Our Pull Request was approved and merged in arch. And has since had additional changes contributed by the community. This pillar will be included with the next release of Salt (fluorine).

23 Launching a project to replace an existing tool: DCT. DCT was built to allow departmental IT staff to make (some) configuration changes to switch ports. Going to replace this with an open-source tool with a modular architecture to support different backends and devices. Discussing use cases and architecture with the University of Wisconsin. Looking for collaborators!

24 Collaborating across our Infrastructure group to stand up GitLab server. We are using this extensively to manage: code for software projects some device configurations server deployment configurations (Ansible) Initial target has been central IT infrastructure group, but we are expanding to provide this as a service to campus, including research, and teaching and learning uses.

25 Code review process as part of merge requests. Review erge Request aintainer Approve? no Contributor Create Branch CI/CD Code linting / Syntax validation Automated Testing Implement erge Request yes erge branch

26 Conducting internal workshops to help colleagues build skills and experience with the technologies and approaches we need to adopt. Git Ansible / Salt NetBox Python Software development practices

27