Network Management Automated Intelligence Open Source Based NMS solution bdnog7 18-22 November 2017 Dhaka, Bangladesh
p Network Management n Parameters n Components n Open Source Tools n Demonstration n Q & A
Parameters p p p p Operation: keeping the network (and the services that the network provides) up and running smoothly. It includes monitoring the network to spot problems as soon as possible, ideally before users are affected. Administration: deals with keeping track of resources in the network and how they are assigned. Maintenance: concerned with performing repairs and upgrades. Maintenance also involves corrective and preventive measures to make the managed network run "ʺbetter. Provisioning: is concerned with configuring resources in the network to support a given service.
So Network Management is the use of a system that constantly monitors a computer network for slow or failing systems and that notifies the network administrator in case of outages via email, SMS or other alarms. subset of the functions involved in network management.
Network Management p System & Service monitoring n Reachability, availability p Resource measurement/monitoring n Capacity planning, availability p Performance monitoring (RTT, throughput) p Stats & Accounting/Metering p Fault Management n Fault detection, troubleshooting, and tracking p Configuration/Change Management p Coordination
Expectations A network in operation needs to be monitored in order to: - Deliver projected SLAs (Service Level Agreements) - SLAs depend on policy p p p p What does your management expect? What do your users expect? What do your customers expect? What does the rest of the Internet expect? What s good enough? 99.999% Uptime? è There'ʹs no such thing as 100% uptime (as we ll see) à
Uptime Expectations What does it take to deliver 99.9 % uptime? 30.5 days x 24 hours = 732 hours a month (732 (732 x.999)) x 60 = 44 minutes only 44 minutes of downtime a month! Need to shutdown 1 hour / week? (732 4) / 732x 100 = 99.4 % Remember to take planned maintenance into account in your calculations, and inform your users/customers if they are included/excluded in the SLA How is availability measured? In the core? End-to-end? From the Internet?
Baselining What is normal for your network? If you ve never measured or monitored your network you will need to know things like: n Typical load on links (è Cacti) n Level of jitter between endpoints (è Smokeping) n Typical percent usage of resources (è LibreNMS) n Typical amounts of noise : p Network scans p Dropped data p Reported errors or failures
Why do all this? Know when to upgrade Is your bandwidth usage too high? Where is your traffic going? Do you need to get a faster line, or more providers? Is the equipment too old? Keep an audit trace of changes Record all changes Makes it easier to find cause of problems due to upgrades and configuration changes Maintain history of network operations n n Using a ticket system lets you keep a history of events. Allows you to defend yourself and verify what happened
Components p Availability p Reliability p Performance p Configuration Mgmt & Monitoring p Network Forensic p Intrusion Detection p. p.. p Coordination
Tools n Diagnostic tools used to test connectivity, ascertain that a location is reachable, or a device is up usually active tools n Monitoring tools tools running in the background ( daemons or services), which collect events, but can also initiate their own probes (using diagnostic tools), and recording the output, in a scheduled fashion.
Tools p Active tools n Ping test connectivity to a host n Traceroute show path to a host n MTR combination of ping + traceroute n SNMP collectors (polling) p Passive tools n log monitoring, SNMP trap receivers p Automated tools n SmokePing record and graph latency to a set of hosts, using ICMP (Ping) or other protocols n MRTG record and graph bandwidth usage on a switch port or network link, at regular intervals n So MANY More...
Tools Reliability p SmokePing n Keeps track of your network latency: n Best of breed latency visualisation. n Interactive graph explorer. n Wide range of latency measurment plugins. n Master/Slave System for distributed measurement. n Highly configurable alerting system. n Live Latency Charts with the most 'ʹinteresting'ʹ graphs. n Free and OpenSource Software written in Perl
Tools SmokePing
Tools SmokePing
Tools Performance p Cacti/MRTG n n n n n n n A tool to monitor, store and present network and system/server statistics Designed around RRDTool with a special emphasis on the graphical interface Almost all of Cacti'ʹs functionality can be configured via the Web. Uses RRDtool, PHP and stores data in MySQL Supports the use of SNMP and graphics with MRTG Authentication Scheme Large Network Deployment
Tools Cacti
Tools Weathermap
Tools Availability p Nagios n server and service availability monitoring p Can monitor pretty much anything p HTTP, SMTP, DNS, Disk space, CPU usage,... p BGP, OSPF, Switch Port, room temperature,.. p Easy to write new plugins (extensions) p Zabbix, ZenOSS, Hyperic,... Many more Open Source... Ø Log, Log, Log... Ø Notification mechanism
Tools. Nagios
Tools. Nagios
Tools. Nagios XI
Tools. Nagios
Tools Availability+ Reliability LibreNMS l l l l SNMP-based auto-discover network monitoring Derived from another project (Observium) Written in PHP as a web application Includes support for a wide range of hardware: Cisco, Linux, FreeBSD, Juniper, Brocade, Foundry, HP and many more See http://docs.librenms.org/support/features/ Over 100 supported! Routers, Switches, Access Points, Security gateways, Hosts, Printers,
LibreNMS Available metrics l l l l CPU, memory and storage statistics Interface traffic, packet and detailed error statistics (L2 and L3) Temperature, fan speed, voltage, amperage, power humidity and frequency sensors Users, processes, load average and uptime statistics
LibreNMS Available metrics cont. l l l l l l Linux distribution detection Real-time interface traffic graphing Device inventory collection (useful!) Detailed IPv4, IPv6, TCP and UDP stack statistics BGP and OSPF information MAC <-> IP address lookup l Find which port an IP/MAC was last seen on
LibreNMS
LibreNMS
Tools Configuration Mgmt & Monitoring The Really Awesome New Cisco config Differ p Rancid n Rancid is a configuration management tool that keeps track of changes in the configurations of any size network equipment (Cisco, HP, Juniper, Foundry, etc.). Works on routers and switches. Automates retrieval of the configurations and archives them as backup tool, audit tool, blame allocation.
Tools RANCID The Really Awesome New Cisco config Differ The data is stored in a VCS (Version Control System) which keeps n Track changes in the equipment configuration n Track changes in the hardware (S/N, modules) n Track version changes in the OS (IOS, CatOS versions) n Find out what your colleagues have done without telling you! n Recover from accidental configuration errors.
Tools RANCID
RANCID
Tools Net Forensic p Network Flow Analysis Tool n NetFlow (C), n cflowd (F), n FlowScan (F), n Sniffer Pro (C), n argus (F), n i-flow (C) n NFSen (F) n AS-STATS
Tools NFSen p Network Flow Analysis Tool n NFSen p Display netflow data: Flows, Packets and Bytes using RRD (Round Robin Database). p Easily navigate through the netflow data. p Process the netflow data within the specified time span. p Create history as well as continuous profiles. p Set alerts, based on various conditions.
Tools NFSen
Tools NFSen
Tools NFSen TCP UDP Flows Packets Bytes Flows Packets Bytes Rank Port Count Port Count Port Count Port Count Port Count Port Count 1 80 39029 80 570630 80 111021671 53 116671 53 150335 12610 142186426 2 445 27833 25 83140 40936 88004359 6881 2388 12610 99433 28712 101344390 3 135 24572 40936 66203 25 52612168 39792 2276 28712 70901 40493 93146942 4 25 7881 445 53175 55893 43525223 15507 1904 40493 65155 46886 27824516 5 23 6761 135 49066 46395 39079355 43040 1611 15699 46682 57563 26436088 6 3128 4786 55893 37615 2889 30261886 60928 1588 1416 40540 62390 25767022 7 443 2999 46395 35068 1317 24692504 51012 1573 57563 37794 54505 25550351 8 22 2517 22 27489 49674 23472247 61295 1447 34018 37747 55893 23548341 9 9415 1275 443 26468 54311 23342821 5060 1309 21694 24942 40633 22940400 10 8080 1081 21651 25614 44879 23306526 49665 1225 46886 19468 40403 19544859
AS-STATS p A netflow/sflow collector p n Storing data in RRD files A cron program n order the ASNs by level of traffic exchanged p A web interface to : n See your traffic per ASN n See your traffic per LINK
How as-stats looks? (Peers)
How as-stats looks? (Link)
What it is good for? p BGP Traffic Engineering p Finding out who you should (try to) peer with p Knowing what's going on in your network p Planning for future expansion
Tools IDS & IPS Computer Security is not something that you can just add on when you need it. Proper planning, installation, monitoring and maintenance all become part of a successful IDS/IPS implementation. p Tri-Sentry (Host Sentry, NetSentry, Service Sentry) p Nessus, Snort p Checkpoint, Cisco IPS, UTM (Cyberoam, Barracuda) $$$ ARE YOU AWARE OF YOUR NETWORK POLLUTION.
Malicious Traffic Detection Tool
Tools Log Management Elasticsearch flexible and powerful open source, distributed real- time search and analytics engine for your log (http://www.elasticsearch.org) n n n n n n n n n n n Easy to scale (Distributed) Everything is one JSON call away (RESTful API) Unleashed power of Lucene under the hood Excellent Query DSL Multi-tenancy Support for advanced search features (Full Text) Configurable and Extensible Document Oriented Schema free Conflict management Active community
Tools Log Management commercial SIEM Logstash Kibana
Elastic Elasticity
Tools Log Management Elasticsearch, Logstash & Kibana (ELK)
Tools Collaboration So, we have many Open Source/Commercial deployments already to monitor our network. All the programs can generate alert/alarm on fault detection. Need to centralize all the information. We need to collaborate these programs Need NOC Its not a big Room/House it s a software Its RT (the ticketing system)
Tools RT Request Tracker p RT is a battle-tested issue tracking system which thousands of organizations use for n bug tracking, n help desk ticketing, n customer service, n workflow processes, n change management, n network operations, n And so on..
Tools RT Request Tracker Whenever, wherever and however there is a problem in the network the relevant monitoring software will send a ticket directly to RT system and system admins will know immediately via email or SMS. This automation will keep track of the SLA. RT has its own Help Desk system and escalation procedure.
Tools... RT n Why are they important? p Track all events, failures and issues n Focal point for help desk communication n Use it to track all communications p Both internal and external n Events originating from the outside: p customer complaints n Events originating from the inside: p System outages (direct or indirect) p Planned maintenance, upgrades, etc.
Tools... RT l l l l l Why are they important? Track all events, failures and issues Focal point for help desk communication Use it to track all communications Both internal and external Events originating from the outside: customer complaints Events originating from the inside: System outages (direct or indirect) Planned maintenance, upgrades, etc.
Tools RT
Tools The Big Cycle
Conclusion RT Cacti SmokePing Nagios NFSen Weather Map Rancid ALL IN ONE NETWORK MANAGEMENT SYSTEM
???