VMware - NSX-vSphere Version: 1.0 11252014 Author: Website: Description: VMware Inc. www.vmware.com NSX-vSphere Content Pack provides powerful filtering, logging visualization and alerting of NSX for vsphere real-time operational activities. This enables users to effectively monitor NSX-vSphere production installations using intuitive Log Insight UI dashboard widgets with just a couple clicks. Highlights: Easy, at a glance, views of an entire NSX-vSphere environment and components. Users will know at a glance whether there is a problem with a NSX-vSphere environment by simply looking at the Overview dashboard. The Content Pack highlights many areas of a NSX-vSphere environment including: Common Installation / Configuration issues - exposes several targeted problem log patterns with actionable remediations - exposes system logs, audit logs and several targeted problem log patterns with actionable remediations Logical Router - exposes system logs, audit logs and several targeted problem log patterns with actionable remediations Distributed Firewall - contains several different dashboards to exposes firewall traffic by total aggregate traffic, s and ruleids Edge Services - separates logs for Edge appliance, health check, communication agent, high availability, load balancer, firewall and Ipsec VPN services Edge Firewall - contains several widgets for monitoring firewall traffic To Setup - forward all logging components to the Log Insight server including: 1. Use the Log Insight vsphere integration to forward logs for all ESXi servers 2. NSX-vSphere Manager 3. NSX-vSphere Controller(s) 4. All NSX-vSphere edge devices Where to begin: The NSX-vSphere - Overview is the main entry point. This dashboard displays all problems/alerts reflected in the entire Content Pack, therefore under normal operating circumstances the dashboard should not contain any data. Once a log appears on a widget, read the widget's information ("i"icon) in order to navigate to the appropriate dashboard for more detailed information associated with the problem. Additionally the Firewall dashboards provide widgets that contain many different views of firewall data. These visualizations are aggregated to show various slices of a NSX-vSphere environment at the, vm and application level. Compatibility: Content Pack is compatible with NSX-vSphere 6.1.x Collapse Dashboards NSX-vSphere - Overview
Name NSX-vSphere Edge System Events by NSX-vSphere Infrastructure Problems system events by Alerts by Logical Router system events by Logical Router Alerts by Bridging Alerts by Distributed Firewall system events by All NSX-vSphere edge system events. Edge Modules include: Gateway, Appliance, Vnic, Health Check, Ipsec Vpn, Loadbalancer, HighAvailability, Commnication Agent, CLI Settings, Log Settings, DHCP, NAT, Firewall, StaticRouting, SSL VPN, DNS and Syslog For more detailed information see the "NSX-vSphere Edge" dashboard. NOTE: System events are the runtime logs of the NSX-vSphere component Common NSX-vSphere infrastructure problems. For more detailed information see the "NSX-vSphere Infrastructure" dashboard. system events which contain non-information. For more detailed information on events and errors see the " - Overview" and other associated dashboards. NOTE: System events are the runtime logs of the NSX-vSphere component alerts widget is based on known error patterns that are generated from the various components that make up the infrastructure. For more detailed information on these errors see the " - Alerts" dashboard. Logical Router errors from the system event log. For more detailed information on errors see the Logical Router dashboards. For more detailed information see the "Logical Router - Overview" and other associated Logical Router dashboards. NOTE: System events are the runtime logs of the NSX-vSphere component Logical Router alerts widget is based on known error patterns that are generated from the various components that make up the Logical Router (DLR) infrastructure. For more detailed information on these errors see the "Logical Router - Alerts" dashboard. Bridging alerts widget is based on known error patterns that are generated from the various components that make up the bridging infrastructure. For more detailed information on any bridging alerts see the "Bridging - Alerts" dashboard. Distributed Firewall errors provides logs provided. For more detailed information see "Distributed Firewall - Overview" and other associated Distributed Firewall dashboards. NOTE: System events are the runtime logs of the NSX-vSphere
Name Distributed Firewall Alerts Firewall Actions component Distributed Firewall alerts widget is based on known error patterns that are generated from the various components that make up thedistributed Firewall infrastructure. For more detailed information on these errors see the "Distributed Firewall - Alerts" dashboard. Firewall Actions shows the distributed firewall traffic by action - drop and reject. This overview only shows drop and reject, for more detailed information see the "Distributed Firewall - Overview" and other associated Distributed Firewall dashboards NSX-vSphere - Infrastructure Name Controller vlan / subnet mismatch by Controller errors This log will be generated during NSX Controller deployment. If the newly instantiated NSX Controller fails to connect to NSX Manager within the 12 minute timeout window, then the NSX Manager will report this log and issue a delete of the NSX Controller VM. Check if the NSX Controller was deployed on the correct dvportgroup. Also if using DHCP for controller IP, check logs on your DHCP server to confirm whether IP was allocated for the controller having issue. This error log will be generated when a controller fails to respond. Likely reasons for this condition is the controller VM has either shutdown/failed or is unreachable over IP. Check network connectivity of this controller. Messaging Infrastructure Problem widget indicates communication errors between NSX Manager and ESXi Host. Messaging Infrastructure Problem by There are two reasons for this error: 1. Entire message bus is down in this scenario there are useful log messages on all hosts Verify that RMQ broker is not running. If it is running, then try to troubleshoot why the messages are not getting through. Look at /var/log/rabbitmq/* log files. If RMQ broker is not running, then simply restart it. Additionally, the file erl_crash.dump on the host file system will contain the whether the broker had crashed or if it was stopped intentionally. 2. The host is down or is not connected to broker Verify that host is down or is not connected to broker. If the host is down or not connected then troubleshoot the host and look at /var/log/rabbitmq/* logs to see what the connection issue is. Some common problems include the connection failures due to a firewall blocking traffic or credential mismatch between broker and host. To check the "rabbitmq" messaging bus status on the host execute the following commands on the ESXi host to see if the host has correct IP for NSX Manager:
Name NSX Manager - Host : Communication Errors esxcfg-advcfg-g /UserVars/RmqIpAddress This event will be generated when NSX Manager fails to receive heartbeat from UserWorld Agent on the host within the threshold period. The output is grouped by host-id. The host-id can be found from vcenter. Check the status of vsfwd process on the host that has lost connection to NSX Manager. NSX Manager - Controller : Communication Errors Host - Controller : Communication Errors # ps -ef grep vsfwd This widget provides a quick view of any communication related errors between NSX Manager and Controller. No data in this widget means there is a healthy communication channel. This widget will group events by controller IP address. This will help you identify the controller that is having issues. This widget filters for logs generated by vxlan module on the ESXi host when it loses control connection to controller. The output is grouped by host name that is reporting the error. This will identify the host that needs attention. On the host, run "/etc/init.d/netcpad status" to check the status of the UserWorld agent process. - Overview Name system events by controller errors data errors by audit events by resource system events are the runtime logs generated by the NSXvSphere components that make up the component. NOTE: System events are the runtime logs of the NSX-vSphere component Controller log errors which are associated with the components of NSX-vSphere data errors are from the ESXi hosts (or s). These errors come from the ESXi components which are responsible for the functionality Audits are events describing a change that was made to a in NSX-vSphere. The changes could be a create, update or delete event This widget shows the field view of the last six audit events. last audit events FieldTable Audits are events describing a change that was made to a in NSX-vSphere. create audit events The changes could be a create, update or delete event Shows all "create" audit events. NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but
Name update audit events delete audit events will give an idea that changes were made to the ing system Shows all "update" audit events. NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but will give an idea that changes were made to the ing system Shows all "delete" audit events. NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but will give an idea that changes were made to the ing system - Alerts Name Failed to create VTEP interface by Failed to create VTEP interface - likely due to VSM negotiating wrong VMODL version with VC This most likely occurred due to a versioning issue between VSM and vcenter - the vmodels did not negotiate correctly. VXLAN configuration problem by VXLAN TCP/IP stack not created by VXLAN dataplane lost connection to controller by VTEP does not have valid IP address by To remediate reconnect the affected Hypervisor to vcenter VXLAN configuration pushed to host before host was prepared. vcenter starts pushing data to the host before the kernel module was loaded. Hypervisor host must be rebooted to initialize configuration in correct order VXLAN TCP/IP stack not created - On stateless hosts this indicates incorrect host profile creation steps VXLAN data plane lost connection to controller. Investigate why the controller is losing the connection to the Hypervisor VTEP interface does not have a valid IP address -OR- VTEP lost its IP address. All vms on the may have lost their connectivity. Ensure that the Interface is not using DHCP. Logical Router - Overview Name Logical Router system errors by Logical Router errors from the system event log. For more detailed information on errors see the Logical Router dashboards. NOTE: System events are the runtime logs of the NSX-vSphere component
Name Logical Router controller errors Logical Router data plane errors by Logical Router audit events by resource Logical Router last audit events FieldTable Controller log errors which are associated with the Logical Router components of NSX-vSphere Logical Router data plane is the ESXi layer of the routing system. These logs are generated by the routing component on the ESXi host (). This widget only shows errors associated with the Logical Router data plane and is grouped by (ESXi host) Logical Router Audits are events describing a change that was made to a Logical Router in NSX-vSphere. The changes could be a create, update or delete event Shows the most recent Logical Router audit events. Each column is an extracted field from the audits. NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but will give an idea that changes were made to the Logical Routing system Shows all Logical Router "create" audit events. Logical Router create audit events Logical Router update audit events NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but will give an idea that changes were made to the Logical Routing system Shows all Logical Router "create" audit events. NOTE: Events may be duplicated as they are generated in multiple loggers in NSX-vSphere. Therefore these numbers may not be entirely accurate but will give an idea that changes were made to the Logical Routing system Logical Router - Alerts Name Failed to create control plane socket by Failed to create/delete a routing related object by Failed to create VXLAN trunk or teaming error by VDR kernel module uses socket to communicate with control plane. When the socket fails to create, VDR module which implements distributed router and bridge, can not work properly. Reload the module or host reboot might solve the issue. /var/log/vmkernel.log might have some information about the failure. Failed to create/delete Instance, LIF, Route, Connection and ARP, DHCP, Bridge. This could be the system failure or the VDR module runs out of memory or the max capacity has been reached if the creation fails. If not due to the capacity issue, host reboot might solve the issue. /var/log/vmkernel.log might have more information about the failure. Scenarios: 1. Failed to create VXLAN trunk VDR enables VxLAN trunking on the vdrport. It can not work properly if it fails.
Name 2. Cound not initialize teaming/update teaming, enable port or get LAG member names VDR module fails to configure the uplink teaming.it can not work properly if it fails. Failed to get LAGv2 info by Unable to configure VXLAN by OSPF MTU Mismatch OSPF Invalid Area ID OSPF Subnet mask mismatch When the issue happens, VDR can not work properly. Rebooting the host might solve the issue. /var/log/vmkernel.log might have the information about the failure. Failed to get LAGv2 info, uplink list. This error happens when VDR configures the uplink teaming. Further investigation on teaming module might be helpful. Rebooting the host might solve the issue. /var/log/vmkernel.log might have the information about the failure. Scenarios: 1. Unable to set VXLAN multicast IP, CP enabled, ref LIF, unref LIF 2. Failed to add hash entry, create permanent ARP VDR module calls functions of VxLAN module to configure VxLAN. Rebooting the host might solve the issue. /var/log/vmkernel.log might have the information about the failure. This message will be generated when there is MTU mismatch on OSPF neighbors that are establishing relationship. Check if the OSPF neighbor state is stuck in "EXSTART". This message will be generated by Edge Router or Logical Router VM when it is configured for OSPF and it receives Hello Packet with invalid Area ID. This indicates configuration error on one of the OSPF neighbors. This message is generated when there is mis-match in subnet mask of OSPF neighbors' interfaces. Please check the subnet mask configuration of the router vnics that are trying to establish neighbor relationship. Bridging - Alerts Name Bridge - Controller Communication errors Bridge communications query the controller to update or query the MAC address table. The MAC address table is used when control VMs are vmotioned to send out RARP packets to notify the physical switch about the new bridge host. When these warning messages occur the bridge still functions correctly until a control VM vmotion occurs. After vmotion is complete it will take up to 5 minutes to converge Typically occurs when a user doesn't create a bridge properly. The proper way to create a bridge is as follows: Bridge create error 1. Create a VDR instance 2. Create a bridge instance under the VDR instance 3. Create 2 LIFs 4. Add the 2 LIFs to the bridge instance 5. Internally there is automation to create a bridge port to keep packet statistics
Name Bridge data path error ERRORs of note: Failed create bridge instance - internal error (contact GSS) Failed to create bridge port - internal error (contact GSS) Cannot ref vxlan network on dvs - please make sure vxlan module is loaded and working (run net-vdl2 -l) When these internal errors occur the bridge is in error state. Try reloading the vdrb module or a system reboot may be required Typically these errors are caused if a user doesn't correctly delete a bridge. Bridge delete error Bridge DVS error Bridge system level error Bridge delete steps are: 1. Remove 2 LIFs from bridge instance 2. Internally remove bridge port 3. Remove bridge instance Please refer each message for details. All warning messages except the following indicate operator error and cause no harm to the system. Failed delete bridge instance - internal error (contact GSS) When these internal errors occur the bridge is in error state. Try reloading the vdrb module or a system reboot may be required When these internal errors occur the bridge is in error state. Try reloading the vdrb module or a system reboot may be required Distributed Firewall - Overview Name Firewall Actions Top Rule Hit Count Firewall audit events by operation Firewall Action shows the distributed firewall traffic by action - pass, drop, reject Top Firewall rules hit. To correlate a ruleid to the firewall rule make sure the ruleid is displayed in the vsphere Web Client by selecting it in the upper right drop-down of the firewall table Distributed Firewall Audits are events describing a change that was made to a Distributed Firewall in NSX-vSphere. The changes could be a create, update or delete event. All Firewall System events by Last Audit events To investigate on actual changes made, visit AuditLogs on NSX Manager Firewall system events are the runtime server logs which are generated by the firewall components. This widget displays all non-informational events FieldTable Last Firewall Audit events for the specified time range Distributed Firewall - Alerts Name Distributed Firewall Firewall critical events:
Name system event errors by 301501 - This is vsm side event if host failed to respond within time out window 301503 - This is vsm side event if vsm failed while provisioning firewall rule 301506 - This is vsm side event if vsm failed to send exclude list update 301031 - Failed to receive/parse/update firewall config. Key value will have context info like generation number and also other debugging info Spoofguard errors: Spoofguard errors by Filter Config errors by Service Profile Config errors by 301502 - This is vsm side event if host failed to respond within time out window 301505 - This is vsm side event if vsm failed to send container update 301061 - Failed to receive/parse/update spoofguard config. Key value will have context info like generation number and also other debugging info 301062 - Failed to apply spoofguard for vnic 301064 - Failed to disable spoofguard for vnic Filter Config errors by 301001 - Host failed to receive/parse filter config or open device /dev/dvfiltertbl. Key value pair would have possible cause of failure and context information 301002 - Failed to apply Filter config to vnic. Key value pair would have possible cause of failure (like failed while opening/parsing/updating filter config) and context information Firewall Service Profile errors: 301011 - Host failed to parse service profile config or open vsip device. Key value pair would have possible cause of failure (like failed while opening/parsing/updating filter config) and context information 301012 - Failed to update service profile config. Key value will have context info like profile name / version 301014 - Failed to delete service profile config. Key value will have context info like profile name / version Flow missed error: Flow Missed errors Dataplane incompatible with ESX version Firewall started, stopped, restarted by 301051 - Flow missed. Key value will have details about count and debugging info like flow dropped for filter or failed to read flow from filter or failed to send flow to vsm NSX-vSphere dataplane is incompatible with ESX version Firewall started, stopped or restarted grouped by VMware Distributed Firewall - Traffic Name Top Firewall Sources Top source IP addresses from all firewall rules that are logging data.
Name Top Firewall Destinations Application ports permitted Application ports denied Top Firewall Sources by bytes - client to server Top Firewall Destinations by bytes - client to server Top Firewall Sources by bytes - server to client Top Firewall Destinations by bytes - server to client Top destination IP addresses from all firewall rules that are logging data. Measures all in / out connections permitted in the NSX-vSphere environment by destination port. The data is the summation of the specified time range. All traffic which is denied due to a firewall rule. The data is grouped by application (or destination) port. NOTE: This widget only shows data associated with ports. Certain types of traffic like ICMP does not have an associated port so it will not show up in this widget. All firewall traffic, in bytes by IP address, from the client to a server. Data is only displayed if the firewall is logging its data. The data is the summation of the specified time range. All firewall traffic, in bytes by IP address, from the client to a server. Data is only displayed if the firewall is logging its data. The data is the summation of the specified time range. All firewall traffic, in bytes by IP address, from the server to a client. Data is only displayed if the firewall is logging its data. The data is the summation of the specified time range. All firewall traffic, in bytes by IP address, from the server to a client. Data is only displayed if the firewall is logging its data. The data is the summation of the specified time range. Distributed Firewall - Hypervisor Data Name Connections Established by over time Connections Established by Number of IN connections from the perspective of the VMware from all firewall rules that are logging data. Represents both Ingress and Egress traffic. To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out" All Firewall connections, in and out, at the VMware layer. Represents both Ingress and Egress traffic. To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out"
Name Bytes Source to Destination Hypervisor by Bytes Destination to Source Hypervisor by Total bytes into a VMware, from client to server, where a vm resides that is listening on a port. Represents both Ingress and Egress traffic. To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out" Total bytes out from a VMware, from server to client, where a vm resides that is listening on a port. Represents both Ingress and Egress traffic. To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out" Distributed Firewall - Rule Data Name Connections by ruleid over time Connections by ruleid Number of IN connections to a VMware by ruleid. To correlate a ruleid to the firewall rule, make sure the ruleid is displayed in the vsphere Web Client by selecting it in the upper right drop-down of the firewall table. Represents both Ingress and Egress traffic. To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out" Total connections IN and OUT at the VMware layer by ruleid. To correlate a ruleid to the firewall rule, make sure the ruleid is displayed in the vsphere Web Client by selecting it in the upper right drop-down of the firewall table. Represents both Ingress and Egress traffic. Bytes Source to Destination Hypervisor by ruleid To narrow this widget to Ingress traffic only, set the vmw_nsx_firewall_traffic_direction filter to "in" For Egress, set the vmw_nsx_firewall_traffic_direction filter to "out" Bytes IN to a VMware that contains a vm listening on a port by ruleid To correlate a ruleid to the firewall rule, make sure the ruleid is displayed in the vsphere Web Client by selecting it in the upper right drop-down of the firewall table Bytes Destination to Bytes OUT from a VMware that is initiating a connection by
Name Source Hypervisor by ruleid ruleid To correlate a ruleid to the firewall rule, make sure the ruleid is displayed in the vsphere Web Client by selecting it in the upper right drop-down of the firewall table NSX-vSphere Edge - Overview Name NSX-vSphere Edge Appliance by NSX-vSphere Edge Health Check by NSX-vSphere Edge Communication Agent by NSX-vSphere Edge High Availability by NSX-vSphere Edge Load Balancer by NSX-vSphere Edge Firewall by NSX-vSphere Edge Ipsec VPN by Shows all logs specific to NSX-vSphere Edge Appliance grouped by log Shows all logs specific to NSX-vSphere Edge Health Check grouped by log Shows all logs specific to NSX-vSphere Edge Communication Agent grouped by log Shows all logs specific to NSX-vSphere Edge High Availability grouped by log Shows all logs specific to NSX-vSphere Edge Load Balancer grouped by log Shows all logs specific to NSX-vSphere Edge Firewall grouped by log Shows all logs specific to NSX-vSphere Edge Ipsec VPN grouped by log NSX-vSphere Edge - Firewall Name Edge Firewall Traffic by action Edge Firewall Traffic by destination Edge Firewall Traffic by source Edge Firewall Traffic by application port Edge Firewall Traffic by ruleid Edge Firewall Traffic by action shows the edge firewall traffic by action - accept, deny Edge Firewall Traffic by destination shows the edge traffic grouped by destination IP address Edge Firewall Traffic by source shows the edge traffic grouped by source IP address Edge Firewall Traffic by application port shows the edge traffic grouped by destination IP port Edge Firewall Traffic by ruleid shows the edge traffic grouped by firewall ruleid