Troubleshooting Cisco Unity Connection

Troubleshooting Cisco Unity Connection Bryan Shapess 2

Agenda Troubleshooting Methodology Troubleshooting Tools CUC Traces Case Studies Message Waiting Indicator Visual Voicemail Single Inbox Message Delivery Note: Versions used for this presentation are Unity Connection 9.1. 3

Troubleshooting Methodology

Where Does Unity Connection Fit? Unity Connection Integrates with: CUCM via SIP or SCCP Third party PBX via PIMG/TIMG Can optionally use External LDAP Directory Can optionally store copy of messages in External Message Store Message Store Unity Directory Telephony (CUCM, SIP, PIMG) 5

Start by Gathering Information Detailed problem description Exact time (last week, yesterday, this morning around 9:00) What was the user doing? (What options did the user select or what digits were entered?) What did the user hear? What did the user expect to happen? Determine frequency Is it reproducible? How often does it occur? Is there a Pattern? Isolate separate issues and prioritize 6

Examine Available Information Use tools such as the application syslog to search for errors Gather and examine traces 7

Prepare for Next Occurrence Configure macro and/or micro traces Notify user(s) of what information to capture Time of issue What they were trying to do Digits entered Recording(s) heard Be prepared to collect traces 8

Troubleshooting Tools

Troubleshooting Tools Remote Port Status Monitor Cisco Utilities Data Link for Informix (CUDLI) Real Time Monitoring Tool (RTMT) Packet Capture EWS Editor 10

Remote Port Status Monitor Available on CiscoUnityTools.com Alternates color with each new call 11

Cisco Utilities Data Link for Informix (CUDLI) Found on www.ciscounitytools.com Read-only access to UnityDirDB, UnityRptDB, UnityDynDB and UnityMBX1DB Provides additional information on tables and columns within those tables 12

CUDLI: Column Info 13

CUDLI: Mouse Over for More Information 14

CUDLI: Options 15

CUDLI: Search 16

CUDLI: Search Search Results 17

CUDLI: Has a Built-In Query Builder 18

RTMT Downloaded from Cisco Unity Connection Administration System Settings Plug-ins Provides performance monitoring data (CPU, memory, disk usage, etc.) Provides alerting capabilities Syslog Viewer provides logging information for the server similar to Windows Event Viewer Trace and Log Central is used to collect trace files 19

Syslog Viewer Sample 20

Syslog Viewer Sample 21

RTMT: Collect Files Found under Trace & Log Central -> Collect Files Refer to macro or micro trace table in troubleshooting guide for traces to retrieve Select services as needed. 22

RTMT: Collect Files Cont d 23

RTMT: Collect Files Select Date/Time Range Specify the Download File Directory Select if you want to Zip Files 24

RTMT: Scheduled Trace Collection Same look as manual Trace Collection Allows for Automatic Trace Collection 25

RTMT: Scheduled Trace Collection 26

RTMT: Scheduled Trace Collection 27

RTMT: Scheduled Trace Collection 28

RTMT: Scheduled Trace Collection 29

Packet Capture Why would you use a packet capture? To see packet headers and payload Help diagnose issues when traces don t seem to help Provide product neutral data Make sure packets are making it to intended destination 30

Packet Capture Built-in to Unity Connection admin cli Syntax: utils network capture [page] [numeric] [file fname] [count num] [size bytes] [src addr] [dest addr] [port num] [host protocol addr] Example syntax to capture SMTP traffic to a file called CUC.cap: utils network capture file CUC count 100000 size all port 25 When finished press <ctrl> + c to stop capture 31

Packet Capture Can download packet capture from RTMT or from CLI Must analyze packet capture in 3rd party tool 32

Packet Capture To view available capture files from CLI admin:file list activelog platform/cli cuc.cap dir count = 0, file count = 1 To download capture file from CLI admin:file get activelog platform/cli/cuc.cap Please wait while the system is gathering files info...done. Sub-directories were not traversed. Number of files affected: 1 Total size in Bytes: 1668 Total size in Kbytes: 1.6289062 Would you like to proceed [y/n]? 33

Packet Capture Wireshark 34

EWS Editor A Microsoft tool which simulates the EWS connection to Exchange Helps diagnose Unity Connection single inbox connection issues to Exchange https://ewseditor.codeplex.com/ Not TAC supported 35

EWS Editor Service URL: http(s)://<exchange CAS Server>/EWS/Exchange.asmx Requested Exchange Version: Exchange2007_SP1 Credentials: Same credentials entered in CUC for Unified Messaging Service Impersonation: Email Address of user s mailbox 36

EWS Editor Will receive this error if you are: Using self signed certificate on Exchange Don t have root certificate trusted on workstation 37

EWS Editor 38

EWS Editor 39

CUC Traces

Macro Traces: Unity Connection A collection of preselected micro traces Used to troubleshoot general areas of functionality Select area where you need traces 41

Micro Traces: Unity Connection Have to select one micro trace grouping at a time Used for more in depth troubleshooting Select Component 42

Micro Traces: Unity Connection Check Individual Tracing Levels 43

What Traces Should I Enable? Diagnostic Traces section in Troubleshooting Guide - http://www.cisco.com/en/us/docs/voice_ip_comm/connection/9x/troubleshooting /guide/9xcuctsg010.html 44

What Traces Should I Enable? When in doubt enable the following Macro Traces: Call Flow Diagnostics Call Control (Miu) Traces Conversation Traces Collect Traces from RTMT: Connection Conversation Manager 45

Unity Connection Traces UC traces end with extension.uc Plain text Format: 16:13:57.240 29183,CUCM-SIP-1-002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008 46

UC Trace Deciphered Format is: Time Thread, Port, Call GUID, Component, Level, Function or Process Trace shows: 16:13:57.240 29183,CUCM-SIP-1-002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008 Event occurred at 4:13 PM 47

UC Trace Deciphered Format is: Date Time Thread, Port, Call GUID, Component, Level, Function or Process Trace shows: 16:13:57.240 29183,CUCM-SIP-1-002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008 Low Level (1) Conversation Engine (CDE) trace 51

Case Studies 53

Case Study 1 MWI Failure Problem Description: A user reports that the red light on the phone is off but when they called Unity Connection it said they had a new message 54

Case Study 1 MWI Failure What trace to enable? Macro Traces Call Control (Miu) Traces Traces for MWI problems What traces to retrieve? Connection Conversation Manager Connection Notifier 55

Case Study 1 MWI Failure Open diag_cunotifier and search for Failed. 14:43:58.086 24630,,,Notifier,12,Bryan Shapess: MWI (ON) failed. Details=Failed MWI [user='bryan Shapess' extension='1000' type=mwi_on mwitype=0=port phonesystem=cucm-sip (bd22ed0b-83ce-4871-a67f-ac3c530ad233)] Error=No MWI ports on switch 56

Case Study 1 MWI Failure View with list of phone systems Phone Systems with port count 57

Case Study 1 MWI Failure View with subscribers and their phone system 58

Case Study 1 MWI Failure 59

Case Study 1 MWI Failure Change phone system to proper phone system. 60

Case Study 2 Visual Voicemail Failure Problem Description: When trying to play a message using Visual Voicemail you receive an error No Line Available To Play Message. Issue happened around 8:55pm 61

Case Study 2 Visual Voicemail Failure 62

Case Study 2 Visual Voicemail Failure What trace to enable? Macro Traces Call Control (Miu) Traces Micro Traces VMWS (All) Phone Service debug logging CallManager Detailed traces What traces to retrieve? Connection Conversation Manager Connection Tomcat Application Cisco Tomcat (Second Page) Phone Console Logs CallManager Detailed SDI Logs 63

Case Study 2 Visual Voicemail Failure Set Phone Service VisualVoicemail log_level to debug Collect console logs from phone web page 64

Case Study 2 Visual Voicemail Failure Find calls to Reverse Trap Number Locate call which matches time user experienced problem 65

Case Study 2 Visual Voicemail Failure 66

Case Study 2 Visual Voicemail Failure Failure caused by defect CSCuj81059 Symptom: On RT phones, having phone firmware 9.2.2 onwards, when line having a busy line/shared line the VVM does not play and gives the following error: "No line available to play message Resolved: 8.6.2SU4 9.1.2ES27 67

Case Study 3 Single Inbox Failure Problem Description: A user reports that when they get a new message it doesn t show up in their Outlook Inbox 68

Case Study 3 Single Inbox Failure What traces to enable? Macro: Single Inbox Trace Micro: EWS (All) What traces to retrieve? Connection Mailbox Sync 69

Case Study 3 Single Inbox Failure Open trace file CuMbxSync 70

Case Study 3 Single Inbox Failure 71

Case Study 3 Single Inbox Failure 72

Case Study 3 Single Inbox Failure 73

Case Study 3 Single Inbox Failure Check Exchange for users whose mailbox is over quota 74

Case Study 3 Single Inbox Failure To better understand single inbox messages, review single inbox annotated logs http://www.ciscounitytools.com/training/connection/cuc8_5.html 75

Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 76

Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings 77