Solution Troubleshooting for Unified Contact Center Enterprise

Transcription

1

2 Solution Troubleshooting for Unified Contact Center Enterprise Holger Esser Technical Solutions Manager CBABU Customer Solutions Success

3 Cisco Spark Ask Question, Get Answers Use Cisco Spark to communicate with the speaker during the event! What if I have a question after visiting Cisco Live?... Cisco Spark How? Spark rooms will be available until July 29, Go to the Cisco Live Mobile app 2. Find this session 3. Click the join link in the session description 4. Navigate to the room, room name = Session 5. Enter messages in the room 3

4 Agenda UCCE Solution Overview Call Flow and Log Review Database Driven Troubleshooting Q / A Appendix

5 UCCE Solutions Overview

6 UCCE Architecture Scripting Presentation Layer Reporting Desktop & CRM Admin Video Agent Shared Container (OpenSocial Gadgets, XMPP, REST API Access) Social Media + Chat Thick Clients Multichannel Recording & Streaming Routing APIs Real-Time Reporting Self Service & Queuing Outbound Desktop Control Call Flows Historical Reporting Contact Center Services VXML Browser Call Control Media Forking TDM / IP Gateways Message Bus Video Network/Data Services 6

7 UCCE Solution Component View Config. Mgt CVP OAMP CCMP App Server Database T1/PRI MediaSense Active Directory AXL TDM GW XCODE CONF T1s SIP SIP SCCP (DSPs) SIP CUBE-E SIP Proxy CUSP UCM SIP SIP CVP Call Server VXML Server Media Server HTTP (VXML) Agent VXML GW VXML Browser SIP (dialer) JTAPI GED 125 HTTP HTTP RE Mobile MediaBroker RE Mobile App Server Generic PG VRU PIMs MR PG SIP Dialer CUCM PIM CTI Server Finesse Pri/Sec GED 188 Media Firewall Web Call-Back App server Firewall HTTP Internet SocialMiner Agent Request API HTTP Supervisor WIM Web Server EIM/WIM Services Server DB server Central Controller Router Logger Campgn Mgr Browsers livedata CSDK Web & Mobile Apps HTTP HTTPS/WSS DTLS / srtp CVP Reporting Server CUIC JDBC Pub/Sub JDBC Admin & Data Server AW/HDS/DDS RP Enterprise Reverse Proxy REMB HTTP Remote Expert Mobile Media Broker H 7

8 Call Flow

9 Objectives CVP LOGS CALLGU = 998FDA A0A0A, DLG = 25 [SIP_LEG] - Processing,, [ICM_TEMPORARY_CONNECT], topic=cisco.cvp.ged125.msg, dialogueid=25, sendseqno=1, label= , correlationid=1020, callguid=998fda a0a0a, rckey=213, rcday=150805, rcseq=0, CallContext:, user.microapp.toextvxml[0]: application=cvpmenu,, LEG = 623E E3-9A38C1FE-3DA10ECE, DNIS = 40100, ANI = 2002 ROUTER LOGS ( : 0 0) Dialog sending VRUQueueService to VRU (serv=-1, rt=0) ( : 0 0) RequestInstr: C=(150805,213), CallState=1 9

10 Message Flow SSE VRU PG Router Router PG VXML GW VRU Logger\HDS IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CUBE or Ingress GW CUCM Agent 10

11 Route Request New Call VRU PG Router Router PG VXML GW VRU Logger\HDS IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse SETUP CUBE or Ingress GW CUCM Agent 11

12 SETUP Route Request Call at VRU VRU PG Router Router PG VXML GW VRU CONNECT IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CONNECT CUBE or Ingress GW CUCM Agent 12

13 SETUP Run Script Req Route Request Request Instruction Run Script Req Call at VRU VRU PG Router Router PG VXML GW VRU CONNECT Logger\HDS Request Instruction IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CONNECT CUBE or Ingress GW CUCM Agent 13

14 SETUP PRE CALL IND Run Script Req Route Request Request Instruction Run Script Req Agent Reserved VRU PG Router Router PG VXML GW VRU CONNECT Logger\HDS Request Instruction IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CONNECT CUBE or Ingress GW CUCM Agent 14

15 INVITE ACK SETUP Run Script Req Route Request Request Instruction Run Script Req Call at Agent VRU PG Router PG VXML GW VRU CONNECT Router Request Instruction IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CONNECT CUBE or Ingress GW CUCM 180 Ringing Agent 15

16 Number of Messages VXML GW 4 VRU PG PG VRU 3 4 Router ~5 Router 3 Logger\HDS IVR ICM SIP CVP 3 ~8 OPC PIM CTISVR JGW AGENT PG ~5 A B Finesse CUBE or Ingress GW CUCM ~50 Agent 16

17 Call s Ingress Gateway ISDN Call Reference CCAPI Cisco-GU Router SIP Dialog CORR VRU Dialog Router Call Key VXML Gateway SIP Leg CVP GU CCAPI CUBE or Ingress GW VXML GW Router VVB CVP GU Inbound VXML Leg Ring Back Leg Agent leg SIP Dialog VRU Dialog IVR SIP CVP Agent PG JGW myc,call PIM, OPC Call CTISvr, Call VXML Server Application Reference ICM OPC PIM CTISVR JGW AGENT PG VRU PG SIP Dialog VRU Dialog Router call Key CUCM Call CI Global Call Call Handler VRU PG VRU PG CUCM 17

18 Call s VRU PG Router Router PG VXML GW VRU Logger\HDS IVR ICM SIP CVP OPC PIM CTISVR JGW AGENT PG A B Finesse CUBE or Ingress GW CUCM Agent 18

19 Call s VRU PG IVR DLG Router PG VXML GW VRU SIP DLG Router Logger\HDS CVP GU IVR DLG IVR ICM SIP CVP SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call A B Finesse ISDN CUBE or Ingress GW CUCM Agent 19

20 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 20

21 What do the messages look like? INVITE SIP/2.0 Via: SIP/2.0/TCP :5060;branch=z9hG4bK6f4f5da88f From: To: Date: Fri, 22 Nov :26:53 GMT Call-: User-Agent: Cisco-CUCM10.0 CSeq: 101 INVITE Cisco-Guid: Session-Expires: 1800 P-Asserted-Identity: Remote-Party-: CISCO-GU CUBE or Ingress GW //-1/998FDA000000/CCAPI/cc_api_call_setup_ind_common: Interface=0xC041A51C, Call Info( Calling Called Incoming Dial-peer=40101,, Call Id= : Nov 22 10:12:10.307: //6681/998FDA000000/CCAPI/ccIFCallSetupRequestPrivate: Called Number=40100 ## Outgoing CCAPI Outgoing Dial-peer=40100, : Nov 22 10:12:10.307: //6682/998FDA000000/CCAPI/ccIFCallSetupRequestPrivate: SPI Call Setup Request Is Success; Interface Type=3, FlowMode=1 CCAPI INBOUND Call CCAPI OUTBOUND Call 21

22 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 22

23 What do the messages look like? Found an incoming CISCO-GU hdr : , Generated a new CVP- hdr : 998FDA A0A0A CALLGU = 998FDA A0A0A LEG = 623E E3-9A38C1FE-3DA10ECE - [INBOUND]: ReqURI (DN) sip:40100@ :5060 FromURI sip:2002@ CALLGU = 998FDA A0A0A, DLG = 25 [SIP_LEG] - Publishing,, [ICM_NEW_CALL], topic=cisco.cvp.ged125.msg, dialogueid=25, sendseqno=1, trunkgroupid=100, trunknumber=0, serviceid=1, dialednumber=40100, uui=, callguid=998fda a0a0a, rckey=, rcday=, rcseq=, location=, locationpkid=, pstntrunkgroupid= , pstntrunkgroupchannelnum= , sipheader=, CallContext:, user.media.id: 998FDA A0A0A, user.cvp_server_info: ,, LEG = 623E E3-9A38C1FE-3DA10ECE, DNIS = 40100, ANI = 2002 CALLGU = 998FDA A0A0A, DLG = 25 [SIP_LEG] - Processing,, [ICM_TEMPORARY_CONNECT], topic=cisco.cvp.ged125.msg, dialogueid=25, sendseqno=1, label= , correlationid=1020, callguid=998fda a0a0a, rckey=213, rcday=150805, rcseq=0, CallContext:, user.microapp.toextvxml[0]: application=cvpmenu,, LEG = 623E E3-9A38C1FE-3DA10ECE, DNIS = 40100, ANI = 2002 CALLGU = 998FDA A0A0A LEG = 998FDA A0A0A [OUTBOUND]: INVITE TO <sip: @ :5060;transport=tcp> FROM 2002 <sip:2002@ :5060> EXPIRES[60] 100REL[Unsupported] VBServlet:service: HTTP Request from : { RECOVERY_VXML=flash:recovery.vxml, CALL_DNIS= , CALL_UUI=, MSG_TYPE=CALL_NEW, ERROR_CODE=NONE(0), CALL_=998FDA A0A0A, CLIENT_TYPE=IOS, CALL_ANI=sip:2002@ :5060 } CISCO-GU CVP GU IVR SIP CVP SIP DLG CVP GU (VRU) ICM CALLGU = 998FDA A0A0A, DLG = 26 [IVR_LEG] - Publishing,, [ICM_REQUEST_INSTRUCTION], topic=cisco.cvp.ged125.msg, dialogueid=26, sendseqno=1, trunkgroupid=200, trunknumber=0, serviceid=2, uui=, correlationid=1020, location=, locationpkid=, pstntrunkgroupid=, pstntrunkgroupchannelnum= , sipheader=,, LEG =, DNIS = , ANI = sip:2002@ :5060 IVR DLG 23

24 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 24

25 What do the messages look like? SIP DLG VRU PG VRU PG Router Call Key VRU->PG:NEW_CALL(154 bytes):d=25 SendSeq#=1 TrkGrp=100 TrkNum=0 Srv=1 DN=40100 ANI=2002 DNIS=40100 CallGU=998FDA A0A0A PstnTrkGrp= PstnTrkGrpChann#= ECCVar:0=998FDA A0A0A ECCVar:29= PG->VRU:TEMPORARY_CONNECT(128 bytes):d=25 SendSeq#=1 Label= Corr=1020 RCK=213 RCKDay= RCKSeq#=0 CallGU=998FDA A0A0A VRU->PG:REQUEST_INSTRUCTION(105 bytes):d=26 SendSeq#=1 TrkGrp=200 TrkNum=0 Srv=2 DNIS= Corr=1020 CallGU=998FDA A0A0A PG->VRU:RUN_SCRIPT_REQ(137 bytes):d=26 SendSeq#=1 Inv=1 Script=GS,Server,V ScriptConfig= ANI=2002 CED= RCK=213 RCKDay= RCKSeq#=1 CallGU=998FDA A0A0A IVR DLG PG->VRU:CONNECT(126 bytes):d=25 SendSeqNo=2 LabelType=1 Label=2001 TransferHint=0 RCK=213 RCKDay= RCKSeq#=0 CallGU=998FDA A0A0A 25

26 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 26

27 What do the messages look like? Router Router SIP DLG Router Call Key IVR DLG (25 x 0 : 0 0) NewCall: C=(150805,213), DN=40100, ANI=2002, CED=, RC=5001, MRD=1, CallAtVRU=1, OpCode=0 RCKSeqNum=0, NIC_DN=40100 CallType(5009, 26): Init CT_SL_Timer: Threshold(20 + 2). (25 x 0 : 0 0) Correlation id for dialog is (1020). (25 x 1020 : 0 0) TransferToVRU: Label= , Cor=1020, VRU=5000, RC=5001 ( : 0 0) Dialog sending VRUQueueService to VRU (serv=-1, rt=0) ( : 0 0) RequestInstr: C=(150805,213), CallState=1 ( : 0 0) RouteComplete: Route: DN=40100, ANI=2002, CED=, Label=2001 Route: C=(150805,213), Labels=1 ( : 0 0) DeviceTargetPreCall_V12: C=(150805,213), Per=5000, SGST=5059, AGST=5054 NetworkCallFlags=0, Invoke=26, GREET= DN=40100, ANI=2002, CED=, CustDBProvidedDigits=, Route=5014, NetworkTarget=0 Correlation? 27

28 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 28

29 What do the messages look like? Found an incoming CISCO-GU hdr : , Generated a new CVP- hdr : 998FDA A0A0A CVP GU CALLGU = 998FDA A0A0A LEG = 623E E3-9A38C1FE-3DA10ECE - [INBOUND]: ReqURI (DN) sip:40100@ :5060 FromURI sip:2002@ IVR ICM SIP CVP CALLGU = 998FDA A0A0A, DLG = 25 [SIP_LEG] - Publishing,, [ICM_NEW_CALL], topic=cisco.cvp.ged125.msg, dialogueid=25, sendseqno=1, trunkgroupid=100, trunknumber=0, serviceid=1, dialednumber=40100, uui=, callguid=998fda a0a0a, rckey=, rcday=, rcseq=, location=, locationpkid=, pstntrunkgroupid= , pstntrunkgroupchannelnum= , sipheader=, CallContext:, user.media.id: 998FDA A0A0A, user.cvp_server_info: ,, LEG = 623E E3-9A38C1FE-3DA10ECE, DNIS = 40100, ANI = 2002 CALLGU = 998FDA A0A0A, DLG = 25 [SIP_LEG] - Processing,, [ICM_TEMPORARY_CONNECT], topic=cisco.cvp.ged125.msg, dialogueid=25, sendseqno=1, label= , correlationid=1020, callguid=998fda a0a0a, rckey=213, rcday=150805, rcseq=0, CallContext:, user.microapp.toextvxml[0]: application=cvpmenu,, LEG = 623E E3-9A38C1FE-3DA10ECE, DNIS = 40100, ANI = 2002 CALLGU = 998FDA A0A0A LEG = 998FDA A0A0A [OUTBOUND]: INVITE TO <sip: @ :5060;transport=tcp> FROM 2002 <sip:2002@ :5060> EXPIRES[60] 100REL[Unsupported] VBServlet:service: HTTP Request from : { RECOVERY_VXML=flash:recovery.vxml, CALL_DNIS= , CALL_UUI=, MSG_TYPE=CALL_NEW, ERROR_CODE=NONE(0), CALL_=998FDA A0A0A, CLIENT_TYPE=IOS, CALL_ANI=sip:2002@ :5060 } VRU LEG CALLGU = 998FDA A0A0A, DLG = 26 [IVR_LEG] - Publishing,, [ICM_REQUEST_INSTRUCTION], topic=cisco.cvp.ged125.msg, dialogueid=26, sendseqno=1, trunkgroupid=200, trunknumber=0, serviceid=2, uui=, correlationid=1020, location=, locationpkid=, pstntrunkgroupid=, pstntrunkgroupchannelnum= , sipheader=,, LEG =, DNIS = , ANI = sip:2002@ :

30 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 30

31 What do the messages look like? VXML GW 1020? CVP GU INVITE SIP/2.0 Via: SIP/2.0/TCP :5060;branch=z9hG4bKSLIsQ4wuxiPtjygFXoQFVg~~2599 To: From: 2002 Call-: Expires: 60 User-Agent: CVP 10.0 (1) Build-481 Call-Info: <sip: :5060>;purpose=x-cisco-origip Remote-Party-: Cisco-Guid: Cisco-Gucid: 998FDA A0A0A App-Info: < :8000:8443> VRU LEG GET HTTP/1.1 GET /cvp/vbservlet?msg_type=call_result&call_=998fda a0a0a&call_seq_num=1&error_code=0&callced=0 GET /cvp/vbservlet?msg_type=call_result&call_=998fda a0a0a&call_seq_num=2&error_code=1 31

32 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 32

33 What do the messages look like? Router Call Key Router Router (25 x 0 : 0 0) NewCall: C=(150805,213), DN=40100, ANI=2002, CED=, RC=5001, MRD=1, CallAtVRU=1, OpCode=0 RCKSeqNum=0, NIC_DN=40100 CallType(5009, 26): Init CT_SL_Timer: Threshold(20 + 2). (25 x 0 : 0 0) Correlation id for dialog is (1020). (25 x 1020 : 0 0) TransferToVRU: Label= , Cor=1020, VRU=5000, RC=5001 ( : 0 0) Dialog sending VRUQueueService to VRU (serv=-1, rt=0) ( : 0 0) RequestInstr: C=(150805,213), CallState=1 ( : 0 0) RouteComplete: Route: DN=40100, ANI=2002, CED=, Label=2001 Route: C=(150805,213), Labels=1 ( : 0 0) DeviceTargetPreCall_V12: C=(150805,213), Per=5000, SGST=5059, AGST=5054 NetworkCallFlags=0, Invoke=26, GREET= DN=40100, ANI=2002, CED=, CustDBProvidedDigits=, Route=5014, NetworkTarget=0 33

34 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 34

35 What do the messages look like? Router Call Key OPC PIM CTISVR JGW AGENT PG Agent PG- PIM DeviceTargetPreCallInd: PreLock: RTRCallKey= NT: 0 dialed number=40100 CED= AST=5054, AG=, CustomerDefinition=1, AW=0 05:27:56:555 pg1a-pim1 Trace: CSTA DELIVERED, TelephonyDriver::Offered: SENT TO OPC Call = Device = 2002 Alerting = Calling = 2002 Called = 2001 Call Agent PG - JGW Call Handler new MyC: 10 CallActiveEv C: ConnCreatedEv C: (394014/1) Addr: 2001 CurCalling: 2002 handleconncreated::new MyDestMyConnection() MyConn: Gen: 19 Val: Dev: 2001/0 Ext: 2001 DevTgStr: Addr: 2001 State: INPROGRESS CCState: OFFERED prevccstate: UNKNOWN loggedin: Y ConnInProgressEv C: Addr:

36 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 36

37 What do the messages look like? CUCM //SIP/SIPUdp/wait_UdpDataInd: Incoming SIP TCP message size 1277 from :[32796]: [ ,NET] INVITE SIP/2.0 Via: SIP/2.0/TCP :5060;branch=z9hG4bKSLIsQ4wuxiPtjygFXoQFVg~~2628 To: From: 2002 Call-: CSeq: 1 INVITE Contact: <sip:2002@ :5060;transport=tcp> User-Agent: CVP 10.0 (1) Build-481 Call-Info: <sip: :5060>;purpose=x-cisco-origip Remote-Party-: <sip:2002@ >;party=calling;screen=yes;privacy=off Cisco-Guid: Session-Expires: 1800 INBOUND CI Cisco-Gucid: 998FDA A0A0A App-Info: < :8000:8443> CVP GU Call Handler Agent Leg Call //SIP/SIPD(10,73,14)/ccbId= /scbId=0/restart0_SIPSetupInd: mtsp.devicename[cvp_sip_trunk] 10,100,230, ^ ^* //SIP/SIPD(10,73,14)/ccbId=0/scbId=0/getKeyBasedOnCiAndBranch: AddressingElement branch is 0 and ci is mapkey is ,100,230, ^ ^* OUTBOUND CI LineControl(745) - Get call instance=1 for CI= ,100,230, ^ ^* 37

38 Call s VRU PG IVR DLG Router Router PG VRU VXML GW SIP DLG Router Call Key CVP GU IVR DLG IVR ICM SIP CVP CVP GU SIP DLG Global Call Call Handler OPC PIM CTISVR JGW AGENT PG Call Call A B Finesse ISDN CUBE or Ingress GW CCAPI INBOUND Call CCAPI OUTBOUND Call INBOUND CI CUCM OUTBOUND CI Agent 38

39 Finesse client log 39

40 Finesse client log 40

41 Finesse firebug log 41

42 What do the messages look like? CallControl : _processcall(): Process the dialog with id: , to extension: 40100, from extension: 2002, call state: ALERTING, calltype: PREROUTE_ACD_IN Header : [ClientServices] MasterPublisher._parseAndPublishXMLEvent() - Received XML event on node '/finesse/api/user/2001': <Update> <data> <user> <dialogs>/finesse/api/user/2001/dialogs</dialogs> <extension>2001</extension> <firstname>john</firstname> <lastname>doe</lastname> <loginid>2001</loginid> Call <loginname>jdoe</loginname> <pendingstate></pendingstate> <roles> <role>agent</role> </roles> <settings> <wrapuponincoming>optional</wrapuponincoming> </settings> <state>talking</state> <statechangetime> t08:13:00.758z</statechangetime> <teamid>5000</teamid> <teamname>bxb</teamname> <uri>/finesse/api/user/2001</uri> </user> </data> <event>put</event> <requestid></requestid> <source>/finesse/api/user/2001</source> </Update> Agent 42

43 Call s Summary Ingress GW : Cisco-Guid: CVP : CISCO-GU = CVP- = 998FDA A0A0A = CALLGU LEG=998FDA A0A0A VRU Leg LEG = 998FDA A0A0A Agent Leg DLG = 25 = SIP Dialog, DLG = 26 VRU Dialog Router : D = 25 SIP Dialog, D = 26 VRU Dialog C=(150805,213) Router Call Key Day,Router Call Key VXML GW : Call-: 998FDA A0A0A VRU Leg Agent pim : RTRCallKey= Router Call Key Day.Router Call Key Call = Call JGW : C: Call Val: Call Handler CUCM : Call-: 998FDA A0A0A Agent Leg CI= Call Handler (Outbound CI) Finesse : dialog with id: Call 43

44 Log Free Troubleshooting (sort of)

45 UCCE AW-HDS Database Tables, Tables, Tables Configuration Tables (Agents, Skills, Peripherals, etc.) Interval Tables (Agent Availability, Not Ready Time, etc.) Termination Call Detail Table (TCD) Each call arriving at the peripheral generates a TCD record at the end of the call Route Call Detail Table (RCD) Each route request generates a an RCD entry as soon as the request is made AW-HDS Do not run complex reporting type queries against the TCD\RCD database tables. If you need to have custom RCD\TCD reporting, create a separate database for such purpose. 45

46 UCCE AW-HDS Database Termination Call Detail Table Important Fields DateTime RouterCallKey / RouterCallKeyDay DialedDigits ANI Peripheral CallGU Various Timings of Call Duration, Arrival, etc AW-HDS 46

47 UCCE AW-HDS Database Route Call Detail Table Important Fields DateTime RouterCallKey / RouterCallKeyDay DialedNumber Peripheral Variables ANI Script (ICM script) FinalObject Various Call Timings AW-HDS 47

48 UCCE Call Journey in the RCD Table Select * FROM [prod_hds].[dbo].[t_route_call_detail].. Dialed Num Router CallKey Day Router CallKey Route Request Type Routing Client Originator Type Msg PriorityOrigin Variable5UserToUserANI Script FinalObject CallType NULL 1CCMusic NULL

49 UCCE Call Journey in the RCD Table Select * FROM [prod_hds].[dbo].[t_route_call_detail].. Dialed Num Router CallKey Day Router CallKey Route Request Type Routing Client Originator Type Msg PriorityOrigin Variable5UserToUserANI Script FinalObject CallType NULL 1CCMusic NULL Script Editor AWDB AWDB AWDB AWDB Dialed_Number Route Routing_Client MasterScript Script 49

50 UCCE Call Journey in the RCD Table Did you see it all the way in the back??? 50

51 UCCE Call Journey in the RCD Table RtrCallKey ANI RoutingClnt DNString BeginCallType BeginScript Script_Author Script_Ver Mod_Date CallTypeName CVP_SCRIPT_ CUCM_RC CustomerCare MICRO HKE\Administ rator 28 07/06/2016 CustomerCare Label TargetLabel AgentName* RtrErrNum RouterErrorDesc RequestType Script EndScriptName FinalNode TargetTypeDesc rf1002 rf1002 NULL 0 Normal Post-Route 5030 Script2 13 Dynamic Label 51

52 UCCE Call Journey in the RCD Table Create the display layout select t_media_routing_domain.enterprisename as MRDomainName, Route_Call_Detail.RouterCallKeySequenceNumber as Seq#, Route_Call_Detail.BeganRoutingDateTime, Route_Call_Detail.RouterCallKey as RtrCallKey, Route_Call_Detail.ANI, 52

53 UCCE Call Journey in the RCD Table Extract the configuration data and join it to the RCD table attributes from Route_Call_Detail left outer join t_script on t_script.script = Route_Call_Detail.Script left outer join t_master_script on t_master_script.masterscript = t_script.masterscript left outer join Routing_Client on Routing_Client.RoutingClient = Route_Call_Detail.RoutingClient left outer join t_call_type on Route_Call_Detail.CallType = t_call_type.calltype left outer join t_agent on t_agent.peripheralnumber = Route_Call_Detail.Label left outer join t_route on t_route.route = Route_Call_Detail.Route 53

54 UCCE Call Journey in the RCD Table RtrCallKey ANI RoutingClnt DNString CUCM_RC Joining the config tables with the RCD contents will yield a logical and humanly readable description of the numerical data. 54

55 UCCE Call Journey in the RCD Table BeginScript CVP_SCRIPT_ MICRO Script_Author Script_Ver HKE\Administ rator 28 FinalNode TargetTypeDesc 13Dynamic Label 55

56 UCCE Call Journey in the RCD Table RtrErrNum RouterErrorDesc 0 Normal Ideally the Router Return Code is 0. All others signify a failure to produce a label and the routing of the contact fails. 56

57 UCCE Call Journey in the TCD Table Select * FROM [prod_hds].[dbo].[t_termination_call_detail].. AgentSkillTarget SkillGroupSkill Target RouterCallKey Day TalkTime ANI RouterCallKey Sequence Number CallGU 12335C NULL NULL E 12335C NULL NULL sip:1004@ : E NULL 57

58 UCCE Call Journey in the TCD Table Seq RouterCallKeyDay ANI DigitsDialed CallTypeName DNIS AgentPeriph AgentSkill TargetName SkillGroup SkillName RouteName PeriphName CallDis Call Disposition Name CallDisp FlagName CustomerCare NULL NULL NULL NULL NULL : CustomerCare NULL NULL NULL NULL NULL CVP_Generic Called Party _2 52 Disconnected Handled CVP_Generic Drop Handled _2 13Primary Route Handled CUCM Generic_ CustomerCare Esser_Holger SupportTeam SupportRoute CUCM Drop Handled Generic_1 13Primary Route Handled VRU PG PG VRU CUCM 58

59 UCCE Call Journey in the TCD Table CallDis Call Disposition Name CallDisp FlagName Called Party 52 Disconnected Handled Drop Handled 13Primary Route Handled Drop Handled 13Primary Route Handled Value Explanation 1 Abandoned in network 2 Abandoned in local queue 3 Abandoned ring 4 Abandoned delay 5 Abandoned interflow 6 Abandoned agent terminal 7 Short 8 Busy 9 Forced busy 10 Disconnect/drop no answer 11 Disconnect/drop busy 12 Disconnect/drop reorder 13 Disconnect/drop handled primary route 14 Disconnect/drop handled other 15 Redirected 16 Cut through Application invalid dialogue 47 Application duplicate dialogue 48 Application invalid invoke 49 Application invalid script selector 50 Application terminate dialogue 51 Task ended during application init 52 Called party disconnected 59

60 UCCE Call Journey in the TCD Table Is a non zero disposition code in the TCD table an indication of a problem? 60

61 UCCE Call Journey in the TCD Table 2,3,6,7,10,13,14,15,19,24,28,29,30,33,52 Illuminati secret code? IQ test? These disposition codes are generally good codes, depending on your use cases. 61

62 Troubleshooting Failure Scenarios

63 Scenario 1 Symptoms Caller calls in and hears I am sorry. and the call drops. Available Information: ANI DNIS Time of Day No system alarms CVP PG RTR 63

64 Scenario 1 Possible Points of Failure VXML GW ICM CVP PG RTR IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 64

65 Scenario 1 Resolution Approach Run TCD\RCD query Result: No Data VXML GW ICM CVP PG RTR IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 65

66 Scenario 1 Resolution Approach Search for ANI in Call Server Logs Result: No Data VXML GW CVP IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging 66

67 Scenario 1 Resolution Approach Enable Gateway debugs VXML GW IOS debug ccsip messages debug ccapi inout debug voip dialpeer all : Jul 5 11:43:56.351: //- 1/xxxxxxxxxxxx/DPM/dpMatchPeersCore: No Outgoing Dial-peer Is Matched; Result=NO_MATCH(-1) : Jul 5 11:43:56.351: //- 1/xxxxxxxxxxxx/DPM/dpMatchSafModulePlugin: dialstring=10000, saf_enabled=0, saf_dndb_lookup=1, dp_result=-1 67

68 Scenario 2 Symptoms Caller calls in and hears I am sorry. and the call drops. Available Information: ANI DNIS Time of Day No system alarms CVP PG RTR 68

69 Scenario 2 Possible Points of Failure VXML GW ICM CVP PG RTR IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 69

70 Scenario 2 Resolution Approach Run TCD\RCD query Result: RCD returns a router error of 65, a script name of CVP_Micro and FinalObject = 14 VXML GW ICM CVP PG RTR IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 70

71 Scenario 2 Resolution Approach Review the RTR log PG ICM PG\Router PIM logs RTR logs RTR 09:34:30:908 ra-rtr Trace: (1 x 0 : 0 0) NewCall: C=(151761, ), DN=10000, ANI=1003, CED=, RC=5001, MRD=1, CallAtVRU=1, OpCode=0 09:34:30:908 ra-rtr Trace: (1 x 1 : 0 0) TransferToVRU: Label= , Cor=1, VRU=5000, RC= :34:31:429 ra-rtr Trace: (1 2 1 : 0 0) RequestInstr: C=(151761, ), CallState=1 09:34:31:429 ra-rtr Trace: (1 2 1 : 0 0) Queuing Call C=(151761, ) for SkillGroup SupportTeam, Pos=1 09:34:31:429 ra-rtr Script Default\\CVP_SCRIPT_MICRO executed for more that 1000 nodes. 09:34:31:430 ra-rtr Trace: (1 2 1 : 0 0) Canceling queuing for C=(151761, ) 09:34:31:430 ra-rtr Trace: (1 2 1 : 0 0) Unqueuing Call C=(151761, ) for SkillGroup SupportTeam, pos=1 09:34:31:430 ra-rtr Script Default\\CVP_SCRIPT_MICRO ( 5004) failed to produce route for dialed number CVP_RC with call type of CustomerCare ( 5000). 09:34:31:431 ra-rtr Trace: (1 2 1 : 0 0) Script Default\\CVP_SCRIPT_MICRO ( 5004) failed to produce route for dialed number CVP_RC with call type of CustomerCare ( 5000). 09:34:31:431 ra-rtr No default label available for dialed number CVP_RC ( 5003). 09:34:31:431 ra-rtr Trace: (1 2 1 : 0 0) No default label available for dialed number CVP_RC ( 5003). 09:34:31:431 ra-rtr Trace: (1 2 1 : 0 0) No Default Route: dn=10000, ani=1 71

72 Scenario 2 Resolution Approach Look at the script RCD returns a router error of 65, a script name of CVP_Micro and FinalObject = 14 ICM PG RTR PG\Router PIM logs RTR logs 72

73 Scenario 3 Symptoms Caller navigates through script and selects a transfer to an agent. Then the call disconnects. Available Information: ANI DNIS Time of Day No system alarms CVP PG RTR 73

74 Scenario 3 Possible Points of Failure VXML GW ICM CVP PG RTR IOS debug ccsip messages debug ccapi inout Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 74

75 Scenario 3 Resolution Approach Run TCD\RCD query Result: TCD returns an agent label = 1002, the peripheral name = CVP and a call disposition of 1 VXML GW ICM CVP PG RTR IOS debug ccsip messages debug voip ccapi inout debug voip dialpeer all Call Server logs Enable debug 41 for SIP messaging PG\Router PIM logs RTR logs 75

76 Scenario 3 Resolution Approach Run TCD\RCD query Result: No Data CVP Call Server logs Enable debug 41 for SIP messaging 481: : Jul :22: : %CVP_9_0_SIP-7-CALL: {Thrd=pool-1-thread-70-SIP-676} CALLGU = 0C210E D E LEG = E3E E6-807FBBD1-7828EA69 - [INBOUND] Handling CONNECT :>>HEADERS: (JMSType)=MsgBus:CONNECT (JMSDestination)=Topic(CVP.ICM.CC.RESP.SYS_SIP1) (JMSTimestamp)= (Server)=CVP9x.SYS_ICM1:ICM:ICM2:CVP9x.MsgBus001 >>BODY: callguid=0c210e d e RouterCallKey= floatingvars=>>headers: (JMSType)=MsgBus:FLOATINGVARS >>BODY: callvars=[,,,, CCMusic,,,,, ] user.microapp.media_server= user.media.id=0c210e d e user.microapp.fromextvxml=[null, null, null, null] user.courtesycallbackenabled=1 user.microapp.toextvxml=[application=helloworld, null, null, null, null] user.microapp.usevxmlparams=n >>STATE: istabular=false iswriteable=true cursor=-1 dnis=10000 timezone=america/los_angeles RouterCallKeySequenceNumber=0 version=cvp_9_0 RouterCallKeyDay= labeltype=1 label=1002 localoffset=- 420 calllegid=e3e e6-807fbbd1-7828ea69 >>STATE: istabular=false iswriteable=false cursor=-1 482: : Jul :22: : %CVP_9_0_SIP-3-SIP_ERROR_SNMP: B2BUA is not configured with a route for making calls to [1002]. Please add this route. [id:5010] 76

77 Appendix CVP Log Parsing Sample Queries Log Collection and Tools Proactive Measures Troubleshooting Tools

78 CVP Log Parsing

79 Tips Search for new call with guid Find the CALLGU Search all lines which includes CALLGU = x (replace x with CALLGU you found above) Compare the is with should be (understand the call flow) 79

80 New call with guid Open CVP Logs Find New Call with guid Find All in All Opened Documents All Lines having new call with guid will be listed 80

81 Find CallGU Highlight the CALLGU Copy CALLGU = X Clicking on this will take you to the line 81

82 Find Lines with CALLGU = x Find CallGU = x Find All in all Opened Documents All Lines having CallGU = x will be listed 82

83 Compare IS with Should Be 83

84 Sample Queries

85 Sample Queries TCD select --Termination_Call_Detail.MRDomain, t_media_routing_domain.enterprisename as MRDomainName, Termination_Call_Detail.RouterCallKeySequenceNumber as Seq, Termination_Call_Detail.DateTime, --Termination_Call_Detail.RouterCallKeyDay, Termination_Call_Detail.RouterCallKey as RtrCallKey, Termination_Call_Detail.ANI, Termination_Call_Detail.DigitsDialed, --Termination_Call_Detail.CallType, t_call_type.enterprisename as CallTypeName, Termination_Call_Detail.DNIS, Termination_Call_Detail.AgentPeripheralNumber as AgentPeriphNbr, --Termination_Call_Detail.AgentSkillTarget, t_device_target.enterprisename as DeviceTargetName, t_agent.enterprisename as AgentSkillTargetName, --Termination_Call_Detail.SkillGroupSkillTarget, t_skill_group.enterprisename as SkillGroupSkillName, --Termination_Call_Detail.Route, t_route.enterprisename as RouteName, --Termination_Call_Detail.ServiceSkillTarget, --t_service.enterprisename as ServiceSkillName, --Termination_Call_Detail.Peripheral, t_peripheral.enterprisename as PeriphName, Termination_Call_Detail.PeripheralCallKey as PeriphCallKey, case when PeripheralCallKey > then PeripheralCallKey - (PeripheralCallKey/ )* else 0 end as GC, -- var float adjust PeripheralCallKey/ as CCM, Termination_Call_Detail.UserToUser, --Termination_Call_Detail.PeripheralCallType, case PeripheralCallType when 1 then 'Post Rte ACD In' when 2 then 'Pre Rte ACD In' when 3 then 'Pre-Route Direct Agent' when 4 then 'Transfer In/Out' when 5 then 'Overflow In' when 6 then 'Other In' when 7 then 'Auto Out/Predictive/Progressive' when 8 then 'Agent Out' when 9 then 'Out (of cluster)' when 10 then 'Agent Inside' when 11 then 'Offered' when 12 then 'Consult' when 13 then 'Consult Offered' when 14 then 'Consult Conf' when 15 then 'Conference' when 16 then 'Unmonitored' when 17 then 'Preview' when 18 then 'Reserve' when 19 then 'Supervisor Assist' when 20 then 'Emergency Call' when 21 then 'Supervisor Monitor' when 22 then 'Supervisor Whisper' when 23 then 'Supervisor Barge in' when 24 then 'Supervisor Intercept' when 25 then 'Route by ICM' when 26 then 'Route By Application Inst.' when 27 then 'Reservation Preview' when 28 then 'Reservation Preview Direct' when 29 then 'Reservation Predictive' when 30 then 'Reservation Callback' when 31 then 'Reservation Personal Callback' when 32 then 'Customer Preview' when 33 then 'Customer Preview Direct' when 34 then 'Customer Predictive' when 35 then 'Customer Callback' when 36 then 'Customer Personal Callback' when 37 then 'Customer IVR' else 'Unkown' end as PeripheralCallTypeName, Termination_Call_Detail.CallDisposition as CallDis, case CallDisposition when 1 then 'Aban Network' when 2 then 'Aban Locl Q' when 3 then 'Aban Ring' when 4 then 'Aban Delay' when 5 then 'Aban Interflow' when 6 then 'Aban Agent Term' when 7 then 'Short' when 8 then 'Busy' when 9 then 'Forced Busy' when 10 then 'Drop no answer' when 11 then 'Drop busy' when 12 then 'Drop Reorder' when 13 then 'Drop Handled Primary Route' when 14 then 'Drop Handled Other' when 15 then 'Redirected' when 16 then 'Cut Through' when 18 then 'Interflow' when 19 then 'RONA' when 20 then 'Intercept Reorder' when 21 then 'Intercept Denial' when 22 then 'Time out' when 23 then 'Voice Energy' when 24 then 'Non-classified Energy' when 25 then 'No Cut Through' when 26 then 'U-Abort' when 27 then 'Failed Software' when 28 then 'Blind Transfer' when 29 then 'Announced Transfer' when 30 then 'Conferenced' when 31 then 'Duplicate Transfer' when 32 then 'Unmonitored Device' when 33 then 'Answering Machine' when 34 then 'Network Blind Transfer'when 52 then 'Called Party Disconnected' else cast(calldisposition as varchar(5)) + ' Unknown -Schema Guide' end as CallDispositionName, --Termination_Call_Detail.CallDispositionFlag, case CallDispositionFlag when 1 then 'Handled' when 2 then 'Abandoned' when 3 then 'Short' when 4 then 'Error' when 5 then 'Redirected' when 6 then 'Requery' when 7 then 'Incomplete' end as CallDisFlagName, --Termination_Call_Detail.NetworkTarget, -----Values t_device_target.description as DvceTgtDesc, t_peripheral_target.dnis as PeriphTgtDNIS, t_peripheral_target.description as PeriphTgtDesc, t_trunk_group.enterprisename as TrunkGroupName, Termination_Call_Detail.InstrumentPortNumber as InstPrtNum, Termination_Call_Detail.NetworkTime as NetWkTime, Termination_Call_Detail.Duration, Termination_Call_Detail.RingTime, Termination_Call_Detail.DelayTime, Termination_Call_Detail.TimeToAband, Termination_Call_Detail.HoldTime, Termination_Call_Detail.TalkTime, Termination_Call_Detail.WorkTime, Termination_Call_Detail.LocalQTime, Termination_Call_Detail.ConferenceTime, Termination_Call_Detail.CED, Termination_Call_Detail.Variable1 as PVar1, Termination_Call_Detail.Variable2 as PVar2, Termination_Call_Detail.Variable3 as PVar3, Termination_Call_Detail.Variable4 as PVar4, Termination_Call_Detail.Variable5 as PVar5, Termination_Call_Detail.Variable6 as PVar6, Termination_Call_Detail.Variable7 as PVar7, Termination_Call_Detail.Variable8 as PVar8, Termination_Call_Detail.Variable9 as PVar9, Termination_Call_Detail.Variable10 as PVar10, Termination_Call_Detail.ICRCallKey, Termination_Call_Detail.AnsweredWithinServiceLevel as InSvcLvl, Termination_Call_Detail.Priority, Termination_Call_Detail.Trunk, Termination_Call_Detail.WrapupData, Termination_Call_Detail.SourceAgentPeripheralNumber as SourcePeriph, Termination_Call_Detail.SourceAgentSkillTarget as SourceSkill, Termination_Call_Detail.BadCallTag, Termination_Call_Detail.ApplicationTaskDisposition as AppDisp, Termination_Call_Detail.ApplicationData as AppData, Termination_Call_Detail.NetQTime, Termination_Call_Detail.RecoveryKey, Termination_Call_Detail.DbDateTime joins to make TCD readable from Termination_Call_Detail left outer join t_media_routing_domain on t_media_routing_domain.mrdomain = Termination_Call_Detail.MRDomain left outer join t_call_type on t_call_type.calltype = Termination_Call_Detail.CallType left outer join t_routing_client on t_routing_client.peripheral = Termination_Call_Detail.Peripheral left outer join t_route on t_route.route = Termination_Call_Detail.Route left outer join t_peripheral on t_peripheral.peripheral = Termination_Call_Detail.Peripheral left outer join t_skill_group on t_skill_group.skilltarget = Termination_Call_Detail.SkillGroupSkillTarget left outer join t_service on t_service.skilltarget = Termination_Call_Detail.ServiceSkillTarget left outer join t_agent on t_agent.skilltarget = Termination_Call_Detail.AgentSkillTarget left outer join t_device_target on t_device_target.networktarget = Termination_Call_Detail.NetworkTarget left outer join t_peripheral_target on t_peripheral_target.networktarget = Termination_Call_Detail.NetworkTarget left outer join t_trunk_group on t_trunk_group.trunkgroup = Termination_Call_Detail.TrunkGroup -----where clause where Termination_Call_Detail.DateTime between '7/1/2015 0:00' and '7/30/ :59' --and Termination_Call_Detail.DNIS like '%xxxxxx' --and RouterCallKey in (xxxxx) and not Termination_Call_Detail.CallDisposition in ( 2,3,6,7,10,13,14,15,19,24,28,29,30,33,52) order by RouterCallKeyDay,RouterCallKey,RouterCallKeySequenceNumber 85

86 Sample Queries RCD select --Route_Call_Detail.MRDomain, t_media_routing_domain.enterprisename as MRDomainName, Route_Call_Detail.RouterCallKeySequenceNumber as Seq#, Route_Call_Detail.BeganRoutingDateTime, --Route_Call_Detail.RouterCallKeyDay, Route_Call_Detail.RouterCallKey as RtrCallKey, Route_Call_Detail.ANI, --Route_Call_Detail.RoutingClient, Routing_Client.EnterpriseName as RtngClnt, --Route_Call_Detail.DialedNumber, Route_Call_Detail.DialedNumberString as DNString, t_dialed_number.description as DNDesc, t_call_type_b.enterprisename as BeginCallType, t_master_script_b.enterprisename as BeginScript, t_script_b.author as Script_Author, t_script_b.version as ScriptVer, t_script_b.datetime as Script_Modified_Date, --Route_Call_Detail.TargetType, --Route_Call_Detail.Route, t_route.enterprisename as RouteName, --t_route.skilltarget, Route_Call_Detail.DateTime, --Route_Call_Detail.CallType, t_call_type.enterprisename as CallTypeName, Route_Call_Detail.Label, Route_Call_Detail.TargetLabel, t_agent.enterprisename as 'AgentName*', --Agent name makes the assumption the agent peripheral = agent extension. Route_Call_Detail.RouterErrorCode as RtrErr#, case Route_Call_Detail.RouterErrorCode when 499 then 'Exceeded Queueu TimeLimit' when 435 then 'Dialog Failed' when 495 then 'Invalid Device Target' when 490 then 'No Agent or SG Found' when 448 then 'Abandoned IVR' when 383 then 'Max Nodes Reached' when 232 then 'No Label for this RoutingClient' when 66 then 'No Default Route' when 65 then 'No Route' when 64 then 'DN Not Scheduled' when 62 then 'No CallType for this DN' when 0 then 'Normal' else cast(routererrorcode as varchar(5)) + ' Unkown -Update sql query with master help' end as RouterErrorDesc, --Route_Call_Detail.RequestType, case RequestType when 1 then 'Pre-Route' when 2 then 'Blind Transfer' when 3 then 'Anounced Transfer' when 4 then 'Overflow' when 5 then 'Re-route' when 6 then 'Post-Route' end as RequestType, Route_Call_Detail.Script, t_master_script.enterprisename as EndScriptName, Route_Call_Detail.FinalObject, t_script.author as ES_Author, t_script.version as ES_Ver, t_master_script.currentversion as ES_CurVer, case when t_script.version <> t_master_script.currentversion then 'Inactive' else 'Active' end as ES_State, t_script.datetime as ES_Modified_Date, --Route_Call_Detail.TargetType, case TargetType when 0 then 'Ended Badly' when 1 then 'Default Routed' when 2 then 'Ended on Agent' when 3 then 'Ended on Service' when 4 then 'Ended on SkillGroup' when 5 then 'Ended on Anouncement' when 6 then 'Ended in Busy node' when 7 then 'Ended in Ring node' when 8 then 'Ended on Label' when 9 then 'Network Default Route' when 10 then 'Ended on Service Array' when 11 then 'Ended Badly' when 12 then 'Ended on Scheduled Target' when 13 then 'AdminSciptDone' when 14 then 'Call Disconnected' when 15 then 'Ended on Release Call Node' when 16 then 'Exceeded Queue Limit' when 17 then 'Ended on Agent to Agent Node' when 18 then 'Ended on Dynamic Label' when 19 then 'Ended on Divert On Busy Node' when 20 then 'Admin Failed Queued Calls' when 21 then 'Routed to agent in specified group' end as TargetTypeDesc, --Route_Call_Detail.OriginatorType, Route_Call_Detail.Priority, --Route_Call_Detail.MsgOrigin, Route_Call_Detail.Variable1 as PV1, Route_Call_Detail.Variable2 as PV2, Route_Call_Detail.Variable3 as PV3, Route_Call_Detail.Variable4 as PV4, Route_Call_Detail.Variable5 as PV5, Route_Call_Detail.Variable6 as PV6, Route_Call_Detail.Variable7 as PV7, Route_Call_Detail.Variable8 as PV8, Route_Call_Detail.Variable9 as PV9, Route_Call_Detail.Variable10 as PV10, Route_Call_Detail.CED, Route_Call_Detail.VruScripts, Route_Call_Detail.CDPD, Route_Call_Detail.UserToUser, Route_Call_Detail.CallSegmentTime, Route_Call_Detail.NetQTime, --Route_Call_Detail.CallTrace, --Route_Call_Detail.RecoveryDay, --Route_Call_Detail.TimeZone, --Route_Call_Detail.NetworkTarget, --Route_Call_Detail.Label, Route_Call_Detail.Originator, --Route_Call_Detail.TargetLabel, Route_Call_Detail.RouterQueueTime, --Route_Call_Detail.BeganCallTypeDateTime, Route_Call_Detail.RequeryResult, Route_Call_Detail.VruProgress, case OriginatorType when 0 then 'Unknown' when 1 then 'Trunk' when 2 then 'Teleset' when 3 then 'VRU' when 4 then 'Trunk Group' end as OriginatorTypeName, Route_Call_Detail.RoutingClientCallKey, Route_Call_Detail.RecoveryKey, Route_Call_Detail.DbDateTime from Route_Call_Detail left outer join t_script on t_script.script = Route_Call_Detail.Script left outer join t_master_script on t_master_script.masterscript = t_script.masterscript left outer join Routing_Client on Routing_Client.RoutingClient = Route_Call_Detail.RoutingClient left outer join t_call_type on Route_Call_Detail.CallType = t_call_type.calltype left outer join t_agent on t_agent.peripheralnumber = Route_Call_Detail.Label left outer join t_route on t_route.route = Route_Call_Detail.Route left outer join t_media_routing_domain on t_media_routing_domain.mrdomain = Route_Call_Detail.MRDomain left outer join t_dialed_number on t_dialed_number.dialednumber = Route_Call_Detail.DialedNumber left outer join t_dialed_number_map on t_dialed_number_map.dialednumber = t_dialed_number.dialednumber left outer join t_call_type_map on t_call_type_map.calltype = t_dialed_number_map.calltype left outer join t_master_script t_master_script_b on t_master_script_b.masterscript = t_call_type_map.masterscript left outer join t_script t_script_b on t_master_script_b.masterscript = t_script_b.masterscript and t_master_script_b.currentversion = t_script_b.version left outer join t_call_type t_call_type_b on t_dialed_number_map.calltype = t_call_type_b.calltype where Route_Call_Detail.DateTime between '7/1/2016 1:00' and '7/30/ :59' --and not RouterErrorCode=XXX --and ANI='XXXXXXXX' --order by VruScripts desc order by BeganRoutingDateTime desc 86

87 Log Collection and Tools

88 Points of Attack Ingress Gateway Gateway logging CUBE or Ingress GW CVP CVP call server logs IVR ICM SIP CVP VRU PG PIM logs OPC logs VRU PG VRU PG Router Router logs VXML Gateway Gateway logging Router VXML GW Agent PG JGW logs OPC logs PIM logs VXML Server Activity logs OPC PIM CTISVR JGW AGENT PG CUCM Call CI Global Call Call Handler CUCM 88

89 Diagnostic Framework Portico 89

90 UCCE Troubleshooting Tools Diagnostic Portico Locally Installed Components Direct Link Bookmark Smart Refresh Remembers scroll position Red <10 min. Yellow >10 min. <30 min. Sub-Component Status Blue= status change 10 mins. Uptime 90

91 UCCE Troubleshooting Tools Diagnostic Portico UCCE-server Support Tool More than just Process Status! Set tracing for single process 91

92 UCCE Troubleshooting Tools Diagnostic Portico UCCE-server Support Tool More than just Process Status! Set tracing for single process 92

93 UCCE Troubleshooting Tools Diagnostic Portico Log Download 93

94 Unified System CLI 94

95 UCCE Troubleshooting Tools Unified System CLI Most Commonly Used Commands admin:system Initializing system mode Retrieving device list. This process may take a few minutes to complete NOTE: Use this command to enter system mode to collect data/logs remotely across UCCE servers. admin:system init Initializing system mode Retrieving device list. This process may take a few minutes to complete admin(system): NOTE: Use this command when you make an update or change devices.csv admin(system):show version Retrieving [version] data from device [icmroggera] ProductType [icm] ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465 Retrieving [version] data from device [icmpg1a] ProductType [icm] ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465 NOTE: Use this command in System mode to show all device versions. admin(system):show processes Peripheral Gateway 1A jtapigw.exe : pcce-pg1a jtapigw jgw1 [ ACTIVE ] : 2.02:01: mdsproc.exe : pcce-pg1a mdsproc (InSvc Pr-Enb Clk) : 2.02:01: opc-cce.exe : pcce-pg1a opc-cce.exe : 2.02:01: Pgagent.exe : pcce-pg1a pgagent (Not active) : 2.02:01: NOTE: Provides sub-component status 95

96 UCCE Troubleshooting Tools Unified System CLI Most Commonly Used Commands Cont admin(system):show devices Device host name: icmroggera Description: RoggerA Password: *********** Alternate user: Enable password: *********** Product type: ICM Group: Device host name: icmpg1a Description: PG1A Password: *********** Alternate user: Enable password: *********** Product type: ICM Group: NOTE: Use this command to list all configured / recognizable devices by the active Unified CLI interface admin(system):show debug Retrieving [debug] data from device [icmroggera] ProductType [icm] Component: icm:router A, subcomponent: icm:rtr admin(system):show config redirect dir c:\temp Retrieving [configuration] data from device [icmroggera] ProductType [icm] Fetching configuration resource for component DumpCfg Configuration: DumpCfg NOTE: This command will snapshot configuration information such as: ConfigExport, Registry, DumpCfg, etc. For PG s, it will snapshot the registry. redirect dir will save the clioutput(#).zip into the temp directory. If that directory is not present, Unified CLI will create it. Trace level = 0 Description: Application data: NOTE: Use this command to show current trace levels. System mode will show all UCCE server components 96

97 UCCE Troubleshooting Tools Unified System CLI Most Commonly Used Commands Cont admin(system):debug level 1 component icm:router A subcomponent icm:rtr Warning: Because running this command can affect system performance, Cisco recommends that you run the command during off-peak hours. Do you want to continue? [y / n]: y Retrieving [debug_level] data from device [icmroggera] ProductType [icm] Component: icm:router A, subcomponent: icm:rtr Debug level is set successfully NOTE: Use this command to set debug tracing Requires Quotes admin(system):show tech-support Warning: Because running this command can affect system performance, Cisco recommends that you run the command during off-peak hours. Do you want to continue? [y / n]: y Output is saved to C:\icm\serviceability\wsccli\.\download\clioutput1.zip NOTE: In System mode, this command will grab a 24hr snapshot of all server and subcomponent logs including OPC Capture on PG s. It will also snapshot server versions, export ICM config, dumpcfg, registries, current debug levels, performance (PerfMon), license info, etc. In other words, it gets EVERYTHING TAC/DE would need. If you have UCM configured in devices.csv, it will also grab a 24hr snapshot of CM/CTI/SDL logs. 97

98 Proactive Measures

99 UCCE Monitoring Capabilities - SNMP High Level Overview VRU PG EMS syslog Syslog Collector Router EMS Logger SNMP AW/ HDS EMS SNMP Network Management System 99

100 UCCE Monitoring Capabilities - SNMP UCCE SNMP Event Flow Router Router Logger CW2K Feed Syslog Syslog Collector CSFS mds EMS CCCA Agent SNMP DM SNMP EMS EMS EMS t_event SNMP NMS 100

101 UCCE Monitoring Capabilities SNMP MIB Overview CISCO-CONTACT-CENTER-APPS-MIB Cisco Unified Contact Center Enterprise <drive>:\icm\snmp CSFS CISCO-CVP-MIB Cisco Unified Customer Voice Portal <drive>:\cisco\cvp\conf HOST-RESOURCES-MIB SYSAPPL-MIB MIB-II Cisco MCS Server Platform MIBs Vendor-supplied (HP/IBM) Logger CCCA Agent SNMP DM SNMP SNMP NMS 101

102 Problem Isolation

103 Problem isolation What kind of questions TAC asks when you open a Case? Versions (full version including ES) Is this a new installation or upgrade? Has this ever worked before? If it was working before, what changes have you made since this started? Call flow / topology? Which agents / numbers / locations are affected? What is the exact prompt/error? 103

104 What do we need to know? Business Impact Contact Center is impacted but which part of the business? How many calls / agents / sites are affected? Engagement Who is involved and why? Main contact and phone number Follow the sun until resolution? Time Saver Provide the Call s, ANI, DNIS, Timestamps, steps done to reproduce the issue, symptoms experienced (for caller and agent), output from TCD and RCD tables What is the expected behavior for working call? 104

105 Tools of the Trade

106 Troubleshooting Tools Monitoring: Serviceability: Troubleshooting: Performance monitors SNMP 2013 Diagnostic Framework Portico Analysis manager System CLI PCCE ONLY 2013 System inventory One Click Log Collection Text Editors Notepad++ jedit Gsplit WireShark 2014 CVP Call Flow Tool 106

107 Questions?

108 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 108

109 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions 109

110 Thank you

111