Genesys Interaction Recording Solution Guide. Troubleshooting

Transcription

1 Genesys Interaction Recording Solution Guide Troubleshooting 7/25/2018

2 Troubleshooting Contents 1 Troubleshooting 1.1 What Happens if MCP fails to send recordings to Recording Processor? 1.2 What Happens if Recording Processor fails to send recordings to SpeechMiner or Interaction Recording Web Services (Web Services)? 1.3 How do I collect logs to debug issues relating to login to the SpeechMiner UI? 1.4 Why do only some of my calls have screen recordings associated with them? 1.5 How do I restore the Find Recordings capability? Genesys Interaction Recording Solution Guide 2

3 What Happens if MCP fails to send recordings to Recording Processor? If you have scheduled maintenance for an extended period of time which could cause lost recordings, Genesys recommends that you reconfigure Media Control Platform (MCP) to stop posting recording, so that you do not have to handle the failed recordings. Important Genesys recommends that your deployment be sized to accommodate 48 hours worth of recording on disk, and that there is plenty of physical memory available on the MCP machine, so that the recording can reload efficiently when MCP restarts. To reconfigure MCP: 1. Stop MCP. 2. Set the recordnumparallelpost parameter in the [mcp] section to Restart MCP. This disables the MCP posting functionality and all recording/metadata files will continue to accumulate in the cache/ record directory. 4. When suitable, stop MCP, set the recordnumparallelpost parameter back to the recommended/default value and restart MCP. MCP will now scan the cache/record directory and post the requests for the recording files that are present. Handling Failed Recordings For handling MCP post failures in the cache directory: 1. Find the MCP's $InstallationRoot$/cache/record/failed directory for the following file types: [audiofilename]-htccmetadata.json [audiofilename] [audiofilename].pem (provided only when encryption is enabled) The audio file name can have extensions of.mp3 or.mp3.bin. For example: 743CTLVFHH0TH7HHB4D1PTS5US00000F_ _ mp3-htccmetadata 743CTLVFHH0TH7HHB4D1PTS5US00000F_ _ mp3 2. Submit these files to: 1. Recording Processor 2. Recording storage (WebDAV) Genesys Interaction Recording Solution Guide 3

4 3. Recording post failures can be due to incorrect provisioning in the IVR Profile, so before posting the metadata file to Recording Processor, look at the metadata file to make sure provisioning is correct. [+] See example { } "mediafiles": [ { "duration": "0", "ivrprofile": "DefaultProfile", "mediadescriptor": { "path": " "storage": "webdav" }, "mediaid": "743CTLVFHH0TH7HHB4D1PTS5US00000G_ _ F mp3", "parameters": { "agentid": "1001", "ani": "5000", "calluuid": "743CTLVFHH0TH7HHB4D1PTS5US00000G", "connid": " eab5010", "datetime": " T15:46:21Z", "dnis": "3001", "id": "743CTLVFHH0TH7HHB4D1PTS5US00000G_ _ ", "recorddn": "2001", "rp.speechminer_auth": "user:password", "rp.speechminer_uri": " "sipsappname": "SIPServer" }, "size": "0", "starttime": " T15:46:21Z", "stoptime": " T15:46:21Z", "tenant": "Environment", "type": "audio/mp3" } ] Check for following items in the json file: mediafiles[0].mediadescriptor.path Make sure this is the correct WebDAV address mediafiles[0].parameters['rp.speechminer_uri'] Make sure this is the correct location of SpeechMiner Interaction Receiver mediafiles[0].parameters['rp.speechminer_auth'] Make sure this is the correct authorization of SpeechMiner Interaction Receiver 4. When encryption is enabled, edit the json file and manually add the pkcs7 headers which is the content of the.pem file (make sure that you replace the line break with \n). [+] See example { "mediafiles": [ { "duration": "0", "ivrprofile": "DefaultProfile", "mediadescriptor": { "path": " "storage": "webdav" }, "mediaid": "743CTLVFHH0TH7HHB4D1PTS5US00000G_ _ F mp3", "parameters": { "agentid": "1001", "ani": "5000", "calluuid": "743CTLVFHH0TH7HHB4D1PTS5US00000G", Genesys Interaction Recording Solution Guide 4

5 } ] } "connid": " eab5010", "datetime": " T15:46:21Z", "dnis": "3001", "id": "743CTLVFHH0TH7HHB4D1PTS5US00000G_ _ ", "recorddn": "2001", "rp.speechminer_auth": "user:password", "rp.speechminer_uri": " "sipsappname": "SIPServer" }, "size": "0", "starttime": " T15:46:21Z", "stoptime": " T15:46:21Z", "tenant": "Environment", "type": "audio/mp3", "pkcs7": "-----BEGIN PKCS7-----\nMIIBJQYJKoZIhvcNAQcDoIIBFjCCARICAQAxgeAwgd0CAQAwgYYwgYAxCzAJBgNV\nBAY 5. When you are ready to submit the files, do the following steps in order: a. Issue an HTTP PUT to WebDAV: curl -u [webdavauth] -T [audiofilename] [mediafiles[0].mediadescriptor.path] Using this example above and assuming the WebDAV authorization is webdav:password, you will see the following returned: curl -u webdav:password -T 743CTLVFHH0TH7HHB4D1PTS5US00000G_ _ F b. Issue an HTTP POST to Recording Processor: curl -u [rpauth] -X POST --header "Content-Type: application/json" [rpuri] where: [rpauth] is the Recording Processor authorization as per the IVR Profile [metadatafile] is the metadata json file in MCP's cache/failed directory [rpuri] is the Recording Processor URI as per the IVR Profile Using the example above, you should see the following returned: curl -u user:password -X POST -htccmetadata.json --header "Content-Type: application/json" What Happens if Recording Processor fails to send recordings to SpeechMiner or Interaction Recording Web Services (Web Services)? If Recording Processor fails to send recordings to SpeechMiner or Interaction Recording Web Services (Web Services if you're using version or earlier) for any reason, Recording Processor will continue trying to send these recordings until it exhausts the number of retries configured. When the number of retries have been exhausted, Recording Processor logs an INFO message to the log file with the corresponding JSON content that is not submitted to SpeechMiner or Interaction Recording Web Services (Web Services). The following steps show Genesys Interaction Recording Solution Guide 5

6 how to search for the specific messages in the log file to recover the post failure to SpeechMiner and Interaction Recording Web Services (Web Services). Prerequisites: Make sure that the Recording Processor logging is set to the INFO level. Important Metadata for the same call could be sent multiple times to Interaction Recording Web Services (Web Services) or SpeechMiner if the call is transferred. Therefore, even if a given metadata record is not sent upstream, the call may still exist in the Interaction Recording Web Services (Web Services) or SpeechMiner database. Handling Failed Recordings to Interaction Receiver (SpeechMiner) 1. Search the Recording Processor log files for the follow message: "Metadata will not be sent to SM". For example: grep "Metadata will not be sent to SM" recordingprocessor_071614_ log [+] Show code snippet returned :27:53,252 process_metadata INFO Metadata will not be sent to SM: {"status": 1, "retries": 6, "dialedphonenumber": "91234", "region": "us", "mediafiles": [{"mediaid": "DHMI6GITSD5D99H23MRA3P74IC00000C_ _ C-10003C2A mp3", "parame {"recorddn": "1234", "ani": "1234", "datetime": " T19:34:43Z", "sipsappname": "SIPServer", "connid": " b5a25700c", "calluuid": "DHMI6GITSD5D99H23MRA3P74IC00000C", "dnis": "91234", "rp.speechminer_auth": "user:password", "id": "DHMI6GITSD5D99H23MRA3P74IC00000C_ _ ", "rp.speechminer_uri": " "accessgroups": [], "duration": "3027", "stoptime": " T19:34:47Z", "starttime": " T19:34:44Z", "mediadescriptor": {"path": " "storage": "webdav"}, "calluuid": "DHMI6GITSD5D99H23MRA3P74IC00000C", "ivrprofile": "DefaultProfile", "size": "11952", "type": "audio/mp3", "tenant": "Environment", "partitions": []}], "eventhistory": [], "callerphonenumber": "1234", "id": "DHMI6GITSD5D99H23MRA3P74IC00000C", "ccid": " f30-bb07-2c95a22aaa89"} 2. Save the json string after "Metadata will not be sent to SM:" as a text file. 3. Issue an HTTP POST to Interaction Receiver for the saved json text file. For example: curl -u <user:password> -X POST --header "Content-Type: application/json" < Where: <user:password> is the user/password authorization for Interaction Receiver < is the URI of Interaction Receiver Genesys Interaction Recording Solution Guide 6

7 Handling Failed Recordings to Interaction Recording Web Services (Web Services) 1. Search the Recording Processor files for the following message: "Metadata will not be sent to HTCC". For example, grep "Metadata will not be sent to HTCC" recordingprocessor_071714_ log [+] Show code snippet returned :26:41,354 process_metadata INFO Metadata will not be sent to HTCC: {"status": 2, "retries": 6, "2001", "region": "us", "mediafiles": [{"mediaid": "J0GOONKU6P74B4ASS9VGS1J2VS000002_ _ "parameters": {"username": "person1", "agentid": "1001", "recorddn": "2001", "ani": "1234", "datetime": " T12:23: "sipsappname": "SIPServer", "connid": " c1394e002", "calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", "d "rp.speechminer_auth": "user:password", "id": "J0GOONKU6P74B4ASS9VGS1J2VS000002_ _ ", "rp.s " "accessgroups": ["/team"], "duration": "15803", "stoptime": " T12:23:31Z", "starttime": " "mediadescriptor": {"path": " "storage": "webdav"}, "calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", "ivrprofile": "DefaultProfile", "partitions": [], "type": "Environment", "size": "62352"}], "eventhistory": [{"calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", "contac {"phonenumber": "1234", "type": "External", "name": "EXTERNAL"}, "occurredat": " T12:23:12Z", "event": "Joined"}, {"calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", {"phonenumber": "1234", "type": "External", "name": "EXTERNAL"}, "occurredat": " T12:23:31Z", "event": "Left"}, {"calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", " {"username": "person1", "phonenumber": "2001", "type": "User", "firstname": "Person", "lastname": "One"}, "occurredat": " T12:23:14Z", "event": "Joined"}, {"calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", {"username": "person1", "phonenumber": "2001", "type": "User", "firstname": "Person", "lastname": "One"}, "occurredat": " T12:23:31Z", "event": "Left"}, {"eventid": " :23:14.613_J0GOONKU6P74B "calluuid": "J0GOONKU6P74B4ASS9VGS1J2VS000002", "occurredat": " T12:23:14Z", "event": "Data", "dat {"WrapUpTime": "0"} }}, {"eventid": " :23:15.677_J0GOONKU6P74B4ASS9VGS1J2VS000002", "calluuid": "J0GOONKU6P74B4ASS9V "occurredat": " T12:23:15Z", "event": "Data", "data": {"added": {"IW_CaseUid": "dcb eaf-4d17-923c-3d961d93cb86", "IW_BundleUid": "afe4a570-ca5d-466b-9f57-03f261d07717 }}, {"eventid": " :23:16.220_J0GOONKU6P74B4ASS9VGS1J2VS000002", "calluuid": "J0GOONKU6P74B4ASS9V "occurredat": " T12:23:16Z", "event": "Data", "data": {"added": {"GSIP_RECORD": "ON", "GSIP_REC_FN": "J0GOONKU6P74B4ASS9VGS1J2VS000002_ _ "} }}, {"eventid": " :23:31.370_J0GOONKU6P74B4ASS9VGS1J2VS000002", "calluuid": "J0GOONKU6P74B4ASS9V "occurredat": " T12:23:31Z", "event": "Data", "data": {"deleted": {"WrapUpTime": "0"} }}, {"eventid": " :23:31.377_J0GOONKU6P74B4ASS9VGS1J2VS000002", "calluuid": "J0GOONKU6P74B4ASS9V "occurredat": " T12:23:31Z", "event": "Data", "data": {"deleted": {"GSIP_RECORD": "ON"} }}], "callerphonenumber": "1234", "id": "J0GOONKU6P74B4ASS9VGS1J2VS000002", "ccid": " f30-bb07 2. Save the json string after "Metadata will not be sent to HTCC:" as a text file. 3. Issue an HTTP POST to Interaction Recording Web Services (or Web Services if you're using version or earlier) for the saved json text file. For example, curl -u <user:password> -X POST --header "Content-Type: application/json" < Where: <user:password> is the user/password authorization for Interaction Recording Web Services (Web Services) < is the URI of Interaction Recording Web Services (Web Services) {ccid} is the contact center identifier on Interaction Recording Web Services (Web Services) Genesys Interaction Recording Solution Guide 7

8 You can find the contact center identifier from the initial Interaction Recording Web Services (Web Services) setup, or you can look up the contact center id from the recording processor log file. In the first few lines of the log file you will find a GUID string as follows: :40:43,746 contact_centers INFO Contact center (: f30-bb07-2c95a22aaa89) discover The following example can be used to return the contact center ID: curl -u user:password -X POST --header "Content-Type: application/json" How do I collect logs to debug issues relating to login to the SpeechMiner UI? If you are having a problem logging into the SpeechMiner UI, you will need to collect the Network and Console log from the Chrome Developer Tools before and after the login attempts with the Preserve log checkbox enabled for example: Make sure that you have captured the requests for the following applications after you click the login button: Web Services Recording Crypto Server SpeechMiner Important Uncheck these options after you have collected the necessary information. Genesys Interaction Recording Solution Guide 8

9 Why do only some of my calls have screen recordings associated with them? Screen Recording only occurs when an agent logs into a workstation that has Screen Recording Service installed. To solve this issue try one of the following: If the Screen Recording Service is installed on your workstation and this problem persists, execute the following command to check the screen recording trigger condition settings for a given agent (for example, julie@genesys.com): curl -u julie@genesys.com: If the code below appears with Value = off, Screen Recording is disabled. If Value = random_voice(x) (where x is 0-100, representing the percentage of calls to record) then, the calls that do not fall into the selected percentage will not be recorded. The SRS is instructed to either record or not-record on a per-call basis. If you are not seeing screen recordings for your deployment, or, for a particular agent, you may wish to temporarily disable random_voice or set to 100 to ensure that all interactions are instructed to have the screens recorded. "statuscode": 0, "settings": [ { "name": "recordingwhen", "value": <to-be-checked> } ] To reconfigure Screen Recording, refer to Advanced Configuration for the Screen Recording Service. Go to C:\Genesys\SRC\Problem Videos. If the Problem Videos folder contains files, verify that you have the following information and contact your Genesys Professional: GSR.log config.json The agent's name. Directory listing How do I restore the Find Recordings capability? Interaction Recording Web Services (or Web Services if you're using version or earlier) uses Elasticsearch for metadata indexing, which is used by MLM, RCS archiving and the SpeechMiner Screen Recording grid. If your Elasticsearch index is out of date, follow these steps to rebuild the index and restore the Find Recordings capability: 1. Select one Interaction Recording Web Services (Web Services) node, and do one of the following: Genesys Interaction Recording Solution Guide 9

10 If you have a regular maintenance window, select an existing Interaction Recording Web Services (Web Services) node and proceed to step #2. If your business runs 24*7, prepare a new dedicated Interaction Recording Web Services (Web Services) node as follows: a. Install Interaction Recording Web Services (Web Services) in the same way a regular Interaction Recording Web Services (Web Services) node is installed. Do not add this node to the Interaction Recording Web Services (Web Services) Load Balancer. b. Edit the Interaction Recording Web Services application.yaml file (if you are using Web Services and Application version or earlier modify the server-settings.yaml file instead), by adding the following configuration. Verify that you add lines under nodes for all the existing Interaction Recording Web Services (Web Services) nodes in your deployment: elasticsearchsettings: usetransportclient: true transportclient: nodes: - {host: <elastic-search-node1>, port: 9300} - {host: <elastic-search-node2>, port: 9300} - {host: <elastic-search-node3>, port: 9300} usesniff: true ignoreclustername: true pingtimeout: nodessamplerinterval: enableindexverificationatstartup: false indexpercontactcenter: true Important By default, the Interaction Recording Web Services (Web Services) node will also serve as the Elasticsearch node. This step is performed to avoid any change to the existing Elasticsearch Cluster. 2. Increase the Hystrix timeout for RecordingOperationApiTaskV2 by adding the following line to the Hystrix configuration: hystrix.command.recordingoperationapitaskv2.execution.isolation.thread.timeoutinmilliseconds=<max time accep Important <max time acceptable in milliseconds> should be 3 times the restored period or more. 3. Restart the selected Interaction Recording Web Services (Web Services) node, navigate to the following URL and wait until it returns the Interaction Recording Web Services (Web Services) version: Genesys Interaction Recording Solution Guide 10

11 4. Perform Reindexing: a. Determine the contact center ID using the following command: curl -u <ops-user>:<ops-pass> selected-web-services-node>: <webservices-listening-port>/api/v2/ops/contact-centers; echo The following output is returned: {"statuscode":0,"uris":[" selected-web-services-node>: <web-serviceslistening-port>/api/v2/ops/contact-centers/<contact-center-id>"]} b. Run the following to regenerate indexing for Voice Recording if required. If you do not need to regenerate indexing for voice recording, skip this step and proceed to 4c. curl -u <ops-user>:<ops-pass> -XPOST -H "Content-Type:application/json" " contact-centers/<contact-center-id>/recordings" -d '{ "operationname":"forceindex", "from": <start-range-in-milliseconds>, "to": <stop-range-in-milliseconds>, "purgeold":false }' c. Run the following command to regenerate indexing for Screen Recording: curl -u <ops-user>:<ops-pass> -XPOST -H "Content- Type:application/json" " -d '{ "operationname":"forceindex", "from": <start-range-in-milliseconds>, "to": <stop-range-in-milliseconds>, "purgeold":false }' Genesys Interaction Recording Solution Guide 11

12 Important If purgeold is true, the entire index will be purged. For this reason, set purgeold to false when only a period of your index is lost. The typical scenario is nightly indexing for a multi-tenancy deployment. When your existing ES map is wrong and you must rebuild the entire index, you should set purgeold to true. If your restore range is too long and you want to restore each period at a time, verify that you set purgeold to false after the first period is restored. Verify that the operation is only executed during the period at which the MLM or RCS archive job is scheduled. 4. Once all re-indexing operations are complete: Remove the change to the Hystrix timeout for the RecordingOperationApiTaskV2 that was added in step 2. Restart the selected Interaction Recording Web Services (Web Services) node. Genesys Interaction Recording Solution Guide 12