Cisco Virtual Media Recorder User Guide

Transcription

1 First Published: Last Modified: Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA USA Tel: NETS (6387) Fax:

2 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: / Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R) 2018 Cisco Systems, Inc. All rights reserved.

3 CONTENTS Preface Preface vii New and Changed Information vii Audience vii Document Organization vii Conventions viii Related Documentation ix CHAPTER 1 Getting Started Using VMR 1 Sign In 1 API Server Connection 3 Sign Out 4 CHAPTER 2 Managing User Accounts 5 Create User Accounts 5 Create Your Own User Account 6 Create Administrator Account 7 Approve New Account 8 Search User Accounts 9 Enable and Disable User Accounts 9 Update User Accounts 10 Update Your Own Account 11 Change Account Password 12 Delete User Account 12 CHAPTER 3 Managing Recordings 13 Create Recording 14 Create Multiple Recordings 15 iii

4 Contents Search Recordings 16 View Recording Details 18 VMR Recording Status 18 Update Recording 22 Play Recording 23 Audit Recordings 24 Start Audit 24 Stop Audit in Progress 24 View Audit Results 24 Stop All Audits in Progress 24 Clear Audit Reports 25 Configure Additional Audit Parameters 25 Delete Recording 25 Download Recording Time Segments 26 CHAPTER 4 Managing Streams 27 Create Stream 28 Search Streams 30 View Stream Details 32 Configure Imported Streams 32 Sample File Formats 34 Update Stream 35 Delete Stream 36 CHAPTER 5 Monitoring VMR Health 37 Introduction 37 Monitoring VMR Endpoints 37 Configure Endpoint Monitoring 38 View Alarms and Events 39 View Alarms in Sensu Uchiwa GUI 39 View Alarms in Command Line Interface 40 Alarm Definitions 40 MemSQL Checks 43 CHAPTER 6 Troubleshooting 47 iv

5 Contents Scheduled Recording Fails 47 Issues with No De-Dup Value 47 NoDeDup Value Not Updated in Database 48 No De-Dup Value Different After Re-Enabling MFC 48 Channel Added with Incorrect De-Dup Value 48 v

6 Contents vi

7 Preface This guide provides instructions for using Cisco Virtual Media Recorder (VMR), the component in the Cisco Cloud DVR (cdvr) solution that supports segment-based media recording and playout. The content in this guide reflects features implemented in VMR and earlier. New and Changed Information, page vii Audience, page vii Document Organization, page vii Conventions, page viii Related Documentation, page ix New and Changed Information This section contains a summary of new and changed information in this guide for VMR Release 1.3.3_053. Feature Location VMR Health Agent (New) Introduction MemSQL Cluster Health Monitoring (New) MemSQL Checks Audience This guide is intended for use by network administrators responsible for installing, configuring, and troubleshooting VMR and related software components. The reader should already be familiar with OpenShift, Kubernetes, Docker, containerized software, and cloud computing concepts in general. Document Organization This document contains the following chapters: vii

8 Conventions Preface Chapter Getting Started Using VMR Managing User Accounts Managing Recordings Managing Streams Monitoring VMR Health Troubleshooting Description Describes how to sign in and sign out of VMR, and information about the API connection. Provides instructions for managing user accounts creating, searching, enabling and disabling, updating and deleting accounts, and updating password. Provides instructions for managing recordings creating, searching, viewing details, playing, updating, deleting, and auditing recordings, and downloading recording time segments. Provides instructions for managing streams creating, searching, viewing details, updating, deleting, and configuring imported streams. Provides information about the VMR health monitoring agent, how to view alarms and events, and alarm definitions. Provides information about troubleshooting issues in VMR. Conventions This guide uses the following conventions. Conventions bold font italic font [ ] {x y z } [ x y z ] string courier font < > Indication Commands and keywords and user-entered text appear in bold font. Document titles, new or emphasized terms, and arguments for which you supply values are in italic font. Elements in square brackets are optional. Required alternative keywords are grouped in braces and separated by vertical bars. Optional alternative keywords are grouped in brackets and separated by vertical bars. A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks. Terminal sessions and information the system displays appear in courier font. Nonprinting characters such as passwords are in angle brackets. viii

9 Preface Related Documentation Conventions [ ]!, # Indication Default responses to system prompts are in square brackets. An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line. Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual. Caution Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data. Warning IMPORTANT SAFETY INSTRUCTIONS Means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device. SAVE THESE INSTRUCTIONS Related Documentation Refer to the following documents for additional information about VMR 1.3.3: Cisco Virtual Media Recorder Deployment Guide Release Notes for Cisco Virtual Media Recorder Open Source Used in Cisco Virtual Media Recorder ix

10 Related Documentation Preface x

11 CHAPTER 1 Getting Started Using VMR The VMR Dashboard is a graphical user interface (GUI) that lets you view, analyze, play, configure, and search recordings and streams via a web browser. This chapter provides the following information about getting started using the VMR Dashboard: Sign In, page 1 API Server Connection, page 3 Sign Out, page 4 Sign In The VMR Dashboard is accessed using a web browser. The Dashboard supports Google Chrome, Apple Safari, and Mozilla FireFox browsers. Note The VMR Dashboard may take longer to load (10 seconds or longer) when using Mozilla Firefox. To avoid this issue, Cisco recommends using Google Chrome or Apple Safari. Note Access to the Dashboard requires that VMR UI-to-API secrets be enabled and configured as described in the Cisco Virtual Media Recorder Deployment Guide. 1

12 Sign In Getting Started Using VMR Perform the following steps to sign in to the VMR Dashboard: Step 1 Open a web browser and enter the URL for the VMR Dashboard in the address bar ( This URL contains the name of the domain (<kubernetes_master_ip>) or IP address that V2PC is using to host VMR. The port is Figure 1: Sign In Step 2 Sign in with the default credentials or click Create Account to create your own account (see Create Your Own User Account). The default credentials are as follows: Note User Name root Password dejaneiro Your administrator can change these credentials. If they do not work or stop working, contact your administrator for updated root sign-in information. The VMR Dashboard opens by default to the Search Recording pane. 2

13 Getting Started Using VMR API Server Connection Figure 2: Search Recording Pane API Server Connection All VMR Dashboard operations are supported by an underlying API, and it automatically connects to the API server at sign in. It then checks every 15 seconds to confirm that the connection is active. 3

14 Sign Out Getting Started Using VMR If connection to the API server is lost, the VMR Dashboard is disabled and an error message is displayed on a dimmed screen. Figure 3: Dashboard Dimmed (Lost API Server Connection) When the connection is restored, the VMR Dashboard automatically detects the connection and becomes available. Sign Out To sign out of the VMR Dashboard, click the (Sign Out) icon in the upper right corner of the dashboard. 4

15 CHAPTER 2 Managing User Accounts This chapter provides instructions for managing user accounts in VMR. It includes the following topics: Create User Accounts, page 5 Search User Accounts, page 9 Enable and Disable User Accounts, page 9 Update User Accounts, page 10 Update Your Own Account, page 11 Change Account Password, page 12 Delete User Account, page 12 Create User Accounts You can create your own user account from the Sign In page or, if you are an administrator, you can create user accounts from the VMR Dashboard and approve them. 5

16 Create Your Own User Account Managing User Accounts Create Your Own User Account Perform the following steps to create your own account: Step 1 The first time you sign in to the VMR Dashboard, the Create Account button is displayed in the upper right portion on the Sign In pane. Click Create Account to open the User Registration dialog. Figure 4: Registration Dialog Step 2 Enter your user information and click Register. Note Your system administrator must approve your account request before you can sign in with your new credentials. For details, see Enable and Disable User Accounts. 6

17 Managing User Accounts Create Administrator Account Create Administrator Account To create an Administrator account, you must first create the new account, and then approve it. Perform the following steps to create the new account: Step 1 Step 2 Sign in to the VMR Dashboard using your Admin credentials. In the navigation pane, expand the Account tab and select Create User. Figure 5: Account > Create User Step 3 Complete the information and click Create. The Created user Successfully! message is displayed in the lower portion on the pane. 7

18 Approve New Account Managing User Accounts Approve New Account Perform the following steps to approve a new user account: Step 1 Step 2 In the navigation pane, expand the Account tab and select Search User. Enter the search criteria to search for the user account and click Search. Figure 6: Account > Search User Step 3 Step 4 Locate the new account in the Search Result table in the lower pane and click the Approve icon in the Action column. On the Activate Account confirmation dialog, click OK. The new Admin user account is now approved and the user can sign in to the VMR Dashboard. 8

19 Managing User Accounts Search User Accounts Search User Accounts Perform the following procedure to search for a user account. Note You must be a Root or Admin user to search user accounts. Step 1 Step 2 Step 3 Step 4 Sign in to the VMR Dashboard. In the navigation pane, expand the Account tab and select Search User. The Search User pane opens. Enter the search criteria in the Search User pane and click Search. Locate the account in the Search Result table in the lower pane. Enable and Disable User Accounts Perform the following steps to enable or disable user accounts: Step 1 Step 2 Step 3 Sign in to the VMR Dashboard. In the navigation pane, expand the Account tab and select Search User. In the Search User pane, use the options in the Status field to filter and search for the account: For an account to be enabled, check the Disabled check box and then click Search. For an account to be disabled, check the Enabled check box and then click Search. Step 4 Step 5 Locate the account in the Search Result table. In the Action column, click the Lock icon and then click OK on the confirmation dialog box. The status of the account changes to Enabled or Disabled, depending on the previous state. 9

20 Update User Accounts Managing User Accounts Update User Accounts If you are a Root or Admin user, you can update other user s account information. Step 1 Step 2 Step 3 Step 4 Sign in to VMR dashboard using your Root or Admin credentials. In the navigation pane, expand the Account tab and select Search User. Select the search criteria and then click Search. Locate the user in the Search Result table in the lower pane and click the Update icon in the Action column. The Update Account dialog box opens showing the current information for the selected user. Figure 7: Update Account Dialog Step 5 Step 6 Modify the User , Full Name or Role, and then click Save Changes. Confirm that the updated user information is displayed in the Search Result table in the lower pane. 10

21 Managing User Accounts Update Your Own Account Update Your Own Account Perform the following steps to update the address or full user name for your own account: Step 1 Step 2 Sign in to VMR dashboard using your credentials. In the navigation pane, expand the Account tab and select My Account. Figure 8: Account > My Account Step 3 Modify your User or Full Name, and then click Update. 11

22 Change Account Password Managing User Accounts Change Account Password Users can change their own password in the VMR Dashboard. Step 1 Step 2 Sign in to the VMR Dashboard using your credentials. In the navigation pane, expand the Account tab and click Change Password. Figure 9: Account > Change Password Step 3 Enter the new password in both the New Password and the Confirm fields, and then click Submit. Note The password must contain 6-20 characters, at least one (a-z) and one (A-Z). Delete User Account If you are a Root or Admin user, you can delete user accounts. Perform the following steps to delete an account: Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Sign in to the VMR Dashboard using your Root or Admin credentials. In the navigation pane, expand the Account tab and select Search User. Enter the search criteria to locate the user account and click Search. In the Search Result table, click the Delete icon in the Action column to delete the account. On the Delete Account confirmation dialog, click OK. Confirm that the deleted user account is removed from the Search Result table. 12

23 CHAPTER 3 Managing Recordings This chapter provides instructions for managing recordings in VMR. It includes the following topics: Create Recording, page 14 Create Multiple Recordings, page 15 Search Recordings, page 16 View Recording Details, page 18 Update Recording, page 22 Play Recording, page 23 Audit Recordings, page 24 Delete Recording, page 25 Download Recording Time Segments, page 26 13

24 Create Recording Managing Recordings Create Recording Perform the following steps to create a recording: Step 1 In the navigation pane, select Recordings > Create Recording. Figure 10: Create Recording Step 2 In the Create Recording pane, enter or select the appropriate parameters for the recording: Field Description Recording ID* Enter a unique ID for the recording. Copy Type Num Copies Account ID* Stream ID* Select Unique or Common. Enter the number of recordings to create. Enter the Account ID set by A8. Select a valid Stream ID for an already provisioned stream. 14

25 Managing Recordings Create Multiple Recordings Field Start Time (UTC)* Duration* Ad Zone* Station ID* Update URL* Listing ID* Description Click in the fields and select the date and time the recording should start from the drop-down list. If you want the recording to start immediately, check the Start immediately check box. Note The start immediately option will use the user's system time. To ensure there is minimum delay, verify that the Network Time Protocol (NTP) server time is in sync with the UTC time in the user's system. Select the check box to indicate the duration for the recording (how long). If you want to specify the End Time, check the Custom check box and specify the date and time the recording ends. Identifies the current Ad Zone associated with the owner of this recording. Identifies the station associated with the broadcast stream. URL used to send update information. Identifies a specific location within the program listing for the program most associated with this recording. Step 3 Click Create to save the recording. A confirmation message is displayed indicating that the recording was successfully created. Create Multiple Recordings Perform the following steps to create multiple recordings: Step 1 Step 2 Perform the steps in Create Recording. In the Num Copies field, set the value to the number of recordings desired (greater than 1). This creates multiple recordings and assigns a unique Recording ID number to each recording, as follows: For the first recording, Recording ID is the value entered. For all other recordings, Recording ID is the value followed by a hyphen and a copy number (-1, -2, and so on). For example, if you specify Recording ID = 100 and Num Copies = 3: The first recording has Recording ID = 100. The second recording (first copy) has Recording ID =

26 Search Recordings Managing Recordings The third recording (second copy) has Recording ID = Step 3 Click Create to save the recordings. A confirmation message is displayed indicating that the recordings were successfully created. Search Recordings Perform the following steps to search for a recording: Step 1 In the navigation pane, select Recordings > Search Recording. Figure 11: Search Recordings Step 2 Use one of the following methods to enter the search criteria: Enter the Recording ID or multiple recording IDs separated by commas (no spaces). You can also use the * wildcard to search recording IDs. Specify any of the following search criteria: Stream ID Enter external Stream ID. 16

27 Managing Recordings Search Recordings Account ID Enter Account ID. Min Start Enter minimum range value of start date and time for recordings to be searched. Max Start Enter maximum range value of start date and time for recordings to be searched. Status Check the check box for the status of recordings to be searched (All, Started, Pending, Complete, Incomplete, Failed, Started With Error, or Erased). Step 3 Click Search. Figure 12: Recordings Search Result Step 4 Review the results in the Recordings Search Result table in the lower pane. 17

28 View Recording Details Managing Recordings View Recording Details Perform the following steps to view detailed information about a recording: Step 1 Step 2 Locate the recording using one of the search options described in Search Recordings. In the Recordings Search Result table, click the link to the recording in the Recording ID column to open the Detail of Recording dialog. Figure 13: Detail of Recording Step 3 When you have completed viewing the recording details, click Close to exit the dialog. VMR Recording Status The recording Status value is computed on-the-fly for each recording based on three fields stored in the Recordings table in MemSQL (the actual status values are not stored): Actual Start Time The Actual Start Time set by the Manifest Agent. This is the start time of the first segment captured time. 18

29 Managing Recordings VMR Recording Status Actual End Time The Actual Stop Time set by the Manifest Agent. This is the stop time when the Manifest Agent stops the recording. Error Code If non-zero, indicates an error. Possible error codes are explained in Table 4-3. Refer to the Recording Status Value Description table for a list of possible combinations of the values and the description of each combination. Table 1: Recording Status Value Description Recording Status Actual Start Time Actual End Time Error Code Erased Time Segment Count Description COMPLETE Not 0 Not 0 = 0 = 0 1+ Recording completed successfully. The recording has ended and the requested timeframe was captured entirely with success. STARTED Not 0 = 0 NA = 0 1+ Recording was started. The recorder has begun capturing content. STARTEDWITH ERROR Not 0 = 0 NA = 0 1+ Recording was started. The recorder has begun capturing content. INCOMPLETE Not 0 Not 0 Not 0 = 0 1+ Recording is incomplete. Only partial content was captured for the recording. Segments are not contiguous over the timeline of the recording from requested start to requested end time. FAILED = 0 Not 0 NA = 0 NA The recorder has failed and unable to service the request. No content was captured for the recording. 19

30 VMR Recording Status Managing Recordings Recording Status Actual Start Time Actual End Time Error Code Erased Time Segment Count Description PENDING = 0 = 0 = 0 = 0 NA Recording is pending. The recording information is present and scheduled to be recorded at a future time. ERASED NA NA NA Set NA Recording has been erased. The recording is no longer available and attempts to reuse the recording Id shall fail. UNKNOWN NA NA NA NA 0 No information is known for this recording Id. Note The Erased Time field is set only if the recording has been marked for deletion. NA refers to Not Applicable; this field can have any value. It does not decide the corresponding status of the Recording. Zero (= 0) time is stored as :00: in the memsql database and displayed as T00:00:00Z in the VMR UI. Possible Causes for Recordings in INCOMPLETE State Lost segments during recordings in progress may lead to the Recording status being marked as INCOMPLETE. Segments may be missing due to: Recording started late (after the StartTime); this is counted as missing segments. This error will not manifest clearly in the system or Memsql tables. MCE time may be leading the VMR time in this case. Segment batches failed; check the Segments table for OverallStatus = 2 for the time span of the recording. These will reveal failed batches and missing segments for the recording. If VMR detects lost segments, which can occur due to a failure in the recording process or during the brief downtime between a node going down and the High Availability (HA) backup node coming online. If the MCE time if out of sync with the VMR time. 20

31 Managing Recordings VMR Recording Status Possible Causes for Recordings in PENDING State If the MCE time is out of sync with the NTP time. If the publishtime in the MPD for the stream is out of sync with the NTP time. Refer to the Possible Error Codes table for a list of error codes and descriptions. Note An Error Code of 0 means it is not set. The actual value of the ErrorCode (for example: 1,2,4 or 5) does not matter for the determination of Recording Status. The logic would only check if the ErrorCode is set or 0. Table 2: Possible Error Codes Error Code Meaning RecErrMissingSegment RecErrUnrecoverable RecErrStartedWithErr RecErrMissingSegment and RecErrStartedWithErr Comments Recording is missing a segment. Archive Agent found that this recording has an unrecoverable error; manual intervention is required. Manifest Agent was unable to start the recording on time. Recording is missing a segment and manifest agent was unable to start the recording on time. 21

32 Update Recording Managing Recordings Update Recording Perform the following procedure to update a recording: Step 1 Step 2 Step 3 Locate the recording using one of the search options described in Search Recordings. View the search results in the Recordings Search Result table in the lower pane. Click the Update Recording icon in the Action column to enable editing the recording. Figure 14: Update Recording Step 4 Note Depending on the current recording status, one or more fields in this dialog may be disabled (dimmed) and cannot be updated. Update the field(s) as needed and then click Save Changes (see Create Recording for field descriptions). 22

33 Managing Recordings Play Recording Play Recording Perform the following procedure to play a recording: Step 1 Step 2 Step 3 Locate the recording using one of the search options described in Search Recordings. View the search results in the Recordings Search Result table in the lower pane. Click the Play icon for the recording in the Actions column. Figure 15: Play Recording Step 4 Step 5 Select the Source Type to display the URL of the source in the Video URL field. To play the recording, click the Play icon in the Video URL field. A playback pane opens and the recording plays in the pane. Figure 16: Recording Playback You can use the control panel along the bottom of the playback window to pause and restart the playback, adjust the volume, enable or disable Closed Captioning, and enable or disable full screen playback. 23

34 Audit Recordings Managing Recordings Audit Recordings Start Audit The VMR Dashboard supports running audits on recordings retrieved as search results. The following related options are available in the Audit column of the Recordings Search Result table. To start auditing a recording, click the Start icon in the Audit column of the Recordings Search Result table. Note If the audit is already in progress, an error message appears at the top of the pane. If this occurs, you can either wait for the current audit run to complete, or stop the audit in progress, as described in Stop Audit in Progress, and restart it later. Stop Audit in Progress To stop an in-progress audit of a recording, click the Stop icon in the Audit column of the Recordings Search Result table. Stopping an audit sends an interrupt signal to the audit tool. Note If an audit is stopped, partial audit results are generated up to the point that the audit was interrupted. View Audit Results To view a summary of audit results and any failed recordings, click the Results icon in the Audit column of the Recordings Search Result table. A summary of the completed audit results appears, including indication of any failed recordings and an option to view additional failure details. Note Audit results are only available for viewing when the current audit run is complete. Stop All Audits in Progress To stop all in-progress audits for recordings, click the Stop All icon located above the Audit column in the Recordings Search Result table. 24

35 Managing Recordings Clear Audit Reports Note Clicking this icon sends a signal to all audits currently running to terminate them. Partial results for such terminated audits are generated up to the point of termination. Clear Audit Reports To clear all results for previous and current audit runs, click the Clear All icon located above the Audit column in the Recordings Search Result table. Note Audit results are stored as files on the API container. Clicking this icon removes all of these files. Configure Additional Audit Parameters In addition to the audit results, the following parameters related to audits are configured on the VMR Dashboard API server: The maximum size of the audit result directory that resides on the API container is 100 MB. The buffer size of the audit result directory is 5 MB. The required size of the audit result directory needed to start running an audit is less than 95 (100-5) MB. The time period after which recording audit result files are cleaned up is 3 days from completion of the corresponding audit run. Delete Recording Perform the following steps to delete a recording: Step 1 Step 2 Step 3 Step 4 Locate the recording using one of the search options described in Search Recordings. View the search results in the Recordings Search Result table in the lower pane. Click the Delete Recording icon for the recording in the Action column to mark the recording for deletion. On the Delete Recording confirmation dialog, click OK to delete the recording. Note When a recording is deleted, its Erased Time is set to the current UTC time. The XRID of the deleted recording cannot be reused for a Create Recording request until the Recorder Manager cleans up the recording (the original and any copies) completely. 25

36 Download Recording Time Segments Managing Recordings Download Recording Time Segments Perform the following steps to download recording time segments: Step 1 Step 2 Locate the recording using one of the search options described in Search Recordings. In the Recordings Search Result pane, click the Download TS icon. Figure 17: DownloadTS Step 3 On the DownloadTS pane, check the check box next to the Time Segments to be downloaded and then click Download. The Time Segments are downloaded and saved in the browser Downloads directory. 26

37 CHAPTER 4 Managing Streams This chapter provides instructions for managing streams in VMR. It includes the following topics: Create Stream, page 28 Search Streams, page 30 View Stream Details, page 32 Configure Imported Streams, page 32 Update Stream, page 35 Delete Stream, page 36 27

38 Create Stream Managing Streams Create Stream Perform the following steps to create a recording: Step 1 In the navigation pane, select Streams > Create Stream. Figure 18: Create Stream Step 2 In the Create Stream pane, enter or select the appropriate parameters for the stream: Field Description Stream ID* Enter an integer value for the unique stream ID. Stream Name URL* Enter a string value for a stream name to associate with Stream ID. Enter the URL for the DASH manifest. 28

39 Managing Streams Create Stream Field No DeDupe Expiry* Archival Start Time ArchiveTime (sec) RearchiveTime (sec) Archival Duration (sec) ArchivalPause (msec) AvgBitrate (MB) MaxBitrate (MB) Description Check the check box if do not want to archive segments from the stream. If the check box is selected, the segments are not de-duped, but the metadata will be archived from that stream. If the check box is cleared, the segments are de-duped. Note This option can only be specified at the time the channel is created. A channel that has No De-Dupe enabled, cannot be modified at a later time. Select the date and time in UTC format that the stream will expire. Select the time the archival process starts in [hh:mm][am/pm] format. Example: 05:00 PM Enter the elapsed time, in seconds, after which the segment belonging to this stream can be archived from active storage. Enter the elapsed time, in seconds, after which the reconstituted segment belonging to this stream can be re-archived. Enter the elapsed time, in seconds, the Archive Agent remains active from the Archival Start Time. Enter the elapsed time, in milliseconds, for the Archive Agent to wait before picking up another archive task. Enter the average bit rate of the transport stream in megabytes. Enter the maximum bit rate of the transport stream in megabytes. Step 3 Click Create to save the stream. A confirmation message is displayed indicating that the stream was successfully created. 29

40 Search Streams Managing Streams Search Streams Perform the following steps to search for a recording: Step 1 In the navigation pane, select Streams > Search Stream. Figure 19: Search Streams Step 2 Use one of the following methods to enter the search criteria: Enter the Stream ID or multiple stream IDs separated by commas (no spaces). You can also use the * wildcard to search stream IDs. Enter the Stream Name or part of the name. To search for multiple stream names, separated by commas (no spaces). Specify any of the following search criteria: Expiration Start The start of the range of Stream Expiration time values to be searched. Expiration End The end of the range of Stream Expiration time values to be searched. Stream Status The current status of the stream (Active, Expired, Enable, Disabled) 30

41 Managing Streams Search Streams Step 3 Click Search. Figure 20: Streams Search Result Step 4 Review the results in the Streams Search Result table in the lower pane. 31

42 View Stream Details Managing Streams View Stream Details Perform the following steps to view detailed information about a stream: Step 1 Step 2 Locate the stream using one of the search options described in Search Streams. In the Streams Search Result table, click the link to the stream in the Stream ID column to open the Detail of Stream dialog. Figure 21: Detail of Stream Step 3 When you have completed viewing the stream details, click Close to exit the dialog. Configure Imported Streams The VMR Dashboard provides the option to import stream information from a file, compare the imported streams with currently provisioned streams, and if appropriate, configure the imported streams on the N8 interface. 32

43 Managing Streams Configure Imported Streams Note Imported stream files can be in either comma separated values (CSV) or JSON format. For examples, see Sample File Formats. Step 1 In the navigation pane, select Streams > Manage Stream. Figure 22: Import Stream File Step 2 Step 3 Step 4 On the Import Stream File pane, click Choose File and navigate to the stream file to be imported. The selected stream file appears in the field. Click Compare to import the file and compare it to the currently provisioned streams. On the confirmation dialog, click OK to proceed with the import and comparison against the database. The Compare Streams Result table displays a side-by-side comparison of the streams in the imported stream file with the streams that are currently provisioned. Figure 23: Compare Stream Result Note If the imported stream file contains duplicate stream IDs, the comparison fails with an error message that includes the duplicate stream IDs. 33

44 Sample File Formats Managing Streams The Imported Streams are listed in the left pane and the currently Configured Streams are listed in the right pane. The comparison process automatically pairs up streams by Stream ID, where possible. The rows in the table are color-coded to show the status of each stream comparison, as follows: White The two streams are provisioned with the same Stream ID and field values. Yellow The two streams are provisioned with the same Stream ID but different field values. Green The imported stream has no corresponding currently provisioned stream. Red The currently provisioned stream has no corresponding imported stream. Step 5 Note Click the Help icon to the right of the Compare Streams Result heading to display a color legend for the table. If indicated by the results of the comparison, click Configure to configure the streams from the imported file to the database. The imported streams are pushed to the N8 interface for provisioning. In addition, any currently provisioned streams that have no corresponding imported stream, based on a comparison of Stream IDs, are set to Expired state. Sample File Formats Click the Help icon to the right of the Import Stream File (CSV/JSON) heading to view examples of CSV and JSON files that can be imported. The following templates show the JSON and CSV file formats that are suitable for import to VMR. JSON File Template "{" "StreamName": "ABC", "StreamID":"10001", "Expiry" :" T05:41: Z", "Transports" :[ { "URL":" "AvgBitRate":1.2, "MaxBitRate":Z.3 } ], "ArchiveConfig":{ "ArchiveTime":10, "ReArchiveTime":S, "ArchivalStartTime":"12:00AM", "ArchivalDuration":12, "ArchivalPause" :4 } } CSV File Template "StreamID","StreamName","URL","Expiry","ArchiveTime","RearchiveTime","Archiv "10001","ABC"," "10002","FOX"," "10003","ESPN","http ://test3.com"," t14:30:00.000z",5,4,"2:45pm",3, "10112","CW","http ://test10112.com"," t14:30:00.000z",5,4,"12:00am" 34

45 Managing Streams Update Stream Update Stream Perform the following procedure to update a stream: Step 1 Step 2 Step 3 Locate the stream using one of the search options described in Search Streams. View the search results in the Streams Search Result table in the lower pane. Click the Update Stream icon in the Action column to enable editing the stream. Figure 24: Update Stream Step 4 Note Depending on the current streaming status, one or more fields in this dialog may be disabled (dimmed) and cannot be updated. Update the field(s) as needed and then click Save Changes (see Create Stream for field descriptions). 35

46 Delete Stream Managing Streams Delete Stream Perform the following steps to delete a stream: Step 1 Step 2 Step 3 Step 4 Locate the stream using one of the search options described in Search Streams. View the search results in the Streams Search Result table in the lower pane. Click the Delete Stream icon for the stream in the Action column to mark the stream for deletion. On the Delete Stream confirmation dialog, click OK to delete the stream. Note When a Stream is deleted, its Expiry Time is set to the current UTC time. The stream is then deleted during the next periodic cleanup. 36

47 CHAPTER 5 Monitoring VMR Health This chapter provides instructions for monitoring the health of the VMR system. It includes the following topics: Introduction, page 37 Monitoring VMR Endpoints, page 37 Configure Endpoint Monitoring, page 38 View Alarms and Events, page 39 Introduction The Health Agent is deployed in VMR as a single Kubernetes pod (ReplicationController). The health-agent pod is comprised of two containers that work together to support the VMR alarm infrastructure: the sensu-client container and the health-agent container. The health-agent service exposes the health-agent container, which facilitates the communication gateway for other VMR components to send status information to the health agent to determine the current health. The alarm manager is responsible for determining the current health of a VMR component. Each registered component is responsible for pushing a snapshot of its current state data to the health agent for review. The known stream list is constantly updated via the rio-api. As each snapshot of state data is received, it is added to an internal cache, allowing for the evaluation of this data over a period of time (default 60 minutes). As a component registers with the health agent, its respective component processor is added to the health agent, which provides the necessary context to evaluate the component's snapshot data for applicable alarm criteria. The processor produces set/clear alarms based solely on the current context of the snapshot data. This alarm data is then returned to the alarm manager for the purpose of reviewing the current alarm state against its cache, and if necessary, record and forward the alarm data to the accompanying sensu-client container. Monitoring VMR Endpoints The VMR Health Agent can monitor the health of VMR endpoints, which include MemSQL, Cleversafe, or COS nodes. The following checks are available: 37

48 Configure Endpoint Monitoring Monitoring VMR Health Individual node health check This event is raised when a node transitions from healthy to unhealthy; and is cleared when the node returns back to healthy state. The event name is check_<node name> where the <node name> is the IP address of the node. Aggregate endpoint health status This event is raised whenever there are any node health transitions. It reports the total nodes healthy as a percentage of all active nodes. When an event is cleared, it will no longer show up in the Uuchiwa (Sensu) GUI Event tab; it will remain in the Client History tab with the event status showing that it is cleared. The event name is check_<endpoint type>_health where the <endpoint type> is either MemSQL or active-storage. Critical endpoint threshold This event is raised as a critical event when the number of unhealthy nodes as a percentage of total active nodes is greater then a configured threshold value (default is 25 percent). This event is cleared when the unhealthy percentages falls below the threshold value. The event name is check_<endpoint type>_threshold where the <endpoint type> is either MemSQL or active-storage. Configure Endpoint Monitoring Perform the following steps to configure the Health Agent to monitor endpoints: Step 1 Step 2 Step 3 Step 4 On Openshift platforms, edit the policy to allow editing: oc login -u system -p admin --insecure-skip-tls-verify=true -n default [openshift-master-ip]:8443 oc project vmroc config use-context [current-context] --namepsace=vmr oc policy add-role-to-group edit system:serviceaccounts -n vmr Note [current-context] can be obtained from the "oc config view" command output. To monitor COS or CleverSafe endpoints, verify that the CS_PORT_LIST value is 80 in the health-agent-rc.yaml file; by default, it will only monitor port 80. In the cisco-vmr.cfg file, specify the active path, archive path, and COS user name and password so the following parameters can be populated with the correct values in the health-agent-rc.yaml file: ACTIVE_PATH = "rio/activepath" ARCHIVE_PATH = "rio/archivepath" BASIC_AUTH = "basicauth= You have the option to configure the Health Agent to only monitor the endpoints, or to monitor and update the endpoints. Modify the following parameters in the health-agent-rc.yaml file to indicate how to monitor the endpoints. HA_DB_EP_MON_ONLY This parameter controls the behavior for monitoring the database (MemSQL). When set to true, the endpoints are only monitored. When set to false, the endpoints are monitored and updated. 38

49 Monitoring VMR Health View Alarms and Events HA_STORAGE_EP_MON_ONLY This parameter controls the behavior for monitoring the storage (for example, COS). When set to true, the endpoints are only monitored. When set to false, the endpoints are monitored and updated. Step 5 Step 6 Step 7 Verify that the health agent is reporting alarms to the Sensu server. In the health-agent-rc.yaml file: HA_ALARM_REPORTER value: sensu Confirm that it is not set to logger, which will only log to ELK. To activate Monitoring COS or CleverSafe endpoints, verify that the Monitoring option is enabled (Do not disable Storage monitoring): HA_DISABLE_STOR_ACTIVE_MON value: false To activate Monitoring Memsql endpoints, verify that the Monitoring option is enabled (Do not disable database monitoring): HA_DISABLE_DB_ACTIVE_MON value: false View Alarms and Events You can view the VMR alarms and events in the Sensu Uchiwa GUI or via the command line interface. View Alarms in Sensu Uchiwa GUI Perform the following steps to view the alarms and event in the Sensu Uchiwa GUI: Step 1 Step 2 Log in to the Uchiwa GUI of the Sensu server that is receiving alarms from VMR health agent. View the information about the alarms, including the time it occurred, the current status, type, severity, node that generated the alarm, and the details. Figure 25: Viewing VMR Alarms in Sensu Uchiwa GUI 39

50 View Alarms in Command Line Interface Monitoring VMR Health Refer to the tables in Alarm Definitions for information describing the alarms. View Alarms in Command Line Interface Perform the following steps to view the alarms via a command line interface: Step 1 Step 2 Open a command line window. Enter the following command: kubectl logs health-agent-<xyz> health-agent Alarm Definitions The following tables provide information about the alarm categories that may be generated and how to troubleshoot the alarms. Table 3: Manifest Agent Alarms Name Raised Cleared Troubleshoot stream_no_ma_check No manifest for stream Pre-conditions: Stream list returned from rio-api dictates the expected streams (updated every minute). No snapshot data is received for a given stream within the list. Snapshot data is received, then halts for a duration > 1 min; raised if the stream still found in the expected list. Snapshot data begins to flow from the manifest agent to the health agent. Once consistent snapshot data is resumed at normal intervals, the alarm is cleared. Attempt to query and pull the manifest.mpd from the MCE directly via curl or using a web browser. Check the status of the linear packager (MCE) configured to confirm its status is online. Ensure network connectivity exists between the VMR worker nodes and MCE worker nodes. 40

51 Monitoring VMR Health Alarm Definitions Name stream_ma_stale_check Manifest for stream is stale Raised Current number of successful manifest polls/queries is equivalent to the cached number of manifest polls. Cleared Number of successful manifest polls/queries differs from the number of cached manifest polls. Troubleshoot Attempt to query and pull the manifest.mpd from the MCE directly via curl or using a web browser. Check the status of the linear packager (MCE) configured to confirm its status is online. Ensure network connectivity exists between the VMR worker nodes and MCE worker nodes. recording_failure_check Recording failure Percentage of active recordings versus active recordings with errors for a given stream falls below 50 percent. When the percentage of active recordings versus recordings with errors for a given stream increases above 50 percent. Look for the presence of accompanying manifest alarms such as the 'no stream ma check', or 'stream ma stale check' to determine whether the state of the stream and related manifest are affected. Examine the VMR Dashboard to investigate the current status of the recordings and check for error states. Investigate the status of the a8 container logs for clues as to the status of the recording or why it may have failed. ma_live_point_drift live point drift The duration of the live point drift, computed by the manifest agent has met or exceeded the pre-defined live drift warning period. (Default period 60 seconds). Drift is defined by the delta between the publish time and live point in the manifest. Once the live point drift duration reduces to a value less than the pre-determined warning threshold. (i.e. drift delta < 60 seconds). Generally, a live point drift is characterized by an upstream issue with either the packager or encoder/transcoder with respect to time. Check to ensure that all resources are time synchronized with the same NTP server. 41

52 Alarm Definitions Monitoring VMR Health Name ma_lp_drift_exceeded live point drift delta exceeded Raised The duration of the live point drift, computed by the manifest agent has exceeded the pre-defined max live drift threshold. (Default period 12 minutes). Drift is defined by the delta between the publish time and live point in the manifest. Cleared Once the live point drift duration reduces to a value less than the pre-defined max threshold. (i.e. drift delta < 12 minutes). Troubleshoot Generally, a live point drift is characterized by an upstream issue with either the packager or encoder/transcoder with respect to time. Check to ensure that all resources are time synchronized with the same NTP server. Note For channel specific events (stream_no_ma_check, stream_ma_stale_check, ma_live_point_drift, ma_lp_drift_exceeded and Manifest CB) in VMR 1.3.3, there is a known issue where all the channel alarms use the same event name with a different message body that includes the channel name. This results in only the latest event for each event type to display in the Uchiwa (Sensu) GUI although multiple events may have failed. To workaround the issue, you should check the ELK log to resolve channel specific events. Table 4: Manifest Agent Circuit Breaker Alarms Name BatchWorkCB Raised When the number of dropped CMPDs being processed by the batch worker over a window of time meets or exceeds the threshold defined by the expected segment duration x the configured multiplier. Cleared When the number of dropped CMPDs processed by the batch worker has been reset, normally expected to correlate with a container restart. Troubleshoot CookedManifestCB DatabaseCB ManifestCB Percentage of Cooked MPD dropped versus configured threshold. Percentage of failed database access versus. configured threshold. Percentage of failed MPD polling versus configured threshold. Container restart Container restart Container restart 42

53 Monitoring VMR Health MemSQL Checks Name Raised Cleared Troubleshoot SegmentRecorderCB Percentage of failed read/write access to Segment Recorder versus threshold. Container restart Table 5: Endpoint Alarms Name Raised Cleared Troubleshoot check_memsql_health When a node transitions from healthy state to unhealthy state, or from unhealthy state back to healthy state. Not cleared check_active-storage_ health When a node transitions from healthy state to unhealthy state, or from unhealthy state back to healthy state. Not cleared check_nodeip_port When a node fails to respond to status check or has unhealthy state. When the node returns the healthy status Make sure HA_DISABLE_STOR_ ACTIVE_MON is set to false. check_active-storage_ threshold check_memsql_threshold Percentage of unhealthy storage nodes exceeding threshold. Percentage of unhealthy MemSQL nodes exceeding threshold. When the percentage drops below threshold. When the percentage drops below threshold. MemSQL Checks Each MemSQL node includes a monitoring service for reporting the health of the MemSQL cluster on an external Sensus server. The monitoring includes base node checks and specific MemSQL checks. You view these checks using the Sensu Uchiwa user interface. The following Sensu checks are available for monitoring the health of the MemSQL cluster. 43

54 MemSQL Checks Monitoring VMR Health Check Name Description Warning Critical check_node_cpu_usage Runs "iostat -c" 70% user memory 90% user memory 40% system memory 60% system memory 30% iowait memory 40% iowait memory check_cluster_state Runs "show cluster status" and count words online, offline, master, slave and orphan ( offline or ophan If number of master and slave partitions check_core_file_exists Runs find /var/lib/memsql -name *core N/A Any file found check_table_mem Runs "SHOW STATUS EXTENDED LIKE 'Alloc_table_memory'" for current table memory 75% 90% Runs "SHOW VARIABLES LIKE 'maximum_table_memory'" for defined table maximum memory check_long_running_queries Runs "show processlist" and looks for process time greater than 600 seconds N/A If any processes greater than 600 seconds check_node_mem_usage Runs 'free -m' 85% 95% check_memsql_node_up Runs "select 1" query N/A If value is not 1 check_node_disk_usage Runs 'df -h' and gathers usage for first partition only 85% 95% check_memsql_logs Reads the following files, and looks for FAILURE, WARN and ERROR Any warning Any failure or error /var/lib/memsql/master- 3306/tracelogs/memsql.log /var/lib/memsql/child- 3306/tracelogs/memsql.log /var/lib/memsql/installs/leaf- 3307/tracelogs/memsql.log /var/lib/memsql/installs/leaf- 3308/tracelogs/memsql.log check_leaf_state Runs "show leaves and count online and offline words N/A Any offline 44

55 Monitoring VMR Health MemSQL Checks Check Name Description Warning Critical check_license Runs memsql-ops license-list -j and looks at expiration date 30 days from now 15 days from now check_aggregator_state Runs "show aggregators" and counts words offline and online N/A Any offline check_server_mem Runs "SHOW STATUS EXTENDED LIKE 'Total_server_memory'" for current server memory 75% 90% Runs "SHOW VARIABLES LIKE 'maximum_memory'" for defined server max memory 45

56 MemSQL Checks Monitoring VMR Health 46

57 CHAPTER 6 Troubleshooting This chapter provides information for resolving issues when using Virtual Media Recorder (VMR). It includes the following topics: Scheduled Recording Fails, page 47 Issues with No De-Dup Value, page 47 Scheduled Recording Fails An error may occur when attempting to play a scheduled recording. This may occur when the feed from the Media Capture Engine (MCE) is corrupt and the manifest file cannot be obtained. The manifest agent logs for the stream that the recording is using displays the following error: <failed on getting manifest returned http status 404cc> In addition, MEMSQL will showthat the recording is in error state (Error code set to 1) and that the recording start time is not set (only end time is set). The XRID= xrid-1. Perform the following to troubleshoot the issue: Use the curl command to obtain the mpd file for the stream that the recording is scheduled to run; must return 200 OK. Repair the feed from MCE and verify that the mpd file can be obtained. Issues with No De-Dup Value The following issues may occur with the No De-Dup feature. Note The No De-Dup field value in V2PC maps to the DeDupe field in the database. 47