Avaya IQ High Availability and Survivability

Transcription

1 Avaya IQ High Availability and Survivability Release 5.2 April 2015

2 2015 Avaya Inc. All Rights Reserved. Notice While reasonable efforts have been made to ensure that the information in this document is complete and accurate at the time of printing, Avaya assumes no liability for any errors. Avaya reserves the right to make changes and corrections to the information in this document without the obligation to notify any person or organization of such changes. Documentation disclaimer Documentation means information published by Avaya in varying mediums which may include product information, operating instructions and performance specifications that Avaya may generally make available to users of its products and Hosted Services. Documentation does not include marketing materials. Avaya shall not be responsible for any modifications, additions, or deletions to the original published version of documentation unless such modifications, additions, or deletions were performed by Avaya. End User agrees to indemnify and hold harmless Avaya, Avaya's agents, servants and employees against all claims, lawsuits, demands and judgments arising out of, or in connection with, subsequent modifications, additions or deletions to this documentation, to the extent made by End User. Link disclaimer Avaya is not responsible for the contents or reliability of any linked websites referenced within this site or documentation provided by Avaya. Avaya is not responsible for the accuracy of any information, statement or content provided on these sites and does not necessarily endorse the products, services, or information described or offered within them. Avaya does not guarantee that these links will work all the time and has no control over the availability of the linked pages. Warranty Avaya provides a limited warranty on Avaya hardware and software. Refer to your sales agreement to establish the terms of the limited warranty. In addition, Avaya s standard warranty language, as well as information regarding support for this product while under warranty is available to Avaya customers and other parties through the Avaya Support website: or such successor site as designated by Avaya. Please note that if You acquired the product(s) from an authorized Avaya Channel Partner outside of the United States and Canada, the warranty is provided to You by said Avaya Channel Partner and not by Avaya. Licenses THE SOFTWARE LICENSE TERMS AVAILABLE ON THE AVAYA WEBSITE, OR SUCH SUCCESSOR SITE AS DESIGNATED BY AVAYA, ARE APPLICABLE TO ANYONE WHO DOWNLOADS, USES AND/OR INSTALLS AVAYA SOFTWARE, PURCHASED FROM AVAYA INC., ANY AVAYA AFFILIATE, OR AN AVAYA CHANNEL PARTNER (AS APPLICABLE) UNDER A COMMERCIAL AGREEMENT WITH AVAYA OR AN AVAYA CHANNEL PARTNER. UNLESS OTHERWISE AGREED TO BY AVAYA IN WRITING, AVAYA DOES NOT EXTEND THIS LICENSE IF THE SOFTWARE WAS OBTAINED FROM ANYONE OTHER THAN AVAYA, AN AVAYA AFFILIATE OR AN AVAYA CHANNEL PARTNER; AVAYA RESERVES THE RIGHT TO TAKE LEGAL ACTION AGAINST YOU AND ANYONE ELSE USING OR SELLING THE SOFTWARE WITHOUT A LICENSE. BY INSTALLING, DOWNLOADING OR USING THE SOFTWARE, OR AUTHORIZING OTHERS TO DO SO, YOU, ON BEHALF OF YOURSELF AND THE ENTITY FOR WHOM YOU ARE INSTALLING, DOWNLOADING OR USING THE SOFTWARE (HEREINAFTER REFERRED TO INTERCHANGEABLY AS YOU AND END USER ), AGREE TO THESE TERMS AND CONDITIONS AND CREATE A BINDING CONTRACT BETWEEN YOU AND AVAYA INC. OR THE APPLICABLE AVAYA AFFILIATE ( AVAYA ). Avaya grants You a license within the scope of the license types described below, with the exception of Heritage Nortel Software, for which the scope of the license is detailed below. Where the order documentation does not expressly identify a license type, the applicable license will be a Designated System License. The applicable number of licenses and units of capacity for which the license is granted will be one (1), unless a different number of licenses or units of capacity is specified in the documentation or other materials available to You. Software means computer programs in object code, provided by Avaya or an Avaya Channel Partner, whether as stand-alone products, pre-installed on hardware products, and any upgrades, updates, patches, bug fixes, or modified versions thereto. Designated Processor means a single stand-alone computing device. Server means a Designated Processor that hosts a software application to be accessed by multiple users. Instance means a single copy of the Software executing at a particular time: (i) on one physical machine; or (ii) on one deployed software virtual machine ( VM ) or similar deployment. License types Designated System(s) License (DS). End User may install and use each copy or an Instance of the Software only on a number of Designated Processors up to the number indicated in the order. Avaya may require the Designated Processor(s) to be identified in the order by type, serial number, feature key, Instance, location or other specific designation, or to be provided by End User to Avaya through electronic means established by Avaya specifically for this purpose. Concurrent User License (CU). End User may install and use the Software on multiple Designated Processors or one or more Servers, so long as only the licensed number of Units are accessing and using the Software at any given time. A Unit means the unit on which Avaya, at its sole discretion, bases the pricing of its licenses and can be, without limitation, an agent, port or user, an or voice mail account in the name of a person or corporate function (e.g., webmaster or helpdesk), or a directory entry in the administrative database utilized by the Software that permits one user to interface with the Software. Units may be linked to a specific, identified Server or an Instance of the Software. Database License (DL). End User may install and use each copy or an Instance of the Software on one Server or on multiple Servers provided that each of the Servers on which the Software is installed communicates with no more than one Instance of the same database. CPU License (CP). End User may install and use each copy or Instance of the Software on a number of Servers up to the number indicated in the order provided that the performance capacity of the Server(s) does not exceed the performance capacity specified for the Software. End User may not re-install or operate the Software on Server(s) with a larger performance capacity without Avaya s prior consent and payment of an upgrade fee. Named User License (NU). You may: (i) install and use the Software on a single Designated Processor or Server per authorized Named User (defined below); or (ii) install and use the Software on a Server so long as only authorized Named Users access and use the Software. Named User, means a user or device that has been expressly authorized by Avaya to access and use the Software. At Avaya s sole discretion, a Named User may be, without limitation, designated by name, corporate function (e.g., webmaster or helpdesk), an or voice mail account in the name of a person or corporate function, or a directory entry in the administrative database utilized by the Software that permits one user to interface with the Software. Shrinkwrap License (SR). You may install and use the Software in accordance with the terms and conditions of the applicable license agreements, such as shrinkwrap or clickthrough license accompanying or applicable to the Software ( Shrinkwrap License ). Copyright Except where expressly stated otherwise, no use should be made of materials on this site, the Documentation, Software, Hosted Service, or hardware provided by Avaya. All content on this site, the documentation, Hosted Service, and the product provided by Avaya including the selection, arrangement and design of the content is owned either by Avaya or its licensors and is protected by copyright and other intellectual property laws including the sui generis rights relating to the protection of databases. You may not modify, copy, reproduce, republish, upload, post, transmit or distribute in any way any content, in whole or in part, including any code and software unless expressly authorized by Avaya. Unauthorized reproduction, transmission, dissemination, storage, and or use without the express

3 written consent of Avaya can be a criminal, as well as a civil offense under the applicable law. Third Party Components Third Party Components mean certain software programs or portions thereof included in the Software or Hosted Service may contain software (including open source software) distributed under third party agreements ( Third Party Components ), which contain terms regarding the rights to use certain portions of the Software ( Third Party Terms ). As required, information regarding distributed Linux OS source code (for those products that have distributed Linux OS source code) and identifying the copyright holders of the Third Party Components and the Third Party Terms that apply is available in the products, Documentation or on Avaya s website at: support.avaya.com/copyright or such successor site as designated by Avaya. You agree to the Third Party Terms for any such Third Party Components. Note to Service Provider The product or Hosted Service may use Third Party Components subject to Third Party Terms that do not allow hosting and require a Service Provider to be independently licensed for such purpose. It is your responsibility to obtain such licensing. Preventing Toll Fraud Toll Fraud is the unauthorized use of your telecommunications system by an unauthorized party (for example, a person who is not a corporate employee, agent, subcontractor, or is not working on your company's behalf). Be aware that there can be a risk of Toll Fraud associated with your system and that, if Toll Fraud occurs, it can result in substantial additional charges for your telecommunications services. Avaya Toll Fraud intervention If You suspect that You are being victimized by Toll Fraud and You need technical assistance or support, call Technical Service Center Toll Fraud Intervention Hotline at for the United States and Canada. For additional support telephone numbers, see the Avaya Support website: or such successor site as designated by Avaya. Suspected security vulnerabilities with Avaya products should be reported to Avaya by sending mail to: securityalerts@avaya.com. Downloading Documentation For the most current versions of Documentation, see the Avaya Support website: or such successor site as designated by Avaya. Contact Avaya Support See the Avaya Support website: for product or Hosted Service notices and articles, or to report a problem with your Avaya product or Hosted Service. For a list of support telephone numbers and contact addresses, go to the Avaya Support website: (or such successor site as designated by Avaya), scroll to the bottom of the page, and select Contact Avaya Support. Trademarks The trademarks, logos and service marks ( Marks ) displayed in this site, the Documentation, Hosted Service(s), and product(s) provided by Avaya are the registered or unregistered Marks of Avaya, its affiliates, or other third parties. Users are not permitted to use such Marks without prior written consent from Avaya or such third party which may own the Mark. Nothing contained in this site, the Documentation, Hosted Service(s) and product(s) should be construed as granting, by implication, estoppel, or otherwise, any license or right in and to the Marks without the express written permission of Avaya or the applicable third party. Avaya is a registered trademark of Avaya Inc. Aura is a registered trademark of Avaya Inc. All non-avaya trademarks are the property of their respective owners. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

4 Contents Chapter 1: Overview of Avaya IQ High Availability... 6 Avaya IQ High Availability... 6 Deployment considerations... 6 Prerequisites... 8 Limitations... 9 Recommendations Chapter 2: High Availability connection High Availability connection Configuring an HA connection Editing an HA connection Chapter 3: Admin Data Synchronization Admin Data Synchronization Scheduling the Admin Data Synchronization Executing on-demand Admin Data Synchronization Performance of Admin Data Synchronization Viewing Admin Data Synchronization log files for troubleshooting Troubleshooting Admin Data Synchronization failures Chapter 4: Historical data synchronization Historical Data Synchronization Executing on-demand Historical Data Synchronization Performance of Historical Data Synchronization Recovering historical data after an outage Viewing Historical Data Synchronization log files for troubleshooting Troubleshooting Historical Data Synchronization failures Chapter 5: Replicating data for High Availability in software-only deployments Replicating data for High Availability in software-only deployments Replicating Avaya IQ data and database to deploy HA in Dual and Multi-host software-only deployments Replicating Avaya IQ data and database to deploy High Availability in a Single host softwareonly deployment Chapter 6: Importing and exporting tools Tools for importing and exporting data Exporting reports Importing reports Troubleshooting exporting and importing data Chapter 7: Avaya IQ support for Communication Manager survivability Avaya IQ Support for Communication Manager Survivability Chapter 8: Survivability connection Creating a connection to a survivable Communication Manager system Avaya IQ High Availability and Survivability April 2015

5 Contents Limitations Avaya IQ connectivity to Communication Manager Setting up connectivity to the main Communication Manager system Setting up connectivity to the Survivable Communication Manager system About reporting group management Known issues Troubleshooting known issues Restarting the CMAdapter Chapter 9: Zero Downtime Upgrade Zero Downtime Upgrade Upgrade scenarios Performing Zero Downtime Upgrade in an HA deployment Performing Zero Downtime Upgrade to deploy HA Recovering from failures during upgrade April 2015 Avaya IQ High Availability and Survivability 5

6 Chapter 1: Overview of Avaya IQ High Availability Avaya IQ High Availability Introduction High Availability (HA) in Avaya IQ is a mechanism that uses two identical Avaya IQ systems connected to the same set of sources to back up and prevent data loss. One of the Avaya IQ systems is designated as the primary or the active system and the other as the secondary or the backup system. The Avaya IQ Operations, Administration, and Maintenance (OAM) tool uses the term Standby to represent the secondary system in an HA deployment. If the active system fails, loses connection to the source or the database server, or needs to be shut down for maintenance, the backup system carries the entire activity load during the outage. During this outage, you can connect to the backup system and continue normal operations. When the system is up again after the outage, using HA feautes, you can copy the source data from the backup system to the system that was down. The HA feature provides the following types of data synchronization: Admin data synchronization Preserves the frequently changing administration data and eliminates the need for manually duplicating the administration on a backup system. Historical data synchronization Preserves the historical data if the Avaya IQ system fails or a link to a source fails. The Admin and Historical Data Synchronization is achieved through jobs running in the Scheduler utility of OAM. Thus, you can use the HA feature to withstand planned and unplanned outages at the sources, the network, and the database, and provide continuous processing of important administration data and historical reporting data. Deployment considerations Both the primary and secondary systems in an HA deployment have identical configurations, provide full Avaya IQ capabilities, collect data from the same source, and are capable of interfacing 6 Avaya IQ High Availability and Survivability April 2015

7 Deployment considerations to the same downstream wall board and work force management systems. These systems, however, operate independently of each other except for the nightly synchronization of the administration data. Since the source sends two simultaneous data streams to both the primary and secondary systems, these systems contain the same call data. This includes the resources information and the historical data from the full spectrum of operation. In an HA deployment, the primary system is usually located at a main data center site; however, you can choose the location for the secondary system. Since both the primary and secondary systems collect data from the same set of sources, you must cautiously plan the placement of the primary and secondary systems to maximize access to sources in the event of a network outage. The following figure shows a typical HA deployment: See the following documents for instructions on how to install an Avaya IQ system: Preparing for Avaya IQ Turnkey Implementations Preparing for Avaya IQ Software-only Implementations Installing and Maintaining Dell R610 Hardware for Avaya IQ Turnkey Deployments Installing and Maintaining Dell R620 Hardware for Avaya IQ Turnkey Deployments Implementing Avaya IQ for Software-only Deployments April 2015 Avaya IQ High Availability and Survivability 7

8 Overview of Avaya IQ High Availability Implementing Avaya IQ on Dell R610 for Turnkey Deployments Implementing Avaya IQ on Dell R620 for Turnkey Deployments Prerequisites To prevent failure in the Admin and Historical Data Synchronization, ensure that the following prerequisites are met: The Avaya IQ software version must be identical on both the systems. You have designated the primary and the secondary systems and have installed the latest license file for HA on both systems prior to creating a connection. Both primary and secondary systems must have the same configurations. For example, if the primary system is configured as a multi-host system, the secondary system must also be configured as a multi-host system. Similarly, for a dual host or single host primary system, the secondary system must also be configured as dual host or single host system respectively. The locale and version of both operating systems must be the same. To configure HA in a multi-host environment, the primary and secondary systems must have the same number of data processing and reporting hosts. However, you can have different Data Collection host configuration between the primary and secondary systems. Both the systems must be connected to the same sources. For example, if the primary system is connected to a Communication Manager system, the secondary system must also be connected to the same Communication Manager system. Both the primary and secondary systems must use the same source name in the HA connection user interface. For proper data synchronization, ensure that the source name does not contain leading and trailing spaces. Both the primary and secondary systems must use the same names for the resource views that are defined under Enterprise > Resources in OAM. That is: - If you change the default view names on the primary system, you must make the same change on the secondary system. Note that you can rename all views except the All Resources view. - If you rename the Functional view on the primary system, you must assign the same name to the Functional view and not any other view on the secondary system. You have created an operating system user that has Read, Write, and Execute access to the $CCR_HOME directory on the primary and secondary Admin host or has all the capabilities of a $CCR_USER user. Both the primary and secondary systems must have SNMP alarming administered so that alarms are sent from either system. 8 Avaya IQ High Availability and Survivability April 2015

9 Limitations Limitations The HA feature in Avaya IQ Release 5.2 is offered with a limited capacity of 7000 administered agents across all data sources. That is, 7000 agents across the main Communication Manager systems, the survivable Communication Manager systems, and any other sources in an Avaya IQ deployment. Exceeding this capacity of 7000 administered agents lowers the synchronization performance. Avaya Aura Contact Center supports only one Avaya IQ connection at a time. Therefore, Avaya Aura Contact Center does not support Avaya IQ HA. You must perform the Historical Data Synchronization within ten days from the time a link to the source fails. Avaya IQ does not allow you to perform: - Aggregation of the data beyond ten days - Aggregation of data prior to creation of the stream If you add HA to an existing Avaya IQ system, the Historical Data Synchronization job does not copy the historical data that was available on the primary system prior to HA deployment on the secondary system. You must execute the summary reports only when the aggregation is complete. Aggregation can take anytime between an hour and several days depending upon the volume of data being aggregated and the level of processing headroom in the database server. You can check the progress of aggregation using the Data Monitoring reports, specifically the Stream Availability - Summary or the Stream Availability - Trend report. The % Data Complete measure in these reports shows the status of aggregation. Custom reports on the primary and secondary systems are not synchronized as part of Admin Data Synchronization or Historical Data Synchronization. However, you can import or export the custom reports from the primary system to the secondary system each time you modify these reports using the custom report import and export tools. For more information see Importing and exporting tools on page 33. In an HA deployment, Enterprise LDAP and CUSLDAP users are not automatically synchronized between the primary and secondary systems. For Active Directory users, you must manually add LDAP connections to both systems and ensure that the LDAP connection attributes are identical on both the primary and the secondary systems. You must also schedule the Enterprise LDAP Synchronization job on the secondary system to run at the same time as the primary system. The HA feature does not manage report data traffic. Therefore, you cannot distribute report users on both the primary and secondary systems to reduce or balance report data traffic. The HA feature primarily restores the data lost due to failure of the active system, loss of connectivity to the source, or outage for maintenance, and synchronizes the data on both the primary and secondary systems. You cannot run the Admin Data Synchronization and the Historical Data Synchronization jobs simultaneously. You must run these jobs one at a time. April 2015 Avaya IQ High Availability and Survivability 9

10 Overview of Avaya IQ High Availability Recommendations To ensure proper data synchronization, conform to the following operational practices: Synchronize the historical data 24 hours after the primary system is up again after an outage. This ensures that all contacts on the secondary system are complete and the agents have logged out. Synchronize the historical data only for the downtime interval. Perform administration only on the primary system, and not on the secondary system. The administration done on the secondary system is lost during the nightly synchronization job as the administration data from the primary system overwrite the administration data on the secondary system. Important: You can perform administration on the secondary system only when the primary system is non-functional and you have to perform administration. This is the only case when you must run an on-demand Admin Data Synchronization job on the primary system when the primary system is functional after an outage. Except this case, you must not run Admin Data Synchronization on the primary system. Perform all custom report design only on the primary system. Run Admin Data Synchronization prior to running the historical reports to ensure that the Avaya IQ system has the latest administration data. 10 Avaya IQ High Availability and Survivability April 2015

11 Chapter 2: High Availability connection High Availability connection High Availability (HA) connection links the primary and secondary systems for synchronizing the administration and historical data. Important: You must configure HA connection on both primary and secondary systems. Ensure that you configure the connection properly; else the data does not get synchronized. An Avaya IQ system supports only one HA connection at a time. You must install the latest license for HA prior to configuring a connection. You can select Help > About on OAM to check whether an Avaya IQ deployment is configured as a primary or secondary system. Configuring an HA connection Procedure 1. Log on to OAM on one of the Avaya IQ systems. 2. Under the Tasks tab, select Connection Management > Administer Connections. The system displays the All Connection Types window. 3. Click High Availability Connection. The system displays the Connection Resources window. 4. Click Add to add a new HA connection. The system displays the Add Resource window. 5. On the Add Resource page: a. Connection Name Enter a name for the connection. This name is displayed in administration to identify the connection. b. Description Enter the description for the connection. c. Click Next. April 2015 Avaya IQ High Availability and Survivability 11

12 High Availability connection 6. On the next Add Resource page, enter values for the following parameters: a. Admin User Enter the user name that is assigned the system administrator role and the administrative privileges to access the primary OAM interface. b. Admin User Password Enter the password for the admin user. c. Admin User Password Confirm Re-enter the admin user password. d. Click Next. 7. On the next Add Resource page, enter values for the following parameters on the remote system: a. Remote High Availability System Type Select the type of remote host as Primary or Secondary. If you configure a connection on a primary system, select the Remote High Availability System Type as Secondary. Similarly, if you configure a connection on a secondary system, select the Remote High Availability System Type as Primary. b. Remote Admin Host IP Address Enter the IP address of the remote Administration host or All Functions host. c. Remote Admin Host User Enter the operating system user name that has Read, Write, and Execute access to the $CCR_HOME directory on the remote Administration host or All Functions host. You can also use the root ID. If you modify the Unix command prompt, make sure that the prompt ends with one of the #', '%', '>', or '$' characters followed by one space. d. Remote Admin Host User Password Enter the password to access the remote Administration host or All Functions host. e. Remote Admin Host User Password Confirm Re-enter the remote Administration host or All Functions host password. f. Remote High Availability System Database IP Address Enter the IP address to access the database on the secondary system. g. Remote Historical Database Name Enter the Oracle System ID or instance name of the remote database. By default, for both software-only and turnkey systems, this name is avayaiq. h. Remote Historical Schema Name Enter the Oracle schema name of the historical user. By default, for software-only systems this name is RPT_HIST_OWNER_USER and for turnkey systems this name is AVAYAIQ. i. Remote Historical Database User Enter the Oracle user name to access the historical database. By default, for software-only systems this name is RPT_HIST_OWNER_USER and for turnkey systems this name is AVAYAIQ. j. Remote Historical Database User Password Enter the password to access the remote database. k. Remote Historical Database User Password Confirm Re-enter the Remote Historical Database User Password. 12 Avaya IQ High Availability and Survivability April 2015

13 Editing an HA connection l. Click Next. 8. On the Finish Up page, click Finish. Editing an HA connection Procedure 1. Log on to OAM on one of the Avaya IQ systems. 2. Under the Tasks tab, select Connection Management > Administer Connections. The system displays the All Connection Types window. 3. Click High Availability Connection. The system displays the Connection Resources window. 4. Click Edit to edit an existing HA connection. The system displays the Edit Resource window with the following tabs: General Connection Maintenance 5. Click General on the Edit Resource page and edit the following fields: a. Connection Name Enter a name for the connection. This name is displayed in administration to identify the connection. b. Description Enter the description for the connection. c. Admin User Enter the user name that is assigned the system administrator role and the administrative privileges to access the primary OAM interface. d. Admin User Password Enter the password for the admin user. e. Admin User Password Confirm Re-enter the admin user password. 6. Click Connection on the Edit Resource page and edit the following remote system parameters: a. Remote Admin Host IP Address Enter the IP address of the remote Administration host or All Functions host. b. Remote Admin Host User Enter the operating system user name that has Read, Write, and Execute access to the $CCR_HOME directory on the remote Administration host or All Functions host. You can also use the root ID. If you modify the Unix command prompt, make sure that the prompt ends with one of the #', '%', '>', or '$' characters followed by one space. c. Remote Admin Host User Password Enter the password to access the remote Administration host or All Functions host. April 2015 Avaya IQ High Availability and Survivability 13

14 High Availability connection d. Remote Admin Host User Password Confirm Re-enter the remote Administration host or All Functions host password. e. Remote High Availability System Database IP Address Enter the IP address to access the database on the secondary system. f. Remote Historical Database Name Enter the Oracle System ID or instance name of the remote database. For both software-only and turnkey systems, this name is usually avayaiq. g. Remote Historical Schema Name Enter the Oracle schema name of the historical user. For software-only systems, this name is usually RPT_HIST_OWNER_USER. For turnkey systems, this name is usually AVAYAIQ. h. Remote Historical Database User Enter the Oracle user name to access the historical database. For software-only systems, this name is usually RPT_HIST_OWNER_USER. For turnkey systems, this name is usually AVAYAIQ. i. Remote Historical Database User Password Enter the password to access the remote database. j. Remote Historical Database User Password Confirm Re-enter the Remote Historical Database User Password. k. Click Next. 7. Click Maintenance on the Edit Resource page to view the type of the configured remote host. You can also add the other generic properties by clicking Add. 8. Click Apply to save the changes. 14 Avaya IQ High Availability and Survivability April 2015

15 Chapter 3: Admin Data Synchronization Admin Data Synchronization Use the Admin Data Synchronization tool to preserve the frequently changing administration data and to eliminate the need for manually duplicating the administration on a backup system. The Admin Data Synchronization tool does: Synchronize roles, groups, permissions, thresholds, service level definitions, and behavior data. The Admin Data Synchronization tool does not: Synchronize the data that originate from a source, such as, devices, routing points, queues, processes, and agents. Synchronize the subsystem data. For example, Admin Data Synchronization does not synchronize the data under Sites > Default Admin Site in the Enterprise tab in OAM. Scheduling the Admin Data Synchronization About this task The secondary system automatically creates an AdminDataSynchronization job which can be scheduled with a predefined frequency. You can only edit the AdminDataSynchronization job to specify a different frequency. You cannot add or delete the AdminDataSynchronization scheduled job. Procedure 1. Log on to OAM on the secondary system. 2. Under the Tasks tab, select Utility > Administer Scheduled Jobs. The system displays the View Scheduled Jobs page. 3. Select AdminDataSynchronization and click Enable ( ) if it is in disabled state. 4. Select AdminDataSynchronization and click Edit ( ). The system displays the Edit System Scheduled Job page. April 2015 Avaya IQ High Availability and Survivability 15

16 Admin Data Synchronization 5. Under Job Parameters, the system, by default, displays the pre-configured connection name in the High Availability Connection Name field. 6. Under Frequency, select the frequency to run the scheduled job. a. Select one of the following: Hourly Runs the job every hour. Daily (Default) Runs the job every day. Weekly Runs the jobevery week. Monthly Runs the job every month. Yearly Runs the job every year. b. Specify the number of days between job executions. 7. Under Range of Recurrence: a. Select the Start date and time when you want to run the Admin Data Synchronization tool. b. Specify the end of the schedule by selecting one of the following options. 8. Click OK. No end date Number of occurrences End by If you select End by, you must select the date and time for ending the scheduled job. Executing on-demand Admin Data Synchronization About this task You can add, modify, or delete an on-demand scheduled job on the primary or secondary system. Important: You must run Admin Data Synchronization only on the secondary system. The only case when you can run Admin Data Synchronization on the primary system is when the primary system is functional after an outage and you have performed administration on the secondary system when the primary system was non-functional. Except this case, you must not run Admin Data Synchronization on the primary system. Procedure 1. Log on to OAM where you want to create an on-demand Admin Data Synchronization job. 16 Avaya IQ High Availability and Survivability April 2015

17 Performance of Admin Data Synchronization 2. Under the Tasks tab, select Utility > Administer Scheduled Jobs. The system displays the View Scheduled Jobs page. 3. Click Add. The system displays the Create Scheduled Job page. 4. Select Admin Data Synchronization in the Job Type list. The system displays the category as On Demand. 5. Enter a name in the Name field for the scheduled job you want to create. 6. Under Job Parameters, enter the connection name in the High Availability Connection Name field. 7. Under Frequency, select one of the following: Now Runs the job immediately after you finish creating the job. At Runs the job at the specified date and time. 8. Click OK. The default frequency is set to One Time. Performance of Admin Data Synchronization Lab testing by Avaya under various system configurations has shown that the Admin Data Synchronization job can take anytime between 5 minutes and 2 hours. The execution time of a job depends on the amount of data that is synchronized. The execution time can vary based on the hardware configuration and load of the system. The amount of data to be synchronized depends on the differential data for roles, user assignments, thresholds, groups and permissions between the primary and secondary systems. Viewing Admin Data Synchronization log files for troubleshooting Condition For any problem that occurs with the Admin Data Synchronization, you can view the following log files for troubleshooting. /var/log/avaya/ccr/adminha.logs April 2015 Avaya IQ High Availability and Survivability 17

18 Admin Data Synchronization /var/log/avaya/ccr/reporting_recorder/reporting_recorders.log If the primary and secondary systems have different set of resources, the Admin data synchronization job succeeds with an error. The error message is logged in the /var/log/ Avaya/CCR/AdminHA.logs file in the following format. For example, <Date> <Time>, [main] ERROR com.avaya.ccr.admindatasync.bulkadmin.datapermmgmt.hadatapermmgmt - <error message> :53:19,616 [main] 936 ERROR com.avaya.ccr.admindatasync.bulkadmin.datapermmgmt.hadatapermmgmt - Cannot find group with name :53:19,640 [main] 667 ERROR com.avaya.ccr.admindatasync.bulkadmin.datapermmgmt.hadatapermmgmt - Cannot find user with ID :53:19,642 [main] 486 ERROR com.avaya.ccr.admindatasync.bulkadmin.datapermmgmt.hadatapermmgmt - No valid roles or users specified In this case, you can ignore this error as the synchronization is successful. However, you can use this error message to validate the differences in the resources. Troubleshooting Admin Data Synchronization failures The following table provides the error messages that are displayed for different Admin Data Synchronization failure scenarios and a possible solution. Error message org.quartz.jobexecutionexception: Admin data synchronization exec() was interrupted while waiting for completion: org.quartz.jobexecutionexception: Job returned a non zero return code: 255 org.quartz.jobexecutionexception: Failed as specified HAConnection does not exist. Solution This error occurs if you enter incorrect login credentials for the admin user who has administrative privileges to access the local OAM application. 1. Enter the correct user name and password in the Admin User and Admin User Password fields while configuring an HA connection. 2. Run the synchronization job again. This error occurs if you enter incorrect HA connection data when you: Configure an HA connection. Table continues 18 Avaya IQ High Availability and Survivability April 2015

19 Troubleshooting Admin Data Synchronization failures Error message ERROR com.avaya.ccr.historicaldatasynch.i TL_operation Error While Obtaining Remote Connection jdbc ERROR com.avaya.ccr.historicaldatasynch.h istoricaldatasynchjob Another Historical or Admin Synch Job running please check scheduler console HA. 0789d9dd301ff1b001302dce66b6077d ERROR com.avaya.ccr.admindatasync.adminda tasynchscheduler.admindatasynchjob Admin data synchronization is Failed. INFO org.quartz.core.jobrunshell Job HA. 0789d9dd300f53ae01300fa2c37d013b threw a JobExecutionException: org.quartz.jobexecutionexception: Admin data synchronization exec() was interrupted while waiting for completion: org.quartz.jobexecutionexception: Solution Run an on-demand Admin Data Synchronization job on the primary or secondary system. 1. Do one of the following: Enter the correct High Availability Connection Name on the Add Resource page while configuring an HA connection. Enter the correct HA connection name for the on-demand Admin Data Synchronization job you want to run. 2. Run the synchronization job again. This error occurs if you do not enter the correct HA connection name on the primary or the secondary system. Ensure that: If you run the Historical Data Synchronization tool on a primary system, enter the name of the HA connection that is administered on the primary system. If you run the Historical Data Synchronization tool on a secondary system, enter the name of the HA connection that is administered on the secondary system. This error occurs if you run more than one Admin Data Synchronization jobs, more than one Historical Data Synchronization jobs, or a combination of Admin and Historical data synchronization jobs simultaneously You must run only one synchronization job at a time. This error occurs if: The user name or password to access the remote Administration host or All Functions host is incorrect. Avaya IQ on the remote Administration host or All Functions host is not running. 1. To resolve this, perform one of the following: Enter the correct user name or password to access the remote Administration host or All Functions host. Check that Avaya IQ on the remote Administration host or All Functions host is running. Table continues April 2015 Avaya IQ High Availability and Survivability 19

20 Admin Data Synchronization Error message Job returned a non zero return code: 255 Solution 2. Run the synchronization job again. 20 Avaya IQ High Availability and Survivability April 2015

21 Chapter 4: Historical data synchronization Historical Data Synchronization You can use the Historical Data Synchronization tool to preserve and synchronize the historical data between the primary and the secondary system if an Avaya IQ system fails or a link to a source fails. You can run the Historical Data Synchronization tool on the primary system and on the secondary system through an on-demand job. The Historical Data Synchronization process synchronizes the data only for a specified time interval. To synchronize historical data from different sources, you must run the Historical Data Synchronization tool separately for each source. To synchronize historical data for inbound and outbound calls, you must first run the Historical Data Synchronization tool for Communication Manager and then for Proactive Contact. To synchronize historical data for inbound Voice Portal calls, you must first run the ETL tool on the system that was under temporary outage, and then execute the Historical Data Synchronization job first for Communication Manager and then for Experience Portal or Voice Portal for the down-time interval. For more information on running the ETL tool, see Running the ETL application section in the Implementing Avaya IQ for Turnkey Deployments or Implementing Avaya IQ for Software-only Deployments documents. Considerations To prevent data loss during Historical Data Synchronization, you must ensure that the backup system is functional for an additional hour both before and after the interval for which you want to perform Historical Data Synchronization. For example, to synchronize historical data on the primary system for one hour, for example, between 7 a.m. and 8 a.m., the backup system must be functional for a minimum of three hours, for example, between 6 a.m. and 9 a.m. The Historical Data Synchronization process synchronizes the data only for a specified time interval. You can notice discrepancy in the call details between the reports generated on the primary and the secondary systems, if a call: initiates before the start time of synchronization and overlaps with the synchronization interval. initiates before the end time of synchronization and continues beyond the synchronization interval. In such cases, the Historical Data Synchronization fails to capture the entire call details for the calls that continue beyond the synchronization interval. You can also notice discrepancy when a parent call initiates an extension call and either of the parent or the extension call overlaps with the synchronization interval. In such a case, the Historical April 2015 Avaya IQ High Availability and Survivability 21

22 Historical data synchronization Data Synchronization fails to capture the call details of that part of the parent call or the extension call that falls outside the synchronization interval. The Historical Data Synchronization does not synchronize the Stream Availability Summary and the Stream Availability Trend reports as these reports contain the information about network outages. You can use these reports to view historical information about the link up and link down times. You must complete the Historical Data Synchronization and the subsequent aggregation recovery within ten days from the time a link to the source fails. For more information about this limitation of recovery from data outages, see Recovering historical data on page 24. After you complete Historical Data Synchronization, the Avaya IQ system performs aggregation for the synchronized data. Aggregation can take anytime between an hour and several days depending upon the volume of data being aggregated and the level of processing headroom in the database server. You can check the progress of aggregation using the Data Monitoring reports, specifically the Stream Availability-Summary or the Stream Availability - Trend report. The % Data Complete measure in these reports shows the status of aggregation. Executing on-demand Historical Data Synchronization You can add, modify, or delete an on-demand scheduled job that performs Historical Data Synchronization. This can be done on either the primary or secondary. Procedure 1. Log on to OAM on the primary or secondary system on which you want to create an ondemand Historical Data Synchronization job. That is, the system that is operational again after an outage. 2. Under the Tasks tab, select Connection Management > Administer Connections. The system displays the All Connection Types window. 3. Click High Availability Connection link. The system displays the Connection Resources window. 4. Configure a connection if you have not already configured one. If you have a pre-configured connection, ensure that you have entered values for all the fields under the Historical Data Synchronization tab. For more information refer to Configuring a connection on page 11 or Editing a connection on page Under the Tasks tab, select Utility > Administer Scheduled Jobs. The system displays the View Scheduled Jobs page. 6. Click Add ( ). The system displays the Create Scheduled Job page. 7. Select Historical Data Synchronization in the Job Type list. 22 Avaya IQ High Availability and Survivability April 2015

23 Executing on-demand Historical Data Synchronization The system displays the category as On Demand. 8. In the Name field, enter a name for the scheduled job. 9. Under Job Parameters, enter the following values: a. In the Source Name to Synchronize from field, enter the name of source for which you are synchronizing the data. To synchronize historical data for inbound calls, enter the name of the Communication Manager source. For outbound calls enter the name of the Proactive Contact source. b. In the High Availability Connection Name field, enter the name of the configured HA connection. If you run the Historical Data Synchronization tool on a primary system, enter the name of HA connection that is administered on the primary system. Similarly, if you run the Historical Data Synchronization tool on a secondary system, enter the name of HA connection that is administered on the secondary system. c. Specify the time interval for synchronizing historical data by selecting a date and time in the following fields: Start Date of Synchronization Window Start Time of Synchronization Window End Date of Synchronization Window End Time of Synchronization Window You can use the Stream Availability report to determine the start date, start time, end date and end time for synchronization. Use the time zone of the Avaya IQ Administration or All Functions host as a reference for the interval time zone. 10. Under Frequency, select one of the following: Now Runs the job immediately after you finish creating the job. At Runs the job at the specified date and time. 11. Click OK. The default frequency is set to One Time. April 2015 Avaya IQ High Availability and Survivability 23

24 Historical data synchronization Performance of Historical Data Synchronization Lab testing by Avaya under various system configurations has shown that the time required for Historical Data Synchronization can vary based on the following factors: Amount of data present in the database system. Amount of data that is transferred between the primary and the secondary system. Hardware configurations of the database system. Load on the database system. Historical Data Synchronization can synchronize a maximum of calls per synchronization interval. You can observe the performance of Historical Data Synchronization specific to your configuration and for different datasets. Dataset signifies the number of calls during the downtime interval. For making better decisions regarding synchronization, observe the performance of Historical Data Synchronization by: Executing the jobs with smaller dataset. Dividing the downtime duration into small intervals for larger datasets, and running the Historical Data Synchronization job for each interval until you synchronize the data for the entire downtime. For example, if the outage is for three hours and there were 120,000 total calls, divide the three hour duration in three intervals so that each interval will have maximum of calls. Executing synchronization jobs for these one hour intervals (where calls <= calls) ensures proper data synchronization. Before comparing reports on the primary and secondary systems for the time period overlapping the downtime, you must synchronize data for the entire downtime. Recovering historical data after an outage Using Historical Data Synchronization, Avaya IQ can recover historical data following an outage. Use the following table as a reference to determine the maximum duration of an outage (in hours) for which an Avaya IQ system can recover historical data. This duration is estimated based on the current size of the database in Gigabytes (GB) and the call rate measured as calls per hour (CPH) during the daily busy hour. The estimates in this table are only approximate for several reasons. For example, the actual call rate during a data outage can vary depending on the time of day or day of week of the outage. Therefore, you should use the following table only as a guideline to determine the behavior and effectiveness of high available systems of various sizes. 24 Avaya IQ High Availability and Survivability April 2015

25 Viewing Historical Data Synchronization log files for troubleshooting If you are uncertain about the typical busy hour call rate in your system, you can estimate the busy hour call are using the following formula. Call rate = Maximum number of concurrently staffed agents/average call duration (in hours). For example, if the system has 1000 simultaneously logged-in agents and the average call duration is 3 minutes (0.05 hours), then the call rate is approximately 1000/0.05 = 20,000 CPH. For example, if the size of database is 200 GB at the time of HA deployment and the call rate is 20,000 CPH, the Avaya IQ system should be able to recover historical data for an outage of up to about 63 hours. If the duration of outage is longer than the estimated value in the table, the data may not be recoverable, although Avaya support staff may be able to assist in recovery of larger data outages. If the estimated value in the table is zero, then this represents a data outage scenario which is likely unrecoverable via the HA feature. Busy Hour Call Rate in calls per hour (CPH) Size of the Database at time of HA Deployment (GB) 0 GB (new Avaya IQ deployed initially as HA) 100 GB 200 GB 3000 GB 400 GB 500 GB 1, , , , , , , , , , , , , Viewing Historical Data Synchronization log files for troubleshooting If a problem occurs with the Historical Data Synchronization in an HA deployment, you can view the following log file for initial troubleshooting. /var/log/avaya/ccr/reporting_recorder/reporting_recorders.log April 2015 Avaya IQ High Availability and Survivability 25

26 Historical data synchronization If you notice discrepancy about agent details between the reports on the primary and the secondary systems, one of the reasons can be that you created an agent and later deleted that agent on Communication Manager when the secondary system was non-functional. In such a case, the Historical Data Synchronization fails to capture the changes in the agent details during the outage. Troubleshooting Historical Data Synchronization failures The following table provides the error messages that are displayed for different Historical Data Synchronization failure scenarios and a possible solution. Error message ERROR com.avaya.ccr.historicaldatasynch.i TL_operation Error While Obtaining Remote Connection jdbc:oracle:thin:@ :1521 :avayaiq,avayaiq java.sql.sqlexception: Listener refused the connection with the following error: ORA-12505, TNS:listener does not currently know of SID given in connect descriptor The Connection descriptor used by the client was: :1521:avayaiq ERROR com.avaya.ccr.historicaldatasynch.i TL_operation Error While Obtaining Remote Connection jdbc:oracle:thin:@ :1521 :avayaiq,avayaiq java.sql.sqlexception: Io exception: Got minus one from a read call java FATAL com.avaya.ccr.historicaldatasynch.i TL_import Failed to set remote database connection :20:27,744 com.avaya.ccr.historicaldatasynch.h istoricaldatasynchjob Solution This error occurs if the remote historical database is down or nonfunctional or cannot be reached. 1. Check the state of the remote database; ensure that the remote database is accessible and functional. 2. Run the synchronization job again. To prevent data loss during Historical Data Synchronization, you must ensure that the backup system, including the remote database, is functional for an additional hour both before and after the interval for which you want to perform Historical Data Synchronization. This error occurs if the remote historical database is down or nonfunctional or cannot be reached. 1. Check the state of the remote database; ensure that the remote database is accessible and functional. 2. Run the synchronization job again. To prevent data loss during Historical Data Synchronization, you must ensure that the backup system, including the remote database, is functional for an additional hour both before and after the interval for which you want to perform Historical Data Synchronization. Table continues 26 Avaya IQ High Availability and Survivability April 2015

27 Troubleshooting Historical Data Synchronization failures Error message [DefaultQuartzScheduler_Worker-9] 341 HistoricalDataSynchJob.java ERROR com.avaya.ccr.historicaldatasynch.h istoricaldatasynchjob Import failed ERROR com.avaya.ccr.historicaldatasynch.i TL_operation Error While Obtaining Remote Connection :avayaiq,avayaiq java.sql.sqlexception: ORA-01017: invalid username/password; logon denied ERROR com.avaya.ccr.historicaldatasynch.i TL_interval ERROR:: Stream key or Source key mismatch between Primary and Secondary. Either invalid keys exist or Source/ Stream names do not match.match SourceDim and StreamDim tables on Primary and Secondary for consistency ERROR com.avaya.ccr.historicaldatasynch.i TL_operation Error While Obtaining Remote Connection :avayaiq,avayaiq java.sql.sqlexception: Io exception: The Network Adapter could not establish the connection Solution This error occurs if you enter an incorrect password to access the remote database while configuring an HA connection. 1. Enter the correct Remote Historical Database User Password on the Add Resource page while configuring an HA connection. 2. Run the synchronization job again. This error occurs if you enter an incorrect source name in the Source Name to Synchronize from field while executing on-demand Historical Data Synchronization. 1. Ensure that both the primary and secondary systems use the same source name in the HA connection user interface. 2. Ensure that the source name does not contain leading and trailing spaces. 3. Run the synchronization job again. This error occurs if you enter an incorrect IP address to access the database on the secondary system while configuring an HA connection. 1. Enter the correct IP address in the Remote High Availability System Database IP Address field on the Add Resource page while configuring an HA connection. 2. Run the synchronization job again. April 2015 Avaya IQ High Availability and Survivability 27

28 Chapter 5: Replicating data for High Availability in software-only deployments Replicating data for High Availability in software-only deployments Important: Use this chapter ONLY if you have: NOT implemented your existing Avaya IQ system as a High Availability (HA) system. Implemented your existing Avaya IQ system in a software-only deployment. Replication of Avaya IQ data and the database ensures proper functioning of the HA feature and proper data synchronization between the existing Avaya IQ system and the backup Avaya IQ system that you add to deploy HA. If you want to deploy HA in your existing Avaya IQ setup: You must replicate the existing Avaya IQ data and the existing Avaya IQ database if the data sources initially defined in the existing Avaya IQ setup are changed prior to deploying HA. You can replicate the existing Avaya IQ data and database if you want to ensure that after deploying HA, the primary and the secondary systems have the same historical report data. Replicating Avaya IQ data and database to deploy HA in Dual and Multi-host software-only deployments Before you begin Ensure that you have: An Avaya IQ system to use as a backup system. The backup system has the same configuration as the existing Avaya IQ system. For example, if the existing system is configured as a multi-host system, the backup system is also configured as a multi-host system. 28 Avaya IQ High Availability and Survivability April 2015

29 Replicating Avaya IQ data and database to deploy HA in Dual and Multi-host software-only deployments Installed the same version of Operating System with all prerequisite RPMs on all the hosts on the backup Avaya IQ system. For the ease of understanding, the existing Avaya IQ system is referred as the primary system and the backup Avaya IQ system is referred as the secondary system in the following procedure. Procedure 1. Clone the primary Avaya IQ database on the database host of the secondary system. 2. Verify that the listener is running and is functioning normally with the registered database instance. 3. Back up the Avaya IQ data available on the primary system. For more information on backing up Avaya IQ data, see Backing up system and database data section in Maintaining and Troubleshooting Avaya IQ. 4. After you complete the backup, disconnect the database host of the primary system from the network. 5. On the database host of the secondary system, replace the host name and IP address values in the following files to correspond to the values of the database host of the primary system. /etc/hosts /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-eth0 $ORACLE_HOME/network/admin/tnsnames.ora $ORACLE_HOME/network/admin/listener.ora Important: You must reboot the database host for the changes to take effect. 6. Copy Avaya IQ base backup and Avaya IQ data backup files from the primary system to the secondary system. 7. Shut down the primary system. 8. Modify the host name and IP address values in the following files on the All Functions or the Administration host in the secondary system to correspond to the values on the All Functions or the Administration host in primary system. /etc/hosts /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-eth0 9. Restart the network services for the changes in host name and IP address to take effect. 10. Install Oracle client application on the secondary system. April 2015 Avaya IQ High Availability and Survivability 29

30 Replicating data for High Availability in software-only deployments 11. Configure the Oracle client to connect to the database host of the secondary system. 12. Restore the Avaya IQ data backed up in Step 3 on the All Functions or the Administration host in the secondary system. For more information on restoring Avaya IQ data, see the Restoring system and database data section in Maintaining and Troubleshooting Avaya IQ. Complete Steps 4, 5, 7, and 8 to ensure successful restore on All Functions or the Administration host, and the database host in the secondary system. 13. Ensure that the Avaya IQ application on the secondary system is running after the restore is complete. If the real-time dashboard reports user interface is not accessible or functional after the restore is complete, restart the RTD Tomcat container from the Lifecycle Manager by executing the following command: /opt/coreservices/lifecycle/bin/lc restart <UUID_of_RTD_Tomcat_container> 14. Run the IP address and host name change script on the All Functions or the Administration host in the secondary system. For more information about changing the host name and IP address of your Avaya IQ system, see Changing hostname and IP address in Maintaining and Troubleshooting Avaya IQ. 15. Connect the secondary system to the same source to which the primary system is connected. 16. Once the secondary system and the corresponding database host is functional with the new set of host names and IP addresses, you must: a. Connect the database host of the primary system to the network b. Start the primary Avaya IQ system and ensure that Avaya IQ application is running on the primary system. At this point, you have two independent Avaya IQ systems associated with two different database hosts but connected to the same data source on two different ports. 17. Install appropriate licenses on these two Avaya IQ systems to designate them as primary and secondary system. 18. Configure HA connections on both the primary and secondary systems. 19. Execute Admin Data Synchronization and Historical Data Synchronization jobs on the primary system to synchronize data between the primary and secondary systems. You must run these jobs one at a time. See Limitations. 30 Avaya IQ High Availability and Survivability April 2015

31 Replicating Avaya IQ data and database to deploy High Availability in a Single host software-only deployment Replicating Avaya IQ data and database to deploy High Availability in a Single host software-only deployment Before you begin Ensure that you have: An Avaya IQ system to use as a backup system. The backup system has the same configuration as the existing Avaya IQ system. For example, if the existing system is configured as a Single host system, the backup system is also configured as a Single host system. Installed the same version of Operating System with all prerequisite RPMs on all the hosts on the backup Avaya IQ system. For the ease of understanding, the existing Avaya IQ system is referred as the primary system and the backup Avaya IQ system is referred as the secondary system in the following procedure. Procedure 1. Clone the database of the primary Avaya IQ system on the secondary system. 2. Verify that the listener is running and is functioning normally with the registered database instance. 3. Back up the Avaya IQ data available on the primary system. For more information on backing up Avaya IQ data, see Backing up system and database data section in Maintaining and Troubleshooting Avaya IQ. 4. Copy Avaya IQ base backup and Avaya IQ data backup files from the primary system to the secondary system. 5. After you copy the backup files, disconnect the primary system from the network. 6. On the secondary system, replace the host name and IP address values in the following files to correspond to the values of the primary system. /etc/hosts /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-eth0 $ORACLE_HOME/network/admin/tnsnames.ora $ORACLE_HOME/network/admin/listener.ora Important: You must reboot the database host for the changes to take effect. 7. Restore the Avaya IQ data backed up in Step 3 on the secondary system. For more information on restoring Avaya IQ data, see the Restoring system and database data section in Maintaining and Troubleshooting Avaya IQ. April 2015 Avaya IQ High Availability and Survivability 31

32 Replicating data for High Availability in software-only deployments 8. Ensure that the Avaya IQ application on the secondary system is running after the restore is complete. If the real-time dashboard reports user interface is not accessible or functional after the restore is complete, restart the RTD Tomcat container from the Lifecycle Manager by executing the following command: /opt/coreservices/lifecycle/bin/lc restart <UUID_of_RTD_Tomcat_container> 9. Run the IP address and host name change script on the secondary system. For more information about changing the host name and IP address of your Avaya IQ system, see Changing hostname and IP address in Maintaining and Troubleshooting Avaya IQ. 10. Connect the secondary system to the same source to which the primary system is connected. 11. Once the secondary system is functional with the new host name and IP address, you must: a. Connect the primary system to the network. b. Start the primary Avaya IQ system and ensure that Avaya IQ application is running on the primary system. At this point, you have two independent Avaya IQ systems associated with two different database hosts but connected to the same data source on two different ports. 12. Install appropriate licenses on these two Avaya IQ systems to designate them as primary and secondary system. 13. Configure HA connections on both the primary and secondary systems. 14. Execute Admin Data Synchronization and Historical Data Synchronization jobs on the primary system to synchronize data between the primary and secondary systems. You must run these jobs one at a time. 32 Avaya IQ High Availability and Survivability April 2015

33 Chapter 6: Importing and exporting tools Tools for importing and exporting data Using the import and export tools, you can synchronize custom reports between the Avaya IQ hosts. For example, in a High Availability (HA) deployment, you can export the custom reports on the primary host and import these custom reports at the secondary host. Exporting reports Before you begin Save the reports that you want to export in one of the Custom Reports, My Folders, or Data Export directories in the Standard Reports UI. You can export a single report or multiple reports from a source Avaya IQ system to another Avaya IQ system. Procedure 1. Log on to the Avaya IQ system as root or as a user with root-level privileges. 2. Enter: cd /opt/avaya/ccr/jars/ui 3. Ensure that the ExportCustomsReports.sh and ImportCustomsReports.sh files and the directory runloadsampledata are present. 4. Enter the following command to back up the Export_Import.log file, if it exists on your system: mv Export_Import.log /<destination_directory_name> When you use the export utility for the first time, it generates the Export_Import.log file. 5. Before running the export utility for the first time, replace the string dataloadertemplate.properties with dataloader.properties in the script runloadsampledata.sh available at: /opt/avaya/ccr/jars/ui/runloadsampledata/ April 2015 Avaya IQ High Availability and Survivability 33

34 Importing and exporting tools 6. Enter one of the following commands: Example To export all reports from a specified directory, enter:./exportcustomreports.sh user:<user_name> folder:<folder_path> <Name_of_export_output_package> To export a single report, enter:./exportcustomreports.sh user:<user_name> report:<report_name_with_absolute_path> <Name_of_export_output_package> When using the folder option, the export script only exports those reports visible in the Avaya IQ reporting interface, not the Cognos interface. When using the report option, you must use the exact name as shown in the Avaya IQ reporting interface, not the Cognos interface. For an Enterprise LDAP user, you must replace <USER_NAME> with the Fully Qualified User Name of that LDAP user. For example, you can specify username@domain.com as a value of <USER_NAME>. You can replace <FOLDER_PATH> with the name of the directory that contains the custom reports that you want to export. For example, you can specify /Custom Reportsor /My Folders or /Data Export directories as <FOLDER_PATH> values. The directory or report name is case-sensitive. The system generates an error log if the user does not exist On completion of the export process, the system exports the reports to the /tmp/ Exported_Reports directory. The system also generates an Export_Import.log file. You can view the log file to verify whether the exported reports are error free. Your requirement is to export all reports from the /Custom Reports/Agent Reports directory. Enter:./exportCustomReports.sh user:avaya folder:/custom Reports/Agent Reports My_Monday_Reports You can view the exported reports in /tmp/exported_reports directory. Your requirement is to export the Agent Occupancy report from the /Custom Reports/ Agent Reports directory. Enter:./exportCustomReports.sh user:steve report:/my Folders/My Always Idle Agents/Agent Occupancy my_agents_occupancy 34 Avaya IQ High Availability and Survivability April 2015

35 Importing reports You can view the exported report in /tmp/exported_reports directory. Importing reports You can import reports from a source Avaya IQ system to another Avaya IQ system. Before you begin If you have multiple reporting hosts, you must perform this procedure only on the reporting host that contains the Cognos Content Manager. Execute the following command on each Reporting host to determine which Reporting host contains the Content Manager: grep "Content Manager is running in" $CCR_HOME/reporting/crn/logs/ cogserver.log tail -1 Procedure 1. Log on to the Reporting host that has Content Manager running as root or as a user with root-level privileges. 2. Ensure that the /tmp directory does not contain any zip file before you start the import process. 3. Ensure that the /tmp/reports_to_import directory, if present on your system, is empty before you start the import process. If the Reports_To_Import directory is not present, enter the mkdir /tmp/ Reports_To_Import command to create the directory. 4. Copy the zip file generated by the ExportCustomsReports.sh command from the source Avaya IQ system to the /tmp/reports_to_import directory on the target Avaya IQ system. Note that the Import tool can process only one zip file at a time. 5. Enter: cd /opt/avaya/ccr/jars/ui 6. Ensure that the ExportCustomsReports.sh and ImportCustomsReports.sh files and the directory runloadsampledata are present. 7. Enter the following command to back up the Export_Import.log file, if it exists on your system: mv Export_Import.log /<destination_directory_name> When you use the import utility for the first time, it generates the Export_Import.log file. 8. Before running the import utility for the first time, replace the string dataloadertemplate.properties with dataloader.properties in the script runloadsampledata.sh available at: April 2015 Avaya IQ High Availability and Survivability 35

36 Importing and exporting tools /opt/avaya/ccr/jars/ui/runloadsampledata/ 9. Enter one of the following commands: Example Enter: To import the reports to a specific directory on the Avaya IQ system, enter:./importcustomreports.sh user:<user_name> overwrite:<overwrite> folder:<destination_folder_path> To import the reports to the default directory on the Avaya IQ system, enter:./importcustomreports.sh user:<user_name> overwrite:<overwrite> For an Enterprise LDAP user, you must replace <USER_NAME> with the Fully Qualified User Name of that user. For example, you can specify username@domain.com as a value of <USER_NAME>. You can specify the OVERWRITE values as yes or no. If you specify yes, then the imported reports overwrites any existing reports with the same name. These values are not case sensitive. The reports are not imported if the user name does not exists. The default directory is the directory of the exported reports on the source Avaya IQ system. If you do not provide the destination path, then the destination path and the source path are the same../importcustomreports.sh user:steve overwrite:yes folder:/ DataExport/Agent Reports The reports are imported to the specified path. If you set the overwrite flag to yes, the existing reports with the same name are overwritten. Enter:./importCustomReports.sh user:steve overwrite:yes The reports are imported to the default path. The default path is the path of the exported reports on the source Avaya IQ system. For example, if you export a report from the Custom Reports directory from the source Avaya IQ system, the reports are imported to the Custom Reports directory on the destination Avaya IQ system. Next steps Validate that you can access reports from all reporting servers after you complete the importing steps. 36 Avaya IQ High Availability and Survivability April 2015

37 Troubleshooting exporting and importing data Troubleshooting exporting and importing data If a problem occurs with the import and export tools, see the following for initial troubleshooting. Problem Import tool does not import reports Export tool does not create a zip file for the exported folder Import succeeds with an error Solution Ensure that the name of the report you want to export does not contain: Leading and trailing spaces #, ~, or * characters Ensure that name of the zip file does not contain any spaces when you execute the./ exportcustomreports.sh command. Check the Export_Import.log file and verify that the errors are not generated during the last execution of the Import tool. If the errors are related to the previous execution of the Import tool, you can ignore this error. Remove or rename the Export_Import.log file prior to running the Import tool. This ensures that the Export_Import.log file generated after running Import tool contain data only for the current run. April 2015 Avaya IQ High Availability and Survivability 37

38 Chapter 7: Avaya IQ support for Communication Manager survivability Avaya IQ Support for Communication Manager Survivability Introduction Survivability in Communication Manager is a feature that allows remote servers to take control of some or all media gateways during Communication Manager fragmentation. Communication Manager fragmentation occurs when it loses communication with media gateways. Communication Manager supports the following Survivable servers: Survivable Core Servers, formerly known as Enterprise Survivable Server (ESS) Survivable Remote Servers, formerly known as Local Survivable Processor (LSP) Avaya IQ supports survivability of Communication Manager Release and later and uses the survivable core and survivable remote servers identically to support survivability. When an active Communication Manager system becomes fragmented, one or more Survivable servers carry some or all of the entire network load during the outage. During this outage, Avaya IQ can connect to the Survivable servers and continue normal operations. Avaya IQ supports Survivability by configuring Communication Manager Survivable server as a separate source. You can also use a standby Avaya IQ Data Collection host to collect data from the Survivable server during Communication Manager fragmentation. To ensure survivability, you must configure a Communication Manager connection using the OAM interface to link the Avaya IQ system to the Survivable server. When the main Communication Manager system is active, the link connecting Avaya IQ to the Survivable server is in Standby mode. When the main Communication Manager system becomes fragmented, it automatically activates the Survivable server. Avaya IQ then automatically detects this activation and changes the status of the link connecting Avaya IQ and the Survivable server from Standby to Active mode. 38 Avaya IQ High Availability and Survivability April 2015

39 Avaya IQ Support for Communication Manager Survivability Once the Survivable server is active, it starts sending operational data to the Avaya IQ Data Collection host. The Data Collection host provides a buffer to preserve the operational data if the WAN connectivity to the Avaya IQ system is lost. After the WAN connectivity is restored, the Data Collection host sends the buffered data from the Survivable server to the Avaya IQ system at the main site. In this case, reporting on the data from the Survivable server is not available until the WAN connectivity is restored. The data from the Survivable server is stored as a separate source. You can integrate this data with the main source data by using the Groups capability in Avaya IQ. Licensing The support for Survivability in Avaya IQ is licensed separately from the High Availability (HA) feature. You can administer a Survivable server only if you have the Survivable feature license. Deployment Considerations You can deploy survivability in the following ways: Survivability-only deployment Survivability with HA deployment Survivability with HA for a multi-site deployment Survivability-only deployment In this type of deployment, the main site consists of the main Communication Manager system and Avaya IQ, and the remote site consists of the Survivable server. As the Survivable server is at a different site than the main Avaya IQ system, Avaya recommends you to connect Avaya IQ and the Survivable server using a Data Collection host. The following figure shows a typical Survivability-only deployment: When the main Communication Manager system is active, it sends operational data to Avaya IQ. When the main Communication Manager system is active, the Avaya IQ Data Collection host is in Standby mode. When Communication Manager fragmentation occurs, the Survivable server starts sending operational events to Avaya IQ through the Avaya IQ Data Collection host. April 2015 Avaya IQ High Availability and Survivability 39

40 Avaya IQ support for Communication Manager survivability The Survivable server sends operational data to Avaya IQ at the main site in real time if WAN and the Avaya IQ is active. Since the Survivable server is administered as a different source than the main Communication Manager system, the operational data collected during fragmentation is stored in the database separate from the main Communication Manager source. Survivability with HA deployment In this type of deployment, the main data center consists of the main Communication Manager system and the primary Avaya IQ system. A secondary Avaya IQ Data Collection host is optional but recommended. Survivable (backup) Data Center consists of Survivable server, secondary Avaya IQ system, and survivable Avaya IQ Data Collection host. Avaya recommends that you deploy Survivability with HA. This type of deployment ensures maximum resiliency of the Avaya IQ system in different failure scenarios and also provides reporting and administration during outages. The following figure shows a Survivability with HA deployment. During normal operations, the main Communication Manager system sends the operational data to both the primary and secondary Avaya IQ system simultaneously. When Communication Manager fragmentation occurs, the Avaya IQ Data Collection host at the backup data center becomes active and sends operational data to the primary and secondary Avaya IQ systems ensuring survivability and high availability of the system. If you do not use a Data Collection host, you can synchronize the data for the for the downtime interval during the Communication Manager fragmentation, by manually running the data synchronization jobs. Multiple sites with Survivability The Survivability deployment with HA for a multi-site deployment is a combination of the typical Survivability-only deployment and the Survivability with HA deployment. 40 Avaya IQ High Availability and Survivability April 2015

41 Avaya IQ Support for Communication Manager Survivability In this deployment, the Avaya IQ Data Collection host is deployed with each additional Survivable Communication Manager Core (ESS). The Avaya IQ Data Collection host is in Standby mode while the main Communication Manager system is active. When Survivable server at the remote site becomes fragmented from main and backup data centers, the Survivable Communication Manager core starts sending operational events to Avaya IQ Data Collection host deployed at the main data center. If the remote site becomes fragmented due to failure in WAN connectivity, reporting on the data from the remote Survivable server is not available until the WAN connectivity is restored. April 2015 Avaya IQ High Availability and Survivability 41