Understand and Troubleshoot Hyper-V Replica in Windows Server "8" Beta

Transcription

1 Understand and Troubleshoot Hyper-V Replica in Windows Server "8" Beta Microsoft Corporation Published: February 2012 Abstract This Understanding and Troubleshooting Guide (UTG) enables you to learn technical concepts, functionality, and troubleshooting methods for Hyper-V Replica in Windows Server 8 Beta. This UTG provides you with: A technical overview and functional description of this feature. Technical concepts to help you successfully install, configure, and manage this feature. User Interface options and settings for configuration and management. Relevant architecture of this feature, with dependencies, and technical implementation. Primary troubleshooting tools and methods for this feature.

2 Copyright information This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes Microsoft. All rights reserved. Active Directory, Hyper-V, Microsoft, MS-DOS, Visual Basic, Visual Studio, Windows, Windows NT, Windows Server, and Windows Vista are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.

3 About the Author Author: Bio: Chuck Timon Chuck Timon has been with Microsoft for 13 years and is a Senior Support Escalation Engineer with Microsoft Commercial Technical Support (CTS) in Charlotte, North Carolina, USA. He specializes in High Availability and Virtualization (Hyper-V) technologies. Chuck also has credits in Microsoft Press books. He authors and provides training to Microsoft employees. He is a frequent contributor to the 'Ask The CORE Team' TechNet blog and is one of the moderators for the High Availability (Clustering) Windows Server TechNet forum.

4 Table of Contents Windows Server "8" Beta Understand and Troubleshoot Guide: Hyper-V Replica... 1 About The Understand and Troubleshoot Guide... 1 Introducing Hyper-V Replica... 1 Description... 1 Technical Overview... 3 Installing/Enabling Hyper-V Replica... 6 Installation Considerations... 6 Installation Process Configuring and Managing Hyper-V Replica Management Considerations Configuration and Management UI Configuration Settings Management Tasks Hyper-V Replica Architecture Key Modules/Components Architectural Diagram Troubleshooting Hyper-V Replica Introduction to Troubleshooting Hyper-V Replica Tools for Troubleshooting Hyper-V Replica General Methodology for Troubleshooting Hyper-V Replica Primary and Replica server connectivity issues Configuring a virtual machine for replication issues Virtual machine Failover issues Virtual machine Reverse Replication issues Virtual machine Initial Replication (IR) and Delta Replication (DR) issues Replication Broker issues Issues Implementing Guest IP functionality Appendix A - Hyper-V Replica PowerShell CMDLETS Appendix B Enabling the Firewall rules in the Replica Server Appendix C - Implementing Self-signed Certificates in Hyper-V Replica... 76

5 Windows Server "8" Beta Understand and Troubleshoot Guide: Hyper-V Replica About The Understand and Troubleshoot Guide Understand and Troubleshoot Guides (UTGs) enable customers to learn about technical concepts, functionality, and general troubleshooting methods for new Windows features and enhancements. The Understand and Troubleshoot Guide supports the user in developing an understanding of key technical concepts, architecture, functionality, and troubleshooting tools and techniques. This understanding will enable more successful beta testing and early adoption experiences during the pre-rtm product evaluation phase, and will support early ramp-up of help desk and technical support roles. Introducing Hyper-V Replica Description What Is Hyper-V Replica? Hyper-V Replica is new functionality added to the Hyper-V Role in Windows Server "8" Beta. Hyper-V Replica enables organizations to implement an affordable Business Continuity and Disaster Recovery (BCDR) solution for virtualized workloads. This allows virtual machines running at a primary site to be efficiently replicated to secondary location (Replica site) across a WAN link. Purpose/Benefits Hyper-V Replica provides a storage-agnostic and workload-agnostic solution that replicates efficiently, periodically, and asynchronously over IP-based networks, typically to a remote site. Hyper-V Replica allows a Hyper-V Administrator, in the event of a failure at a primary site (e.g. fire, natural disaster, power outage, server failure etc ), to execute a failover of Windows Server 1

6 production workloads to replica servers at a secondary location within minutes, thus incurring minimal downtime. The configurations at each site do not have to be the same with respect to server or storage hardware. Hyper-V Replica provides a System Administrator the option to restore virtualized workloads to a point in time depending on the Recovery History selections for the virtual machine. Hyper-V Replica provides the necessary management APIs that enable IT management vendors to build an enterprise class Disaster Recovery (DR) solution for their customers. Hyper-V Replica enables Infrastructure as a Service (IaaS) for hosting providers that host dedicated/virtual servers for their customers. With Hyper-V Replica, Hosters can provide solutions that offer DR as a service to their customers (specifically Small and Medium Business (SMB) customers). Here is a list of terms that are relevant to the Hyper-V Replica functionality: Term Recovery Time Objective (RTO) Recovery Point Objective (RPO) Application-consistent Replica Standard Replica Primary server Replica server Planned Failover Failover Definition The Recovery Time Objective (RTO) is the duration of time within which a business process must be restored after a disaster (or disruption) in order to avoid unacceptable consequences associated with a break in business continuity. This can also be referred to in business terms as a Service Level Agreement (SLA) The Recovery Point Objective (RPO) describes the acceptable amount of data loss measured in time Recovery to a point-in-time that is consistent from the application s perspective. To accomplish this, the Volume Shadow Copy Service (VSS) is used A crash consistent replica of the primary virtual machine The Hyper-V machine (Hyper-V Failover Cluster) that hosts virtualized production workloads The Hyper-V machine (Hyper-V Failover Cluster) that hosts the replica virtual machines for a Primary server A Planned Failover is a controlled event where an Administrator gracefully moves a virtual machine from a Primary site to a replica site A Failover is an unplanned event where the Primary site experienced a problem and replica virtual machines had to be brought Microsoft Corporation. All rights reserved.

7 Test Failover online at a replica site A process an Administrator executes on a replica virtual machine to verify it s functionality Technical Overview Prerequisites To take advantage of the Hyper-V Replica, which is included as part of the Hyper-V server role, the following pre-requisites must be met: Hardware that supports the Hyper-V Role on Windows Server "8" Beta Sufficient storage on both the Primary and Replica servers to host the files used by virtualized workloads Network connectivity between the locations hosting the Primary and Replica servers Properly configured firewall rules to permit replication between the Primary and Replica sites An X.509v3 certificate to support Mutual Authentication with certificates (if desired or needed) Functional Description Administrators can use Hyper-V Replica to provide a virtual machine level replication solution which efficiently replicates data over a LAN/WAN to a remote (Replica) site without relying on software or hardware technologies outside of the Windows Server "8" Beta operating system. At a very high level (details to follow in the Architecture section), Hyper-V Replica is summarized in the following: Windows Server 3

8 Replication Engine: The Replication Engine, in many respects, is the 'heart' of Hyper-V Replica. It manages the replication configuration details and handles initial replication, delta replication, failover, and test-failover operations. It also tracks virtual machine and storage mobility events and takes appropriate actions as needed (i.e. it pauses replication events until migration events complete and then resumes where they left off) Change Tracking: The Change Tracking module provides a virtual machine level change tracking mechanism on the primary server by keeping track of the write-operations, which happen in the virtual machine. This component is designed in such a way that it makes the scenario work irrespective of where the virtual machine VHD file(s) resides (VHD files can be hosted on Direct Attached Storage (DAS), a SAN LUN, an SMB share on a File Server, or a Cluster Shared Volume (CSV) Network Module: The Networking Module provides a secure and efficient (data compression by default) network channel to transfer virtual machine replicas between Primary and Replica site. Network communications are built on top of HTTP\HTTPS protocols and support integrated as well as certificate-based authentication with optional support for encryption Hyper-V Replica Broker role: The Hyper-V Replica Broker role is configured in a Windows Server "8" Beta Failover Cluster. This functionality supports seamless replication even in the event of a migration of a replica virtual machine from one cluster node to another. This is achieved by interacting with the Windows Server Failover Clustering (WSFC) service and the Hyper-V network module. The Hyper-V Replica Broker redirects all virtual machine Microsoft Corporation. All rights reserved.

9 specific events to the appropriate node in the replica cluster. The Broker queries the cluster database to determine which node should handle which events. This ensures all events are redirected to the correct node in the cluster in the event a Quick Migration, Live Migration or Storage Migration process was executed Management Experience: This includes the following components: o o o o o Hyper-V Manager UI: The replication settings are available in the Hyper-V Manager and provide an end-to-end experience for replication configuration, inbox monitoring, test failover, planned failover, unplanned failover and reverse-replication experiences Failover Cluster Manager UI: When Primary or Replica servers are part of a Hyper-V Failover Cluster, all management for the virtual machines and the Hyper-V Replica configurations should be done from the Failover Cluster Manager interface Scripting: Hyper-V Replica functionality is integrated within the Hyper-V PowerShell Module Hyper-V Replica APIs: These are part of the Hyper-V WMI interface. This interface can also be used by Third Party management software applications Remote Management: The Hyper-V Manager UI is included as part of the Remote Server Administration Tools (RSAT) that can be installed on supported Windows 8 Consumer Preview operating systems so administrators can remotely manage virtual machine replication Security Considerations There is no requirement for servers running the Hyper-V role to be part of an Active Directory domain unless those servers are part of a Failover Cluster. If the servers are part of an Active Directory domain, then that is a defined security boundary, and it is the responsibility of the Domain Administrator to configure domain-level security policies to protect the domain and to allow Hyper-V to function properly. Hyper-V Replica can also be implemented between un-trusted domains/workgroups. Security in Hyper-V is implemented at the following levels: Hyper-V uses a new Simple Authorization model where a Hyper-V Administrators group is created locally on each server running the Hyper-V role when the role is installed. The Authorization Manager is still present, and the Hyper-V Administrators group, along with members of the local administrators group in the server, is added by default Hyper-V administrators can configure Replica servers to accept incoming connections from specific servers, thus restricting access, using a Fully Qualified Domain Name (FQDN) (e.g. P1.contoso.com) or by using a wildcard attached to a domain suffix (e.g. *.contoso.com). If the Replica server is a Failover Cluster, security is enforced at the cluster level Windows Server 5

10 Firewall rules must be configured to allow incoming connections on the Hyper-V Replica servers for the designated port configured by the Hyper-V Administrator (Refer to Appendix B) Mutual Authentication is accomplished by using Integrated Authentication in an Active directory environment (Kerberos) between trusted domains. Certificates can be used outside of Active Directory. This would be common in an X-premise environment where a Hosting company is providing DR services to multiple tenants. Additional security is provided by way of a Replication Authorization Tag. Consider a scenario where a Hosting company is providing DR services to multiple Third Party Tenants (e.g. Contoso.com, Fabrikam.com and WoodgroveBank.com). Each of these tenants is using the same Replica Cluster at the Hosting Company's datacenter to host Replica virtual machines. A Replication Authorization Tag (Security Tag in the UI) is used to prevent one company from gaining access to another company's replica machines. An example configuration might look like this - Note: If the Primary site is running virtualized workloads in a Hyper-V Failover Cluster, then the Client Access Point (CAP) associated with the Hyper-V Replica Broker resource in the cluster would be used as the Primary Server. If Certificates were being used, the CAP associated with the Hyper-V Replica broker would also be required to have its own certificate. Installing/Enabling Hyper-V Replica Installation Considerations Hyper-V Replica is implemented as part of the Hyper-V Role. The Hyper-V Role can be enabled on a standalone Hyper-V server or on servers that are members of a Failover Cluster. Hyper-V servers can function as members of a Workgroup or as member servers in the same or different Active Directory domains. It is not required that Primary and Replica servers be part of the same Active Directory domain. If the Hyper-V server role is enabled on servers that are members of a Failover Cluster, those servers must be joined to the same Active Directory domain. The Hyper-V server role can be enabled using the Server Manager console or the Deployment Image and Servicing Management (DISM) command line tool. The Hyper- V role has no dependencies on any other server roles. Installing the Hyper-V role also installs the Hyper-V Management interface, which is part of the Remote Server Administration Tools (RSAT) feature Microsoft Corporation. All rights reserved.

11 Deployment Scenarios There are four primary deployment scenarios for Hyper-V Replica: Head Office and Branch Office (HO-BO) Enterprise Datacenter Hosting Provider Datacenter Customer Office and Hosting Provider Datacenter (Cross-Premise) Head Office and Branch Office This scenario typically involves Mid-Market customers who have a main corporate Head Office and one or more Branch Offices located in different physical locations. This type of customer typically has a limited budget for purchasing hardware, WAN connectivity, and hiring IT Staff. As part of a cost saving initiative, a customer may decide to implement Microsoft virtualization technologies to migrate corporate applications running on physical hardware to virtualized workloads running on Microsoft servers running the Hyper-V role. One or more of servers are hosted in the Head Office location either as standalone servers or as part of one or more Hyper-V Failover Clusters. Management decides to implement a Disaster Recovery (DR) plan and deploys additional servers running the Hyper-V role (Hyper-V Failover Clusters) at one or more Branch Offices to function as Replica servers. Using Microsoft's Hyper-V Replica functionality, the customer implements a DR solution. Implementing this DR solution involves these basic steps: 1. Configure a Replica Server (Replica Failover Cluster) at one or more designated Branch Offices 2. Create the replication relationships between the virtual machines running in the Head Office and the Branch Office(s) 3. Perform the initial replication for all virtual workloads Windows Server 7

12 4. Perform a Test Failover on each virtual machine ensuring it will start on the Branch Office server(s) 5. Configure the server(s) in the Head Office as Replica servers as part of a reverse replication strategy for the Branch Office Replica servers 6. Execute a Planned Failover of all virtual machines to the Branch Office(s) to ensure all virtual machines come online properly (Note: A Planned Failover requires a Virtual Machine be shutdown first resulting in an outage if it is a production workload) 7. Execute another Planned Failover back to the Head Office from the Branch Office(s) to verify reverse replication 8. Monitor replication for all virtual machines and generate reports as needed 9. Test virtual machine mobility scenarios to ensure they function properly. These scenarios can include live migration of virtual machines between cluster nodes, or migration of virtual machine storage between File Servers 10. Test replication using virtual machines that have been restored from backup 11. Modify corporate maintenance (Change Management) plans to include using Hyper-V Replica functionality Enterprise Datacenter The Enterprise Datacenter scenario is very similar to the Head Office and Branch Office scenario in terms of the actual steps an administrator executes to implement Disaster Recovery using Hyper-V Replica. The main difference would be scale. Enterprise environments typically include one, or more, large, geographically dispersed datacenters supporting a greater number of virtualized workloads running on more servers. Additionally, enterprise environments may implement Third Party or 'homegrown' applications that take advantage of Hyper-V Replica APIs (Application Programming Interface) in an effort to streamline internal management processes. Hosting Provider Datacenter The Hosting Provider Datacenter scenario is very similar to the Enterprise scenario in terms of the actual steps an administrator executes to implement Disaster Recovery using Hyper-V Replica. Hosting companies have additional concerns in that they are dealing with multiple customers (tenants) on a shared internal infrastructure. This requires implementing stricter isolation policies within the datacenter and a billing system that can accurately track resource usage Microsoft Corporation. All rights reserved.

13 Customer Office and Hosting Provider Datacenter (Cross-Premises) The Customer Office and Hosting Provider Datacenter (Cross-Premises) scenario takes the Hosting Provider Datacenter scenario one-step further in that Disaster Recovery is provided as a service (Infrastructure as a Service (IaaS)) to customers external to the hosting company itself. This modifies the administrative process as follows: Windows Server 1. Hosting company configures a Replica Server (Replica Failover Cluster) in the Hosting company datacenter 2. Hosting company develops a customer portal providing access to customers for tracking their services and adding services as needed 3. Customer subscribes to hosting company disaster recovery service 4. Hosting company provides required server and certificate information. Customer creates the replication relationships between the virtual machines running in the customer environment and the designated Replica server(s) in the hosting company's datacenter (using a secure Virtual Private Network (VPN) connection) 5. Customer performs the initial replication or all virtual workloads 6. Customer performs a Test Failover on each virtual machine ensuring it will start on the Replica server(s) 7. Customer configures their server(s) as part of a reverse replication strategy on their own premise server(s) 8. Customer executes a Planned Failover of all virtual machines to the hosting company datacenter Replica server(s) to ensure all virtual machines come online properly 9. Customer executes another Planned Failover back to their on premise server(s) to verify reverse replication 9

14 10. Customer monitors replication for all virtual machines and generates reports as needed 11. Customer tests virtual machine mobility scenarios provided hosting company provides those services 12. Customer and hosting company test replication using virtual machines that have been restored from backup 13. Customer modifies maintenance (Change Management) plans to include using Hyper-V Replica functionality Microsoft Corporation. All rights reserved.

15 Installation Process Hyper-V Replica is installed as part of the Hyper-V Role. Installation UI/Wizard To add the Hyper-V Role to a server installation, use the Add Role and Feature Wizard (ARFW), which is part of Server Manager or use the Deployment Image and Servicing Management (DISM) command line tool. A reboot is required. This can occur automatically if the installation is being executed in the Server Manager interface by checking a box (Restart each target machine automatically if needed) in the wizard (Confirmation screen). This allows the server to reboot, if needed. Installation Command Line Interface (CLI) The Hyper-V Role can also be installed from the command line interface by using the Deployment Image and Service Management (DISM) command line tool. At a command prompt, type: dism /online /enable-feature /featurename: Microsoft-Hyper-V. A reboot is required to complete the installation. The Server Manager PowerShell cmdlet Install- WindowsFeature can also be used. PowerShell: Install-WindowsFeature -Name Hyper-V -IncludeManagementTools When adding the Hyper-V Role using the ARFW, there is an opportunity to create Virtual Switches in Hyper-V. This step can be accomplished in the wizard when installing the Role or it can be done afterwards using the Hyper-V Manager interface. Windows Server 11

16 You can allow the Hyper-V server to receive Live Migrations You can also specify the default location for virtual hard disks and virtual machine configuration files Microsoft Corporation. All rights reserved.

17 A reboot is required to complete the configuration of the Hyper-V role. Verifying Installation Once the installation is completed (following a reboot of the server), the Hyper-V role will show as installed in Server Manager and the Hyper-V Manager will be listed under Administrative tools. To verify proper functioning of Hyper-V, run the Hyper-V Best Practices Analyzer (BPA) and then create a new virtual machine and verify it starts correctly. Windows Server 13

18 Uninstalling/Disabling To remove the Hyper-V Role, use the Remove Roles and Features Wizard (RRFW) in Server Manager, use the Deployment Image and Servicing Management (DISM) command line tool, or use the Server Manager PowerShell Cmdlet Uninstall-WindowsFeature. A reboot is required to complete the process. Configuring and Managing Hyper-V Replica Management Considerations Hyper-V servers are managed locally using the Hyper-V Manager interface or remotely using the Hyper-V Remote Server Administration Tools (RSAT) or System Center Virtual Machine Manager (SCVMM). If the configuration is a Hyper-V Failover Cluster, the Failover Cluster Management interface is used. Configuration and Management UI Hyper-V Replica settings are configured in the Hyper-V Manager interface for standalone Hyper-V servers and the Failover Cluster Manager interface for Hyper-V Failover Clusters. Configuration Settings Configuring a Standalone Hyper-V Replica Server 1. In the Hyper-V Manager interface, Click on Hyper-V Settings in the Actions pane 2. In the Hyper-V Settings dialog box, Click on Replication Configuration 3. In the Details pane, Select Enable this computer as a replica server 4. Choose an Authentication method to include the port that will be used (The default ports are 80 for HTTP and 443 for HTTPS). There are default firewall rules already in place that just need to be enabled when replication is enabled on the Hyper-V server. Important: Review Appendix B to enable the Firewall rules 5. Configure Authorization and storage. This includes designating a specific location to store replica virtual machine files if the default location is not to be used. Should you not desire to allow all Primary servers to be serviced, there is an option to allow only specific servers (Primary servers) to send replication requests. If you want to allow all servers within the domain, a wildcard character can be used (e.g. *.contoso.com). When using a wildcard, only one storage location can be specified. If individual server entries are used, different storage locations for replica files can be configured. Complete all entries for the Primary Server, Storage Locations, and Security Tag information. Click Apply or OK when finished Microsoft Corporation. All rights reserved.

19 An example configuration is shown here: Configuring a Hyper-V Replica Failover Cluster A Failover Cluster consists of multiple physical servers also known as nodes. Each node must have the Hyper-V role installed as well as the Failover Clustering feature. Installation of the Hyper-V role has already been covered. To install the Failover Clustering feature: 1. In Server Manager, start the Add Roles and Features Wizard (ARFW) by first choosing the Dashboard view and then choose Add Roles. 2. Step through the ARFW choosing the local node and making the following choices: a. Installation Type: Role-based or Feature-based installation b. Server Selection: Choose local node c. Server Roles: Make no selections and click Next Windows Server 15

20 d. Features: Choose Failover Clustering e. Confirmation: Review the information and click Install (a reboot is not required when installing the Failover Clustering feature) f. Results: Ensure the result is a successful installation of the Failover Clustering feature. 3. Complete this action across all nodes that will be members in the cluster Note: The Failover Clustering feature can also be installed from the command line interface (CLI) by using the Deployment Image and Service Management (DISM) command line tool. At a command prompt, type: dism /online /enable-feature /featurename:failover-clustering. A reboot is not required to complete the installation. After the Failover Clustering feature is installed and a Failover Cluster is created, the Hyper-V Replica Broker Role needs to be configured. To create the Hyper-V Replica Broker role: 1. Open the Failover Cluster Manager interface (located under Tools in the Server Manager Menu bar) 2. In the left-pane, connect to the cluster 3. With the name of the cluster highlighted in the left-hand pane, click on Configure Role in the Actions pane 4. The High Availability Wizard initializes 5. Select Hyper-V Replica Broker in the Select Role screen 6. Complete the wizard by providing a properly formatted NetBIOS name and IP address (as needed) which serves as the connection point (Client Access Point (CAP))when configuring virtual machines for replication 7. Ensure the Role comes Online and is able to failover between all nodes in the cluster Configuring the Failover Cluster as a Replica cluster With the Hyper-V Replica Broker Role configured in the cluster, the cluster can be configured as either a Primary cluster or a Replica cluster. This is accomplished using the Highly Available Hyper-V Replica role. To accomplish this: 1. Open the Failover Cluster Manager interface (located under Tools in the Server Manager Menu bar) 2. In the left-pane, connect to the cluster 3. With the name of the cluster highlighted in the left-hand pane, click on Roles in the details pane under the Navigate category 4. Right-click on the Role and choose Replication Settings Microsoft Corporation. All rights reserved.

21 5. In the Hyper-V Replica Clustering Broker Configuration screen, make the appropriate selections as needed to configure the cluster as a Replica server. This action implements the configuration across all nodes in the cluster. Keep in mind this does not automatically enable the correct firewall rules in each node of the cluster nor does it properly configure authentication if certificates are being used. Those actions will have to be completed by the administrator on each node in the cluster. When using certificate based authentication in a Failover Cluster, all of the cluster nodes and the Client Access Point (CAP) supporting the Hyper-V Replica role will need machine certificates. Enabling a Virtual Machine for Replication Replication is configured on a per virtual machine basis. Virtual machines running on Primary site Hyper-V servers (or Hyper-V Failover Clusters) are configured to replicate to Replica servers (or Replica Failover Clusters) at remote sites. 1. In Hyper-V Manager, in the Details pane, Click on a virtual machine 2. With a virtual machine selected, Right-click and choose Enable Replication. This starts the Enable Replication wizard Windows Server 17

22 3. In the Specify Replica Server screen, enter either the NetBIOS or the Fully-Qualified Domain Name (FQDN) for the Replica Server in the Replica Server box. (Note: If the Replica server is a Failover Cluster, ensure the connection point for the Hyper-V Replica Broker is used). Click Next 4. If Remote WMI is enabled, then Hyper-V Replica automatically fills the port and authentication settings 5. In the Choose Replication VHDs screen, deselect the disk(s) you do not want to replicate for the virtual machine and then click Next 6. Replication changes are sent to a Replica server every 5 minutes. In the Configure Recovery History screen, make selections for the number and types of recovery points to be sent to the Replica server. If Only the latest point for recovery is chosen, then there will be only the parent VHD that is sent during Initial Replication and all changes Microsoft Corporation. All rights reserved.

23 are merged into that VHD. If Additional recovery points is chosen, set the number of desired additional recovery points (Standard Replicas) that will be saved on the Replica server. The final selection in this screen, Replicate incremental VSS copy every <select hours>: (available only if Recovery History is enabled) can be used to save Application-consistent Replicas for the virtual machine on the Replica server. After completing the selections, Click Next 7. In the Choose Initial Replication Method screen, several methods can be used to perform an initial replication for the virtual machine to the Replica server. The default selection is to Send initial copy over the network. This starts replication immediately over the network to the Replica server. If immediate replication is not desired, it can be scheduled to occur at a specific time on a specific date. If the administrator does not want to consume the bandwidth due perhaps to the large amount of data that needs to be replicated, he can choose Send initial copy using external media. This method copies all the virtual machine data to a portable storage device that can then be sent to a Replica site and the initial replication can be completed there by importing the data. The final option available to an administrator is Use an existing virtual machine on the Replica server as the initial copy. A restored virtual machine or snapshot of a virtual machine on the Replica server will be used as the initial copy. Make a selection and then Click Next Windows Server 19

24 8. In the Completing the Enable Replication Relationship Wizard, review the information in the Summary and then Click Finish 9. A pop-up dialog box is presented indicating replication was successfully enabled for the virtual machine. In the dialog box is a Settings button that can be used to configure the network the replica virtual machine will connect to on the Replica server. This can be useful because the replica virtual machine is not connected to any network on the Replica server by default The Enabling Replication process will start and should complete successfully (i.e. a virtual machine is created on the Replica server and delta replication commences. If the Replica server is a Failover Cluster, the virtual machine is made highly available). If any errors occur, refer to the troubleshooting section later in this document Microsoft Corporation. All rights reserved.

25 Once the replication relationship is established and if Send initial copy over the network was selected (Default setting), the Status column for the configured virtual machine will indicate Sending Initial Replica. Progress as a percentage of the total replication is also displayed. Note: If a virtual machine is being replicated to a Failover Cluster (using the Hyper-V Replica Broker), a highly available virtual machine is created in the cluster. This allows the replica to be migrated to other nodes in the cluster if needed. If Send initial copy using external media was selected, the Status for the virtual machine will indicate an Initial Replication in Progress. In this case, the progress reflects the copying of the virtual machine files to a location other than the Replica Server (perhaps a portable storage device) so the files can be transported to the Replica server location and then the Initial Replication can be completed by importing the files on that server. Once the import process completes, the Replication State for the virtual machine in the Primary Site will reflect a new Replication State of Replication Enabled. If Use existing restored virtual machine as initial copy was selected, the virtual machine files restored from a backup on the Replica server will be used for Initial Replication (IR). Before delta replication actually begins, a quick resynch is executed. Hyper-V Replica implements the standard Hyper-V virtual machine folder structure on the Replica server in the location specified when the Replica server (Replica Failover Cluster) is configured. A Hyper-V Replica folder is created as the root folder in the specified location. Under the Hyper-V Replica folder is a series of subfolders as seen here - Windows Server 21

26 More Information: The Snapshots folder is not created unless Recovery History is configured for the virtual machine. The folders support the necessary files for each virtual machine configured to replicate with the Replica server. The virtual machines are identified by Globally Unique Identifiers (GUID) as seen here - Hyper-V Failover Cluster as a Primary or Replica Server Failover Clustering has proven its value in making virtualized workloads highly available. We saw this in Windows Server 2008 using Quick Migration and then in Windows Server 2008 R2 with the addition of Live Migration. Failover Clustering can also play an important role as a Replica Cluster. To accommodate this, a new role has been added in Failover Clustering called the Hyper-V Replica Broker. A new resource type, Virtual Machine Replication Broker, was added to support this new Role Microsoft Corporation. All rights reserved.

27 Failover Replication Broker Architecture The Hyper-V Replica Broker runs in a Replica cluster and provides a Replica server name (connection point (a.k.a. Client Access Point (CAP))) for initial virtual machine placement when contacted by a Primary server. After a virtual machine is initially replicated to the Replica Cluster, the Hyper-V Replica Broker provides the virtual machine to Replica Server (cluster node) mapping to ensure the Primary server can replicate data for the virtual machine to the correct node in the cluster in support of mobility scenarios on the Replica side (e.g. Live\Quick Migration, or Storage Migration). The Hyper-V Replica Broker is used to configure the replication settings for all nodes in the cluster. In standalone Hyper-V servers, the Hyper-V Manager is used to configure replication settings. The Failover Cluster Manager is used to configure replication settings in the Replica cluster. Using the Hyper-V Replica Role, Replication Settings across the entire cluster are set. Windows Server 23

28 The replication settings are the same as those for standalone Hyper-V servers. Network Considerations for Hyper-V Replica Scenarios There are scenarios where the Replica server, or Replica cluster, will reside at a Disaster Recovery (DR) site located across a WAN link and the DR site uses a completely different network-addressing scheme than the Primary site. In this configuration, when virtual machines are failed over to a DR site, a new IP configuration will be needed for each network configured in the virtual machine. To accommodate this scenario, there is built-in functionality in Hyper-V Replica where virtual machines network settings can be modified to include configuration information for a different network at a DR site. To take advantage of this, the Hyper-V Administrator must modify the network configuration for each replicated virtual machine on the Replica server. If connectivity to networks at the replica site is required, the settings for all networks a virtual machine is connected to must be modified. The Hyper-V Administrator can provide both IPv4 and IPv6 configuration information for a virtual machine. The Failover TCP/IP setting, which is available after replication is enabled for the virtual machine, is used to provide the network configuration information in the virtual machine Microsoft Corporation. All rights reserved.

29 The addressing information provided is used when a Failover action (Planned Failover or Failover) is executed. The configuration of the Guest virtual machine IP settings in this manner only applies to Synthetic Network Adapters and not Legacy Network Adapters. The operating system running in the Guest virtual machine must be one of the following - Windows Server "8" Beta, Windows Server 2008 R2, Windows Server 2008, Windows Server 2003 SP2 (or higher), Windows 7, Vista SP2 (or higher), and Windows XP SP2 (or higher). The latest Windows Server "8" Beta Integration Services must be installed in the virtual machine. The information is reflected in the virtual machine configuration file located on the Replica server. Windows Server 25

30 Management Tasks The Hyper-V Manager interface is used to manage standalone Hyper-V Primary, Replica servers, and the virtualized workloads running on those servers. The Failover Cluster Manager interface is used if the Primary or Replica servers are part of a Hyper-V Failover Cluster. Hyper-V Replica management tasks can be categorized as follows: Note: Hyper-V Server Primary Site Management Tasks Hyper-V Server Replica Site Management Tasks Virtual Machine Primary Site Management Tasks Virtual Machine Replica Site Management Tasks Modifying Virtual Machine Replication Settings In the above list, Hyper-V Failover Cluster can be substituted for 'Hyper-V Server'. Hyper-V Server Primary Site Management tasks involving the Hyper-V Server at a Primary Site include: Ensure the Hyper-V server (Hyper-V Failover Cluster) at the Primary site is configured as a Replica server to support Reverse Replication for a Planned Failover event To configure the Hyper-V server at the Primary site as a Replica server: 1. In the Hyper-V Manager interface, Click on Hyper-V Settings in the Actions pane 2. In the Hyper-V Settings dialog box, Click on Replication Configuration 3. In the Details pane, Select Enable this computer as a Replica server 4. Choose an Authentication method to include the port that will be used (if not using the default port) 5. Configure Authorization and storage. This includes designating a specific location to store replica virtual machine files if the default location is not to be used. Should you not desire to allow all Hyper-V Primary servers to be serviced, you have the option to allow only specific Hyper-V servers (Primary servers) to send replication requests. Click Apply or OK when finished Note: In a Replica cluster, use the Hyper-V Replica Broker role to configure the cluster nodes for replication. Monitor the Replication Health of virtual machines configured for replication Microsoft Corporation. All rights reserved.

31 To monitor the Replication Health of a virtual machine configured for replication: 1. Open Hyper-V Manager 2. In the details pane, Right-click on one of the Column Headings and select Add\Remove Columns 3. Choose Replication Health in the Available Columns list, click Add to move it to the Displayed Columns list 4. Move the new column to the desired location in the listing and click OK Monitor Hyper-V Replica specific Performance counters using Performance Monitor To monitor Hyper-V Replica performance: 1. Click the Start button, then click Run and type perfmon.msc and press ENTER 2. In the navigation tree, expand Monitoring Tools, and then click Performance Monitor 3. In the menu bar above the Performance Monitor graph display, either click the Add button (+) or right-click anywhere in the graph and click Add counters from the menu. The Add Counters dialog box opens 4. In the Available Counters section, select counters to view in the Performance Monitor display. The counters for Hyper-V Replica are virtual machine specific and are listed under Hyper-V F Counter VM 5. Choose the desired counters and instances (virtual machines) then click the Add button to add the counters 6. When finished, click OK For more information Performance Monitor, visit the Performance Monitor Getting Started Guide ( on TechNet. Evaluate Hyper-V Replica log data using the Microsoft-Windows-Hyper-V- VMMS\Admin log To review Hyper-V Replica log data: 1. In the Server Manager Menu Bar, Click on Tools and choose Event Viewer from the list 2. In the navigation tree, expand Application and Services Logs, expand Microsoft, expand Windows, expand Hyper-V-VMMS 3. Click on Admin Windows Server 27

32 Hyper-V Replica event messages are registered in the Hyper-V-VMMS channel. Hyper-V Server Replica Site Management tasks involving the Hyper-V Server at a Replica Site include: Ensure the Hyper-V server (Hyper-V Failover Cluster) at the Replica site is configured as a Replica server To configure the Hyper-V server at the Primary site as a Replica server: 1. In the Hyper-V Manager interface, Click on Hyper-V Settings in the Actions pane 2. In the Hyper-V Settings dialog box, Click on Replication Configuration 3. In the Details pane, Select Enable this computer as a Replica server 4. Choose an Authentication method to include the port that will be used (if not using the default port) 5. Configure Authorization and storage. This includes designating a specific location to store replica virtual machine files if the default location is not to be used. Should you not desire to allow all Hyper-V Primary servers to be serviced, you have the option to allow only specific Hyper-V servers (Primary servers) to send replication requests. Click Apply or OK when finished Note: In a Replica cluster, use the Hyper-V Replica Broker role to configure the cluster nodes for replication. Monitor the Replication Health of virtual machines configured for replication To monitor the Replication Health of a virtual machine configured for replication: 1. Open Hyper-V Manager 2. In the details pane, Right-click on one of the Column Headings and select Add\Remove Columns 3. Choose Replication Health in the Available Columns list, click Add to move it to the Displayed Columns list 4. Move the new column to the desired location in the listing and click OK Monitor Hyper-V Replica specific Performance counters using Performance Monitor To monitor Hyper-V Replica performance: Microsoft Corporation. All rights reserved.

33 1. Click the Start button, then click Run and type perfmon.msc and press ENTER 2. In the navigation tree, expand Monitoring Tools, and then click Performance Monitor 3. In the menu bar above the Performance Monitor graph display, either click the Add button (+) or right-click anywhere in the graph and click Add counters from the menu. The Add Counters dialog box opens 4. In the Available Counters section, select counters to view in the Performance Monitor display. The counters for Hyper-V Replica are virtual machine specific and are listed under Hyper-V Replica Counter VM 5. Choose the desired counters and instances (virtual machines) then click the Add button to add the counters 6. When finished, click OK For more information Performance Monitor, visit the Performance Monitor Getting Started Guide ( on TechNet. Evaluate Hyper-V Replica log data using the Microsoft-Windows-Hyper-V- VMMS\Admin log To review Hyper-V Replica log data: 1. In the Server Manager Menu Bar, Click on Tools and choose Event Viewer from the list 2. In the navigation tree, expand Application and Services Logs, expand Microsoft, expand Windows, expand Hyper-V-VMMS 3. Click on Admin Hyper-V Replica event messages are registered in the Hyper-V-VMMS channel. Virtual Machine - Primary Site Management tasks involving virtual machines at the Primary Site include: Windows Server Planned Failover - This action initiates a failover of a virtual machine from a Primary to a Replica server. This is a 'planned' event as opposed to a Failover action, which is unplanned. Since it is a 'planned' event, there should be no data loss. This action executes a series of checks prior to executing the failover. One check determines if the Primary server has also been configured as a Replica server. This is done because the assumptions are first, the virtual machine being failed over to a Replica server will eventually be moved back to the Primary server and second, the Primary server will become the Replica server for the virtual machine that is being failed over. This action 29

34 provides an Administrator the flexibility to execute the failover of a virtual machine to a replica server in a controlled manner before a disaster occurs To execute a Planned Failover for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Planned Failover 4. A list of Pre-Requisites and Actions is presented. If the Virtual Machine has not been shut down and the Primary Server not configured as a Replica Server, complete those tasks before proceeding. By default, Start the replica virtual machine after Failover is checked (uncheck if this is not the desired action for the virtual machine after a Planned Failover completes) 5. Click on the Failover button. 6. If the Failover is successful, a pop-up dialog box appears reporting the Failover completed successfully (Note: If the option to start the virtual machine after the Planned Failover was left checked, then the virtual machine will be started on the Replica server). Close the dialog box. 7. If the Planned Failover does not complete successfully, review the information contained in the General Methodology for troubleshooting the virtual machine Failover process in the troubleshooting section. Pause Replication - This action pauses replication for the selected virtual machine. The Replication Health column in the Hyper-V Manager interface (if selected for display) reflects a Warning Status To Pause Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine that is not paused 3. With the virtual machine selected, Right-click and select Replication and then choose Pause Replication 4. The Replication Health, if visible, in Hyper-V Manager, will be updated and the Replication Health for the virtual machine will indicate a Warning. The State column still shows the Virtual Machine as Running 5. Right-click on the Virtual Machine, select Replication and then click on View Replication Health. The health report reflects an accurate Replication State which should be Replication Paused Microsoft Corporation. All rights reserved.

35 Resume Replication (Available only if replication has been paused for a virtual machine) - This action resumes replication for the selected virtual machine (the action must be executed in the same site where replication was Paused). The Hyper-V Replica Network Services component re-establishes a connection to the Replica server (if needed) and replication resumes. If the virtual machine was in a Resynch Required state, Resume Replication performs a resynchronization. A resynchronization essentially compares blocks between the Primary and Replica VHDs and then sends the delta blocks to the Replica. Scenarios where this can happen include, but may not be limited to, a failure occurred on the Primary server when changes were being made to the replication log or, if the Primary is a Failover Cluster, an unplanned cluster failover occurred. The Replication Health column in Hyper-V Manager interface (if selected for display) reflects a status of Normal To Resume Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a paused virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Resume Replication 4. The Replication Health, if visible, in Hyper-V Manager, is updated and the Replication Health for the virtual machine is Normal View Replication Health - This action provides data about the replication events for a virtual machine. A sample Replication Health Report on a Primary virtual machine is Windows Server 31

36 shown here - A Replication Health Report can be saved as a CSV file. A Replication Health Report indicates if it is being viewed as either a Primary or a Replica virtual machine (see a sample of a Replication Health Report on a Replica virtual machine later in this guide) To view Replication Health for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose View Replication Health 4. The Replication Health Report for the virtual machine is displayed. The report can be saved as a CSV file by clicking on Save as Button Remove Replication - This action stops replication for the virtual machine. All connections for the virtual machine to the Replica server are terminated. The Replication Health in Hyper-V Manager on the Primary server, if selected for viewing, is Not Applicable. A corresponding action must be accomplished on the Replica server. Failure to execute this same action on the Replica server will result in errors should a Hyper-V Administrator attempt to re-enable replication for the virtual machine (more information is provided in the Troubleshooting section) Microsoft Corporation. All rights reserved.

37 To Remove Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and choose Remove Replication 4. Acknowledge the pop-up Warning by clicking on Remove Replication 5. The Replication Health column, if displayed, indicates Not Applicable for the virtual machine 6. Connect to the Replica server and execute Steps 1-5. This will remove replication for the virtual machine on the Replica server and will initiate a merge for all the replica information for the virtual machine 7. The Replication Health column, if displayed, indicates Not Applicable for the virtual machine 8. Additional cleanup action is required on the Replica server. In Hyper-V Manager, Right-click on the virtual machine and choose Delete. Acknowledge the pop-up Warning by clicking on Delete. This removes the virtual machine reference in Hyper-V Manager. Some data files remain on the Replica server in the storage location specified for the replication data. To recover storage space, manually remove the data. Enable Replication (Available only if replication is not enabled for a virtual machine) - This action enables replication for a virtual machine Windows Server To Enable Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Enable Replication 4. Review the information in the Before You Begin screen and click Next 5. In the Specify Replica Server screen, provide the name for the Replica Server using the Fully Qualified Domain Name (FQDN) or NetBIOS format. There is also the option to Browse Active Directory for the server. If the Replica Server is configured correctly, the Specify Connection Parameters screen is populated. If not, an error is registered and an option to Configure Server is available to configure the server to be a Replica Server. If data compression is not desired, Uncheck the box Compress the data that is transmitted over the network. Click Next 6. In the Choose Replication VHDs screen, ensure all disks to be replicated are Checked and then click Next (i.e. uncheck those disk you do not want replicated. An example might be a disk functioning as a repository for the 33

38 virtual machine page file) 7. In the Configure Recovery History screen, select as desired. For an explanation of the options, review the section on Enabling a virtual machine for replication. Click Next 8. In the Choose Initial Replication Method screen, select as desired. For an explanation of the options, review the section on Enabling a virtual machine for replication. Click Next 9. Review the information in the Summary screen, and click Finish Once replication has been enabled for a virtual machine, the Replication Health column, if visible, in Hyper-V Manager will be updated. Once the Initial Replication (IR) has been completed, the Replication Health for a virtual machine will be Normal. Virtual Machine - Replica Site Management tasks involving virtual machines at the Replica Site include: Failover - This action executes a process that starts a virtual machine on the Replica server using a replica (Recovery Point) selected by the Hyper-V Administrator. This is an unplanned event unlike the Planned Failover action, which is a planned event. Executing a Failover for a virtual machine could result in data loss depending on which recovery point is selected To Failover a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Failover 4. In the next screen, choose a recovery point from the drop-down listing of all the recovery points associated with the virtual machine and then click Failover 5. The virtual machine starts and the Replication Health indicates Warning. If the Primary server remains the same, the Replication Health for the virtual machine that was recovered is also Warning. To complete the process and remove the Warnings, either Cancel Failover or configure Reverse Replication and allow Initial Replication to complete. If a new Replica Server is needed, configure Reverse Replication to the new Replica server. Test Failover - This action allows a Hyper-V Administrator to test a virtual machine on the Replica server without interrupting the production workload running on the Primary server. The network configuration for the test virtual machine is disconnected by default so as not to interfere with the production workload. If network connectivity Microsoft Corporation. All rights reserved.

39 is to be tested, the recommendation is to create a separate test network and connect the test virtual machine to that network. The virtual machine created and started has the same name as the original virtual machine with a modifier of Test added on to the end To start a Test Failover for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Test Failover 4. In the next screen, choose a recovery point from the drop-down listing of all the recovery points associated with the virtual machine and then click Test Failover. A new test virtual machine is created but is not started. At this point, the virtual machine can be started and then a connection can be made to the virtual machine and a verification process can be completed Stop Test Failover (Available only if a test is already running for the selected virtual machine) - This action stops a test that is in progress for the selected virtual machine. The virtual machine is stopped and deleted from Hyper-V Manager (Note: If the Test Failover is being executed on a Replica cluster, the Test-Failover role that is created in Failover cluster Manager will have to be manually deleted) To stop a Test Failover for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select the virtual machine being tested, not the test virtual machine that is running. Right-click on the virtual machine, select Replication and then choose Stop Test Failover 3. The test virtual machine is stopped if it is running and is removed from Hyper-V Manager as the test is completed Pause Replication - This action pauses replication for the selected virtual machine. The Replication Health column in Hyper-V Manager, if selected for viewing, indicates a Warning To Pause Replication for a virtual machine: 1. Open Hyper-V Manager Windows Server 35

40 2. In the details pane, select a virtual machine that is not paused 3. With the virtual machine selected, Right-click and select Replication and then choose Pause Replication 4. The Replication Health, if visible, in Hyper-V Manager, is updated and indicates a Warning Resume Replication (Available only if replication has been paused for a virtual machine on the Replica server) - This action resumes replication for the selected virtual machine. If a 'resynch' is required for the virtual machine, that action will be initiated on the Primary server. The Replication State column, if selected for viewing in Hyper- V Manager, indicates Replication Enabled To Resume Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a paused virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Resume Replication 4. The Replication Health, if visible, in Hyper-V Manager, is updated and indicates Normal View Replication Health - This action provides data about the replication events for a virtual machine. A sample Replication Health Report for a Replica virtual machine is shown here Microsoft Corporation. All rights reserved.

41 A Replication Health Report can be saved as a CSV file. A Replication Health Report indicates if it is being viewed on either a Primary or a Replica server To View Replication Health for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose View Replication Health 4. The Replication Health Report for the virtual machine is displayed. The report can be saved as a CSV file by clicking on Save as Remove Replication - If a Remove Replication action is executed on the Replica server, a corresponding action must be executed on the Primary Server. This action stops replication for the virtual machine. Prior to re-enabling replication, the virtual machine must be deleted in Hyper-V Manager on the Replica server. This destroys the virtual machine on the Replica Server. If the virtual machine is not deleted, a Replication error is reflected in Hyper-V Manager and associated error logs are registered (more information is provided in the Troubleshooting section) To Remove Replication for a virtual machine: 1. Open Hyper-V Manager Windows Server 37

42 2. In the details pane, select a virtual machine 3. With the virtual machine selected, Right-click and select Replication and then choose Remove Replication 4. Acknowledge the pop-up Warning by clicking on Remove Replication 5. The Replication Health column, if displayed, indicates Not Applicable 6. A process will start to merge all recovery point data into the VHD that was initially replicated 7. Connect to the Primary server. In Hyper-V Manager, the virtual machine Replication Health indicates Normal 8. Execute Steps 1-5. This removes replication for the virtual machine on the Primary server 9. The Replication Health column, if displayed, now indicates Not Applicable on the Primary server. If Step 8 is not accomplished before the next 5 minute replication interval, Replication Health will indicate Critical 10. Additional cleanup action is required on the Replica server. In Hyper-V Manager, Right-click on the virtual machine and choose Delete. Acknowledge the pop-up Warning by clicking on Delete. This removes the virtual machine reference in Hyper-V Manager. Some data files remain on the Replica server in the storage location specified for the virtual machine. To recover storage space, manually remove this data Cancel Failover - This action is available if a Failover action was executed for a virtual machine. This allows a Hyper-V Administrator to cancel the Failover action if, for example, he decides the recovery point chosen was not the desired one. After cancelling the Failover, another recovery point can be selected and another Failover process initiated. A Failover can only be cancelled if the virtual machine state is Failed over - Waiting Completion. If a Reverse Replication has been completed, the Failover can no longer be cancelled To Cancel Failover for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine that has a Replication Health of Warning but viewing the Health Report shows Failover Complete 3. With the virtual machine selected, Right-click, select Replication and then choose Cancel Failover 4. Acknowledge the pop-up Warning by clicking on Yes. On the Replica server, the virtual machine is turned off and the Replication Health indicates Normal. On the Primary server, the Replication Health shows Critical. 5. To clear the Critical health state, right-click on the virtual machine and select Replication and then choose Resume Replication Microsoft Corporation. All rights reserved.

43 Reverse Replication - This action is available if a Failover action was executed for a virtual machine. This allows the Hyper-V Administrator to designate a Hyper-V server as a Replica server for the virtual machine that was recovered To enable Reverse Replication for a virtual machine: 1. Open Hyper-V Manager 2. In the details pane, select a virtual machine that has a Health of Warning 3. With the virtual machine selected, Right-click, select Replication, and then choose Remove Recovery Points. This merges all the recovery points into the original VHD. This completes Failover but Replication Health still indicates Warning 4. With the virtual machine selected, Right-click, select Reverse Replication 5. Complete the Reverse Replication wizard by either selecting the Primary server (the default) as the Replica y server or choosing another Replica server. Keep in mind that if the selected Hyper-V server has not been enabled as a Replica server and the appropriate firewall rule enabled, the Reverse Replication process will fail Remove Recovery Points - This action is available only during a Failover scenario. When this action is executed, all recovery points (snapshots) for a Replica virtual machine are deleted. When the action is executed, a pop-up dialog box is presented to the user indicating all recovery points will be removed and Cancel Failover will no longer be available. The user must acknowledge the pop-up by clicking either Yes or No. If Yes is selected, the Failover is committed and the recovery points are merged down into the base VHD for the virtual machine. At this point Reverse Replication can be configured to clear the Warning for Replication Health and an Initial Replication can begin to the new Replica Server Modifying Virtual Machine Replication Settings Some replication settings for a virtual machine can be modified on the Primary server even after initial replication has been enabled. Information that can or cannot be modified includes: Windows Server 39

44 Replication settings o o o Port on the Replica server Authentication Type Data compression Recovery Points o o Number of Recovery Points to save Replication of incremental snapshots using VSS Replication VHDs - cannot be modified. VHDs can only be selected or deselected for replication when replication is initially enabled Resynchronization o o Manually - The user will have to initiate a resynchronization Automatically Start Resynchronization - resynchronization will occur automatically when needed (i.e. when the virtual machine state indicates a Resynch Required Microsoft Corporation. All rights reserved.

45 o Automatically scheduled (Default) - resynchronization, if needed, will only occur during the timeframe configured A subset of these settings are visible, but cannot be modified, on the virtual machine on the Replica server. Hyper-V Resynchronization (Resync) Process One of the settings that can be configured for a virtual machine on the Primary Server is Resynchronization. Resynch is applicable when both Delta Replication (DR) fails and Full Initial Replication (IR) is costly in terms of network bandwidth and the time it would take to complete a Full Initial Replication. Resynch minimizes the amount of data sent over the network by computing checksums of the Primary and Replica virtual machine VHDs and sending only the deltas. Resynch is not a replacement for normal DR. After a Resynch completes, normal DR should resume. Here are typical scenarios where Delta Replication may fail: 1. Change tracking issues on the Primary Server Windows Server 41

46 a. Power-hole scenario where the Primary Server fails either at the Operating System or physical hardware level b. If the Primary is a Failover Cluster, an unexpected termination of the cluster service occurred and a Move was executed for the Highly Available Virtual Machine 2. Replication issues: a. Unable to convert the replicated tracking log file to a VHD on the Replica Server b. Unable to link a VHD on the Replica Server with a Parent VHD c. Tracking log file is missing on either the Primary or Replica Server (Note: a tracking log file is maintained for each VHD selected for replication in the virtual machine configuration) 3. Time Travel on the Primary or Replica Server a. The Primary or Replica virtual machine is restored from backup and the VHD chain is broken. A Resync will be required to get the DR back on track 4. Reverse Replication after Failover process is executed a. An in-line IR Resync will be triggered should the virtual machine already exist on the Primary Server The Resynchronization process is optimized for VHD sizes up to 500 Gigabytes. The Resynch workflow is designed to work with Hyper-V Replica Networking so it can take advantage of built-in Authorization, Compression, Chunking\Retry mechanisms and error handling features. Resync is also designed to consume migration, virtual machine state change and VHD tree change notification events for either a virtual machine or the virtual machine storage. The Resync process can be resumed in the event of an outage (e.g. network outage, VMMS crash, etc ). Resynchronization uses a Fixed Block Chunking algorithm where Source (Primary Server) and Seed (Replica Server) files are divided into fixed chunks (2 Megabytes); the CRC64 signatures for each chunk are generated and then compared to determine which block(s) from the Source need to be applied to the Seed. This same process is used by the Microsoft Data Protection Manager (DPM) product. There is logic built into the Resynch process where if the process will take longer than 6 hours to complete using a 50% dedicated 1.5Mbps link, the Administrator will be asked to perform a full IR. Hyper-V Replica Architecture Key Modules/Components The Hyper-V Replica architecture is divided into two pieces: Microsoft Corporation. All rights reserved.

47 Hyper-V Replica Component architecture illustrates the Hyper-V Replica components, which are integrated into the Hyper-V Virtual Machine Management Service (VMMS) itself Hyper-V Replica Communications architecture illustrates the communication exchanges between Primary and Replica servers Architectural Diagram Hyper-V Replica Component Architecture As already mentioned, Hyper-V Replica is new functionality that is included as part of the Hyper-V role in Windows Server "8" Beta. The main components are: Hyper-V Replica Management Interface Hyper-V Replica PowerShell Objects Hyper-V Replica WMI API Hyper-V Replica Replication Manager (RM) Windows Server 43

48 Hyper-V Replica Replication Tracker (RT) Hyper-V Replica Networking Hyper-V Replica Broker Manager Hyper-V Replica Management Layer The Hyper-V Manager interface is used to configure, manage, and monitor Hyper-V Replica. Hyper-V Administrators use Hyper-V Manager to perform selection, configuration, provisioning, failover, and test failover operations monitoring, and reporting activities for virtual machines. All Hyper-V Replica management is built exclusively on WMI. Hyper-V Replica PowerShell Objects The Hyper-V Replica PowerShell interface is an implementation of the Admin Task Model and allows Hyper-V administrators to accomplish administrative tasks in managing configuration, provisioning, failover, test failover, operations monitoring, and reporting activities using PowerShell scripts. The Replica PowerShell objects are built exclusively on the Hyper-V Replica WMI (Windows Management Interface) API. Hyper-V Replica WMI API The Hyper-V Replica WMI API is part of the Hyper-V WMI provider and provides programmability of the underlying stack. Remoteable UI (using Windows Server "8" Beta Remote Server Administration Tools (RSAT)) and PowerShell are achieved by integrating with the Windows Remote Management Client and Listener (WinRM). The Hyper-V WMI Provider communicates SOAP (Simple Object Access Protocol) Messages using the WS-MAN (Web Services-Management) protocol with the Virtual Machine Management Service (VMMS) that runs on the remote Hyper-V Host. For more details on the WMI provider, refer to WMI Architecture. For more details on the remoting architecture, refer to Windows Remote Management Architecture. Hyper-V Replica Replication Manager (RM) The Virtual Machine Management Service (VMMS) is a collection of management components that manage virtual machines on the host computer system. VMMS follows an object model where subsystems have a singleton manager class that is activated by the Service Module. Hyper-V Replica components plug into the VMMS object model. The Hyper-V Replica Replication Manager is a component that plugs into the Hyper-V architecture and is responsible for interacting with Hyper-V to implement an efficient virtual machine level change tracking mechanism. This Hyper-V Replica Replication Manager object registers for migration and AVHD (differencing disk) reference count notifications, and it monitors the internal Hyper-V state through queries and callbacks. It stores and retrieves per virtual machine configuration and runtime state. The Replication Manager manages all the metadata describing each protected virtual machine and interacts with the Virtual Machine Configuration Repository to store metadata. The metadata includes: Microsoft Corporation. All rights reserved.

49 Configuration metadata including Standard-replica and Application-consistent replicas, encryption, compression, and source to destination storage mappings The collection of virtual machines running on a Primary server. For any given virtual machine, all replicated Virtual Hard Disk (VHD) path information Current replica state (information about the last successful Application-consistent or Standard replica recovery point replication event) The Hyper-V Replica Replication Manager implements all the handlers for initial replication, failover, and test-failover tasks. It initiates and tears down the sequence for initial replication (IR) (in-band or out-of-band), delta replication (DR), failover, and test failover testing. Hyper- V Replica Replication Manager is responsible for scheduling repetitive and one-time tasks and managing the status of these tasks. The following task handlers are implemented: Initial Replication handler Delta Replication handler Failover handler Failback handler Test Failover handler The Hyper-V Replica Replication Manager handles all virtual machine lifecycle change events in support of virtual machine mobility scenarios (cluster failover, live migration, Export- Import, etc ). To accomplish this, the Replication Manager registers for synchronous callbacks from the Virtual Machine Migration Manager for migration events. These are required so the Replication Manger can quiesce and suspend replication processes before migration begins. Conversely, the Replication Manager resumes replication when migration has been completed or cancelled. The Hyper-V Replica Replication Manager interacts with Hyper-V and the Hyper-V Replica Change Tracker to implement the core of the replication tasks. This combination of functionality is referred to as the Hyper-V Replica Replication Engine (RE). The Hyper-V Replica Replication Engine: Exposes private API s to the handlers in the Replication Task Manager that initiate the call sequence for IR and DR. Implements a virtual machine lifecycle change tracking mechanism that notifies the Replication State Manager whenever the virtual machine configuration changes in a way that affects the replication sequence. Examples of this include a failover of a virtual machine from one primary to another when deployed in a clustered configuration. Performs IR and DR task sequences. Hyper-V Replica Replication Tracker (RT) The Hyper-V Replica Replication Tracker is part of the Hyper-V Replica Replication Engine. The Hyper-V Replica Replication Tracker encapsulates the replication state of a virtual Windows Server 45

50 machine and periodically replicates updates to the virtual hard disks associated with a virtual machine to a Replica server. By default, virtual machine Delta Replication happens every 5 minutes and is not configurable. The changes to a VHD on the Primary server are tracked in a Hyper-V Replication Log (*.hrl) that is located in the same directory as the VHD it is associated with. Each VHD configured in a virtual machine that is selected for replication has an associated tracking log. This log(s) is (are) sent to the Replica server. When a log is in transit to the Replica server, the changes in the primary virtual machine are tracked in another log file in the same directory. The selections made by a Hyper-V Administrator when configuring Recovery History for a virtual machine, determine how the replication logs are handled on the Replica server. As an example, if the selection Store only the latest point for recovery is made, all of the replication log data is merged into the VHD file(s) that was (were) initially replicated to the Replica server for the virtual machine. Once an acknowledgement is received from the Replica server, the log, which is tracking the changes, is sent and the process repeats itself. The Hyper-V Replica Replication Manager also provides virtual machine state updates (like migration) to the Replication Tracker. The Replication Tracker persists its state in the virtual machine configuration thus ensuring the replica state is preserved consistently across virtual machine configuration changes. An example of a virtual machine configuration file with this information is shown here - The Replication tracker enables VMMS to store its configuration metadata inside the virtual machine configuration, specifically metadata that must be preserved during a virtual machine mobility scenario (e.g. cluster failover, live migration, export-import, etc ). The Replication Tracker implements the following sequence for Delta Replication (DR): Microsoft Corporation. All rights reserved.

51 Hyper-V Replica replicates changes every 5 minutes for all virtual machines that have replication enabled to their respective Replica Servers. The replication logs (are sent, provided the previous replication was acknowledged by the Replica Server. There are three specific cases o o o Case 1: Administrator has chosen to store only the latest recovery point Case 2: Administrator has chosen to store multiple recovery points (Standard Replicas) Case 3: Administrator has chosen to store multiple recovery points (Standard Replica) and wants Application-Consistent recovery points as well which uses the Volume Shadow Copy Service in the virtual machine to create snapshots for transport to the Replica Server For Case 1, every 5 minutes the current log file(s), which is (are) tracking changes to the virtual machine are frozen for transportation to the Replica Server. A new log file(s) is generated to continue tracking changes. The frozen log file(s) is(are) sent to the Replica Server and merged into the replica VHDs For Case 2, as before, log files are sent every 5 minutes and a new log file(s) is generated to continue tracking changes. However, unlike case 1, in the replica server, a recovery snapshot is created every 1 hour to which the log files are written. It is worth noting that this value (1 hr) is not configurable by the user. The number of such recovery snapshots created is determined by the selection made by the Admin when replication is configured. Once the limit is reached, a merge is initiated which merges the oldest snapshot to the base replica VHD For Case 3, like case 1 and 2 earlier, log files are sent from the primary server every 5 minutes. However as the user wants an application consistent recovery point, a VSS snapshot is taken at the desired interval that is sent to the replica site. The behavior in the replica server is similar to case 2 however when an app consistent copy arrives, the entire replica snapshot now becomes app consistent. To illustrate with an example, if the administrator has configured to take an hourly app consistent copy, then every recovery snapshot in the replica server is app consistent (as the 12 th log file which arrives at a 5min interval in a given hour is app consistent). If the user has picked a 2 hour app consistency window, then every other recovery snapshot in the recovery snapshot tree is app consistent. At any time on the Primary Server, if a new log cannot be created, changes continue to be tracked in the existing log and an error is registered. Replication for the virtual machine is suspended and a Warning is reflected in the Replication Health Report for the virtual machine. The virtual machine may transition into a Resync Required state at some point When multiple virtual machines on the same host are configured for replication, the Hyper-V Replica Networking component throttles the DR send requests so there are no spikes on the host Windows Server 47

52 Hyper-V Replica Broker Manager The Hyper-V Replica Broker Manager is a singleton object that runs as part of the Hyper-V VMMS service. This module plays an important role if the Replica server is a Failover Cluster. The Replica Broker Manager maintains a network listener that uses Hyper-V Replica Networking services to communicate with the Hyper-V Replica Broker Role that is configured in the Failover Cluster. The Hyper-V Replica Broker role tracks the movement of Replica virtual machines hosted in a Failover Cluster. As part of virtual machine mobility scenarios, virtual machines can be migrated between nodes in the cluster. The Hyper-V Replica Broker Manager provides information that ensures proper, continued replication of virtual machines. Hyper-V Replica Networking Refer to the section that discusses the Hyper-V Replica Communications Architecture. Hyper-V Replica Communications Architecture Hyper-V Replica Communications Architecture Microsoft Corporation. All rights reserved.

53 The Hyper-V Replica communications architecture is implemented at the Hyper-V Replica Network transport layer. This layer provides a bi-directional transport channel based on the HTTP Client-Server model to send virtual machine replicas and control messages between Primary and Replica servers. Replica Network Services is responsible for authorizing access to a Replica server and mutually authenticating the Primary and Replica servers. It also provides the ability to encrypt and compress data that is sent by the primary server. In summary, Replica Network Services provides a secure, authorized, authenticated and efficient network transport channel for replicating virtualized workloads between Hyper-V servers. Hyper-V Replica Network Services supported features include: Authorizing the Primary server on the Replica server Mutual Authentication using certificates (HTTPS server) Mutual Authentication using Kerberos (HTTP server) Encryption of data stream(s) if certificates are being used for authentication Compression of data stream(s) Partial network throttling using Quality of Service (QOS) Policy Directing replication to a particular port based on Hyper-V Replication configuration and port configuration in the firewall. Some of the supported features are visible in the configuration interface for the Replica Server. Windows Server 49

54 Encryption of communications between Primary and Replica sites is not enabled by default. If Windows Integrated Authentication (Kerberos) is selected, the Kerberos encrypt function is not used by Hyper-V Replica. This method uses the Windows Security Support Provider Interface (SSPI) to generate the message exchange between the Primary and Replica server. Kerberos authentication for services is achieved by using Service Principal Names (SPNs). Hyper-V Replica Network Services creates and registers a new SPN (Service Principal Name) for which the service class is 'Hyper-V Replica Service'. If encryption is desired, then Certificate-based Authentication should be selected allowing the underlying SSL layer to encrypt the packets using a selected certificate. Note: A valid X509v3 digital certificate is required for Mutual Authentication using certificates. Once the Replica server configuration is completed, communications between a Primary and Replica server do not commence until a virtual machine hosted on the Primary server is configured for replication. Before this happens, the Replication Health (if displayed) for a virtual machine is Not Applicable. Each virtual machine hosted on a Primary server must be individually configured for replication using the Enable Replication wizard. Once Microsoft Corporation. All rights reserved.

55 configured, the Replication Health for a virtual machine changes to Normal. Once reliable communications are established between a Primary and Replica server, control and data messages are exchanged. Send Message Process The Hyper-V Replica Network Services layer creates a control channel to send and receive control messages from the Primary to the Replica server. Hyper-V Replica Network Services first checks to see if a control connection already exists. If one does, the connection is used. If one does not, a new connection is created and stored in an internal table. Hyper-V Replica Network Services then generates a packet for the control message and sends it across the network to the Replica server. Hyper-V Replica Network Services on the Replica server forwards the package to the Hyper-V Replica Replication Engine (RE), which acts upon it and Windows Server 51

56 sends a response back within a timeout interval (120 seconds). Unlike data packets, control packets are not chunked or compressed. Furthermore, the control channel messages are synchronous and therefore there is no retry logic built into Hyper-V Replica Network Services for control channels if a connection fails. Send Data Process Once a session is established with a Replica server and control packets have been successfully exchanged, the Hyper-V Replica Replication Engine will start transferring data from the Primary to the Replica server for the configured virtual machine based on the virtual machine ID (GUID). The Hyper-V Replica Network Services layer will check if a data connection exists for the virtual machine to the Replica server and use it. If it does not, a connection is created and the connection information is stored in an internal table. Before any data is sent, a control packet is sent to the replica server that contains a list of files that will be sent from the Primary server. The Replica server response contains information about which, if any, of the files already exist. The Hyper-V Replica Replication Engine on the Primary server will use the network interface provided by the Hyper-V Replica Network Service to send the data (files). These files will be either for an Initial Replication (IR) or for a Delta Replication (DR). The Hyper-V Replica Network Service layer chunks the data (2 MB chunks) and compresses the data. The data is encrypted if using the Mutual Authentication scheme involving certificates. Once the chunks of data are received by the Hyper-V Replica Network Service layer on the Replica server, they are decrypted, deflated, and glued back together and placed in the location that was provided in the virtual machine configuration. Once the entire payload is received, the Hyper-V Replica Network Service on the Replica server notifies the Replication Engine on the Primary that the transfer has been completed. If a virtual machine migration occurs while a data transfer is in progress, the Hyper-V Replica Network Service will clean-up any existing connections and reinitiate the connection to the Replica server without any manual intervention once the migration is completed. Once a connection is available, the Hyper-V Replica Network Service will send the same replica that may have been interrupted because of the migration. This is accomplished by doing a file comparison on what is already available on the Replica server against the files that need to be sent by the Primary server. If a storage migration in the Primary server occurs while a data transfer is in progress, the Hyper-V Replica Network Service will stop sending any replicas to the replica server until the migration is completed and the data channel has been cleaned up. Once a migration completes, the Hyper-V Replica Network Service re-opens the connection to the Replica server and sends the same replica that had not been sent already. This is accomplished by doing a file comparison on what is already available on the Replica server against the files that need to be sent from the Primary server. Data channels are persistent channels between Primary and Replica servers. Control channels, on the other hand, are short lived. To persist the data channel in an environment where 'proxies' potentially exist, an echo process has been put in place. Most 'proxies' allow for a 120-second 'keep-live' for a connection. To persist the data channel between a Primary Microsoft Corporation. All rights reserved.

57 and Replica server, echo packets are sent every 90 seconds. A Data channel is torn down when replication is paused, replication is removed for a virtual machine, or a virtual machine enabled for replication is deleted. Retry Logic for Delta Replication In addition to a connection timeout default (120 seconds), there is built-in retry logic within the Hyper-V Replica Network Service when a DR is occurring. Network Error RETRY LOGIC Replication Paused\Low disk Space on the Replica server Network Error - Replica Broker Non-Recoverable Error Network Authentication Error Authorization Error Virtual Machine not found Low memory condition INTERVAL Exponentially increase the retry interval from start of first attempt at DR (1, 2, 4, 8, 10 minutes). If network error persists, retry every 30 minutes. This is a case where replication has been paused for a virtual machine or the Replica server is running low on disk space. Retries occur every replication interval or when the user manually triggers a replication (Resume Replication). Check the Hyper- V VMMS\Admin log for additional information. If the Replica server is a Hyper-V Failover Cluster and the Replication Broker role is configured, retries occur at 1, 2, and 3 minutes. Failing to start a DR, the Replica Broker is re-contacted to determine the correct location of the Replica server. Virtual machine Replication Status will be shown as Critical - suspended and administrator intervention is required. Examples would include a broken VHD chain or the Replica virtual machine is in an invalid state. This is a non-recoverable error and no retires will be attempted. An event log message is registered and administrator intervention is required. This is a non-recoverable error and no retires will be attempted. An event log message is registered and administrator intervention is required. In the case of a standalone Hyper-V server, this is a non-recoverable error and administrator intervention is required. If the Replica server is a Hyper-V Failover Cluster with the Replica Broker configured, the same logic is applied as indicated above (Network Error - Replica Broker). If a low memory condition occurs on the Primary server, the retry logic is the same as indicated above for Paused Replication. Windows Server 53

58 Cancel\Pause from Primary Server There is no retry logic in this case. Troubleshooting Hyper-V Replica Introduction to Troubleshooting Hyper-V Replica This section explains how to troubleshoot Hyper-V Replica. Use this guide when: You have problems with connectivity between Primary and Replica servers You have problems enabling a virtual machine for replication You have problems with virtual machine replication whether it is Initial Replication (IR) or Delta Replication (DR) You have problems executing management actions associated with virtual machines on a Primary or Replica server You have problems with the Replication Broker configured in a Hyper-V Failover Cluster. You need to collect Performance monitoring data for replicated virtual machines. Tools for Troubleshooting Hyper-V Replica Utilities and Commands for Troubleshooting Hyper-V Replica Performance Monitor Performance Monitor contains Hyper-V counters specific to Hyper-V Replica. These counters monitor replication statistics for configured virtual machines. The specific counter is Hyper- V Failover Replication Counter VM. The data that can be collected for each selected virtual machine includes: Average Replication Latency Average Replication Size Last Replication Size Network Bytes Recv Network Bytes Sent Replication Count Replication Latency Microsoft Corporation. All rights reserved.

59 Resynchronized Bytes Windows Server 55

60 Hyper-V Replica Integration into the Hyper-V Best Practice Analyzer (BPA) Rules pertaining to Hyper-V Replica are included in the Hyper-V Best Practice Analyzer. The following BPA Rule details are provided to assist with troubleshooting: Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Detail A Replica server must be configured to accept replication requests Red Configuration This computer is designated as a Hyper-V Replica server but is not configured to accept incoming replication data from primary servers. This server cannot accept replication traffic from primary servers. Use Hyper-V Manager to specify which primary servers this Replica server should accept replication data from. Detail Replica servers should be configured to identify specific primary servers authorized to send replication traffic Yellow Configuration As configured, this Replica server accepts replication traffic from all primary servers and stores them in a single location. All replication from all primary servers is stored in one location, which might introduce privacy or security problems. Use Hyper-V Manager to create new authorization entries for the specific primary servers and specify separate storage locations for each of them. You can use wildcard characters to group primary servers into sets for each authorization entry. Detail Compression is recommended for replication traffic Yellow Configuration The replication traffic sent across the network from the primary server to the Replica server is uncompressed. Replication traffic will use more bandwidth than necessary. This impacts the following virtual machines:<list of VMs> Configure Hyper-V Replica to compress the data transmitted over the network in the settings for the virtual machine in Hyper-V Manager. You can also use tools outside of Hyper-V to perform compression. Detail Configure guest operating systems for VSS-based backups to enable applicationconsistent snapshots for Hyper-V Replica Red Configuration Application-consistent snapshots require that Volume Shadow Copy Services (VSS) is enabled and configured in the guest operating systems of virtual machines participating in replication. Even if application-consistent snapshots are specified in the replication Microsoft Corporation. All rights reserved.

61 Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Windows Server configuration, Hyper-V will not use them unless VSS is configured. This impacts the following virtual machines:<list of VMs> Use Hyper-V Manager to install integration services in the virtual machine. Detail Integration services must be installed before primary or Replica virtual machines can use an alternate IP address after a failover Red Configuration Virtual machines participating in replication can be configured to use a specific IP address in the event of failover, but only if integration services are installed in the guest operating system of the virtual machine. In the event of a failover (planned, unplanned, or test), the Replica virtual machine will come online using the same IP address as the primary virtual machine. This configuration might cause connectivity issues. This impacts the following virtual machines:<list of VMs> Use Hyper-V Manager to install integration services in the virtual machine. Detail To participate in replication, servers in failover clusters must have a Hyper-V Replica Broker configured Red Configuration For failover clusters, Hyper-V Replica requires the use of a Hyper-V Replica Broker name instead of an individual server name. If the virtual machine is moved to a different failover cluster node, replication cannot continue. Use Failover Cluster Manager to configure the Hyper-V Replica Broker. In Hyper-V Manager, ensure that the replication configuration uses the Hyper-V Replica Broker name as the server name. Detail Virtual hard disks with paging files should be excluded from replication Yellow Configuration Paging files should be excluded from participating in replication, but no disks have been excluded. Virtual hard disks that experience a high volume of input/output activity will unnecessarily require much greater resources to participate in replication. This impacts the following virtual machines:\n{0} If you have not already done so, create a separate virtual hard disk for the Windows paging file. If initial replication has already been completed, use Hyper- V Manager to remove replication. Then, configure replication again and exclude the virtual hard disk with the paging file from replication. Detail Configure the Failover TCP/IP settings that you want the Replica virtual machine to use in the event of a failover Yellow 57

62 Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Configuration Replica virtual machines configured with a static IP address should be configured to use a different IP address from their primary virtual machine counterpart in the event of failover. Clients using the workload supported by the primary virtual machine might not be able to connect to the Replica virtual machine after a failover. Also, the primary virtual machine's original IP address will not be valid in the Replica virtual machine network topology. Use Hyper-V Manager to configure the IP address that the Replica virtual machine should use in the event of failover. This impacts the following virtual machine(s): <List of VMs> Detail Authorization entries should have distinct tags for primary servers with virtual machines that are not part of the same security group. Yellow Configuration The server will accept replication requests for the replica virtual machine from any of the servers in the authorization list associated with the same replication tag as of the VM. There might be privacy and security concerns with a virtual machine accepting replication from primary servers belonging to different authorization entries. This impacts the following authorization entries:<list of VMs> Use different tags in the authorization entries for primary servers with virtual machines that are not part of the same security group. Modify the Hyper-V settings to configure the replication tags. Detail Certificate-based authentication is configured, but the specified certificate is not installed on the Replica server or failover cluster nodes Red Configuration The security certificate that Hyper-V Replica has been configured to use to provide certificate-based replication is not installed on the Replica server (or any failover cluster nodes). In the event of a cluster failover or move to another node, Hyper-V replication will pause if the new node does not also have the appropriate certificate installed. This impacts the following nodes: <List of nodes> Install the configured certificate on the Replica server (and all associated nodes in the failover cluster, if any). Detail Replication is paused for one or more virtual machines on this server Yellow Operation Replication is paused for one or more of the virtual machines. While the primary virtual machine is paused, any changes that occur will be accumulated and will be sent to the Replica virtual machine once replication is resumed. s long as replication is paused, accumulated changes occurring in the primary virtual machine will consume available disk space on the primary server. After Microsoft Corporation. All rights reserved.

63 Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Windows Server replication is resumed, there might be a large burst of network traffic to the Replica server. This impacts the following virtual machines: <List of VMs> Confirm that pausing replication was intended. If replication was paused to address low disk space or network connectivity, resume replication as soon as those issues are resolved. Detail Initial replication is complete, but no test failover has been attempted Red Operation No test failovers have been attempted since completing initial replication. A test failover confirms that failover will succeed and that all workload operations on the primary virtual machine continue properly after failover to the Replica virtual machine. This impacts the following virtual machines: <List of VMs> Use Hyper-V Manager to conduct a test failover. Detail There has been no test failover in at least one month Yellow Operation Test failovers should be carried out at least monthly to verify that failover will succeed and that virtual machine workloads will operate as expected after failover. A test failover confirms that failover will succeed and that all workload operations on the primary virtual machine continue properly after failover to the Replica virtual machine. This impacts the following virtual machines: <List of VMs> Use Hyper-V Manager to conduct a test failover. Detail Certificate-based authentication is recommended for replication. Yellow Configuration One or more virtual machines selected for replication are configured for Kerberos authentication. The replication network traffic from the primary server to the replication server is unencrypted. This impacts the following virtual machines:<list of VMs> If another method is being used to perform encryption, you can ignore this. Otherwise, modify the virtual machine settings to choose certificate-based authentication. Detail Configure a policy to throttle the replication traffic on the network Yellow Configuration There might not be a limit on the amount of network bandwidth that replication is allowed to consume. Network bandwidth could become completely dominated by replication traffic, affecting other critical network activity. This impacts the following ports: <List of Ports> If you use another method to throttle network traffic, you can ignore this. 59

64 Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Impact Resolution Summary Rule Title Severity Category Issue Otherwise, use Group Policy Editor to configure a policy that will throttle the network traffic to the relevant port of the Replica server. Detail Resynchronization of replication should be scheduled for off-peak hours. Yellow Configuration Resynchronization of replication for the primary VMs is not scheduled for offpeak hours. Replication logs and Replication Point Objective will increase when the VM is in a resynchronize-required state for a longer time. At the same time, resynchronization will affect the IOPS bandwidth on the primary and the replica server, hence might affect production workloads. Use Hyper-V Manager VM Replication settings to configure the autoresynchronize replication window of the primary VM within the off-peak hours. Detail VHDX-based virtual hard disks are recommended for virtual machines that have recovery history enabled in replication settings. Yellow Configuration VHD-based virtual hard disks are being used for the virtual machines that are enabled for replication with recovery history turned on. Under some circumstances, the VHDs on the replica server could experience consistency issues. This impacts the following virtual machine(s): <List of VMs> Use the new virtual hard disk format (VHDX) for the virtual machines that are enabled for replication with recovery history turned on. You can convert a virtual hard disk from VHD format to VHDX format. The VHDX format has reliability mechanisms that help protect the disk from corruptions due to system power failures. However, do not convert the virtual hard disk if it is likely to be attached to an earlier release of Windows at some point. Windows releases earlier than {1} do not support the VHDX format. Detail Recovery snapshots should be removed after failover. Yellow Operation A failed over virtual machine has one or more recovery snapshots. Available space may run out on the physical disk that stores the snapshot files. If this occurs, no additional disk operations can be performed on the physical storage. Any virtual machine that relies on the physical storage could be affected. This impacts the following virtual machines: <List of VMs> For each failed over virtual machine, use the Complete-VMFailover cmdlet in Windows PowerShell to remove the recovery snapshots and indicate Failover completion. Detail A large number of recovery points has been configured Yellow Configuration Hyper-V Replica has been configured to store more than nine previous recovery Microsoft Corporation. All rights reserved.

65 Impact Resolution points. Maintaining too many recovery points could cause the Replica server to run out of available disk space. This impacts the following virtual machines: <List of VMs> Review the number of recovery points configured, taking into account factors such as the number of virtual machines on the server and the oldest recovery point that is really required. Tip: Periodically run the Hyper-V BPA to verify the Hyper-V server configuration complies with known 'best practices'. The BPA can also be used as a troubleshooting tool. Logging Options Hyper-V Replica is part of the Hyper-V Role. All events pertaining to Hyper-V Replica are logged in the Hyper-V-VMMS\Admin log located under Applications and Services Logs\Microsoft\Windows. In addition, an Analytic log can be enabled for Hyper-V-VMMS. To enable this log, first make the Analytic and Debug logs viewable in the Event Viewer. Open Event Viewer, then in the View menu, click Show Analytic and Debug logs. Windows Server 61

66 An Analytic log is visible under Hyper-V-VMMS In the Actions pane, click on Enable Log. Once enabled, it appears in Performance Monitor as an Event Trace Session located under Data Collector Sets. To view the information collected, first stop the tracing session by disabling the log, and then save the log and re-open it in Event Viewer or use other tools to convert it as desired Microsoft Corporation. All rights reserved.

67 General Methodology for Troubleshooting Hyper-V Replica Primary and Replica server connectivity issues General Methodology for troubleshooting Hyper-V Replica connectivity issues between Primary and Replica servers Symptom: Hyper-V Replica functionality is disrupted and the Hyper-V VMMS\Admin log reports general network connectivity errors between the Primary and Replica server 1. Verify the Replica server is booted and running. 2. Check network connectivity and name resolution functionality between the Primary and Replica server by executing ping and nslookup tests. If ping test fails, resolve network connectivity issues. If name resolution fails, check DNS 3. Ensure the Replica server is listening on the Replica Server Port. This can be accomplished by running a netstat -ano command on the Replica server after verifying the appropriate firewall rule has been Enabled or the custom firewall rule has been configured to allow Inbound communications on the configured port 4. Inspect the System Event Log on the Primary and Replica servers to determine if there is any failure condition associated with network functionality 5. Run the Hyper-V Best Practice Analyzer (BPA) and inspect the report for any configuration or operational issues Windows Server 63

68 Configuring a virtual machine for replication issues General Methodology for troubleshooting configuring a virtual machine for replication Symptom: Configuring a virtual machine for replication fails. 1. Verify the Replica server is booted and running. 2. Check network connectivity between and name resolution functionality the Primary and Replica server by executing a ping and nslookup tests. If the ping test fails, resolve network connectivity issues. If name resolution fails, check DNS 3. Ensure the Replica server is listening on the Replica Server Port and the Authentication Type is configured correctly. 4. If the Replica server configuration matches the parameters entered in the Enable Replication wizard, verify the Firewall on the Replica server has been configured to allow Inbound communications on the Replica Server Port 5. Inspect the System Event Log on the Primary and Replica servers to determine if there is any failure condition associated with network functionality 6. Inspect the Hyper-V VMMS\Admin Log for any events related to network connectivity on both the Primary and Replica servers Virtual machine Failover issues General Methodology for troubleshooting the virtual machine Planned Failover process A virtual machine Planned Failover process is a planned event where a running virtual machine on the Primary server is moved to a designated Replica server. Symptom: The Check that virtual machine is turned off Pre-Requisite test fails. 1. Ensure the virtual machine has been shut down prior to executing a Planned Failover to a Replica server Symptom: The Check configuration for allowing revers replication test fails. 1. Ensure the Primary server has also been configured as a Replica server. The assumption is that if a Planned Failover is executed to a Replica server, the virtual machine will use the Primary server as the new Replica server. This configuration in the virtual machine is included as part of the Planned Failover process Microsoft Corporation. All rights reserved.

69 Symptom: Send un-replicated data to Replica server fails. 1. Verify network connectivity to the Replica server using the procedures outlined in the General Methodology for troubleshooting Hyper-V Replica connectivity issues between Primary and Replica servers section Virtual machine Reverse Replication issues General Methodology for troubleshooting configuring a virtual machine for Reverse Replication Symptom: Reverse Replication configuration for a virtual machine results in a failure. 1. Verify network connectivity to the Hyper-V server being used as a Replica server using the procedures outlined in the General Methodology for troubleshooting Hyper-V Replica connectivity issues between Primary and Replica servers section Virtual machine Initial Replication (IR) and Delta Replication (DR) issues General Methodology for troubleshooting Initial Replication (IR) for a virtual machine Symptom: Initial Replication (IR) for a virtual machine fails. 1. Verify network connectivity to the Replica server using the procedures outlined in the General Methodology for troubleshooting Hyper-V Replica connectivity issues between Primary and Replica servers section 2. Ensure the protocol configuration between the Primary and Replica server match 3. Verify the Primary server is authorized to replicate with the Replica server this includes verifying the Security Tags match 4. Ensure the Authentication method matches between the Primary and Replica server 5. If there is an error on the Replica server indicating there is insufficient storage space, verify there is sufficient storage space available on the drive hosting the virtual machine replica file(s). If there is insufficient storage space, add additional storage space Windows Server 65

70 General Methodology for troubleshooting Delta Replication (DR) for a virtual machine Symptom: Delta Replication (DR) for a virtual machine fails 1. Verify network connectivity to the Replica server using the procedures outlined in the General Methodology for troubleshooting Hyper-V Replica connectivity issues between Primary and Replica servers section 2. Ensure the protocol configuration between the Primary and Replica server match 3. Verify the Primary server is authorized to replicate with the Replica server 4. Ensure the Authentication method matches between the Primary and Replica server 5. Check for any error(s) on the Replica server indicating there is insufficient storage space available to host the virtual machine replica files 6. Check for any error(s) on the Replica server indicating the virtual machine files could not be located Symptom: Application-consistent replicas are not generated by the Primary server and replicated to the Replica server 1. Verify the virtual machine has been configured to replicate application-consistent replicas to the Replica server 2. Verify the Integration Services version of the Guest matches what is installed in the Host (if there is a mismatch, a Warning message will be registered in the Hyper-V- Integration Admin log) 3. Check the virtual machine Integration Services and verify the Backup (Volume snapshot) integration component is enabled in the Guest 4. Review the system event log in the Guest and determine if there any errors pertaining to the Volume Shadow Copy Service (VSS) 5. Test VSS in the Guest by executing a backup of the operating system 6. Execute a backup on the Hyper-V host and verify the Guest can be backed up Microsoft Corporation. All rights reserved.

71 Replication Broker issues General Methodology for troubleshooting Replication Broker issues Symptom: When enabling a virtual machine for replication, a connection to the Client Access Point (CAP) being used by the Hyper-V Replica Cluster Replication Broker cannot be made. 1. Ensure all the resources supporting the Hyper-V Replica Clustering Replication Broker are Online in the cluster. If there are any failures for the resources in the group, troubleshoot the failures using standard Failover Cluster troubleshooting procedures 2. Move the resource group containing the Hyper-V Replica Clustering Replication Broker to another node in the cluster and attempt to enable replication for a virtual machine using the Client Access Point for the Hyper-V Replica Clustering Replication Broker Issues Implementing Guest IP functionality General methodology for troubleshooting Guest IP functionality Symptom: After initiating a Failover for a virtual machine, the configured Failover TCP/IP settings for the virtual machine in the Replica server are not implemented and a connection to the virtual machine cannot be made. 1. Ensure the Integration Components in the virtual machine have been updated. This problem could occur in down-level operating systems running in a virtual machine on a Windows Server "8" Beta Hyper-V server 2. Check the Hyper-V-Integration\Admin event log for an Event ID: 4010 Warning message reporting a problem with the Hyper-V Data Exchange functionality with the virtual machine experiencing this problem. Additionally, an Event ID: 4132 Error message will be recorded indicating a problem applying IP settings to a network adapter in the virtual machine experiencing this problem 3. Update the Integration Components in the virtual machine Windows Server 67

72 Appendix A - Hyper-V Replica PowerShell CMDLETS The following Hyper-V PowerShell cmdlets apply to Hyper-V Replica. Cmdlet Description Verb-Noun Complete-VMFailover This cmdlet completes the Failover process of the virtual machine. The virtual machine s current recovery point is committed and all other recovery points are removed. Failover cannot be cancelled once the recovery points are removed. Complete-VMFailover Syntax Complete-VMFailover [-VMName] <String> [-ComputerName <String[]>] [- PassThru] Example Complete-VMFailover -VMName C0-FIN-FS1 Cmdlet Description Verb-Noun Get-VMNetworkAdapterFailoverConfiguration Retrieves the Failover IP settings on a virtual machine network adaptor. Get-VMNetworkAdapterFailoverConfiguration Syntax Get-VMNetworkAdapterFailoverConfiguration [-VMName] <String[]> [[- VMNetworkAdapterName] <String[]>] [-ComputerName <String[]>] Example Cmdlet Description Verb-Noun Get-VMReplication Retrieve the replication relationship information of one or more virtual machines. Get-VMReplication Syntax Get-VMReplication [[-VMName] <String[]>] [-ComputerName <String[]>] [- ReplicaServerName <String>] [-PrimaryServerName <String>] [-State <String>] [-ReplicationTag <String>] Microsoft Corporation. All rights reserved.

73 Example Cmdlet Description Verb-Noun Get-VMReplicationAuthorizationEntry This cmdlet returns the details of an authorization entry as identified by the input parameter. The possible ways to get a particular authorization entry is by using AllowedPrimaryServer, or security tag. Get-VMReplicationAuthorizationEntry Syntax Get-VMReplicationAuthorizationEntry [[-AllowedPrimaryServer] <String>] [- ComputerName <String[]>] [-ReplicaStorageLocation <String Example Get-VMReplicationAuthorizationEntry -AllowedPrimaryServer hyp3.contoso.local -ReplicaStorageLocation e:\virtualmachines -SecurityTag CONTOSO Cmdlet Description Verb-Noun Syntax Example Get-VMReplicationServer This cmdlet retrieves the replication settings of the Replica server. If the server has not been configured as replica server, then the command returns default object. Get-VMReplicationServer Get-VMReplicationServer [[-ComputerName] <String[]>] Cmdlet Windows Server Import-VMInitialReplication 69