Best Practices in DR using VCS & VVR Shrikant Ghare, Yatin Nayak

Transcription

1 Best Practices in DR using VCS & VVR Shrikant Ghare, Yatin Nayak Customer Forum, March 2012

2 Agenda 1 DR using VCS 2 DR using VVR 3 Q & A DR using VCS & VVR - Best Practices 2

3 DR using VCS 1 DR using VCS Quick overview 2 Best Practices for DR configurations 3 Advanced DR Configurations DR using VCS Best Practices 3

4 VCS DR Configuration Process Application Recovery Data Availability DR Testing DR using VCS Best Practices 4

5 DR Configuration: Application Recoverability wac gcoip gconic ClusterService SG DNS GCO wac gcoip gconic ClusterService SG DNS Oracle Oracle Mount Mount DiskGroup DiskGroup Oracle SG HTC Oracle SG HTC HTC PVOL Sync/Async HTC SVOL Production Cluster DR Cluster DR using VCS Best Practices 5

6 DR Configuration: Data Availability Cluster A Cluster B SMPL LLT/GCO paircreate COPY MCU ESCON or FC link RCU synchronized SMPL PVOL down, horctakeover Link restored, pairresync SMPL PVols SVols configuration using HTC PVOL up, horctakeover State diagram for HTC SVOLs DR using VCS Best Practices 6

7 VCS integrates with several Data Availability options Symantec EMC HDS/HP IBM Others Veritas Volume Replicator Campus Cluster NBU RealTime SRDF, SRDF/Star, VPLEX MirrorView RecoverPoint Hitachi TrueCopy/HP XP CA, Hitachi/HP XP 3DC Hitachi Universal Replicator 3PAR, EVA CA DB2 HADR IBM Global Mirror, IBM Metro Mirror, DS4K RemoteMirror SVC CopyServices, XIVMirror Oracle DataGuard, DataGuard Broker NetApp SnapMirror DR using VCS Best Practices 7

8 Disaster Readiness Test: Firedrills Simulate DR Failover: FireDrill Reset Mount Initiate Test Application Snapshot Fire Drill Primary Site Secondary Site Support for all replication technologies HW rep: SRDFSnap, HTCSnap etc. SW rep: DiskGroupSnap, RVGSnapshot, etc. Flavors: Gold/Silver/Bronze Volume Snapshot DR using VCS Best Practices 8

9 DR using VCS 1 DR using VCS Quick overview 2 Best Practices for DR configurations 3 Advanced DR Configurations DR using VCS Best Practices 9

10 VCS Stretch Cluster: Best Practices APP 1 Single VCS Cluster Define Site boundaries VCS: SystemZones, VxVM: Host Site ID, Disk/Enclosure Site ID Configure the failover behavior for stretch clusters AutoFailoverPolicy Recommended value: 2 for manual failover across SystemZones Zone A Zone B DR using VCS Best Practices 10

11 VCS GCO: Best Practices Configuring DR infrastructure: Use automation tools/wizards Java GUI/gcoconfig & VCS CLIs Reduce time Avoid human errors Manual Failover Configuring the failover behavior for Global Service Groups Use Manual ClusterFailoverPolicy (default) APP 1 GCO For short distances between DR sites, ClusterFailoverPolicy can be Connected Use Preswitch during planned migrations DR using VCS Best Practices 11

12 VCS GCO: Best Practices Steward Configure Steward for 2-site DR setups GCO phonon.symantec.com phonon.symantec.com Configure DNS agent for consistent name resolution after DR failover APP 1 GCO DR using VCS Best Practices 12

13 VCS Agents: Best Practices Install latest version of DR agent from SORT Freeze SGs before upgrading/installing replication agent Define the takeover behavior of the replication resource SplitTakeover = 0 AutoTakeover = 0 APP 1 GCO link suspended/broken Stale Data DR using VCS Best Practices 13

14 VCS Agents: Best Practices Use separate Device Group per Application APP 1 APP 2 APP 3 For out-of-band arrays, configure all the IP addresses in the resource definition clarspa.symantec.com clarspb.symantec.com DR using VCS Best Practices 14

15 VCS Agents: Best Practices For Virtual machines managing in-band arrays, map the Control Devices in Raw device mode to the Guest e.g. Physical RDM in Vmware Adjust the OnlineTimeout of the resource For EMC SRDF, use sigma script to assist in setting DevFOTime Increase ActionTimeout for Oracle DataGuard EMC RecoverPoint (CDP/R), use latest FailoverImage e.g. latest (recommended), bookmark_name, TIME=Timestamp Set PanicSystemOnDGLoss on DiskGroup DR using VCS Best Practices 15

16 Disaster Readiness Test: Best Practices APP 1 Identify Firedrill flavor (recommended: Gold) GCO Inter-array replication Local mirror/snapshot Use space optimized snapshots wherever possible E.g. EMC CLARiiON Snapview, EMC Symmetrix Snap DR using VCS Best Practices 16

17 Disaster Readiness Test: Best Practices Use fdsetup wizard to configure Firedrill Service Group Schedule periodic Firedrills using fdsched Configure only one System for Firedrill SG DR using VCS Best Practices 17

18 DR using VCS 1 DR using VCS Quick overview 2 Best Practices for DR configurations 3 Advanced DR Configurations DR using VCS Best Practices 18

19 Database Tier Middleware Tier Web Tier Multi Tier applications FINANCE VBS PROD VBS DR VBS 1. Disaster event: (GCO) Cluster Fault AppHA AppHA AppHA 2. On DR site, run VBS Showplan to check whether VBS can be started 3. Perform VBS Start on DR Site VCS 4. For the GCO Tier force takeover is done to start DB Tier SG GCO SFRAC SFHA SFCFS Production Site DR using VCS Best Practices Cluster Fault Event DR Site 5. Subsequently App and Web Tier SGs Started 19

20 DR for Virtual Machines VM images replicated to DR site To maintain consistent application environment (no config drifts) across Data Centers Replicated Network parameters may be incorrect in the DR site because of different subnet GCO Best Practices Use VCS Virtualization Agents to manage Virtual machines E.g. Solaris Ldoms, Solaris Zones, AIX WPARs Use VCS Virtualization agents DROpts attributes IP Address, Netmask, Gateway, DNS, etc. VM Network Reconfiguration after DR failover DR using VCS Best Practices 20

21 Multi site replication Bunker/Secondary Site APP 4 Sync Async Recovery link Primary Site Tertiary Site Use VCS agents to handle multi site replication configurations SRDF (Concurrent), SRDFstar, HTC3DC, RVGPrimary (with Bunker), etc. Configure single stretch cluster covering the synchronous sites Separate cluster for the asynchronous site, connected by VCS GCO DR using VCS Best Practices 21

22 Agenda 1 DR using VCS 2 DR using VVR 3 Q & A DR using VCS & VVR - Best Practices 22

23 Goals Planning Configuring VVR Elective best practices Optimizing Advanced Features DR using VVR 23

24 Options: where VVR fits in Primary Site Secondary Site Application-Based APP APP Host-Based Storage-Based DR using VVR 24

25 VVR objects Primary Datacenter Secondary Site APP RLINK Primary RVG Secondary RVG Production Data Storage Replicator Log (SRL) Storage Replicator Log (SRL) Secondary Data DR using VVR 25

26 VVR - Flexible replication modes to meet your SLA Synchronous Maximum Protection: Zero RPO Ideal for small distances (< 100 KM) Asynchronous Maximum Performance Ideal for any distance between sites Bunker Maximum Protection + Maximum Performance Cost effective 3-site solution DR using VVR 26

27 VVR Asynchronous High performance replication across any distance Primary Datacenter Secondary Site (DR site) SAN WAN Application host Write I/O I/O ack Storage array Cache & order I/O Application host Storage array NOT in Performance Path Next I/O Write I/O in order Performance Path Transfer I/O (distance = more time) Store I/O Acknowledge Transfer ack (distance = more time) DR using VVR 27

28 Best Practices in VVR - Overview VRAdvisor General practices Planning Configuring Configuring a secondary HA-DR agents Best Practices overview Bunker replication Compression Advanced features Optimizing Off-host processing VVR memory pool tuning DR using VVR 28

29 DR using VVR 29

30 VRAdvisor a VVR Planning tool How can I achieve desired RPO/RTO? How do I tolerate network outages? Use VRAdvisor Graphic interface No need to install VVR DR using VVR 30

31 VRAdvisor a VVR Planning tool The problems Tolerance to network & secondary outages Tolerance to I/O spikes RPO & RTO planning What VRAdvisor does Data collection and analysis Graphical charts SRL size planning Network bandwidth recommendations for sync & async modes of replication specified RPO & RTO What-if analysis DR using VVR 31

32 VRAdvisor a VVR Planning tool DR using VVR 32

33 VRAdvisor a VVR Planning tool DR using VVR 33

34 Best Practices in VVR Configuring Planning Configuring Advanced features Best Practices in VVR Optimizing General practices Configuring a VVR secondary Setting up HA-DR agents DR using VVR 34

35 Configuring VVR 101 Create the primary Replicated Volume Group (RVG) #> vradmin g diskgroup createpri rvg volumelist srl Add the secondary site to this RVG #> vradmin g diskgroup addsec rvg primary secondary is now configured, and can be started by #> vradmin g diskgroup a startrep rvg secondary Monitor replication status by running #> vradmin l repstatus rvg DR using VVR 35

36 General Practices for Configuring Better availability 1 RVG per application 1 RVG per disk group Better performance Use different physical disks for SRL and data volumes Stripe the SRL over several physical disks Tier-1/SSD storage for SRL: all writes go through the SRL! Better protection Mirror all data volumes and SRLs across arrays DR using VVR 36

37 Best Practices in VVR Configuring Planning Setting up Advanced features Best Practices in VVR Optimizing General practices Configuring a VVR secondary Setting up HA-DR agents DR using VVR 37

38 Best Practices for Configuring a VVR secondary Virtual IP Plan ahead for application clustering by configuring the replication IP addresses as virtual IPs Network bandwidth Enable bandwidth limit for VVR, if network bandwidth is constrained or shared Use autosync with SmartMove for efficient initial synchronization (SF 6.0+) Use compression feature (SF 6.0+) Firewall Open all the VVR heartbeat and data ports (vrport) For NAT-based firewalls, vol_vvr_use_nat tunable must be set to 1 DR using VVR 38

39 Best Practices in VVR Configuring Planning Setting up Advanced features Best Practices in VVR Optimizing General practices Configuring a VVR secondary Setting up HA-DR agents DR using VVR 39

40 HA-DR Agent hierarchy an example Mount Service Group (Parallel & Global) RVGShared Service Group (Parallel) RVGLogowner Service Group (Failover) CFSmount Online local firm dependency RVGShared Online local firm dependency RVGLogowner IP RVGSharedPri CVMVoldg NIC Online local firm dependency GCO Service Group (Failover) CFSfsckd App IP NIC CVMCluster CVMVxconfigd CVM Service Group (Parallel) DR using VVR 40

41 Best Practices in VVR Optimizing replication Planning Setting up Advanced features Best Practices in VVR Optimizing VVR memory pool tuning Performance tuning DR using VVR 41

42 VVR Memory Pools - overview Primary Network Secondary USER KERNEL USER KERNEL I/O ENGINE READBACK RVIOMEM RDBACK READ WRITE Replica Update State Machine WAN MESSAGE SERVER DATA VOLs SRL REPLICAS NMCOM NMCOM 4145 TCP ANON UDP NMCOM NMCOM SRL SECONDARY I/O ENGINE DATA VOLs DR using VVR 42

43 Best Practices in VVR Advanced Features Planning Setting up Advanced features Best Practices in VVR Tuning Using bunker replication Using off-host processing Using compression DR using VVR 43

44 Best Practices for bunker replication Production Site VVR Asynchronous (any distance, over IP) VVR Disaster Recovery Site Synchronous (small distance; IP or FC) Replay log (during DR) Bunker Site VVR Storage needed only for bunker SRL - Follow bunker proximity guidelines - Dedicate separate disks to bunker SRL - Dedicate separate controllers to bunker SRL DR using VVR 44

45 Summary Planning VVR configuration VRAdvisor VVR Elective best practices Configuring Adding VVR secondary HA-DR agents Optimizing VVR memory pool tuning Advanced Features Bunker replication DR using VVR 45

46 Thank you! Shrikant Ghare Yatin Nayak Copyright 2010 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. 46

47 Multi Tier applications: Planned migration (TBD) FINANCE VBS PROD VBS DR VBS 1. VBS contains mix of Global & Local SGs AppHA AppHA AppHA 2. Global SG is ONLINE on Production Site VCS 3. VBS Showplan on DR site would show that VBS on DR cannot go online SFRAC SFHA SFCFS ONLINE Production Site Advanced DR Configurations GCO DR Site 4. Stop the VBS on Production site 5. Start the VBS on DR site 47

48 Planning for Disaster Recovery Ensure Business Continuance after Disaster Reduce downtime cost and recovery time (RTO) Reduce data loss (RPO) Automatic detection and notification of disaster/site failures Automated application component startup in the right order Ability to do planned migration of application to DR site Veritas Cluster Server (VCS) for DR automation Single solution for local High Availability & Disaster Recovery Planning for Disaster Recovery 48

49 DR Events in VOM Advanced DR Configurations 49

50 DR Events in VOM Advanced DR Configurations 50

51 Futures DR SLA computation Instant computation of DR SLAs like RPO and RTO VOM based recovery plan for SG and VBS Multi Tier Firedrills Futures 51

52 Configuration: Best Practices Identify hosts for replication management and install appropriate vendor tools on all control hosts These could be the same as Application s SystemList, or different For in-band arrays, map the control devices to Control hosts E.g. EMC Gatekeepers, Hitachi Command Device For virtual machine control hosts, map the control devices in Raw device mode e.g. Physical RDM Configure security wherever possible Install secure vendor tools wherever available e.g. NaviSecCli Generate appropriate array security files Best Practices for DR Configurations 52

53 Multi Tier Application: DR Best Practices Use VBS to manage multi-tier applications Use Showplan to check the readiness of the Business Service on DR Site for planned/unplanned operations There should be at least one global service group configured in VBS DR setup on at least one of the VBS Cluster Tier Recommended: Global SGs at all tiers The failover policy of the global service group should be Manual Multiple tiers can be consolidated into the same tier at the DR site Advanced DR Configurations 53

54 DR Configurations: Topologies Veritas Cluster Server (VCS) for Disaster Recovery VCS Benefits Local HA Metropolitan HA (Stretch Cluster) Steward Wide-Area DR (Global Cluster) APP SAP 4 APP 3 APP 2 APP 1 SAP APP 4 APP 3 APP 2 APP 1 Reduce Failover time Operator dependency Operator error Complexity Provide comprehensive data and application availability Synch or Mirroring Asynchronous DR using VCS Best Practices 54