The Clone Task Force Local Replication, CLARiiON Environment

Transcription

1 The Clone Task Force Local Replication, CLARiiON Environment EMC Proven Professional Knowledge Sharing May, 2007 Fernando Moreno Liso System Executive Engineer CLARiiON Specialist Comparex Espana, S.A Copyright 2007 EMC Corporation 1

2 Table of Contents AUTHOR S BIOGRAPHY...3 ABSTRACT...4 ARTICLE: THE CLONE TASK FORCE PLANNING THE CLONES...5 CASE 1. AIX 5.3, ORACLE CASE 2. SLES 8, ORACLE 9I PRE-REQUISITES NAVICLI JAVA SECURITY SYNCHRONIZING...9 AIX... 9 Linux FRACTURING CONSISTENCY INTEGRITY WITH ORACLE INTEGRITY WITH ORACLE 9I BACKING UP DEVELOPING AND TESTING CONCLUSIONS...18 Copyright 2007 EMC Corporation 2

3 Author s Biography Fernando was born in Spain, and is married with a son and a daughter. After completing his degree, he was a systems engineer with HP Net servers learning about IT architecture. Some years later, as a system administrator at the same company specializing in Microsoft environments, he learned about the soft side of IT architectures. At Comparex, he focuses on storage subsystems. He specialized in HP and EMC platforms achieving HP Master Accredited Systems Engineer and EMC Proven Professional Customer Engineer (CLARiiON Specialist) status. Fernando now works as a storage services consultant to improve operational facilities and data availability in IT departments. Disclaimer: The views, processes or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation s views, processes or methodologies. Copyright 2007 EMC Corporation 3

4 Abstract Designing: Needs and Challenge. Data availability and protection must be considered when designing and implementing new storage information architectures. Since most systems are in production when these architectures are implemented, downtime must be avoided while still performing availability task routines. Backups can also fail so the testing of processes under crash conditions is also paramount. The work described in this article, based upon EMC best practices, describes the challenge of implementing local replication, backup, restore, and recovery of AIX and Linux (SUSE Linux Enterprise Server or SLES for short) environments with Oracle and SAP applications through the scheduling of automatic tasks that involve cloning, backup and re-charge of test systems production databases. Our project was a success. Through the use of scheduling tools, navicli commands and database scripting, the replica execution process was vastly simplified. In fact, only one script was required to recover an entire database twice a day. Implementing the method. The process was simple. We synchronized and fractured three groups of clones automatically for each environment (AIX or Linux, SLES) every eight hours, and assigned one of the fractured clone groups to a backup server. Then, we restored to a test server and recovered and checked the data (SAP and Oracle databases) within another instance name. Although the process was simple, the method was difficult to develop. We had to consider the implementation and integration of several tools in sequence including crontab, navicli commands, and SAP and Oracle scripts. We had to suspend different versions of databases, sync, consistent fracture, resume database, assign clones, backup to a disk library, de-assign clones, restore data, and start instances in test servers all without impacting production systems. Additionally, all these processes needed to run reliably and automatically with only one click. Article: The Clone Task Force There are some special requirements to make CLARiiON s local replication and high availability easy and effortless to manage. The objective of this article is to describe the local replication implemented in our main production environment. This implementation was necessary to protect the information system s data. We wanted to protect data every eight hours and to retain copies for as much time as possible (23 hours) so we implemented a regular system for cloning the information without impacting performance. The CLARiiON offered many advantages. Copyright 2007 EMC Corporation 4

5 We scheduled fractures every day as follows: Fractured first clones at 16:00; synchronized them at 15:00. Fractured second clones at 00:00; synchronized them at 23:00 Fractured third clones at 08:00; synchronized them at 07:00. 3 Different Fractures every 8h and kept 23h. We also wanted to give developers an opportunity to test with recent production data. Administrators required the same opportunity to backup data up to 23 hours. One of those fractured clones was assigned to a host for testing and backup tasks and then replaced to the cloning rotation once a day. The custom environment described in this article is the base infrastructure to implement local replication in CLARiiON. It consists of the following: - 1 CLARiiON CX700 array (Release 19). - Navisphere Manager, Agent and CLI, SnapView and PowerPath Soft. - Logical Units (LUN) over FC disks assigned to production hosts. - The same 3 n of LUN for each clone as required, the same size, over ATA disks for cloning purposes. - 2 LUN of 0,125GB for clone private feature properties configured. - 1 host of each OS to mount the clones (AIX and SLES). - 1 CLARiiON Disk Library for backing up tasks. In addition, the customer environment included SAP and Oracle databases on the production LUN. We had to know Oracle and SAP commands to make the replication consistent and to have the instances up and running in the background or development environments. 1. Planning the Clones We employed two ways of cloning. - The first consisted of an SAP R/3 environment containing production IS management system under Oracle 10 database and AIX 5.3 OS in a Power5 partitioned server. - The second consisted of another SAP environment containing HR and FI management systems under Oracle 9.2 databases and SLES 8 Linux Enterprise at a Blade center. Copyright 2007 EMC Corporation 5

6 The storage design of those databases follows: Case 1. AIX 5.3, Oracle 10 An IBM cluster (HACMP 5.2) with two AIX 5.3 nodes held the following storage infrastructure, all supporting the PA1 Information Data System: SAP1 SAP2 sapdata LUN10 270GB Raid5 Instancia LUN20 20GB Raid1 archives LUN30 10GB Raid5 Mirrlog LUN40 2,5GB Raid1/0 Origlog LUN50 2,5GB Raid1/0 We built the following infrastructure on the CX700 to maintain the clone replication: SAP IS PA1 - AIX 5.3, Oracle 10 FC Disks ATA Disks Raid5 Source LUN Comments Size (GB) Raid Clone Group Clone1 Clone2 Clone3 LUN10 Sapdatas (1-5) clones_pa1_lun10 LUN1010 LUN1011 LUN1012 LUN20 Instance 20 1 clones_pa1_lun20 LUN1020 LUN1021 LUN1022 LUN30 Archives/Redo Logs Offline 10 5 clones_pa1_lun30 LUN1030 LUN1031 LUN1058 LUN40 Redo Logs Online (OrigLog A&B) 2,5 1/0 clones_pa1_lun40 LUN1040 LUN1041 LUN1042 LUN50 Mirrored Logs Online (MirrLog A&B) 2,5 1/0 clones_pa1_lun50 LUN1050 LUN1051 LUN1052 We prepared the AIX and Oracle scripts for synchronizing and fracturing these clones regularly and consistently. Copyright 2007 EMC Corporation 6

7 Case 2. SLES 8, Oracle 9i A heartbeat SLES8 Linux cluster with two nodes sharing the (PH2) SAP Eco-Fin and HR information data system: Dbpro2a Dbpro2b Sapdatas1-4 LUN60 200GB Raid5 Sapdatas5-8 LUN70 300GB Raid5 Logs LUN80 20GB Raid1/0 Also, to maintain the clone replication, we built the following infrastructure on the CX700: SAP HR&FI PH2 - SLES8, Oracle 9 FC Disks ATA Disks Raid5 Source LUN Comments Size (GB) Raid Clone Group Clone1 Clone2 Clone3 LUN60 Sapdatas (1-4) clones_ph2_lun60 LUN1060 LUN1061 LUN1062 LUN70 Sapdatas (5-8) clones_ph2_lun70 LUN1070 LUN1071 LUN1072 LUN80 Logs Online & Offline 20 1/0 clones_ph2_lun80 LUN1080 LUN1081 LUN1082 Then, we showed the synchronizing and fracturing procedures periodically and consistently. Copyright 2007 EMC Corporation 7

8 2. Pre-Requisites Here are some considerations before initiating the cloning process. 2.1 NaviCLI As long as Release 19 is implemented in the CX, we needed the supported navicli version to interact with the array. For both, AIX and Linux, the installed navicli version is (6.5) navicli getagent Agent Rev: (6.5) 2.2 Java We had to install Java for users who would be executing the scripts for synchronizing and fracturing the clones because the Navisphere Command Line Interface (CLI) for clones uses Java runtime, and because Java was already implemented for SAP in earlier versions than required or recommended for cloning CLI. For AIX: java -jar -version java version "1.4.2" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2) Classic VM (build 1.4.2, J2RE IBM AIX 5L for PowerPC (64 bit JVM) build caix (SR6) (JIT enabled: jitc)) The installation directory for this version is: /usr/java/j2re1.4.2_12/bin/ For Linux: java -jar -version java version "1.4.2_12" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_12-b03) Java HotSpot(TM) Client VM (build 1.4.2_12-b03, mixed mode) Installation directory for this version is: /usr/java/j2re1.4.2_12/bin 2.3 Security The user who launched the synchronization and fracture scripts required privileges at the CX. We had to implement the CLI key and CLI security file to work around this issue. navicli.jar -AddUserSecurity -password *password* -scope 0 Copyright 2007 EMC Corporation 8

9 3. Synchronizing Synchronization occurs one hour before the fracture of each group of clones in each environment. The first synchronization lasted longer than one hour but after this, we experienced medium rates (a high rate could impact performance) and our results were adequate an hour before fracturing. When synchronizing a clone, we lost the availability to recover from the time it was fractured so we did have not a retention backup each 8 hours of 24 but rather 23 hours. That was the time we kept a clone copy of the data, three copies a day. The archive logs did the rest. This is the synchronization process scheduled in the production hosts: AIX crontab l # SNAPCLONE 15:00, SINC CLONE * * * /root/snapclone1500 > /tmp/snapclone1500.log # SNAPCLONE 23:00, SINC CLONE * * * /root/snapclone2300 > /tmp/snapclone2300.log # SNAPCLONE 07:00, SINC CLONE3 0 7 * * * /root/snapclone0700 > /tmp/snapclone0700.log Each of those scripts contains the following: :/root $cat snapclone1500 echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones1 de PA1... clones_pa1_lun10 -cloneid o clones_pa1_lun20 -cloneid o clones_pa1_lun30 -cloneid o clones_pa1_lun40 -cloneid o clones_pa1_lun50 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs1 PA1 Sincronizados" Copyright 2007 EMC Corporation 9

10 :/root $cat snapclone2300 echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones2 de PA1... clones_pa1_lun10 -cloneid o clones_pa1_lun20 -cloneid o clones_pa1_lun30 -cloneid o clones_pa1_lun40 -cloneid o clones_pa1_lun50 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PA1 Sincronizados" :/root $cat snapclone0700 echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones3 de PA1... clones_pa1_lun10 -cloneid o clones_pa1_lun20 -cloneid o clones_pa1_lun30 -cloneid o clones_pa1_lun40 -cloneid o clones_pa1_lun50 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs3 PA1 Sincronizados" Linux crontab l # SNAPCLONE 15:00, SINC CLONE * * * /root/scripts/snapclone1500 > /root/scripts/snapclone1500.log # SNAPCLONE 23:00, SINC CLONE * * * /root/scripts/snapclone2300 > /root/scripts/snapclone2300.log # SNAPCLONE 07:00, SINC CLONE3 0 7 * * * /root/scripts/snapclone0700 > /root/scripts/snapclone0700.log Copyright 2007 EMC Corporation 10

11 And the scripts called from crontab: :/root/scripts $cat snapclone1500 echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones1 PH2... snapview -syncclone -name clones_ph2_lun60 -cloneid o snapview -syncclone -name clones_ph2_lun70 -cloneid o snapview -syncclone -name clones_ph2_lun80 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs1 PH2 Sincronizados" :/root/scripts $cat snapclone2300 echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones2 PH2... snapview -syncclone -name clones_ph2_lun60 -cloneid o snapview -syncclone -name clones_ph2_lun70 -cloneid o snapview -syncclone -name clones_ph2_lun80 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PH2 Sincronizados" :/root/scripts $cat snapclone0700 echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones3 PH2... snapview -syncclone -name clones_ph2_lun60 -cloneid o snapview -syncclone -name clones_ph2_lun70 -cloneid o snapview -syncclone -name clones_ph2_lun80 -cloneid o echo "EXIT --- FIN DEL SCRIPT --- CLONEs3 PH2 Sincronizados" Copyright 2007 EMC Corporation 11

12 4. Fracturing When we fractured a clone group, we considered the Oracle databases consistency and integrity. We had to bind or enchain some tasks in our scheduled scripts. 4.1 Consistency When fracturing more than one clone simultaneously, and to preserve the point in time across the set of clones, we used the -consistentfractureclones option in our scripts. In this way, we ensured the I/O requests would delay until all of the clones had been fractured. Fractures occurred three times a day, after synchronization of the respective clones one hour before. Synchronization of the file systems was also needed to flush cache data into a disk. Fractures occurred at 00:00, 08:00 and 16:00 when clones were already in a synchronized or consistent state. Main fracture scripts are called by crontab on each system. AIX crontab l # SNAPCLONE 16:00, FRACT CLONE * * * /root/snapclone1600 > /tmp/snapclone1600.log # SNAPCLONE 00:00, FRACT CLONE2 0 0 * * * /root/snapclone0000 > /tmp/snapclone0000.log # SNAPCLONE 08:00, FRACT CLONE3 0 8 * * * /root/snapclone0800 > /tmp/snapclone0800.log Additional scripts follow on the next page. Copyright 2007 EMC Corporation 12

13 Each of these scripts contains lines for each fracture: :/root $cat snapclone1600 echo Fecha: date +"%Y%m%d%H%M%S" rm /tmp/bckctrl.dbf echo Suspend BBDD... su - orapa1 "-c sqlplus echo Limpieza de buffers y Sincronizacion datos filesystems en cache... /usr/sbin/sync /usr/sbin/sync echo Fractura Consistente de Clones1 de PA1 java -jar /usr/lpp/navicli/navicli.jar -address snapview -consistentfractureclones -CloneGroupNameCloneId clones_pa1_lun clones_pa1_lun clones_pa1_lun clones_pa1_lun clones_pa1_lun o echo Resume BBDD... su - orapa1 "-c sqlplus echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PA1 fracturados" And the same at 00:00, 08:00 for second ( ) and third ( ) snap clones respectively. sanpsuspend.sql and snapresume.sql scripts will be discussed in the integrity section below. Linux crontab l # SNAPCLONE 16:00, FRACT CLONE * * * /root/scripts/snapclone1600 > /root/scripts/snapclone1600.log # SNAPCLONE 00:00, FRACT CLONE2 0 0 * * * /root/scripts/snapclone0000 > /root/scripts/snapclone0000.log # SNAPCLONE 08:00, FRACT CLONE3 0 8 * * * /root/scripts/snapclone0800 > /root/scripts/snapclone0800.log Additional scripts follow on the next page. Copyright 2007 EMC Corporation 13

14 One of these scripts is implemented as follows: Linux:/root/scripts $cat snapclone0000 echo Fecha: date +"%Y%m%d%H%M%S" echo Suspend BBDD... su - oraph2 "-c rm /tmp/bckctrl.dbf" su - oraph2 "-c sqlplus echo Limpieza de buffers y Sincronizacion datos filesystems en cache... /bin/sync /bin/sync echo Fractura de Clones2 de PH2 snapview -consistentfractureclones -CloneGroupNameCloneId clones_ph2_lun clones_ph2_lun clones_ph2_lun o echo Resume BBDD... su - oraph2 "-c sqlplus echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PH2 fracturado" We changed the clone id to fracture for the first and third fracture clones scripts. Inside these scripts we introduced other commands (snapsuspend and snapresume) to put the Oracle database in special mode, and to give integrity to the cloned database. 4.2 Integrity with Oracle 10 We achieved Oracle database integrity by freezing (suspend mode) the instance we wanted to backup through cloning tasks. It involved the following: Begin backup of Table spaces Suspend database It is easier with Oracle release 10 since alter database begin backup allowed putting the whole tablespace in begin backup mode. Then, we put put the database in suspend mode. Additional scripts follow on next page. Copyright 2007 EMC Corporation 14

15 :/root $cat snapsuspend.sql connect / as sysdba!echo database backup controlfile to trace; alter database backup controlfile to trace;!echo database backup controlfile to file tmp bckctrl; alter database backup controlfile to '/tmp/bckctrl.dbf';!echo system switch logfile; alter system switch logfile;!echo database begin backup; alter database begin backup;!echo SYSTEM SUSPEND; alter system suspend;!echo ; :/root $cat snapresume.sql connect / as sysdba alter system resume; alter database end backup; alter system archive log current; alter database backup controlfile to trace; alter database backup controlfile to '/tmp/bckctrl.dbf'; 4.3 Integrity with Oracle 9i For Oracle s 9.2 release you must consider table space. To build the Oracle script you can list them with this command: select to_char(sysdate, 'MM/DD/YY') datum, tablespace_name, status from sys.dba_tablespaces; Snapclone calls snapsuspend.sql which includes these statements: Linux:/root/scripts $cat snapsuspend.sql connect / as sysdba alter database backup controlfile to trace; alter system switch logfile;!echo TABLESPACES BEGIN BACKUP; start beginbackup.sql!echo ALTER SYSTEM SUSPEND; alter system suspend; Copyright 2007 EMC Corporation 15

16 beginbackup.sql contains the list of sql statements to put all the tablespaces previously listed into begin backup mode: Linux:/root/scripts $cat beginbackup.sql!echo TABLESPACE SYSTEM begin backup; alter tablespace SYSTEM begin backup;!echo TABLESPACE PSAPROLL begin backup; alter tablespace PSAPROLL begin backup;. And so on until the last tablespace. When the clone set is already fractured, snapclone calls snapresume.sql which calls endbackup.sql to complete the process, where: Linux:/root/scripts $cat snapresume.sql connect / as sysdba alter system resume; start endbackup.sql alter system archive log current; alter database backup controlfile to trace; alter database backup controlfile to '/tmp/bckctrl.dbf'; And: Linux:/root/scripts $cat endbackup.sql alter tablespace SYSTEM end backup; alter tablespace PSAPROLL end backup;. And so on until the last tablespace. The last two scripts must reside in the Oracle user home directory with the appropriate permissions. 5. Backing Up We assigned clones to backup hosts through the Veritas pre-script bpstart_notify.policy executed just after one of the fractures: navicli -h storagegroup -addhlu -gname SG_AIXBACKUP -hlu 0 -alu 1010 navicli -h storagegroup -addhlu -gname SG_AIXBACKUP -hlu 1 -alu 1020 navicli -h storagegroup -addhlu -gname SG_AIXBACKUP -hlu 2 -alu 1030 navicli -h storagegroup -addhlu -gname SG_AIXBACKUP -hlu 3 -alu 1040 navicli -h storagegroup -addhlu -gname SG_AIXBACKUP -hlu 4 -alu 1050 cfgmgr powermt config rc.agent start varyonvg vg01 varyonvg vg02 varyonvg vg03 varyonvg vg04 varyonvg vg05 sh /root/scripts/pa1mounts.sh Copyright 2007 EMC Corporation 16

17 PA1mounts.sh included the mount command of each of the file systems required for backup (SAPdatas and logs) and defined in /etc/filesystems (Aix) or /etc/fstab (Linux). The backup put the clone information on a CDL as if it were on a tape for longer retention and without impacting production hosts. In Linux, we scheduled a reboot after assigning the clones to the backup host. It is the only way we have found to effectively rescan the scsi channel with SLES8. So, we scheduled the activation of clones and the reboot through crontab * * * /root/scripts/snapactivarclones1 > /tmp/snapactivarclones1.log * * * /sbin/shutdown -r now The rest of the script is the same as in AIX but we activated volume groups with vgchange a y vgname and we flushed file systems cache with /bin/sync Once backup was completed, we had to de-assign clones and remove any clone disk reference for subsequent assignment the next day. You can do it automatically by including these lines in your post backup script (bpend_notify.policy in case of Veritas Netbackup for the example we use). For AIX, assuming hdisk0 and hdisk1 are internal disks and hdisk3 to hdisk12 are the 10 paths for the 5 disk clones: #!/bin/ksh /usr/sbin/sync /usr/sbin/sync # Umount all the file systems sh /root/scripts/pa1umounts.sh # stop I/O rc.agent stop # De-activate VGs Vg= vg01 vg02 vg03 vg04 vg05 For volume in $Vg Varyoffvg $volume done # Remove PowerPath devices powermt remove dev=all lsdev -Cc disk -Fname grep power xargs -n1 rmdev -dl rmdev -dl powerpath0 hdisktoremove="hdisk3 hdis4 hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10 hdisk11 hdisk12" for r in $hdisktoremove do rmdev -dl $r done rc.agent start sleep 1 # De-assign clones. navicli -h storagegroup -removehlu -gname JARA_RUMPUP -hlu 0 -o navicli -h storagegroup -removehlu -gname JARA_RUMPUP -hlu 1 -o navicli -h storagegroup -removehlu -gname JARA_RUMPUP -hlu 2 -o navicli -h storagegroup -removehlu -gname JARA_RUMPUP -hlu 3 -o navicli -h storagegroup -removehlu -gname JARA_RUMPUP -hlu 4 -o Copyright 2007 EMC Corporation 17

18 It was not necessary to remove PowerPath devices for Linux hosts. We kept the paths in a dead state since they would be the same pseudo devices after re-assigning and rebooting. 6. Developing and Testing We tested the reliability of the replication in two ways: Recovering from one of the fractured clones Restoring from CDL virtual tape backup In both AIX and Linux, we assigned (the same way we assigned for backing up) one of the fractured clones to the test hosts and recovered a new database instance with the same data information weekly, or we restored one of the fractured clones from tape. To automate this task, you must: Remember that the name of the database instance should be different than the original. Change the Tnsnames and Listener files according to new host. Change the SAP profile to accomplish new host. Secure an SAP license for the new hardware machine to bring up the test or develop environment. Change the Oracle user according to new test environment. Perform Recover Database to run the new database. Modify or disable R3 jobs of source instance. 7. Conclusions Once implemented, the cloning procedure we see as cron daemon for UNIX servers provided the main functionality to schedule scripts that called another one to fully automate the process. It was not difficult to achieve a replication system for our data in conjunction with CLI and Oracle concepts. We recommend that you monitor the progression of synchronization and fracture to selfcontrol the course of action. We also recommend that you get statistics from analyzer to minimize any possible impact on performance if more than one clone synchronization could be on. From SAP systems, RZ20, ST03 or SM50 would help to monitor the system. Copyright 2007 EMC Corporation 18