ZFS S TORAGE EXADATA

Transcription

1 THE PERFECT M ARRIAGE ARRIAGE: ZFS ZFS S TORAGE EXADATA TORAGE APPLIANCE AND Charles Kim, Viscosity North America Nitin Vengurlekar, Viscosity North America Justin Nugent, Viscosity North America ABSTRACT Storage requirements are exploding across the Oracle industry, often beyond DBA control. We are constantly asked to maximize existing resources and do more with less, and the reality is data volumes are increasing exponentially. What kind of options do we have? As fellow database administrators (DBA) and database machine administrators (DMA), we want to better leverage the space that we have available on the Exadata. Space is a premium on the Exadata, and we want to squeeze every byte we can. How can we do that? The best-suited answer is the ZFS Storage Appliance. We plan to demonstrate how you can couple the ZFS Storage Appliance (ZFSSA) with the Exadata to fully maximize space utilization by tier-ing storage on the Exadata and ZFSSA. We will also discuss data migration strategies to archive data from DATA and RECO disk groups and to the ZFSSA. Along the way, we can take advantage of Exadata Hybrid Columnar Compression (EHCC) to reduce our footprint even more. We plan to provide insight into optimal RMAN backup settings, direct NFS (dnfs) settings, and quick scripts used to drive throughput on the ZFSSA across all the compute nodes. You will also learn about the power of implementing snapshots and clones to provision databases on the fly, and replication to achieve another level of protection. ZFSSA is not just for the Exadata. We will reveal additional use cases for the ZFSSA that will be relevant for your IT infrastructure. TARGET AUDIENCE The primary audience of this white paper is aimed at database architects, database machine administrators and managers who support databases on the Exadata and are interested in stretching the storage footprint on the Exadata. The audience will also be for DMAs who are interested in optimal backup strategies and snapshot/cloning features available with the Exadata on the ZFSSA. EXECUTIVE SUMMARY Our goal is to expose the practicality and the usefulness of the ZFS Storage Appliance to anyone who is interested in expanding the storage options and footprint on the Exadata. Attendees will learn how to setup direct NFS for the ZFS Storage Appliance and to drive performance in backups the power of creating snapshots and cloning databases instantly how to take advantage of Data Guard databases to instantiate another copy of the database on the fly RMAN strategies to take back for their enterprise 1 Session # 414

2 BACKGROUND A full rack Exadata X3 Rack can house X TB of space in a normal redundancy and X TB of space in a high redundancy disk group. A half rack Exadata X3 Rack can house X TB of space in a normal redundancy and X TB of space in a high redundancy disk group. One of the biggest business drivers for moving to the Exadata is for database consolidation. As we consolidate databases and notice that we need additional capacity, what do you do if you are hitting 80 or 90 percent space utilization on the DATA_DG disk group? Better yet, your company is interested in purchasing the Exadata and you realize that your database footprint is higher than what Oracle provides on the full Exadata. Not only do we have to calculate the size of the databases housed on the Exadata, we have to take into consideration the space required for database backups. With Exadata, we can only perform backups over network. We can perform backups to the RECO_DG disk group or to DBFS file system. Neither of these solutions allow for the backup to leave the Exadata ecosystem and do not take into consideration of complete loss of storage or failures at the disk levels beyond ASM normal or high redundancy. Network backups can be over 1GigE, 10GigE or over 40GigE Infiniband. In our discussions, we will only cover 10gigE and Infiniband (more so on the Infiniband side). We can perform what is coined as Disk to Disk (D2D) backups from the Exadata to the ZFSSA over extremely low latency 40 GigE Infiniband leveraging Oracle s Direct NFS. The ZFSSA provides de-duplication and Exadata Hybrid Columnar Compression (EHCC) support. For larger customers, our goals are to leverage Image Copy backups and perform Forever Incremental backups. TECHNICAL DISCUSSIONS AND EXAMPLES When we talk about storage in the context of this white paper, we are referring to the Sun ZFS Storage 7000 Appliances. Previously, the similar line of storage appliances were called Sun Storage 7000 Unified Storage System as shown below: Legacy Storage Systems Current Storage Appliances Sun Storage 7000 Unified Storage Systems (7110, 7210, 7310, and 7410) à Sun ZFS Storage 7000 Appliances (7120, 7320, and 7420) First thing to note, the ZFSSA line of products are evolving and changing each year. At the time of this white paper, the ZFSSA 7420 model can scale up to 2.59 PB of raw capacity, can accommodate up to 2 controllers with up to 2 TB read flash cache per storage controller, and can hold up to 292 GB write flash cache per disk shelf. Most importantly, the ZFSSA 7420 accepts up to Quad Gigabit Ethernet UTP; Dual 10 GigE, QDR Infiniband HCA, and 8Gb FC HBA. What we are interested in the most is the Dual 10GigE and QDR Infiniband HCA to drive performance of our space utilization on the ZFSSA. Once you purchase the ZFSSSA for your Exadata, you will have to make choice on data protection based on performance and capacity. If you want the best protection and performance, choose the mirrored option. Choose the appropriate configuration option based on your business requirements. If you are leveraging the ZFSSA for just backups, you may opt not to do mirroring and choose double parity instead. Similarly, if you choose to only place archived datafiles (not active datafiles), you can choose the double parity option also. 2 Session # 414

3 On the ZFSSA with 2 trays of disks fully populated, we are able to provision 64.5 TB of useable space. On the Exadata, the only protocol that we will want is NFS. Once shares are allocated from the ZFSSA, we can Direct NFS mount the shares on the Exadata. RECLAIM RESOURCES AND SPACE FROM DBFS One you have the ZFSSA in place, you can completely eliminate the DBFS (Database File System) from the Exadata. Not only are you eliminating the clusterware resource from the Exadata but also the database that houses the DBFS file system contents. If you have a huge DBFS file system requirement on the Exadata, the tablespace that houses the DBFS file system table must be moved from the SYSTEM_DG to either the DATA_DG or the RECO_DG. Removing the DBFS database can free up quite a bit of valuable space from the DATA_DG or the RECO_DG. ACHIEVE INFORMATION LIFE CYCLE MANAGEMENT ON THE EXATATA AND ZFSSA ZFSSA Storage costs are commodity compared to adding storage cells to existing Exadata. On each ZFSSA, disk space is scalable up to 1.72 PB of Disk Drives (1720 TB), and flash storage is scalable up to 10 TB. Our goal is to leverage high transactional data on DATA_DG disk group as Tier 1 storage. We can move less active data that is over 1 or 2 years old (or even as soon as 3-6 months) from Tier 1 to Tier 2 storage (RECO_DG). Ultimately, move data that is older than 2-3 years from Tier 2 to Tier 3 (ZFSSA). As we move from Tier 1 to Tier 2 storage, we can leverage Warehouse Compression for Tier 2 storage. If your goal is to achieve the highest level of performance, do not want to compress the contents of your DATA_DG at all. As you move from Tier 1 to Tier 2, you can leverage various compression options. We will also take advantage of any partitioning strategy to move data from different tiers of storage. As we move from Tier 1 to Tier 2, we can leverage EHCC compression in warehouse compression mode. You should choose Warehouse Compression LOW if load time service levels are more critical than query performance. Furthermore, we can achieve a deeper level of HCC (Archive High) as we move data from Tier 2 to Tier 3 storage. With the archive compression mode, our goal is to sacrifice performance for maximum storage gains. You should only use archive compression if the tables/partitions are rarely accessed. As we take into consideration Tier 3 storage, we can architect for situations where oracle datafiles can be made READ-ONLY. For tables that are partitioned and have archiving requirements, older partitions become great candidates for read-only 3 Session # 414

4 tablespaces where we can skip READ-ONLY datafiles during RMAN backups.. Our strategic goal is to to backup the READ-ONLY tablespaces once and be able to forget about it from any future backups. After three years, we can even convert data to external tables or external datapump files. CREATE NFS SHARES We can create NFS shares from the ZFSSA from the GUI called the Browser User Interface (BUI) in the ZFSSA world or command line. In this section, we will demonstrate the simplicity of creating shares with both approaches. We will start with the BUI method first. Once you login to your ZFSSA, click on the Shares link on the top. Your screen will change to display all the projects and file systems. Click on the + sign on left of the Filesystems. The Create Filesystem window will pop up for you to edit. In this window, you will want to associate the Filesystem to a project (take the default if you do not have projects created yet) and name your share. Click on the Apply button when you are finished. We have successfully created our first NFS share. We can achieve the exact same result from command-line (CLI). Whether you are a BUI or a CLI administrator, everyone should learn how to manage the ZFSSA from the CLI. There may be situations where the BUI may not be available for you especially in environments where you have to traverse through series of firewalls or have issues with port availability. From the CLI, you can easily create shares: zfs1:> shares zfs1:shares> select default zfs1:shares default> filesystem db_u200 4 Session # 414

5 zfs1:shares default/db_u200 (uncommitted)> commit After we create our share, we want to set the appropriate permissions and security settings: zfs1:shares default> select db_u200 zfs1:shares default/db_u200> set aclinherit=discard aclinherit = discard (uncommitted) zfs1:shares default/db_u200> set sharenfs="rw,anon=0" sharenfs = rw,anon=0 (uncommitted) zfs1:shares default/db_u200> set root_permissions=755 root_permissions = 755 (uncommitted) zfs1:shares default/db_u200> commit Now are complete with our setup. Let s look at all the properties of the new share that we just created: zfs1:shares default/db_u200> show Properties: aclinherit = discard aclmode = discard (inherited) atime = true (inherited) casesensitivity = mixed checksum = fletcher4 (inherited) compression = off (inherited) dedup = false (inherited) compressratio = 100 copies = 1 (inherited) creation = Tue Feb :36:49 GMT+0000 (UTC) logbias = latency (inherited) mountpoint = /export/db_u200 (inherited) normalization = none quota = 0 quota_snap = true readonly = false (inherited) recordsize = 128K (inherited) reservation = 0 reservation_snap = true rstchown = true (inherited) secondarycache = all (inherited) shadow = none nbmand = false (inherited) sharesmb = off (inherited) sharenfs = rw,anon=0 snapdir = hidden (inherited) utf8only = false vscan = false (inherited) sharedav = off (inherited) shareftp = off (inherited) sharesftp = off (inherited) sharetftp = (inherited) pool = Default canonical_name = Default/local/default/db_u200 exported = true (inherited) nodestroy = false space_data = 130K space_unused_res = 0 space_snapshots = 0 space_available = 58.3G space_total = 130K root_group = other root_permissions = 755 root_user = nobody origin = PREPARE EXADATA COMPUTE NODES FOR DIRECT NFS Before we can mount any file system over the network, we have preliminary system administration tasks that need to be performed on all the Exadata compute nodes. To start, we must start Portmap, NFS and NFSLOCK services. In addition, 5 Session # 414

6 we must enable the services with the chkconfig command to be persistent across server reboots. To start the services, execute the service command from the sbin directory and pass two arguments: the name of the service followed by what we want it to do (start, stop or status). Here s the snippet of code depicts the exact commands that we want to execute to enable our NFS services: - root: cat zfs.1 export PATH=$PATH:/usr/local/bin:/sbin dcli -l root -g /home/oracle/dbs_group /sbin/chkconfig portmap on dcli -l root -g /home/oracle/dbs_group /sbin/service portmap start dcli -l root -g /home/oracle/dbs_group /sbin/chkconfig nfs on dcli -l root -g /home/oracle/dbs_group /sbin/service nfs start dcli -l root -g /home/oracle/dbs_group /sbin/chkconfig nfslock on dcli -l root -g /home/oracle/dbs_group /sbin/service nfslock start Upon execution, you should see results that s similar to what you see here: - root: ksh./zfs.1 exaddb01: Starting portmap: [ OK ] exaddb02: Starting portmap: [ OK ] exaddb01: Starting NFS services: [ OK ] exaddb01: Starting NFS quotas: [ OK ] exaddb01: Starting NFS daemon: [ OK ] exaddb01: Starting NFS mountd: [ OK ] exaddb01: Starting RPC idmapd: [ OK ] exaddb02: Starting NFS services: [ OK ] exaddb02: Starting NFS quotas: [ OK ] exaddb02: Starting NFS daemon: [ OK ] exaddb02: Starting NFS mountd: [ OK ] exaddb02: Starting RPC idmapd: [ OK ] exaddb01: Starting NFS statd: [ OK ] exaddb02: Starting NFS statd: [ OK ] After the services are started and enabled, you can leverage the chkconfig command again followed by the --list option (double dashes) and confirm that the services are configured to re-start as part of the server reboot process. In particular, we want these services to re-start in run levels 3,4,5. [root@exaddb ]# /usr/local/bin/dcli -l root -g /home/oracle/dbs_group /sbin/chkconfig --list egrep -i "nfs port" exaddb01: nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off exaddb01: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off exaddb01: portmap 0:off 1:off 2:on 3:on 4:on 5:on 6:off exaddb02: nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off exaddb02: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off exaddb02: portmap 0:off 1:off 2:on 3:on 4:on 5:on 6:off Next, we want to enable Direct NFS on all the Exadata compute nodes. You can leverage dcli to simply enable Oracle Direct NFS on all of the compute nodes simultaneously. In this example, we are executing dcli (Distributed Command Line Interface) that can be used to run set of commands on all or a subset of compute nodes in parallel. dcli can save you enormous amount of time and can help mitigate human errors that often occur when running a same command multiple times on each individual compute node. dcli -l oracle -g /home/oracle/dbs_group make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk dnfs_on Another requirement for Direct NFS is that the /etc/hosts.allow and /etc/hosts.deny files must have permissions for read write at the owner level and read-only for group and world access. [root@exaddb ]# /usr/local/bin/dcli -l root -g /home/oracle/dbs_group chmod 644 /etc/hosts.allow [root@exaddb ]# /usr/local/bin/dcli -l root -g /home/oracle/dbs_group chmod 644 /etc/hosts.deny We leverage the dcli executable again to confirm that our commands executed successfully. [root@exaddb ]# /usr/local/bin/dcli -l root -g /home/oracle/dbs_group ls -l /etc/hosts.allow 6 Session # 414

7 /etc/hosts.deny exaddb01: -rw-r--r-- 1 root root 161 Jan /etc/hosts.allow exaddb01: -rw-r--r-- 1 root root 347 Jan /etc/hosts.deny exaddb02: -rw-r--r-- 1 root root 161 Jan /etc/hosts.allow exaddb02: -rw-r--r-- 1 root root 347 Jan /etc/hosts.deny Next, we must modify the Linux kernel parameters in the /etc/sysctl.conf file. The first parameter, net.core.wmem_max parameter, resides in the middle of the file. Notice the line that is commented out. That was the original value that was configured with the Exadata: Modify Kernel Parameters (in the middle of the file): #net.core.wmem_max = # -- Modified per ZFS net.core.wmem_max = The new value is what we want to configure for Direct NFS with the ZFSSA. In addition to the net.core.wmem_max parameter, we will also configure the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters to 4 Mbs. Since these are new parameters, we can add them to the end of the sysctl.conf file. # -- At the end of the file # -- # -- Added for ZFS net.ipv4.tcp_wmem= net.ipv4.tcp_rmem= The Exadata compute nodes do not need to be re-started. Once the kernel parameters are set, we can dynamically reload the kernel parameters with the sysctl p command. You can also pass the a option to sysctl command to display all the runtime kernel parameter settings. Again, we can leverage dcli to execute the kernel reload across all Exadata compute nodes. In a quarter rack, this may not be a big deal, but as you start supporting half racks or even a full rack Exadata, dcli will be an important tool for you to master: [root@exaddb ]# /usr/local/bin/dcli -l root -g /home/oracle/dbs_group /sbin/sysctl -p exaddb01: net.ipv4.ip_forward = 0 exaddb01: net.ipv4.conf.default.rp_filter = 1 exaddb01: kernel.sysrq = 1 exaddb01: kernel.softlockup_panic = 1 exaddb01: kernel.core_uses_pid = 1 exaddb01: kernel.shmmax = exaddb01: kernel.shmall = exaddb01: kernel.msgmni = 2878 exaddb01: kernel.msgmax = 8192 exaddb01: kernel.msgmnb = Once the kernel parameters are set and all ancillary preliminary requirements are satisfied, we can proceed to the database component. To enable Direct NFS, we must execute the make command with the dnfs_on option. exaddb01:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib TOOLSDEV1 - oracle: make -f ins_rdbms.mk dnfs_on rm -f /u01/app/oracle/product/11.2.0/dbhome_1/lib/libodm11.so; cp /u01/app/oracle/product/11.2.0/dbhome_1/lib/libnfsodm11.so /u01/app/oracle/product/11.2.0/dbhome_1/lib/libodm11.so You will need to repeat the DNFS settings on every Exadata compute node. 7 Session # 414

8 Another pair of services that need check for are, CPUSPEED and IRQBALANCE. On the Exadata, these services are disabled by default, which optimizes throughput for some network devices. CONFIGURING AND MOUNTING THE NFS SHARE: On the Exadata, Direct NFS (dnfs) is the preferred solution over the Linux kernel managed NFS. Instead of leveraging the operating system kernel NFS client, we can configure Oracle Database(s) to access NFS V3 servers using an Oracle internal Direct NFS client to optimize the I/O access path to NFS server to provide improved scalability and reliability We will have to configure dnfs on the Exadata compute nodes. $ cat /etc/oranfstab server: zfssa-dr-h1 path: path: export: /export/backup1 mount: /zfs/backup1 export: /export/backup2 mount: /zfs/backup2 For our final step for our dnfs configuration, we will manually NFS mount the /dnfs share with the following settings: mount -t nfs -o rw,bg,hard,nointr,noacl,rsize=131072,wsize= ,tcp,vers=3,timeo= :/export/dNFS /dnfs For complete NFS options and settings, please consult the following MOS ID: Mount Options for Oracle files when used with NFS on NAS devices [ID ] BACKUP ARCHITECTURE A typical backup architecture will have the Exadata being backed up over Infiniband to the ZFSSA. The following diagram depicts a customer case study where a pair of ZFS storage appliances were purchased to backup the production Exadata, the Development Exadata, production RAC/non-RAC databases that resided on HP severs and development databases environments that were on HP servers. 8 Session # 414

9 QDR Infiniband (IB) fabric provides 40 Gb of bandwidth per port between the database servers and the ZFSSA. All the Exadata RMAN backups are performed over IB. In this scenario, the Production ZFSSA replicates production backups to what we called the DEV/DR ZFSSA (non-prod ZFSSA), another ZFS storage appliance with twice the number of disks and capacity than the Production ZFSSA. We also purchased the ZFS replication software to replicate backups over the private non-routable 10gigE VLAN. The backups on the Production ZFSSA replicated it shares to the DEV/DR ZFSSA so that we can keep one production sized database copy on Production ZFSSA and another copy on the DEV/DR ZFSSA. All the non-production databases were also backed up to the DEV/DR ZFSSA. SETTTINGS FOR RMAN BACKUPS With RMAN, we can perform image copy backups or backup set backups. Depending on which option we choose, we can tune our backup processes with the following hidden _underscore parameters: _backup_disk_bufcnt Number of buffers used to process backup sets _backup_disk_bufsz Size of the buffers used to process backup sets _backup_file_bufcnt Number of buffers used to process image copies _backup_file_bufsz Size of the buffers used to process image copies The following settings are best practice recommendations for RMAN backups over IB on the Exadata: Type of Backup _backup_disk_bufcnt _backup_disk_bufsz _backup_file_bufcnt _backup_file_bufsz Backup Set Backup Backup Set Restore Image Copy Backup Image Copy Restore Additional guidelines that you should pay careful attention to are: 1. Use 2-4 RMAN channels per each tray of disks 9 Session # 414

10 2. Load balance RMAN channels across shares and controllers 3. Load balance RMAN channels across Exadata compute nodes PERFORM RMAN BACKUPS Here s an code example of a compressed level 0 RMAN compressed backup backupset. Notice that we are leveraging 4 channels with _backup_disk_bufcnt /_backup_disk_bufsz hidden pair of parameters. Depending on whether you are performing image copy backups or backup set backups, you will leverage different _underscore parameters. run { allocate channel d1 type disk; allocate channel d2 type disk; allocate channel d3 type disk; allocate channel d4 type disk; set limit channel d1 kbytes = ; set limit channel d2 kbytes = ; set limit channel d3 kbytes = ; set limit channel d4 kbytes = ; sql 'alter system set "_backup_disk_bufcnt"=64'; sql 'alter system set "_backup_file_bufcnt"=64'; sql 'alter system set "_backup_disk_bufsz"= '; backup as compressed backupset incremental level 0 tag=dbatools_bkup_0z_02feb13_0300 filesperset 1 format '/u01/app/oracle/admin/dbatools/bkups/%d.%s.%p.%t.l0.db' (database) ; # "create pfile=''/u01/app/oracle/admin/dbatools/bkups/init_dbatools_02feb13_0300.ora'' from spfile"; sql "alter system archive log current"; sql "alter system switch logfile"; sql "alter system switch logfile"; change archivelog all validate; sql "alter database backup controlfile to trace"; backup as compressed backupset format '/u01/app/oracle/admin/dbatools/bkups/%d.%s.%p.%t.a' skip inaccessible (archivelog all not backed up 2 times); backup tag=dbatools_ctl_02feb13_0300 format '' (current controlfile); backup backupset from tag DBATOOLS_CTL_02Feb13_0300 format '/u01/app/oracle/admin/dbatools/bkups/%d.%s.%p.%t.ctl'; delete noprompt archivelog until time 'sysdate - 2' backed up 2 times to device type disk; release channel d1; release channel d2; release channel d3; release channel d4; } RMAN Backups are fully documented on the DataGuardBook.com website. Look for the driver shell script called rman2disk.ksh from Two main driver files that are needed for image copy backups are: Replace rman2disk.sql.baseline Replace rman2disk.sql.for_recover Here s a snippet for the rman2disk.sql.baseline script: run { allocate channel d4 type disk; 10 Session # 414

11 sql 'alter system set "_backup_file_bufcnt"=64'; sql 'alter system set "_backup_file_bufsz"= '; backup as copy incremental level 0 tag='forever' format '/u01/app/oracle/admin/###_oracle_sid_###/bkups/%u' (database) ; release channel d4; } This script simply performs a full image copy backup of the database to the dnfs file system. All the RMAN scripts are intelligently designed to follow a symbolic link from the $ORACLE_BASE/admin/$ORACLE_SID directories as directory paths may not exist from one server to another. Now let s look at the rman2disk.sql.for_recover code example: run { allocate channel d4 type disk; #recover copy of database with tag 'FOREVER' from tag 'DAILY'; recover copy of database with tag 'FOREVER'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG 'FOREVER' format '/u01/app/oracle/admin/###_oracle_sid_###/bkups/%d.%s.%p.%t.l1.4r.db' DATABASE; release channel d4; } This script performs two main tasks. First it will perform a recovery of the image copy (update it with the last incremental backup). Then, a level 1 incremental backup is taken with the sole purpose to be recovered against the image copy later on. If we stagger the incremental update (recover copy of database with tag) command prior to performing our level 1 incremental backup, we can achieve another 24 hours of protection on the primary production database from end user errors or even corruptions. Many customers do not have the luxury of keeping flashback logs for 24 hours. If we delay our incremental update, we can switch the datafile to copy or even restore the datafile from the ZFS file system and be able to achieve a higher level of SLA. OTHER INDUSTRY USE CASES The ZFS storage appliance is not just for the Exadata. We can leverage it anywhere we can leverage a NAS appliance. Oracle E-Business Suite has a requirement for a shared apps tier. We can create shares on the ZFSSA to be presented to the E- Business Suite application server. E-Business Suite also has a requirement for a share on the database server to be shared with the application server. Again, a share can be presented from the ZFSSA. One thing you will hear more than anything else will be that the ZFSSA can expand and protect Oracle Database Appliance (ODA). On the ODA, we get far more than additional capacity and place to backup the database too. Oracle also provides full HCC support on the ZFSSA. If number of disks and controllers are purchased to accommodate IOPs and Throughput, we can run production databases on the ZFSSA. The workload of mission critical databases can run on the ZFSSA if proper capacity sizing exercises are performed. Other use cases for the ZFSSA include: Plug and Play with your existing environment on 10gigE as NFS Appliance Can run RAC or non-rac databases over fiber channel (not just over dnfs) Rapidly provision Physical Standby or Active Standby with Snapshots and Clones Rapidly clone databases with snapshots 11 Session # 414

12 SNAPSHOTS Snapshots are point-in-time read-only copy of the file system or LUN. Snapshots initially do not consume any space but as the active share data content changes, the previous unreferenced blocks will be kept as part of the snapshot (copy-on-write). Over time, if the active data volumes change, the snapshot can grow as big as the file system at the time the snapshot was created. Shares support the ability to rollback to a previous snapshot. When you perform a rollback, newer snapshots and clones of the snapshot will be destroyed. To create a new snapshot of a share, drill down into the share properties and click on the + sign next to the Snapshots label on the header of the Snapshot section. When you click on the + sign, the Create Snapshot window will pop up. Designate a name to your snapshot and click on the Apply button. As a general rule, you should determine a naming strategy to name your snapshots since we can have unlimited number of snapshots so that you can apply purging rules or find them easily if you have hundreds of snapshots. CLONES A clone is a writeable copy of a snapshot share. Similar to a snapshot, a clone will not consume any space initially but as new changes occur on the cloned copy, the size of the clone will grow relative to the changes that happen on the cloned copy. Space is shared between snapshots and clones. You can have multiple clones per snapshot. The negative caveat is that when you destroy a snapshot, you will also have to destroy all the clones associated with the snapshot. To create a clone of a snapshot, simply click on the Create Clone + sign in the Clones section. The Create Clone window will appear for you to provide some basic information. 12 Session # 414

13 Similar to the snapshot, you must designate a name to the snapshot clone. The name of the clone cannot have a # or similar metadata characters. A snapshot clone is considered to be a snapshot in all-practical purposes when it comes to administration. You can have unlimited number of clones for a snapshot. WORKING WITH SNAPSHOTS AND CLONES WITH DATA GUARD On the other side of a Data Guard configuration, if you place your physical standby data files on the ZFSSA, you can create snapshots and clones of the physical standby database and rapidly provision cloned databases for DEV, QA, hot fix testing, or stress testing purposes. The process for creating cloned databases from a snapshot is quite simple and extremely fast. Simply stop the managed recovery process (MRP) to place the standby database in a consistent state. From the ZFSSA BUI, take a snapshot of the project or filesystem where the physical standby resides. Once the snapshot is taken, you can resume MRP on the physical standby database. From the snapshot taken above, create a clone of the file systems from the BUI. Next, you can dnfs mount the cloned file system to the new database server at the physical standby site. As the final step, you simply open the cloned physical standby database in read-write mode. Unlike the snapshot standby feature of Oracle Database 11g, we do not have to worry about running out of space for the archived logs or the amount of time it will take to flashback the database. Another beauty of this architecture is that we can create multiple snapshots and literally unlimited number of clones. The number of snapshots and clones that we create is 100% dependent on the amount of space available on the ZFSSA. Also imagine multiple components of the application that can be tested at the same time. How often do you hear that the one part of the QA team has to wait for the other part of the QA team to finish so that they do not step on each other. Also, imagine application releases being tested simultaneously with multiple clones of the database to deliver a faster application release cycle. If you also happen to be running your DEV environment next to your physical standby, you can have full production sized DEV databases for testing. If you do not modify that much data in your DEV environment, which is what we would expect, the actual space consumption would only be the size of your changes that you make. You can also refresh your DEV/QA environments exponentially faster. What would normally take days or a day or hours can now be done in minutes. 13 Session # 414

14 APPENDICES SETTINGS ON ZFS SHARE For purposes of performing RMAN image copy backups and backup sets, set the synchronous write bias to Throughput. For incremental backups and incremental updates to image copies, set the synchronous write bias to Latency. Just like any storage vendor, Oracle ZFSSA can also set quotas and minimum reservations. We can specify in our reservations to include the snapshots as part of the quota. Notice that we have the capability to set data deduplication on or off. We will want to setup deduplication if we store database files or other file that contain duplicate data. For compression, here are the various compression options on the ZFSSA. For RMAN backups from Exadata to the ZFSSA, we will want LZJB if we want to enable compression from the storage tier. LZJB (Fastest) lzjb A simple run-length encoding that only works for sufficiently simple inputs, but doesn't consume much CPU. GZIP-2 (Fast) gzip-2 A lightweight version of the gzip compression algorithm. GZIP (Default) gzip The standard gzip compression algorithm. GZIP-9 (Best Compression) gzip-9 Highest achievable compression using gzip. This consumes a significant amount of CPU and can often yield only marginal gains. DISABLING SERVICES FOR 10GIGE These services, cpuspeed and irqbalance, can reduce NFS throughput over 10 Gb Ethernet. If these services are not being used, or their use is less valuable than maximizing NFS performance over 10 Gb Ethernet, the services can be manually disabled after boot or dynamically disabled with the chkconfig and service commands: 14 Session # 414

15 # chkconfig cpuspeed off # service cpuspeed stop # chkconfig irqbalance off [root@exaddb ]# /sbin/chkconfig --list egrep -i "cpu irq" irqbalance 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@exaddb ]# /sbin/service irqbalance status irqbalance (pid 6756) is running... REFERENCES Oracle White Paper April 2012: Backup and Recovery Performance and Best Practices using the Sun ZFS Storage Appliance with the Oracle Exadata Database Machine Oracle White Paper - September 2011: Protecting Oracle Exadata with the Sun ZFS Storage Appliance: Configuration Best Practices UPDATES This document will also continue to evolve. You can also expect this document to grow and grow over time. Please check for updates from the viscosityna.com website. AUTHOR BIOGRAPHY Charles Kim is an Oracle ACE Director, an Oracle Certified Professional, an Oracle Certified RAC Expert, and a VMware Certified Professional. Charles regularly presents on advanced topics for IOUG and Oracle OpenWorld such as RAC/ASM, Exadata, ZFS Storage Appliance, Linux, Data Guard, and 7x24 High Availability Considerations. His experience spans variety of Unix platforms in both SMB and Fortune 500 companies with specialization in high volume, mission-critical, real-time OLTP databases. Charles works predominately in the Maximum Availability Architecture (MAA) space (RAC, ASM, Data Guard, and other High Availability solutions). Charles released his first book titled Oracle Database 11g New Features for DBA and Developers on November Charles also co-authored the Linux Recipes for Oracle DBAs with APress to be released in November Unleashing his expertise in the maximum availability sector, Charles is also the co-author of the Oracle Data Guard 11g Handbook with Oracle Press. Charles is also the author of the MAA case study at Oracle's Website ( Charles holds certifications in Oracle, Red Hat Linux, and Microsoft, has over 21 years of IT experience and has worked with Oracle since Charles blogs regularly at and provides technical solutions to Oracle DBAs and developers. 15 Session # 414