Planning & Executing your Upgrade to NBU Suresh Suseendran, Lead Architect Praveen Vunnava, Lead Architect

Transcription

1 Planning & Executing your Upgrade to NBU Suresh Suseendran, Lead Architect Praveen Vunnava, Lead Architect Planning & Executing your Upgrade to NBU 7.6 1

2 Agenda 1 Top 5 Reasons for Upgrading to NBU Pre-Upgrade Steps/ Best Practices 3 Upgrade Best Practices 4 Post Upgrade Best Practices 5 Upgrade Tools/Scripts Planning & Executing your Upgrade to NBU 7.6 2

3 Top 5 Reasons for Upgrading to NBU 7.6 3

4 Top 5 Reasons for Upgrading to NBU 7.6 Superior Integration with VMware environments Next Generation Deduplication Architecture Storage Life Cycle Windows and Targeted Auto Image Replication Intelligent Policy Framework for Oracle Replication Director for VMware, Application, and Block Array Support 4

5 Pre-Upgrade Steps/Best Practices 5

6 The Customer Experience with NetBackup Upgrades Install & upgrade takes too long Maintenance windows are too narrow Can t collect product inventory to meet business needs Limited tools available to identify and mitigate risks No time to wait on hold with Support Tasks Customers Perform 1. Installation, configuration and upgrades 2. Planned and unplanned maintenance 3. Reduce and eliminate risk 4. Troubleshooting 5. Define and manage standard builds 6. Obtain and deploy license / manage compliance 7. Server to storage connectivity (path mgmt) 8. Performance tuning post upgrade 9. Application level protection 10. Implement new features post upgrade 11. Planning for all activities

7 What is Symantec Operations Readiness Tools? SORT is a website designed to: Support Symantec enterprise products Automate and simplify some of the most time-consuming administrator tasks Use SORT to: Determine if systems are ready for product installation or upgrade Generate checklists Generate system specific custom reports Media, Master, and client systems Get a list of deployed hot fixes (and EEB s) on a set of systems Identifies if any EEB related fixes have been addressed or not Determine fixed-in state with the NetBackup Hot Fix and EEB Release Auditor Quickly link to NetBackup documents, Support, SymConnect, etc View Future Platform and Feature Plans for early notification of planned changes to supported items

8 Improving the Total Customer Experience with SORT Enhance the overall product experience for Symantec Enterprise users in three key areas Manage Risks Improve Operations Efficiency Align to Best Practices Real-time patch notifications Risk assessments and mitigation pre-upgrade (EEB s) Actionable server error reports Pre-Upgrade checks to validate readiness of the environment Trouble-free installations/ upgrades Product and license key inventory Advanced support-based searches Extended error code solutions Server and environmental specific recommendations Tune and optimize customer environment (Shared memory tunable verification & recommendations)

9 Pre-Upgrade Steps / Procedure : SORT Data Collector

10 Pre-Upgrade Steps / Procedure : SORT Data Collector

11 Pre-Upgrade Steps / Procedure : SORT Data Collector

12 Pre-Upgrade Best Practices SORT should be used as the starting point Generate an installation check list from Symantec Operations Readiness Tool (SORT). ( Review Upgrade guide ( Review Upgrade Portal ( ) NetBackup Catalog Consistency Utility (NBCC) (

13 Pre-Upgrade Best Practices Pre-checks Perform catalog consistency check to validate the catalog and NBDB Perform sanity check on EMM NBDB unload may be required to fix some EMM inconsistencies: Open a support case to perform this (if required) Fix any configuration related issues Fix any dedupe disk pool / ost related issues

14 Pre-Upgrade Best Practices Catalog conversion Catalog size (Please review the Upgrade Guide for guidelines on time estimation) Catalog size is important : catalog migration is part of upgrade process Database rebuild: NBU 7.6 utilizes version of Sybase SQL Anywhere database Modify server.conf to increase the -ch setting (change it to at least half the physical memory of the master) so that conversion process would complete faster. Please modify it back after the upgrade is complete Note: Time required to upgrade from 7.5.x is longer than performing upgrades from pre-7.5 versions since the database (NBDB) is small prior to NBU 7.5

15 Pre-Upgrade Best Practices Perform consistent backup of NetBackup catalog for any rollback or recovery purpose. Perform consistent backup of OS for any rollback or recovery purpose. If NetBackup binaries are under OS drive & RAID(1) control & protect the drive by breaking the mirror, preserve one drive in case of any roll back required. Verify enough space is available for both catalog conversion and also policy information (class folders/directories) : Upgrade Guide Run NBSU on master & Media servers to gather configuration info & other related information about the master / media servers.

16 Pre-Upgrade Best Practices Perform a nbdb online backup to save emm DB in a separate location (preferably on a different server) other than the same volume as catalog. Verify and make copies of the following prior to Upgrade: Configuration touch files Custom scripts Any other miscellaneous customization performed to NBU configuration files Validate & download all respective patches that are required from OS perspective to support NetBackup x version utilize SORT for this.

17 Pre-Upgrade Best Practices Cleanup all NetBackup configuration files and remove unwanted entries In case of clustered master server, validate the compatibilities of VCS & OS binaries & backup all configuration files etc. Recommended to have a minimum of 20 to 30% free space available on the following file systems NetBackup binary, catalog, log volumes, MSDP. Enable logging for daemons like EMM, nbpem, nbstserv, bpdbm, bprd in a proactive manner to collect and analyze logs in case of any failures. Make sure the emm.conf, server.conf file entries are having appropriate values to support the environment.

18 Pre-Upgrade Best Practices NetBackup Appliances In case of Symantec appliances validate and confirm the following : Make sure IPMI configuration is enabled and accessible Minimum of 20 to 30 % of disk space should be available for Catalog & MSDP volume. Perform overall health check of the appliance hardware components to find out for any kind of hardware issues and address any kind of hardware issues by opening support cases accordingly.

19 Pre-Upgrade Planning - Catalog DB Sybase rebuild NBU 7.6 utilizes version During the upgrade,.db files are migrated to the same version for better performance. Requires about 7.5 minutes per GB (12 minutes for BMR_DATA.db). total Size of EMM DB multiply by 7.5 BMR DB multiply by 12. BMR_DATA.db 7.5+ file size can become large Check the size of the file and if greater than 1GB, requires to perform a pre-install purge. TECH will detail the steps. Failure to purge wont cause any issues but rebuild will take longer times. Post 7.6 upgrade, the purge is built in DO NOT loose patience and kill this process, or DB recovery will be needed. 19 Symantec Confidential: Internal Use Only

20 Symantec Confidential: Internal Use Only Pre-Upgrade Planning - MSDP conversion The MSDP metadata and reference management changes in NetBackup require that the existing database records be converted to the new format during an upgrade installation. Installer will ask if conversion should be started automatically or manually. As a best practice, it is recommended to let the installer start the process. SPAD/SPOOLD will not run until the conversion is complete, and the conversion process will start them when it is done. 20

21 Symantec Confidential: Internal Use Only NetBackup Pre Upgrade Tools Migration_check.pl/.exe SORT data-collectors are similar Performs image counts in best available way, and understands what to count. Can be run during and/or after the upgrade to check results Install_prep.pl/.exe Record current state of policies/slp s/dssu schedules Deactivate so that scheduled tasks won t happen Reactivate once upgrade is complete Parallel_cat_import.pl/.exe Performs the Guided Method many-at-once client import 21

22 Symantec Confidential: Internal Use Only NetBackup Pre Upgrade Tools : MSDP pddeobjectcount.pl Performs MSDP object image counts in best available way, and provides an estimation of the overall MSDP upgrade time frame. Should be run before the upgrade. Detailed information of how to execute and interpret the results of the tool is explained in the technote : ECH

23 Upgrade Best Practices 23

24 Upgrade Best Practices Perform the NetBackup Upgrade in the following order: NBU OpsCenter server NBU master server NBU media servers NBU clients 24

25 Upgrade Best Practices : Installer Pre-Checks The installer now checks for certain conditions that are known to cause issues, and will prevent the install from starting Does enough free space exist in the data directory to actually rebuild the.db files? Does enough free space exist in the MSDP pool to perform the conversion? Do links exist in the NetBackup directory structure on Windows? Updates to the latest version (must have internet connection for this to work) EEB s, any known fixes as well any identified upgrade issues so far.

26 Symantec Confidential: Internal Use Only Upgrade Best Practices Run a hot catalog backup. Validate the performance of the Catalog file system and make sure it is not fragmented and yielding best performance for read & write I/O. Stop all customized or third-party scripts. Disable OpsCenter data collection for this master server. Disable all storage lifecycle policies (SLPs). Deactivate all NetBackup policies. Deactivate all disk staging storage units for all pre-netbackup 7.5.x environments. 26

27 Symantec Confidential: Internal Use Only Upgrade Best Practices Perform any cluster-specific tasks For clustered NBU Master servers, take the following NetBackup resources offline: MSCS clusters: Take all of the NetBackup group resources offline except for the disk, the virtual name, and the virtual IP address. VCS clusters: Take the NetBackup resource offline. 27

28 Symantec Confidential: Internal Use Only Upgrade Best Practices MSDP/NBU Appliances Do not stop or restart services on a MSDP media server or appliance during the MSDP data conversion. Please wait for the conversion to complete In case appliance related Upgrade please make note of these : Rollback after upgrade is complete is not supported Upgrade takes a long time plan about 2 hours+ Mostly due to time for ISO install After rollback unable to attempt upgrade again for 3+ hours 28

29 Post Upgrade Best Practices 29

30 Symantec Confidential: Internal Use Only Post Upgrade Best Practices Run a NBCC and validate the consistency of the catalog, to validate the sanity of EMM. Once the catalog migration and the MSDP conversion finish, reactivate the following in the order as shown: All disk staging storage units. All NetBackup policies. All storage lifecycle policies (SLPs). OpsCenter data collection for this master server. Perform test backups & restores for few of the policies/critical clients. 30

31 Symantec Confidential: Internal Use Only Post Upgrade Best Practices Perform a full backup of the catalog file system & file system hosting NBU binaries. In case of a clustered master server, fail over the services between nodes and verify the functionality & performance. Update parameters in configuration files for MSDP / EMM to get optimized performance. Configure SLP windows to enable scheduling of duplication or replication jobs. 31

32 Symantec Confidential: Internal Use Only Post Upgrade Best Practices If you upgraded a master server that allowed nonroot users to administer NetBackup, you must reconfigure the permissions and the group. The default permissions and group on the newly installed files allow only a root user to perform NetBackup administration. If you made changes to NetBackup scripts before the upgrade, apply those changes to the new, upgraded versions of the scripts. Upgrade any add-on products (such as NetBackup language packages) on all upgraded clients. All add-on products should be at the same version as the NetBackup client. 32

33 Symantec Confidential: Internal Use Only Post Upgrade Best Practices If any touch files were leveraged prior to the upgrade, some of the touch file parameters have been migrated to Master Server Host Properties (For example: SLP parameters) Please review Upgrade Guide on all of the different parameters Verify the changes to any commands to make sure that your custom scripts are not affected : Upgrade Guide 33

34 Upgrade Tools/Scripts 34

35 Symantec Confidential: Internal Use Only Upgrade Tools/Scripts Pre Upgrade Scripts / Tools : Migration_check.pl/.exe Install_prep.pl/.exe Parallel_cat_import.pl/.exe pddeobjectcount.pl Post Upgrade Scripts / Tools : To start the conversion of MSDP database manually use this script : /usr/openv/pdde/pdconfigure/scripts/installers/pdde_convert.sh Excute this script to cleanup storage artifacts from the conversion process : /usr/openv/pdde/pdcr/bin/stconv cleanup 35

36 NBU Upgrade Use Cases 36

37 Use Case 1 NBU Appliances Architecture 37

38 Use Case 1 Pre-Upgrade Steps Generate the SORT checklist for upgrade ( Verify compatibility of all the components identified in the checklist report generated from SORT website NBU master, media and clients Identify any Late Breaking News and EEB s that are required as part of the upgrade Verify the space requirements for upgrade process to complete successfully Identify the catalog size so that approximate time estimate can be made for catalog conversion as part of the NBU master server upgrade on NBU 5230 appliance 7.5 minutes per GB of EMM database 12 minutes per GB of BMR database Identify any client platforms that might be impacted as part of the upgrade 38

39 Use Case 1 Pre-Upgrade Steps Run pddeobjectcount.pl script on all of the NBU 5230 media server appliances Provides the time estimate required to convert the MSDP database to new refdb format Include an additional hour to the estimate to be conservative Run NBCC on both master server appliances Perform sanity check on EMM database of both master server appliances NBDB consistency check Fix any issues that were identified as part of both NBCC and NBDB consistency checks Perform a successful backup of OpsCenter database 39

40 Use Case 1 Pre-Upgrade Steps Cleanup all NetBackup configuration files and remove unwanted entries (touch files) Create copies of any customs scripts that were implemented Stop NBU and create copies of NBDB (copy it to a remote server) on both NBU 5230 master server appliances Disable OpsCenter data collection for both NBU 5230 master server appliances Modify server.conf file on both master server appliances to change - ch setting to half the physical memory of the NBU 5230 appliance to help speed up the catalog conversion during upgrade of the master server Identify any nbrb.conf settings prior to upgrade to make sure that they are addressed after the upgrade is complete : nbrb.conf settings are migrated into EMM database 40

41 Use Case 1 Pre-Upgrade Steps Delete any unwanted policies, storage unites, entries in EMM database (for example: de-commissioned media servers etc..) Run a catalog backup job on both master server appliances at both data centers (A & B) Deactivate the following components (Install_prep.pl): NBU policies including SLPs Any custom script execution Any cron jobs defined for NBU 5230 appliance maintenance tasks Identify any CRON or 3 rd party scheduled jobs and deactivate them 41

42 Use Case 1 Upgrade Upgrade the environment in the following order: NBU OpsCenter server NBU 5230 master server appliance at Data Center B NBU 5230 master server appliance at Data Center A NBU 5230 media server appliance at remote site A NBU 5230 media server appliance at remote site B NBU 5230 media server appliances at Data Center A NBU 5230 media server appliances at Data Center B NBU clients at remote sites & data centers 42

43 Use Case 1 Post Upgrade Activities Run catalog consistency check Activate the following components: All of the policies including SLPs Any custom script execution Any CRON jobs defined for NBU 5230 appliance maintenance tasks Any CRON or 3 rd party scheduled jobs OpsCenter data collection on the master server appliances Run catalog backup Run acceptance testing Backups, Restores of a few select clients & data types Verify and test SLP/optimized duplication 43

44 Use Case 1 Post Upgrade Activities Update server.conf on NBU 5230 master server appliances to change -ch setting back to optimal value Update custom scripts to accommodate for changes to any NBU commands & settings with the upgrade (if any) Update parameters in configuration files for MSDP / EMM to get optimized performance (SLP parameters, SEGK size, thresholds, high/low water marks, max i/o limits on MSDP storage pools etc..) Make sure any changes to nbrb.conf are updated in EMM database after the upgrade (nbrbutil listsettings) Implement new features like targeted AIR, SLP windows for better efficiencies across the entire architecture 44

45 Thank you! Suresh Suseendran, Lead Architect Praveen Vunnava, Lead Architect Copyright 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. 45