Patching Exadata Database Machine - the DBA gets it all!

Transcription

1 Patching Exadata Database Machine - the DBA gets it all! DOAG Conference 2011 Konrad Häfeli Senior Technology Manager Trivadis AG BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN 1

2 AGENDA 1. Patching Overview 2. Patch Application Methods 3. Patch Administrator/Manager? 4. Patching Exadata V2 to X Conclusion 2

3 Exadata 1. Turn Key solution? 2. Appliance? generally "closed and sealed" not serviceable by the owner 3. Black Box? 3

4 Preconfigured, balanced System 1. In terms of: Processors Memory Storage I/O Interconnect 4

5 Exadata responsibility?! Who is responsible for maintaining the supportability? Who is responsible for patching? The operator is responsible! Keep on patching! 5

6 Source of Support: MOS Note The Maininformation for Version Support Compatibility Matrix 6

7 Exadata Patch Stacks/Types Patches apply to 3 Stacks: Database Server RDBMS Grid Infrastructure Bundle Patch InfiniBand Switches Infiniband Switch Patches Exadata Storage Server Operating System Firmware Storage Server Patches KVM and Cisco No Oracle patches 7

8 Database Server Patches Delivered in bundle patches created specifically for Exadata Database and Grid Infrastructure Homes Monthly released (until Oct 2011, then 2 months) Contain a recently released Patch Set Update (PSU), which contain a recently released Critical Patch Update (CPU) Bundle patches are cumulative Installation with OPatch utility 8

9 Exadata Storage Server Patches Updates for OS, Kernel, InfiniBand, ILOM, firmware and new features Must occur only with an Exadata Storage Server patch provided as a single downloadable patch from My Oracle Support Do NOT manually update firmware or software on storage servers. Includes the minimal OS pack for database server Quarterly released Installation using a script supplied with the patch called patchmgr rolling and non-rolling application 9

10 Infiniband Switch Patches Install only Oracle patches MOS note is the reference Prerequisites and instructions for installing a patch are provided in a README Once or twice a year released Installation depends on the version, check first InfiniBand switch software version has no dependency on Exadata Storage Server software version 10

11 Patch Guideline Exadata Patching Overview and Patch Testing Guidelines MOS Note Systems that are in production or are in late testing stages before production should plan to periodically adopt more current patch releases. It is not required or necessary to install every new patch release. A patch should be installed on a production system only after it has been validated in a proper test environment, and no less than one month after release to allow field experience to solidify. However, if the system requires a fix that is available in a newer version, then plans to adopt a newer version should be accelerated 11

12 AGENDA 1. Patching Overview 2. Patch Application Methods 3. Patch Administrator/Manager? 4. Patching Exadata V2 to X Conclusion 12

13 Reduce Patching Risks Patching used to be a bit difficult/complicated Since later releases feature to reduce Downtime and Risk Feature Minimum Version Cell Rolling Apply Bundle Patch Merge RAC Rolling Installable OPatch Auto Installable EM Installable Dataguard Standby-First Installable OPlan GI_BP DB_BP GI_BP BP DB_BP BP DB_BP BP BP2 13

14 Rolling Bundle Patch Installed with OPatch utility Since opatch auto Before apply or auto depending the patch OPatch auto One command per node Database has to be registered -oh <ORACLE_HOME> patches specific home Stop Oracle Home Patch Oracle Home Stop CRS Unlock Grid Home Patch Grid Home Start CRS Lock Grid Home Start Oracle Home 14

15 Storage Server Patches Installed with utility: patchmgr Includes operating system updates, firmware updates, new functionality Patchmgr uses dcli (distributed command line interface) Rolling apply Stage Cells CELL 1 Offline ASM CELL1 Patch CELL 1 Online ASM or non rolling apply 15

16 Storage Server Patches rolling vs. non-rolling ROLLING Pro No downtime In case of problems just one cell affected Contra Long apply time No/reduced redundancy during apply 1.5 to 3 hours per cell NON-ROLLING Pro 1.5 to 3 hours overall Reduced patch window Contra System downtime during apply In case of problems all cells affected 16

17 Infiniband Switch Patches Infiniband switches are running CentOS Linux 5.2 Infiniband patches can be installed in a rolling fashion Patch the first switch, and wait for reboot to complete, then repeat patch process on remaining switches Older versions (1.1.3) were installed by placing update files on a web or FTP server, and downloading RPM package updates to the switch Later versions (1.3.3) were installed by placing the update package on the filesystem of the switch, and updating from the ILOM of the IB switch 17

18 AGENDA 1. Patching Overview 2. Patch Application Methods 3. Patch Administrator/Manager? 4. Patching Exadata V2 to X Conclusion 18

19 Exadata Administrator Who is in charge for patching? Oracle Software Stack allows Separation of roles Database Home Grid Home (Cluster and AMS functionality) Does it really need a SysAdmin? What is to do? - OS Maintenance (not only minimal pack) - Usermanagement - Crontab setup/enabling - Sendmail config - Network configuration (changes) - 19

20 DBA or not? To get the most synergy combine the jobs DBA can do it No fear to look over the border Interdisciplinary work done with checklists Job enrichment no matter from which side DBA SysAdmin SysAdmin DBA 20

21 Teamwork Patching big systems is time consuming Checking dozens of readmes is error-prone Doublecheck your work in a team 21

22 AGENDA 1. Patching Overview 2. Patch Application Methods 3. Patch Administrator/Manager? 4. Patching Exadata V2 to X Conclusion 22

23 Reason for patching Lifecycle of an old V2 quarter rack RDBMS needed which requires Storage Server Version x Prepare the V2 for scaling out to a half Rack (with an X2-2 Quarter) Complex Upgrade/Patch of all Components 23

24 Patch process: a story with ups and downs Get Infos from «gurus»: A lot of tipps but most not really relevant for my actual configuration RTFM Read the fine MOS-Note ;-)

25 Infiniband Switch Upgrade 25

26 Infiniband Switch Patching Infiniband Switches are supposed to be patched first The procedure must be taken in three steps Users of FW version will need to upgrade to (patch ) before upgrading to # nm2version NM2-36p version: Build time: Sep :52:51 ComExpress info: Manufacturing Date: Serial Number: "NCD2T0059" Hardware Revision: 0x0006 Firmware Revision: 0x0102 Packageload via Webserver 26

27 Infiniband Switch Patching (2) Exadata patch , Infiniband switch software Infiniband NM2 36p payload patch tmp]# disablesm Stopping IB Subnet Manager.. [ OK ] [root@bdm2sw-ib2 tmp]# tmp]# /tmp/ibswitchcheck.sh spsh pre Sun(TM) Integrated Lights Out Manager Current Version [root@bdm2sw-ib2 version ILOM 3.0 of switch r47111 tmp]# is:./ibswitchcheck.sh and is a leaf post Switch Copyright target 2009 version Sun Microsystems, is: Inc. All rights reserved. [SUCCESS] Use Current is subject Switch version to meets license of minimal switch terms. is: patching level and [SUCCESS] is a leaf. Switch target version is: > load [SUCCESS] -source Switch /tmp/sundcs_36p_repository_1.3.3_2.pkg is at target patching level. NOTE:. Firmware upgrade will upgrade the SUN DCS 36p firmware. [SUCCESS] ILOM will /conf/configvalid enter a special mode is 1 to load new firmware. No other tasks should be performed in ILOM until the firmware Overall status of post check is SUCCESS 27

28 Storage Server Upgrade 28

29 Storage Server Patching Download mentioned the Patches Exadata Storage Server Patching consists of: Patch Exadata Storage Server software (Note ) Check: Exadata Critical Issues MOS Note: All Issues solved with Storage Server Image Master Note for Oracle Database Machine and Exadata Storage Server (Doc ID ) Before and after patch application, run HealthCheck MOS Note: to verify software, hardware, and firmware versions and configuration best practices 29

30 Uprade sequence Switch the Primary-DBs to Standby-Site Leaves in a MAA-Environment the Standby-Site free for «offline» Patching Run Checkscripts to verify software, hardware, and firmware versions and configuration best practices - MOS Note exachk is the current version. HealthCheck is frozen and retained for backward compatibility with HP hardware Old HealthChecks: #./run_os_commands_as_root.sh \ -a /home/oracle/healthcheck \ -b /u01/app/11.2.0/grid \ -c /u01/app/11.2.0/grid \ -d /u01/app/oracle/product/11.2.0/dbhome_1 30

31 Uprade sequence (2) New exachk # oracle@bdm2db01:~/exachk/ [EXATEST1]./exachk -a Report. ============================================================= Node name - bdm2db02 ============================================================= Collecting - CPU Information Collecting - CRS active version Collecting - CRS oifcfg Collecting - CRS software version Collecting - Cluster interconnect (clusterware) Collecting - Compute node PCI bus slot speed for infiniband HCAs Collecting - Exadata storage cells [DBMV2] Collecting - Kernel parameters. 31

32 Uprade sequence (3) Following README for patch Review the support note Find the model of the cell or database host # dmidecode -s system-product-name Applying the Patch with No Deployment-wide Downtime also known as a "rolling update (Does not need any database downtime) in worst-case conditions or when unexpected conditions occur, it can lead to Oracle ASM repair timeout being reached - resulting in Oracle ASM dropping the grid disks on the cell - Re-adding these grid disks is a time consuming manual operation - It also triggers a rebalance by Oracle ASM. It is recommended that only light loads or no loads be running on the system during patching to avoid timeouts or having the grid disks dropped by Oracle ASM after disk repair times expire 32

33 Uprade sequence (4) ~]# imageinfo Kernel version: el5 #1 SMP Tue Jun 30 16:48:30 EDT 2009 x86_64 Cell version: OSS_ _LINUX.X64_ Cell rpm version: cell _linux.x64_ Active image version: Active image activated: :24: Active image status: success Active system partition on device: /dev/md5 Active software partition on device: /dev/md7 In partition rollback to : Possible Cell boot usb partition: /dev/sdm1 Cell boot usb version: Inactive image version: undefined Rollback to the inactive partitions: Impossible 33

34 Uprade sequence (5) Setup Patchdepot on DB-Server Config a cell_group file with all cell-ips Check Prerequirements #./patchmgr -cells cell_group -patch_check_prereq Stop all DB- and Cell-Services dcli -g dbs_group -l root \ "/u01/app/11.2.0/grid/bin/crsctl stop crs -f dcli -g dbs_group -l root "ps -ef grep grid" dcli -g cell_group -l root \ "cellcli -e alter cell shutdown services all" 34

35 Uprade sequence (6) Patch all Cells #./patchmgr -cells cell_group patch. 14:33-25-May:2011 :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell :14-25-May: of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes... 16:14-25-May: of 5 :SUCCESS: DONE: Check the state of patch on cells. Needs approx. 1 ¾ hour # cat /etc/redhat-release Enterprise Linux Enterprise Linux Server release 5.5(Carthage) Cleanup #./patchmgr -cells cell_group -cleanup 35

36 DB-Server minimal Pack (OS and Firmware) 36

37 DB-Server minimal Pack Start Console via ILOM start /SP/console Check imagehistory and stop services # /usr/local/bin/imagehistory # /u01/app/11.2.0/grid/bin/crsctl stop crs f # cd /opt/oracle.oswatcher/osw #./stoposw.sh Adapt /etc/security/limits.conf (memory limits to ¾ physical memory) Unzip db_patch_ zip (included in Storage Patch) Start the patch./install.sh -force 37

38 DB-Server minimal Pack (2) # /usr/local/bin/imageinfo Kernel version: el5 #1 SMP Tue Jun 30 16:48:30 EDT 2009 x86_64 Image version: Image activated: :52: Image status: success System partition on device: /dev/sda1 # cat /etc/redhat-release Enterprise Linux Enterprise Linux Server release 5.3 (Carthage) # uname -r el5 # rpm -qa grep ofa ofa el # /opt/megaraid/megacli/megacli64 -LDInfo -Lall -a0. Default Cache Policy: WriteThrough, ReadAheadNone, Direct, Current Cache Policy: WriteThrough, ReadAheadNone, Direct, 38

39 Troubles with one database node At this point bdm2db02 failed to reboot. We had this message in the ILOM console during boot process (hard to get hold of, all runs through very quickly ): Adapter at Baseport is not responding No MegaRAID Adapter Installed The controller had to be replaced! After replacement it had as to be expected a too low firmware version: # dcli -l root -g all_group "/opt/oracle.supporttools/checkhwnfwprofile -c strict"> /tmp/ck.out [root@bdm2db01 tmp]# more ck.out bdm2db01: [SUCCESS] The hardware and firmware profile matches one of the supported profiles bdm2db02: [WARNING] The hardware and firmware are not supported. 39

40 Troubles with disk controller Upgrade Controller firmware ~]# /opt/oracle.supporttools/checkhwnfwprofile -U /opt/oracle.cellos/iso/cellbits/ Now updating the disk controller firmware... Now disabling cache of the disk controller... Battery capacity is below the threshold value So policy Change to WB will not come into effect immediately Fix Diskcontroller configuration to WriteBack /opt/megaraid/megacli/megacli64 -LDSetProp WB -Lall -a0 /opt/megaraid/megacli/megacli64 -LDSetProp NoCachedBadBBU -Lall -a0 /opt/megaraid/megacli/megacli64 -LDSetProp NORA -Lall -a0 /opt/megaraid/megacli/megacli64 -LDSetProp Direct -Lall -a0 40

41 Upgrade DB Server 41

42 Uprade sequence to Database Upgrade on Exadata Database Machine (Doc ID ) There are four main sections to the upgrade: Prepare the Existing Environment - environment must be at certain minimum levels before upgrade to Install and Upgrade Grid Infrastructure to always performed in a RAC rolling manner Install Database Software - into a new ORACLE_HOME directory with no impact to the current env Upgrade Database to requires database-wide downtime - Rolling upgrade with Logical Standby or Golden Gate may be used to reduce database downtime 42

43 Download required files Files staged on first database server only Patch Oracle Database 11g, Release 2 ( ) Patch Set 1 for Linux x p _112020_linux-x86-64_1of7.zip - Oracle Database - p _112020_linux-x86-64_2of7.zip - Oracle Database - p _112020_linux-x86-64_3of7.zip - Oracle Grid Infrastructure - p _112020_linux-x86-64_7of7.zip - Deinstall tool 43

44 Download required files (2) Files staged on all database servers. Patch OPatch latest update Patch Fix for CRS rolling upgrade bug Latest Database bundle patch for Exadata Data Guard only - Bundle patch overlay fix for unpublished bug See Document for details. - The overlay patch required must match the bundle patch installed. At the time of publication there are two overlay patches available. If an overlay patch for the bundle patch you will install is not listed above, either contact Oracle Support to check on availability of an overlay patch for your bundle patch, or utilize the workaround described in Document This is described in more detail later in this document. Patch Overlay fix for unpublished bug on top of BP5 is used within this document 44

45 OS Upgrade DB Server For instructions for updating key software components on database servers from OL 5.3 to OL 5.5, refer to Document Steps: Step 1: Obtain RPM bundle, stage needed RPMs, apply workaround(s) Step 2: Install updated kernel Step 3: Shutdown Services Step 4: Update additional RPMs Step 5: Update additional packages Step 6: Clean up, restart processes, healthcheck 45

46 AGENDA 1. Patching Overview 2. Patch Application Methods 3. Patch Administrator/Manager? 4. Patching Exadata V2 to X Conclusion 46

47 Conclusion 47

48 Conclusion Even Exadata as an engineered solution needs software maintenance Patching should be done regularly to proactive avoid troubles Integrate it into the Lifecycle-Management with a well defined patch process Patching involves near countless steps, define and use checklists Regularly check your Systems state with the provided Healthcheck Scripts Evaluate the latest patch feature, as there is a fast improvement cycle Having redundancy in terms of a MAA system reduces patch risks and time 48

49 Conclusion (2) Do not forget Mr. Murphy: Software is buggy, but hardware breaks This Patch/Upgrade Scenario was definitely one of the most challenging ones, your case can only be easier ;-) Therefore no matter who does the job: The DBA gets I all! (If he wants it) 49

50 THANK YOU. Trivadis AG VISIT US AT THE TRIVADIS-STAND: Floor 3, No. 304 Konrad Häfeli Papiermühlestrasse 73 CH-3014 Bern Tel Fax BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN 50