SharePoint Patching: Notes From the Field Sam Hassani, Premier Field Engineer, Microsoft UK
Agenda The story of SharePoint patching Answering the what do I need question Dismantle a released update Update scenarios Planning and upgrading Monitoring and troubleshooting
Terminology
Type of Updates Released Hotfix COD Critical on Demand Public Hotfix e.g. US DST Hotfix 941422 Post Service Pack rollup Rollup of hotfixes build during SP-lock timeframe when service pack is locked to not accept new fixes Cumulative Update Since December * rolling 2 month basis Broadly affecting support issues Service Pack Service Packs include all of the updates for MOSS plus deliver important customer-requested stability and performance improvements, while incorporating further enhancements to user security.
Build Numbers 12 12 00 6219 6219 Major Major Minor Minor Build Build Major = Office 12 : SharePoint 2007 Minor = RTM Build = Internal build number 4518 = RTM 6219 = SP1 6320(WSS)/6322(MOSS) = Infrastructure Update 6341 = February Cumulative Update Revision = Update type 1000 1000 Revisio Revisio nn 1000 = Service Pack 30xx = Private build 5000 = Cumulative Update 500x = COD Build
Check Your Build How? WSS Check Version OWSSVR.DLL WSS Localised Check Installed KB Numbers Content DB Versions Table MOSS Check Version Microsoft.SharePoint.Portal.DLL MOSS Localised Check Installed KB Numbers Content DB Versions Table DLC Check Version Microsoft.Office.Policy.DLL OSRV Check Version of Microsoft.Office.Server.DLL
Check Your Build How? DEMO Check Your Build
Patching Post IU http://support.microsoft.com/kb/953749/ Improvements to the time that is required to update and upgrade Windows SharePoint Services sites PSConfig no longer updates Site Collection Version number unless a Schema update is required Significant Performance Improvement February Cumulative Updates for WSS and MOSS http://support.microsoft.com/kb/961756 http://support.microsoft.com/kb/961755 Latest service pack + latest WSS Cumulative Update +latest MOSS Cumulative Update = completely up-todate.
How it used to be... How it was - MOSS CoreSer ver GLB Build CoreSer ver Loc 6324.500 0 6327.500 0 6331.500 0 Jul y Au g Oc t 955593 Build STSSTSGLB LOC Jul 95559 y 4 Au 957109 g Oc 95769 t 1 DLCLoc pjsrv wfeglb pjsrva pp-glb 956056 957693 Ifswf e -glb Xlsrva pp -glb 955586 95558 6 95339 7 958567 How it was - WSS 6324.50 00 6327.50 00 6332.50 00 DLCGLB 958569 95769 6 95769 4
The December CU and Beyond... How it is- MOSS Build CoreSer CoreSer ver -glb ver -loc 6324.50 Jul 955593 00 y 6327.50 Au 956056 00 g 6331.50 Oct 957693 958567 00 6335.50 De 960011 960011 How it is-cwss 00 6341.50 Fe STS961755 STS961755 Build 02 b glb loc 6324.50 July 955594 00 6327.50 Au 957109 00 g 6332.50 Oct 957691 00 6335.50 De 960010 960010 00 c 6341.50 Feb 961756 961756 DLCGLB DLC -loc pjsrv wfeglb pjsrva Ifswf pp-glb e -glb Xlsrva pp -glb 955586 95558 6 95339 7 958569 960011 96001 1 961755 96175 5 95769 6 96001 1 96175 5 95769 4 96001 1 96175 5 9600 11 9617 55 96001 1 96175 5
What s In the MSP? Orca Part of the Window s development tools portion of the SDK http://msdn2.microsoft.com/en-us/libr ary/aa370557.aspx Open MSI file View MSP as transform
What s In the MSP? DEMO Dismantling a Patch
Typical Scenarios New Farm Updating to......minimum Build...Recommended Build...Latest and Greatest Build Existing Server in Existing Farm New Server in Existing Farm
Updating To... Minimum required build: The oldest supported build Currently this is SP1 for WSS 3.0/MOSS 2007 Once a Service Pack is released, the n-1 version is supported for 1 year after Service Pack release. After this time, patches for n-1 version will not be shipped Microsoft recommended build: Product groups pubic recommendations: http://blogs.msdn.com/sharepoint The current recommended build is the Infrastructure Update Latest and greatest build: Running with maximum number of fixes
Culture Shift Which build should I be on? Minimum required Microsoft recommended Latest and greatest "If it ain't broke don't fix it attitude is becoming less prevalent (based on Case Histories) Software vendors are more diligent leading to increased consumer trust
Version Distribution SP1 + IU Improved Support from PSS RTM SP1 + Dec CU
New Farm What is the best way to build it? Slipstream it Any package can be slipstreamed - extract the MSPs into the update folder Benefits of fixes in the provisioning and configuration wizard code paths. When you configure CA it is configured with the fixes and updates already in place. Long term is no different to manual updates but is more efficient
New Server in Existing Farm Adding a new server to an existing farm: Replacement Scale-out process New server must be same version as existing servers How do we achieve this? Slipstream installation Manually install patches Do the above before running configuration wizard to join the farm
Existing Server in Existing Farm Patching an existing server in an existing farm: New Microsoft recommendations To correct an error that has been experienced (per a KB article) To be running the most recent code How do we achieve this? Manually install patches Do the above before running configuration wizard
Slipstream Creation Requirement for deploying on Windows Server 2008 Deployment guides Windows SharePoint Services 3.0 http://technet2.microsoft.com/windowsserver /WSS/en/library/91649a7e-6b5a-4e5a-9ee5-519 51f4b857f1033.mspx Office SharePoint Server 2007 http://technet2.microsoft.com/office/en-us/li brary/f484f5f2-35bb-4d70-bf56-dd1c4c287c72103 3.mspx
Slipstream Creation
Pre Upgrade Steps Take a complete farm backup and verify it restores It is not supported to restore an earlier version onto a newer version e.g. RTM to SP 1 Backup any SharePoint out of box files you may have customized Run orphan repair tool, http://support.microsoft.com/kb/923904 Defragment databases, http://support.microsoft.com/kb/943345 Ensure adequate disk space exists on each server Installation of the new binaries, and upgrade logging will take disk space
Pre Upgrade Steps Load balance sites across databases if needed Whitepaper released at SP 1, Performance Recommendations for Storage Planning and Monitoring, http://go.microsoft.com/fwlink/?linkid=105623&clcid=0 x409 Verify no upgrade Timer Jobs are currently running In place upgrade needs to complete before installing an update Gradual upgrade needs to complete current site upgrade before installing an update Take a full farm backup immediately after you know the update has applied successfully
Deployment Overview All updates follow the same steps and procedures Make sure the account you will be installing the update has adequate permissions Mechanics Install the update on the SharePoint server(s) Installs the updated binaries Run SharePoint Configuration Wizard or psconfig.exe Updates the web applications, databases, and versions The database upgrade is the longest running piece Until the database upgrade is complete access to content is blocked to prevent corruption
Deployment Options In Place Single server Basic installation Automatic updates In place installation for a farm Install the update on each server until the following dialog appears Click OK on the dialog on the first server Wait until it has completed Continue to each of the remaining servers
Deployment options DB attach Document all databases assigned to each web application Remove all of the databases from the web applications Install the update on each server following the in place method Re-attach the content databases to respective web applications Note: Attaching the content database has been reported to be faster and give more flexibility and control than normal in place upgrading of a farm
Minimising Downtime High performance db attach MSIT Scenario: 4 Terabytes of data to upgrade 80 content databases at 50GB per database Upgrade Procedure: 1. 2. 3. 4. 5. 6. Create the 4 farms (Identical target patch version) Backup databases Attach databases to upgrade farms (in parallel) Verify upgrade has completed successfully Detach databases from upgrade farms (in parallel) Attach upgraded databases to production farm Complete with quadrupled throughput!
Minimising Downtime Duplicate Read-Only Farm Internet Scenario: 24x7 Internet Presence Required Upgrade Procedure: 1. 2. 3. 4. 5. 6. 7. Create 2 farms (Identical original patch version) Backup databases Attach databases to target farm Change routing / DNS Verify access to Read Only farm Upgrade production farm Switch back to Upgraded farm Complete with continued uptime!
Sequence of Installation WSSv3 Updates must be applied before the MOSS 2007 Updates FAQ Can I install WSS and MOSS updates and run the configuration wizard once? FAQ Can I uninstall any update if there are problems?
Upgrade Order Web01 = COMPLETE + Central Admin Web01 = WFE + Central Admin Upgrade this server first Upgrade index01 first if it is also a WFE since it has the most roles/services If index01 does not have WFE enabled upgrade web01 first Multiple Central Admin Upgrade 1st Central Admin before 2nd Multiple Query/Index If Search/Index contains Central Admin upgrade it first Otherwise, whichever server has the most roles/services is before others
Verify Update was Successful The Upgrade Log file Finished upgrading SPFarm Name=<Configuration Database> In-place upgrade session finishes. Root object = SPFarm=<Configuration Database>, recursive = True. 0 errors and 0 warnings encountered. If you cannot find the entries look for Exception or Error, correct the problem and run the configuration wizard or psconfig.exe again Check build version
Monitoring and Troubleshooting Most support issues reported on deploying updates relates to failures running the Configuration wizard SharePoint Products and Technologies Configuration wizard psconfig.exe
Monitoring and Troubleshooting Log files PSCDiagnostics log files, e.g. PSCDiagnostics_1_21_2008_14_27_10_84_4533747 83.log Upgrade log files, e.g. Upgrade.log Event Logs PSCDiag and Upgrade log files are stored in: c:\program Files\Common Files\Microsoft Shared\web server extensions\12\logs
Monitoring and Troubleshooting?? psconfig -cmd upgrade -inplace b2b -wait -force
Monitoring and Troubleshooting Upgrade Log Refreshing the upgrade log Sequences and Actions http://support.microsoft.com/kb/948780/en-us FAQ What is SyncUpgradeTimerJob: sleeping for 10 seconds
Monitoring and Troubleshooting Sequences can repeat in certain scenarios Multiple content databases Multiple web applications If the SPTimerv3 account can not write to the default Upgrade.log, it may write to \Documents and Settings\SPTimerv3 account\ Local Settings\Temp\Upgrade.log The Upgrade.log file is cumulative
Common Failures Issue: Server Error when accessing SharePoint site Cause: WFEs are upgraded but Databases are not Resolution: Detach/reattach version mismatched database(s).
Common Failures Issue: PSConfig fails with inadequate install permissions Cause: Not logged in with adequate permissions to install the update Overall #2 SP1 deployment failure Resolution: Ensure necessary permissions when deploying any update Administrators on the WFEs Administrators on the SQL Server or granted the database role db_owner on all SharePoint databases
Common Failures Issue: PSConfig fails during site collection upgrade sequence Cause: [NewWebParts] [12.1.6301.0] [DEBUG] [1/10/2009 8:15:40 AM]: Begin Upgrade()...... [SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Unknown server error number: d Resolution:. Disable ForeFront Security for SharePoint Run PSconfig, or detach/reattach content databases
Common Failures Issue: PSConfig fails at Step 8 Cause: [SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Upgrade [SPSearchDatabase Name=OSST_SEARCH Parent=SPDatabaseServiceInstance Name=sql08,1433] failed....... [SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Object reference not set to an instance of an object. Resolution : Corrupt dll in Dec CU [Build 3334]
Final Thoughts... Understand what you have... Understand where you want to be... While we cannot upgrade a live farm we do have processes that will move us closer to the nirvana of a zero downtime upgrade