<Insert Picture Here> Oracle Solaris Patch Update

Transcription

1 <Insert Picture Here> Oracle Solaris Patch Update Gerry Haskins Director, Software Patch Services Revenue Product Engineering, Systems 30 th August 2010

2 Contents Recommended Patching Strategy Patching Strategy Considerations

3 Contents Background Information Solaris Fundamentals Patches & Solaris Updates Interim Diagnostics & Relief (IDRs) Dynamic Tension SplitGate Process Improvements & Benefits Solaris 10 Kernel PatchID Sequence Deferred Activation Patching Problematic Solaris 10 patches Live Upgrade Bug Fix Process Overview Patch Quality Metrics

4 Contents News Zones Patching Performance Improvements Other Patch Utility Improvements My Oracle Support (MOS) Patch Entitlement Solaris 8 Vintage Patch Service PatchFinder Other Recent Patching Enhancements Patch Education Resources Patching Tips, Tools, & Services The Next Generation: Image Packaging System How Can I Further Improve Your Patching Experience?

5 Recommended Patching Strategy

6 Recommended Patching Strategy What to apply? Apply the latest available Solaris 10 Update Release during the next major maintenance window. Each Solaris Update contains all available patches preapplied to the release image. Solaris Updates provide intensely tested, high quality, functionally rich, and performant software baselines on which to standardize deployments.

7 Recommended Patching Strategy For customers whose change control procedures make it difficult to upgrade, a Solaris Update Patch Bundle is available for each Solaris Update starting with Solaris 10 5/08 (Update 5) which will patch pre-existing packages to the same software level as the corresponding Solaris Update. For example, all Zones and ZFS functionality are entirely contained within patches, so applying the latest patches gives you all the latest functionality for these features. The only difference is that the Solaris Update Patch Bundle doesn't include new packages included in the Solaris Update, which may be required for some new features.

8 Recommended Patching Strategy Keep as up to date as possible with the Solaris OS Recommended Patch Cluster to get the latest Solaris Security, Data Corruption, and System Availability fixes. The Recommended Patch Cluster provides the minimum amount of change required to get all the latest Solaris OS Security, Data Corruption, and System Availability fixes. The Enterprise Installation Standards (EIS) patch baseline, which is available in Oracle Enterprise Manager Ops Center, is a superset of the Recommended Patch Cluster and hence is a reasonable alternative.

9 Recommended Patching Strategy Keep as up-to-date as possible with firmware updates. Firmware has become increasingly important for SPARC and well as x86. Firmware updates resolve many key issues and may deliver significant performance improvements. Apply any additional environment specific patches required.

10 Recommended Patching Strategy How to apply? Always install the latest patch and package utility patches first to ensure correct patch application. This is done automatically when applying the Solaris OS Recommended Patch Cluster.

11 Recommended Patching Strategy Apply patches and upgrades to an Alternate Boot Environment (ABE) to minimize risk and downtime. This avoids the need to follow most of the manual Special Install Instructions contained in patch READMEs which are typically only needed if patching a live boot environment. Using an ABE provides a simple rollback mechanism in case of issues with the changes made simply boot back into the original boot environment to resume production if issues arise. Use Oracle Solaris Live Upgrade (LU) for most environments, including Zones environments. In mirrored environments, split the mirror and use Live Upgrade to update one side. In Oracle Solaris Cluster environments, a rolling update of the cluster nodes may be preferred.

12 Recommended Patching Strategy How to further mitigate risk? The best way for customers to minimize risk is to have a test environment which mimics the production environment as closely as possible and includes good functional test coverage and peak load testing. While Oracle tests all software prior to release, it's impossible to test all possible configuration permutations, including 3rd party and home grown applications. Therefore, testing in the specific customer environment prior to live deployment is recommended.

13 Patch Strategy Considerations

14 Patching Strategy Considerations Typical objective is to maximize production system availability by optimizing proactive maintenance to prevent issues. Change implies risk. But minimizing risk is not as simple as minimizing change. Need to consider best tested and best quality baselines upon which to standardize deployments to minimize risk: Solaris Updates are intensely tested by many teams across Oracle Sun and so provide a good baseline upon which to standardize deployments. Each Solaris Update includes all bug fixes which were available at the time it was built.

15 Patching Strategy Considerations The Solaris OS Recommended Patch Cluster provides all the latest available Solaris OS Security, Data Corruption, and System Availability fixes, including any released since the contents of the latest Solaris Update was finalized. The cluster is tested as a unit prior to release. It is archived quarterly as the Solaris OS Critical Patch Updates (CPU).

16 Patching Strategy Considerations In contrast, Dim Sum patching (e.g. picking a Kernel patch which is 6 months old, a libc patch which is 2 years old, and an LDAP patch which is a week old) may result in a software combination which has never been tested before as a set. However, rigorous processes to ensure patch dependencies are correctly defined, coupled with the patch test and verification process means that issues with Dim Sum patching are very rare. The more customers who standardize on the Recommended Patching Strategy, the more they will leverage the safety in numbers of having a large number of systems at a similar software level, as issues should be found, analyzed, and fixed faster than if they are on unique software levels.

17 Patching Strategy Considerations Why not apply all patches? Applying all patches is a perfectly reasonable strategy. However, Jim Jimmo Moore, Solaris Sustaining Director EMEA, states that almost all issues reported more than 18 months after a Solaris release has shipped are corner cases which only occur in highly specific configurations. One can argue that the quantity of change included in Solaris 10 Updates resets this 18 month clock to some extent. Although code changes in patches go through an intensive review, verification, and test process, there's still a finite risk of a fix introducing regressions in specific configurations.

18 Patching Strategy Considerations Since change implies risk, it's debatable whether applying all corner case fixes for other customers' configurations in between Solaris Update releases is the optimal system maintenance strategy to minimize risk and maximize system availability.

19 Patching Strategy Considerations What about timing of patch application? Patches are intensely tested, but issues specific to certain configurations can still occasionally slip through. Some customers like to wait until a patch has been released for a period of time e.g. 30 days before applying it unless it fixes a urgent security issue. Analysis of the time between patch release and the withdrawal of problematic patches shows little correlation to any specific sweet spot, although serious pervasive issues are usually found within 10 days of release. My Oracle Support (MOS) enables customers to see how often a patch has been downloaded and to rate patches which is useful in increasing confidence levels.

20 Patching Strategy Considerations Oracle Sun releases over 4,500 patches every year, for Solaris 8, 9, and 10, SPARC and x86, Middleware products, Developer products, Storage products, etc. Just 17 have been withdrawn after release in the last year due to serious issues. A patch is withdrawn if it does more harm than good to most customers. For corner case issues, information is typically added to Special Install Instructions section of the patch READMEs. For Security, Data Corruption, or System Availability issues, an Oracle Alert (security) or Sun Alert (data corruption, system availability) will also be issued. Patches which fix such issues will automatically be added to the Solaris OS Recommended Patch Cluster as soon as they are available.

21 Background Information: Solaris Fundamentals why things are the way they are Solaris Updates & Patches Key Solaris 10 patching issues resolved

22 Solaris Fundamentals There is a single customer visible code branch for each Solaris Named Release. That is, there is one set of patches for all Solaris 8 releases, another set of patches for all Solaris 9 releases, and another set of patches for all Solaris 10 releases. This means that the same Solaris 10 patches can be applied to all systems running Solaris 10, irrespective of which Solaris 10 Update release is installed on them. This simplifies System Administration and helps provide a more homogeneous OS environment.

23 Solaris Fundamentals All code changes, e.g. for a new Solaris 10 bug fix, are putback to the tip of the Solaris 10 source tree. The resultant patch can be applied to all preceding Solaris 10 releases. Note: Engineering sometimes refer to a patch according to the time it was produced (e.g. A Solaris 10 Update 9 patch which was produced in the Update 9 time frame). However such patches will apply to all preceding Solaris 10 releases too. All top-of-tree patches will also be pre-applied into future Solaris 10 Update release images.

24 Solaris Fundamentals Therefore each Solaris Update contains all bug fixes which were available when the Update was built. Thus, each Solaris Update is successively better quality. This is what enables Oracle to provide ultra long-term support for each release of Solaris at reasonable cost to Oracle.

25 Solaris Fundamentals Since there is effectively a single customer visible source code branch for each Solaris Named Release, any change to pre-existing packages will be released as patches. Thus, a patch may contain a mix of feature and bug fix code. There is a many-to-many relationship between patches and packages. But within a Named Release, there's only ever one active (i.e. non-obsoleted) patch for a specific executable object. A patch may require other patches if there are one-way code dependencies between them. Where code putbacks would cause mutual inter-dependencies between patches, one patch will accumulate the contents of, and explicitly obsolete, the other(s).

26 Solaris Updates & Patches New features are introduced in each Solaris 10 Update release. Some large new features may introduce new packages. But all features will make at least some change to pre-existing packages (e.g. hooks for the feature). These changes will be delivered in patches. Many features, such as Zones and ZFS functionality, are completely contained within patches. Hence any Solaris 10 system can utilize the latest Zones and ZFS functionality by installing the appropriate patches.

27 Solaris Updates & Patches Each Update release is built from the original Solaris 10 package versions, plus all available patches pre-applied to those packages, plus any new packages introduced. For core Solaris, if you need a new package which is not available in the release you are running, you need to install or upgrade to a release containing that package. Note: Do not apply a package from one Update release to a system running another release as any pre-applied patches in the package will confuse patch dependency resolution and is likely to lead to system corruption. This is due to the many-to-many relationship between most patches and packages.

28 Solaris Updates & Patches Live Upgrade packages are an exception to this rule. For example, to upgrade a Solaris 8 system to Solaris 10, you would install the Solaris 10 version of the Live Upgrade package on the Solaris 8 system and apply the Solaris 10 LU patch. The LU patch is constructed so that it only applies to the LU package and has no dependencies on OS version specific patches, hence the patch dependency resolution issue described above is avoided. For core Solaris, the package version string remains constant throughout the lifetime of a Solaris Named Release and is only incremented from one Named Release to another. This is what determines patch applicability. For non-core Solaris, some package versions may be up'rev'd in a Solaris Update, typically for code originating from 3 rd parties.

29 Solaris Updates & Patches So Solaris 10 patches may include new feature code as well as bug fixes. New features are almost always switched off by default in patches. NewBoot is a notable exception. The rule is to avoid surprises. Caveats will be detailed in the Special Install Instructions section of the patch README files. The scope of new features is likely to diminish in future Solaris 10 Updates which will reduce the risk of introducing functional regressions.

30 Solaris Updates & Patches Where multiple code changes intersect, they will typically be included in the same patch. This can result in large Kernel patches being released at the end of each Solaris 10 Update release. These Kernel patches may contain significant amounts of latent feature code. These patches are very intensely tested by Oracle Sun as part of the Solaris 10 Update QA processes, as well as being tested as individual patches.

31 Solaris Updates & Patches These large patches undergo a process called rejuvenation, whereby future code changes will be included in a series of new, smaller, more manageable patches for different areas of functionality, each of which has a requirement on the parent patch from which it was rejuvenated. The result is that there are stepping stones to key functional enhancement baselines contained in the large Kernel patches released after each Update release, with smaller patches containing bug fixes released in between Update releases.

32 Solaris Updates & Patches The Kernel patches included in Solaris 10 Update releases are large and complex. It is not recommended to attempt to back them out although we do test that this is possible. Note: If an issue occurs during patch application, attempting to back out the patch without proper Root Cause Analysis of the issue is likely to compound the problem (e.g. If the system has run out of space).

33 Solaris Updates & Patches Since the Kernel patches associated with Solaris 10 Update releases are rejuvenated, all subsequent Kernel and other child patches will require them. Therefore, there is little sense in backing them out as they will be required to get fixes and features in future Kernel patches.

34 Solaris Updates & Patches Customer visible Solaris 10 Kernel patches Larger changes Larger changes Update 4 Kernel Patch 12001[12]-14 Small targeted patches Small targeted patches 12711[12] [12]-11 Larger changes Update 6 Kernel Patch Update 5 Kernel Patch 12712[78] [12] [12] [89] [78]-09

35 Interim Diagnostics & Relief (IDRs) An Interim Diagnostic or Relief (IDR) may be provided to a customer to help diagnose an issue or to provide relief for an issue while a generic release quality patch is being prepared. IDRs are temporary code branches and hence, unlike patches, are tied to a specific software level from where the code branch was taken e.g. A Kernel patch revision. IDRs lock down the packages to which they apply to prevent them from being accidentally overwritten. Therefore, IDRs must be removed before any patches can be applied to those packages. Thus, for ease of administration, it is best to avoid deploying IDRs in production if possible.

36 Interim Diagnostics & Relief (IDRs) If a system is to be patched, a new IDR may be required if the patch and IDR code intersect and the issue isn't yet fixed in a patch. Customers need to request this. IDRs are only supported until a permanent fix is available in a patch, at which point the IDR should be removed and the patch applied in its place. IDRs are typically customer specific, although a variant, an Interim Security Relief (ISR), may be put on general release to provide rapid relief to a Public security vulnerability while a generic release quality patch is being prepared.

37 Dynamic tension There's dynamic tension between the desire to provide customers with cool new features such as Zones enhancements, ZFS, NewBoot, Secure By Default, Trusted extensions, iscsi, OPL and other hardware support in Solaris 10 Updates, and the desire to provide production customers with rock solid stability. Getting the balance correct is tricky. New features must not introduce regressions in preexisting functionality. The scope of features allowed into Solaris 10 Update releases is larger than what was allowed into Solaris 8 or 9 Updates. This caused problems up to and including Solaris 10 11/06 (Update 3)

38 Dynamic tension The resultant issues have been solved through innovative solutions such as the SplitGate source code gate management process, Live Upgrade, and Deferred Activation Patching. In particular, Zones introduced a significant amount of complexity to the patching and upgrade processes, including performance issues which are now largely addressed with Zones Parallel Patching, Zones Update On Attach, Turbo Packaging. From Solaris 10 8/07 (Update 4) onwards, and for earlier releases patched up to this level, the major patching issues have been resolved.

39 SplitGate Process Improvement The process to manage the internal source code gate for core Solaris ( ON ) was changed after the Solaris 10 11/06 (Update 3) release to the new SplitGate process to provide better separation of immature feature code from customer bug fixes. SplitGate replaced the older Feature Foldback process. SplitGate has made a very significant improvement to Solaris 10 patch quality, starting immediately after Kernel patch (SPARC) / (x86).

40 SplitGate Benefits Kernel patch quality improvement: Release quality Solaris 10 Kernel Patches using old Feature Foldback ON source gate management model: SPARC: 21 out of 66 = 32% X86: 12 out of 66 = 18% Release quality Solaris 10 Kernel Patches using SplitGate ON source gate management model (as of July 26, 2010): SPARC: 58 out of 67 = 87% X86: 62 out of 67 = 93%

41 Solaris 10 Kernel PatchID Sequence SPARC x xx Update xx Sustaining Update Sustaining Update Sustaining Update Sustaining SplitGate Model Update Sustaining Update Sustaining Post-U * Update * Update * Update * Feature Foldback Model *Not Released as standalone patch

42 Deferred Activation Patching Up to Solaris 10 8/07 (Update 4), there were serious problems patching a live Zones boot environment due to the potential for code newly applied in patches being invoked during the patching process which might be incompatible with processes running in memory. This was due to the increased footprint of the patch utilities in a Zones environment. The problem with patching a live boot environment is solved by Deferred Activation Patching, whereby loopback filesystem (lofs) mounts are used to overlay the old object on top of the patched object to keep the system in a fully consistent state during patching. Deferred Activation Patching functionality is provided in the patch utilities patch from revision (SPARC) / (x86) onwards.

43 Deferred Activation Patching The only patches which currently specify application in Deferred Activation Patching mode are the Kernel patches associated with Solaris 10 Update releases: Kernel patch 12001[12]-14 from Solaris 10 8/07 (Update 4) Kernel patch 12712[78]-11 from Solaris 10 5/08 (Update 5) Kernel patch 13713[78]-09 from Solaris 10 10/08 (Update 6) Kernel patch 13955[56]-08 from Solaris 10 5/09 (Update 7) Kernel patch 14144[45]-09 from Solaris 10 10/09 (Update 8). Kernel patches from future Solaris 10 Update releases are also likely to employ Deferred Activation Patching. Deferred Activation Patching (DAP) is implicitly invoked for any other patch requiring a DAP patch which is applied before rebooting the system.

44 Deferred Activation Patching When the system is rebooted, the loopback filesystem (lofs) mounts will disappear, exposing the patched objects. Prior to the reboot, a lot of lofs mounts will be visible. This is normal and expected behavior after application of a Deferred Activation Patch, prior to rebooting. The contents of patches applied in DAP mode will not be activated until the system is rebooted.

45 Problematic Solaris 10 patches Deferred Activation Patching is not retrospective, so for systems using a Solaris version older than Solaris 10 8/07 (Update 4), some complex patches need to be managed: Kernel patch (SPARC) / (x86) uses loopback filesystem (lofs) mounts within the scripts in the patch itself to ensure system consistency during application to a live boot environment, and was the forerunner of the final Deferred Activation Patching solution. A reboot is required after installing this patch before other patches can be applied. This is enforced by the patch utilities.

46 Problematic Solaris 10 patches The only other circumstance where a reboot is required before further patching can be performed using 'patchadd' on the live boot environment is for x86 systems running an early version of Solaris 10, as a potential inconsistency exists between Kernel patches below running in memory and libc changes delivered in patch and -03, and xx which obsoletes it. Code in the scripts of these patches ensures that a later Kernel patch, e.g , must be active before these patches can be applied. There may be additional reboot constraints for higher level patch automation tools due to their own footprint on the target system i.e. Interaction between the calls they make to newly patched objects and what is running in memory.

47 Problematic Solaris 10 patches Obsolete Zones patch (SPARC) / (x86) must be installed to fix CR before Kernel patch (SPARC) / (x86) can be applied. The above problematic patches are only an issue for early versions of Solaris 10 and do not apply to Solaris 10 8/07 (Update 4) and later.

48 Live Upgrade Infodoc , provides information on Live Upgrade (LU) patches. For systems running Solaris 10 8/07 (Update 4) or earlier, the Solaris 10 Live Upgrade Zones Starter Patch Bundle on SunSolve provides the prerequisite patches to start using LU in a Zones on UFS. These patches need to be applied to the live boot environment. Systems running Solaris 10 5/08 (Update 5) or later have working LU functionality for Zones on UFS built in. Use Case: Use LU to create an ABE, apply the latest LU patches from Infodoc to the ABE to get latest LU functionality (e.g. To support ZFS Root migration), boot into the ABE and use LU again to create an ABE (e.g. For ZFS Root migration).

49 Background Information: Overview of the Bug Fix Process

50 Bug Fix Process - Prioritization Ensure any serious issue affecting your environment has a Customer Escalation for you associated with it. Just because a CR (Change Request, a.k.a. bug report) exists for the issue, doesn't mean it'll be automatically fixed immediately. Asking Oracle Support to ensure a call record for your company is associated with the CR will help increase its priority. The more customers who Escalate a CR, the higher the priority it will typically be given to fix. Don't rely on the fact that someone else may have escalated it.

51 Bug Fix Process - Analysis Ensure information on issues is as complete and accurate as possible as this will speed up the process: Include all error messages, log files, core dumps, Explorer, etc. What precisely is the problem observed? When is it observed / not observed? Where is it observed / not observed? What changes were made recently? What is the configuration? Zones? IPv6? VxVM? SC? etc. All Oracle Sun Sustaining and most QA staff are trained in the Sun Global Resolve problem solving methodology which is based upon Kepner-Tregoe's Analytic Trouble-Shooting methodology. The better the input data provided by customers, the faster the analysis and resolution will be.

52 Bug Fix Process Known Issue? My Oracle Support, SunSolve and Google are good sources of info to check if it's a known issue and if there's a solution or workaround already available

53 Bug Fix Process - Fix Once sufficient information is received to enable successfully analysis of the issue, the bug fix process begins. The bug fix process is rigorous to ensure quality. An IDR (Interim Diagnostics or Relief) may be produced to provide relief to Escalating Customers or, if required, to help diagnose the problem. The design for the bug fix will be peer reviewed. The code for the bug fix will be peer reviewed. The bug fix will be unit and link tested by the Sustaining engineer. The bug fix will go through functional Pre-Integration Functional Regression Testing and Performance Regression Testing.

54 Bug Fix Process Code Test The bug fix will initially be integrated into the development branch of Solaris (e.g. Solaris 11 ). The bug fix will be tested by dozens of QA teams across Oracle Sun who test each bi-weekly build of the development branch. If no issues are found after 4 weeks (2 builds) soak time in the development branch, the bug fix may be approved to go back into a production release. The soak time in the development branch helps prevent buggy code getting into a production release.

55 Bug Fix Process Code Test The bug fix may now be scheduled for integration into the relevant production release(s) on which the issue was reported. For Solaris 10, the back-ported bug fix goes through a second round of Pre-Integration Testing. A patch is created containing the bug fix.

56 Bug Fix Process Patch Test The patch is submitted to the Patch Pipeline which manages the patch test, verification, and release processes. It is called a T-Patch or Test Patch. Over 230 audits are run to check the structural integrity of the patch. Each engineer who contributed code to the patch must test the patch and explicitly verify that their fix(es) work as intended. The T-Patch is given to Escalating Customers to verify it fixes their issue. The Patch System Test (PST) team perform automated Install/Backout testing and System Testing on all Solaris, SunCluster, and some other patches. This includes Load and Stress testing, Performance Regression testing, etc.

57 Bug Fix Process Patch Release For Solaris 10, all patches are included in builds of the next Solaris Update. For Solaris 10, each Solaris Update build is intensely tested by dozens of QA teams across Oracle Sun. This increases our confidence level in the patch contents. When the patch test and verification process is complete, the T-Patch designation is removed from the patch, and the patch is released on My Oracle Support (MOS) and SunSolve and made available for download by patch automation tools. All this takes time, but is designed to ensure the quality of patches produced for production releases.

58 Bug Fix Process - Timing If a Solaris 10 Update is about to ship, putbacks to Solaris 10 may be delayed until after the Solaris Update ships, as the Source Gates are closed to all putbacks except Release Stoppers at the end of each Solaris Update. This results in a hiatus on Solaris 10 patch production for up to 12 weeks. Sustaining engineers try to time putbacks to avoid this hiatus. An overview of patch testing is available on

59 Bug Fix Process - Overview Customer Verify IDR Verify T-Patch Support Customer downloads released Patch Publish patch on SunSolve Patch OK? y Fix Available? y Provide T-Patch n Sustaining Analyze Provide IDR Integrate fix in Nevada Fix OK? y Integrate fix in Solaris 10 Patch System Test Design Review Code Review Functional Verification Functional Verification Pre-integration Functional & Perf Testing System Test Pre-integration Functional & Perf Testing System Test Build Image Performance Test Build Image Performance Test Test H/W Test T-Patch H/W Test

60 Patch Quality Metrics Solaris 10 Update 3 problematic SplitGate Process Improvement Introduced Deferred Activation Patching Enhancement Introduced (Patch System Test) Solaris patch quality now better than ever Over 5 years of patch quality data

61 News

62 Zones Patching Performance Performance Improvements: Use Live Upgrade to patch or upgrade an inactive boot environment. Zones Parallel Patching enhancement to the patch utilities which improves non-global Zones patching performance by ca. 3x. Available in the patch utilities patch from revision 11925[45]-66. Zones Update on Attach (available in Kernel patch 13713[78]-09) is even more performant. Further enhancements available in later patches.

63 Zones Patching Performance Zones Update on Attach is also useful to fix non global zones which are out of sync with the global zone (e.g. due to running out of disk space during patch application). SVR4 Turbo Packaging enhancement significantly speeds up Install, Upgrade, Live Upgrade Alternate Boot Environment creation, Zones creation, and also speeds up the application of very large patches. The enhancement is delivered in the following patches (and later): SPARC: , , , X86: , , ,

64 Other Patch Utility Improvements Improved 'patchadd -M' design to reduce risk of nonglobal zones getting out of sync with respect to patch level e.g. If run out of disk space during patching. Live Upgrade support for SVM fixed. Limited Live Upgrade support for Zone roots on ZFS in Solaris 10 10/08 (Update 6). Enhanced support for an expanded number of configurations in post-u6 patches, Update 7, and beyond.

65 My Oracle Support (MOS) Over 30,000 Oracle Sun patches are now available in My Oracle Support (MOS). Customers with an Oracle support contract can search for and download Oracle Sun patches from MOS: Search by Product, PatchID, Sun CR#, or Type=patchset (i.e. for Patch Clusters/Bundles) is supported. Be aware of a nomenclature clash: Sun Obsolete = Oracle Superseded Sun Withdrawn = Oracle Obsolete Since the current SVR4-based package and patching architecture uses the term SUNW_OBSOLETE, we cannot easily transition to the Oracle definitions, hence some confusion may result.

66 Patch Entitlement Solaris Patch Entitlement: The Solaris business model changed several years ago from selling Solaris and providing patches at no cost, to making the Solaris releases available at no cost and charging for patches. This policy has not changed. The implementation has been tightened up. Customers need an appropriate support contract for each system on which they wish to apply patches or patch clusters, including test and development systems. Hardware warranties and hardware-only support contracts do not provide entitlement to Solaris patches. See &

67 Solaris 8 Vintage Patch Service Solaris 8 began End of Service Life Phase 2 on April 1, 2009: Only customers who sign up to the Solaris 8 Vintage Patch Service will be able to access Solaris 8 patches produced after April 1, including patches which provide security fixes. Recommendation is for customers to migrate to the latest Solaris 10 Update, currently Solaris 10 10/09 (Update 8) As a migration aid, customers may wish to sign up for Solaris 8 (and/or 9) Containers, which enables a Solaris 8 (and/or 9) environment to be run in a Zone on a Solaris 10 system. See &

68 PatchFinder Sophisticated PatchFinder tool for Oracle Sun patches, For example: Identify patch revisions which introduce new security fixes. Search for patches for specific objects. Search for all Solaris OS Recommended patches released since your last patching maintenance cycle. Derive patch statistics e.g. number of Oracle Sun patches released in last year, number of withdrawn patches, etc. Add patches from multiple searches to a patch basket. Automatically and intelligently resolve patch dependencies for all patches in the patch basket and download. View patch access entitlements i.e. which support contracts are needed to access specific patches. See

69 Other Recent Patching Improvements Merged Solaris Recommended and Sun Alert patch clusters, see Solaris OS Recommended Patch Clusters now archived quarterly as Solaris Critical Patch Updates (CPUs), see Solaris Update Patch Bundles now update /etc/release to make it easier to identify that a system has been patched to the software level of a Solaris Update release. Patching Pre-flight Check tool to ensure system is ready to be patched e.g. sufficient space, no leftover lock files, etc. See

70 Patch Education Resources Recognized need to better communicate patching best practice and issues to customers and the field. Initiated: Big Admin Patching Hub, Patch Corner blog, Tidy-up and overhaul of other customer facing patch related policies and documentation Customer Patch Forums: New: Oracle Solaris Installation, Booting, & Patching Forum: Strategy, Best Practices, etc.: Issues:

71 Patch Education Resources Overhauled SunSolve Patches & Updates page, Patching Best Practices Videos / Course, and reference guide This is a comprehensive how to patching guide written by patch experts.

72 Patching Tips, Tools, & Services The Solaris OS Recommended Patch Cluster provides all Solaris OS patches which fix Security, Data Corruption, and System Availability issues: Sign up for Sun Alert notifications on

73 Patching Tips, Tools, & Services Applying rebootimmediate and reconfigimmediate patches to the live boot environment using 'patchadd' should be interpreted as requiring a reboot before normal operations are resumed. It's usually OK to continue to apply further patches before initiating the reboot. On the rare occasion where this is not the case, such as / , the patch will contain code to prevent the user continuing without rebooting. Higher level patch automation tools may have additional constraints due to their own footprint on the target system. Don't run other applications until the reboot is performed as the system is in a potentially inconsistent state. See Patching an Alternate Boot Environment avoids such issues.

74 Patching Tips, Tools, & Services Zones Update on Attach is very useful to sync up non-global Zones which are out of sync with the global zone regarding their patch level (e.g. because the non-global zone ran out of disk space during patching). Zones Update on Attach is also useful as a method to improve Zones patching performance (with some constraints). Detach non-global Zones, apply Solaris patches including Kernel patch or higher to the global zone, reboot to activate the new Kernel, reattach the non-global Zones with Update on Attach, and voilà, the non-global Zones will be brought up to the same patch level at the global Zone. Further enhancements coming in latest patches. Zones Update on Attach will only update software, not down-rev it. Any 'patchrm' operation must be performed while the zones are attached, e.g. if 'patchrm' is being used to remove a withdrawn patch.

75 Patching Tips, Tools, & Services Patch Check Advanced (pca) is a simple and highly popular 3 rd party patch automation tool, developed and supported by Martin Paul. 'pca' is available from Oracle Enterprise Manager Ops Center is a comprehensive virtualization and provisioning management tool which has a patch management component, based on the Aduva product which Sun acquired a couple of years ago. It supersedes UCE. Ops Center support for Live Upgrade is now available.

76 Patching Tips, Tools, & Services 'smpatch' and Update Manager are older Sun patch management tools based on Sun's PatchPro technology. A major enhancement to the back end of these tools was rolled out in 2009 to significantly increase their reliability and robustness.

77 Patching Tips, Tools, & Services Oracle Services provide patching services to customers, including tailoring a patching strategy to customer needs and providing customers with lists of key patches to install on their specific systems. Many of these services for Sun products are based on the TLP (Traffic Light Patching) tool, SRAS (Sun Risk Analysis Services) and the EIS (Enterprise Installation Standards). EIS is the same methodology used for factory pre-installs of Oracle Sun hardware and installations by Oracle field personnel. The EIS patch set is based on the Solaris OS Recommended Patch Cluster with additional patches added for products such as SunCluster, SunVTS, SSP, SMS, QFS, SAM-FS, and firmware updates.

78 Patching Tips, Tools, & Services The monthly EIS Patch Baselines are now available through Ops Centre and UCE. New proactive patching services are currently in development.

79 Image Packaging System Solaris 10 and earlier employs a 2-tier SVR4-based package and patching model. This increases complexity. The strategic solution for is to move to a single tier architecture, Image Packaging System (IPS), in OpenSolaris and Nevada (Solaris 11 ). See Dr. Stephen Hahn's blog, To get bug fixes, users will update packages. Different streams of packages will be available, from bug fix only to bleeding edge development. Bart Smaalders and David Comay who are working on the project are very familiar with the issues with the current patch and package architecture. They are designing solutions in IPS to address the root causes.

80 Image Packaging System The primary target audience for OpenSolaris currently is developers and the current Image Packaging System functionality is concentrating on their needs. As the target audience for OpenSolaris / Nevada matures towards ISVs and production customers, one can expect to see the Image Packaging System functionality mature into this space.

81 How Can I Further Improve Your Patching Experience? Solaris Patch Utilities? Patch Automation Tools? SunSolve? Patch Services? Patch Best Practices / Education? Patch Quality? Faster Time to Release Patches? Other? Please let me know - Gerry.Haskins@oracle.com

82 <Insert Picture Here>