IBM AIX Operating System Service Strategy Details and Best Practices

IBM AIX Operating System Service Strategy Details and Best Practices Julie Craft AIX Development jcraft@us.ibm.com 06/10/2009 1

Contents Contents... 2 Introduction... 3 Enhanced Service Strategy Releases... 3 Contents of a Service Pack... 3 Changing the names of the service packs... 4 Changing the M in VRMF... 5 Technology Levels Must be Applied as a Group... 5 Recommendation to apply the whole SP instead of just individual PTFs... 6 Service Pack schedules... 6 Technology Level Lifecycle and Support... 6 Interim Fix Guidelines/Policy for Maintenance... 8 Certified Interim Fixes... 8 Security Fixes... 9 APAR Numbers for each TL... 9 New Automatic Interim Fix Removal Function... 9 Hardware Support with a Service Pack... 11 Moving to a New TL... 11 Staying Current... 12 Maximum Stability Model... 12 Additional Information... 12 Summary... 12 06/10/2009 2

Introduction IBM has enhanced the AIX operating system (OS) Release and Service Strategy as part of an ongoing effort to improve the manageability and stability of the AIX operating system for our clients. The enhanced strategy will provide clients with: Longer support for each AIX Technology Level update (formerly known as Recommended Maintenance Levels) Improved serviceability for AIX throughout the life of each Technology Level Support for some new hardware on existing Technology Levels The previous paragraph came from the IBM AIX Operating System Release Service Strategy paper, which should be read before reading this paper. This paper will go deeper into some of the changes, give some best practice scenarios and discuss some of the more advanced function available. Enhanced Service Strategy Releases The AIX enhanced service strategy will start with AIX 5.3 TL6 and continue with AIX 6.1. AIX 5.2 will not use the new service strategy; TL10 is the last technology level available on 5.2 and we will continue to provide service packs until the end of support. Contents of a Service Pack Service packs contain fixes for: Customer reported problems (APARs) that cannot wait until the next TL Critical problems found by development or test teams Very, very limited number of changes to support new hardware. Examples: A new device driver, a new ODM entry to allow for configuration of a new class or type of device, small changes in the kernel to recognize a new processor speed, etc. The only changes that are allowed in an SP are limited to minimal corrections that do not change behavior or add new functionality. The development team employs a fix rating system to enforce this. New function, both for hardware exploitation and software features, is only shipped in Technology Levels or new releases. While AIX TL s contain RSCT and CSM fileset updates, those filesets are not updated in the AIX Service Packs. The reason for this is scheduling. The RSCT and CSM teams line up their schedules to meet the AIX TL schedules, but do not do the same for Service Packs. The fixes are placed into the next TL. Occasionally, usually with applications or products that rely more heavily on CSM or RSCT, you may be required to download a fix directly from the RSCT or CSM download sites. This situation may be more suited for your environment because it does not require application of a specific AIX SP, only the individual fix for CSM or RSCT. We are continuing to explore possibilities here to simplify the maintenance of AIX and its components. 06/10/2009 3

Changing the names of the service packs Starting with TL6, Service Packs will be labeled with their release date, using the YYWW format, where: YY = 2 digit year WW = 2 digit week An example: AIX 5.3 TL6 SP3 was released in 2007 and the 32nd week of the year, therefore: 5300-06-03-0732 ' ' ' '---------------------YYWW ' ' '-----------------SP3 ' '------------TL level '--AIX release The oslevel s command was updated to report this new name. The nice thing about this is not only can you tell when the SP was released, but you will also know which SP to move to on a new TL. When moving up to a new TL, you must move to a SP that is the same or later than your current SP. The SP number itself (IE SP3) will not be the same, because the Service Packs will be numbered consecutively as they are released, but the dates will tell you where you need to be on the new TL. In general, it is recommended that you apply the latest Service Pack when moving to a new TL. New Release Strategy Service Pack Name Detail (AIX V5.3 shown) 2007 2008 2009 2010 Fall Spring Fall Spring Fall Spring Technology 0812 SP HP 0830 SP 0845 SP SP4 SP6 SP8 0903 SP SP9 Level 7 0830 0845 SP SP SP2 SP4 0903 SP SP5 Technology Level 8 Service pack names will be changed to reflect the release dates: VR00-TL-SP-YYWW 5300-07-04-0812 We will line SP s up so the upgrade path will be clear Figure 1: Service Strategy Service Pack Name Detail *All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. * Not showing all SP s above and example dates are not necessarily accurate 06/10/2009 4

Applying individual PTF updates after a Service Pack will not be shown with the oslevel s command. Instead, use oslevel s g 5300-06-03-0732 to show what filesets are greater than the specified service pack. The installp command was changed, as were the updates themselves, to not allow a system to apply any updates that are younger than what is currently installed. This would cause regression. When moving to a new TL, we recommend moving to the latest SP, that way you are guaranteed it will pass the installp regression test. Using the example above, if you are currently at TL7 SP8 (5300-07-08-0845), then you would not be allowed to install TL8 SP2 (5300-08-02-0830), because 0845 was released after 0830, and therefore will have fixes and hardware enablement that 0830 does not have. The oslevel s command will now print out the new SP format, but the other options will not change. # oslevel 5300 # oslevel r 5300-06 # oslevel s 5300-06-03-0736 The installp command will also stop an update entirely if it sees any updates that are older (released before) in the list it is trying to apply. This is to make sure that part of a SP is not installed, if that was not your intention. If you see a message from installp about not being able to install a fileset update because of regression, then go to a newer SP (the latest) and try the install again. Changing the M in VRMF Starting with AIX 5.3 TL7, any filesets that are updated will get a new M in the VRMF. For example, if the bos.rte.lvm fileset was updated, its update in TL7 would be 5.3.7.0 and the first update, in SP1, will be 5.3.7.1. The fileset updates won t necessarily correspond to the SP because an update for fileset bos.rte.install might not come out until SP3, but it would be called bos.rte.install 5.3.7.1. Likewise, if bos.rte.lvm had not changed since SP1, it would still be bos.rte.lvm 5.3.7.1 in SP3. Technology Levels Must be Applied as a Group Technology Levels must be applied as a group, using the smitty update_all or install_all_updates commands. Installing a Technology Level is an all or nothing operation. Initially, the plan was to add requisites to glue the TL together, but this was not done because of the complications of circular requisites. But, installing a partial Technology Level will not be recognized from a support standpoint. Before applying a TL, you should always create a backup and plan on restoring that backup if you need to rollback to your previous level. Or, use alt_disk_install or multibos as a way to get back to your previous level. Technology Level updates should always be committed because they cannot be rejected. Committing the updates saves space in the / and /usr file systems and also makes it easier to track and reject service pack PTFs. 06/10/2009 5

After the TL has been successfully applied and tested, another backup should be taken for disaster recovery situations. Recommendation to apply the whole SP instead of just individual PTFs For planned maintenance windows, customers are encouraged to apply Service Packs as a group, to simplify inventory and make it easier to report levels to service representatives and auditors. Fix Central and SUMA will now download the entire SP for a specified APAR (search function). This was also done to simplify the ordering process. For more information on Fix Central changes, see the paper: Fix Central changes supporting AIX Service Strategy at http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html. Even though the entire SP is downloaded, individual fileset updates or APARs can still be applied with SMIT ( smitty install_by_fix ) or from the command line (installp or instfix). There is nothing in the Service Pack that glues all the updates in a Service Pack together. While most of the testing that occurs in the regression test lab is done as a group, applying individual updates is still fully supported. For unplanned application of fixes (like security or HIPERs), changing the minimum amount of code is certainly the desired outcome. For example: An interim fix was just released for a security issue and it requires an update from a service pack. The service pack would require a reboot of the system whereas applying just the individual PTF would not. In these cases, just applying the individual APARs/PTFs may be the best solution. Service Packs, because they will always require a reboot, are recommended for planned maintenance windows or if the application of the fix will require a reboot anyway. Service Pack schedules Because there will be more Technology Levels shipping new fixes and because the Service Packs will be lined up to come out at the same time (with the same YYWW), the SP release schedules will be increased to approximately every 12 weeks. Occasionally, if a very critical problem is found (high severity security, for example) and it cannot wait for the next SP, a standalone PTF will be released and will be made available via APAR search on Fix Central. The PTF will be included in the next Service Pack. Technology Level Lifecycle and Support With a valid Software Maintenance Agreement (SWMA), customers can continue to open PMRs to ask questions and report problems with AIX 5.2, AIX 5.3 and AIX 6.1. However, if a code defect is identified (i.e. an APAR is created) then the release and TL must be in the New Fixes Available period to request a fix. This applies to both interim fixes and Service Pack updates. More information about the System p Product Lifecycle dates can be found at: http://www.ibm.com/software/support/systemsp/lifecycle 06/10/2009 6

The following chart illustrates the entire lifecycle of a technology level, showing both the times where new fixes are delivered and when it is still supported via PMR for how to and usage questions and problem determination. Technology Level Lifecycle New Fixes Available Period Support Period How to Q&A, problem determination New Fixes created (Interim Fixes and Service Packs) Availability of the TL End of New Fixes Available Period End of support for the AIX release Figure 2: Technology Level Lifecycle The chart below is a listing of the technology levels that have been released in the last two years. The chart highlights the differences in the timeframes of when fixes were available. AIX 5.2 TL8 and TL9 fell under the old service strategy and had a shorter time period of fixes, about one year from release. The new strategy allows new fixes to become available for approximately two years after release. Use this chart for comparison and planning. Release Release Date Fixes Available Until AIX 5.2 TL8 Feb-06 Feb-07 AIX 5.2 TL9 Jul-06 Nov-07 AIX 5.2 TL10 Jun-07 4/30/2009 ** AIX 5.3 TL5 Aug-06 11/30/2007 AIX 5.3 TL6 Jun-07 5/30/2009 AIX 5.3 TL7 Nov-07 2 years later AIX 5.3 TL8 April-08 2 years later AIX 5.3 TL9 Nov-08 2 years later AIX 5.3 TL10 May-09 2 years later AIX 6.1 TL0 Nov-07 2 years later AIX 6.1 TL1 April-08 2 years later AIX 6.1 TL2 Nov-08 2 years later AIX 6.1 TL3 May-09 2 years later 06/10/2009 7

Figure 3: New Fixes Available NOTE: All future dates are subject to change. * New Fixes will be made available until two years later or when the new TL is offered. In some cases this may mean slightly less than two years of support, in some cases it may mean slightly more. AIX will only make new fixes available for up to four Technology Levels for each release at any given time. ** For 5.2, the date for Fixes Available Until and End of Support are the same date. When selecting a TL for upgrade, keep the New Fixes Available period in mind. The recommendation is to move to the latest TL and the latest SP when performing an upgrade, to get the longest period of support. Interim Fix Guidelines/Policy for Maintenance Interim fixes are fixes made available to our customers to offer relief for a problem until the customer is able to update to a Service Pack or Technology level that contains the fix. The general rule is that a customer can request an interim fix for Technology Levels that are in the New Fixes Available period; it will then be evaluated by development. Development will make every attempt to supply an interim fix, however, there may be some that cannot be supplied due to complexity. Here are the guidelines that the Service team will be following: If a client is at the latest level and finds a problem they can request an Interim Fix If a client finds a problem that is already in a later SP - they will be asked to install the latest SP (or, at a minimum, the Service Pack that includes their fix or individual PTF that contains the fix) If it is a critical problem and/or the client can't install a SP - an interim fix can be requested, however it is recommend they move to the latest SP during their next maintenance window If a client finds a new, unique problem at any level that is making new fixes available - they can get an Interim Fix on their current level (if it's possible) If a client finds a new unique problem on a prior SP and has multiple Interim Fixes installed, which are included in a later SP - they must move to the latest SP before getting another Interim Fix If a client has multiple Interim Fixes and only some of them are in a later SP - they may be asked to move to the latest SP and another Interim Fix will be bundled with their prior Interim Fixes on the latest SP Clients should update to the SP that contains the fix as soon as they are able Certified Interim Fixes Interim fixes are offered to customers as a faster way to get temporary relief for a problem. Interim fixes are shipped in emgr format instead of installp format. Interim fixes are also tracked via the lslpp L and emgr l commands. Some interim fixes get additional functional and regression testing before they are released. These interim fixes are usually intended for a wider audience than other interim fixes, which are specific to a single customer. Examples of certified Interim Fixes are security fixes that are released thru a vulnerability advisory or subscription notification. 06/10/2009 8

Security Fixes Security fixes for security vulnerabilities are published thru advisories. Customers will continue to be notified of security fixes by notices from IBM Subscription services at: http://www14.software.ibm.com/webapp/set2/subscriptions/pqvcmjd The security fixes published in the vulnerability advisories are posted here: ftp://aix.software.ibm.com/aix/efixes/security In the past, security interim fixes were offered for a longer period than other fixes to allow customers time to transition to the current levels. But, because the enhanced service strategy allows new fix availability for a longer period (even longer than the previous security fix policy of 3 total TLs), security fixes will be made available for the same time period as all other new fixes (see chart above). This offers consistency to the service strategy in regards to types of fixes offered in the New Fixes Available period. To assist our customers with the transition to AIX 5.3 Technology Levels that support the enhanced service strategy, we will continue to generate 5.3 TL5 security interim fixes until 5.3 TL9 is shipped in 2H08 (approximately 10/2008). AIX 5.2 TL8 and AIX 5.2 TL9 support of security interim fixes will continue until 6/2008. AIX 5.2 TL10 will continue to be supported via Service Packs and interim fixes until end of support in April, 2009. APAR Numbers for each TL In the past, if a problem existed in two releases, only one APAR number was created for each release. Starting with AIX 5.3 TL7, a new APAR number will be created for every Technology Level where the problem is fixed. When PTFs were only made available on the current TL, only one APAR number was required. Now that multiple service streams exist per release, an APAR must be created for each one where a new fileset update (PTF) will be shipped. If you ve been given an APAR number for a particular problem, do a Fix Search in Fix Central to verify the APAR number with the Technology Level. The search will return results for the other sister APAR numbers, APAR numbers for the same release (e.g. 5.3) but different TLs, and also cousin APARs for other releases (e.g. 6.1). The search will also show availability of the APAR and allow signup for a subscription notice if it has not been released. For more information on searching for APAR and fixes, see the Fix Central paper at: http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html. New Automatic Interim Fix Removal Function Because of the issues with multiple APARs numbers in a release (APARs differ per TL as described above) installp was not able to map the fixes it was shipping to the interim fixes that were installed. Starting with 5.3 TL10 and 6.1 TL3, installp and emgr have been enhanced to map the APARs and automatically remove interim fixes if the fix is present in the TL, SP or PTF that installp is applying. Interim fixes will now be packaged with a reference number. That number will allow installp to map it back to APARs for all the Technology Levels where the fix was shipped. When installp performs a preview install/update or an actual update, information will be provided to list any interim fixes that will be automatically removed. Preview installs are always highly recommended, even if you don t currently have any interim fixes applied. But, if you do, 06/10/2009 9

then installp will list any interim fixes that are installed and that will be automatically removed or any interim fixes that are installed and cannot be removed because the corresponding fix was not available in the updates you were installing. In this later case, the install will fail stating there are filesets locked by emgr, like it has in the past. For example, the information about interim fixes that will be removed will appear directly after the build date verification in the installp (apply or preview) output: +-----------------------------------------------------------------------------+ BUILDDATE Verification... +-----------------------------------------------------------------------------+ Verifying build dates...done The updates being installed contain the following interim fix labels, which will be removed prior to installing the updates: IZ39706 IZ23965 NOTE: Rejecting an update that contains an interim fix will not re-install the interim fix on the system. Please note that if an interim fix is removed as part of the installp update, and then that update is later removed (rejected), the interim fix that was previously installed will NOT be re-installed. You would need to manually install the interim fix again at that point. So, always make sure you have a copy of your interim fix, use alt_disk_install or create a mksysb image before the update for recovery. Because this is new function, a decision was made to not ship it in Service Packs on the older Technology Levels. But, for this particular case, we will allow and support installing one individual fileset from a TL. The bos.rte.install 5.3.10.0 update may be installed on 5.3 TL7, TL8 or TL9 by itself, and the bos.rte.install 6.1.3.0 update may be installed on TL1 or TL2 by itself. This will give you the new automatic interim fix removal function but will not require you to go all the way to 5.3 TL10 or 6.1 TL3. You may also download the bos.rte.install fileset update individually from Fix Central by searching on bos.rte.install. From the http://www-933.ibm.com/support/fixcentral/ main site, choose: Product Group System p, Product AIX, Version 5.3, Fix Type Fix Search http://www14.software.ibm.com/webapp/set2/fixsrch/fixsearch?release=53 Enter search items: fileset bos.rte.install Then, in your search results you should select "Fileset information for: bos.rte.install". You will be taken to all the bos.rte.install fixes. Choose the latest update for your release so you ll get any additional fixes or function included. All the updates are cumulative, so you can apply to any level since GOLD. You will need to uncheck the download options of Include requisites, Include fixes that correct regressions and Replace superseded fixes with the latest. If you leave these options checked, then the entire TL will be listed for download. 06/10/2009 10

It is also recommended to watch the bos.rte.install fileset to pick up any fixes or new function. You can always update the bos.rte.install fileset to a lever higher than your current TL, so if new function ships in a later TL, you can obtain it without updating your entire system. All interim fixes shipping will have a reference field that allows installp to map the APARs to the interim fix. When the interim fix is installed (if you have the new bos.rte.install fileset applied) then a message will be printed by emgr to alert you to the fact that automatic removal is enabled: +-----------------------------------------------------------------------------+ Processing APAR reference file +-----------------------------------------------------------------------------+ ATTENTION: Interim fix is enabled for automatic removal by installp. To verify that your interim fix has a reference number, you can check the contents of the /usr/emgrdata/dbs/aparref.db file: # cat /usr/emgrdata/dbs/aparref.db IZ39706 : 220 This file contains a list of interim fix labels with their corresponding reference number values. Later, we ll update the emgr l command to show this reference number. So, again, it s a good idea to get the latest level of bos.rte.install to make sure you have all the latest fixes and enhancements. The bos.rte.install 5.3.10.0 or bos.rte.install 6.1.3.0 or higher update must be installed in order to get the automatic interim fix removal function. If this fileset update is installed along with other PTFs, in a Technology Level, for example, it will automatically re-invoke itself and it will then perform the mapping and automatic removal, if applicable. Hardware Support with a Service Pack When new hardware is released, the required TL s or SP s will be published in the release notes and RFA (Release For Announce). If the TL is still in the New Fixes Available period, then the latest SP will have the hardware support. If new boot media is required for an existing TL, then it will be made available for web download (ISO image) via the PRPQ process. If you currently have the TL in your environment, then updating to the required SP and creating a mksysb backup media or image (NIM) will allow boot and install of the new hardware systems. Moving to a New TL You should move to a new TL: If your existing TL is out or is about to go out of the New Fixes Available period 06/10/2009 11

You want to use new function and/or features in a new TL. Hardware exploitation, such as large page space or new software function, such as multibos, will only be released in a TL. You are going to test a new level for distribution into your production environment and want to get the longest period of new fixes available. In this case, you should move to the latest TL. Staying Current If you are currently running on a TL where new fixes are being made available, then any SP will be supported for your interim fix needs. But, the risk in not updating to a service pack that contains your fix during a planned maintenance window is that you will require another interim fix to the same file. See the Interim Fix Guidelines for more information. Updating twice a year to a new SP or TL is recommended to stay current. Maximum Stability Model Realizing that customers appreciated and relied on the CSP for planning their maintenance, and realizing that having a specific SP to recommend to customers that want to maintain the most stable environment and do not plan on moving up to a new SP before moving to a new TL, we would recommend using the SP that roughly would be delivered in the same time frame as the CSP. We would recommend planning around SP3. This is not to say that other Service Packs in the TL are less stable than SP3, but because the TL has been out a few months and any major issues have usually been identified by that point in time, and there is still more than a year left for service, SP3 is a good one to target and our recommendation for maximum stability. This recommendation is meant as guidance for planning. Individual situations may require applying earlier or later SP s, depending on circumstances. Additional Information The IBM Service and Support Best Practices for UNIX servers Web page at http://www14.software.ibm.com/webapp/set2/sas/f/best includes additional information on the AIX Release and Service Strategy, and best practices for managing the AIX and System p servers. The Fix Level Recommendation Tool (FLRT) is a planning tool to help administrators determine what key components of your System p server are at the minimum recommended fix level. It can be found at: http://www14.software.ibm.com/webapp/set2/flrt/home. Summary The new AIX service strategy will offer customers more options for maintenance in their environment. Two years of new fixes per TL will help simplify and reduce cost for maintenance. Supporting new hardware on existing TL s will reduce complexity in customer environments. 06/10/2009 12

IBM Corporation 2009 IBM Corporation Marketing Communications Systems Group Route 100 Somers, New York 10589 Produced in the United States of America June 2009 All Rights Reserved This document was developed for products and/or services offered in the United States. IBM may not offer the products, features, or services discussed in this document in other countries. The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in your area. All statements regarding IBM s future directions and intent are subject to change or withdrawal without notice and represent goals and objectives only. IBM, the IBM logo, AIX, AIX 5L, POWER5+, System p are trademarks or registered trademarks of International Business Machines Corporation in the United States or other countries or both. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others. Copying or downloading the images contained in this document is expressly prohibited without the written consent of IBM. Information concerning non-ibm products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of the non-ibm products should be addressed with the suppliers. The IBM home page on the Internet can be found at http://www.ibm.com. The IBM System p home page on the Internet can be found at http://www.ibm.com/systems/p. The IBM AIX home page on the Internet can be found at: http:///www.ibm.com/servers/aix. PSW03011-USEN-00 06/10/2009 13