Updating to a new Technology Level or Service Pack

Technote (FAQ)

Question

What is the recommended process for upgrading to a new Technology Level or Service Pack in AIX ?

Answer

-- Updating to a New Technology Level or Service Pack --

Update_All in 5.3, 6.1, and AIX 7

This document describes the recommended preparation and process when considering updating your system to a new technology level or adding a service pack to an existing technology level. In all we will review some key words and terminology, run through recommended pre-checks, discuss the update_all process using both SMIT and command line, and finally post-checks & FAQ.

Key Words / Terminology :

V.R.M.F. (Version, Release, Maintenance Level, Fix Level)
This represents your operating system level and is mentioned because of the change made in the 5300-07 technology level update. Previous to 5300-07 updates are recognized by their last numerical level entry only.
Ex.bos.rte.install 5.3.0.60

Beginning with 5300-07 the 3rd number, or “Maintenance Level” number will also be included and will represent the Technology level of the fileset, followed by the “Fix Level”.
Ex.bos.rte.install 5.3.7.0

*Note that the “Fix Level” entry does not necessarily represent the “Service Pack” level. Any similarity is simply coincidental.

Oslevel 
You will also notice a change to how the operating system level reads. With the introduction of 5300-06 you will notice more information provided when running the ‘oslevel’ command. # oslevel -s
6100-06-06-1140
The addition of the last four digits simply represents the year (11) and week (40) of the release of your technology level/service pack (-06-06).

Update_all
This is the recommended method of installation when upgrading your technology level or service pack level only. This installation method should not be used when attempting to upgrade the version or release level of your operating system.
Ex.5300-0x to 6100-0x - do NOT use update_all
6100-0x to 7100-0x - do NOT use update_all

Recommended Pre-Checks :

1. The back-out
It is recommended to have at least one back-out method available to you in case a restore is required. Recommended back out methods include : mksysb restore, sysback restore, altinst_rootvg clone, and multibos. More information on any of these back-out methods can be found at the publib documentation site by using this link to click through to the appropriate infocenter and search features found here : http://publib16.boulder.ibm.com/pseries/

2. Boot Image Verification
The hd5 logical volume holds your boot image. There are a few checks to make against your boot image before starting your update_all process.
First, find out which disk contains your hd5 logical volume.

# lslv -m hd5
hd5:N/A
LP       PP1     PV1
0001     0001    hdisk0
The listing under the “PV1” column indicates on which disk hd5 is located. Any other entries under “PV2”..etc, simply represent mirrored copies.

Next, verify your boot image can be successfully recreated using the hdisk# from above and by using /dev/ipldevice.

# bosboot -ad /dev/hdisk0
bosboot: Boot image is 35795 512 byte blocks.

# bosboot -ad /dev/ipldevice
bosboot: Boot image is 35795 512 byte blocks.If either of these two commands fail for any reason, please call the support center for resolution before proceeding.

Finally, with the continual introduction of new devices and hardware the boot images are growing larger. You will want to make sure your hd5 logical volume has enough free and contiguous space to hold the boot image. Currently the recommended allocated space to have for hd5 is 32meg. This is of concern in environments with smaller disk sizes, and should be checked before running an update. # lsvg -l rootvg |grep hd5
hd5 boot      1    1    1    closed/syncd       N/A

# lsvg rootvg |grep SIZE
VG STATE:     active         PP SIZE:      32 megabyte(s)

The single partition (in bold (1)) is 32meg in size. This should be enough to contain the boot image. Smaller partition sizes will require more partitions to be allocated.

Your hd5 partitions also need to be contiguous partitions. Check this by running the following command :

# lspv -M hdisk0 |grep hd5
hdisk0:1 hd5:1
hdisk0:2 hd5:2
hdisk0:3 hd5:3
You can see in this example that the hd5 logical volume covers the first 3 partitions on the disk and they are all contiguous. If your partitions are not contiguous, or are not covered on the first partitions of the disk please call the software support line for assistance with correcting this.
Again, you may only have 1 partition that is large enough to handle the boot image, or you may have multiple smaller partitions. Either is fine.

3. Firmware
Firmware should be kept up to date and be checked whenever a technology level update is considered. In general firmware updates should be applied before software updates, but that is not a rule set in stone. You should always refer to the firmware download site installation information and follow those instructions.
You can access the firmware download site here : http://www-933.ibm.com/support/fixcentral/

If there are any questions concerning the firmware update, installation instructions, or if there are software considerations please contact the hardware support center first.

4. Fileset Consistency
You should run the following command to check fileset consistency : # lppchk -v
Ideally this should return to the command line with no output. If you do receive output and are unfamiliar with how to resolve it, please call the support center for assistance before running your upgrade.

5. Space
In order to see if you have enough space for your update you can run the update_all operation in preview mode. This will provide you with anestimated system resources listing of the necessary space. In order to do this you will first have to install the fileset “bos.rte.install”. Following are two examples of how to update this fileset first, so a preview update_all can be executed. The example below shows initially “committing” the bos.rte.install update. If you would feel more comfortable using the “apply” feature in this case, feel free to do so. Note that if you do “Apply” you will need to “Commit” before continuing with the actual update_all process.

From an unmounted cdrom device (cd0).

# installp -acd /dev/cd0 bos.rte.install
From a download directory (/fixes/6100-06)
# installp -acd /fixes/6100-06 bos.rte.install

Using SMITTY

# smitty install_all
* INPUT device / directory for software         /dev/cd0
* SOFTWARE to install                           [bos.rte.install]
PREVIEW only? (install operation will NOT occur) no
Once the installation is complete you will then be able to run the preview operation on the remaining filesets to get an estimate of the required space for your update. # smitty update_all
* INPUT device / directory for software         /dev/cd0
* SOFTWARE to update                            _update_all
PREVIEW only? (update operation will NOT occur) yes

The preview will show something similar to this in regards to the space requirements :
RESOURCES
---------------
Estimated system resource requirements for filesets being installed:
(All sizes are in 512-byte blocks)

Filesystem	Needed Space	Free Space
/	402	239312
/usr	94352	165440
/var	1	237384
/tmp	52304	259240
/opt	6648	158576
Total	153707	1059952

Running the update using SMIT or command line :
This section will cover the update_all process. In all cases the “input device” (location of the software) will be referred to as being a local cdrom drive (/dev/cd0). You can always use a download directory or nfs mount point, simply substitute the input device as necessary. This process is also executed presuming you are logged in as root (as opposed to using ‘sudo’).

1. SMITTY :
By menu selection options you would take the following path :

# smittySoftware Installation and Maintenance
Install and Update Software
Update Installed Software to Latest Level (Update All)

-or use the fastpath-# smitty update_all

INPUT device / directory for software                    /dev/cd0
SOFTWARE to update                                         _update_all
PREVIEW only? (update operation will NOT occur)            no
COMMIT software updates?                                   yes
SAVE replaced files?                                       no
AUTOMATICALLY install requisite software?                  yes
EXTEND file systems if space needed?                       yes
VERIFY install and check file sizes?                       no
DETAILED output?                                           no
Process multiple volumes?                                  yes
ACCEPT new license agreements?                             yes*
Preview new LICENSE agreements?                                 No
The only required change would be the option for “ACCEPT new license agreements”. All other given defaults should remain as they are. Pressing here will bring up a confirmation dialog box to which you press again. The only other interaction you may have is if you need to switch out a volume of the media package. When complete, the upper left corner “Command:” field will have one of two outcomes :

OK : Your installation has completed without error. You can either scroll down or exit out and view the /smit.log file for the log of the installation.

FAILED : One or more filesets failed for some reason. You can either scroll down or exit out and view the /smit.log file for the log of the installation. Corrective action should be taken before rebooting.

2. Command line :
You will use the ‘install_all_updates’ command if opting to run this from command line.
This command does have a representative manpage so feel free to review the flags, however this command was intended to be easily executed so running a basic update will not be a very long command.

# install_all_updates -Y -cd /dev/cd0

Once complete you will find the log of the installation in /var/adm/ras/install_all_updates.log. By default the fileset updates will be installed in the “APPLIED” state. The "-c" flag has been added to the above command to specify that all filesets install in the “COMMITTED” state. See the FAQ at the end of this document for more information concerning “APPLIED” vs “COMMITTED”.



Post checks :
Presuming your update_all was successful you will want to check the following commands. If you receive unexpected output please contact the support center for assistance.
* Please see FAQ statement 11 and 11a below regarding post-update_all issues.

# oslevel -r        This should return with the expected TL output# oslevel -s        This should return with the expected TL and SP output.# lppchk -v         This should come back ideally with no output.

FAQ and Opinions

This section is contains the most commonly asked questions and a few recommendations concerning the update_all. Most answers will be kept short and to the point (as much as possible).

1. Is it okay to install my Technology Level in the “APPLIED” state ? What if I want to reject the TL update if there is a problem ? 
With the introduction of Technology Levels and the “all or nothing” packaging format, these updates are bringing in on the upwards of 400+ fileset updates for each TL. Attempting to perform a “reject” process on so much code simply does not work well, and is not supported. Recommended back-out methods are discussed earlier in this document.

1a. Does the same hold true for Service Packs ? 
The Service Pack updates are certainly much smaller groups of updates....typically numbering around 40-50 per update. While you certainly will have a better chance of successfully rejecting 40 filesets instead of 400, it would still be best to have one of the back-out methods mentioned earlier.

2. Why does my update_all have problems when I use an NFS mounted cdrom ?
When running an update_all over NFS, you create your own mount point for the cdrom drive. This does not allow it to recognize the fact that there are multiple volumes available that contain the correct requisite filesets. Multiple update_all operations may be needed on each volume when NFS mounting a cdrom drive. A better option may be to ‘bffcreate’ the contents of all CDs down to a directory, then NFS mount that directory.

3. The update_all failed. What now ?
Do not reboot. You can review the log to find out what failed and why. If you are comfortable and familiar with these situations you can correct the failures and re-update the filesets. If not, please call the helpdesk and open a PMR for assistance. Have the log available to email in if necessary.

4. I need to run my update today but I may not be able to reboot until next week. Is that a problem ? 
Plans should be made to reboot as soon as the update is complete and checks have been made to ensure there were no failures. System files will have been replaced, but the corresponding kernel and library updates will not be loaded until boot time. You will likely encounter problems if you delay rebooting.

5. Is it recommended to reboot before issuing a TL upgrade ?
If this is possible, absolutely. There are systems out there that have not been rebooted in a year or more. Who is to say that something has not happened in that time that simply would not show up until a reboot. Rebooting the system first assures a good boot image, a stable system, and would isolate any problem that normally would not be caught until the post-update reboot as either a preexisting issue, or an issue directly related to the TL update itself.

6. Some say to use base AIX install media when updating the TL, others say the TL fix downloads or CDs should be used. Which is right ?
The recommendation is to use the TL fix downloads from FixCentral, or the TL CDs that can be ordered either online or from AIX SupportLine. You can also use the base AIX installation media, however without getting into a long answer, the recommendation is using the TL fix packages.

7. How long will the update_all take ?
This is a question that does not quite have a definite answer. The length of time required for the update_all depends on how many filesets are being updated as well as the available system resources such as processors and memory.
Giving a ballpark figure however, going up one TL should only take about 30-60 minutes plus reboot time. Some admins prefer to be more on the conservative side and block a 2-3 hour downtime. Consideration should also be made to the amount of time it would take to restore, depending on the backup method selected.

8. Is it okay to run the update_all while people are online ?
Updating could affect running processes. As such, applications should be down and users offline as a general rule.

9. Where can I download new service packs or technology levels ?
TL and SP updates can be acquired from the FixCentral website located here :http://www-933.ibm.com/support/fixcentral/

10. Will product X at level Y.Z work with my new technology level ?
Some products may only be certified up to a certain operating system level while other products may require an update. The best thing to do would be to contact product X's support center. If it is an IBM product feel free to contact our support center and open a PMR requesting to speak to the product X team. Any 3rd party products should be cleared by their support before upgrading.

11. The 'oslevel -s' and/or 'oslevel -r' commands do not report what I expect after the udpate. How can I determine what is missing ?
If your 'oslevel' commands do not report correctly after your update_all you can add the "-l" flag to determine which filesets still need to be updated in order to complete your upgrade.
The syntax :
# oslevel -rl
# oslevel -sl

Example :
# oslevel -rl 6100-01
-or-
# oslevel -sl 6100-00-01-0748

The filesets listed will show an "Actual Level" heading (your current level) and a "Maintenance Level" heading (the level you need to be at to satisfy the TL/SP).

11a. Some filesets occasionally require multiple update_all operations.
There are cases where a second update_all operations need to be executed to pick up the full TL update. This is not defective behavior. Please also take note that this tip is not intended to resolve a "FAILED" status of your update_all. If you notice that your TL update was successful, but some filesets did not get updated and are on your media or in your download location, you may need to run the update_all a second time to pick them up.

12. What are other best practices with regards to keeping the AIX operating system updated?
IBM maintains a best practices site for service and support for Power Systems.
Visit http://www.ibm.com/support/customercare/sas/f/best for more information.