SANDEEP TANTI'S UNIX-AIX WORLD: Patch Management in AIX

1 Preamble

2 Preinstallation checks

2.1 Check consistency and TL/ML

2.2 Check for applicability

2.3 Check rootvg

2.4 Check for cluster

3 Preinstallation tasks

3.1 Backup methods

3.2 Backup with alt_disk_install

3.3 Backup to tape

3.4 Backup to TSM

4 Installation of ML/TL, Service pack and APAR

5 Final notes

6 Conclusions

Preamble

This manual is directed to give some ideas and some framework for process of management of patches for AIX. In all the cases, not cover in this manual you should contact your local UNIX guru or search for help from consultant. Information in this article is based of AIX 5L v5.2 and 5.3. For Your version please refer appropriate manuals.

Preinstallation checks

Check consistency and TL/ML

Let's 1st check if packages, installed on the machine are OK

# lppchk -v

If you get report of something wrong try to get more info about the case and resolve the problem before continue with installation

# lppchk -v -m3

Check the current installed ML/TL

# instfix -i|grep ML

# instfix -i |grep TL

Keep the above information in file (not on the server, please)

Check for applicability

Check if APAR you want to install is applicable to your system. Check if fileset, mentioned in APAR is installed on your system

# lslpp -L

If not you can skip the rest of the document and feel happy, because it's need to do nothing on this machine :-) But if fileset is installed check if APAR is installed

# insfix -ivk

If it's installed be happy and jump to the next server, otherwise check if software is in use. For example you have installed drivers for ATM card, but you have no such card in the server or have installed software for IPsec, but not configured and not in use. In this case you can (again) be happy and continue with other task. Otherwise continue with steps in this document

Check rootvg

See if rootvg is mirrored and all lv's are mirrored correctly (excluding dump and boot volumes). If your rootvg is not mirrored you can skip later in document part for alt_disk_install, but in generally you should thing twice if situation like this is OK for business

# lsvg -p rootvg # lsvg rootvg # lsvg -l rootvg

Check for cluster

Check if the server is node of the cluster, because some of the operations can disrupt the proper work of cluster software

# smit hacmp

Check if the cluster processes are active

# lssrc -g cluster

Check if cluster software is installed

# lslpp -l | grep -i cluster

Preinstallation tasks

Create a new LV to keep ML/TL, Service pack and APAR's you will download from IBM web site. Be aware this file system can need up to 3.6 gigabytes. If you do not have enough free LP you can use any (almost) already existing partition on the server or arrange some disk space on other machine and export it via NFS. After finish the installation do not forget to remove the LV or directory if you do not need the files for other installation. In case of usage of NFS you should be aware process of installation is slower and any network problem can disrupt your installation and put you in situation of recovery system from backup or alternate disk installation. Upload (download) in this directory TL (last one), Service pack (only last one) and APARs for prepare the installation. Extract them in current directory. because they normally come as tar.gz file use this trick to keep some diskpace

# gzip -d -c TL...tar.gz|tar xvf -

Sound's good to extract and install Service pack separately (in different directory)

Backup methods

For safe yourself from problems after installation, boot problems, etc you have few opportunities. I personaly know 3:

Make alternate disk installation. This is actually process of splitting mirrored rootvg, create "copy" of current installation to unused disk and add the second this in boot list to be sure you have working copy of your operating system in case of problem
if you have tape device on your server use mksysb to backup you rootvg on the tape and use this backup in case of problem
if you have in the network TSM (Tivoli Storage Manager) and NIM (Network Installation manager) servers installed you can backup system on the TSM and in case of problem reinstall operating system from NIM server

If you can't use any of the above methods you should be prepared for installation from CD/DVD of the entire operating system and recover it to the level to the situation before installation of APAR. Honestly speaking you can use mksysb, saving backup file on NFS mounted directory, but this will be no so helpful for easy and fast restore of the system

Backup with alt_disk_install

So lets start with the 1st method - alternate disk installation Remove the secondary dump device. On normal systems with mirrored rootvg exist 2 different dump devices. So to free one of the disks we should remove the dump device, resided on that disk. First we should find this device. To see the name of the secondary dump device

# lsvg -l rootvg | grep dump

Then let's change the settings for secondary dump device to /dev/sysdumpnull . Do not use /dev/null

# smit dump

And now we can remove the dump Logical Volume

# smit lvm

Next step is to unmirror the rootvg. To do this we need to check which physical disks are included in rootvg. Here and below we assume disks, included in our rootvg are hdisk0 and hdisk1. In case of more that 2 disks in your rootvg process of doing this operation become more complex and it's out of scope of this document. The similar is the situation if you have many unmirrored LV in rootvg and the can't be allocated only to one PV.

# lsvg -p rootvg

Choose what disk to be leaved in rootvg based on free space, boot list and logical volumes that resign on only one of the disks

# lspv hdisk0 ; lspv hdisk1 ; lspv -l hdisk0 ; lspv -l hdisk1

See the current bootlist

# bootlist -m normal -o

Remove the second disk from the mirror

# unmirrorvg rootvg hdisk1

Check if there are any LVs left on the second disk

# lspv -l hdisk1

And if exist migrate the remaining LVs to the first disk.

# migratepv hdisk1 hdisk0

Remove the second disk from the root VG. For do this it's need to clear the boot record of the second PV

# chpv -c hdisk1

Check if the bootlist is set first to the disk with the current rootvg, and on second place the disk with the altinst_rootvg (and optionally cd0, if exist in current bootlist)

# bootlist -m normal hdisk0 hdisk1

Add new boot image to the first PV to have “fresh” boot record and be not afraid of not boot from rootvg

# bosboot -ad /dev/hdisk0

Removes the second PV from rootvg

# reducevg rootvg hdisk1

Now we have one free PV on with we can create alternative disk installation. Now is time to check the existing FS and exclude the unnecessary ones from the backup, including the FS created for the download of the TL/ML. You can check if there is already an exclude list made for tape backup

# lsvg -l rootvg

# cat /etc/exclude.rootvg

# cp /etc/exclude.rootvg /etc/exclude.altdisk

# vi /etc/exclude.altdisk

The file should look like this

/opt/myfiles/tmp/*

Do not put in exclude list /tmp, /var/tmp and similar directories, because this can cause problems if you will need to use alternate disk installation for recovery purpose. If you have names of LV more that 11 characters you should rename them to be no more that 11, because of limitation of software for alternate disk installation. To do this you should first unmount the filesystem, rename the LV and mount it again. You also should avoid naming VG, LV and devices starting with letters “alt” to avoid modification, deletion or damage the items mentioned above. The command to create alternate disk installation is sample, but you should be careful of syntax, because the results can be catastrophic

# nohup alt_disk_install -BC -e /etc/exclude.altdisk hdisk1 &

We put all in background to avoid disruption of operation in case of network problems. If you do not have/use exclude files you can omit -e key and path to exlude file. For more information about the command, please see “Commands Reference, Volume 1, a – c” of AIX OS. To monitoring the process of creation use

# tail -f nohup.out

After successful end of above operation you can test alternate disk installation (using wakeup key)

# alt_disk_install -W hdisk1

List the content of PV

# lspv -l hdisk1

And if everything is ok put the installation in “sleep”

# alt_disk_install -S

Backup to tape

The second way to backup our rootvg is to use mksysb utility. To do this we should be sure have tape device in the macine

# lsdev -Cc tape

And we have tape in to device

# tctl -f /dev/rmt0 rewind

# tctl -f /dev/rmt0 status

Alternatively you can use mt instead of tctl. Then we can create exclude list for filesystems we do not want to exist on the tape image

# vi /etc/exclude.rootvg

After this we can run the backup from command line or via smit

# smit mksysb

# nohup mksysb -m -e -i /dev/rmt0 &

If your tape device is not /dev/rmt0 change it in above command. If you want to create image on NFS mounted directory do the above operations, changing the target device to file instead of tape device

# smit mksysb

# nohup /usr/bin/mksysb '-m' '-e' '-i' '-X' /backup/image_date &

Backup to TSM

If You want to use TSM for backup use command

# smit sysback

For the rest of process consult your local storage/backup administrator and/or documentation

Installation of ML/TL, Service pack and APAR

Check for some processes you should stop by hand in time of installation to avoid problems and crashes like Oracle Database server, Tivoli Gateway or TEC server. Before start installation you should commit all the software, installed on the server to make easy roll-back in case of problem in time of installation

# smit commit

# installp -c all

Before install TM/ML and Service pack is wise to install installp software itself to have better overview later when you try to preview the process of installation of TL/ML

# smit install - bos.rte.install

# installp -a -g bos.rte.install

Next step is to update all the software from TL/ML and Service pack

# smit update_all

Do not forget to set only apply the package, but not commit and keep the copy of replaced files. After installation of packages check again your system:

# lppchk -v

# oslevel -r

# instfix -i|grep ML

# instfix -i|grep TL

# instfix -i|grep SP

# errpt |more

# bootlist –m normal -o

And if everything looks fine reboot your server

# shutdown -Fr

Final notes

Recreate the mirror of rootvg
After few days of stable work and some tests from application users it's time to recreate mirror of rootvg (if you are using this way of backup). Let's destroy alternate disk installation

# alt_disk_install -X

Check which harddisk was previously included in rootvg, and add it

# extendvg -f rootvg hdisk1

Recreate secondary dump device. Extract the name and size of the primary dump device

# lsvg –l rootvg

Calculate the size of the necessary paging space with the command (in some cases is good idea to consult with your application administrators and to increase the size of secondary dump device):

# sysdumpdev -e

Create LV similar to the first one with type of LV sysdump

# smit lvm

Do not forget to change the secondary dump device from /dev/sysdumpnull to freshly created one

# smit dump

Then you can mirror rootvg. To avoid “locking” of your terminal set process in background

# smit lvm

# nohup mirrorvg '-S' rootvg hdisk1 &

Create boot image on hdisk1

# bosboot -ad /dev/hdisk1

Add hdisk1 to bootlist

# bootlist -m normal hdisk0 hdisk1

Please check the bootlist you get when you check the machine before installation and add other devices to the bootlist if it's necessary. Synchronize LVs on both disks

# nohup syncvg -v rootvg &

Conslusion

This manual is not directed to replace your company policies/procedures, but give you tested way to do the work :-)