Tanti Technology

My photo
Bangalore, karnataka, India
Multi-platform UNIX systems consultant and administrator in mutualized and virtualized environments I have 4.5+ years experience in AIX system Administration field. This site will be helpful for system administrator in their day to day activities.Your comments on posts are welcome.This blog is all about IBM AIX Unix flavour. This blog will be used by System admins who will be using AIX in their work life. It can also be used for those newbies who want to get certifications in AIX Administration. This blog will be updated frequently to help the system admins and other new learners. DISCLAIMER: Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility. If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.

Friday, 13 December 2013

AIX TECHNOLOGY LEVEL(TL) UPDATE METHODS

As an AIX system administrator, you know the importance of keeping the systems up to date. Unfortunately, this is always a little tricky and, sometimes, due to the size of the environments, very hard to accomplish. This article shows the many ways available to update an AIX server.

Introduction

Staying current with the latest AIX Technology Level (TL) is always the best option to better availability, reliability and security. TL is a set of fixes, and new features added to an AIX version or new hardware support.

You should considered moving to a new TL version for the following reasons:

  • A new function provided in a new TL is needed.
  • If the existing TL is out or is about to go out of new fixes and service packs.
  • The system currently has a need for a fix, which is present on the new TL.
This article goes through the multiple ways of doing a TL update and the fallback options, as well. It shows the approved and supported TL upgrades methods provided by IBM.

Before you begin

TLs must be applied as a group. Installing a TL is an "all or nothing" operation. Installing a partial TL is not recognized from a support standpoint.

Before applying a TL, always create a backup of you current installation, and plan on a worst case scenario—on restoring that backup if needed to rollback the system to the previous level.

Also, TL updates should always be committed because they cannot be rejected. If a TL has been applied and needs to go back to the previous version, then a fallback plan is needed.

The general rule of thumb is to always create a backup before beginning. It can be any kind of image backup (mksysb image, a sysback image, etc).

After a valid backup image is available, the upgrade process can be started.

Also, it is a good practice to create a health checklist, that is, save as much as information from the system (netstat, ifconfig, lsvg, lsdev, lscfg, prtconf, etc.) and keep it somewhere other than the server that is being upgraded. Keep in mind that this information is only going to be used as support material in case of problems.

If the TL version to be installed is previous to AIX 5.3 TL10 and AIX 6.1 TL3, make sure all interim fixes (ifix) have been removed from the system. Listing 1 shows how to check for installed ifixes, and listing 2 shows how to uninstall an ifix:
Listing 1. Checking for installed ifixes
# emgr –l
Listing 2. Removing an ifix
# emgr –r –L 
For further information on how to use the emgr command, check IBM documentation or the man pages.

If you are updating to AIX 5.3 TL10 or AIX 6.1 TL3, these steps do not need to be executed since installp and emgr have been enhanced to automatically remove the ifix when present in the TL. Otherwise, the ifixes will have to be manually removed.

Check if all filesets are applied and are valid, as shown in Listing 3.
Listing 3. Checking installed filesets consistency
# instfix –i | grep ML
# lppchk –v
If problems are reported from running these commands, stop now and fix the problems. Applying a TL to an inconsistent AIX will not only damage the operating system, but it can also make it unbootable.

Next, all filesets in the APPLIED state needs to be COMMITTED. To commit them, follow the example in Listing 4.
Listing 4. Committing all APPLIED filesets
# installp -c -f -g –X
Or, use smitty commit and follow the instructions on the screen.

Here are most common methods that are used in AIX TL up gradations.

  1. Alternate Disk
  2. Multibos
  3. Same Disk
  4. Using NIM
Going forward you will learn about each method in detail.

Alternate disk

This is one of the most used practices to apply a new TL. It is referenced in IBM technical documents and books. Using an alternate disk gives the following options:

  1. Using a secondary disk and apply the TL without disruption, only rebooting after the TL upgrade is done—IBM recommends that all processes be terminated and users to be logged off (this procedure is covered in this article).
  2. Rebooting to a secondary disk and running the TL upgrade to what was the primary disk—some system administrator often use this to make sure the system is running on a consistent state and has no problems or malfunctions after a reboot is performed.
Either way, a free disk is needed to install using alternate disk.

For this alternate disk update method, the server to be updated has a rootvg with two mirrored disks. Therefore,unmirror it before beginning and use the second disk for the alternate disk installation, as shown in Listing 5. 
Listing 5. Unmirroring rootvg
# unmirrorvg rootvg
If a secondary dump device is configured to use the secondary disk, move its LPs to the remaining disk usingmigratepv or unconfigure the secondary dump device.

Remove the disk from the rootvg (assuming that X is the disk device number), as seen on Listing 6.
Listing 6. Removing disk from the rootvg
# reducevg rootvg hdiskX
Now create the alternate disk and apply the TL update to it. The TL files can be placed locally, as covered in this article, in a remote NFS share or in a CD-ROM.

Smitty fastpath smitty alt_clonecan be used, or the alt_disk_copy command line. Figure 1 shows the initial smitty screen.
Figure 1. Initial smitty alt_clone screen
image001

Remember, if you are unsure about a field, pressing F1 provides help.

Moving forward, use hdisk1 and the TL files are locally under the /stage_TL filesystem, as shown in Figure 2.
Figure 2. Added hdisk1, bundle to install, directory with images and the acceptance of license agreements
image002

All operations will be logged to /var/adm/ras/alt_disk_inst.log. To watch its progress, enter tail –f to it.

The server will need to be rebooted after the update process, so make sure the bootlist is showing the target alternate disk as the first boot device (as seen in Listing 7).
Example 7. Checking boot device order
# bootlist –m normal –o
hdisk1 blv=hd5
Once the server has been rebooted, issue oslevel –s or oslevel –r and check if the OS level is now at the updated TL, as demonstrated in Listing 8.
Listing 8. Check running AIX version
# oslevel –s
5300-10-01-0921
If the update is considered successful, the rootvg can be mirrored again. Listing 9 shows how to mirror the rootvg again.
Listing 9. Mirroring back rootvg
# exportvg old_rootvg
# extendvg –f rootvg hdisk0
# mirrorvg rootvg
Create a new boot image on hdisk0 and add it to the boot list, as seen in Listing 10.
Listing 10. Creating a boot image
# bosboot –ad /dev/hdisk0
# bootlist –m normal –o hdisk0 hdisk1

Multibos

This is by far the coolest way to have AIX upgraded. It was introduced with AIX 5.3 TL3. This is great in cases where only one disk is available on rootvg and no free disks for alternate disks are available.

Multibos creates and maintain two different and bootable AIX instances within the same rootvg. It is similar to alternate disk. In this case, the biggest difference is that multibos will create and copy only the following Logical Volumes (LVs):

  • /;
  • /usr;
  • /var;
  • /opt, and;
  • hd5 (Boot logical volume).
All others LVs will be shared with the original root volume group, although more logical volumes can be specified and copied. Multibos supports a new TL update, but AIX version upgrades are not supported by multibos.

In addition to the tasks mentioned in the Before you begin section, make sure that enough free disk space to copy each BOS logical volume to the same root volume group disk is available, otherwise multibos will not work.

Create a new standby BOS instance by running the multibos command. Check its options and documentation before you begin. Listing 11 shows how to create a new standby BOS instance.
Listing 11. Preview of multibos standby BOS creation
# multibos –sXp
This shows a preview of what multibos is about to execute. For further information always check its log file (/etc/multibos/logs/op.alog). If everything seems to be OK with the preview, execute it again without the preview flag (-p) as shown in Listing 12. 
Example 12. Multibos standby BOS creation
# multibos –sX
It will take a few minutes to copy all the contents, and after it’s completed all new LVs will be prefixed by "bos_". Listing 13 shows how the rootvg will look like after the new standby BOS has been created.
Listing 13. Multibos standby BOS created
# lsvg –l rootvg
rootvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     1     1    closed/syncd  N/A
hd6                 paging     192   192   1    open/syncd    N/A
hd8                 jfs2log    1     1     1    open/syncd    N/A
hd4                 jfs2       1     1     1    open/syncd    /
hd2                 jfs2       17    17    1    open/syncd    /usr
hd9var              jfs2       8     8     1    open/syncd    /var
hd3                 jfs2       4     4     1    open/syncd    /tmp
hd1                 jfs2       1     1     1    open/syncd    /home
hd10opt             jfs2       1     1     1    open/syncd    /opt
lvdump1             sysdump    22    22    1    open/syncd    N/A
lvdump2             sysdump    22    22    1    open/syncd    N/A
bos_hd5             boot       1     1     1    closed/syncd  N/A
bos_hd4             jfs2       1     1     1    open/syncd    /bos_inst
bos_hd2             jfs2       17    17    1    open/syncd    /bos_inst/usr
bos_hd9var          jfs2       8     8     1    open/syncd    /bos_inst/var
bos_hd10opt         jfs2       1     1     1    open/syncd    /bos_inst/opt
It is a good idea to enter the newly created BOS instance shell and verify its current TL, as shown in Listing 14. To exit from the multibos environment just type 'exit':
Listing 14. Entering the multibos shell and checking AIX version
# multibos –S
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Multibos Shell Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
Multibos Root Shell
+-----------------------------------------------------------------------------+
Starting multibos root shell ...
Active boot logical volume is hd5.
Script command is started. The file is /etc/multibos/logs/scriptlog.090904032855.txt.
MULTIBOS> oslevel –s
5300-06-08-0831
If all prerequisite tasks have been completed (see Before you begin section), the TL update can be started. The command used in the Listing 15 will update your newly created standby BOS instance.
Listing 15. Updating the standby BOS instance
# multibos –Xac –l /stage_TL
After the command is completed, enter it again on the multibos shell and check the current TL, as seen in Listing 16.
Listing 16. Entering multibos shell
# multibos –S
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Multibos Shell Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
Multibos Root Shell
+-----------------------------------------------------------------------------+
Starting multibos root shell ...
Active boot logical volume is hd5.
Script command is started. The file is /etc/multibos/logs/scriptlog.090904035718.txt.
MULTIBOS> oslevel –s
5300-10-01-0921
Configure and ensure the boot list is pointing to the standby BOS as the first boot device, as shown in Listing 17.
Listing 17. Setting up boot to your new standby BOS instance
# multibos –B

# bootlist –m normal –o 
hdisk0 blv=bos_hd5
hdisk0 blv=hd5
If the update procedure failed and a fallback is needed, set and verify the boot list back to the previous boot LV and reboot—this will bring back the older AIX version. This procedure is shown in Listing 18.
Listing 18. Changing the boot device back to your original rootvg
# bootlist –m normal –o hdisk0 blv=hd5 hdisk0 blv=bos_hd5
# bootlist –m normal –o
hdisk0 blv=hd5
hdisk0 blv=bos_hd5
But, if no problems were found and the standby BOS is not necessary any longer, it can be removed by issuing the command shown in Listing 19.
Listing 19. Removing the old rootvg
# multibos -R

Same disk

This is the simplest method available. The downside of this method is that two reboots are needed in case of a fallback.

In this example, a backup to an alternate disk will be done before the update process. So, jump to smitty alt_clone again, as shown in the Figure 3 and select the desired backup disk and hit enter with the default values.
Figure 3. smitty alt_clone menu
image003

The command line can also be used, as shown in Listing 20.
Listing 20. Command-line for cloning rootvg
 # alt_disk_copy -P "all" -d "hdisk1" -B
To follow the progress of the alternate task look at the alternate disk log file, /var/adm/ras/ alt_disk_inst.log.

After the alternate disk is done, the update process can be performed. Use the smitty fastpath smitty update_all or from the command line use install_all_updates. Listing 21 shows the update process.

Enter the directory containing the TL filesets:
Listing 21. Same disk update process
>
 # cd /stage_TL

And create a Table of Contents file (ToC):

 # inutoc .
While still inside the directory containing the filesets, run our smitty update_all command as shown in Figure 4.
Figure 4: smitty update_all initial screen
image004

The first screen will ask where the filesets are, add a “.” (dot) and press enter.
Figure 5 shows how the smitty menu will look like.
Figure 5. smitty update_all menu with options
image005

After the update process is done, reboot the server.

Once the server has been rebooted, issue oslevel –s or oslevel –r and check if the OS level is now at the TL level that was applied as observed in Listing 22.
Listing 22: Checking AIX version after update
 # oslevel –s
 5300-10-01-0921
If the update was considered successfully the rootvg can be mirrored again. Follow the example in Listing 23 to re-mirror the rootvg and create a boot image on hdisk1 and add it to the boot list.
Listing 23. Re-mirroring the rootvg
 # exportvg alt_inst_rootvg
 # extendvg –f rootvg hdisk1
 # mirrorvg rootvg
 # bosboot –ad /dev/hdisk1
 # bootlist –m normal –o hdisk0 hdisk1

Using a NIM Server

This is also one of most popular ways of updating an AIX server. However, in this case an up and running NIM Server is needed.

This article does not intend to show how to configure a NIM server or its pieces: spots, lpp_sources, machines, etc.

As was done for the previous methods, refer to the Before you begin section to review if all requirements were met. If yes, follow the example in Figure 6 to get started:
Figure 6. Entering smitty nim menu
image006

This will enter the main NIM server smitty menu as shown in Figure 7. On the first screen select “Perform NIM Software Installation and Maintenance Tasks”
Figure 7. Main NIM server smitty menu
image007

Next, select “Alternate Disk Installation”, as shown in Figure 8.
Figure 8. Select "Alternate Disk Installation"
image008

As "Alternate Disk Installation" option is selected chose "Clone rootvg to an Alternate Disk".
Figure 9. Select "Clone the rootvg to an Alternate Disk"
image009

The next screen is where all the settings are done.

Select the client machine (server to be updated), type the disk which the TL will be applied—since this is going to be a alternate disk, make sure the disk is not used by any other volume group (VG). See Figure 10 for all options available on this menu.
Figure 10. NIM settings for Alternate Disk Install
image010

After you confirm and enter the values, NIM will automatically start updating the client.

Its progress can be seen from the client by looking at two log files (/var/adm/ras/nimlog and /var/adm/ras/alt_disk_inst.log).

Note:During the update process the screen won't be showing anything, but the process will be running on background. Make sure the process is completed before a reboot is done on the server—look at the log files.

After the update process is done, reboot the server.

Once the server has been rebooted, issue oslevel –s or oslevel –r and check if the OS level is now at the TL level that was applied.

 # oslevel –s
 5300-10-01-0921
If the update is considered successful, the rootvg can be mirrored again:

 # exportvg alt_inst_rootvg
 # extendvg –f rootvg hdisk1
 # mirrorvg rootvg

Create a new boot image on hdisk1 and add it to the boot list:

 # bosboot –ad /dev/hdisk1
 # bootlist –m normal –o hdisk0 hdisk1

AIX 6.1 Installation Tips

This document contains the latest tips for successful installation of AIX 6.1.


Technology Levels, Service Packs, APARs and PTFs mentioned in this document, when available, can be obtained from the following web site.

The AIX installation CD-ROMs and the level of AIX pre-installed on new systems may not contain the latest fixes available at the time you install the system, and may contain errors. Some these fixes may be critical to the proper operation of your system. We recommend that you update to the latest service pack level, which can be obtained fromhttp://www.ibm.com/eserver/support/fixes/fixcentral/main/pseries/aix.
When updating to a new Technology Level, it is a good practice to first update the bos.rte.install update in a separate installation session.
With certain combinations of updates, the update process may have to be run a second time in order to apply all updates in a package. Checking the output of the 'oslevel -r' and 'oslevel -s' commands for the expected values after an update is recommended. Until the update is run a second time, the output of the oslevel command may not indicate that the package is fully installed.
The compare_report command, which is documented in the AIX Commands Reference, can be used to determine which available updates are newer than those installed on your system.
Any library or executable that is updated by an interim fix (emgr) or service update, which is in use by an active process, will not be reflected in that process until it is restarted. In addition, any process that is using a library and does a dlopen() of the same library after the library has been updated, could experience inconsistencies.


Build date failure updating from 6100-07-07 to 6100-08-02

Attempting to update from 6100-07-07 to 6100-08-02 results in build date errors for the following filesets.
  • devices.pciex.df1020e214103c04.rte
  • devices.pciex.df1020e214103c04.diag
  • devices.pciex.df1020e214100f04.rte
  • devices.pciex.df1020e214100f04.diag
To resolve this issue, download the fix for APAR IV42218, and include the images into the same directory as the 6100-08-02 Service Pack prior to updating.
This issue only occurs when updating from the 6100-07-07 Service Pack.

6100-08 Installation

When applying the 6100-08 Technology Level with Service Pack 1, you may have to run 'smitty update_all' a second time to update bos.aso. Until this is done, the 'oslevel -s' command may not indicate the correct level.

6100-08 Service Pack 1

The 6100-08-01 Service Pack is considered highly recommended. 6100-08-01 is included in the 6100-08 Technology Level available on
Fix Central, but systems installed prior to the availability of Service Pack 1 should install 6100-08-01. To determine if the 6100-08-01 Service Pack is installed, run the command:
    oslevel -s
The output should indicate "6100-08-01-1245".

Free Space Requirements for devices.common.IBM.sni

The devices.common.IBM.sni product requires that 256MB of available, free partitions in the rootvg volume group. If this space has not been previously allocated for the sni product, a test will be performed at installation time to make sure that this space is available, and if it is not, the installation will fail. If installation is successful, a test will be performed when the lpar is restarted after installation to determine if the sni hardware is available, and if it is, a 256MB logical volume will be allocated as /var/adm/sni.

Integrated Ethernet Software Requirements

The ethernet adapter on the integrated multifunction card of the Power 7 model 770 and 780 systems requires the following minimum software levels for AIX 6.1.
Service PackAdditional APAR
6100-05-07IV11845
6100-06-06IV11842
6100-07-00None

6100-07 Installation

When applying the 6100-07 Technology Level on systems below the 6100-06 level, you will have to run 'smitty update_all' a second time to update Java6.sdk. Until this is done, the 'oslevel' command will not indicate the correct level.

6100-07 Service Pack 1

The 6100-07-01 Service Pack is considered highly recommended. 6100-07-01 is included in the 6100-07 Technology Level available on
Fix Central, but systems installed prior to the availability of Service Pack 1 should install 6100-07-01. To determine if the 6100-07-01 Service Pack is installed, run the command:
    oslevel -s
The output should indicate "6100-07-01-1141".

6100-07 Update When Using Cluster Aware AIX

Due to enhancements in Cluster Aware AIX (CAA) in the 6100-07 and 7100-01 Technology Levels, CAA must be stopped prior to the update. Attempting to update to 6100-07 or 7100-01 with an active cluster will fail. Note that PowerHA SystemMirror 7.1 exploits CAA. This procedure must be used when performing one of the following with a CAA cluster deployed.
  • Updating from AIX 6.1 6100-06 to 6100-07
  • Updating from AIX 7.1 7100-00 to 7100-01
  • Migrating from AIX 6.1 6100-06 to AIX 7.1 7100-01

Update Procedure for PowerHA SystemMirror 7.1

  1. Backup the cluster configuration using the PowerHA SystemMirror snapshot facility.
  2. Stop the PowerHA Cluster Services through SMIT. Stop cluster services on all nodes in the cluster.
  3. Remove the CAA cluster using the command 'rmcluster -n ClusterName'.
  4. Update the AIX Technology Level and reboot the cluster nodes.
  5. Verify and Synchronize the CAA cluster configuration from PowerHA SystemMirror.
  6. Restart the PowerHA SystemMirror 7.1 Cluster.

Update Procedure for Cluster Aware AIX

  1. Save the CAA cluster configuration.
  2. Remove the CAA cluster using the command 'rmcluster -n ClusterName'.
  3. Update the AIX Technology Level and reboot the cluster nodes.
  4. Redeploy the CAA cluster using the mkcluster command.
Refer to the AIX_TL_update_information_for_PowerHA_7.1.pdf document at the following location for additional information.

Missing fixdata in 6100-05-03 and 6100-03-07 Service Packs

The 6100-05-03 and 6100-03-07 Service Packs are missing fixdata, causing 'oslevel -s' to output the incorrect service pack level. These problems are resolved by the following APARs.

6100-06 Service Pack 1

The 6100-06-01 Service Pack is considered highly recommended. 6100-06-01 is included in the 6100-06 Technology Level available on
Fix Central, but systems installed prior to the availability of Service Pack 1 should install 6100-06-01. To determine if the 6100-06-01 Service Pack is installed, run the command:
    oslevel -s
The output should indicate "6100-06-01-1043".

System hang after update to 6100-03

Systems with Encypted File System (EFS) support enabled may fail to boot after updating to the 6100-03 Technology Level. This problem occurs if the clic.rte.kernext and clic.rte.lib filesets are installed at a level below 4.6.0.0. EFS may be enabled if the 'efsenable' command was ever executed on the system, even if there are no encrypted file systems currently in use. The presence of the file /var/efs/efsenabled indicates that EFS is enabled.
To avoid this issue, update the clic.rte.kernext and clic.rte.lib filesets from the clic.rte image available by ordering APAR IZ56564, or from the AIX Expansion Pack media dated 5/2009 or later. The clic.rte filesets need to be updated before the system is rebooted after updating to the 6100-03 Technology Level.
If the 6100-03 Technology Level is installed without updating the clic.rte filesets to the 4.6.0.x level, the system may fail to reboot. Should this occur, you can recover using the following procedure.
  1. Reboot system into service mode
  2. Select 'Task Selection', then chose 'Shell Prompt'
  3. Move file /var/efs/efsenabled to /var/efs/efsenabled.SAVE
  4. Reboot system into normal mode
  5. Update clic.rte.kernext and clic.rte.lib to the 4.6.0.x level
  6. Move back /var/efs/efsenabled from /var/efs/efsenabled.SAVE
This problem is resolved by including the clic.rte package in the 6100-03-00-0920 Technology Level available for download as of 8/7/2009. The Update media labeled LCD8-0840-07 (CD-ROM) or LCD8-0912-06 (DVD), dated 9/2009, also includes this fix.

Port 80 in Use by Web-Based System Manager

Starting with the 6100-02 Technology Level, port 80 is used by default by the Web-Based System Manager web server. This will cause other applications, such as WebSphere's IBM HTTP Server, that use port 80 to fail to start. The Web-Based System Manager web server can be disabled using the following commands.
    stopsrc -s http4websm
    rmitab webserverstart
This issue will be resolved by the following APARs.
Technology
Level
APAR
6100-02IZ49158
6100-03IZ47722

Feature 4758 Cryptographic Adapter not supported

The 4758 Cryptographic Adapter is not supported on AIX 6.1. After migration from AIX 5.3 or earlier to AIX 6.1, the bos.pkcs11 fileset will not be migrated. There is no AIX 6.1 version of bos.pkcs11, and the 5.3 or earlier level of bos.pkcs11 is not supported on AIX 6.1 and therefore should be deinstalled.

Creating a NIM SPOT on 6100-00-02 can cause system damage

Attempting to create a NIM SPOT on the 6100-00-02 Service Pack can fail with an error similar to:
warning: 0042-175 c_instspot: An unexpected result was returned by the
         "/usr/sbin/unmount" command:
unmount: 0506-349 Cannot unmount /dev: The requested resource is busy.
The failure causes loss of some ODM information which leaves the NIM master in an unpredictable state that will manifest itself on the next system reboot. Some volume group information is lost, which may be recovered by running the command:
    importvg rootvg -y hdisk0
Additionally, network configuration can be lost and must be restored manually.
Do not create a NIM SPOT until the fix for APAR IZ14395 is installed.
This problem is fixed in the 6100-00-03 Service Pack.

APAR IZ09375

In addition to 6100-00 Service Pack 1, APAR IZ09375 should be installed on all POWER 6 based systems running AIX 6.1. APAR IZ09375 fixes a problem in AIX 6.1 that can result in various symptoms, including a system crash or data errors under heavy paging activity. 
APAR IZ09375 will be available as part of 6100-00 Service Pack 2. Prior to Service Pack 2 availability, the fix can be obtained from:ftp://ftp.software.ibm.com/aix/efixes/iz09375

6100-00 Service Pack 1

Service Pack 1 is considered a highly recommended update for customers installing AIX 6.1. To determine if you already have Service Pack 1 installed, use the command:
    oslevel -s
The output should include "6100-00-01". Service Pack 1 can be obtained from the Fix Central web site.