AIX - Software Maintenance

AIX - Software Maintenance

In this software Installation we can do following things

1. Fileset installation
2. Applying the Fileset
3. Reject the Installed Fileset
4. Commit the applied Fileset
5. Remove the Committed Fileset
6. Verify the Installed Fileset ( whether properly installed or not)
7. Clear the braked Fileset installation
8. List the Installed Fileset
9. List the Installed Fileset contents ( What are the files available in File Set)
10. To finding the file, Extracted from which Fileset.
11. To find the command file path
12. To view the fileset history
1. Fileset Installation

The Filesets has additional support files for AIX. Suppose we need additional files, we can install it from Base operating system (bos) CD, Bonus software CD, Expansion Software CD, LPP’s (Licensed program product) CD.

NOTE:

If we want to check the AIX Server performance, we need topas command, this topas command is not by default come under AIX OS installation, So that topas file we have to install from some other CD’s.

First we should mount the CDROM drive in server.

When we mount the CDROM, we required mount point like one directory.

So we should create the directory. Eg. /cdmnt (Directory name will be whatever may be)

#mkdir mount point
#mkdir cdmnt

Then u can mount the CDROM drive in cdmnt directory

#mount –v cdrfs –o ro /dev/cd0 /mount point
#mount –v cdrfs –o ro /dev/cd0 /cdmnt

Verify the Cdrom drive mount process.

#mount

The above command will show the cdrom drive mount point i.e. /cdmnt and cdrfs file system.

Then u go to /cdmnt folder

#cd cdmnt
#pwd
cdmnt
#

Now topas file will be under bos.permaget.tool file set. Now we should find the file set in software cd.

To find the file set using command #installp –Ld /dev/cd0 | grep bos.perf*

If the above bos.permaget.tool file set is available in cd, it will show otherwise it won’t show.

If file set is found then we have to install the fileset.

To install the file set using #installp –agxpd /dev/cd0 file set (for preview installation)
i.e. #installp –agxd /dev/cd0 bos.permaget.tool (for installation)

see difference
Options –agxpd for preview

-agxd for installation

When we type the above command, fileset installation will start and topas file extracted from the fileset and it will store to /usr/sbin/topas (super user or administrator commands stored in /usr/sbin)

After that we can use topas performance monitor utility.

#topas

The above command will display the system performance status, like cpu usage, memory usage, io usage (hdisk’s) network usage, paging space usage .


2. Applying Fileset (It is not permanent installation)

This applying Fileset is just file set installation, i.e. When we updating the old fileset to new fileset, by default installation has done in apply state and old fileset configuration is moved to /usr/lpp/package(fileset) path, After commit the fileset this /usr/lpp/package(fileset) will updated . Because if any problems in updated fileset, we can easily reject(remove) the updated fileset, If updated fileset is working properly we can commit the updated fileset, i.e. permanent installation, this case we cannot reject the fileset, but we can uninstall the fileset.

NOTE:

Suppose we are installing fileset for the first time, this first time installation done in commit state, and When we updating the fileset, installation has done in apply state, suppose that updated fileset giving some problem, we can easily reject the updated fileset.





For fileset installation using command installp

Suppose u want to install one fileset bos.rte

Preview (It will show the installation preview – This preview is not required our testing purpose we are using)
#installp –agxpd /dev/cd0 fileset name

#installp –agxpd /dev/cd0 bos.rte (for preview)

Flags:

a - Apply
g – Perquisites (supporting file)
x – Expand the space
p – Preview
d – Specify the device

After preview we can install the Fileset

#installp –agxd /dev/cd0 bos.rte

After Installation, old installed fileset configuration is moved to /usr//lpp/package (fileset), and updated fileset also available in /usr/lpp/package.


3. Reject the Applied Fileset.

After updating the fileset that installed fileset is moved to applied state, suppose that updated filesets are not working properly then we have to remove the updated fileset and pervious filesets are to be configuring to current working condition.

Now we will reject the updated fileset using

#installp –rgp fileset name (For preview)

#installp –rg fileset name

#installp –rg bos.rte.command

The above command is used to remove the updated fileset and automatically previous fileset is moved to working condition.



4. Commit the Applied fileset (It is permanent installation)

After updating the fileset that installed fileset is moved to applied state, suppose applied filesets are working properly, then we have to commit the updated fileset for permanent installation. Otherwise we will remove the updated fileset using reject option.

Now we will commit the updated fileset using

#installp –cgx fileset name

#installp –cgx bos.rte.command

The above command is used to commit the updated fileset.



5. Remove the Committed fileset (It is complete fileset removal not reject)

After commit the fileset, that fileset is moved to permanent installation, we cannot reject, we have to uninstall the fileset only.

Now we will uninstall the committed fileset using:

#installp –ugp Fileset name

#installp –ugp bos.rte.command (For preview)

#installp –ug bos.rte.command (For un installation)

The above command is used to uninstall the committed fileset.



6. Verify the installed fileset (Whether properly installed or not)

After installation we want to verify whether filesets are properly installed are not, using following command

#lppchk –v fileset name

#lppchk –v bos.rte.command

If any problems it will show the error message otherwise no error message, then we will confirm fileset is installed properly.


7. Clear the braked fileset installation:

While installing the filesets, some problem has occurred, i.e. Cd is not reading or space is not available in volume group.

This time half of the fileset only installed in server, so before next installation, we should remove the half installed fileset using

#Installp –C

To clean the brokend fileset installation


8. List the installed fileset

Suppose we want to see what are the filesets available in Server, Using

#lslpp –l

The above command list the all installed fileset

Suppose you want to find particular fileset in server, Using

#lslpp –l | grep Fileset name

The above command is list the specified fileset only, if file set is not available or not installed, it won’t show anything.


9. List the installed fileset content (What are the files available in fileset)

Suppose we want to see bos.rte.command fileset content, using

#lslpp –f fileset name

#lslpp –f bos.rte.command

The above command is display the bos.rte.command file

Basically bos.rte.command fileset has user commands So it will show the /usr/bin/ls,






10. To find the file, which is installed from which fileset.

Using # lslpp –w /usr/bin/ls

The above command shows the bos.rte.command fileset name, because ls command file is extracted from bos.rte.command fileset

#lslpp –w /usr/sbin/savevg

The above shows some other fileset name.

11. To find the command file path:

Using #which ls

The above command is display the given file path i.e. the above command output will be

/usr/bin/ls

Suppose we want to savevg path, using

#which savevg

/usr/sbin/savevg

12. To view the fileset history:

Using #lslpp –h fileset name

#lslpp –h bos.rte.command

The above command output will be full details of fileset.

Fileset version, installed time, date

NOTE:
Whenever we installing the fileset or fix, in that current path should have the .toc (table of content) file. If this file is not available we cannot install any fileset or fix from that current path

If file is not available, we can create the .toc file using command
#inutoc .

The above command is used to create the .toc file

AIX - SRC AND DAEMONS

AIX - SRC AND DAEMONS

Group of subsystems called daemons, using src command we can perform the following activities with subsystems and daemons.

1. Start daemon
2. List daemon
3. stop daemon
4. refresh (restart) daemon

AIX - Start daemon

Using srcstart command we can start the daemon.Actually nfs daemon has 5 subsystems (services), when you configured nfs in server, that 5 subsystems are always start state. If one subsystem is not working properly or not started, then the nfs is not working properly.

Nfs daemons:

Server: 1. nfsd, 2. rpc.mountd

Client: 3.biod, 4. rpc.statd, 5. rpc.lockd

If clients are not able to access the network file system from the server, that time you have to start or restart the nfsd and rpc.mountd subsystems.

#startsrc –s nfsd
#startsrc –s rpc.mountd

Flag –s is mentioned for single, because we are starting single nfsd subsystem

If we want start complete nfs daemons using –g flag

#startsrc –g nfs
Nfsd – started
Biod – started
Rpc.mountd – started
Rpc.lockd – started
Rpc.statd – started
#
The above command is start all the 5 subsystems at the same time, but it is not recommended.
Like this we start single subsystem or group of subsystems.
AIX - Stop Daemon

Using stopsrc command we can stop the daemon,If we want to stop nfs daemon, then your command will be

#stopsrc –g nfs

Nfsd – stopped
Biod – stopped
Rpc.mountd – stopped
Rpc.lockd – stopped
Rpc.statd – stopped

Or we can stop single subsystem using –s flag

#stopsrc –s nfsd
Nfsd – stopped, like this we can stop the daemons

AIX - Paging Space

AIX - Paging Space

Paging space is configured for additional system performance i.e. if real memory is occupies some system process, that time further process is moved to paging space temporary. If real memory getting free, then process moved from paging space to real memory. This paging space is assigned from HDD space.

Normally paging space is configured double size of the real memory

If system has 512MB RAM, then paging space size will be 1BG.



In this paging space we can perform following:

1. List paging space
2. Create paging space
3. Change paging space
4. Activate paging space
5. Deactivate paging space
6. Remove paging space


AIX - Paging - Activate Paging Space

Using swapon command we can activate the paging space

#swapon /dev/newpaging (To activate paging space)

The paging space is active stage, and then one entry will be available in /etc/swapspaces file.


5. Deactivate paging space

Using swapoff command we can deactivate the paging space

#swapoff /dev/newpaging (To deactivate paging space)

The paging space is deactivated, then that paging space entry is removed from /etc/swapspaces

AIX - Paging - Change Paging Space

Using chps command we can increase, decrease, auto on, and auto off the paging space.

#chps –s 5 newpaging (5 LP’s added with newpaging) – To increase

#chps –d 3 newpaging (3 LP’s reduced from newpaging – To Decrease

#chps -ay newpaging (To start the paging space at startup)

#chps –an newpaging (To stop the paging space automatic startup at boot)
Using mkps command we can create the paging space

AIX - Paging - Create Paging Space

#mkps –s 10 rootvg hdisk2
s- size
10 – no of PP size
Rootvg – specifying the VG
Hdisk2 – paging space created in PV hdisk2 (PP allocated from hdisk2)

Then paging space has created and name will be paging00

Using mklv command we can create paging space using different name.
Generally paging name will be paging00, paging01, paging02,

#mklv –y newpaging –t paging rootvg 10 hdisk2
#lsps –a
Paging00 , newpaging
AIX - Paging - Deactivate Paging Space

Using swapoff command we can deactivate the paging space

#swapon /dev/newpaging (To deactivate paging space)

The paging space is deactivated, then that paging space entry is removed from /etc/swapspaces

AIX - Paging - List Paging Space

Using lsps command we can view the paging space details

#lsps –a (It will display the paging space details)

AIX - Paging - Remove Paging Space

Using rmps command we can remove the paging space

#rmps newpaging

NOTE:

Before removing the paging space, paging space should be offline, then we can remove the paging space, otherwise it will giv

AIX - ODM (Object Data Manager)

AIX - ODM (Object Data Manager)


ODM generally used for Maintain the System information, whenever we install and reconfigure the hardware device and software, that information’s added in ODM,

Mainly ODM used to store the following information’s

1. Hardware vital product data information
2. Software information
3. Smit menu
4. Nim
5. TCPIP
6. Error log
1. Device configuration information
2. Display information for SMIT (menus, selectors, and dialogs)
3. Vital product data for installation and update procedures
4. Communications configuration information
5. System resource information.
But we cannot see any ODM information in file format, but this ODM information available in many files and different path,
1. /etc/objrepos
2. /usr/lib/objrepos
3. /usr/share/lib/objrepos
NOTE:
For ODM some variable entry available in /etc/environment file.U should not edit anything in this file. If you want to check the variable entry using command
#env
The above command is used to display the variable details
ODM has two kinds of Databases
1. Pre-defined (PdDv)
2. Customized(CuDv)


1. Predefined
In this predefined Database has one Object class (PdDv) and Objects i.e.
PdDv (Object class)
Object: (Physical Information’s)
Floppy drive
Harddisk drive
Mouse like this all the devices.


2. Customized

In this customized Database has one Object class (CuDv) and Objects i.e.

CuDv (Object class)

Objects: (Installed configuration information’s)

Hdisk0
Hdisk1
Fd0
Sa0
Rmt0
Cd0
Scsi0 like this all the devices

ODM States:

ODM has two states

1. Available (device status will be 1)
2. Defined (device status will be 0)

Suppose we are installing one HDD in server. That device is currently accessible this is available state.

Suppose that installed device is removed from server or hdd has failed. This case that device is moved to defined state. i.e. that device configuration is available in server but currently device is not accessible.

We can see this available and defined device status using odmget command

#odmget CuDv ( It will display the customized device status)




ODM Commands:

For object:

1. odmadd
2. odmchange
3. odmget
4. odmdelete

1. odmadd

The above command is used to manually we can add object information in ODM

2. odmchange

The above command is used to change configurations for object in ODM
3. odmget

#odmget –q name=hdisk0 CuDv

The above command is used to display the specified object hdisk0 information in ODM

We can see this available and defined device status using odmget command

#odmget CuDv (It will display the customized device status)


4. odmdelete

#odmdelete –o CuDv –q name=hdisk0
The above command is used to delete the specified object hdisk0 in ODMe

For object class:

1. odmcreate
2. odmdrop
3. odmshow




1. odmcreate

The above command is used to manually we can add object class (PdDv) in ODM

2. odmdrop

The above command is used to stop the specified Object class in ODM
3. odmshow
The above command is used to display the specified object class in ODM

NOTE:
You should not add, delete, change, for object and object class in ODM
(100 % we don’t have permissions for ODM)

AIX - ODM Management



ODM information is stored in the directories /etc/objrepos (default ODM directory, $ODMDIR env variable), /usr/lib/objrepos and /usr/share/lib/objrepos.

odmadd
Adds objects to created object classes.

odmchange
Changes the contents of a selected object in the specified object class.

odmcreate
Produces the .c (source) and .h (include) files necessary for ODM application development and creates empty object classes.

odmdelete
Deletes selected objects from a specified object class.

odmdrop
Removes an object class.

odmget
Retrieves objects from the specified object classes and places them into an odmadd input file.

odmshow
Displays an object class definition.
odmshow CuDv : displays the object class definition for the Customized Device Database.

NFS

NFS

AIX - Network - Assign IP address to NIC

Assign IP address to NIC

Using mktcpip or smitty tcpip command we can assign IP address to NIC

#smitty tcpip

It will ask following details

Hostname –
IP address –
Subnet mask –
DNS server –
DNS Server name –
Gateway –
#
Or
#ifconfig en0 inet 192.168.1.1 netmask 255.255.255.0 up (Configure en0 starts Immediately)


After finishing this process ip address assigned to NIC

Or

We can assign one more IP address to same network card

#ifconfig en0 192.168.1.33 alias (adding alias IP to en0)
#ifconfig –en0 192 168.1.33 –alias (Removing alias IP from en0)
AIX - Network - Check status of the NIC
AIX - Network - Check status of the NIC

Using ifconfig command we can check the status of the NIC
#ifconfig –a (To show status of all network interfaces for IP)
AIX - Network - Check the Network Statistics
AIX - Network - Check the Network Statistics


Using netstat command we can check the network statistics

#netstat –a (To show the state of all sockets)

#netstat –c (To show the network buffers cache)

#netstat –D (To show the net drops of packets)

#netstat –i (To display interface statistics)

#netstat –rn (To show routing table – ip will be given instead of host names)

#netstat –s (To show statistics of the protocols)


AIX - Network - Clear Gateway

Using route command we can clear the gateway

#route –f (To clear the gateway)

AIX - Network - Disable IP address

Using ifconfig command we can disable the IP address
#ifconfig en0 down (Turns off network card en0)
Enable IP address

#ifconfig en0 up (Turns on network card en0)

AIX - Network - NIC Configuration

Using cfgmrg command we can configure to server

Cfgmgr command is device management related command. If we want to install new network card, first we should connect network in server. While booting the server boot process is starting the cfgmrg, so that time network card will be installed in server.

We have installed Ethernet card 1.

That Ethernet card is divided into three parts

Ent0 – Physical adapter
En0 – Logical name
Et0 – Architecture of the card (802.3)

Or

Using mkinet command we can install the NIC

AIX - Network - Remove IP Address

Using ifconfig command we can remove network interface from network list
#ifconfig en0 detach (Removes en0 card from the network interface list)




AIX - Network - Trace Host

Using traceroute command we can trace the route to the host

#traceroute ServerA (To trace the route to ServerA)


AIX - Network -Changing the HOST Name

Using hostname command we can change the host name

#hostname (It will display existing host name)
ServerA

#hostname ServerB

#hostname ServerB

ServerA host name is changed to ServerB
Or
Using chdev command to change the hostname for inet0

#chdev –l inet0 –a hostname=ServerB


AIX - Network -Check the NIC device status

#entstat en0 (To display the status of Ethernet device en0)
#entstat –d en0 (To display detailed information about Ethernet device en0)
Posted by Pa
AIX - NFS (Network File System)

In this NFS we can share the file systems from server to clients, across the network, using some nfs related commands we can configure the NFS in server end and client end. So clients can access the server mounted files systems through network, with specified permissions, such as read only, read write.

NOTE:
Before configuring the NFS, U should check the /etc/hosts file entry, and NFS daemons and all the nfs daemon subsystems.

--- nfsd, rpc.mountd subsystems is running on server end.

--- rpc.statd, rpc.lockd, rpc.mountd subsystems running on client end


AIX - NFS - Changing Exported File System Permissions

Using chnfsexp command we can change the permission for exported file systems

#smitty chnfsexp

One smitty screen will appear, and then you specify the permission and configure the file system.
AIX - NFS - Client End

Then we should import the file system to client end.

5. Import
6. List the imported file systems
7. Remove the imported file systems



AIX - NFS - Export file system

Using smitty mknfsexp command we can export the file system

#smitty mknfsexp

One window will appear then you have to put the configuration

a. File system name (what file system you want to export i.e. /usr)

b. Specify the target host name (destination host name i.e. serverB)
If this box is empty, then this exporting file system is allowed to all the clients.

c. Specify the access permissions (Read only, Read write)

d. Host allowed root access (Source Server name, ServerA)

e. Then press enter button to execute the above configuration finally result will be OK, if you got OK output then your configurations are exported properly. If you got FAILED error message then you have to check the configuration.

NOTE: After exporting the file system, that exported file system will be added in /etc/exports, and /etc/xtab because while boot process, server is checking exported file systems from above files, if entry found in the above file then file system are exporting on boot process.
AIX - NFS - Import File Systems

Using smitty mknfsmnt command we can import the file system.

#smitty mknfsmnt

One smitty configuration screen will appear then you have to specify the following things.

a. specify the path name of mount point (client end mount point by default /mnt will be available or we have to create directory and we can specify that directory name to mount the file system /mnt)

b. Specify the path name of remote directory (This is server file system name i.e. /usr, which file system was exported by server)

c. Host where remote directory resides (This is exporting source server name i.e. ServerA)

d. Mount type name (Specify the file system type jfs2)

e. Then press enter button to execute the above configuration, finally it will exported


AIX - NFS - List Exported File System

Using lsnfsexp command we can list the exported file system

#lsnfsexp

/usr – rw

AIX - NFS - List Imported Files System

Using mount command we can check the imported file system status.

#mount

Above command is used to display the mounted file system information along with file system type, if nfs mounted file system available then it will show file system type name nfs so easily we can identify which nfs mounted file system.



AIX - NFS - Remove Exported File System

Using smitty rmnfsexp command we can remove the exported file system

#smitty rmnfsexp

One smitty screen will appear, and then you press Esc+4, it will check and shows what are the file systems are exported. Then you can remove the exported file system.

After this command execution, list the exported file system using command lsnfsexp, now removed file systems is not listed.

NOTE: while removing the file system, the file system entry removed from /etc/exports and /etc/xtab

AIX - NFS - Remove Imported File System

Using smitty rmnfsmnt command we can remove the mounted file system from client end.
#smitty rmnfsmnt

Then one smitty screen will appear. Specify the mounted file system name or press Esc+4 keys, for system analyze and show the mount file system, then you mention from the list also. After this command execution, which mounted file system will be removed from the client PC. Now you put mount command, that removed file system is not listed.

Like this we can export and import the network file system.

AIX - NFS - Server End Activities

Server End

First we should export the file system from server end.

1. Export
2. List the exported file systems
3. Remove the exported file systems
4. Change the exported file system permissions

AIX - Installation

AIX - Installation - Migration

This method upgrades from earlier versions of the AIX BOS to AIX 6.1 (see the release notes for restrictions). The migration installation method is used to upgrade from an existing version or release of AIX to a later version or release of AIX. A migration installation preserves most file systems, including the root volume group, logical volumes, and system configuration files. It overwrites the /tmp file system.
AIX - Installation - New and Complete Overwrite

This method installs AIX 6.1 on a new machine or completely overwrites any BOS version that exists on your system.

AIX - Installation - Perform a new and complete overwrite BOS installation from CD

Step 1. Prepare your system

-There must be adequate disk space and memory available. AIX 5L Version 5.2 and AIX 5L Version 5.3 require 128MB of memory and 2.2GB of physical disk space.

-Make sure your hardware installation is complete, including all external devices.

Step 2. Boot from the AIX product CD

-Insert the AIX Volume 1 CD into the CD-ROM device.

-Make sure all external devices attached to the system, such as CD-ROM drives, tape drives, DVD drives, and terminals, are turned on. Only the CD-ROM drive from which you will install AIX should contain the installation media.

-Power on the system.

-When the system beeps twice, press F5 on the keyboard or 5 on an ASCII terminal. If you have a graphics display, you will see the keyboard icon on the screen when the beeps occur. If you have an ASCII terminal, you will see the word keyboard when the beeps occur.
-Select the system console by pressing F1 or 1 on an ASCII terminal and press Enter.

-Select the English language for the BOS installation menus by typing a 1 in the Choice field. Press Enter to open the Welcome to Base Operating System Installation and Maintenance screen.

-Type 2 to select 2 Change/Show Installation Settings and Install in the Choice field and press Enter.


Welcome to Base Operating System
Installation and Maintenance


Type the number of your choice and press Enter. Choice is
indicated by >>>.

1 Start Install Now with Default Settings

2 Change/Show Installation Settings and Install

3 Start Maintenance Mode for System Recovery

88 Help ?
99 Previous Menu
>>> Choice [1]: 2

Step 3. Set and verify BOS installation settings


-In the Installation and Settings screen, verify that the installation settings are correct by checking the method of installation (new and complete overwrite), the disk or disks you want to install, the primary language environment settings, and the advanced options.

If the default choices are correct, type 0 and press Enter to begin the BOS installation. The system automatically reboots after installation is complete.

-Go to Step 4. Configure the system after installation.

Otherwise, go to sub-step 2.
To change the System Settings, which includes the method of installation and disk where you want to install, type 1 in the Choice field and press Enter.


Installation and Settings


Either type 0 and press Enter to install with current settings,
or type the number of the setting you want to change
and press Enter.


1 System Settings:
Method of Installation..................New and Complete Overwrite
Disk Where You Want to Install..hdisk0


>>> Choice [0]: 1


Type 1 for New and Complete Overwrite in the Choice field and press Enter. The Change Disk(s) Where You Want to Install screen now displays.


Change Disk(s) Where You Want to Install


Type one or more numbers for the disk(s) to be used for
installation and press Enter. To cancel a choice, type the
corresponding number and Press Enter. At least one
bootable disk must be selected. The current choice is
indicated by >>>.

Name Location Code Size(MB) VG Status Bootable

1 hdisk0 04-B0-00-2,0 4296 None Yes
2 hdisk1 04-B0-00-5,0 4296 None Yes
3 hdisk2 04-B0-00-6,0 12288 None Yes

>>> 0 Continue with choices indicated above

66 Disks not known to Base Operating System Installation
77 Display More Disk Information
88 Help ?
99 Previous Menu

>>> Choice [0]:

In the Change Disk(s) Where You Want to Install screen:

-Select hdisk0 by typing a 1 in the Choice field and press Enter. The disk will now be selected as indicated by >>>. To unselect the destination disk, type the number again and press Enter.

-To finish selecting disks, type a 0 in the Choice field and press Enter. The Installation and Settings screen now displays with the selected disks listed under System Settings.

-Change the Primary Language Environment Settings to English (United States). Use the following steps to change the Cultural Convention, Language, and Keyboard to English.

-Type 2 in the Choice field on the Installation and Settings screen to select the Primary Language Environment Settings option.

-Type the number corresponding to English (United States) as the Cultural Convention in the Choice field and press Enter.

-Select the appropriate keyboard and language options.

-Verify that the selections are correct in the Overwrite Installation Summary screen, as follows


Overwrite Installation Summary

Disks: hdisk0
Cultural Convention: en_US
Language: en_US
Keyboard: en_US
64 Bit Kernel Enabled: No
JFS2 File Systems Created: No
Desktop: CDE
Enable System Backups to install any system: Yes

Optional Software being installed:

>>> 1 Continue with Install
88 Help ?
99 Previous Menu

>>> Choice [1]:

-Press Enter to begin the BOS installation. The system automatically reboots after installation is complete


Step 4. Configure the system after installation

-After a new and complete overwrite installation, the Configuration Assistant opens on systems with a graphics display. On systems with an ASCII display, the Installation Assistant opens.

-Select the Accept Licenses option to accept the electronic licenses for the operating system.

-Set the date and time, set the password for the administrator (root user), and configure the network communications (TCP/IP).

Use any other options at this time. You can return to the Configuration Assistant or the Installation Assistant by typing configassist or smitty assist at the command line.

-Select Exit the Configuration Assistant and select Next. Or, press F10 or ESC+0 to exit the Installation Assistant.

-If you are in the Configuration Assistant, select Finish now. Do not start the Configuration Assistant when restarting AIX and select Finish.


At this point, the BOS Installation is complete, and the initial configuration of the system is complete.

AIX - Installation - Preservation

This method replaces an earlier version of the BOS but retains the root volume group, the user-created logical volumes, and the /home file system. The system file systems /usr, /var, /tmp, /opt, and / (root) are overwritten. Product (application) files and configuration data stored in these file systems will be lost. Information stored in other non-system file systems will be preserved.

AIX - Installing the Base Operating System

There are multiple ways to install the AIX® base operating system.

The Base Operating System (BOS) installation program first restores the run-time bos image, then installs the appropriate filesets, depending on your selections. The installation program automatically installs required message filesets, according to the language you choose

If you are reinstalling on an older system, the DVD media can only be used to boot or reinstall on 64-bit systems. To determine if your system is a 32-bit system or a 64-bit system, run the prtconf command with the -c flag.

The following installation methods are available on AIX:

New and Complete Overwrite

Preservation

Migration

HMC Tips IX - System Plan

HMC Tips IX - System Plan

1. How to make a system plan from a running machine ?

# mksysplan -f marc.sysplan -m Machine-Name -v

where
marc.sysplan is the file name.

2. How to list a system plan ?

# lssysplan

3. How to delete a particular system plan ?

# rmsysplan

4. How to reploy a system plan on a managed server ?

# deploysysplan

5. How to copy a system plan from/into the HMC ?

# cpsysplan

HACMP BEST PRACTICES

HACMP
HIGH AVAILABILITY CLUSTER MULTIPROCESSING
BEST PRACTICES


Table of Contents
I. Overview1

II. Designing High Availability1

Risk Analysis2

III. Cluster Components3

Nodes3

Networks3

Adapters5

Applications5

IV. Testing9

V. Maintenance 9

Upgrading the Cluster Environment10

VI. Monitoring12

VII. HACMP in a Virtualized World13

Maintenance of the VIOS partition – Applying Updates18

VIII.Summary19

IX. References 21

X. About the Authors 21








WHITE PAPER

Overview
IBM High Availability Cluster Multiprocessing (HACMP TM) product was first shipped in 1991 and is now
in its 14th release, with over 60,000 HACMP clusters in production world wide. It is generally recognized
as a robust, mature high availability product. HACMP supports a wide variety of configurations, and provides
the cluster administrator with a great deal of flexibility. With this flexibility comes the responsibility
to make wise choices: there are many cluster configurations that are workable in the sense that the cluster
will pass verification and come on line, but which are not optimum in terms of providing availability. This
document discusses the choices that the cluster designer can make, and suggests the alternatives that make
for the highest level of availability*.


Designing High Availability

“…A fundamental design goal of (successful) cluster design is the elimination of single points of failure (SPOFs)…”
A High Availability Solution helps ensure that the failure of any component of the solution, be it hardware,
software, or system management, does not cause the application and its data to be inaccessible to the user
community. This is achieved through the elimination or masking of both planned and unplanned downtime.
High availability solutions should eliminate single points of failure (SPOF) through appropriate design,
planning, selection of hardware, configuration of software, and carefully controlled change management
discipline.
While the principle of "no single point of failure" is generally accepted, it is sometimes deliberately or inadvertently
violated. It is inadvertently violated when the cluster designer does not appreciate the consequences
of the failure of a specific component. It is deliberately violated when the cluster designer chooses
not to put redundant hardware in the cluster. The most common instance of this is when cluster nodes are
chosen that do not have enough I/O slots to support redundant adapters. This choice is often made to reduce
the price of a cluster, and is generally a false economy: the resulting cluster is still more expensive
than a single node, but has no better availability.
A cluster should be carefully planned so that every cluster element has a backup (some would say two of
everything!). Best practice is that either the paper or on-line planning worksheets be used to do this planning,
and saved as part of the on-going documentation of the system. Fig 1.0 provides a list of typical
SPOFs within a cluster.
“….cluster design decisions should be based on whether they contribute to availability (that is, eliminate a SPOF) or
detract from availability (gratuitously complex) …”
* This document applies to HACMP running under AIX®, although general best practice concepts are also applicable to HACMP
running under Linux®.

Fig 1.0 Eliminating SPOFs



Risk Analysis

Sometimes however, in reality it is just not feasible to truly eliminate all SPOFs within a cluster. Examples,
may include : Network ¹, Site ². Risk analysis techniques should be used to determine those which simply
must be dealt with as well as those which can be tolerated. One should :
Study the current environment. An example would be that the server room is on a properly sized
UPS but there is no disk mirroring today.
Perform requirements analysis. How much availability is required and what is the acceptable likelihood
of a long outage.
Hypothesize all possible vulnerabilities. What could go wrong?
Identify and quantify risks. Estimate the cost of a failure versus the probability that it occurs.
Evaluate counter measures. What does it take to reduce the risk or consequence to an acceptable
level?
Finally, make decisions, create a budget and design the cluster.
1 If the network as a SPOF must be eliminated then the cluster requires at least two networks. Unfortunately, this only eliminates
the network directly connected to the cluster as a SPOF. It is not unusual for the users to be located some number of hops away
from the cluster. Each of these hops involves routers, switches and cabling – each of which typically represents yet another SPOF.
Truly eliminating the network as a SPOF can become a massive undertaking.
2 Eliminating the Site as a SPOF depends on distance and the corporate disaster recovery strategy. Generally, this involves using
HACMP eXtended Distance (XD, previously known as HAGEO). However, if the sites can be covered by a common storage area
network—say, buildings within a 2km radius—then Cross-site LVM mirroring function as described in the HACMP Administration
Guide is most appropriate, providing the best performance at no additional expense. If the sites are within the range of PPRC
(roughly, 100km) and compatible ESS/DS/SVC storage systems are used, then one of the HACMP/XD: PPRC technologies is
appropriate. Otherwise, consider HACMP/XD: GLVM. These topics are beyond the scope of this white paper.


Cluster Components

Here are the recommended practices for important cluster components.

Nodes
HACMP supports clusters of up to 32 nodes, with any combination of active and standby nodes. While it
is possible to have all nodes in the cluster running applications (a configuration referred to as "mutual
takeover"), the most reliable and available clusters have at least one standby node - one node that is normally
not running any applications, but is available to take them over in the event of a failure on an active
node.
Additionally, it is important to pay attention to environmental considerations. Nodes should not have a
common power supply - which may happen if they are placed in a single rack. Similarly, building a cluster
of nodes that are actually logical partitions (LPARs) with a single footprint is useful as a test cluster, but
should not be considered for availability of production applications.
Nodes should be chosen that have sufficient I/O slots to install redundant network and disk adapters.
That is, twice as many slots as would be required for single node operation. This naturally suggests that
processors with small numbers of slots should be avoided. Use of nodes without redundant adapters
should not be considered best practice. Blades are an outstanding example of this. And, just as every cluster
resource should have a backup, the root volume group in each node should be mirrored, or be on a
RAID device.
Nodes should also be chosen so that when the production applications are run at peak load, there are still
sufficient CPU cycles and I/O bandwidth to allow HACMP to operate. The production application
should be carefully benchmarked (preferable) or modeled (if benchmarking is not feasible) and nodes chosen
so that they will not exceed 85% busy, even under the heaviest expected load.
Note that the takeover node should be sized to accommodate all possible workloads: if there is a single
standby backing up multiple primaries, it must be capable of servicing multiple workloads. On hardware
that supports dynamic LPAR operations, HACMP can be configured to allocate processors and memory to
a takeover node before applications are started. However, these resources must actually be available, or
acquirable through Capacity Upgrade on Demand. The worst case situation – e.g., all the applications on
a single node – must be understood and planned for.

Networks
HACMP is a network centric application. HACMP networks not only provide client access to the applications
but are used to detect and diagnose node, network and adapter failures. To do this, HACMP uses
RSCT which sends heartbeats (UDP packets) over ALL defined networks. By gathering heartbeat information
on multiple nodes, HACMP can determine what type of failure has occurred and initiate the appropriate
recovery action. Being able to distinguish between certain failures, for example the failure of a network
and the failure of a node, requires a second network! Although this additional network can be “IP
based” it is possible that the entire IP subsystem could fail within a given node. Therefore, in addition


there should be at least one, ideally two, non-IP networks. Failure to implement a non-IP network can potentially
lead to a Partitioned cluster, sometimes referred to as 'Split Brain' Syndrome. This situation can
occur if the IP network(s) between nodes becomes severed or in some cases congested. Since each node is
in fact, still very alive, HACMP would conclude the other nodes are down and initiate a takeover. After
takeover has occurred the application(s) potentially could be running simultaneously on both nodes. If the
shared disks are also online to both nodes, then the result could lead to data divergence (massive data corruption).
This is a situation which must be avoided at all costs.
The most convenient way of configuring non-IP networks is to use Disk Heartbeating as it removes the
problems of distance with rs232 serial networks. Disk heartbeat networks only require a small disk or
LUN. Be careful not to put application data on these disks. Although, it is possible to do so, you don't want
any conflict with the disk heartbeat mechanism!
Important network best practices for high availability :
Failure detection is only possible if at least two physical adapters per node are in the same physical
network/VLAN. Take extreme care when making subsequence changes to the networks, with regards
to IP addresses, subnetmasks, intelligent switch port settings and VLANs.
Ensure there is at least one non-IP network configured.
Where possible use Etherchannel configuration in conjunction with HACMP to aid availability. This
can be achieved by ensuring the configuration contains a backup adapter which plugs into an alternate
switch. However, note: HACMP see Etherchannel configurations as single adapter networks. To
aid problem determination configure the netmon.cf file to allow ICMP echo requests to be sent to
other interfaces outside of the cluster. See Administration guide for further details.
Each physical adapter in each node needs an IP address in a different subnet using the same subnet
mask unless Heartbeating over IP Aliasing is used.
Currently, there is NO support in HACMP for Virtual IP Addressing (VIPA), IPv6 and IEEE802.3
standard et interfaces.
Ensure you have in place the correct network configuration rules for the cluster with regards IPAT
via Replacement/Aliasing, Etherchannel, H/W Address Take-over (HWAT), Virtual Adapter support,
service and persistent addressing. For more information check the HACMP Planning Guide
documentation.
Name resolution is essential for HACMP. External resolvers are deactivated under certain event
processing conditions. Avoid problems by configuring /etc/netsvc.conf and NSORDER variable in
/etc/environment to ensure the host command checks the local /etc/hosts file first.
Read the release notes stored in : /usr/es/sbin/cluster/release_notes. Look out for new
and enhanced features, such as collocation rules, persistent addressing and Fast failure detection.
Configure persistent IP labels to each node. These IP addresses are available at AIX® boot time and
HACMP will strive to keep them highly available. They are useful for remote administration, monitoring
and secure Node-to-Node communications. Consider implementing a host-to-host IPsec tunnel
between persistent labels between nodes. This will ensure sensitive data such as passwords are
not sent unencrypted across the network. An example: when using C-SPOC option "change a users
password".
If you have several virtual clusters split across frames, ensure boot subnet Addresses are unique per
cluster. This will avoid problems with netmon reporting the network is up when indeed the physical
network outside the cluster maybe down.

Adapters
As stated above, each network defined to HACMP should have at least two adapters per node. While it is
possible to build a cluster with fewer, the reaction to adapter failures is more severe: the resource group
must be moved to another node. AIX provides support for Etherchannel, a facility that can used to aggregate
adapters (increase bandwidth) and provide network resilience. Etherchannel is particularly useful for
fast responses to adapter / switch failures. This must be set up with some care in an HACMP cluster.
When done properly, this provides the highest level of availability against adapter failure. Refer to the IBM
techdocs website: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101785 for further
details.
Many System p TM servers contain built-in Ethernet adapters. If the nodes are physically close together, it
is possible to use the built-in Ethernet adapters on two nodes and a "cross-over" Ethernet cable (sometimes
referred to as a "data transfer" cable) to build an inexpensive Ethernet network between two nodes for
heart beating. Note that this is not a substitute for a non-IP network.
Some adapters provide multiple ports. One port on such an adapter should not be used to back up another
port on that adapter, since the adapter card itself is a common point of failure. The same thing is true
of the built-in Ethernet adapters in most System p servers and currently available blades: the ports have a
common adapter. When the built-in Ethernet adapter can be used, best practice is to provide an additional
adapter in the node, with the two backing up each other.
Be aware of network detection settings for the cluster and consider tuning these values. In HACMP terms,
these are referred to as NIM values. There are four settings per network type which can be used : slow,
normal, fast and custom. With the default setting of normal for a standard Ethernet network, the network
failure detection time would be approximately 20 seconds. With todays switched network technology this
is a large amount of time. By switching to a fast setting the detection time would be reduced by 50% (10
seconds) which in most cases would be more acceptable. Be careful however, when using custom settings,
as setting these values too low can cause false takeovers to occur. These settings can be viewed using a variety
of techniques including : lssrc –ls topsvcs command (from a node which is active) or odmget
HACMPnim |grep –p ether and smitty hacmp.

Applications
The most important part of making an application run well in an HACMP cluster is understanding the
application's requirements. This is particularly important when designing the Resource Group policy behavior
and dependencies. For high availability to be achieved, the application must have the ability to
stop and start cleanly and not explicitly prompt for interactive input. Some applications tend to bond to a
particular OS characteristic such as a uname, serial number or IP address. In most situations, these problems
can be overcome. The vast majority of commercial software products which run under AIX are well
suited to be clustered with HACMP.

Application Data Location
Where should application binaries and configuration data reside? There are many arguments to this discussion.
Generally, keep all the application binaries and data were possible on the shared disk, as it is easy
to forget to update it on all cluster nodes when it changes. This can prevent the application from starting or
working correctly, when it is run on a backup node. However, the correct answer is not fixed. Many application
vendors have suggestions on how to set up the applications in a cluster, but these are recommendations.
Just when it seems to be clear cut as to how to implement an application, someone thinks of a new
set of circumstances. Here are some rules of thumb:
If the application is packaged in LPP format, it is usually installed on the local file systems in rootvg. This
behavior can be overcome, by bffcreate’ing the packages to disk and restoring them with the preview option.
This action will show the install paths, then symbolic links can be created prior to install which point
to the shared storage area. If the application is to be used on multiple nodes with different data or configuration,
then the application and configuration data would probably be on local disks and the data sets on
shared disk with application scripts altering the configuration files during fallover. Also, remember the
HACMP File Collections facility can be used to keep the relevant configuration files in sync across the cluster.
This is particularly useful for applications which are installed locally.

Start/Stop Scripts
Application start scripts should not assume the status of the environment. Intelligent programming should
correct any irregular conditions that may occur. The cluster manager spawns theses scripts off in a separate
job in the background and carries on processing. Some things a start script should do are:
First, check that the application is not currently running! This is especially crucial for v5.4 users as
resource groups can be placed into an unmanaged state (forced down action, in previous versions).
Using the default startup options, HACMP will rerun the application start script which may cause
problems if the application is actually running. A simple and effective solution is to check the state
of the application on startup. If the application is found to be running just simply end the start script
with exit 0.
Verify the environment. Are all the disks, file systems, and IP labels available?
If different commands are to be run on different nodes, store the executing HOSTNAME to variable.
Check the state of the data. Does it require recovery? Always assume the data is in an unknown state
since the conditions that occurred to cause the takeover cannot be assumed.
Are there prerequisite services that must be running? Is it feasible to start all prerequisite services
from within the start script? Is there an inter-resource group dependency or resource group sequencing
that can guarantee the previous resource group has started correctly? HACMP v5.2 and later has
facilities to implement checks on resource group dependencies including collocation rules in
HACMP v5.3.
Finally, when the environment looks right, start the application. If the environment is not correct and
error recovery procedures cannot fix the problem, ensure there are adequate alerts (email, SMS,
SMTP traps etc) sent out via the network to the appropriate support administrators.
Stop scripts are different from start scripts in that most applications have a documented start-up routine
and not necessarily a stop routine. The assumption is once the application is started why stop it? Relying
on a failure of a node to stop an application will be effective, but to use some of the more advanced features
of HACMP the requirement exists to stop an application cleanly. Some of the issues to avoid are:
Be sure to terminate any child or spawned processes that may be using the disk resources. Consider
implementing child resource groups.
Verify that the application is stopped to the point that the file system is free to be unmounted. The
fuser command may be used to verify that the file system is free.
In some cases it may be necessary to double check that the application vendor’s stop script did actually
stop all the processes, and occasionally it may be necessary to forcibly terminate some processes.
Clearly the goal is to return the machine to the state it was in before the application start script was
run.
Failure to exit the stop script with a zero return code as this will stop cluster processing. * Note: This
is not the case with start scripts!
Remember, most vendor stop/starts scripts are not designed to be cluster proof! A useful tip is to have stop
and start script verbosely output using the same format to the /tmp/hacmp.out file. This can be achieved
by including the following line in the header of the script: set -x && PS4="${0##*/}"'[$LINENO]

'
Application Monitoring
HACMP provides the ability to monitor the state of an application. Although optional, implementation is
highly recommended. This mechanism provides for self-healing clusters. In order to ensure that event
processing does not hang due to failures in the (user supplied) script and to prevent hold-up during event
processing, HACMP has always started the application in the background. This approach has disadvantages
:
There’s no wait or error checking
In a multi-tiered environment there is no easy way to ensure that applications of higher tiers have
been started.
Application monitoring can either check for process death, or run a user-supplied custom monitor method
during the start-up or continued running of the application. The latter is particularly useful when the application
provides some form of transaction processing - a monitor can run a null transaction to ensure that
the application is functional. Best practice for applications is to have both process death and usersupplied
application monitors in place.
Don’t forget to test the monitoring, start, restart and stop methods carefully! Poor start, stop and monitor
scripts can cause cluster problems, not just in maintaining application availability but avoiding data corruption
3.
In addition, HACMP also supplies a number of tools and utilities to help in customization efforts like preand
post- event scripts. Care should be taken to use only those for which HACMP also supplies a man
page (lslpp -f cluster.man.en_US.es.data) – those are the only ones for which upwards compatibility
is guaranteed. A good best practice example for this use would be for application provisioning.
3 Having monitoring scripts exit with non zero return codes when the application has not failed in-conjunction with poor start / stop
scripts can result in undesirable behavior (i.e. data corruption). Not only is the application down but is in need of emergency repair
which may involve data restore from backup.
4 CoD support includes : On/Off CoD inc. Trial, CUoD and CBU for high-end only. See
http://www-03.ibm.com/servers/eserver/about/cod for further details.

Application Provisioning
HACMP has the capability of driving Dynamic LPAR and some Capacity on Demand (CoD) operations 4
to ensure there is adequate processing and memory available for the application(s) upon start-up. This is
shown in Fig 1.1.
Fig 1.1 Application Provisioning example.
This process can be driven using HACMP smit panels. However, this approach does have several limitations
:
Support for POWER4 TM architecture only (Whole CPU's and 256 Memory Chunks)
No provisions or flexibility for shutting down or "stealing from" other LPARs
CoD activation key must have been entered manually prior to any HACMP Dynamic Logical Partitioning
(DLPAR) event
Must have LPAR name = AIX OS Hostname = HACMP node name
Large memory moves will be actioned in one operation. This will invariably take some time and
hold up event processing
LPAR hostname must be resolvable at HMC
The HACMP diver script hmc_cmd does not log the DLPAR / CoD commands it sends to the HMC.
Debugging is limited and often is it necessary to hack the script - which is far from ideal!
If the acquisition / release fails the operation is not repeated on another HMC if defined
Given these drawbacks, I would recommend this behavior is implemented using user supplied custom
scripts. Practical examples can be explored in the AU61G Education class - see Reference section.

Testing
Simplistic as it may seem, the most important thing about testing is to actually do it.
A cluster should be thoroughly tested prior to initial production (and once clverify runs without errors or
warnings). This means that every cluster node and every interface that HACMP uses should be brought
down and up again, to validate that HACMP responds as expected. Best practice would be to perform the
same level of testing after each change to the cluster. HACMP provides a cluster test tool that can be run
on a cluster before it is put into production. This will verify that the applications are brought back on line
after node, network and adapter failures. The test tool should be run as part of any comprehensive cluster
test effort.
Additionally, regular testing should be planned. It’s a common safety recommendation that home smoke
detectors be tested twice a year - the switch to and from daylight savings time being well-known points.
Similarly, if the enterprise can afford to schedule it, node fallover and fallback tests should be scheduled
biannually. These tests will at least indicate whether any problems have crept in, and allow for correction
before the cluster fails in production.
On a more regular basis, clverify should be run. Not only errors but also warning messages should be
taken quite seriously, and fixed at the first opportunity. Starting with HACMP v5.2, clverify is run
automatically daily @ 00:00 hrs. Administrators should make a practice of checking the logs daily, and reacting
to any warnings or errors.)

Maintenance
Even the most carefully planned and configured cluster will have problems if it is not well maintained. A
large part of best practice for an HACMP cluster is associated with maintaining the initial working state of
the cluster through hardware and software changes.
Prior to any change to a cluster node, take an HACMP snapshot. If the change involves installing an
HACMP, AIX or other software fix, also take a mksysb backup. On successful completion of the change,
use SMIT to display the cluster configuration, print out and save the smit.log file. The Online Planning
Worksheets facility can also be used to generate a HTML report of the cluster configuration.
All mission critical HA Cluster Enterprises should, as best practice, maintain a test cluster identical to the
production ones. All changes to applications, cluster configuration, or software should be first thoroughly
tested on the test cluster prior to being put on the production clusters. The HACMP cluster test tool can be
used to at least partially automate this effort.
Change control is vitally important in an HACMP cluster. In some organizations, databases, networks and
clusters are administered by separate individuals or groups. When any group plans maintenance on a
cluster node, it should be planned and coordinated amongst all the parties. All should be aware of the
changes being made to avoid introducing problems. Organizational policy must preclude “unilateral”
changes to a cluster node. Additionally, change control in an HACMP cluster needs to include a goal of
having all cluster nodes at the same level. It is insufficient (and unwise!) to upgrade just the node running
the application. Develop a process which encompasses the following set of questions :
Is the change necessary?
How urgent is the change?
How important is the change? (not the same as urgent)
What impact does the change have on other aspects of the cluster?
What is the impact if the change is not allowed to occur?
Are all of the steps required to implement the change clearly understood and documented?
How is the change to be tested?
What is the plan for backing out the change if necessary?
Is the appropriate expertise be available should problems develop?
When is the change scheduled?
Have the users been notified?
Does the maintenance period include sufficient time for a full set of backups prior to the change and
sufficient time for a full restore afterwards should the change fail testing?
This process should include an electronic form which requires appropriate sign-offs before the change can
go ahead. Every change, even the minor ones, must follow the process. The notion that a change, even a
small change might be permitted (or sneaked through) without following the process must not be permitted.
To this end, the best practice is to use the HACMP C-SPOC facility where possible for any change, especially
with regards to shared volume groups. If the installation uses AIX password control on the cluster
nodes (as opposed to NIS or LDAP), C-SPOC should also be used for any changes to users and groups.
HACMP will then ensure that the change is properly reflected to all cluster nodes.

Upgrading the Cluster Environment
OK, so you want to upgrade? Start by reading the upgrade chapter in the HACMP installation documentation
and make a detailed plan. Taking the time to review and plan thoroughly will save many 'I forgot to
do that!' problems during and after the migration/upgrade process. Don’t forget to check all the version
compatibilities between the different levels of software/firmware and most importantly the application
software certification against the level of AIX and HACMP. If you are not sure check with IBM support
and/or user the Fix Level Recommendation Tool (FLRT) which is available at :
http://www14.software.ibm.com/webapp/set2/flrt/home.
Don’t even think about upgrading AIX or HACMP without first taking a backup and checking that it is
restorable. In all cases, it is extremely useful to complete the process in test environment before actually
doing it for real. AIX facilities such as alt_disk_copy and multibos for creating an alternative rootvg
which can activated via a reboot are very useful tools worth exploring and using.
Before, attempting the upgrade ensure you carry out the following steps :
Check that cluster and application are stable and that the cluster can synchronize cleanly

Take a cluster snapshot and save it to a temporary non cluster directory
(export SNAPSHOTPATH=)
Save event script customization files / User Supplied scripts to a temporary non cluster directory. If
you are unsure that any custom scripts are included, check with odmget HACMPcustom.
Check that the same level of cluster software (including PTFs) are on all nodes before beginning a
migration
Ensure that the cluster software is committed (and not just applied)
Where possible the Rolling Migration method should be used as this ensures maximum availability. Effectively,
cluster services are stopped one node at a time using the takeover option (Now move resource
groups’ in HACMP v5.4). The node/system is updated accordingly and cluster services restarted. This operation
is completed one node at a time until all nodes are at the same level and operational. Note : While
HACMP will work with mixed levels of AIX or HACMP in the cluster, the goal should be to have all nodes
at exactly the same levels of AIX, HACMP and application software. Additionally, HACMP prevents
changes to the cluster configuration when mixed levels of HACMP are present.
Starting with HACMP v5.4, PTFs can now be applied using a ‘Non disruptive upgrade’ method. The process
is actually identical to the rolling migration, however, resource groups are placed into an ‘Unmanaged’
State to ensure they remain available. Note: During this state the application(s) are not under the control
of HACMP (ie. Not highly Available!). Using the default start-up options, HACMP relies on an application
monitor to determine the application state and hence appropriate actions to undertake.
Alternatively, the entire cluster and applications can be gracefully shutdown to update the cluster using
either the ‘snapshot’ or ‘Offline’ conversion methods. Historically, upgrading the cluster this way has resulted
in fewer errors! but requires a period of downtime!

Monitoring
HACMP provides a rich set of facilities for monitoring a cluster, such as Tivoli Integration filesets and
commands such as cldisp, cldump & clstat. The actual facilities used may well be set by enterprise
policy (e.g., Tivoli is used to monitor all enterprise systems). The SNMP protocol is the crux to obtaining
the status of the cluster. HACMP implements a private Managed Information Base (MIB) branch maintained
via a SMUX peer subagent to SNMP contained in clstrmgrES daemon, as shown in Fig 2.0.
Fig 2.0 SNMP and HACMP
The clinfo daemon status facility does have several restrictions and many users/administrators of HACMP
clusters implement custom monitoring scripts. This may seem complex but actually it’s remarkably
straight forward. The cluster SNMP MIB data can be pulled simply over an secure session by typing : ssh
$NODE snmpinfo -v -m dump -o /usr/es/sbin/cluster/hacmp.defs risc6000clsmuxpd
> $OUTFILE. The output can be parsed through perl or shell scripts to produce a cluster status report. A
little further scripting can parse the output again in HTML format so the cluster status can be obtained
through a cgi web driven program, as shown in Fig 2.1. Further details are covered in the AU61 World-
Wide HACMP Education class. Other parties also have HACMP aware add-ons for SNMP monitors,
these include : HP OpenView, Tivoli Universal Agent and BMC PATROL (HACMP Observe Knowledge
Module by to/max/x).
Furthermore, HACMP can invoke notification methods such as a SMS, pager and e-mail messages on cluster
event execution and execute scripts on entry of error log reports. Best practice is to have notification of
some form in place for all cluster events associated with hardware, software failures and significant actions
such as adapter, network & node failures.
Fig 2.1 Custom HACMP Monitor

HACMP in a Virtualized World
HACMP will work with virtual devices, however some restrictions apply when using virtual Ethernet or
virtual disk access. Creating a cluster in a virtualized environment will add new SPOFs which need to be
taken into account. HACMP nodes inside the same physical footprint (frame) must be avoided if high
availability is to be achieved; this configuration should be considered only for test environments. To eliminate
the additional SPOFs in a virtual cluster the use of a second VIOS should be implemented in each
frame with the Virtual Client (VIOC) LPARs located within different frames, ideally some distance apart.
Redundancy for disk access can be achieved through LVM mirroring or Multi-Path I/O (MPIO). LVM mirroring
is most suited to eliminate the VIOC rootvg as a SPOF as shown in Fig 3.0. The root volume group
can be mirrored using standard AIX practices. In the event of VIOS failure, the LPAR will see stale partitions
and the volume group would need to be resynchronized using syncvg. This procedure can also util-
ize logical volumes as backing storage to maximize flexibility. For test environments, whereby each VIOC
is located in the same frame LVM mirroring could also be used for datavgs as well.
Fig 3.0 Redundancy using LVM Mirroring
For shared data volume groups, the MPIO method should be deployed. See Fig 4.0. A LUN is mapped to
both VIOS in the SAN. From both VIOSs, the LUN is mapped again to the same VIOC. The VIOC LPAR
will correctly identify the disk as an MPIO capable device and create one hdisk device with two paths. The
configuration is then duplicated on the backup frame/node. Currently, the virtual storage devices will
work only in failover mode, other modes are not yet supported. All devices accessed through a VIO server
must support a “no_reserve” attribute. If the device driver is not able to “ignore” the reservation, the device
can not be mapped to a second VIOS. Currently, the reservation held by a VIO server can not be broken
by HACMP, hence only devices that will not be reserved on open are supported. Therefore, HACMP
requires the use of enhanced concurrent mode volume groups (ECVGs) The use of ECVGs is generally
considered best practice!
Fig 4.0 Redundancy using MPIO
In a virtualized networking environment, a VIOS is needed for access to the outside world via a layer-2
based Ethernet bridge which is referred to an a Shared Ethernet Adapter (SEA). Now, the physical network
devices along with the SEA are the new SPOFs. How are these SPOFs eliminated? Again through the
use of a second VIOS. Etherchannel technology from within the VIOS can use used to eliminate both the
network adapters and switch as a SPOF. To eliminate the VIOS as a SPOF there are two choices :
1. Etherchannel (configured in backup mode ONLY - No Aggregation) in the VIOC. See Fig 5.0
2. SEA failover via the Hypervisor. See Fig 6.0.
There are advantages and disadvantages with both methods. However, SEA failover is generally considered
best practice as it provides the use of Virtual LAN ID (VID) tags and keeps the client configuration
cleaner.
From the client perspective only a single virtual adapter is required and hence IPAT via Aliasing must be
used. IPAT via Replacement and H/W Address Takeover (HWAT) are not supported. Having a second virtual
adapter will not eliminate a SPOF as the adapter is not real! The SPOF is the Hypervisor! Generally,
single interface networks are not best practice as this limits the error detection capabilities of HACMP. In
this case, it can’t be avoided so to aid additional analysis, add external IP-addresses to the netmon.cf file.
In addition, at least two physical adapters per SEA should be used in the VIOS in an Etherchannel configuration.
Adapters in this channel can also form an aggregate, but remember that most vendors require
adapters which form an aggregate to share the same backplane (A SPOF! - so don’t forget to define a
backup adapter). An exception this this rule is Nortel’s Split Multi-Link Trunking. Depending on your environment
this technology maybe worth investigating.
Fig 5.0 Etherchannel in Backup Mode
Fig 6.0 SEA Failover
And finally a view of the big picture. Be methodical in your planning. As you can see from Fig 7.0 even a
simple cluster design can soon become rather complex!
Fig 7.0 A HACMP Cluster in a virtualized world

Maintenance of the VIOS partition – Applying Updates
The VIOS must be updated in isolation, i.e. with no client access. A simple way of achieving this is to start
by creating a new profile for the VIO server by copying the existing one. Then delete all virtual devices
from the profile and reactivate the VIOS using the new profile. This ensures that no client partition can
access any devices and the VIOS is ready for maintenance.
Prior to restarting the VIOS, manual failover from the client must be performed so all disk access and networking
goes through the alternate VIOS. Steps to accomplish this are as follows. For:
MPIO storage, disable the activate path by typing :
chpath -l hdiskX -p vscsiX -s disable
LVM mirrored disks, set the virtual SCSI target devices to 'defined' state in the VIO server partition.
SEA failover can be initiated from the active VIOS by typing:
chdev -attr ha_mode=standby
Etherchannel in the VIOC, initiate a force failover using smitty etherchannel.
After the update has been applied the VIOS must be rebooted. The client should then be redirected to the
newly updated VIOS and the same procedure followed on the alternative VIOS. It’s important that each
VIOS used has the same code level.

Summary
‘Some final words of advice ....’
Spend considerable time in the planning stage. This is where the bulk of the documentation will be produced
and will lay the foundation for a successful production environment! Start by building a detailed
requirements document⁵. Focus on ensuring the cluster does what the users want /need it to do and that
the cluster behaves how you intend it to do. Next, build a technical detailed design document⁶. Details
should include a thorough description of the Storage / Network / Application / Cluster environment (H/
W & S/W configuration) and the Cluster Behavior (RG policies, location dependencies etc). Finally, make
certain the cluster undergoes comprehensive and thorough testing⁷ before going live and further at regular
intervals.
Once the cluster is in production, all changes must be made in accordance with a documented Change
Management procedure, and the specific changes must follow the Operational Procedures using (where
possible) cluster aware tools⁸.
Following the above steps from the initial start phase will greatly reduce the likelihood of problems and
change once the cluster is put into production. In addition, to conclude this white paper, here is a general
summary list of HACMP do’s and don’ts.
Do :
Where feasible, use IPAT via Aliasing style networking and enhanced concurrent VGs.
Ensure the H/W & S/W environment has a reasonable degree of currency. Take regular cluster
snapshots and system backups.
Configure application monitors to enhance availability and aid self healing.
Implement a test environment to ensure changes are adequately tested.
Implement a reliable heartbeat mechanism and include at least one non IP network.
Ensure there are mechanisms in place which will send out alerts via SNMP, SMS or email when failures
are encountered within the cluster.
Implement verification and validation scripts that capture common problems (or problems that are
discovered in the environment) eg. volume group settings, NFS mount/export settings, application
changes. In addition, ensure that these mechanisms are kept up-to-date.
Make use of available HACMP features, such as: application monitoring, extended cluster verification
methods, ‘automated’ cluster testing (in TEST only), file collections, fast disk takover and fast
failure detection.
Do not :
Introduce changes to one side of the cluster whilst not keeping the other nodes in sync. Always ensure
changes are synchronized immediately. If some nodes are up and others down, ensure the
change is made and synchronized from an active node.
Attempt change outside of HACMPs control using custom mechanisms. Where possible use CSPOC.
Configure applications to bind in any way to node specific attributes, such as IP Addresses, hostnames,
CPU IDs etc. It is best practice to move the applications from node-to-node manually before
putting them in resource groups under the control of HACMP.
Make the architecture too complex or implement a configuration which hard to test.
Deploy basic application start and stop scripts which do not include pre-requisite checking and error
recovery routines. Always ensure these scripts verbosely log to stdout and stderr.
Implement nested file systems that create dependencies or waits and other steps that elongate failovers.
Provide root access to untrained and cluster unaware administrators.
Change failure detection rates on networks without very careful thought and consideration.
Action operations such as # kill `ps –ef | grep appname | awk ‘{print $2}’` when
stopping an application. This may also result in killing the HACMP application monitor as well.
Rely on standard AIX volume groups (VGs) if databases use raw logical volumes. Consider instead
implementing Big or Scaleable VGs. This way, user, group and permission information can be stored
in the VGDA header and will reduce the likelihood of problems during failover.
Rely on any form of manual effort or intervention which maybe involved in keeping the applications
highly available.
⁵A written cluster requirements document allows you to carry out a coherent and focused discussion with the users about what they
want done. It also allows you to refer to these requirements while you design the cluster and while you develop the cluster test
plan.
⁶A written cluster design document describes from a technical perspective, exactly how you intend to configure the cluster environment.
The behavior of the environment should meet all requirements specified in ⁵.
⁷A written test plan allows you to test the cluster against the requirements (which describes what you were supposed to build) and
against the cluster design document (which describes what you intended to build). The test plan should be formatted in a way
which allows you to record the pass or failure of each test as this allows you to easily know what’s broken and it allows you to
eventually demonstrate that the cluster actually does what the users wanted it to do and what you intended it to do.
⁸Do not make the mistake of assuming that you have time to write the operational documentation once the cluster is in production.

AIX - LVM - Change File System

AIX - LVM - Change File System

Using chfs command we can rename and increase the size of the file system online

We want to increase file system size

#chfs –a size=+block size /filesystem name
#chfs –a size=+32m /filesystem name
#chfs-a size=+4g /filesystem name

+block size – We can mention the size using blocks (2048 blocks = 1MB)
+32m – We can mention the size using MB
+4g – We can mention the size using GB


After increasing the file system size, we can verify using commands

#df –k
#ls –q /newfs

Rename the file system using chfs command

#lsfs /newfs
It will display /newfs details
#

Now we want to rename /newfs to /testfs then your command will be

#chfs –m /testfs /newfs

After executing the above command /newfs renamed as /testfs

We can auto mount the file system while booting using chfs command

#chfs –A /testfs

AIX - LVM - Create File System

Create FS

Using crfs command we can create a file system. File systems belongs to LV’s

Whenever we create the file system we should mention the LV name

#crfs –v jfs2 –d testlv –m /newfs (Normal creation)

The /newfs file system created on testlv.

#crfs –v jfs2 –g testvg –a size=64465 –m /newfs (Directly we can create /fs from VG
This case lv name will be lv00 or lv01 like this after that also we can rename the lv name)


IBM - AIX - List File System

We can list file system details using lsfs command

#lsfs – List all filesystems in the /etc/filesystems entry

#lsfs –q (List all filesystems with detailed info)

#lsfs –a (list all filesystems (default)

#lsfs –l (specify the output in the list format)

#lsfs –c (specify the output in the column format)

#lsfs –v jfs (List all jfs filesystems)

AIX - LVM - Mount File System
After creating the file system, we should mount the file system, without file system mount we cannot access the file system using mount command we can mount the file system.

Before mount lsfs command is wont display the /newfs file system

#lsfs –a

That /newfs file system detail is not available

#mount /newfs

Now you execute lsfs command, it will show the /newfs details

#lsfs –a

AIX- Device Management

AIX- Device Management

In this device management, we can do following things.

1. Make/Add a Device
2. Change Device
3. List Device
4. Remove Device
5. List Device properties
6. List Device configuration


1. Make/Add a Device

We want to install new device in server, this case we should run following command
Suppose we want to install new Hard disk

#smitty mkdev (not recommended)

After executing the above command, one sort will come; in this sort we should fill some details about hard disk drive, such as model, make, capacity, etc.

The above command is not recommended because first we should know all the details about HDD, so instead of that we can use cfgmgr command.

When we run the cfgmgr command it will detect all the newly installed devices,

First you connect hard disk and run the cfgmgr command it will detect the hdd.

#cfgmgr (It will detect the all the devices)

#cfgmgr scsi0

Directly you want to detect hard disk then directly u should check the hdd parent device scsi0

When you run the above command it will detect only Scsi0 devices i.e. HDD, Cdrom, and Tape.

Like this we can configure newly installed devices.


2. Change Device

Suppose we want to change device configuration, this case we should run following command.

We want to change PVID (Physical volume ID) for Hdd

#chdev –l hdisk0 –a pv=clear (To clear the PVID)
#chdev –l hdisk0 –a pv=yes (To assign the PVID)

Whenever we installing the new hdd, system will generate some Identification number for hdd, we can change this PVID using chdev commands

Like this we can change duplex mode for network card and block size for tape media.

#chdev –l inet0 –a hostname=si (To change the host name for inet0)

#chdev –l rmt0 –a block_size=512 (To Change tape drive block_size is 512 blocks)

#chdev –l ent0 –a media_speed=100_full_duplex (To Change Ethernet card duplex speed)



3. List Device

Suppose we want to list device configuration details, this case se should run following command.

Whenever we list the device information that information is listed from ODM because ODM is storing the device information.

#lsdev –C (To list the Customized device configuration information from ODM)
#lsdev – P (To list the Predefined device configuration information from ODM)

The above commands list the object class device information; it will list all the device information.



Suppose we want to see particular device (object) information,

#lsdev –Cc disk
#lsdev –Cc processor
#lsdev –Cc adapter
#lsdev –Cc memory

C – Customized (object class)
c – for device (object)

If any confusion please refer the ODM notes


4. Remove Device

Suppose you want to remove device from server. Suppose it is one HDD.

First you have removed the HDD from server. Then you checking device configuration details using

#lsdev –Cc disk

Or

#odmget CuDv

It will show the device status

There is two options is available in device removal

#rmdev –dl hdisk5

The above command is used to remove hdisk5 device from ODM (complete removal from OS)

After executing the above command this hdisk5 configuration is not available in ODM

#odmget CuDv | grep hdisk5

It wont display anything because device configuration is removed from ODM

Suppose you want to remove device from the server not from ODM
This case you should use the following command.

#rmdev –l hdisk5

The above command is used to remove hdisk5 device from server, this device configuration is available in ODM (not from OS)

After executing the above command this hdisk5 configuration is available in ODM and device status is defined

#odmget CuDv | grep hdisk5

It will show Hdisk5 configuration details but device status is 0


If you want install hdisk5 device again, you should run the following command

#cfgmgr

#cfgmgr scsi0 (Directly you can search device on Parent device scsi0)

After executing above commands the device hdisk5 become to Available state.
And device status is 1

5. List device Properties (attributes)


Suppose you want to check the device attributes details, you should use following commands.


You want to see memory details,

#lsattr –El mem0 (For memory)
#lsattr –El processor0 (For processor)
#lsattr –El ent0 (For Ethernet card)
#lsattr –El rmt0 (For Tape drive)

E – Effective
l - Device
Like this all the devices.


6. List device Configuration


Suppose you want to check the device configuration details, you should use following commands.

Suppose you want to see HDD FRU number

#lscfg (it will show full system information)

#lscfg –vl hdisk0 (It will show HDD configuration details FRU)
#lscfg –vl eth0 (It will so MAC address details of eth0)

Like this all the devices.

NOTE:
#bindprocessor –q (It will show processor information)

Suppose you are using backup device, currently connected and configured with server.

First you switch on the server, at this time tape driver is not switched on. While booting the server, it is checking the tape drive, but tape is now powered on so tape drive is moved to defined state, status will be 0, Now tape drive is switched on now we want to activate the tape drive, we can activate the tape drive without rebooting the server using
#mkdev –l rmt0 (After executing the command tape drive is come to available state)