PowerHA/SystemMirror
PowerHA implementation steps
PowerHA Limits
Log Files
Useful HACMP Commands
Notes:
Installation changes
The following AIX configuration changes are made:
1. Files modified:
/etc/hosts
/etc/inittab
/etc/rc.net
/etc/services
/etc/snmpd.conf
/etc/snmpd.peers
/etc/syslog.conf
/etc/trcfmt
/var/spool/cron/crontab/root
2. The hacmp group is added.
3. Also, using cluster configuration and verification, the file /etc/hosts can be changed by adding or modifying entries.
4. The following network options are set to “1” by RSCT topsvcs startup:
- nonlocsrcroute
- ipsrcrouterecv
- ipsrcroutesend
- ipsrcrouteforward
- ip6forwarding
5. The verification utility ensures that the value of each network option is consistent across all cluster nodes for the following settings:
- tcp_pmtu_discover
- udp_pmtu_discover
- ipignoreredirects
- routerevalidate
The cluster communications daemon
With the introduction of clcomdES, there is no need for an /.rhosts file to be configured. The cluster communications daemon is started by inittab, with the entry being created by the installation of PowerHA. The daemon is controlled by the system resource controller, so startsrc, stopsrc and refresh work. In particular, refresh is used to re-read /usr/es/sbin/cluster/etc/rhosts and moving the log files. The cluster communication daemon uses port 6191.
Resource group components:
Service IP Label
Volume Group
Filesystem
Application Server
NFS mounts
NFS exports
Resource group Startup options:
Online on home node only.
Online on first available node.
Online on all available nodes.
Online using distribution policy.
Resource group Fallover options:
Fall over to next priority node in list:
Fallover using dynamic node priority:
Bring offline (on error only)
Resource group Fallback options:
Fall back to higher priority node in list
Never fall back
Resource group attributes
Settling time
Delayed fallback timers
Distribution policy
Dynamic node priorities
Resource group processing order
Priority override location
Resource group dependencies - parent / child
Resource group dependencies - location
Resource Groups operations
Bring a resource group offline
Bring a resource group online
Move a resource group to another node/site
Suspend/resume application monitoring
Sources of HACMP information
HACMP manuals come with the product –READ THEM!
Sales Manual: www.ibm.com/common/ssi
/usr/es/sbin/cluster/release_notes
IBM courses:
– HACMP Administration I: Planning and Implementation (AU54/Q1554)
– HACMP Administration II: Administration and Problem Determination (AU61/Q1561)
– HACMP V5 Internals (AU60/Q1560)
IBM Web Site:
– http://www-03.ibm.com/systems/p/ha/
Non-IBM sources (not endorsed by IBM but probably worth a look):
– http://www.matilda.com/hacmp/
– http://groups.yahoo.com/group/hacmp/
PowerHA implementation steps
PowerHA Limits
| Component | Maximum number/other limits |
|---|---|
| Nodes | 32 |
| Resource groups | 64 |
| Networks | 48 |
| Network interfaces, devices, and labels | 256 |
| Cluster resources | While 128 is the maximum that clinfo can handle, there can be more in the cluster |
| Parent-Child dependencies | Max of 3 levels |
| Site | 2 |
| Interfaces | 7 interfaces per node per network |
| Application monitors per site | 128 |
| Persistent IP alias | One per node per network |
| XD_data networks | 4 per cluster |
| GLVM Modes | Synchronous, Asynchronous, non concurrent |
| GLVM Devices | All PVs supported by AIX, no need to be same local and remote |
Log Files
| File | Description |
|---|---|
| /var/hacmp/adm/cluster.log | Generated by cluster scripts and daemons |
| /var/hacmp/log/hacmp.out | Generated by event scripts and utilities |
| /var/hacmp/adm/history/cluster.mmddyyyy | Cluster history files generated daily |
| /var/hacmp/clcomd/clcomd.log | Generated by clcomd daemon |
| /var/hacmp/clcomd/clcomddiag.log | Generated by clcomd daemon, debug information |
| /var/hacmp/clverify/clverify.log | Generated by Cluster Verification utility |
| /var/hacmp/log/autoverify.log | Generated by Auto Verify and Synchronize |
| /var/hacmp/log/clavan.log | Generated by Application Availability Analysis tool |
| /var/hacmp/log/clinfo.log | Generated by client node running clinfo |
| /var/hacmp/log/cl_testtool.log | Generated by the Cluster Test Tool |
| /var/hacmp/log/clconfigassist.log | Generated by Two-Node Cluster Configuration Assistant |
| /var/hacmp/log/clstrmgr.debug | Generated by the clstrmgr daemon |
| /var/hacmp/log/clstrmgr.debug.long | Detail information from the clstrmgr daemon |
| /var/hacmp/log/clutils.log | Generated by cluster utilities and file propagation |
| /var/hacmp/log/cspoc.log | Generated by CSPOC commands |
| /var/hacmp/log/cspoc.log.remote | Detail information from CSPOC commands |
| /var/hacmp/log/migration.log | Generated by cluster migration |
| /var/hacmp/log/sa.log | Generated by Application Discovery |
| "odmget HACMPlogs" | Displays a complete list of HACMP Log Files |
| /var/ha/log/topsvcs.default | Topology Services starup configuration log. |
| /var/ha/log/topsvcs.dd.hhmmss.lang | Topology Services start/stop log |
| /var/ha/log/topsvcs.dd.hhmmss | Topology Services activity log |
| /var/ha/log/nim.topsvcs.IF.clustername | NIM hearthbeat activity log for each interface. |
| /var/ha/log/nmDiag.topsvcs.IF.cluster | NIM diagnostic log for each interface. |
| /var/ha/log/grpsvcs.default.nodenum_instnum | Group Services startup log |
| /var/ha/log/grpsvcs_nodenum_instnum | Group Services activity log. |
Useful HACMP Commands
| Command | Purpose |
|---|---|
| clstat | Displays topology and resource groups status. clinfoES and snmpd must be running. |
| cldump | Displays topology & resource group status and configuration. snmpd must be running. |
| cldisp | Like cldump, but application oriented. snmpd must be running. |
| cltopinfo (cllsif) | Displays topology configuration. |
| clRGinfo (clfindres) | Displays resource group status. |
| clsshowres | Displays resource groups configuration. |
| clshowsrv | Calls lsrsc to display status of: HACMP subsystems (clshowrsrv -a) HACMP and RSCT subsystems (clshowsrv -v) |
| clcycle | Rotates selected log files. |
| clgetactivenodes | Displays active nodes. Must specify which nodo to ask (-n node) |
| clsnap | Save HACMP log files and configuration information. |
| cl_ls_shared_vgs | List shared vgs. |
| cl_lsfs | List shared fs. |
| cllsgrp | List the resource groups. |
| cllsres | Show short resource group information. |
| clRGmove | Brings a RG Offline/Online or move it |
| lssrc -ls clstrmgrES | Displays Cluster Services |
Notes:
Installation changes
The following AIX configuration changes are made:
1. Files modified:
/etc/hosts
/etc/inittab
/etc/rc.net
/etc/services
/etc/snmpd.conf
/etc/snmpd.peers
/etc/syslog.conf
/etc/trcfmt
/var/spool/cron/crontab/root
2. The hacmp group is added.
3. Also, using cluster configuration and verification, the file /etc/hosts can be changed by adding or modifying entries.
4. The following network options are set to “1” by RSCT topsvcs startup:
- nonlocsrcroute
- ipsrcrouterecv
- ipsrcroutesend
- ipsrcrouteforward
- ip6forwarding
5. The verification utility ensures that the value of each network option is consistent across all cluster nodes for the following settings:
- tcp_pmtu_discover
- udp_pmtu_discover
- ipignoreredirects
- routerevalidate
The cluster communications daemon
With the introduction of clcomdES, there is no need for an /.rhosts file to be configured. The cluster communications daemon is started by inittab, with the entry being created by the installation of PowerHA. The daemon is controlled by the system resource controller, so startsrc, stopsrc and refresh work. In particular, refresh is used to re-read /usr/es/sbin/cluster/etc/rhosts and moving the log files. The cluster communication daemon uses port 6191.
Resource group components:
Service IP Label
Volume Group
Filesystem
Application Server
NFS mounts
NFS exports
Resource group Startup options:
Online on home node only.
Online on first available node.
Online on all available nodes.
Online using distribution policy.
Resource group Fallover options:
Fall over to next priority node in list:
Fallover using dynamic node priority:
Bring offline (on error only)
Resource group Fallback options:
Fall back to higher priority node in list
Never fall back
Resource group attributes
Settling time
Delayed fallback timers
Distribution policy
Dynamic node priorities
Resource group processing order
Priority override location
Resource group dependencies - parent / child
Resource group dependencies - location
Resource Groups operations
Bring a resource group offline
Bring a resource group online
Move a resource group to another node/site
Suspend/resume application monitoring
Sources of HACMP information
HACMP manuals come with the product –READ THEM!
Sales Manual: www.ibm.com/common/ssi
/usr/es/sbin/cluster/release_notes
IBM courses:
– HACMP Administration I: Planning and Implementation (AU54/Q1554)
– HACMP Administration II: Administration and Problem Determination (AU61/Q1561)
– HACMP V5 Internals (AU60/Q1560)
IBM Web Site:
– http://www-03.ibm.com/systems/p/ha/
Non-IBM sources (not endorsed by IBM but probably worth a look):
– http://www.matilda.com/hacmp/
– http://groups.yahoo.com/group/hacmp/
