Hacmp cluster related commands list

PowerHA(HACMP) Commands
How to start cluster daemons (options in that order:  clstrmgr, clsmuxpd, broadcast message, clinfo, cllockd)	clstart -m -s -b -i -l
How to show cluster state and substate (depends on clinfo)	clstat
SNMP-based tool to show cluster state	cldump
Similar to cldump, perl script to show cluster state	cldisp
How to list the local view of the cluster topology	cltopinfo
How to list the local view of the cluster subsystems	clshowsrv -a
How to show all necessary info about HACMP	clshowsrv -v
How to show HACMP version	lslpp -L | grep cluster.es.server.rte
How to verify the HACMP configuration	/usr/es/sbin/cluster/diag/clconfig -v -O
How to list app servers configured including start/stop scripts	cllsserv
How to locate the resource groups and display their status	clRGinfo -v
How to rotate some of the log files	clcycle
A cluster ping program with more arguments	cl_ping
Cluster rsh program that take cluster node names as argument	clrsh
How to find out the name of the local node	get_local_nodename
rHow to check the HACMP ODM	clconfig
How to put online/offline or move resource groups	clRGmove
How to list the resource groups	cllsgrp
How to create a large snapshot of the hacmp configuration	clsnapshotinfo
How to show short resource group information	cllsres
How to list the cluster manager state	lssrc -ls clstrmgrES
Cluster manager states	·         ST_NOT_CONFIGURED Node never started ·         ST_INIT Node configured but down - not  running ·         ST_STABLE Node up and running ·         ST_RP_RUNNING  ·         ST_JOINING  ·         ST_BARRIER  ·         ST_CBARRIER  ·         ST_VOTING  ·         ST_RP_FAILED Node with event error
How to show heartbeat information	lssrc -ls topsvcs
How to check logs related to hacmp	odmget HACMPlogs
How to list all information from topology HACMP	lssrc -ls topsvcs
How to show all info about group	lssrc -ls grpsvcs
How to list the logs	cllistlogs
How to list the resources defined for all resource group	clshowres
How to show resource information by resource group	clshowres -g'RG'
How to show resource information by node	clshowres -n'NODE'
How to locate the resource groups and display status (-s)	clfindres
How to list interface name/interface device   name/netmask      associated with a specified ip label / ip address of a specific  node	clgetif
Cluster verification utility	clverify
How to list cluster topology information	cllscf
X utility for cluster configuration	xclconfig
X utility for hacmp management	xhacmpm
X utility for cluster status	xclstat
How to force shutdown cluster immediately without releasing resources	lclstop -f -N
How to do graceful shutdown immediately with no takeover	clstop -g -N
How to do graceful shutdown immediately with takeover	clstop -gr -N
How to sync the cluster topology	cldare -t
How to do the mock sync of topology	cldare -t -f
How to sync the cluster resources	cldare -r
How to do the mock sync of resources	cldare -r -f
How to list the name and security level of the cluster	cllsclstr
How to list the info about the cluster nodes	cllsnode
How to list info about node69	cllsnode -i node69
How to list the PVID of the shared hard disk for resource group dataRG	cllsdisk -g dataRG
How to list all cluster networks	cllsnw
How to list the details of network ether1	cllsnw -n ether1
How to show network ip/nonip interface information	cllsif
How to list the details of network adapter node1_service	cllsif -n node1_service
How to list the shared vgs which can be accessed by all nodes	cllsvg
How to list the shared vgs in resource group dbRG	cllsvg -g dbRG
How to list the shared lvs	cllslv
How to list the shared lvs in the resource group dbRG	cllslv -g dbRG
How to list the PVID of disks in the resource group appRG	cllsdisk -g appRG
How to list the shared file systems	cllsfs
How to list the shared file systems in the resource group sapRG	cllsfs -g sapRG
How to show info about all network modules	cllsnim
How to show info about ether network module	cllsnim -n ether
How to list the runtime parameters for the node node1	cllsparam -n node1
How to add a cluster definition with name dcm and id 3	claddclstr -i 3 -n dcm
How to create resource group sapRG with nodes n1,n2 in cascade	claddgrp -g sapRG -r cascading -n n1 n2
Creates an application server ser1 with startscript as /usr/start and stop script as /usr/stop	claddserv -s ser1 -b /usr/start -e /usr/stop
How to change cluster definitions name to dcmds and id to 2	clchclstr -i 2 -n dcmds
How to change the cluster security to enhanced	clchclstr -s enhanced
How to delete the resource group appRG and related resources	clrmgrp -g appRG
How to remove the node node69	clrmnode -n node69
How to remove the adapter named node69_svc	clrmnode -a node69_svc
How to remove all resources from resource group appRG	clrmres -g appRG
How to remove the application server app69	clrmserv app69
How to remove all applicaion servers	clrmserv ALL
How to list the nodes with active cluster manager processes from cluster manager on node node1clgetaddr node1 returns a pingable address from node node1	clgetactivenodes -n node1
How to list the info about resource group sapRG	clgetgrp -g sapRG
How to list the participating nodes in the resource group sapRG	clgetgrp -g sapRG -f nodes
How to get the ip label associated to the resource group	clgetip sapRG
How to list the network for ip 192.168.100.2, netmask 255.255.255.0	clgetnet 192.168.100.2 255.255.255.0
How to list the VG of LV nodelv	clgetvg -l nodelv
How to add node5 to the cluster	clnodename -a node5
How to change the cluster node name node5 to node3	clnodename -o node5 -n node3

How many SAN switches connected to my AIX server

Hope you all are aware of it nowadays we are mostly using SAN disks. It is highly important to know out disks and the adapter details. Fiber channel is used for lightning fast communication and to access the data. Data Read/Write to the San Disks have some protocols. Will see it later. Here is the commands to check the disks and Adapter details

# lsdev |grep fc
# lsattr -El fscsi0 |grep -i scsi_id

Example 1

# lsdev |grep fc
fcs0       Available 24-T1       Virtual Fibre Channel Client Adapter
fcs1       Available 24-T1       Virtual Fibre Channel Client Adapter

# lsattr -El fscsi0 |grep -i scsi_id
scsi_id      0xdf0707  Adapter SCSI ID                       False

# lsattr -El fscsi1 |grep -i scsi_id
scsi_id      0xde0718  Adapter SCSI ID                       False

0xdf & 0xde means that the 2 FC adapter are connected to 2 different SAN Switch

How to clear the PV id in AIX

When you assign a disk to a volume group,the disk automatically gets a PVID(Physical Volume ID).

For instance , the following example shows that hdisk2 has a PVID because it is a member in  hacmp_VG and hdisk1 has no PVID

because it is not a member in any VG:

# lspv

hdisk2          00f6c0100e3c830e      hacmp_VG    active
hdisk0          00f6c0105a881049      rootvg      active
hdisk1            none                 None

let`s remove hacmp_VG  and check the state of the PVID of hdisk2″

# reducevg hacmp_VG hdisk2
ldeletepv: Volume Group deleted since it contains no physical volumes.

# lsvg
rootvg

# lspv
hdisk2          00f6c0100e3c830e     None
hdisk0          00f6c0105a881049     rootvg          active
hdisk1            none               None

Now,we removed the hacmp_VG and hdisk2 still has the same PVID!

To remove the hdisk2 PVID:

#chdev -l hdisk2 -a pv=clear
hdisk2 changed

let`s check it again:

#lspv
hdisk2          none                   None
hdisk0          00f6c0105a881049       rootvg          active
hdisk1          none                   None

Now,we removed hdisk2 PVID.

Troubleshooting on boot process

Troubleshooting on boot process:

Accessing a system that will not boot: Press F5 on a PCI based system to boot from the tape/CDROMàInsert volume 1 of the installation media àselect the maintenance mode for system recoveryà Access a root volume groupàselect the volume groupà

Damaged boot image:Access a system that will not bootàCheck the / and /tmp file system sizeàdetermine the boot disk using lslv –m hd5àRecreate the boot image using bosboot –a –d /dev/hdisknàcheck for CHECKSTOP errors on errlog. If such errors found probably failing hardware. àshutdown and restart the system

Corrupted file system, Corrupted jfs log: Access a system that will not bootàdo fsck on all filw systemsà format the jfs log using /usr/sbin/logform /dev/hd8àRecreate the boot image using bosboot –a –d /dev/hdiskn

Super block corrupted: If fsck indicates that block 8 is corrupted, the super block for the file system is corrupted and needs to be repaired ( dd count=1 bs=4k skip=31 seek=1 if=/dev/hdn of=/dev/hdn)àrebuild jfslog using /usr/sbin/logform /dev/hd8àmount the root and usr file systems by (mount /dev/hd4 /mnt, mount /usr)àCopy the system configuration to backup directory(cp /mnt/etc/objrepos* /mnt/etc/objrepos/backup)àcopy the configuration from the RAM fs(cp /etc/objrepos/Cu* /mnt/etc/objrepos)àunmount all file systemsàsave the clean ODM to the BLV using savebase –d /dev/hdiskàreboot

Corrupted /etc/inittab file: check the empty,missing inittab file. Check problems with /etc/environment, /bin/sh,/bin/bsh,/etc/fsck,/etc/profileàReboot

Runlevelà selected group of processes. 2 is muti user and default runlevel. S,s,M,m for Maintenance mode

Identifying current run levelàcatt /etc/.init.state

Displaying history of previous run levels: /usr/lib/acct/fwtmp < /var/adm/wtmp |grep run-level

Changing system run levels: telinit M

Run level scripts allow users to start and stop selected applications while changing the run level. Scripts beginning with k are stop scripts and S for start scripts.

Go to maintenance mode by using shutdown -m

Rc.boot fle: The /sbin/rc.boot file is a shell script that is called by the init. rc.boot file configures devices, booting from disk, varying on a root volume group, enabling fle systems, calling the BOS installation programs.

/etc/rc file: It performs normal startup initialization. It varyon all vgs, Activate all paging spaces(swapon –a), configure all dump devices(sysdumpdev –q), perform file system checks(fsck –fp), mount all

/etc/rc.net: It contains network configuration information.

/etc/rc.tcpip: it start all network related daemons(inted, gated, routed, timed, rwhod)

back up in AIX

backup
Command typically used to backup files on an AIX platform. Distribution tapes for AIX are in this format. This command will span across multiple tapes.
Backup by INODE is not recommended because it eliminates the capability to selectively restore files by filename.
If you have more data to backup than can fit on one tape, you will probably want to perform INCREMENTAL backups for files that have been modified within the last 24 hours for each day of the work week, followed by making a complete backup of the system on the weekend.
The error message Volume on /dev/rfd0 is not in backup format is generated when an attempt is made to extract a file NOT created by the backup command:
If your backup was successful, the last item logged by 'backup' is the time the backup finished and the number of 512 byte block that were archived.

Options What it does
-i Read standard input
-v List names of files being backed up
-f Name of output device (typically de/v/rmt0 for tape drive)
-e Won't attempt to compress or pack files that are greater than 24 MB.
-q Won't prompt user to insert tape
-p Pack data. Backup images by default are NOT packed. If you wish to pack your data, use this option. The restore command has enough intelligence to determine if a backup image is packed or not and take appropriate action


Examples What it does
find /u -print | backup -ivf/dev/rmt0 Backup the file system /u to tape device using ABSOLUTE pathnames.

find . -print | backup -iepqvf/dev/rmt0 | tee /tmp/BACKUP.LOG

Backup files and directories under current directory (.) using RELATIVE pathnames. The tee (|) will take as input the output of the backup command (filenames being copied) and write them to the log file 'BACKUP.LOG' for later review.


find /faxdata -mtime -1 -type f -print | backup -ivf/dev/rmt0.1 Find any REGULAR files (-type f) that have been modified within the past 24 hours (-mtime -1) and write them to the tape device but don't rewind (/dev/rmt*.1) the tape after backup has finished.


backup -ivqf/dev/fd0 Backup files interactively. Select the file(s) to backup by typing in the file's name. When done, press ^D to finish.

CPIO backup command in AIX

Copies files to/from an archive storage device. This command is often used to copy files from UNIX to AIX systems. This command will span across multiple tapes. cpio stands for CoPy Input to Output.




Options What it does
-c Writes header information in ascii format for portability across UNIX systems. The message Out of phase--get help is generated when an attempt is made to extract a file not in cpio format OR if the archive tape was written with the -c option. The cpio command doesn't have this option specified when attemping to extract files from the media (eg., cpio -icdumpv < /dev/rmt0)
-r When restoring will prompt for new path/file to restore to
-v List the file name as it's being copied
-i Read from standard input, i.e. keyboard
-u Copies unconditionally. Older file replaces a newer one. Files with older modification times will replace files with newer modification times.
-d Creates directories, if applicable
-t For table of contents
-C1 Performs block I/O in 512 byte blocks, if the block size of the device that wrote the data on the tape was set to 512 bytes. Make sure you read with the same block size. If the block size that the data was written to and about to be read from are two different sizes, this message will be displayed: Cannot read from the specified device