NIM FAQ

To boot from network for NIM installation from NIM server/boot server 192.168.0.2
and client 192.168.0.220 (provided nim allocation and bosinst done from nim server).

# bootlist -m normal ent0 speed=auto duplex=auto gateway=0.0.0.0 bserver=192.168.0.2 \
client=192.168.0.220

To change NIM client limitation
# nim -o change -a restrict_nfs_exports=no master

How to check the bosinst.data file
# /usr/lpp/bosinst/bicheck /export/lpp_source/no_prompt

How to check a spot missing fileset
# nim -o fix_query 530TL9BASEspot|grep ML

How to configure the NIM Master only (No other resource). Run ifconfig -a first to find
which network you want for NIM Configuration. In this case it is en0
# nimconfig -anetname=MASTER_NET -apif_name=en0 -a platform=`bootinfo -p` \
-a netboot_kernel=mp -a cable_type=tp

How to copy software to an existing lpp_source from the command line
1a. Copy the software from the media to the lpp_source directory.
2b. Perform the NIM check operation on the lpp_source by entering the following command
1c. # nim -o check Lpp_sourceName

How to create a LPP Source from CD. (To create from an existing directory use the full path
of the existing directory ex. /export/lpp_source/510ML2lpp in stead of /dev/cd0)
# gencopy -X -b "-qv" -d /dev/cd0 -t /export/lpp_source/510ML8lpp -f all 2>&1

How to create a NIM lpp_source from existing directory
# nim -o define -t lpp_source -a server=master -a location=/export/lpp_source/530TL5lpp \
530TL5lpp

How to create a spot from existing lpp_source
# nim -o define -t spot -a server=master -a location=/export/spot/530ML5spot \
-a source=530ML5lpp 530ML5spot

How to create a software bundle for using with NIM
# gencopy -X -b "-qv" -d /export/lpp_source/gpfs -t /export/lpp_source/GPFS -f Fi le 2>&1

How to create AIX 4.3.3 spot from AIX 5.3 and higher
1a. # export INST_DEBUG=yes
1b. # nim -o define -t spot -a server=master -a location=/export/spot \
-a source=/export/lpp_source/430ML3lpp 430ML3spot

How to create a spot from a mksysb image name god.mksysb
(provided god.mksysb already defined as mksysb resource)
# nim -o define -t spot -a source=god.mksysb -a server=master -a location=/export/spot god.spot

To define a NIM client with 64 Bit Kernel and network to 1000 Full Duplex hard coded
(NIM Servername=testnim, client=agni, comments option is optional)
# nim -o define -t standalone -a if1="testnim agni 0" -a cable_type1=tp \
-a net_settings1="1000 full" -a platform=chrp -a netboot_kernel=64 \
-a comments="192.168.190.202" agni

To define a NIM client with 32 Bit(mp) Kernel and network to auto
(NIM Servername=testnim, client=agni, comments option is optional)
# nim -o define -t standalone -a if1="testnim agni 0" -a cable_type1=tp \
-a net_settings1="auto auto" -a platform=chrp -a netboot_kernel=mp \
-a comments="192.168.190.202" agni

Display NIM Machines
# lsnim -c machines

Steps to fix error code 0503-497 bosboot creation error
( 0503-497 installp: An error occurred during bosboot verification processing)
# cd /dev
# lsvg -p rootvg <-- br="" hdisk0="" hdisk1=""># rm ipldevice
# ln hdisk0 ipldevice
# bosboot -ad /dev/ipldevice <--- br="" error="" same=""># bosboot -ad /dev/hdisk0 <--- br="" error="" same=""># ls -l |grep ipl -> ipl_blv is missing
# ln rhd5 ipl_blv

Find the SPOT service pack level
# nim -o fix_query spot_name |grep SP ( replace spot_name with actual spot name)

To find out what is missing from the SPOT
# nim -o fix_query -a fix_query_flags="cq" -a fixes=5300-07_AIX_ML \
530TL7SP3spot |grep ":-:"

How to fix "Request denied - Method_req"
# nim -Fo change -a cpuid= ClientName

How to display mksysb file contents
# lsmksysb -c -f /mksysb_name

How to check tape is mksysb tape or not

Procedure 1
# lsmksysb -f /dev/rmt0 (if the tape is in rmt0)

Procedure 2
# restore -Tvf /dev/rmt0.1 -s4

How to find out allocation count of lpp_source resource
# lsnim -a lpp_source

How to find out allocation count of spot resource
# lsnim -a spot

How to find out allocation count of spot resource
# lsnim -t mksysb - to see all the mksysb images

How to make a lpp Source from original CD
# gencopy -X -b "-qv" -d /dev/cd0 -t /export/lpp_source/710TL0SP1lpp all

How to list all mksysb resources
# lsnim -t mksysb

To display detailed information about the NIM attributes named lpp_source and Rstate
# lsnim -p -a lpp_source -a Rstate

To display the operations which can be performed on the paging object type
# lsnim -p -t paging -O

To display all customized objects in the networks object class
# lsnim -c networks

To display allocated NIM resources for machine sigma
# lsnim -c resources sigma

To display a list of operations that can be applied to sigma
# lsnim -O sigma

To display a list of resources available to sigma
# lsnim -L sigma

To display the members of the machine group test1 with state and group exclusion status
# lsnim -g test11

To display basic information about the members of the resource group myres11
# lsnim -m myres11

To display a long listing of members of the machine group MacGrp1, with any hidden NIM
internal information
# lsnim -m -Fl MacGrp1

To display all members of machine group MacGrp1 which has a spot allocated
# lsnim -ma spot MacGrp1

To display all members of machine group MacGrp1 which has a lpp_source allocated
# lsnim -ma lpp_source MacGrp1

To create mksysb image from tape to /images directory
Step1 # tctl -f /dev/rmt0.1 rewind
Step2 # tctl -f /dev/rmt0.1 fsf 3
Step3 # dd if=/dev/rmt0.1 of=/images/mksysb.name.image bs=4m

To perform the boot diag operation from the client
# nimclient -o diag -a spot=SPOTName

To perform the diag operation from the master
# nim -o diag -a spot=SPOTName MachineName

To issue the maint_boot operation from the client
# nimclient -o maint_boot -a spot=SPOTNAME

To issue the maint_boot operation from the master
# nim -o maint_boot -a spot=SPOTNAME CLIENT_NAME

To verify that the maintenance boot operation worked from client
# nimclient -l -l ClientMachineName

To verify that the maintenance boot operation worked from NIM Master
# lsnim -l ClientMachineObjectName
(If the operation was successful, the client's Cstate output will look similar
to the following: Cstate = maintenance boot has been enabled)

To find NIM bootp failure issue

Bootp failure : Make sure it's running
# lssrc -t bootps

output should look like below

bootps /usr/sbin/bootpd bootpd /etc/bootptab active

If this is set to "inoperative" try starting it

# vi /etc/inetd.conf make sure 'bootps' and 'tftp' are uncommented and save the file.

# refresh -s inetd

# lssrc -t bootps

Checking spots few commands

# nim -o lppchk -a lppchk_flags="v" 5305_spot
# nim -o lppchk -a lppchk_flags="l" 5305_spot
# nim -o lppchk -a lppchk_flags="c" 5305_spot

NIM command example to allocate and install mksysb

# nim -o bos_inst -a source=mksysb -a mksysb=clientname_mksysb \
-a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp \
-a accept_licenses=yes -a no_client_boot=yes -a force_push=no clientname

NIM command to define a mksysb resource
# nim -o define -t mksysb -a server=master -a location=/images/cust/custimg.img custimgname

NIM Command example to install BOS

a) # nim -o bos_inst -a source=rte -a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp \
-a bosinst_data=No_Prompt -a script=MYSCRIPT -a fb_script=InstallSoft \
-a accept_licenses=yes -a preserve_res=yes -a no_client_boot=yes \
-a set_bootlist=no -a force_push=no CLIENT_NAME
or

b) # nim -o bos_inst -a source=rte -a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp \
-a bosinst_data=No_Prompt -a script=MYSCRIPT -a fb_script=InstallSoft \
-a accept_licenses=yes -a no_client_boot=yes -a force_push=no CLIENT_NAME

NIM deallocate all resource allocate to a client
a) # nim -Fo reset client_name
b) # nim -Fo deallocate -a subclass=all client_name

Disabling master push permissions from the command line
# nimclient -P

To re-enable push permission on the client, enter the following on the client machine
# nimclient -p

NIM Fix query comands example
# nim -o fix_query 520ML5spot|grep ML
# nim -o fix_query -a fix_query_flags=c 520ML5spot|grep ":-:"

NIM Insatll commands example for bos

Example 1.

# nim -o bos_inst -a source=rte -a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp \
-a bosinst_data=No_Prompt -a script=FTPSCR -a fb_script=Install_Drivers \
-a accept_licenses=yes -a preserve_res=yes -a no_client_boot=yes \
-a set_bootlist=no -a force_push=no venus

Example 2.

# nim -o bos_inst -a source=rte -a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp\
-a bosinst_data=No_Prompt -a script=FTPSCR -a fb_script=Install_Drivers\
-a accept_licenses=yes -a no_client_boot=yes -a force_push=no venus

Example 3. for mksysb

# nim -o bos_inst -a source=mksysb -a mksysb=venus_mksysb\
-a spot=530ML7SP3spot -a lpp_source=530ML7SP3lpp\
-a accept_licenses=yes -a no_client_boot=yes -a force_push=no venus

To enable the 'use_clients_if_host' option on the master
# nim -o change -a use_clients_if_host=yes master

To disable the 'use_clients_if_host' option on the master
# nim -o change -a use_clients_if_host=no master

To view the 'if' attribute for mac1
# lsnim -a if mac1

NIM Problem Unable to create any type of network boot image
# smitty nim_control_boot ==> Limit Boot Image Creation to Defined Interfaces?
Change it to "Yes"

NIM showlog command example
# nim -o showlog -a full_log=yes -a log_type=nimerr 530TL4spot

NIM take over example from nim server testnim1
# /usr/lpp/bos.sysmgt/nim/methods/m_takeover testnim1

To update NIM packages from CD
# nim -o update -a packages=all -a source=/dev/cd0 530ML5lpp

To update NIM packages from a downloaded directory /download
# nim -o update -a packages=all -a source=/download 530ML5lpp

To upgrade a client using nimadm
# nimadm -c 550-b-lp01 -s 530ML7spot -l 530ML7lpp -d "hdisk10 hdisk11" -Y -j nimvg

ODMDELETE COMMAND TO DELETE NIM OBJECTS. USE AT YOUR OWN RISK.

Suppose you want to delete the entry with TRYME mksysb and lsnim shows the name as

mksysb.TRYME and you are unable to delete it normal way.

MAKE SURE YOU BACKUP NIM DATABSE BEFORE THIS. READ THE LAST LINE ALSO

# odmget nim_attr >/tmp/nim_attr.out

# vi /tmp/nim_attr.out and look for TRYME entry

Note down the id no for Ex. id=1161733976

# odmdelete -o nim_attr -q id=1161733976

Now Delete it from nim_object

#odmget nim_object >/tmp/nim_object.out

vi that file and note down the id for TRYME

# odmdelete -o nim_object -q id=1162344443

now from websm screen or smitty nim add the routing information to NIM

MASTER object

resources -> master ->properties ->nim interface. ( Add the interface again)

Recovering the /etc/niminfo file from the command line
# nimconfig -r

To rebuild the /etc/niminfo file from a running NIM client
# niminit -a master_port=PortNumber -a master=MasterHostName \
-a name=ClientMachineObjectName

To define a SPOT from a directory that previously had a SPOT installed in it. This won't work
if you copied the spot directory from another server.
# /usr/lpp/bos.sysmgt/nim/methods/m_mkspot -o -a server=server \
-a location=location -a source=no spotname

Example:

A SPOT named spot1 was created on the NIM master in the /export/spot directory.
Later, the NIM database became corrupted and has to be rebuilt. The SPOT files
are still on the machine, but the SPOT has to be redefined to NIM.
# /usr/lpp/bos.sysmgt/nim/methods/m_mkspot -o -a server=master \
-a location=/export/spot -a source=no spot1

To remove a mksysb image from NIM Server. It will remove the mksysb file too
# nim -o remove -a rm_image=yes mksysbname

To Remove a bundle/package from a lpp_source

1a. Find the Package/Bundle name
# installp -Ld /export/lpp_source/530TL12SP2lpp|grep -i devices.vtdev.scsi.rte
1b. Now remove the Package from the LPP Source
# nim -o update -a rm_images=yes -a packages=devices.vtdev.scsi 530TL12SP2lpp

To list all duplicate and conflicting updates in the /myimages image source directory
# /usr/lib/instl/lppmgr -d /myimages -u

To remove all duplicate and conflicting updates in the /export/lpp_source/530TL10Sp2lpp
lpp_source directory
# /usr/lib/instl/lppmgr -d /export/lpp_source/530TL10Sp2lpp -u -r

To list of all superseeded filesets from /export/lpp_source/530TL10Sp2lpp
lpp_source directory
# /usr/lib/instl/lppmgr -d /export/lpp_source/530TL10Sp2lpp -x

To remove all superseeded fileset from /export/lpp_source/530TL10Sp2lpp
lpp_source directory
# /usr/lib/instl/lppmgr -d /export/lpp_source/530TL10Sp2lpp -x -r

To remove all superseeded and duplicate fileset /export/lpp_source/530TL10Sp2lpp
lpp_source directory
# /usr/lib/instl/lppmgr -d /export/lpp_source/530TL10Sp2lpp -x -u -r

To restore a single file from a mksysb image to current directory
This example is for /etc/passwd file
# restore -xvqf client_mksysb ./etc/passwd

To create an image.data file for editing before creating a mksysb
# mkszfile ( will create ./image.data which can be used for mksysb command)
# mksysb -m (will create the disk map under /tmp/vgdata/rootvg)
# mksysb -X (will extend /tmp filesystem if necessary)

To display detail NIM network information
# lsnim -c networks -l

To display BOS installation status information
# lsnim -a info -a Cstate ClientName

OR

# lsnim -l ClientName

To update a lpp_source using gencopy from a downloaded directory
# gencopy -X -b "-qv" -d /download -t /export/lpp_source/530ML6lpp/530ML6lpp -f all 2>&1

How to update a spot image
# nim -o cust -F -a lpp_source=530TL12SP2lpp -a fixes=update_all 530TL12SP2spot

To define NIM client itself to the master
# niminit -a name=krishna -a master=nimtest1 -a pif_name=en0 -a platform=chrp \
-a netboot_kernel=mp

To check the above was successful
# nimclient -l -l krishna

To change NIM CPUID
# nim -o change -a validate_cpuid=no master

IBM TECHINFO-0020751 ( www.ibm.com). This is very useful document I found from IBM website.
On a Network Server 500 or 700 running AIX, I moved a PCI Ethernet card to another PCI slot. I
was not able to remove the old device, however. The device was in the defined state, but it
could not be removed even if I manually deleted the device. This leaves an "en1" device,
and a configured "en2" device which should be "en1".

This is a special case where an object defined in the ODM could not be deleted by deleting the
device. You can selectively alter the ODM with the following steps:

Step 1
Before running any "" or "odmadd" commands, it is recommended that you save the original
copies of the Cu* files, like this:
# cd /etc/objrepos
# cp CuAt cuat.old
# cp CuDep cudep.old
# cp CuDv cudv.old
# cp CuDvDr cudvdr.old
# cp CuVPD cuvpd.old
Step 2

In general, to find out what is in the ODM, you can run the odmget command. There are five
files on which you would want to run odmget. These files are in /etc/objrepos; their names
are CuAt, CuDep, CuDv, CuDvDr, and CuVPD. The syntax for odmget would be:
# odmget CuAt > /tmp/cuat
# odmget CuDep > /tmp/cudep
# odmget CuDv > /tmp/cudv
# odmget CuDvDr > /tmp/cudvdr
# odmget CuVPD > /tmp/cuvpd

Then you can use pg, vi, or another editor on the /tmp/cu* files to "see" what is in the ODM.
You can also "grep" for items, if you know what you are looking for, like this: (This assumes that
you put the output of the odmget commands in the /tmp directory, in files beginning with the letters
"cu".)

grep 'ent_name' /tmp/cu*
where "ent_name" is the word for which you are looking. For example: grep ent2 /tmp/cu*

Step 3

WARNING: If you run " -o CuAt", you will delete EVERYTHING in /etc/objrepos/CuAt!
Consider backing up each /etc/objrepos/Cu* file if you have not already done so.

You can put the commands below in a file, make it executable, and run it.
It will delete anything in the ODM that is associated with et2. You will want to also delete all ent2 entries.

DEVICE=et2
# odmdelete -q "name = $DEVICE" -o CuAt
# odmdelete -q "parent = $DEVICE" -o CuDv
# odmdelete -q "name = $DEVICE" -o CuDv
# odmdelete -q "name = $DEVICE" -o CuDep
# odmdelete -q "dependency = $DEVICE" -o CuDep
# odmdelete -q "value1 = $DEVICE" -o CuDvDr
# odmdelete -q "value3 = $DEVICE" -o CuDvDr

Lost Root Password Recovery Using NIM

Lost root password recovery in AIX is relatively easy using nim.  There are some differences if the server in question has been imaged with a mksysb.  In this case, the ‘Access a root volume’ option may not successfully mount rootvg.  Here is how to get around that:

On the nim server, configure the nim client for a maintenance boot:

# nim -o maint_boot -a spot=install

 image> <host machinename>

Boot the client into SMS and, after configuring the network settings, etc., boot the system normally into the ‘Welcome To Base OS’ menu.

You will now see an option to ‘Start Maintenance Mode for System Recovery’ (option 3).  Choose this option and then ‘Access this volume and Start a Shell’.  Select 0 Continue. If the shell is given successfully use the ‘passwd’ command to change the root password.

# passwd
# sync;sync;sync;reboot

This is how lost password recovery is supposed to work.  However, if you get an error about getrootfs failing, use the following technique:

Identify the rootvg hdisk with lspv, import the disk and mount / .

# lspv
# importvg hdisk(x)
# mount /dev/hd4 /mnt

Next, we’ll create a temporary user with root’s ID.

# echo temproot::0:0::/:/usr/bin/ksh >> /mnt/etc/passwd
# sync
# cd /
# umount /mnt

Reboot and login as ‘temproot’.  Change root’s password.

# passwd root

Lastly, and most importantly, edit the /etc/passwd file and REMOVE temproot.

Have a better way to handle this?  Let us know in the comments.

AIX Basics: Growing a Filesystem

This begins what will be an ongoing series of posts relating to basic AIX tasks.  The first is a quick explanation of how to add and takeaway space from a filesystem:

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 964 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    3.00    3.00    1%    14    1%    /home

Check the VG for available space.

# lsvg rootvg
VOLUME GROUP:      rootvg                  VG IDENTIFIER:
VG STATE:                    active                    PP SIZE: 512 megabyte(s)
VG PERMISSION:      read/write           TOTAL PPs: 558 (285696 megabytes)
MAX LVs:                       256                        FREE PPs: 454 (232448 megabytes)
LVs:                                    14                         USED PPs: 104 (53248 megabytes)
…

Now, we add space to our filesystem.

# chfs -a size=+5G /home
Filesystem size changed to 16777216

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1     8.00     7.99     1%     14 1%     /home

Alternatively, we could specify the total size of our filesystem.

# chfs -a size=10G /home
Filesystem size changed to 20971520

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    10.00    9.99    1%    14    1%    /home

Be careful when removing space from a filesystem.  It is possible to lose data if the filesystem is close to full, depending on how much you want to decrease by.  This can only be done with JFS2 filesystems.

# chfs -a size=8G /home

or

# chfs -a size=-2G /home

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    8.00    7.99    1%    14    1%    /home

AIX Basics: ‘du’ & ‘df’ – why can’t they just get along?

A right of passage as a unix admin is to be deceived by the ‘du’ and ‘df’ commands.  Every single one of us has tried to clear space in a full file system only to run ‘df’ and be greeted by that ever-present 100%.  What gives?

The answer lies in the way that each command gathers disk allocation information.

‘du’ is a user level program that navigates the file system tree and adds up the blocks allocated to files, directories and links as reported by stat().  ’du’ doesn’t have access to be “aware” of everything, so it doesn’t see metadata.  ’df’ refers to the disk allocation maps and has all necessary accesses.

In every case, ‘df’ is a more reliable indicator of file system allocation status.  Here is an actual case of a common example of why:

# df -g
..
/dev/logicalv     72.00      0.00  100%    13579    96% /filesystem
..

# du -s /filesystem
32671176        /filesystem

# fuser -d /filesystem
/filesystem  123456

Process 123456 has a file(s) open in /filesystem that has been deleted.

Notice that ‘df’ reports a full 72G %used and ‘du’ reports only 32G used. The application had multiple files opened when the files were deleted from the filesystem. Since that space is still not free, ‘df’ correctly reports it as used. ‘du’, however, only has visibility into present file allocations and doesn’t see metadata, so it reports it free.

This conversation can also extend into ensuring files are not currently opened before deleting them during cleanup. This can be done with fuser and is explained in more detail in this fuser post.

‘du’ is perfectly adequate to quickly find the larger files in a file system for cleanup, but, when it comes to accurately reporting the allocation status of a file system, stick with ‘df’.

AIX Basics: fuser

Removing files and unmounting file systems in unix should always be preceded by the ‘fuser’ command. Too many times I see situations where an application has a file or files opened and they are removed from the file system.  This causes even greater disparity between the ‘du’ and ‘df’ commands and confusion in general.

The filesystem will not release file space until the process in question has closed it.  ’df’ reflects this and ‘du’ does not.  The ‘-d’ option will aid in troubleshooting this situation:

# df -g
..
/dev/logicalv     72.00      0.00  100%    13579    96% /filesystem
..

# du -s /filesystem
32671176        /filesystem

# fuser -d /filesystem
/filesystem  123456

In this situation, process 123456 has a deleted file open from /filesystem.  Ideally, the admin would have done the following before deleting a file:

# fuser /filesystem/file
/filesystem/file: 123456

This shows that process 123456 is using the file.  Before deleting the file it needs to be released.

‘fuser’ can also be used to terminate all processes using a filesystem.  This should only be used when permission has been obtained to bring the application down.

# fuser -kxuc /filesystem/file

This will kill all processes using /filesystem/file.

Disk replacement on VIO server

Replace failed disk in VIOS

This article describes two common scenarios for replacing failing local disk in VIOS.

Scenario 1:   Failed disk in VIO server which is used by VIO client(s)

The failing disk contains LVs used for rootvg of VIO clients. The rootvg is mirrored to another disk presented by a second VIO server. This scenario is illustrated below:

Procedure to replace failing disk on VIO server:

On VIOS: as padmin user:

Record information which will be needed for later operations and recreation of devices:
$ lsdev -virtual

To get volume group name in which failed disk participate:
$ lspv

To get list of  logical volumes on disk:
$ lspv -lv

To get info about logical volumes, e.g. size (number of LPs):
$ lsvg -lv
$ lsvg

To get info about LVs, VTD names, vhost numbers and virtual clients:
$ lsmap –all

On client(s): Identify affected disk(s)(LVs on bad disk on VIOS):
# lscfg -vl  (for all virtual SCSI disks)
hdisk1           U9117.MMA.999999-V2-C12-T1-L8200000000000000  Virtual SCSI Disk Drive

Take note of the following:
V# – LPAR ID (this should be the LPAR ID of the affected VIOS)
C# – slot number
L# – LUN ID

The affected disk may be listed as removed or missing depending on the failure.
# lsvg -p rootvg

Remove the bad disk from the mirror:
# unmirrorvg rootvg
# reducevg rootvg
# rmdev -dl hdisk#

On VIOS:

Remove all VTDs and LVs that reside on the failed disk:
$ rmvdev -vtd -rmlv
or
$ rmdev -dev
$ rmdev -dev

Check if all logical volumes are removed from bad disk:
$ lspv -lv

Remove the disk from the respective volume group:
$ reducevg
Note: If the volume group consists of only one disk then the whole VG will need to be removed from ODM. In that case use the following commands:

$ deactivatevg
$ exportvg
Replace the failed disk:
$ diagmenu
–> select “Task Selection”
–> select “Hot Plug Task”
–> select “SCSI and SCSI RAID Hot Plug Manager”
–> Replace/Remove a Device Attached to an SCSI Hot Swap Enclosure

Configure the new disk:
$ cfgdev

Add the new disk to the volume group or recreate the VG in case it was removed:
$ extendvg
or
$ mkvg -vg  

Recreate the LVs with the same names and size which we got in the beginning.
$ mklv -lv

Recreate the VTDs:
$ mkvdev -vdev -vadapter -dev

On client(s):

Discover new disk(s) and rebuid mirror:
# cfgmgr
# extendvg rootvg
# mirrorvg rootvg
Build boot image on both mirrored disks (just in case):
# bosboot -ad /dev/
# bosboot -ad /dev/

Set bootlist:
# bootlist -m normal <list names of both hdisks>
# bootlist -m normal -o

Scenario 2:  Bad disk in rootvg of VIO server
Usually rootvg utilize some kind of disk protection. Most often rootvg consists of disks which are LVM mirrored. To replace a mirrored hdisk in rootvg of VIO server you can use VIO commands or root AIX commands (to become root, use oem_setup_env command). In this example we will use VIO commands since this is the recommended way of managing VIOS.

Break the mirror:
$ unmirrorios  , where is the bad disk
Check if any LV remained on the bad disk:
$ lspv -lv
If there are any (e.g. lg_dumplv – dump device) migrate them to the other disk or remove them (dump device can be recreated later):

$ migratepv -lv
or
$ rmlv -f

Take out failed disk from rootvg:
$ reducevg rootvg
Use ”Hot Plug” procedure to replace the failed disk:

$ diagmenu
–> select “Task Selection”
–> select “Hot Plug Task”
–> select “SCSI and SCSI RAID Hot Plug Manager”
–> Replace/Remove a Device Attached to an SCSI Hot Swap Enclosure

Configure the new disk:
$ cfgdev

Verify that the new disk came back with the same number as the previous one:
$ lspv
$ extendvg rootvg
$ mirrorios -defer  (Note that if you do not use -defer option, your VIO server will be rebooted after mirroring completes)
Check bootlist to ensure that both disks are included as boot devices:

$ bootlist -mode normal -ls
hdisk0 blv=hd5
hdisk1 blv=hd5

Use the command below to include both disks if they do not show up in the bootlist:

$ bootlist -mode normal hdisk0 hdisk1