Lost Root Password Recovery Using NIM

Lost root password recovery in AIX is relatively easy using nim.  There are some differences if the server in question has been imaged with a mksysb.  In this case, the ‘Access a root volume’ option may not successfully mount rootvg.  Here is how to get around that:

On the nim server, configure the nim client for a maintenance boot:

# nim -o maint_boot -a spot=install

 image> <host machinename>

Boot the client into SMS and, after configuring the network settings, etc., boot the system normally into the ‘Welcome To Base OS’ menu.

You will now see an option to ‘Start Maintenance Mode for System Recovery’ (option 3).  Choose this option and then ‘Access this volume and Start a Shell’.  Select 0 Continue. If the shell is given successfully use the ‘passwd’ command to change the root password.

# passwd
# sync;sync;sync;reboot

This is how lost password recovery is supposed to work.  However, if you get an error about getrootfs failing, use the following technique:

Identify the rootvg hdisk with lspv, import the disk and mount / .

# lspv
# importvg hdisk(x)
# mount /dev/hd4 /mnt

Next, we’ll create a temporary user with root’s ID.

# echo temproot::0:0::/:/usr/bin/ksh >> /mnt/etc/passwd
# sync
# cd /
# umount /mnt

Reboot and login as ‘temproot’.  Change root’s password.

# passwd root

Lastly, and most importantly, edit the /etc/passwd file and REMOVE temproot.

Have a better way to handle this?  Let us know in the comments.

AIX Basics: Growing a Filesystem

This begins what will be an ongoing series of posts relating to basic AIX tasks.  The first is a quick explanation of how to add and takeaway space from a filesystem:

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 964 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    3.00    3.00    1%    14    1%    /home

Check the VG for available space.

# lsvg rootvg
VOLUME GROUP:      rootvg                  VG IDENTIFIER:
VG STATE:                    active                    PP SIZE: 512 megabyte(s)
VG PERMISSION:      read/write           TOTAL PPs: 558 (285696 megabytes)
MAX LVs:                       256                        FREE PPs: 454 (232448 megabytes)
LVs:                                    14                         USED PPs: 104 (53248 megabytes)
…

Now, we add space to our filesystem.

# chfs -a size=+5G /home
Filesystem size changed to 16777216

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1     8.00     7.99     1%     14 1%     /home

Alternatively, we could specify the total size of our filesystem.

# chfs -a size=10G /home
Filesystem size changed to 20971520

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    10.00    9.99    1%    14    1%    /home

Be careful when removing space from a filesystem.  It is possible to lose data if the filesystem is close to full, depending on how much you want to decrease by.  This can only be done with JFS2 filesystems.

# chfs -a size=8G /home

or

# chfs -a size=-2G /home

# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 3.00 2.73 10% 11642 2% /
/dev/hd2 4.00 1.90 53% 50028 10% /usr
/dev/hd9var 3.00 1.14 62% 7020 3% /var
/dev/hd3 3.00 1.76 42% 963 1% /tmp
/dev/fwdump 1.00 1.00 1% 4 1% /var/adm/ras/platform
/dev/hd1    8.00    7.99    1%    14    1%    /home

AIX Basics: ‘du’ & ‘df’ – why can’t they just get along?

A right of passage as a unix admin is to be deceived by the ‘du’ and ‘df’ commands.  Every single one of us has tried to clear space in a full file system only to run ‘df’ and be greeted by that ever-present 100%.  What gives?

The answer lies in the way that each command gathers disk allocation information.

‘du’ is a user level program that navigates the file system tree and adds up the blocks allocated to files, directories and links as reported by stat().  ’du’ doesn’t have access to be “aware” of everything, so it doesn’t see metadata.  ’df’ refers to the disk allocation maps and has all necessary accesses.

In every case, ‘df’ is a more reliable indicator of file system allocation status.  Here is an actual case of a common example of why:

# df -g
..
/dev/logicalv     72.00      0.00  100%    13579    96% /filesystem
..

# du -s /filesystem
32671176        /filesystem

# fuser -d /filesystem
/filesystem  123456

Process 123456 has a file(s) open in /filesystem that has been deleted.

Notice that ‘df’ reports a full 72G %used and ‘du’ reports only 32G used. The application had multiple files opened when the files were deleted from the filesystem. Since that space is still not free, ‘df’ correctly reports it as used. ‘du’, however, only has visibility into present file allocations and doesn’t see metadata, so it reports it free.

This conversation can also extend into ensuring files are not currently opened before deleting them during cleanup. This can be done with fuser and is explained in more detail in this fuser post.

‘du’ is perfectly adequate to quickly find the larger files in a file system for cleanup, but, when it comes to accurately reporting the allocation status of a file system, stick with ‘df’.

AIX Basics: fuser

Removing files and unmounting file systems in unix should always be preceded by the ‘fuser’ command. Too many times I see situations where an application has a file or files opened and they are removed from the file system.  This causes even greater disparity between the ‘du’ and ‘df’ commands and confusion in general.

The filesystem will not release file space until the process in question has closed it.  ’df’ reflects this and ‘du’ does not.  The ‘-d’ option will aid in troubleshooting this situation:

# df -g
..
/dev/logicalv     72.00      0.00  100%    13579    96% /filesystem
..

# du -s /filesystem
32671176        /filesystem

# fuser -d /filesystem
/filesystem  123456

In this situation, process 123456 has a deleted file open from /filesystem.  Ideally, the admin would have done the following before deleting a file:

# fuser /filesystem/file
/filesystem/file: 123456

This shows that process 123456 is using the file.  Before deleting the file it needs to be released.

‘fuser’ can also be used to terminate all processes using a filesystem.  This should only be used when permission has been obtained to bring the application down.

# fuser -kxuc /filesystem/file

This will kill all processes using /filesystem/file.