Procedure to mount and unmount NFS filesystems on AIX

1)  Show what is being exported on the source server

     showmount -e

    Note:  If the command above does not show the correct mount points
    that needs to be exported.  You can run the following command to attempt
    to export the filesystems.

     exportfs -a

2)  To unmount the filesystem on the source server that is being NFS on other systems.

     a)  unmount the NFS mount points on the target server.

          umount (filesystems)   target servers
         
     b)  umount the filesystem on the source server once the target servers
         are unmounted.

           umount (filesystems)
          
3)  Mounting NFS mount points on target server.

     a) mount (IP):(mount point)  (mount point)

System dump devices - AIX

Traditionally the default dump device for system dumps was: /dev/hd6 (paging space) and still is on a lot of systems. If there is not enough space to copy over the dump file after a crash, then the system administrator is prompted upon restart to copy the dump file over to some removable media , like a tape or DVD. This can be time consuming and it is sometimes the case that you want to get your system back up quickly. I can sympathise with system administrators who just ignore the prompt to get the system back up due to business pressure, thus deleting the dump, so then one does not know why it crashed in the first place. If you do not have enough space on your dump device to copy the dump, then during the start-up process, the copydumpmenu menu utility is invoked to give the system administrator the opportunity to copy the dump to a removable media, for example to a tape device if present. The copydumpmenu utility can also be called from the command line when the system is up. The copy directory by default is /var/adm/ras with the file-name:vmcore..BZ , where X is a sequence number. The dump file is a BZ (BZIP) and not a Z compressed file format.

The snap command can be used to gather information about the dump file, be-sure to include the -D flag, it gathers the information from the primary dump device.

With systems now having more memory available, this has provided more flexibility as to where the primary dump device could be placed. Typically, for systems with over 4 GB of memory there is now a dedicated dump device, called: lg_dumplv

# lsvg -l rootvg |grep sysdump

 lg_dumplv sysdump 8 8 open/syncd N/A

Using the sysdumpdev command, one can determine what devices are used for the system dumps.

The following output shows a system using AIX 7.1 having the lg_dumplv as its primary dump device:
# sysdumpdev -l 

primary /dev/lg_dumplv 

secondary /dev/sysdumpnull 

copy directory /var/adm/ras 

forced copy flag TRUE 

always allow dump TRUE 

dump compression ON 

type of dump traditional

Looking more closely at the above output fields. Notice that an extra field is now present for AIX 6.1 onwards: type of dump. Currently set to traditional, here you can have it set at (firmware) fw-assisted, if your hardware supports it. For the secondary field, there is no dump device. This is denoted by using the sysdumpnull device. This means all system dumps are lost if it goes to that device. The copy directory is /var/adm/ras, this is where the system dump will be copied to , for either further examination, or to be copied off to go to IBM support. Note that 'always allow dump' is set to true, this must be the case if a dump is to be successfully initiated. Dump compression is on by default.

Common settings using sysdumpdev are:
To change the primary device use: sysdumpdev -P -p
To change the secondary device use: sysdumpdev -P -s
To change the copy directory use: sysdumpdev -D
To change the always dump condition use: sysdumpdev -k for false, sysdumpdev -K for true
To change the type of dump use: sysdumpdev -t


Few Commands:

1. To view the current dump configuration :

# sysdumpdev -l

primary /dev/hd6
secondary /dev/sysdumpnull
copy directory /var/adm/ras
forced copy flag TRUE
always allow dump FALSE
dump compression OFF

2. To change the primary dump device temporarily :

# sysdumpdev -p /dev/dumplv

3. To change the primary dump device permanently :

# sysdumpdev -P -p /dev/dumplv

4. To change the secondary dump device temporarily :

# sysdumpdev -s /dev/dumplv

5. To change the secondary dump device permanently :

# sysdumpdev -P -s /dev/dumplv

6. To set the copy flag :

# sysdumpdev -K

7. To unset the copy flag :

# sysdumpdev -k

8. To estimate the dump size :

# sysdumpdev -e

9. To list the last dump information :

# sysdumpdev -L
Device name: /dev/lg_dumplv
Major device number: 12
Minor device number: 4
Size: 42123543 bytes
Date/Time: Wed Jan 01 12:03:00 CDT 2009
Dump status: 0
dump completed successfully
Dump copy filename: /var/adm/ras/vmcore.1

10. To copy the saved vmcoren file to tape :

# snap -gfkD -o /dev/rmt0

11. To read the dump file :

# crash dump unix
>

12. To change the dump file location and if the copy fails it should ask external media to copy the dump file:

# sysdumpdev -D /opt/dumpfiles

13. To change the dump file location and if the copy fails it should ignore the system dump:

# sysdumpdev -d /opt/dumpfiles

14. To specify the dumps should not be compressed :

# sysdumpdev -c

15. To specify the dumps should be always compress :

# sysdmpdev -C

16. To find out whether a new systemp dump has occured before the last reboot :

# sysdumpdev -z 

The compressed dump is now on the LV lg_dumplv. The dump was not copied across to the copy directory when issuing a user initiated dump. To copy the most recent system dump from a system dump device to a directory, use the savecore command. For example, to copy the dump to the directory /var/adm/ras. I could use:

# savecore -d /var/adm/ras vmcore.0.BZ

If you need to uncompress the file use the dmpuncompress utility. The format of the command is:

dmpuncompress < filename>

After uncompressing, the dump file is now ready for further investigation using kdb or for transfer to IBM support.

# dmpuncompress vmcore.0.BZ
replaced with vmcore.0

Alternatively you can use the smit dump menu option and select,Copy a system dump. The following screen displays:

Copy dump image to: Type or select values in entry fields. Press Enter after making all desired changes. [Entry Fields] * Copy dump image from: [/dev/lg_dumplv] / * Copy dump image to: [/var/adm/ras/dump_fil> * Input and output file blocksize for copy [4096] # Size in bytes of dump image 63894528 Date of last dump Thu Oct 27 18-02-28 B>

The fields are populated with the current dump that is on the primary dump device. This is the default setting, after the copy, the dump file is present in: /var/adm/ras:

# ls -l dump_file_copy.BZ -rw-r--r-- 1 root system 63894528 Oct 27 18:15 dump_file_copy.BZ

After a dump has occurred there may well be a minidump generated as a well. Contained in the errorlog output listing earlier in the article, there was an entry for:

F48137AC 1027180411 U O minidump COMPRESSED MINIMAL DUMP

The minidump is a small compress dump that will be present in: /var/adm/ras. This file contains a snapshot of the system when the system was dumped or crashed. This file can be used for diagnosing if the main dump is not present, due to the dump being removed or not captured.

Using find command

The command find is used to search a given directory for a file or a given expression mentioned in the command. we can also do necessary actions on the output files using xargs

Some important options:

     -xdev                                          Stay on the same file system (dev in fstab).
    -exec cmd {} \;                           Execute the command and replace {} with the full path
    -iname                                        Like -name but is case insensitive
    -ls                                                Display information about the file (like ls -la)
    -size n                                         n is +-n (k M G T P)
    -cmin n                                      File's status was last changed n minutes ago.

find . -type f ! -perm -444	Find files not readable by all
find . -type d ! -perm -111	Find dirs not accessible by all
find /home/user/ -cmin 10 -print	Files created or modified in the last 10 min.
find . -name '*.[ch]' | xargs grep -E 'expr'	Search 'expr' in this dir and below.
find / -name "*.core" | xargs rm	Find core dumps and delete them
find / -name "*.core" -print -exec rm {} \;	Other syntax
find . \( -name "*.png" -o -name "*.jpg" \) -print	iname is not case sensitive
find . -type f -name "*.txt" ! -name README.txt -print	Exclude README.txt files
find /var/ -size +1M -exec ls -lh {} \;
find /var/ -size +1M -ls	Find in /var files above 1M and longlist them
find . -size +10M -size -50M -print
find /usr/ports/ -name work -type d -print -exec rm -rf {} \;	Clean the ports

Find files with SUID; those file have to be kept secure.

Some more Examples:

1 .To list all files in the file system with a given base file name, type: 
find / -name .profile -print

This searches the entire file system and writes the complete path names of all files named .profile.
The / (slash) tells the find command to search the root directory and all of its subdirectories. 
In order not to waste time, it is best to limit the search by specifying the directories where you think the
files might be.

2. To list files having a specific permission code in the current directory tree, type:
find . -perm 0600 -print

This lists the names of the files that have only owner-read and owner-write permission. The . (dot) tells the find command to search the current directory and its subdirectories. See the chmod command for an explanation of permission codes.

3. To search several directories for files with certain permission codes, type: 
find manual clients proposals -perm -0600 -print

This lists the names of the files that have owner-read and owner-write permission and possibly other permissions. The manual, clients, and proposals directories and their subdirectories are searched. In the previous example, -perm 0600 selects only files with permission codes that match 0600 exactly.
In this example, -perm -0600 selects files with permission codes that allow the accesses indicated by 0600 and other accesses above the 0600 level. This also matches the permission codes 0622 and 2744.

4 .To list all files in the current directory that have been changed during the current 24-hour period, type:
find . -ctime 1 -print

5 .To search for regular files with multiple links, type:
find . -type f -links +1 -print

This lists the names of the ordinary files (-type f) that have more than one link (-links +1). Note: Every directory has at least two links: the entry in its parent directory and its own . (dot) entry. The ln command explains multiple file links.

6 . To find all accessible files whose path name contains find, type: 
find . -name '*find*' -print

7. To remove all files named a.out or *.o that have not been accessed for a week and that are not mounted using nfs, type:
find / \( -name a.out -o -name '*.o' \) -atime +7 ! -fstype nfs -exec rm {} \;

Note: The number used within the -atime expression is +7. This is the correct entry if you want the command to act on files not accessed for more than a week (seven 24-hour periods).

8 . To print the path names of all files in or below the current directory, except the directories named SCCS or files in the SCCS directories, type: 
find . -name SCCS -prune -o -print

To print the path names of all files in or below the current directory, including the names of SCCS directories, type:
find . -print -name SCCS -prune

9. To search for all files that are exactly 414 bytes long, type:
find . -size 414c -print

10. To find and remove every file in your home directory with the .c suffix, type:
find /u/arnold -name "*.c" -exec rm {} \;

Every time the find command identifies a file with the .c suffix, the rm command deletes that file. The rm command is the only parameter specified for the -exec expression. The {} (braces) represent the current path name.

11 .In this example, dirlink is a symbolic link to the directory dir. You can list the files in dir by refering to the symbolic link dirlink on the command line. To do this, type:
find -H dirlink -print

12 . In this example, dirlink is a symbolic link to the directory dir. To list the files in dirlink, traversing the file hierarchy under dir including any
symbolic links, type:
find -L dirlink -print

13 . To determine whether the file dir1 referred by the symbolic link dirlink is newer than dir2, type:
find -H dirlink -newer dir2
Note: Because the -H flag is used, time data is collected not from dirlink but instead from dir1, which is found by traversing the symbolic link.

14. To produce a listing of files in the current directory in ls format with expanded user and group name, type : find . -ls -long

15 .To list the files with ACL/EA set in current directory, type:
find . -ea

Sendmail Configuration in AIX

Daemon : sendmail

To start the daemon :

# startsrc -s sendmail -a "-bd -q30m"
where
bd - To start the sendmail as a SMTP mail relay router
q - Is the interval in which the sendmail daemon processes the saved messages

To start the daemon automatically after the system boot:

a. # vi /etc/rc.tcpip

b. Uncomment the below line
start /usr/lib/sendmail "$src_running" "-bd -q${qpi}"

To display the status of the daemon :

# lssrc -s sendmail
# ps -ef | grep sendmail

To stop the daemon :

# stopsrc -s sendmail
# kill -1 `cat /etc/sendmail.pid`

Configuration File:

/etc/sendmail.cf - Where the hostname, Relay server name,... are stored.

Alias File :

/etc/aliases - Where the group(alias) to member mapping is stored.


To Add the hostname in the sendmail configuration :

a. Vi /etc/sendmail.cf

b. Change "#DwYourHostName" to "Dw{hostname of local server}"

c. # refresh -s sendmail


To Add the mail (relay) server in the sendmail configuration :

a. Vi /etc/sendmail.cf

b. Change "#DSrelayhostname" to "DS{hostname of the Relay Server}"

c. # refresh -s sendmail


To send the mails,

# echo "Test Message" | sendmail -v raja@server1.domain.com

If you add any alias in /etc/aliases file, then do the following

# sendmail -bi

This will make the sendmail daemon to re-read the aliases file.

To display the list of messages in the mail queue :

# mailq (or) # sendmail -bp

Directory containing log files and temp files associated with messages in the mail queue :

/var/spool/mqueue

To delete the first 1000 messages in the root's mail queue :

# mail -u root , then enter "d 1-1000"

Submitting system dump testcase to IBM

If you find any critical hardware or software issuess in errpt of a AIX server, you need to call IBM support .

 IBM support contact number - 1-800-426-7378 (1-800-IBM-SERV) 

Check the information about the last recorded system dump (sysdumpdev –L). If this command reports no previous dump was recorded, then you will not be able to submit a system dump testcase. A dump will not be recorded if: 

a) The system was halted or rebooted and did not crash.

b) The system hung but was not reset properly. 

c) The dump device was changed after the crash and the previously recorded dump has been lost.

# sysdumpdev -L
0453-039
Device name: /dev/lg_dumplv
Major device number: 10
Minor device number: 11
Size: 21010432 bytes
Date/Time: Tue Feb 12 13:05:33 CST 2002
Dump status: 0
dump completed successfully

Ensure the Date/Time is the actual time when the system crashed or hung. If the time stamp is old, the dump will not contain any information about the latest outage and there is no need to create a system dump testcase.

Check the dump status: If the status is 0 (Succesful dump) OR -2 (Dump device is too small) OR -3 (Dump crashed or didn't start): Proceed to step to create the system dump testcase.

If the status is -4 (Dump failed due to i/o error): There might not be a system dump if Size is 0 bytes. This usually indicates a bad disk or disk adapter. Check the error report for disk or disk adapter errors. If hardware errors exist, consider also opening a ticket with IBM Hardware Support. Even if the size of the dump is 0 bytes, proceed to step to create the system dump testcase so that AIX Software Support can assist with determining why the dump failed.

If the status is -1 (No dump device is defined): There is no system dump. A dump device should be configured so that a dump can be captured the next time the system crashes.

The snap command will be be used to create the testcase in the file system of your choice. The command creates a number of sub-directories including a directory named dump that will include a compressed copy of the system dump. The -c option on the snap command will create a compressed pax archive of the sub-directories named snap.pax.Z and this is the testcase file that should be ftp'ed to IBM. You will need to find a file system with sufficient space to hold all of the testcase data. In general, the file system you choose should have enough free space to hold about 2.5 times the size of the compressed system dump if the system is using dump compression. Dump compression is always used on AIX 6.1 or higher, but is optional on AIX 5. The sysdumpdev -L command can be used to obtain the size of the system dump to use in determining how much space will be required to hold the testcase. 

AIX 6.1 or higher:

The output from sysdumpdev -L will contain the following line that shows the size of the
compressed system dump.
Size: N (this is the size of the compressed dump in bytes)
Find a file system with at least 2.5 times this value.

AIX 5

If the system is using dump compression, the output from sysdumpdev -L will include
the following two lines:
Size: N (this is the size of the compressed dump in bytes)
Uncompressed: N (this is the size of the uncompressed dump in bytes)
Find a file system with at least 2.5 times the size of the compressed system dump.
If the system is not using dump compression, the output from sysdumpdev -L will
include the following line:
Size: (this is the size of the uncompressed dump in bytes)
In this case, find a file system with at least 1/2 the size of the uncompressed dump.

In the instructions that follow, the file system you choose to contain the testcase will be referred to as /thefs. Replace all occurrences of /thefs with the path to the filesystem you have chosen. If you choose the /tmp filesystem to contain the testcase data, the snap command will automatically create a subdirectory in /tmp named ibmsupt and this is where the testcase data will be stored. The snap.pax.Z file will be located at /tmp/ibmsupt/snap.pax.Z.

If you choose a filesystem other than /tmp, create a directory named ibmsupt in that filesystem and later you will give the snap command the location of this ibmsupt directory with the -d option. The snap.pax.Z file will be located at /thefs/ibmsupt/snap.pax.Z.

# mkdir /thefs/ibmsupt

  Run the snap commmand to generate the testcase data. This will create a file named snap.pax.Z which will be located inside the ibmsupt directory. If your chosen file system is /tmp, run:

# snap -r ; snap -ac

If your chosen file system is not /tmp, run:

# snap -acd /thefs/ibmsupt

 
Ftp the testcase file to IBM.
Rename the snap.pax.Z file to the PMR number using this format:

#####.###.000.snap.pax.Z
For example, if the PMR number is 12345,678 then name the file 12345.678.000.snap.pax.Z
Ftp the file to testcase.software.ibm.com or use the web interface at
https://testcase.boulder.ibm.com/ to upload the file into the directory /toibm/aix.
# cd /thefs/ibmsupt
# ftp testcase.software.ibm.com
user : anonymous
password : your full email address
> cd /toibm/aix
> bin
> hash
> put 12345.678.000.snap.pax.Z
> bye
Note: If a permission denied error is returned from ftp, first make certain that the ftp current directory is /toibm/aix. If it is, then usually a permissions denied error means an attempt is being made to overwrite an existing file and the permissions on the file will not allow this. In this case, try renaming the file to something like this:
12345.678.000.2.snap.pax.Z

 
Verify the upload with your ITS personnel and a quick follow-up with the technician will solve your problem quickly.

AIX Interview Questions :

AIX Interview Questions :

There is a certain amount of overlap between these questions. The interviewer should not ask them all, but ask a sufficient variety of questions to ascertain the applicant’s knowledge of both AIX in general and the interviewer’s operating environment.

Basic Hardware, AIX, and other Applications

Question: What levels of AIX have you worked with?
Answer:
 AIX 4.3.3 and AIX 5.1 good
 If only AIX 4.2.1, 4.1, or 3.2, bad

Question: What types of machines have you worked with?
Answer: Look for questions pertaining to the hardware, to ascertain level of competence and understanding.
 Take note of applicants who mention older technologies like “SP2” or newer technologies like “Regatta”

Question: What applications have you used with AIX?
Answer: Look for applications like HACMP, ADSM/TSM, SAP, i2, Manugistics, MQ Series, WebSphere, etc.
 An applicant with considerable HACMP experience is likely to have quite a bit of AIX knowledge, especially LVM, disks, general TCPIP, and application enablement.


User Management

Question: Suppose that there are some users that need to run certain commands normally only accessible by root. How would you grant them access?
Answer: sudo
Question: Discuss your philosophy on granting user access to root or to an application user ID.
Answer: For root, prefer to 

Base Kernel

Question: The system is performing very slowly; you have discovered that this is due to an increasing number of defunct processes are being created. How can you determine the cause?
Answer:
1. Check to see if new defunct processes are being created, owned by the init process, with no processes being deleted at all.
2. If some defunct processes are being cleared out and the overall number of defunct processes is still growing, then this is due to an application creating more defunct processes than the init process can clean out. 
3. However, if NO defunct processes owned by init (PID 1) are being cleared out, this points to an incorrect wait entry in the /etc/inittab file. The init process is stuck on this wait entry and will not cycle through the defunct processes until that entry finishes. The solution is to change that “wait” entry to a “once” entry.

Question: You get a message that the system can’t fork any additional processes? What do suspect to be the problem?
Answer: Either paging space is full or maxuproc (maximum number of user processes) is set too low.

Question: You’re running some commands and getting an error that says “srcmstr daemon is not running”. Obviously, it is, because the machine appears to be functional. What is the likely cause? 
Answer: Someone has updated some of the base kernel filesets without rebooting the machine.

Question: How do you run 64-bit applications on AIX 4.3.3? On AIX 5.1? What else might you have to consider on an AIX 5 machine?
Answer: 
• AIX 4.3.3 only comes with a 32-bit kernel, so you need to ensure that 64-bit application support is enabled (via the load64bit entry in /etc/inittab or done via SMIT).
• For AIX 5.1, either use the same method as AIX 4.3.3 OR configure the system to use a true 64-bit kernel. If you use a true 64-bit kernel, any 32-bit application will NOT run.
ODM

Question: Describe one situation where it might be appropriate to edit the ODM.
Answer: E.g., need to make parameter changes to a device that is open, and that cannot be modified without closing the device (i.e., a network device), and where the machine can be easily rebooted following the change.

Question: Describe how to manually edit the ODM. What commands should be used?
Answer: 
1. Use the odmget command to grab the appropriate entries from the ODM, into a text file. 
2. Edit these entries. 
3. Use the odmdelete command to delete the current entries.
4. Use the odmadd command to add the new entries.
5. If these are device configuration files, use the savebase command to save them into the boot image.

Question: When might the savebase command be used?
Answer: After making manual changes to the CuAt, CuDv, CuDvDr, CuDep, or CuVPD ODM classes (aka the device configuration database), these classes must be saved into the boot image in the boot logical volume. If they are not saved, there is a risk run that the changes won’t be saved at boot. Note that the chdev command (and others) will cause savebase to be run.

Booting

Question: What key would you use at boot time to signal the machine to boot into service mode?
Answer: F5 or 5 will signal the machine to boot from the internal boot list. F1 or 1 will bring the machine into SMS mode. The F[x] keys are used for graphical consoles; the others are used on ASCII consoles.

Question: What is the effect of setting the normal mode bootlist to cd0 first and hdisk0 second?
Answer: The machine will boot from the CDROM into maintenance mode automatically if a boot CD has been inserted. If not, the machine will boot from the disk into normal mode.

Question: A machine hangs at boot because the network is inaccessible or an NIS server is accessible. How can you recover?
Answer: With inaccessible network (LED 581 for a long time), the machine will generally recover after timing out. With a NIS problem, probably have to boot into service mode

Disk Device Configuration

Question: You are adding a number of new internal SCSI disks to a pSeries 680 and a new drawer of SSA drives. How does cfgmgr operate different when configuring each?
Answer: Internal SCSI drives are discovered one at a time by order of their SCSI ID (i.e., their slot location). SSA drives are discovered by their SSA serial number, which has nothing to do with their slot in the drawer.

Question: After a reboot or after running cfgmgr, a particular disk is listed twice. Give some possible reasons why and how you might narrow them down.
Answer:
1. The disk can be detected, but it has failed and since cfgmgr can’t verify the physical volume identifer, it marks the old disk Defined and then creates a new disk entry with no PVID.
2. You are running in a multi-path (either SAN or twintailed SCSI) environment and the system is (properly) seeing the disk multiple times.
• If one disk is Defined and the other Available, or lspv shows the second instance as having no PVID, then it’s probably #1. Also check the error report.
• If the disk is seen as being Available twice (and lspv lists it twice), it’s probably #2.

Question: What considerations must you give when creating RAID-0 or RAID-1 disks in an SSA array?
Answer: You cannot cable RAID-0 or RAID-1 arrays such that there are multiple adapters in the loop (as opposed to RAID-5).

Question: What’s the difference between RAID-0 and RAID-1?
Answer: RAID-0 is hardware striping. RAID-1 is mirroring.

Question: When using an SSA array, what are the tradeoffs to consider between using a large RAID-5 array and using LVM mirroring?
Answer: RAID-5 would waste fewer disks, but it would be slower.

LVM/JFS

Question: How do you properly mirror rootvg to protect the OS from crashing should a disk fail?
Answer: 
1. Use the mirrorvg command to mirror the logical volumes
2. Run the bosboot command to reboot the boot image and update the disk boot record on both disks that contain a copy of the boot logical volume.
3. Run the bootlist command to add both disks to the boot list.
4. Mirror all non-rootvg paging spaces as well as those in rootvg.
Dump device?
Paging space? 

Question: Why is it important to mirror non-rootvg paging spaces, and what might be the effect if this is not done?
Answer: The OS allocates pages in a round-robin fashion across all paging devices. If a non-rootvg disk containing a paging space fails, the system will likely crash unless it is mirrored. Putting paging spaces on RAID disks is not recommended due to the performance implications.

Question: What kind of considerations can be made with regard to mirroring rootvg in a non-mission critical environment where maximizing disk space is important?
Answer: You might be able to get away with not mirroring any paging spaces, the boot logical volume, or any non-critical file systems. If a disk crashes, you might have to perform some maintenance in service mode.


Question: Provide the steps to move a file system from one volume group to another WITHOUT recreating and restoring it.
Answer:
1. Unmount the file system
2. Use the cplv command to copy the logical volume to the new volume group.
3. Use the chfs command to update the dev and log entries for the file system.
4. Run fsck.

Question: Why is directly editing /etc/filesystems, to change information for a file system, a bad idea?
Answer: The entries from /etc/filesystems are usually stored in the logical volume control block for each logical volume. If you ever had to run importvg for this volume group (i.e., after a reinstall or if using HACMP), the changes would be lost.

Question: You can’t unmount a file system (AIX says it’s in use). How do you determine what process is keeping it open?
Answer: fuser or lsof.

Question: The “df” command shows that the file system is 100% full, but “du” shows no files in the file system. Why, and how do you debug this?
Answer: A running process is holding open a file descriptor that references a large amount of space either not saved as a file or references a file that has been deleted. Use “lsof” or “fuser” to track down this process.

Question: The “df” and “du” commands indicate that the file system is 50% full, but you notice that the total size of all files, as given by “ls” is greater than the size of the file system. How is this possible?
Answer: These are sparse files, where the size of the file is larger than the amount of space it takes up. The file contains a large of “null blocks”.

Question: What steps should you follow to replace a failed hdisk?
Answer:
1. Remove all allocated logical volumes (if not mirrored) or LV copies (if mirrored).
2. Remove disk from the volume group.
3. Remove disk definition (and pdisk, if it’s an SSA drive mapped 1-to-1)
4. Physically replace disk
5. Run cfgmgr.
6. Add disk to volume group and recreate LVs or LV copies.

Network

Question: Basic network configuration
Answer: "smit mktcpip"?

Question: What file is used to tell AIX to use local /etc/hosts entries instead of DNS?
Answer: /etc/netsvc.conf. Specifically, the entry is “hosts=local,bind”.

Question: There is a conflict between a machine’s IP address and the DNS entry. Unfortunately, the DNS server is a Win2K server managed by a little old lady who works once a week. How do you get around this?
Answer: Put an entry in /etc/hosts, and then use /etc/netsvc.conf to force AIX to read /etc/hosts first.

Question: You’ve properly exported a file system on a server. You’ve properly set up the NFS file system definition on the client, and the proper daemons are running. However, when you try to mount it, you get an error:
mount: 1831-009 aixnim not in hosts database
mount: 1831-008 giving up on:
aixnim:/nim
A route to the remote host is not available.

What two options do you have to resolve this?
Answer: 
1. Check /etc/resolv.conf for an entry to the appropriate nameserver.
2. Add it to /etc/hosts.

Question: What is resolv.conf used for?
Answer: To provide a list of nameservers and domains to search for hosts.

Question: Your machines are in the xyz.com domain. You want to be able to look up hosts in the abc.com domain. What should you do so that the ping command will find a route to those hosts?
Answer: 
1. Add the nameserver for that domain to /etc/resolv.conf.
2. Make sure you have a route to those nameservers.

Question: Where is the default gateway stored in AIX? Where are other routes stored?
Answer: As ODM entries in the CuAt ODM class.

SMIT

Question: How would you use SMIT to create scripts for future use?
Answer: Use the script.script file for commands that have been executed. Or, use the -x flag with smit.

Question: You’re using SMIT and are getting a strange error during its execution. How do you debug what SMIT is doing?
Answer: Use the -D flag on SMIT, and look at the smit.log file for the AIX commands being executed by SMIT and the error messages being generated.

Performance

Question: A project manager claims that a system is CPU-bound. What commands do you run to verify this, and what sort of output are you looking for? 
Answer: 
1. Run vmstat and look at the wait, idle, CPU utilization, runq and blockq parameters. 
2. A CPU-bound system will exhibit relatively low wait and idle percentages and a high degree of CPU usage. 
3. In addition, the runq parameter will average the number of CPUs on the system and the blockq parameter will be, on the average, high.

Question: You have a 32-way Regatta. Why is vmstat not an accurate tool to measure CPU utilization for each processor?
Answer: vmstat takes a system average. The sar command can show the CPU utilization for a particular processor.

Question: If you have Workload Manager policies defined for several classes of processes, and the machine is under little load, what happens?
Answer: Nothing. WLM policies are not enforced until there is contention for resources.

Installation Issues

Question: What is the difference between a migration install, a preservation install, or an overwrite install?
Answer: 
• A overwrite install recreates rootvg and the file systems and installs everything from scratch. Other non-OS file systems are deleted.
• A preservation install preserves rootvg. It only overwrites the base OS (to the base level from the installation image) and thus only affects /, /usr, /var, and /tmp. Your own modified config files are generally saved to a special location and can be recovered.
• A migration install is used to take an existing AIX machine to a new AIX level (i.e., 4.3 to 5.1).

Question: You need to update an AIX 4.3.2 machine to AIX 4.3.3. How do you do this?
Answer: Take the AIX 4.3.3 CDs and perform an “update_all”.

Question: Define under what circumstances two different versions of a product can coexist on the same machine in AIX.
Answer:
1. The names of the filesets for that product are different when stored in the ODM.
2. The product does not store names in the ODM.

Question: What considerations do you need to make when cloning system X to system Y?
Answer: Make sure that all the drivers needed for system Y and its associated hardware are included in the image for system X.

Question: A user wants you to install some set of commands on system X; these commands already exist on system Y. How would you tell which filesets to install?
Answer: Run “lslpp -w ” to see what fileset provides that command.

Question: You’ve installed a new fileset. How do you tell which files are provided?
Answer: Run “lslpp -f ” to see what files are installed by that fileset.

AIX Patches and Maintenance

Question: Discuss your strategy for applying and committing maintenance
Answer:
• A prudent administrator will generally apply fixes first if those fixes have never been used before in the environment.
• After an evaluation period, the fixes can be committed.
• Subsequent installations of those fixes on other machines can then be committed.
• If these fixes are related to the base OS, a mksysb should be created first.

Question: What are the AIX Maintenance Levels and how do they differ from normal fixes?
Answer: AIX Maintenance Levels are similar to an NT service pack. They are generally considered safe to install “all at once”. Normal AIX fix collections may or may not contain fixes that could interfere with each other; however, maintenance levels are generally considered safe.

Question: Describe the difference between a PTF, and APAR, and a maintenance level.
Answer: An APAR is a specific patch that may update one or more filesets. A PTF is an IBM term for a collection of APARs commonly shipped together as a common fileset update. A maintenance level is a collection of APARs (also generally ordered as an APAR).

Question: Your machine is running AIX 4.3.3 with maintenance level 10 applied. “oslevel” (no option) reports “4.3.2.0”. Why?
Answer: There are some filesets installed that are at a level BELOW what is defined for 4.3.3.

Question: What is the difference between applying and committing an APAR?
Answer: Applying it saves the old versions of the files so that you can back off the new version. Committing the APAR removes the old versions.

Question: How do you back off a committed set of patches?
Answer: You have to forcibly install the base level of the affected filesets, while NOT reinstalling any prerequisites, and then re-patch back to the appropriate level.



Question: You’re installing a new IBM 6228 fiber card into an existing 4.3.3 system for the first time. What problems might you encouter?
Answer: The AIX 4.3.3 CDs don’t provide any base support for IBM 6228 fiber cards. You must download the drivers in the form of an APAR and install them separately. AIX 5.1 provide fiber support in the base CDs.

Misc. / Other Products

Question: What TERM setting would you use on an IBM 3153 monitor?
Answer: ibm3151

Question: List some common sites to download freeware tools for AIX?
Answer: aixpdslib.seas.ucla.edu
purdue.edu
www.bullfreeware.com
IBM repository of Linux freeware for AIX

Question: A group of developers wants to know if they will be able to install different versions of the Java runtime on a machine. Specifically, they want to install the base level filesets for versions 1.2.2, 1.3.0, and 1.3.1, as well as a separate set of patches. What do you tell them?
Answer: JDK/JRE 1.2.2, 1.3.0, and 1.3.1 can all coexist on a machine, as they are completely separate sets of filesets. However, patches for these versions cannot coexist with the base levels.

Question: Versions 3.6.4 and 3.6.6 of the IBM CSet++ compiler are comprised of the same basic set of fileset names, meaning that they normally cannot be installed on the same machine at the same time. What solution (unsupported by IBM) do you suggest? What considerations must you give to this environment?
Answer: Install the second set of software on a second machine. Take all the files installed by those filesets and copy them to the other machine into another directory. The ODM will not recognize that the second set of filesets are installed, but the product will still be functional. Note that this is unsupported by IBM, but there’s a fine line between “unsupported” and “functional”. Note that you can’t very easily apply patches to this environment.
Question: What does the LIBPATH variable do?
Answer: It controls the order in which libraries will be discovered so as to arbitrate between libraries with the same name.

Question: When IBM some particular sequence of commands or environment is “not supported”, what do they mean?
Answer: That it may work, but that won’t SUPPORT it because they haven’t TESTED it.

HACMP interview questions

a. What characters should a hostname contain for HACMP configuration?
The hostname cannot have following characters: -, _, * or other special characters.

b. Can Service IP and Boot IP be in same subnet?
No. The service IP address and Boot IP address cannot be in same subnet. This is the basic requirement for HACMP cluster configuration. The verification process does not allow the IP addresses to be in same subnet and cluster will not start.

c. Can multiple Service IP addresses be configured on single Ethernet cards?
Yes. Using SMIT menu, it can be configured to have multiple Service IP addresses running on single Ethernet card. It only requires selecting same network name for specific Service IP addresses in SMIT menu.

d. What happens when a NIC having Service IP goes down?
When a NIC card running the Service IP address goes down, the HACMP detects the failure and fails over the service IP address to available standby NIC on same node or to another node in the cluster.

e. Can Multiple Oracle Database instances be configured on single node of HACMP cluster?
Yes. Multiple Database instances can be configured on single node of HACMP cluster. For this one needs to have separate Service IP addresses over which the listeners for every Oracle Database will run. Hence one can have separate Resource groups which will own each Oracle instance. This configuration will be useful if there is a failure of single Oracle Database instance on one node to be failed over to another node without disturbing other running Oracle instances.

f. Can HACMP be configured in Active - Passive configuration?
Yes. For Active - In Passive cluster configuration, do not configure any Service IP on the passive node. Also for all the resource groups on the Active node please specify the passive node as the next node in the priority to take over in the event of failure of active node.

g. Can file system mounted over NFS protocol be used for Disk Heartbeat?
No. The Volume mounted over NFS protocol is a file system for AIX, and since disk device is required for Enhanced concurrent capable volume group for disk heartbeat the NFS file system cannot be used for configuring the disk heartbeat. One needs to provide disk device to AIX hosts over FCP or iSCSI protocol.

h. Which are the HACMP log files available for troubleshooting?

Following are log files which can be used for troubleshooting:
1. /var/hacmp/clverify/current//* contains logs from current execution of cluster verification.
2. /var/hacmp/clverify/pass//* contains logs from the last time verification passed.
3. /var/hacmp/clverify/fail//* contains logs from the last time verification failed.
4. /tmp/hacmp.out file records the output generated by the event scripts of HACMP as they execute.
5. /tmp/clstmgr.debug file contains time-stamped messages generated by HACMP clstrmgrES activity.
6. /tmp/cspoc.log file contains messages generated by HACMP C-SPOC commands.
7. /usr/es/adm/cluster.log file is the main HACMP log file. HACMP error messages and messages about HACMP related events are appended to this log.
8. /var/adm/clavan.log file keeps track of when each application that is managed by HACMP is started or stopped and when the node stops on which an application is running.
9. /var/hacmp/clcomd/clcomd.log file contains messages generated by HACMP cluster communication daemon.
10. /var/ha/log/grpsvcs. file tracks the execution of internal activities of the grpsvcs daemon.
11. /var/ha/log/topsvcs. file tracks the execution of internal activities of the topsvcs daemon.
12. /var/ha/log/grpglsm file tracks the execution of internal activities of grpglsm daemon.