Performance Monitoring and Tuning in AIX

Performance Monitoring and Tuning in AIX
Performance Monitoring :

1. How to find out the system-wide memory usage ?
# svmon -G -i 2 5

2. How to list top 10 memory consuming processes ?

You can use any of the below commands
# svmon -Put 10
# ps aux head -1; ps aux sort -rn +3 head

3. How to list top 10 cpu consuming processes ?

# ps aux head -1; ps aux sort -rn +2 head -10

4. What is the best command for general performance monitoring :

# topas

You can even use 'jtopas', which is a java based system monitoring tool.

5. How to start trace for the entire system ?

# trace -a

6. How to stop trace ?

# trcstop

7. Where is the log file for trace tool located ?

/var/adm/ras/trcfile

8. What is the command used to generate trace report from a trace log file ?

# trcrpt

9. How to generate report on utilization statistics related to an LPAR ?

# lparstat

10. How to display the LPAR configuration report ?

# lpstat -i

11. What are the mostly used commands to find the cpu, memory,disk i/o statistics ?

# sar -> CPU, Memory statistics
# vmstat -> CPU, Memory statistics
# iostat -> CPU, Disk I/O satistics
# topas -> CPU, Memory, Network and Disk I/O statistics
# ps aux -> CPU, Memory statistics

12. How to display processes related to a specific user ?

# ps -fu username

13. How to list all the 64bit processes running in a system ?

# ps -efM

14. How to enable Interface Specific Network Options in AIX ?
# no -o use_isno=1

By enabling use_isno option, you can set buffer settings on a specific interface, giving you better control over performance management of network interfaces.

15. What is 'thewall' and how to set ?
'thewall' in AIX defines the upper limit for network kernel buffers.

When running AIX 5L V5.3 running a 32 bit kernel is 1GB or half the size of real memory depending on which of the two is the smallest. If you have AIX 5L V5.3 running a 64bit kernel the size of thewall will be 65GB or half the size of real memory, depending on which of the two is smaller.
To display the size of the the wall,

# no -o thewall

Note:
the size of thewall is static from AIX 5L Version 5.1 and later, and cannot be changed, to reduce the upper limit of memory used for networking make use of the maxmbuf tunable.

16. What is maxbuf tunable variable and how to set it ?

The maxmbuf tunable used by AIX specifies the maximum amount of memory that can be used by the networking subsystem.

It can displayed by using the below command,
# lsattr -El sys0 -a maxbuf

By default the maxmbuf tunable is disabled, it is set to 0, this means that the value of thewall will be used to define the maximum amount of memory used for network communications. By setting a non zero value to maxmbuf will override the value of thewall. This is the only way of reducing the value set by thewall.

The value of maxbuf's is defined by 1Kb units. To set its value to 1GB,
# chdev -l sys0 -a maxmbuf=1000000

17. How to find out the media speed of a network interface ?

# netstat -v ent0 grep Media

18. How to view the statistics for a specific network adapter ?

# entstat -d ent1

You can also use "netstat -v ent1".

19. How to reset the above network statistics ?

# entstat -r ent1

20. How to start iptrace on a specific network adapter ?

To Start :
# startsrc -s iptrace -a "-i en0 iptrc.out" &

To Stop:
# stopsrc -s iptrace

21. How to generate report from the iptrace's output file ?

# ipreport -r -s iptrc.out > ipreport

22. How to get the NFS statistics ?

NFS server RPC statistics : # nfsstat -sr
NFS server NFS statistics : # nfsstat -sn
NFS client RPC statistics : # netstat -cr
NFS client NFS statistics : #netstat -cn
Statistics on mounted file systems : # nfsstat -m
To reset the nfsstat statistics : # nfsstat -z

23. How to list the current values of all the network tunables?

# no -a

24. How to display the current value of a specific network tunable?

# no -o tcp_recvspace

25. How to display all the values (current, default, boot, min, max..) values of a network tunable ?

# no -L tcp_recvspace

26. What is the file that holds the next boot tunables's values ?

/etc/tunables/nextboot

27. What is the file that automatically generated with all the values of the network tunables that were set immediately after the reboot ?

/etc/tunables/lastboot

28. How to change the current value of a network tunable's value as well as add the entry to the /etc/tunables/nextboot file ?

Use the 'p' flag in the no command.

For Ex., # no -p -o tcp_recvspace=16k

29. How to display all the NFS network variables ?

# nfs -a

30. How to enable the collection of disk input/output statistics ?

# chdev -l sys0 -a iostat=true

31. How to display the 5 busiest logical volumes in a VG ?

# lvmstat -v datavg -c 5

32. How to display, enable and disable the statistics collection for a VG ?

To enable: # lvmstat -v datavg -e
To disable: # lvmstat -v datavg -d
To show : # lvmstat -v datavg

33. How to display the statistics for a LV ?

# lvmstat -l lv001

34. How to report disk statistics ?

# sar -d 5 60
Abovc command displays the disk i/o statistics 60 times in 5 sec interval.

35. How to list top 10 real memory consuming processes ?

# svmon -Put 10

36. How to list top 10 paging space consuming processes ?

# svmon -Pgt 10

37. How to list the files opened by a process ?

# svmon -pP Process_id

38. How to find out the memory usage of a specific process ?

# svmon -wP

39. How to display the paging (swap) usage ?

# swap -s allocated = 4718592 blocks used = 1475527 blocks free = 3243065 blocks

#swap -l
device maj,min total free
/dev/paging02 38, 4 4608MB 3166MB
/dev/paging01 38, 3 4608MB 3168MB
/dev/paging00 10, 14 4608MB 3167MB
/dev/hd6 10, 2 4608MB 3167MB

NIM

NIM
Required Filesets:

For Server - bos.sysmgt.nim.master and bos.sysmgt.nim.spot
For Client - bos.sysmgt.nim.client

Few Resource Definitions:

SPOT - Shared Product Object Tree is a directory containing files required to boot a machine and the boot image

LPP_SOURCE - Licensed Program Product source is a directory containing images/filesets that AIX uses to load software

MKSYSB - Mksysb resource used to build a machine

Requirements for NIM Server:

Disk Space :
1. 3 GB per base lpp_source resource
2. 500 MB + per mksysb resource
3. 500 MB per SPOT resource
4. Additional buffer space for future growth

Other Requirements:
# Minimum 512 MB real memory
# 10 or 100 MBPS ethernet adapter

My Recommendations for NIM VG and Filesystems :

1. Create a seperate VG called 'nimvg' with enough space.

2. Create the following filesystems in nimvg based upon your requirement

a. /tftpboot - To hold boot images
b. /export/nim - To hold the resources like SPOT, LPP, Mksysb

Directory Structure :
/export/nim/lpp_source - To hold lpp source resources
/export/nim/spot - To hold spot resources
/export/nim/mksysb - To hold the mksysb backup for clients

Naming Schemes:

Follow the below schemes to easily identify during regular operations :

spot530TL6 - SPOT for AIX V 5.3 TL 6
spot530TL9 - SPOT for AIX V 5.3 TL 9
lpp_source530TL6 - LPP_SOURCE for AIX V 5.3 TL 6
lpp_source530TL9 - LPP_SOURCE for AIX V 5.3 TL 6
client_server1 - Mksysb image of the host server1
client_server2 - Mksysb image of the hsot server2


How to setup the NIM Master :

0. Create the /tftpboot and /export/nim file systems as per yoru requirement

1. Initial setup of NIM Master
a. ODM database
b. Boot Area: /tftpboot directory that is used to store boot files (images)
c. /etc/niminfo - Is the Key configuration file that exists on both master and clients
d. nimesis daemon - This is the daemon which used to communicate with the nim clients

2. Insert the AIX CD into the master server's CD Drive

3. Create LPP_SOURCE and SPOT resources

Commands to manage NIM master and clients:

To setup NIM Server:
# nim_master_setup -B -a device=/dev/cd0 -a file_system=/nim -a volume_group=nimvg

To setup NIM installation in a client:
# smitty nim_bosinst

To view the status of NIM installation in a NIM client:
# lsnim -l client_hostname

To define a lpp_source resource:
# nim -o define -t lpp_source -a source=/dev/cd0 -a server=master -a location=/nim/lpp_source/AIX_5_3_4 AIX_5_3_4

To define a spot resource:
# nim -o define -t spot -a server=master -a location=/export/nim/spot -a source=lpp_source530 spot530
To remove a resource:
# nim -o remove AIX_5_3_4

To initialize a NIM client for diag operation:
# nim -o diag client_hostname

To initialize a NIM client for maintenance operation:
# nim -o maint client_hostname

To unconfigure a NIM server:
# nim -o unconfig master_server

To allocate a SPOT to a NIM client:
# nim -o allocate -a spot=AIX_5_3 client_hostname

To deallocate a SPOT from a NIM client:
# nim -o deallocate -a spot=AIX_5_3 client_hostname

To remove a NIM client after deallocating all its resources:
# nim -o remove client_hostname

To reboot a client:
# nim -o reboot client_hostname

To list all the NIM resources:
# lsnim

To list detailed information about a nim client:
# lsnim -l client_hostname

To list the resources allocated to a NIM client:
# lsnim -c resources client_hostname

NIM
To list all the mksysb resources
# lsnim -t mksysb
# lsnim -t spot
To list all the machines
# lsnim -t standalone
OR
# lsnim -c machines
To reset the NIM status of system host-10
# nim -o reset -a force=yes host-10
or
# nim -Fo reset host10
To Force Deallocate all the resources from system host-01
# nim -Fo deallocate -a subclass=all host-01
To rebuild the /etc/niminfo file in the master
# nimconfig -r
To rebuild the /etc/niminof file in the NIM client
# niminit -a master= -a name=
To remove a machine from the NIM environment
# nim -o remove
To define a mksysb resource
# nim -o define -t mksysb -a server=master -a location=

To enable Base OS install in a client using mksysb resource
# nim -o bos_inst -a source=mksysb -a spot=spot_53ML4 -a accept_licenses=yes
-a mksysb=
To add additional software to lppsource1
# nim -o update -a packages=all source=/dev/cd0 lppsource1
To update the spot and lppsource to the latest level
# nim_update_all -l -s -d -u -B



Some useful NIM SMIT Fast paths
________________________________________
smit nim_mkmac # Adding new machines
smit nim_bosinst # For doing BOS install operation on a machine
Common NIM Error codes
________________________________________
608 - tftp retrieve of client info file failure
Action If a 608 hang is encountered, verify that the ClientName.info file exists in the /tftpboot directory. If it does not exist, retry the NIM operation to create it. If it does exist, verify that tftp access to the /tftpboot directory is not restricted in the /etc/tftpaccess.ctl file. is also possible that the network adapter was not configured properly in the boot environment
611 - Remote mount of NFS file system failure
Action 611 hangs occur when the client machine is unable to mount a resource from a server. Ensure that NFS is running on the resource server. Verify that the resources specified for the operation are exported properly by checking the /etc/exports and /etc/xtab files on the server. Also, confirm that the resources have permissions set correctly for reading.
613 - Failure setting up route tables
Action 613 hangs usually occur because a route is incorrectly defined for a network in the NIM database. Verify that the correct gateways are specified between networks, and all gateways are functional. Use debug-enabled network boot images to determine which routes could not be defined.