Performance Monitoring

Performance Monitoring :

1. How to find out the system-wide memory usage ?
# svmon -G -i 2 5

2. How to list top 10 memory consuming processes ?

You can use any of the below commands
# svmon -Put 10
# ps aux head -1; ps aux sort -rn +3 head

3. How to list top 10 cpu consuming processes ?

# ps aux head -1; ps aux sort -rn +2 head -10

4. What is the best command for general performance monitoring :

# topas

You can even use 'jtopas', which is a java based system monitoring tool.

5. How to start trace for the entire system ?

# trace -a

6. How to stop trace ?

# trcstop

7. Where is the log file for trace tool located ?

/var/adm/ras/trcfile

8. What is the command used to generate trace report from a trace log file ?

# trcrpt

9. How to generate report on utilization statistics related to an LPAR ?

# lparstat

10. How to display the LPAR configuration report ?

# lpstat -i

11. What are the mostly used commands to find the cpu, memory,disk i/o statistics ?

# sar -> CPU, Memory statistics
# vmstat -> CPU, Memory statistics
# iostat -> CPU, Disk I/O satistics
# topas -> CPU, Memory, Network and Disk I/O statistics
# ps aux -> CPU, Memory statistics

12. How to display processes related to a specific user ?

# ps -fu username

13. How to list all the 64bit processes running in a system ?

# ps -efM

14. How to enable Interface Specific Network Options in AIX ?
# no -o use_isno=1

By enabling use_isno option, you can set buffer settings on a specific interface, giving you better control over performance management of network interfaces.

15. What is 'thewall' and how to set ?
'thewall' in AIX defines the upper limit for network kernel buffers.

When running AIX 5L V5.3 running a 32 bit kernel is 1GB or half the size of real memory depending on which of the two is the smallest. If you have AIX 5L V5.3 running a 64bit kernel the size of thewall will be 65GB or half the size of real memory, depending on which of the two is smaller.
To display the size of the thewall,

# no -o thewall

Note:
the size of thewall is static from AIX 5L Version 5.1 and later, and cannot be changed, to reduce the upper limit of memory used for networking make use of the maxmbuf tunable.

16. What is maxbuf tunable variable and how to set it ?

The maxmbuf tunable used by AIX specifies the maximum amount of memory that can be used by the networking subsystem.

It can displayed by using the below command,
# lsattr -El sys0 -a maxbuf

By default the maxmbuf tunable is disabled, it is set to 0, this means that the value of thewall will be used to define the maximum amount of memory used for network communications. By setting a non zero value to maxmbuf will override the value of thewall. This is the only way of reducing the value set by thewall.

The value of maxbuf's is defined by 1Kb units. To set its value to 1GB,
# chdev -l sys0 -a maxmbuf=1000000

17. How to find out the media speed of a network interface ?

# netstat -v ent0 grep Media

18. How to view the statistics for a specific network adapter ?

# entstat -d ent1

You can also use "netstat -v ent1".

19. How to reset the above network statistics ?

# entstat -r ent1

20. How to start iptrace on a specific network adapter ?

To Start :
# startsrc -s iptrace -a "-i en0 iptrc.out" &

To Stop:
# stopsrc -s iptrace

21. How to generate report from the iptrace's output file ?

# ipreport -r -s iptrc.out > ipreport

22. How to get the NFS statistics ?

NFS server RPC statistics : # nfsstat -sr
NFS server NFS statistics : # nfsstat -sn
NFS client RPC statistics : # netstat -cr
NFS client NFS statistics : #netstat -cn
Statistics on mounted file systems : # nfsstat -m
To reset the nfsstat statistics : # nfsstat -z

23. How to list the current values of all the network tunables?

# no -a

24. How to display the current value of a specific network tunable?

# no -o tcp_recvspace

25. How to display all the values (current, default, boot, min, max..) values of a network tunable ?

# no -L tcp_recvspace

26. What is the file that holds the next boot tunables's values ?

/etc/tunables/nextboot

27. What is the file that automatically generated with all the values of the network tunables that were set immediately after the reboot ?

/etc/tunables/lastboot

28. How to change the current value of a network tunable's value as well as add the entry to the /etc/tunables/nextboot file ?

Use the 'p' flag in the no command.

For Ex., # no -p -o tcp_recvspace=16k

29. How to display all the NFS network variables ?

# nfs -a

30. How to enable the collection of disk input/output statistics ?

# chdev -l sys0 -a iostat=true

31. How to display the 5 busiest logical volumes in a VG ?

# lvmstat -v datavg -c 5

32. How to display, enable and disable the statistics collection for a VG ?

To enable: # lvmstat -v datavg -e
To disable: # lvmstat -v datavg -d
To show : # lvmstat -v datavg

33. How to display the statistics for a LV ?

# lvmstat -l lv001

34. How to report disk statistics ?

# sar -d 5 60
Abovc command displays the disk i/o statistics 60 times in 5 sec interval.

35. How to list top 10 real memory consuming processes ?

# svmon -Put 10

36. How to list top 10 paging space consuming processes ?

# svmon -Pgt 10

37. How to list the files opened by a process ?

# svmon -pP Process_id

38. How to find out the memory usage of a specific process ?

# svmon -wP

39. How to display the paging (swap) usage ?

# swap -s allocated = 4718592 blocks used = 1475527 blocks free = 3243065 blocks

#swap -l
device maj,min total free
/dev/paging02 38, 4 4608MB 3166MB
/dev/paging01 38, 3 4608MB 3168MB
/dev/paging00 10, 14 4608MB 3167MB
/dev/hd6 10, 2 4608MB 3167MB

Paging Space (also called Swap space)

Memory hierarchy

The instructions and data that the CPU processes are taken from memory. Memory comes in several layers.
Registers - the top layer, it is high speed storage cells (can conatin 32-64 bit data)

Caches - If data can not be found in registers it will be looked in  the next level, which is cace
         L1 cache the fastes an smallest (usually on CPU chip) 32-256 KB
         L2 cache, if the needed data not in L1, CPU is trying to find it in L2, it can be megabytes in size
RAM - If the needed data not in the hardware caches then TLB (Translation Lookaside Buffer) will be checked, after the RAM
         TLB - cache of recently accessed addresses

Disk - If the address is not in RAM, then a page fault occurs and the data is retrieved from the hard disk.
         A page fault is a request to load a 4KB data page from disk.

The way demand paging works is that the kernel only loads a few pages at a time into real memory. When the CPU is ready for another page, it looks at the RAM. If it cannot find it there, a page fault occurs, and this signals the kernel to bring more pages into RAM from disk.

If the CPU is waiting data from real memory, the CPU is still considered as being in busy state. If data is needed from disk then CPU is in I/O wait state.
   

Paging Space (also called Swap space)

The RAM and the paging space are divided into 4 KB sections called page frames. (A page is a unit of virtual memory that holds 4 KB of data.) When the system needs more RAM, page frames of information are moved out of RAM and onto the hard disk. This is called paging out. When those page frames of information are needed again, they are taken from the hard disk and moved back into the RAM. This is called paging in.

When the system spends more time shuffling page frames in and out of RAM instead of doing useful work, the system is thrashing.

When the amount of available paging space falls below a threshold, called the paging space warning level, the system sends all the processes (except the kernel processes) a SIGDANGER signal. This signal tells the processes to terminate gracefully. When the amount of empty paging space falls further below a second threshold, called the paging space kill level, the system sends a SIGKILL signal to processes that are using the most paging space. (terminate nongracefully)

When AIX is installed, it automatically creates paging space on the installation disk, which is usually the hard disk hdisk0. The name of this paging space is always hd6. The file /etc/swapspaces contains a list of the paging space areas that will be activated at system startup.

swapon is a term from the days before page frames were used. At that time, around 1982, AIX swapped entire programs out of RAM and onto the hard disk. Today, a portion of the program is left in RAM, and the rest is paged out of the program onto the hard disk. The term swapon stuck, so today, we sometimes refer to paging out and paging in as swapping

Once you page out a computation page, it continues to take up space on the paging file as long as the process exist, even if the page is subsequently paged back in. In general you should avoid paging at all.

How much paging space do you need on your system? What is the rule of thumb?
Database administrators usually like to request the highest number of everything and might instruct you to double the amount of paging space as your RAM (the old rule of thumb). Generally speaking, if my system has greater than 4GB of RAM, I usually like to create a one-to-one ratio of paging space versus RAM. Monitor your system frequently after going live. If you see that you are never really approaching 50 percent of paging space utilization, don't add the space.

The number and types of applications will dictate the amount of paging space needed. Many sizing “rules of thumb” have been published, but the only way to correctly size your machine's paging space is to monitor the amount of paging activity.

Tips for paging space:
- Only 1 paging space per disk
- Use disks with the least activity
- Paging spcaces should the same size
- Do not extend a paging spcae to multiple PV's

Ideally, there should be several paging spaces of equal size each on different physical volumes. The paging space is allocated in a round robin manner and will use all paging areas equally. If you have two paging areas on one disk, then you are no longer spreading the activity across several disks.Because of the round robin technique that is used, if they are not the same size, then the paging space usage will not be balanced.


bootinfo -r                    displays the real memory in kilobytes (this also works: lsattr -El sys0 -a realmem)
lscfg -vp |grep -p DIMM        displays the memory DIMM


lsattr -El sys0 -a realmem     (list attributes) see how much real memory you have
ps aux | sort +4 -n            lists how much mem is used by the processes
svmon -P | grep -p        you can see how much paging spce a process is using
svmon -P -O sortseg=pgsp       shows paging space usage of processes

mkps -s 4 -n -a rootvg hdisk0  creates a paging space (give the name automatically:paging00)
            -n                 activates it immediately,
            -a                 it will be activated at next restart as well (adds it to /etc/swapspaces)
            -s                 size 4 lp
lsps -a                        list all paging spaces and the usage of a paging space
lsps -s                        summary of all paging spaces combined (all the paging spaces are added together)
chps -s 3 hd6                  dynamically increase the size of a paging space with 3 lps
chps -d 1 paging00             dynamically decrease the size of a paging space with 1 lp (it will create a temporary paging space)

/etc/swapspaces                contains a list of the paging space areas
vmstat -s

smitty mkps                    adding paging space
smitty chps                    changing paging space
swapon /dev/paging02           dynamically activate, or bring online, a paging space (or smitty pgsp)
swapoff /dev/paging03          deactivate a paging space

------------------------------

removing a paging space:
    swapoff /dev/paging03      deactivate a paging space (the /dev is needed)
    rmps paging03              removes a paging space (the /dev is not needed)

------------------------------

For flushing the paging space:
(it shows high percentage, but actually nothing is using it)
1. chps -d 1 hd6               it will decrease the size of the paging spave by 1 lp (it will create a temp. paging space, copy the conntent...)
                               (if not enough space in the vg, it will not do that)
2. chps -s 1 hd6               increase paging space to its original size

------------------------------

Fork:
When there is a message regarding cannot fork... it is probably caused by low paging space

When a process calls fork(), the operating system creates a child process of the calling process.
The child process created by fork() is a sort of replica of the calling process. Some server processes, or daemons, call fork() a few times to create more than one instance of themselves. An example of this is a web server that pre-forks so it can handle a certain number of incoming connections without having to fork() the moment they arrive.

CHANGING Normal VG TO Big VG & Big VG TO Scalable VG

CHANGING Normal VG TO Big VG & Big VG TO Scalable VG

Changing Normal VG to Big VG:

If you reached the MAX PV limit of a Normal VG and playing with the factor (chvg -t) is not possible anymore you can convert it to Big VG.
It is an online activity, but there must be free PPs on each physical volume, because VGDA will be expanded on all disks:

root@bb_lpar: / # lsvg -p bbvg
bbvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            511         2           02..00..00..00..00
hdisk3            active            511         23          00..00..00..00..23
hdisk4            active            1023        0           00..00..00..00..00

root@bb_lpar: / # chvg -B bbvg
0516-1214 chvg: Not enough free physical partitions exist on hdisk4 for the
        expansion of the volume group descriptor area.  Migrate/reorganize to free up
        2 partitions and run chvg again.

In this case we have to migrate 2 PPs from hdisk4 to hdsik3 (so 2 PPs will be freed up on hdisk4):

root@bb_lpar: / # lspv -M hdisk4
hdisk4:1        bblv:920
hdisk4:2        bblv:921
hdisk4:3        bblv:922
hdisk4:4        bblv:923
hdisk4:5        bblv:924
...

root@bb_lpar: / # lspv -M hdisk3
hdisk3:484      bblv:3040
hdisk3:485      bblv:3041
hdisk3:486      bblv:3042
hdisk3:487      bblv:1
hdisk3:488      bblv:2
hdisk3:489-511

root@bb_lpar: / # migratelp bblv/920 hdisk3/489
root@bb_lpar: / # migratelp bblv/921 hdisk3/490

root@bb_lpar: / # lsvg -p bbvg
bbvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            511         2           02..00..00..00..00
hdisk3            active            511         21          00..00..00..00..21
hdisk4            active            1023        2           02..00..00..00..00

If we try again changing to Big VG, now it is successful:
root@bb_lpar: / # chvg -B bbvg
0516-1216 chvg: Physical partitions are being migrated for volume group
        descriptor area expansion.  Please wait.
0516-1164 chvg: Volume group bbvg2 changed.  With given characteristics bbvg2
        can include up to 128 physical volumes with 1016 physical partitions each.

If you check again, freed up PPs has been used:
root@bb_lpar: / # lsvg -p bbvg
bbvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            509         0           00..00..00..00..00
hdisk3            active            509         17          00..00..00..00..17
hdisk4            active            1021        0           00..00..00..00..00

---------------------------------------

Changing Normal (or Big) VG to Scalable VG:

If you reached the MAX PV limit of a Normal or a Big VG and playing with the factor (chvg -t) is not possible anymore you can convert that VG to Scalable VG. A Scalable VG allows a maximum of 1024 PVs and 4096 LVs and a very big advantage that the maximum number of PPs applies to the entire VG and is no longer defined on a per disk basis.

!!!Converting to Scalable VG is an offline activity (varyoffvg), and there must be free PPs on each physical volume, because VGDA will be expanded on all disks.

root@bb_lpar: / # chvg -G bbvg
0516-1707 chvg: The volume group must be varied off during conversion to
        scalable volume group format.

root@bb_lpar: / # varyoffvg bbvg
root@bb_lpar: / # chvg -G bbvg
0516-1214 chvg: Not enough free physical partitions exist on hdisk2 for the
        expansion of the volume group descriptor area.  Migrate/reorganize to free up
        18 partitions and run chvg again.


After migrating some lps to free up required PPs (in this case it was 18), then changing to Scalable VG is successful:
root@bb_lpar: / # chvg -G bbvg
0516-1224 chvg: WARNING, once this operation is completed, volume group bbvg
        cannot be imported into AIX 5.2 or lower versions. Continue (y/n) ?
...
0516-1712 chvg: Volume group bbvg changed.  bbvg can include up to 1024 physical volumes with 2097152 total physical partitions in the volume group.

unix url

http://www.tutorialspoint.com/java/index.htm

Social Engineering Skills for Unix Administrators

unixadminschool.com

 you can watch how the Unix was developed at AT&T 
Garage - http://www.youtube.com/watch?v=aXhSGd-lNK8

 REFERNCE LINKS
http://wiki.fite.cat/index.php/Spacewalk
http://msat.disruptivefoss.org/.../spacewalk-sw_install.html...

 http://lmgtfy.com/?q=setup+redhat+cluster

 http://www.pimp-my-rig.com/.../rhel-cluster-luci...

RHEL 6 – Controlling Cache Memory / Page Cache Size

gurkulindia.com

List STALE partitions across Volume Groups for each Logical Volume in AIX

I wrote a tiny script that can report the number of stale partitions across multiple Volume Groups. If invoked with the -a flag the script will list LPs, PPs and Stale partitions for all Logical Volumes in all active Volume Groups. Using the -v flag only information for the specified Volume Group will be displayed.

Usage:

stale.sh -v rootvg
stale.sh -a

Sample output:

VG: rootvg   LV: hd5     LPs: 1      PPs: 2         STALE 0
VG: rootvg   LV: hd4     LPs: 6      PPs: 12        STALE 0
VG: rootvg   LV: hd2     LPs: 29     PPs: 58        STALE 0
VG: rootvg   LV: hd3     LPs: 4      PPs: 8         STALE 0
VG: rootvg   LV: hd1     LPs: 2      PPs: 4         STALE 0

The script is here:

#!/bin/ksh
#
# OS: AIX
# stale.sh
# Created by: aixdoc.wordpress.com
# THERE IS NO WARRANTY FOR THIS SCTIPT
############################################################################################
# This script if invoked with the -a flag will print for all Logical Volumes in all active #
# Volume Groups the number of LPs, PPs and stale partitions for each Logical Volume.       #
# Using the flag -v [ VG Name ] only information for the specified Volume Group will be    #
# displayed.             #
############################################################################################
# Sample output:             #
# ...                     #
# VG: rootvg           LV: hd6           LPs: 16         PPs: 32        STALE 0            #
# VG: rootvg           LV: hd8           LPs: 1          PPs: 2         STALE 0            #
# VG: rootvg           LV: hd5           LPs: 1          PPs: 2         STALE 0            #
# VG: rootvg           LV: hd4           LPs: 6          PPs: 12        STALE 0            #
# VG: rootvg           LV: hd2           LPs: 29         PPs: 58        STALE 0            #
# ...              #
############################################################################################ 

# Check if we run on AIX

if [[ `uname -s` != AIX  ]]
then print 'This script is designed for AIX only'
exit 1
fi

# Check if tmp file exists, if yes delete the file.

if [[ -a  /tmp/lslv_stale.out ]]
  then
 rm /tmp/lslv_stale.out
fi

# If no command line argument is specified, print usage

if [[ $# -eq 0 ]]; then
 print 'Usage: stale.sh [-a] List all LVs in all open VGs [-v VG_Name] List LVs in the specified VG'
  exit
fi

# If the -a flag is specified, get the list of all LVs from all active VGs. 

while [[ $1 = -* ]]; do
 case $1 in
 -a )

for vg in `lsvg -o`
 do
   lsvg -l $vg | awk 'NR>2 { print $1 }' >> /tmp/lslv_stale.out
     done

# Now run lslv against each Logical Volume and format the output as required.

for list_lv in `cat /tmp/lslv_stale.out`
  do
    lslv $list_lv |  awk -vORS=' ' 'NR==1 { printf "%-20s %-40s", "VG: " $6, "LV: " $3 } NR==7 {  printf "%-15s %-15s", "LPs: " $2, "PPs: " $4} NR==8 { printf "%-5s %-5s\n",  $1, $3 }'
            done;;

# Using the -v flag report only information for the specified VG
 -v )

lsvg -l $2 | awk 'NR>2 { print $1 }' >> /tmp/lslv_stale.out
for list_lv in `cat /tmp/lslv_stale.out`
  do
    lslv $list_lv |  awk -vORS=' ' 'NR==1 { printf "%-20s %-40s", "VG: " $6, "LV: " $3 } NR==7 {  printf "%-15s %-15s", "LPs: " $2, "PPs: " $4} NR==8 { printf "%-5s %-5s\n",  $1, $3 }'
 done;;

         *) print 'Usage: stale.sh [-a] List all LVs in all open VGs [-v VG_Name] List LVs in the specified VG'
return 1
esac
shift
done

# Delete temporary file

rm /tmp/lslv_stale.out

Installing the Network Installation Manger (NIM) master server

This post will describe the steps nesessery to setup a basic NIM master server.

Assuming the installation source will be the AIX Volume 1 installation media, in my case mounted in the /mnt/iso directory.

 
# installp -agXd /mnt/iso bos.sysmgt.nim.master
...
# lslpp -l | grep gt.nim
  bos.sysmgt.nim.client      6.1.5.1  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.master      6.1.5.1  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.client      6.1.5.1  COMMITTED  Network Install Manager -

Now we configure our mashine as a NIM Master.

smitty nimconfig

The following screen appears

                                                Configure Network Installation Management Master Fileset

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* Network Name                                       [NET_EN0] 
* Primary Network Install Interface                  [en0]                                                                                             +

  Allow Machines to Register Themselves as Clients?  [yes]                                                                                             +
  Alternate Port Numbers for Network Communications   
       (reserved values will be used if left blank)
    Client Registration                              []                                                                                                 #
    Client Communications                            []                                                                                                 #

F1=Help                               F2=Refresh                             F3=Cancel                             F4=List
F5=Reset                              F6=Command                             F7=Edit                               F8=Image
F9=Shell                              F10=Exit                               Enter=Do

The highlighted values are mandatory.

Now check the NIM server status using the lsnim command

# lsnim
master         machines        master
boot           resources       boot
nim_script     resources       nim_script
EN0_NET        networks        ent

# lsnim -l master
master:
   class               = machines
   type                = master
   max_nimesis_threads = 20
   comments            = machine which controls the NIM environment
   platform            = chrp
   netboot_kernel      = 64
   if1                 = EN0_NET  
   cable_type1         = N/A
   Cstate              = ready for a NIM operation
   prev_state          =
   Mstate              = currently running
   serves              = boot
   serves              = nim_script
   master_port         = 1058
   registration_port   = 1059
   reserved            = yes



 # lsnim -l EN0_NET
EN0_NET:
   class      = networks
   type       = ent
   Nstate     = ready for use
   prev_state = information is missing from this object's definition
   net_addr   = 192.168.2.0
   snm        = 255.255.255.0
   routing1   = default 192.168.2.1

Next we define the lpp_source running “smitty nim_mkres” and selecting “lpp_source”

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* Resource Name                                      [LPP_61_ML5]
* Resource Type                                       lpp_source
* Server of Resource                                 [master]                                                                      +
* Location of Resource                               [/nim/aix61ml5/lppsource]                                                     /
  NFS Client Security Method                         []                                                                            +
  NFS Version Access                                 []                                                                            +
  Architecture of Resource                           []                                                                            +
  Source of Install Images                           [/mnt/iso]                                                                    +/
  Names of Option Packages                           []
  Show Progress                                      [yes]                                                                         +
  Comments                                           []














F1=Help                               F2=Refresh                             F3=Cancel                             F4=List
F5=Reset                              F6=Command                             F7=Edit                               F8=Image
F9=Shell                              F10=Exit                               Enter=Do                              



                                 

Now verify the LPP source status

 # lsnim -l LPP_61_ML5
LPP_61_ML5:
   class       = resources
   type        = lpp_source
   arch        = power
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /nim/aix61ml5/lppsource
   simages     = yes
   alloc_count = 0
   server      = master

Next we create the SPOT using “smitty nim_mkres” and selecting SPOT

                                           Define a Resource

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* Resource Name                                      [SPOT_61_ML5]
* Resource Type                                       spot
* Server of Resource                                 [master]                  +
  Source of Install Images                           [LPP_61_ML5]              +
* Location of Resource                               [/nim/aix61ml5]           /
  NFS Client Security Method                         []                        +
  NFS Version Access                                 []                        +
  Expand file systems if space needed?                yes                      +
  Comments                                           []

  installp Flags
  PREVIEW only? (install operation will NOT occur)    no                       +
  COMMIT software updates?                            no                       +
  SAVE replaced files?                                yes                      +
  AUTOMATICALLY install requisite software?           yes                      +
  OVERWRITE same or newer versions?                   no                       +
  VERIFY install and check file sizes?                no                       +

The process may take some time, after it is finished we can see the new SPOT resource:

# lsnim -l SPOT_61_ML5
SPOT_61_ML5:
   class         = resources
   type          = spot
   plat_defined  = chrp
   arch          = power
   bos_license   = yes
   Rstate        = ready for use
   prev_state    = verification is being performed
   location      = /nim/aix61ml5/SPOT_61_ML5/usr
   version       = 6
   release       = 1
   mod           = 5
   oslevel_r     = 6100-05
   alloc_count   = 0
   server        = master
   Rstate_result = success
   mk_netboot    = yes
   mk_netboot    = yes

Now we will define a NIM client using the fastpath smitty nim_mkmac

                                                        Define a Machine

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

[TOP]                                                   [Entry Fields]
* NIM Machine Name                                   [power2]
* Machine Type                                       [standalone]                      +
* Hardware Platform Type                             [chrp]                            +
  Kernel to use for Network Boot                     [64]                              +
  Communication Protocol used by client              [nimsh]                           +
  Primary Network Install Interface                   
*   Cable Type                                        tp                               +
    Network Speed Setting                            []                                +
    Network Duplex Setting                           []                                +
*   NIM Network                                       EN0_NET                                                                     
*   Host Name                                         power2                                                                      
    Network Adapter Hardware Address                 [0]                                                                          
    Network Adapter Logical Device Name              []                                                                           
  IPL ROM Emulation Device                           []                                +/
  CPU Id                                             []                                                                         
  Machine Group                                      []                                +
  
  Managing System Information
  WPAR Options
    Managing System                                  []
    -OR- 

Again, verify the state of the created resource:

# lsnim -l power2
power2:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = 64
   if1            = EN0_NET power2 0
   cable_type1    = tp
   Cstate         = ready for a NIM operation
   prev_state     = ready for a NIM operation
   Mstate         = currently running

The lsnim command show the summary of all the defined resources.

lsnim
master          machines        master
boot            resources       boot
nim_script      resources       nim_script
EN0_NET         networks        ent
LPP_61_ML5      resources       lpp_source
SPOT_61_ML5     resources       spot
power2          machines        standalone

AIX-Encrypted File System

This post will in short describe how to setup an Encrypted Filesystem on AIX 6.1.

EFS offers 2 modes of operation:

Root Admin mode
This is the default mode. Root can reset user and group keystore passwords.

Root Guard mode
Root doeas not have access to user’s encrypted files and cannot change their passwords.

Note: NFS exports of EFS filesystems are not supported.

1. Prerequisites:
RBAC has to be enabled. Should be by default on AIX 6.1. If not use chdev to enable it.

# lsattr -El sys0 | grep RBAC
enhanced_RBAC   true         Enhanced RBAC Mode        True

CryptoLite needs to be installed

# lslpp -l | grep clic       
  clic.rte.kernext           4.7.0.1  COMMITTED  CryptoLite for C Kernel
  clic.rte.lib               4.7.0.1  COMMITTED  CryptoLite for C Library
  clic.rte.kernext           4.7.0.1  COMMITTED  CryptoLite for C Kernel

2. EFS Commands:

efsenable – Enables EFS on a given system. This is run only once
efskeymgr – Encryption Key Management tool
efsmgr – File encryption and decryption

3. Setup:
To enable EFS on the system use:

# efsenable -a
Enter password to protect your initial keystore:
Enter the same password again:

If your password for EFS will be identical with your login password the EFS Kernel extention will be loaded automatically into the kernel. Thus
you will be able to access the encrypted files without having to provide a password.
Otherwise `efskeymgr -o ksh` has tto be executed in order to load the key’s.

In order to have the ability to encrypt files, the filesystem that will hold this files needs to be EFS enabled (efs=yes) and Extended Attribute V2 has to be activated.

This can be verified using lsfs -q

# lsfs -q /archive
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/fslv12     --         /archive               jfs2  262144  rw         yes  no 
  (lv size: 262144, fs size: 262144, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

# chfs -a efs=yes /archive

# lsfs -q /archive
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/fslv12     --         /archive               jfs2  262144  rw         yes  no 
  (lv size: 262144, fs size: 262144, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v2, Quota: no, DMAPI: no, VIX: yes, EFS: yes, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

Now we will have a look at the keys associated  with the current shell.

# efskeymgr -V
List of keys loaded in the current process:
 Key #0:
                           Kind ..................... User key
                           Id   (uid / gid) ......... 0
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. 00f06152:be7cae83:a02379a0:82e30ab8:f6295ea1
 Key #1:
                           Kind ..................... Group key
                           Id   (uid / gid) ......... 7
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. 4a09752d:e19078be:354e4268:268c7d56:18928ecb
 Key #2:
                           Kind ..................... Admin key
                           Id   (uid / gid) ......... 0
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. 6f6e40e3:89c418ac:2e555ac4:60fdb6b5:630201f9

4. Encrypt file

Now we will create a file, try to encrypt it, have a problem with umask and finally encrypt the file.

# echo "I like black tee with milk." > secret.txt
# ls -U
total 8
-rw-r------    1 root     system           30 Jul 17 10:08 secret.txt
drwxr-xr-x-    2 root     system          256 Jan 27 19:01 tmp

        Encrypt file
          |        
# efsmgr -e secret.txt
./.efs.LZacya: Security authentication is denied.

# umask 077

# efsmgr -e secret.txt
# ls -U
total 16
drwxr-xr-x-    2 root     system          256 Jul 17 10:01 lost+found
-rw-r-----e    1 root     system           30 Jul 17 11:17 secret.txt
          |
          Indicates that this file is encrypted

Display file encryption information:

# efsmgr -l secret.txt
EFS File information:
 Algorithm: AES_128_CBC
List of keys that can open the file:
 Key #1:
  Algorithm       : RSA_1024
  Who             : uid 0
  Key fingerprint : 00f06152:be7cae83:a02379a0:82e30ab8:f6295ea1

Now I set the file permission’s to 644 and try to read the file as another user.

# chmod 644 secret.txt
# ls -la
-rw-r--r--    1 root     system          145 Jul 17 11:23 secret.txt

user1 # file secret.txt
secret.txt: 0653-902 Cannot open the specified file for reading.
user1 # cat secret.txt
cat: 0652-050 Cannot open secret.txt.

As root we will list the inode number of the file, get the block pointer and read directly from the filesystem using fsdb to see if the file is stored  encrypted.

      Display inode no.
      |
# ls -iU
total 32
    3 drwxr-xr-x-    2 root     system          256 Jul 17 10:01 lost+found
    5 -rw-r--r--e    1 root     system          145 Jul 17 11:23 secret.txt

# istat 5 /dev/fslv12
Inode 5 on device 10/27 File
Protection: rw-r--r--   
Owner: 0(root)          Group: 0(system)
Link count:   1         Length 145 bytes

Last updated:   Tue Jul 17 13:23:52 GMT+02:00 2012
Last modified:  Tue Jul 17 13:23:52 GMT+02:00 2012
Last accessed:  Tue Jul 17 13:23:52 GMT+02:00 2012

Block pointers (hexadecimal):
29        
# fsdb /dev/fslv12
Filesystem /dev/fslv12 is mounted.  Modification is not permitted.

File System:                    /dev/fslv12

File System Size:               261728  (512 byte blocks)
Aggregate Block Size:           4096
Allocation Group Size:          8192    (aggregate blocks)

> display 0x29
Block: 41     Real Address 0x29000
00000000:  119CB74E 637C6FE0 C0BF2DCD 36B775BB   |...Nc|o...-.6.u.|
00000010:  569B5A6C 43476ED3 F4BFE938 7C662A3B   |V.ZlCGn....8|f*;|
00000020:  B5D89C51 FA2BE7B6 CEAF2D3E 555EAA06   |...Q.+....->U^..|
00000030:  4FF23413 B11D1170 982690B3 5F1BCA9A   |O.4....p.&.._...|
00000040:  4AD3CEA5 A3CBFAD9 C730EE00 9BD1F409   |J........0......|
00000050:  71203B85 A51320C6 04A97DA4 43002DA7   |q ;... ...}.C.-.|
00000060:  994CC67B A1AC31DF 2C8201AD 3E5B50F7   |.L.{..1.,...>[P.|
00000070:  6BA7B01D EC5CB918 17E13F46 2935FA98   |k....\....?F)5..|
00000080:  718DF155 D6E69A41 EF592B60 EA5F7B24   |q..U...A.Y+`._{$|
00000090:  32521FE2 7AD8EC61 1A94413D A8338A26   |2R..z..a..A=.3.&|
000000a0:  62E4A319 D6251A66 F19D4739 2FC7E83A   |b....%.f..G9/..:|
000000b0:  DE0F878A 1F95AB89 5C7F3520 C65B7896   |........\.5 .[x.|
000000c0:  915A7655 EC269DFF 68E2B08A 871114A9   |.ZvU.&..h.......|
000000d0:  E30B195F 280F7DCD 4F8BE094 4B5603D8   |..._(.}.O...KV..|
000000e0:  962303B0 D957A2A5 24A2A3A5 6260EA5E   |.#...W..$...b`.^|
000000f0:  A4C62B7D FB9B1841 893D253F 72E61065   |..+}...A.=%?r..e|
-hit enter for more-
00000100:  01A150FD AD54677D A856E9B1 320257E1   |..P..Tg}.V..2.W.|
00000110:  5F023AA3 0191E0D6 4B64583B D9F2A4C7   |_.:.....KdX;....|
00000120:  F988937A E0117EB2 26E61976 E4860D7D   |...z..~.&..v...}|
00000130:  0C724A4E 50616226 BDE06FEB 10A19564   |.rJNPab&..o....d|
00000140:  17C90BB7 774338B3 8525ED90 5EADFD8B   |....wC8..%..^...|
00000150:  636FC1AF D46C2E64 6AC37082 3B0168BE   |co...l.dj.p.;.h.|
00000160:  24C0CD2E D8587254 F6DBC1BA 93BE6AD6   |$....XrT......j.|
00000170:  E89EEFF9 08000B07 E3827C10 AE0FD7DB   |..........|.....|
00000180:  162D0E6D EF94D85A 3F09CD85 A19A31FF   |.-.m...Z?.....1.|
00000190:  49E13BFC 5328F670 E0B50878 942CC4BB   |I.;.S(.p...x.,..|
000001a0:  BF1D6C4F 9DA72F3D 8DC90691 328A7053   |..lO../=....2.pS|
000001b0:  99C31EEB 1CD2208A CBF609C1 4DB86819   |...... .....M.h.|
000001c0:  E2746288 5E152ECA 0E2BD9DF D1D1D210   |.tb.^....+......|
000001d0:  7ADDF0EC 522E93E2 CAA0A36F B3CBFB05   |z...R......o....|
000001e0:  4EA56F3C ECBA1A0C AA132269 2024E065   |N.o<......"i $.e|
000001f0:  00BC51B0 88BBCD8A 9C644F66 6A16DBC8   |..Q......dOfj...|

Above we see that the file on the disk is encrypted.

5. Decrypting a file

Decrypt file
          |
# efsmgr -d secret.txt
# ls -U
total 24
drwxr-xr-x-    2 root     system          256 Jul 17 10:01 lost+found
-rw-r--r---    1 root     system          145 Jul 17 12:07 secret.txt

6. Encryption Inheritance

If you enable Encryption Inheritance on a directory all newly created files in that directory will be automatically encrypted.

To enable Encryption inheritance use:

# efsmgr -E /archive

# ls -U / | grep archive
drwxr-xr-xe    3 root     system          256 Jul 17 12:09 archive

# touch next.txt

# ls -U 
total 32
drwxr-xr-x-    2 root     system          256 Jul 17 10:01 lost+found
-rw-------e    1 root     system            0 Jul 17 12:09 next.txt
-rw-r--r---    1 root     system          145 Jul 17 12:07 secret.txt

7. Grant access to another user
Say we are  user1 and want to have a look at who has EFS access to the file.

user1 $ efsmgr -l secret.txt
EFS File information:
 Algorithm: AES_128_CBC
List of keys that can open the file:
 Key #1:
  Algorithm       : RSA_1024
  Who             : uid 0
  Key fingerprint : 00f06152:be7cae83:a02379a0:82e30ab8:f6295ea1

To grant access to a user use:

Add access to the specified file to a user or group(u/g)   
          |
# efsmgr -a secret.txt -u user1
                        |
                        Add user to EFS access list        

user1 $ cat secret.txt
I like black tee with milk.

EFS Docmentation can be found in the following Redbooks:

AIX 6.1 Diffrence Guide SG24-7559-00 Page 40
AIX V6 Advanced Security Features SG24-7430-00 Page 59