Tanti Technology

My photo
Bangalore, karnataka, India
Multi-platform UNIX systems consultant and administrator in mutualized and virtualized environments I have 4.5+ years experience in AIX system Administration field. This site will be helpful for system administrator in their day to day activities.Your comments on posts are welcome.This blog is all about IBM AIX Unix flavour. This blog will be used by System admins who will be using AIX in their work life. It can also be used for those newbies who want to get certifications in AIX Administration. This blog will be updated frequently to help the system admins and other new learners. DISCLAIMER: Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility. If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.

Monday 12 May 2014

Netstat Detailed


Netstat command displays various network related information such as network connections, routing tables, interface statistics, masquerade connections, multicast memberships etc.,
In this article, let us review 10 practical unix netstat command examples.

1. List All Ports (both listening and non listening ports)

List all ports using netstat -a

# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
udp        0      0 *:bootpc                *:*                                

Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     6135     /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5140     /var/run/acpid.socket

List all tcp ports using netstat -at

# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
tcp        0      0 localhost:ipp           *:*                     LISTEN
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN

List all udp ports using netstat -au

# netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 *:bootpc                *:*
udp        0      0 *:49119                 *:*
udp        0      0 *:mdns                  *:*

2. List Sockets which are in Listening State

List only listening ports using netstat -l

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:ipp           *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN
udp        0      0 *:49119                 *:*

List only listening TCP Ports using netstat -lt

# netstat -lt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN

List only listening UDP Ports using netstat -lu

# netstat -lu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 *:49119                 *:*
udp        0      0 *:mdns                  *:*

List only the listening UNIX Ports using netstat -lx

# netstat -lx
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     6294     private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     6203     public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     6302     private/ifmail
unix  2      [ ACC ]     STREAM     LISTENING     6306     private/bsmtp

3. Show the statistics for each protocol

Show statistics for all ports using netstat -s

# netstat -s
Ip:
    11150 total packets received
    1 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    11149 incoming packets delivered
    11635 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed.
Tcp:
    582 active connections openings
    2 failed connection attempts
    25 connection resets received
Udp:
    1183 packets received
    4 packets to unknown port received.
.....

Show statistics for TCP (or) UDP ports using netstat -st (or) -su

# netstat -st

# netstat -su

4. Display PID and program names in netstat output using netstat -p

netstat -p option can be combined with any other netstat option. This will add the “PID/Program Name” to the netstat output. This is very useful while debugging to identify which program is running on a particular port.
# netstat -pt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        1      0 ramesh-laptop.loc:47212 192.168.185.75:www        CLOSE_WAIT  2109/firefox
tcp        0      0 ramesh-laptop.loc:52750 lax:www ESTABLISHED 2109/firefox

5. Don’t resolve host, port and user name in netstat output

When you don’t want the name of the host, port or user to be displayed, use netstat -n option. This will display in numbers, instead of resolving the host name, port name, user name.
This also speeds up the output, as netstat is not performing any look-up.
# netstat -an
If you don’t want only any one of those three items ( ports, or hosts, or users ) to be resolved, use following commands.
# netsat -a --numeric-ports

# netsat -a --numeric-hosts

# netsat -a --numeric-users

6. Print netstat information continuously

netstat will print information continuously every few seconds.

# netstat -c
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 ramesh-laptop.loc:36130 101-101-181-225.ama:www ESTABLISHED
tcp        1      1 ramesh-laptop.loc:52564 101.11.169.230:www      CLOSING
tcp        0      0 ramesh-laptop.loc:43758 server-101-101-43-2:www ESTABLISHED
tcp        1      1 ramesh-laptop.loc:42367 101.101.34.101:www      CLOSING
^C

7. Find the non supportive Address families in your system

netstat --verbose
At the end, you will have something like this.
 netstat: no support for `AF IPX' on this system.
 netstat: no support for `AF AX25' on this system.
 netstat: no support for `AF X25' on this system.
 netstat: no support for `AF NETROM' on this system.

8. Display the kernel routing information using netstat -r

# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.0     *               255.255.255.0   U         0 0          0 eth2
link-local      *               255.255.0.0     U         0 0          0 eth2
default         192.168.1.1     0.0.0.0         UG        0 0          0 eth2
Note: Use netstat -rn to display routes in numeric format without resolving for host-names.

9. Find out on which port a program is running

# netstat -ap | grep ssh
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        1      0 dev-db:ssh           101.174.100.22:39213        CLOSE_WAIT  -
tcp        1      0 dev-db:ssh           101.174.100.22:57643        CLOSE_WAIT  -
Find out which process is using a particular port:
# netstat -an | grep ':80'

10. Show the list of network interfaces

# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500 0         0      0      0 0             0      0      0      0 BMU
eth2       1500 0     26196      0      0 0         26883      6      0      0 BMRU
lo        16436 0         4      0      0 0             4      0      0      0 LRU
Display extended information on the interfaces (similar to ifconfig) using netstat -ie:
# netstat -ie
Kernel Interface table
eth0      Link encap:Ethernet  HWaddr 00:10:40:11:11:11
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:f6ae0000-f6b00000

VMSTAT - Virtual Memory Statistics



If someone asks me to check how Linux/UNIX system is performing now, first think I do it vmstat. Lot of people just checks for CPU and memory utilization statistics in vmstat. But in real, it gives more information that just CPU and memory information. In this posting, I am going to explain the detail of vmstat.

vmstat stands for virtual memory statistics; it collects and displays summary information about memory, processes, interrupts, paging and block I/O information. By specifying the interval, it can be used to observe system activity interactively.

Most commonly people will use 2 numeric arguments in vmstat; first is delay or sleep between updates and the second is how many updates you want to see before vmstat quits. Please note this is not the full syntax of vmstat and also it can vary between OSs. Please refer your OS man page for more information.

To run vmstat with 7 updates, 10 seconds apart type

#vmstat 10 7

Please note, in some systems, reported metrics might be slightly diffent. The heading that I am writing now are reported in Oracle Linux (Unbreakable Oracle Linux)


Process Block: Provides details of the process which are waiting for something (it can be CPU, IO etc; can be potentially for any resource)
r  -->  Processes waiting for CPU. More the count we observe, more processes waiting to run. If we just observe a spike in the count, we shouldn’t treat them as bottleneck. If value is constantly high (most people treats 2 * CPU count  as high), it hints that CPU is the bottleneck.
b  -->  Uninterruptible sleeping processes, also known as “blocked” processes. These processes are most likely waiting for I/O but could be for something else too
w  -->  number of processes that can be run but have been swapped out to the swap area. This parameter gives hint about memory bottleneck. Please remember, only some system reports this count

Memory Block: Provides detailed memory statistics
Swpd  -->  Amount of virtual memory or swapped memory used
Free  -->  Amount of free physical memory (RAM)
Buff  -->  Amount of memory used as buffers. This memory is used to store file metadata such as i-nodes and data from raw block devices
Cache  -->  Amount of physical memory used as cache (Mostly cached file).

Note: Most of the systems report memory block value in KB. Remember I said most; not all. So check man page.

Swap Block: Provided memory swap information
si  -->  Rate at which the memory is swapped back from the disk to the physical RAM
so  -->  Rate at which the memory is swapped out to the disk from physical RAM

Note: Most of the systems reports swap block value in KB. Check man page

I/O Block: I/O related information
bi  -->  Rate at which the system sends data to the block devices (in blocks/sec)
bo  -->  Rate at which the system reads the data from block devices (in blocks/sec)

System Block: System information
in  -->  Number of interrupts received by the system per second
cs  -->  Rate of context switching in the process space (in number/sec)

CPU block: Most used CPU related information
Us  -->  Shows the percentage of CPU spent in user processes. Most of the user/application/database processes comes under user processes category
Sy  -->  Percentage of CPU used by system processes, such as all root/kernal processes
Id  -->  Percentage of free CPU
Wa  -->  Percentage spent in “waiting for I/O”

Lot of people have problems here. Some people say us + sy +id + wa=100 and some other says us + sy +id =100. I stick to second (I/O doesn’t consume CPU). 

Interpretation with respect to performance:

The first line of the output is an average of all the metrics since the system was restarted. So, ignore that line since it does not show the current status. The other lines show the metrics in real time.

Ideally r/b/w values under procs block with close to 0 or 0 itself. If one or value counter values are constantly reporting high values, it means that system may not have sufficent CPU or Memory or I/O bandwidth.

If value of swpd under swap is too high and it keeps changing, the it means that system is running short of memory.

The data under “io” indicates the flow of data to and from the disks. This shows how much disk activity is going on, which does not necessarily indicate some problem(obviously data has to go to disk in order to be persistent). If we see some large number under “proc” and then “b” column (processes being blocked) and high I/O, the issue could be a I/O contention.

Rule of thumb in Performance
Adding more CPU, Memory or I/O bandwidth to the system is not the solution to the problem always; this is just postment of problem to future and it can blow anytime. The real solution is tune the application(every compoent in the architecture) as far as possible. Adding hardware capacity or buying a powerful hardware should be the last option.

IOSTAT



Iostat command reports CPU and I/O statistics. iostat command is used during performance analysis to narrow down the problematic areas in the system.
Sample output of iostat command and its explanation:
$ iostat
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.99    0.78    7.43   12.77    0.00   53.03

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              27.40       797.19       201.27     800902     202208
sr0               0.03         1.24         0.00       1248          0
The first line displays general information — Linux kernel version, hostname etc.,
Next two lines displays CPU statistics, in which the first 3 column displays average CPU usage. The 4th column displays the percentage of time the CPU were idle and waited for I/O response. 5th column displays percentage of waiting time of virtual CPU. 6th column displays the percentage of time the CPU is idle.
Remaining line displays the device utilization report. First line in that is headers, such as number of Transfer per second, number of blocks read per second, blocks write per second, total block reads, total block writes.


4 iostat Examples

Continuous execution of iostat with specific interval

Execute iostat continuously at specific interval, and up to N times
$ iostat interval count
Execute iostat every 10 seconds for 5 times.
$ iostat 10 5
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          24.24    1.51    7.49    4.97    0.00   61.79

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              14.06       249.94       121.22    1337998     648912
sr0               0.01         0.23         0.00       1248          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.13    0.00    6.21    0.60    0.00   80.06

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.51         4.01        28.86         40        288
sr0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.11    0.00    5.71    0.00    0.00   83.18

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.30         0.00         3.20          0         32
sr0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.61    0.00    6.51    0.40    0.00   81.48

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.10         0.00        68.87          0        688
sr0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.03    0.00    5.91    0.20    0.00   78.86

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.50         0.00        30.46          0        304
sr0               0.00         0.00         0.00          0          0

View only the device report

View only the device report only once.
$ iostat -d
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              12.17       207.67       108.62    1352766     707584
sr0               0.00         0.19         0.00       1248          0
View the device report continuously for every 2 seconds, for 5 times.
$ iostat -d 2 5
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              11.12       186.33       100.49    1358494     732640
sr0               0.00         0.17         0.00       1248          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.00       108.00         0.00        216          0
sr0               0.00         0.00         0.00          0          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               5.00         0.00        96.00          0        192
sr0               0.00         0.00         0.00          0          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.01         0.00         8.04          0         16
sr0               0.00         0.00         0.00          0          0

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sr0               0.00         0.00         0.00          0          0

View device statistics for particular devices

When you have multiple devices, and wanted information only about certain devices then you can do that with -p option.
$ iostat -p sda
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.71    0.91    8.63    3.20    0.00   63.55

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               9.72       154.75        90.78    1375518     806888
sda1              0.00         0.11         0.00        946          0
sda2              0.00         0.00         0.00         14          0
sda3              4.74       115.66        30.38    1028073     270024
sda4              0.01         0.17         0.00       1522          0
sda5              4.96        38.75        60.40     344427     536864
For viewing multiple devices statistics, you can use
$ iostat -p sda, sdb

View extended statistics report

You can view extended statistics report using -x option.
$ iostat -x sda
Linux 2.6.31-17-generic (sathiya-laptop)  05/25/10  _i686_ (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.20    0.73    8.62    2.75    0.00   64.70

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               3.65     7.00    4.99    3.64   131.34    85.14    25.10     0.35   40.25   4.55   3.93

Syntax and Options

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ device [...] | ALL ] [ -p [ device [,...] | ALL ] ] [ interval [ count ] ]
Short OptionOption Description
-cDisplay the CPU utilization report
-dDisplay the device utilization report
-hMake the NFS report displayed by option -n easier to read by a human
-kDisplay statistics in kilobytes per second instead of blocks per second. Data dis-played are valid only with kernels 2.4 and later
-mDisplay statistics in megabytes per second instead of blocks or kilobytes per sec-ond. Data displayed are valid only with kernels 2.4 and later
-NDisplay the registered device mapper names for any device mapper devices. Usefulfor viewing LVM2 statistics
-nDisplay the network filesystem (NFS) report. This option works only with kernel2.6.17 and later
-p [ { device [,...] | ALL } ]The -p option displays statistics for block devices and all their partitions that are used by the system. If a device name is entered on the command line, then statistics for it and all its partitions are displayed. Last, the ALL keyword indicates that statistics have to be displayed for all the block devices and partitions defined by the system, including those that have never been used. Note that this option works only with post 2.5 kernels
-tPrint the time for each report displayed. The timestamp format may depend on thevalue of the S_TIME_FORMAT environment variable (see below)
-VPrint version number then exit
-xDisplay extended statistics. This option works with post 2.5 kernels since it needs /proc/diskstats file or a mounted sysfs to get the statistics. This option may also work with older kernels (e.g. 2.4) only if extended statistics are available in /proc/partitions (the kernel needs to be patched for that)
-zTell iostat to omit output for any devices for which there was no activity duringthe sample period

 * Source Article from : Internet

SVMON


To print out global statistics, use the -G flag. In this example, we will repeat it
five timesat two-second intervals.

# svmon -G -i 2 5
       m e m o r y            i n  u s e            p i n        p g  s p a c e
  size inuse  free   pin   work  pers  clnt   work  pers  clnt     size   inuse
 16384 16250   134  2006  10675  2939  2636   2006     0     0    40960   12674
 16384 16254   130  2006  10679  2939  2636   2006     0     0    40960   12676
 16384 16254   130  2006  10679  2939  2636   2006     0     0    40960   12676
 16384 16254   130  2006  10679  2939  2636   2006     0     0    40960   12676
 16384 16254   130  2006  10679  2939  2636   2006     0     0    40960   12676

The columns on the resulting svmon report are described as follows:

memory
     Statistics describing the use of real memory, shown in 4 K pages.
     size
         Total size of memory in 4 K pages.
     inuse
         Number of pages in RAM that are in use by a process plus the number of
         persistent pages that belonged to a terminated process and are still resident
         in RAM. This value is the total size of memory minus the number of pages
         on the free list.
     free
         Number of pages on the free list.
     pin
         Number of pages pinned in RAM (a pinned page is a page that is always
         resident in RAM and cannot be paged out).
in use
     Detailed statistics on the subset of real memory in use, shown in 4 K frames.
    work
         Number of working pages in RAM.
     pers
         Number of persistent pages in RAM.
     clnt
         Number of client pages in RAM (client page is a remote file page).
pin
     Detailed statistics on the subset of real memory containing pinned pages, shown in
     4 K frames.
     work
         Number of working pages pinned in RAM.
     pers
         Number of persistent pages pinned in RAM.
     clnt
         Number of client pages pinned in RAM.
pg space
     Statistics describing the use of paging space, shown in 4 K pages. This data is
     reported only if the -r flag is not used. The value reported starting with operating
     system version 4.3.2 is the actual number of paging-space pages used (which
     indicates that these pages were paged out to the paging space). This differs from
     the vmstat command in that vmstat's avm column which shows the virtual
     memory accessed but not necessarily paged out.
     size
         Total size of paging space in 4 K pages.
     inuse
         Total number of allocated pages.

In our example, there are 16384 pages of total size of memory. Multiply this number
by 4096 to see the total real memory size (64 MB). While 16250 pages are in use,
there are 134 pages on the free list and 2006 pages are pinned in RAM. Of the total
pages in use, there are 10675 working pages in RAM, 2939 persistent pages in RAM,
and 2636 client pages in RAM. The sum of these three parts is equal to the inuse
column of the memory part. The pin part divides the pinned memory size into working,
 persistent and client categories. The sum of them is equal to the pin column of the
memory part.There are 40960 pages (160 MB) of total paging space, and 12676
pages are in use. The inuse column of memory is usually greater than the inuse
column of pg spage because memory for file pages is not freed when a program
completes, while paging-space allocation is.
=====================
Memory bottlenecks

      The following section describes memory bottleneck solutions with the
      following commands: vmstat, svmon, ps.

      1.vmstat

      Run the following command:

      vmstat 1

      NOTE: System may slow down when pi and po are consistently non-zero.
      pi number of pages per second paged in from paging space
      po number of pages per second paged out to paging space

      When processes on the system require more pages of memory than are
      available in RAM, working pages may be paged out to paging space and then
      paged in when they are needed again. Accessing a page from paging space
      is considerably slower than accessing a page directly from RAM. For this
      reason, constant paging activity can cause system performance degradation.

      NOTE: Memory is over-committed when the fr:sr ratio is high.
      fr number of pages that must be freed to replenish the free list or to
      accommodate an active process sr number of pages that must be examined
      in order to free fr number of pages

      An fr:sr ratio of 1:4 means for every one page freed, four pages must be
      examined. It is difficult to determine a memory constraint based on this ratio
      alone and what constitutes a high ratio is workload/application dependent.

      NOTE: Memory is over-committed to the point of thrashing when
      po*SYS>fr.

      The system considers itself to be thrashing when po*SYS > fr where SYS is a
      system parameter viewed with the schedtune command. The default value is 0
      if a system has 128MB or more. Otherwise,the default is 6. Thrashing occurs
      when the system spends more time paging than performing work. When this
      occurs, selected processes may be suspended temporarily, and the system
      may be noticeably slower.

      2.svmon

      As root run the following command:
      # svmon -Pau 10 | more

      Sample Output:

      Pid Command Inuse Pin Pgspace
      13794 dtwm 1603 1 449
      Pid: 13794
      Command: dtwm
      Segid Type Description Inuse Pin Pgspace Address Range
      b23 pers /dev/hd2:24849 2 0 0 0..1
      14a5 pers /dev/hd2:24842 0 0 0 0..2
      6179 work lib data 131 0 98 0..891
      280a work shared library 1101 0 10 0..65535
      181 work private 287 1 341 0..310:65277..65535
      57d5 pers code,/dev/hd2:61722 82 0 0 0..135

      This command lists the top ten memory using processes and gives a report
      about each one. In each process report, look where Type = work and
      Description = private. Check how many 4K (4096 byte) pages are used under
      the Pgspace column. This is the minimum number of working pages this
      segment is using in all of virtual memory. A Pgspace number that grows, but
      never decreases, may indicate a memory leak. Memory leaks occur when an
      application fails to deallocate memory.

      341 * 4096 = 1,396,736 or 1.4MB of virtual memory
      3. ps

      Run the following command:
      ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r
      size
      amount of memory in KB allocated from page space for the memory segment
      of Type = work and Description = private for the process as would be
      indicated by svmon.

      RSS
      amount of memory, in KB, currently in use (in RAM) for the memory segment
      of Type = work and Description = private plus the memory segment(s) of Type
      = pers and Description = code for the process as would be indicated by
      svmon.

      trs
      amount of memory, in KB, currently in use (in RAM) for the memory
      segment(s) of Type = pers and Description = code for the process as would
      be indicated by svmon.

      %mem
      RSS value divided by the total amount of system RAM in KB multiplied by
      100.

Tuesday 6 May 2014

AIX NIM Client Installation


Before we can install a client we have to prepare the server and the client for this operation. On the server the resources have to be allocated for the client, and the client has to boot over the network.

NIM server

smitty nim
  • Perform NIM administration tasks
    • Manage Machines
      • Manage Network Install Resource Allocation
        • Allocate Network Install Resources
          • ITO_AIX_53-06-07-0818_LPP_FULL (lpp_source)
          • ITO_AIX_53-06-07-0818_LPP_SPOT_FULL (spot)
      • Perform Operations on Machines
        • bos_inst
          • Source for BOS Runtime Files rte → spot
          • Initiate Boot Operation on Client? Yes → no
          • ACCEPT new license agreements? No → Yes
When done correctly there are two changes on the server. First there is a line added in the /etc/bootptab:
server1.company.local:bf=/tftpboot/server1.company.local:ip=10.10.10.100:ht=ethernet:sa=10.10.10.200:sm=255.255.0.0:
And, you can check the /etc/exports to see if the correct NFS shares are exported.

NIM Client

Setup IPL:
  • Boot the client and go into the SMS (System Management Services) menu by pressing 1 when you get the screen with stripes in it:
  • Choose the setup Remote IPL (Initial Program Load) option (2)
  • Select the network adapter you want to use for the network installation
  • Select network service for NIM this is bootp (1)
  • Select IP parameters (1)
    • Select Client IP Address (this is the NIM clients IP address that should also exist with the machine definition in the NIM serv er)
    • Select Server IP Address (this is the NIM server IP address)
    • Select Gateway IP Address (this is the NIM client gateway)
    • Select Subnet mask (this is the NIM client subnet mask)
    • Exit the IP parameters menu (ESC)
  • Select Adapter configuration (2)
    • Select Speed, Duplex (1) (set to auto,auto)
    • Select Spanning Tree Enabled (2) Make sure SPT is off
    • Select Protocol (3) Make sure the standard protocol is being used
  • Exit the Adapter Configuration menu (ESC)
    • Select Advanced Setup: bootp (4)
    • Select bootp retries (1) change the number of retries to 2
    • Select TFTP retries (3) change the number of retries to 2
    • Exit the Advanced setup: bootp menu (ESC)
  • Select Ping test (3) verify the network/ adapter configuration and try to ping the NIM server (1)
    • If the system replies ping success then you have a physical network connection to the NIM server.
    • If the system replies that the ping failed check the physical network configuration and the configuration on the NIM server
    • If it was success type M (return to main menu)
Setup Boot from network (still in SMS menu):
  • Select Boot Options(5)
  • Select Configure Boot Device order(2)
  • Select 1st boot device(1)
  • Select Network(6)
  • Select BOOTP(1)
  • Select the IP configured network adapter
  • Select make first boot device
  • Type x to exit the SMS menu and let the server boot
If the NIM client and server side are properly configured the NIM client will boot from network.
It essentially will use bootp request to NIM server which will begin to push the installation package to the NIM client by using TFTP protocol.

Installation

Select a new and complete overwrite with needed software packages:
aixinstall.jpg 
Check AIX Post Install to see what you need to adjust after the installation is done.

AIX NIM Client


This page shows you how to add a standalone client to the NIM environment, and how to reset and remove it again. This is mostly done through smitty.

AIX NIM Client Add

smitty nim
  • Perform NIM Administration Tasks
    • Manage Machines
      • Define a Machine
(smit fastpath: smit nim_mkmac)
Fill in the hostname the NIM server can resolve! If not, NIM will create a new network or disply other miscellaneous errors.
NOTE: without proper DNS A and PTR records the NFS mount will fail 
aixnimclientadd.jpg

AIX NIM Client Reset

In case the client has a running operation or something like that you'll first have to reset the state of the specific machine:
smitty nim
  • Perform NIM Administration Tasks
    • Manage Machines
      • Perform Operations on Machines
Select the machine you want to reset and press ENTER and select the reset operation: 
aixnimclientreset.jpg

Commandline

Or use this command:
bash-3.2# nim -o reset lpar01

AIX NIM Client Remove

Now you can remove the machine:
smitty nim
  • Perform NIM Administration Tasks
    • Manage Machines
      • Remove a Machine
Select the machine you're trying to remove and confirm (couple of times).

AIX NIM concept


NIM: Network Installation Manager
This page is to cover all steps taken and done to do get to the following objective:
  • Design a new NIM environment
    • Based on AIX 6.1
    • Define name conventions
    • Support AIX 5.2; 5.3 and 6.1
    • Support 6.1 VIOS
  • Setup a new NIM environment based on AIX 6.1
  • Recreate all clients in the new NIM environment
  • Clone AIX installations (bare metal as well as LPARs) to new hardware
  • Setup SUMA
  • Update all necessary installations to the latest stable technology level
  • Design and implement disaster recovery
  • Design and implement a Standard Operating Environment

NIM Design

NIM terminology

Master

The machine where you set up and maintain your NIM environment. It's also possible to initiate installations from this machine (push mode).

Client

A target for NIM master-initiated operations such as installation, updates etc. It's also possible to initiate its own installation or update (pull mode).

Resource server

Any machine (master and or client) configured to hold a particular software resource. In most environments, the master is also the (only) resource server.

Push mode

Initiated from the master. To be successful, the client must have AIX and tcp/ip configured.

Pull mode

Initiated from the client. To use, you need access to the clients' SMS menu.

SPOT (Shared Product Object Tree)

A directory of code (installed filesets) that is used during client booting procedure. It's equivalent in content to the code that resides in the /usr file system on a system running AIX. It also contains the code needed to generate the boot images (kernels, which will be stored in the /tftboot directory) that the client uses until it can mount the SPOT over NFS.

lpp_source

A directory similar to AIX installation CDs. It contains the Licensed Program Products (LLPs) in Backup File Format (BFF) and RPM filesets that can be installed.

mksysb

A file containing the image of the root volume group of a machine. It's used to restore a machine, or to install it from scratch (cloning).

bosinst_data

A flat ASCII file similar to the bosinst.data file used for restoring backup images. It automates the installation process by providing the answers to the interactive installation questions.

script

A file which runs after the installation on your client to perform customization such as file system resizing, additional user creation etc.

Name conventions

lpp_source objects

  • lpp_5300_05_01
    • AIX 5.3 met Technology Level (TL) 5 en Service Pack (SP) 1

spot objects

  • spot_5300_05_01
    • AIX 5.3 met Technology Level (TL) 5 en Service Pack (SP) 1

mksysb generic objects

  • mksysb_5300_05_01
    • AIX 5.3 met Technology Level (TL) 5 en Service Pack (SP) 1

mksysb particular objects

  • mksysb_lpar1_20090527
    • backup of lpar1 on 27 may 2009

network objects

  • net_10_1_0_0
  • net_10_246_60_00

NIM Setup

Directory structure

According to the redbook on NIM there are several options to maintain the storage needed by NIM. Because we have a SAN which is able to increase allocated volumes and LUNs I create a single volume and two LUNs, one for booting and one for the NIM data. Because I have a separate LUN for NIM, it's possible to increase the filesystem when necessary. That way, I'll always keep one filesystem and volumegroup for booting, and one of each for the NIM data. In case you need to increase the storage space needed for NIM data perform the following steps:
  • Increase the SAN volume
  • Increase the LUN
  • Increase the filesystem
For more info on the last step see AIX Storage.

Create a special volume group to host all NIM related data and create one large filesystem in this volume group. After creation of the filesystem create directories to store all NIM data:
bash-3.2# lsvg -l sanvg
sanvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       1       1    open/syncd    N/A
fslv00              jfs2       796     796     1    open/syncd    /nim
bash-3.2# ls -l /nim
total 0
drwxr-xr-x    2 root     system          256 Jun 09 06:52 images
drwxr-xr-x    2 root     system          256 Jun 09 06:49 lost+found
drwxr-xr-x    2 root     system          256 Jun 09 06:52 lpp_source
drwxr-xr-x    2 root     system          256 Jun 09 06:52 spot
drwxr-xr-x    2 root     system          256 Jun 09 07:06 tftpboot
Note that the tftpboot directory will hold the boot images when needed for clients. Because the number of boot images may increase with the number of operating system levels I created a special directory on the dedicated volume group and let the directory in root link to this directory:
ln -s /nim/tftpboot/ /tftpboot
ls -l
lrwxrwxrwx    1 root     system           14 Jun 09 07:06 tftpboot -> /nim/tftpboot/
For more information on creating AIX volume groups and filesystems (and logical volumes) see AIX Storage

Master installation

The NIM master software was installed through smitty:
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
bos.sysmgt.nim.master       6.1.3.0         USR         APPLY       SUCCESS
bos.sysmgt.nim.master       6.1.3.0         USR         COMMIT      SUCCESS

---- end ----
Now the software is installed but not yet configured/initialized. The command lsnim will tell you by showing you no objects at all:
bash-3.2# lsnim
0042-011 lsnim: unable to access the /etc/niminfo file

NIM Master Initialization

Note: Before starting the initialization, make sure the NIM master can be resolved with the nameserver, for the A-record, as well as for the reverse record.
NIM initialization is done through the nimconfig command:
bash-3.2# nimconfig -a netname=net_10_1_0_0 -a pif_name=en0 -a netboot_kernel=mp -a cable_type=tp -a client_reg=no
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 274446.
  • netname
    • Required: the name of the first nim network that is defined
  • pif_name
    • Required: The primary network install interface
  • netboot_kernel
    • Specifies the type of kernel to use when booting the client over the network.
      • mp = multiprocessor
      • up = uniprocessor
  • cable_type
    • Required if pif refers to an ethernet network
      • tp = Twisted Pair
      • bnc = Coaxial cable
      • dix = Thick cable, also known as “10Base5”
  • client_reg
    • Defines whether the clients may register themselves with the master or not
That NIM is initialized now can be checked using the lsnim command:
bash-3.2# lsnim
master           machines        master
boot             resources       boot
nim_script       resources       nim_script
net_10_1_0_0     networks        ent
bash-3.2# lsnim -l master
master:
   class               = machines
   type                = master
   max_nimesis_threads = 20
   comments            = machine which controls the NIM environment
   client_reg          = no
   platform            = chrp
   netboot_kernel      = mp
   if1                 = net_10_1_0_0 ms-nim01.company.local 001A6444B9C8
   cable_type1         = tp
   Cstate              = ready for a NIM operation
   prev_state          =
   Mstate              = currently running
   serves              = boot
   serves              = nim_script
   master_port         = 1058
   registration_port   = 1059
   reserved            = yes
bash-3.2# lsnim -l boot
boot:
   class       = resources
   type        = boot
   comments    = represents the network boot resource
   Rstate      = ready for use
   location    = /tftpboot
   alloc_count = 0
   server      = master
   reserved    = yes
bash-3.2# lsnim -l nim_script
nim_script:
   class       = resources
   type        = nim_script
   comments    = directory containing customization scripts created by NIM
   Rstate      = ready for use
   location    = /export/nim/scripts
   alloc_count = 0
   server      = master
   reserved    = yes
bash-3.2# lsnim -l net_10_10_0_0
net_10_1_0_0:
   class      = networks
   type       = ent
   Nstate     = ready for use
   prev_state = information is missing from this object's definition
   net_addr   = 10.10.0.0
   snm        = 255.255.0.0
   routing1   = default 10.10.1.2
And by checking the /etc/niminfo file:
bash-3.2# cat /etc/niminfo
# nimconfig
export NIM_NAME=master
export NIM_CONFIGURATION=master
export NIM_MASTER_PORT=1058
export NIM_REGISTRATION_PORT=1059
export NIM_MASTER_HOSTNAME=ms-nim01.company.local

Verify other services

The NIM master uses the bootpd and tftpd services. Check if they're already running:
bash-3.2# lssrc -ls inetd
Subsystem         Group            PID          Status
 inetd            tcpip            135338       active
Debug         Not active
Signal        Purpose
 SIGALRM      Establishes socket connections for failed services.
 SIGHUP       Rereads the configuration database and reconfigures services.
 SIGCHLD      Restarts the service in case the service ends abnormally.
Service       Command                  Description              Status
 xmquery      /usr/bin/xmtopas         xmtopas -p3              active
 wsmserver    /usr/websm/bin/wsmserver wsmserver -start         active
 time         internal                                          active
 daytime      internal                                          active
 time         internal                                          active
 daytime      internal                                          active
 ntalk        /usr/sbin/talkd          talkd                    active
 exec         /usr/sbin/rexecd         rexecd                   active
 login        /usr/sbin/rlogind        rlogind                  active
 shell        /usr/sbin/rshd           rshd                     active
 telnet       /usr/sbin/telnetd        telnetd -a               active
 ftp          /usr/sbin/ftpd           ftpd                     active
In case they're not (as shown in the example) edit the /etc/inetd.conf file to uncomment the corresponding lines:
bootps   dgram  udp     wait    root    /usr/sbin/bootpd        bootpd /etc/bootptab
tftp     dgram  udp6    SRC     nobody  /usr/sbin/tftpd         tftpd -n
And refresh the inetd daemon:
bash-3.2# refresh -s inetd
0513-095 The request for subsystem refresh was completed successfully.
And check to see if the services are running now:
bash-3.2# lssrc -ls inetd
Subsystem         Group            PID          Status
 inetd            tcpip            135338       active
Debug         Not active
Signal        Purpose
 SIGALRM      Establishes socket connections for failed services.
 SIGHUP       Rereads the configuration database and reconfigures services.
 SIGCHLD      Restarts the service in case the service ends abnormally.
Service       Command                  Description              Status
 tftp         /usr/sbin/tftpd          tftpd -n                 active
 bootps       /usr/sbin/bootpd         bootpd /etc/bootptab     active
 xmquery      /usr/bin/xmtopas         xmtopas -p3              active
 wsmserver    /usr/websm/bin/wsmserver wsmserver -start         active
 time         internal                                          active
 daytime      internal                                          active
 time         internal                                          active
 daytime      internal                                          active
 ntalk        /usr/sbin/talkd          talkd                    active
 exec         /usr/sbin/rexecd         rexecd                   active
 login        /usr/sbin/rlogind        rlogind                  active
 shell        /usr/sbin/rshd           rshd                     active
 telnet       /usr/sbin/telnetd        telnetd -a               active
 ftp          /usr/sbin/ftpd           ftpd                     active
As you can see, they are running now.

TFTP and BOOTP debugging

To enable debugging for tftp and bootp edit the inetd.conf again and change the bootp and tftp lines according to these:
bootps  dgram   udp     wait    root    /usr/sbin/bootpd       bootpd -d -d -d /etc/bootptab
tftp     dgram  udp6    SRC     nobody  /usr/sbin/tftpd         tftpd -nvi
and restart the inetd daemon: 
bash-3.2# refresh -s inetd
0513-095 The request for subsystem refresh was completed successfully.
In case you have enabled syslogging you'll now see the messages in your syslog:
Daemon  Daemon  Information  bootpd[430226]: exiting after 15 minutes of inactivity  18:49:14 Jul 01 2009
Daemon  Daemon  Information  tftpd[286866]: [0000010b] EZZ7046I : Status Transaction completed successfully  18:35:04 Jul 01 2009
Daemon  Daemon  Information  tftpd[286866]: [0000010b] EZZ7029I : Status Read request for 10.10.3.9: /tftpboot/ms-lpar01.company.local  18:34:14 Jul 01 2009
Daemon  Daemon  Information  tftpd[286866]: [0000010b] EZZ7044I : 10.10.3.9 RRQ   18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Finished processing boot request.  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Gateway IP address (bp->bp_giaddr) = 10.10.3.7  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Server IP address (bp->bp_siaddr) = 10.10.3.7  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Client IP address (bp->bp_ciaddr) = 10.10.3.9  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: The following addresses are included in the bootp reply  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: sending RFC1048-style reply  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: RFC1048 vendor data ( bp_vend[64] )  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: vendor magic field is 99.130.83.99  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: bootfile = /tftpboot/ms-lpar01.company.local  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: found 10.10.3.9 ms-lpar01.company.local  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: request from IP addr 10.10.3.9  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Received boot request.  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: bootptab mtime is Wed Jul 1 18:20:07 2009  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Finished processing boot request.  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Gateway IP address (bp->bp_giaddr) = 10.10.3.7  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Server IP address (bp->bp_siaddr) = 10.10.3.7  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Client IP address (bp->bp_ciaddr) = 10.10.3.9  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: The following addresses are included in the bootp reply  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: sending RFC1048-style reply  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: RFC1048 vendor data ( bp_vend[64] )  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: vendor magic field is 99.130.83.99  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: bootfile = /tftpboot/ms-lpar01.company.local  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: found 10.10.3.9 ms-lpar01.company.local  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: request from IP addr 10.10.3.9  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: Received boot request.  18:34:14 Jul 01 2009
Daemon  Daemon  Information  bootpd[430226]: bootptab mtime is Wed Jul 1 18:20:07 2009 

TFTP Security

Because tftp uses no authentication it is important to limit the directories the service can access, and because I have created a link from /tftpboot to /nim/tftpboot I have to add extra directories. To do so, create the file /etc/tftpaccess.ctl and define the correct directories:
bash-3.2# vi /etc/tftpaccess.ctl
bash-3.2# cat /etc/tftpaccess.ctl
# NIM access for network boot
allow:/tftpboot
allow:/nim/tftpboot

Test TFTP

bash-3.2# cat /tftpboot/test
bash-3.2# touch /tftpboot/test2
bash-3.2# tftp -r - 0 /tftpboot
...testtest2Received 64 bytes in 0.0 seconds
bash-3.2#

bash-3.2# tftp -o - 0 /tftpboot/ms-lpar01.company.local.info
#------------------ Network Install Manager ---------------
# warning - this file contains NIM configuration information
#       and should only be updated by NIM
export NIM_NAME=ms-lpar01
export NIM_HOSTNAME=ms-lpar01.company.local
export NIM_CONFIGURATION=standalone
export NIM_MASTER_HOSTNAME=ms-nim01.company.local
export NIM_MASTER_PORT=1058
export NIM_REGISTRATION_PORT=1059
export NIM_SHELL="nimsh"
export NIM_MASTERID=000153BAD400
export NIM_LICENSE_ACCEPT=yes
export RC_CONFIG=rc.bos_inst
export NIM_BOSINST_ENV="/../SPOT/usr/lpp/bos.sysmgt/nim/methods/c_bosinst_env"
export NIM_BOSINST_RECOVER="/../SPOT/usr/lpp/bos.sysmgt/nim/methods/c_bosinst_env -a hostname=ms-lpar01.company.local"
export SPOT=ms-nim01.company.local:/nim/spot/spot_5300_05/usr
export NIM_CUSTOM="/../SPOT/usr/lpp/bos.sysmgt/nim/methods/c_script -a location=ms-nim01.company.local:/export/nim/scripts/ms-lpar01.script"
export NIM_BOS_IMAGE=/SPOT
export NIM_BOS_FORMAT=spot
export NIM_HOSTS=" 127.0.0.1:loopback:localhost  10.10.3.9:ms-lpar01.company.local  10.10.3.7:ms-nim01.company.local "
export NIM_MOUNTS=" ms-nim01.company.local:/nim/lpp_source/lpp_5300_05:/SPOT/usr/sys/inst.images:dir "
export ROUTES=" default:0:10.10.1.2 "
Received 1283 bytes in 0.0 seconds

bash-3.2# tftp -r - 0 /etc
Error code 2: Access violation
Looks like everything works fine.

Define lpp and spot resources

Define lpp source AIX 6.1 SP 1

Define lpp source for AIX 6.1 SP 01, with the CD-rom as source:
bash-3.2# nim -o define -t lpp_source -a server=master -a location=/nim/lpp_source/lpp_6100_00_01 -a source=/dev/cd0 lpp_6100_00_01
Preparing to copy install images (this will take several minutes)...

/nim/lpp_source/lpp_6100_00_01/RPMS/ppc/cdrecord-1.9-7.aix5.2.ppc.rpm
......
/nim/lpp_source/lpp_6100_00_01/installp/ppc/X11.loc.BE_BY.6.1.0.0.I

Now checking for missing install images...

All required install images have been found. This lpp_source is now ready.
warning: 0042-265 c_mk_lpp_source: The image source "/dev/cd0"
        that was used to define the lpp_source is missing one or more
        of the following from the list of default packages:
                wio.common
                wio.fcp
As you can see, there are two 'default' packages missing:
  • wio.common
  • wio.fcp
Both are used for workload partition, which I'm not running so I don't need them, which means I can ignore this.

Complete software from all CDs

When you create a LPP source you create one by default with only the most used filesets. If you want all software to be in the LPP_source you can add all sources like this:
Keep cd1 in the drive and:
  • smitty nim
    • Perform NIM Administration Tasks
      • Manage Resources
        • Perform Operations on Resources
        • And select the lpp_source (6100_00_01)
        • And select update
        • Confirm that you're adding software
        • And select the input device (/dev/cd0)
        • Select the software packages to add and press ENTER (ALL)
It will first ask for the first CD to copy the missing filesets, and after that it will ask for all cds:
bffcreate:  Please insert volume 2 into device /dev/cd0 and press Enter
           to continue or enter "q" to quit.

Define lpp_source AIX 5L 5.3 TL 5

Define lpp source for AIX 5.3 TL 05, with a directory as source:
bash-3.2# nim -o define -t lpp_source -a server=master -a location=/nim/lpp_source/lpp5300_05 -a source=/tmp/5300-05/cd1/ lpp_5300_05
Preparing to copy install images (this will take several minutes)...

/nim/lpp_source/lpp5300_05/RPMS/ppc/cdrecord-1.9-7.aix5.2.ppc.rpm
......
/nim/lpp_source/lpp5300_05/installp/ppc/bos.adt.5.3.0.0.I

Now checking for missing install images...

All required install images have been found. This lpp_source is now ready.
warning: 0042-265 c_mk_lpp_source: The image source "/tmp/5300-05/cd1/"
        that was used to define the lpp_source is missing one or more
        of the following from the list of default packages:
                bos.swma
                csm.msg.en_US
                ifor_ls.html.en_US
                rsct.msg.en_US

bash-3.2# lsnim
master           machines        master
boot             resources       boot
nim_script       resources       nim_script
net_10_1_0_0     networks        ent
lpp_5300         resources       lpp_source
lpp_5300_05      resources       lpp_source
Again, there are packages missing which I don't need.

Update LPP Source

First copy an existing LPP source

bash-3.2# cd /nim/lpp_source/
bash-3.2# ls
lpp_5300_05     lpp_6100_00_01
bash-3.2# cp -r lpp_5300_05/ lpp_5300_06_07
bash-3.2# ls
lpp_5300-06-07  lpp_5300_05     lpp_6100_00_01

Define the LPP source

bash-3.2# nim -o define -t lpp_source -a server=master -a location=/nim/lpp_source/lpp_5300_06_07 lpp_5300_06_07
Preparing to copy install images (this will take several minutes)...

Now checking for missing install images...

All required install images have been found. This lpp_source is now ready.

Update the LPP source with a downloaded TL/SP

bash-3.2# nim -o update -a packages=all -a source=/nim/installsources/5300-06-07/ lpp_5300_06_07

/nim/lpp_source/lpp_5300_06_07/installp/ppc/bos.rte.install.5.3.0.67.U
......
/nim/lpp_source/lpp_5300_06_07/installp/ppc/bos.loc.pc.Lv_LV.5.3.0.10.U
bash-3.2#

Check the updated LPP source

Checking an updated LPP source consists of 3 steps, first check the lpp source, then remove duplicate packages, and then recheck the lpp source. I don't know for sure if the checking is necessary, but it only takes a few seconds, so better safe then sorry.
bash-3.2# nim -o check lpp_5300_06_07
bash-3.2# nim -o lppmgr -a lppmgr_flags=-rbux lpp_5300_06_07
lppmgr: Source table of contents location is /nim/lpp_source/lpp_5300_06_07/installp/ppc/.toc
lppmgr: Building table of contents in /nim/lpp_source/lpp_5300_06_07/installp/ppc ..
lppmgr: Building table of contents completed.
lppmgr: Generating list of superseded filesets..
lppmgr: Generating duplicate list..
lppmgr: Generating base level duplicate list..

Results:
======================= start list =============================
Java14.license.1.4.2.0.U
.......
sysmgtlib.framework.core.5.3.0.50.U
======================== end list ==============================

lppmgr: Building table of contents in /nim/lpp_source/lpp_5300_06_07/installp/ppc ..
lppmgr: Building table of contents completed.
rm: removing /nim/lpp_source/lpp_5300_06_07/installp/ppc/Java14.license.1.4.2.0.U
......
rm: removing /nim/lpp_source/lpp_5300_06_07/installp/ppc/sysmgtlib.framework.core.5.3.0.50.U
bash-3.2# nim -o check lpp_5300_06_07
bash-3.2# lsnim -l lpp_5300_06_07
lpp_5300_06_07:
   class       = resources
   type        = lpp_source
   arch        = power
   Rstate      = ready for use
   prev_state  = verification is being performed
   location    = /nim/lpp_source/lpp_5300_06_07
   simages     = yes
   alloc_count = 0
   server      = master
This is the explanation of the lppmgr_flags option:
  • Remove duplicate updates (-u flag).
  • Remove duplicate base levels (-b flag).
  • Eliminate update imagess which are the same level as base images of the same fileset. Such update images can create conflicts that lead to installation failure (-u flag).
  • Remove message and locale filesets other than the language you specify (-k flag).
  • Remove superseded filesets (-x flag).
  • Remove non-system images from a NIM lpp_source resource (-X flag).

Check content LPP_Source

Check to see what packages on what level are installed in a particular lpp_source:
bash-3.2# nim -o showres lpp_5300_06_07 | grep xlC
  xlC.aix50.rte               6.0.0.13                   I  N usr
  xlC.cpp                     6.0.0.0                    I  N usr
  xlC.msg.en_US.cpp           6.0.0.0                    I  N usr
  xlC.rte                     6.0.0.0                    I  N usr

Define SPOT sources

If you want to work with different kind of versions you have to create a non-/usr SPOT. It is recommended to use a NIM lpp_source to create it from. Note that you don't need to provide the name of the directory for the location, it will automatically create a directory by the name of the SPOT object.
bash-3.2# nim -o define -t spot -a server=master -a location=/nim/spot/ -a source=lpp_6100_00_01 -a installp_flags=-aQg spot_6100_00_01

 Creating SPOT in "/nim/spot/" on machine "master" from "lpp_6100_00_01" ...

 Restoring files from BOS image.  This may take several minutes ...

 Installing filesets ...

......

 Be sure to check the output from the SPOT installation
 to verify that all the expected software was successfully
 installed.  You can use the NIM "showlog" operation to
 view the installation log file for the SPOT.


+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.

  Selected Filesets
  -----------------
  bos.64bit 6.1.0.1                           # Base Operating System 64 bit...
  bos.diag.com 6.1.0.0                        # Common Hardware Diagnostics
  bos.diag.rte 6.1.0.0                        # Hardware Diagnostics

......

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
  349  Selected to be installed, of which:
      349  Passed pre-installation verification
   23  Additional requisites to be automatically installed
  ----
  372  Total to be installed

......

installp:  * * * A T T E N T I O N ! ! !
        Software changes processed during this session require
        any diskless/dataless clients to which this SPOT is
        currently allocated to be rebooted.

 Checking filesets and network boot images for SPOT "spot_5300_05".
 This may take several minutes ...

Check SPOT sources

bash-3.2# nim -o check spot_5300_05

Check SPOT level

As far as I know there is no strict way of determining the oslevel including the fixpack from the SPOT resource. However, you can determine the technology level of the spot, which you can use to search for installed filesets. This will give you an almost certainty of which level your SPOT is made.
First check the technology level:
bash-3.2# lsnim -l spot_5300_06_07
spot_5300_06_07:
   class         = resources
   type          = spot
   plat_defined  = chrp
   arch          = power
   bos_license   = yes
   Rstate        = ready for use
   prev_state    = verification is being performed
   location      = /nim/spot/spot_5300_06_07/usr
   version       = 5
   release       = 3
   mod           = 0
   oslevel_r     = 5300-06
   alloc_count   = 0
   server        = master
   if_supported  = chrp.64 ent
   if_supported  = chrp.mp ent
   Rstate_result = success
The oslevel_r is set to 5.3 TL 6.
Now see what packages are installed for AIX 5.3 TL6:
bash-3.2# nim -o fix_query -a fix_query_flags=-c  spot_5300_06_07 | grep :=: | grep "AIX 5300-06"
5300-06_AIX_ML:bos.rte.ILS:5.3.0.50:5.3.0.50:=:AIX 5300-06 Update
......
5300-06_AIX_ML:devices.pci.1410e601.rte:5.3.0.50:5.3.0.50:=:AIX 5300-06 Update
......
53-06-040748_SP:bos.rte.devices_msg:5.3.0.61:5.3.0.61:=:AIX 5300-06-04 Service Pack
......
53-06-050806_SP:devices.pciex.14103f03.rte:5.3.0.1:5.3.0.1:=:AIX 5300-06-05 Service Pack
......
53-06-060811_SP:perl.rte:5.8.2.62:5.8.2.62:=:AIX 5300-06-06 Service Pack
......
53-06-070818_SP:perl.rte:5.8.2.62:5.8.2.62:=:AIX 5300-06-07 Service Pack
As you can see, the output displays all filesets that are installed for AIX 5.3 TL 6, and ends with the software packages from the latest fixpack.

Debug SPOT sources

In case the bootp and tftp stages work fine, but there still goes something wrong with the installation you have two options to debug the NIM operations.

NIM 3-digit LED codes

During the NIM process it produces codes which are displayed in the LED. These codes are (The values are presented in the order in which they are displayed in the three-digit LED):
  • 299 Boot image successfully received at the NIM client.
  • 600 Starting network boot (portion of /sbin/rc.boot).
  • 602 Configuring network parent devices.
    • 603 Script defsys, cfgsys, or cfgbus located in /usr/lib/methods/ failed.
  • 604 Configuring physical network boot device.
    • 605 Configuration physical network boot device failed.
  • 606 Running /usr/sbin/ifconfig on logical network boot devices.
    • 607 /usr/sbin/ifconfig failed.
  • 608 Attempting to retrieve the client.info file with tftp from the SPOT server.
    • 609 The client.info file does not exist or could not be accessed, or it is zero length.
  • 610 Attempting to mount a remote file system using NFS.
    • 611 The client is unable to mount a remote file system (NIM resource) using NFS.
  • 612 Accessing remote files. Unconfiguring network boot devices.
    • 613 The route command failed.
  • 614 Configuration of logical paging devices.
  • 615 Configuration of logical paging device failed.
  • 616 Converting from diskless to dataless configuration.
    • 617 Diskless to dataless configuration failed.
  • 618 Configuring remote (NFS) paging device.
    • 619 Configuration of remote (NFS) paging space failed.
  • 620 Updating special device files and ODM in permanent file system.
  • 622 Control returned to the /sbin/rc.boot program.
    • 623 The BOS installation program has encountered a fatal error.
  • 624 Control passed to the BOS installation Program.
  • c40 Extracting data files from media.
  • c42 Extracting data files from diskette.
  • c44 Initializing install data base with target disk information.
  • c46 Normal install processing.
  • c48 Prompting user for input.
  • 622 Control returned to the /sbin/rc.boot program.
    • 623 The BOS installation program has encountered a fatal error.
  • 624 Control passed to the BOS installation Program.
  • c40 Extracting data files from media.
  • c42 Extracting data files from diskette.
  • c44 Initializing install data base with target disk information.
  • c46 Normal install processing.
  • c48 Prompting user for input.
These errors are most common:
608 - tftp retrieve of client info file failure If a 608 hang is encountered, we verify that the ClientName.info file exists in the /tftpboot directory. If it does not exist, we retry the NIM operation to create it. If it does exist, we verify that tftp access to the /tftpboot directory is not restricted in
the /etc/tftpaccess.ctl file. It is also possible that the network adapter was not configured properly in the boot environment. In this case, we use debug-enabled network boot images to look for errors in the boot environment.
611 - Remote mount of NFS file system failure LED 611 hangs occur when the client machine is unable to mount a resource from the NIM master/resource server. First ensure that NFS is running on the master/resource server. We verify that the resources specified for the operation are exported properly by checking the /etc/exports and /etc/xtab files on the server. Also, we confirm that the resources have permissions set correctly for reading. Debug-enabled network boot images can also be used to determine exactly which mount command is failing on the client. We can also check the value of the nfso server portcheck option and usage of NFS reserved ports.For further information about the use of NFS reserved ports and the nfso server portcheck option, refer to 5.3.9, “NFS reserved ports” on page 441.
  • This message can also occur when the nim server can't reverse lookup the client. So always create the client in your DNS with an A and a PTR record.
613 - Failure setting up route tables 613 hangs usually occur because a route is incorrectly defined for a network in the NIM database. We verify that the correct gateways are specified between networks, and all gateways are functional. We use debug-enabled network boot images to determine which routes could not be defined.

Enable SPOT debug

The other option is to enable debugging on the SPOT source.
  • Run the following command:
nim  -Fo  check  -a  debug=yes  SPOTName 
where SPOTName is the name of your SPOT.
  • Obtain the address for entering the debugger by running the following command to get the address:
lsnim  -a  enter_dbg  SPOTName 
where SPOTName is the name of your SPOT. The displayed output will be similar to the following: 
spot1: 
    enter_dbg  =  "chrp.mp  0x001840d4" 
    enter_dbg  =  "chrp.up  0x00160b7c" 
    enter_dbg  =  "rs6k.mp  0x001840d4" 
    enter_dbg  =  "rs6k.up  0x00160b7c" 
    enter_dbg  =  "rspc.mp  0x001840d4" 
    enter_dbg  =  "rspc.up  0x00160b7c" 
Write down the enter_dbg address for the client you are going to boot. For example, if your client is an chrp-uniprocessor machine, you would write down the address 160b7c.
  • Attach a tty device to your client system (port 1).
  • Set up and perform the NIM operation that will require the client to boot over the network. Boot the client over the network.
  • After the client gets the boot image from the SPOT server, the debug screen will appear on the tty. At the > prompt, enter:
st  Enter_dbg_Value  2 
where Enter_dbg_Value is the number you wrote down in step 2 as your machine type’s enter_dbg value. Specifying a 2 at the address of the enter_dbg value prints the output to your tty.
  • Type g (for go) and press Enter to start the boot process.
  • Use Ctrl-s to temporarily stop the process as you watch the output on the tty. Use Ctrl-q to resume the process.
  • To rebuild your boot images in non-debug mode, use the following command:
nim  -Fo  check  SPOTName 
where SPOTName is the name of your SPOT.
If the boot image is left in debug mode, every time a client is booted from these boot images, the machine will stop and wait for a command at the debugger > prompt. If you attempt to use these debug-enabled boot images and there is not a tty attached to the client, the machine will appear to be hanging for no reason.

Client operations

Add / reset / remove clients

To see how to add, reset or remove a NIM standalone client please check AIX NIM Client.

Install a client

To see how to install a client with NIM see AIX NIM Client Installation.

Update a client

To update a client from the client you have to do the following steps:
  • list the resources available for the client
bash-3.2# nimclient -l -L lpar01
AIX_5300_LPP                            lpp_source
ITO_AIX_5300_LPP                        lpp_source
__smit_bundle_368870                    installp_bundle
ITO_AIX_5300_SPOT                       spot
ITO_AIX_53-05-CSP_SPOT                  spot
ITO_AIX_53-06-07-0818_LPP               lpp_source
ITO_AIX_53-06-07-0818_SPOT              spot
ITO_AIX_53-06-07-0818_LPP_FULL          lpp_source
ITO_AIX_53-06-07-0818_LPP_SPOT_FULL     spot
AIX_5300-09_LPP_FULL                    lpp_source
  • allocate the needed resource for the client
bash-3.2# nimclient -o allocate -a lpp_source=ITO_AIX_53-06-07-0818_LPP_FULL
  • check to see if the resource is indeed allocated
bash-3.2# nimclient -l -c resources lpar01
ITO_AIX_53-06-07-0818_LPP_FULL     lpp_source
  • start the update
bash-3.2# nimclient -o cust -a lpp_source=ITO_AIX_53-06-07-0818_LPP_FULL -a fixes=update_all

+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.

  Selected Filesets
  -----------------
  bos.adt.libm 5.3.0.40                       # Base Application Development...
  bos.adt.libm 5.3.0.61                       # Base Application Development...

  << End of Success Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    2  Selected to be installed, of which:
        2  Passed pre-installation verification
  ----
    2  Total to be installed

+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+

installp: APPLYING software for:
        bos.adt.libm 5.3.0.40


. . . . . << Copyright notice for bos.adt >> . . . . . . .
 Licensed Materials - Property of IBM

 5765G0300
   (C) Copyright International Business Machines Corp. 1987, 2006.
   (C) Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
   (C) Copyright Regents of the University of California 1982, 1983, 1986, 1987.
   (C) Copyright ISQUARE, Inc. 1990.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for bos.adt >>. . . .

Filesets processed:  1 of 2  (Total time:  2 secs).

installp: APPLYING software for:
        bos.adt.libm 5.3.0.61


. . . . . << Copyright notice for bos.adt >> . . . . . . .
 Licensed Materials - Property of IBM

 5765G0300
   Copyright International Business Machines Corp. 1987, 2008.
   Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
   Copyright Regents of the University of California 1982, 1983, 1986, 1987.
   Copyright ISQUARE, Inc. 1990.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for bos.adt >>. . . .

Finished processing all filesets.  (Total time:  12 secs).

+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
bos.adt.libm                5.3.0.40        USR         APPLY       SUCCESS
bos.adt.libm                5.3.0.61        USR         APPLY       SUCCESS
bash-3.2# oslevel -s
5300-06-07-0818
bash-3.2#
Note: The other time I got an error regarding I had to accept a new license. Adding the parameter -a accept_licenses=yes did the trick.

Install additional software from lpp_source

  • We need software that is currently not installed:
bash-3.2# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools xlC.aix50.rte xlC.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.adt.base              5.3.0.62  COMMITTED  Base Application Development
                                                 Toolkit
  bos.adt.lib               5.3.0.61  COMMITTED  Base Application Development
                                                 Libraries
  bos.perf.libperfstat      5.3.0.61  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         5.3.0.62  COMMITTED  Performance Statistics
                                                 Interface
  bos.perf.proctools        5.3.0.63  COMMITTED  Proc Filesystem Tools
  xlC.aix50.rte              8.0.0.0  COMMITTED  C Set ++ Runtime for AIX 5.0
  xlC.rte                    8.0.0.0  COMMITTED  C Set ++ Runtime

Path: /etc/objrepos
  bos.perf.libperfstat      5.3.0.50  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         5.3.0.62  COMMITTED  Performance Statistics
                                                 Interface
lslpp: Fileset bos.adt.libm not installed.
  • list the resources available for the client
bash-3.2# nimclient -l -L ms-lpar02
lpp_5300_05         lpp_source
spot_5300_05        spot
lpp_6100_00_01      lpp_source
spot_6100_00_01     spot
lpp_5300_06_07      lpp_source
spot_5300_06_07     spot
  • allocate the needed resource for the client
bash-3.2# nimclient -o allocate -a lpp_source=lpp_5300_06_07
  • check to see if the resource is indeed allocated
bash-3.2# nimclient -l -c resources ms-lpar02
lpp_5300_06_07     lpp_source
  • Start the installation
bash-3.2# nimclient -o cust -a lpp_source=lpp_5300_06_07 -a filesets=bos.adt.libm

+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.

  Selected Filesets
  -----------------
  bos.adt.libm 5.3.0.40                       # Base Application Development...
  bos.adt.libm 5.3.0.61                       # Base Application Development...

  << End of Success Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    2  Selected to be installed, of which:
        2  Passed pre-installation verification
  ----
    2  Total to be installed

+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+

installp: APPLYING software for:
        bos.adt.libm 5.3.0.40


. . . . . << Copyright notice for bos.adt >> . . . . . . .
 Licensed Materials - Property of IBM

 5765G0300
   (C) Copyright International Business Machines Corp. 1987, 2006.
   (C) Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
   (C) Copyright Regents of the University of California 1982, 1983, 1986, 1987.
   (C) Copyright ISQUARE, Inc. 1990.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for bos.adt >>. . . .

Filesets processed:  1 of 2  (Total time:  4 secs).

installp: APPLYING software for:
        bos.adt.libm 5.3.0.61


. . . . . << Copyright notice for bos.adt >> . . . . . . .
 Licensed Materials - Property of IBM

 5765G0300
   Copyright International Business Machines Corp. 1987, 2008.
   Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
   Copyright Regents of the University of California 1982, 1983, 1986, 1987.
   Copyright ISQUARE, Inc. 1990.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for bos.adt >>. . . .

Finished processing all filesets.  (Total time:  13 secs).

+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
bos.adt.libm                5.3.0.40        USR         APPLY       SUCCESS
bos.adt.libm                5.3.0.61        USR         APPLY       SUCCESS
Note: You can install multiple packages at once like this:
nimclient -o cust -a lpp_source=lpp_5300_06_07 -a filesets="bos.adt.libm bos.adt.base"
* Now the software is installed:
bash-3.2# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools xlC.aix50.rte xlC.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.adt.base              5.3.0.62  COMMITTED  Base Application Development
                                                 Toolkit
  bos.adt.lib               5.3.0.61  COMMITTED  Base Application Development
                                                 Libraries
  bos.adt.libm              5.3.0.61  APPLIED    Base Application Development
                                                 Math Library
  bos.perf.libperfstat      5.3.0.61  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         5.3.0.62  COMMITTED  Performance Statistics
                                                 Interface
  bos.perf.proctools        5.3.0.63  COMMITTED  Proc Filesystem Tools
  xlC.aix50.rte              8.0.0.0  COMMITTED  C Set ++ Runtime for AIX 5.0
  xlC.rte                    8.0.0.0  COMMITTED  C Set ++ Runtime

Path: /etc/objrepos
  bos.perf.libperfstat      5.3.0.50  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         5.3.0.62  COMMITTED  Performance Statistics
                                                 Interface
bash-3.2# oslevel -s
5300-06-07-0818
As you can see, the software is applied now, to commit it, run this command:
installp -c bos.adt.libm

Create a SOE

AIX SOE is the term IBM uses for imaging of AIX machines. The purpose is to create an overall document which describes into detail how AIX machines are installed and managed. Because that document should be creates by every company on their own I will focus on only creating the so-called golden image… or as you could say… One image to rule them all.
This image is created using the configuration outlined in AIX Post Install. Follow the guidelines under cloning AIX to see how an image can be created and restored.

Clone AIX

AIX cloning or imaging is an easy way to set up a standard environment for your AIX installations. Create one installation which counts for all and restore that image everytime you need a new installation. That way you can never forget a setting etc.

Create a mksysb image

We do this through the smity fastpath “smitty nim_mkres”
  • Select mksysb
Setup these values according to your environment:
aixmksysb01.jpg
And see the progress of the backup:
aixmksysb02.jpg
When nim is ready with creating the mksysb resource it will tell you so:
aixmksysb03.jpg 
Now you have a new resource:
bash-3.2# lsnim
master              machines        master
......
soe_20090713        resources       mksysb

Restore a mksysb image

We do this through the smity fastpath “smitty nim_bosinst”
  • Select a target, this is the machine you're gonna restore the image to
  • Select mksysb as thge installation type
  • Select the mksysb resource you just created
  • Select the spot related to the mksysb resource (should be of the same os level)
Adjust the parameters according to your environment, but at least set these settings to “yes”:
  • ACCEPT new license agreements?
    • This one is set twice under:
      • Customization FB Script to run at first reboot
      • installp Flags
If the lpar you selected is still running you'll see this message:
aixmksysbrestore02.jpg
The machine will reboot now, so you have to go to the console of the lpar, because you'll have to set the console, accept English as the installation language, and accept the default settings:
aixmksysbrestore01.jpg
After that the installation starts. You can monitor the installation through the console and through the nim commandline:
bash-3.2# lsnim -l ms-lpar01
ms-lpar01:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = mp
   if1            = net_10_1_0_0 ms-lpar01 0
   net_settings1  = auto auto
   cable_type1    = tp
   Cstate         = Base Operating System installation is being performed
   prev_state     = BOS installation has been enabled
   Mstate         = in the process of booting
   info           = BOS install 38% complete : 42% of mksysb data restored.
   boot           = boot
   mksysb         = soe_20090713
   nim_script     = nim_script
   spot           = spot_5300_06_07
   cpuid          = XXXXXXXXXXXX
   control        = master
   Cstate_result  = success
This is the lpar after the installation first reboot. Network and everything is being configured:
aixmksysbrestore03.jpg 

SUMA

SUMA Commands

I didn't test these commands myself yet. They have been used in a production environment however, so I think you could trust them.
  • update suma on server to use proxy
suma -c -a HTTP_PROXY=http://proxy:3128/
suma -c -a HTTPS_PROXY=http://proxy:3128/
suma -c -a FTP_PROXY=http://proxy:3128/
  • download updates to lpp_source
suma -x -a Action=Clean -a RqType=TL -a RqName=5300-09 -a DLTarget=/export/eznim/lpp_source/AIX_5300-09_LPP_FULL/
  • other updates MP/SP
suma -x -a Action=Preview -a RqType=SP -a RqName=5300-06-07 -a FilterDir=/export/eznim/lpp_source/ITO_AIX_53-06-07-0818_LPP_FULL/
suma -x -a Action=Download -a RqType=ML -a RqName=5300-09 -a DLTarget=/export/eznim/lpp_source/ITO_AIX_53-09

NIM Troubleshooting

NIM Error

0042-291 NIMkid: The NIM master is not currently allowing
registration of NIM clients.  Please seek assistance from your
network install administrator.
Solution: Set the parameter in the NIM Server so clients can register themselves:
  • smitty nim
  • Perform NIM Administration Tasks
  • Configure NIM Environment Options
  • Manage Client Registration
  • Allow Machines to Register Themselves as Clients?