Thumb Rules – AIX ADMIN

1.    Collect System Information before doing any change on your server.

2.    Ensure the current OS level is consistent:     “oslevel -s;oslevel -r;instfix -i|grep ML;instfix -i|grep SP;lppchk -v” [If the os is inconsistent, then first bring the os level to consistent state and then proceed with the change].

3.    Check the lv/filesystems consistency “df -k”(df should not hang); all lvs should be in sync state “lsvg -o|lsvg -il”.

4.    Check errpt & ‘alog -t console -o’ to see if there are any errors.

5.    Perform ‘pwdck’ & ‘grpck’ to verify password & group file consistency.

6.    Ensure the root user has not expired.

7.    Ensure your id on the server is working.

8.    Ensure mksysb(OS image backup) is taken. Verify mksysb with “lsmksysb -l -f /mksysbimg”(check size).

9.    Check the /etc/exclude.rootvg to see if any important filesystem/dir is excluded from the mksysb backup.

10.    Take the TSM backup of mksysb as well.

11.    Ensure the scheduled TSM backup of filesystems is done.

12.    Verify boot list & boot device:   “bootlist -m normal -o”  “ipl_varyon -i”

13.    Ensure hd5(bootlv) is 32MB (contigous PPs)  [very important for migration]

14.    Remember Application/Database team are responsible for their Application/Database backup and restore. Therefore alert the application teams so that they take their backups and other precautions before we perform our change.

15.    Ensure the ‘console’ for the server is working.

16.    Initiate the change on console. If there is any network disconnection during the change, you can reconnect to the console and get the menus back.

17.    Remember three essential things are in place before you perform any change “backup(mksysb); system information; console”

18.    Ensure you are on right server(hostname;ifconfig -a) before you perform change.

19.    Remember one change at one time; multiple changes could cause problem & can complicate troubleshooting.

20.    Pre-inform the monitoring team about your change; so that there are no vague alerts/PRs.

21.    Ensure there is enough free filesystem space(/usr, /var, / ), required for the change.

22.    Before and After the change, ensure the filesystems are well below the thresholds(75%).
23.    Perform changes only for the approved CRs(change requests).

24.    Check the history of the servers(PRs or CRs)…to see if there were any issues or change failures for these servers.

25.    Ensure that there is no clash of SAs over changes; one change being performed by multiple SAs. Its better to verify with the ‘who -u’ command, to see if there are any SAs already working on the server.

26.    If there are two disks in rootvg, then perform alt disk clone for one disk. This is fastest & safest backout method in case of any failure. Though you perform alt disk clone, ensure you as well take mksysb.

27.    For migration change, check if there is SAN(IBM/EMC..) used, if so, then you have to follow the procedure of exporting vgs, uninstall sdd* fileset;and after migration reinstall sdd* fileset, reimport vgs etc.

28.    Ensure hd5(bootlv) is 32MB (contigous PPs)  [very important for migration]

29.    EXPECT THE UNEXPECTED : Ensure you have the clear backout plan in place.

30.    Have the patches/filesets pre-downloaded and verified.

31.    Check/verify the repositories on NIM/central server; check if these repositories were tested/used earlier.

32.    Verify the connectivity between the client and NIM server. In some accounts the admin network is to be exclusively enabled(cabled) for the changes.

33.    Take care of the network zones(public, private, behind firewall); each zone may have their separate NIM server.

34.    Take care of the static routes(which may differ across zones). Ensure the static routes are not disturbed after the change.

35.    Ensure there are no other conflicting changes from other departments such as SAN, network, firewall, application.. which could dampen your change.

36.    Maintain/record the commands run/console output in the notepad(named after the change).

37.    Update the change record with the output/artefact. You can execute the commands with date command, so that the date & time when the command is run is recorded “date;uptime;oslevel -r”.

38.    If the system has not rebooted from longtime(> 100 days); then perform ‘bosboot’ & then reboot the machine(verify the fs/appln. after reboot), & then commence with the migration/upgrade. [Donot reboot the machine if the bosboot fails!]

39.    Check filesystems count “df -k|wc -l”  ; verify the count after migration or reboot.

40.    Ensure there are no schedule reboots in crontab. If there is any then comment it before you proceed with the change.

41.    Look for the log messages carefully; donot ignore warnings.

42.    Perform preview(TL/SP upgrade) before you perform actual change; see if there are any errors reported in preview(look for keyword ‘error’ / ‘fail’); look for the tail/summary of messages;

43.    Though the preview may report as ‘ok’ at the header, still you have to look in the messages and read the tail/summary of preview.

44.    If preview reports any dependency/requisite fileset missing then have those downloaded as well.

45.    Ensure you have enough free space in rootvg. Min of 1-2 GB to be free in rootvg(TL upgrade/OS migration).

46.    Ensure application team have tested their application on the new TL/SP/OS to which you are upgrading your system.

47.    If you have multiple putty sessions opened; then name the sessions accordingly [Putty -> under behaviour -> window title]; this will help you in quickly getting to the right session.

48.    On sev1 issues, update the SDM in the ST multichat at regular intervals.

49.    Over the conf voice call, if they verbally request you to perform any change, get the confirmation in writing in the multi ST chat.

50.    Update the PR in regular intervals.

51.    Update your team with the issue status(via mail).

52.    Document any new learnings(from issues/changes) and share it with team.

53.    Ensure proper validated CR procedure is in place;  Precheck -> Installation -> Backout -> Verification

54.    Save the severity multichats; enable autologging of multichats in Sametime.

55.    If the junior SA resource is performing the change for first time(first time in any account); then let the techlead club him with a senior SA.

56.    SAs First time execution (or first time in an account) or to say any first time had to be dealt carefully. Techleads have to take extra precautions.

57.    Any important information/document, please ensure that you update it in your account teamroom.

58.    Ensure for TL upgrades, you go by TL by TL, shortcut to direct TL could sometimes cause problem.

59.    Check if the server is running any cluster (HACMP), if so then you have to follow different procedure.