Secrets of an AIX Administrator, Part 3: Tips for Patching Your AIX Server

Nowhere is the truism "the only constant thing is change" more relevant than in the IT world. We have to deal with the constant presence of external threats such as exploits and vulnerabilities that must be removed to prevent servers from being compromised. Then, too, software is continually updated as new tools are released and features are added to existing software and OSs. So as technology constantly evolves, keeping AIX servers up to date with the latest versions of the OS is crucial. In this article, we’ll look at how to patch AIX servers and maintain software on them.

Breaking Down the OS

Before we delve into the actual topic of patching, I want to back up a bit and discuss AIX's versioning structure. AIX has one of the easiest structures for keeping track of and understanding the levels of software on a server. Unlike Microsoft Windows, with its non-intuitive version numbers—for example, Windows 98 was released long before Windows 7—or an OS whose patch levels aren’t easily discerned when you look at the individual pieces of the OS at a glance (say, when watching a Solaris patch cluster as it runs), AIX has a simple way of iterating each version of its OS and components.

The IBM AIX OS comprises four main levels: version, release, technology level (TL), and service pack (SP). The version and release numbers denote the main levels of AIX and typically change as major features or hardware enhancements are released. The next level under Release is TL; TLs are groupings of large quantities of OS filesets that contain new functions or features. Finally, under TLs are SPs, which are more minor revisions (i.e., fixes) within each TL. You can view the levels installed on your system by running the oslevel -s command.

Likewise, AIX filesets generally reflect the OS level. You can obtain the levels for a particular fileset by running the lslpp -l command against the fileset; the four numbers returned in the command output generally reflect the version, release, TL, and SP pretty well. However, other pieces of software (i.e., DB2) often don’t have any relationship to the level of the AIX OS.

Commit-Apply-Commit

The first step to maintaining the AIX OS on servers is to understand the difference between a committed and an applied fileset. When a fileset is committed on a server, there’s no way to roll back to a previous version of the software, short of a reinstall. The committed fileset is locked in place. When a fileset is applied, however, it can be rejected and removed from the server in the event that applying the fileset causes something to go wrong on the server.

When it’s time to apply some patches or upgrade a server, especially when dealing with TLs and SPs, I recommend starting with a mksysb backup of the server first. Then, after the backup is done, commit any applicable filesets to cement the state of the OS. Following this procedure will provide a solid foundation and restore point for the rest of the process.

Install the software in the applied state so that in the event that something is off-kilter with the patches or upgrade, it can be promptly rejected. If you can afford to do so (even if the software doesn’t require it), I advise that you also reboot the server afterward to make sure that the system periodically gets tested and any latent problems can be shaken out. After a few weeks of the server running well and going through a vetting period, commit the software to complete the process and set the new baseline for the server.

Stick with the Big Groupings

Sometimes an application will require an individual OS fileset to be upgraded in order to work properly. For example, the pre-check for installing Oracle will often call out several filesets to be patched before the database software can be installed. However, I never advise upgrading filesets on a one-off basis; I always stick with the practice of upgrading through TLs or SPs.

When admins deviate from this practice and place a number of incongruous updates on their servers, it makes administration more difficult in the long run. Commands like oslevel or instfix will return inconsistent numbers for the software levels that are running. Future patches may balk at having some components already at higher levels than they should be. Plus, if you have to contact IBM Support for help, it will make explaining your server much more complicated than simply saying, "I'm running AIX 6.1, TL 06, SP 05."

The only cases in which I would endorse deviating from this practice are if something is seriously broken on the server or if a high-profile vulnerability could impact the server. In such cases, it’s appropriate to add an IBM eFix and resume the recommended patching practice after IBM integrates the fix into a future TL or SP.

Use NIM or a Software Repository

One big headache that plagues AIX systems admins is working in an environment where servers are all across the board with their OSs. I’ve been in some shops that ran every version of AIX, from 5.1 to 7.1, with multiple TLs and SPs in each version and release. The variation of OS versions made keeping track of servers a nightmare: finding which systems needed to be patched, which ones had vulnerabilities, and picking and choosing which needed immediate patching, especially when IBM emailed warnings about high-impact alerts.

The Network Installation Manager (NIM) utility makes it a breeze to install the same level of OS onto multiple servers at once. Find a good level of the AIX OS that works for your servers, install NIM onto a small LPAR, and turn that system into a bastion host that serves as a software fix repository. By doing this, you’ll only need to be concerned with one or two iterations of the OS, and you’ll find it easier to mass-deploy a homogenous OS configuration.

Watch for RPMs Versus Filesets

With the advent of Linux affinity in the AIX OS about a decade ago, it became possible to compile and install Red Hat packages onto Power Systems. While most OS patches pertain to AIX filesets, there are a fair number of applications that work closely with the OS that use RPM packages. These applications include utilities like sudo, wget, and gcc, which all have a role in administration.

When you go to patch your servers, be sure to also check the RPM packages installed on your system to determine whether they should also be updated. Use both the rpm and lslpp commands to check all those other pieces of software.

Use nimadm_migrate and alt_disk_install Strategies

One of the best features of AIX that has come out in the past few years is a combination of two technologies: nimadm_migrate, which is the AIX System Management Interface Tool (SMIT) fastpath to migrating a client box to a new version of the OS, and alt_disk_install. These two commands allow admins to take the currently running OS, clone it onto a blank hdisk, and then even apply patches onto that drive. This way, the server stays completely up and there’s no risk to the currently running root volume group (rootvg) as the server is migrated to a new SP, TL, or even a completely new version and release of the OS. All it takes is a reboot to switch over to the fresh OS. And in the event something goes wrong, the server just needs to be rebooted onto the original disk to put things back the way they were.

Stay Current with Patching

Keeping your servers up to date with the AIX OS and patching them on a routine schedule can stave off unnecessary downtime, give your users more features, and periodically "shake the tree" to see if anything falls out. We'll continue our exploration of AIX management in the next article, when we look at a few techniques to make networking more understandable and manageable.

Secrets of an AIX Administrator, Part 4: Networking

In AIX, I’ve found that networking is probably one of the most complicated subjects around. Some components of networking are addressed by making changes to flat files, while others are handled through the Object Data Manager (ODM). There are typically at least three commands or ways of doing something, from adding a route to turning off an interface. And what seems like a subtle or innocuous action can wind up causing headaches on the server in due time—for example, a reboot that brings out duplicate default routes.

I have extensive experience in both setting up and messing up networking on AIX servers worldwide. Along the way, I’ve taken copious notes on how to manage networking, so that others will hopefully avoid the same plights I encountered. The following is a condensed version of the knowledge I've gleaned about AIX networking through hard-won experience.

Understand Physical and Logical Relationships

Although most forms of UNIX and Linux will have only one device under the /dev directory to describe each networking interface, AIX is different in that it will create both a device for the physical interface (e.g., ent0) and a device for the logical interface (e.g., en0). Each device serves a unique purpose; for example, media speeds are set on the physical interface, whereas IP addresses are defined on the logical interface.

Knowing this about the physical and logical devices, your one key take-away is that if you need to do something to the physical interface, you’ll need to shut down, disable, or remove the logical interface before AIX will accept any commands to manipulate the physical interface.

Establish EtherChannel Devices First

Before you define any IPs on your servers, to create redundancy and get higher performance, I recommend setting up devices that use Cisco EtherChannel technology wherever possible. Much like bonding in Linux or IP network multipathing (IPMP) in Solaris, EtherChannel lets you gather multiple physical interfaces into one connection. The interfaces can be aggregated so that traffic goes down both pipes simultaneously, or they can work in an active-passive mode conditional upon failover. The "smitty etherchannel" fastpath is the easiest way I’ve found to define EtherChannel devices.

One thing worth noting is that the EtherChannel device, although logical in nature, still has a physical and a logical component. For example, if you choose to use ent0 and ent1 to create an EtherChannel adapter, the physical device will be created as ent2 with a logical device of en2. The EtherChannel adapter behaves almost identically to any other standard network adapter as far as setting network tunable parameters or establishing IP aliases, but the physical interface will show some additional information when viewed using the lsdev or lsattr command.

Use SMIT for Your Initial IPs

I’ve always subscribed to the philosophy that the proficiency of an AIX administrator can generally be determined by how often or seldom the admin relies upon the System Management Interface Tool (SMIT) to do his or her work. But there are a few tasks where, rather than memorize long commands with dozens of flags, I will turn to SMIT to accomplish my work. Setting up initial IP addresses is one of those tasks.

The "smitty tcpip" fastpath is the quickest route to navigating to the menus for setting up everything network related. From the SMIT Minimum Configuration & Startup screen, define your first IP address and default gateway through the menu. Then, as you need to add IPs to other interfaces, use the "smitty chinet" fastpath for those other devices. Using the Minimum Configuration & Startup screen to define additional interfaces could adversely affect the routing on the server.

Use the ODM Wisely

Most networking information in AIX is stored within the ODM. Only a few flat files that pertain to networking can be modified manually, such as /etc/hosts or the /etc/rc.tcpip and /etc/rc.nfs files. This can make it difficult to manage, change, or remove items like network routing from the server quickly and easily.

Aside from the initial plumbing-up of IPs on interfaces, the one other thing I’ll do through the ODM is to assign any IP aliases onto devices. I do so using the following command:
 

chdev -l enX -a alias4=$IP,$NETMASK

This way, following a reboot, all IPs on that specific adapter will come up automatically. However, beyond this point, I stick with modifying other flat files.

Case in point: I’ve found the task of adding additional routes to a server by means of the ODM to be a beast. I've had to hunt through obscure corners of the CuAt database when trying to remove or adjust a single route. Instead of trying to jump through hoops in adding or removing routes through the ODM, I typically modify the /etc/rc.net file under "Part II - Traditional Configuration" and add all my routes to this area using the route command. This way, I can delete active routes off the server using a route delete command and strip them from the file without having to do a reboot or worrying about the ODM.

Smoother Networking

The tips I've provided here should help make your AIX networking routine go a little more smoothly. In my next article, I'll pull together the information I've provided in this article and the other "Secrets of an AIX Administrator" articles (see the Learning Path for an articles list and links), and I’ll conclude this series by giving you some tips on what to do following your first reboot after building a new server.

Secrets of an AIX Administrator, Part 2: File Systems

On the list of the most common activities that AIX systems administrators perform, file system maintenance comes right after user ID maintenance and password resets. I can’t count the number of times I’ve created, grown, shrunk, or deleted file systems in the course of my professional career. When you think about it, this makes sense: Almost every user needs space to store his or her data.

AIX’s flexibility for managing data comes with a downside: There are many ways to make an error in file system maintenance that will set the stage for future problems on the server. It takes only one or two commands to make a server practically impossible to grow, so that you can’t keep up with demands for additional places for data. Growing a file system the wrong way—for example, growing an IBM PowerHA file system on a server without using the PowerHA command set—can also create more work in the long run. To help you avoid file system maintenance mistakes that could result in inefficiencies on your server, this article offers some pearls of file system wisdom that I’ve acquired over the years.

Use the Root Volume Group for Rootvg Data Only

The most common mistake administrators—especially novice admins—make is to put a file system inside the root volume group (rootvg) or to use space that’s within the rootvg for application-related work. Here are the three big reasons you shouldn’t do this:

• If you’re using internal disks, you’ll be more limited as to how large you can make your file systems.
• In the event that the file systems need to be exported to another system, you’ll wind up exporting your entire rootvg.
• Putting a file system within rootvg can make backups performed using mksysb or other AIX backup utilities exceedingly large.

I recommend using rootvg space only for applications in limited circumstances, such as small web servers, personal AIX systems, and temporary testing environments.

Break Down Volume Groups

When you begin to plan a Power Systems server environment, think about how to use your disk space efficiently and purposefully. Separate the data into volume groups according to guidelines such as how dynamically VGs will change, types of data and access levels, and types of storage. For example, AIX 6.1 and 7.1 provide a new flag, -X, for the mkvg and chvg commands, which can restrict VGs to using a certain type of storage, such as solid-state drive (SSD) disks, thereby preventing a mixed-storage environment that could cause a hit on the server’s I/O. Use these tools with foresight to design a maintainable system.

Use JFS2 and Scalable VGs

The more modern versions of AIX include two features of the Logical Volume Manager (LVM) that I recommend be used almost everywhere. The first is scalable volume groups. In the past, AIX had limitations as to how many physical volumes could be in a volume group, how large the volume group could get, and how large the physical partition size could be. Scalable volume groups eliminate almost every reasonable constraint previously associated with managing volume groups.

The second feature is the enhanced journaled file system structure, better known as JFS2. Just as scalable volume groups improved management of larger disks, JFS2 took the limitations of JFS and rendered them obsolete by permitting file systems to grow larger than the 2TB limit. JFS2 also gave us better tools, such as the ability to shrink the size of a file system. If you create your volume groups and file systems using these two features, you’ll eliminate 90 percent of the growth and sustainability problems that existed just a decade ago.

Keep Physical Volumes the Same Size Whenever Possible

Aesthetically speaking, there’s nothing worse than pulling up information on the physical volumes in a volume group and seeing all sorts of random disk sizes. I’ve seen servers with disks ranging from 20GB to 500GB—all in the same volume group. This discrepancy in sizes can be caused by disks being mixed and matched across local and SAN storage, people responding to emergency file system growth requests, and sloppy SAN administrators. However, this kind of non-uniform disk layout can have other implications, such as limiting growth (e.g., a logical volume’s inter-disk policy is reached) and impacting I/O (e.g., a large, hot disk takes 80 percent or more of the I/O for a volume group).

I recommend working with SAN administrators and those in charge of storage to come up with a uniform disk-size policy for volume groups. Set the physical volume size to something appropriate—for example, 50GB—and stick to that size policy. If you set the policies for your logical volumes to be striped across all the disks (assuming there’s some type of RAID configuration on the back end, as is the case in most SAN environments), you’ll avoid creating any hot spots. This configuration has the added benefit of making the root disk easily visible when you’re booting through a System Management Services (SMS) menu if you pick a unique size, such as 48GB, for its disks.

Simple Steps to Better File Management

By following my advice, you’ll find that once-troublesome file-system requests will become one of the easiest tasks you to perform. The tips I provided will also help you in the long run with patching your servers.

The Secrets of an AIX Administrator Series

In future issues of POWER IT Pro, we’ll move on to other areas of managing servers and share handy techniques for improving systems, including proper patching protocols.

Secrets of an AIX Administrator, Part 1

I often get e-mail messages from people all over the world who’ve benefitted from the articles I’ve written about IBM AIX systems administration. Some just say, “Thank you,” while others are looking for more details about a particular topic or asking for clarification on a point I’ve made. Some are looking for information on a subject I haven’t yet addressed or are actually requesting a specific topic. In all the incoming email messages, there are a surprising number of people who ask a simple, yet complex, question: “How can I become an AIX systems administrator?”

As I reply back to all of these people and try to help those looking for assistance, I ponder the challenges newcomers to AIX administration must face. In the past, I could sense that some people were in dire straits and knew that things were going to turn out badly. For others, I could tell they had been given a bad task and knew that they would have to paddle upstream all the way. Some seemed just plain lost. But then I thought to myself, “Wouldn’t it be great if someone could share all of the experiences that can’t be gained from a classroom or training environment so that newcomers wouldn’t have to go through the struggles and mistakes that other administrators have made?”

With these experiences in mind, I knew I had to write this series of articles on the secrets of being an AIX systems administrator. My goal is to cover the things I wish I had known about AIX before I became an admin. These articles will cover the practical knowledge that can’t be acquired within training classes or books or gleaned from the man pages on a server. And, hopefully, it will save you from some of the headaches and heartaches I’ve faced in my many years of taking care of servers in all sorts of environments, from education and telecommunications to agriculture and outsourcing.

Breaking into the Business

To kick off this series, it’s best to start at the beginning. How does someone get into the position of being an AIX systems administrator? Do you have to have some accreditation from IBM? Is a computer science degree needed? Does a secret handshake or shadowy ritual grant you the powers of root access?

I’ve found that no two administrators took the exact same path to get where they are. On the one hand, I’ve met folks such as electrical engineers who have extensive programming knowledge, possess master’s degrees and become admins. On the other hand, there are plenty of people—like me—who come from non-technical focuses and have minimal formal training. (I majored in history and minored in political science, and took only one computer science class in college, which was primarily an introduction to Microsoft Windows and Office.) But there are some commonalities and steps that any admin will recommend for those interested in becoming an AIX systems administrator.

Check your motives. Ask yourself why you want to be an AIX systems administrator. There are plenty of rewards to becoming an admin, such as decent compensation, telecommuting, and flexible schedules; finding the right company or institution can make the job even better. But being an admin comes with plenty of pain, such as on-call rotations, long nights, off-hours work, and the perception that because you can telecommute, you can be contacted any time. So don’t charge lightly into this kind of commitment—you must first determine what’s important to you in your life.

Get some exposure. If you decide you want to become an AIX admin, start by getting some exposure to an AIX server. Ideally, you should find a simple system and have your own user ID for ­experimentation purposes. But servers aren’t typically available for just anyone to pick up; you’ll have to hunt one down. Your company might have a server available and some flexibility in setting up an account for you, but I’ve found that the easiest place to find access is through a university or a local community college. In most cases, the university will give you a bare-bones account with a few megabytes of storage for free just for taking some classes. You might have to contact the department that supports the servers to find out if they have an AIX server or other forms of UNIX, and you’ll need to be specific about what kind of access you’ll want.

Read all you can. There’s a tremendous wealth of free information out there on AIX systems administration. Publications such as POWER IT Pro keep people up-to-date with the latest news and information about AIX technology with practical tips and articles. IBM’s Redbooks on AIX are available on the web for free download in PDF format. And most admins will have a small library of books from publishers such as O’Reilly that they will lend out or recommend that you purchase. You can beef up your book knowledge quickly and easily, but real-world experience will always trump theory and handbooks.

Install Linux or play with other flavors of UNIX. Since its inception in the early 1990s, Linux has become hugely prolific. And with the advent of AIX 5L, there’s integration between AIX and Linux. Even if you have access to an AIX server, by building your own Linux box at home, you’ll be able to play around and experiment with commands as the root user—blowing a few things up in a safe environment—before you lay hands on an AIX production server that’s worth millions of dollars.

Find a mentor. Try to locate a skilled AIX admin who’s generous with sharing information and has a patient teacher’s spirit. Don’t ever think you know more than him or her; approach your admin with a kind, inquisitive attitude and learn as much as you can from seeing how he or she reacts to various situations over time. Take copious notes and ask questions whenever you can. And buy your mentor a meal or drink after work to show your appreciation for his or her knowledge and wisdom.

Apply and prepare for rejection. When trying to find a position, there’s a great paradox that happens to nearly everyone: You can’t get a job as an admin without experience, and you can’t get experience without a job. This is why you’ll need to beef up your resume by demonstrating all the time and energy you’ve invested into learning AIX. You might need to quantify this by passing a certification exam, getting references from qualified people, or finding a low-level entry position and working your way up from there. Be ready to get rejected, but don’t give up hope; stay determined and watchful.

If you follow these steps, you’ll be on the right path for a solid career in this field. And from there, you’ll understand more of the secrets of being an AIX systems administrator.

And what are the secrets, you ask? Let’s start with the all-important user ID.

User ID Utopia

One of the first things that AIX systems administrators get tasked with in order to familiarize themselves with their environments is user ID management. The theory is that this is one of the best ways for admins to get to know their user communities and the function and purpose of their servers. However, more often than not, user ID management is seen almost as a janitorial task—a sort of necessary evil that everyone has to take part in from time to time. That’s because it’s no fun working through endless password resets, changing huge groups of users who weren’t set up properly initially, or manually managing dozens of user IDs on many servers.

There are some good rules for managing user IDs that aren’t typically found in manuals or training classes. I’ve acquired these guidelines after years of seeing things go amiss, taking drastic measures to fix problems, and being kept awake at all hours of the night. In general, if you use these secrets, your life as an AIX admin will go much more smoothly.

Protect the root password. The more people who know the root password, the greater the likelihood of someone running the wrong command. Guard it as though it’s a bank account number.

Homogenize user IDs. Try to get a spreadsheet or system going to keep all of the user ID numbers for all of the user IDs across your servers in sync. This way, you won’t need to be brought in to set permissions or ownership should someone need to transfer files from one server to another or use an NFS mount across systems. If you’re administering more than just a few servers, consider using a utility such as NIS or LDAP to handle things from a single location.

Make important user IDs local. Make sure that any user IDs and home directories, such as those used by applications, are defined locally to the server. I’ve seen some environments in which network outages have crippled mission-critical servers because all of their resources were located externally to the systems. If your business won’t work without some specific user IDs being present, keep them local.

Establish all groups up front. One of the strangest problems an admin can encounter is getting the message, “Cannot set process credentials,” when someone attempts to log in. This message happens when the user’s primary group doesn’t exist on the server. To solve this problem and make management easier in the long run, give all the user IDs all of the possible groups they need up front.

Use proper home directory management. If you have user IDs that consume a good deal of space in their home directories, make the home directories into their own file systems, and then set permissions and ownership as you create the user IDs. Put the home directories in places in which they make sense; I’ve seen home directories placed in all sorts of locations such as \etc and \tmp, where they could potentially harm the server or cause general confusion. Keep them in \home or another file system structure that stands out for the applications on the server.

Set password expiration warnings. If you’re using best computer security practices or have to conform to federal laws or regulations, your user IDs will have regularly scheduled password expiration dates. Do yourself a favor and set a reminder email to go to your users a week or so before the change so that they can take action before they get locked out.

Install sudo. Lastly, I believe that there’s no greater application for managing user IDs than sudo. Sudo provides a safe and secure way to give access to privileged commands while minimizing risk to the server. The sudo configuration file can be ported across multiple servers and even onto different versions of AIX, UNIX, or Linux systems. I believe in using it on any server I administer, if for no other reason than providing another audit trail to becoming the root user.

More Secrets to Come

You now know some of the secrets of an AIX systems administrator when it comes to the most basic level of administration in caring for user IDs. In future issues of POWER IT Pro, we’ll move on to other areas of managing servers and share handy techniques for improving systems, including file systems and LVM maintenance.