VIOS Advisor

Have You Seen the VIOS Advisor?

Download the tool here. I'll let IBM developerWorks provide the introduction.

"The VIOS advisor is an application that runs within the customer's VIOS for a user specified amount of time (hours), which polls and collects key performance metrics before analyzing results and providing a health check report and proposes changes to the environment or areas to investigate further.

"The goal of the VIOS advisor is not to provide another monitoring tool, but instead have an expert system view performance metrics already available to the customer and make assessments and recommendations based on the expertise and experience available within the IBM systems performance group.

     Download vios_advisor.zip from the link provided in download section.
     Unzip vios_advisor.zip on a workstation that has a web browser.
     ftp vios_advisor onto the VIOS you wish to monitor. (Place in any directory.
     chmod +x vios_advisor to give the application execution priviliges.

 "The application "vios_advisor" takes only one paramter, which is the duration of the the monitoring period, in minutes.

 "For example, to monitor the VIOS for 30 minutes, run:

      vios_advisor 30

   Usage Statement:

      Usage: vios_advisor
      duration_in_minutes:
      Recommended monitoring time = >= 30 min
      Minimum monitoring time = 5 min (only recommended for settings verification)
      Maximum monitoring time = 1440 min (24 hours)

      -v : Version

"The vios_advisor application is silent (does not produce any output to screen) and upon termination, will generate an xml file in the current running directory labeled:

     vios_advisor.xml

"Copy over the vios_advisor.xml file to the workstation where the zip file: vios_advisor.zip was extracted, and place the file in the vios_advisor folder. Open the vios_advisor.xml file with the web-browser of your choice to see the report.

"The measured overhead for the VIOS Advisor is minimal. An increase in CPU consumption of 0.1 cores was measured on a POWER7 server. Memory consumption will vary based on the number of physical I/O devices in the VIOS, but expect the advisor to consume 2-20 MB of memory."

I downloaded the advisor and extracted the files from the .zip file. Then I selected the vios_advisor_example file that was located in the newly created directory. This was the output in my browser:

I copied the vios_advisor file to my VIOS and ran chmod on it so that I could run the tool. Then I ran a quick test to make sure it worked:

$ chmod u+x vios_advisor
$ vios_advisor -v
vios_advisor  Version: 121211B

Then I ran:

 $ vios_advisor

 Usage: vios_advisor

      duration_in_minutes:
            Recommended monitoring time = >= 30 min
            Minimum monitoring time =         5 min  (only recommended for settings verification)
            Maximum monitoring time =      1440 min  (24 hours)

       -v :    Version

Since this was a test, I chose the minimum of five minutes to verify the settings.

$  vios_advisor 5

At the end of the test, I received a file called vios_advisor.xml. I copied that back to my PC, putting it in the directory that my vios_advisor.zip file was extracted to. Then I examined the report.

I'm sure IBM will continue to enhance the tool, helped along by user feedback.

So have you tried the VIOS advisor? How would you improve it?

DLPAR==Resolving an Issue with Dual HMCs

Recently, a customer was unable to run a DLPAR command against some of the LPARs on their frame. That in itself isn't unusual. Generally in these situations the network isn't communicating between the HMC and the LPAR, or perhaps RMC daemons need to be restarted somewhere.

This environment had dual HMCs connected to the managed system. HMC1 controlled some of the LPARs and HMC2 controlled others, but not by design. Although there was no rhyme or reason to it, for simplification let's say that HMC1 was controlling LPAR1 and LPAR3 and HMC2 was controlling LPAR2 and LPAR4. The correct setup would have been HMC1 and HMC2 controlling LPAR1, LPAR2, LPAR3 and LPAR4. In reality approximately 40 LPARs were on the frame, with each HMC controlling approximately half of the LPARs.

If you were on HMC1, you could DLPAR LPAR1 and LPAR3 with no issues. If you were on HMC2, you could DLPAR LPAR2 and LPAR4 with no issues. The problem was that the only way to know which HMC was controlling which LPAR was to either login to the HMC command line and run lspartition –dlpar, or use the HMC GUI and select HMC Management > View Network Topology. There was no way to know which HMC you needed to login to to manage which LPAR. This headache needed to be resolved.

Initially we did some troubleshooting with IBM Support. That resulted in us running things like:

            /usr/sbin/rsct/install/bin/recfgct
            /usr/sbin/rsct/bin/rmcctrl –p

We tried getting root access via pedbg. We also tried collecting a snap:

            /usr/sbin/rsct/bin/ctsnap

Eventually, once we escalated high enough up the support food chain, someone noticed a very basic HMC setup problem:

            The LPAR IBM.MgmtDomainRM default file shows this msg where it's attempting to create a    IBM.MCP entry for hmc1. It fails with Error number 14, duplicate key for localhost.

            2610-652 The specified time limit has been exceeded.
            Mon Feb 17 13:04:32 CST 2014(439849)      ../../../../../src/rsct/rm/MgmtDomainRM/MCP_cfg.c/01438/1.22  2613-034 Error number 14 was          returned when attempting to define an IBM.MCP resource.
            2610-014 The key token localhost is a duplicate.

During the initial build of the HMCs, they had been given their unique hostname and IP address, but somehow someone made a change that resulted in both hostnames being reset to localhost. Since these HMCs had the same hostname and ran on the same network, only one of the two was capable of managing LPARs at any given time. The other one would always fail.

Needless to say, if you run multiple HMCs, make sure they have unique hostnames. And in any environment, it's essential to establish good change control so people aren't making changes to systems without proper approvals and documentation