How to Add a Node to an HACMP Cluster

Contents

1. Introduction
2. STEP 1: Additional Boot Addresses
3. STEP 2: Import LUNs of shared volume groups into new node
4. STEP 3: Add the new node to the cluster
5. STEP 4: Restart cluster services on new node
6. STEP 5: Add new IP addresses to the cluster configuration
7. STEP 6: Start HACMP on new node
8. STEP 7: Remove Dependencies between Resource Groups
9. STEP 8: Add new node to all resource groups
10. STEP 9: Restore dependencies between Resource Groups
11. STEP 10: Final synchronization of the cluster

1. Related Information

Introduction

This article describes how to add a third node »hoppy« to an HACMP cluster. We find the following situation:

barney# clRGinfo

-----------------------------------------------------------------------------

Group Name     Group State                  Node           

-----------------------------------------------------------------------------

RES_GRP_01     ONLINE                       barney        

               OFFLINE                      betty        

RES_GRP_02     ONLINE                       barney        

               OFFLINE                      betty        

Conclusion: barney and betty are members of two resource groups, but all resources are currently active on barney.

STEP 1: Additional Boot Addresses

Before we can start we need new boot IP addresses for all HACMP networks. The boot addresses have to be assigned to the network interfaces of our new nodehoppy in AIX:

hoppy# smitty chinet

                                                        [Entry Fields]

  Network Interface Name                              en0

  INTERNET ADDRESS (dotted decimal)                  [<boot ip address>]

  Network MASK (hexadecimal or dotted decimal)       [<netmask>>]

  Current STATE                                       up                +

  Use Address Resolution Protocol (ARP)?              yes               +

Note: Usually the boot addresses are not equal to the host specific ip addresses of the interfaces (such as the hostname's ip address). They become persistent addresses instead. However, in a virtualized environment we could use the host specific IP addresses as boot addresses and could avoid the need for persistent addresses at all.

We add the new node's boot and persistent addresses to /etc/hosts on all nodes. The old node's IP addresses as well as all cluster service addresses have to be added to the new node's /etc/hosts.

Finally we copy /usr/es/sbin/cluster/netmon.cf to the new node:

hoppy# scp barney:/usr/es/sbin/cluster/netmon.cf /usr/es/sbin/cluster

STEP 2: Import LUNs of Shared Volume Groups into New Node

All storage assigned to the resource groups RES_GRP_01 and RES_GRP_02 has to be visible to the new node. Be sure to set the reserve_policy to no_reserve for all shared storage on the new node:

hoppy# cfgmgr

hoppy# chdev -a reserve_policy=no_reserve hdiskN1

hoppy# chdev -a reserve_policy=no_reserve hdiskN2

hoppy# chdev -a reserve_policy=no_reserve hdiskN3

                      .

                      .

                      .

Now we are ready to import all shared Volume Groups into the new node.

hoppy# importvg -n -V  -y  

Get information about major number, Volume Group, and PVID from existing nodes.

STEP 3: Add the New Node to the Cluster

Now we are ready to add hoppy as new node to the cluster. We do this on one of the existing nodes.

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Topology Configuration

       -> Configure HACMP Nodes

         -> Add a Node to the HACMP Cluster

                                                            [Entry Fields]

     * Node Name                                         [hoppy]

      Communication Path to Node                         [<boot address of new node>]

STEP 4: Restart Cluster Services on new Node

We need to restart the cluster services on the new node. First we verify the configuration and synchronize it to the other nodes:

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Verification and Synchronization

clstat shows new node as down.

To be sure we check /usr/es/sbin/cluster/etc/rhosts on all nodes -> all cluster addresses should be in it!

At this point the cluster services on the new node can be restarted:

hoppy# stopsrc -g cluster

hoppy# stopsrc -s clcomdES     

hoppy# startsrc -s clcomdES

hoppy# startsrc -g cluster

STEP 5: Add new IP Addresses to the Cluster Configuration

First we let HACMP discover what we have...

barney# smitty hacmp

   -> Extended Configuration

     -> Discover HACMP-related Information from Configured Nodes

Then we add the new ip addresses to the cluster

barney# smitty hacmp

   -> Extended Configuration 

     -> Extended Topology Configuration 

       -> Configure HACMP Communication Interfaces/Devices

         -> Add Communication Interfaces/Devices

           -> Add Pre-defined Communication Interfaces and Devices

             -> Communication Interfaces

               -> Select Network

                                                               [Entry Fields]

        * IP Label/Address                                   [<new boot address>]

        * Network Type                                        ether

        * Network Name                                        (selected network)

        * Node Name                                          []

          Network Interface                                  []

When we are done with the boot addresses we can put the new node's persistent addresses to the interfaces:

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Topology Configuration

       -> Configure HACMP Persistent Node IP Labels/Addresses

         -> Add a Persistent Node IP Label

                                                        [Entry Fields]

         * Node Name                                            hoppy

         * Network Name                                       [(selected network)]            +

         * Node IP Label/Address                              [<persistent IP address>]       +

           Netmask(IPv4)/Prefix Length(IPv6)                  []

If our persistent address is in a different network than the boot address and there are routes through the interface (what is the classical setup) they have to be defined in the ODM on hoppy. For a default route to a gateway 111.111.111.1 the command would be

hoppy# chdev -l inet0 -a route=net,-hopcount,0,,0,111.111.111.1 -P

Again we need to synchronize the cluster

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Verification and Synchronization

STEP 6: Start HACMP on New Node

Finally we are ready to start the cluster on the new node.

hoppy# smitty clstart

hoppy# mkdev -l inet0¹

---

¹ this command activates the route set with STEP 5

STEP 7: Remove Dependencies between Resource Groups

Any online on same node dependencies have to be removed before we are able to add the new node to resource groups:

barney# smitty hacmp

   -> Extended Resource Configuration

     -> Configure Resource Group Run-Time Policies

       -> Configure Dependencies between Resource Groups

         -> Configure Online on the Same Node Dependency

           -> Remove Online on the Same Node Dependency Between Resource Groups

STEP 8: Add New Node to All Resource Groups

First we need to copy the application start/stop scripts to the new node (hoppy):

hoppy# scp barney:/path/to/application_servers/* /path/to/application_servers

Then we are ready to add new node to the HACMP configuration:

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Resource Configuration

       -> HACMP Extended Resource Group Configuration

         -> Change/Show a Resource Group

Select resource groups one by one and add the new node to the end of the list of Participating Nodes:

                                                                [Entry Fields]

        Resource Group Name                                 (Resource Group)

        New Resource Group Name                            []

        Participating Nodes (Default Node Priority)        [barney betty hoppy]                                     

Note: Really add the new node to the end of the list. Don't use the F4 selection, because the order might be wrong! To be sure we verify the node order in the resource group with

barney# clshowres | egrep 'Resource Group Name|Participating Node Name'

Resource Group Name                                 RES_GRP_01

Participating Node Name(s)                          barney betty hoppy

Resource Group Name                                 RES_GRP_02

Participating Node Name(s)                          barney betty hoppy 

Finally we synchronize the cluster:

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Verification and Synchronization

STEP 9: Restore Dependencies between Resource Groups

If you had to remove any dependencies between resource groups in STEP 7 now it's time to add the dependencies again:

barney# smitty hacmp

   -> Extended Resource Configuration

     -> Configure Resource Group Run-Time Policies

       -> Configure Dependencies between Resource Groups

         -> Configure Online on the Same Node Dependency

           -> Add Online on the Same Node Dependency Between Resource Groups

Use to get a list of known resource groups.

STEP 10: Final Synchronization of the Cluster

Now synchronize the cluster...

barney# smitty hacmp

   -> Extended Configuration

     -> Extended Verification and Synchronization

and clRGinfo shows three nodes:

barney# clRGinfo

-----------------------------------------------------------------------------

Group Name     Group State                  Node           

-----------------------------------------------------------------------------

RES_GRP_01     ONLINE                       barney        

               OFFLINE                      betty        

               OFFLINE                      hoppy        

RES_GRP_02     ONLINE                       barney        

               OFFLINE                      betty        

               OFFLINE                      hoppy