Tanti Technology

My photo
Bangalore, karnataka, India
Multi-platform UNIX systems consultant and administrator in mutualized and virtualized environments I have 4.5+ years experience in AIX system Administration field. This site will be helpful for system administrator in their day to day activities.Your comments on posts are welcome.This blog is all about IBM AIX Unix flavour. This blog will be used by System admins who will be using AIX in their work life. It can also be used for those newbies who want to get certifications in AIX Administration. This blog will be updated frequently to help the system admins and other new learners. DISCLAIMER: Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility. If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.

Tuesday, 22 October 2013

How to Add a New Resource Group to an Active Cluster

How to Add a New Resource Group to an Active Cluster


Contents

  1. Introduction
  2. Create new Volume Groups
  3. Add Resources: New Service IP Adresses
  4. Add Resources: New Application Start/Stop Scripts
  5. Create a New Resource Group
  6. Add the Resources to the New Resource Group
  7. Final Synchronization of the Cluster
  1. Related Information

1. Introduction

This article describes how to add a new resource group »RES_GRP_03« to an HACMP cluster. Primary node for the new resource group shall be »betty«. We find the following situation:
betty# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node           
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       barney        
               OFFLINE                      betty        
 
RES_GRP_02     ONLINE                       barney        
               OFFLINE                      betty       
Conclusion: There are two resource groups - both online on barney.

2. Create new Volume Groups

First we need to scan for new disks:
betty# cfgmgr
and set a PVID to all new disks:
betty# chdev -l hdiskA1 -a pv=yes
betty# chdev -l hdiskA2 -a pv=yes
betty# chdev -l hdiskB1 -a pv=yes
                      :
                      :
and change the reservation policy for use with HACMP:
betty# chdev -l hdiskA1 -a reserve_policy=no_reserve
betty# chdev -l hdiskA2 -a reserve_policy=no_reserve
betty# chdev -l hdiskB1 -a reserve_policy=no_reserve
                      :
                      :
Now we are ready to run the configuration manager on the other node and set the reservation policy of the disks:
barney# cfgmgr
barney# chdev -l hdiskA1 -a reserve_policy=no_reserve
barney# chdev -l hdiskA2 -a reserve_policy=no_reserve
barney# chdev -l hdiskB1 -a reserve_policy=no_reserve
                      :
                      :
Since the new resource group will be primarily online on betty, we create the volume group and the filesystems on betty. Anyway, there is no technical reason for doing this on the primary node. We could do this on the other node as well. The volume groups will be created as follows:
betty# mkvg -C -V 130 -S -s 32 -y RG03data hdiskA1 hdiskA2 hdiskB1 hdiskB2
betty# varyonvg RG03data
betty# chvg -Qn RG03data
Please note: The volume groups we create are enhanced-concurrent-capable. By default these volume groups have the following properties: Quorum is on, auto-varyon is off. Further note the volume group's major number we set with the »-V« option. Get free major numbers on both nodes with the »lvlstmajor« command.
With the volume groups created and vary'd on we can go on creating logical volumes. Usually we would use logical volumes mirrored over two datacenters. So we ensure that all first LV copies (PV1 in »lslv -m« output) reside in Datacenter A and all second LV copies (PV2 in »lslv -m« output) reside in Datacenter B. We use superstrictness and the most narrow upper bound possible to reduce the risk of a mirroring mess, e.g.
betty# mklv -t jfs2 -e x -a e -u 2 -s s -c 2 -y lvrg03fs01 RG03data 256 hdiskA1 hdiskB1 hdiskA2 hdiskB2
where hdiskA1 and hdiskA2 reside in datacenter A and hdiskB1 and hdiskB2 reside in datacenter B.
Filesystems on top of our logical volumes should be created with the mount = false option, e.g.
betty# crfs -v jfs2 -m /filesystem/rg3/1 -d /dev/lvrg03fs01 -A no -p rw -a agblksize=512 -a logname=INLINE
betty# mount /filesystem/rg3/1
betty# crfs -v jfs2 -m /filesystem/rg3/2 -d /dev/lvrg03fs02 -A no -p rw -a agblksize=512 -a logname=INLINE 
betty# mount /filesystem/rg3/1
Once all filesystems are created we unmount everything we just created and close the volume group:
betty# umount /filesystem/rg3/1
betty# umount /filesystem/rg3/2
betty# varyoffvg RG03data
Now we are ready to import all Volume Groups we created into the other node.
barney# importvg -n -V  -y
Get information about major number, Volume Group, and PVID from the other node.

3. Add Resources: New Application Start/Stop Scripts

For the new application server we first need application start and stop scripts. We assume that we've got these scripts and copied them to /etc/hacmp on betty. Then we can add a new application server:
betty# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resources Configuration
         -> Configure HACMP Applications
            -> Configure HACMP Application Servers
               -> Add an Application Server
 
                             Add Application Server
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                                        [Entry Fields]
* Server Name                                        [app_srv03]
* Start Script                                       [/etc/hacmp/start_srv03]
* Stop Script                                        [/etc/hacmp/stop_srv03]
  Application Monitor Name(s)                                                         +
Since the scripts are local we have to copy them over to barney:
betty# scp -p /etc/hacmp/st*_srv03 barney:/etc/hacmp/

4. Add Resources: New Service IP Adresses

For the new Resource Group one or more service ip addresses are needed. First we have to add the new service ip addresses to /etc/hosts on both nodes:
# vi /etc/hosts
111.111.111.130 haservice3 haservice3.domain.com
111.111.111.131 haservice31 haservice31.domain.com
Then we let HACMP discover information:
betty# smitty hacmp
-> Extended Configuration
   -> Discover HACMP-related Information from Configured Nodes
Then we have to define the new IP labels as a HACMP resource.
betty# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resources Configuration
         -> Configure HACMP Service IP Labels/Addresses
            -> Add a Service IP Label/Address
 
  +--------------------------------------------------------------------------+
  |                  Select a Service IP Label/Address type                  |
  |                                                                          |
  | Move cursor to desired item and press Enter.                             |
  |                                                                          |
  |   Configurable on Multiple Nodes                                         |
  |   Bound to a Single Node                                                 |
  |                                                                          |
  | F1=Help                 F2=Refresh              F3=Cancel                |
  | F8=Image                F10=Exit                Enter=Do                 |
  | /=Find                  n=Find Next                                      |
  +--------------------------------------------------------------------------+
The service address needs to move with the application - so we select "Configurable on Multiple Nodes" here.
    Add a Service IP Label/Address configurable on Multiple Nodes (extended)
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
                                                        [Entry Fields]
* IP Label/Address                                    haservice3               +
  Netmask(IPv4)/Prefix Length(IPv6)                  []
* Network Name                                        net_ether_01
  Alternate HW Address to accompany IP Label/Address []

5. Create a New Resource Group

betty# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resource Group Configuration
         -> Add a Resource Group
 
                        Add a Resource Group (extended)
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
                                                        [Entry Fields]
* Resource Group Name                                [RES_GRP_03]
* Participating Nodes (Default Node Priority)        [betty barney]                              +
 
  Startup Policy                                      Online On Home Node Only                   +
  Fallover Policy                                     Fallover To Next Priority Node In The List +
  Fallback Policy                                     Never Fallback                             +
If you pick the partitipating nodes with from the list, you might end up with th wrong order. The home node must be the first in the line!
Don't be confused by the clRGinfo output:
betty# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node           
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       barney        
               OFFLINE                      betty        
 
 
RES_GRP_02     ONLINE                       barney        
               OFFLINE                      betty        
clRGinfo: Operation failed, error = 0.
clRGinfo will show the new resource group once we synchronized the cluster.

6. Add the Resources to the New Resource Group

Before we start to add resources to our new resource group it might be a good idea to synchronize the cluster first.
The last step is to add resources to our resource groups. If you didn't run the HACMP discovering tool yet, you have to run it now. Otherwise HACMP can't find our newly created volume groups. If you exactly did the steps in the order of this article then the discovering tool has already be run.
This said, we add the resources:
betty# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resource Group Configuration 
         -> Change/Show Resources and Attributes for a Resource Group
 
  +--------------------------------------------------------------------------+
  |        Change/Show Resources and Attributes for a Resource Group         |
  |                                                                          |
  | Move cursor to desired item and press Enter.                             |
  |                                                                          |
  |   RES_GRP_01                                                             |
  |   RES_GRP_02                                                             |
  |   RES_GRP_03                                                             |
  |                                                                          |
  | F1=Help                 F2=Refresh              F3=Cancel                |
  | F8=Image                F10=Exit                Enter=Do                 |
  | /=Find                  n=Find Next                                      |
  +--------------------------------------------------------------------------+
 
      Change/Show All Resources and Attributes for a Custom Resource Group
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
[TOP]                                                   [Entry Fields]
  Resource Group Name                                [RES_GRP_03]
 
  Participating Nodes (Default Node Priority)        [betty barney]                              +
 
  Startup Policy                                      Online On Home Node Only                   +
  Fallover Policy                                     Fallover To Next Priority Node In The List +
  Fallback Policy                                     Never Fallback                             +
 
  Service IP Labels/Addresses                        [haservice3 hasevice31]                     +
  Application Servers                                [app_srv03]                                 +
 
  Volume Groups                                      [RG03data]                                  +
  Use forced varyon of volume groups, if necessary    true                                       +
  Automatically Import Volume Groups                  false                                      +
 
  Filesystems (empty is ALL for VGs specified)       []                                          +
  Filesystems Consistency Check                       logredo                                    +
  Filesystems Recovery Method                         sequential                                 +
  Filesystems mounted before IP configured            false                                      +
  Filesystems/Directories to Export                  []                                          +
                                                                                                 +
  Filesystems/Directories to NFS Mount               []
  Network For NFS Mount                              []                                          +
[MORE...10]

7. Final Synchronization of the Cluster

Now synchronize the cluster...
betty# smitty hacmp
   -> Extended Configuration
     -> Extended Verification and Synchronization
and clRGinfo shows three resource groups:
betty# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node           
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       barney        
               OFFLINE                      betty        
 
RES_GRP_02     ONLINE                       barney        
               OFFLINE                      betty        
 
RES_GRP_03     ONLINE                       betty      
               OFFLINE                      barney     
You will also notice that the service IP addresses have been acquired and that all our filesystems have been mounted on the primary node.


Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)