Wednesday, 17 April 2013

All About HACMP/PowerHA


PowerHA implementation steps

PowerHA Limits

ComponentMaximum number/other limits
Resource groups64
Network interfaces, devices, and labels256
Cluster resourcesWhile 128 is the maximum that clinfo can handle, there can be more in the cluster
Parent-Child dependenciesMax of 3 levels
Interfaces7 interfaces per node per network
Application monitors per site128
Persistent IP aliasOne per node per network
XD_data networks4 per cluster
GLVM ModesSynchronous, Asynchronous, non concurrent
GLVM DevicesAll PVs supported by AIX, no need to be same local and remote

 Log Files

/var/hacmp/adm/cluster.logGenerated by cluster scripts and daemons
/var/hacmp/log/hacmp.outGenerated by event scripts and utilities
/var/hacmp/adm/history/cluster.mmddyyyyCluster history files generated daily
/var/hacmp/clcomd/clcomd.logGenerated by clcomd daemon
/var/hacmp/clcomd/clcomddiag.logGenerated by clcomd daemon, debug information
/var/hacmp/clverify/clverify.logGenerated by Cluster Verification utility
/var/hacmp/log/autoverify.logGenerated by Auto Verify and Synchronize
/var/hacmp/log/clavan.logGenerated by Application Availability Analysis tool
/var/hacmp/log/clinfo.logGenerated by client node running clinfo
/var/hacmp/log/cl_testtool.logGenerated by the Cluster Test Tool
/var/hacmp/log/clconfigassist.logGenerated by Two-Node Cluster Configuration Assistant
/var/hacmp/log/clstrmgr.debugGenerated by the clstrmgr daemon
/var/hacmp/log/clstrmgr.debug.longDetail information from the clstrmgr daemon
/var/hacmp/log/clutils.logGenerated by cluster utilities and file propagation
/var/hacmp/log/cspoc.logGenerated by CSPOC commands
/var/hacmp/log/cspoc.log.remoteDetail information from CSPOC commands
/var/hacmp/log/migration.logGenerated by cluster migration
/var/hacmp/log/sa.logGenerated by Application Discovery
"odmget HACMPlogs"Displays a complete list of HACMP Log Files
/var/ha/log/topsvcs.defaultTopology Services starup configuration log.
/var/ha/log/topsvcs.dd.hhmmss.langTopology Services start/stop log
/var/ha/log/topsvcs.dd.hhmmssTopology Services activity log
/var/ha/log/nim.topsvcs.IF.clusternameNIM hearthbeat activity log for each interface.
/var/ha/log/nmDiag.topsvcs.IF.clusterNIM diagnostic log for each interface.
/var/ha/log/grpsvcs.default.nodenum_instnumGroup Services startup log
/var/ha/log/grpsvcs_nodenum_instnumGroup Services activity log. 

Useful HACMP Commands

clstatDisplays topology and resource groups status.
clinfoES and snmpd must be running.
cldumpDisplays topology & resource group status and configuration.
snmpd must be running.
cldispLike cldump, but application oriented.
snmpd must be running.
cltopinfo (cllsif)Displays topology configuration.
clRGinfo (clfindres)Displays resource group status.
clsshowresDisplays resource groups configuration.
clshowsrvCalls lsrsc to display status of:
HACMP subsystems (clshowrsrv -a)
HACMP and RSCT subsystems (clshowsrv -v)
clcycleRotates selected log files.
clgetactivenodesDisplays active nodes.
Must specify which nodo to ask (-n node)
clsnapSave HACMP log files and configuration information.
cl_ls_shared_vgsList shared vgs.
cl_lsfsList shared fs.
cllsgrpList the resource groups.
cllsresShow short resource group information.
clRGmoveBrings a RG Offline/Online or move it
lssrc -ls clstrmgrESDisplays Cluster Services


Installation changes 

The following AIX configuration changes are made:

1. Files modified:


2. The hacmp group is added.  

3. Also, using cluster configuration and verification, the file /etc/hosts can be changed by adding or modifying entries. 

4. The following network options are set to “1” by RSCT topsvcs startup:

- nonlocsrcroute
- ipsrcrouterecv
- ipsrcroutesend
- ipsrcrouteforward
- ip6forwarding

5. The verification utility ensures that the value of each network option is consistent across all cluster nodes for the following settings:

- tcp_pmtu_discover
- udp_pmtu_discover
- ipignoreredirects
- routerevalidate

The cluster communications daemon

With the introduction of clcomdES, there is no need for an /.rhosts file to be configured. The cluster communications daemon is started by inittab, with the entry being created by the installation of PowerHA. The daemon is controlled by the system resource controller, so startsrc, stopsrc and refresh work. In particular, refresh is used to re-read /usr/es/sbin/cluster/etc/rhosts and moving the log files. The cluster communication daemon uses port 6191.

Resource group components:

Service IP Label
Volume Group
Application Server
NFS mounts
NFS exports

Resource group Startup options:

Online on home node only.
Online on first available node.
Online on all available nodes.
Online using distribution policy.

Resource group Fallover options:

Fall over to next priority node in list:
Fallover using dynamic node priority:
Bring offline (on error only)

Resource group Fallback options:

Fall back to higher priority node in list
Never fall back

Resource group attributes

Settling time
Delayed fallback timers
Distribution policy
Dynamic node priorities
Resource group processing order
Priority override location
Resource group dependencies - parent / child
Resource group dependencies - location

Resource Groups operations 

Bring a resource group offline
Bring a resource group online
Move a resource group to another node/site
Suspend/resume application monitoring

Sources of HACMP information

HACMP manuals come with the product –READ THEM!
Sales Manual:

IBM courses:

– HACMP Administration I: Planning and Implementation (AU54/Q1554)
– HACMP Administration II: Administration and Problem Determination (AU61/Q1561)
– HACMP V5 Internals (AU60/Q1560)

IBM Web Site:


Non-IBM sources (not endorsed by IBM but probably worth a look):


0 blogger-disqus:

Post a Comment