RHEL Kernel image to large

Interesting issue today in trying to install Redhat 6 on a POWER System, so that if you see this error -


Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-35.el6)
Enter "help" to get some basic usage information
boot: linux
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
Claim failed for initrd memory at 02000000 rc=ffffffff


Now after some searching on Bugzilla I found what looked like my problem, from this I have done a little testing and resolved the issue as detailed below:


Boot the POWER server and enter the OpenFirmware prompt with selection 8 at the IBM menu on start, then run the following command at the OK prompt -


0 > printenv real-base

If you see the following the I
-------------- Partition: common -------- Signature: 0x70 ---------------
real-base                2000000             2000000


Now if I have understood this correctly the firmware is expecting a image of 32MB and in our case for RHEL6 it is 16MB (or smaller) so we need to set it correctly, as below -


0 > setenv real-base 1000000
0 > printenv real-base
-------------- Partition: common -------- Signature: 0x70 ---------------
real-base                1000000              2000000


Then reboot with a -

0 > reset-all


Then boot of the RHEL image as normal and you should find all install from this point will work OK, in this case it was a virtual server so until the LPAR is removed from the HMC.


Ref -

• 2000000 is 32MB
• 1800000 is 24MB
• 1000000 is 16MB
• c00000 is 12MB

PowerHA Failure Rate

Changing Module failure rate -

Within PowerHA you can increase the rate in while HA detects the failure of the various hearbeat network, there are 3 predefined values of slow, normal and fast.

# smitty cm_config_networks
    - then select 'Change aNework Module using Predefined Values' and select the module you wish to change, in this case 'ether'

                               [Entry Fields]
* Network Module Name          ether
  Description                  Ethernet Protocol
  Failure Detection Rate       Normal                     +

  NOTE: Changes made to this panel must be
        propagated to the other nodes by
        Verifying and Synchronizing the cluster

As you can see once the setting has been changed it is important to sync and verify the cluster 'smitty cm_ver_and_sync.select'.
If you wish to tune this further then you can select the 'Custome Values' option -

                                     [Entry Fields]
* Network Module Name                   ether
  Description                           [Ethernet Protocol]
  Address Type                          Address                       +
  Path                                  [/usr/sbin/rsct/bin/hats_nim]  /
  Parameters                            []
  Grace Period                          [60]                           #
  Supports gratuitous arp               [true]                         +
  Entry type                            [adapter_type]
  Next generic type                     [transport]
  Next generic name                     [Generic_UDP]
  Supports source routing               [true]                         +
  Failure Cycle                         [10]                           #
  Interval between Heartbeats (seconds) [1.00]

 
  Heartbeat rate is the rate at which cluster servic
  es sends 'keep alive' messages between adapters in
  the cluster. The combination of heartbeat rate and
  failure cycle determines how quickly a failure can
  be detected and may be calculated using this
  formula:
  (heartbeat rate) * (failure cycle) * 2 seconds

  NOTE: Changes made to this panel must be
        propagated to the other nodes by
        Verifying and Synchronizing the cluster

So the default value above is 10 * 1.00 * 2 = 20.00 seconds before a failure of the network is declaired.

Then once you have this set, using the 'Show a Network Module' will give you a sumary current values.

PowerHA (HACMP) Notes

Useful PowerHA (HACMP) Notes -

Here are a few little notes that I have picked up over the years that relate to some of the functions of PowerHA and HACMP.  They have proved useful to me in some situations and hopefully will to you too.


HA ODMs

The HA ODMs are the object databases that HA uses to ensure that the cluster is all in sysnc, if any of them differ in some why in some cases 'lazy update' with tidy up the records syncing the nodes.  But in some cases this will cause a command/action to fail so you will need to go any sync the cluster.


DCD  - The default HA ODM - HACMPtopsvcs. Look for 'instanceNum' – should be the same on all cluster nodes else it shows things have not syncd correctly!


ACD - Active – this is stored in /usr/es/sbin/cluster/etc/objrepos/active to read this ODM change you ODM path to this file, remember to change it back!


SCD - Staging


So to read the DCD ODM -


# odmget HACMPtopsvcs


HACMPtopsvcs:
        hbInterval = 1
        fibrillateCount = 4
        runFixedPri = 1
        fixedPriLevel = 38
        tsLogLength = 5000
        gsLogLength = 5000
        instanceNum = 4


The topsvcs daemon will also show the current instance number on the node.


# lssrc -ls topsvcs|grep Instance
  Configuration Instance = 4


# odmget HACMPnode
    - the line ‘version’ shows the current HA version
    Version 11 = 6.1
    Version 10 = 5.5
    Version 9 = 5.4
    Version 8 = 5.3
    Version 7 = 5.2


Resource Group Dependencies

If you are having trouble with RG dependencise then the following command will list resource group start inter dependencies
"clrgdependency -t'PARENT_CHILD' -sp"


# clrgdepdency -t [PARENT_CHILD | NODECOLLOCATION | ANTICOLLOCATION |SITECOLLOCATION ] -sl
# clrgdepdency -t PARENT_CHILD -sl
Parent Child
rg1 rg2
rg1 rg3
# clrgdepdency -t NODECOLLOCATION -sl
# clrgdepdency -t ANTILOCATION -sl
HIGH:INTERMEDIATE:LOW
rg01::rg03frigg
# clrgdepdency -t SITECOLLOCATION -sl
rg01 rg03 frigg

Heat-Beating

# lssrc -ls topsvcs
- Shows the status of the system heart-beating. 

PowerHA

PowerHA Problem Determination

Here a few notes that I made of some of the more usefull day to day commands to help look at problems in your PowerHA (HACMP) clusters. If you are running older version of HACMP such as 5.4 and below some of the paths may differ. Most of this should be correct for 5.5, 6.1 and 7.1, though I need to do some revision and testing for 7.1.

Cluster Status

Its normally a good idea to get a status of the cluster and try to work out how serious the problem is. Depending on the cluster/nodes status some of the commands will need to be run on a active node, so in the case of a fail-over the standby node -

/usr/es/sbin/cluster/utilities/clfindres - shows current status of resource groups.

/usr/es/sbin/cluster/clstat - shows cluster status and substate in real time, needs clinfo to be running.

Now these 2 commands are good to show you the current status of the cluster, but you can also get more information with -

lssrc -ls clstrmgrES - shows the cluster manager state

lssrc -ls topsvcs - shows heartbeat information

If you want even more details about the cluster then the following commands should be able to help you further -

/usr/es/sbin/cluster/utilities/cltopinfo - show current topology status and some information about the cluster.

/usr/es/sbin/cluster/utilities/clshowres - shows short resource group information.

/usr/es/sbin/cluster/utilities/cldisp – shows cluster information such as monitor and rery intervals.

/usr/es/sbin/cluster/utilities/cllscf - list the network configuration of a HACMP cluster.

/usr/es/sbin/cluster/utilities/cllsif - show network interface information.

/usr/es/sbin/cluster/utilities/clRGinfo -p -t - show current RG state.

/usr/es/sbin/cluster/utilities/clRGinfo –m - shows RG monitor status

Cluster Logs -

Once you have a idea of the status of the cluster you can start looking at the log files to try to determin the problem, now a good place to start is first the cluster.log

/usr/es/adm/cluster.log - I tend to filter this as follows so that I can get a idea of the events that have occured, but also look out for 'config_too_long'.

cat /usr/es/adm/cluster.log |grep ' EVENT ' |more - Should look something like this -

Jan 13 18:51:29 <node1> HACMP for AIX: EVENT COMPLETED: node_up <node1> 0
Jan 13 18:51:30 <node1> HACMP for AIX: EVENT START: node_up_complete <node1>
Jan 13 18:51:30 <node1> HACMP for AIX: EVENT START: start_server cluster_AS
Jan 13 18:51:30 <node1> HACMP for AIX: EVENT COMPLETED: start_server cluster_AS 0
Jan 13 18:51:30 <node1> HACMP for AIX: EVENT COMPLETED: node_up_complete <node1> 0
Jan 13 18:59:16 <node1> HACMP for AIX: EVENT START: node_down <node1>
Jan 13 18:59:16 <node1> HACMP for AIX: EVENT START: stop_server cluster_AS
Jan 13 19:00:34 <node1> HACMP for AIX: EVENT COMPLETED: stop_server cluster_AS 0
Jan 13 19:01:04 <node1> HACMP for AIX: EVENT START: release_service_addr

From these log entries you can see what events have completed and failed, generally you will look out for 'config_too_log', EVENT FAIL or events that have not completed. As all events that have a start should also follow in the log at some point with a completed, else this indicates a problem too.

Once you have picked any problem you might have you can use these as a key to look further into this log, or the hacmp.log as this lists the same events but in much more detail, this should give you some more clues as to where to look from there. Depending on what you find the problem may well be RSCT related or HA core.

Now if the problem relates to the HA core then you can look further into logs such as the following -

/var/hacmp/log/clstrmgr.debug - clstrmgrES activity, even more detail then hacmp.out, good place to look if events are missing from other logs.

/var/hacmp/log/cpsoc.log - the cspoc log for the node that the command was issued on. It contains time-stamped, formatted messages generated by HACMP C-SPOC commands.

/var/hacmp/clcomd/clcomd.log - cluster communication daemon log.

/var/hacmp/clverify/clverify.log - cluster verification log, good place to look to make sure the cluster was working before the problem.

/var/hacmp/adm/history/cluster.mmddyyyy - It contains time-stamped, formatted messages generated by HACMP scripts.

RSCT (Reliable Scalable Cluster Technology)

These are stored in /var/ha/log, and the RSCT relates to to site to site comunication and heart-beating, this means that the RSCT logs are a good place to look if you are having problem with the fail-over adapters or the HB'ing disks. The logs are broken up into a number of related areas

grpsvcs.* - Group Services log; hacmp cluster comunication log.

nim.topsvcs.<interface>* - Network Interface Module Topology Services log; deals with specific interface comunication and login, there will be one of these of each cluster interface on each node, plus the disk/serial HB'ing devices.

nmDiag.nim.topsvcs.<interface>* - Network Module Diagnostic message; this relates only to the network devices.

topsvcs.* - Topology Services log; summary and some further detail of all the network topology that is occuring with in the cluster, a good place to look to get a status of the adapters and cluster status.

topsvcs.default - Daily Topology Services log which is run at the begining of the day to confirm the topsvcs status.

First Post

Well this is the first post of this new blog, so lets cover what I'm hoping to do here.  I've been working with IBM's version of UNIX called AIX since 1998 though not just at IBM, I have had some other jobs at other companies and even worked with other flavours.  Such as HP-UX, Solaris and the now dead Dynix/Ptx, along with these I have also used various flavours of Linux, though mainly Ubuntu, Redhat and SLES.

Over the years I have collected a lot of knowledge and information, some of it like Dynix now dead and gone, but alot of it still very useful and more importantly as things change I to am learning new things all the time.

Hopefully I can pass some of that information on and create a useful resource not just for myself but for others too.