Performance AIX and POWER Notes

Recommend Performance Notes

Below are some details of performance tuning that I have come across in relation to different situations, while while might be true in one case may not work for another.  So please be sure to perform testing on your environment before making these recomendations.

Hardware memory prefetch -
Hardware memory prefetch helps to improve the performance of applications that reference memory sequentially by prefetching memory.

Hardware memory prefetch consdierations - There might be adverse performance effects, depending on the workload characteristics.Determine whether to disable hardware memory prefetch in a production environment after sufficient testing with the application.

In this case it was a JAVA application, which is recomended that you turn off the hardware memory prefetch.  The example below shows how to disable across a reboot using the -b flag, the command creates an entry in the /etc/inittab file -

# dscrctl -n -b -s 1

 
The value of the DSCR OS default will be modified on subsequent boots
 

# dscrctl -q
Current DSCR settings:
        Data Streams Version = V2.06
        number_of_streams = 16
        platform_default_pd = 0x5 (DPFD_DEEP)
        os_default_pd = 0x1 (DPFD_NONE)

# cat /etc/inittab|grep dscrset
dscrset:2:once:/usr/sbin/dscrctl -n -s 1 >/dev/null 2>/dev/console

Memory and Page Size Considerations
IBM Java can take advantage of medium (64K) and large (16M) page sizes that are supported by the current AIX versions and POWER processors. Using medium or large pages instead of the default 4K page size can significantly improve application performance. The performance improvement of using medium or large pages is a result of a more efficient use of the hardware translation caches which are used when translating application page addresses to physical page addresses. Applications that are frequently accessing a fast amount of memory benefit most from using pages sizes that are larger than 4K.

# vmo -o lgpg_regions=16384 -o lgpg_size=16777216 -o v_pinshm=1
- create 16MB large page objects and pin memory, when done should use -Xlp on Java cmd line

VIOS Update

Following command are recommend for updating your VIOS running levels 2.1.0 and above. As a reboot is required on the system, make sure that any active environments are shutdown too (unless Dual VIO);
 
First commit older updates with the following command - 

 # updateios -commit

Then update the server from the media or device so as below its /mnt but it could be /dev/cd# -

# updateios -accept -install -dev /mnt

To apply the changes to the VIOS you will need to reboot (Any active LPARs will need to be shutdown too) -
# shutdown -force -restart
 
Once rebooted, you will need to re-accept the license-  
# license -accept
 
Confirm the update with -
# ioslevel

More on Etherchannel

If you wish to add or remove additional devices to your live Etherchannel adapter then the following command will help -

Add adapter ent4 into the EtherChannel device called ent5 - 
/usr/lib/methods/ethchan_config -a ent5 ent0

To remove ent4 from the EtherChannel device ent5 - 

/usr/lib/methods/ethchan_config -f ent7

Etherchannel with SEA

Configuring your SEA with Etherchannel.

 First check the shared virtual ethernet adapters you have -

# lsmap -all -net
SVEA   Physloc
------ --------------------------------------------
ent3   U9117.MMB.101697P-V1-C2-T1

SEA                 NO SHARED ETHERNET ADAPTER FOUND


Then we need to check what adapters we can use, now in our case we will be using the first 2 adapters from a dual port 1Gb card -  

# lsdev -type adapter
name             status      description
ent0             Available   2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
ent1             Available   2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
  
Then we need to create the Etherchannel device -   

# mkvdev -lnagg ent0 ent1
ent2 Available

This creates the device in standard mode, though you can switch it over to round robin -

# chdev -l ent2 -a mode=round_robin  

Then we can create a SEA "bridge" between the physical Etherchannel device ent2 and the virtual ent3

 # mkvdev -sea ent2 -vadapter ent3 -default ent3 -defaultid  1
            main:  86 Recived SEA events bytes 163
ent9 Available
 
 Once that is done, you can set-up initial TCP/IP config (en9 is the interface for the SEA ent9)

# mktcpip -hostname <vio-name> -inetaddr <ip-address> -interface en9 -start -netmask <subnet> -gateway <gateway-ip>
  
Now your server ready to go.

SmartCloud Entry - Projects, Users and Accounts

Now we have everything the way we want, we can think about working with our projects, users and accounts.   From this we can manage our usage of the SmartCloud and the workloads we are going to be deploying, creating, managing and re-sizing.
First begin by creating the users that you want, all users are created equal until you put them in the relavant accounts and projects.

Users
Select Users -> Select New User ->  

• Fill in the details as you require.   

 If you select 'admin privileges' then they have visibility of advanced features like networking and users creation.

Projects
Once you have your users you need to create a project for them to belong to -
Select Projects -> Select New Project -> 

• Give a name and description. 

Once its created you can select the users to add to the project and there 'Role' in that project.
Accounts
Now we just need to create a account for this project and users so that we can control the billing further.
Select Accounts -> Select New Account -> 

• Give a name and description
• Select the Account owner.
• Give a starting balance and low balance alert threshold.
•  Then add further account member's, this is generated from the project members added earlier.  

Now this is set-up any new workloads to that project will get billed to the relvant selected accounts and the cost taken from their funding. If the users make changes, the project owner will be able to see, amaned and approve/deny these and the account get charges relating to those.  If notifications have been set and e-mail set-up then e-mails will be sent about events on the SmartCloud relevant to those users.
 

SmartCloud Entry - Further Configuration

Billing and Metering

Now that we have the our SmartCloud up, we need to get familiar with how it works and what we can do with it.  Now most of us will be interested in setting up request approval, billing, and user/project control. But before we do those things we need to enable them and check the values are what we require.  Now in the previous blog entry we mentioned how to turn these on -


To turn on billing, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/billing.properties -> com.ibm.cfs.billing.enabled=true

To turn on metering, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/metering.properties -> com.ibm.cfs.metering.enabled=true

Once this is enabled we can also look at changing and update the cpu, memory, disk value.

This is the directory tht is used by IBM SmartCloud Entry as a product catalog -

.skc/products

Three files that effecgt billing charges rates - 

.skc/products/ram.xml
.skc/products/cpu.xml
.skc/products/disk.xml

 (taken from the readme) -

The default collector will look at a specific virtual machine property (as defined by the Cloud) and use the value of the property as the units of product to buy, at the products price. For example, IBM VMControl 2.3.x exposes a VM property called "Processor.VirtualQuantity", this property can be used to know the amount of CPUs allocated to a VM, and hence used to charge the VM for a "CPU Cloud product."

For example take a look at the following product:

<cloudProduct id="com.ibm.cfs.services.billing.products.cpu">
<name>CPU</name>
<description>The amount of CPUs used in a virtual machine at a rate of $1 per hour.</description>
<pricing currency="USD" interval="60" price="0.0167"/>
<collector property="Processor.VirtualQuantity"/>
</cloudProduct>

The product above specifies that the default SKC collector will collect charges on virtual machines using the amount of CPUs assigned to them, exposed via the "Processor.VirtualQuantity" property, at a rate of 0.0167 cents every 60 seconds or roughly $1 per hour. The reason why it is important to collect at an interval less than the actual described rate (i.e., seconds instead of hours), is so that users looking at their bills get a higher resolution approximation of their actual charges, having an accurate look at the charges might be important for accounting purposes, or in situations where account credit is limited.

E-mail

SmartCloud Entry can send e-mail notification for several user and admin related events, such as the completion of a workload, account creation, and more.  In order to get these notifications it is necessary to configure a properties file in the home directory.

To setup notification for SmartCloud Entry follow the steps described below:

1. Open the email.properties file in the SCE home directory.
2. Set the com.ibm.cfs.email.relay.host property to the hostname of the relay host that IBM Starter Kit for Cloud should use for outgoing SMTP emails.
3. Optionally, you can set the e-mail subject prefix for all SmartCloud Entry e-mails and the “from” name, by setting the following properties respectively:
• com.ibm.cfs.email.subject.prefix
• com.ibm.cfs.email.from.name
• com.ibm.cfs.email.from.address
4. Save the email.properties file and restart the SmartCloud Entry server.

You can globally disable e-mail notifications in SmartCloud Entry completely by setting the com.ibm.cfs.email.default.notifications property in the email.properties file to false.

Configuring Advanced Features

Virtual Appliances   

The administrator can pre-configure the appliances to simplify its deployment for the users. Configuration the properties can be made through the SmartCloud Entry web user interface or through the deployment.properties file in the SCE home directory.

In order to simplify the deployment process for the users, it is recommended that an administrator configures all of the virtual appliances prior to making SmartCloud Entry available to its users. Virtual appliance customisation often requires knowledge of the low level details of the environment or having advanced knowledge of the data centre of which the user may not be familiar.  

Configuring a staging project

By default, SmartCloud Entry will scan the cloud for new appliances periodically.  When SCE finds a new appliance it places it in the Public project where anyone can deploy it.  SCE also allows configuring of a 'Staging' project where newly discovered appliances will show up rather than having them put directly into the Public project.  This allows the adminstrator to configure the the appliances and to then move them to a different project (Public or a specific project) to give the correct visibility to the users.  

So to configure this staging project add or uncomment this line in the deployment.properties file: 

com.ibm.cfs.staging.project=Staging   

Deployment Visibility

You can also choose whether you want your users to see the advanced deployment options or just the basic options.
The advanced options gives the user a finer-grained level of control over the level of resources consumed by the appliance. You may or may not want to show this advanced panel to the end-user, so to configure the visibility of the advanced deployment options, please follow these steps:  

• Open the deployment.properties file in the SCE home directory.
• Set the deployment.advanced.form.enabled property to false. This will disable the advanced deployment option and it will not be shown to the end user. However, administrators will continue to see the advanced deployment option in the user interface. The default value of this property is true.
• Save the deployment.properties file and restart the SmartCloud Entry server. 

These are by no means all the advanced features of SmartCloud Entry, if you want to know more then I would recommend that you take a look at the Administrators Guide

 
 

LPAR Migration Error Message- HSCLA24E

This week we where trying to migrate some VIO hosted LPARs using XIV disk from one POWER 7 system to another.  The disk is hosted on a VIO server via the fabric, then using vscsi devices to map up to the servers.  Unfortunately the migration failed and the message we got was -
 HSCLA24E The migrating partition's virtual SCSI adapter 2 cannot
 be hosted by the existing virtual I/O server (VIOS) partition on the
 destination managed system.  To migrate the partition, set up the
 necessary VIOS hosts on the destination managed system, then try
 the operation again.

So we did some searching and found the following -
HSCLA24E error:
1) On the source Virtual I/O Server partition, do not set the adapter as required and do not select Any client partition can connect when you create a virtual SCSI adapter. (can cause the error code).
2) max transfer size of the used hdisk may not be different on source and destination VIOS.
3) The VIO Servers may not have set the reserve_policy to single_path, no_reserve is required.
4) Destination VIO Servers are not able to see the disks the client needs.
5) the same VTD (Virtual target devices) names may not exist on the destination system.
 
But as we have XIV disk you also need to ensure that the following matches up on the source and destination system else it will fail as above:
 
Boot disks -
algorithm       fail_over
max_transfer    0x100000
queue_depth     256
Data disks -
algorithm       round_robin
max_transfer    0x100000
queue_depth     256
 
Setting the 'fail_over' on the boot disks is important as while you might be able to see the disk, 'round_robin' will cause the OS to fail to boot. If the information is different between the VIO Server you are migrating too and from then the migration will fail, so this means your need to perform some pre-activities to ensure it works.

IBM SmartCloud Entry on POWER.

These are my build notes take from work that I have done, along with this I have tried to include any important information from the build guides and other information that I have found. From this you should be able to deploy a working POWER SmartCloud Entry.

First you will need to deploy the basic VMControl, IBM Systems Director on POWER frame work. The details of this are covered in my other blogs, So make sure you have that working before you go further else you could spend time debuging a issue that is nothing to do with SmartCloud Entry.

Installation -

Ensure you have Java 1.6 installed (from the command line issue java -version and verify it is 1.6.x)

JRE is min required, A full SmartCloud Entry will work as well;

For power (AIX) installs, a 32-bit JRE is requied; for x86 (windows or linux) 64-bit JRE is required and can be checked with -

# java -version

Obtain the installation package and copy it over to the server that you wish to run SmartCloud Entry on, in this example we are running it on our Systems Director/NIM sever.

# cd /tmp/sce

# ls

skc220_aix_installer.bin

Then issue a chmod for execute authority on the installer

Chmod +x skc220_aix_installer.bin

• Install IBM SmartCloud Entry as follows:
  ./skc220_aix_installer.bin
• Select your Locale and press Enter.
• Then accept the license once you have paged through all the screens.
• Keep the default option or for example enter option 3 for another location such as - /SCE
• Type Y to verify the details if changed.
• Press Enter to accept the default to Install File Sets.
• Choose install folder. Press Enter and Y to verify.
• Note: If accepting default - /opt/ibm/SCE
• Enter a location for the Property File Install Folder if you changed it above.
• Verify the Pre-Installation Summary, Press Enter.
• Verify the Installer finished successfully. To enter configuration options through the wizard, press Enter. This is the default for the installer.
  Note: You will not be able to connect to the cloud until the hostname of the target cloud is defined.
• Accept the default for the initial admin user of admin. Press Enter.
• Enter the password twice for the admin user. Press Enter.
• Accept the default value for cloud properties of IBM Systems Director, VMControl. Press Enter.
• Enter Cloud hostname of your IBM Systems Director VMControl Server. Press Enter.
• Enter the cloud username and password. Default is root. Press Enter to accept that default and then enter the password twice for verification. Press Enter.
• Verify the installation completed successfully. Press Enter to exit the Installer.

Setting up SmartCloud Entry -

Now before we start SKC edit the /opt/ibm/SCE/skc.ini file to update the very first line -console, as follows and save it -

-console

7777 ← Add this line

-clean

-vmargs

Now when you want to get to the SCE console just run this from a command prompt on the skc server -

# tn localhost 7777

If you want to enable approvals and billing (and metering), which are off by default then do the following -

To turn on billing, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/billing.properties -> com.ibm.cfs.billing.enabled=true

To turn on metering, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/metering.properties -> com.ibm.cfs.metering.enabled=true

Starting SmartCloud Entry -

Now all this is complete we can look at starting the service.

Open URL - http://localhost:8080/cloud/web/login.html

Or you can used - http://localhost:8080/cloud/web/index.html?preventCache=true

Note: this is to keep the browser from caching objects for SmartCloud Entry - useful during development drivers.

Login as admin using the default password you set earlier.

Once you have logged in you will need to set the 'Cloud Status' on the right hand side, so select – Edit Cloud connection. Then enter the relvant details, be this hostname or IP address, user id and password of the system, in our case it was the root user. If the test comes back OK then you can update the settings with those entries.

Updates to SmartCloud Entry -

Now it is installed we can download the update package (zip file) from IBM Fix Central to the computer running the IBM Starter Kit for Cloud application. Create a directory on the local file system of the computer running the IBM SmartCloud Entry application. For example in - /tmp/sce and then extract the update package (zip) to the directory created. Once completed run the following from the IBM SmartCloud Entry OSGi console:

# addrepo file:PATH_TO_DIRECTORY

Where PATH_TO_DIRECTORY is the full file path to the directory the update zip was extracted to. From the example directory given above, type: addrepo file:/tmp/sce

# installupdates

Then CLI output from the installupdates command should indicate installation success.

After installing the updates to the application, IBM SmartCloud Entry should be restarted. To restart:

Type close into the OSGi console in which the application is running.

Once the application ends, the console window will close indicating the application has stopped.

Start IBM SmartCloud Entry, and the installed fixes should now be active.

Updates for IBM Systems Director -

Once the updates above have been completed we can add further updates to IBM Systems Director that are specific for SCE. So once again download the packages to a directory on the servers and extract them from the zip. When extracted run the following -

# smcli importupd -vr /tmp/sce/ISDskc9

.ATKUSC206I Generating SDDs for path: "/tmp/sce/ISDskc9".

..ATKUPD293I Update "com.ibm.director.core.database.feature_6.2.1.2-20111009-LAFix" was successfully imported to the library......

ATKUPD573I Running compliance for all new updates that were found.

..ATKUPD286I The import updates task has completed successfully.

This will import all the fixes in to the system to allow you to update them as follows –

# smcli installupd -v -n pclm2_sdnim -u com.ibm.director.core.database.feature_6.2.1.2-20111009-LAFix

Which you can do for each fix, then as below for the next set of fixes -

# smcli importupd -vr /tmp/sce/VMCUpdate2312

.ATKUSC206I Generating SDDs for path: "/tmp/sce/VMCUpdate2312".

.ATKUPD293I Update "com.ibm.director.mgr.virtualization.server.feature_6.2.1.2-e20110803-LAFix" was successfully imported to the library......

ATKUPD573I Running compliance for all new updates that were found.

.....ATKUPD286I The import updates task has completed successfully.

# smcli installupd -v -n pclm2_sdnim -u com.ibm.director.mgr.virtualization.server.feature_

6.2.1.2-e20110803-LAFix

Now this is all complete you will need to stop SmartCloud Entry and then restart Systems Director.

I will deal with the Cloud tasks and options in a later update.

POWER Cloud intergrating IWD

Now that we have built our POWER Cloud frame-work we can look at integrating this in with IWD, I have been testing with both 3.1 and 3.  As things move on, I will be looking at testing this with Starter Kit for Cloud and IBM Service Delivery Manager.

- IWD Setup for POWER - 

- System Requirements - 
1/2/4 Procs Ent1 - Uncapped-168 - 4/4/8 Virt - 8/12/16Gb Mem - 30-60Gb storage, choose large if it is to be the NIM Master too.  AIX 6.1.5.3 
As we will be using IBM Workload Deployer this has been updated to match the cpu/memory/storage requirements, NIM Server requires Director Common Agent installed if it is not on the same node + VMControl sub agent POWER System 6/7 1 Dedicated HMC model CR5 and above.
 
- NOTES - 
In my blog post above 'POWER Cloud Install Notes' the pool sizes are rather small examples only, ideally you need about 40GB per deployed image, so even a small environment is going to be about 200-500GB in size.  Generally we will be pushing our installs to the cloud_boot pool and using the cloud_data for the System Director/NIM server so this should stay fairly static.  Though this of course is just the way I have set it up, so feel free to change it to suit your own needs 

- Final Configuration Updates - 
Once you have the POWER Cloud underlying infrastructure set-up you need to make the following updates to the system, this will ensure that there will be no issues between the POWER VMControl and IWD.
Check ulimit –a to ensure that “file(blocks)” is unlimited. The “nofiles(descriptor)” setting should be at least 6000 or unlimited. These parameters are set by editing /etc/security/limits; unlimited setting is “-1” in that file.
 
NIM tuning suggestions:

 # no -p -o tcp_recvspace=524288
# no -p -o tcp_sendspace=524288
# no -p -o udp_recvspace=655360
# no -p -o udp_sendspace=65536
# no -p -o arpqsize=128
# no -p -o rfc1323=1
# no -p -o sb_max=8388608
# chdev -l sys0 -a maxuproc='1024' 

The /tmp file system should be enlarged to have at least 15GB of free space. If you plan to store installation or fix pack files in /tmp, consider making the file system 20GB. The ISD will use the /tmp file system during IBM Workload Deployer deployments to temporarily cache catalog images as they are being passed to NIM. These catalog images can be larger than 10GB.
You will need a group of IPs (with resolvable hostnames) for your virtual system deployments. You will place these IP entries within an “IP group” within the IBM Workload Deployer appliance. In addition to the cloud system customers, these IPs/hostnames must be accessible from the HMC and from the IBM Workload Deployer appliance. Test the DNS entries for these IP address and hostnames using nslookup (for example). 

File systems (minimums) -
File system                 Minimum size     Comments
/  (root file system)     1GB
 /usr                             4GB
 /var                             2GB
 /opt                             6GB                   ISD installs in /opt & needs at least 4GB more.
/tmp                             15GB                 At least 1GB is required for ISD installation.

Workload Deployer V3.0, images will be staged here temporarily during deployment, so must have at least 15GB of free space at that time.
 
On the IBM Systems Director lpar, make these customizations: 
    Edit <sysdir_installation_root>/lwi/conf/overrides/USMIKernel.properties
    Add these two lines -

    Kernel.REST.Max.Concurrent.Requests=40

    max.cli.threads=17 

    Edit <sysdir_installation_root>/lwi/conf/overrides/director.javaopt 
     Change the Java heap to -         –Xms2048m –Xmx4096m

 This setting is based on an 8GB IBM Systems Director server, in my testing I have been running about 12GB initially.  Along with this I increased the processing to 2 full processors, 4 virtual. 
Set paging space for IBM Systems Director to be 3GB minimum. 

- IWD Deployment to POWER Issues - 
Now we did hit one issue once we connected to IWD with our system, in that the upload of the appliance to our image repository was successful once we had sufficient space, at the creation of the LPAR failed with the following message on IWD - 
Virtual machine could not be registered
VMControl Message: Start asynch work run for deploy of virtual appliance :7926
VMControl Message: Start asynch work run for deploy of virtual appliance :7926
VMControl Message: Workload, DB2 Enterprise 9.7 (PowerVM), was created. 

Looking at the agent logs - /opt/ibm/director/agent/logs/error-log-#.html - found the following -
       CLASS NetworkAdapter:
 Primary Adapter = true
 IP Address = 172.19.105.227
 Host Name = hiciwd227
 Gateway = 172.19.105.1
 Netmask = 255.255.248.0
 DNS1 IP Address = 172.19.105.30
 DNS2 IP Address = 172.19.105.30
 Domain Name =
 MAC Address = null
 Cable Type = N/A
 Speed = auto
 Duplex = auto
 Network Type = ent
 Display name= networkport.6]
 ::class.method=NIMMaster.validateDeploy() ::thread=HttpServer : 6 ::loggername=com.ibm.director.im.rf.nim.master
 ::class.method=unknown ::thread=HttpServer : 6 ::loggername=SystemErr
2011/11/24 17:08:41.730 INFO 2011/11/24 17:08:41.729 INFO Image /export/nim/appliances/4d3571a0-3ffe-4623-8026-d59e69a03ee6/image1.mksysb referenced by NIM Reso
urce appliance-1_image-1 was found. ::class.method=NIMMaster.validateAppliance() ::thread=HttpServer : 6 ::loggername=com.ibm.director.im.rf.nim.master
 ::class.method=unknown ::thread=HttpServer : 6 ::loggername=SystemErr
2011/11/24 17:08:42.239 INFO 2011/11/24 17:08:42.238 SEVERE DNZIMN620E Error occurred while trying to resolve hostname from IP 172.19.105.227. ::class.method=NI
MMaster.getResults() ::thread=HttpServer : 6 ::loggername=com.ibm.director.im.rf.nim.master
 ::class.method=unknown ::thread=HttpServer : 6 ::loggername=SystemErr 

As you can see it can't resolve the address that it has assigned to the new deployed LPAR, and this was because these addresses where not in DNS.  You can fix the issue by adding them in to DNS or /etc/hosts on the NIM server. 

- IWD Example Image -