Performance AIX and POWER Notes

Recommend Performance Notes

Below are some details of performance tuning that I have come across in relation to different situations, while while might be true in one case may not work for another.  So please be sure to perform testing on your environment before making these recomendations.

Hardware memory prefetch -
Hardware memory prefetch helps to improve the performance of applications that reference memory sequentially by prefetching memory.

Hardware memory prefetch consdierations - There might be adverse performance effects, depending on the workload characteristics.Determine whether to disable hardware memory prefetch in a production environment after sufficient testing with the application.

In this case it was a JAVA application, which is recomended that you turn off the hardware memory prefetch.  The example below shows how to disable across a reboot using the -b flag, the command creates an entry in the /etc/inittab file -

# dscrctl -n -b -s 1

 
The value of the DSCR OS default will be modified on subsequent boots
 

# dscrctl -q
Current DSCR settings:
        Data Streams Version = V2.06
        number_of_streams = 16
        platform_default_pd = 0x5 (DPFD_DEEP)
        os_default_pd = 0x1 (DPFD_NONE)

# cat /etc/inittab|grep dscrset
dscrset:2:once:/usr/sbin/dscrctl -n -s 1 >/dev/null 2>/dev/console

Memory and Page Size Considerations
IBM Java can take advantage of medium (64K) and large (16M) page sizes that are supported by the current AIX versions and POWER processors. Using medium or large pages instead of the default 4K page size can significantly improve application performance. The performance improvement of using medium or large pages is a result of a more efficient use of the hardware translation caches which are used when translating application page addresses to physical page addresses. Applications that are frequently accessing a fast amount of memory benefit most from using pages sizes that are larger than 4K.

# vmo -o lgpg_regions=16384 -o lgpg_size=16777216 -o v_pinshm=1
- create 16MB large page objects and pin memory, when done should use -Xlp on Java cmd line

VIOS Update

Following command are recommend for updating your VIOS running levels 2.1.0 and above. As a reboot is required on the system, make sure that any active environments are shutdown too (unless Dual VIO);
 
First commit older updates with the following command - 

 # updateios -commit

Then update the server from the media or device so as below its /mnt but it could be /dev/cd# -

# updateios -accept -install -dev /mnt

To apply the changes to the VIOS you will need to reboot (Any active LPARs will need to be shutdown too) -
# shutdown -force -restart
 
Once rebooted, you will need to re-accept the license-  
# license -accept
 
Confirm the update with -
# ioslevel

More on Etherchannel

If you wish to add or remove additional devices to your live Etherchannel adapter then the following command will help -

Add adapter ent4 into the EtherChannel device called ent5 - 
/usr/lib/methods/ethchan_config -a ent5 ent0

To remove ent4 from the EtherChannel device ent5 - 

/usr/lib/methods/ethchan_config -f ent7

Etherchannel with SEA

Configuring your SEA with Etherchannel.

 First check the shared virtual ethernet adapters you have -

# lsmap -all -net
SVEA   Physloc
------ --------------------------------------------
ent3   U9117.MMB.101697P-V1-C2-T1

SEA                 NO SHARED ETHERNET ADAPTER FOUND


Then we need to check what adapters we can use, now in our case we will be using the first 2 adapters from a dual port 1Gb card -  

# lsdev -type adapter
name             status      description
ent0             Available   2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
ent1             Available   2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
  
Then we need to create the Etherchannel device -   

# mkvdev -lnagg ent0 ent1
ent2 Available

This creates the device in standard mode, though you can switch it over to round robin -

# chdev -l ent2 -a mode=round_robin  

Then we can create a SEA "bridge" between the physical Etherchannel device ent2 and the virtual ent3

 # mkvdev -sea ent2 -vadapter ent3 -default ent3 -defaultid  1
            main:  86 Recived SEA events bytes 163
ent9 Available
 
 Once that is done, you can set-up initial TCP/IP config (en9 is the interface for the SEA ent9)

# mktcpip -hostname <vio-name> -inetaddr <ip-address> -interface en9 -start -netmask <subnet> -gateway <gateway-ip>
  
Now your server ready to go.

SmartCloud Entry - Projects, Users and Accounts

Now we have everything the way we want, we can think about working with our projects, users and accounts.   From this we can manage our usage of the SmartCloud and the workloads we are going to be deploying, creating, managing and re-sizing.
First begin by creating the users that you want, all users are created equal until you put them in the relavant accounts and projects.

Users
Select Users -> Select New User ->  

• Fill in the details as you require.   

 If you select 'admin privileges' then they have visibility of advanced features like networking and users creation.

Projects
Once you have your users you need to create a project for them to belong to -
Select Projects -> Select New Project -> 

• Give a name and description. 

Once its created you can select the users to add to the project and there 'Role' in that project.
Accounts
Now we just need to create a account for this project and users so that we can control the billing further.
Select Accounts -> Select New Account -> 

• Give a name and description
• Select the Account owner.
• Give a starting balance and low balance alert threshold.
•  Then add further account member's, this is generated from the project members added earlier.  

Now this is set-up any new workloads to that project will get billed to the relvant selected accounts and the cost taken from their funding. If the users make changes, the project owner will be able to see, amaned and approve/deny these and the account get charges relating to those.  If notifications have been set and e-mail set-up then e-mails will be sent about events on the SmartCloud relevant to those users.
 

SmartCloud Entry - Further Configuration

Billing and Metering

Now that we have the our SmartCloud up, we need to get familiar with how it works and what we can do with it.  Now most of us will be interested in setting up request approval, billing, and user/project control. But before we do those things we need to enable them and check the values are what we require.  Now in the previous blog entry we mentioned how to turn these on -


To turn on billing, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/billing.properties -> com.ibm.cfs.billing.enabled=true

To turn on metering, configure these in the properties files in the .skc dir in the user home dir

.skc/approval.properties -> com.ibm.cfs.request.lifecycle.enabled=true

.skc/metering.properties -> com.ibm.cfs.metering.enabled=true

Once this is enabled we can also look at changing and update the cpu, memory, disk value.

This is the directory tht is used by IBM SmartCloud Entry as a product catalog -

.skc/products

Three files that effecgt billing charges rates - 

.skc/products/ram.xml
.skc/products/cpu.xml
.skc/products/disk.xml

 (taken from the readme) -

The default collector will look at a specific virtual machine property (as defined by the Cloud) and use the value of the property as the units of product to buy, at the products price. For example, IBM VMControl 2.3.x exposes a VM property called "Processor.VirtualQuantity", this property can be used to know the amount of CPUs allocated to a VM, and hence used to charge the VM for a "CPU Cloud product."

For example take a look at the following product:

<cloudProduct id="com.ibm.cfs.services.billing.products.cpu">
<name>CPU</name>
<description>The amount of CPUs used in a virtual machine at a rate of $1 per hour.</description>
<pricing currency="USD" interval="60" price="0.0167"/>
<collector property="Processor.VirtualQuantity"/>
</cloudProduct>

The product above specifies that the default SKC collector will collect charges on virtual machines using the amount of CPUs assigned to them, exposed via the "Processor.VirtualQuantity" property, at a rate of 0.0167 cents every 60 seconds or roughly $1 per hour. The reason why it is important to collect at an interval less than the actual described rate (i.e., seconds instead of hours), is so that users looking at their bills get a higher resolution approximation of their actual charges, having an accurate look at the charges might be important for accounting purposes, or in situations where account credit is limited.

E-mail

SmartCloud Entry can send e-mail notification for several user and admin related events, such as the completion of a workload, account creation, and more.  In order to get these notifications it is necessary to configure a properties file in the home directory.

To setup notification for SmartCloud Entry follow the steps described below:

1. Open the email.properties file in the SCE home directory.
2. Set the com.ibm.cfs.email.relay.host property to the hostname of the relay host that IBM Starter Kit for Cloud should use for outgoing SMTP emails.
3. Optionally, you can set the e-mail subject prefix for all SmartCloud Entry e-mails and the “from” name, by setting the following properties respectively:
• com.ibm.cfs.email.subject.prefix
• com.ibm.cfs.email.from.name
• com.ibm.cfs.email.from.address
4. Save the email.properties file and restart the SmartCloud Entry server.

You can globally disable e-mail notifications in SmartCloud Entry completely by setting the com.ibm.cfs.email.default.notifications property in the email.properties file to false.

Configuring Advanced Features

Virtual Appliances   

The administrator can pre-configure the appliances to simplify its deployment for the users. Configuration the properties can be made through the SmartCloud Entry web user interface or through the deployment.properties file in the SCE home directory.

In order to simplify the deployment process for the users, it is recommended that an administrator configures all of the virtual appliances prior to making SmartCloud Entry available to its users. Virtual appliance customisation often requires knowledge of the low level details of the environment or having advanced knowledge of the data centre of which the user may not be familiar.  

Configuring a staging project

By default, SmartCloud Entry will scan the cloud for new appliances periodically.  When SCE finds a new appliance it places it in the Public project where anyone can deploy it.  SCE also allows configuring of a 'Staging' project where newly discovered appliances will show up rather than having them put directly into the Public project.  This allows the adminstrator to configure the the appliances and to then move them to a different project (Public or a specific project) to give the correct visibility to the users.  

So to configure this staging project add or uncomment this line in the deployment.properties file: 

com.ibm.cfs.staging.project=Staging   

Deployment Visibility

You can also choose whether you want your users to see the advanced deployment options or just the basic options.
The advanced options gives the user a finer-grained level of control over the level of resources consumed by the appliance. You may or may not want to show this advanced panel to the end-user, so to configure the visibility of the advanced deployment options, please follow these steps:  

• Open the deployment.properties file in the SCE home directory.
• Set the deployment.advanced.form.enabled property to false. This will disable the advanced deployment option and it will not be shown to the end user. However, administrators will continue to see the advanced deployment option in the user interface. The default value of this property is true.
• Save the deployment.properties file and restart the SmartCloud Entry server. 

These are by no means all the advanced features of SmartCloud Entry, if you want to know more then I would recommend that you take a look at the Administrators Guide

 
 

LPAR Migration Error Message- HSCLA24E

This week we where trying to migrate some VIO hosted LPARs using XIV disk from one POWER 7 system to another.  The disk is hosted on a VIO server via the fabric, then using vscsi devices to map up to the servers.  Unfortunately the migration failed and the message we got was -
 HSCLA24E The migrating partition's virtual SCSI adapter 2 cannot
 be hosted by the existing virtual I/O server (VIOS) partition on the
 destination managed system.  To migrate the partition, set up the
 necessary VIOS hosts on the destination managed system, then try
 the operation again.

So we did some searching and found the following -
HSCLA24E error:
1) On the source Virtual I/O Server partition, do not set the adapter as required and do not select Any client partition can connect when you create a virtual SCSI adapter. (can cause the error code).
2) max transfer size of the used hdisk may not be different on source and destination VIOS.
3) The VIO Servers may not have set the reserve_policy to single_path, no_reserve is required.
4) Destination VIO Servers are not able to see the disks the client needs.
5) the same VTD (Virtual target devices) names may not exist on the destination system.
 
But as we have XIV disk you also need to ensure that the following matches up on the source and destination system else it will fail as above:
 
Boot disks -
algorithm       fail_over
max_transfer    0x100000
queue_depth     256
Data disks -
algorithm       round_robin
max_transfer    0x100000
queue_depth     256
 
Setting the 'fail_over' on the boot disks is important as while you might be able to see the disk, 'round_robin' will cause the OS to fail to boot. If the information is different between the VIO Server you are migrating too and from then the migration will fail, so this means your need to perform some pre-activities to ensure it works.