Using Application Clusters

The ability to run multiple @task eservers against the same database is not supported by default. Unless each server is aware of other servers in the cluster and coordinates its activities with these other instances of @task, deploying multiple servers could result in data corruption and unpredictable behavior. Additional setup is required to make this work.

An @task cluster consists of 2 or more instances of @task running on different hosts that access the same database.

When installed properly, an @task cluster can dramatically increase scalability by distributing the workload across multiple servers to improve performance and availability by allowing seamless fail-over when a host goes down.

Basic Overview of Cluster Functionality

The first server to start becomes the master node. The master node is responsible for controlling the cluster. All cluster synchronization and communication is routed through the master node. If it goes down (or the connection is lost) a new node is elected as the master node. When the original master node is restarted it rejoins the cluster as a new node, but not as the master node. After a node is elected as master node, it remains the master node until it is shutdown.

Fail-over, in the case when a server goes down, happens automatically. @task API requests are automatically rerouted. There is no need to modify client code. HTTP requests will have to be retried manually by web server software or a network load balancer depending on the customer installation.

If the server that goes down is the master node, there may be a delay as JBoss elects a new master node and recreates the messaging hooks to support cluster communication. However, requests to the cluster are queued during this period, and requests will be serviced normally once the master node configuration is complete.

Cluster Support Checklist

When setting up a clustered environment, all machines in the cluster must:

Have the same operating system. JBoss clustering will not work unless all machines in the cluster have the same operating system installed. Please note if your database is running on a separate server, it DOES NOT have to be running the same operating system as the servers running JBoss and @task
Have the same versions of JBoss and @task. It is important to ensure that all servers in the cluster have identical copies of both JBoss software and @task.
Share the same network subnet address and allow for IP Multicast.
Are isolated from other potential instances of JBoss. The network should be properly isolated so that nodes outside your production cluster are not "discovered" by nodes running inside your production cluster. Enforcing this will eliminate any crosstalk that could occur with other machines on site being used for QA or staging purposes.
Have access to a common file system. Because @task relies on direct file system access for many of its functions, it is critical that all servers have access to the ATTASK_DOC_HOME directory using the same path. This means that the ATTASK_DOC_HOME environment variable on each server should contain a file system path that points to the SAME network-mounted directory shared by all servers in the cluster.

JBoss Cluster Setup

@task software is bundled with 6 separate cluster support folders. The naming convention for the folders is as follows: <operating system>-<database type>-cluster-support. It is important to match up the correct folder for the database.

Supported operating systems include:

unix - refers to any version of Unix (Solaris, HP/UX, AIX, etc.), Linux (Red Hat, Debian, etc.), or Apple Macintosh OS X.
windows - refers to any version of Windows (XP, 2000, 2003, etc.)

N O T E : Theoperating system refers to the operating system of the servers running JBoss and @task.

Supported databases include:

MSSQL - Microsoft SQL Server 2000/2005
Oracle - Oracle 9i or later
MySQL - MySQL 5.0 or later

N O T E : The database can run on a different operating system from that of the servers running JBoss and @task. It is not important which operating system the database is running on.

The included directories are:

/unix-mssql-cluster-support
/unix-oracle-cluster-support
/unix-mysql-cluster-support
/windows-mssql-cluster-support
/windows-oracle-cluster-support
/windows-mysql-cluster-support

JBoss and @task Clustering Setup

1. Backup your current JBoss installation. Find your JBoss installation directory (JBOSS_HOME) and copy the "JBOSS_HOME/server/default/" directory and its contents to another location on the file system outside JBOSS_HOME. This will allow you to revert the cluster configuration should you choose to do so.

2. In JBOSS_HOME, delete the "JBOSS_HOME/server/default/deploy/jms" directory on each server in the cluster. This directory, if left on any of the servers in the cluster, will conflict with the JMS setup used by the cluster.

3. Shutdown @task Make sure, if you have multiple instances already running, that all instances of @task that will be in the cluster are shutdown.

4. Copy the contents of "/<operating system>-<database type>-cluster-support" to the "JBOSS_HOME/server/default" directory on each server in the cluster. It is important that each server have the identical configuration files.

5. Please note, that the contents of this directory are meant to copy over the top of the JBOSS_HOME/server/default.

6. Start @task

7. Login to @task using an account with administrator privileges. Enable clustering in the @task Configuration by changing the configuration parameter `Cluster Enabled' to `yes'.

8. Restart all @task servers

Client Configuration Changes Using The @task API With Clustering

This instruction assumes you are already familiar with how to connect to the @task API in a single-instance mode.

EJB API

The JNDI parameter "java.naming.provider.url" must be changed in order to connect to an @task cluster.

This value should be changed to a comma-delimited list containing the values <host address>:1100.

For example, suppose you have an @task cluster with 3 servers with hostnames host1, host2, and host3. The java.naming.provider.url property should have a value of:

java.naming.provider.url=host1:1100,host2:1100,host3:1100

This property will enable the API client to try each of these hosts, in order for an initial connection to the EJBHome interface. Once a connection is made, the API client will handle its own load balancing and fail-over. No additional code is needed on the client side to take advantage of these features.

SOAP API

Currently, there are no tools that allow for SOAP clients to automatically discover multiple hosts in an @task cluster or that handle automatic fail-over.

Connecting to an @task server in a cluster is exactly the same as connecting to a single server. Any rollover or load balancing policies must be implemented by the client.

FAQ

Clustering is not working and I'm seeing the following error in my logs:

ERROR [HypersonicDatabase] Starting failed
jboss:database=localDB,service=Hypersonic
java.sql.SQLException: General error: java.lang.NullPointerException
at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
at org.hsqldb.jdbcDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
at java.sql.DriverManager.getConnection(DriverManager.java:171)

This means that the Hypersonic database (the internal datastore that handles JMS messages) has become corrupt. To fix this problem, shutdown all @task servers in the cluster and delete the contents of the JBOSS_HOME/server/default/data/hypersonic directory. These files will be recreated once JBoss is restarted.

This will cause any outstanding messages to be deleted. If this problem persists, contact customer support.