There is safety in numbers: how to build fail-safe clusters

| April 10, 2012

Very often, websites can't cope with any extra load, whether from high traffic or performance reducing tasks. As a result, companies lose money, as well as taking a reputation hit. Programmers blame system administrators. System administrators blame managers. Managers blame service providers. And vice versa. In the end, in this day and age, you can't afford down time: it will hurt your bottom line. So, how do you solve this issue?

One of the best solutions for such issues is using a cluster architecture for your applications. We'll show you how to create fail-safe clusters within GlassFish and within Jelastic.

1. Background

You need two hardware nodes, n1 and n2, for GlassFish servers and then machine lb1 for load balancing and machine db1 for the database. Make sure, that all machines can "see" each other (ping).

All software will work under Linux control. Also we'll use a preset image of a virtual machine with CentOS installed. GlassFish uses SSH to control remote nodes. That's why we have to install SSH-server on each machine of the cluster. Moreover we need separate accounts in every OS (specify their names, for example: glassfish).

# adduser glassfish
# passwd glassfish

GlassFish uses nonstandard ports, so you have to specify required rules in the firewall or just switch it off. Let's choose the easiest variant:

# service iptables save
# service iptables stop
# chkconfig iptables off

And of course, install JVM.

Unfortunately, there are a lot of configurations and settings, when you create a cluster manually. So, be patient!

2. Configure the cluster

We configure the cluster via command line with the asadmin utility.

1. Download GlassFish distribution


and unzip the package.

2. Start the administrative domain instance (domain1):

./asadmin start-domain domain1

GlassFish disables remote connections to the admin domain. That's why all cluster nodes, except DAS nodes, can't cooperate with it.

Typical error:

Failed to rendezvous with DAS on n1:4848. Please check if this server is running, that the host and port are correct, and that this server is configured to allow remote access.

To fix it type the next string:

./asadmin enable-secure-admin

and restart administrative domain.

Create your cluster:

./asadmin create-cluster c1

Create two application servers on this hardware node:

./asadmin create-local-instance --cluster c1 i1
./asadmin create-local-instance --cluster c1 i2

3. Start the cluster:

./asadmin start-cluster c1

4. Add new instances on the second hardware node :

./asadmin –host n1 –port 4848 create-local-instance –cluster c1 i3
./asadmin –host n1 –port 4848 create-local-instance –cluster c1 i4

5. Go to GlassFish admin page, find the instance you have just created, change its type to SSH and change Node Host to n2. Also you have to specify the login and password for SSH access to machine n2. As you can see below, all the clusters are working now:

3. Load balancing and HA

1. Install NGINX:

#yum install nginx
#nano /etc/nginx/nginx.conf

2. Set round-robbin balancing for 4 servers on NGINX with sticky session support:

upstream backend {
server max_fails=5 fail_timeout=15s;
server max_fails=5 fail_timeout=5s;
server max_fails=5 fail_timeout=5s;
server max_fails=5 fail_timeout=5s;

3. Hosts n1 and n2 should have their common parent domain, for example: Edit the files /etc/hosts on both machines:

n1: localhost

n2: localhost

4. Check multicast on each hardware node:

./asadmin validate-multicast

5. Change n2 to in DAS admin console.
6. Web-descriptor of your application should contain a suitable instruction (<distributable>) for providing replication of http-sessions. Don't forget to set the Availability flag, when you deploy your app.

4. Configure database

We'll use PostgreSQL as an example.

Install the database, create a new user and database:

# yum install postgresql-8.4
# service postgresql initdb
# service postgresql start

Specify allowed subnets in the pg_hba.conf (/var/lib/pgsql/data/pg_hba.conf) file :

host all all md5

5. Application deployment

Now you can deploy your application (for example: jforum).

Obviously, this is time consuming!

That's why automated all these steps for you in Jelastic.

You can create fail-safe clusters with just a few clicks from within your dashboard:

Make things easy for yourself.
Use Jelastic.
Create fail-safe clusters. Easily.