File Synchronization Between Several Application Servers in the Cluster
Time invested and a lot of hard work hopefully results in your site or app becoming more and more popular. That’s great news, but now you need to keep up with the increased traffic. A solution to this is to migrate your application on multiple web servers. Of course, it is more difficult to run such applications as WordPress, Drupal, Joomla, Liferay, Redmine and others that are similar, across multiple web servers.
As result, you face another challenge due to the increased complexity over a single-server installation of your application. The main problem is that these applications by default save all uploaded files only to one server and do not synchronize these files between server nodes in the cluster. In other words, only the server which was processing the file upload request will contain newly added content. The rest of the servers will fall out-of-sync.
In today’s post, we want to address the issue of keeping uploaded files in sync or, in other words, we will provide solution for the file synchronization problem.
For now this is just a workaround which helps to automate content synchronization on multiple app servers. In the future, our team will integrate it into the core system after we process feedback on this functionality.
The current solution is a combination of bookmarklet and server side script. Basically, server side script is used with a help of such a utility as lsyncd together with cron.
Lsyncd is a light-weight, live mirror solution used to synchronize app servers. Being wisely coupled with inotify, lsyncd initiates file sync only if it has detected any actual changes in the system. In such a way, the load on CPU is decreased and you do not burn up many resources on processing the synchronization.
With a help of our bookmarklet, lsyncd is automatically installed on each app server available in the environment. Before its installation, you will be asked to choose the folders you want to synchronize. As a result, when a change is made in the chosen folders on any of the app servers, those changes will be synced to each server node automatically.
This robust and efficient solution makes the file synchronization system as simple as possible. So now let’s see how to turn this into reality, step-by-step.
In our example we will use the WordPress application deployed to the environment with two servers. This instruction is also fully appropriate for other PHP, Java or Ruby applications such as Drupal, Joomla, Liferay, Redmine and others.
1. You can install your WordPress application manually or using our JPS widget for one click installation. The instruction and the widget can be found in the WordPress document.
2. Now let’s create a cluster, to increase the number of app servers.
By using instructions in our documentation, you can easily add extra app servers, enable high availability or even configure a clustered solution:
In our case, we have simply added one more app server to the environment. And, you can use a more complex scenario by setting the clustered solution with your WordPress application.
3. Click Open in browser for your environment.
4. Complete the WordPress installation by providing the required information.
Test File Upload Without Synchronization
Let’s check the file upload processing by the servers, without synchronization enabled.
1. Go to the admin panel and make your custom changes: upload images or files, edit the themes etc. In our case we uploaded one image to the Media Library.
2. Return to the Jelastic dashboard and click Config next to your server node.
3. Navigate to the webroot > ROOT > wp-content folder.
At one of the app servers you will see a newly added uploads folder that contains your newly added files.
Note that the folder with uploaded files depends on the app that you are running.
Here is the list of such folders for some widely used applications:
4. If you navigate to the same folder (wp-content, in our case) of the second node you’ll see that there is no uploads directory there at all.
As a result, you will see that the added image is located only at the server that processed the file upload.
Apply File Synchronization
Now let’s proceed to solving the problem of file synchronization in your cluster.
1. Firstly, drag this bookmarklet to your browser bookmarks bar:
2. Navigate to your Jelastic dashboard and initiate the script by clicking on File Synchronization in your bookmarks bar.
3. In the opened window, choose the environment with your application and check the folder (or several folders) you want to sync. Click Install.
Note: do not select the whole webroot folder because this will disable the synchronization
4. Wait a short time (about 2 mins) for the settings to be applied. Initially, you will see that on both server nodes lsyncd folder, install.sh and replication.tar files appear in the webroot directory.
5. In a moment, check the webroot > ROOT > wp-content folder on all app servers.
As you can see, the previously uploaded image is located in the uploads folder on each node.
It’s that easy. As a result, the synchronization will be automatically performed every time you add or change anything in your WordPress (or any other application).
Change Topology, Keeping Synchronization
If you change the environment topology by adding more app servers, you need to initiate the script again in order to keep the synchronization process on all nodes.
1. Click Change environment topology for the environment with your application.
2. Add extra servers and Apply the changes.
3. Perform 2-4 steps of Apply File Synchronization block in order to run internal process for setting synchronization.
Note: After additional running of the script, the previous configurations will be reset and the newly chosen settings applied.
The synchronization process can be monitored via logs.
1. Navigate to the webroot/lsyncd/var/log folder.
2. The status of sync can be viewed in the lsyncd.log and lsyncd.status files.
We trust that these instructions are helpful to you. Use Jelastic to create new apps and create engaging content and get even more users! That won’t be a problem if you are hosting your app with our platform, as there is no need to follow or process your changes. Just make them and leave the rest for Jelastic.
Can you solve the problem of file synchronization in any other way? Please, share your experience in the comments below.