Jelastic Platform-as-a-Service provides certified support of various stacks (application servers, databases, load balancers, cache servers and others) and this list can be extended with custom Docker-based templates. In this article, we’ll cover the steps on how to build a software stack as a container image (using Apache Kafka as a sample) and make it available as a custom template within the dedicated platform installed on-premise or on top of preferred cloud infrastructure.
Apache Kafka is a distributed streaming platform written in Java and Scala, which utilizes a publish/subscribe messaging model. In Kafka, the producers write messages to the topics and consumers (subscribers) of the appropriate topics read them. It includes a set of utilities for creating themes, sections, ready-made publishers, subscribers, etc.
Now, let’s follow the steps required to get Kafka up and running as one of the available templates within topology wizard of your Private Jelastic PaaS installation.
This guide is aimed for the platform owners, i.e. admin access to the Jelastic Cluster Admin Panel (JCA) is required. You can install your own Private Jelastic PaaS in one of the following ways:
- Automated installation of Lite Edition PaaS on Digital Ocean
- Automated installation of Lite Edition PaaS on Google Cloud
- Request Full PaaS installation on the preferred infrastructure for private use (performed by Jelastic)
- Request Business PaaS installation for service providers (performed by Jelastic)
Also, you’ll need a Docker Hub account to store your custom images and Docker Engine CE to build them (can be installed locally or at your Jelastic account). In this tutorial we’ll cover the latter option:
1. Locate the Docker Engine CE package in the Marketplace in the Dev & Admin Tools section.
2. Within the installation frame, choose the Create a clean standalone engine option and click Install. If required, change the Environment name.
3. Once the environment is created, it can be used for building templates. Access the Engine Node via Web SSH to start.
We have prepared a ready-to-work example for the Kafka stack template building, publishing and testing. You can download this pre-configured example from GitHub repository https://github.com/jelastic/kafka-image-building and follow the video tutorial that shows the whole process from scratch including Jelastic Private PaaS installation on the top of DigitalOcean infrastructure.
Below you can review the steps of building and publishing Kafka stack provided in the form of step-by-step instruction.
Composing Kafka Dockerfile
Create a Dockerfile via any preferable text-editor, e.g. vim:
Fill it with the content below.
Building an Image
1. Let’s create a dedicated directory to work at, for example:
Enter the directory and add a Dockerfile from the previous section.
2. The next step is to create a directory tree where the system service files will be located. These files are added to the image due to the ADD src/. / line in Dockerfile.
$mkdir -p src/etc/sudoers.d src/etc/systemd/system src/etc/jelastic src/var/lib/jelastic/overrides
The directory structure will be created as follows:
3. Now, you need to create several configuration files to set up your stack. Use vim text editor to create listed below files and provide the required content.
Define start/stop service commands that are allowed to be executed as user kafka in src/etc/sudoers.d/kafka file.
The image uses systemd to initialize multiple daemon services. Thus we are in need to create a system container as for our example, since the utilized Kafka distribution has ZooKeeper as an integrated component, and the two services are running in a single container. As a result, two appropriate systemd service files should be created (kafka.service and zookeeper.service).
As you can see in the [Unit] section, the kafka service depends on ZooKeeper and only starts after the zookeeper service.
Create another service file for ZooKeeper similar to the previous step.
The src/etc/jelastic/favourites.conf file configures shortcuts to the most common files and directories at the left pane of Configuration File Manager.
The redeploy.conf file lists files to keep during the container lifecycle.
It is used by JEM to determine the template-specific logic (service initialization and restart in the case of Kafka).
4. Ensure that the stack template icons are available via the sourceUrl specified in the last line of the Dockerfile. However, if needed, you’ll be able to change them later on at JCA > Templates > Edit Template.
5. Build an image with the next command:
$docker image build -t <dockerhub-account>/kafka:2.4.1 .
6. Once it is built, login to your Docker Hub account and push the image to it:
$docker image push <dockerhub-account>/kafka:2.4.1
Adding and Testing Template
Once your image is available at Docker Hub, it is time to add it as a custom template via the Jelastic Cluster Admin Panel. You can check the dedicated tutorial on how to add a template to the platform.
Also, we’ll cover the main steps below:
1. In order to add a new template, go to the JCA > Templates section and click Add > From Docker Repository.
2. Type the Repository name the same as you’ve used at the building stage (i.e. <dockerhub-account>/kafka). Press Enter or click the magnifying glass icon to proceed.
3. If everything is done properly, the image tags will be pulled from the Docker Hub repository.
All the template metadata is automatically pulled from the image (icons will be applied automatically during addition). So, choose a tag that should be published as default and click Add.
4. Now, your template should appear on the list. We recommend testing the image before publishing it. Select Kafka and click Actions > Preview Unpublished in the tools bar.
Your dashboard account with all the software templates available (including unpublished ones) will be opened.
5. Create a New Environment with the Kafka template (at the Extra Services) and click Create.
6. Once the environment is created, you can check whether the Kafka broker server works properly. Open the Web SSH terminal and run the commands as follows:
- create a new test-topic topic
$/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --zookeeper localhost:2181 --partitions 1 --replication-factor 1
- write a few messages into the created topic
$/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic
- exit with Ctrl+C shortcut and try to read the messages that were sent by the producer (the --from-beginning flag is required to view messages sent before):
$/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning
Congratulations! As you can see, all of the messages sent to test-topic were obtained by the consumer, which means the Kafka stack is working properly.
7. Now, you can return to the JCA and Publish the template to make it available at the platform dashboard for all your PaaS customers.
See the video tutorial to follow the whole process.
That’s all! You have passed through all of the steps required to create a custom software stack template for your Jelastic Private PaaS and get it published at the platform dashboard. Now, you can use this guide to adopt your own solutions and add them to the platform.