Let’s Encrypt SSL Certificate Zero Downtime Renewal in WordPress Cluster

| March 18, 2021
Let's Encrypt SSL Certificate Automatic Renewal

Nowadays, an SSL is a necessity for any website in production, thus certificate issuing and renewal are routine tasks that can be easily automated by tools like Let’s Encrypt. These processes are performed using open-source certbot with the standalone authenticator. The main drawback here is the possible downtime of the web application server during the automatic certificate renewal and domain validation.

In order to overcome the problem with uptime during SSL certificates update, Jelastic created a built-in Let’s Encrypt Add-On with an advanced zero-downtime renewal algorithm.

Downtime Reasons while Updating SSL Certificate with certbot

The certbot command-line tool's main purpose is to configure an HTTPS server and automatically obtain a trusted certificate without human intervention. It performs the following tasks:

  • proves that you are in control of the website
  • receives a certificate and configures it on a web server
  • tracks certificate expiration and updates it
  • helps users revoke their certificate (if needed)
  • forces certificate renewal when required

First, certbot initiates issuance of SSL certificate by sending Certificate Signing Requests (CSR). Then, it starts a Certificate Management Agent (CMA) which is a Python web server that listens to the specified port to handle a validation request from the Let's Encrypt CA. Before obtaining a certificate, it is necessary to prove that renewal was invoked by the domain administrator. The validation request always happens on port 80, which is usually occupied by the application listening to HTTP traffic. So, the web server which it is running on must be temporarily stopped to run CMA instead. And consequently such implementation causes application downtime.

Another issue that can cause downtime is that the Python web server can become stuck once it tries to handle a non-validation HTTP request that can come from any user. It happens since certbot runs the web server in a single-threaded mode and consequently it can’t handle multiple incoming requests.

Solution to Avoid Downtime during SSL Certificate Update

In order to omit the mentioned issues, Jelastic created a built-in Let’s Encrypt Add-On that includes an improved algorithm. It presupposes using iptables rules to integrate an intermediate reverse proxy layer into the traffic chain to filter HTTP traffic. The proxy is configured to route the request to the Python web server from Let’s Encrypt CA only. Other traffic is routed to the user’s application.

Iptables Routing to Proxy Port

The add-on does not stop the application's web server and redirects requests that come to website port 80 to the proxy port (e.g. 12345) via the iptables rules until validation is completed.

iptables -I INPUT -p tcp -m tcp --dport ${PROXY_PORT} -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport ${LE_PORT} -j ACCEPT
iptables -t nat -I PREROUTING -p tcp -m tcp ! -s 127.0.0.1/32 --dport 80 -j REDIRECT --to-ports ${PROXY_PORT}

Where:

${PROXY_PORT} - 12345

${LE_PORT} - 12346

Intermediate Reverse Proxy

Tinyproxy is used as an intermediate proxy. It is a lightweight HTTP/HTTPS proxy-server built from scratch to be fast and compact. The startup speed is a key advantage of this software. Thus, we can ensure uninterruptible work for the user's application by switching traffic routes through the intermediate proxy layer. Its config file looks very simple and can be found in the Let’s Encrypt add-on repository:

User tinyproxy
Group tinyproxy

Port 12345
Timeout 6000

MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 0

BindSame yes
DisableViaHeader Yes

ConnectPort 80
ConnectPort 12346

AddHeader "X-Forwarded-Proto" "http"
ReversePath "/" "http://127.0.0.1/"
ReversePath "/.well-known/acme-challenge/" "http://127.0.0.1:12346/.well-known/acme-challenge/"
ReverseOnly Yes
ReverseMagic Yes

Here the ReversePath directives enable reverse proxy support and define the HTTP traffic filtering rules. Any request that will come with the substring "/.well-known/acme-challenge/" will be routed to the Python web server that listens to port 12346 to get a validation file from the hidden .well-known/acme-challenge/ directory. Only Let's Encrypt CA generates such requests. And the rest of the HTTP traffic will be routed to the web application that listens to port 80. This is how the Jelastic Let’s Encrypt add-on proves that CSR was initiated by the domain administrator.

Use Case: Zero Downtime SSL Update in WordPress Websites

Now, let's see how it works in a real case scenario using a WordPress application as a sample. While WordPress package (Cluster or Standalone) installation, enable “Install Let’s Encrypt SSL with Auto-Renewal” option.

Let’s Encrypt SSL Certificate WordPress Cluster installation
  • WordPress Cluster

In the case of a WordPress clustered environment, traffic is redirected from all load balancers to the Master node (the first or initially created container in the layer), where the validation server is located.

Let’s Encrypt SSL Certificate Renewal WordPress Cluster
  • WordPress Standalone

Traffic flow in a WordPress standalone container (with LLSMP or LEMP certified templates) is shown in the illustration below.

Let’s Encrypt SSL Certificate Renewal WordPress Standalone

According to the filtering rules, proxy outputs traffic either to the Python web server port 12346 or to port 80 of the user's web server.

Renewing Let’s Encrypt Certificate Using Webroot Authenticator

There is an additional method to issue new certificates or update existing ones which is called webroot. This method is preferable for highly loaded websites (thus soon will be added to WordPress cluster by default) as it does not interrupt web servers’ work and omit the risk of downtime or performance degradation.

The webroot can be used with certified PHP software stacks as in the WordPress packages. When launched, it obtains a path to the site's root directory. Then certbot creates a directory structure there for storing temporary file:

${webroot-path}/.well-known/acme-challenge

Let’s Encrypt SSL Certificate Renewal with webroot

Let's Encrypt server checks for the presence of this file in the specified location with a request of this type:Let's Encrypt server checks for the presence of this file in the specified location with a request of this type:

http://example.com/.well-known/acme-challenge/KJr5D^ReTW4kY_Z6UIIkjlmlmOkyQgPr_7ArlLgtZE7Fg

If the requested temporary file (for example: "KJr5D^ReTW4kY_Z6UIIkjlmlmOkyQgPr_7ArlLgtZE7Fg") exists, then domain ownership will be successfully verified, and the certificate will be issued and uploaded to the users' website.

After completing the validation procedure, the certbot obtains the certificate file. Usually, the Let’s Encrypt SSL certificate file is stored in the /etc/letsencrypt/live directory where all the files are symlinks to the files in /etc/letsencrypt/archive directory.

So as you can see, there are options on how to achieve full uptime while updating SSL certificates. And to make it even more easy and automated, Jelastic implemented the required algorithm in Let's Encrypt add-on, as well as made this solution built-in to the WordPress cluster and standalone package. Get your own experience using free Let's Encrypt SSL certificates with zero downtime renewal procedure empowered by Jelastic PaaS.

Related Articles

Free Let’s Encrypt SSL Certificates: Out-of-Box Integration with the Most Popular Software Stacks
Highly-Available WordPress Hosting with Automatic Clusterization on All Stack Layers
WordPress Hosting in Elastic Standalone Container
WordPress Security Optimization for High-Performing Websites