Setting up a scalable and highly available Wordpress environment with Rancher and Docker | Rancher L

A few weeks ago Rancher added a Docker Load Balancing service in 0.16 version which allows users to distribute traffic between Docker containers. Last week Rancher 0.20 version was released with a new great functionality which takes Load Balancing to a new level: Services.

A Rancher Service is a group of containers running the same Docker image that you can scale up/down as needed. Rancher allows you to define Load Balancer services that you can link to any service group in order to automatically distribute traffic to the containers running in the linked group.

When you scale up a service Rancher will randomly deploy the new container on any active host, and automatically add it to the linked Load Balancer so it will receive traffic when started.

In my last post I was talking about building a distributed and persistent storage solution for Docker and Rancher using GlusterFS. Today we are using that storage in conjunction with Rancher Services/LB feature to show you how to deploy Highly Available, scalable and distributable web applications.

In this post we are deploying a group of WordPress containers for publishing a blog site on AWS. We are also using a Percona XtraDB Cluster and the GlusterFS storage from last post so there is no Single Point of Failure in our setup.

At the end we are testing scalability and HA capabilities for this solution. This is why the post is divided in these sections:

1. Preparing AWS environment and creating Docker hosts
2. Deploying the storage cluster
3. Deploying the database cluster
4. Deploying the WordPress & LB services
5. Testing scalability
6. Testing HA capabilities

Prerequisites

Preparing AWS environment

Before deploying the WordPress environment you need to satisfy the following requirements in AWS:

• Create an Access Key to use Rancher AWS provisioning feature. You can get an Access Key by clicking your username in AWS console, and then Security Credentials option.

• Go to EC2 Console, Security Groups section and click Create Security Group button. Configure a Security Group named WordPress for the default VPC and with the following inbound rules:

• Allow 22/tcp, 2376/tcp and 8080/tcp ports from any source, needed for Docker machine to provision hosts
• Allow 500/udp and 4500/udp ports from any source, needed for Rancher network
• Allow 9345/tcp and 9346/tcp ports from any source, needed for UI features like graphs, view logs, and execute shell
• Allow 80/tcp port from any source, needed to publish the WordPress site

• Create a RancherOS instance (look for RancherOS AMI in Community AMIs). Configure it to run Rancher Server by defining the following user data and associate it to the WordPress Security Group. Once the instance is running you can browse to Rancher UI: http://RANCHER_INSTANCE_PUBLIC_IP:8080/

#!/bin/bash

docker run -d -p 8080:8080 rancher/server:v0.20.0

Creating Docker hosts

In Rancher UI, click + Add Host button and choose Amazon EC2 provider. You need to specify, at least, the following information:

• Quantity: deploy a total of three hosts
• Hosts names, for example wordpress
• Amazon Access Key and Secret Key that you got before
• Instance Type: choose at least t2.small instances, as PXC needs more than 1GB RAM for replicating a joining node
• EC2 Region, Zone and VPC/Subnet ID. Be sure to choose the same region, zone and VPC/subnet ID where Rancher Server is deployed
• Type the Security Group name that we created before: WordPress

After a while you will see your three hosts up and running.

Deploying the storage cluster

As described in my last post, you can deploy the storage following these instructions. First, click + Add Container button on wordpress1 host and enter the following information:

• Name: gluster1
• Image: nixel/rancher-glusterfs-server:latest

Expand Advanced Options and follow these steps:

• Volumes section – Add this volume: /gluster_volume:/gluster_volume
• Networking section – Choose Managed Network on Docker0
• Security/Host section – Enable Give the container full access to the host checkbox

Now wait for gluster1 container to be created and copy its Rancher IP address, you are needing it now. Then click + Add Container button on wordpress2 host to create the second GlusterFS server container with the following configuration:

• Name: gluster2
• Image: nixel/rancher-glusterfs-server:latest

Expand Advanced Options and follow these steps:

• Command section – Add an Environment Variable named GLUSTER_PEER which value is the gluster1 container IP. In my case it is 10.42.25.141

• Volumes section – Add this volume: /gluster_volume:/gluster_volume
• Networking section – Choose Managed Network on Docker0
• Security/Host section – Enable Give the container full access to the host checkbox

Now wait for gluster2 container to be created and open its menu, then click View Logs option. You will see the following messages at the bottom of log screen confirming that shared volume was successfully created.

Deploying the database cluster

Now it is time to create the database containers, which are running a Percona XtraDB Cluster (PXC). As PXC needs at least three nodes for quorum you are deploying three DB containers now, each one running on a different Docker host for better availability and performance. Click + Add Container on wordpress1 host and enter the following information:

• Name: pxc1
• Image: nixel/rancher-percona-xtradb-cluster:latest

Expand Advanced Options and follow these steps:

• Command section – Add an Environment Variable named PXC_BOOTSTRAP which value is YES
• Volumes section – Add this volume: /var/lib/mysql:/var/lib/mysql
• Networking section – Choose Managed Network on Docker0

• Security/Host section – Enable Give the container full access to the host checkbox

Show pxc1 logs and copy the values of PXC_NODES, PXC_SST_PASSWORD and PXC_ROOT_PASSWORD variables that you are needing now. You will also see messages informing that PXC cluster has been bootstraped.

Now add a new PXC container with this configuration

• Name: pxc2
• Image: nixel/rancher-percona-xtradb-cluster:latest

Expand Advanced Options and follow these steps:

• Command section – Add the following Environment Variables:
  • PXC_BOOTSTRAP which value is NO
  • PXC_NODES which value is pxc1 Rancher IP, in my case it is 10.42.154.159
  • PXC_SST_PASSWORD which value was in pxc1 log, in my case it is zMpUCmmyW7B9KUf7AWuo
  • PXC_ROOT_PASSWORD which value was in pxc1 log, in my case it is wT9IhUFCPMH8ghzl2ftt
• Volumes section – Add this volume: /var/lib/mysql:/var/lib/mysql
• Networking section – Choose Managed Network on Docker0
• Security/Host section – Enable Give the container full access to the host checkbox

Finally wait for pxc2 to start, and add the third PXC container with this configuration:

• Name: pxc3
• Image: nixel/rancher-percona-xtradb-cluster:latest

Expand Advanced Options and follow these steps:

• Command section – Add the following Environment Variables:
  • PXC_BOOTSTRAP which value is NO
  • PXC_NODES which value is a comma separated list of pxc1 and pxc2 containers IPs. In my case I’m typing this: 10.42.154.159,10.42.170.177
  • PXC_SST_PASSWORD which value was in pxc1 log, in my case it is zMpUCmmyW7B9KUf7AWuo
  • PXC_ROOT_PASSWORD which value was in pxc1 log, in my case it is wT9IhUFCPMH8ghzl2ftt
• Volumes section – Add this volume: /var/lib/mysql:/var/lib/mysql
• Networking section – Choose Managed Network on Docker0
• Security/Host section – Enable Give the container full access to the host checkbox

Wait for pxc3 to start. Then you can check how all MySQL processes are started on all PXC containers by showing their logs:

Deploying the wordpress and LB services

As I introduced before, we are launching one Rancher Service for wordpress and another one for Load Balancers. This will allow us to easily scale up wordpress as Rancher will automatically deploy the new containers and add them to the Load Balancers by only clicking one button.

All Rancher Services belong to a Rancher Environment. Environments are namespaces where services are deployed, so you can have different sets of services. To create the wordpress service go to Services section on Rancher UI and click + Add Environment button. Create a new environment named TEST.

The process of creating a service is the same as creating a new container. Just click + Add Service and configure it as follows:

• Name: wordpress
• Scale: create, for example, 10 containers
• Image: nixel/rancher-wordpress-ha:latest

Expand Advanced Options and follow these steps:

• Command section – Add the following Environment Variables:
  • GLUSTER_PEER which value is a comma separated list of gluster1 and gluster2 containers IPs. In my case it is 10.42.25.141,10.42.229.233
  • WORDPRESS_DB_HOSTS which value is a comma separated list of pxc1, pxc2 and pxc3 containers IPs, note that you need to specify DB port which is 3306 by default. In my case I’m typing this: 10.42.154.159:3306,10.42.170.177:3306,10.42.2.38:3306
  • WORDPRESS_DB_PASSWORD which value is the same as PXC_ROOT_PASSWORD obtained from pxc1 logs. In my case it is wT9IhUFCPMH8ghzl2ftt
• Networking section – Choose Managed Network on Docker0
• Security/Host section – Enable Give the container full access to the host checkbox

Note that we are not defining any volume because we are using GlusterFS storage, and no ports are exposed because Load Balancers are publishing our WordPress service for us.

Now click Start button on wordpress service and Rancher will automatically create and start 10 wordpress containers for you.

Now we are creating the wordpress Load Balancers. To do so, click + Add Load Balancer button and enter this configuration:

• Name: wordpress-lb
• Scale: create, for example, 3 Load Balancers. They are being deployed on all Docker hosts for best availability and performance.
• Target service: link it to wordpress service
• Listeners: Enter 80/http source port and map it to 80/http target port
• Health Check: select GET method and type /healthcheck.txt URI

After creating the LB service group you have to start it by clicking the Start button. At this point you have your HA wordpress environment up and running.

Go to Hosts section and you will see your wordpress and LB containers deployed. Note that each Docker host is running a LB container, so you can use DNS balancing to get better availability.

Completing wordpress installation

Now that all our containers are started we have to browse to WordPress site and complete the installation. First of all configure a DNS registry to resolve all your Docker hosts IPs so LBs will receive web traffic. In my case, for testing purposes, I’m adding the following to my /etc/hosts file (I’m simulating a LB failover later):

52.24.85.40 
#52.24.137.235 
#52.10.158.122 

Then browse to http:// (replace it with your current domain) and complete wordpress installation. Once finished you will see the wordpress Hello World post.

You can customize your wordpress installation now, but we are doing a more interesting thing: testing the scalability and HA capabilities.

Testing scalability

In case you need to handle more traffic and hence to scale up your environment you only have to click + Scale Up button on wordpress service group, and Rancher will automatically handle containers creation and will add them to the Load Balancers for you. I wish I would write more about this step but it’s that easy to scale up services now

To test how Rancher added wordpress11 container to the LBs you may want to stop all wordpress1-10 containers and check that your site is still online.

In case you need to scale down your infrastructure just delete some containers. In my case I’m deleting the wordpress11 container I have just created.

Testing HA capabilities

Our wordpress environment will be reliable if, at least, all these conditions are met:

1. There is 1 wordpress container running, and
2. there are 2 PXC containers running (because of PXC quorum), and
3. there is 1 gluster container running, and
4. there is 1 wordpress-lb agent running

To test this let’s simulate the worst case scenario:

• ensure that gluster1, pxc1, pxc2, wordpress10, and LB agent deployed on wordpress1 host are running
• stop gluster2, pxc3, wordpress1-9 containers, and LB agents deployed on wordpress2 and wordpress3 hosts.

Now browse to your blog and you will see that it’s still online. Login on wordpress so you will enter administration page.

Then do the following in order to simulate a failover on all services, from storage to LB:

1. Start gluster2, wait 15 seconds, and then stop gluster1 container
2. Start pxc3, wait 15 seconds, and then stop pxc1 container
3. Start wordpress1 and stop wordpress10 container.
4. Stop LB Agents on wordpress1 and wordpress2 hosts, and keep started LB agent running on wordpress3 host (change your hosts file if required)

Navigate to wordpress administration. It is not only reachable but you also are able to use the same session because you were not logged out, so no matter which container is covering your services because you won’t note any service disruption.

Conclusion

Rancher has added a new service feature to make it easier to distribute traffic across different Docker containers. Now you can scale up/down your environments from Rancher UI in just one step, and Rancher will automatically create new containers and add them to the appropriated Load Balancers.

This simplifies the way to create Highly Available, scalable, and distributable environments ready for production use. As an example we have deployed a WordPress installation with redundant services for storage, database, web application, and Load Balancers. We have simulated common production failover/degradation scenarios and proved that our setup was fault tolerant and therefore users wouldn’t notice any service disruption.

During our next Online Meetup, we’ll be providing a hands-on demonstration of how to build this type of application using Rancher and Docker Compose.  Please join us:

REGISTER NOW

Manel Martinez is a Linux systems engineer with experience in the design and management of scalable, distributable and highly available open source web infrastructures based on products like KVM, Docker, Apache, Nginx, Tomcat, Jboss, RabbitMQ, HAProxy, MySQL and XtraDB. He lives in spain, and you can find him on Twitter @manel_martinezg.

TwitterHacker NewsLinkedInGoogle+RedditFacebook