Bringing High Availability to OpenStack Keystone and Glance | www.hastexo.com

Bringing High Availability to OpenStack Keystone and Glance

Submitted by martin on Tue, 2012-04-03 15:15

• OpenStack

If you want to bring high availability to your OpenStack setup, then this is for you: hastexo has started to develop resource agents to integrate Keystone and Glance into the Pacemaker cluster manager.

What's this about?

OpenStack has become a bright star shining on the heaven of cloud computing. Over the last months, the developers did a magnificent job to get the next release, OpenStack Essex, ready for production. One of the things that haven't been that much in focus recently, however, was the subject of High Availability for OpenStack. Florian Haas did a nice writeup of the most interesting questions concerning this. After all, we've got to start somewhere, and that's what I just did. 

Today, i'm announcing the availability of a first public preview version of Pacemaker resource agents for OpenStack Identity (Keystone) and OpenStack ImageService (Glance). I wrote OCF resource agents for keystone, glance-register and glance-api. They allow system administrators to run Keystone and Glance as highly available services within the Pacemaker cluster manager. They can start, stop and monitor the status of a given instance of any of these services.

Why do I need it?

Pacemaker (in conjunction with Corosync) is the most advanced solution for clustering available for the Linux operating system these days. It can control the nodes of a multi-node cluster and make sure that an application is restarted on another cluster node if its original host goes down for whatever reason. It's highly flexible and allows for complex configuration. 

To be able to control the cluster services on a given cluster node, Pacemaker needs an interface to exactly that application. It doesn't call tools directly, but instead uses the so-called resource agents for this. Pacemaker calls such an agent with either the "start", the "stop" and the "monitor" argument and expects the agent to just start, stop or monitor the given application. 

The OCF agents I wrote do exactly this: They allow Pacemaker to start, stop and monitor Keystone and Glance.

There are upstart-jobs for this, what's wrong with them?

Absolutely nothing! The upstart jobs are great in doing what they are supposed to do. However, upstart itself is not cluster aware and there is no way for node a to know whether glance is actually running on node b. With Pacemaker and these OCF agents, this becomes possible.

It's correct to assume that on Ubuntu systems, one could even use Pacemaker's upstart interface to take care of this problem; however, Upstart is not available on all distributions and neither is Pacemaker's upstart interface. The OCF agents are distribution-independent and can be used on every distribution where Pacemaker is available. 

Last but not least, these agents provide for a way more extensive monitoring than the standard init scripts and upstart jobs: By providing Keystone login credentials, you can make "log into" the services in question and see whether they really work (that way, it's impossible that the actual process appears in the process table but the service as such is broken).

How do I use the agents?

The three agents are available from here:

keystone: https://github.com/madkiss/keystone/blob/ha/tools/ocf/keystone

glance-api: https://github.com/madkiss/glance/blob/ha/tools/ocf/glance-api

glance-registry: https://github.com/madkiss/glance/blob/ha/tools/ocf/glance-registry

Download them and be sure to store them in /usr/lib/ocf/resource.d/openstack. Create that directory prior to downloading the agents. Also make sure that the scripts are executable. And be sure to download the agents to all nodes in your cluster. 

Additionally, make sure that no system-controlled instances of these services are running. On Ubuntu systems, you can disable the automatic startup of keystone, glance-api and glance-registry by creating the files /etc/init/keystone.override, /etc/init/glance-api.override and /etc/init/glance-registry.override, and making sure the only content of these files is "manual".  Then, jump onto Pacemaker's CRM shell and set up your resources. This is an example setup:

primitive p_glance_api ocf:openstack:glance-api \
	params config="/etc/glance/glance-api.conf" \
	os_auth_url="http://localhost:5000/v2.0/" \
	os_password="hastexo" \
	os_tenant_name="admin" \
	os_username="admin" \
	user="glance" \
	client_binary="/usr/bin/glance" \
	op monitor interval="30s" timeout="30s"

primitive p_glance_registry ocf:openstack:glance-registry \
	params config="/etc/glance/glance-registry.conf" \
	os_auth_url="http://localhost:5000/v2.0/" \
	os_password="hastexo" \
	os_tenant_name="admin" \
	os_username="admin" \
	user="glance" \
	op monitor interval="30s" timeout="30s"

primitive p_keystone ocf:openstack:keystone \
	params config="/etc/keystone/keystone.conf" \
	os_auth_url="http://localhost:5000/v2.0/" \
	os_password="hastexo" \
	os_tenant_name="admin" \
	os_username="admin" \
	user="keystone" \
	client_binary="/usr/bin/keystone" \
	op monitor interval="30s" timeout="30s"

For more information on the agent's parameters, you can do

crm ra info ocf:openstack:agent

Replace "agent" with either "keystone", "glance-api" or "glance-registry".

Please note: In this example, we're using the admin credentials to connect to the services for monitoring. For security reasons, it's recommended that you create a separate user in Keystone for this and use that user's login credentials.

Feedback

I wholeheartedly invite everybody to share his opinion on these resource agents. Should you have suggestions on how to improve them - no matter whether it's just an idea or an actual patch - please don't hestitate to leave a comment behind on this page! I'd love to hear from you :-)