Corosync and Pacemaker: Best Practices for Deploying a High‑Availability (HA) Cluster

Let’s consider several possible solutions.

• Oracle Clusterware for Oracle Unbreakable Linux is fairly expensive; it really only makes sense if you’re working with Oracle products.
• Veritas InfoScale Availability is a commercial solution, formerly known as Veritas Cluster Server. A VCS Access license for a four‑node cluster runs about $20–30k.
• Red Hat Cluster Suite is open source, but using it properly requires wading through a lot of documentation, which will take a significant amount of time.

I want to show how to build application-level clustering with high availability quickly, on a tight budget, and without deep expertise in clustering. In my view, the best option is to use Corosync with Pacemaker. This combo is free, easy to learn, and fast to roll out.

Corosync is software that lets you build a unified cluster from multiple physical or virtual servers. Corosync monitors and disseminates the state of all members (nodes) in the cluster.

This product allows you to:

• Monitor application status
• Notify applications when the cluster’s active node changes
• Broadcast identical messages to processes on all nodes
• Provide access to a shared database for configuration and metrics
• Send notifications about changes made in the database

Pacemaker is a cluster resource manager. It lets you run and manage services and resources within a cluster of two or more nodes. In short, its key advantages are:

• Detects and remedies failures at the node and service level.
• Storage-agnostic: forget about shared storage like a bad dream.
• Resource-agnostic: if you can script it, you can cluster it.
• Supports STONITH (Shoot-The-Other-Node-In-The-Head): a failed node is fenced/isolated and won’t receive traffic until it reports healthy again.
• Supports quorum-based and resource-driven clusters of any size.
• Works with virtually any redundancy topology.
• Can automatically replicate its configuration to all cluster nodes—no manual edits on each node.
• Lets you define resource start order and colocation/compatibility on a single node.
• Supports advanced resource types: clones (one resource running on many nodes) and multi-state roles (master/replica, primary/secondary), which is relevant for DBMSs like MySQL, MariaDB, PostgreSQL, and Oracle.
• Provides a unified CRM cluster shell with scripting support.

The idea is to build the cluster with Corosync and use Pacemaker to monitor its state.

In theory

Let’s tackle the following tasks.

• Deploy a two-node high-availability cluster to handle client requests, using a shared virtual IP address and the Nginx web service.
• Test the failure of one node in the cluster and verify that resources fail over to the surviving node when the active node goes down.
• Failure detection on the active node: 30 seconds.

Practical implementation

It’s time to put the plan into action. In my example, I used Red Hat Enterprise Linux 7. You can use any other Linux distribution—the principles of building the cluster are the same.

First, install the packages required for the software to function correctly on both nodes (perform the installation on each node).

The next command also needs to be run with root privileges, but if you run it via sudo, the installer script won’t work correctly and the hacluster user won’t be created.

Verify the hacluster (Pacemaker) user and change its password:

Log in to the nodes as the hacluster user. If you see an access error to a node (Error: Unable to communicate with ha-node1), it most likely means the Linux account has restrictions that block the use of remote shells. You need to lift the SUDO restriction for the duration of the installation.

Once you see the following, you can move on:

Now, on both nodes, verify that corosync, pacemaker, and pcs are installed; enable them to start at boot, and start the Pacemaker configuration service.

Log in to the nodes as the hacluster user:

Creating a two-node cluster:

Enable and start all clusters on every node:

For a two-node setup, enable STONITH (fencing). It’s used to forcibly power off nodes that fail to cleanly stop their workloads. Configure the cluster to ignore quorum.

Check the status on both nodes ($ sudo pcs status) and see:

Add the virtual IP address and nginx as resources (hopefully you remember the 30‑second timeout), then query the cluster status.

Take the first node offline for a few minutes. Check the status of the second node to confirm the first is unavailable.

Once the first node is available, manually fail over the virtual IP and Nginx to it using the command

All that’s left is to check the cluster status and verify that the address is assigned to the first node.

You can view the list of all supported services with the command $ sudo crm_resource --list-agents ocf.

Nginx‑based clustering

Nginx must be installed and configured on both nodes. Connect to the cluster and create the following resource:

Here we define a service resource that is monitored every 30 seconds; if it doesn’t respond for one minute, the resource will fail over to another node.

Conclusion

We’ve built a simple yet highly fault-tolerant cluster for the application’s frontend. Remember: building a cluster is a creative endeavor, and it all comes down to your ingenuity and imagination. For a sysadmin, there’s no such thing as an impossible task!