Codership Tutorials on Galera Cluster

Table of Contents
Introduction & Installation.........................................................................................................................3
Getting Started with Galera Cluster.........................................................................................................3
Installing Galera Cluster...........................................................................................................................7
Galera Cluster for MySQL—Binary Installation............................................................................11
Galera Cluster for MySQL - Source Installation..................................................................................18
MariaDB Galera Cluster - Binary Installation.....................................................................................21
MariaDB Galera Cluster - Source Installation.....................................................................................25
Installing a Galera Cluster on AWS........................................................................................................30
System Configuration............................................................................................................................38
Configuring the InnoDB Buffer Pool¶...............................................................................................41
Replication Configuration......................................................................................................................43
Starting the Cluster................................................................................................................................45
Getting Started Galera with Docker.......................................................................................................49
Testing a Cluster..................................................................................................................................54
Introduction & Installation

Getting Started with Galera Cluster

Galera Cluster is a synchronous replication solution to improve availability and

performance of the MySQL database service. All nodes in Galera Cluster are identical
and fully representative of the cluster. They allow for unconstrained transparent client
access, operating as a single, distributed MySQL server. It provides,

 Transparent client connections, so it’s highly

compatible with existing applications;
 Synchronous data safety semantics—if a
client received confirmation, transactions will
be committed on every node; and
 Automatic write conflict detection and
resolution, so that nodes are always
consistent.

Galera Cluster is well suited for LAN, WAN, container and cloud environments. The
following tutorial articles provide you with the basics to setting up and deploying Galera
Cluster. Bear in mind before you get started that you need root access to at least three
Linux or FreeBSD hosts and their IP addresses.

Note

With the latest release Galerea Cluster begins the 4.x branch, introducing a number of
new features. For more information on these features, see What’s New.

How Galera Cluster Works

The primary focus is data consistency. The transactions are either applied on every node
or not all. So, the databases stay synchronized, provided that they were properly
configured and synchronized at the beginning.

The Galera Replication Plugin differs from the standard MySQL Replication by
addressing several issues, including multi-master write conflicts, replication lag and
slaves being out of sync with the master.
In a typical instance of a Galera Cluster, applications can write to any node in the cluster
and transaction commits, (RBR events), are then applied to all the servers, through
certification-based replication.

Certification-based replication is an alternative approach to synchronous database

replication, using group communication and transaction ordering techniques.

Note

For security and performance reasons, it’s recommended that you run Galera Cluster on
its own subnet.

Node Initialization
Individual nodes in Galera Cluster are MySQL, MariaDB or Percona XtraDB. But,
deploying a node is not exactly the same as the standard standalone instance of the
database server. You need to take a few additional steps in order to properly install and
configure the software. The software runs on any unix-like operating system. These
articles provide guides to installing and configuring nodes for Galera Cluster.

 Installation

Once you have your server hardware ready,

including at least three hosts running either
Linux or FreeBSD and their respective IP
 addresses. This article provides guides to
preparing the server and installing Galera
Cluster. When you install Galera Cluster, you
must choose between three implementations
available. For each implementations, you can
install the software using Debian- and RPM-
based binary packages or by building the
node from source.

o Galera Cluster for MySQL: The

reference implementation from
Codership, Oy. You can use either
the Install MySQL Binary
Installation or the Source Build
methods.
o Percona XtraDB Cluster: The
Percona alternative
implementation of Galera Cluster,
which uses XtraDB in place of
MySQL. You can use either the
Install XtraDB Binary Installation
or the Source Build methods.
o MariaDB Galera Cluster: The
MariaDB Ab alternative
implementation of Galera Cluster,
which uses MariaDB in place of
MySQL. You can use either the
Install MariaDB Binary Installation
or the Source Build.
 System Configuration

Before you can start the cluster, you need to

configure the individual nodes. This article
covers general parameters that you must set
in the my.cnf configuration file in order to use
Galera Cluster. It also provides a guide to
configuring swap space in order to protect
the node from crashing due to large write-
 sets requiring more memory than the server
has available.

 Replication Configuration

With the system-level configurations

complete, this article provides a guide to
configuring the database server to connect
and communicate with the cluster and
explains the syntax format used in cluster
addresses.

Cluster Initialization
With the software installed on the relevant servers in your your infrastructure, you can
now initialize Galera Cluster, by bootstrapping the Primary Component then starting all
the other nodes as you would any other database server instance. These tutorial articles
provide guides to starting the cluster, ways of testing that it’s operational and, when you
need to, how to restart the entire cluster.

 Starting the Cluster

In order to start Galera Cluster, you need to

bootstrap the Primary Component.
Afterwards, you can start nodes using the
same command as you would the standard
standalone MySQL, MariaDB or Percona
XtraDB database server. This article provides a
guide to initializing the cluster.

 Testing a Cluster

With your cluster online, you may want to test

out some of the features in order to ensure
it’s working properly and to better see how it
works in planning out your own deployment.
This tutorial provides a rough guide to testing
replication and similar cluster operations.

 Restarting the Cluster

On occasion, you may need to restart the

entire cluster, such as in the case of a power
 failure where every node is shut down and
there is no database server process left
running. This article provides guides to
finding your most advanced node and
restarting the Primary Component on that
node.

Installing Galera Cluster

Galera Cluster requires server hardware for a minimum of three nodes.

If your cluster runs on a single switch, use three nodes. If your cluster spans switches,
use three switches. If your cluster spans networks, use three networks. If your cluster
spans data centers, use three data centers. This ensures that the cluster can maintain a
Primary Component in the event of network outages.

Hardware Requirements

For server hardware, each node requires at a minimum the following components:

 1 GHz single core CPU;

 512 MB RAM; and
 100 Mbps network connectivity

Note

Galera Cluster may occasionally crash when run on limited hardware due to insufficient
memory. To prevent this, make sure that you have allocated a sufficient amount of swap
space. For more information on how to create swap space, see Configuring Swap Space.

Software Requirements

For software, each node in the cluster requires at a minimum the following:

 Linux or FreeBSD operating system installed;

 MySQL or MariaDB server with the wsrep API patch; and
 Galera Replication Plugin installed.

Note
Binary installation packages for Galera Cluster include the database server with the
wsrep API patch. When building from source, though, you must apply this patch
manually.

Preparing the Server

Before you begin the installation process, there are a few tasks that you need to do to
prepare the servers for Galera Cluster. You must perform the following steps on each
node in your cluster.

Disabling SELinux for mysqld¶

If SELinux (Security-Enhanced Linux) is enabled on the servers, it may block mysqld from
performing required operations. You must either disable SELinux for mysqld or configure
it to allow mysqld to run external programs and open listen sockets on unprivileged
ports—that is, operations that an unprivileged user may do.

To disable SELinux for mysqld, execute the following from the command-line:

# semanage permissive -a mysqld_t

This command switches SELinux into permissive mode when it registers activity from the
database server. While this is fine during the installation and configuration process, it is
not in general a good policy to disable security applications.

Rather than disable SELinux, so that your may use it along with Galera Cluster, you will
need to create an access policy. This will allow SELinux to understand and allow normal
operations from the database server. For information on how to create such an access
policy, see SELinux.

Note

For more information on writing SELinux policies, see SELinux and MySQL.
Firewall Configuration¶

Next, you will need to update the firewall settings on each node so that they may
communicate with the cluster. How you do this varies depending upon your distribution
and the particular firewall software that you use.
Note

If there is a NAT firewall between the nodes, you must configure it to allow for direct
connections between the nodes, such as through port forwarding.

As an example, to open ports between trusted hosts using iptables, you would execute
something like the following on each node:

# iptables --append INPUT --protocol tcp \

--source 64.57.102.34 --jump ACCEPT

# iptables --apend INPUT --protocol tcp \

--source 193.166.33.20 --jump ACCEPT

# iptables --append INPUT --protocol tcp \

--source 193.125.4.10 --jump ACCEPT

This causes packet filtering on the kernel to accept TCP connections between the given
IP addresses.

Warning

The IP addresses in the example are for demonstration purposes only. Use the real
values from your nodes and netmask in the iptables configuration for your cluster.

The updated packet filtering rules take effect immediately, but are not persistent. When
the server reboots, it reverts to default packet filtering rules, which do not include your
updates. To use these rules after rebooting, you need to save them as defaults.

For systems that use init, run the following command:

# service save iptables

For systems that use systemd, you need to save the current packet filtering rules to the
path that the iptables unit reads when it starts. This path can vary by distribution, but
you can normally find it in the /etc directory.

 /etc/sysconfig/iptables
 /etc/iptables/iptables.rules

When you find the relevant file, you can save the rules using the iptables-save
command, then redirecting the output to overwrite this file.
# iptables-save > /etc/sysconfig/iptables

When iptables starts it now reads the new defaults, with your updates to the firewall.

For more information on setting up the firewall for Galera Cluster and other programs
for configuring packet filtering in Linux and FreeBSD, see Firewall Settings.
Disabling AppArmor¶

By default, some servers—for instance, Ubuntu—include AppArmor, which may prevent

mysqld from opening additional ports or running scripts. You must disable AppArmor or
configure it to allow mysqld to run external programs and open listen sockets on
unprivileged ports.

To disable AppArmor, run the following commands:

$ sudo ln -s /etc/apparmor.d/usr.sbin.mysqld
/etc/apparmor.d/disable/usr.sbin.mysqld

You will then need to tell AppArmor to reload profile:

$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

In some cases you may also need to restart AppArmor. If your system uses init scripts,
run the following command:

$ sudo service apparmor restart

If instead, your system uses systemd, run the following command instead:

$ sudo systemctl restart apparmor

Installing Galera Cluster

There are two versions of Galera Cluster for MySQL: the original Codership reference
implementation and MariaDB Galera Cluster. For each database server, binary packages
are available for Debian- and RPM-based Linux distributions, or you can build them
from source.
Galera Cluster for MySQL¶
  Galera Cluster for MySQL—Binary Installation

 Galera Cluster for MySQL - Source Installation

MariaDB Galera Cluster¶

 MariaDB Galera Cluster - Binary Installation

 MariaDB Galera Cluster - Source Installation

Note

In the event that you build or install Galera Cluster over an existing standalone instance
of MySQL or MariaDB, there are some additional steps that you need to take in order to
update your system to the new database server. For more information, see Migration.

Galera Cluster for MySQL—Binary Installation

Galera Cluster for MySQL may be installed on Linux servers using binary packages. These
files can be downloaded directly from the Codership repository, or by way of a package
manager: apt-get, yum or zypper.

Enabling the Codership Repository

To install Galera Cluster for MySQL with a package manager, you first will have to enable
the Codership repository on the server. There are a few ways to do this, depending on
which Linux distribution and package manager you use. The sections below provide
details on how to use each of the three main supported package managers to install
Galera Cluster for MySQL.

Enabling the apt Repository

For Debian and Debian-based Linux distributions, the procedure for adding a repository
requires that you first install the Software Properties. The package names vary depending
on the distribution. For Debian, at the command-line, execute the following command:
apt-get install python-software-properties

For Ubuntu or a distribution derived from Ubuntu, you would execute instead the
following:

apt-get install software-properties-common

If your server uses a different Debian-based distribution, and neither of these

commands work on your system, try consulting your distribution’s package listings for
the appropriate package name.

Once you have the Software Properties installed, you can then enable the Codership
repository for your system. Start by adding the GnuPG key for the repository. This is
done by executing the following from the command-line:

apt-key adv --keyserver keyserver.ubuntu.com --recv BC19DDBA

Next, add the Codership repository to your sources list. Using a simple text editor, create
file called, galera.list in the /etc/apt/sources.list.d/ directory. Add these lines to that
file, with the necessary adjustments for the version used:

# Codership Repository (Galera Cluster for MySQL)

deb https://releases.galeracluster.com/mysql-wsrep-VERSION/DIST RELEASE main

deb https://releases.galeracluster.com/galera-3/DIST RELEASE main

In the example above, you would change the repository addresses. The VERSION should
be set to MySQL-wsrep version you want to install. For example, it might be something
like, 5.6. The DIST should be replaced with the name of the Linux distribution on the
server. This could be ubuntu. Last, replace RELEASE with the distribution release (e.g.,
wheezy).

If you don’t know which release you have installed on your server, you can generally find
this using the following command:

lsb_release -a

Version 4 of Galera was recently released. If you’d like to install it, the configuration lines
in galera.list should read similar to the following:
# Codership Repository (Galera Cluster for MySQL)

deb https://releases.galeracluster.com/galera-4/ubuntu focal main

deb https://releases.galeracluster.com/mysql-wsrep-8.0/ubuntu focal main

Again, you may have to adust the version and release numbers, depending on which
you want to install. Please note that this will require at least version 18.04 of Ubuntu.

To be assured the proper version is installed and updated, set which repository you
prefer to the Codership repository (this is not only recommended, it is required). To do
this, using a text editor, create a file called, galera.pref in the /etc/apt/preferences.d/
directory. The contents should look like the following:

# Prefer Codership repository

Package: *

Pin: origin releases.galeracluster.com

Pin-Priority: 1001

This is needed to make sure the patched versions are preferred. This might be
important, for instance, if a third-party program requires libmysqlclient20 and the OS-
version for the library is newer.

Finally, you should update the local cache of the repository. Do this by entering the
following from the command-line:

apt-get update

Once you’ve done all of these tasks, the packages in the Codership repository will be
available for installation. For information on installing them using apt-get, skip ahead
on this page to the section entitled, Installing Galera Cluster for MySQL.

Enabling the yum Repository

For rpm-based distributions of Linux (e.g., CentOS, Red Hat and Fedora), you will need
to enable the Codership repository. Using a simple text editor, create a file called,
galera.repo in the /etc/yum.repos.d/ directory. The contents of that file should look
something like the following:

[galera]
name = Galera

baseurl = https://releases.galeracluster.com/galera-3/DIST/RELEASE/ARCH

gpgkey = https://releases.galeracluster.com/GPG-KEY-galeracluster.com

gpgcheck = 1

[mysql-wsrep]

name = MySQL-wsrep

baseurl =
https://releases.galeracluster.com/mysql-wsrep-VERSION/DIST/RELEASE/ARCH

gpgkey = https://releases.galeracluster.com/GPG-KEY-galeracluster.com

gpgcheck = 1

In this sample repository configuration file, you would change the repository addresses
for the baseurl. The VERSION should be set to the whichever MySQL-wsrep version you
want (e.g., it might be 5.6). The DIST should be changed to the name of the Linux
distribution you’re using on your sever (e.g., centos or fedora). The RELEASE should be
replaced with the distribution’s release number. It might be 6 for CentOS, but 20 or 21
for Fedora. Last, the ARCH indicates the architecture of your hardware. This could be
changed to x86_64 for 64-bit systems.

After you’ve created, modified, and saved this repository file, you’ll be able to install the
packages from the Codership repository using yum. For an explanation on installing, skip
ahead on this page to the section entitled, Installing Galera Cluster for MySQL.

Enabling the zypper Repository

For servers with Linux distributions that use zypper for package management (e.g.,
openSUSE and SUSE Linux Enterprise Server), you can enable the Codership repository
by importing the GPG key and then creating a galera.repo file in the local directory.

To import the GPG key, execute the following from the command-line on your server:

rpm --import "https://releases.galeracluster.com/GPG-KEY-galeracluster.com"

Next, you’ll need to use a simple text editor to create a repository file called,
galera.repo in the local directory. The contents should look something like the
following:

[galera]

name = Galera

baseurl = https://releases.galeracluster.com/galera-3/DIST/RELEASE/ARCH

[MySQL-wsrep]

name = MySQL-wsrep

baseurl =
https://releases.galeracluster.com/mysql-wsrep-VERSION/DIST/RELEASE/ARCH

You’ll have to adjust these baseurl repository addresses. Replace where you see
VERSION above with the version of MySQL-wsrep you want to install (e.g., it might be
5.6). Next, substitute DIST in both addresses with the Linux distribution name (e.g.,
opensuse or sles). Where it says RELEASE, enter instead the distribution release number.
Last, change ARCH to the architecture of the server’s hardware. For example, you would
enter x86_64 for 64-bit systems. When you’re finished these modifications, save the file.

You’ll now need to add the Codership repository to zypper by executing the following
two lines from the command-line:

zypper addrepo galera.repo

zypper refresh

Once that’s all done, you’ll be able to install the packages from the Codership repository
using zypper. This is covered in the next section.

Installing Galera Cluster for MySQL

There are two packages involved in the installation of Galera Cluster for MySQL: the
MySQL database server, but one that has been built to include the wsrep API; and the
Galera Replication Plugin. The yum and zypper repositories include Galera Arbitrator with
the Galera Replication Plugin, but for Debian-based distributions using apt-get you’ll
need to include add it to your installation instruction.
So, for Debian-based distributions using the apt-get package manager, execute the
following from the command-line:

apt-get install galera-3 galera-arbitrator-3 mysql-wsrep-5.6

If you want to install version 4 of Galera Cluster, you would instead execute the
following from the command-line:

apt-get install galera-4 galera-arbitrator-4 mysql-wsrep-8.0

On servers using the yum package manager (i.e., Red Hat, Fedora and CentOS
distributions), you would instead execute this command:

yum install galera-3 mysql-wsrep-5.6

Please note that on CentOS 6 and 7, this command may generate a transaction check
error. For more information on that error and how to resolve it, see the section below on
MySQL Shared Compatibility Libraries.

If you want to install version 4 of Galera Cluster using yum, you would instead execute
the following from the command-line:

yum install galera-4 mysql-wsrep-8.0

Please note that on CentOS 8 and RedHat 8, you need to disable MySQL and MariaDB
modules before installing Galera Cluster from a repository under
https://releases.galeracluster.com/. In order to do this, execute the following from the
command-line:

dnf module disable mysql mariadb

If you’re using the zypper utility, which would be used with openSUSE and SUSE Linux
Enterprise Server, you would execute the following from the command-line:

zypper install galera-3 mysql-wsrep-5.6

Once you’ve executed the line appropriate to your distribution and package manager,
Galera Cluster for MySQL should be installed on your server. You will then have to
repeat this process for each node in your cluster, including enabling the repository files
mentioned earlier.

Incidentally, when deciding which packages from the Codership repository to install, the
package manager may elect to install a newer major verion of Galera Cluster, newer than
the one you intended to install. Before confirming the installation of packages, make
sure that the package manager is planning to install the Galera Cluster version you want.

If you installed Galera Cluster for MySQL over an existing stand-alone instance of
MySQL, there are some additional steps that you’ll need to take to update your system
to the new database server. For more information, see Migrating to Galera Cluster.

MySQL Shared Compatibility Libraries

When installing Galera Cluster for MySQL on CentOS, versions 6 and 7, you may
encounter a transaction check-error that blocks the installation. The error message may
look something like this:

Transaction Check Error:

file /usr/share/mysql/czech/errmsg.sys from install

mysql-wsrep-server-5.6-5.6.23-25.10.e16.x86_64 conflicts

with file from package mysql-libs-5.1.73-.3.e16_5.x86_64

This relates to a dependency problem between the version of the MySQL shared
compatibility libraries that CentOS uses, and the one that Galera Cluster requires. To
resolve this, you’ll have to upgrade, which can be done with the Codership repository
using yum.

There are two versions available for this package. Which version you’ll need will depend
on which version of the MySQL wsrep database server you want to install.

For CentOS 6, you would enter something like the following from the command-line:

yum upgrade -y mysql-wsrep-libs-compat-VERSION

You would, of course, replace VERSION here with 5.5 or 5.6, depending on the version of
MySQL you want to use. For CentOS 7, to install MySQL version 5.6, you would execute
the following from the command-line:

yum upgrade mysql-wsrep-shared-5.6

For CentOS 7, to install MySQL version 5.5, you will also need to disable the 5.6 upgrade.
To do this, enter the following from the command-line:

yum upgrade -y mysql-wsrep-shared-5.5 -x mysql-wsrep-shared-5.6

When yum finishes the upgrade, you can then install the MySQL wsrep database server
and the Galera Replication Plugin as described above.

Galera Cluster for MySQL - Source Installation

Galera Cluster for MySQL is the reference implementation from Codership Oy. Binary
installation packages are available for Debian- and RPM-based distributions of Linux. If
your Linux distribution is based upon a different package management system, if your
server uses a different unix-like operating system, such as Solaris or FreeBSD, you will
need to build Galera Cluster for MySQL from source.

Note

If you built Galera Cluster for MySQL over an existing standalone instance of MySQL,
there are some additional steps that you need to take in order to update your system to
the new database server. For more information, see Migrating to Galera Cluster.

Installing Build Dependencies

When building from source code, make cannot manage or install dependencies for either
Galera Cluster or the build process itself. You need to install these first. For Debian-
based systems, run the following command:

# apt-get build-dep mysql-server

For RPM-based distributions, instead run this command:

# yum-builddep MySQL-server

If neither command works on your system or that you use a different Linux distribution
or FreeBSD, the following packages are required:
  MySQL Database Server with wsrep API: Git, CMake, GCC and GCC-C++,
Automake, Autoconf, and Bison, as well as development releases of libaio and
ncurses.
 Galera Replication Plugin: SCons, as well as development releases of Boost, Check
and OpenSSL.

Check with the repositories for your distribution or system for the appropriate package
names to use during installation. Bear in mind that different systems may use different
names and that some may require additional packages to run. For instance, to run
CMake on Fedora you need both cmake and cmake-fedora.

Building Galera Cluster for MySQL

The source code for Galera Cluster for MySQL is available through GitHub. You can
download the source code from the website or directly using git. In order to build
Galera Cluster, you need to download both the database server with the wsrep API
patch and the Galera Replication Plugin.

To download the database server, complete the following steps:

1. Clone the Galera Cluster for MySQL database server source code.

2. # git clone https://github.com/codership/mysql-wsrep

3. Checkout the branch for the version that you want to use.

4. # git checkout 5.6

The main branches available for Galera Cluster for MySQL are:

o 5.6
o 5.5

You now have the source files for the MySQL database server, including the wsrep API
patch needed for it to function as a Galera Cluster node.

In addition to the database server, you need the wsrep Provider, also known as the
Galera Replication Plugin. In a separator directory, run the following command:

# cd ..

# git clone https://github.com/codership/galera.git

Once Git finishes downloading the source files, you can start building the database
server and the Galera Replication Plugin. The above procedures created two directories:
mysql-wsrep/ for the database server source and for the Galera source galera/

Building the Database Server

The database server for Galera Cluster is the same as that of the standard database
servers for standalone instances of MySQL, with the addition of a patch for the wsrep
API, which is packaged in the version downloaded from GitHub. You can enable the
patch through the wsrep API, requires that you enable it through the WITH_WSREP and
WITH_INNODB_DISALLOW_WRITES CMake configuration options.

To build the database server, cd into the mysql-wsrep/ directory and run the following
commands:

# cmake -DWITH_WSREP=ON -DWITH_INNODB_DISALLOW_WRITES=ON ./

# make

# make install

Building the wsrep Provider

The Galera Replication Plugin implements the wsrep API and operates as the wsrep
Provider for the database server. What it provides is a certification layer to prepare
write-sets and perform certification checks, a replication layer and a group
communication framework.

To build the Galera Replicator plugin, cd into the galera/ directory and run SCons:

# scons

This process creates the Galera Replication Plugin, (that is, the libgalera_smm.so file). In
your my.cnf configuration file, you need to define the path to this file for the
wsrep_provider parameter.

Note

For FreeBSD users, building the Galera Replicator Plugin from source raises certain Linux
compatibility issues. You can mitigate these by using the ports build at
/usr/ports/databases/galera.

Post-installation Configuration
After the build completes, there are some additional steps that you must take in order
to finish installing the database server on your system. This is over and beyond the
standard configurations listed in System Configuration and Replication Configuration.

Note

Unless you defined the CMAKE_INSTALL_PREFIX configuration variable when you ran
cmake above, by default the database server installed to the path /usr/local/mysql/. If
you chose a custom path, adjust the commands below to accommodate the change.

1. Create the user and group for the database server.

2. # groupadd mysql
3. # useradd -g mysql mysql

4. Install the database.

5. # cd /usr/local/mysql
6. # ./scripts/mysql_install_db --user=mysql

This installs the database in the working directory. That is, at

/usr/local/mysql/data/. If you would like to install it elsewhere or run it
from a different directory, specify the desired path with the --basedir and
--datadir options.

7. Change the user and group for the directory.

8. # chown -R mysql /usr/local/mysql

9. # chgrp -R mysql /usr/local/mysql

10. Create a system unit.

11. # cp /usr/local/mysql/supported-files/mysql.server \
12. /etc/init.d/mysql
13. # chmod +x /etc/init.d/mysql
14. # chkconfig --add mysql

This allows you to start Galera Cluster using the service command. It also
sets the database server to start during boot.

In addition to this procedure, bear in mind that any custom variables you enabled
during the build process, such as a nonstandard base or data directory, requires that
you add parameters to cover this in the configuration file, (that is, my.cnf).

Note
This tutorial omits MySQL authentication options for brevity.

MariaDB Galera Cluster - Binary Installation

MariaDB Galera Cluster is the MariaDB implementation of Galera Cluster. Binary

installation packages are available for Debian-based and RPM-based distributions of
Linux through the MariaDB repository (MariaDB Repository Generator).

Enabling the MariaDB Repository

In order to install MariaDB Galera Cluster through your package manager, you need to
enable the MariaDB repository on your server. There are two different ways to
accomplish this, depending on which Linux distribution you use.

Enabling the apt Repository

For Debian and Debian-based Linux distributions, the procedure for adding a repository
requires that you first install the software properties. The package names vary
depending on your distribution. For Debian, at the command-line execute the following:

# apt-get install python-software-properties

For Ubuntu or a distribution derived from Ubuntu, execute instead this command:

$ sudo apt-get install software-properties-common

If you’re use a different Debian-based distribution and neither of these lines above work,
consult your distribution’s package listings for the appropriate package name.

Once you have the software properties installed, you can enable the MariaDB repository
for your server.

First, add the GnuPG key for the MariaDB repository by executing the following from the
command-line:

# apt-key adv --recv-keys --keyserver \

keyserver.ubuntu.com 0xcbcb082a1bb943db

Next, add the MariaDB repository to your sources list. You can do this by entering
something like the following from the command-line:

# add-apt-repository 'deb
https://mirror.jmu.edu/pub/mariadb/repo/version/distro release main'

You wouldn’t enter exactly the line above. You’ll have to adjust the repository address:

 version indicates the version number of MariaDB that you want to use. (e.g., 5.6).
 distro is the name of the Linux distribution you’re using’ (e.g., ubuntu).
 release should be changed to your distribution release (e.g., wheezy).
If you don’t know which release is installed on your server, you can determine this by
using the entering the following from the command-line:

$ lsb_release -a

1. You should also update the local cache on the server. You can do this by entering
the following:

# apt-get update

For more information on the MariaDB repository, package names and available mirrors,
see the MariaDB Repository Generator.

Packages in the MariaDB repository are now available for installation through apt-get.

Enabling the yum Repository

For RPM-based distributions (e.g., CentOS, Red Hat and Fedora), you can enable the
MariaDB repository by creating a text file with .repo as the file extension to the
/etc/yum/repos.d/ directory.

Using a simple text editor, create a new .repo file containing something like the
following:

# MariaDB.repo

[mariadb]

name = MariaDB

baseurl = https://yum.mariadb.org/version/package

gpgkey = httpss://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck = 1

For the value of baseurl, you’ll have to adjust the web address:

 version should be changed to the version of MariaDB you want to use (e.g., 5.6).
  package will have to be changed to the package name for your operating system
distribution, release and architecture. For example, rhel6-amd64 would reference
packages for a Red Hat Enterprise Linux 6 server running on 64-bit hardware.

For more information on the repository, package names or available mirrors, see the
MariaDB Repository Generator. It will generate the actual text you will need to put in
your repository configuration file. In fact, by clicking through the choices presented, you
can just copy the results and paste them into your configuration file without any
modification.

Installing MariaDB Galera Cluster

There are three packages involved in the installation of MariaDB Galera Cluster: the
MariaDB database client, a command-line tool for accessing the database; the MariaDB
database server, built to include the wsrep API patch; and the Galera Replication Plugin.

For Debian-based distributions, from the command-line run the following command:

# apt-get install mariadb-client \

mariadb-galera-server \

galera

For RPM-based distributions, execute instead from the command line the following:

# yum install MariaDB-client \

MariaDB-Galera-server \

galera

Once you’ve done this, MariaDB Galera Cluster will be installed on your server. You’ll
need to repeat this process for each node in your cluster.

Note

If you installed MariaDB Galera Cluster over an existing stand-alone instance of

MariaDB, there are some additional steps that you’ll need to take to update your system
to the new database server. For more information on this, see Migrating to Galera
Cluster.
MariaDB Galera Cluster - Source Installation

MariaDB Galera Cluster is the MariaDB implementation of Galera Cluster for MySQL.
Binary installation packages are available for Debian- and RPM-based distributions of
Linux. If your Linux distribution is based on a different package management system, or
if it runs on a different unix-like operating system where binary installation packages are
not available, such as Solaris or FreeBSD, you will need to build MariaDB Galera Cluster
from source.

Note

If you built MariaDB Galera Cluster over an existing standalone instance of MariaDB,
there are some additional steps that you need to take in order to update your system to
the new database server. For more information, see Migration.

Preparing the Server

When building from source code, make cannot manage or install dependencies for either
Galera Cluster or the build process itself. You need to install these packages first.

 For Debian-based distributions of Linux, if MariaDB is available in your

repositories, you can run the following command:

 # apt-get build-dep mariadb-server

 For RPM-based distributions, instead run this command:

 # yum-builddep MariaDB-server

In the event that neither command works for your system or that you use a different
Linux distribution or FreeBSD, the following packages are required:

 MariaDB Database Server with wsrep API: Git, CMake, GCC and GCC-C++,
Automake, Autoconf, and Bison, as well as development releases of libaio and
ncurses.
 Galera Replication Plugin: SCons, as well as development releases of Boost, Check
and OpenSSL.

Check with the repositories for your distribution or system for the appropriate package
names to use during installation. Bear in mind that different systems may use different
names and that some may require additional packages to run. For instance, to run
CMake on Fedora you need both cmake and cmake-fedora.
Building MariaDB Galera Cluster
The source code for MariaDB Galera Cluster is available through GitHub. Using Git you
can download the source code to build MariaDB and the Galera Replicator Plugin locally
on your system.

1. Clone the MariaDB database server repository.

2. # git clone https://github.com/mariadb/server

3. Checkout the branch for the version that you want to use.

4. # git checkout 10.0-galera

The main branches available for MariaDB Galera Cluster are:

o 10.1
o 10.0-galera
o 5.5-galera

Starting with version 10.1, MariaDB includes the wsrep API for Galera
Cluster by default.

Warning

MariaDB version 10.1 is still in beta.

You now have the source files for the MariaDB database server with the wsrep API
needed to function as a Galera Cluster node.

In addition to the database server, you also need the wsrep Provider, also known as the
Galera Replicator Plugin. In a separate directory run the following command:

# cd ..

# git clone https://github.com/codership/galera.git

Once Git finishes downloading the source files, you can start building the database
server and the Galera Replicator Plugin. You now have the source files for the database
server in a server/ directory and the Galera source files in galera/.

Building the Database Server

The database server for Galera Cluster is the same as that of the standard database
servers for standalone instances of MariaDB, with the addition of a patch for the wsrep
API, which is packaged in the version downloaded from GitHub. You can enable the
patch through the WITH_WSREP and WITH_INNODB_DISALLOW_WRITES CMake configuration
options.

To build the database server, cd into the server/ directory and run the following
commands:

# cmake -DWITH_WSREP=ON -DWITH_INNODB_DISALLOW_WRITES=ON ./

# make

# make install

Note

In addition to compiling through cmake and make, there are also a number of build
scripts in the BUILD/ directory, which you may find more convenient to use. For example,

# ./BUILD/compile-pentium64-wsrep

This has the same effect as running the above commands with various build options
pre-configured. There are several build scripts available in the directory, select the one
that best suits your needs.

Building the wsrep Provider

The Galera Replication Plugin implements the wsrep API and operates as the wsrep
Provider for the database server. What it provides is a certification layer to prepare
write-sets and perform certification checks, a replication layer and a group
communication framework.

To build the Galera Replication Plugin, cd into the galera/ directory and run SCons.

# scons

This process creates the Galera Replication Pluigin, (that is, the libgalera_smm.so file). In
your my.cnf configuration file, you need to define the path to this file for the
wsrep_provider parameter.

Note
For FreeBSD users, building the Galera Replication Plugin from source raises certain
issues due to Linux dependencies. You can mitgate these by using the ports build
available at /usr/ports/databases/galera or by installing the binary package:

# pkg install galera

Post-installation Configuration
After the build completes, there are some additional steps that you must take in order
to finish installing the database server on your system. This is over and beyond the
standard configuration process listed in System Configuration and Replication
Configuration.

Note

Unless you defined the CMAKE_INSTALL_PREFIX configuration variable when you ran
cmake above, by default the database is installed to the path /usr/local/mysql/. If you
chose a custom path, adjust the commands below to accommodate the change.

1. Create the user and group for the database server.

2. # groupadd mysql
3. # useradd -g mysql mysql

4. Install the database.

5. # cd /usr/local/mysql
6. # ./scripts/mysql_install_db --user=mysql

This installs the database in the working directory, (that is, at

/usr/local/mysql/data). If you would like to install it elsewhere or run the
script from a different directory, specify the desired paths with the --
basedir and --datadir options.

7. Change the user and group permissions for the base directory.

8. # chown -R mysql /usr/local/mysql

9. # chgrp -R mysql /usr/local/mysql

10. Create a system unit for the database server.

11. # cp /usr/local/mysql/supported-files/mysql.server \
12. /etc/init.d/mysql
 13. # chmod +x /etc/init.d/mysql
14. # chkconfig --add mysql

This allows you to start Galera Cluster using the service command. It also
sets the database server to start during boot.

In addition to this procedure, bear in mind that any further customization variables you
enabled during the build process, such as a nonstandard base or data directory, may
require you to define additional parameters in the configuration file, (that is, my.cnf).

Installing a Galera Cluster on AWS

Amazon Web Services (AWS) is one of the largest providers of virtual servers and other
web services. Through an account with AWS, using their EC2 (Elastic Compute Cloud)
platform, you can create virtual servers, known as instances, and choose the operating
system distribution. From there you can log remotely into the instance like you would a
physical server, and install MySQL or MariaDB and Galera software. This means that you
can create multiple instances to form a Galera Cluster.

This tutorial will explain the steps to creating instances on AWS, and how to install and
configure the database and Galera software on each node. It’ll end by showing you how
to start the cluster.

We’re assuming you already have an account with AWS and know the basics of how to
use the EC2 platform to create an instance and related aspects. We’ll focus instead on
aspects relevant to forming a Galera Cluster.

Encryption Key
For this article, we’ll create three nodes to be used in the cluster. To access these nodes,
you’ll need an encryption key. You can use an existing key, but we recommend creating
a new one specifically for the Galera. There are many tools to do so, such as ssh-keygen.
See Generating an SSH Key for instructions on how to do this.

Once you have your encryption key, log into AWS and navigate to the EC2 console.
Under Security in the left margin, click on the Key Pairs link. Click on the white Import
Key Pair button. You can click on Choose File to load a .pem file from your computer, or
you can copy and paste the raw text from your key into the large box labeled, Public Key
Contents. Give the key a name like, galera-cluster. When finished, click Import.
Creating AWS Instances
With an encryption key in place, you’re ready to create three instances on AWS. You
could create more, but we’ll do three, which is the minimum recommended for a healthy
cluster. In the EC2 console, click on Instances in the left margin. That will show your
existing instances in the main pane. Now click on the large blue button at the top
labeled, Launch Instances.

The first choice you’ll have to make when creating a new instance is the operating
system distribution. For the examples in this article, we chose CentOS 7 (x86_64) - with
Updates HVM. This is a 64-bit operating system.

The second step is to choose an instance type. Because we’re using this cluster as a
training tool, we chose t2.micro since it’s Free Tier Eligible. This allows us to use it for a
year without having to pay anything.

Next, you’ll be asked configure the details of the instance. The first box asks how many
instances you want to launch: enter 3. You can leave all of the other settings at their
default values.

Adding storage is next. If you chose the free tier as did we, you’ll have to accept the
default of 8 gigabytes of storage. You may need more if you’re creating a cluster for
production use. For our training purposes, this is plenty. The next screen is about adding
tags, you can click past that.

Then you’ll be asked to configure a security group. This is basically AWS’s firewall. Even
if you already have one for your existing instances, create a new one just for Galera.
Name it something easy like, galera-servers. You’ll be able to give several inbound traffic
rules related to the three nodes we’ll create and the ports they will need. Until we create
the three instances, though, we won’t know their IP addresses. So, for now, just add a
rule that allows you to log into the nodes. To do this, select SSH type. For the source,
click on the pull-down menu and choose My IP—or choose Custom and type in the IP
address from which you’ll log in (e.g., 12.127.17.72/32).

With that all done, click on the blue button at the bottom right to review and launch the
instance. It will show you the choices you made. If everything is fine, click Launch.
A dialog box will then appear asking you to provide an encryption key. Click Choose an
Existing Key Pair and select the one you created for Galera. Then read and click the
check-box for the warning and then click Launch Instance. That’ll begin the process of
starting the instance.

Now, click on the Instances link in the margin again to see state of the instances. It’ll
take a little while before the new instances are running. You may want to label each
instance something like, galera - node 1.

Installing Software on Nodes

When all three nodes are running, you’re ready to install and configure the database
and Galera software. Check the details of each instance to get the external IP address.
Then use ssh to log into each node, using your the encryption key.

It’s best to keep all three nodes the same. So update the software that’s already installed
on each node with a command-line package-management utility. Since we installed
CentOS on our nodes, we used yum on each like so:

sudo yum -y update

We also installed rsync, which Galera will use to synchronize nodes when they first join
the cluster, or recovering nodes that have been offline for a while. We also installed
firewalld, which provides a firewall.

sudo yum -y install rync firewalld

When you’ve finished updating and installing basic software on all three nodes, you can
then proceed to installing the database and Galera software. For our purposes, we chose
to install MariaDB. If you’d prefer to use MySQL, see the article on Installing MySQL and
Galera Cluster.
To install MariaDB, we first used the MariaDB repository (MariaDB Repository
Generator). We chose the CentOS distribution, the CentOS 7 (x86_64) release to match
what we’ve installed on our nodes. We then chose the MariaDB version 10.4. The
repository tool gave us the lines we needed to copy into a new repo file,
/etc/yum.repos.d/mariadb.repo.

Using yum again, we executed the following from the command-line on each node:

sudo yum install MariaDB-server MariaDB-client

That will install everything we need, including the Galera software since it’s shipped with
MariaDB as of version 10.4. When you’re finished doing this one each node, you’ll be
ready to configure the MariaDB and Galera.

Configuring Software on Nodes

Before starting MariaDB on each node, you need to configure it. To do this, edit the
my.cnf file—or /etc/my.cnf.d/server.cnf as we did. Add the following lines under the
[mysqld] heading:

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

bind-address=0.0.0.0

user=mysql
default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=128M

binlog_format=ROW

log-error=/var/log/mysqld.log

There are some basic parameters related to MariaDB and using the InnoDB storage
engine—which is the only one that Galera uses. For parameters particular to Galera, add
a [galera] section to the same configuration file:

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

wsrep_node_name='galera1'

wsrep_node_address="172.31.19.208"

wsrep_cluster_name='galera-training'

wsrep_cluster_address="gcomm://172.31.19.208,172.31.26.197,172.31.15.54"

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_slave_threads=4

wsrep_sst_method=rsync

There are several parameters here, but we’ll talk about just a few of them. First, wsrep_on
enables Galera. The file path for the value of wsrep_provider may have to be adjusted
to your server.
The wsrep_node_name needs to be unique for each node (e.g., galera1, galera2, and
galera3). The wsrep_node_address is the IP address for the node. For AWS, use the
internal IP addresses, not the external ones. You’ll have to adjust this for each node.
Make sure the wsrep_node_name and the wsrep_node_address correspond to the names
you gave for each node in AWS. It will make it easier later to shut down a node for
maintenance.

The wsrep_cluster_name is a name you’ll assign for the cluster. It has to be the same on
all three nodes. The wsrep_cluster_address contains the IP addresses of all of the
nodes in the cluster, including the node on which it’s entered.

That’s all that we need to do to configure MariaDB and Galera. However, don’t start
MariaDB, yet. You first need to configure security on each node and on AWS.

Security Settings on Nodes

Security is important, but it can be the biggest obstacle for creating a new Galera
Cluster. You have to open the lines of communications between the nodes. The two
main factors will be SELinux and the firewall. You could disable both, but we recommend
using them and enabling what you need.

Galera Cluster uses a few TCP ports: 3306 (MariaDB’s default), 4444, 4567, and 4568—
and 4567 for UDP. For SELinux, you will need to open these ports by executing the
following from the command-line on each node:

semanage port -a -t mysqld_port_t -p tcp 3306

semanage port -a -t mysqld_port_t -p tcp 4444

semanage port -a -t mysqld_port_t -p tcp 4567

semanage port -a -t mysqld_port_t -p udp 4567

semanage port -a -t mysqld_port_t -p tcp 4568

semanage permissive -a mysqld_t

As for the firewall, you’ll have to first enable and start it by entering the first two lines
below. The six lines that follow those two will open the needed ports. The last line
reloads the firewalld daemon.

systemctl enable firewalld

systemctl start firewalld

firewall-cmd --zone=public --add-service=mysql --permanent

firewall-cmd --zone=public --add-port=3306/tcp --permanent

firewall-cmd --zone=public --add-port=4444/tcp --permanent

firewall-cmd --zone=public --add-port=4567/tcp --permanent

firewall-cmd --zone=public --add-port=4567/udp --permanent

firewall-cmd --zone=public --add-port=4568/tcp --permanent

firewall-cmd --reload

After these lines have been executed on each node, they’re now ready to interact with
each other. You just need to go back to AWS and add some entries to the security
group you created earlier so that AWS would interfere.

Security Settings on AWS

From the AWS EC2 console, under Security in the left margin, click on Security Groups
and then select the one you created for your Galera cluster. Under the Actions pull-down
menu, select Edit Inbound Rules. This will open a large dialog box like the one in the
screenshot below.
For your first entry, click Add Rule and select the type, MySQL/Aurora and then enter the
internal IP address for the first node (e.g., 172.31.19.208/32). Add a description to keep
your sanity (e.g., Galera - Node 1). Next, add another rule, but this time select the type,
Custom TCP Rule. Enter in the port field, 4444. Then enter the internal IP address and a
description of the first node. Now add another custom TCP entry the same way, but for
the port, enter 4567 - 4568. Last, add a custom UDP entry for port 4567.

Repeat these four entries for each node, adjusting the IP addresses and descriptions for
each. When you’re finished, click on Save.

Now that you’ve authorized each Galera node to communicate with the others on the
required ports, you’re ready to start Galera

Starting Galera
When first starting a new Galera cluster, you’ll have to start one node first. The problem
is that if there are no other nodes in the cluster, it will fail—unless you start MariaDB on
the first node telling it that it’s the first. To do this, you would use the --wsrep-new-
cluster option.

To make it easy, though, in version 4 of Galera, you can use the galera_new_cluster
script. Just execute it from the command-line on the first node, only. Incidentally, if you
ever shut-down all nodes in the cluster and want to use galera_new_cluster again, you
may have to edit /var/lib/mysql/grastate.dat and change the safe_to_bootstrap
variable in that file from 0 to 1.

Running galera_new_cluster on the first node will start MariaDB and Galera, but only
on that one node. On the other nodes, execute the following from the command-line:

systemctl start mariadb

Wait until the second node has started before starting the third. It doesn’t matter, but
it’s better to keep the clutter to a minimum when learning. Once MariaDB has started on
each node, enter the first line below from the command-line on one of the nodes.
Incidentally, there’s no password at this point, so just hit enter when asked.

mysql -p -u root -e "SHOW STATUS LIKE 'wsrep_cluster_size'"

Enter password:

+--------------------+-------+

| Variable_name | Value |

+--------------------+-------+

| wsrep_cluster_size | 3 |

+--------------------+-------+

You can see from the results here that there are three nodes in the cluster. That’s what
we want. You’ve successfully installed a Galera cluster using AWS instances. You can
start entering data and using the cluster.

If you first want to play with the cluster a bit, log into all three nodes with the mysql
client. Create a database on one node and check the other two nodes to see that it’s
there. Then create a table on another node and insert a row of data into that table on
the third node. Then execute a SELECT statement on each node to see that it’s
replicating the data across the cluster.

System Configuration
After you finish installing Galera Cluster on your server, you’re ready to configure the
database itself to serve as a node in a cluster. To do this, you’ll need to edit the MySQL
configuration file.

Using a text editor, edit the /etc/my.cnf file. You’ll need to include entries like the ones
shown in this sample excerpt:

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

binlog_format=ROW

bind-address=0.0.0.0

default_storage_engine=innodb

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=122M

wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_cluster_name="example_cluster"

wsrep_cluster_address="gcomm://IP.node1,IP.node2,IP.node3"

wsrep_sst_method=rsync

[mysql_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid
Depending on your system and the location of your installation of MySQL or MariaDB,
you will need to adjust the valuables for variables (e.g., the path to the data directory).

Configuring the Database Server

In addition to settings for the system, there are other basic configurations that you will
need to set in the /etc/my.cnf file. Make these changes before starting the database
server.

First, make sure that mysqld is not bound to 127.0.0.1. This is the IP address for
localhost. If the bind-address variable is in the file, comment it out by adding a hash
sign (i.e., #) at the start of the line:

# bind-address = 127.0.0.1

Next, ensure the configuration file includes the conf.d/ by adding a line with !
includedir at the start, followed by the file path:

!includedir /etc/mysql/conf.d/

Now, set the binary log format to use row-level replication, as opposed to statement-
level replication. You’d do this by adding the following line:

binlog_format=ROW

Don’t change this value later as it affects performance and consistency. The binary log
can only use row-level replication for Galera Cluster.

Galera Cluster will not work with MyISAM or other non-transactional storage engines.
So, make sure the default storage engine is InnoDB using the default_storage_engine
variable like so:

default_storage_engine=InnoDB

Next, ensure the InnoDB locking mode for generating auto-increment values is set to
interleaved lock mode. This is designated by a value of 2 for the appropriate variable:

innodb_autoinc_lock_mode=2
Don’t change this value afterwards. Other modes may cause INSERT statements to fail
on tables with AUTO_INCREMENT columns.

Warning

When innodb_autoinc_lock_mode is set to traditional lock mode (i.e., a value of 0) or to

consecutive lock mode (i.e., a value of 1) it can cause unresolved deadlocks and make
the system unresponsive in Galera Cluster.

After all of that, make sure the InnoDB log buffer is written to file once per second,
rather than on each commit, to improve performance. To do this, set the
innodb_flush_log_at_trx_commit variable to 0 like so;

innodb_flush_log_at_trx_commit=0

Warning

Although setting innodb_flush_log_at_trx_commit to a value of 0 or 2 improves

performance, it also introduces potential problems. Operating system crashes or power
outages can erase the last second of transaction. Although normally you can recover
this data from another node, it can still be lost entirely in the event that the cluster goes
down at the same time.

After you make all of these changes and additions to the configuration file, you’re ready
to configure the database privileges.

Configuring the InnoDB Buffer Pool¶

The InnoDB storage engine uses its own memory buffer to cache data and for indexes of
tables. You can configure this memory buffer through the innodb_buffer_pool_size
parameter. The default value is 128 MB. To compensate for the increased memory usage
of Galera Cluster over a standalone MySQL database server, you should scale your usual
value back by five percent.

innodb_buffer_pool_size=122M

Configuring Swap Space

Memory requirements for Galera Cluster are difficult to predict with any precision. The
particular amount of memory it uses can vary significantly, depending upon the load the
given node receives. In the event that Galera Cluster attempts to use more memory than
the node has available, the mysqld instance will crash.
The way to protect a node from such crashes is to ensure that there is sufficient swap
space available on the server. This can be either in the form of a swap partition or swap
files. To check the available swap space, execute the following from the command-line:

# swapon --summary

Filename Type Size Used Priority

/dev/sda2 partition 3369980 0 -1

/swap/swap1 file 524284 0 -2

/swap/swap2 file 524284 0 -3

If swap is not configured, nothing will be returned from this command. If your system
doesn’t have swap space available or if the allotted space is insufficient, you can fix this
by creating swap files.

First, create an empty file on your disk, set the file size to whatever size you require. You
can do this with the fallocate tool like so:

fallocate -l 512M /swapfile

Alternatively, you can manage the same using dd utility like this:

dd if=/dev/zero of=/swapfile bs=1M count=512

Be sure to secure the swap file by changing the permissions on the filesystem with chmod
like this:

chmod 600 /swapfile

$ ls -a / | grep swapfile

-rw------- 1 root root 536870912 Feb 12 23:55 swapfile

This sets the file permissions so that only the root user can read and write to the file. No
other user or group member can access it. Using the ls command command above
shows the results.

Now you’re read to format the swap file. You can do this with the mkswap utility. You’ll
then need to active the swap file.

mkswap /swapfile

swapon /swapfile

Using a text editor, update the /etc/fstab file to include the swap file by adding the
following line to the bottom:

/swapfile none swap defaults 0 0

After you save the /etc/fstab file, you run swapon again to see the results:

swapon --summary

Filename Type Size Used Priority

/swapfile file 524284 0 -1

Replication Configuration

In addition to the configuration for the database server, there are some specific options
that you need to set to enable write-set replication. You must apply these changes to
the configuration file (i.e., my.cnf) for each node in the cluster.

 wsrep_cluster_name: Use this parameter to set

the logical name for the cluster. You must use
the same name for each node in the cluster.
The connection will fail on nodes that have
different values for this parameter.

 wsrep_cluster_address: Use this parameter to

define the IP addresses for the cluster in a
comma-separated list.
 Note

There are additional schemata and options

available through this parameter. For more
information on the syntax, see Cluster
Addresses below.

 wsrep_node_name: Use this parameter to

define the logical name for the individual
node—for convenience.

 wsrep_node_address: Use this parameter to

set explicitly the IP address for the individual
node. It’s used when auto-guessing doesn’t
produce desirable results.

[mysqld]
wsrep_cluster_name=MyCluster
wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
wsrep_node_name=MyNode1
wsrep_node_address="192.168.0.1"

Backend Schema
There are two backend schemata available with Galera Cluster.

 dummy: This provides a pass-through back-end

for testing and profiling purposes. It doesn’t
connect to other nodes and will ignore any
values given to it.
 gcomm: This provides the group
communications back-end for use in
production. It accepts an address and has
several settings that may be enabled through
the option list, or by using the
wsrep_provider_options parameter.

Cluster Addresses
For the cluster address section, you have to provide a comma-separate list of IP
addresses for all of the nodes in the cluster. You would do this using the
wsrep_cluster_address parameter. Cluster addresses are listed in the configuration file
using a particular syntax, like so:

<backend schema>://<cluster address>[?<option1>=<value1>[&<option2>=<value2>]]

Below is an example of how this line from the configuration file might look:

wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"

Here, the backend schema is gcomm. The cluster addresses (i.e., 192.168.0.1, etc.) are
listed next, separted by commas. You can add options after that, within the quotes. You
would start with a question mark, followed by each option setting. Option key/value
pairs are separated by an ampersand. This is covered in the Options section below.

The IP addresses given in the configuration file should include any current members of
the cluster. The list may also include the IP addresses of any possible cluster members.
Members can belong to no more than one Primary Component;

If you start a node without proving an IP address for this parameter, the node will
assume that it’s the first node of a new cluster. It will initialize the cluster as though you
launched mysqld with the --wsrep-new-cluster option.

Options
When setting the IP address in the configuration file using the wsrep_cluster_address
parameter, you can also set some options. You can set backend parameters, such as the
listen address and timeout values.

Note

The wsrep_cluster_address options list is not durable. The node must resubmit the
options on each connection to a cluster. To make these options durable, set them in the
configuration file using the wsrep_provider_options parameter.

The options set in the URL take precedent over parameters set elsewhere. Parameters
you set through the options list are prefixed by evs (i.e., Extended Virtual Synchrony), pc
(i.e., Primary Component) and gmcast.

For more information on the available parameters, see Galera Parameters.

When listing options, start with a question mark after the IP address list. Then provide
the options in a key=value format. Key/value pairs must be separated by an ampersand.
Below is an example of how this might look:
wsrep_cluster_address="gcomm://192.168.0.1, 192.168.0.2, 192.168.0.3 ?
gmcast.segment=0 & evs.max_install_timeouts=1"

In this example, the segment option for gcomm and the max_install_timeouts option for
evs are set.

Incidentally, if the listen address and port are not set in the parameter list, gcomm will
listen on all interfaces. The listen port will be taken from the cluster address. If it’s not
specified in the cluster address, the default port is 4567.

Starting the Cluster

After you finish installing MySQL (or MariaDB or Percona XtraDB Cluster to Galera
Cluster) and Galera and have added the necessary settings for the configuration file
needed for Galera Cluster, the next steps are to start the nodes that will form the cluster.
To do this, you will need to start the mysqld daemon on one node, using the --wsrep-
new-cluster option. This initializes the new Primary Component for the cluster. Each
node you start after that will connect to the component and begin replication.

Before you attempt to initialize the cluster, there are a few things you should verify are
in place on each node and related services:

 At least three servers with the same version of

MySQL, MariaDB, or Percona XtraDB Cluster
installed on each;

 If you’re using firewalls, make sure the ports

4444, 4567, and 4568 for TCP traffic, and 4567
for UPD traffic are open between the hosts;

 SELinux and AppArmor, whichever your

system uses or both, has to be set to allow
access to mysqld; and,

 Set the parameter for wsrep_provider to the

location of libgalera_smm.so. That line in the
configuration file might look like this:

 wsrep_provider=/usr/lib64/
libgalera_smm.so
Once you have at least three hosts ready, you can initialize the cluster.

Note

When migrating from an existing, stand-alone instance of MySQL, MariaDB or Percona

XtraDB Cluster to Galera Cluster, there will be some additional steps that you must take.
For more information on what you need to do, see Migration.

Starting the First Node

By default, a node don’t start as part of the Primary Component. Instead, it assumes that
the Primary Component is already running and it is merely joining an existing cluster.
For each node it encounters in the cluster, it checks whether or not it’s a part of the
Primary Component. When it finds the Primary Component, it requests a state transfer
to bring its database into sync with the cluster. If it can’t find the Primary Component, it
will remains in a non-operational state.

The problem is that there is no Primary Component when a cluster starts, when the first
node is initiated. Therefore, you need explicitly to tell that first node to do so with the --
wsrep-new-cluster argument. Althought this initiate node is said to be the first node, it
can fall behind and leave the cluster without necessarily affecting the Primary
Component.

Note

When you start a new cluster, any node can serve as the first node, since all the
databases are empty. When you migrate from MySQL to Galera Cluster, use the original
master node as the first node. When restarting the cluster, use the most advanced node.
For more information, see Migration and Quorum Reset.

To start the first node–which should have MySQL, MariaDB or Percona XtraDB Cluster,
and Galera installed–you’ll have to launch the database server on it with the --wsrep-
new-cluster option. There are different ways to do this, depending on the operating
system. For systems that use init, execute the following from the command-line:

On Galera:

mysqld_bootstrap --wsrep-new-cluster

On MariaDB:

mysqld --wsrep-new-cluster
On Percona XtraDB Cluster

systemctl start [email protected]

Warning

Use the --wsrep-new-cluster argument only when initializing the Primary Component.
Don’t use it to connect a new node to an existing cluster.

For operating systems that use systemd, you would instead enter the following from the
command-line:

/usr/bin/mysqld_bootstrap

Both of these start the mysqld daemon on the node.

In MariaDB, use the command below to start MariaDB, Galera, and to establish the
Primary Component:

galera_new_cluster

Once the first node starts the database server, verify that the cluster has started, albeit a
one-node cluster, by checking wsrep_cluster_size. With the database client, execute the
following SQL statement:

SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 1 |
+--------------------+-------+

This status variable indicates the number of nodes that are connected to the cluster.
Since only the first node has been started, the value is 1 here. After you start other
nodes that will be part of this same cluster, execute this SQL statement again–on thee
first node or any node you’ve verified are in the cluster. The value should reflect the
number of nodes in the cluster.

Once you get the first node started and the Primary Component initialized, don’t restart
mysqld. Instead, wait until you’ve added more nodes to the cluster so that it can stay
viable without the first node. If you must restart the first node before adding other
nodes, shutdown mysqld and then bootstrap start it again (e.g., execute
galera_new_cluster). If it won’t start as easily as it did the first time, you may have to
edit the file containing the Galera Saved State (i.e., /var/lib/mysql/grastate.dat). The
contents of that file will look something like this:

# GALERA saved state

version: 2.1
uuid: bd5fe1c3-7d80-11e9-8913-4f209d688a15
seqno: -1
safe_to_bootstrap: 0

The variable safe_to_bootstrap is set to 0 on the first node after it’s been bootstrapped
to protect against you inadvertently bootstrapping again while the cluster is runnning.
You’ll have to change the value to 1 to be able to bootstrap anew.

Adding Nodes to the Cluster

Once you have successfully started the first node and thereby initialized a new cluster,
the procedure for adding all the other nodes is even simpler. You just launch mysqld as
you would normally–without the --wsrep-new-cluster option. You would enter
something like the following from the command-line, depending on your operating
system and database system (see above for other methods):

systemctl start mariadb

When the database server initializes as a new node, it will try to connect to the cluster
members. It knows where to find these other nodes based on the IP addresses listed in
the wsrep_cluster_address parameter in the configuration file.

You can verify that the node connection was successful checking the wsrep_cluster_size
status variable. In the database client of any node in the cluster, run the following SQL
statement:

SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 2 |
+--------------------+-------+
This indicates that the two nodes are now connected to the cluster. When the nodes in
the cluster agree on the membership state, they initiate state exchange. In state
exchange, a new node will check the cluster state. If the state of a new node differs from
the cluster state–which is normally the case–the new node requests a state snapshot
transfer (SST) from the cluster and it installs it on its local database. After this is done,
the new node is ready for use.

Getting Started Galera with Docker

Docker is an open platform for developers and sysadmins to build, ship, and run
distributed applications. Consisting of Docker Engine, a portable, lightweight runtime
and packaging tool, and Docker Hub, a cloud service for sharing applications and
automating workflows, Docker enables apps to be quickly assembled from components
and eliminates the friction between development, QA, and production environments. As
a result, it can ship faster and run the same app, unchanged, on laptops, data center
VMs, and any cloud.

let’s first look at how to build a basic Docker Image, which we will extend later. Then
we’ll deploy on a test cluster on a local machine. The examples here have been tested
on Ubuntu 14.04 with Docker 1.5.

Build a Basic Docker Image

In Docker, Dockerfiles are used to describe the Docker images we are going to use to
start our Galera Cluster. We are using the following Dockerfile:

FROM ubuntu:14.04
MAINTAINER Erkan Yanar <[email protected]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update

RUN apt-get install -y software-properties-common
RUN apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 BC19DDBA
RUN add-apt-repository 'deb https://releases.galeracluster.com/ubuntu trusty
main'
RUN apt-get update
RUN apt-get install -y galera-3 galera-arbitrator-3 mysql-wsrep-5.6 rsync lsof
COPY my.cnf /etc/mysql/my.cnf
ENTRYPOINT ["mysqld"]
This image builds on top of the Ubuntu 14.04 image. It simply installs Galera using the
Codership repository and copies the my.cnf over.

The my.cnf is quite simple.

[mysqld]
user = mysql
bind-address = 0.0.0.0
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_sst_method = rsync
default_storage_engine = innodb
binlog_format = row
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
query_cache_size = 0
query_cache_type = 0

A pre-built image is available from Docker Hub. You can pull it by running:

sudo docker pull erkules/galera:basic

(All commands in this article need to run as root.)

Deploy on a Test Cluster on a Local Machine

Next, we are going to start a Galera Cluster on the local host. The instructions below are
for demonstration purposes only and will not work when deploying on multiple hosts,
as networking between containers needs to be set up. Configuring Docker networking
across multiple hosts will be described in a following post.

Starting a Cluster
There have been a number of blog posts showing how to start Galera Cluster on a single
host. This post is going to show the simplest way to do that in Docker by using simple
commands, which will not work for a multi-host installation. First, if working on Ubuntu,
we need to put AppArmor’s Docker profile in complain mode in advance.

$ sudo aa-complain /etc/apparmor.d/docker

Then we can start the first Galera node by instructing Docker to create a container and
run mysqld in it.
$ sudo docker run --detach=true --name node1 -h node1 erkules/galera:basic --
wsrep-cluster-name=local-test --wsrep-cluster-address=gcomm://

In addition to defining the internal name and hostname, we also define the name of the
cluster.

MySQLs error log is not configured explicitly, and Docker records STDOUT and STDERR
of every container. So, using sudo docker logs node1, the log output from the first node
can be displayed without having to enter the container. For the next two containers, we
use a simple Docker trick. The –link option writes the name and the IP of host1 into the
/etc/hosts file of the container. This way, we can connect the remaining nodes to node1
without having to obtain its IP from its container.

$ sudo docker run --detach=true --name node2 -h node2 --link node1:node1

erkules/galera:basic --wsrep-cluster-name=local-test --wsrep-cluster-
address=gcomm://node1
$ sudo docker run --detach=true --name node3 -h node3 --link node1:node1
erkules/galera:basic --wsrep-cluster-name=local-test --wsrep-cluster-
address=gcomm://node1

Now we have a running Galera cluster. We can check the number of nodes in the Cluster
by running the mysql client from inside one of the containers:

$ sudo docker exec -ti node1 mysql -e 'show status like "wsrep_cluster_size"'

+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+

We built a simple Galera Cluster on one host, without using SSH and without the need
to configure any IP addresses. This setup does not support restarting the container—
you should remove the container and recreate it instead.

Deploying Galera on Mutiple Docker Hosts

Now let’s discuss how to deploy Galera on multiple Docker hosts. By design, Docker
containers are reachable using port-forwarded TCP ports only, even if the containers
have IP addresses. So we will set up port forwarding for all TCP ports that are required
for Galera to operate.

The following TCP port are used by Galera:

3306-MySQL port
4567-Galera Cluster
4568-IST port
4444-SST port

Before we start, we need to stop enforcing AppArmor for Docker:

$ aa-complain /etc/apparmor.d/docker

Building a Multi-Node Cluster using the Default Ports

Building a multi-node cluster using the default ports is not complicated. Besides
mapping the ports 1:1, we also need to set –wsrep-node-address to the IP address of the
host.

We assume following 3 nodes

nodea 10.10.10.10
nodeb 10.10.10.11
nodec 10.10.10.12

A simple cluster setup would look like this:

nodea$ docker run -d -p 3306:3306 -p 4567:4567 -p 4444:4444 -p 4568:4568

--name nodea erkules/galera:basic
--wsrep-cluster-address=gcomm:// --wsrep-node-address=10.10.10.10
nodeb$ docker run -d -p 3306:3306 -p 4567:4567 -p 4444:4444 -p 4568:4568
--name nodeb erkules/galera:basic
--wsrep-cluster-address=gcomm://10.10.10.10 --wsrep-node-address=10.10.10.11
nodec$ docker run -d -p 3306:3306 -p 4567:4567 -p 4444:4444 -p 4568:4568
--name nodec erkules/galera:basic
--wsrep-cluster-address=gcomm://10.10.10.10 --wsrep-node-address=10.10.10.12
nodea$ docker exec -t nodea mysql -e 'show status like "wsrep_cluster_size"'

+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+

In this example, we used the image from the previous blog post. Docker is going to
download the image if it is not already present on the node.

Building a Multi-Node Cluster using Non-Default Ports

In the long run, we may want to start more than one instance of Galera on a host in
order to run more than one Galera cluster using the same set of hosts.

For the purpose, we set Galera Cluster to use non-default ports and then map MySQL’s
default port to 4306:

MySQL port 3306 is mapped to 4306

Galera Cluster port 4567 is changed to 5567
Galera IST port 4568 is changed to 5678
Galera SST port 4444 is changed to 5444

The docker command line part is straightforward. Please note the additional command-
line options used to configure Galera

nodea$ docker run -d -p 4306:3306 -p 5567:5567 -p 5444:5444 -p 5568:5568

--name nodea erkules/galera:basic --wsrep-cluster-address=gcomm://
--wsrep-node-address=10.10.10.10:5567 --wsrep-sst-receive-
address=10.10.10.10:5444
--wsrep-provider-options="ist.recv_addr=10.10.10.10:5568"
nodeb$ docker run -d -p 4306:3306 -p 5567:5567 -p 5444:5444 -p 5568:5568
--name nodeb erkules/galera:basic
--wsrep-cluster-address=gcomm://10.10.10.10:5567
--wsrep-node-address=10.10.10.11:5567 --wsrep-sst-receive-
address=10.10.10.11:5444
--wsrep-provider-options="ist.recv_addr=10.10.10.11:5568"
nodec$ docker run -d -p 4306:3306 -p 5567:5567 -p 5444:5444 -p 5568:5568
--name nodec erkules/galera:basic
--wsrep-cluster-address=gcomm://10.10.10.10:5567
--wsrep-node-address=10.10.10.12:5567 --wsrep-sst-receive-
address=10.10.10.12:5444
--wsrep-provider-options="ist.recv_addr=10.10.10.12:5568"
nodea$ docker exec -t nodea mysql -e 'show status like "wsrep_cluster_size"'

+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+

The following Galera Cluster configuration options are used to specify each port:

4567 Galera Cluster is configured using `–wsrep-node-address`

4568 IST port is configured using `–wsrep-provider-options=”ist.recv_addr=”`
4444 SST port is configured using `–wsrep-sst-receive-address`

Summary
That concludes this tutorial. As you can see, it’s easy to run Galera on Docker and inside
Docker on multiple hosts, even with non-standard ports. It is also possible to use
solutions such as weave, socketplane.io and flannel that provide a multi-host network
for the containers.

Testing a Cluster

When you have a cluster running, you may want to test certain features to ensure that
they are working properly or to prepare yourself for handling actual problems that may
arise.

Replication Testing
There are a few step to do to test that Galera Cluster is working as expected. First, using
the database client, verify that all nodes have connected to each other. To do this,
execute the SHOW STATUS statement like so:

SHOW STATUS LIKE 'wsrep_%';

+---------------------------+------------+
| Variable_name | Value |
+---------------------------+------------+
...
| wsrep_local_state_comment | Synced (6) |
| wsrep_cluster_size | 3 |
| wsrep_ready | ON |
+---------------------------+------------+

Because of the LIKE operator, only variables beginning with wsrep_ are returned. The
three variables pertinent here are the ones shown in the example above.

 wsrep_local_state_comment: The value

Synced indicates that the node is
connected to the cluster and
operational.
 wsrep_cluster_size: The numeric value
returned indicates the number of nodes
in the cluster.
 wsrep_ready: A value of ON indicates that
this node in which the SQL statement
was executed is connected to the cluster
and able to handle transactions.

For the next test, try creating a table and inserting data into it. Use a database client on
node1 to enter these SQL statements:

CREATE DATABASE galeratest;

USE galeratest;
CREATE TABLE test_table (
id INT PRIMARY KEY AUTO_INCREMENT,

msg TEXT ) ENGINE=InnoDB;

INSERT INTO test_table (msg)

VALUES (“Hello my dear cluster.”);

INSERT INTO test_table (msg)

VALUES (“Hello, again, cluster dear.”);

 These statements will create the database
galeratest and the table test_table within
it. The last two SQL statements inserts data
into that table. After doing this, log into
node2 and check that the data was replicated
correctly. You would do this with by
executing the following SQL statement on
node2:

SELECT * FROM galeratest.test_table;

id msg

1
Hello my dear cluster. Hello, again, cluster dear.
2

The results returned from the SELECT

statement indicates that the data entered on
node1 was replicated on node2.

Split-Brain Testing
There are a few steps to test Galera Cluster
for split-brain situations on a two-node
cluster. First, disconnect the network
connection between the two nodes. At this
point, the Quorum will be lost and the nodes
won’t serve requests.

Now, reconnect the network connection. The

quorum will remain lost and the nodes still
won’t serve requests.

To reset the quorum, on one of the database

clients, execute the following SQL statement:
 SET GLOBAL
wsrep_provider_options=’pc.bootstrap=
1’;

At this point the quorum should be reset and

the cluster recovered.

Failure Simulation
You can also test Galera Cluster by simulating
various failure situations on three nodes. To
simulate a crash of a single mysqld process,
execute the following from the command-line
on one of the nodes:

$ killall -9 mysqld

To simulate a network disconnection, use

iptables or netem to block all TCP/IP traffic
to a node.

To simulate an entire server crash, run each

mysqld in a virtualized guest, and abrubtly
terminate the entire virtual instance.

If you have three or more Galera Cluster

nodes, the cluster should be able to survive
the simulations.