With our 3 controller servers running on a bare bones Ubuntu install, it’s time to start getting the services required for OpenStack up and running. Before any of the actual cloud services can be installed, we need a shared database and message queue. Because our design goal here is both redundancy and load balancing, we can’t just install a basic MySQL package.
A little researched showed that there are 3 options for a MySQL compatible multi-master cluster: MySQL w/ a wsrep patch, MariaDB, or Percona XtraDB Cluster. In all cases Galera is used as the actual clustering mechanism. For ease of installation (and because it has solid Ubuntu package support), we decided to use MariaDB + Galera for the clustered database.
MariaDB Cluster install
MariaDB has a handy tool to build the apt commands for mirror selection Here, so we’ll use that to build a set of commands to install MariaDB and Galera. We’ll also install rsync at this time if not already installed, since we’ll be using that for Galera cluster sync.
apt-get install software-properties-common
apt-key adv –recv-keys –keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
add-apt-repository ‘deb http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu trusty main’
apt-get update
apt-get install mariadb-galera-server galera rsync
MariaDB will prompt for a root password during install, for ease of use we’re using “openstack” as the password for all service accounts.
The configuration is shared on all 3 nodes, so first we’ll build the configuration on the controller-0 node, then copy it over to the others.
/etc/mysql/conf.d/cluster.cnf
[mysqld]
query_cache_size=0
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_type=0
bind-address=10.1.1.20
# Galera Provider Configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# Galera Cluster Configuration
wsrep_cluster_name=”openstack”
wsrep_cluster_address=”gcomm://10.1.1.20,10.1.1.21,10.1.1.22″
# Galera Synchronization Congifuration
wsrep_sst_method=rsync
# Galera Node Configuration
wsrep_node_address=”10.1.1.20″
wsrep_node_name=”controller-0″
We’ll also comment out the “bind-address = 127.0.0.1” line from /etc/mysql/my.cnf so that the DB server will listen on our specified address instead of just localhost. (Note: While you can bind address to 0.0.0.0 for all interfaces, this will interfere with our colocated HAProxy unless mysql is moved to a port other than 3306. Since we want to keep mysql on the default port, we’ll just bind it to the internal IP for now)
The last bit of prep needed is to copy the contents of /etc/mysql/debian.cnf from the first node to the other two, since Ubuntu/Debian uses a system maintenance account and the credentials are randomly generated on install time.
Now it’s time to bring up and test the cluster!
First, stop all nodes with “service mysql stop“. Double check that the mysqld is actually stopped, the first time we did that it was still running on the 2nd and 3rd node for some reason.
Initialize the cluster on the primary node: “service mysql start –wsrep-new-cluster”
Once that’s up and running, start mysql on the next two nodes and wait for it to fully come up: “service mysql start”
Now, it’s time to test the database replication and multi-master config. We’ll do this by writing data to the first node, confirming it replicated onto the second, writing data to the second, confirming on the third, etc…
root@controller-0:~# mysql -u root -popenstack -e “CREATE DATABASE clustertest;”
root@controller-0:~# mysql -u root -popenstack -e “CREATE TABLE clustertest.table1 ( id INT NOT NULL AUTO_INCREMENT, col1 VARCHAR(128), col2 INT, col3 VARCHAR(50), PRIMARY KEY(id) );”
root@controller-0:~# mysql -u root -popenstack -e “INSERT INTO clustertest.table1 (col1, col2, col3) VALUES (‘col1_value1′,25,’col3_value1’);”
root@controller-1:/etc/mysql/conf.d# mysql -u root -popenstack -e “SELECT * FROM clustertest.table1;”
+—-+————-+——+————-+
| id | col1 | col2 | col3 |
+—-+————-+——+————-+
| 1 | col1_value1 | 25 | col3_value1 |
+—-+————-+——+————-+
root@controller-1:/etc/mysql/conf.d# mysql -u root -popenstack -e “INSERT INTO clustertest.table1 (col1,col2,col3) VALUES (‘col1_value2′,5000,’col3_value2’);”
root@controller-2:/etc/mysql/conf.d# mysql -u root -popenstack -e “SELECT * FROM clustertest.table1;”
+—-+————-+——+————-+
| id | col1 | col2 | col3 |
+—-+————-+——+————-+
| 1 | col1_value1 | 25 | col3_value1 |
| 4 | col1_value2 | 5000 | col3_value2 |
+—-+————-+——+————-+
root@controller-2:/etc/mysql/conf.d# mysql -u root -popenstack -e “INSERT INTO clustertest.table1 (col1,col2,col3) VALUES (‘col1_value3′,99999,’col3_value3’);”
root@controller-0:/etc/mysql/conf.d# mysql -u root -popenstack -e “SELECT * FROM clustertest.table1;”
+—-+————-+——-+————-+
| id | col1 | col2 | col3 |
+—-+————-+——-+————-+
| 1 | col1_value1 | 25 | col3_value1 |
| 4 | col1_value2 | 5000 | col3_value2 |
| 5 | col1_value3 | 99999 | col3_value3 |
+—-+————-+——-+————-+
HAProxy + MariaDB integration
Now that we have a clustered database, it’s time to tie it to the HAProxy so that requests can be load balanced, and the MySQL server taken offline if necessary. We don’t need MySQL to be externally accessible, so we’ll only be configuring this service on the management (10.1.1.0/24) network. Let’s start by adding an haproxy user on each of the 3 database servers. (This user should probably be restricted to the load balancer hosts in a more secure production environment) This user has no password and cannot actually connect to any databases, it simply allows a connection to the mysql server for a health check. (Note that you must use the “CREATE USER” DDL in order for changes to be replicated. If you insert directly into the mysql tables, the changes will NOT be replicated to the rest of the cluster)
root@controller-0:~# mysql -u root -popenstack
MariaDB [mysql]> CREATE USER ‘haproxy’@’%’;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> exit
Bye
Once done, we’ll add (on each node) the health check to each of the 3 HAProxy servers in /etc/haproxy/haproxy.cfg: (Configured to listen on the internal VIP presented by keepalived)
listen galera 10.1.1.10:3306
balance source
mode tcp
option tcpka
option mysql-check user haproxy
server controller-0 10.1.1.20:3306 check weight 1
server controller-1 10.1.1.21:3306 check weight 1
server controller-2 10.1.1.22:3306 check weight 1
(Note: If you forgot to tell MySQL to explicitly bind to the above IP addresses, haproxy will fail since MySQL is already bound to 3306. If you need multiple IP addresses, you can move MySQL to port 3307 and proxy to that instead)
We’ll reload the haproxy config with “service haproxy reload” and see if we can connect to MySQL on the VIP:
root@controller-0:/var/log# mysql -h 10.1.1.10 -u root -popenstack
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 226
Server version: 5.5.38-MariaDB-1~trusty-wsrep-log mariadb.org binary distribution, wsrep_25.10.r3997
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
MariaDB [(none)]>
We can also see our MySQL services being monitored by the load balancer statistics page:
RabbitMQ Message Queue
The message queue of choice for most OpenStack installations appears to be RabbitMQ. Since it supporters clustering natively, the the OpenStack services will load balance to the message queue without any additional proxy, this step goes fairly quickly.
Install the RabbitMQ server on all the controller nodes with “apt-get install ntp rabbitmq-server“. Once the install is completed, stop the rabbitmq service on all nodes (“service rabbitmq-server stop“) and confirm that the service isn’t running.
RabbitMQ wants to do hostname based clustering, and since we’re not running any DNS in this environment, we need to add the following lines to /etc/hosts:
10.1.1.20 controller-0
10.1.1.21 controller-1
10.1.1.22 controller-2
Additionally, these IP addresses must match the hostnames of your servers. If your servers still have the default “ubuntu” hostname, clustering will fail.
In order for the servers to cluster, the Erlang cookie needs to be the same on all nodes. Copy the file /var/lib/rabbitmq/.erlang.cookie from the first node to the other two nodes. (Since we don’t have root login enabled, we used scp to the ubuntu account, the moved it into place locally. You’ll need to make sure the file has user and group “rabbitmq” after copying.) Now we can restart the service on all nodes with “service rabbitmq-server start”.
With these steps in place, clustering rabbitmq is simple. We’ll start on the 2nd node. (controller-1):
root@controller-1:/var/lib/rabbitmq# service rabbitmq-server stop
* Stopping message broker rabbitmq-server [ OK ]
root@controller-1:/var/lib/rabbitmq# service rabbitmq-server start
* Starting message broker rabbitmq-server [ OK ]
root@controller-1:/var/lib/rabbitmq# cd
root@controller-1:~# rabbitmqctl stop_app
Stopping node ‘rabbit@controller-1’ …
…done.
root@controller-1:~# rabbitmqctl join_cluster rabbit@controller-0
Clustering node ‘rabbit@controller-1’ with ‘rabbit@controller-0’ …
…done.
root@controller-1:~# rabbitmqctl start_app
Starting node ‘rabbit@controller-1’ …
…done.
root@controller-1:~# rabbitmqctl cluster_status
Cluster status of node ‘rabbit@controller-1’ …
[{nodes,[{disc,[‘rabbit@controller-0′,’rabbit@controller-1’]}]},
{running_nodes,[‘rabbit@controller-0′,’rabbit@controller-1’]},
{partitions,[]}]
…done.
Now that the first two nodes are clustered, we’ll do the same with the 3rd:
root@controller-2:/var/lib/rabbitmq# service rabbitmq-server stop
* Stopping message broker rabbitmq-server [ OK ]
root@controller-2:/var/lib/rabbitmq# service rabbitmq-server start
* Starting message broker rabbitmq-server [ OK ]
root@controller-2:/var/lib/rabbitmq# rabbitmqctl stop_app
Stopping node ‘rabbit@controller-2’ …
…done.
root@controller-2:/var/lib/rabbitmq# rabbitmqctl join_cluster rabbit@controller-0
Clustering node ‘rabbit@controller-2’ with ‘rabbit@controller-0’ …
…done.
root@controller-2:/var/lib/rabbitmq# rabbitmqctl start_app
Starting node ‘rabbit@controller-2’ …
…done.
root@controller-2:/var/lib/rabbitmq# rabbitmqctl cluster_status
Cluster status of node ‘rabbit@controller-2’ …
[{nodes,[{disc,[‘rabbit@controller-0′,’rabbit@controller-1’,
‘rabbit@controller-2’]}]},
{running_nodes,[‘rabbit@controller-0′,’rabbit@controller-2’,
‘rabbit@controller-1’]},
{partitions,[]}]
…done.
And we’re done! With the clustered DB and MQ building blocks in place, we’re ready to start installing the actual OpenStack services on our controllers. It’s worth noting that RabbitMQ has a concept of a disc and a RAM node and nodes can be changed at any time. Nothing in any of the documentation for OpenStack suggests that a RAM node is required for performance, but presumably if there becomes a question of scale, it would be worth looking into.