a9s MariaDB Cluster Recovery

It occurs that a MariaDB cluster member is not following the master node. In this case, we need to bootstrap this lost member again.

You can achieve this by doing these steps (execute all commands on the lost cluster member):

Replace the <replace-me> value for the following option below depending on the version:

• For a9s MariaDB >= 10.6 use mariadb.
• For a9s MariaDB < 10.6 use mysql.

1. Execute monit stop all to stop all processes

2. Make sure that there are no <replace-me> processes running anymore. You can use ps aux to search for still running processes:

   ps aux | grep <replace-me>

   The PID is in the second column. To kill the process and its child processes, use:

   kill -9 {PID}

   Example:

   kill -9 11496

3. Remove the directory /var/vcap/store/<replace-me>*

   rm -rf /var/vcap/store/<replace-me>*

4. Make sure that both

• /var/vcap/store/<replace-me>

• /var/vcap/store/<replace-me>-temp are gone:

  ls /var/vcap/store

5. Run the script /var/vcap/jobs/<replace-me>/bin/pre-start as root user
6. Execute monit start all to start the processes again.