a9s MySQL Cluster Recovery

It occurs that a MySQL cluster member is not following the master node. In this case, we need to bootstrap this lost member again.

You can achieve this by doing these steps (execute all commands on the lost cluster member):

1. Execute monit stop all to stop all processes

2. Make sure that there are no mysql processes running anymore. You can use ps aux to search for still running processes:

   ps aux | grep mysql

   The PID is in the seconds column. To kill the process and its child processes, use:

   kill -9 {PID}

   Example:

   kill -9 11496

3. Remove the directory /var/vcap/store/mysql*

   rm -rf /var/vcap/store/mysql*

4. Make sure that both

• /var/vcap/store/mysql

• /var/vcap/store/mysql-temp are gone:

  ls /var/vcap/store

5. Run the script /var/vcap/jobs/mysql/bin/pre-start as root user
6. Execute monit start all to start the processes again.