Skip to main content
Version: Latest

Manual Logical Backup Recovery

This document briefly describes how to manually restore a logical backup from a MongoDB deployment to a second MongoDB deployment on another site.

Requirements​

There are some requirements that must be met in order to accomplish this:

  • Either the ** aws-cli or the azure-cli:** With the credentials necessary to download the backup.
  • Access to the bucket where the backup is stored: The CLI must be configured with credentials that permit at least reading the files.
  • Access to the old database where the backup metadata is stored: This is necessary to recover the encryption key used to encrypt the backup files. An available a9s Backup Manager is necessary to be able to read data from the database, or to take note of the encryption key beforehand.

How to Discover the Service Instance GUID​

In order to identify the Backup ID on a9s Backup Manager, the Cloud Foundry Service Instance guid must be identified. If the Cloud Foundry CLI can not be used (cf service <service-name> --guid), follow the step below.

Required ID's
  • We need the Service Instance GUID to find the Backup Manager Service Instance ID .
  • We need the Backup Manager Service Instance GUID to find the Backup ID.
  • The Backup ID is formed by the Backup Manager Service Instance GUID + - + <timestamp>.

Using Cloud Foundry and Service Broker Databases​

Access the ccdb (Cloud Controller Database) database.

After connected, find the GUID of the desired Service Instance with:

SELECT name, guid FROM service_instances WHERE name = '<service-instance-name>';

How to Discover the a9s Backup Manager Service Instance GUID​

The Backup ID has the following format: <backup-manager-service-instance-guid>-<timestamp>.

The <backup-manager-service-instance-guid> is generated by the a9s Backup Manager to identify the Service Instance. In order to identify the correct base backup to use when recovering a Service Instance, we need to find out the Service Instance ID on the a9s Backup Manager.

note

The Service Instance ID generated by the a9s Backup Manager is not the same used by Cloud Foundry to identify the Service Instance.

Using the a9s Backup Manager API​

The a9s Backup Manager API supports querying for the backups of a given Service Instance. You can use this method if the a9s Backup Manager for the failing site is still available.

Retrieve the a9s Backup Manager password for the admin user:

credhub get -n /backup_manager_password

Then, trigger the following request against your Backup Manager:

curl -u <backup-manager-username>:<backup-manager-password> \
  http://<backup-manager-host>:3000/instances/<service-instance-guid>

This call returns the backup and restore history. You can select which backup to use based on the created_at and updated_at fields, and use the backup_id field content.

Using a9s Backup Manager Ruby Console (IRB)​

The a9s Backup Manager includes a script that runs the Interactive Ruby Shell already configured to access the current a9s Backup Manager database. To execute this script, access the a9s Backup Manager instance, become root and execute the following command:

/var/vcap/jobs/anynines-backup-manager/bin/rails_c

Inside the IRB shell, execute the following commands:

irb(main):001:0> Instance.where(instance_id: "<service-instance-guid>").first.guid

The <service-instance-guid> is the Cloud Foundry Service Instance GUID (How to Discover the Service Instance GUID).

Using the a9s Backup Manager Database​

Use this method if your a9s Backup Manager is not available.

First, you need to access the PostgreSQL CLI at a9s-pg deployment and then access its internal a9s Backup Service database. The name of the database is specified in the backup-service manifest configured during the a9s Framework deployment.

SELECT guid FROM instances WHERE instance_id='<service-instance-guid>';

The <service-instance-guid> is the Cloud Foundry Service Instance GUID (How to Discover the Service Instance GUID).

Download Files​

The logical backup storage follows the structure below:

AWS S3:

  • <bucket>: Or container where the backups are stored.
    • <backup-manager-service-instance-id>.json: Files holding metadata about the backup.
    • <backup-manager-service-instance-id>-<index>: Encrypted and split backup file.

Azure:

  • <bucket>: Or container where the backups are stored.
    • <backup-manager-service-instance-id>: Encrypted backup file.

Download Backup​

The first step is to identify which available backup from your storage backend will be used.

AWS S3:

aws s3 ls s3://<bucket>/<backup-manager-service-instance-guid>

This command should list the metadata files, each file looks like <backup-manager-service-instance-guid>-<timestamp>.json. Each .json file is the metadata about a backup. Since we are restoring to the latest available backup, get the name of the backup with the most recent timestamp.

The name of the backup (Backup ID) is the name of the file without .json.

It is important to note that backup files are split into multiple files containing 1GB each, this means that if your base backup has more than 1GB of size, you will need to put together these files before restoring, for the moment let's download all files belonging to a backup.

To list these files, execute the following:

aws s3 ls s3://<bucket>/<backup-manager-service-instance-guid>

And the following command to download all files that belongs to the backup:

aws s3 cp --exclude="*" --include="<backup-manager-service-instance-guid>*" --recursive \
  s3://<bucket> <tmp-base-backup-dir>

Azure:

az storage blob list --container-name <container> --prefix <backup-manager-service-instance-guid>- | jq '.[].name'

This command will list all the available backups, where each file name looks like <backup-manager-service-instance-guid>-<timestamp>. Since we are restoring to the latest available backup, get the name of the backup with the most recent timestamp.

The name of the backup (Backup ID) is the name of the file.

To download the file, execute:

az storage blob download --container-name <container> --file <backup-id> --name <backup-id>

Prepare Files​

The files are split (if you are using AWS S3) and encrypted. Before restoring, you will need to prepare these files.

The first step, after downloading, is to decrypt them. Once the files are decrypted, you must join them, if necessary, before starting the recovery process.

Retrieve Backup Encryption Key​

All backup files are encrypted before they are stored in the backend storage. So, in order to be able to extract these files it is necessary to decrypt them.

Unless the encryption key used to encrypt the backup was changed using the a9s Service Dashboard (in which case, the encryption key should have been securely stored by the user), access to the a9s Backup Manager of the old site will be necessary in order to retrieve the key used to encrypt the backup. This can be done with the following command:

bosh -d <deployment-name> ssh backup-manager
sudo su -

/var/vcap/jobs/anynines-backup-manager/bin/rails_c

irb(main):001:0> Backup.where(backup_id: "<backup-file-id>").first.credentials[:encryption][:key]

The accepted value for backup-file-id is shown in the table below:

NameDescription
backup-manager-service-instance-id-<timestamp>The id that the Backup Manager internally assigns to the instance with the timestamp appended at the end. This directly references a specific backup, which the command uses during execution.
note

While the value of backup-file-id can be similar to backup-<storage>-filename there is no guarantee that it will be consistent from storage to storage. Thus, it is recommended to use the values described in the table above.

For more information, see the Decrypt a Backup Whose Encryption Key is Unknown section of the documentation.

Unfortunately, if it is not possible to retrieve the credentials then it will be impossible to decrypt the files, which is necessary for the restore process.

Decrypt All Files​

AWS S3:

To decrypt the backup, execute the command below for all files that belong to the backup. All split files must be decrypted together and in the correct ascending order.

Usage of Compound Commands

The following commands make use of the following compound command cat $(ls -1v ...), which assumes the use of a Linux distribution operating system. Therefore, if you are using a different operating system for your environment, it is recommended that you adjust this compound command to accordingly.

cat $(ls -1v <backup-id>-*) \
  | openssl enc -aes256 -d -pass 'pass:<key>' \
  | gunzip -c > <dst-dir>/<backup-id>.dump

For example:

cat $(ls -1v b6f4c071-ef44-4af2-9608-531b4ce4823f-1548967947154-*) \
  | openssl enc -aes256 -d -pass 'pass:NYHD8MVmA55HEqoaYHpQaxfwEMcQ1wkI' \
  | gunzip -c > /var/vcap/store/mongodb-restore/b6f4c071-ef44-4af2-9608-531b4ce4823f-1569935509279.dump

Azure:

Execute the following command on the backup file:

cat <file-name> \
  | openssl enc -aes256 -d -pass 'pass:<key>' \
  | gunzip -c > <dst-dir>/<backup-id>.dump

For example:

cat b6f4c071-ef44-4af2-9608-531b4ce4823f-1569935509279 \
  | openssl enc -aes256 -d -pass 'pass:NYHD8MVmA55HEqoaYHpQaxfwEMcQ1wkI' \
  | gunzip -c > /var/vcap/store/mongodb-restore/b6f4c071-ef44-4af2-9608-531b4ce4823f-1569935509279.dump

Copy Files to the Recovery Instance​

note

The new Service Instance, where the old Service Instance backup will be restored, must be clear/clean and should not have any encryption keys associated with the Service Instance itself. If this is not the case, there might be a potential problem after the restoration.

Before copying the files to your Service Instance, make sure you have enough space to store the backup file and the created database. Furthermore, make sure you are restoring this dump file in the current master node. The recovery process must be started in the master node, then cloned by the standby nodes.

First, create a directory under /var/vcap/store/

bosh -d <service-instance-deployment-name> ssh mongodb/<node-index-or-id>
sudo su -
mkdir /var/vcap/store/mongodb-restore
chown root.vcap -R /var/vcap/store/mongodb-restore
chmod g+w /var/vcap/store/mongodb-restore

With the directory prepared, copy the backup file (dump file) to the Virtual Machine.

In the examples below the backup file is transferred using bosh scp:

Single Instance: Single dump file.

file_dump="<backup-id>.dump

bosh -d <service-instance-deployment-name> scp ${file_dump} mongodb/<node-index>:/var/vcap/store/mongodb-restore

Cluster Instance: Mongodb directory in tar.

file_dump="<backup-id>.dump

mongodb_dump_directory="${file_dump}.folder"
mkdir ${mongodb_dump_directory}
tar -xf ${file_dump} -C ${mongodb_dump_directory}

bosh -d <service-instance-deployment-name> scp -r ${mongodb_dump_directory} mongodb/<node-index>:/var/vcap/store/mongodb-restore

Backup Recovery​

Single Instance Recovery​

Parameters:

  • username: Username of MongoDB's admin user.
bosh -d <deployment_name> manifest > <deployment_manifest>
bosh int <deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_username
  • password: Password of the admin user of the MongoDB.
bosh -d <deployment_name> manifest > <deployment_manifest>
bosh int <deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_password
  • host: Domain of the MongoDB virtual machine.
<deployment_name>-mongodb-<node-index>.node.dc1.<consul_domain>
# Add if necessary
## --ssl --sslAllowInvalidCertificates - When the MongoDB uses TLS.

dump_path=<backup-id>.dump
/var/vcap/packages/mongodb/bin/mongorestore --quiet --host <host> --port <port> -u <username> -p <password> --gzip --archive=${dump_path}

Restore Cluster Instance​

Remove all service keys​

Before starting the restoration, remove all service keys of the new deployment. We also recommend identifying the new cluster's primary node in order to avoid a scenario where your cluster ends with no discernible primary after restoring the data.

Stop All MongoDB Process at the Nodes​

note
  • a9s MongoDB cluster dump is a tar file and must be placed in all nodes in the /var/vcap/store/mongodb folder, to ensure they have the same content.
  • Currently, it is not possible to restore a MongoDB cluster Service Instance from one MongoDB version to another. Hence, it is necessary to restore the dump content to the same version.

1 - Stop all nodes in the cluster (execute for each node):

bosh -d <service-instance-deployment-name> ssh mongodb/<node-index> -c "sudo monit stop mongodb"

2 - Check if all mongodb processes are stopped (execute for each node):

bosh -d <service-instance-deployment-name> ssh mongodb/<node-index> -c "sudo monit summary"

If all mongodb processes are stopped, you can proceed with the next step.

Applying the a9s MongoDB Cluster Dump​

  1. Restore the nodes (execute the following snippet for each node):
bosh -d <service-instance-deployment-name> ssh mongodb/<node-index>
dump_folder_path=<backup-id>.dump.folder

rm -rf /var/vcap/store/mongodb/*
mv -f ${dump_folder_path}/* /var/vcap/store/mongodb
rm /var/vcap/store/mongodb/mongod.lock

Start a9s MongoDB Processes at the Nodes​

Execute the following for each node:

bosh -d <service-instance-deployment-name> ssh mongodb/<node-index> -c "sudo monit start mongodb"

Change MongoDB User Credentials​

Due to the fact that, when restoring the cluster Service Instance, the MongoDB database comes with the old Service Instance's credentials we need to update them in accordance to the new Service Instance.

  • Get MongoDB User credentials from the old Service Instance.
note

If it is not possible to get the <old_deployment_username> and the <old_deployment_password>, it is not possible to restore a cluster Service Instance properly.

Find the <old_deployment_username>:

bosh -d <old_deployment_name> manifest > <old_deployment_manifest>
bosh int <old_deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_username

Find the <old_deployment_password>:

bosh -d <old_deployment_name> manifest > <old_deployment_manifest>
bosh int <old_deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_password
  • Get MongoDB User credentials from the current Service Instance.

Find the <username>:

bosh -d <current_deployment_name> manifest > <current_deployment_manifest>
bosh int <current_deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_username

Find the <password>:

bosh -d <current_deployment_name> manifest > <current_deployment_manifest>
bosh int <current_deployment_manifest> --path /instance_groups/name=mongodb/jobs/name=mongodb/properties/mongodb/admin_password
bosh -d <service-instance-deployment-name> ssh mongodb/<master_node_index>
sudo su -
  • Access MongoDB via the MongoDB's shell client.
Command
sudo su -
# When the Service Instance is SSL, use `--tls` `--tlsCAFile <your-ca-file-path>`.
/var/vcap/packages/mongosh/mongosh -u <old_deployment_username> -p <old_deployment_password> "admin"
  • Update MongoDB's username
db.system.users.update({"user":"<old_deployment_username>"}, {$set:{"user":"<username>"}})
  • Access MongoDB via the MongoDB's shell client with the new username
Command
sudo su -
# When the Service Instance is SSL, use `--tls` `--tlsCAFile <your-ca-file-path>`.
/var/vcap/packages/mongosh/mongosh -u <username> -p <old_deployment_password> "admin"
  • Update MongoDB's password.
db.changeUserPassword("<username>", "<password>")

Reconfigure Replication​

bosh -d <service-instance-deployment-name> ssh mongodb/<master_node_index>
sudo su -
  • Access MongoDB via the MongoDB's shell client
Command
sudo su -
# When the Service Instance is SSL, use `--tls` `--tlsCAFile <your-ca-file-path>`.
/var/vcap/packages/mongosh/mongosh -u <username> -p <password> "admin"
  • Reconfigure the replication configuration.

It is necessary configure the host with the node domains. These domains are structured in the format <deployment_name>-mongodb-<node_index>.node.dc1.<consul_domain>:<port>.

cfg = rs.conf()
cfg.members[0].host = <deployment_name>-mongodb-0.node.dc1.<consul_domain>:<port>
cfg.members[1].host = <deployment_name>-mongodb-1.node.dc1.<consul_domain>:<port>
cfg.members[2].host = <deployment_name>-mongodb-2.node.dc1.<consul_domain>:<port>
rs.reconfig(cfg, {"force": true})

Check MongoDB Health​

Find the Cluster's Primary Node​

Check which node has the primary property set to true.

bosh -d <deployment_name> ssh mongodb/0 -c 'echo "rs.isMaster()" | /var/vcap/packages/mongodb/bin/mongo' --results

Example: In this case, the node 0is the PRIMARY. Have a look at the output:

(...)
  "secondary" : false,
  "primary" : "<deployment_name>-mongodb-<node-index>.node.dc1.<consul_domain>:<port>",
  "me" : "<deployment_name>-mongodb-<node-index>.node.dc1.<consul_domain>:<port>",
(...)

Check MongoDB status​

  • Log into the primary node.
bosh -d <service-instance-deployment-name> ssh mongodb/<master_node_index>
sudo su -
  • Execute db.stats() in MongoDB's shell client.
Command
# When the Service Instance is SSL, use `--tls` `--tlsCAFile <your-ca-file-path>`.
/var/vcap/packages/mongosh/mongosh -u <username> -p <password> "admin"
db.stats()
  • Check if the database information complies with database backed up.

Check Replication Status​

  1. Get into the primary node.
bosh -d <service-instance-deployment-name> ssh mongodb/<master_node_index>
sudo su -
  1. Execute rs.status() in MongoDB's shell client.
Command
# When the Service Instance is SSL, use `--tls` `--tlsCAFile <your-ca-file-path>`.
/var/vcap/packages/mongosh/mongosh -u <username> -p <password> "admin" rs.status()
  1. Check if there is 1 PRIMARY node and 2 SECONDARY nodes. Output:
(...)
	"members" : [
		{
			"name" : "<deployment_name>-mongodb-0.node.dc1.<consul_domain>:<port>",
			"health" : 1,
			"stateStr" : "PRIMARY",
		},
		{
			"name" : "<deployment_name>-mongodb-1.node.dc1.<consul_domain>:<port>",
			"health" : 1,
			"stateStr" : "SECONDARY",
		},
		{
			"name" : "<deployment_name>-mongodb-2.node.dc1.<consul_domain>:<port>",
			"health" : 1,
			"stateStr" : "SECONDARY",
			"uptime" : 52519,
		}
	],
(...)