Disaster Recovery
Currently, a9s PostgreSQL is the only Data Service that supports performing a disaster recovery execution into a newly created service instance. All other a9s Data Services will need some manual interaction, which are not specifed and may vary from Data Service to Data Service.
Trying to apply the information contained on this document to an a9s Data Service aside from a9s PostgreSQL can leave your instances in an irrecuperable state.
a9s Elasticsearch does not support disaster recovery at any level.
This document helps you to create a fork of an existing service instance, based on the last saved state. This procedure can also be used to recover the last saved state of a deleted service instance.
For an Application Developer to be able to perform a disaster recovery, both the a9s Backup Manager and the service instances need to be prepared. In other words your Platform Operator must configure the a9s Backup Manager and a fresh and compatible instance must be ready for use.
As long the disaster recovery store is reachable, the steps described in this document can be used to recover the last saved state of a given instance, even if the whole system is not available on the primary site.
To know more about configuration techniques for the disaster recovery store, consult the Platform Operator documentation for Disaster Recovery.
This document describes the process using the a9s Service Dashboard API, but it can also be done using the a9s Cloud Foundry CLI plugin for disaster recovery.
Overview
In a nutshell, the Disaster Recovery, from the perspective of the Application Developer, can be seen as a series of steps that can be summarized as follows:
- Procuring the Backup GUID
- Procuring the encryption key
- Performing the operation
This document will walk your through these general steps for both the a9s Cloud Foundry CLI plugin for disaster recovery and the a9s Service Dashboard API.
Recommendations
We strongly recommend saving the information described here specifically the Backup GUID and encryption key, in a secure location. We recommend you do this as soon as your instance is created, or has a major operation performed on it (e.g after updates, recreation, changes).
The reason for this is to be prepared in case of a total disaster, such as the entire instance being lost. Having the information mentioned above ensures that you can recover the last saved state of said instance.
Note: The a9s Backup Manager does not provide the encryption key, as this must be set by you.
Downloading and Installing the a9s Cloud Foundry CLI Plugin
You can get the plugin from our public S3 bucket. The plugin is available for the following operating systems:
- Linux (sha:493bd71f62baea006f9de8ac1df6a4e5382c3a3a3e95303482ce9072a13491fc)
- MacOS (sha:3616fdf1dadd8452e439a3d49796cfb1b8f085c1e041adce08ae70e69ed164e5)
- Windows (sha:d609d4bb70eade48141e145153ac1a59b6ab9e8a9b0d075706d1103063d9fbd3)
Once you have downloaded the binary file you can install it into the Cloud Foundry CLI with the following command:
cf install-plugin <source-path>/cf-disaster-recovery-plugin -f
You may notice that we are using the -f flag in the command. This is necessary
because this plugin is not signed by Cloud Foundry, therefore the -f flag is
needed to ignore the unsigned certificate when installing the binary to the CLI.
Prepare the Service Instance
When performing disaster recovery, it is necessary to have some information about the original instance holding the data to be recovered, as we need to locate the correct backup collection and, if necessary, create a fresh instance with the same settings:
| Key | Descryption |
|---|---|
| Dashboard URL | The URL to the Dashboard of the Service Instance. Used to retrieve the Backup GUID. |
| Backup GUID | The GUID which the a9s Backup Manager is using to refer to backups of a given instance. |
| Backup Encryption Key | The encryption key used to encrypt the backup |
| Service Name | The name of the service used to create the Service Instance. |
| Plan Name | The name of the plan used to create the Service Instance |
| Custom Service Parameters | Custom parameter set during creating or updating of the Service Instance |
As mentioned before we strongly recommend retrieving this information as soon as you create or update your instance, especially when updating the encryption key, and storing it in a secure place. This way you will be able to recover from more catastrophic scenarios.
If you are using Cloud Foundry, Most of the information can be retrieved using the
Cloud Foundry CLI with a simple cf service <service-instance-name>, except:
- The intance's encryption key
- The instance's Backup GUID
Note: It is worth noting that the following caveats:
The new Service Instance has a completely different Backup GUID. This means that any backup that is created in the new Service Instance will have a different Backup GUID.Therefore it is imperative to properly distinguish them during the process and when saving them for future reference.
The new Service Instance will not have an user defined encryption key until you set one yourself. This encryption key will not affect the current disaster recovery, but it is important to note it down. It is imperative to properly distinguish both encryption key during the process and when saving them for future reference.
Setting the Backup's Encryption Key
In order to perform a disaster recovery without help from an operator (assuming the disaster recovery has been properly configured), the instance's encryption key must be set. This can be done through different methods:
- as a parameter of the request when retrieving the Backup GUID
- via the a9s Cloud Foundry CLI plugin for disaster recovery
- via the a9s Service Dashboard API
- via the a9s Dashboard
If using the default encryption key or the encryption key is unknown, the only way to retrieve it is via the a9s Backup Manager database with the help of your operator. For more information, please check the operator Disaster Recovery documentation.
Note: Keep in mind that setting a new encryption key will overwrite and replace any previously existing encryption key. This new encryption key is not retroactively applied, so any prior backups cannot be decrypted with it.
Note: It is worth noting that while the a9s Backup Manager can change the encryption key, it does not return it, thus it is necessary to save it manually, in a secure place.
Getting the Backup GUID
Both the a9s Service Dashboard API and the a9s Cloud Foundry CLI plugin for disaster recovery can be used to retrieve the Backup GUID.
Note: The Backup GUID provided by both the a9s Service Dashboard API and the a9s Cloud Foundry CLI plugin for disaster recovery differs from the Service Instance GUID provided by Cloud Foundry.
Getting the Backup GUID Using the API Directly
Retrieving all the necessary data using the API is only possible if the service instance still exists, for this reason, we recommended performing these steps directly after the creation, update or updating the encryption key, and store the information in a safe location.
Authentication
To use the API, you need to be authenticated and authorized. You will also need to create a cookie, which will store your authenticated session. The necessary steps to do so are explained in the a9s Service Dashboard documentation.
After the authentication step, you will have the bearer_token and url environment variables
and the cookie file called session.cookie.
Endpoint
POST /service-instances/:cf-instance-guid/backups/prepare_disaster_recovery
Parameters
| Key | Description |
|---|---|
encryption_key | Optional The key which the Backup Manager is using to encrypt backups. |
Response:
| Key | Description |
|---|---|
backup_guid | The GUID which the Backup Manager is using to refer to backups. |
cURL Example
curl -X POST --cookie session.cookie --cookie-jar session.cookie --location --insecure \
--header "Authorization: ${bearer_token}" \
--header "Content-Type: application/json" \
"${url}/backups/prepare_disaster_recovery" \
-d '{"encryption_key": "<encryption_key_value>"}'
Prepare Disaster Recovery via the CF CLI Plugin
You can prepare a disaster recovery backup by runnning the following command:
$ cf prepare-disaster-recovery <service-instance-name> [<encryption-key>]
where an already existing service instance is used as a parameter, with the
encryption-key being optional.
After execution the CF CLI will provide with all the necessary information for the disaster recovery backup. This response contains information such as the Backup GUID of the disaster recovery backup.
While the encryption-key is an optional parameter, keep in mind that if it is
not set it needs to be known, as the Backup Manager will not return
the current encryption key.
Perform the Disaster Recovery
To be able to perform a disaster recovery, you must first create a new service instance to host the data that you want to recover. This requires the new service instance to be created with the same parameters and with the same or bigger plan as the old service instance to avoid unexpected behavior when restoring the data (e.g.: a persistent disk that can not store the whole data). This information must be gathered in the Prepare the Service Instance section.
Cloud Foundry Example:
cf create-service <service name> <plan_name> my_psql_service -c <custom service parameters>
E.g.:
cf create-service a9s-postgresql13 postgresql-replica-small my_psql_service -c '{"continuous_archiving": "enabled"}'
After that, a restore can be triggered, with the additional information shown below:
| Key | Description |
|---|---|
| Backup GUID | The GUID which the a9s Backup Manager is using to refer to backups. |
| Backup Encryption Key | The encryption key used to decrypt the Backup. |
Note: All fields in this step are mandatory.
Note: If an instance is configured with continuous archiving it can only be recovered to an instance configured with continuous archiving.
Note: You can check the status of the recovery task using the a9s Dashboard of the service instance recovering the data. After the task is finished with success, the instance is ready for use.
Note: After the restore finishes, it is necessary to create a new service binding to the new service instance.
Perform the Disaster Recovery Using the API Directly
Authentication
To use the API, you need to be authenticated and authorized. You will also need to create a cookie, which will store your authenticated session. The necessary steps to do so are explained in the a9s Service Dashboard documentation.
After the authentication step, you will have the bearer_token and url environment variables
and the cookie file called session.cookie.
Endpoint
POST /service-instances/:cf-instance-guid/backups/perform_disaster_recovery
Parameters
| Name | Type | Description |
|---|---|---|
backup_guid | string | The GUID which the a9s Backup Manager is using to refer to backups. |
encryption_key | string | The encryption key that was used to encrypt the backups. |
Create new Service Instance
First, you create a new service instance, as shown in the example below:
cf create-service a9s-postgresql13 postgresql-replica-small my_recovered_psql_service -c '{"continuous_archiving": "enabled"}'
Perform Disaster Recovery with cURL
To perform a disaster recovery via cURL, the request must be executed against the full
url of the new service instance as described in the example below:
curl -X POST --cookie session.cookie --cookie-jar session.cookie --location --insecure \
--header "Authorization: ${bearer_token}" \
"${url}/backups/perform_disaster_recovery" \
--data "backup_guid=${backup_guid}" \
--data "encryption_key=${encryption_key}"
Perform Disaster Recovery via the CF CLI Plugin
You can restore a disaster recovery backup by runnning the following command:
$ cf disaster-recovery-restore <service-instance> <backup-guid> <encryption-key>
where the information received in the prepare method's response are the needed parameters. This will restore a disaster recovery backup on an existing service instance (not the original instance).