Creating Backups with Elasticsearch on Azure

Creating backups on Azure with Elasticsearch requires a different approach and as such the specification of parameters is different from the usual proceedings.

Some settings are still configured on the backup-service whereas the credentials for the Azure account itself must be specified in the deployment manifest of the service instance because the credentials have to be present on all cluster nodes.

Prerequisites

You need access to Azure. There are three things you need regarding that (in terms of the az CLI):

  • account-name
  • account-key
  • container-name

Configuring the Backup Service

The setting required to configure the backup-service are container and provider:

plugin_configuration:
  elasticsearch-backup:
    container: 'your-container-name'
    provider: azure
    name:
      backup: elasticsearch-backup
      restore: elasticsearch-restore

Configuring Service Instances

Unfortunately the way of how and where to specify credentials differ in version 5 and version 6. However, the way of specifying it in the deployment manifest is the same. Only the key names differ there.

Elasticsearch Version 5

Properties of an ES instance in version 5:

elasticsearch:
    keystore5:
      'cloud.azure.storage.default.account': 'your-account-name'
      'cloud.azure.storage.default.key': 'your-account-key'

Elasticsearch Version 6 (+)

Properties of an ES instance in version 6:

elasticsearch:
    keystore:
      'azure.client.default.account': 'your-account-name'
      'azure.client.default.key': 'your-account-key'

Verify Your Setup

To check whether the backup works you can use the az CLI. Find out the deployment name which you can do by having a look at your bosh tasks or by creating a cf service-key and having a look at the hostname. In this example the deployment name is d13d357. Use the az CLI like so to find backups for your service instance:

az storage blob list --account-key 'your-account-key' --account-name 'your-account-name' --container-name 'your-container-name' --prefix=d13d357

Globally Setting Default Credentials

If you want all service instances to share one set of azure credentials you can do so by modifying the properties of the template-uploader job in your Elasticsearch service deployment, e.g.:

  properties:
    template-uploader:
      template-vars:
+       /elasticsearch_backup_azure_account_name: 'your-account-name'
+       /elasticsearch_backup_azure_account_key: 'your-account-key'
+     template-ops-files:
+     - ops/es-backup-azure-credentials.yml
+     - ops/es5-backup-azure-credentials.yml

Note: After adding those lines, the template uploader errand must be executed.