a9s Data Services Administrative Tasks

This document explains the most common tasks an a9s Data Service Operator should know.

Map a Service Instance GUID to a BOSH Deployment Name

In order to find out which BOSH deployment belongs to which Service Instance, the following command can be used (jq must be installed):

curl --user admin:[deployer-api-password] [deployer-api-endpoint]/deployments.json | jq '.[] | \
  select(.deployment_attributes.instance_guid == "[service-instance-guid]") | .name'

Update All Service Instances

There are three scenarios where the Platform Operator might want to update the existing Service Instance deployments:

1. A new version of this anynines-deployment repo is available and contains new BOSH releases or new configurations
2. The Platform Operator uploaded a new stemcell to the BOSH director
3. The Platform Operator changed the BOSH Cloud Config in their setup

In case of the first two scenarios the Platform Operator first has to execute the templates-uploader errand so that the templates in the a9s BOSH Deployer are updated. These changes in the templates will be picked up as Maintenance Updates by the a9s Data Services Framework.

Based on that, the Platform Operator can use the a9s Deployment Updater errand to update all outdated deployments.

In case of the third scenario, only the a9s Deployment Updater errand needs to be executed and all outdated deployments are updated.

Simply run this errand by executing the following command:

bosh -d <deployment_name> run-errand deployment_updater

Updating stopped Service Instances

The Stop/Start feature (currently available only for a9s PostgreSQL), introduced the stopped state to the Service Instance's lifecycle.

For Service Instances in a stopped state, any attempt to update them will fail, as stopped is not recognized as a state allowed to perform updates.

Further information about the Stop/Start feature is available here.

For more information about how to configure the a9s Deployment Updater see a9s Deployment Updater - Properties.

Caveats

• When changing the template_name_v2 property for a service plan, the deployment_updater cannot recognize this change and needs to run for all instances.

Update a Specific Service Instance

Instead of updating all Service Instances of a service, it is also possible to update only one Service Instance. To do so, the GUID of the Service Instance must first be identified. When using Cloud Foundry, this can be done by running:

$ cf service service-instance-name-in-cf --guid
34e68cdf-62cc-4ea6-a3e8-714026dba1f8

In this example, the Service Instance GUID is 34e68cdf-62cc-4ea6-a3e8-714026dba1f8. Next, the endpoint of the Service Broker and the Service Broker admin password must be identified. Once this information is available, an update of the Service Instance can be triggered by executing:

$ curl --user admin:[service-broker-password] -X PATCH \
  [service-broker-hostname]:3000/v2/service_instances/[service-instance-guid] \
  -d '{"plan_id":"6b1973db-e057-4a71-9832-a4b3f27a0d8f", "service_id": "7ee52a02-8839-43c2-a550-728ad736bbda"}'

The service_id and plan_id parameters can be fetched from the service catalog of the broker.

Interact with the Backup Manager

As an a9s Data Service Operator, it is possible to interact with the Backup Manager API in order to trigger backups and restores.

To trigger a backup of all Service Instances, execute:

curl <user>:<password>@<backup manager endpoint>/backup_agent/backup_all -H "Content-Type: application/json" -d {}

To trigger a backup for a specific Service Instance, execute:

curl <user>:<password>@<backup manager endpoint>/backup_agent/backup -H "Content-Type: application/json" \
  -d '{"instance_guid": "<service-instance-guid>"}'

List all backups:

curl <user>:<password>@<backup manager endpoint>/instances

Trigger a restore:

Next to the Service Instance GUID, the backup id to be restored is required. This backup id can be found by calling the /instances/<service-instance-guid> endpoint first and getting the field id for a specific backup (not the field backup_id!). The backup id is an integer.

Once this information is available, the restore of the backup can be triggered by running:

curl <user>:<password>@<backup manager endpoint>/backup_agent/restores -H "Content-Type: application/json" \
  -d '{"instance_id": "<service-instance-guid>", "backup_id": <id>}'

Decrypt a Backup Whose Encryption Key Is Unknown

To decrypt an existing backup whose encryption key is unknown, access to the a9s Backup Manager is required. With access to the a9s Backup Manager, the following steps can be followed to obtain the encryption key and decrypt the backup.

1. Download the appropriate backup directly from the Backup Store. The backup name should be in the format <deployment-name>-<unix-timestamp>. To find the backup later in the database, the Created at date of the corresponding backup is required. This date can either be found on the a9s Service Dashboard or obtained by converting the unix-timestamp of the backup file to a date.

2. Connect to the a9s Backup Manager:

bosh -d backup-service ssh backup-manager

3. Become root:

sudo -i

4. Open the Rails console of the a9s Backup Manager:

/var/vcap/jobs/anynines-backup-manager/bin/rails_c

5. Find the correct encryption key for the backup. The name of the backup file typically contains the backup_id, which can be used to find the related encryption key:

Backup.where(
  backup_id: "89fe4350-784e-4c03-a779-e88067d53cd8-1694685740323"
  ).first.credentials[:filter_plugins][0][:password]

Please note that above snippet returns an Array. The encryption plugin should be the first (index 0) but it is possible that it is in another position. This depends on the specific configuration of the Backup Manager.

If the correct backup_id is missing, the backup can be found by filtering for the creation date. The date must be in the format Year-Month-Day Hour:Minute:Second. The time must be UTC. As an example, the date 2018-11-26 13:45:53 is used below:

Backup.where("created_at >= ?", "2018-11-26 13:45:53")

6. Decrypt the backup. As an example, the backup file ~/Downloads/d70a4d9-1543239953810 and the password from the previous step 12345678 are used:

cat ~/Downloads/d70a4d9-1543239953810 | openssl enc -aes256 -md md5 -d -pass 'pass:12345678' | \
  gunzip -c > ~/Downloads/d70a4d9-1543239953810.decrypted

Get the Error Backtrace from a Backup or Restore

If a backup or restore fails, the backtrace of the error is saved in the database. The following steps describe how to read the error backtrace.

1. Connect to the a9s Backup Manager:

bosh -d backup-service ssh backup-manager

2. Become root:

sudo -i

3. Open the Rails console of the a9s Backup Manager:

var/vcap/jobs/anynines-backup-manager/bin/rails_c

4. Get the Instance where the error happened:

instance = Instance.where(instance_id: "instance_guid").first

5. Get the Backup that fails. To do so, there are multiple options:

• If it is the last backup that failed:

backup = instance.backups.last

• If the backup_id is known, e.g. d25ed99-1543410104023:

backup = instance.backups.where(backup_id: "d25ed99-1543410104023")

• If the backup_id is not known, it can be found by filtering for the creation date of the Backup/Restore. The date must be in the format Year-Month-Day Hour:Minute:Second. The time must be UTC. As an example, the date 2018-11-26 13:45:53 is used below:

backup = instance.backups.where("created_at >= ?", "2018-11-26 13:45:53").first

6. Finally load the message and decode it:

Base64.decode64(backup.backup_agent_task.msg)

Backups of a9s-pg

The backup of the a9s-pg can now be handled with the a9s Backup Manager. See a9s_pg_backup for details.

Delete Obsolete Backup Metadata Files

Obsolete metadata files, from already deleted backups, can be deleted by executing the delete_metadata_files script within the backup-manager VM as shown in the steps below:

1. Connect to the a9s Backup Manager:

bosh -d backup-service ssh backup-manager

2. Become root:

sudo -i

3. Execute the script:

/var/vcap/jobs/anynines-backup-manager/bin/delete_metadata_files

Rotate Database Encryption Salts

To rotate the database encryption salts of the a9s Service Broker, the a9s BOSH Deployer or the Backup Manager, the following steps must be executed. The example below uses the mariadb-service:

1. Duplicate the current encryption salt

OLD_SALT=`credhub get -n "/<BOSH director name>/mariadb-service/mariadb_service_broker_db_salt" | \
  grep value | awk '{print $2}'`
credhub set -n "/<BOSH director name>/mariadb-service/mariadb_service_broker_db_salt_old" -t password -w "${OLD_SALT}"

OLD_SALT=`credhub get -n "/<BOSH director name>/mariadb-service/mariadb_service_deployer_db_salt32" | \
  grep value | awk '{print $2}'`
credhub set -n "/<BOSH director name>/mariadb-service/mariadb_service_deployer_db_salt32_old" -t password \
  -w "${OLD_SALT}"

OLD_SALT=`credhub get -n "/<BOSH director name>/backup-service/backup_manager_encryption_key" | \
  grep value | awk '{print $2}'`
credhub set -n "/<BOSH director name>/backup-service/backup_manager_encryption_key_old" -t password \
  -w "${OLD_SALT}"

2. Regenerate the encryption salt

credhub generate -n "/<BOSH director name>/mariadb-service/mariadb_service_broker_db_salt" -t password -l 32
credhub generate -n "/<BOSH director name>/mariadb-service/mariadb_service_deployer_db_salt32" -t password -l 32
credhub generate -n "/<BOSH director name>/backup-service/backup_manager_encryption_key" -t password -l 32

3. Redeploy the service

bosh -d mariadb-service deploy mariadb-service/mariadb-service.yml

4. Execute the errands

bosh -d mariadb-service run-errand migrate-deployer-encrypted-database-fields
bosh -d mariadb-service run-errand migrate-service-broker-encrypted-database-fields
bosh -d backup-service run-errand migrate-backup-manager-encrypted-database-fields

Rotate Consul Certificates

Prerequisites

Find Out BOSH Director Name

BOSH_NAME=`bosh env --json | jq '.Tables[0].Rows[0].name' -r`

Ensure the Current CredHub CA Entry Is Complete

The current CA value for the CredHub entry must not be empty:

credhub get -n "/${BOSH_NAME}/consul-dns/cdns_ca" --output-json | jq .value.ca

If the previous command returns null, the following commands must be executed to copy the value of the current certificate into the value for the current CA:

credhub get -k private_key -n "/${BOSH_NAME}/consul-dns/cdns_ca" > /tmp/cdns_ca.private.pem
credhub get -k certificate -n "/${BOSH_NAME}/consul-dns/cdns_ca" > /tmp/cdns_ca.cert.pem

credhub set -n "/${BOSH_NAME}/consul-dns/cdns_ca" -t certificate -c /tmp/cdns_ca.cert.pem -p /tmp/cdns_ca.private.pem \
  -r /tmp/cdns_ca.cert.pem

Rotate an Expiring Consul CA and Certificate

To rotate an expiring Consul CA and certificate, the following steps must be followed:

Duplicate Current CA

credhub get -k private_key -n "/${BOSH_NAME}/consul-dns/cdns_ca" > /tmp/cdns_ca.private.pem
credhub get -k certificate -n "/${BOSH_NAME}/consul-dns/cdns_ca" > /tmp/cdns_ca.cert.pem
credhub get -k ca -n "/${BOSH_NAME}/consul-dns/cdns_ca" > /tmp/cdns_ca.ca.pem

credhub set -n "/${BOSH_NAME}/consul-dns/cdns_ca_old" -t certificate -c /tmp/cdns_ca.cert.pem \
  -p /tmp/cdns_ca.private.pem -r /tmp/cdns_ca.ca.pem

Regenerate Current CA

To prevent CA rotation every year, the duration parameter must be changed.

credhub generate --duration=365 -n "/${BOSH_NAME}/consul-dns/cdns_ca" -c a9sConsulCA --is-ca -t certificate

Redeploy Environment (With Old CA, New CA, and Old Certificate)

consul-dns

The following Ops file must be applied to the consul-dns deployment, and the consul-dns deployment must be redeployed.

To prevent SSL certificate rotation every year, the duration parameter in the following Ops file must be changed.

note

<bosh-director-name> is replaced with the director name from step 1 in all following Ops files.

- type: replace
  path: /instance_groups/name=consul/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"
- type: replace
  path: /variables/name=~1cdns_ssl/options/duration?
  value: 365

data-services

The following Ops file must be applied to the x-service deployments, and the x-service deployments must be redeployed. The templates-uploader errand and the force_deployment_updater errand must be run after the deployments have been redeployed.

- type: replace
  path: /instance_groups/name=spi/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"
- type: replace
  path: /instance_groups/name=broker/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"
- type: replace
  path: /instance_groups/name=deployer-api/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

- type: replace
  path: /instance_groups/name=templates-uploader/jobs/name=template-uploader/properties/template-uploader/template-vars/~1cdns_ssl.ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

# delete for a9s Prometheus
- type: replace
  path: /instance_groups/name=service-dashboard/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

# force update instances
- type: replace
  path: /instance_groups/name=force_deployment_updater/jobs/name=deployment-updater/properties/strategy?
  value:
    update:
      instance_type: provisioned

The a9s Prometheus deployment does not contain a service dashboard with a running Consul job. The Ops entry with this replacement must be deleted: /instance_groups/name=service-dashboard/jobs/name=consul/properties/consul/ssl_ca

The force update instances Ops entry guarantees that all instances will be updated even though instances are not outdated. The Ops entry is necessary, in the case of a Consul certificate rotation, because it is not possible to guarantee that the instances are outdated, once only a CredHub value is changed.

a9s-pg

The following Ops file must be applied to the a9s-pg deployment, and the deployment must be redeployed.

- type: replace
  path: /instance_groups/name=pg/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

backup-service

The following Ops file must be applied to the backup-service deployment, and the deployment must be redeployed.

- type: replace
  path: /instance_groups/name=backup-manager/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"
- type: replace
  path: /instance_groups/name=backup-monit/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

service-guard

The following Ops file must be applied to the service-guard deployment, and the deployment must be redeployed.

- type: replace
  path: /instance_groups/name=guard/jobs/name=consul/properties/consul/ssl_ca
  value: "((/<bosh-director-name>/consul-dns/cdns_ca_old.ca))((/<bosh-director-name>/consul-dns/cdns_ca.ca))"

Delete Consul Certificate

credhub delete -n /cdns_ssl

Redeploy Environment (With Old CA, New CA, and New Certificate)

The environment must be redeployed after the Consul certificate has been deleted.

warning

• The appropriate Ops file from step Redeploy Environment (With Old CA, New CA, and Old Certificate) must be applied to the corresponding deployment.
• The Service Instances must be updated using the force_deployment_updater errand.

If some Service Instance is still using the old Consul certificate after using the force_deployment_updater errand, a command from Update a Specific Service Instance can be used to trigger the updating process for this service.

Redeploy Environment (Without Old CA)

The environment must be redeployed after the Consul certificate has been deleted.

warning

• In this case, the Ops file from step Redeploy Environment (With Old CA, New CA, and Old Certificate) must not be applied.
• The service templates must be uploaded with the templates-uploader errand, and the Service Instances must be updated using the force_deployment_updater errand.

Update the CF Gorouter Request Timeout

When using the Cloud Foundry (CF) Gorouter, it is necessary to be aware that there is a timeout, and it might reject requests exceeding this timeout.

By default, the timeout is 15 minutes. Therefore, requests longer than this timeout will be canceled. In order to change the default timeout, it is required to update the Cloud Foundry deployment (specifically the gorouter job).

The request_timeout_in_seconds property needs to be added or modified in the Cloud Foundry deployment manifest when the routing BOSH release properties are updated. This value must be configured as a number (integer) representing the timeout in seconds.

Example:

(...)
  jobs:
  - name: gorouter
    properties:
      request_timeout_in_seconds: 3600 # 1 hour
(...)

Network Update

a9s Data Services support network update and relocation of the pool of addresses available for the Service Instances. The operator can update the BOSH Cloud Config and apply the changes by updating the Service Instances with the Deployment Updater Errand.

warning

The a9s Data Services do not support network changes that affect the majority part of the nodes in a cluster without downtime. This happens because after the first node is updated, it may have different addresses for the remaining part of the cluster, and when the second node goes down for update, no part of the cluster will have a quorum to continue working as it should. Therefore, it is recommended to use at least 3 availability zones with distinct network definitions for each one, and during the update, one availability zone should be updated at a time.

For example:

* Modify the az1
* Apply the update to all Service Instances
* Modify the az2
* Apply the update to all Service Instances
* Modify the az3
* Apply the update to all Service Instances

a9s KeyValue Network Update

The a9s KeyValue cluster instances are affected directly when updating the network because it violates the Cluster Deployment Update Strategy principle, which requires all nodes (primary and secondary) to be healthy during the cluster update. The network update process updates one availability zone at a time. This violates the principle because the node with a new network IP at the availability zone being updated will be unreachable during the cluster update, and the cluster will not be fully healthy for some time. However, the 2 other nodes are still accessible.

note

Because the network update updates all availability zones at once, it causes a large downtime, since all nodes will have different IPs. Hence, the cluster will not have a quorum and the KeyValue deployment will be unreachable until all nodes are updated.

The best way to do the update:

• Update one availability zone at a time.
• The a9s KeyValue stop-cluster-update-on-failure property must be set to false because, if a node update fails while the cluster is not healthy, the deployment should not fail and should continue updating the cluster.

Known issues:

• The deployment update might delay a little bit because the cluster will not be fully healthy and it will reach the cluster-update-node-timeout. The time will depend on the cluster-update-node-timeout value and node update order determined by BOSH.
• It might cause data loss because of the stop-cluster-update-on-failure property set as false. Read the Cluster Deployment Update Strategy section to understand better.