a9s Backup Service
This section documents the a9s Backup Service, its components and its use cases.
a9s Backup Process
While the a9s Backup Framework supports AWS S3 as a backup store, this does not extend to every AWS S3 Storage Class. Currently, the supported AWS S3 storage class is S3 Standard.
Creating a New Backup
By creating a new backup of an active service instance using the a9s service dashboard, the endpoint /backup_agent/backup of the a9s backup manager will be triggered.
Afterwards, the backup manager starts the backup process of the selected service instance by inserting the backup job into the backup agent queue. The backup agent, co-located with the deployment of the service instance itself, start to create the backup by connecting to the virtual machine(s) of the service instance.
Subsequently, the created backup will be first compressed and encrypted using the encryption key, defined by the user using the a9s service dashboard. Finally, the backup will be uploaded to the configured backup store (e.g. AWS S3).
Restore a Backup
By restoring a service instance using the a9s service dashboard, the endpoint /backup_agent/restores of the a9s Backup Manager will be triggered.
The backup manager handles the incoming request and triggers the restore of the service instance by inserting the restore job into the backup agent queue.
Once the restore job has been started, the backup agent downloads the selected backup from the configured backup store and prepares the backup by decrypting and unpacking the downloaded archive file.
Finally, the backup agent imports the backup data into the running service instance by connecting to the virtual machine(s) of the service instance.
Concerns
Memory Usage
The a9s Backup Agent has a monit alert limit for the memory usage:
if totalmem > <backup-agent.thresholds.alert_if_above_mb> Mb then alert
To be able to see if the backup process used too much memory, the monit logs can be
examined. They are located at /var/vcap/monit/monit.log. If the process is exceeding
the memory alert limit, monit will add a newline to this file.
If that happens regularly, it is suggested to update the Service Instance to a plan with
more memory available.
The current memory usage can be seen with the metrics.
Backup Storage With Self-Signed Certificates
This affects a9s Search services.
a9s Service Instances are not able to communicate to a backup storage with self-signed certificates out of the box, as the service instance does not trust the certificates by default. Thus, once the service instance tries to communicate, this will be rejected, since certificate is recognized as "untrusted".
In order to allow the service instance to trust the self-signed certificates, they must be present in the Java Keystore, as OpenSearch uses Java.
To solve this, a9s Data Service has a mechanism that allows adding self-signed certificates to the Keystore, as long as these certificates are added to the BOSH Director's trusted certificates.
As such, the Platform Operator must add the self-signed certificates to the BOSH Director's trusted certificates by following the official BOSH documentation - Installing Certificates on VMs. After applying the changes, the new a9s service instances will trust, and thus be able to use, your self-signed certificates.