a9s Parachute v2
a9s Parachute v2 uses Data Service specific actions to stop data ingestion as a measure to avoid exposing the a9s Data Services to a full disk.
a9s Parachute v2 uses monit to watch the ephemeral and persistent disk usage of an a9s Data Service instance. When the
disk usage threshold hits a configured limit, it executes a lock script to lock the a9s Data Service Instance.
During pre-start of the a9s Parachute v2 process, an unlock script is executed to revert the actions that have been
taken to prevent further data ingestion.
The following a9s Data Services are using a9s Parachute v2:
- a9s KeyValue 8
- a9s LogMe2
- a9s MariaDB
- a9s Messaging 4
- a9s MongoDB 7.0
- a9s PostgreSQL 15
- a9s PostgreSQL 17
- a9s Search
Configuration
The a9s Parachute v2 can be configured via the following BOSH properties:
| Property | Default Value | Description |
|---|---|---|
plugins | A list of plugins that can be used by a9s Parachute. For each plugin the following settings must be set: name, plugins_path, username, password | |
ephemeral.usage_limit | 80 | Limit in percent for the ephemeral disk usage. 0 deactivates the check. |
ephemeral.fault_tolerance.times | 5 | The number of test failures for the ephemeral disk before an error event is triggered. |
ephemeral.fault_tolerance.cycles | 10 | The amount of cycles in which test failures are counted for the ephemeral disk. |
persistent.usage_limit | 80 | Limit in percent for the persistent disk usage. 0 deactivates the check. |
persistent.fault_tolerance.times | 5 | The number of test failures for the persistent disk before an error event is triggered. |
persistent.fault_tolerance.cycles | 10 | The amount of cycles in which test failures are counted for the ephemeral disk. |
Observability
There are multiple option to monitor a9s Parachute. Below, you can find a list of the different options:
- You can receive a9s Parachute related metrics via a9s Logstash. For more information, see a9s Parachute Metrics
- You can fetch a9s Parachute related metrics via the API v1
- a9s Parachute can be monitored via the a9s Service Dashboard
Unlock a Service Instance Manually
To manually unlock a Service Instance, the following steps are necessary:
- SSH into one node of the affected Service Instance via
bosh -d <deployment_name> ssh <instance_name>/<index> - Become root via
sudo -i - Remove the a9s Parachute lock file via
rm -f /var/vcap/sys/run/a9s-parachute/a9s-parachute-activated - Unlock the Service Instance via
/var/vcap/jobs/a9s-parachute/bin/unlock
Keep in mind that the a9s Parachute doesn't clear up disk space, so you have to free up the affected disk manually.
a9s Data Service Specifics
a9s Parachute stops selected Monit processes to prevent further data ingestion, while keeping certain processes alive to ensure the VM remains accessible. The table below lists which processes are stopped and which remain running when a9s Parachute 2 is active.
| Data Service | Monit Processes Stopped | Monit Processes Kept Alive |
|---|---|---|
| a9s KeyValue | valkey, backup-agent | consul, nginx, logstash, a9s-bee |
| a9s LogMe2 | opensearch, backup-agent | consul, nginx, logstash, a9s-bee |
| a9s Messaging | rabbitmq, backup-agent | consul, nginx, logstash, a9s-bee |
| a9s MongoDB | mongodb, backup-agent | consul, nginx, logstash, a9s-bee |
| a9s PostgreSQL | postgresql-ha, postgresql-info-webservice, backup-agent | consul, nginx, logstash, a9s-bee |
| a9s Search | opensearch, backup-agent | consul, nginx, logstash, a9s-bee |
Depending on the Data Service, other service-specific actions may be applied. Below, you can find a brief overview of these actions.
a9s MariaDB
When the disk usage threshold hits a configured limit, a lock script is executed and sets the whole database server to
read-only mode. This is done via the command SET GLOBAL read_only = ON;.
Additionally, a9s Parachute creates its lock file /var/vcap/sys/run/a9s-parachute/a9s-parachute-activated to indicate
that a9s Parachute was activated.
When the a9s Parachute process starts, the lock file is deleted and the read-only mode of the database
server is reverted via the command SET GLOBAL read_only = OFF;.
Resource Considerations
For more information, see a9s Parachute Resource Considerations.