a9s Parachute v1

The a9s Parachute v1 is intended to ensure that certain services are stopped properly in the event of a disk filling up, in order to prevent data loss. For this purpose, it runs on every a9s Data Service Instance. The a9s Parachute v1 monitors both the ephemeral and the persistent disk, and as soon as the disk usage exceeds a configured threshold, it stops the configured services.

a9s Parachute v1

The following a9s Data Services are using a9s Parachute v1:

• a9s Prometheus

Configuration

The a9s Parachute can be configured via the following BOSH properties:

Property	Default Value	Description
a9s-parachute.ephemeral.usage_limit	80	Limit in percent for the ephemeral disk usage. 0 deactivates the check. Setting this value > 90 is considered unsafe and could lead to a system crash.
a9s-parachute.persistent.usage_limit	80	Limit in percent for the persistent disk usage. 0 deactivates the check. Setting this value > 90 is considered unsafe and could lead to a system crash.
a9s-parachute.services_to_stop		An array with the names of the monit processes to stop when the usage limit for a disk is reached. If no process names are specified, an error is raised.

Observability

To make the a9s Parachute observable, the following lock files are created in the /var/vcap/sys/run/a9s-parachute directory on the VM of the respective node of the Service Instance when the a9s Parachute is triggered:

• a9s-parachute-activated: Indicates that the a9s Parachute has been triggered and the configured services have been stopped
• a9s-parachute-activated-ephemeral: Indicates that the a9s Parachute for the ephemeral disk has been triggered
• a9s-parachute-activated-persistent: Indicates that the a9s Parachute for the persistent disk has been triggered

Note that if the a9s Parachute was not triggered, the above file will not exist and the directory /var/vcap/sys/run/a9s-parachute will be empty.

There are multiple option to monitor a9s Parachute. Below, you can find a list of the different options:

• You can receive a9s Parachute related metrics via a9s Logstash. For more information, see a9s Parachute Metrics
• You can fetch a9s Parachute related metrics via the API v1
• a9s Parachute can be monitored via the a9s Service Dashboard

Manually Restart a Service Instance's Node

To manually restart the processes of a Service Instance's node and re-enable the a9s Parachute so that it stops the configured processes again when a disk fills up, the following steps are necessary:

1. SSH into the affected node of the Service Instance via bosh -d <deployment_name> ssh <instance_name>
2. Become root via sudo -i
3. Remove all a9s Parachute lock files via rm -rf /var/vcap/sys/run/a9s-parachute
4. Enable the a9s Parachute checks via monit start ephemeral and monit start persistent
5. Start the stopped processes via monit start <process_name>

Keep in mind that the a9s Parachute does not clear up disk space, so the disk space must be freed up manually.

Resource Considerations

Every a9s Data Services Service Instance has an a9s Parachute process running. This process will stop the node process when the persistent disk hits the configured usage limit (default: 80%). This is valid for all Services.

The persistent disk is mounted using the ext4 file system, which reserves a portion of the file system (5% by default) for the root user to prevent, among other things, normal users (e.g. vcap) from filling the disk and crashing the system. Additionally, some Data Services such as a9s LogMe2 have their own built-in safeguards against disk exhaustion. Any non-root processes (e.g. postgres, logstash) may stall when the disk is almost full. Anything above 90% is considered unsafe and could lead to a system crash. A larger disk size should be considered instead.

The calculation for used disk space does not consider the reserved disk space; it is calculated over the overall disk usage.