a9s Smoke Tests
For each a9s Data Service, the smoke-tests
errand is included.
This errand can be used to verify the main a9s Data Service features work in your environment.
It ensures:
- Service Instances can be created.
- Service Instances can be bound to CF applications.
- Applications can use the a9s Data Service (e.g. insert data into a9s PostgreSQL Service Instance).
- Service Instances can be updated to a bigger plan (vertical scaling).
- Data is still in the database after vertical scaling.
- Data can be backed up and restored.
- The a9s Service Dashboard is working with the Service Instances.
- Metrics can be streamed from the Service Instances.
Important
The smoke-tests
errand tries to remove the services and applications upon its completion, but the smoke-tests
errand
cannot guarantee that there are no leftovers. It is the Platform Operator's responsibility to monitor the smoke-tests
CF org and remove leftovers. Leftovers might slow down the system and raise costs for the environment (for example in
AWS).
General Smoke Tests Properties
The following list contains the general configurations for the smoke-test
errand.
Parameter | Type | Default | Description |
---|---|---|---|
service-smoke-tests.service.type | string | null | Service type that is going to be tested (e.g. mongodb , postgresql , rabbitmq , redis ). |
service-smoke-tests.service.do_check_plan_update | boolean | false | boolean |
service-smoke-tests.service.do_check_migration | boolean | false | boolean |
service-smoke-tests.service.max_parallel | integer | 1 | The amount of plans that should be tested in parallel. By default all plans are executed sequential. |
service-smoke-tests.service.skip_ssl_validation | boolean | true | When set to true , create ssl validation is skipped. |
service-smoke-tests.service.asg_prefix | string | a9s-asg | The prefix for the Cloud Foundry ASGs created by the a9s CF Service Guard. |
service-smoke-tests.service.custom_parameters | JSON | {} | Custom parameters which are applied to all Data Services, e.g. '{"graphite": ["prometheus-platform.service.dc1.((iaas.consul.domain)):9109"]}' . |
service-smoke-tests.service.asg_prefix
The prefix for the Cloud Foundry ASGs created by the a9s CF Service Guard. When the value of the property asg_prefix
has been set in the
a9s CF Service Guard, the same value must
be set here.
service-smoke-tests.service.custom_parameters
With setting the custom_parameters
, it is possible to apply some custom parameters to the Service Instances during the
tests. The custom parameters will be applied to all Data Services which are being tested.
service-smoke-tests:
service:
custom_parameters: '{"graphite": ["prometheus-platform.service.dc1.((iaas.consul.domain)):9109"]}'
Retry Strategy
With the following configurations, the retry strategy for the different tests can be adapted.
Parameter | Type | Default | Description |
---|---|---|---|
service-smoke-tests.service.retries.retry_backoff | string | none | The common step retry strategy. Must be one of none , linear or exponential . none means for every step retry it waits the baseline time. linear means for every step retry it waits the retry count multiplied with the baseline. And exponential means for every step retry it waits the retry count to the power of 2 multiplied with the baseline. |
service-smoke-tests.service.retries.max_retries | integer | 60 | Maximum count of retries for a common step. |
service-smoke-tests.service.retries.baseline_backoff | integer | 10 | The baseline for the common strep retry strategy. |
service-smoke-tests.service.retries.service_step_max_retries | integer | 0 | Maximum count of retries for a service step. |
service-smoke-tests.service.retries.service_step_retry_backoff | string | none | The service step retry strategy. Must be one of none , linear or exponential . none means for every step retry it waits the baseline time. linear means for every step retry it waits the retry count multiplied with the baseline. And exponential means for every step retry it waits the retry count to the power of 2 multiplied with the baseline. |
service-smoke-tests.service.retries.service_step_baseline_backoff | integer | 10 | The baseline for the service step retry strategy. |
service-smoke-tests.service.retries.wait_service_is_ready_max_retries | integer | 260 | Maximum attempts while waiting for a service create or update operation to finish. A check for the status is executed every ~30s. |
Cloud Foundry properties
There are several possible configurations for the Cloud Foundry related behaviour of the smoke-test
errand.
Parameter | Type | Default | Description |
---|---|---|---|
service-smoke-tests.cf.restart_method | string | restart | Whether the service binding checker app should be restarted or restaged. The possible values are restart or restage . |
service-smoke-tests.cf.api_endpoint | string | null | The CF API endpoint. |
service-smoke-tests.cf.username | string | null | The CF username. |
service-smoke-tests.cf.password | string | null | The CF password. |
service-smoke-tests.cf.enable_service_access | boolean | true | When set to true the service, to be tested, will automatically enabled for the test organization in the CF marketplace, otherwise it must be manually enabled. |
service-smoke-tests.cf.org | string | null | CF organization in which the test apps will be pushed. |
service-smoke-tests.cf.space | string | null | The CF space in which the test apps will be pushed. |
service-smoke-tests.cf.assign_space_developer_role | boolean | true | When set to true the SpaceDeveloper role is assigned to the configured user in the configured organization and space, otherwise it must be manually assigned. The SpaceDeveloper role is necessary to execute the a9s Service Dashboard tests. |
service-smoke-tests.cf.apps_domain | string | null | The name of the CF apps domain where the test apps will be pushed. |
service-smoke-tests.cf.apps_use_ssl | boolean | true | When set to true the communication with the apps go through a SSL secured connection, otherwise they use a non SSL connection. |
service-smoke-tests.cf.assign_space_developer_role
When set to true
the SpaceDeveloper
role is assigned to the configured Cloud Foundry user in the configured
organization and space, otherwise it must be manually assigned.
The SpaceDeveloper
role is necessary to execute the a9s Service Dashboard tests. See also:
service-smoke-tests.service.do_check_dashboard_sso
The role is not removed from the used Cloud Foundry user afterwards.
Please be aware that if this property is set to true
, this potentially extends the permissions of the Cloud Foundry
user being used.
Per Service Configuration
The a9s Smoke Tests support a per-service configuration of what to test. To apply the per-service configuration, the
property service-smoke-tests.service.services
requires an array with an entry for each version of the Data Service
that should be tested. For example for the a9s PostgreSQL Data Service, the different versions like PostgreSQL 15,
PostgreSQL 17, etc. can be configured for testing. The hash for each Data Service has the following properties that can
be configured:
- name: <data-service-name>
plans: <plans-to-test>
plan_updates: <plan-updates-to-test>
configurations:
- custom_parameters: '{"some": "JSON"}'
do_check_basic_tests: 'ENABLED' or 'DISABLED'
do_check_dedicated_credentials: 'ENABLED' or 'DISABLED'
do_check_service_cases: 'ENABLED' or 'DISABLED'
do_check_backup: 'BASIC', 'EXTENDED', 'PITR' or 'DISABLED'
do_check_service_guard: 'ENABLED' or 'DISABLED'
do_check_dashboard_sso: 'ENABLED' or 'DISABLED'
do_check_backup_download: 'ENABLED' or 'DISABLED'
do_check_graphite_streaming: 'ENABLED' or 'DISABLED'
Using the following example, the a9s PostgreSQL 17 service can be tested using the default service configuration:
- name: ((postgresql_service.services.a9s-postgresql17.name))
plans: ((postgresql_service.services.a9s-postgresql17.plans-to-test))
plan_updates: ((postgresql_service.services.a9s-postgresql17.planupdates-to-test))
configurations:
- custom_parameter: '{"continuous_archiving" : "enabled" }'
do_check_dedicated_credentials: ENABLED
do_check_service_cases: true
do_check_plan_update: true
do_check_migration: true
do_check_backup: PITR
do_check_security_guard: true
do_check_dashboard_sso: true
do_check_graphite_streaming: true
Furthermore, it is also possible to manually configure the smoke-tests for a Data Service:
- name: a9s-postgresql17
plans:
- postgresql-single-small-ssl
- postgresql-replica-small-ssl
plan_updates:
- postgresql-single-small-ssl:
- postgresql-single-big-ssl
- postgresql-replica-small-ssl
- postgresql-replica-small-ssl:
- postgresql-replica-big-ssl
configurations:
- custom_parameter: '{"continuous_archiving" : "enabled" }'
do_check_dedicated_credentials: ENABLED
do_check_service_cases: true
do_check_plan_update: true
do_check_migration: true
do_check_backup: PITR
do_check_security_guard: true
do_check_dashboard_sso: true
do_check_graphite_streaming: true
Below is a detailed list of all available options for the configurations
field. By default every check is set to
DISABLED
and needs to be explicitly enabled.
Parameter | Accepted Values | Description |
---|---|---|
custom_parameters | a JSON string, e.g. '{"some": "JSON"}' | Applies custom parameters on each service instance of that Data Service during the tests. |
do_check_basic_tests | ENABLED , DISABLED | Checks that the service instance can be deployed and the Cloud Foundry app is able to bind to it. |
do_check_dedicated_credentials | ENABLED , DISABLED | Checks that multiple Cloud Foundry apps are able to bind independently of each other to the service instance. |
do_check_service_cases | ENABLED , DISABLED | Executes Data Service specific smoke tests. Currently, there is only a specific testing case for a9s Prometheus. |
do_check_backup | BASIC , EXTENDED , PITR , DISABLED | Tests the backup and restore procedure for all of the Data Services. It is possible to test different backup types (basic, PITR or extended backup cases). |
do_check_service_guard | ENABLED , DISABLED | Checks if the corresponding security groups are properly created for all of the Data Services. |
do_check_dashboard_sso | ENABLED , DISABLED | Tests the Single-Sign-On (SSO) functionality of the a9s Service Dashboard. |
do_check_backup_download | ENABLED , DISABLED | Checks that the backups can be downloaded via a9s Service Dashboard. |
do_check_graphite_streaming | ENABLED , DISABLED | Checks that the service instance metrics are properly streamed by the graphite server. |
a9s PostgreSQL
Migration Tests
You can enable/disable migration tests using the property service-smoke-tests.service.do_check_migration
. The tests
are enabled by default.
If enabled, the smoke tests will run migration tests. It is basically testing the custom parameter copy_from
that is
available to the Application Developer, see Migration documentation.
For every service that you want to run migration tests, you must set the plan_migrations
property. For example:
properties:
service-smoke-tests:
service:
do_check_migration: true
services:
- name: a9s-postgresql17
plan_migrations:
- origin_service: a9s-postgresql15
origin_plan: postgresql-single-nano
target_plan: postgresql-single-nano
The property plan_migrations
can be null
to not test anything for the current service.
Otherwise, it must be an array of objects with the following keys:
origin_service
: The service name where theorigin_plan
exists.origin_plan
: The plan name for the Service Instance we do want to fill with data and migrate data from.target_plan
: The plan name within the current service we create an Service Instance with and do want to migrate data to.
For every entry in the array the test will run the following high level test logic:
- Spawn up an Service Instance with service
origin_service
and planorigin_plan
and fill it with data. - Spawn up a new Service Instance with current service and plan
target_plan
using the custom paramcopy_from
. - Validate new Service Instance copied the data from the first Service Instance.
The service names you reference must exist under properties.service-smoke-tests.service.services
, otherwise the smoke
tests might fail during migration tests since the service offering must be enabled during smoke tests initialization.
a9s Prometheus
Custom Scrape Config Test
You can enable/disable the scrape config tests using the property service-smoke-tests.service.do_check_service_cases
.
The tests are disabled by default.
If enabled, the smoke tests will run tests to ensure that a scrape config can be configured in an a9s Prometheus Service
Instance. It is basically testing the custom parameter scrape_configs
, which is available to the Application
Developer, see Custom Scrape Configs.
To enable these tests, you have to set the property service-smoke-tests.service.do_check_service_cases
to ENABLED
,
e.g. with an Ops file.
---
- type: replace
path: /instance_groups/name=smoke-tests/jobs/name=service-smoke-tests/properties/service-smoke-tests/service/do_check_service_cases?
value: ENABLED
The custom parameter scrape_configs
must be modifiable by the Application User, otherwise the smoke tests might fail
during testing a custom scrape config since this property will be set during the smoke test run.
See Custom Parameter Configuration how to enable
the Application Developer to use custom parameters.