User Provided Certificates
By default - if no certificates have been provided by the Application Developer - the certificates configured by the Platform Operator are used. However, when creating or updating a service instance using a TLS/SSL service plan, it is possible to provide your own X.509 certificate.
This section describes how an Application Developer can provide their own certificates - from here on called user provided certificates - for a service instance using a TLS/SSL service plan of an a9s Data Service.
Please be aware that different services may have special limitations which are described explicitly in the Limitations section.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
Requirements
When you want to configure your own certificates for your service instance, you need the following properties
| Property | Note |
|---|---|
certificate | The single certificate (server/leaf certificate) that is used for the service instance. The X.509 certificate MUST be in PKCS#1 format and PEM encoded. |
private_key | The private key for the certificate. The private key MUST be in PKCS#1 format and PEM encoded. The private key MUST NOT be encrypted. |
ca | The ordered certificate chain for the certificate. The individual X.509 certificates MUST be in PKCS#1 format and PEM encoded. The first certificate MUST be the CA certificate that signed the certificate in certificate. The last certificate MUST be the root CA certificate. The certificate chain can also only consist of the root CA. |
Common Name And Subject Alternative Name
The Common Name (CN) of the user provided certificates MUST be set to *.node.dc1.<iaas.consul.domain>.
The Subject Alternative Name (SAN) of the user provided certificates MUST contain the DNS name entries
*.node.dc1.<iaas.consul.domain> and *.service.dc1.<iaas.consul.domain>.
This is needed as it is used to verify all internal communication between the cluster nodes. Furthermore as the service
instance hostnames are propagated to the applications, via the service bindings, they provide a way to verify the server
they are connecting to.
In the example below, it is possible to see the hostnames of a9s Search cluster service instance; which MUST be covered in the user provided certificates as stated above.
d75caca-os-0.node.dc1.<iaas.consul.domain>
d75caca-os-1.node.dc1.<iaas.consul.domain>
d75caca-os-2.node.dc1.<iaas.consul.domain>
d75caca.service.dc1.<iaas.consul.domain>
All a9s Data Services with a vendored dashboard MUST contain these additional SAN entries:
*.<dashboad_hostname>.<iaas.cf.system_domain>*.<iaas.cf.system_domain>.
This affects the following a9s Data Services:
- a9s Messaging
<dashboad_hostname>=a9s-rabbitmq-dashboard
- a9s LogMe2
<dashboad_hostname>=a9s-logme2-dashboard
The <iaas.consul.domain> and <iaas.cf.system_domain> are part of the environment configuration and it is set by the
Platform Operator during the environment installation. Please contact your Platform Operator for more information.
Configuration of User Provided Certificates
A service instance can only be configured with user provided certificates during its creation.
You can use the cf create-service command with the -c parameter when using the cf CLI to provide a file containing
the TLS/SSL specific configuration parameters in a valid JSON object. Optionally you can provide these parameters as a
valid escaped JSON object string. Following best practices, we recommend using a JSON file to provide these parameters.
The TLS/SSL specific configuration parameters that must be specified using the -c parameter are:
| Key | Value | Description |
|---|---|---|
tls.<cert_name>.certificate | A PEM-encoded X.509 certificate | The leaf certificate to be deployed to your service instance. |
tls.<cert_name>.private_key | A private key | The private key for the certificate specified in tls.<cert_name>.certificate. |
tls.<cert_name>.ca | A PEM-encoded X.509 certificate | The CA certificate with which your certificate has been or will be signed. |
The placeholder <cert_name> in the TLS/SSL specific configuration is Data Service specific and the following values
are required for the individial Data Service:
| Data Service | Certificate Names |
|---|---|
| a9s PostgreSQL | postgresql |
| a9s MongoDB | mongodb |
| a9s Messaging | rabbitmq |
| a9s MariaDB | client, server |
| a9s Search | opensearch |
| a9s Redis | redis |
| a9s LogMe2 | opensearch, opensearch_dashboards, fluentd |
| a9s MySQL | client, server |
You can use the same certificate, private_key and ca for multiple service instances, and in the case of a9s LogMe2
and a9s MariaDB even for the different use cases in the same service instance.
Example
In the following example, we will create an a9s PostgreSQL Service Instance using an SSL plan and a X.509 certificate.
First we will have a look at the structure of the required JSON file and its content.
Keep in mind that the content must be valid JSON and therefore you need to escape the PEM encoded X.509 certificate file
or private key; which you can do, for example, by using awk:
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' <path to your PEM-encoded certificate file>
The JSON file should look like the example below:
{
"tls": {
"postgresql": {
"ca": "-----BEGIN CERTIFICATE-----\nMIIFbjCCA1YCCQCjw+EXPZsMZjANBgkqhkiG9w0BAQsFADB4MQswCQYDVQQGEwJE\nRTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJicnVlY2tlbjEWMBQG\nA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAYBgNVBAMMEXRlc3Qu\nYW55bmluZXMuY29tMCAXDTIyMDcyODEwMjk1OFoYDzIwNTAwODEwMTAyOTU4WjB4\nMQswCQYDVQQGEwJERTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJi\ncnVlY2tlbjEWMBQGA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAY\nBgNVBAMMEXRlc3QuYW55bmluZXMuY29tMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A\nMIICCgKCAgEA7KwHFPQrW9oqzheKK2Vo0TqEexlmN92v6u7Rn6FIZD4FFOPqyg9q\n1P/FQld3UUaMio6l7wxyCCzojB5cWpcBrWWb6Lcr4e0aXLOi8B6vVixKftZt0jcE\nQCv8nPHacYcHXw4LwhVMVedXvhwtLbopRVOuRdbdLkfWujew1gnafQ/C2+7ozv7Z\n/1ADXCREjQ4KlPnn1d87ZK7muR4TjZFEh0ZZ7gA3UjimywzQw8wCUIMBXDH8OInr\nYKHY4+yyq7oaliM68H7WYfJURSQZ/Zv1jVCYLs1AXXWMYSy4hnbzGdLrOfg62Lcc\noYrnJc6yV4e0xOXniCYCy9TxOD5kwXkm3yKO+adpD3TR7BqLCKG9KyN9Ih8vrQlC\ni4azZnMQ2EGlEDoV6wN35etEMocC4hB8TRUYQ1KY3TTVSKb9wPAn/4cAuh6F+cPK\nmq19D+z+PWlVaQHwPFwO2p2koTIycd6gjctHlGS/lAWPsoI1nnTtZ2u5PX3r5/B0\n144XuFTDNmI+7ch3jcggIU6hWcWd4qZ++UE1LltuhTGpzqPw/3Vt9TNSkTvpy8B/\nasZPKPZiDkwFsQQY385dtvkCobpJYNC3wtHnr/t4q+gklYR5mO4xMDgNf+bKjTuI\n0Iut0Aj+0uoJxFvTTfLM4EtPLlW2z7YPkY0A4+OaFV8LjPO1ZPyFyrMCAwEAATAN\nBgkqhkiG9w0BAQsFAAOCAgEAfLXs0mymh4jdUc/ghOdXQFkCqMg5E5wSWlfEgzC0\n8oJwAw4TLjynvrMpZtms6DWf0o63NtrNoQPMcJ6+R8oD3Vqn7Q7BDR+F+cKg2c0e\n44xocV5o5vX8xwZ63GScS+ZSIunGvP537XV5Lzyv3dI9e+qjs893mzgAO02ACT34\nTXlDn/bTKAmOOqRaj6yP2HsInCsJetB5pHi3R3buyy9DbAXt6qtt+eySv1U7JU81\nipMoUbv2PdLLfgfJOVGCldpduOwsyFTwHB78MBiT755QWBalYJ/sw47OaYSDFKCX\nmdO913OpXkUZfF3l4l/CkXEUsY6+pMSCoguwgwln3eVPfmNjqG4LtZGOgKMepmcG\nkRNSGsvVKXglcIsPgogkOxUf569FEK5kEtYw0cMn8MCrv358H1xUFXVcitSLKpGM\n3klsZ1tU/ZnwxbSLcuozHhd+YOjy9KSRmUZCA+ORKVhrweWpugbnv0ZRKzXGLC9a\ny1Rj//v5jWMiEAyZNjeSnCNfLx3jlovR3SjcnZZFrVidQi8cZCnAMzzVArnHi/FI\nxpkP5EAqh8q7DmrMY5X11eEuyCmbgPmjji/IVcOJ3NC9hZeOIiSazU3CvDEX6Blm\nmL/NTgnwTnQn5dKSahQgLtFaIve4VNP4aPFXgGdYScd6U8Luo34tD1EWg45kXnkG\nA38=\n-----END CERTIFICATE-----\n",
"certificate": "-----BEGIN CERTIFICATE-----\nMIIEaTCCAlECCQCjNrxOOvCTgzANBgkqhkiG9w0BAQsFADB4MQswCQYDVQQGEwJE\nRTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJicnVlY2tlbjEWMBQG\nA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAYBgNVBAMMEXRlc3Qu\nYW55bmluZXMuY29tMB4XDTIyMDcyODEwMzE0N1oXDTM2MDQwNTEwMzE0N1owdTEL\nMAkGA1UEBhMCREUxETAPBgNVBAgMCFNhYXJsYW5kMRUwEwYDVQQHDAxTYWFyYnJ1\nZWNrZW4xFjAUBgNVBAoMDWFueW5pbmVzIEdtYkgxCzAJBgNVBAsMAkRTMRcwFQYD\nVQQDDA4qLmFueW5pbmVzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBALBfuvTiCfj0eJ03UaDLPT91qpZvsvKeML9TZ5C5MKG5kOp4GbAl+qcg9SEY\nouYD1niKfrpi2CN1iT7xVhJTQImSxfQO0GXumGlOz/PwCvBm2bTHBzBZl389sDGA\n6od00ailkv1On4DBfMD6okEWgWyWLZ8PdMDcat0SoKxwt4P4ddtvhtYNSNtiM7Po\nag1Lv0Ocha6B1kD8GbJY/murLrCW7ZBUF3i2p075imfEwJiu9U34t79rnPhzaHAX\nshL9VAEFidNAXahK4q2EudjFum+Bd5Ud0EdIfzz1Dn9EUYfFn0mPF+CCfIu1x8n6\nFOwz5Q/0+NRAIHAy9waB8vmP/e8CAwEAATANBgkqhkiG9w0BAQsFAAOCAgEAwfkC\nH9e8eQX22gpzCp282YFbCq5Zl0eKONZ3hDm0G+ej1hVUIomKVrwJdz8fcYSf5KZw\nn95HuMPNL5w8SgJ3gR5ZNegWlNYL/sKhIk3NDSyv6helSs6dbufy2CMXg51jlpUp\nJWy4Xa4VLv5kPHxi5gZili4eYCh9Z97kxb8K6P3wXnUOhpyOj4LvzyIwXIxG7gXj\nfp0sQ5+GrlwBhNKnPnF3h8fWU4iQjiRp9hwixKO2uetyc3t47U4FBT4NyYuvGbR/\nVYlekHPCCFn7o31p1VJ/qYGqaMoZEfJZv6t2w9WMf4BpNXnyJ1TFCRSJ0QtZC35E\n4ZuHf9O4rVIBTQL0G7edHbXpxUh0au6uvk8Z0QtwDi8p+1gCg85DICUOmqPxALGi\nDzMjDApWMWyru5PeAC6BnVWuF/Gwhq46X4roQZJLke8eAIdYbptCAltuwlIz1foj\n2DsRESyzcyzk/wziSg3Ai42r1dEODcMd3Na0339EpdG/lbK4+hzPcC4Tv/iUH1Xh\nGnyuH+gi40+92VDSGbvFLTlb/alNBY3TWAjJMCrhE3WgcxVTdFkBS/gV3bPghhaw\nVL/2Nv7oTllDWwAZoJ/WKKu0s1a129iwIJ0xkK0NlDkY13DgTQIngZiXTi3Nq9ct\nB4O8e7EElIvg+68UN7u3nyGI9yzVHqoBHjzwIcw=\n-----END CERTIFICATE-----\n",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAsF+69OIJ+PR4nTdRoMs9P3Wqlm+y8p4wv1NnkLkwobmQ6ngZ\nsCX6pyD1IRii5gPWeIp+umLYI3WJPvFWElNAiZLF9A7QZe6YaU7P8/AK8GbZtMcH\nMFmXfz2wMYDqh3TRqKWS/U6fgMF8wPqiQRaBbJYtnw90wNxq3RKgrHC3g/h122+G\n1g1I22Izs+hqDUu/Q5yFroHWQPwZslj+a6susJbtkFQXeLanTvmKZ8TAmK71Tfi3\nv2uc+HNocBeyEv1UAQWJ00BdqErirYS52MW6b4F3lR3QR0h/PPUOf0RRh8WfSY8X\n4IJ8i7XHyfoU7DPlD/T41EAgcDL3BoHy+Y/97wIDAQABAoIBAD8x7dD0dNJI9jaO\nrabJa6ajSH4ig6G+4ISNI9yNHkm0LaJ1ae7djNP5URuMskFsOppyNTofVIZQBN8S\npd5nCgbBGkB6Vl1PgjipToV/CsxcbcECVycR0uB6f/kc0eu7BaxBt0sfsrL1oPec\nViqwH6uCm9IUPack0v1nQT7IHFkiQwE9Sg8FDgXoUtaIbzb7oRd/dFStUap6lrvC\nNscjpItU0m9hdwTRUqjqv/Cg066TdYEFdtdL9F8oOtsu0Ix3SkQY3xkUS+9z0FZC\nljHOv3nzLIOBnn0BfXqy2am/evwkwOw8PzDp5RP6FStEWJgwzINvPD77fMrhy511\ndYhtDsECgYEA1jpMlKSYJ674z07Dqk1wuM/joY8EZHRVnp7kWtzOtHcRZjG1Ab/R\nzvEtDr09t1mgBHfC0xTFkzntTj3hTMuKNmqP0ycPL/k5HGdGyCuW7Al7JLweUlbN\nNEG/adxn5EI9QFjs7RmfVFDBlc8yUbwIZSmSyqb5b4fjB77I/6me+n8CgYEA0sPd\n8btgiQl0yZmAsu9hAnMbE3rPB/cfIWoYGN7KNY2xEuQKcW0v2z3th0td44KdxwiL\nlY26V1USzY/MbnHRBGOZw6WBQNSYw9TfjMN6XVv84KGXv7cM31rhsascvY7L+SX/\nKsk69jEQZaPlnRGsPoHkVOBgZnySlXpW6L0V5JECgYBhpVEir6Nq8yYV2CD3jzTC\nCIAJM9ccsqoUEvijMeJF/7++hQmsMnK/kM6o2Tk4SXHWl4AGFoG4Cb9Q2oPHLT9i\nblAAPt19UGvntmtc8gFotSmcJOLtRQNjvlQxPHVeZZLlsaLMr9Ef7W9PiZG2D68D\n7V8rX9ByNc4VybB0WCxOoQKBgQCo83fbyh/FWj4zSQjA52E1bH387Io/UVq0F27/\nqMAhk9apVQIGEMe3EPpyZPj7Yn4FyZTTKsyAK2MqQyZWNoid4xWknuxwwrs+6ErX\ndO+HHTEmBIM3nI4GEb0wBgHA9lIOA+Z7LtpD9eq1/18VvM/9P3SAkWjVXvDGQE6g\nvMpq8QKBgEaFtGhwuPe5/j6HsOJPvOgMPjM3LdQ18nO/NVDNQE8A0KftCdaioAy8\n5wGi2lDy1zmb+dZEMR9rEZpemNeE4jHSMOOPSwz4ZAhb2xpKymbtl/cwSrkiFeoh\n5SCm+DPRKOnxsj+NC4U9/Js8YwribuEI4MOdlOjM/hNxseDCEguE\n-----END RSA PRIVATE KEY-----\n"
}
}
}
You can use the following command to create a service instance with the mentioned JSON file:
cf create-service a9s-postgresql13 postgresql-single-small-ssl my-pg -c ssl_config.json
Creating service instance my-pg in org organization / space test as user@example.com...
OK
Create in progress. Use 'cf services' or 'cf service my-pg' to check operation status.
Rotation of User Provided Certificates
Leaf Certificate Rotation
As certificates for service instances with SSL plans expire, there is a need to rotate them regularly, and user provided leaf certificates are not an exception. The rotation of the user provided leaf certificate and its key can be done by following these steps.
This will only work if the current CA certificate is valid, not expired and it is used to sign the new leaf certificate. This process may cause some downtime in single deployments, as the service instances are updated with the certificate information. If the CA is still valid, there is no downtime on cluster deployments.
Providing your own certificates allows you to be more in control, but in exchange you are responsible for the rotation, since the automation doesn't work with this option, as it will return the same certificate each time it is executed. Additionally the forced rotation functionality is not supported for user provided certificates.
Step 1: Update the leaf certificate
First, add the new leaf certificate and the corresponding private key to your JSON file. Ensure that these are properly parsed, as mentioned before. Your config file should look like:
{
"tls": {
"postgresql": {
"ca": "<old_ca>",
"certificate": "<new_leaf_certificate>",
"private_key": "<new_private_key>"
}
}
}
Step 2: Redeploy your service instance
You can apply the new leaf certificate and private key to your SSL service instance by executing the following command:
cf update-service my-pg -c updated_ssl_config.json
CA Certificate Rotation
Just as Leaf certificates expire for service instances with SSL plans, so do CA certificates. Unfortunately the rotation of a CA certificate is not as easy as the rotation of a leaf certificate because simply replacing it will lead to broken clustered service instances.
To update the service we recommend to use the following method for a rotation with minimal downtime.
Step 1: Append the new CA to the old CA
Concatenate the old and the new CA (first old CA, then new CA) and store it in a temporary file to make it easier to parse them properly. This allows the service instance's nodes and the clients to trust the wildcard certificate presented by the service instance that can be either signed by the new or the old CA.
Add the concatenated CAs to your JSON file. Ensure that these are properly parsed as mentioned before. Your config file should look like:
{
"tls": {
"postgresql": {
"ca": "<concatenated_cas>",
"certificate": "<leaf_certificate>",
"private_key": "<private_key>"
}
}
}
Step 2: Roll out both, the old and the new CA, to your service service instance
You can apply the concatenated CAs to your SSL service instance by executing the following command:
cf update-service my-pg -c updated_ssl_config.json
Before proceeding with the next step, all service bindings MUST be renewed to propagate the new CA to the clients, unless the new CA is already trusted by the client's runtime.
Step 3: Replace the leaf certificate
Add the new leaf certificate and the corresponding private key to your JSON file. Ensure that these are properly parsed, as mentioned before. Your config file should look like:
{
"tls": {
"postgresql": {
"ca": "<concatenated_cas>",
"certificate": "<new_leaf_certificate>",
"private_key": "<new_private_key>"
}
}
}
Step 4: Roll out the new leaf certificate to your service instance
You can apply the new leaf certificate and private key to your SSL service instance by executing the following command:
cf update-service my-pg -c updated_ssl_config.json
Step 5: Remove the old CA
Add the new and only the new CA to your JSON file. Ensure that these are properly parsed, as mentioned before. The old CA is no longer needed since all existing service instances are only using the new wildcard certificate that is signed by the new CA.
{
"tls": {
"postgresql": {
"ca": "<new_ca>",
"certificate": "<new_leaf_certificate>",
"private_key": "<new_private_key>"
}
}
}
Step 6: Roll out only the new CA to each service instance
You can apply the concatenated CAs to your TLS/SSL service instance by executing the following command:
cf update-service my-pg -c updated_ssl_config.json
Before proceeding with the next step, all service bindings MUST be renewed to propagate the new CA to the clients, unless the new CA is already trusted by the client's runtime.
Limitations
Deleting User Provided Certificates
The deletion of user provided certificates is not supported by the a9s Data Services Framework. As a consequence, if you want to use certificates provided by the Platform Operator you have to create a new service instance using a TLS/SSL service plan and migrate your data.
Migrating User Provided Certificates
The a9s Data Services Framework does not support the certificate migration for a service instance using a TLS/SSL service plan from user provided certificates to certificates provided by the Platform Operator and vice versa.
Limitations for MariaDB
MariaDB 10.4 has a pre-defined structure for how it expects the different certificates. While the structure is described in length in the vendors' documentation, for our intents and purposes, this is the most relevant piece of information:
There are three certificates that you need to create in order to secure Galera Cluster: the Certificate Authority (CA) key and cert; the server certificate, to secure mysqld activity and replication traffic; and the client certificate to secure the database client and stunnel for state snapshot transfers.
Due to the nature of our automation, you have to specify a different certificate for client and intercluster.
Limitations for OpenSearch
OpenSearch expects that the certificates' private keys are in the format PKCS#8. This restriction applies when
user provided certificates are using.
You can check the vendor's documentation for more information: