User Provided Certificates

By default - if no certificates have been provided by the Application Developer - the certificates configured by the Platform Operator are used. However, when creating or updating a service instance using a TLS/SSL service plan, it is possible to provide your own X.509 certificate.

This section describes how an Application Developer can provide their own certificates - from here on called user provided certificates - for a service instance using a TLS/SSL service plan of an a9s Data Service.

Please be aware that different services may have special limitations which are described explicitly in the Limitations section.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Requirements

When you want to configure your own certificates for your service instance, you need the following properties

Property	Note
certificate	The single certificate (server/leaf certificate) that is used for the service instance. The X.509 certificate MUST be in PKCS#1 format and PEM encoded.
private_key	The private key for the certificate. The private key MUST be in PKCS#1 format and PEM encoded. The private key MUST NOT be encrypted.
ca	The ordered certificate chain for the certificate. The individual X.509 certificates MUST be in PKCS#1 format and PEM encoded. The first certificate MUST be the CA certificate that signed the certificate in certificate. The last certificate MUST be the root CA certificate. The certificate chain can also only consist of the root CA.

Common Name And Subject Alternative Name

The Common Name (CN) of the user provided certificates MUST be set to *.node.dc1.<iaas.consul.domain>. The Subject Alternative Name (SAN) of the user provided certificates MUST contain the DNS name entries *.node.dc1.<iaas.consul.domain> and *.service.dc1.<iaas.consul.domain>. This is needed as it is used to verify all internal communication between the cluster nodes. Furthermore as the service instance hostnames are propagated to the applications, via the service bindings, they provide a way to verify the server they are connecting to.

In the example below, it is possible to see the hostnames of a9s Search cluster service instance; which MUST be covered in the user provided certificates as stated above.

d75caca-os-0.node.dc1.<iaas.consul.domain>
d75caca-os-1.node.dc1.<iaas.consul.domain>
d75caca-os-2.node.dc1.<iaas.consul.domain>
d75caca.service.dc1.<iaas.consul.domain>

All a9s Data Services with a vendored dashboard MUST contain these additional SAN entries:

• *.<dashboad_hostname>.<iaas.cf.system_domain>
• *.<iaas.cf.system_domain>.

This affects the following a9s Data Services:

• a9s Messaging
  • <dashboad_hostname> = a9s-rabbitmq-dashboard
• a9s LogMe2
  • <dashboad_hostname> = a9s-logme2-dashboard

note

The <iaas.consul.domain> and <iaas.cf.system_domain> are part of the environment configuration and it is set by the Platform Operator during the environment installation. Please contact your Platform Operator for more information.

Configuration of User Provided Certificates

caution

A service instance can only be configured with user provided certificates during its creation.

You can use the cf create-service command with the -c parameter when using the cf CLI to provide a file containing the TLS/SSL specific configuration parameters in a valid JSON object. Optionally you can provide these parameters as a valid escaped JSON object string. Following best practices, we recommend using a JSON file to provide these parameters.

The TLS/SSL specific configuration parameters that must be specified using the -c parameter are:

Key	Value	Description
tls.<cert_name>.certificate	A PEM-encoded X.509 certificate	The leaf certificate to be deployed to your service instance.
tls.<cert_name>.private_key	A private key	The private key for the certificate specified in tls.<cert_name>.certificate.
tls.<cert_name>.ca	A PEM-encoded X.509 certificate	The CA certificate with which your certificate has been or will be signed.

The placeholder <cert_name> in the TLS/SSL specific configuration is Data Service specific and the following values are required for the individial Data Service:

Data Service	Certificate Names
a9s KeyValue	valkey
a9s LogMe2	opensearch, opensearch_dashboards, fluentd
a9s MariaDB	client, server
a9s Messaging	rabbitmq
a9s MongoDB	mongodb
a9s MySQL	client, server
a9s PostgreSQL	postgresql
a9s Redis®*	redis
a9s Search	opensearch

info

You can use the same certificate, private_key and ca for multiple service instances, and in the case of a9s LogMe2 and a9s MariaDB even for the different use cases in the same service instance.

Example

In the following example, we will create an a9s PostgreSQL Service Instance using an SSL plan and a X.509 certificate. First we will have a look at the structure of the required JSON file and its content. Keep in mind that the content must be valid JSON and therefore you need to escape the PEM encoded X.509 certificate file or private key; which you can do, for example, by using awk:

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' <path to your PEM-encoded certificate file>

The JSON file should look like the example below:

{
  "tls": {
    "postgresql": {
      "ca": "-----BEGIN CERTIFICATE-----\nMIIFbjCCA1YCCQCjw+EXPZsMZjANBgkqhkiG9w0BAQsFADB4MQswCQYDVQQGEwJE\nRTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJicnVlY2tlbjEWMBQG\nA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAYBgNVBAMMEXRlc3Qu\nYW55bmluZXMuY29tMCAXDTIyMDcyODEwMjk1OFoYDzIwNTAwODEwMTAyOTU4WjB4\nMQswCQYDVQQGEwJERTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJi\ncnVlY2tlbjEWMBQGA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAY\nBgNVBAMMEXRlc3QuYW55bmluZXMuY29tMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A\nMIICCgKCAgEA7KwHFPQrW9oqzheKK2Vo0TqEexlmN92v6u7Rn6FIZD4FFOPqyg9q\n1P/FQld3UUaMio6l7wxyCCzojB5cWpcBrWWb6Lcr4e0aXLOi8B6vVixKftZt0jcE\nQCv8nPHacYcHXw4LwhVMVedXvhwtLbopRVOuRdbdLkfWujew1gnafQ/C2+7ozv7Z\n/1ADXCREjQ4KlPnn1d87ZK7muR4TjZFEh0ZZ7gA3UjimywzQw8wCUIMBXDH8OInr\nYKHY4+yyq7oaliM68H7WYfJURSQZ/Zv1jVCYLs1AXXWMYSy4hnbzGdLrOfg62Lcc\noYrnJc6yV4e0xOXniCYCy9TxOD5kwXkm3yKO+adpD3TR7BqLCKG9KyN9Ih8vrQlC\ni4azZnMQ2EGlEDoV6wN35etEMocC4hB8TRUYQ1KY3TTVSKb9wPAn/4cAuh6F+cPK\nmq19D+z+PWlVaQHwPFwO2p2koTIycd6gjctHlGS/lAWPsoI1nnTtZ2u5PX3r5/B0\n144XuFTDNmI+7ch3jcggIU6hWcWd4qZ++UE1LltuhTGpzqPw/3Vt9TNSkTvpy8B/\nasZPKPZiDkwFsQQY385dtvkCobpJYNC3wtHnr/t4q+gklYR5mO4xMDgNf+bKjTuI\n0Iut0Aj+0uoJxFvTTfLM4EtPLlW2z7YPkY0A4+OaFV8LjPO1ZPyFyrMCAwEAATAN\nBgkqhkiG9w0BAQsFAAOCAgEAfLXs0mymh4jdUc/ghOdXQFkCqMg5E5wSWlfEgzC0\n8oJwAw4TLjynvrMpZtms6DWf0o63NtrNoQPMcJ6+R8oD3Vqn7Q7BDR+F+cKg2c0e\n44xocV5o5vX8xwZ63GScS+ZSIunGvP537XV5Lzyv3dI9e+qjs893mzgAO02ACT34\nTXlDn/bTKAmOOqRaj6yP2HsInCsJetB5pHi3R3buyy9DbAXt6qtt+eySv1U7JU81\nipMoUbv2PdLLfgfJOVGCldpduOwsyFTwHB78MBiT755QWBalYJ/sw47OaYSDFKCX\nmdO913OpXkUZfF3l4l/CkXEUsY6+pMSCoguwgwln3eVPfmNjqG4LtZGOgKMepmcG\nkRNSGsvVKXglcIsPgogkOxUf569FEK5kEtYw0cMn8MCrv358H1xUFXVcitSLKpGM\n3klsZ1tU/ZnwxbSLcuozHhd+YOjy9KSRmUZCA+ORKVhrweWpugbnv0ZRKzXGLC9a\ny1Rj//v5jWMiEAyZNjeSnCNfLx3jlovR3SjcnZZFrVidQi8cZCnAMzzVArnHi/FI\nxpkP5EAqh8q7DmrMY5X11eEuyCmbgPmjji/IVcOJ3NC9hZeOIiSazU3CvDEX6Blm\nmL/NTgnwTnQn5dKSahQgLtFaIve4VNP4aPFXgGdYScd6U8Luo34tD1EWg45kXnkG\nA38=\n-----END CERTIFICATE-----\n",
      "certificate": "-----BEGIN CERTIFICATE-----\nMIIEaTCCAlECCQCjNrxOOvCTgzANBgkqhkiG9w0BAQsFADB4MQswCQYDVQQGEwJE\nRTERMA8GA1UECAwIU2FhcmxhbmQxFTATBgNVBAcMDFNhYXJicnVlY2tlbjEWMBQG\nA1UECgwNYW55bmluZXMgR21iSDELMAkGA1UECwwCRFMxGjAYBgNVBAMMEXRlc3Qu\nYW55bmluZXMuY29tMB4XDTIyMDcyODEwMzE0N1oXDTM2MDQwNTEwMzE0N1owdTEL\nMAkGA1UEBhMCREUxETAPBgNVBAgMCFNhYXJsYW5kMRUwEwYDVQQHDAxTYWFyYnJ1\nZWNrZW4xFjAUBgNVBAoMDWFueW5pbmVzIEdtYkgxCzAJBgNVBAsMAkRTMRcwFQYD\nVQQDDA4qLmFueW5pbmVzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBALBfuvTiCfj0eJ03UaDLPT91qpZvsvKeML9TZ5C5MKG5kOp4GbAl+qcg9SEY\nouYD1niKfrpi2CN1iT7xVhJTQImSxfQO0GXumGlOz/PwCvBm2bTHBzBZl389sDGA\n6od00ailkv1On4DBfMD6okEWgWyWLZ8PdMDcat0SoKxwt4P4ddtvhtYNSNtiM7Po\nag1Lv0Ocha6B1kD8GbJY/murLrCW7ZBUF3i2p075imfEwJiu9U34t79rnPhzaHAX\nshL9VAEFidNAXahK4q2EudjFum+Bd5Ud0EdIfzz1Dn9EUYfFn0mPF+CCfIu1x8n6\nFOwz5Q/0+NRAIHAy9waB8vmP/e8CAwEAATANBgkqhkiG9w0BAQsFAAOCAgEAwfkC\nH9e8eQX22gpzCp282YFbCq5Zl0eKONZ3hDm0G+ej1hVUIomKVrwJdz8fcYSf5KZw\nn95HuMPNL5w8SgJ3gR5ZNegWlNYL/sKhIk3NDSyv6helSs6dbufy2CMXg51jlpUp\nJWy4Xa4VLv5kPHxi5gZili4eYCh9Z97kxb8K6P3wXnUOhpyOj4LvzyIwXIxG7gXj\nfp0sQ5+GrlwBhNKnPnF3h8fWU4iQjiRp9hwixKO2uetyc3t47U4FBT4NyYuvGbR/\nVYlekHPCCFn7o31p1VJ/qYGqaMoZEfJZv6t2w9WMf4BpNXnyJ1TFCRSJ0QtZC35E\n4ZuHf9O4rVIBTQL0G7edHbXpxUh0au6uvk8Z0QtwDi8p+1gCg85DICUOmqPxALGi\nDzMjDApWMWyru5PeAC6BnVWuF/Gwhq46X4roQZJLke8eAIdYbptCAltuwlIz1foj\n2DsRESyzcyzk/wziSg3Ai42r1dEODcMd3Na0339EpdG/lbK4+hzPcC4Tv/iUH1Xh\nGnyuH+gi40+92VDSGbvFLTlb/alNBY3TWAjJMCrhE3WgcxVTdFkBS/gV3bPghhaw\nVL/2Nv7oTllDWwAZoJ/WKKu0s1a129iwIJ0xkK0NlDkY13DgTQIngZiXTi3Nq9ct\nB4O8e7EElIvg+68UN7u3nyGI9yzVHqoBHjzwIcw=\n-----END CERTIFICATE-----\n",
      "private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAsF+69OIJ+PR4nTdRoMs9P3Wqlm+y8p4wv1NnkLkwobmQ6ngZ\nsCX6pyD1IRii5gPWeIp+umLYI3WJPvFWElNAiZLF9A7QZe6YaU7P8/AK8GbZtMcH\nMFmXfz2wMYDqh3TRqKWS/U6fgMF8wPqiQRaBbJYtnw90wNxq3RKgrHC3g/h122+G\n1g1I22Izs+hqDUu/Q5yFroHWQPwZslj+a6susJbtkFQXeLanTvmKZ8TAmK71Tfi3\nv2uc+HNocBeyEv1UAQWJ00BdqErirYS52MW6b4F3lR3QR0h/PPUOf0RRh8WfSY8X\n4IJ8i7XHyfoU7DPlD/T41EAgcDL3BoHy+Y/97wIDAQABAoIBAD8x7dD0dNJI9jaO\nrabJa6ajSH4ig6G+4ISNI9yNHkm0LaJ1ae7djNP5URuMskFsOppyNTofVIZQBN8S\npd5nCgbBGkB6Vl1PgjipToV/CsxcbcECVycR0uB6f/kc0eu7BaxBt0sfsrL1oPec\nViqwH6uCm9IUPack0v1nQT7IHFkiQwE9Sg8FDgXoUtaIbzb7oRd/dFStUap6lrvC\nNscjpItU0m9hdwTRUqjqv/Cg066TdYEFdtdL9F8oOtsu0Ix3SkQY3xkUS+9z0FZC\nljHOv3nzLIOBnn0BfXqy2am/evwkwOw8PzDp5RP6FStEWJgwzINvPD77fMrhy511\ndYhtDsECgYEA1jpMlKSYJ674z07Dqk1wuM/joY8EZHRVnp7kWtzOtHcRZjG1Ab/R\nzvEtDr09t1mgBHfC0xTFkzntTj3hTMuKNmqP0ycPL/k5HGdGyCuW7Al7JLweUlbN\nNEG/adxn5EI9QFjs7RmfVFDBlc8yUbwIZSmSyqb5b4fjB77I/6me+n8CgYEA0sPd\n8btgiQl0yZmAsu9hAnMbE3rPB/cfIWoYGN7KNY2xEuQKcW0v2z3th0td44KdxwiL\nlY26V1USzY/MbnHRBGOZw6WBQNSYw9TfjMN6XVv84KGXv7cM31rhsascvY7L+SX/\nKsk69jEQZaPlnRGsPoHkVOBgZnySlXpW6L0V5JECgYBhpVEir6Nq8yYV2CD3jzTC\nCIAJM9ccsqoUEvijMeJF/7++hQmsMnK/kM6o2Tk4SXHWl4AGFoG4Cb9Q2oPHLT9i\nblAAPt19UGvntmtc8gFotSmcJOLtRQNjvlQxPHVeZZLlsaLMr9Ef7W9PiZG2D68D\n7V8rX9ByNc4VybB0WCxOoQKBgQCo83fbyh/FWj4zSQjA52E1bH387Io/UVq0F27/\nqMAhk9apVQIGEMe3EPpyZPj7Yn4FyZTTKsyAK2MqQyZWNoid4xWknuxwwrs+6ErX\ndO+HHTEmBIM3nI4GEb0wBgHA9lIOA+Z7LtpD9eq1/18VvM/9P3SAkWjVXvDGQE6g\nvMpq8QKBgEaFtGhwuPe5/j6HsOJPvOgMPjM3LdQ18nO/NVDNQE8A0KftCdaioAy8\n5wGi2lDy1zmb+dZEMR9rEZpemNeE4jHSMOOPSwz4ZAhb2xpKymbtl/cwSrkiFeoh\n5SCm+DPRKOnxsj+NC4U9/Js8YwribuEI4MOdlOjM/hNxseDCEguE\n-----END RSA PRIVATE KEY-----\n"
    }
  }
}

You can use the following command to create a service instance with the mentioned JSON file:

cf create-service a9s-postgresql13 postgresql-single-small-ssl my-pg -c ssl_config.json

Creating service instance my-pg in org organization / space test as user@example.com...
OK

Create in progress. Use 'cf services' or 'cf service my-pg' to check operation status.

Rotation of User Provided Certificates

Leaf Certificate Rotation

As certificates for service instances with SSL plans expire, there is a need to rotate them regularly, and user provided leaf certificates are not an exception. The rotation of the user provided leaf certificate and its key can be done by following these steps.

caution

This will only work if the current CA certificate is valid, not expired and it is used to sign the new leaf certificate. This process may cause some downtime in single deployments, as the service instances are updated with the certificate information. If the CA is still valid, there is no downtime on cluster deployments.

danger

Providing your own certificates allows you to be more in control, but in exchange you are responsible for the rotation, since the automation doesn't work with this option, as it will return the same certificate each time it is executed. Additionally the forced rotation functionality is not supported for user provided certificates.

Step 1: Update the leaf certificate

First, add the new leaf certificate and the corresponding private key to your JSON file. Ensure that these are properly parsed, as mentioned before. Your config file should look like:

{
  "tls": {
    "postgresql": {
      "ca": "<old_ca>",
      "certificate": "<new_leaf_certificate>",
      "private_key": "<new_private_key>"
    }
  }
}

Step 2: Redeploy your service instance

You can apply the new leaf certificate and private key to your SSL service instance by executing the following command:

cf update-service my-pg -c updated_ssl_config.json

CA Certificate Rotation

Just as Leaf certificates expire for service instances with SSL plans, so do CA certificates. Unfortunately the rotation of a CA certificate is not as easy as the rotation of a leaf certificate because simply replacing it will lead to broken clustered service instances.

To update the service we recommend to use the following method for a rotation with minimal downtime.

Step 1: Append the new CA to the old CA

Concatenate the old and the new CA (first old CA, then new CA) and store it in a temporary file to make it easier to parse them properly. This allows the service instance's nodes and the clients to trust the wildcard certificate presented by the service instance that can be either signed by the new or the old CA.

Add the concatenated CAs to your JSON file. Ensure that these are properly parsed as mentioned before. Your config file should look like:

{
  "tls": {
    "postgresql": {
      "ca": "<concatenated_cas>",
      "certificate": "<leaf_certificate>",
      "private_key": "<private_key>"
    }
  }
}

Step 2: Roll out both, the old and the new CA, to your service service instance

You can apply the concatenated CAs to your SSL service instance by executing the following command:

cf update-service my-pg -c updated_ssl_config.json

Renew Service Bindings

Before proceeding with the next step, all service bindings MUST be renewed to propagate the new CA to the clients, unless the new CA is already trusted by the client's runtime.

Step 3: Replace the leaf certificate

Add the new leaf certificate and the corresponding private key to your JSON file. Ensure that these are properly parsed, as mentioned before. Your config file should look like:

{
  "tls": {
    "postgresql": {
      "ca": "<concatenated_cas>",
      "certificate": "<new_leaf_certificate>",
      "private_key": "<new_private_key>"
    }
  }
}

Step 4: Roll out the new leaf certificate to your service instance

You can apply the new leaf certificate and private key to your SSL service instance by executing the following command:

cf update-service my-pg -c updated_ssl_config.json

Step 5: Remove the old CA

Add the new and only the new CA to your JSON file. Ensure that these are properly parsed, as mentioned before. The old CA is no longer needed since all existing service instances are only using the new wildcard certificate that is signed by the new CA.

{
  "tls": {
    "postgresql": {
      "ca": "<new_ca>",
      "certificate": "<new_leaf_certificate>",
      "private_key": "<new_private_key>"
    }
  }
}

Step 6: Roll out only the new CA to each service instance

You can apply the concatenated CAs to your TLS/SSL service instance by executing the following command:

cf update-service my-pg -c updated_ssl_config.json

Renew Service Bindings

Before proceeding with the next step, all service bindings MUST be renewed to propagate the new CA to the clients, unless the new CA is already trusted by the client's runtime.

Limitations

Deleting User Provided Certificates

The deletion of user provided certificates is not supported by the a9s Data Services Framework. As a consequence, if you want to use certificates provided by the Platform Operator you have to create a new service instance using a TLS/SSL service plan and migrate your data.

Migrating User Provided Certificates

The a9s Data Services Framework does not support the certificate migration for a service instance using a TLS/SSL service plan from user provided certificates to certificates provided by the Platform Operator and vice versa.

Limitations for MariaDB

MariaDB 10.4 has a pre-defined structure for how it expects the different certificates. While the structure is described in length in the vendors' documentation, for our intents and purposes, this is the most relevant piece of information:

There are three certificates that you need to create in order to secure Galera Cluster: the Certificate Authority (CA) key and cert; the server certificate, to secure mysqld activity and replication traffic; and the client certificate to secure the database client and stunnel for state snapshot transfers.

Due to the nature of our automation, you have to specify a different certificate for client and intercluster.

Limitations for OpenSearch

OpenSearch expects that the certificates' private keys are in the format PKCS#8. This restriction applies when user provided certificates are using.

You can check the vendor's documentation for more information:

• https://opensearch.org/docs/latest/security-plugin/configuration/tls/
• https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/

---

*Redis is a registered trademark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by anynines GmbH is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and anynines GmbH.