a9s Kubernetes Configurations
This document describes the special configuration necessary for an a9s Kubernetes Service deployment.
Just include the ops file
The credentials will be fetched via Instance Metadata.
You need to include the ops files
Additionally you need to provide the following variables:
/vcenter_ip: is the vCenter Server IP or FQDN.
/vcenter_dc: is the name of the vCenter Datacenter on which Kubernetes node VMs are deployed.
/vcenter_ds: is the default datastore to use for provisioning volumes using storage classes/dynamic provisioning.
/vcenter_master_user: is the vCenter username for vSphere Cloud Provider.
/vcenter_master_password: is the password for vCenter user.
/vcenter_insecure_flag: is set to 1 if vCenter uses a self-signed certificate.
vcenter_vms is required to specify the
working directory (folder in which VMs are provisioned).
This means that it's currently not possible or not an easy way to determine
whether there are leftovers.
The template-uploader-errand BOSH release provides ops files to configure
StorageClass. Read the corresponing sections which ops
files to add to
template-ops-files for a specific platform.
Default Cluster Roles / RBAC
We implemented a default ClusterRole for a9s Kubernetes. The ClusterRole defines
RBAC rules, that get automatically applied, when creating an a9s Kubernetes
service-binding that is created, gets a different
access-token. This user is automatically bound to this ClusterRole
The a9s Kubernetes BOSH property path is
Per default, the ClusterRole gives you access to every resource and every API in the Cluster. To restrict the access to different resources/APIs, the ClusterRole needs to be adjusted according to the official Kubernetes - Using RBAC Authorization documentation.
But please watch out for configurations issues. For example, if you remove access to
storageclasses for the
service-binding, the end user might not be able to install
Last Mile Encryption
Last mile encryption refers to the use of encryption (namely TLS) for communication between the a9s-router and the a9s Kubernetes cluster, by default encryption is already enabled between the Internet and the a9s-router. In order to configure the "last mile encryption" with our Kubernetes offering enable the following properties in the Kubernetes service manifest:
jobs: - name: kubernetes-spi properties: backup-agent: ... consul: ... kubernetes-spi: dashboard: ... endpoint: ... intermediate-ca: cert: ((/a9s_router_ca.certificate)) key: ((/a9s_router_ca.private_key))
In order for the above credhub reference to work correctly you must supply
the relevant organization's intermediate certificate as the credhub credential
/a9s_router_ca. HAProxy can be enabled to force last mile encryption
The following should be included into the cloud-config of the BOSH director:
vm_extensions: - cloud_properties: vmx_options: disk.enableUUID: "1" name: enable-disk-UUID
Paste this configuration to a file and update the cloud config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:
bosh update-cloud-config path/to/cloudconfig.yml
Since a9s Kubernetes service instances are using BOSH DNS instead of Consul DNS, it is necessary to add some special configuration to the BOSH Director.
First of all, the BOSH Director needs to have "local dns" enabled. See the official BOSH Documentation for more information.
If this is enabled, all the a9s Kubernetes service instances need the bosh-dns job provided by the bosh-dns release. This can be done via runtime configuration. Be carefully, this runtime configuration will only be applied to a9s Kubernetes instances and not for other services or service instances provided by anynines. If you enable this for all the services and service instances, this could cause severe problems in your platform. So we suggest to only enable bosh-dns addon for the a9s Kubernetes deployments.
addons: - include: jobs: - name: kube-apiserver release: a9s-k8s - name: kubelet release: a9s-k8s jobs: - name: bosh-dns properties: api: client: tls: ((/dns_api_client_tls)) server: tls: ((/dns_api_server_tls)) cache: enabled: true health: client: tls: ((/dns_healthcheck_client_tls)) enabled: true server: tls: ((/dns_healthcheck_server_tls)) release: bosh-dns name: bosh-dns releases: - name: bosh-dns sha1: d514ab3ae376778e106e17c22b78a8705690ae1d url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.17.0 version: 1.17.0 variables: - name: /dns_healthcheck_tls_ca options: common_name: dns-healthcheck-tls-ca is_ca: true type: certificate - name: /dns_healthcheck_server_tls options: ca: /dns_healthcheck_tls_ca common_name: health.bosh-dns extended_key_usage: - server_auth type: certificate - name: /dns_healthcheck_client_tls options: ca: /dns_healthcheck_tls_ca common_name: health.bosh-dns extended_key_usage: - client_auth type: certificate - name: /dns_api_tls_ca options: common_name: dns-api-tls-ca is_ca: true type: certificate - name: /dns_api_server_tls options: ca: /dns_api_tls_ca common_name: api.bosh-dns extended_key_usage: - server_auth type: certificate - name: /dns_api_client_tls options: ca: /dns_api_tls_ca common_name: api.bosh-dns extended_key_usage: - client_auth type: certificate
Paste this runtime configuration to a file and update the runtime config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:
bosh update-runtime-config path/to/runtimeconfig.yml