a9s Kubernetes Configurations

This document describes the special configuration necessary for an a9s Kubernetes Service deployment.

AWS

Just include the ops file ops/iaas/aws/kubernetes-cloud-provider.yml. The credentials will be fetched via Instance Metadata.

vSphere

You need to include the ops files ops/iaas/vsphere/kubernetes-cloud-provider.yml and ops/iaas/vsphere/use-vm-extensions.yml.

Additionally you need to provide the following variables:

  • /vcenter_ip: is the vCenter Server IP or FQDN.
  • /vcenter_dc: is the name of the vCenter Datacenter on which Kubernetes node VMs are deployed.
  • /vcenter_ds: is the default datastore to use for provisioning volumes using storage classes/dynamic provisioning.
  • /vcenter_master_user: is the vCenter username for vSphere Cloud Provider.
  • /vcenter_master_password: is the password for vCenter user.
  • /vcenter_insecure_flag: is set to 1 if vCenter uses a self-signed certificate.
  • /vcenter_vms

The variable vcenter_vms is required to specify the working directory (folder in which VMs are provisioned). This means that it's currently not possible or not an easy way to determine whether there are leftovers.

Storage Classes

The template-uploader-errand BOSH release provides ops files to configure StorageClass. Read the corresponing sections which ops files to add to template-ops-files for a specific platform.

Default Cluster Roles / RBAC

We implemented a default ClusterRole for a9s Kubernetes. The ClusterRole defines RBAC rules, that get automatically applied, when creating an a9s Kubernetes instance. Every service-binding that is created, gets a different username and access-token. This user is automatically bound to this ClusterRole via ClusterRoleBinding.

The a9s Kubernetes BOSH property path is properties.kubernetes-spi.default_clusterroles.

Per default, the ClusterRole gives you access to every resource and every API in the Cluster. To restrict the access to different resources/APIs, the ClusterRole needs to be adjusted according to the official Kubernetes - Using RBAC Authorization documentation.

But please watch out for configurations issues. For example, if you remove access to storageclasses for the service-binding, the end user might not be able to install linkerd manually.

Cloud Config

The following should be included into the cloud-config of the BOSH director:

vm_extensions:
- cloud_properties:
    vmx_options:
      disk.enableUUID: "1"
  name: enable-disk-UUID

Paste this configuration to a file and update the cloud config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:

bosh update-cloud-config path/to/cloudconfig.yml

Runtime Config

Since a9s Kubernetes service instances are using BOSH DNS instead of Consul DNS, it is necessary to add some special configuration to the BOSH Director.

First of all, the BOSH Director needs to have "local dns" enabled. See the official BOSH Documentation for more information.

If this is enabled, all the a9s Kubernetes service instances need the bosh-dns job provided by the bosh-dns release. This can be done via runtime configuration. Be carefully, this runtime configuration will only be applied to a9s Kubernetes instances and not for other services or service instances provided by anynines. If you enable this for all the services and service instances, this could cause severe problems in your platform. So we suggest to only enable bosh-dns addon for the a9s Kubernetes deployments.

addons:
- include:
    jobs:
    - name: kube-apiserver
      release: a9s-k8s
    - name: kubelet
      release: a9s-k8s
  jobs:
  - name: bosh-dns
    properties:
      api:
        client:
          tls: ((/dns_api_client_tls))
        server:
          tls: ((/dns_api_server_tls))
      cache:
        enabled: true
      health:
        client:
          tls: ((/dns_healthcheck_client_tls))
        enabled: true
        server:
          tls: ((/dns_healthcheck_server_tls))
    release: bosh-dns
  name: bosh-dns
releases:
- name: bosh-dns
  sha1: d514ab3ae376778e106e17c22b78a8705690ae1d
  url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.17.0
  version: 1.17.0
variables:
- name: /dns_healthcheck_tls_ca
  options:
    common_name: dns-healthcheck-tls-ca
    is_ca: true
  type: certificate
- name: /dns_healthcheck_server_tls
  options:
    ca: /dns_healthcheck_tls_ca
    common_name: health.bosh-dns
    extended_key_usage:
    - server_auth
  type: certificate
- name: /dns_healthcheck_client_tls
  options:
    ca: /dns_healthcheck_tls_ca
    common_name: health.bosh-dns
    extended_key_usage:
    - client_auth
  type: certificate
- name: /dns_api_tls_ca
  options:
    common_name: dns-api-tls-ca
    is_ca: true
  type: certificate
- name: /dns_api_server_tls
  options:
    ca: /dns_api_tls_ca
    common_name: api.bosh-dns
    extended_key_usage:
    - server_auth
  type: certificate
- name: /dns_api_client_tls
  options:
    ca: /dns_api_tls_ca
    common_name: api.bosh-dns
    extended_key_usage:
    - client_auth
  type: certificate

Paste this runtime configuration to a file and update the runtime config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:

bosh update-runtime-config path/to/runtimeconfig.yml