Skip to main content
Version: 32.0.0

a9s Kubernetes Configurations

This document describes the special configuration necessary for an a9s Kubernetes Service deployment.

AWS​

Just include the ops file ops/iaas/aws/kubernetes-cloud-provider.yml. The credentials will be fetched via Instance Metadata.

vSphere​

You need to include the ops files ops/iaas/vsphere/kubernetes-cloud-provider.yml and ops/iaas/vsphere/use-vm-extensions.yml.

Additionally you need to provide the following variables:

  • /vcenter_ip: is the vCenter Server IP or FQDN.
  • /vcenter_dc: is the name of the vCenter Datacenter on which Kubernetes node VMs are deployed.
  • /vcenter_ds: is the default datastore to use for provisioning volumes using storage classes/dynamic provisioning.
  • /vcenter_master_user: is the vCenter username for vSphere Cloud Provider.
  • /vcenter_master_password: is the password for vCenter user.
  • /vcenter_insecure_flag: is set to 1 if vCenter uses a self-signed certificate.
  • /vcenter_vms

The variable vcenter_vms is required to specify the working directory (folder in which VMs are provisioned). This means that it's currently not possible or not an easy way to determine whether there are leftovers.

vSphere Container Storage Interface (CSI)​

The CSI driver is responsible for volume provisioning, attaching and detaching the volume to VMs, mounting, formatting and unmounting volumes from the pod within the node VM, and so on. The CSI driver is built as an out-of-tree CSI plugin for Kubernetes. For more information read the vSphere CSI Documentation

The vSphere CSI driver is enabled as an addon for each a9s Kubernetes instance. To enable add vsphere under the apply-a9s-specs job properties of the instance manifest:

- name: apply-a9s-specs
  release: a9s-k8s
  properties:
    addons:
    - vsphere

To configure the vSphere container storage using CSI add your vSphere datacenter information to the service manifest under the properties of the kubernetes-spi job. The following should be included in the Kubernetes service manifest for the information to be passed via the SPI to the instances:

- name: kubernetes-spi
  properties:
    ...
    kubernetes-spi:
      cloud-provider:
        vsphere:
          virtualcenter_ip: 1.1.1.1
          username: "username"
          password: "password"
          insecure_key: true
          datacenter: "datacenter"
  ...

AWS Container Storage Interface (CSI)​

The Amazon Elastic Block Store Container Storage Interface (CSI) Driver provides a CSI interface used by Container Orchestrators to manage the lifecycle of Amazon EBS volumes. For more information read the Amazon Elastic Block Store (EBS) Documentation

The AWS EBS CSI driver is enabled as an addon for each a9s Kubernetes instance. To enable add aws under the apply-a9s-specs job properties of the instance manifest:

- name: apply-a9s-specs
  release: a9s-k8s
  properties:
    addons:
    - aws

To configure the AWS container storage using CSI add your AWS credential information to the service manifest under the properties of the kubernetes-spi job. The following should be included in the Kubernetes service manifest for the information to be passed via the SPI to the instances:

- name: kubernetes-spi
  properties:
    ...
    kubernetes-spi:
      cloud-provider:
        aws:
          access_key: my_access_key
          key_id: my_key_id
  ...

Storage Classes​

The template-uploader-errand BOSH release provides ops files to configure StorageClass. Read the corresponding sections which ops files to add to template-ops-files for a specific platform.

Default Cluster Roles / RBAC​

We implemented a default ClusterRole for a9s Kubernetes. The ClusterRole defines RBAC rules, that get automatically applied, when creating an a9s Kubernetes instance. Every service-binding that is created, gets a different username and access-token. This user is automatically bound to this ClusterRole via ClusterRoleBinding.

The a9s Kubernetes BOSH property path is properties.kubernetes-spi.default_clusterroles.

Per default, the ClusterRole gives you access to every resource and every API in the Cluster. To restrict the access to different resources/APIs, the ClusterRole needs to be adjusted according to the official Kubernetes - Using RBAC Authorization documentation.

But please watch out for configurations issues. For example, if you remove access to storageclasses for the service-binding, the end user might not be able to install linkerd manually.

Last Mile Encryption​

Last mile encryption refers to the use of encryption (namely TLS) for communication between the a9s-router and the a9s Kubernetes cluster, by default encryption is already enabled between the Internet and the a9s-router. In order to configure the "last mile encryption" with our Kubernetes offering enable the following properties in the Kubernetes service manifest:

jobs:
  - name: kubernetes-spi
    properties:
      backup-agent:
        ...
      consul:
        ...
      kubernetes-spi:
        dashboard:
          ...
        endpoint:
          ...
        intermediate-ca:
          cert: ((/a9s_router_ca.certificate))
          key: ((/a9s_router_ca.private_key))

In order for the above credhub reference to work correctly you must supply the relevant organization's intermediate certificate as the credhub credential for /a9s_router_ca. HAProxy can be enabled to force last mile encryption by setting a9s-route-registrar.endpoint.verify from none to required.

Cloud Config​

The following should be included into the cloud-config of the BOSH director:

vm_extensions:
- cloud_properties:
    vmx_options:
      disk.enableUUID: "1"
  name: enable-disk-UUID

Paste this configuration to a file and update the cloud config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:

bosh update-cloud-config path/to/cloudconfig.yml

Runtime Config​

Since a9s Kubernetes service instances are using BOSH DNS instead of Consul DNS, it is necessary to add some special configuration to the BOSH Director.

First of all, the BOSH Director needs to have "local dns" enabled. See the official BOSH Documentation for more information.

If this is enabled, all the a9s Kubernetes service instances need the bosh-dns job provided by the bosh-dns release. This can be done via runtime configuration. Be carefully, this runtime configuration will only be applied to a9s Kubernetes instances and not for other services or service instances provided by anynines. If you enable this for all the services and service instances, this could cause severe problems in your platform. So we suggest to only enable bosh-dns addon for the a9s Kubernetes deployments.

addons:
- include:
    jobs:
    - name: kube-apiserver
      release: a9s-k8s
    - name: kubelet
      release: a9s-k8s
  jobs:
  - name: bosh-dns
    properties:
      api:
        client:
          tls: ((/dns_api_client_tls))
        server:
          tls: ((/dns_api_server_tls))
      cache:
        enabled: true
      health:
        client:
          tls: ((/dns_healthcheck_client_tls))
        enabled: true
        server:
          tls: ((/dns_healthcheck_server_tls))
    release: bosh-dns
  name: bosh-dns
releases:
- name: bosh-dns
  sha1: d514ab3ae376778e106e17c22b78a8705690ae1d
  url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.17.0
  version: 1.17.0
variables:
- name: /dns_healthcheck_tls_ca
  options:
    common_name: dns-healthcheck-tls-ca
    is_ca: true
  type: certificate
- name: /dns_healthcheck_server_tls
  options:
    ca: /dns_healthcheck_tls_ca
    common_name: health.bosh-dns
    extended_key_usage:
    - server_auth
  type: certificate
- name: /dns_healthcheck_client_tls
  options:
    ca: /dns_healthcheck_tls_ca
    common_name: health.bosh-dns
    extended_key_usage:
    - client_auth
  type: certificate
- name: /dns_api_tls_ca
  options:
    common_name: dns-api-tls-ca
    is_ca: true
  type: certificate
- name: /dns_api_server_tls
  options:
    ca: /dns_api_tls_ca
    common_name: api.bosh-dns
    extended_key_usage:
    - server_auth
  type: certificate
- name: /dns_api_client_tls
  options:
    ca: /dns_api_tls_ca
    common_name: api.bosh-dns
    extended_key_usage:
    - client_auth
  type: certificate

Paste this runtime configuration to a file and update the runtime config of the BOSH director that will deploy a9s Kubernetes instances by using the following command:

bosh update-runtime-config path/to/runtimeconfig.yml

Third Party Addons​

The operator can configure a9s Kubernetes service to have pre-installed addons like cifs for every instance. The configuration consists of 2 steps:

  1. Set values into a9s Kubernetes SPI
  2. Include addons into a9s Kubernetes plans.

There is an example of setting values for a9s Kubernetes SPI.

Important to mention that you plan to integrate other software into a9s Kubernetes, please, be careful with their restrictions. Before integrating other software read their documentation regarding restrictions. Those restrictions can easily break service.

kubernetes-spi:
  dashboard: [...]
  third-party-addons:
    addon1:
      master: # this part will be executed at master nodes
        apply_specs: |
          ${kubectl} apply -f ${file0}
        files:
        - content: |
            apiVersion: v1
            kind: Pod
            metadata:
              name: k8s-unit-tests-dummy-file0
              labels:
                app: k8s-unit-tests-dummy-file0
            spec:
              containers:
              - name: busybox
                image: busybox
                args:
                - sleep
                - "1000000"
          name: file0 # this name will be used in the apply_specs part
        pre_start: |
          echo 'Set up aws cli'
          curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
          unzip awscliv2.zip
          ./aws/install

          echo 'Set credentials'
          /usr/local/bin/aws configure set region us-east-1
          /usr/local/bin/aws configure set aws_access_key_id [AWS_ACCESS_KEY_ID]
          /usr/local/bin/aws configure set aws_secret_access_key [AWS_SECRET_ACCESS_KEY]

          echo 'Create bucket'
          # **NOTE**: Using capital letters in AWS S3 Bucket names will not work!
          # Please also note [other AWS S3 Bucket restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html).

          if [ $(/usr/local/bin/aws s3 ls|grep example-bucket-pre-start-master|wc -l) == 0 ] ; then
            /usr/local/bin/aws s3 mb s3://example-bucket-pre-start-master
          fi
        pre_stop: |
          if [[ "${BOSH_DEPLOYMENT_NEXT_STATE}" == "delete" ]] ; then
            /usr/local/bin/aws s3 rb s3://example-bucket-pre-start-master
          fi
      workers: # this part will be executed at worker nodes
        pre_start: |
          echo 'Set up aws cli'
          curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
          unzip awscliv2.zip
          ./aws/install

          echo 'Set credentials'
          /usr/local/bin/aws configure set region us-east-1
          /usr/local/bin/aws configure set aws_access_key_id [AWS_ACCESS_KEY_ID]
          /usr/local/bin/aws configure set aws_secret_access_key [AWS_SECRET_ACCESS_KEY]

          echo 'Create bucket'
          # **NOTE**: Using capital letters in AWS S3 Bucket names will not work!
          # Please also note [other AWS S3 Bucket restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html).

          if [ $(/usr/local/bin/aws s3 ls|grep example-bucket-pre-start-worker|wc -l) == 0 ] ; then 
            /usr/local/bin/aws s3 mb s3://example-bucket-pre-start-worker
          fi
        pre_stop: |
          if [[ "${BOSH_DEPLOYMENT_NEXT_STATE}" == "delete" ]] ; then
            if [ $(/usr/local/bin/aws s3 ls|grep example-bucket-pre-start-worker|wc -l) -gt 0 ] ; then 
              /usr/local/bin/aws s3 rb s3://example-bucket-pre-start-worker
            fi
          fi
    addon2:
      master:
        apply_specs: ''
        files: []
        pre_start: ''
        pre_stop: ''
      workers:
        pre_start: ''
        pre_stop: ''
    addon3:
      master:
        apply_specs: ''
        files: []
        pre_start: 'apt update -y'
        pre_stop: ''

The operator can configure a9s Kubernetes service to have pre-installed addons. Important to say that names of the files have to follow the rule of naming variables in BASH called as variables in the apply_specs part. To understand how works pre_start, pre_stop and apply_specs, please, look at BOSH documentation regarding Pre-start Lifecycle Hook and Pre-stop Lifecycle Hook and Post-deplooy Lifecycle Hook(inside of this Lifecycle Hook will be executed apply_specs part). Please, pay attention at the Environment Variables in the pre-stop hook, for example, if we put the code in the condition when BOSH_DEPLOYMENT_NEXT_STATE is equal delete, the code will be executed when a VM is going to be deleted.

Another important thing that we introduce is a validation of the addon structure. So a9s Kubernetes SPI will check the next statements:

  • Addon has master key
  • Addon has apply_specs, files, pre_start, and pre_stop keys for master
  • Addon has pre_start, and pre_stop keys for workers
  • Files have filled content and name
  • File content should be valid YAML content.

If you don't need to set values for any of the keys, you can pass an empty string, please, look at addon2 in the previous example. If you don't need to execute anything at the workers node, you can skip it, please, look at addon3 in the previous example.

There is an example of setting values for a9s Kubernetes Plan:

plans:
  kubernetes-cluster-small:
    guid: 33bb7116-b2b3-4fc6-b8bc-420e58a4d175
    description: 'a small Kubernetes HA cluster'
    free: true
    template_name_v2: kubernetes
    default_placeholder_values:
      vm_type_master: small
      persistent_disk_type_master: a9svs-kubernetes-master-single-small
      vm_type_worker: a9svs-kubernetes-worker-single-small
      master_node_count: 3
      worker_node_count: 3
      third-party-addons:
        - addon1

In this example, we are enabling addon1 for the instances with the plan kubernetes-cluster-small. Do not enable more addons than you set in the a9s Kubernetes SPI because you will get an instance configuration error.

Disclaimer​

We do not take any responsibility for a misconfigured Third Party Addon since it was not directly implemented by anynines. Any misbehavior of the system which was caused by a misconfiguration of a Third Party Addon(s) has to be solved by the customer. We do not provide support for such a case.