If you already completed the tutorials Deploy Kubernetes 1.9 from scratch on VMware vSphere and Install and configure a multi-master Kubernetes cluster with kubeadm, you should have a pretty good understanding of how a multi-master Kubernetes cluster is structured. You are now probably looking for a way to automate the deployment of your lab so you don't have to follow all these painful steps each time you want to deploy a clean environment.
Kubespray is a Kubernetes incubator project. It is composed of Ansible playbook and automates the deployment of a Kubernetes cluster on an existing infrastructure.
In this lab, we will use Terraform to deploy our infrastructure on VMware vSphere and, in a second stage, Terraform will call Kubespray to install and configure Kubernetes for us. This automation not only supports the deployment, but also adds worker nodes to the Kubernetes cluster, removes worker nodes from the Kubernetes cluster, upgrades the version of Kubernetes, and destroys the Kubernetes cluster.
We will take advantage of the fact that our infrastructure will be running on VMware vSphere and that Kubespray supports the configuration of vSphere Cloud Provider. This will allow us to use the vSphere storage as persistent volumes in the Kubernetes cluster as seen in the article Use vSphere Storage as Kubernetes persistent volumes.
Requirements
For this lab, you will need a configured VMware vSphere environment. You will also need a Ubuntu 16.04 desktop client machine. This machine will be used to execute the Terraform script and to access the Kubernetes dashboard. This machine needs to be on a network that has access to the vCenter API as Terraform will need this to deploy and configure the virtual machines.
For the Kubernetes infrastructure, we are going to deploy three master nodes. They will have the IPs 10.10.40.110, 10.10.40.111, and 10.10.40.112. In front of these master nodes, we will deploy a HAProxy load balancer with the IP 10.10.40.113.
Regarding the Kubernetes worker nodes, we will deploy them on the IP range 10.10.40.120-10.10.40.123.
We will also need a Ubuntu 16.04 vSphere template and a specific user for the vSphere Cloud Provider. The step to configure these will be described later on.
Installing the requirements on the Ubuntu 16.04 client machine
1- Install Git.
$ sudo apt-get install git
2- Install Unzip.
$ sudo apt-get install unzip
3- Install Python 2.7.
$ sudo apt-get install python
4- Install Pip.
$ sudo apt-get install python-pip
5- Install Ansible.
$ pip install ansible
6- Install the Python netaddr library.
$ pip install netaddr
7- Install Terraform.
$ wget https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
$ unzip terraform_0.11.7_linux_amd64.zip
$ sudo mv terraform /usr/local/bin
8- Install Kubectl.
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
sudo apt-key add -
$ sudo vim /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
$ sudo apt-get update
$ sudo apt-get install kubectl
Configuring the requirements on VMware vSphere
Create a Ubuntu 16.04 template
1- Create a new virtual machine.
2- Enter the name "ubuntu-16.04-terraform-template" as the name of the virtual machine.
3- Choose the position of the virtual machine.
4- Choose the compatibility of the virtual machine.
5- Choose Ubuntu as a guest OS type.
6- Set the type of the SCSI controller to "VMware Paravirtual".
7- Select the network of the virtual machine.
8- Select the Ubuntu 16.04 ISO.
9- Connect the CD drive when the virtual machine boots.
10- Complete the creation of the virtual machine.
11- Power on the virtual machine.
12- Open the console of the virtual machine.
13- Select the language of the installer.
14- Install Ubuntu 16.04 server.
15- Choose the language of the system.
16- Choose the location of the system.
17- Choose the mapping of the keyboard.
18- Configure the network card with a temporary IP. We will unconfigure it later on.
19- Choose the default hostname.
20- Leave the domain name empty.
21- Choose the name of your user.
22- Choose a password for your user.
23- Choose if you would like to encrypt your disk or not.
24- Configure the timezone of the system.
25- Select the partitioning method.
26- Select the disk to install Ubuntu 16.04.
27- Configure a proxy if you are using one to access the Internet.
28- Choose to install the security update automatically.
29- Install OpenSSH.
30- Install GRUB.
31- Complete the installation.
32- Once the virtual machine has rebooted, SSH to it from the client machine.
$ ssh sguyennet@10.10.40.254
33- Upgrade the system.
$ sudo apt-get update
$ sudo apt-get upgrade
34- Upgrade the kernel to a version above 4.8. You can skip this step if you are not planning to use Cilium as an overlay network for your Kubernetes cluster.
$ sudo apt-get install linux-image-4.15.0-15-generic \
linux-image-extra-4.15.0-15-generic
$ sudo reboot
35- Allow your user to use sudo without a password.
$ sudo visudo
...
sguyennet ALL=(ALL) NOPASSWD: ALL
36- Generate a private and a public key on the client machine. Leave the passphrase empty.
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sguyennet/.ssh/idrsa):
Created directory '/home/sguyennet/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sguyennet/.ssh/idrsa.
Your public key has been saved in /home/sguyennet/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6pK2EnnNianYPjb/0YoEhZlz23tQIJwzkQ0bUytkgcg sguyennet@ubuntu
The key's randomart image is:
+---[RSA 2048]----+
|.. oXOo |
|.E.*=.o |
| = =o. . |
| + + . |
| ...=o.S |
| o.+ += |
| o +..+ o |
|. B.+o + |
| o.+++ |
+----[SHA256]-----+
37- Copy the public key to the template virtual machine.
$ ssh-copy-id sguyennet@10.10.40.254
38- Verify that you can SSH to the template virtual machine without entering a password.
$ ssh sguyennet@10.10.40.254
39- Remove the ens192 network interface configuration of the template virtual machine.
$ sudo vim /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
40- Restart the networking of the template virtual machine.
$ sudo systemctl restart networking
41- Shutdown the template virtual machine.
$ sudo shutdown now
42- Take a snapshot of the template virtual machine. This snapshot will be used to do a linked clone of the template into several virtual machines.
43- Convert the virtual machine to a template.
44- Create a folder for the template.
45- Move the template to the new folder.
Create a user to access the VMware vSphere storage from Kubernetes
1- Browse to the administration page.
2- Add a new user called "k8s-vcp@vsphere.local".
Create roles for the vSphere Cloud Provider user
1- Create a role to view the profile-driven storage.
2- Create a role to manage the Kubernetes nodes virtual machines.
3- Create a new role to manage the Kubernetes volumes.
Assign permission to the vSphere Cloud Provider user
1- Add the read-only permission at the datacenter level. Remove the propagation of the permission.
2- Add the profile-driven storage view at the vCenter level. Remove the propagation of the permission.
3- Add the manage node permission at the cluster level. This cluster is the cluster where the Kubernetes nodes will be deployed. Keep the propagation of the permission.
4- Add the manage volumes permission at the datastore level. This datastore will be the datastore where the Kubernetes volumes will be created. Remove the propagation of the permission.
Create a directory for the vSphere Cloud Provider
1- Browse to the datastore files tabulation and create a new folder. The datastore needs to be the one to which you assigned permission to in the previous steps. This folder will store the virtual disks created by the vSphere Cloud Provider.
2- Name the folder "kubevols".
Create a resource pool for the Kubernetes cluster
1- Add a new resource pool in the cluster to which you assigned permission to in the previous steps.
Launching our first Kubernetes cluster
Clone the terraform script
1- Go back to the client machine.
2- Clone the terraform-vsphere-kubespray GitHub repository.
$ git clone https://github.com/sguyennet/terraform-vsphere-kubespray.git
Configure the terraform script
1- Go to the terraform-vsphere-kubespray directory.
$ cd terraform-vsphere-kubespray
2- Edit the terraform.tfvars configuration file and fill in the different variables. Enable the anti-affinity rule for the Kubernetes master virtual machine only if your vSphere cluster supports DRS. For the network plugin, you can choose between various options like Cilium, Weave, or Flannel.
$ vim terraform.tfvars
# vCenter connection
vsphere_vcenter = "vcsa.inkubate.io"
vsphere_user = "administrator@vsphere.local"
vsphere_password = "**********"
vsphere_unverified_ssl = "true"
vsphere_datacenter = "inkubate-lab"
vsphere_drs_cluster = "Compute-01"
vsphere_resource_pool = "Compute-01/Resources/kubernetes-kubespray"
vsphere_enable_anti_affinity = "true"
vsphere_vcp_user = "k8s-vcp@vsphere.local"
vsphere_vcp_password = "**********"
vsphere_vcp_datastore = "Datastore-02"
# Kubernetes infrastructure
vm_user = "sguyennet"
vm_password = "**********"
vm_folder = "kubernetes-kubespray"
vm_datastore = "Datastore-01"
vm_network = "pg-inkubate-production-static"
vm_template = "terraform-template/ubuntu-16.04-terraform-template"
vm_linked_clone = "false"
k8s_kubespray_url = "https://github.com/kubernetes-incubator/kubespray.git"
k8s_kubespray_version = "v2.5.0"
k8s_version = "v1.10.2"
k8s_master_ips = {
"0" = "10.10.40.110"
"1" = "10.10.40.111"
"2" = "10.10.40.112"
}
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
}
k8s_haproxy_ip = "10.10.40.113"
k8s_netmask = "24"
k8s_gateway = "10.10.40.1"
k8s_dns = "10.10.40.1"
k8s_domain = "inkubate.io"
k8s_network_plugin = "weave"
k8s_weave_encryption_password = "**********"
k8s_master_cpu = "1"
k8s_master_ram = "2048"
k8s_worker_cpu = "1"
k8s_worker_ram = "2048"
k8s_haproxy_cpu = "1"
k8s_haproxy_ram = "1024"
k8s_node_prefix = "k8s-kubespray"
3- Initialize the terraform script. This step is going to download the necessary Terraform provider.
$ terraform init
3- Check what Terraform is going to deploy.
$ terraform plan
4- Deploy the Kubernetes cluster.
$ terraform apply
...
Apply complete! Resources: 17 added, 0 changed, 0 destroyed.
5- List the Kubernetes nodes.
$ kubectl --kubeconfig config/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
k8s-kubespray-master-0 Ready master 1m v1.10.2
k8s-kubespray-master-1 Ready master 1m v1.10.2
k8s-kubespray-master-2 Ready master 2m v1.10.2
k8s-kubespray-worker-0 Ready node 1m v1.10.2
k8s-kubespray-worker-1 Ready node 1m v1.10.2
k8s-kubespray-worker-2 Ready node 1m v1.10.2
Scaling the cluster
Add one or several worker nodes
1- Edit the terraform.tfvars configuration file and add a new worker node to the list of worker IPs.
$ vim terraform.tfvars
...
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
"3" = "10.10.40.123"
}
2- Add the new worker node to the cluster.
$ terraform apply -var 'action=add_worker'
...
Apply complete! Resources: 3 added, 0 changed, 2 destroyed.
3- Check that the worker node was added to the Kubernetes cluster.
$ kubectl --kubeconfig config/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
k8s-kubespray-master-0 Ready master 31m v1.10.2
k8s-kubespray-master-1 Ready master 31m v1.10.2
k8s-kubespray-master-2 Ready master 32m v1.10.2
k8s-kubespray-worker-0 Ready node 31m v1.10.2
k8s-kubespray-worker-1 Ready node 31m v1.10.2
k8s-kubespray-worker-2 Ready node 31m v1.10.2
k8s-kubespray-worker-3 Ready node 1m v1.10.2
Remove one or several worker nodes
1- Edit the terraform.tfvars configuration file and remove a new worker node from the list of worker IPs.
$ vim terraform.tfvars
...
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
}
2- Remove the new worker node from the cluster. Before being removed, the node will be drained and all the pods running on this node will be rescheduled on the other worker nodes.
$ terraform apply -var 'action=remove_worker'
...
Apply complete! Resources: 1 added, 0 changed, 3 destroyed.
3- Check that the worker node was removed from the Kubernetes cluster.
$ kubectl --kubeconfig config/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
k8s-kubespray-master-0 Ready master 36m v1.10.2
k8s-kubespray-master-1 Ready master 36m v1.10.2
k8s-kubespray-master-2 Ready master 36m v1.10.2
k8s-kubespray-worker-0 Ready node 36m v1.10.2
k8s-kubespray-worker-1 Ready node 36m v1.10.2
k8s-kubespray-worker-2 Ready node 36m v1.10.2
Upgrading the cluster to a new version of Kubernetes
1- Edit the terraform.tfvars configuration file and modify the Kubernetes version.
$ vim terraform.tfvars
...
k8s_version = "v1.10.3"
...
2- Open a new terminal on the client machine to monitor what is happening during the upgrade of the Kubernetes cluster.
$ watch -n 1 kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
k8s-kubespray-master-0 Ready master 1h v1.10.3
k8s-kubespray-master-1 Ready master 1h v1.10.3
k8s-kubespray-master-2 Ready master 1h v1.10.3
k8s-kubespray-worker-0 Ready node 1h v1.10.3
k8s-kubespray-worker-1 Ready,SchedulingDisabled node 1h v1.10.2
k8s-kubespray-worker-2 Ready node 1h v1.10.2
3- Upgrade the Kubernetes version. The upgrade will be done node by node. The worker nodes will be drained and all the pods on the drained node will be rescheduled on the other nodes. This should avoid downtime of your application running in the Kubernetes cluster as long as you scale your application to at least two replicas.
$ terraform apply -var 'action=upgrade'
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Checking that the vSphere Cloud Provider is working
Create a storage class for our application
1- Create the file redis-sc.yaml.
$ vim redis-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: thin-disk
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: thin
2- Create the storage class.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f redis-sc.yaml
Create a persistent storage claim for the Redis master node
1- Create the file redis-master-claim.yaml.
$ vim redis-master-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: redis-master-claim
annotations:
volume.beta.kubernetes.io/storage-class: thin-disk
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
2- Create the persistent storage claim.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f redis-master-claim.yaml
3- Check that the virtual disk for the Redis master pod was created in the "kubevols" directory.
Create a persistent storage claim for the Redis slave node
1- Create the file redis-slave-claim.yaml.
$ vim redis-slave-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: redis-slave-claim
annotations:
volume.beta.kubernetes.io/storage-class: thin-disk
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
2- Create the persistent storage claim.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f redis-slave-claim.yaml
3- Check that the virtual disk for the Redis slave pod was created in the "kubevols" directory.
Launch the application
1- Create the file guestbook-all-in-one.yaml.
$ vim guestbook-all-in-one.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
tier: backend
role: master
spec:
ports:
# the port that this service should serve on
- port: 6379
targetPort: 6379
selector:
app: redis
tier: backend
role: master
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: redis-master
# these labels can be applied automatically
# from the labels in the pod template if not set
# labels:
# app: redis
# role: master
# tier: backend
spec:
# this replicas value is default
# modify it according to your case
replicas: 1
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# matchLabels:
# app: guestbook
# role: master
# tier: backend
template:
metadata:
labels:
app: redis
role: master
tier: backend
spec:
containers:
- name: master
image: gcr.io/google_containers/redis:e2e # or just image: redis
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
volumeMounts:
- name: redis-master-data
mountPath: /data
volumes:
- name: redis-master-data
persistentVolumeClaim:
claimName: redis-master-claim
---
apiVersion: v1
kind: Service
metadata:
name: redis-slave
labels:
app: redis
tier: backend
role: slave
spec:
ports:
# the port that this service should serve on
- port: 6379
selector:
app: redis
tier: backend
role: slave
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: redis-slave
# these labels can be applied automatically
# from the labels in the pod template if not set
# labels:
# app: redis
# role: slave
# tier: backend
spec:
# this replicas value is default
# modify it according to your case
replicas: 1
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# matchLabels:
# app: guestbook
# role: slave
# tier: backend
template:
metadata:
labels:
app: redis
role: slave
tier: backend
spec:
containers:
- name: slave
image: gcr.io/google_samples/gb-redisslave:v1
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
# If your cluster config does not include a dns service, then to
# instead access an environment variable to find the master
# service's host, comment out the 'value: dns' line above, and
# uncomment the line below.
# value: env
ports:
- containerPort: 6379
volumeMounts:
- name: redis-slave-data
mountPath: /data
volumes:
- name: redis-slave-data
persistentVolumeClaim:
claimName: redis-slave-claim
---
apiVersion: v1
kind: Service
metadata:
name: frontend
labels:
app: guestbook
tier: frontend
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
type: NodePort
ports:
# the port that this service should serve on
- port: 80
selector:
app: guestbook
tier: frontend
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: frontend
# these labels can be applied automatically
# from the labels in the pod template if not set
# labels:
# app: guestbook
# tier: frontend
spec:
# this replicas value is default
# modify it according to your case
replicas: 4
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# matchLabels:
# app: guestbook
# tier: frontend
template:
metadata:
labels:
app: guestbook
tier: frontend
spec:
containers:
- name: php-redis
image: gcr.io/google-samples/gb-frontend:v4
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
# If your cluster config does not include a dns service, then to
# instead access environment variables to find service host
# info, comment out the 'value: dns' line above, and uncomment the
# line below.
# value: env
ports:
- containerPort: 80
3- Start the application.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f guestbook-all-in-one.yaml
4- Get the port on which the application is running.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf describe service frontend | grep NodePort
5- Browse to http://[ip_of_one_of_your_kubernetes_node]:[your_application_port].
6- Add some messages to the guestbook.
7- Destroy the application.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf delete -f guestbook-all-in-one.yaml
8- Check that the application was deleted.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf get pods
9- Create a new application.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f guestbook-all-in-one.yaml
10- Get the new port on which the application is running.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf describe service frontend | grep NodePort
5- Browse to http://[ip_of_one_of_your_kubernetes_node]:[your_application_port].
Your messages should still be there.
Accessing the Kubernetes dashboard
1- Create an admin user manifest.
$ vim kubernetes-dashboard-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
2- Create the admin user.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf create -f kubernetes-dashboard-admin.yaml
3- Get the admin user token.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf -n kube-system describe secret $(kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf -n kube-system get secret | grep admin-user | awk '{print $1}')
4- Copy the token.
5- Start the proxy to access the dashboard.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf proxy
6- Browse to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy.
7- Select Token and paste the token from step 4.
Installing Heapster
Heapster is a small monitoring tool. It collects the performance metrics of the different pods running in the cluster and displays them in the Kubernetes dashboard.
1- Create a manifest for Heapster.
$ vim heapster.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: heapster
spec:
serviceAccountName: heapster
containers:
- name: heapster
image: gcr.io/google_containers/heapster-amd64:v1.4.2
imagePullPolicy: IfNotPresent
command:
- /heapster
- --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: Heapster
name: heapster
namespace: kube-system
spec:
ports:
- port: 80
targetPort: 8082
selector:
k8s-app: heapster
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:heapster
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system
2- Deploy Heapster.
$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf create -f heapster.yaml
Destroying the Kubernetes cluster
1- Go to the terraform-vsphere-kubespray directory.
$ cd ~/terraform-vsphere-kubespray
2- Destroy the deployment
$ terraform destroy
Conclusion
You now have a way to easily deploy, scale, upgrade and destroy a Kubernetes cluster on VMware vSphere. This will allow you to do even more testing without worrying about breaking the cluster as you can spin up a new one automatically.
If you find an issue with the Terraform script, please let me know by opening an issue on GitHub: https://github.com/sguyennet/terraform-vsphere-kubespray