During the second day of VMWorld 2017 in Las Vegas, VMware, Google, and Pivotal unveiled their partnership regarding Pivotal Container Service aka PKS. This product will be designed to deploy, in an automatic fashion, enterprise grade Kubernetes clusters on VMware vSphere. PKS will be based on the Pivotal open source project Kubo. Kubo stands for Kubernetes Bosh as it leverages Bosh Director to deploy Kubernetes.

If you would like a taste of what PKS will be, here is how to deploy Kubernetes on VMware vSphere with Kubo.

Preriquisites

For this lab we will need an Ubuntu 16.04 client machine with access to the VMware vSphere management network. This client machine will have the IP 10.10.20.50 and will be used to deploy Bosh Director and deploy a Kubernetes cluster.
Once deployed, the Bosh Director virtual machine will have the IP 10.10.20.51. The HAProxy master IP of our Kubernetes cluster deployment will be 10.10.20.53.

Warning: You will need at least a 1TB datastore to do this lab.

Install the Bosh client

1- Install the dependencies for the Bosh client.

$ sudo apt-get install build-essential zlibc zlib1g-dev ruby ruby-dev \
openssl libxslt-dev libxml2-dev libssl-dev libreadline6 libreadline6-dev \
libyaml-dev libsqlite3-dev sqlite3

2- Download the Bosh client on the Ubuntu 16.04 client machine.

$ wget https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-2.0.28-linux-amd64

3- Change the permissions of the Bosh client.

$ sudo chown root:root bosh-cli-2.0.28-linux-amd64
$ sudo chmod 755 bosh-cli-2.0.28-linux-amd64

4- Move the Bosh client to /usr/local/bin.

$ sudo mv bosh-cli-2.0.28-linux-amd64 /usr/local/bin/bosh

Deploy the Bosh Director virtual machine

1- Create a kubo ressource pool in VMware vSphere.

2- Clone the Bosh deployment GitHub project.

$ git clone https://github.com/cloudfoundry/bosh-deployment

3- Enter the bosh-deployment directory.

$ cd bosh-deployment

4- Deploy Bosh Director.

$ bosh create-env bosh.yml \
--state=bosh-state.json \
--vars-store=bosh-creds.yml \
-o vsphere/cpi.yml \
-o uaa.yml \
-o misc/powerdns.yml \
-o credhub.yml \
-v director_name=bosh \
-v internal_cidr=10.10.20.0/24 \
-v internal_gw=10.10.20.1 \
-v internal_ip=10.10.20.51 \
-v network_name='VM Network' \
-v vcenter_dc=inkubate-lab \
-v vcenter_ds='Datastore-03' \
-v vcenter_ip=10.10.20.10 \
-v vcenter_user='administrator@vsphere.local' \
-v vcenter_password='[your_vsphere_password]' \
-v vcenter_templates=kubo \
-v vcenter_vms=kubo \
-v vcenter_disks=Datastore-03 \
-v vcenter_cluster=Compute-01 \
-v dns_recursor_ip=10.10.20.1

5- Move the Bosh Director virtual machine to the kubo resource pool.

6- Move the stemcell virtual machine to the kubo resource pool.

Access Bosh Director with the Bosh client

1- Create an environment alias.

$ bosh alias-env kubo -e 10.10.20.51 --ca-cert <(bosh int ./bosh-creds.yml --path /director_ssl/ca)

2- Create a Bosh client environment variable.

$ export BOSH_CLIENT=admin

3- Create a Bosh client secret environment variable.

$ export BOSH_CLIENT_SECRET=$(bosh int ./bosh-creds.yml --path /admin_password)

4- Verify that you can connect to Bosh Director.

$ bosh -e kubo env

Deploy a Kubernetes cluster

1- Go back to the home directory.

$ cd ~

2- Clone the Kubo deployment GitHub project.

$ git clone -b v0.7.0 https://github.com/cloudfoundry-incubator/kubo-deployment

3- Enter the kubo-deployment directory.

$ cd kubo-deployment

4- Create a cloud-config file.

$ bosh int configurations/vsphere/cloud-config.yml \
-o manifests/ops-files/k8s-haproxy-static-ips-vsphere.yml \
-v director_name=bosh \
-v internal_cidr=10.10.20.0/24  \
-v internal_gw=10.10.20.1 \
-v dns_recursor_ip=10.10.20.1 \
-v internal_ip=10.10.20.52 \
-v kubernetes_master_host=10.10.20.53 \
-v worker_haproxy_ip_addresses=10.10.20.54 \
-v reserved_ips=[10.10.20.1-10.10.20.51] \
-v network_name='VM Network' \
-v deployments_network='VM Network' \
-v vcenter_cluster='Compute-01' \
-v vcenter_rp="k8s-01" > cloud-config-01.yml

5- Update cloud config.

$ bosh -e kubo update-cloud-config cloud-config-01.yml

6- Create a kubo-deployment file.

$ bosh int manifests/kubo.yml \
-o manifests/ops-files/master-haproxy-vsphere.yml \
-o manifests/ops-files/worker-haproxy.yml \
-v deployments_network='VM Network' \
-v kubo-admin-password="[your_password]" \
-v kubelet-password="[your_password]" \
-v kubernetes_master_port=443 \
-v kubernetes_master_host=10.10.20.53 \
-v deployment_name=kubo-cluster-01 \
-v worker_haproxy_tcp_frontend_port=1234 \
-v worker_haproxy_tcp_backend_port=4231 > kubo-deployment-01.yml

7- Download the stemcell.

$ wget https://s3.amazonaws.com/bosh-core-stemcells/vsphere/bosh-stemcell-3421.11-vsphere-esxi-ubuntu-trusty-go_agent.tgz

8- Upload the stemcell to Bosh Director.

$ bosh -e kubo upload-stemcell ./bosh-stemcell-3421.11-vsphere-esxi-ubuntu-trusty-go_agent.tgz

9- Move the stemcell to the kubo resource pool.

10- Download the Kubo release.

$ wget https://github.com/cloudfoundry-incubator/kubo-release/releases/download/v0.7.0/kubo-release-0.7.0.tgz

11- Upload the Kubo release to Bosh Director.

$ bosh -e kubo upload-release kubo-release-0.7.0.tgz

12- Create the k8s-01 resource pool.

13- Deploy a Kubernetes cluster.

$ bosh -e kubo -d kubo-cluster-01 deploy kubo-deployment-01.yml

14- Verify the deployment.

$ bosh -e kubo deployments

15- List the instances of the deployment.

$ bosh -e kubo instances

Access the Kubernetes cluster

Install the credhub client

1- Go back to your home directory.

$ cd ~

2- Download credhub.

$ wget https://github.com/cloudfoundry-incubator/credhub-cli/releases/download/1.4.0/credhub-linux-1.4.0.tgz

3- Extract the credhub tarball.

$ tar xvzf credhub-linux-1.4.0.tgz

4- Modify the permissions of credhub.

$ sudo chown root:root credhub
$ sudo chmod 755 credhub

5- Move credhub to /usr/local/bin.

$ sudo mv credhub /usr/local/bin

Install the kubectl client

1- Download kubectl.

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

2- Modify the permissions of kubectl.

$ sudo chown root:root kubectl
$ sudo chmod 755 kubectl

3- Move kubectl to /usr/local/bin.

$ sudo kubectl /usr/local/bin

Login to Bosh Director credhub

1- Go to the bosh-deployment directory.

$ cd bosh-deployment

2- Generate the CA certificate.

$ bosh -e kubo int "./bosh-creds.yml" --path="/uaa_ssl/ca" > credhubca.crt

3- Generate the certificate.

$ bosh -e kubo int "./bosh-creds.yml" --path="/credhub_tls/ca" > credhub.crt

4- Login with credhub.

$ credhub login \
-u credhub-cli \
-p $(bosh -e kubo int "./bosh-creds.yml" --path "/credhub_cli_password") \
-s "https://10.10.20.51:8844" --ca-cert credhubca.crt --ca-cert credhub.crt

Create the Kubernetes context

1- Generate the Kubernetes certificate.

$ bosh int <(credhub get -n "/bosh/kubo-cluster-01/tls-kubernetes" --output-json) \
--path=/value/ca > kubecert.crt

2- Set the Kubernetes cluster.

$ kubectl config set-cluster "kubo-cluster-01" \
--server="https://10.10.20.53" \
--certificate-authority=kubecert.crt \
--embed-certs=true

3- Set the Kubernetes credentials.

$ kubectl config set-credentials "kubo-cluster-admin" \
--token="[your_password]"

4- Set the Kubernetes context.

$ kubectl config set-context "kubo-cluster-01" \
--cluster="kubo-cluster-01" \
--user="kubo-cluster-admin"

List the Kubernetes pods

1- Load the Kubernetes context.

$ kubectl config use-context "kubo-cluster-01"

2- List the Kubernetes system pods.

$ kubectl get pods --namespace=kube-system

Scale Kubernetes cluster

Scale in (remove a worker node)

1- Go in the kubo-deployment directory.

$ cd ~/kubo-deployment

2- Edit the kubo-deployment-01.yml file.

$ vim kubo-deployment-01.yml

3- On line 83, remove an instance.

...
instances: 2
...

4- Update the Kubernetes deployment.

$ bosh -e kubo -d kubo-cluster-01 deploy kubo-deployment-01.yml

Scale out (add a worker node)

1- Go in the kubo-deployment directory.

$ cd ~/kubo-deployment

2- Edit the kubo-deployment-01.yml file.

$ vim kubo-deployment-01.yml

3- On line 83, add an instance.

...
instances: 3
...

4- Update the Kubernetes deployment.

$ bosh -e kubo -d kubo-cluster-01 deploy kubo-deployment-01.yml