Install and manage automatically a Kubernetes cluster on VMware vSphere with

 Terraform and Kubespray 10%

(http://blog.inkubate.io) 3289 words - 12 min read.
 HOME (HTTP://BLOG.INKUBATE.IO)


Install and manage

automatically a Kubernetes
cluster on VMware vSphere
with Terraform and
Kubespray
20 June 2018

If you already completed the tutorials Deploy Kubernetes 1.9 from scratch on VMware
vSphere (https://blog.inkubate.io/deploy-kubernetes-1-9-from-scratch-on-vmware-
vsphere/) and Install and con gure a multi-master Kubernetes cluster with kubeadm
(https://blog.inkubate.io/install-and-con gure-a-multi-master-kubernetes-cluster-
with-kubeadm/), you should have a pretty good understanding of how a multi-master
Kubernetes cluster is structured. You are now probably looking for a way to automate
the deployment of your lab so you don't have to follow all these painful steps each
time you want to deploy a clean environment.

 Kubespray is a Kubernetes incubator
Install and manage project. It isacomposed
automatically of Ansible
Kubernetes cluster playbook
on VMware and
vSphere with
 Terraform and Kubespray
automates the deployment of a Kubernetes cluster on an existing infrastructure.
10%
(http://blog.inkubate.io) 3289 words - 12 min read.

In this lab, we will use Terraform to deploy our infrastructure on VMware vSphere
and, in a second stage, Terraform will call Kubespray to install and con gure
Kubernetes for us. This automation not only supports the deployment, but also adds
worker nodes to the Kubernetes cluster, removes worker nodes from the Kubernetes
cluster, upgrades the version of Kubernetes, and destroys the Kubernetes cluster.

We will take advantage of the fact that our infrastructure will be running on VMware
vSphere and that Kubespray supports the con guration of vSphere Cloud Provider.
This will allow us to use the vSphere storage as persistent volumes in the Kubernetes
cluster as seen in the article Use vSphere Storage as Kubernetes persistent volumes
(https://blog.inkubate.io/use-vsphere-storage-as-kubernetes-persistant-volumes/).

Requirements
For this lab, you will need a con gured VMware vSphere environment. You will also
need a Ubuntu 16.04 desktop client machine. This machine will be used to execute the
Terraform script and to access the Kubernetes dashboard. This machine needs to be

 on a network that hasInstall
access andto the vCenter
manage API as
automatically Terraformcluster
a Kubernetes will need this to
on VMware deploy
vSphere with
 Terraform and Kubespray
and con gure the virtual machines.
10%
(http://blog.inkubate.io) 3289 words - 12 min read.

For the Kubernetes infrastructure, we are going to deploy three master nodes. They
will have the IPs 10.10.40.110, 10.10.40.111, and 10.10.40.112. In front of these master
nodes, we will deploy a HAProxy load balancer with the IP 10.10.40.113.

Regarding the Kubernetes worker nodes, we will deploy them on the IP range
10.10.40.120-10.10.40.123.

We will also need a Ubuntu 16.04 vSphere template and a speci c user for the vSphere
Cloud Provider. The step to con gure these will be described later on.

Installing the requirements on

the Ubuntu 16.04 client
machine
1- Install Git.

$ sudo apt-get install git

2- Install Unzip.

$ sudo apt-get install unzip

3- Install Python 2.7.

$ sudo apt-get install python

4- Install Pip.

$ sudo apt-get install python-pip 

 5- Install Ansible. Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
$ pip install ansible

6- Install the Python netaddr library.

$ pip install netaddr

7- Install Terraform.

$ wget https://releases.hashicorp.com/terraform (http://releases.hashicorp.com/te

$ unzip terraform_0.11.7_linux_amd64.zip

$ sudo mv terraform /usr/local/bin

8- Install Kubectl.

$ curl -s https://packages.cloud.google.com/apt (http://packages.cloud.google.com

sudo apt-key add -

$ sudo vim /etc/apt/sources.list.d/kubernetes.list

deb http://apt.kubernetes.io/ kubernetes-xenial main

$ sudo apt-get update

$ sudo apt-get install kubectl


 Configuring the requirements
Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

on VMware vSphere
Create a Ubuntu 16.04 template
1- Create a new virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

2- Enter the name "ubuntu-16.04-terraform-template" as the name of the virtual

machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

3- Choose the position of the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

4- Choose the compatibility of the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

5- Choose Ubuntu as a guest OS type.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

6- Set the type of the SCSI controller to "VMware Paravirtual".


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

7- Select the network of the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

8- Select the Ubuntu 16.04 ISO.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

9- Connect the CD drive when the virtual machine boots.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

10- Complete the creation of the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

11- Power on the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

12- Open the console of the virtual machine.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

13- Select the language of the installer.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

14- Install Ubuntu 16.04 server.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

15- Choose the language of the system.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

16- Choose the location of the system.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

17- Choose the mapping of the keyboard.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

18- Con gure the network card with a temporary IP. We will uncon gure it later on.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

19- Choose the default hostname.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

20- Leave the domain name empty.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

21- Choose the name of your user.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

22- Choose a password for your user.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

23- Choose if you would like to encrypt your disk or not.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

24- Con gure the timezone of the system.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

25- Select the partitioning method.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

26- Select the disk to install Ubuntu 16.04.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

27- Con gure a proxy if you are using one to access the Internet.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

28- Choose to install the security update automatically.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

29- Install OpenSSH.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

30- Install GRUB.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

31- Complete the installation.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

32- Once the virtual machine has rebooted, SSH to it from the client machine.

$ ssh [email protected]

33- Upgrade the system.

$ sudo apt-get update

$ sudo apt-get upgrade

34- Upgrade the kernel to a version above 4.8. You can skip this step if you are not
planning to use Cilium as an overlay network for your Kubernetes cluster.

$ sudo apt-get install linux-image-4.15.0-15-generic \

linux-image-extra-4.15.0-15-generic


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ sudo reboot
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
35- Allow your user to use sudo without a password.

$ sudo visudo
...
sguyennet ALL=(ALL) NOPASSWD: ALL

36- Generate a private and a public key on the client machine. Leave the passphrase
empty.

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sguyennet/.ssh/idrsa):
Created directory '/home/sguyennet/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sguyennet/.ssh/idrsa.
Your public key has been saved in /home/sguyennet/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6pK2EnnNianYPjb/0YoEhZlz23tQIJwzkQ0bUytkgcg sguyennet@ubuntu
The key's randomart image is:
+---[RSA 2048]----+
|.. oXOo |
|.E.*=.o |
| = =o. . |
| + + . |
| ...=o.S |
| o.+ += |
| o +..+ o |
|. B.+o + |
| o.+++ |
+----[SHA256]-----+

37- Copy the public key to the template virtual machine.

$ ssh-copy-id [email protected]


 38- Verify that you can SSHand
Install to manage
the template virtual
automatically machine without
a Kubernetes cluster on entering a
VMware vSphere with

password.
Terraform and Kubespray
3289 words - 12 min read.
10%
(http://blog.inkubate.io)

$ ssh [email protected]

39- Remove the ens192 network interface con guration of the template virtual
machine.

$ sudo vim /etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback

40- Restart the networking of the template virtual machine.

$ sudo systemctl restart networking

41- Shutdown the template virtual machine.

$ sudo shutdown now

42- Take a snapshot of the template virtual machine. This snapshot will be used to do
a linked clone of the template into several virtual machines.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

43- Convert the virtual machine to a template.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

44- Create a folder for the template.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

45- Move the template to the new folder.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Create a user to access the VMware vSphere storage

from Kubernetes
1- Browse to the administration page.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

2- Add a new user called "[email protected]".


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Create roles for the vSphere Cloud Provider user

1- Create a role to view the pro le-driven storage.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

2- Create a role to manage the Kubernetes nodes virtual machines.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

3- Create a new role to manage the Kubernetes volumes.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Assign permission to the vSphere Cloud Provider user

1- Add the read-only permission at the datacenter level. Remove the propagation of
the permission.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

2- Add the pro le-driven storage view at the vCenter level. Remove the propagation
of the permission.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

3- Add the manage node permission at the cluster level. This cluster is the cluster
where the Kubernetes nodes will be deployed. Keep the propagation of the
permission.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

4- Add the manage volumes permission at the datastore level. This datastore will be
the datastore where the Kubernetes volumes will be created. Remove the propagation
of the permission.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Create a directory for the vSphere Cloud Provider

1- Browse to the datastore les tabulation and create a new folder. The datastore
needs to be the one to which you assigned permission to in the previous steps. This
folder will store the virtual disks created by the vSphere Cloud Provider.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

2- Name the folder "kubevols".


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Create a resource pool for the Kubernetes cluster

1- Add a new resource pool in the cluster to which you assigned permission to in the
previous steps.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Launching our first

Kubernetes cluster
Clone the terraform script
1- Go back to the client machine.

2- Clone the terraform-vsphere-kubespray GitHub repository.

$ git clone https://github.com/sguyennet/terraform-vsphere-kubespray.git (http://

Configure the terraform script

1- Go to the terraform-vsphere-kubespray directory. 
 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ cd terraform-vsphere-kubespray
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
2- Edit the terraform.tfvars con guration le and ll in the different variables. Enable
the anti-af nity rule for the Kubernetes master virtual machine only if your vSphere
cluster supports DRS. For the network plugin, you can choose between various
options like Cilium, Weave, or Flannel.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ vim terraform.tfvars
Terraform and Kubespray
# vCenter connection3289 words - 12 min read.
10%
(http://blog.inkubate.io)
vsphere_vcenter = "vcsa.inkubate.io"
vsphere_user = "[email protected]"
vsphere_password = "**********"
vsphere_unverified_ssl = "true"
vsphere_datacenter = "inkubate-lab"
vsphere_drs_cluster = "Compute-01"
vsphere_resource_pool = "Compute-01/Resources/kubernetes-kubespray"
vsphere_enable_anti_affinity = "true"
vsphere_vcp_user = "[email protected]"
vsphere_vcp_password = "**********"
vsphere_vcp_datastore = "Datastore-02"

# Kubernetes infrastructure
vm_user = "sguyennet"
vm_password = "**********"
vm_folder = "kubernetes-kubespray"
vm_datastore = "Datastore-01"
vm_network = "pg-inkubate-production-static"
vm_template = "terraform-template/ubuntu-16.04-terraform-template"
vm_linked_clone = "false"
k8s_kubespray_url = "https://github.com/kubernetes-incubator/kubespray.g
k8s_kubespray_version = "v2.5.0"
k8s_version = "v1.10.2"
k8s_master_ips = {
"0" = "10.10.40.110"
"1" = "10.10.40.111"
"2" = "10.10.40.112"
}
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
} 
 Install and manage automatically a Kubernetes cluster on VMware vSphere with
k8s_haproxy_ip = "10.10.40.113"
 Terraform and Kubespray
k8s_netmask = "24"3289 words - 12 min read.
10%
(http://blog.inkubate.io)
k8s_gateway = "10.10.40.1"
k8s_dns = "10.10.40.1"
k8s_domain = "inkubate.io"
k8s_network_plugin = "weave"
k8s_weave_encryption_password = "**********"
k8s_master_cpu = "1"
k8s_master_ram = "2048"
k8s_worker_cpu = "1"
k8s_worker_ram = "2048"
k8s_haproxy_cpu = "1"
k8s_haproxy_ram = "1024"
k8s_node_prefix = "k8s-kubespray"

3- Initialize the terraform script. This step is going to download the necessary
Terraform provider.

$ terraform init

3- Check what Terraform is going to deploy.

$ terraform plan

4- Deploy the Kubernetes cluster.

$ terraform apply
...
Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

5- List the Kubernetes nodes.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

NAME

$ kubectl --kubeconfig config/admin.conf get nodes
Terraform and Kubespray
STATUS ROLES AGE VERSION
10%
(http://blog.inkubate.io) 3289 words - 12 min read.
k8s-kubespray-master-0 Ready master 1m v1.10.2
k8s-kubespray-master-1 Ready master 1m v1.10.2
k8s-kubespray-master-2 Ready master 2m v1.10.2
k8s-kubespray-worker-0 Ready node 1m v1.10.2
k8s-kubespray-worker-1 Ready node 1m v1.10.2
k8s-kubespray-worker-2 Ready node 1m v1.10.2

Scaling the cluster

Add one or several worker nodes
1- Edit the terraform.tfvars con guration le and add a new worker node to the list of
worker IPs.

$ vim terraform.tfvars
...
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
"3" = "10.10.40.123"
}

2- Add the new worker node to the cluster.

$ terraform apply -var 'action=add_worker'

...
Apply complete! Resources: 3 added, 0 changed, 2 destroyed.

3- Check that the worker node was added to the Kubernetes cluster.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

NAME

$ kubectl --kubeconfig config/admin.conf get nodes
Terraform and Kubespray
STATUS ROLES AGE VERSION
10%
(http://blog.inkubate.io) 3289 words - 12 min read.
k8s-kubespray-master-0 Ready master 31m v1.10.2
k8s-kubespray-master-1 Ready master 31m v1.10.2
k8s-kubespray-master-2 Ready master 32m v1.10.2
k8s-kubespray-worker-0 Ready node 31m v1.10.2
k8s-kubespray-worker-1 Ready node 31m v1.10.2
k8s-kubespray-worker-2 Ready node 31m v1.10.2
k8s-kubespray-worker-3 Ready node 1m v1.10.2

Remove one or several worker nodes

1- Edit the terraform.tfvars con guration le and remove a new worker node from the
list of worker IPs.

$ vim terraform.tfvars
...
k8s_worker_ips = {
"0" = "10.10.40.120"
"1" = "10.10.40.121"
"2" = "10.10.40.122"
}

2- Remove the new worker node from the cluster. Before being removed, the node
will be drained and all the pods running on this node will be rescheduled on the other
worker nodes.

$ terraform apply -var 'action=remove_worker'

...
Apply complete! Resources: 1 added, 0 changed, 3 destroyed.

3- Check that the worker node was removed from the Kubernetes cluster.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

NAME

$ kubectl --kubeconfig config/admin.conf get nodes
Terraform and Kubespray
STATUS ROLES AGE VERSION
10%
(http://blog.inkubate.io) 3289 words - 12 min read.
k8s-kubespray-master-0 Ready master 36m v1.10.2
k8s-kubespray-master-1 Ready master 36m v1.10.2
k8s-kubespray-master-2 Ready master 36m v1.10.2
k8s-kubespray-worker-0 Ready node 36m v1.10.2
k8s-kubespray-worker-1 Ready node 36m v1.10.2
k8s-kubespray-worker-2 Ready node 36m v1.10.2

Upgrading the cluster to a

new version of Kubernetes
1- Edit the terraform.tfvars con guration le and modify the Kubernetes version.

$ vim terraform.tfvars
...
k8s_version = "v1.10.3"
...

2- Open a new terminal on the client machine to monitor what is happening during
the upgrade of the Kubernetes cluster.

$ watch -n 1 kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf

NAME STATUS ROLES AGE VERSION
k8s-kubespray-master-0 Ready master 1h v1.10.3
k8s-kubespray-master-1 Ready master 1h v1.10.3
k8s-kubespray-master-2 Ready master 1h v1.10.3
k8s-kubespray-worker-0 Ready node 1h v1.10.3
k8s-kubespray-worker-1 Ready,SchedulingDisabled node 1h v1.10.2
k8s-kubespray-worker-2 Ready node 1h v1.10.2

3- Upgrade the Kubernetes version. The upgrade will be done node by node. The
worker nodes will be drained and all the pods on the drained node will be rescheduled
on the other nodes. This should avoid downtime of your application running in the

 Kubernetes cluster asInstall
long and
as you scale
manage your application
automatically to atcluster
a Kubernetes leaston
two replicas.
VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
$ terraform apply -var 'action=upgrade'
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Checking that the vSphere

Cloud Provider is working
Create a storage class for our application
1- Create the le redis-sc.yaml.

$ vim redis-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1 (http://storage.k8s.io/v1)
metadata:
name: thin-disk
provisioner: kubernetes.io/vsphere-volume (http://kubernetes.io/vsphere-volume)
parameters:
diskformat: thin

2- Create the storage class.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f r

Create a persistent storage claim for the Redis master

node
1- Create the le redis-master-claim.yaml.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ vim redis-master-claim.yaml
Terraform and Kubespray
kind: PersistentVolumeClaim
10%
(http://blog.inkubate.io) 3289 words - 12 min read.
apiVersion: v1
metadata:
name: redis-master-claim
annotations:
volume.beta.kubernetes.io/storage- (http://volume.beta.kubernetes.io/storage-
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi

2- Create the persistent storage claim.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f r

3- Check that the virtual disk for the Redis master pod was created in the "kubevols"
directory.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Create a persistent storage claim for the Redis slave

node
1- Create the le redis-slave-claim.yaml.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ vim redis-slave-claim.yaml
Terraform and Kubespray
kind: PersistentVolumeClaim
10%
(http://blog.inkubate.io) 3289 words - 12 min read.
apiVersion: v1
metadata:
name: redis-slave-claim
annotations:
volume.beta.kubernetes.io/storage- (http://volume.beta.kubernetes.io/storage-
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi

2- Create the persistent storage claim.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f r

3- Check that the virtual disk for the Redis slave pod was created in the "kubevols"
directory.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Launch the application

1- Create the le guestbook-all-in-one.yaml.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ vim guestbook-all-in-one.yaml
apiVersion: v1
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
kind: Service
metadata:
name: redis-master
labels:
app: redis
tier: backend
role: master
spec:
ports:
# the port that this service should serve on
- port: 6379
targetPort: 6379
selector:
app: redis
tier: backend
role: master
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: redis-master
# these labels can be applied automatically
# from the labels in the pod template if not set
# labels:
# app: redis
# role: master
# tier: backend
spec:
# this replicas value is default
# modify it according to your case
replicas: 1
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# matchLabels:
# app: guestbook
# role: master
# tier: backend
template:
metadata: 
 labels: Install and manage automatically a Kubernetes cluster on VMware vSphere with
app: redis Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
role: master
tier: backend
spec:
containers:
- name: master
image: gcr.io/google_containers/redis:e2e (http://gcr.io/google_container
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
volumeMounts:
- name: redis-master-data
mountPath: /data
volumes:
- name: redis-master-data
persistentVolumeClaim:
claimName: redis-master-claim
---
apiVersion: v1
kind: Service
metadata:
name: redis-slave
labels:
app: redis
tier: backend
role: slave
spec:
ports:
# the port that this service should serve on
- port: 6379
selector:
app: redis
tier: backend
role: slave
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
 name: redis-slave Install and manage automatically a Kubernetes cluster on VMware vSphere with

# these labels canTerraform and Kubespray
be applied automatically 10%
(http:/ 3289 words - 12 min read.
#/blog.inkubate.io)
from the labels in the pod template if not set
# labels:
# app: redis
# role: slave
# tier: backend
spec:
# this replicas value is default
# modify it according to your case
replicas: 1
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# matchLabels:
# app: guestbook
# role: slave
# tier: backend
template:
metadata:
labels:
app: redis
role: slave
tier: backend
spec:
containers:
- name: slave
image: gcr.io/google_samples/gb-redisslave:v1 (http://gcr.io/google_sampl
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
# If your cluster config does not include a dns service, then to
# instead access an environment variable to find the master
# service's host, comment out the 'value: dns' line above, and
# uncomment the line below.
# value: env
ports:
- containerPort: 6379 
 volumeMounts:
Install and manage automatically a Kubernetes cluster on VMware vSphere with
- name: redis-slave-data
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
mountPath: /data
volumes:
- name: redis-slave-data
persistentVolumeClaim:
claimName: redis-slave-claim
---
apiVersion: v1
kind: Service
metadata:
name: frontend
labels:
app: guestbook
tier: frontend
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
type: NodePort
ports:
# the port that this service should serve on
- port: 80
selector:
app: guestbook
tier: frontend
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: frontend
# these labels can be applied automatically
# from the labels in the pod template if not set
# labels:
# app: guestbook
# tier: frontend
spec:
# this replicas value is default
# modify it according to your case
replicas: 4
# selector can be applied automatically
# from the labels in the pod template if not set
# selector: 
 # matchLabels:Install and manage automatically a Kubernetes cluster on VMware vSphere with
# app: guestbook
Terraform and Kubespray 10%
(http:/ 3289 words - 12 min read.
#/blog.inkubate.io)
tier: frontend
template:
metadata:
labels:
app: guestbook
tier: frontend
spec:
containers:
- name: php-redis
image: gcr.io/google-samples/gb-frontend:v4 (http://gcr.io/google-samples
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
# If your cluster config does not include a dns service, then to
# instead access environment variables to find service host
# info, comment out the 'value: dns' line above, and uncomment the
# line below.
# value: env
ports:
- containerPort: 80

3- Start the application.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f g

4- Get the port on which the application is running.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf describe s

5- Browse to http://[ip_of_one_of_your_kubernetes_node]:
[your_application_port] (). 
 6- Add some messages to the
Install andguestbook.
manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
7- Destroy the application.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf delete -f

8- Check that the application was deleted.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf get pods

9- Create a new application.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf apply -f g

10- Get the new port on which the application is running.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf describe s

5- Browse to http://[ip_of_one_of_your_kubernetes_node]:
[your_application_port] ().

Your messages should still be there.

Accessing the Kubernetes

dashboard
1- Create an admin user manifest.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

$ vim kubernetes-dashboard-admin.yaml
apiVersion: v1
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1 (http://rbac.authorization.k8s.io/v
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system

2- Create the admin user.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf create -f

3- Get the admin user token.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf -n kube-sy

4- Copy the token.

5- Start the proxy to access the dashboard.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf proxy


 6- Browse to http://localhost
Install and8001/api/v1/namespaces/kube-
manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray
system/services/https:kubernetes-dashboard:/proxy ().
10%
(http://blog.inkubate.io) 3289 words - 12 min read.

7- Select Token and paste the token from step 4.


 Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Installing Heapster
Heapster is a small monitoring tool. It collects the performance metrics of the
different pods running in the cluster and displays them in the Kubernetes dashboard.

1- Create a manifest for Heapster.


 $ vim heapster.yaml Install and manage automatically a Kubernetes cluster on VMware vSphere with

apiVersion: v1
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: heapster
spec:
serviceAccountName: heapster
containers:
- name: heapster
image: gcr.io/google_containers/heapster-amd64:v1.4.2 (http://gcr.io/goog
imagePullPolicy: IfNotPresent
command:
- /heapster
- --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service (http://kubernetes.io/cluster-service): 'true'
kubernetes.io/name (http://kubernetes.io/name): Heapster
name: heapster
namespace: kube-system
spec:
ports: 
 - port: 80 Install and manage automatically a Kubernetes cluster on VMware vSphere with

targetPort: 8082Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.
selector:
k8s-app: heapster
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1 (http://k8s.io/v1beta1)
metadata:
name: heapster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:heapster
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system

2- Deploy Heapster.

$ kubectl --kubeconfig ~/terraform-vsphere-kubespray/config/admin.conf create -f

Destroying the Kubernetes

cluster
1- Go to the terraform-vsphere-kubespray directory.

$ cd ~/terraform-vsphere-kubespray

2- Destroy the deployment

$ terraform destroy


 Conclusion
Install and manage automatically a Kubernetes cluster on VMware vSphere with
 Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

You now have a way to easily deploy, scale, upgrade and destroy a Kubernetes cluster
on VMware vSphere. This will allow you to do even more testing without worrying
about breaking the cluster as you can spin up a new one automatically.

If you nd an issue with the Terraform script, please let me know by opening an issue
on GitHub: https://github.com/sguyennet/terraform-vsphere-kubespray
(https://github.com/sguyennet/terraform-vsphere-kubespray)

Tags: Cloud (/tag/cloud/), Automation (/tag/automation/), Kubernetes (/tag/kubernetes/), vSphere

(/tag/vsphere/), VMware (/tag/vmware/), Terraform (/tag/terraform/), Container (/tag/container/), Ansible
(/tag/ansible/), Kubespray (/tag/kubespray/)

Share this post:

 (https://twitter.com/share?
text=Install%20and%20manage%20automatically%20a%
and-manage-automatically-a-kubernetes-cluster-on-
vmware-vsphere-with-terraform-and-kubespray/) 
(https://www.facebook.com/sharer/sharer.php?
u=https://blog.inkubate.io/install-and-manage-
automatically-a-kubernetes-cluster-on-vmware-
vsphere-with-terraform-and-kubespray/) 
(https://plus.google.com/share?
url=https://blog.inkubate.io/install-and-manage-
automatically-a-kubernetes-cluster-on-vmware-
vsphere-with-terraform-and-kubespray/)

 Install and manage automatically a Kubernetes cluster on VMware vSphere with

 PREVIOUS (/INSTALL-AND-CONFIGURE-METALLB-AS-A-LOAD-BALANCER-FOR-
Terraform and Kubespray 10%
/blog.inkubate.io) 3289 words - 12 min read.
KUBERNETES/)
(http:/
Install and con gure MetalLB as a load balancer for Kubernetes (/install-and-con gure-
metallb-as-a-load-balancer-for-kubernetes/)
NEXT  (/HOW-TO-USE-HARBOR-PRIVATE-REGISTRY-WITH-KUBERNETES/)
How to install and use VMware Harbor private registry with Kubernetes (/how-to-use-harbor-
private-registry-with-kubernetes/)

Author

SIMON GUYENNET (/AUTHOR/SIMON/)

Comments


 Install and manage automatically a Kubernetes cluster on VMware vSphere with

ALSO ON BLOG.INKUBATE.IO
Terraform and Kubespray 10%
(http://blog.inkubate.io) 3289 words - 12 min read.

Monitor Kubernetes How to install and Use vSphere Storage Deploy Ku

with Prometheus use VMware … as Kubernetes … on VMwar

3 years ago • 4 comments 3 years ago • 4 comments 3 years ago • 32 comments 3 years ago • 15
Prometheus is a monitoring Harbor is a container image Hatchway is a VMware open During the sec
tool originally created by registry developed by source project. The goal of VMWorld 201
SoundCloud. The project … VMware. It was recently … this project is to allow to … Vegas, VMwar

119 Comments blog.inkubate.io 🔒 Disqus' Privacy Policy 

1 Login

 Recommend 2 t Tweet f Share Sort by Best

Join the discussion…

LOG IN WITH
OR SIGN UP WITH DISQUS ?

 Creative Commons