This document describes how to use kubeadm to manually install Kubernetes with Contiv-VPP networking on one or more bare metal or VM hosts.
- VmWare VMs: the vmxnet3 driver is required on each interface that will be used by VPP. Please see here for instructions how to install the vmxnet3 driver on VmWare Fusion.
DPDK setup must be completed on each node as follows:
-
Load the PCI UIO driver:
$ sudo modprobe uio_pci_generic -
Verify that the PCI UIO driver has loaded successfully:
$ lsmod | grep uio uio_pci_generic 16384 0 uio 20480 1 uio_pci_genericPlease note that this driver needs to be loaded upon each server bootup, so you may want to add
uio_pci_genericinto the/etc/modulesfile, or a file in the/etc/modules-load.d/directory. For example, the/etc/modulesfile could look as follows:# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. uio_pci_generic
You need to find out the PCI address of the network interface that
you want VPP to use for multi-node pod interconnect. On Debian-based
distributions, you can use lshw(*):
$ sudo lshw -class network -businfo
Bus info Device Class Description
====================================================
pci@0000:00:03.0 ens3 network Virtio network device
pci@0000:00:04.0 ens4 network Virtio network device
* On CentOS/RedHat/Fedora distributions, lshw may not be available by default, install it by
yum -y install lshw
Finally, you need to set up the vswitch to use the network adapters:
You can perform the above steps using the node setup script.
After the nodes you will be using in your K8s cluster are prepared, you can install the cluster using kubeadm.
For first-time installation, see Installing kubeadm. To update an
existing installation, you should do a apt-get update && apt-get upgrade
or yum update to get the latest version of kubeadm.
On each host with multiple NICs where the NIC that will be used for Kubernetes management traffic is not the one pointed to by the default route out of the host, a custom management network for Kubernetes must be configured.
In K8s 1.10, support for huge pages in a pod has been introduced. For now, this feature must be either disabled or memory limit must be defined for vswitch container.
To disable huge pages, perform the following steps as root:
- Using your favorite editor, disable huge pages in the kubelet configuration
file (
/etc/systemd/system/kubelet.service.d/10-kubeadm.confor/etc/default/kubeletfor version 1.11+):
Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
- Restart the kubelet daemon:
systemctl daemon-reload
systemctl restart kubelet
To define memory limit, append the following snippet to vswitch container in deployment yaml file:
limits:
hugepages-2Mi: 1024Mi
memory: 1024Mi
or set contiv.vswitch.defineMemoryLimits to true in helm values.
Before initializing the master, you may want to tear down up any previously installed K8s components. Then, proceed with master initialization as described in the kubeadm manual. Execute the following command as root:
kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
Note: kubeadm init will autodetect the network interface to advertise
the master on as the interface with the default gateway. If you want to use a
different interface (i.e. a custom management network setup), specify the
--apiserver-advertise-address=<ip-address> argument to kubeadm init. For
example:
kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
Note: The CIDR specified with the flag --pod-network-cidr is used by
kube-proxy, and it must include the PodSubnetCIDR from the IPAMConfig
section in the Contiv-vpp config map in Contiv-vpp's deployment file
(contiv-vpp.yaml). Pods in the host network namespace
are a special case; they share their respective interfaces and IP addresses with
the host. For proxying to work properly it is therefore required for services
with backends running on the host to also include the node management IP
within the --pod-network-cidr subnet. For example, with the default
PodSubnetCIDR=10.1.0.0/16 and PodIfIPCIDR=10.2.1.0/24, the subnet
10.3.0.0/16 could be allocated for the management network and
--pod-network-cidr could be defined as 10.0.0.0/8, so as to include IP
addresses of all pods in all network namespaces:
kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
If Kubernetes was initialized successfully, it prints out this message:
Your Kubernetes master has initialized successfully!
After successful initialization, don't forget to set up your .kube directory
as a regular user (as instructed by kubeadm):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configIf you have already used the Contiv-VPP plugin before, you may need to pull the most recent Docker images on each node:
bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
Install the Contiv-VPP network for your cluster as follows:
-
If you do not use the STN feature, install Contiv-vpp as follows:
kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml -
If you use the STN feature, download the
contiv-vpp.yamlfile:wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yamlThen edit the STN configuration as described here. Finally, create the Contiv-vpp deployment from the edited file:
kubectl apply -f ./contiv-vpp.yaml
Beware contiv-etcd data is persisted in /var/etcd by default. It has to be cleaned up manually after kubeadm reset.
Otherwise outdated data will be loaded by a subsequent deployment.
You can also generate random subfolder, alternatively:
curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
After some time, all contiv containers should enter the running state:
root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
NAME READY STATUS RESTARTS AGE IP NODE
...
contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
In particular, make sure that the Contiv-VPP pod IP addresses are the same as
the IP address specified in the --apiserver-advertise-address=<ip-address>
argument to kubeadm init.
Verify that the VPP successfully grabbed the network interface specified in
the VPP startup config (GigabitEthernet0/4/0 in our case):
$ sudo vppctl
vpp# sh inter
Name Idx State Counter Count
GigabitEthernet0/4/0 1 up rx packets 1294
rx bytes 153850
tx packets 512
tx bytes 21896
drops 962
ip4 1032
host-40df9b44c3d42f4 3 up rx packets 126601
rx bytes 44628849
tx packets 132155
tx bytes 27205450
drops 24
ip4 126585
ip6 16
host-vppv2 2 up rx packets 132162
rx bytes 27205824
tx packets 126658
tx bytes 44634963
drops 15
ip4 132147
ip6 14
local0 0 down
You should also see the interface to kube-dns (host-40df9b44c3d42f4) and to the
node's IP stack (host-vppv2).
By default, your cluster will not schedule pods on the master for security reasons. If you want to be able to schedule pods on the master, e.g. for a single-machine Kubernetes cluster for development, run:
kubectl taint nodes --all node-role.kubernetes.io/master-
More details about installing the pod network can be found in the kubeadm manual.
To add a new node to your cluster, run as root the command that was output by kubeadm init. For example:
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
More details can be found int the kubeadm manual.
After some time, all contiv containers should enter the running state:
root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
NAME READY STATUS RESTARTS AGE IP NODE
contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
contiv-vswitch-h6759 2/2 Running 0 14h 192.168.56.105 cvpp-slave2
contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
etcd-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
kube-apiserver-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
kube-controller-manager-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
kube-dns-545bc4bfd4-fr6j9 3/3 Running 0 14h 10.1.134.2 cvpp
kube-proxy-q8sv2 1/1 Running 0 14h 192.168.56.106 cvpp
kube-proxy-s8kv9 1/1 Running 0 14h 192.168.56.105 cvpp-slave2
kube-scheduler-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
In particular, verify that a vswitch pod and a kube-proxy pod is running on each joined node, as shown above.
On each joined node, verify that the VPP successfully grabbed the network
interface specified in the VPP startup config (GigabitEthernet0/4/0 in
our case):
$ sudo vppctl
vpp# sh inter
Name Idx State Counter Count
GigabitEthernet0/4/0 1 up
...
From the vpp CLI on a joined node you can also ping kube-dns to verify node-to-node connectivity. For example:
vpp# ping 10.1.134.2
64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms
Statistics: 5 sent, 5 received, 0% packet loss
You can go ahead and create a simple deployment:
$ kubectl run nginx --image=nginx --replicas=2
Use kubectl describe pod to get the IP address of a pod, e.g.:
$ kubectl describe pod nginx | grep IP
You should see two ip addresses, for example:
IP: 10.1.1.3
IP: 10.1.1.4
You can check the pods' connectivity in one of the following ways:
- Connect to the VPP debug CLI and ping any pod:
sudo vppctl
vpp# ping 10.1.1.3
- Start busybox and ping any pod:
kubectl run busybox --rm -ti --image=busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ #
/ # ping 10.1.1.3
- You should be able to ping any pod from the host:
ping 10.1.1.3
to enable pod deployment on the master, untaint the master first:
kubectl taint nodes --all node-role.kubernetes.io/master-
In order to verify inter-node pod connectivity, we need to tell Kubernetes to deploy one pod on the master node and one POD on the worker. For this, we can use node selectors.
In your deployment YAMLs, add the nodeSelector sections that refer to
preferred node hostnames, e.g.:
nodeSelector:
kubernetes.io/hostname: vm5
Example of whole JSONs:
apiVersion: v1
kind: Pod
metadata:
name: nginx1
spec:
nodeSelector:
kubernetes.io/hostname: vm5
containers:
- name: nginx
image: nginx
apiVersion: v1
kind: Pod
metadata:
name: nginx2
spec:
nodeSelector:
kubernetes.io/hostname: vm6
containers:
- name: nginx
image: nginx
After deploying the JSONs, verify they were deployed on different hosts:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx1 1/1 Running 0 13m 10.1.36.2 vm5
nginx2 1/1 Running 0 13m 10.1.219.3 vm6
Now you can verify the connectivity to both nginx PODs from a busybox POD:
kubectl run busybox --rm -it --image=busybox /bin/sh
/ # wget 10.1.36.2
Connecting to 10.1.36.2 (10.1.36.2:80)
index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
/ # rm index.html
/ # wget 10.1.219.3
Connecting to 10.1.219.3 (10.1.219.3:80)
index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
To uninstall the network plugin itself, use kubectl:
kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
In order to remove the persisted config, cleanup the bolt and etcd storage:
rm -rf /var/etcd/contiv-data
rm -rf /var/bolt/bolt.db
- First, drain the node and make sure that the node is empty before shutting it down:
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
- Next, on the node being removed, reset all kubeadm installed state:
rm -rf $HOME/.kube
sudo su
kubeadm reset
- If you added environment variable definitions into
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf(e.g. in Step 1.3), clean them up now.
Some of the issues that can occur during the installation are:
- Forgetting to create and initialize the
.kubedirectory in your home directory (As instructed bykubeadm init --token-ttl 0). This can manifest itself as the following error:W1017 09:25:43.403159 2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") - Previous installation lingering on the file system.
'kubeadm init --token-ttl 0fails to initialize kubelet with one or more of the following error messages:... [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused. ...
If you run into any of the above issues, try to clean up and reinstall as root:
sudo su
rm -rf $HOME/.kube
kubeadm reset
kubeadm init --token-ttl 0
rm -rf /var/etcd/contiv-data
rm -rf /var/bolt/bolt.db