RPi 4 K8s multi-master cluster using Ubuntu and Kubespray

First let's install Ubuntu 18.04 LTS for (Arm64) in all your Raspberry Pi 4. In my case I installed it in ten Raspberry Pi 4, for a multi-master installation the recommended minimum size is four raspberry pis (as shown in the picture above): two masters, two workers (three etcds, two in master nodes one in a worker node).

It will be saved in /home/your-username/.ssh/id_rsa

2. Copy your ssh key to all raspberry pi 4 nodes, repeat this command for all raspberry pi ips:

ssh-copy-id 192.168.0.10

You'll get:

(answer your password)

3. Edit /etc/sudoers in all pi nodes and add a line after the %sudo one to avoid password questions when using ssh to pi nodes from the controller node, replace "your-controller-username" by your real controller user name:

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL
your-controller-username ALL=(ALL:ALL) NOPASSWD:ALL

4. Install python3, pip and git in your controller node:

sudo apt install python3-pip git
sudo pip3 install --upgrade pip
pip --version

5. Add a function in your .bashrc script to be able to run commands to all raspberry pis at the same time:

cd
nano .bashrc

Add this function at the end of the .bashrc file:

function picmd {                                                                                                         
   echo "pi1"                                                                                                              
   ssh 192.168.0.10 "$@"                                                                                                   
   echo "pi2"                                                                                                              
   ssh 192.168.0.11 "$@"                                                                                                   
   echo "pi3"                                                                                                              
   ssh 192.168.0.12 "$@"                                                                                                   
   echo "pi4"                                                                                                              
   ssh 192.168.0.13 "$@"                                                                                                                                                                                                    }     

Apply .bashrc changes

source .bashrc

Try it from your controller node writing: picmd date, you should see all pi node responses, something like this:

ubuntu@CONTROLLER-NODE:~$ picmd date
pi1
Sat Jan 25 14:50:31 UTC 2020
pi2
Sat Jan 25 14:50:33 UTC 2020
pi3
Sat Jan 25 14:50:35 UTC 2020
pi4

You can now use picmd command to execute commands (like apt update) in all your raspberry pi nodes without logging in all of them one by one.

 

Installing Kubespray and the Kubernetes cluster

1. In your controller node home directory clone kubespray repo:

git clone https://github.com/kubernetes-sigs/kubespray.git

2. Enter into kubespray directory and take a look which kuberspray version are you going to install:

cd kubespray/ && git describe --tags 

It's important to know which kubespray version you are installing due kubespray don't support upgrades from an older release straight to the latest release, you have to upgrade it version after version without jumping any.

4. Install kubespray requirements:

sudo pip install -r requirements.txt

5. Create a new inventory from the sample folder:

cp -rfp inventory/sample inventory/mycluster  

6. Declare a IPS variable with all your raspberry cluster ips, for instance:

declare -a IPS=(192.168.0.10 192.168.0.11 192.168.0.12 92.168.0.13)

7. Generate kubespray configuration based in your node ips:

CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

8. Review the generated hosts.yml file:

nano inventory/mycluster/hosts.yml

In this file, in the 'hosts:' section, be aware that node1, node2, etc... will be the hostname of your raspberry pis based on the ips you provided. Kubespray will change them, if you prefer to name your raspberry pi hostnames differently, change them in this file. Initially, hosts.yml file should look like:

all:
  hosts:
    node1:
      ansible_host: 192.168.0.10
      ip: 192.168.0.10
      access_ip: 192.168.0.10
    node2:
      ansible_host: 192.168.0.11
      ip: 192.168.0.11
      access_ip: 192.168.0.11
    node3:
      ansible_host: 192.168.0.12
      ip: 192.168.0.12
      access_ip: 192.168.0.12
    node4:
      ansible_host: 192.168.0.13
      ip: 192.168.0.13
      access_ip: 192.168.0.13
  children:
    kube-master:
      hosts:
        node1:
        node2:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

9. Optionally you can review other generated configuration files:

nano inventory/mycluster/group_vars/all/all.yml
nano inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

10. Install your multi master Kubernetes cluster using kubespray:

ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Kubespray installation process can last several minutes or more than half an hour depending on your hardware and the configuration you set, relax and enjoy a cup of tea.

 

Classic error, number of masters and etcds databases

It's because in your hosts.yml configuration the number of masters and etcd database nodes are wrong adjust them and launch kubespray installer again.

 

Skip Kubespray packages upgrades

In all your raspberry pi nodes, block to upgrade packages installed by Kubespray. When you upgrade your Kubernetes Kubespray cluster to a newer version, Kubespray will be in charge of update them too. Run this command in all your raspberry pi nodes:

sudo apt-mark hold apt-transport-https aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli ipset ipvsadm libipset3 libltdl7 libpython-stdlib libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib pigz python python-apt python-minimal python2 python2-minimal python2.7 python2.7-minimal socat