Installing highly available kubernetes cluster

Hi everyone my name is Elvin and today i am going show you how to install highly available kubernetes cluster. now before we start installation proccess let me explain what is kubernetes and why do we need it?

Firstly we should understand old deployment way. before cloud and virtualization exist we can deploy developer application we bought new physical server. Then we install operation system and for example if it is java application we should install jre and after yes we are in production. but old tradtional way is problamatic.

1. if we need more ram and cpu we should poweroff server this action can brings down time
2. if server have any physical problem and trying solve this problem also can bring down time

3. snapshot,backup and we have a lots of problem with it.

in IT field next big revolution is virtualization. virtualization sofware be possible that we can run multiple operation system in the same physical machine. and we call physical machine is host and other multiple OS is called VM (virtual machine). virtualization feature save us from all three problems.

Next barrier was about deployment. yes our os managment was very cool and effective. but we have problem about deployment. there are problems beetwen ops team and dev team. For example if developer write application UBUNTU 16.04 and ops team want to run it 18.04 version of ubuntu. like this we have tons of problems and that slow down development proccess.

Then another bir revolution happened

i dont explain in detail but i can say.

1. all application was made containerized. it saved developer and sysadmin deal with dependency and some OS problems.
2. next jenkins github gitlab like this tools come and help us continus integration.
3. and dealing with microservices architectura kubernetes help us.

yes today is my main purpose is install highly avalible cluster. and for that reason we need minumum 3 master node, minumum 2 haproxy, one virtual ip, minumum one worker node. my servers is showing below.

let’s start with HA nodes. we will install haproxy for load balancing all three master nodes. Our kubernetes api endpoint will be haproxy ip addresses. but as you see we will have 2 haproxy that is because we will make haproxy node highly available. we shoud use only one ip for endpoint and we achive it with keepalived. and keepalived service provide us with Virtual IP. let’s install

apt update && apt install -y keepalived haproxy

now i will configure my keepalived.

cat >> /etc/keepalived/keepalived.conf <<EOF
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
#interval  how often  script run
interval 3
#timeout mean how long wait script return answer for example we can have sciprt and it can take 10 second
timeout 10
#fall mean how many times script return unsucsessful answer and then change keepalived VIP to another
fall 5
#rise how many times return successful
rise 2
fail 2
weight -2
}
vrrp_instance VI_1 {
state BACKUP
interface ens160
virtual_router_id 1
priority 100
advert_int 5
authentication {
auth_type PASS
auth_pass mysecret
}
virtual_ipaddress {
10.100.3.210
}
track_script {
check_apiserver
}
}
EOF

Now we have check_apiserver.sh. This basic script send request with curl to localhost and VIP if fail then keepalived give ip to other node.

cat >> /etc/keepalived/check_apiserver.sh <<EOF
#!/bin/sh
errorExit() {
echo "*** $@" 1>&2
exit 1
}
curl --silent --max-time 2 --insecure https://localhost:6443/ -o /dev/null || errorExit "Error GET https://localhost:6443/"
if ip addr | grep -q 10.100.3.210; then
curl --silent --max-time 2 --insecure https://10.100.3.210:6443/ -o /dev/null || errorExit "Error GET https://10.100.3.210:6443/"
fi
EOF
chmod +x /etc/keepalived/check_apiserver.sh

Disable swap

swapoff -a; sed -i ‘/swap/d’ /etc/fstab

Disable Firewall

systemctl disable — now ufw ## or you can add ports

Enable and Load Kernel modules

{
cat >> /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
}

Add Kernel settings

{
cat >>/etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system
}

Install containerd runtime

{
apt update
apt install -y containerd apt-transport-https
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
systemctl restart containerd
systemctl enable containerd
}

Add apt repo for kubernetes

{
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
}

Install Kubernetes components

{
apt update
apt install -y kubeadm kubelet kubectl
}

INIT CLUSTER

#Only run it master 01
kubeadm init --control-plane-endpoint="10.100.3.210:6443" --upload-certs --apiserver-advertise-address=10.100.3.201
##kubeadm init --control-plane-endpoint=”[VIP]” — upload-certs — apiserver-advertise-address=[server ip]

Next is applying calico network plugin

curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O
kubectl apply -f calico.yaml