14 KiB
14 KiB
Installing k8s with kubeadm
PLEASE NOTE The default behavior of a kubelet is to fail to start if swap memory is detected on a node.
Disable it by turning it off on the fly
sudo swapoff -a
or to make the change persistent across reboots remove it from the fstab
Container runtime - containerd
Go to - https://github.com/containerd/containerd/blob/main/docs/getting-started.md and follow the instructions This must be done on all the nodes, master and worker.
- Download the containerd runtime
wget https://github.com/containerd/containerd/releases/download/v2.2.0/containerd-2.2.0-linux-amd64.tar.gz
- Extract it to /usr/local
$ sudo tar Cxzvf /usr/local containerd-1.6.2-linux-amd64.tar.gz
bin/
bin/containerd-shim-runc-v2
bin/containerd-shim
bin/ctr
bin/containerd-shim-runc-v1
bin/containerd
bin/containerd-stress
- systemd download
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
Place it in the correct location For debian trixie it was
/usr/lib/systemd/system/
Then reload and enable
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
- runc Download the runc binary from https://github.com/opencontainers/runc/releases
wget https://github.com/opencontainers/runc/releases/download/v1.4.0/runc.amd64
now install it
sudo install -m 755 runc.amd64 /usr/local/sbin/runc
- Install CNI plugin
Download the the associated cni-plugin from https://github.com/containernetworking/plugins/releases
wget https://github.com/containernetworking/plugins/releases/download/v1.8.0/cni-plugins-linux-amd64-v1.8.0.tgz
and extact under /opt/cni/bin
mkdir -p /opt/cni/bin
sudo tar Czxvf /opt/cni/bin cni-plugins-linux-amd64-v1.8.0.tgz
- Install kubelet kubeadm kubectl
Update the apt package index and install packages needed to use the Kubernetes apt repository:
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:
If the directory /etc/apt/keyrings does not exist, it should be created before the curl command, read the note below.
sudo mkdir -p -m 755 /etc/apt/keyrin
else continue
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
Added the apt repo for k8s 1.34
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
Install the packages
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
Enable kubelet before running adm
sudo systemctl enable --now kubelet
- Configure cgroup driver
Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.
Feel free to go read more at https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd and Advanced topics at https://github.com/containerd/containerd/blob/main/docs/getting-started.md
but in short...
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
- Network configuration Enable IPv4 packet forwarding To manually enable IPv4 packet forwarding:
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --system
Validate that it is set to 1
sysctl net.ipv4.ip_forward
- Create the Cluster
sudo kubeadm init --apiserver-advertise-address=The_Master_Node_IP --pod-network-cidr=POD_NODE_CIDR_BLOCK
sudo kubeadm init --apiserver-advertise-address=192.168.50.20 --pod-network-cidr=10.244.0.0/16
THe output should be similar to the following
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.50.20:6443 --token 3m8ogx.4w779pfudv9sr9h0 \
--discovery-token-ca-cert-hash sha256:2b70f7b1939e9a3c17a94ddc94a8714a5782dc476bf50ba915a6c848710dd0ba
You want to keep that join for later.
Now create your kube conf
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Look at pods and nodes
Node01-master:~$ kubectl get nodes
kubectl get pods -A
NAME STATUS ROLES AGE VERSION
node01-master NotReady control-plane 161m v1.34.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bc5c9577-wrfwh 0/1 Pending 0 161m
kube-system coredns-66bc5c9577-ztwtc 0/1 Pending 0 161m
kube-system etcd-node01-master 1/1 Running 0 161m
kube-system kube-apiserver-node01-master 1/1 Running 0 161m
kube-system kube-controller-manager-node01-master 1/1 Running 0 161m
kube-system kube-proxy-tqcnj 1/1 Running 0 161m
kube-system kube-scheduler-node01-master 1/1 Running 0 161m
Node01-master:~$
Not everything is running. we have no CNI
We could create new pods but it will not start cause no networking
Node01-master:~$ kubectl run nginx --image=nginx:latest
pod/nginx created
Node01-master:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 0/1 Pending 0 5s
Node01-master:~$ kubectl describe pod nginx -n default
Name: nginx
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: run=nginx
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
nginx:
Image: nginx:latest
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7gvx (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-s7gvx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 22s default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Node01-master:~$
This is not a taint issue. I can add the worker node to the cluster using kubeadm join, and it will still be the same.
Node01-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node01-master NotReady control-plane 167m v1.34.2
node02-worker NotReady <none> 87s v1.34.2
Node01-master:~$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
nginx 0/1 Pending 0 3m36s
Node01-master:~$
- Install network add-on (Cillium)
Using https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/ as ref We only do this on the master node
CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
Node01-master:~$ echo $CLI_VERSION
v0.18.9
Node01-master:~$
Download the binary
curl -L --remote-name \
https://github.com/cilium/cilium-cli/releases/download/${CLI_VERSION}/cilium-linux-amd64.tar.gz
Extract to /usr/local/bin and validate
Node01-master:~$ sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
cilium
Node01-master:~$ cilium version
cilium-cli: v0.18.9 compiled with go1.25.5 on linux/amd64
cilium image (default): v1.18.3
cilium image (stable): v1.18.4
cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found
Node01-master:~$
Now install cilium
Node01-master:~$ cilium install
ℹ️ Using Cilium version 1.18.3
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has been installed
Node01-master:~$
As you are installing this into k8s it is using the kubeconf so no sudo needed.
Validate its installed
Node01-master:~$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet cilium-envoy Desired: 2, Ready: 2/2, Available: 2/2
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 2
cilium-envoy Running: 2
cilium-operator Running: 1
clustermesh-apiserver
hubble-relay
Cluster Pods: 3/3 managed by Cilium
Helm chart version: 1.18.3
Image versions cilium quay.io/cilium/cilium:v1.18.3@sha256:5649db451c88d928ea585514746d50d91e6210801b300c897283ea319d68de15: 2
cilium-envoy quay.io/cilium/cilium-envoy:v1.34.10-1761014632-c360e8557eb41011dfb5210f8fb53fed6c0b3222@sha256:ca76eb4e9812d114c7f43215a742c00b8bf41200992af0d21b5561d46156fd15: 2
cilium-operator quay.io/cilium/operator-generic:v1.18.3@sha256:b5a0138e1a38e4437c5215257ff4e35373619501f4877dbaf92c89ecfad81797: 1
Node01-master:~$
If you want, there is also a connectivity test that can be runm, but it can take some time to complete (more than a 120 tests).
If you now look at your cluster, it should be up and running
Node01-master:~$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cilium-test-1 client-64d966fcbd-ctrb8 1/1 Running 0 2m46s 10.0.1.63 node02-worker <none> <none>
cilium-test-1 client2-5f6d9498c7-9th7m 1/1 Running 0 2m46s 10.0.1.146 node02-worker <none> <none>
cilium-test-1 echo-same-node-7889ff4c85-5hcqb 2/2 Running 0 2m46s 10.0.1.155 node02-worker <none> <none>
cilium-test-ccnp1 client-ccnp-86d7f7bd68-s89zf 1/1 Running 0 2m46s 10.0.1.228 node02-worker <none> <none>
cilium-test-ccnp2 client-ccnp-86d7f7bd68-tmc76 1/1 Running 0 2m45s 10.0.1.202 node02-worker <none> <none>
default nginx 1/1 Running 0 18m 10.0.1.91 node02-worker <none> <none>
kube-system cilium-5s7hd 1/1 Running 0 4m55s 192.168.50.20 node01-master <none> <none>
kube-system cilium-bj2rz 1/1 Running 0 4m55s 192.168.50.21 node02-worker <none> <none>
kube-system cilium-envoy-fgqm6 1/1 Running 0 4m55s 192.168.50.20 node01-master <none> <none>
kube-system cilium-envoy-mtgzs 1/1 Running 0 4m55s 192.168.50.21 node02-worker <none> <none>
kube-system cilium-operator-68bd8cc456-fb2kg 1/1 Running 0 4m55s 192.168.50.21 node02-worker <none> <none>
kube-system coredns-66bc5c9577-wrfwh 1/1 Running 0 3h2m 10.0.1.176 node02-worker <none> <none>
kube-system coredns-66bc5c9577-ztwtc 1/1 Running 0 3h2m 10.0.1.206 node02-worker <none> <none>
kube-system etcd-node01-master 1/1 Running 0 3h2m 192.168.50.20 node01-master <none> <none>
kube-system kube-apiserver-node01-master 1/1 Running 0 3h2m 192.168.50.20 node01-master <none> <none>
kube-system kube-controller-manager-node01-master 1/1 Running 0 3h2m 192.168.50.20 node01-master <none> <none>
kube-system kube-proxy-tqcnj 1/1 Running 0 3h2m 192.168.50.20 node01-master <none> <none>
kube-system kube-proxy-wm8r7 1/1 Running 0 16m 192.168.50.21 node02-worker <none> <none>
kube-system kube-scheduler-node01-master 1/1 Running 0 3h2m 192.168.50.20 node01-master <none> <none>
Node01-master:~$
Node01-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node01-master Ready control-plane 3h3m v1.34.2
node02-worker Ready <none> 17m v1.34.2
Node01-master:~$