Installation
Docker
Containerd runs as a daemon managing container lifecycle. Kubernetes interacts with containerd via the CRI:
sudo pacman -Syu
sudo pacman -S containerd
Configure containerd to use systemd cgroups for compatibility with Kubernetes:
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl enable --now containerd
Kubernetes
yay -S kubeadm-bin kubelet-bin kubectl-bin
sudo systemctl enable kubelet
Configuration
Before initialization, disable swap to meet Kubernetes requirements:
sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab
Set ip_forward to 1
sudo sysctl -w net.ipv4.ip_forward=1
sysctl net.ipv4.ip_forward
sudo tee /etc/sysctl.d/99-kubernetes.conf <<'EOF'
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
sysctl net.ipv4.ip_forward
If you are working on an air-gapped system first download the needed container images:
kubeadm config images list
registry.k8s.io/kube-apiserver:v1.36.5
registry.k8s.io/kube-controller-manager:v1.36.5
registry.k8s.io/kube-scheduler:v1.36.5
registry.k8s.io/kube-proxy:v1.36.5
registry.k8s.io/coredns/coredns:v1.14.2
registry.k8s.io/pause:3.10.2
registry.k8s.io/etcd:3.6.8-0
Download on a different server:
docker pull registry.k8s.io/kube-apiserver:v1.36.5
docker pull registry.k8s.io/kube-controller-manager:v1.36.5
docker pull registry.k8s.io/kube-scheduler:v1.36.5
docker pull registry.k8s.io/kube-proxy:v1.36.5
docker pull registry.k8s.io/coredns/coredns:v1.14.2
docker pull registry.k8s.io/pause:3.10.2
docker pull registry.k8s.io/etcd:3.6.8-0
docker save -o kube-apiserver.tar registry.k8s.io/kube-apiserver:v1.36.5
docker save -o kube-controller-manager.tar registry.k8s.io/kube-controller-manager:v1.36.5
docker save -o kube-scheduler.tar registry.k8s.io/kube-scheduler:v1.36.5
docker save -o kube-proxy.tar registry.k8s.io/kube-proxy:v1.36.5
docker save -o coredns.tar registry.k8s.io/coredns/coredns:v1.14.2
docker save -o pause.tar registry.k8s.io/pause:3.10.2
docker save -o etcd.tar registry.k8s.io/etcd:3.6.8-0
And load them on the kube server:
sudo ctr -n k8s.io images import kube-apiserver.tar
sudo ctr -n k8s.io images import kube-controller-manager.tar
sudo ctr -n k8s.io images import kube-scheduler.tar
sudo ctr -n k8s.io images import kube-proxy.tar
sudo ctr -n k8s.io images import coredns.tar
sudo ctr -n k8s.io images import pause.tar
sudo ctr -n k8s.io images import etcd.tar
The initialize the cluster with kubeadm:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
Kublet refuses to start
[kubelet-start] Starting the kubelet error: error execution phase wait-control-plane: cannot obtain client without bootstrap: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: context deadline exceeded To see the stack trace of this error execute with --v=5 or higher
sudo journalctl -u kubelet -n 100 --no-pager
Sep 24 16:51:25 linux-system kubelet[3344449]: /dev/zram0 partition 64917500 0 100
Sep 24 16:51:25 linux-system kubelet[3344449]: >
Sep 24 16:51:25 linux-system kubelet[3344449]: E0924 16:51:25.920406 3344449 run.go:72] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap or set --fail-swap-on flag to false"
Sep 24 16:51:25 linux-system systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 16:51:25 linux-system systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 24 16:51:36 linux-system systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 22.
Sep 24 16:51:36 linux-system systemd[1]: Started kubelet: The Kubernetes Node Agent.
Sep 24 16:51:36 linux-system (kubelet)[3344809]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_EXTRA_ARGS
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.130772 3344809 server.go:545] "Kubelet version" kubeletVersion="v1.36.2"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.131071 3344809 server.go:547] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.131151 3344809 watchdog_linux.go:94] "Systemd watchdog is not enabled"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.131175 3344809 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.132310 3344809 server.go:985] "Client rotation is on, will bootstrap in background"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.135873 3344809 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.142125 3344809 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.153910 3344809 server.go:1448] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.169450 3344809 server.go:808] "--cgroups-per-qos enabled, but --cgroup-root was not specified. Defaulting to /"
Sep 24 16:51:36 linux-system kubelet[3344809]: I0924 16:51:36.170090 3344809 swap_util.go:119] "Swap is on" /proc/swaps contents=<
Sep 24 16:51:36 linux-system kubelet[3344809]: Filename Type Size Used Priority
Sep 24 16:51:36 linux-system kubelet[3344809]: /dev/zram0 partition 64917500 0 100
Sep 24 16:51:36 linux-system kubelet[3344809]: >
Sep 24 16:51:36 linux-system kubelet[3344809]: E0924 16:51:36.170134 3344809 run.go:72] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap or set --fail-swap-on flag to false"
Sep 24 16:51:36 linux-system systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 16:51:36 linux-system systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 24 16:51:46 linux-system systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 23.
sudo crictl ps -a
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
sudo crictl logs $(sudo crictl ps -a --name kube-apiserver -q | head -1)
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
FATA[0000] ID cannot be empty
sudo crictl logs $(sudo crictl ps -a --name etcd -q | head -1)
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
FATA[0000] ID cannot be empty
free -h
sudo systemctl stop dev-zram0.swap
swapon --show
sudo systemctl restart kubelet
sudo kubeadm init phase kubeconfig admin
Verify everything is up now:
sudo crictl ps
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
81996b392b3fe 137b99d7bbba0 15 minutes ago Running kube-scheduler 0 7a7b243979be2 kube-scheduler-omarchy kube-system
ab320e249b1fb ee85eb1f0edd2 15 minutes ago Running etcd 0 d022d30443f0f etcd-omarchy kube-system
6098b703f178e cf2bbfca0a634 15 minutes ago Running kube-controller-manager 0 83b926a6a0469 kube-controller-manager-omarchy kube-system
09e5a0ef9b57f d350c55cb17bb 15 minutes ago Running kube-apiserver 0 225f4780d15c7 kube-apiserver-omarchy kube-system
sudo ss -lntp | grep 6443
LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",pid=3388266,fd=4))
Back to Configuration
Set up kubectl for the current user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
And another Error
export KUBECONFIG=$HOME/.kube/config
kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "kubernetes-admin" cannot list resource "nodes" in API group "" at the cluster scope
kubectl get clusterrolebinding kubernetes-admin -o yaml
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io "kubernetes-admin" is forbidden: User "kubernetes-admin" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
kubectl auth can-i create clusterrolebindings
Warning: resource 'clusterrolebindings' is not namespace scoped in group 'rbac.authorization.k8s.io'
sudo kubeadm init phase kubeconfig admin
W0924 17:59:31.770334 3473452 version.go:108] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp: lookup dl.k8s.io on 127.0.0.53:53: server misbehaving W0924 17:59:31.770589 3473452 version.go:109] falling back to the local client version: v1.36.2 [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
sudo kubeadm init phase bootstrap-token
I0924 18:00:30.279517 3475225 version.go:260] remote version is much newer: v1.37.1; falling back to: stable-1.36 [bootstrap-token] Using token: 3tfmec.awkj2m07xti1w9zh [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
sudo kubeadm init phase kubeconfig admin --v=5
I0924 18:01:25.568038 3477036 initconfiguration.go:116] skip CRI socket detection, fill with the default CRI socket unix:///var/run/containerd/containerd.sock I0924 18:01:25.568683 3477036 interface.go:443] "Looking for default routes with IP addresses" IPVersion=4 I0924 18:01:25.568739 3477036 interface.go:448] "Default route transits interface" interface="wlp99s0" I0924 18:01:25.568893 3477036 interface.go:209] "Interface is up" interface="wlp99s0" I0924 18:01:25.569025 3477036 interface.go:257] "Found addresses for interface" interface="wlp99s0" numAddresses=3 addresses=[{"IP":"192.168.2.104","Mask":"////AA=="},{"IP":"2409:8754:30c0:800:2a0c:7be4:86c1:90af","Mask":"//////////8AAAAAAAAAAA=="},{"IP":"fe80::11a8:a303:438f:657f","Mask":"//////////8AAAAAAAAAAA=="}] I0924 18:01:25.569042 3477036 interface.go:224] "Checking for matching global IP" address="192.168.2.104/24" I0924 18:01:25.569059 3477036 interface.go:231] "IP found" IP="192.168.2.104" I0924 18:01:25.569070 3477036 interface.go:263] "Found valid address" IPVersion=4 IP="192.168.2.104" interface="wlp99s0" I0924 18:01:25.569082 3477036 interface.go:454] "Found active IP" IP="192.168.2.104" I0924 18:01:25.569120 3477036 kubelet.go:195] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd" I0924 18:01:25.569164 3477036 version.go:191] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt I0924 18:01:25.840932 3477036 version.go:260] remote version is much newer: v1.37.1; falling back to: stable-1.36 I0924 18:01:25.841011 3477036 version.go:191] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.36.txt I0924 18:01:26.056834 3477036 certs.go:472] validating certificate period for CA certificate I0924 18:01:26.056944 3477036 certs.go:472] validating certificate period for front-proxy CA certificate I0924 18:01:26.057004 3477036 kubeconfig.go:111] creating kubeconfig file for admin.conf I0924 18:01:26.057313 3477036 certs.go:472] validating certificate period for ca certificate [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
zram-generator keeps restarting and killing kubelet:
sudo mkdir -p /etc/systemd/zram-generator.conf.d
sudo ln -s /dev/null /etc/systemd/zram-generator.conf.d/90-omarchy.conf
ls -l /etc/systemd/zram-generator.conf.d/
sudo systemctl daemon-reload
sudo systemctl stop dev-zram0.swap
swapon --show
sudo reboot
Container Network Interface Plugin
sudo pacman -S cni-plugins
sudo modprobe br_netfilter
lsmod | grep br_netfilter
ls -l /proc/sys/net/bridge/
sudo sysctl -w net.bridge.bridge-nf-call-iptables=1
sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1
sudo tee /etc/modules-load.d/kubernetes.conf <<'EOF'
br_netfilter
EOF
sudo tee /etc/sysctl.d/99-kubernetes.conf <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
kubectl rollout restart daemonset/kube-flannel-ds -n kube-flannel
kubectl get pods -n kube-flannel -w
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
Another Error
kubectl logs -n kube-flannel -l app=flannel --tail=100 Defaulted container
"kube-flannel" out of: kube-flannel, install-cni-plugin (init): dial tcp 10.96.0.1:443: connect: connection refused
Proxy is missing and Flannel pod keeps crashing ->
kubectl get pods -n kube-flannel
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-t96zb 0/1 Error 7 (5m16s ago) 11m
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-t96zb 0/1 CrashLoopBackOff 6 (101s ago) 7m48s
kube-system etcd-omarchy 1/1 Running 0 103m
kube-system kube-apiserver-omarchy 1/1 Running 0 103m
kube-system kube-controller-manager-omarchy 1/1 Running 0 103m
kube-system kube-scheduler-omarchy 1/1 Running 0 103m
kubectl get pods -n kube-system -l k8s-app=kube-proxy
No resources found in kube-system namespace.
Restart the proxy manually:
sudo kubeadm init phase addon kube-proxy
I0924 19:00:29.535214 3587402 version.go:260] remote version is much newer: v1.37.1; falling back to: stable-1.36
[addons] Applied essential addon: kube-proxy
kubectl rollout restart daemonset/kube-flannel-ds -n kube-flannel
daemonset.apps/kube-flannel-ds restarted
Now it is working:
kubectl get pods -n kube-flannel
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-drwhr 1/1 Running 0 5s
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
omarchy Ready <none> 108m v1.36.2 192.168.2.104 <none> Omarchy 7.2.5-3-omarchy (amd64) containerd://2.3.5
Changing the LAN IP
The LAN IP is baked into the k8s cluster - meaning when your laptop connect to another Wifi you need to re-init the entire cluster ~ not an issue for a dev environment. E.g. my address changed from 192.168.2.* to 192.168.0.* and everything broke:
sudo kubeadm reset -f
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.151
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
Now re-checking if the cluster is up - shows that CoreDNS does not become healthy:
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-r5ng6 1/1 Running 0 14h
kube-system coredns-589f44dc88-k2fqt 0/1 Running 0 3m48s
kube-system coredns-7c456bc5d4-6vblh 0/1 Running 0 3m48s
kube-system coredns-7c456bc5d4-kpktj 0/1 Running 0 3m48s
kube-system etcd-omarchy 1/1 Running 2 14h
kube-system kube-apiserver-omarchy 1/1 Running 2 14h
kube-system kube-controller-manager-omarchy 1/1 Running 1 14h
kube-system kube-proxy-tgprg 1/1 Running 0 14h
kube-system kube-scheduler-omarchy 1/1 Running 6 14h
```bash
kubectl logs -n kube-system deploy/coredns --tail=25
Found 2 pods, using pod/coredns-7c456bc5d4-fmcd7
[INFO] plugin/ready: Plugins not ready: "kubernetes"
This is a Firewall problem - either allow only the ports CoreDNS needs:
# pod -> node IP, e.g. apiserver (192.168.0.151:6443) or kubelet 10250
sudo ufw allow in from 10.244.0.0/24 proto tcp to any port 6443
sudo ufw allow in from 10.244.0.0/24 proto tcp to any port 10250
Or, alternatively, allow everything:
# pod -> anything on the node (kube-system metrics, coredns :9153 prometheus, coredns :53, etc.)
sudo ufw allow from 10.244.0.0/16
sudo iptables -S INPUT | grep -E "10\.244\.0\.|ufw"
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
kubectl get pod -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-7c456bc5d4-fmcd7 1/1 Running 0 73m
coredns-7c456bc5d4-hxjhl 1/1 Running 0 73m
Hello World
kubectl create deployment hello --image=nginx:alpine
kubectl get pods -o wide -w
kubectl expose deployment hello --port=80 --type=ClusterIP
kubectl get svc hello -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello ClusterIP 10.100.91.198 <none> 80/TCP 26m app=hello
kubectl get endpoints hello -o wide
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
hello 10.244.0.3:80 27m
curl http://10.244.0.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.
Further configuration is required for the web server, reverse proxy,
API gateway, load balancer, content cache, or other features.</p>
<p>For online documentation and support please refer to
<a href="https://nginx.org/">nginx.org</a>.<br/>
To engage with the community please visit
<a href="https://community.nginx.org/">community.nginx.org</a>.<br/>
For enterprise grade support, professional services, additional
security features and capabilities please refer to
<a href="https://f5.com/nginx">f5.com/nginx</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
kubectl delete pod hello --force --grace-period=0 --ignore-not-found