Create Kubernetes cluster on CentOS Stream
Introduction
This is a step-by-step installation of a bare-metal Kubernetes cluster on a set of CentOS Stream servers or virtual machines. It’s definitely not the good way of doing it but it’s a first step to start touching self hosted Kubernetes when already familiar with CentOS.
Common
Install CentOS Stream on master and worker nodes. You may leave SELinux in enforcing mode. Many tutorials advise to disable SELinux but it appears it’s not longer needed.
During the installation process you MUST disable the swap. The installer will complain about it but you can safely ignore the warning.
Setup passwordless SSH connection to root on the nodes. You need to copy your own ssh public key as authorized in the root account.
You should have the certificate of your own Certificate Authority as ca.crt available.
Setup the firewall rules (as root@node):
firewall-cmd --permanent --add-port=6443/tcp # Kubernetes API (Master)
firewall-cmd --permanent --add-port=2379-2380/tcp # etcd (Master) Only for multi-master
firewall-cmd --permanent --add-port=8090/tcp # Platform Agent (Master/Worker)
firewall-cmd --permanent --add-port=8091/tcp # Platform API (Operator)
firewall-cmd --permanent --add-port=10250/tcp # Kubernets API (Master/Worker)
firewall-cmd --permanent --add-port=10251/tcp # Only for multi-master
firewall-cmd --permanent --add-port=10252/tcp # Only for multi-master
firewall-cmd --permanent --add-port=10255/tcp # K8S API Read-only
firewall-cmd --permanent --add-port=5000/tcp # Private Registry
firewall-cmd --permanent --add-port=30000-32767/tcp # NodePort
firewall-cmd --permanent --add-port=8285/udp # Flannel (Master/Worker) Needed ?
firewall-cmd --permanent --add-port=8472/udp # Flannel (Master/Worker)
firewall-cmd --permanent --add-port=5353/udp # DNS Ingress Controler ?
firewall-cmd --permanent --add-port=5353/tcp # DNS Ingress Controler ?
firewall-cmd --add-masquerade --permanent # (Master/Worker)
firewall-cmd --reloadMake iptable see bridged traffic (as root@node):
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --systemCopy your CA certificate as trusted on the node from you host:
scp ca.crt root@node:/etc/pki/ca-trust/source/anchors/Set it as trusted (as root@node);
update-ca-trustInstall docker from its repo (as root@node):
dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
dnf install -y --allowerasing docker-ceBy default, the docker configuration uses cgroup and it need to be switched to systemd. For that, you need to edit /usr/lib/systemd/system/docker.service and add the native.cgroupdriver as shown below.
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemdEnable docker daemon and start it (as root@node):
systemctl enable --now dockerNow install Kubernetes from its repo (as root@node):
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOFInstall kubeadm, kubelet and kubectl. Enable kubelet and start it (as root@node):
dnf install -y kubeadm kubelet kubectl
systemctl enable --now kubeletMaster node
This section describes the steps specific to the master node only.
You need to initialize the master node with kubeadm (as root@master). Keep track of the reported join command as you will need it for the worker nodes. The parameter --pod-network-cidr=10.244.0.0/16 is required for flannel network. You also need to update your user environment as reported by the command. This user is referenced as user in this document. You also need to update the root account with this environment.
kubeadm init --pod-network-cidr=10.244.0.0/16In Kubernetes we need to chose a network implementation. Install the flannel network (as user@master):
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlThis step is optional and should not be done for a production environment. Enable the use of the master node as a worker node (as user@master):
kubectl taint nodes --all node-role.kubernetes.io/master-Worker node
This section describes the steps specific to the master node only.
For each worker node, initialize the worker node and join the cluster (as root@worker):
kubeadm join 192.168.0.11:6443 --token e66.change.me.1yn --discovery-token-ca-cert-hash sha256:325.change.me.79e If you no longer have the token you can create a new one on the master node (as root@worker):
kubeadm token create --print-join-command
kubeadm token list Test the cluster
you can test that everything is working with these simple tests (as user@master). That will do the following:
- Get the nodes
- Get the pods
- Deploy and run the busybox image and make it sleep for a long long time
- Execute a simple command on the running busybox pod
- Deploy dns utils
- Execute dns debugging commands on the running pod
kubectl get nodes
kubectl get pod
kubectl run busybox --image=busybox:1.28.4 --command -- sleep 99999
kubectl exec busybox -- cat /etc/resolv.conf
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
kubectl exec -i -t dnsutils -- nslookup kubernetes
kubectl exec -i -t dnsutils -- cat /etc/resolv.confDeployement of the Dashboard
Deploy the dashboard (as user@master):
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yamlConfigure access to the dashoard (as user@master):
mkdir ~/dashboard
cat > ~/dashboard/dashboard-admin.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOFInstall the access:
kubectl apply -f ~/dashboard/dashboard-admin.yaml
kubectl get secret -n kubernetes-dashboard $(kubectl get serviceaccount admin-user -n kubernetes-dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decodeLaunch the proxy on the master node (as user@master):
kubectl proxyThe dashboard is only accessible from the inside of the cluster. So we need to forward the 8001 port using SSH from your point of connexion to the master node:
ssh -L 8001:localhost:8001 root@masterThe dashboad is then available from the host. You need to copy paste the token provided by the get secret command to the dashboard.
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/Copy/paste the secret key reported by the kubectl get secret command in the login popup.
Create a private registry
Create a PersistentVolume and PersistentVolumeClaim
Create a PersistentVolume on the master and the corresponding PersistentVolumeClaim. First, you must create the folder on the master node (as root@master):
mkdir /data/registryCreate the PersistentVolume and PersistentVolumeClaim for the registry:
cat > registry-storage.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
name: registry-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
storageClassName: local
local:
path: /data/registry
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- master
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
spec:
storageClassName: local
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
EOFDeploy the registry
Create a certificate for the registry (for master node) and sign it with your CA:
openssl genrsa -out "registry.key" 2048
openssl req -new -subj "/C=XX/L=City/O=You/CN=master/emailAddress=foo@bar.com" -key "registry.key" -out "registry.csr"
openssl x509 -req -days 3650 -in "registry.csr" -CA "ca.crt" -CAkey "ca.key" -CAcreateserial -out "registry.crt" -sha256 Create a secret to be used for securing the deployment of the registry:
kubectl create secret tls registry-tls --key="registry.key" --cert="registry.crt"
kubectl describe secret registry-tlsCreate the yaml of the deployment of the registry:
cat > registry.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: private-repository
labels:
app: private-repository
spec:
replicas: 1
selector:
matchLabels:
app: private-repository
template:
metadata:
labels:
app: private-repository
spec:
volumes:
- name: certs-vol
secret:
secretName: registry-tls
- name: registry-vol
persistentVolumeClaim:
claimName: registry-pvc
containers:
- image: registry:2
name: private-repository
imagePullPolicy: IfNotPresent
env:
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: "/certs/tls.crt"
- name: REGISTRY_HTTP_TLS_KEY
value: "/certs/tls.key"
ports:
- containerPort: 5000
volumeMounts:
- name: certs-vol
mountPath: /certs
- name: registry-vol
mountPath: /var/lib/registry
EOFAnd finally, deploy it:
kubectl apply -f registry.yamlCreate the service
Create the YAML of the registry service:
cat > registry-service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: registry-service
spec:
type: NodePort
selector:
app: private-repository
ports:
- port: 5000
targetPort: 5000
nodePort: 30500
EOFInstall the registry service:
kubectl apply -f registry-service.yamlThe registry is now accessible from master node on port 30500.
Create a secret
Create a secret for connecting to the local registry:
kubectl create secret docker-registry ocirsecret --docker-server=master:30500 --docker-username='a' --docker-password='b' --docker-email='a@b.com'Push an image from podman to our private registry
Pushing a podman image to our private registry to be used in the cluster is done with the podman push command. You first need to retrieve the image ID using podman image ls and then push it:
podman image ls
podman push <<IMAGE ID>> docker://master:30500/blah:1.0The previous command may fail if your certificate was not properly generated or configured. You may override this check with --tls-verify=false
Conclusion
This was clearly not the correct way to deploy a bare metal Kubernetes cluster. CentOS Stream should not be used for this purpose. Fedora Core OS is most probably a better choice but installation of Fedora Core OS is very different from CentOS Stream if you are already familiar with regular Linux distributions.
The manual steps for creating the cluster are good for learning and understanding but it is not the way to go for product. One may try Ansible and the load of pre-existing playbooks.
