Deploying a Kubernetes cluster with containerd and an insecure private Docker registry

Table of Contents

I was recently offered the opportunity to publish an extension of a paper ( A data infrastructure for heterogeneous telemetry adaptation. Application to Netflow-based cryptojacking detection) that I presented in ICIN-2023. My idea, as part of the extension, was to create new tests and generate new metrics. However, the framework I was presenting was deployed in a Kubernetes cluster with a private Docker registry on an infrastructure to which I do not have access anymore. The deadline was tight, and even if I have worked with Kubernetes before, I have never deployed a cluster from zero. I want to help others who may be in a similar situation in which time is pressing. So I created this step-by-step quick guide to deploy a Kubernetes cluster with an insecure private Docker registry. I am aware that this may not be the best way to do it, but it sure works! So if you just want to play with Kubernetes and Docker registry or need to test any application, this is probably a good enough option.

Machine specifications #

In this tutorial we use two VirtualBox machines with Ubuntu 20.04.6 LTS (Focal Fossa) and the following specifications:

Processors: 3 vCPU
RAM: 8 GB
Disk: 33 GB

You can download the server install image (.iso file) from Ubuntu 20.04.6 LTS.

One virtual machine is going to be the K8s control-plane, and the other machine will be a worker node.

For clarity of where we are executing commands, we follow the next color convention:

Control-Plane:
- Normal user: cp
- Root user: root
Worker
- Normal user: worker
- Root user: root

I will use this color for outputs.

Deploying a Kubernetes cluster (Control-Plane) #

The first thing to do is update and upgrade the system and install the necessary packages. With ROOT user, execute:

apt-get update && apt-get upgrade -y
apt-get install -y vim
apt install curl apt-transport-https vim git wget gnupg2 software-properties-common lsb-release ca-certificates uidmap -y
apt-get update && apt-get upgrade -y

Now we need to disable swap and load some modules.

swapoff -a
modprobe overlay
modprobe br_netfilter

Update kernel networking.

cat << EOF | tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

Install the necessary keys.

mkdir -p /etc/apt/keyrings;
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg;
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

As we are going to deploy Kubernetes with containerd, we need to install the containerd software.

apt-get update && apt-get install containerd.io -y;
containerd config default | tee /etc/containerd/config.toml;
sed -e 's/SystemdCgroup = false/SystemdCgroup = true/g' -i /etc/containerd/config.toml;
systemctl restart containerd

Add a new repo for Kubernetes, add gpg keys for the packages, and update the new repo declared.

echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update

Now we install the Kubernetes software and prevent packages from being upgraded. I am using version 1.25.1 of kubeadm, kubelet, and kubectl.

apt-get install -y kubeadm=1.25.1-00 kubelet=1.25.1-00 kubectl=1.25.1-00;
apt-mark hold kubelet kubeadm kubectl

We need to install a Container Network Interface (CNI). We will use Calico as the network plugin, so we download the plugin template.

wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

We add a local DNS alias for our control-plane, supposing that we want the alias to be “k8scp” and the principal interface to be “enp0s3”.

echo "$(ip addr show | grep "enp0s3" | sed -En 's/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p') k8scp" >> /etc/hosts

Create a kubeadm-config template and initialize the control plane.

echo "apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.25.1
controlPlaneEndpoint: "k8scp:6443"
networking:
    podSubnet: 192.168.0.0/16" > kubeadm-config.yaml

kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out

The control-plane is now installed. We are going to allow non-root access to the cluster and install the Calico plugin with the template we downloaded before, execute WITHOUT ROOT USER.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
sudo cp /root/calico.yaml .
kubectl apply -f calico.yaml

Update notation for the containerd runtime-endpoint.

sudo crictl config --set \
runtime-endpoint=unix:///run/containerd/containerd.sock \
--set image-endpoint=unix:///run/containerd/containerd.sock

Delete taints from the control-plain so the server can run non-infrastructure pods.

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
kubectl describe node | grep -i taint

Finally, we can add bash autocompletion and test the connection with the cluster.

sudo apt-get install bash-completion -y
echo "source <(kubectl completion bash)" >> $HOME/.bashrc
kubectl get nodes

Congratulations you have installed a Kubernetes control-plane node. Now, let’s add a worker!

Adding a worker node #

Most of the steps followed for the control-plane are the same for the worker. With ROOT user execute:

apt-get update && apt-get upgrade -y
apt install curl apt-transport-https vim git wget gnupg2 \
software-properties-common lsb-release ca-certificates uidmap -y
swapoff -a
modprobe overlay
modprobe br_netfilter
cat << EOF | tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
| sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update && apt-get install containerd.io -y
containerd config default | tee /etc/containerd/config.toml
sed -e 's/SystemdCgroup = false/SystemdCgroup = true/g' -i /etc/containerd/config.toml
systemctl restart containerd
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update
apt-get install -y kubeadm=1.25.1-00 kubelet=1.25.1-00 kubectl=1.25.1-00
apt-mark hold kubelet kubeadm kubectl

Now to connect the worker to the control-plane, we need to create a token and a Discovery Token CA Cert Hash. So we go to the control-plane node and execute:

sudo kubeadm token create
openssl x509 -pubkey \
-in /etc/kubernetes/pki/ca.crt | openssl rsa \
-pubin -outform der 2>/dev/null | openssl dgst \
-sha256 -hex | sed 's/ˆ.* //'

The output of previous commands is:

token: 8c64fq.86598iiufdjhmm5t
sha256: bb3fbc40faa7e9465435eb2c8e028a2f8e7affa37d4b1498497b239f561ebad6

Coming back to the worker node we add a DNS alias with the IP address of the control-plane. Edit /etc/host and add the following line:

<IP of control-plane> k8scp

Finally, we use the token and hash to join the cluster.

kubeadm join \
--token 8c64fq.86598iiufdjhmm5t \
k8scp:6443 \
--discovery-token-ca-cert-hash \
sha256:bb3fbc40faa7e9465435eb2c8e028a2f8e7affa37d4b1498497b239f561ebad6

Configuring a private insecure registry with Docker #

Installing the registry in the worker node #

We are going to install a private insecure Docker registry. First, as ROOT, we install the necessary packages.

apt update
apt -y python3-pip containerd docker.io docker-registry apache2-utils
apt-get -y install jq
pip install yq

Add “k8sregistry” as dns alias.

echo "$(ip addr show | grep "enp0s3" | sed -En 's/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p') k8sregistry" >> /etc/hosts

To be able to access the Docker registry from the worker machine, we need to add the IP to the insecure-registries field in the file /etc/docker/daemon.json.

echo "{
  \"insecure-registries\":
    [\"$(ip addr show | grep "enp0s3" | sed -En 's/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'):5000\", \"k8sregistry:5000\"],
  \"dns\":
    [\"8.8.8.8\", \"8.8.4.4\"]
}" > /etc/docker/daemon.json

We restart the Docker daemon and the docker-registry.

systemctl restart docker
systemctl restart docker-registry

To enable basic authentication in the Docker registry, we execute the following command as non-ROOT:

sudo -E python3 -m yq -yi '.auth.htpasswd.path |= "/etc/docker/registry/.htpasswd"' /etc/docker/registry/config.yml

Now, as ROOT, we create a user for the Docker registry and login with the user.

htpasswd -Bc /etc/docker/registry/.htpasswd regadmin
docker login http://k8sregistry:5000/v2/

Now we have to see the authentication token generated; we are going to use it in the control-plane.

1
2
3
4
5
6
7
8


cat /root/.docker/config.json
{
        "auths": {
                "k8sregistry:5000": {
                        "auth": "cmVnYWRtaW46cmVnYWRtaW4="
                }
        }
}

Configuring the control-plane and worker to access the registry from Kubernetes #

We can access the docker registry using docker in the worker node, but Kubernetes is not going to be able to pull images unless we configure containerd in the control-plane and worker nodes. Add “k8sregistry” as an alias by modifying "<IP of the worker machine>".

echo "<IP of the worker machine> k8sregistry" >> /etc/hosts

Both in the control-plane and the worker nodes we need to:

Edit the containerd configuration in /etc/containerd/config.toml (i.e., modify the line marked adding the path indicated):

1
2


[plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

Create the following directories and files:

mkdir /etc/containerd/certs.d/
mkdir /etc/containerd/certs.d/k8sregistry:5000
touch /etc/containerd/certs.d/k8sregistry:5000/hosts.toml

Configure hosts.toml by executing:

echo "server = \"http://k8sregistry:5000\"

[host.\"http://k8sregistry:5000\"]
  capabilities = [\"pull\", \"resolve\"]
  skip_verify = true
  [host.\"http://k8sregistry:5000\".header]
    authorization = \"Basic cmVnYWRtaW46cmVnYWRtaW4=\"" > /etc/containerd/certs.d/k8sregistry:5000/hosts.toml

Finally, we restart the containerd service:

systemctl restart containerd

We are done! #

You can test it by pushing an image to the private insecure registry. In the worker node:

sudo docker pull nginx
sudo docker tag nginx:latest k8sregistry:5000/mynginx:1.0
sudo docker push k8sregistry:5000/mynginx:1.0

And creating a deployment using that image. In the control-plane:

kubectl create deployment testdeployment --image=k8sregistry:5000/mynginx:1.0 \
--dry-run=client -o yaml > test-deployment.yaml
kubectl create -f test-deployment.yaml

Debugger Detection Techniques: The Summary of a Summary

4 mins· 0 · 0

malware-analysis reverse-engineering

The Art of Password Cracking: Rainbow Tables

9 mins· 0 · 0

cracking documentation pentesting