Kubernetes the Hard Way — subject

Brief

You are handed three or four bare Linux machines and nothing else — no kubeadm, no managed service, no install scripts. Your assignment: turn them into a working Kubernetes cluster by writing every certificate, kubeconfig, and systemd unit yourself, so that afterwards you can name every component in the architecture and say what breaks without it. The original tutorial is the reference prose; this subject is the assignment and the checks. When stuck on a mechanism, read the tutorial's chapter for that stage — but the artifacts must be yours.

Instructions

Environment

Choose one:

• Local VMs (recommended for redoing cheaply): 3–4 VMs via your hypervisor of choice (QEMU/KVM, VirtualBox, Multipass…), Debian-family ARM64 or x86_64, ≥2 GB RAM each.
• Cloud instances: any provider's smallest viable instances on a shared network.

Roles: one jumpbox (your workbench — all commands run from here), one server (control plane), two workers (node-0, node-1). Requirements before anything else:

• Root SSH from the jumpbox to every machine.
• A machines.txt database (IP, FQDN, hostname, pod-subnet per worker — e.g. 10.200.0.0/24 for node-0, 10.200.1.0/24 for node-1) that your loops script against.
• Hostnames set and /etc/hosts distributed so every machine reaches every other by name.
• Kubernetes release binaries (kube-apiserver, kube-controller-manager, kube-scheduler, kubectl, kubelet, kube-proxy), etcd, containerd + runc + CNI plugins, and crictl downloaded once to the jumpbox.

PKI — provision by hand

Create a CA (openssl or cfssl — your choice), then issue one certificate per identity, each with the correct subject and SANs:

• admin — your kubectl identity (O=system:masters).
• One kubelet cert per worker — CN=system:node:<host>, O=system:nodes, SANs covering the node's hostname and IP (Node authorizer requires this exact form).
• kube-apiserver — SANs must include every name a client will dial: the server's IP/hostname, 127.0.0.1, the first IP of the service range (e.g. 10.32.0.1), and the kubernetes.default.svc.cluster.local name chain.
• kube-controller-manager (CN=system:kube-controller-manager), kube-scheduler (CN=system:kube-scheduler), kube-proxy (O=system:node-proxier).
• service-accounts — the key pair the apiserver uses to sign service-account tokens.

Distribute: each worker gets the CA cert plus its own kubelet key pair; the server gets the CA, apiserver, and service-accounts pairs.

Configs

• Generate a kubeconfig per component (kubelet ×2, kube-proxy, controller-manager, scheduler, admin), each embedding the CA, the component's cert, and the right endpoint (workers dial the server by name; control-plane components dial 127.0.0.1). Place them where each component expects them.
• Write the encryption config (an EncryptionConfiguration with an AES-CBC key) for at-rest encryption of Secrets; it ships to the server.

Control plane

1. etcd — install on the server, systemd unit, listening on localhost is fine for the single-server layout.
2. kube-apiserver — systemd unit wired to etcd, the cert pairs, the service-account key, the encryption config, and the service cluster IP range (e.g. 10.32.0.0/24).
3. kube-controller-manager and kube-scheduler — systemd units using their kubeconfigs; the controller-manager also gets the cluster CIDR (e.g. 10.200.0.0/16), the CA for signing, and the service-account key.
4. RBAC for kubelet access — a ClusterRole granting the apiserver access to kubelet APIs (nodes/pods proxy, logs, exec…) bound to the apiserver's kubelet-client identity, applied via kubectl.

Workers

On each worker: install containerd (+ runc), kubelet, kube-proxy as systemd units; write the CNI bridge + loopback configs carrying that worker's pod subnet; load required kernel modules and sysctls; enable swap off (or configure kubelet to tolerate it — know which you chose and why).

Pod networking & DNS

• The bridge CNI plugin handles intra-node traffic. Inter-node, add static routes on the server and each worker: every machine must route each pod subnet to the worker that owns it.
• Deploy cluster DNS (CoreDNS) as a Deployment + Service so pods resolve kubernetes.default.

Teardown and redo

Destroy everything — VMs included — and rebuild the entire cluster from your own notes without opening the tutorial. Keep the notes you used; where they failed you, fix them.

Constraints

• No kubeadm, no Helm for the control plane, no provisioning scripts you didn't write. Anything you automate, you must have first done manually once.
• Keep a written run-book as you go — the redo milestone depends on it.
• Version pinning: pick one Kubernetes minor release and use its matching etcd, containerd, and CNI plugin versions throughout.

Acceptance

Mapped to the exercise's milestones:

1. Fleet — from the jumpbox, ssh root@<hostname> succeeds by name for server and both workers; hostname on each returns the expected value.
2. PKI — openssl verify -CAfile ca.crt <cert> passes for every issued cert; apiserver cert's SANs include the service IP and kubernetes.default.svc (openssl x509 -text shows them); each worker holds only its own key pair.
3. Configs — kubectl config view --kubeconfig <file> shows the right server/user for each component; the encryption config is on the server and referenced by the apiserver unit.
4. etcd — etcdctl member list shows one healthy member; the systemd unit survives a reboot.
5. Control plane — kubectl cluster-info and a kubectl get --raw /healthz return healthy from the jumpbox using the admin kubeconfig; the RBAC binding for kubelet access exists.
6. Workers — kubectl get nodes lists both workers Ready with the pinned version.
7. Network & smoke tests — all pass:
   • a kubectl run/Deployment schedules and runs on a worker;
   • kubectl port-forward, kubectl logs, kubectl exec all work against it;
   • a NodePort Service answers from outside the cluster;
   • pods on node-0 can reach pod IPs on node-1 (the static routes work);
   • a busybox pod resolves kubernetes.default via cluster DNS;
   • create a Secret, then hexdump its key from etcd directly: the value must be k8s:enc:aescbc-prefixed ciphertext, not plaintext.
8. Redo — the second build reaches all checks above with the tutorial closed; time it, and note every place your run-book was wrong.

Related

• Kubernetes the Hard Way — the exercise note this is the subject of.
• Kubernetes the Hard Way (source) — the guided tutorial whose prose this assignment deliberately leaves out.