00 /What You Will Learn
- Explain the difference between a container, a Pod, a ReplicaSet, and a Deployment.
- Spin up a multi-node Kubernetes cluster on your machine using kind with dedicated port mappings.
- Expose backend pods to your localhost using a NodePort Service on port 18300.
- Trigger pod self-healing by deleting a running pod and watching Kubernetes create a replacement.
- Compare plain Docker container crash vs Kubernetes declarative desired-state self-healing.
Before you start
Finish Lab 00 (Docker Basics) and Lab 01 (Docker Compose Basics) first. You should know what a container and a port mapping are. Give Docker at least 4 GB of memory, and make sure kind and kubectl are installed.
01 /Why It Matters
Picture a production Tatkal ticketing system on a busy morning. At 09:59 AM, the backend is handling a few hundred requests a minute. At 10:00:02 AM, tens of thousands of passengers hit the booking endpoint at the exact same second. If one backend container crashes from an unhandled exception or memory spike, standalone Docker leaves it dead. The socket closes immediately.
Incoming traffic shifts to the surviving containers, their response times spike, and ticket requests begin timing out. If you rely on an on-call engineer to notice the alert, log into the server, and run docker start manually, several minutes of ticket bookings are lost before anyone can intervene.
When an application crashes under heavy load, relying on human reflexes is broken by design. Kubernetes replaces manual restarts with a declarative control loop. You state your intent once: keep 3 replicas of tatkal-api running at all times. The controller manager checks the cluster continuously. If a pod terminates, it spots the deficit within seconds and starts a replacement pod on a healthy node automatically.
Plain Docker runs containers on one machine. If a container dies, Docker leaves it dead unless you added a restart policy on that host. Kubernetes runs an entire cluster. If a container crashes or a worker machine goes down, Kubernetes detects the gap and schedules fresh pods on surviving machines automatically.
Think of Kubernetes as a railway terminal. The Control Plane is the Station Master office: it does not carry passengers itself, but it holds the timetable and directs traffic. The Worker Nodes are the platforms where the trains run. A Pod is like a tiffin carrier: it is the atomic package that Kubernetes schedules and moves around. Inside that tiffin carrier is your application container holding your code. If a carrier is dropped or damaged, the Station Master immediately orders a duplicate tiffin carrier prepared on a healthy platform.
02 /What We Are Building
A 3-node kind cluster inside Docker (1 control plane node, 2 worker nodes). The Tatkal API runs as a 3-replica Deployment exposed via NodePort on 127.0.0.1:18300.
The three-layer hierarchy: Deployment, ReplicaSet, and Pod
1. The Deployment (The Blueprint Contract)
You write a Deployment when you want to define what application should run and what version it is on. The Deployment does not manage individual containers directly. Instead, it manages ReplicaSets.
2. The ReplicaSet (The Headcount Guard)
The ReplicaSet has one single job in life: count. If the blueprint says 3 replicas, it counts how many matching pods currently exist. If it counts 2, it asks the cluster to create 1. If it counts 4, it deletes 1.
3. The Pod (The Running Worker)
The Pod is the actual running process on a worker machine. When you delete a pod with kubectl delete pod, you are deleting the worker. The ReplicaSet instantly spots that the count dropped from 3 to 2, and creates a replacement pod within seconds.
Under the Hood: From kubectl apply to a Running Pod
1. API Server and etcd Storage
When you run kubectl apply, your laptop talks to the kube-apiserver over HTTPS. The API server authenticates the request, validates the YAML schema, and persists the Deployment object into etcd (the distributed key-value store). No container exists yet.
2. Controller Manager Reconciles
The Deployment controller running inside kube-controller-manager notices the new Deployment and creates a ReplicaSet. The ReplicaSet controller notices that 3 pods are desired but 0 exist, so it writes 3 unassigned Pod objects into etcd with status Pending.
3. Scheduler Selects Nodes
The kube-scheduler continuously watches for pods without an assigned node. It evaluates CPU and memory capacity across the worker nodes, scores them, and writes the chosen node name into the Pod object in etcd.
4. Kubelet and Containerd Start the Workload
The kubelet running on the chosen worker node detects that a pod has been assigned to it. It instructs containerd through the Container Runtime Interface (CRI) to pull the image, set up Linux network namespaces and cgroups, and start the container process.
The Postal Route: How 127.0.0.1:18300 reaches your container
Step 1: Laptop to kind Node
Your laptop sends a request to http://127.0.0.1:18300. Through the extraPortMappings setting in kind-config.yaml, Docker forwards this port into port 30080 on the kind worker container.
Step 2: NodePort to kube-proxy
Port 30080 is a NodePort opened across every node in the cluster. Inside that node, kube-proxy has programmed Linux kernel firewall rules (iptables) that watch for traffic arriving on port 30080.
Step 3: kube-proxy to Pod Container
kube-proxy picks one of the three healthy Tatkal API pods at random, rewrites the destination IP to that pod internal IP address, and delivers the packet to container port 8080.
Self-healing replaces failing pods automatically, but when several replicas crash and restart at the exact same moment, their boot sequences synchronize. Each new pod opens several database connections during application startup to populate its connection pool. If dozens of replacement pods boot together, this sudden rush of connection attempts can overwhelm the primary database listener. The database may reject new connections with fatal errors, causing the replacement pods to fail their startup health checks and crash again. In production systems, pairing self-healing with connection poolers, readiness delay, and startup exponential backoff prevents replacement pods from creating a restart storm on the database.
The cluster definition in kind-config.yaml:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: da-k60
nodes:
- role: control-plane
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
extraPortMappings:
- containerPort: 30080
hostPort: 18300
listenAddress: "127.0.0.1"
protocol: TCP
- role: worker
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
- role: worker
image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5The Deployment manifest in k8s/deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tatkal-api
labels:
app: tatkal-api
spec:
replicas: 3
selector:
matchLabels:
app: tatkal-api
template:
metadata:
labels:
app: tatkal-api
spec:
containers:
- name: api
image: tatkal-api:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 2
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 10
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 10Key parts of the configuration
extraPortMappings
Maps host port 18300 on 127.0.0.1 directly to containerPort 30080 on the worker node. This lets curl from your laptop reach the NodePort Service.
replicas: 3
Declares the desired number of pods. If one terminates, the ReplicaSet controller detects the difference and creates a replacement.
labels and matchLabels
Kubernetes connects resources using label tags. The Deployment attaches app: tatkal-api to each pod. The ReplicaSet uses matchLabels like a search filter to count its pods.
readinessProbe
Checks /ready via HTTP. Traffic is routed only after the container passes this probe, preventing 502 Bad Gateway errors during startup.
tolerationSeconds: 10
Shortens the node eviction window from the default 300 seconds down to 10 seconds for rapid failover testing. This is a preview of the node eviction mechanism we test in depth in Lab 65.
03 /Words You Need
| Word | Plain meaning |
|---|---|
| Pod | The smallest deployable unit in Kubernetes. A wrapper holding one or more containers sharing network and storage. |
| Deployment | A declarative controller that manages ReplicaSets to reach and maintain your desired pod count and versions. |
| ReplicaSet | The headcount controller whose sole job is to ensure a specified number of identical pod replicas are running. |
| NodePort | A Service port opened on every cluster node (ports 30000-32767), forwarding external traffic to backend pods. |
| ClusterIP | A stable internal virtual IP assigned to a Service, accessible only to components inside the cluster. |
| kubelet | The node worker agent that receives pod manifests from the control plane and starts containers via the container runtime (such as containerd). |
| Control Plane | The master brain of the cluster: API server, scheduler, controller-manager, and etcd database. |
| kind | Kubernetes IN Docker. Runs local multi-node clusters where each Docker container acts as a full Kubernetes node. |
| Self-Healing | The automated control loop detecting a missing or failed pod and launching an exact replacement automatically. |
| Desired State | The declarative specification in YAML telling Kubernetes what the cluster should look like at all times. |
| kube-proxy | The network agent on each node configuring Linux kernel iptables rules to route Service traffic to pods. |
| Atomic | Indivisible. An all-or-nothing unit that cannot be split into smaller pieces. |
| Ephemeral | Temporary. Container filesystems and pod IP addresses are short-lived and discarded when a pod terminates. |
| Namespace | An isolated workspace inside Kubernetes or Linux that separates resources and network interfaces. |
| Primary | The authoritative leader database or server that handles writes, contrasted with read replicas. |
| Failover | Automatically redirecting traffic to surviving healthy pods when one instance crashes. |
| Eviction | Terminating pods on a failing or constrained node so they can be recreated on healthy nodes. |
04 /Predict Before You Run
Question 1: You kill a plain Docker container without a restart policy. What happens to incoming requests?
The container process stops immediately. The OS closes listening sockets. Every request sent to that port fails with connection refused until a human restarts it.
Question 2: You delete a pod in a 3-replica Kubernetes Deployment with kubectl delete pod. Does traffic drop completely?
No. The NodePort Service continues routing requests to the other 2 surviving pods. Simultaneously, the Deployment controller detects 2 pods instead of 3 and immediately schedules a replacement.
Question 3: When you send 12 requests through NodePort 18300 to a 3-replica deployment, does each pod receive exactly 4 requests?
No. kube-proxy uses randomized packet routing (via iptables or IPVS mode), not strict round-robin. You will see traffic reach multiple different pods, but the distribution varies per run.
05 /Step by Step
Step 0: Get the lab
Download the zip with the button at the top of this page. Unzip it into a new folder and open a terminal inside that folder. Every command below runs from there.
mkdir k8s-first-cluster && cd k8s-first-cluster
unzip ~/Downloads/k8s-first-cluster.zipmkdir k8s-first-cluster; cd k8s-first-cluster
Expand-Archive -Path "$HOME/Downloads/k8s-first-cluster.zip" -DestinationPath .Run the automated test runner ./test-first-cluster.sh or .\test-first-cluster.ps1 to execute all 7 steps, verify assertions, and clean up. To inspect by hand:
Step 1: Check your tools and set isolated KUBECONFIG
export KUBECONFIG="$PWD/.kubeconfig-da-k60"
docker info > /dev/null && kind version && kubectl version --client$env:KUBECONFIG = "$PWD\.kubeconfig-da-k60"
docker info > $null; kind version; kubectl version --clientYou should see: Docker running, kind v0.33.0 or higher, and kubectl client version. Setting KUBECONFIG ensures this lab never modifies your global ~/.kube/config.
Step 2: Negative control with plain Docker
docker compose up -d
curl -s http://127.0.0.1:18400/healthz
docker compose kill
curl -s http://127.0.0.1:18400/healthz || echo "Plain docker died and stayed dead"docker compose up -d
curl.exe -s http://127.0.0.1:18400/healthz
docker compose kill
curl.exe -s http://127.0.0.1:18400/healthz; if ($LASTEXITCODE -ne 0) { Write-Host "Plain docker died and stayed dead" }You should see: the plain container answers 200 OK initially, but after kill, every request fails immediately with connection refused. Docker does not self-heal.
Step 3: Create the 3-node Kubernetes cluster
kind create cluster --name da-k60 --config kind-config.yaml --kubeconfig "$PWD/.kubeconfig-da-k60"
kubectl get nodeskind create cluster --name da-k60 --config kind-config.yaml --kubeconfig "$PWD\.kubeconfig-da-k60"
kubectl get nodesYou should see: three nodes (da-k60-control-plane, da-k60-worker, da-k60-worker2) in Ready status.
Step 4: Build and load the Tatkal API image
docker build -t tatkal-api:1.0 app/
kind load docker-image tatkal-api:1.0 --name da-k60docker build -t tatkal-api:1.0 app/
kind load docker-image tatkal-api:1.0 --name da-k60You should see: image built successfully and transferred into the kind node images.
Step 5: Deploy the Tatkal API and Service
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl rollout status deployment/tatkal-api --timeout=60skubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl rollout status deployment/tatkal-api --timeout=60sYou should see: deployment "tatkal-api" successfully rolled out with 3 replicas Ready.
Step 6: Send requests through NodePort 18300
for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:18300/; echo; done1..6 | ForEach-Object { (Invoke-RestMethod http://127.0.0.1:18300/).pod }You should see: responses returning different pod IDs as kube-proxy distributes requests across the 3 replicas.
Step 7: Watch self-healing in action
POD=$(kubectl get pods -l app=tatkal-api -o jsonpath="{.items[0].metadata.name}")
echo "Deleting $POD..."
kubectl delete pod $POD --now
kubectl get pods -l app=tatkal-api$pod = (kubectl get pods -l app=tatkal-api -o jsonpath="{.items[0].metadata.name}")
Write-Host "Deleting $pod..."
kubectl delete pod $pod --now
kubectl get pods -l app=tatkal-apiYou should see: the deleted pod terminating and a brand-new replacement pod already running or in ContainerCreating. Within seconds, 3 replicas are Ready again.
06 /What You Should See
Real recorded output from the automated test run (test-first-cluster). Plain Docker failure reproduced, cluster bootstrapped, 3 pods deployed, and self-healing verified:
The numbers above come from one run on one laptop with Docker Desktop. Your machine will show different timings and counters. What must match is the shape of the result, the [OK] lines.
07 /Look Inside
1. Inspect nodes and internal IP addresses
Run: kubectl get nodes -o wide
You should see: the control-plane and two worker nodes, their internal container IPs, OS image, and kubelet version.
2. Inspect pods distribution across nodes
Run: kubectl get pods -o wide
You should see: the 3 replicas scheduled across both worker nodes (da-k60-worker and da-k60-worker2), each with an isolated cluster pod IP.
3. Inspect the NodePort Service endpoints
Run: kubectl get endpoints tatkal-api
You should see: the Service pointing to all 3 pod IP addresses on port 8080.
08 /Break It, Then Fix It
Experiment 1: Scale replicas. Change replicas: 3 to replicas: 5 in k8s/deployment.yaml and run kubectl apply -f k8s/deployment.yaml. What happens?
Answer: The Deployment controller detects that 5 replicas are wanted while 3 exist. It creates 2 additional pods immediately without restarting or disturbing the existing 3 pods.
Experiment 2: Break image name. Change image: tatkal-api:1.0 to image: tatkal-api:nonexistent in k8s/deployment.yaml and re-apply. What do you see?
Answer: kubectl get pods shows ErrImagePull followed by ImagePullBackOff. The 3 existing healthy pods stay in Running and keep serving requests. Kubernetes never terminates running pods until replacement pods pass readiness checks.
09 /What Changes in Real Projects
- Never use NodePort as your public production ingress. NodePort binds a port (30000-32767) across every cluster node. In production clouds (AWS, GCP, Azure), use an Ingress Controller (Nginx, Traefik, Envoy) backed by a managed Cloud Load Balancer with TLS termination.
- kind runs inside Docker on your local laptop. It is designed for development, testing, and continuous integration. It does not provide real hardware node isolation or cloud storage provisioners.
- Always specify CPU and memory requests and limits. Without
resources.requestsandresources.limits, a single leaking container can starve other pods on the same node and trigger out-of-memory kernel kills. - Configure Pod Disruption Budgets (PDB). In production clusters, node draining for kernel upgrades or AMI rotations can terminate pods. A PDB guarantees that a minimum number or percentage of replicas remain available at all times.
- Container filesystems are ephemeral. If a pod crashes and restarts, any files written inside the container are lost. Stateful databases must use PersistentVolumes (PV) backed by cloud block storage (AWS EBS, GCP Persistent Disk).
Choosing Kubernetes over Docker Compose trades lightweight simplicity for platform overhead. First, you pay roughly 1.3 GB of idle memory on your machine just to run etcd, the API server, and worker node background services before your first app pod boots. Second, you swallow a 1.3 GB node image download on the first run. Third, debugging is significantly more complex. In Compose, you run docker logs on one container. In Kubernetes, you have to trace pods, replica sets, Service endpoints, kube-proxy iptables rules, and overlay networks before you can explain why a simple HTTP request hung.
10 /Interview Corner
- How does Kubernetes self-healing work? The controller manager runs a reconciliation loop comparing the cluster current state with the declared desired state in etcd. If a pod terminates, the ReplicaSet controller detects the deficit and instructs the scheduler to bind a new pod to an eligible node. The node kubelet then creates the container.
- What is the difference between a Pod and a Container? A container is a single isolated process environment. A Pod is the smallest scheduling unit in Kubernetes, grouping one or more containers that share the same network namespace (localhost), IP address, and storage volumes.
- Why should you never deploy bare Pods in production? Bare pods lack lifecycle controllers. If a node hosting a bare pod fails, or if the pod process crashes without restart, Kubernetes does not reschedule it. A Deployment manages a ReplicaSet, guaranteeing that desired replica counts survive node failures and enabling zero-downtime rolling updates.
Follow-up questions the interviewer will ask
- How does Kubernetes know a pod has crashed and needs replacement?
- What is the difference between a Pod and a Container?
- Why use a Deployment instead of creating Pods directly?
11 /Clean Up
kind delete cluster --name da-k60 --kubeconfig "$PWD/.kubeconfig-da-k60"
docker compose down -v
rm -f "$PWD/.kubeconfig-da-k60"
unset KUBECONFIGkind delete cluster --name da-k60 --kubeconfig "$PWD\.kubeconfig-da-k60"
docker compose down -v
Remove-Item -Force "$PWD\.kubeconfig-da-k60" -ErrorAction SilentlyContinue
Remove-Item Env:\KUBECONFIG -ErrorAction SilentlyContinueThe automated runner cleans up the cluster and baseline container automatically in its finally block.