Kubernetes has one job: make reality match what you declared.
You write a YAML file that says 'I want 3 replicas of this app running, with 512MB RAM each, exposed on port 3000.' That is your desired state. Kubernetes reads it, stores it in etcd, and then spends every second of every day trying to make the cluster look like that declaration. This is called the reconciliation loop.
THE CONTROL PLANE — the brain of Kubernetes:
API Server: Every kubectl command, every Helm chart, every CI/CD pipeline talks to the API server. It is the single entry point for all cluster operations. It validates requests, authenticates them, and persists state to etcd.
etcd: A distributed key-value store that holds the entire cluster state. Every pod spec, every service definition, every config map lives in etcd. If etcd goes down, the cluster cannot make any scheduling decisions — existing pods keep running but nothing new can be created or modified. This is why etcd backup is non-negotiable in production.
Scheduler: When a new pod is created, it starts as Pending with no node assigned. The scheduler watches for unscheduled pods and assigns them to nodes based on: available resources (CPU, memory), node affinity rules, taints and tolerations, pod anti-affinity (don't put two replicas on the same node). The scheduler does not start the pod — it just picks the node.
Controller Manager: Runs dozens of controllers in a single process. The Deployment controller watches for Deployment objects and creates ReplicaSets. The ReplicaSet controller watches for ReplicaSets and creates Pods. The Node controller watches for node failures. Each controller runs its own reconciliation loop.
THE WORKER NODES — where your code actually runs:
kubelet: An agent running on every node. It watches the API server for pods assigned to its node, pulls the container image, starts the container via the container runtime (containerd or CRI-O), and reports back pod status. If a container crashes, kubelet reports it to the API server, which triggers the CrashLoopBackOff backoff logic.
kube-proxy: Manages iptables or IPVS rules on each node to implement Service networking. When you create a Service with ClusterIP, kube-proxy creates rules on every node so that traffic to the ClusterIP gets load-balanced to the correct pods.
Container Runtime: containerd or CRI-O. Actually pulls images and runs containers. Docker was removed as a runtime in Kubernetes 1.24.
WHAT ACTUALLY HAPPENS WHEN YOU RUN kubectl apply:
1. kubectl sends the YAML to the API server
2. API server validates and stores it in etcd
3. Deployment controller detects new desired state, creates a ReplicaSet
4. ReplicaSet controller creates 3 Pod objects (Pending state, no node assigned)
5. Scheduler assigns each pod to a node
6. kubelet on each node detects a pod assigned to it
7. kubelet pulls the container image
8. kubelet starts the container
9. Pod status changes to Running
kubectl returns after step 2. Steps 3-9 happen asynchronously. This is why kubectl apply returns instantly but your pod takes 30-60 seconds to be Running.
RESOURCE REQUESTS VS LIMITS — the source of most production incidents:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
Requests are what the scheduler uses. It finds a node where 256Mi is available and schedules the pod there. The pod might use more — requests are not a hard cap.
Limits are hard caps enforced by the kernel. If the container exceeds memory limit, the OOM killer terminates it immediately — no warning, no graceful shutdown. Exit code 137. If it exceeds CPU limit, it gets throttled — slowed down, not killed.
The production mistake everyone makes: setting memory limits too close to average usage. Your app uses 300MB normally but spikes to 700MB during report generation. With a 512Mi limit it gets OOMKilled every time someone runs a report. You see CrashLoopBackOff at 9AM every day. The logs look healthy because the OOM kill happens at the kernel level before the app can write anything.
Fix: set limits at 2-3x your average usage. Use kubectl top pod to measure actual usage before setting limits.
PROBES — the thing that separates zero-downtime deployments from outages:
Liveness Probe: Is this container still alive? If it fails, kubelet kills and restarts the container. Use for deadlock detection — your app is running but stuck in an infinite loop and not processing requests.
Readiness Probe: Is this container ready to receive traffic? If it fails, the pod is removed from the Service endpoints — traffic stops going to it but the container is not restarted. Use for startup time — your app takes 20 seconds to load config on startup. Without a readiness probe, Kubernetes sends traffic before it is ready and users get errors.
Startup Probe: Is the container still starting up? Disables liveness and readiness probes until startup succeeds. Use for slow-starting legacy apps.
The critical production insight: a failing readiness probe does NOT restart your pod. A failing liveness probe does. Mixing them up causes either unnecessary restarts or traffic being sent to unhealthy pods.
DEPLOYMENT STRATEGIES:
Rolling Update (default): Replaces pods one by one. Zero downtime if health checks are configured correctly. maxSurge controls how many extra pods can exist during rollout. maxUnavailable controls how many pods can be down during rollout.
Recreate: Kills all old pods then creates new ones. Causes downtime. Only use when you cannot run two versions simultaneously (database schema changes that are not backward compatible).
Blue/Green: Run two identical environments. Switch traffic at the load balancer level. Instant rollback. Expensive — doubles infrastructure cost.
Canary: Send a small percentage of traffic to the new version. Monitor error rates. Gradually increase traffic. Best for high-risk deployments.