Kubernetes
Kubernetes (K8s) is an open-source platform that runs containers across a cluster of machines and keeps them in the state you declared. You write YAML that says “run three copies of this image”, and Kubernetes places them on nodes, restarts them when they crash, replaces them when a node dies, and rolls out new versions. It grew out of Google’s internal Borg system, was open-sourced in 2014, and is now the default way to run containerized platforms, including data platforms.
The building blocks
| Object | What it is | Data-engineering example |
|---|---|---|
| Pod | One or more containers that share network and volumes; the unit Kubernetes schedules | A Spark executor, an Airflow task |
| Deployment | Keeps N identical, stateless Pods running and handles rolling updates | An ingestion API, a Kafka Connect worker |
| ReplicaSet | Created by a Deployment; owns the actual Pod count for one version | (Managed for you) |
| StatefulSet | Pods with stable names and their own persistent volumes | Kafka brokers and KRaft controllers, Postgres |
| Job / CronJob | Run Pods to completion, once or on a schedule | A nightly backfill, a dbt run |
| Service | Stable DNS name and virtual IP in front of a changing set of Pods | trino.analytics.svc |
| ConfigMap / Secret | Configuration and credentials injected into Pods | Warehouse connection strings |
| PersistentVolumeClaim | A request for durable storage | Broker log directories |
| Namespace | A folder for objects, with its own quotas and permissions | team-ingest, airflow |
The control plane (API server, etcd, scheduler, controller manager) stores the desired state and runs the reconciliation loops. Each worker node runs a kubelet that starts containers through a CRI runtime such as containerd.
Desired state in action
| NO. | STEP | COMPONENT | TYPE | INFO |
|---|---|---|---|---|
| 1 | 01 | deployment-controller | CREATE | ReplicaSet ingest-7d9f (replicas 3) |
| 2 | 02 | replicaset-controller | CREATE | pod-a, pod-b, pod-c (Pending, no node yet) |
| 3 | 03 | scheduler | BIND | pod-a → node1, pod-b → node2, pod-c → node3 (spread) |
| 4 | 04 | node-controller | NOTREADY | node3 missed heartbeats; its Pods are evicted |
| 5 | 05 | replicaset-controller | CREATE | have 2, want 3 → pod-d |
| 6 | 05 | scheduler | BIND | pod-d → node1 (most free capacity) |
- A cluster with three empty worker nodes. The scheduler is waiting for Pods to place.
- You apply a Deployment asking for 3 replicas. Its controller creates a ReplicaSet for this version of the image.
- The ReplicaSet creates 3 Pods. They are Pending: they exist in etcd but have no node yet.
- The scheduler filters nodes that fit the Pods' CPU and memory requests, scores them, and binds one Pod per node. Kubelets start the containers.
- Node 3 dies. After missed heartbeats it is marked NotReady, and after a grace period (5 minutes by default) pod-c is evicted. Actual state (2) no longer matches desired state (3).
- Nobody gets paged: the ReplicaSet creates pod-d and the scheduler places it on Node 1. This is self-healing: the same reconcile loop as the first deploy.
The manifest behind the figure:
apiVersion: apps/v1
kind: Deployment
metadata: { name: ingest }
spec:
replicas: 3
selector: { matchLabels: { app: ingest } }
template:
metadata: { labels: { app: ingest } }
spec:
containers:
- name: ingest
image: registry.example.com/ingest:2.4
resources:
requests: { cpu: "500m", memory: "1Gi" } # what the scheduler reserves
limits: { memory: "1Gi" } # exceeding it = OOMKilled
readinessProbe: { httpGet: { path: /health, port: 8080 } }A rolling update is the same machinery: change the image tag and the Deployment creates a second ReplicaSet, scaling the new one up and the old one down a few Pods at a time while readiness probes pass.
Kubernetes for data workloads
- Spark:
spark-submit --master k8s://...runs the driver and executors as Pods; the Spark Operator (Kubeflow) adds aSparkApplicationresource. - Airflow: the official Helm chart, with the KubernetesExecutor (one Pod per task) or
KubernetesPodOperatorfor running any image as a task. See Pipeline Orchestration. - Flink: the Flink Kubernetes Operator manages jobs, savepoints and upgrades.
- Kafka: Strimzi runs brokers as StatefulSets and manages topics and users as resources.
- Trino, ClickHouse, Postgres (CloudNativePG): operators or Helm charts for each.
The pattern is the operator: a custom resource plus a controller that encodes how to run one system, extending the same reconciliation idea.
Trade-offs
Kubernetes is powerful and heavy. Running the control plane yourself, upgrading every few months (minor versions get about 14 months of support), networking, storage classes and security policies are real work. Most teams should use a managed service (EKS, GKE, AKS) or a lightweight distribution (k3s) and stay on stateless workloads first. Stateful systems such as databases and Kafka run well on Kubernetes today, but only with a mature operator and tested backups.
Common mistakes
- No resource requests. The scheduler then packs Pods blindly and noisy neighbours starve each other. Always set CPU and memory requests; set a memory limit.
- Running one-off scripts in Deployments. If it finishes, it is a Job; a Deployment will restart it forever.
latestimage tags. Nodes cache images, so different Pods can run different code. Use immutable tags or digests.- Storing state in a Pod’s filesystem. Pods are cattle; use a PersistentVolumeClaim or object storage.
- Secrets as plain env vars in Git. Use an external secrets manager (External Secrets Operator, Vault, cloud KMS) or sealed secrets.
Learning Resources
Official
- Kubernetes concepts and interactive tutorials
- Running Spark on Kubernetes
- Kubernetes training and certifications (CKA, CKAD)
Papers
Books
- Kubernetes in Action, 2nd ed., Marko Lukša (Manning)
- Kubernetes: Up and Running, 3rd ed. (O’Reilly)
- Production Kubernetes (O’Reilly)
Courses
- Kubernetes Course: Full Beginners Tutorial (freeCodeCamp)
- Kubernetes Tutorial for Beginners, TechWorld with Nana
- Kubernetes the Hard Way, Kelsey Hightower: build a cluster by hand to understand every part