$ kubectl apply -f ingest.yaml

Kubernetes

Kubernetes (K8s) is an open-source platform that runs containers across a cluster of machines and keeps them in the state you declared. You write YAML that says “run three copies of this image”, and Kubernetes places them on nodes, restarts them when they crash, replaces them when a node dies, and rolls out new versions. It grew out of Google’s internal Borg system, was open-sourced in 2014, and is now the default way to run containerized platforms, including data platforms.

The building blocks

Object What it is Data-engineering example
Pod One or more containers that share network and volumes; the unit Kubernetes schedules A Spark executor, an Airflow task
Deployment Keeps N identical, stateless Pods running and handles rolling updates An ingestion API, a Kafka Connect worker
ReplicaSet Created by a Deployment; owns the actual Pod count for one version (Managed for you)
StatefulSet Pods with stable names and their own persistent volumes Kafka brokers and KRaft controllers, Postgres
Job / CronJob Run Pods to completion, once or on a schedule A nightly backfill, a dbt run
Service Stable DNS name and virtual IP in front of a changing set of Pods trino.analytics.svc
ConfigMap / Secret Configuration and credentials injected into Pods Warehouse connection strings
PersistentVolumeClaim A request for durable storage Broker log directories
Namespace A folder for objects, with its own quotas and permissions team-ingest, airflow

The control plane (API server, etcd, scheduler, controller manager) stores the desired state and runs the reconciliation loops. Each worker node runs a kubelet that starts containers through a CRI runtime such as containerd.

Desired state in action

$ kubectl apply -f ingest.yaml # replicas: 3
Deployment → ReplicaSet → Pods, then a node dies
CONTROL PLANE DEPLOYMENT ingest · replicas: 3 REPLICASET ingest-7d9f · want 3 SCHEDULER fits Pods onto nodes pod-a pod-b pod-c Pending 3 Pods bound to nodes NODE 14 CPU · 16 GB NODE 24 CPU · 16 GB NODE 34 CPU · 16 GB pod-aingest:2.4 · Running pod-bingest:2.4 · Running pod-cingest:2.4 · Running NODE 3 · NotReady pod-c lost · 2 of 3 running pod-d (replacement)ingest:2.4 · Running
NO.STEPCOMPONENTTYPEINFO
101deployment-controllerCREATEReplicaSet ingest-7d9f (replicas 3)
202replicaset-controllerCREATEpod-a, pod-b, pod-c (Pending, no node yet)
303schedulerBINDpod-a → node1, pod-b → node2, pod-c → node3 (spread)
404node-controllerNOTREADYnode3 missed heartbeats; its Pods are evicted
505replicaset-controllerCREATEhave 2, want 3 → pod-d
605schedulerBINDpod-d → node1 (most free capacity)
  1. A cluster with three empty worker nodes. The scheduler is waiting for Pods to place.
  2. You apply a Deployment asking for 3 replicas. Its controller creates a ReplicaSet for this version of the image.
  3. The ReplicaSet creates 3 Pods. They are Pending: they exist in etcd but have no node yet.
  4. The scheduler filters nodes that fit the Pods' CPU and memory requests, scores them, and binds one Pod per node. Kubelets start the containers.
  5. Node 3 dies. After missed heartbeats it is marked NotReady, and after a grace period (5 minutes by default) pod-c is evicted. Actual state (2) no longer matches desired state (3).
  6. Nobody gets paged: the ReplicaSet creates pod-d and the scheduler places it on Node 1. This is self-healing: the same reconcile loop as the first deploy.

The manifest behind the figure:

apiVersion: apps/v1
kind: Deployment
metadata: { name: ingest }
spec:
  replicas: 3
  selector: { matchLabels: { app: ingest } }
  template:
    metadata: { labels: { app: ingest } }
    spec:
      containers:
        - name: ingest
          image: registry.example.com/ingest:2.4
          resources:
            requests: { cpu: "500m", memory: "1Gi" }   # what the scheduler reserves
            limits:   { memory: "1Gi" }                # exceeding it = OOMKilled
          readinessProbe: { httpGet: { path: /health, port: 8080 } }

A rolling update is the same machinery: change the image tag and the Deployment creates a second ReplicaSet, scaling the new one up and the old one down a few Pods at a time while readiness probes pass.

Kubernetes for data workloads

  • Spark: spark-submit --master k8s://... runs the driver and executors as Pods; the Spark Operator (Kubeflow) adds a SparkApplication resource.
  • Airflow: the official Helm chart, with the KubernetesExecutor (one Pod per task) or KubernetesPodOperator for running any image as a task. See Pipeline Orchestration.
  • Flink: the Flink Kubernetes Operator manages jobs, savepoints and upgrades.
  • Kafka: Strimzi runs brokers as StatefulSets and manages topics and users as resources.
  • Trino, ClickHouse, Postgres (CloudNativePG): operators or Helm charts for each.

The pattern is the operator: a custom resource plus a controller that encodes how to run one system, extending the same reconciliation idea.

Trade-offs

Kubernetes is powerful and heavy. Running the control plane yourself, upgrading every few months (minor versions get about 14 months of support), networking, storage classes and security policies are real work. Most teams should use a managed service (EKS, GKE, AKS) or a lightweight distribution (k3s) and stay on stateless workloads first. Stateful systems such as databases and Kafka run well on Kubernetes today, but only with a mature operator and tested backups.

Common mistakes

  • No resource requests. The scheduler then packs Pods blindly and noisy neighbours starve each other. Always set CPU and memory requests; set a memory limit.
  • Running one-off scripts in Deployments. If it finishes, it is a Job; a Deployment will restart it forever.
  • latest image tags. Nodes cache images, so different Pods can run different code. Use immutable tags or digests.
  • Storing state in a Pod’s filesystem. Pods are cattle; use a PersistentVolumeClaim or object storage.
  • Secrets as plain env vars in Git. Use an external secrets manager (External Secrets Operator, Vault, cloud KMS) or sealed secrets.

Learning Resources

Official

Papers

Books

Courses