DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ kubectl apply -f ingest.yaml
# Kubernetes **Kubernetes** (K8s) is an open-source platform that runs containers across a cluster of machines and keeps them in the state you declared. You write YAML that says "run three copies of this image", and Kubernetes places them on nodes, restarts them when they crash, replaces them when a node dies, and rolls out new versions. It grew out of Google's internal Borg system, was open-sourced in 2014, and is now the default way to run containerized platforms, including data platforms. ## The building blocks | Object | What it is | Data-engineering example | |---|---|---| | **Pod** | One or more containers that share network and volumes; the unit Kubernetes schedules | A Spark executor, an Airflow task | | **Deployment** | Keeps N identical, stateless Pods running and handles rolling updates | An ingestion API, a Kafka Connect worker | | **ReplicaSet** | Created by a Deployment; owns the actual Pod count for one version | (Managed for you) | | **StatefulSet** | Pods with stable names and their own persistent volumes | Kafka brokers and KRaft controllers, Postgres | | **Job / CronJob** | Run Pods to completion, once or on a schedule | A nightly backfill, a dbt run | | **Service** | Stable DNS name and virtual IP in front of a changing set of Pods | `trino.analytics.svc` | | **ConfigMap / Secret** | Configuration and credentials injected into Pods | Warehouse connection strings | | **PersistentVolumeClaim** | A request for durable storage | Broker log directories | | **Namespace** | A folder for objects, with its own quotas and permissions | `team-ingest`, `airflow` | The **control plane** (API server, etcd, scheduler, controller manager) stores the desired state and runs the reconciliation loops. Each **worker node** runs a **kubelet** that starts containers through a CRI runtime such as [containerd]({{< relref "../../containerization/containerd" >}}). ## Desired state in action {{< fig kicker="$ kubectl apply -f ingest.yaml # replicas: 3" title="Deployment → ReplicaSet → Pods, then a node dies" >}} CONTROL PLANE DEPLOYMENT ingest · replicas: 3 REPLICASET ingest-7d9f · want 3 SCHEDULER fits Pods onto nodes pod-a pod-b pod-c Pending 3 Pods bound to nodes NODE 14 CPU · 16 GB NODE 24 CPU · 16 GB NODE 34 CPU · 16 GB pod-aingest:2.4 · Running pod-bingest:2.4 · Running pod-cingest:2.4 · Running NODE 3 · NotReady pod-c lost · 2 of 3 running pod-d (replacement)ingest:2.4 · Running
NO.STEPCOMPONENTTYPEINFO
101deployment-controllerCREATEReplicaSet ingest-7d9f (replicas 3)
202replicaset-controllerCREATEpod-a, pod-b, pod-c (Pending, no node yet)
303schedulerBINDpod-a → node1, pod-b → node2, pod-c → node3 (spread)
404node-controllerNOTREADYnode3 missed heartbeats; its Pods are evicted
505replicaset-controllerCREATEhave 2, want 3 → pod-d
605schedulerBINDpod-d → node1 (most free capacity)
  1. A cluster with three empty worker nodes. The scheduler is waiting for Pods to place.
  2. You apply a Deployment asking for 3 replicas. Its controller creates a ReplicaSet for this version of the image.
  3. The ReplicaSet creates 3 Pods. They are Pending: they exist in etcd but have no node yet.
  4. The scheduler filters nodes that fit the Pods' CPU and memory requests, scores them, and binds one Pod per node. Kubelets start the containers.
  5. Node 3 dies. After missed heartbeats it is marked NotReady, and after a grace period (5 minutes by default) pod-c is evicted. Actual state (2) no longer matches desired state (3).
  6. Nobody gets paged: the ReplicaSet creates pod-d and the scheduler places it on Node 1. This is self-healing: the same reconcile loop as the first deploy.
{{< /fig >}} The manifest behind the figure: ```yaml apiVersion: apps/v1 kind: Deployment metadata: { name: ingest } spec: replicas: 3 selector: { matchLabels: { app: ingest } } template: metadata: { labels: { app: ingest } } spec: containers: - name: ingest image: registry.example.com/ingest:2.4 resources: requests: { cpu: "500m", memory: "1Gi" } # what the scheduler reserves limits: { memory: "1Gi" } # exceeding it = OOMKilled readinessProbe: { httpGet: { path: /health, port: 8080 } } ``` A **rolling update** is the same machinery: change the image tag and the Deployment creates a second ReplicaSet, scaling the new one up and the old one down a few Pods at a time while readiness probes pass. ## Kubernetes for data workloads - **Spark**: `spark-submit --master k8s://...` runs the driver and executors as Pods; the **Spark Operator** (Kubeflow) adds a `SparkApplication` resource. - **Airflow**: the official Helm chart, with the **KubernetesExecutor** (one Pod per task) or `KubernetesPodOperator` for running any image as a task. See [Pipeline Orchestration]({{< relref "../../pipeline-orchestration" >}}). - **Flink**: the **Flink Kubernetes Operator** manages jobs, savepoints and upgrades. - **Kafka**: **Strimzi** runs brokers as StatefulSets and manages topics and users as resources. - **Trino**, **ClickHouse**, **Postgres** (CloudNativePG): operators or Helm charts for each. The pattern is the **operator**: a custom resource plus a controller that encodes how to run one system, extending the same reconciliation idea. ## Trade-offs Kubernetes is powerful and heavy. Running the control plane yourself, upgrading every few months (minor versions get about 14 months of support), networking, storage classes and security policies are real work. Most teams should use a managed service (**EKS**, **GKE**, **AKS**) or a lightweight distribution (**k3s**) and stay on stateless workloads first. Stateful systems such as databases and Kafka run well on Kubernetes today, but only with a mature operator and tested backups. ## Common mistakes - **No resource requests.** The scheduler then packs Pods blindly and noisy neighbours starve each other. Always set CPU and memory requests; set a memory limit. - **Running one-off scripts in Deployments.** If it finishes, it is a **Job**; a Deployment will restart it forever. - **`latest` image tags.** Nodes cache images, so different Pods can run different code. Use immutable tags or digests. - **Storing state in a Pod's filesystem.** Pods are cattle; use a PersistentVolumeClaim or object storage. - **Secrets as plain env vars in Git.** Use an external secrets manager (External Secrets Operator, Vault, cloud KMS) or sealed secrets. ## Learning Resources ### Official - [Kubernetes concepts](https://kubernetes.io/docs/concepts/) and [interactive tutorials](https://kubernetes.io/docs/tutorials/) - [Running Spark on Kubernetes](https://spark.apache.org/docs/latest/running-on-kubernetes.html) - [Kubernetes training and certifications (CKA, CKAD)](https://kubernetes.io/training/) ### Papers - [Large-scale cluster management at Google with Borg (EuroSys 2015)](https://research.google/pubs/large-scale-cluster-management-at-google-with-borg/) ### Books - [Kubernetes in Action, 2nd ed., Marko Lukša (Manning)](https://www.manning.com/books/kubernetes-in-action-second-edition) - [Kubernetes: Up and Running, 3rd ed. (O'Reilly)](https://www.oreilly.com/library/view/kubernetes-up-and/9781098110192/) - [Production Kubernetes (O'Reilly)](https://www.oreilly.com/library/view/production-kubernetes/9781492092292/) ### Courses - [Kubernetes Course: Full Beginners Tutorial (freeCodeCamp)](https://www.youtube.com/watch?v=d6WC5n9G_sM) - [Kubernetes Tutorial for Beginners, TechWorld with Nana](https://www.youtube.com/watch?v=X48VuDVv0do) - [Kubernetes the Hard Way, Kelsey Hightower](https://github.com/kelseyhightower/kubernetes-the-hard-way): build a cluster by hand to understand every part