# DataBurst Data Engineering Wiki

> Free, open data engineering wiki: distributed systems, storage, file formats, streaming and data architecture, explained with interactive animations.

Each page is also available as raw markdown by appending `source.txt` to its URL.

- [Apache Hive](https://wiki.databurst.tech/docs/roadmap/query-engine/hive/): Apache Hive in 2026: HiveQL, the Hive Metastore, partition pruning (animated), Hive 4 on Tez, and why the Metastore outlived the engine.
- [Basic SQL](https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/): Basic SQL explained with animations: SELECT, WHERE, joins, GROUP BY and HAVING, NULL handling, INSERT/UPDATE/DELETE, and the logical order a query runs in.
- [Block Storage](https://wiki.databurst.tech/docs/roadmap/storage/block-storage/): How block storage works, animated: a write travels from the app through the file system to numbered blocks on a network volume. EBS, Ceph RBD and NVMe compared.
- [CAP](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/): CAP theorem explained with an interactive animation: what a distributed database does during a network partition, CP vs AP, eventual consistency and PACELC.
- [Columnar](https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/): Column-oriented analytical databases vs wide-column stores like Cassandra, and the LSM-tree write path they share, animated: memtable, WAL, flush, compaction.
- [Configuration Management](https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/): Configuration management explained: idempotent desired state, push vs pull, agentless vs agent-based, and how Ansible, Puppet, Chef and Salt compare in 2026.
- [Docker](https://wiki.databurst.tech/docs/roadmap/containerization/docker/): How Docker builds images from a Dockerfile, why layers and the build cache matter, how containers run from images, and Docker Compose for local data stacks.
- [ETL](https://wiki.databurst.tech/docs/roadmap/data-integration/etl/): ETL (extract, transform, load) animated: how data is cleaned in a staging engine before loading, when ETL still beats ELT, and common pitfalls.
- [File Formats](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/): Overview of the data file formats every data engineer meets: CSV, JSON, XML and Parquet, how they differ, and when each one is the right choice.
- [git](https://wiki.databurst.tech/docs/roadmap/version-control-system/git/): Git explained for data engineers: commits, the three areas, branches as pointers, merge vs rebase animated, everyday commands, and a workflow that works.
- [Github](https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/): GitHub for data teams: pull requests, branch rulesets, CODEOWNERS, GitHub Actions for CI/CD, security features, and where Copilot and Codespaces fit.
- [Kubernetes](https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/): Kubernetes for data engineers: Pods, Deployments, ReplicaSets, the scheduler and self-healing, animated, plus how Spark, Airflow and Kafka run on a cluster.
- [Logging](https://wiki.databurst.tech/docs/roadmap/observability/logging/): Logging done right: structured JSON logs, levels, correlation IDs, the collect-ship-store pipeline, and how Loki, Elasticsearch, OpenSearch and Splunk compare.
- [OAuth](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/): OAuth 2.0 explained, animated: the authorization code flow with PKCE, access and refresh tokens, scopes, client credentials, and how OpenID Connect adds login.
- [Programming Language](https://wiki.databurst.tech/docs/roadmap/programming-language/): Which programming languages data engineers need in 2026: Python and SQL first, then where Java, Scala, Go and Rust fit in the data tooling ecosystem.
- [Python](https://wiki.databurst.tech/docs/roadmap/programming-language/python/): Python for data engineering in 2026: the libraries that matter (Polars, DuckDB, PySpark, Airflow, dbt), uv for packaging, typing, performance and the GIL.
- [Relational](https://wiki.databurst.tech/docs/roadmap/databases/relational/): How relational databases work: tables, primary and foreign keys, joins, ACID transactions, B-tree indexes, and when PostgreSQL or MySQL is the right choice.
- [Relational Data Warehouse](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/): What a relational data warehouse is: OLTP vs OLAP, ETL into modeled tables, Inmon vs Kimball, MPP engines, and where it still fits in 2026.
- [Roadmap Step-by-Step](https://wiki.databurst.tech/docs/roadmap/): Step-by-step data engineering roadmap: fundamentals, storage, databases, distributed systems, pipelines and architecture, each with animated lessons.
- [Star Schema (Ralph Kimball)](https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/): How a Kimball star schema works: fact tables, dimension tables, grain, surrogate keys and slowly changing dimensions, with animated examples and SQL.
- [TCP/IP](https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/): How TCP/IP works, animated: the four-layer model, IP addresses and ports, the TCP three-way handshake, sequence numbers, retransmission and teardown.
- [Advanced SQL](https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/): Advanced SQL with animated window frames: CTEs and recursive CTEs, window functions, PARTITION BY, QUALIFY, GROUPING SETS, MERGE and materialized views.
- [Data Lake](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/): What a data lake is: raw files on object storage, schema-on-read, raw/clean/curated zones, catalogs, lake vs warehouse, and how to avoid a data swamp.
- [Docker Swarm](https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/): Docker Swarm mode explained: managers, workers, services and tasks, Raft consensus, overlay networks and the routing mesh, and when Swarm still fits.
- [Document Based](https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/): How document databases like MongoDB store JSON documents, when to embed vs reference, how they scale, and what to watch for when pipelines read them.
- [ELT](https://wiki.databurst.tech/docs/roadmap/data-integration/elt/): ELT (extract, load, transform) explained with an animation: land raw data first, model it with SQL and dbt inside the warehouse, and rebuild any time from raw.
- [Gitlab](https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/): GitLab explained: one platform for Git hosting, merge requests and built-in CI/CD, the self-managed option, .gitlab-ci.yml pipelines, runners and scanning.
- [Infrastructure Provisioning](https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/): How Terraform and OpenTofu provision infrastructure: providers, state, plan and apply animated, drift, remote backends and locking, plus the alternatives.
- [Java](https://wiki.databurst.tech/docs/roadmap/programming-language/java/): Why Java runs the big-data engine room: Kafka, Flink, Trino, Iceberg on the JVM, modern Java features (LTS releases, virtual threads), GC and JVM tuning basics.
- [JWT](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/): JSON Web Tokens explained, animated: header, payload and signature, how an API verifies a JWT with JWKS, which claims to check, and common security mistakes.
- [Monitoring](https://wiki.databurst.tech/docs/roadmap/observability/monitoring/): Metrics-based monitoring explained: counters, gauges and histograms, Prometheus pull scraping and PromQL, cardinality, SLOs and alerting for data pipelines.
- [NoSQL](https://wiki.databurst.tech/docs/roadmap/databases/nosql/): What NoSQL means, why non-relational databases exist, and how key-value, document, columnar, time-series, vector, graph and NewSQL systems differ.
- [Object Storage](https://wiki.databurst.tech/docs/roadmap/storage/object-storage/): How object storage like Amazon S3 works, animated: a PUT and a GET, keys and metadata, consistency, multipart uploads, and why data lakes use it.
- [PACELC](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/): PACELC extends CAP: during a partition choose availability or consistency, else choose latency or consistency. Animated write latency and real systems compared.
- [Podman](https://wiki.databurst.tech/docs/roadmap/containerization/podman/): Podman explained: a daemonless, rootless, Docker-compatible container engine with pods, Quadlet systemd units and Kubernetes YAML. When to pick it over Docker.
- [Serialization](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/): Schema-first binary serialization for data engineers: how Avro and Protobuf encode records compactly and let producers and consumers evolve schemas safely.
- [Snowflake](https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/): What a snowflake schema is, how normalizing dimensions into sub-tables trades extra joins for less redundancy, and when to choose it over a star schema.
- [SparkSQL](https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/): How Spark SQL turns SQL or DataFrame code into a distributed job, animated: Catalyst analysis, optimization, physical planning, codegen and AQE.
- [Apache Mesos](https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/): Apache Mesos and two-level scheduling: resource offers, frameworks like Marathon and Spark, why everyone moved to Kubernetes, and its 2025 retirement.
- [BitBucket](https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/): Bitbucket explained: Atlassian's Git host with Jira integration, Bitbucket Pipelines CI/CD, Cloud vs Data Center, and how it compares with GitHub and GitLab.
- [CDC](https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/): Change data capture animated: how Debezium reads the Postgres WAL or MySQL binlog and streams every insert, update and delete to Kafka as ordered change events.
- [containerd](https://wiki.databurst.tech/docs/roadmap/containerization/containerd/): What containerd is, where it sits between Docker, Kubernetes and runc, how the CRI works since dockershim's removal, and which CLIs to use (ctr, nerdctl).
- [Data Vault](https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/): Data Vault 2.0 explained: hubs, links and satellites, hash keys, insert-only loading and how new sources are added without redesign, with an animated model.
- [GraphQL](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/): GraphQL explained, animated: schemas, queries and resolvers, how it avoids over- and under-fetching compared to REST, the N+1 problem, and when not to use it.
- [Key-Value](https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/): How key-value stores like Redis, Valkey and DynamoDB work: get/put by key, hash partitioning, in-memory vs durable stores, TTLs, and typical caching use cases.
- [Modern Data Warehouse](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/): The modern data warehouse pattern: a data lake for raw data plus a cloud warehouse for serving, with ELT, and how it compares to a lakehouse.
- [Sharding](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/): Sharding explained with an animation: range vs hash partitioning, choosing a shard key, hot spots, rebalancing, cross-shard queries and when not to shard.
- [Tracing](https://wiki.databurst.tech/docs/roadmap/observability/tracing/): Distributed tracing explained with an animated trace waterfall: spans, context propagation, sampling, OpenTelemetry, and Jaeger, Tempo and Zipkin compared.
- [Trino](https://wiki.databurst.tech/docs/roadmap/query-engine/trino/): What Trino is and how it works: coordinator and workers, connectors and catalogs, federated queries, fault-tolerant execution, and the PrestoSQL fork.
- [Clock](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/): Time in distributed systems: clock drift and NTP, why timestamps mis-order events, Lamport and vector clocks animated, hybrid clocks and TrueTime.
- [Data Fabric](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/): What a data fabric is: a metadata-driven layer that connects, catalogs, governs and serves data across many systems, and how it differs from data mesh.
- [Presto](https://wiki.databurst.tech/docs/roadmap/query-engine/presto/): What Presto (PrestoDB) is today: Facebook's distributed SQL engine, its split from Trino, the Linux Foundation project, and the Velox-based C++ native workers.
- [REST](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/): REST API design explained: resources and URLs, HTTP methods and idempotency, status codes, pagination, versioning, OpenAPI, and REST vs GraphQL vs gRPC.
- [Reverse ETL](https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/): Reverse ETL animated: how modeled warehouse data is diffed and synced into CRMs, ad platforms and support tools, plus API limits, keys and common pitfalls.
- [Time Series](https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/): What time-series databases optimize for: time partitioning, compression, downsampling, retention and cardinality, with Prometheus, InfluxDB, TimescaleDB.
- [ACID](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/): ACID transactions explained: atomicity, consistency, isolation and durability, isolation levels and their anomalies, with an animated lost update and its fix.
- [Data Mart](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/): What a data mart is: a subject-focused slice of the warehouse for one team, dependent vs independent marts, conformed dimensions and how to avoid data silos.
- [gRPC](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/): gRPC explained: Protocol Buffers service definitions, code generation, HTTP/2 transport, the four streaming modes, deadlines, and when to pick gRPC over REST.
- [Replication](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/): How database replication works: leaders and followers, sync vs async copies, replication lag and stale reads, failover, and multi-leader vs leaderless designs.
- [Vector](https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/): How vector databases find similar items, animated: embeddings, cosine distance, exact vs approximate search, HNSW, and pgvector vs Qdrant vs Milvus.
- [Caching](https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/): Caching explained, animated: cache-aside hits, misses and LRU eviction, write strategies, TTLs and invalidation, cache stampedes, and Redis vs Valkey vs CDN.
- [Data Lakehouse](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/): How a data lakehouse works: open table formats (Iceberg, Delta Lake, Hudi) add ACID commits, snapshots and time travel to Parquet files on object storage.
- [Distributed Transactions](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/): Distributed transactions explained: two-phase commit animated with a coordinator crash, why 2PC blocks, consensus-based commit, sagas and the outbox pattern.
- [Graph](https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/): How graph databases model nodes and relationships, why multi-hop queries are fast, Cypher and the GQL standard, and when Neo4j beats SQL joins.
- [Data Mesh](https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/): Data mesh explained: domain ownership, data as a product, a self-serve platform and federated governance, plus when it helps and when it is overkill.
- [Indexing](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/): How database indexes work: B-tree lookup vs full table scan animated, index types (hash, LSM, GIN, BRIN) and local vs global indexes in sharded systems.
- [NewSQL](https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/): What NewSQL (distributed SQL) databases are: sharded, Raft-replicated relational systems with ACID transactions, like Spanner, CockroachDB, TiDB and YugabyteDB.
- [Version Control System](https://wiki.databurst.tech/docs/roadmap/version-control-system/): Why data engineers version everything (pipelines, SQL, infrastructure, schemas) and how a version control system like Git records, branches and merges changes.
- [Version Control System Hosting](https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/): What Git hosting platforms add on top of Git: pull requests, code review, branch protection and CI. An animated PR flow plus GitHub, GitLab and Bitbucket.
- [Consistency vs Eventual Consistency](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/): Strong vs eventual consistency: consistency models from linearizable to eventual, quorums (R + W > N) animated, conflict resolution and real settings.
- [Normalization or Denormalization](https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/): Normalization vs denormalization: update anomalies animated, normal forms by example, why OLTP normalizes and warehouses, NoSQL and sharded systems denormalize.
- [Databases](https://wiki.databurst.tech/docs/roadmap/databases/): A map of database families for data engineers: relational, key-value, document, columnar, wide-column, graph, time-series, vector and distributed SQL.
- [Storage](https://wiki.databurst.tech/docs/roadmap/storage/): Block, file and object storage compared: how each addresses data, typical latency and cost, and why data lakes are built on object storage like S3.
- [Query Engine](https://wiki.databurst.tech/docs/roadmap/query-engine/): How distributed SQL query engines work, animated: coordinator, splits, parallel scans, partial aggregation, exchange. Hive, Spark SQL, Trino, Presto.
- [Data Architecture](https://wiki.databurst.tech/docs/roadmap/data-architecture/): Data processing architectures for batch and real-time data: how Lambda and Kappa combine or replace batch and stream layers, and how to choose.
- [Data Processing](https://wiki.databurst.tech/docs/roadmap/data-processing/): How distributed data processing works, animated: split, map, shuffle and reduce, plus how Spark, Flink, Beam, dbt and DuckDB fit batch and streaming workloads.
- [Pipeline Orchestration](https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/): Pipeline orchestration animated: DAGs, task dependencies, retries and backfills, plus how Airflow 3, Dagster, Prefect, Argo and Kestra compare in 2026.
- [Messaging and Streaming Platform](https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/): Kafka-style logs animated: producers, partitions, offsets and consumer group rebalancing, plus queues vs logs and how Kafka, Pulsar, RabbitMQ and NATS compare.
- [Data Governance](https://wiki.databurst.tech/docs/roadmap/data-governance/): Data governance for data engineers: ownership, quality, lineage, catalogs, access control, privacy and data contracts, and how to automate them in the pipeline.
- [Data Visualization](https://wiki.databurst.tech/docs/roadmap/data-visualization/): Data visualization for data engineers: picking the right chart for the question, dashboard rules, common mistakes, and BI tools like Power BI and Superset.
- [Containerization](https://wiki.databurst.tech/docs/roadmap/containerization/): What containers are, how they differ from virtual machines, how images and OCI runtimes fit together, and why data teams ship pipelines in them.
- [Container Orchestration](https://wiki.databurst.tech/docs/roadmap/container-orchestration/): What container orchestrators do (scheduling, self-healing, scaling, rollouts), how desired-state reconciliation works, and Kubernetes vs Swarm vs Mesos.
- [CI/CD](https://wiki.databurst.tech/docs/roadmap/cicd/): CI/CD for data engineers: pipeline stages animated, continuous delivery vs deployment, testing SQL and dbt in CI, GitOps, and the main CI tools compared.
- [Infrastructure as Code](https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/): Infrastructure as Code explained: declarative vs imperative, provisioning vs configuration management, state and drift, and the main tools data teams use.
- [Observability](https://wiki.databurst.tech/docs/roadmap/observability/): Observability for data systems: logs, metrics and traces, how OpenTelemetry ties them together, and data observability for freshness, volume and schema.
- [Scala](https://wiki.databurst.tech/docs/roadmap/programming-language/scala/): Scala in data engineering in 2026: its role in Apache Spark, functional programming on the JVM, Scala 2.13 vs 3, and when Scala still beats PySpark.
- [Go](https://wiki.databurst.tech/docs/roadmap/programming-language/go/): Go for data engineers: why Kubernetes, Terraform and Prometheus are written in it, goroutines and channels for concurrent ingestion, and when to choose Go.
- [Rust](https://wiki.databurst.tech/docs/roadmap/programming-language/rust/): Rust in data engineering: memory safety without a garbage collector, the Rust data tools (Polars, DataFusion, delta-rs, uv), and Python bindings via PyO3.
- [Avro](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/): How Apache Avro works, animated: JSON schemas, compact binary records, writer vs reader schema resolution, schema registries and safe schema evolution.
- [CSV](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/): What CSV is, how RFC 4180 quoting works, why naive comma splitting corrupts data, and how to load CSV safely with explicit types, encodings and delimiters.
- [DNS](https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/): How DNS resolves a name to an IP, animated: stub and recursive resolvers, root, TLD and authoritative servers, record types, TTL caching and debugging.
- [Docs](https://wiki.databurst.tech/docs/)
- [Firewalls](https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/): How firewalls work, animated: stateless vs stateful packet filtering, connection tracking, default deny, egress rules, cloud security groups and WAFs.
- [HTTP](https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/): HTTP explained with an animated request/response: methods, status codes, headers, caching with ETags, and how HTTP/1.1, HTTP/2 and HTTP/3 differ.
- [HTTPS](https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/): How HTTPS works, animated: the TLS 1.3 handshake, key exchange, certificates and chains of trust, plus common certificate errors and how to debug them.
- [JSON](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/): JSON for data engineers: objects and arrays, JSON Lines for big files, JSON Schema, number and date pitfalls, and how to flatten nested JSON into tables.
- [Kappa](https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/): Kappa architecture explained with an animation: one stream processing path over a replayable log, and reprocessing by replaying into a new job version.
- [Lambda](https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/): Lambda architecture explained with an animation: batch, speed and serving layers, how results are merged, and why two codebases became its weak point.
- [Parquet](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/): How Apache Parquet works, animated: columnar storage, row groups, column chunks, min/max statistics, predicate pushdown, encodings and compression.
- [Protocol Buffers](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/): How Protocol Buffers work, animated: .proto schemas, field numbers on the wire, how old readers skip new fields, and the rules that keep schema changes safe.
- [SSH](https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/): How SSH works, animated: key exchange, host keys and known_hosts, public-key login with ed25519, plus SSH config, agent forwarding and tunnels to databases.
- [UDP](https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/): UDP explained: connectionless datagrams, the 8-byte header, why DNS, streaming and QUIC use it, and an animated side-by-side comparison with TCP.
- [VPN](https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/): How a VPN works, animated: tunneling and encapsulation, WireGuard vs IPsec vs OpenVPN, site-to-site vs remote access, and zero-trust alternatives.
- [XML](https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/): XML for data engineers: elements, attributes and namespaces, XSD validation, streaming parsers for large files, and how to turn XML trees into tables.