DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ git switch -c fix/orders-freshness
# DataOps **DataOps** applies DevOps habits to data work: every change to pipelines, SQL models and schemas goes through **version control**, is **tested automatically** (both the code and the data it produces), is **deployed by CI/CD** through separate environments, and is **monitored** in production so problems are found by the team before users find them, and fixed fast. It is not a tool or a job title. It is a way of working that makes data changes small, frequent, safe and easy to undo. ## The loop {{< fig kicker="$ gh pr create --title 'Add discount column to fct_orders'" title="One change, from branch to incident and back" >}} BRANCHPULL REQUESTCIDEPLOY PIPELINE RUNMONITORALERTFIX CHANGE HISTORY gita17f3c90 fct_ordersv40v41 BRANCH feat/discountedit dbt model PULL REQUEST PR #219code review CI pytest, lint ✓dbt test on sample ✓ DEPLOY merge → prodsame artifact 9e2b v42 PIPELINE RUN orders_daily02:00 · runs MONITOR ✗ discount 40% nullwas 0% yesterday ALERT owner + on-call#orders-data FIX restore v41patch in a branch ← git revert ← restore to v41
  1. Every change goes around the same loop. Below it, two histories record what changed: the code in git, the data in table versions.
  2. An engineer adds a discount column in a branch and opens a pull request for review.
  3. CI runs code tests and lint, then builds the changed models on a sample of production data and runs data tests on the result.
  4. After merge, the same tested artifact is deployed. Git gets commit 9e2b; the next write will create table version v42.
  5. The scheduled pipeline run picks up the new code and writes fct_orders.
  6. Monitoring notices what the sample in CI did not: 40% of discounts are null in real data, up from 0%.
  7. An alert reaches the owner. The table is restored to v41 in seconds, the code is reverted, and the real fix starts in a new branch.
{{< /fig >}} ## The practices | Practice | What it means for data | Typical tools | |---|---|---| | Version control | SQL, DAGs, schemas, configs and infrastructure all live in git; no edits in production | Git, GitHub/GitLab | | Automated tests | Unit tests for code, **data tests** for outputs: keys, nulls, accepted values, reconciliation | pytest, dbt tests, Great Expectations, Soda | | CI/CD | Every pull request is built and tested; merges deploy automatically | GitHub Actions, GitLab CI, dbt slim CI | | Environments | Dev, CI and prod are separated; test on clones or branches of real data, not on prod | Warehouse zero-copy clones, Iceberg/Delta branches, lakeFS | | Infrastructure as code | Warehouses, buckets, permissions and clusters are declared, reviewed and reproducible | Terraform, OpenTofu, Pulumi | | Observability | Freshness, volume, schema and quality metrics on every table; run durations and failures per job | Orchestrator alerts, Elementary, Monte Carlo, OpenLineage | | Fast recovery | Small changes, easy rollback of code and data, a runbook for common incidents | git revert, table time travel, backfills | A useful scorecard is the four [DORA](https://dora.dev/) metrics applied to data: how often you deploy, how long a change takes to reach production, how often a change causes an incident, and how long it takes to recover. ## Change history: code and data When a number looks wrong, the first question is "what changed?". DataOps answers it with two histories: - **Code history in git.** Every model, DAG and schema change is a commit linked to a pull request, a reviewer and a CI run. `git log models/fct_orders.sql` shows who changed the logic and why; `git revert` undoes it. - **Data history in table versions.** Lakehouse table formats keep a snapshot per write. You can list them, query the past and roll back: ```sql -- Delta Lake DESCRIBE HISTORY fct_orders; SELECT * FROM fct_orders VERSION AS OF 41; RESTORE TABLE fct_orders TO VERSION AS OF 41; -- Apache Iceberg (Spark) SELECT * FROM fct_orders VERSION AS OF 4348897519830128344; -- snapshot id CALL catalog.system.rollback_to_snapshot('db.fct_orders', 4348897519830128344); ``` Snowflake (`AT (OFFSET => -3600)`, `UNDROP TABLE`) and BigQuery (`FOR SYSTEM_TIME AS OF`) offer time travel on their native tables too. Snapshots are kept only for a retention window, and expiring them is also how deleted data really disappears, so set that window on purpose (see [Data Governance]({{< relref "/docs/roadmap/data-governance" >}})). Together with lineage, the two histories let you answer "this table changed at 02:14 because commit 9e2b changed this join" instead of guessing. ## Common mistakes - **Treating DataOps as a tool purchase.** An observability platform without tests in CI, owners and runbooks just produces more alerts. - **Testing code but not data.** A model that compiles can still double revenue. Data tests in CI and in production are the core of DataOps. - **CI on toy data only.** A hand-made fixture with five rows won't show a 40% null rate. Add a sample or clone of real data, plus production monitoring. - **Hotfixes in production.** Editing a table or a DAG by hand breaks the history and the next deploy overwrites it. Even urgent fixes go through a branch. - **No rollback plan for data.** Reverting code doesn't un-write bad rows. Know how you restore a table version or re-run a backfill before you need it. ## Learning Resources - [The DataOps Manifesto](https://dataopsmanifesto.org/) - [DORA research program](https://dora.dev/) - [Delta Lake: table history and time travel](https://docs.delta.io/latest/delta-utility.html) - [Apache Iceberg: Spark queries and time travel](https://iceberg.apache.org/docs/latest/spark-queries/) - [Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/), DataOps undercurrent: Reis and Housley See also: [CI/CD]({{< relref "/docs/roadmap/cicd" >}}), [Observability]({{< relref "/docs/roadmap/observability" >}}), [Data Quality]({{< relref "/docs/roadmap/data-governance/data-quality" >}}), [Pipeline Orchestration]({{< relref "/docs/roadmap/pipeline-orchestration" >}})