$ git switch -c fix/orders-freshness

DataOps

DataOps applies DevOps habits to data work: every change to pipelines, SQL models and schemas goes through version control, is tested automatically (both the code and the data it produces), is deployed by CI/CD through separate environments, and is monitored in production so problems are found by the team before users find them, and fixed fast. It is not a tool or a job title. It is a way of working that makes data changes small, frequent, safe and easy to undo.

The loop

$ gh pr create --title 'Add discount column to fct_orders'
One change, from branch to incident and back
BRANCHPULL REQUESTCIDEPLOY PIPELINE RUNMONITORALERTFIX CHANGE HISTORY gita17f3c90 fct_ordersv40v41 BRANCH feat/discountedit dbt model PULL REQUEST PR #219code review CI pytest, lint ✓dbt test on sample ✓ DEPLOY merge → prodsame artifact 9e2b v42 PIPELINE RUN orders_daily02:00 · runs MONITOR ✗ discount 40% nullwas 0% yesterday ALERT owner + on-call#orders-data FIX restore v41patch in a branch ← git revert ← restore to v41
  1. Every change goes around the same loop. Below it, two histories record what changed: the code in git, the data in table versions.
  2. An engineer adds a discount column in a branch and opens a pull request for review.
  3. CI runs code tests and lint, then builds the changed models on a sample of production data and runs data tests on the result.
  4. After merge, the same tested artifact is deployed. Git gets commit 9e2b; the next write will create table version v42.
  5. The scheduled pipeline run picks up the new code and writes fct_orders.
  6. Monitoring notices what the sample in CI did not: 40% of discounts are null in real data, up from 0%.
  7. An alert reaches the owner. The table is restored to v41 in seconds, the code is reverted, and the real fix starts in a new branch.

The practices

Practice What it means for data Typical tools
Version control SQL, DAGs, schemas, configs and infrastructure all live in git; no edits in production Git, GitHub/GitLab
Automated tests Unit tests for code, data tests for outputs: keys, nulls, accepted values, reconciliation pytest, dbt tests, Great Expectations, Soda
CI/CD Every pull request is built and tested; merges deploy automatically GitHub Actions, GitLab CI, dbt slim CI
Environments Dev, CI and prod are separated; test on clones or branches of real data, not on prod Warehouse zero-copy clones, Iceberg/Delta branches, lakeFS
Infrastructure as code Warehouses, buckets, permissions and clusters are declared, reviewed and reproducible Terraform, OpenTofu, Pulumi
Observability Freshness, volume, schema and quality metrics on every table; run durations and failures per job Orchestrator alerts, Elementary, Monte Carlo, OpenLineage
Fast recovery Small changes, easy rollback of code and data, a runbook for common incidents git revert, table time travel, backfills

A useful scorecard is the four DORA metrics applied to data: how often you deploy, how long a change takes to reach production, how often a change causes an incident, and how long it takes to recover.

Change history: code and data

When a number looks wrong, the first question is “what changed?”. DataOps answers it with two histories:

  • Code history in git. Every model, DAG and schema change is a commit linked to a pull request, a reviewer and a CI run. git log models/fct_orders.sql shows who changed the logic and why; git revert undoes it.
  • Data history in table versions. Lakehouse table formats keep a snapshot per write. You can list them, query the past and roll back:
-- Delta Lake
DESCRIBE HISTORY fct_orders;
SELECT * FROM fct_orders VERSION AS OF 41;
RESTORE TABLE fct_orders TO VERSION AS OF 41;

-- Apache Iceberg (Spark)
SELECT * FROM fct_orders VERSION AS OF 4348897519830128344;  -- snapshot id
CALL catalog.system.rollback_to_snapshot('db.fct_orders', 4348897519830128344);

Snowflake (AT (OFFSET => -3600), UNDROP TABLE) and BigQuery (FOR SYSTEM_TIME AS OF) offer time travel on their native tables too. Snapshots are kept only for a retention window, and expiring them is also how deleted data really disappears, so set that window on purpose (see Data Governance).

Together with lineage, the two histories let you answer “this table changed at 02:14 because commit 9e2b changed this join” instead of guessing.

Common mistakes

  • Treating DataOps as a tool purchase. An observability platform without tests in CI, owners and runbooks just produces more alerts.
  • Testing code but not data. A model that compiles can still double revenue. Data tests in CI and in production are the core of DataOps.
  • CI on toy data only. A hand-made fixture with five rows won’t show a 40% null rate. Add a sample or clone of real data, plus production monitoring.
  • Hotfixes in production. Editing a table or a DAG by hand breaks the history and the next deploy overwrites it. Even urgent fixes go through a branch.
  • No rollback plan for data. Reverting code doesn’t un-write bad rows. Know how you restore a table version or re-run a backfill before you need it.

Learning Resources

See also: CI/CD, Observability, Data Quality, Pipeline Orchestration