$ cat what-is-data-engineering.md

What is data engineering?

Data engineering is the work of building and running the systems that collect data from where it is created, store it, clean and reshape it, and deliver it to the people and programs that need it, reliably and on time. A data engineer turns scattered, messy source data (app databases, event streams, files, third-party APIs) into trustworthy tables, feeds and features that analysts, data scientists, ML models and business tools can use without thinking about where the data came from.

Put simply: every chart, report and model depends on data that someone moved, checked and organized first. That someone is the data engineer.

The data engineering lifecycle

A useful way to see the whole job is the data engineering lifecycle, popularized by the book Fundamentals of Data Engineering. Data passes through five stages, and a set of undercurrents support every stage. Follow one order in an online shop through it:

$ trace order --id 1042
One order, from the shop app to the dashboard
GENERATION app database order #1042 Alice · $59.00 INGESTION CDC · batch copied out every change STORAGE data lake raw, kept as it landed TRANSFORM SQL · Spark cleaned, joined → fct_orders SERVING to its users dashboard ML model reverse ETL UNDERCURRENTS apply to every stage above security data management DataOps data architecture orchestration software engineering
  1. Generation: Alice buys a $59 item. The shop app writes order #1042 into its own database. Data engineers don't own this system, but they must understand it.
  2. Ingestion: the order is copied out of the app database, either in nightly batches or continuously with change data capture (CDC).
  3. Storage: the raw copy lands in a data lake or warehouse and is kept, so it can be reprocessed later.
  4. Transformation: the raw rows are cleaned, deduplicated and joined with customers and products into a tidy fct_orders table.
  5. Serving: the result feeds a sales dashboard, a recommendation model, and reverse ETL back into the CRM.
  6. Undercurrents: security, data management, DataOps, architecture, orchestration and software engineering run under all five stages.
Stage Question it answers Typical tools
Generation Where does the data come from? App databases, event trackers, SaaS tools, IoT devices
Ingestion How do we get it out, and how often? CDC (Debezium), Kafka, Airbyte, custom extract jobs
Storage Where do we keep it, and in what format? Object storage (S3), warehouses, lakehouse tables, Parquet
Transformation How do we make it correct and useful? SQL, dbt, Spark, Flink
Serving Who uses it, and how? BI dashboards, ML features, APIs, reverse ETL

The undercurrents are what separate a pipeline that works once from one that works every day: security (who may see what), data management (quality, lineage, catalogs, governance), DataOps (testing, CI/CD and monitoring for data), data architecture (how the pieces fit and scale), orchestration (scheduling and dependencies between jobs) and software engineering (version control, code review, clean code).

The five Vs of big data

The classic way to describe what makes data hard to handle:

V Meaning Online shop example
Volume How much data there is Millions of orders and billions of page-view events
Velocity How fast it arrives and must be processed Thousands of clicks per second during a sale
Variety How many shapes and formats it comes in Database rows, JSON events, CSV files, product images
Veracity How trustworthy and accurate it is Duplicate orders, missing prices, wrong time zones
Value Whether it is worth the effort Better stock planning and recommendations

You don’t need all five to need a data engineer. Even a small shop with messy data from three tools has a veracity problem.

One dataset, many questions

Several roles work with data, and they often look at the same tables. What differs is the question they ask and what they build:

$ grep -r orders ./team
Same orders table, five different jobs
orders idamountcity 104259.00Lyon 104312.50Oslo 104480.00Lima DATA ENGINEER "Do all orders arrive complete and on time?" builds: pipelines, storage, raw → clean tables ANALYTICS ENGINEER "What is our one trusted definition of revenue?" builds: tested SQL models (dbt), fct_orders, metrics DATA ANALYST "Which cities grew last quarter, and why?" builds: SQL queries, dashboards, reports DATA SCIENTIST "Which customers are likely to stop buying?" builds: experiments, statistical and predictive models ML / AI ENGINEER "How do we show each shopper good picks in 50 ms?" builds: model serving, feature pipelines, LLM apps delivered by the data engineer
  1. An online shop has one orders table. Let's see who uses it.
  2. The data engineer makes sure the orders get there: complete, on time, in the right shape.
  3. The analytics engineer models the clean data into trusted, tested tables and shared metric definitions.
  4. The data analyst answers business questions with SQL and dashboards.
  5. The data scientist looks for patterns and builds models that predict what happens next.
  6. The ML / AI engineer puts models into production, where they answer in milliseconds.
  7. Every role above depends on the first one. Without reliable pipelines, the others have no data to work with.

Titles blur in practice: in a small company one person may do all five jobs; in a large one each is a team. Newer roles are appearing too, such as inference engineers, who make models cheaper and faster to run, and forward deployed engineers, who build data and AI solutions on site with a customer. All of them rely on the same foundations this roadmap covers.

How to use this roadmap

The roadmap is ordered so each stage builds on the one before it. You don’t have to finish a stage before peeking at the next, but the order saves time: it is hard to understand Kafka without knowing networking, or a lakehouse without knowing file formats.

  1. Foundations: Programming Language, Operating System, Networking, General Software Engineering Skills, Web Frameworks and API Development, Version Control and Version Control Hosting.
  2. Distributed systems, SQL and databases: Distributed Systems Concepts, SQL Fundamentals and Databases.
  3. Modeling, architecture and storage: Data Modeling, Data Management Architecture, Data Integration, Storage, File Formats and Serialization, Query Engines and Data Architecture.
  4. Processing, orchestration and governance: Data Processing, Pipeline Orchestration, Messaging and Streaming Platforms and Data Governance.
  5. Visualization, infrastructure and observability: Data Visualization, Containerization, Container Orchestration, CI/CD, Infrastructure as Code and Observability.

The stages mirror what data engineering job ads actually ask for: a programming language (usually Python and SQL), databases, distributed processing (Spark, Kafka), an orchestrator, a cloud platform and containers. New to a term? Look it up in the Glossary.

Learning Resources

Books

Roadmap