$ docker build -t etl:1.0 . && docker run etl:1.0

Docker

Docker is the toolset that made containers mainstream: you describe an environment in a Dockerfile, docker build turns it into an image, and docker run starts a container from that image. Images are pushed to a registry (Docker Hub, GHCR, ECR) so any machine can pull and run exactly the same bits.

Under the hood, Docker Engine (dockerd) delegates to containerd and runc to actually start containers. Docker itself is the friendly layer on top: the CLI, the builder (BuildKit, the default since Engine 23), networking, volumes and Compose.

What happens when you type docker run

$ docker run -d --name pg -e POSTGRES_PASSWORD=secret postgres:18
Client, daemon, registry: one docker run
CLIENT $ docker run postgres:18 your terminal: only sends requests DOCKER HOST · LINUX dockerd images · networks · volumes REGISTRY docker.io GHCR · ECR · Quay REST API over /var/run/docker.sock image not local: pull missing layers containerd unpacks, supervises runc namespaces + cgroups, exits pg · postgres:18 · Up macOS / Windows: this box is a hidden Linux VM ports are forwarded to localhost · on Linux, no VM
  1. Three parts: the client (the docker CLI), the daemon on the Docker host, and a registry that stores images.
  2. The CLI sends the request to dockerd over a REST API on a Unix socket. "Cannot connect to the Docker daemon" means nobody is listening there.
  3. The daemon has no postgres:18 locally, so it pulls the missing layers from Docker Hub.
  4. containerd takes over the container's lifecycle: it unpacks the image and supervises the process.
  5. runc asks the kernel for namespaces and cgroups, starts the process inside them, and exits.
  6. The container is running: a fenced-off postgres process on the host's kernel.
  7. On macOS and Windows, everything inside the dashed box runs in a hidden Linux VM (Docker Desktop, Podman machine, OrbStack, Colima, WSL 2). Only the CLI runs natively.

The hidden VM explains several laptop surprises: published ports are forwarded from the VM to your localhost, bind mounts cross the VM boundary and are slower for big folders, the VM has its own CPU and memory limit (check the Docker Desktop settings), and Apple Silicon wants arm64 images while amd64-only images run emulated.

Images are stacks of layers

Each instruction in a Dockerfile that changes the filesystem produces a layer: a read-only tarball of changes, identified by its content hash. An image is an ordered list of layers. A container adds one thin writable layer on top.

$ docker build -t etl:1.0 .
From Dockerfile to image to running containers
Dockerfile 1 FROM python:3.12-slim 2 COPY requirements.txt . 3 RUN pip install -r requirements.txt 4 COPY etl.py . IMAGE etl:1.0 (READ-ONLY) L1 python:3.12-slim~45 MB L2 requirements.txt1 KB L3 pandas, pyarrow…~180 MB L4 etl.py6 KB CONTAINER Arw layer CONTAINER Brw layer both share L1 to L4 on disk edit etl.py, rebuild: L1 to L3: CACHED L4: REBUILT (6 KB) push: only L4 uploads
  1. A four-line Dockerfile for a small Python ETL job. Nothing is built yet.
  2. FROM pulls the base image. Its layers become the bottom of our stack.
  3. COPY and RUN each add a layer. The dependency layer is the big one.
  4. COPY etl.py adds the last layer. Together, L1 to L4 are the read-only image etl:1.0.
  5. docker run starts a container: the image plus a thin writable layer. Files the job writes go there and vanish with the container.
  6. A second container from the same image shares every image layer. Only its own writable layer is new, so starting it is near-instant.
  7. Change only etl.py and rebuild: layers above the change are rebuilt, the rest come from the build cache. That is why dependency installs go before code copies.

Reading an image name

A full image reference has up to five parts. postgres on its own is shorthand for docker.io/library/postgres:latest.

Part In docker.io/library/postgres:18-alpine@sha256:… Meaning
Registry docker.io Where the image lives. Default is Docker Hub; others: ghcr.io, quay.io, <account>.dkr.ecr.<region>.amazonaws.com
Namespace library The owner. library is Docker Hub’s official images; otherwise a user or org
Repository postgres The image name
Tag 18-alpine A human label for one version. Mutable: it can be moved to new bytes. Default latest means “default”, not “newest”
Digest @sha256:… The hash of the image manifest. Immutable: always the exact same bytes

Pin tags (postgres:18) everywhere and digests in production. docker images --digests shows the digest of what you pulled.

A practical Dockerfile for a data job

# Stage 1: build wheels in the full image, which has compilers
FROM python:3.12 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip wheel --wheel-dir /wheels -r requirements.txt

# Stage 2: small runtime image, no compilers
FROM python:3.12-slim
RUN useradd --create-home etl
WORKDIR /app
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels
COPY etl.py .
USER etl
ENTRYPOINT ["python", "etl.py"]

Two habits matter most: put rarely-changing steps (dependencies) before often-changing ones (code) so the cache works, and use a multi-stage build so build tools never reach the final image.

The cache rule in one line: when a layer changes, it and every layer after it rebuild. Compare:

# Slow: any code edit invalidates COPY, so pip reinstalls everything
COPY . .
RUN pip install -r requirements.txt

# Fast: dependencies stay cached until requirements.txt changes
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

docker build -t myapp:1.0 . builds it; the trailing . is the build context, the folder sent to the builder (filtered by .dockerignore).

Anatomy of docker run

docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5433:5432 -v pgdata:/var/lib/postgresql postgres:18
Flag Meaning
-d Detached: run in the background and print the container ID
--name pg A name to type instead of the random ID
-e KEY=value Environment variable: configuration and passwords
-p 5433:5432 Publish a port: host 5433 → container 5432
-v pgdata:/var/lib/postgresql Mount the named volume pgdata at that path (created if missing)
--network labnet Attach to a user-defined network, where containers find each other by name
--rm Delete the container when it exits (good for one-off jobs)
-it Interactive terminal, for shells and psql
postgres:18 The image. Everything after it is the command and its arguments, not Docker flags

Ports, volumes and networks

$ docker rm -f pg && docker run ... -v pgdata:/var/lib/postgresql postgres:18
The container dies, the volume survives
YOUR LAPTOP psql localhost:5432 DOCKER HOST pg postgres on :5432 -p 5432:5432 INSERT 2 rows -v pgdata:/var/lib/postgresql volume pgdata users: Alice, Sam docker rm -f pg container + writable layer gone pg (new) same -v pgdata SELECT * FROM users; 2 rows: the data survived
  1. A Postgres container is running, but nothing on your laptop can reach it yet: containers are isolated by default.
  2. -p 5432:5432 opens a door: traffic to localhost:5432 on the host is forwarded to port 5432 inside the container.
  3. Rows you insert are written to /var/lib/postgresql, which is the named volume pgdata, not the container's own layer.
  4. docker rm -f pg destroys the container and its writable layer. The volume is not touched.
  5. Run the same command again: a brand-new container mounts the same pgdata volume.
  6. The rows are still there. Containers are disposable; volumes hold the state.

Where data should live:

Mount Example Best for Survives docker rm?
Named volume -v pgdata:/var/lib/postgresql Databases. Docker manages where it lives Yes
Bind mount -v "$(pwd)/src":/app Live-editing code, passing input files Yes, it is your folder
tmpfs --tmpfs /tmp Scratch space, secrets that must not touch disk No, it lives in RAM
Nothing No: writes go to the container layer

Manage volumes with docker volume ls, docker volume inspect pgdata, docker volume rm pgdata and docker volume prune (unused volumes only).

Postgres 18 changed the mount path. For postgres:18 and newer, mount the volume at /var/lib/postgresql; for postgres:17 and older it is /var/lib/postgresql/data. Most tutorials still show the old path; on 18 it either fails to start or leaves your data outside the volume.

Networks. On a user-defined network (docker network create labnet), containers reach each other by name: a web app connects to db:5432. The default bridge network has no name resolution. Inside a container, localhost means that container itself, not your laptop and not the database container.

Docker Compose for local stacks

docker compose (Compose v2, built into the CLI) starts several containers from one YAML file. It is the usual way to run a local data stack: Postgres as a source, Kafka, MinIO as fake S3, and Airflow, all on one private network where containers reach each other by service name.

services:
  postgres:
    image: postgres:17
    environment: { POSTGRES_PASSWORD: dev }
    volumes: [pgdata:/var/lib/postgresql/data]
  minio:
    image: minio/minio
    command: server /data
    ports: ["9000:9000"]
volumes:
  pgdata:

MinIO stopped publishing community Docker images in late 2025, so minio/minio no longer gets updates; it is fine for a throwaway local stack, but for anything longer-lived consider an S3-compatible alternative such as SeaweedFS or Garage.

Compose is for development and single-host setups. For many hosts, use Kubernetes.

Ten commands you’ll use daily

A container moves through created → running → stopped → removed: run creates and starts it, stop halts it with its files intact, start brings it back, rm deletes it.

Command What it does
docker run IMAGE Create and start a container
docker ps -a List containers, including stopped ones
docker logs -f NAME Follow its output
docker exec -it NAME sh Open a shell inside a running container (debug only)
docker stop NAME Stop it; its files stay
docker start NAME Start a stopped container again
docker rm -f NAME Delete it and its writable layer
docker inspect NAME Every detail (IP, mounts, env, exit code) as JSON
docker images List local images
docker rmi IMAGE Delete an image

And for building and shipping images:

Command What it does
docker build -t etl:1.0 . Build an image from the Dockerfile in .
docker image history etl:1.0 Show the layers and their sizes
docker tag etl:1.0 ghcr.io/acme/etl:1.0 Give the image a registry name
docker push ghcr.io/acme/etl:1.0 Upload it to a registry
docker system df, docker system prune See and reclaim disk used by stopped containers, unused images and cache

Hands-on lab: Postgres and a web UI in Docker

About an hour, no programming. You need Docker Desktop (macOS, Windows) or Docker Engine with the Compose plugin (Linux); docker version must show both a Client and a Server section. Pre-pull the images on a good connection: docker pull postgres:18 adminer alpine.

1. Run Postgres.

docker run -d --name pg \
  -e POSTGRES_PASSWORD=secret \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql \
  postgres:18
docker ps            # pg should show as Up
docker logs pg       # look for: database system is ready to accept connections

Port 5432 already taken by a local Postgres? Use -p 5433:5432.

2. Connect and query.

docker exec -it pg psql -U postgres
CREATE TABLE users (id serial PRIMARY KEY, name text NOT NULL, city text);
INSERT INTO users (name, city) VALUES ('Alice', 'Berlin'), ('Sam', 'Lisbon');
SELECT * FROM users;
\q

Any SQL client on your laptop can also connect to localhost:5432 as postgres / secret.

3. Destroy the container, keep the data.

docker rm -f pg
docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 \
  -v pgdata:/var/lib/postgresql postgres:18          # same command as step 1
docker exec -it pg psql -U postgres -c "SELECT * FROM users;"

Alice and Sam are still there. Repeat with -v pgdata2:/var/lib/postgresql and the table is gone: a new volume is an empty database.

4. Build your own image with seed data. In an empty folder, create init.sql:

CREATE TABLE cities (id serial PRIMARY KEY, name text NOT NULL, country text);
INSERT INTO cities (name, country) VALUES ('Berlin', 'Germany'), ('Lisbon', 'Portugal');

and a Dockerfile:

FROM postgres:18
# *.sql and *.sh files here run on the first start of an empty database
COPY init.sql /docker-entrypoint-initdb.d/
docker build -t cities-db:1.0 .
docker run -d --name cities -e POSTGRES_PASSWORD=secret -p 5433:5432 cities-db:1.0
docker exec -it cities psql -U postgres -c "SELECT * FROM cities;"

Init scripts run only when the data directory is empty, so they are skipped when you mount a volume that already holds a database.

5. Connect a web server to the database. Two containers on a user-defined network, finding each other by name. Adminer is a small web UI for databases.

docker rm -f pg cities                  # free port 5432 and the pgdata volume
docker network create labnet
docker run -d --name db  --network labnet -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql postgres:18
docker run -d --name web --network labnet -p 8080:8080 adminer

Open http://localhost:8080 and log in with System PostgreSQL, Server db, user postgres, password secret. The server is the container name, not an IP and not localhost. Note that db publishes no port: only containers on labnet can reach it.

6. The same with one Compose file. Remove the two containers (docker rm -f db web), then save this as compose.yaml:

services:
  db:
    image: postgres:18
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - pgdata:/var/lib/postgresql
  web:
    image: adminer
    ports:
      - "8080:8080"
    depends_on: [db]
volumes:
  pgdata:
docker compose up -d       # creates a network; services reach each other by service name
docker compose ps
docker compose down        # removes containers and network, keeps the volume
docker compose down -v     # also deletes the volume and the data

Clean up afterwards with docker volume rm pgdata pgdata2 and docker system prune.

Troubleshooting

You see Check with Fix
Cannot connect to the Docker daemon docker version (no Server section) Start Docker Desktop, or sudo systemctl start docker on Linux
port is already allocated / address already in use docker ps, lsof -i :5432 Stop the other process, or map a different host port: -p 5433:5432
Postgres exits: superuser password is not specified docker logs pg Add -e POSTGRES_PASSWORD=...
Postgres 18 fails with a mount or data-directory error docker inspect pg (Mounts) Mount at /var/lib/postgresql, not /var/lib/postgresql/data
init.sql did not run docker volume ls The volume already held a database. Use a new volume, or docker volume rm it
App container cannot reach localhost:5432 docker network inspect labnet Put both on one user-defined network and connect to db:5432
“It worked a minute ago” docker ps -a The container exited; read why with docker logs NAME
Pull is slow or blocked docker pull output Use a mirror, or docker save -o images.tar postgres:18 on one machine and docker load -i images.tar on another
Anything else docker logs NAME Read the last lines first

If you remember three words: an image is the recipe (read-only layers), a container is a running instance (an isolated process), and a volume is the data that outlives the container.

Common mistakes

  • latest in production. Tags are mutable. Pin a version, or better, the image digest (@sha256:…).
  • Secrets in layers. A password in a COPYed file or ENV stays in the image history even if a later step deletes it. Use BuildKit secrets (RUN --mount=type=secret) and runtime env vars.
  • Data in the container layer. Without a volume, docker rm deletes your database. Mount a named volume for anything you want to keep.
  • localhost inside a container. It is the container itself. Use the other container’s name on a shared network.
  • No .dockerignore. Without it, .git, local data dumps and virtualenvs are sent to the builder and may end up in the image.
  • Licensing surprise. Docker Engine and CLI are open source, but Docker Desktop requires a paid subscription for larger companies. Check the current terms, or use Podman Desktop, Rancher Desktop, Colima or OrbStack.

Learning Resources

Official

Books

Courses and tutorials