# Docker
**Docker** is the toolset that made containers mainstream: you describe an environment in a **Dockerfile**, `docker build` turns it into an **image**, and `docker run` starts a **container** from that image. Images are pushed to a **registry** (Docker Hub, GHCR, ECR) so any machine can pull and run exactly the same bits.
Under the hood, Docker Engine (`dockerd`) delegates to [containerd]({{< relref "../containerd" >}}) and `runc` to actually start containers. Docker itself is the friendly layer on top: the CLI, the builder (**BuildKit**, the default since Engine 23), networking, volumes and Compose.
## What happens when you type `docker run`
{{< fig kicker="$ docker run -d --name pg -e POSTGRES_PASSWORD=secret postgres:18" title="Client, daemon, registry: one docker run" >}}
Three parts: the client (the docker CLI), the daemon on the Docker host, and a registry that stores images.
The CLI sends the request to dockerd over a REST API on a Unix socket. "Cannot connect to the Docker daemon" means nobody is listening there.
The daemon has no postgres:18 locally, so it pulls the missing layers from Docker Hub.
containerd takes over the container's lifecycle: it unpacks the image and supervises the process.
runc asks the kernel for namespaces and cgroups, starts the process inside them, and exits.
The container is running: a fenced-off postgres process on the host's kernel.
On macOS and Windows, everything inside the dashed box runs in a hidden Linux VM (Docker Desktop, Podman machine, OrbStack, Colima, WSL 2). Only the CLI runs natively.
{{< /fig >}}
The hidden VM explains several laptop surprises: published ports are forwarded from the VM to your `localhost`, bind mounts cross the VM boundary and are slower for big folders, the VM has its own CPU and memory limit (check the Docker Desktop settings), and Apple Silicon wants `arm64` images while `amd64`-only images run emulated.
## Images are stacks of layers
Each instruction in a Dockerfile that changes the filesystem produces a **layer**: a read-only tarball of changes, identified by its content hash. An image is an ordered list of layers. A container adds one thin **writable layer** on top.
{{< fig kicker="$ docker build -t etl:1.0 ." title="From Dockerfile to image to running containers" >}}
A four-line Dockerfile for a small Python ETL job. Nothing is built yet.
FROM pulls the base image. Its layers become the bottom of our stack.
COPY and RUN each add a layer. The dependency layer is the big one.
COPY etl.py adds the last layer. Together, L1 to L4 are the read-only image etl:1.0.
docker run starts a container: the image plus a thin writable layer. Files the job writes go there and vanish with the container.
A second container from the same image shares every image layer. Only its own writable layer is new, so starting it is near-instant.
Change only etl.py and rebuild: layers above the change are rebuilt, the rest come from the build cache. That is why dependency installs go before code copies.
{{< /fig >}}
## Reading an image name
A full image reference has up to five parts. `postgres` on its own is shorthand for `docker.io/library/postgres:latest`.
| Part | In `docker.io/library/postgres:18-alpine@sha256:…` | Meaning |
|---|---|---|
| Registry | `docker.io` | Where the image lives. Default is Docker Hub; others: `ghcr.io`, `quay.io`, `.dkr.ecr..amazonaws.com` |
| Namespace | `library` | The owner. `library` is Docker Hub's official images; otherwise a user or org |
| Repository | `postgres` | The image name |
| Tag | `18-alpine` | A human label for one version. Mutable: it can be moved to new bytes. Default `latest` means "default", not "newest" |
| Digest | `@sha256:…` | The hash of the image manifest. Immutable: always the exact same bytes |
Pin tags (`postgres:18`) everywhere and digests in production. `docker images --digests` shows the digest of what you pulled.
## A practical Dockerfile for a data job
```dockerfile
# Stage 1: build wheels in the full image, which has compilers
FROM python:3.12 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip wheel --wheel-dir /wheels -r requirements.txt
# Stage 2: small runtime image, no compilers
FROM python:3.12-slim
RUN useradd --create-home etl
WORKDIR /app
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels
COPY etl.py .
USER etl
ENTRYPOINT ["python", "etl.py"]
```
Two habits matter most: put rarely-changing steps (dependencies) **before** often-changing ones (code) so the cache works, and use a **multi-stage** build so build tools never reach the final image.
The cache rule in one line: when a layer changes, it **and every layer after it** rebuild. Compare:
```dockerfile
# Slow: any code edit invalidates COPY, so pip reinstalls everything
COPY . .
RUN pip install -r requirements.txt
# Fast: dependencies stay cached until requirements.txt changes
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
```
`docker build -t myapp:1.0 .` builds it; the trailing `.` is the **build context**, the folder sent to the builder (filtered by `.dockerignore`).
## Anatomy of `docker run`
```bash
docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5433:5432 -v pgdata:/var/lib/postgresql postgres:18
```
| Flag | Meaning |
|---|---|
| `-d` | Detached: run in the background and print the container ID |
| `--name pg` | A name to type instead of the random ID |
| `-e KEY=value` | Environment variable: configuration and passwords |
| `-p 5433:5432` | Publish a port: **host** 5433 → **container** 5432 |
| `-v pgdata:/var/lib/postgresql` | Mount the named volume `pgdata` at that path (created if missing) |
| `--network labnet` | Attach to a user-defined network, where containers find each other by name |
| `--rm` | Delete the container when it exits (good for one-off jobs) |
| `-it` | Interactive terminal, for shells and `psql` |
| `postgres:18` | The image. Everything after it is the command and its arguments, not Docker flags |
## Ports, volumes and networks
{{< fig kicker="$ docker rm -f pg && docker run ... -v pgdata:/var/lib/postgresql postgres:18" title="The container dies, the volume survives" >}}
A Postgres container is running, but nothing on your laptop can reach it yet: containers are isolated by default.
-p 5432:5432 opens a door: traffic to localhost:5432 on the host is forwarded to port 5432 inside the container.
Rows you insert are written to /var/lib/postgresql, which is the named volume pgdata, not the container's own layer.
docker rm -f pg destroys the container and its writable layer. The volume is not touched.
Run the same command again: a brand-new container mounts the same pgdata volume.
The rows are still there. Containers are disposable; volumes hold the state.
{{< /fig >}}
**Where data should live:**
| Mount | Example | Best for | Survives `docker rm`? |
|---|---|---|---|
| Named volume | `-v pgdata:/var/lib/postgresql` | Databases. Docker manages where it lives | Yes |
| Bind mount | `-v "$(pwd)/src":/app` | Live-editing code, passing input files | Yes, it is your folder |
| tmpfs | `--tmpfs /tmp` | Scratch space, secrets that must not touch disk | No, it lives in RAM |
| Nothing | | | No: writes go to the container layer |
Manage volumes with `docker volume ls`, `docker volume inspect pgdata`, `docker volume rm pgdata` and `docker volume prune` (unused volumes only).
**Postgres 18 changed the mount path.** For `postgres:18` and newer, mount the volume at `/var/lib/postgresql`; for `postgres:17` and older it is `/var/lib/postgresql/data`. Most tutorials still show the old path; on 18 it either fails to start or leaves your data outside the volume.
**Networks.** On a user-defined network (`docker network create labnet`), containers reach each other **by name**: a web app connects to `db:5432`. The default bridge network has no name resolution. Inside a container, `localhost` means **that container itself**, not your laptop and not the database container.
## Docker Compose for local stacks
`docker compose` (Compose v2, built into the CLI) starts several containers from one YAML file. It is the usual way to run a local data stack: Postgres as a source, Kafka, MinIO as fake S3, and Airflow, all on one private network where containers reach each other by service name.
```yaml
services:
postgres:
image: postgres:17
environment: { POSTGRES_PASSWORD: dev }
volumes: [pgdata:/var/lib/postgresql/data]
minio:
image: minio/minio
command: server /data
ports: ["9000:9000"]
volumes:
pgdata:
```
MinIO stopped publishing community Docker images in late 2025, so `minio/minio` no longer gets updates; it is fine for a throwaway local stack, but for anything longer-lived consider an S3-compatible alternative such as SeaweedFS or Garage.
Compose is for development and single-host setups. For many hosts, use [Kubernetes]({{< relref "../../container-orchestration/kubernetes" >}}).
## Ten commands you'll use daily
A container moves through `created → running → stopped → removed`: `run` creates and starts it, `stop` halts it with its files intact, `start` brings it back, `rm` deletes it.
| Command | What it does |
|---|---|
| `docker run IMAGE` | Create and start a container |
| `docker ps -a` | List containers, including stopped ones |
| `docker logs -f NAME` | Follow its output |
| `docker exec -it NAME sh` | Open a shell inside a running container (debug only) |
| `docker stop NAME` | Stop it; its files stay |
| `docker start NAME` | Start a stopped container again |
| `docker rm -f NAME` | Delete it and its writable layer |
| `docker inspect NAME` | Every detail (IP, mounts, env, exit code) as JSON |
| `docker images` | List local images |
| `docker rmi IMAGE` | Delete an image |
And for building and shipping images:
| Command | What it does |
|---|---|
| `docker build -t etl:1.0 .` | Build an image from the Dockerfile in `.` |
| `docker image history etl:1.0` | Show the layers and their sizes |
| `docker tag etl:1.0 ghcr.io/acme/etl:1.0` | Give the image a registry name |
| `docker push ghcr.io/acme/etl:1.0` | Upload it to a registry |
| `docker system df`, `docker system prune` | See and reclaim disk used by stopped containers, unused images and cache |
## Hands-on lab: Postgres and a web UI in Docker
About an hour, no programming. You need Docker Desktop (macOS, Windows) or Docker Engine with the Compose plugin (Linux); `docker version` must show both a Client and a Server section. Pre-pull the images on a good connection: `docker pull postgres:18 adminer alpine`.
**1. Run Postgres.**
```bash
docker run -d --name pg \
-e POSTGRES_PASSWORD=secret \
-p 5432:5432 \
-v pgdata:/var/lib/postgresql \
postgres:18
docker ps # pg should show as Up
docker logs pg # look for: database system is ready to accept connections
```
Port 5432 already taken by a local Postgres? Use `-p 5433:5432`.
**2. Connect and query.**
```bash
docker exec -it pg psql -U postgres
```
```sql
CREATE TABLE users (id serial PRIMARY KEY, name text NOT NULL, city text);
INSERT INTO users (name, city) VALUES ('Alice', 'Berlin'), ('Sam', 'Lisbon');
SELECT * FROM users;
\q
```
Any SQL client on your laptop can also connect to `localhost:5432` as `postgres` / `secret`.
**3. Destroy the container, keep the data.**
```bash
docker rm -f pg
docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 \
-v pgdata:/var/lib/postgresql postgres:18 # same command as step 1
docker exec -it pg psql -U postgres -c "SELECT * FROM users;"
```
Alice and Sam are still there. Repeat with `-v pgdata2:/var/lib/postgresql` and the table is gone: a new volume is an empty database.
**4. Build your own image with seed data.** In an empty folder, create `init.sql`:
```sql
CREATE TABLE cities (id serial PRIMARY KEY, name text NOT NULL, country text);
INSERT INTO cities (name, country) VALUES ('Berlin', 'Germany'), ('Lisbon', 'Portugal');
```
and a `Dockerfile`:
```dockerfile
FROM postgres:18
# *.sql and *.sh files here run on the first start of an empty database
COPY init.sql /docker-entrypoint-initdb.d/
```
```bash
docker build -t cities-db:1.0 .
docker run -d --name cities -e POSTGRES_PASSWORD=secret -p 5433:5432 cities-db:1.0
docker exec -it cities psql -U postgres -c "SELECT * FROM cities;"
```
Init scripts run only when the data directory is empty, so they are skipped when you mount a volume that already holds a database.
**5. Connect a web server to the database.** Two containers on a user-defined network, finding each other by name. [Adminer](https://hub.docker.com/_/adminer) is a small web UI for databases.
```bash
docker rm -f pg cities # free port 5432 and the pgdata volume
docker network create labnet
docker run -d --name db --network labnet -e POSTGRES_PASSWORD=secret \
-v pgdata:/var/lib/postgresql postgres:18
docker run -d --name web --network labnet -p 8080:8080 adminer
```
Open and log in with System **PostgreSQL**, Server **db**, user `postgres`, password `secret`. The server is the container **name**, not an IP and not `localhost`. Note that `db` publishes no port: only containers on `labnet` can reach it.
**6. The same with one Compose file.** Remove the two containers (`docker rm -f db web`), then save this as `compose.yaml`:
```yaml
services:
db:
image: postgres:18
environment:
POSTGRES_PASSWORD: secret
volumes:
- pgdata:/var/lib/postgresql
web:
image: adminer
ports:
- "8080:8080"
depends_on: [db]
volumes:
pgdata:
```
```bash
docker compose up -d # creates a network; services reach each other by service name
docker compose ps
docker compose down # removes containers and network, keeps the volume
docker compose down -v # also deletes the volume and the data
```
Clean up afterwards with `docker volume rm pgdata pgdata2` and `docker system prune`.
## Troubleshooting
| You see | Check with | Fix |
|---|---|---|
| `Cannot connect to the Docker daemon` | `docker version` (no Server section) | Start Docker Desktop, or `sudo systemctl start docker` on Linux |
| `port is already allocated` / `address already in use` | `docker ps`, `lsof -i :5432` | Stop the other process, or map a different host port: `-p 5433:5432` |
| Postgres exits: `superuser password is not specified` | `docker logs pg` | Add `-e POSTGRES_PASSWORD=...` |
| Postgres 18 fails with a mount or data-directory error | `docker inspect pg` (Mounts) | Mount at `/var/lib/postgresql`, not `/var/lib/postgresql/data` |
| `init.sql` did not run | `docker volume ls` | The volume already held a database. Use a new volume, or `docker volume rm` it |
| App container cannot reach `localhost:5432` | `docker network inspect labnet` | Put both on one user-defined network and connect to `db:5432` |
| "It worked a minute ago" | `docker ps -a` | The container exited; read why with `docker logs NAME` |
| Pull is slow or blocked | `docker pull` output | Use a mirror, or `docker save -o images.tar postgres:18` on one machine and `docker load -i images.tar` on another |
| Anything else | `docker logs NAME` | Read the last lines first |
If you remember three words: an **image** is the recipe (read-only layers), a **container** is a running instance (an isolated process), and a **volume** is the data that outlives the container.
## Common mistakes
- **`latest` in production.** Tags are mutable. Pin a version, or better, the image digest (`@sha256:…`).
- **Secrets in layers.** A password in a `COPY`ed file or `ENV` stays in the image history even if a later step deletes it. Use BuildKit secrets (`RUN --mount=type=secret`) and runtime env vars.
- **Data in the container layer.** Without a volume, `docker rm` deletes your database. Mount a named volume for anything you want to keep.
- **`localhost` inside a container.** It is the container itself. Use the other container's name on a shared network.
- **No `.dockerignore`.** Without it, `.git`, local data dumps and virtualenvs are sent to the builder and may end up in the image.
- **Licensing surprise.** Docker Engine and CLI are open source, but **Docker Desktop** requires a paid subscription for larger companies. Check the current terms, or use Podman Desktop, Rancher Desktop, Colima or OrbStack.
## Learning Resources
### Official
- [Docker documentation and guides](https://docs.docker.com/guides/)
- [Dockerfile reference](https://docs.docker.com/reference/dockerfile/) and [build best practices](https://docs.docker.com/build/building/best-practices/)
- [Docker Compose documentation](https://docs.docker.com/compose/)
### Books
- [Docker Deep Dive, Nigel Poulton](https://nigelpoulton.com/books/)
- [Docker in Action, 2nd ed. (Manning)](https://www.manning.com/books/docker-in-action-second-edition)
### Courses and tutorials
- [Docker Curriculum](https://docker-curriculum.com/)
- [Complete Docker Course (YouTube)](https://www.youtube.com/watch?v=RqTEHSBrYFw)
- [Docker Tutorial for Beginners (YouTube)](https://www.youtube.com/watch?v=3c-iBn73dDE)