Docker
Docker is the toolset that made containers mainstream: you describe an environment in a Dockerfile, docker build turns it into an image, and docker run starts a container from that image. Images are pushed to a registry (Docker Hub, GHCR, ECR) so any machine can pull and run exactly the same bits.
Under the hood, Docker Engine (dockerd) delegates to containerd and runc to actually start containers. Docker itself is the friendly layer on top: the CLI, the builder (BuildKit, the default since Engine 23), networking, volumes and Compose.
What happens when you type docker run
- Three parts: the client (the
dockerCLI), the daemon on the Docker host, and a registry that stores images. - The CLI sends the request to
dockerdover a REST API on a Unix socket. "Cannot connect to the Docker daemon" means nobody is listening there. - The daemon has no
postgres:18locally, so it pulls the missing layers from Docker Hub. - containerd takes over the container's lifecycle: it unpacks the image and supervises the process.
- runc asks the kernel for namespaces and cgroups, starts the process inside them, and exits.
- The container is running: a fenced-off postgres process on the host's kernel.
- On macOS and Windows, everything inside the dashed box runs in a hidden Linux VM (Docker Desktop, Podman machine, OrbStack, Colima, WSL 2). Only the CLI runs natively.
The hidden VM explains several laptop surprises: published ports are forwarded from the VM to your localhost, bind mounts cross the VM boundary and are slower for big folders, the VM has its own CPU and memory limit (check the Docker Desktop settings), and Apple Silicon wants arm64 images while amd64-only images run emulated.
Images are stacks of layers
Each instruction in a Dockerfile that changes the filesystem produces a layer: a read-only tarball of changes, identified by its content hash. An image is an ordered list of layers. A container adds one thin writable layer on top.
- A four-line Dockerfile for a small Python ETL job. Nothing is built yet.
- FROM pulls the base image. Its layers become the bottom of our stack.
- COPY and RUN each add a layer. The dependency layer is the big one.
- COPY etl.py adds the last layer. Together, L1 to L4 are the read-only image etl:1.0.
- docker run starts a container: the image plus a thin writable layer. Files the job writes go there and vanish with the container.
- A second container from the same image shares every image layer. Only its own writable layer is new, so starting it is near-instant.
- Change only etl.py and rebuild: layers above the change are rebuilt, the rest come from the build cache. That is why dependency installs go before code copies.
Reading an image name
A full image reference has up to five parts. postgres on its own is shorthand for docker.io/library/postgres:latest.
| Part | In docker.io/library/postgres:18-alpine@sha256:… |
Meaning |
|---|---|---|
| Registry | docker.io |
Where the image lives. Default is Docker Hub; others: ghcr.io, quay.io, <account>.dkr.ecr.<region>.amazonaws.com |
| Namespace | library |
The owner. library is Docker Hub’s official images; otherwise a user or org |
| Repository | postgres |
The image name |
| Tag | 18-alpine |
A human label for one version. Mutable: it can be moved to new bytes. Default latest means “default”, not “newest” |
| Digest | @sha256:… |
The hash of the image manifest. Immutable: always the exact same bytes |
Pin tags (postgres:18) everywhere and digests in production. docker images --digests shows the digest of what you pulled.
A practical Dockerfile for a data job
# Stage 1: build wheels in the full image, which has compilers
FROM python:3.12 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip wheel --wheel-dir /wheels -r requirements.txt
# Stage 2: small runtime image, no compilers
FROM python:3.12-slim
RUN useradd --create-home etl
WORKDIR /app
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels
COPY etl.py .
USER etl
ENTRYPOINT ["python", "etl.py"]Two habits matter most: put rarely-changing steps (dependencies) before often-changing ones (code) so the cache works, and use a multi-stage build so build tools never reach the final image.
The cache rule in one line: when a layer changes, it and every layer after it rebuild. Compare:
# Slow: any code edit invalidates COPY, so pip reinstalls everything
COPY . .
RUN pip install -r requirements.txt
# Fast: dependencies stay cached until requirements.txt changes
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .docker build -t myapp:1.0 . builds it; the trailing . is the build context, the folder sent to the builder (filtered by .dockerignore).
Anatomy of docker run
docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5433:5432 -v pgdata:/var/lib/postgresql postgres:18| Flag | Meaning |
|---|---|
-d |
Detached: run in the background and print the container ID |
--name pg |
A name to type instead of the random ID |
-e KEY=value |
Environment variable: configuration and passwords |
-p 5433:5432 |
Publish a port: host 5433 → container 5432 |
-v pgdata:/var/lib/postgresql |
Mount the named volume pgdata at that path (created if missing) |
--network labnet |
Attach to a user-defined network, where containers find each other by name |
--rm |
Delete the container when it exits (good for one-off jobs) |
-it |
Interactive terminal, for shells and psql |
postgres:18 |
The image. Everything after it is the command and its arguments, not Docker flags |
Ports, volumes and networks
- A Postgres container is running, but nothing on your laptop can reach it yet: containers are isolated by default.
-p 5432:5432opens a door: traffic to localhost:5432 on the host is forwarded to port 5432 inside the container.- Rows you insert are written to
/var/lib/postgresql, which is the named volume pgdata, not the container's own layer. docker rm -f pgdestroys the container and its writable layer. The volume is not touched.- Run the same command again: a brand-new container mounts the same pgdata volume.
- The rows are still there. Containers are disposable; volumes hold the state.
Where data should live:
| Mount | Example | Best for | Survives docker rm? |
|---|---|---|---|
| Named volume | -v pgdata:/var/lib/postgresql |
Databases. Docker manages where it lives | Yes |
| Bind mount | -v "$(pwd)/src":/app |
Live-editing code, passing input files | Yes, it is your folder |
| tmpfs | --tmpfs /tmp |
Scratch space, secrets that must not touch disk | No, it lives in RAM |
| Nothing | No: writes go to the container layer |
Manage volumes with docker volume ls, docker volume inspect pgdata, docker volume rm pgdata and docker volume prune (unused volumes only).
Postgres 18 changed the mount path. For postgres:18 and newer, mount the volume at /var/lib/postgresql; for postgres:17 and older it is /var/lib/postgresql/data. Most tutorials still show the old path; on 18 it either fails to start or leaves your data outside the volume.
Networks. On a user-defined network (docker network create labnet), containers reach each other by name: a web app connects to db:5432. The default bridge network has no name resolution. Inside a container, localhost means that container itself, not your laptop and not the database container.
Docker Compose for local stacks
docker compose (Compose v2, built into the CLI) starts several containers from one YAML file. It is the usual way to run a local data stack: Postgres as a source, Kafka, MinIO as fake S3, and Airflow, all on one private network where containers reach each other by service name.
services:
postgres:
image: postgres:17
environment: { POSTGRES_PASSWORD: dev }
volumes: [pgdata:/var/lib/postgresql/data]
minio:
image: minio/minio
command: server /data
ports: ["9000:9000"]
volumes:
pgdata:MinIO stopped publishing community Docker images in late 2025, so minio/minio no longer gets updates; it is fine for a throwaway local stack, but for anything longer-lived consider an S3-compatible alternative such as SeaweedFS or Garage.
Compose is for development and single-host setups. For many hosts, use Kubernetes.
Ten commands you’ll use daily
A container moves through created → running → stopped → removed: run creates and starts it, stop halts it with its files intact, start brings it back, rm deletes it.
| Command | What it does |
|---|---|
docker run IMAGE |
Create and start a container |
docker ps -a |
List containers, including stopped ones |
docker logs -f NAME |
Follow its output |
docker exec -it NAME sh |
Open a shell inside a running container (debug only) |
docker stop NAME |
Stop it; its files stay |
docker start NAME |
Start a stopped container again |
docker rm -f NAME |
Delete it and its writable layer |
docker inspect NAME |
Every detail (IP, mounts, env, exit code) as JSON |
docker images |
List local images |
docker rmi IMAGE |
Delete an image |
And for building and shipping images:
| Command | What it does |
|---|---|
docker build -t etl:1.0 . |
Build an image from the Dockerfile in . |
docker image history etl:1.0 |
Show the layers and their sizes |
docker tag etl:1.0 ghcr.io/acme/etl:1.0 |
Give the image a registry name |
docker push ghcr.io/acme/etl:1.0 |
Upload it to a registry |
docker system df, docker system prune |
See and reclaim disk used by stopped containers, unused images and cache |
Hands-on lab: Postgres and a web UI in Docker
About an hour, no programming. You need Docker Desktop (macOS, Windows) or Docker Engine with the Compose plugin (Linux); docker version must show both a Client and a Server section. Pre-pull the images on a good connection: docker pull postgres:18 adminer alpine.
1. Run Postgres.
docker run -d --name pg \
-e POSTGRES_PASSWORD=secret \
-p 5432:5432 \
-v pgdata:/var/lib/postgresql \
postgres:18
docker ps # pg should show as Up
docker logs pg # look for: database system is ready to accept connectionsPort 5432 already taken by a local Postgres? Use -p 5433:5432.
2. Connect and query.
docker exec -it pg psql -U postgresCREATE TABLE users (id serial PRIMARY KEY, name text NOT NULL, city text);
INSERT INTO users (name, city) VALUES ('Alice', 'Berlin'), ('Sam', 'Lisbon');
SELECT * FROM users;
\qAny SQL client on your laptop can also connect to localhost:5432 as postgres / secret.
3. Destroy the container, keep the data.
docker rm -f pg
docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 \
-v pgdata:/var/lib/postgresql postgres:18 # same command as step 1
docker exec -it pg psql -U postgres -c "SELECT * FROM users;"Alice and Sam are still there. Repeat with -v pgdata2:/var/lib/postgresql and the table is gone: a new volume is an empty database.
4. Build your own image with seed data. In an empty folder, create init.sql:
CREATE TABLE cities (id serial PRIMARY KEY, name text NOT NULL, country text);
INSERT INTO cities (name, country) VALUES ('Berlin', 'Germany'), ('Lisbon', 'Portugal');and a Dockerfile:
FROM postgres:18
# *.sql and *.sh files here run on the first start of an empty database
COPY init.sql /docker-entrypoint-initdb.d/docker build -t cities-db:1.0 .
docker run -d --name cities -e POSTGRES_PASSWORD=secret -p 5433:5432 cities-db:1.0
docker exec -it cities psql -U postgres -c "SELECT * FROM cities;"Init scripts run only when the data directory is empty, so they are skipped when you mount a volume that already holds a database.
5. Connect a web server to the database. Two containers on a user-defined network, finding each other by name. Adminer is a small web UI for databases.
docker rm -f pg cities # free port 5432 and the pgdata volume
docker network create labnet
docker run -d --name db --network labnet -e POSTGRES_PASSWORD=secret \
-v pgdata:/var/lib/postgresql postgres:18
docker run -d --name web --network labnet -p 8080:8080 adminerOpen http://localhost:8080 and log in with System PostgreSQL, Server db, user postgres, password secret. The server is the container name, not an IP and not localhost. Note that db publishes no port: only containers on labnet can reach it.
6. The same with one Compose file. Remove the two containers (docker rm -f db web), then save this as compose.yaml:
services:
db:
image: postgres:18
environment:
POSTGRES_PASSWORD: secret
volumes:
- pgdata:/var/lib/postgresql
web:
image: adminer
ports:
- "8080:8080"
depends_on: [db]
volumes:
pgdata:docker compose up -d # creates a network; services reach each other by service name
docker compose ps
docker compose down # removes containers and network, keeps the volume
docker compose down -v # also deletes the volume and the dataClean up afterwards with docker volume rm pgdata pgdata2 and docker system prune.
Troubleshooting
| You see | Check with | Fix |
|---|---|---|
Cannot connect to the Docker daemon |
docker version (no Server section) |
Start Docker Desktop, or sudo systemctl start docker on Linux |
port is already allocated / address already in use |
docker ps, lsof -i :5432 |
Stop the other process, or map a different host port: -p 5433:5432 |
Postgres exits: superuser password is not specified |
docker logs pg |
Add -e POSTGRES_PASSWORD=... |
| Postgres 18 fails with a mount or data-directory error | docker inspect pg (Mounts) |
Mount at /var/lib/postgresql, not /var/lib/postgresql/data |
init.sql did not run |
docker volume ls |
The volume already held a database. Use a new volume, or docker volume rm it |
App container cannot reach localhost:5432 |
docker network inspect labnet |
Put both on one user-defined network and connect to db:5432 |
| “It worked a minute ago” | docker ps -a |
The container exited; read why with docker logs NAME |
| Pull is slow or blocked | docker pull output |
Use a mirror, or docker save -o images.tar postgres:18 on one machine and docker load -i images.tar on another |
| Anything else | docker logs NAME |
Read the last lines first |
If you remember three words: an image is the recipe (read-only layers), a container is a running instance (an isolated process), and a volume is the data that outlives the container.
Common mistakes
latestin production. Tags are mutable. Pin a version, or better, the image digest (@sha256:…).- Secrets in layers. A password in a
COPYed file orENVstays in the image history even if a later step deletes it. Use BuildKit secrets (RUN --mount=type=secret) and runtime env vars. - Data in the container layer. Without a volume,
docker rmdeletes your database. Mount a named volume for anything you want to keep. localhostinside a container. It is the container itself. Use the other container’s name on a shared network.- No
.dockerignore. Without it,.git, local data dumps and virtualenvs are sent to the builder and may end up in the image. - Licensing surprise. Docker Engine and CLI are open source, but Docker Desktop requires a paid subscription for larger companies. Check the current terms, or use Podman Desktop, Rancher Desktop, Colima or OrbStack.
Learning Resources
Official
- Docker documentation and guides
- Dockerfile reference and build best practices
- Docker Compose documentation