DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ docker build -t etl:1.0 . && docker run etl:1.0
# Docker **Docker** is the toolset that made containers mainstream: you describe an environment in a **Dockerfile**, `docker build` turns it into an **image**, and `docker run` starts a **container** from that image. Images are pushed to a **registry** (Docker Hub, GHCR, ECR) so any machine can pull and run exactly the same bits. Under the hood, Docker Engine (`dockerd`) delegates to [containerd]({{< relref "../containerd" >}}) and `runc` to actually start containers. Docker itself is the friendly layer on top: the CLI, the builder (**BuildKit**, the default since Engine 23), networking, volumes and Compose. ## What happens when you type `docker run` {{< fig kicker="$ docker run -d --name pg -e POSTGRES_PASSWORD=secret postgres:18" title="Client, daemon, registry: one docker run" >}} CLIENT $ docker run postgres:18 your terminal: only sends requests DOCKER HOST · LINUX dockerd images · networks · volumes REGISTRY docker.io GHCR · ECR · Quay REST API over /var/run/docker.sock image not local: pull missing layers containerd unpacks, supervises runc namespaces + cgroups, exits pg · postgres:18 · Up macOS / Windows: this box is a hidden Linux VM ports are forwarded to localhost · on Linux, no VM
  1. Three parts: the client (the docker CLI), the daemon on the Docker host, and a registry that stores images.
  2. The CLI sends the request to dockerd over a REST API on a Unix socket. "Cannot connect to the Docker daemon" means nobody is listening there.
  3. The daemon has no postgres:18 locally, so it pulls the missing layers from Docker Hub.
  4. containerd takes over the container's lifecycle: it unpacks the image and supervises the process.
  5. runc asks the kernel for namespaces and cgroups, starts the process inside them, and exits.
  6. The container is running: a fenced-off postgres process on the host's kernel.
  7. On macOS and Windows, everything inside the dashed box runs in a hidden Linux VM (Docker Desktop, Podman machine, OrbStack, Colima, WSL 2). Only the CLI runs natively.
{{< /fig >}} The hidden VM explains several laptop surprises: published ports are forwarded from the VM to your `localhost`, bind mounts cross the VM boundary and are slower for big folders, the VM has its own CPU and memory limit (check the Docker Desktop settings), and Apple Silicon wants `arm64` images while `amd64`-only images run emulated. ## Images are stacks of layers Each instruction in a Dockerfile that changes the filesystem produces a **layer**: a read-only tarball of changes, identified by its content hash. An image is an ordered list of layers. A container adds one thin **writable layer** on top. {{< fig kicker="$ docker build -t etl:1.0 ." title="From Dockerfile to image to running containers" >}} Dockerfile 1 FROM python:3.12-slim 2 COPY requirements.txt . 3 RUN pip install -r requirements.txt 4 COPY etl.py . IMAGE etl:1.0 (READ-ONLY) L1 python:3.12-slim~45 MB L2 requirements.txt1 KB L3 pandas, pyarrow…~180 MB L4 etl.py6 KB CONTAINER Arw layer CONTAINER Brw layer both share L1 to L4 on disk edit etl.py, rebuild: L1 to L3: CACHED L4: REBUILT (6 KB) push: only L4 uploads
  1. A four-line Dockerfile for a small Python ETL job. Nothing is built yet.
  2. FROM pulls the base image. Its layers become the bottom of our stack.
  3. COPY and RUN each add a layer. The dependency layer is the big one.
  4. COPY etl.py adds the last layer. Together, L1 to L4 are the read-only image etl:1.0.
  5. docker run starts a container: the image plus a thin writable layer. Files the job writes go there and vanish with the container.
  6. A second container from the same image shares every image layer. Only its own writable layer is new, so starting it is near-instant.
  7. Change only etl.py and rebuild: layers above the change are rebuilt, the rest come from the build cache. That is why dependency installs go before code copies.
{{< /fig >}} ## Reading an image name A full image reference has up to five parts. `postgres` on its own is shorthand for `docker.io/library/postgres:latest`. | Part | In `docker.io/library/postgres:18-alpine@sha256:…` | Meaning | |---|---|---| | Registry | `docker.io` | Where the image lives. Default is Docker Hub; others: `ghcr.io`, `quay.io`, `.dkr.ecr..amazonaws.com` | | Namespace | `library` | The owner. `library` is Docker Hub's official images; otherwise a user or org | | Repository | `postgres` | The image name | | Tag | `18-alpine` | A human label for one version. Mutable: it can be moved to new bytes. Default `latest` means "default", not "newest" | | Digest | `@sha256:…` | The hash of the image manifest. Immutable: always the exact same bytes | Pin tags (`postgres:18`) everywhere and digests in production. `docker images --digests` shows the digest of what you pulled. ## A practical Dockerfile for a data job ```dockerfile # Stage 1: build wheels in the full image, which has compilers FROM python:3.12 AS build WORKDIR /app COPY requirements.txt . RUN pip wheel --wheel-dir /wheels -r requirements.txt # Stage 2: small runtime image, no compilers FROM python:3.12-slim RUN useradd --create-home etl WORKDIR /app COPY --from=build /wheels /wheels RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels COPY etl.py . USER etl ENTRYPOINT ["python", "etl.py"] ``` Two habits matter most: put rarely-changing steps (dependencies) **before** often-changing ones (code) so the cache works, and use a **multi-stage** build so build tools never reach the final image. The cache rule in one line: when a layer changes, it **and every layer after it** rebuild. Compare: ```dockerfile # Slow: any code edit invalidates COPY, so pip reinstalls everything COPY . . RUN pip install -r requirements.txt # Fast: dependencies stay cached until requirements.txt changes COPY requirements.txt . RUN pip install -r requirements.txt COPY . . ``` `docker build -t myapp:1.0 .` builds it; the trailing `.` is the **build context**, the folder sent to the builder (filtered by `.dockerignore`). ## Anatomy of `docker run` ```bash docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5433:5432 -v pgdata:/var/lib/postgresql postgres:18 ``` | Flag | Meaning | |---|---| | `-d` | Detached: run in the background and print the container ID | | `--name pg` | A name to type instead of the random ID | | `-e KEY=value` | Environment variable: configuration and passwords | | `-p 5433:5432` | Publish a port: **host** 5433 → **container** 5432 | | `-v pgdata:/var/lib/postgresql` | Mount the named volume `pgdata` at that path (created if missing) | | `--network labnet` | Attach to a user-defined network, where containers find each other by name | | `--rm` | Delete the container when it exits (good for one-off jobs) | | `-it` | Interactive terminal, for shells and `psql` | | `postgres:18` | The image. Everything after it is the command and its arguments, not Docker flags | ## Ports, volumes and networks {{< fig kicker="$ docker rm -f pg && docker run ... -v pgdata:/var/lib/postgresql postgres:18" title="The container dies, the volume survives" >}} YOUR LAPTOP psql localhost:5432 DOCKER HOST pg postgres on :5432 -p 5432:5432 INSERT 2 rows -v pgdata:/var/lib/postgresql volume pgdata users: Alice, Sam docker rm -f pg container + writable layer gone pg (new) same -v pgdata SELECT * FROM users; 2 rows: the data survived
  1. A Postgres container is running, but nothing on your laptop can reach it yet: containers are isolated by default.
  2. -p 5432:5432 opens a door: traffic to localhost:5432 on the host is forwarded to port 5432 inside the container.
  3. Rows you insert are written to /var/lib/postgresql, which is the named volume pgdata, not the container's own layer.
  4. docker rm -f pg destroys the container and its writable layer. The volume is not touched.
  5. Run the same command again: a brand-new container mounts the same pgdata volume.
  6. The rows are still there. Containers are disposable; volumes hold the state.
{{< /fig >}} **Where data should live:** | Mount | Example | Best for | Survives `docker rm`? | |---|---|---|---| | Named volume | `-v pgdata:/var/lib/postgresql` | Databases. Docker manages where it lives | Yes | | Bind mount | `-v "$(pwd)/src":/app` | Live-editing code, passing input files | Yes, it is your folder | | tmpfs | `--tmpfs /tmp` | Scratch space, secrets that must not touch disk | No, it lives in RAM | | Nothing | | | No: writes go to the container layer | Manage volumes with `docker volume ls`, `docker volume inspect pgdata`, `docker volume rm pgdata` and `docker volume prune` (unused volumes only). **Postgres 18 changed the mount path.** For `postgres:18` and newer, mount the volume at `/var/lib/postgresql`; for `postgres:17` and older it is `/var/lib/postgresql/data`. Most tutorials still show the old path; on 18 it either fails to start or leaves your data outside the volume. **Networks.** On a user-defined network (`docker network create labnet`), containers reach each other **by name**: a web app connects to `db:5432`. The default bridge network has no name resolution. Inside a container, `localhost` means **that container itself**, not your laptop and not the database container. ## Docker Compose for local stacks `docker compose` (Compose v2, built into the CLI) starts several containers from one YAML file. It is the usual way to run a local data stack: Postgres as a source, Kafka, MinIO as fake S3, and Airflow, all on one private network where containers reach each other by service name. ```yaml services: postgres: image: postgres:17 environment: { POSTGRES_PASSWORD: dev } volumes: [pgdata:/var/lib/postgresql/data] minio: image: minio/minio command: server /data ports: ["9000:9000"] volumes: pgdata: ``` MinIO stopped publishing community Docker images in late 2025, so `minio/minio` no longer gets updates; it is fine for a throwaway local stack, but for anything longer-lived consider an S3-compatible alternative such as SeaweedFS or Garage. Compose is for development and single-host setups. For many hosts, use [Kubernetes]({{< relref "../../container-orchestration/kubernetes" >}}). ## Ten commands you'll use daily A container moves through `created → running → stopped → removed`: `run` creates and starts it, `stop` halts it with its files intact, `start` brings it back, `rm` deletes it. | Command | What it does | |---|---| | `docker run IMAGE` | Create and start a container | | `docker ps -a` | List containers, including stopped ones | | `docker logs -f NAME` | Follow its output | | `docker exec -it NAME sh` | Open a shell inside a running container (debug only) | | `docker stop NAME` | Stop it; its files stay | | `docker start NAME` | Start a stopped container again | | `docker rm -f NAME` | Delete it and its writable layer | | `docker inspect NAME` | Every detail (IP, mounts, env, exit code) as JSON | | `docker images` | List local images | | `docker rmi IMAGE` | Delete an image | And for building and shipping images: | Command | What it does | |---|---| | `docker build -t etl:1.0 .` | Build an image from the Dockerfile in `.` | | `docker image history etl:1.0` | Show the layers and their sizes | | `docker tag etl:1.0 ghcr.io/acme/etl:1.0` | Give the image a registry name | | `docker push ghcr.io/acme/etl:1.0` | Upload it to a registry | | `docker system df`, `docker system prune` | See and reclaim disk used by stopped containers, unused images and cache | ## Hands-on lab: Postgres and a web UI in Docker About an hour, no programming. You need Docker Desktop (macOS, Windows) or Docker Engine with the Compose plugin (Linux); `docker version` must show both a Client and a Server section. Pre-pull the images on a good connection: `docker pull postgres:18 adminer alpine`. **1. Run Postgres.** ```bash docker run -d --name pg \ -e POSTGRES_PASSWORD=secret \ -p 5432:5432 \ -v pgdata:/var/lib/postgresql \ postgres:18 docker ps # pg should show as Up docker logs pg # look for: database system is ready to accept connections ``` Port 5432 already taken by a local Postgres? Use `-p 5433:5432`. **2. Connect and query.** ```bash docker exec -it pg psql -U postgres ``` ```sql CREATE TABLE users (id serial PRIMARY KEY, name text NOT NULL, city text); INSERT INTO users (name, city) VALUES ('Alice', 'Berlin'), ('Sam', 'Lisbon'); SELECT * FROM users; \q ``` Any SQL client on your laptop can also connect to `localhost:5432` as `postgres` / `secret`. **3. Destroy the container, keep the data.** ```bash docker rm -f pg docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 \ -v pgdata:/var/lib/postgresql postgres:18 # same command as step 1 docker exec -it pg psql -U postgres -c "SELECT * FROM users;" ``` Alice and Sam are still there. Repeat with `-v pgdata2:/var/lib/postgresql` and the table is gone: a new volume is an empty database. **4. Build your own image with seed data.** In an empty folder, create `init.sql`: ```sql CREATE TABLE cities (id serial PRIMARY KEY, name text NOT NULL, country text); INSERT INTO cities (name, country) VALUES ('Berlin', 'Germany'), ('Lisbon', 'Portugal'); ``` and a `Dockerfile`: ```dockerfile FROM postgres:18 # *.sql and *.sh files here run on the first start of an empty database COPY init.sql /docker-entrypoint-initdb.d/ ``` ```bash docker build -t cities-db:1.0 . docker run -d --name cities -e POSTGRES_PASSWORD=secret -p 5433:5432 cities-db:1.0 docker exec -it cities psql -U postgres -c "SELECT * FROM cities;" ``` Init scripts run only when the data directory is empty, so they are skipped when you mount a volume that already holds a database. **5. Connect a web server to the database.** Two containers on a user-defined network, finding each other by name. [Adminer](https://hub.docker.com/_/adminer) is a small web UI for databases. ```bash docker rm -f pg cities # free port 5432 and the pgdata volume docker network create labnet docker run -d --name db --network labnet -e POSTGRES_PASSWORD=secret \ -v pgdata:/var/lib/postgresql postgres:18 docker run -d --name web --network labnet -p 8080:8080 adminer ``` Open and log in with System **PostgreSQL**, Server **db**, user `postgres`, password `secret`. The server is the container **name**, not an IP and not `localhost`. Note that `db` publishes no port: only containers on `labnet` can reach it. **6. The same with one Compose file.** Remove the two containers (`docker rm -f db web`), then save this as `compose.yaml`: ```yaml services: db: image: postgres:18 environment: POSTGRES_PASSWORD: secret volumes: - pgdata:/var/lib/postgresql web: image: adminer ports: - "8080:8080" depends_on: [db] volumes: pgdata: ``` ```bash docker compose up -d # creates a network; services reach each other by service name docker compose ps docker compose down # removes containers and network, keeps the volume docker compose down -v # also deletes the volume and the data ``` Clean up afterwards with `docker volume rm pgdata pgdata2` and `docker system prune`. ## Troubleshooting | You see | Check with | Fix | |---|---|---| | `Cannot connect to the Docker daemon` | `docker version` (no Server section) | Start Docker Desktop, or `sudo systemctl start docker` on Linux | | `port is already allocated` / `address already in use` | `docker ps`, `lsof -i :5432` | Stop the other process, or map a different host port: `-p 5433:5432` | | Postgres exits: `superuser password is not specified` | `docker logs pg` | Add `-e POSTGRES_PASSWORD=...` | | Postgres 18 fails with a mount or data-directory error | `docker inspect pg` (Mounts) | Mount at `/var/lib/postgresql`, not `/var/lib/postgresql/data` | | `init.sql` did not run | `docker volume ls` | The volume already held a database. Use a new volume, or `docker volume rm` it | | App container cannot reach `localhost:5432` | `docker network inspect labnet` | Put both on one user-defined network and connect to `db:5432` | | "It worked a minute ago" | `docker ps -a` | The container exited; read why with `docker logs NAME` | | Pull is slow or blocked | `docker pull` output | Use a mirror, or `docker save -o images.tar postgres:18` on one machine and `docker load -i images.tar` on another | | Anything else | `docker logs NAME` | Read the last lines first | If you remember three words: an **image** is the recipe (read-only layers), a **container** is a running instance (an isolated process), and a **volume** is the data that outlives the container. ## Common mistakes - **`latest` in production.** Tags are mutable. Pin a version, or better, the image digest (`@sha256:…`). - **Secrets in layers.** A password in a `COPY`ed file or `ENV` stays in the image history even if a later step deletes it. Use BuildKit secrets (`RUN --mount=type=secret`) and runtime env vars. - **Data in the container layer.** Without a volume, `docker rm` deletes your database. Mount a named volume for anything you want to keep. - **`localhost` inside a container.** It is the container itself. Use the other container's name on a shared network. - **No `.dockerignore`.** Without it, `.git`, local data dumps and virtualenvs are sent to the builder and may end up in the image. - **Licensing surprise.** Docker Engine and CLI are open source, but **Docker Desktop** requires a paid subscription for larger companies. Check the current terms, or use Podman Desktop, Rancher Desktop, Colima or OrbStack. ## Learning Resources ### Official - [Docker documentation and guides](https://docs.docker.com/guides/) - [Dockerfile reference](https://docs.docker.com/reference/dockerfile/) and [build best practices](https://docs.docker.com/build/building/best-practices/) - [Docker Compose documentation](https://docs.docker.com/compose/) ### Books - [Docker Deep Dive, Nigel Poulton](https://nigelpoulton.com/books/) - [Docker in Action, 2nd ed. (Manning)](https://www.manning.com/books/docker-in-action-second-edition) ### Courses and tutorials - [Docker Curriculum](https://docker-curriculum.com/) - [Complete Docker Course (YouTube)](https://www.youtube.com/watch?v=RqTEHSBrYFw) - [Docker Tutorial for Beginners (YouTube)](https://www.youtube.com/watch?v=3c-iBn73dDE)