DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ ip route | grep default # default via 192.168.1.1 dev wlan0 proto dhcp
# DHCP **DHCP** (Dynamic Host Configuration Protocol) is how a machine that has just joined a network gets its settings: an **IP address** and subnet mask, the **default gateway**, and the **DNS servers** to use. A DHCP server, usually your router at home or the cloud provider's VPC, hands out addresses from a pool and **leases** each one for a limited time. The exchange takes four messages, remembered as **DORA**: Discover, Offer, Request, Acknowledge. ## DORA, step by step {{< fig kicker="$ sudo dhclient -v wlan0" title="Getting an address: Discover, Offer, Request, Acknowledge" >}} LAPTOP MAC 3c:22:fb:7a:10:9e DHCP SERVER router 192.168.1.1 no IP: 0.0.0.0 pool .10 – .200 DISCOVER 0.0.0.0 → 255.255.255.255 broadcast: is there a DHCP server here? OFFER 192.168.1.42/24 gateway .1 · DNS .1 · lease 24 h REQUEST 192.168.1.42 broadcast: I'll take that one, please ACK lease starts it's yours for 24 hours 192.168.1.42/24 .42 → 3c:22:…:9e 12 h later(T1 = 50%) REQUEST renew .42 · unicast ACK lease extended, 24 h again
NO.STEPFROMTOTYPEINFO
1010.0.0.0:68255.255.255.255:67DISCOVERUDP broadcast · client MAC 3c:22:fb:7a:10:9e
202192.168.1.1:67client:68OFFERyiaddr 192.168.1.42 · mask /24 · router .1 · DNS .1 · lease 86400 s
3030.0.0.0:68255.255.255.255:67REQUESTrequested IP 192.168.1.42 · server id 192.168.1.1
404192.168.1.1:67client:68ACKlease confirmed · client configures its interface
505192.168.1.42:68192.168.1.1:67REQUESTrenewal, sent directly to the server that granted the lease
605192.168.1.1:67192.168.1.42:68ACKlease timer restarts at 24 h
  1. A laptop joins the Wi-Fi. It has a MAC address but no IP, and it doesn't know where the DHCP server is.
  2. Discover: it broadcasts to everyone on the LAN over UDP, from 0.0.0.0 port 68 to 255.255.255.255 port 67.
  3. Offer: the server picks a free address from its pool and offers it with the mask, gateway, DNS servers and lease time.
  4. Request: the laptop asks for that address. Still a broadcast, so any other DHCP server knows its offer was declined.
  5. Acknowledge: the server records the lease against the laptop's MAC, and the laptop configures its interface.
  6. Halfway through the lease, the laptop renews directly with the server. Same IP, fresh 24 hours. If it never renews, the address returns to the pool.
{{< /fig >}} ## What a lease gives you | Setting | Example | Check it with | |---|---|---| | IP address and prefix | `192.168.1.42/24` | `ip -br addr` | | Default gateway | `192.168.1.1` | `ip route \| grep default` | | DNS servers | `192.168.1.1` | `resolvectl status` | | Search domain | `corp.example.com` | `cat /etc/resolv.conf` | | Lease time | 24 hours | `nmcli device show wlan0`, or the DHCP client's lease file | Leases are why a laptop or a cloud VM can come back with a different IP after a long shutdown, and why two machines never get the same address on a working network. If no server answers at all, the machine gives itself a link-local address from `169.254.0.0/16` and you have "connected, but no internet". ## Static IPs and reservations Clients are fine with any address. Servers are not: a database, a Kafka broker or a printer must stay at a known address. Two ways to get that: - **Static configuration**: set the IP, gateway and DNS by hand on the machine, using an address outside the DHCP pool so nothing else is given it. - **DHCP reservation** (static lease): tell the DHCP server "MAC `3c:22:fb:7a:10:9e` always gets `.42`". The machine still uses DHCP, so gateway and DNS changes still reach it, but its address never changes. Usually the better option on networks you manage. In practice you put a **DNS name** in front of either, so clients never hard-code the IP. ## DHCP in the cloud Cloud VMs use DHCP too, but you rarely see it. In AWS, an instance's private IP is fixed for its lifetime, and the VPC's DHCP service hands it out at boot. What you configure is the **DHCP option set**: which DNS servers and domain name instances receive. This matters when a pipeline must resolve hostnames in an on-prem domain (for example `db.corp.example.com`) through a hybrid DNS resolver. Azure and GCP work similarly; for a fixed public address you allocate a static or elastic IP separately. ## Common mistakes - **Giving a server a dynamic address** and pointing pipelines at its IP. After the next reboot or lease expiry the job connects to the wrong machine, or to nothing. - **Picking a static IP inside the DHCP pool.** Sooner or later the server hands the same address to someone else and both machines break intermittently. - **Two DHCP servers on one LAN** (a forgotten home router plugged into the office switch). Clients get random gateways; run one server per subnet, or relays. - **Forgetting DHCP doesn't cross routers.** Each subnet needs its own server or a **DHCP relay** on the router that forwards the broadcasts. ## Try it ```bash ip -br addr # the address the lease gave you ip route | grep default # "proto dhcp" means it came from DHCP resolvectl status | grep "DNS Server" # DNS servers from the lease (systemd) nmcli device show wlan0 | grep -i dhcp # lease details (NetworkManager) ipconfig getpacket en0 # macOS: the full DHCP reply sudo tcpdump -n -i any port 67 or port 68 # watch DORA while you reconnect ``` ## Learning Resources - [RFC 2131: Dynamic Host Configuration Protocol](https://www.rfc-editor.org/rfc/rfc2131): the specification - [DHCP on Wikipedia](https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol): clear diagrams of every message - [AWS: DHCP option sets in Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html) See also: [IP addressing]({{< relref "ip-addressing" >}}), [DNS]({{< relref "/docs/roadmap/networking/protocols/dns" >}}).