DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ ip -br addr # wlan0 UP 192.168.1.42/24
# IP addressing An **IP address** says where a machine sits on a network, the way a street address says where a house is. An IPv4 address is 32 bits, written as four numbers from 0 to 255 (`192.168.1.42`). The **prefix length** after the slash (`/24`) says how many of those bits name the **network**; the rest name the **host** inside it. That one number decides which machines you can reach directly and which traffic must go through a router. ## Reading 192.168.1.42/24 {{< fig kicker="$ ip route get 203.0.113.10" title="Network bits, host bits, and where a packet goes" >}} 192.168.1.42/24 19211000000 16810101000 100000001 4200101010 network part · first 24 bits · same for the whole LAN host part · last 8 bits network 192.168.1.0/24 mask 255.255.255.0 256 addrs · hosts .1 – .254 LAPTOP192.168.1.42 NAS192.168.1.77 GATEWAY192.168.1.1 to .77: same network → direct INTERNET203.0.113.10 to 203.0.113.10: other network → default gateway
  1. An IPv4 address with a prefix length, as ip addr prints it.
  2. Underneath, it is 32 bits: four 8-bit numbers (octets). 42 is 00101010 in binary.
  3. /24 means the first 24 bits are the network. Every machine on this LAN shares them. The last 8 bits pick the host.
  4. So the network is 192.168.1.0/24, mask 255.255.255.0: 256 addresses. .0 names the network and .255 is broadcast, leaving .1 to .254 for hosts.
  5. Sending to 192.168.1.77? Same network bits, so the laptop finds its MAC with ARP and sends the frame directly.
  6. Sending to 203.0.113.10? Different network, so the packet goes to the default gateway, the router that knows the way out.
{{< /fig >}} The rule your OS applies to every packet: compare the destination's network bits with your own. Same network, deliver directly; different network, hand it to the gateway. `ip route` shows that table, and `ip route get ` tells you which way one destination will go. ## CIDR sizes **CIDR** (Classless Inter-Domain Routing) is the `address/prefix` notation. Each bit you add to the prefix halves the block. | CIDR | Mask | Addresses | Typical use | |---|---|---|---| | `/32` | 255.255.255.255 | 1 | One host, e.g. in an allow-list or firewall rule | | `/28` | 255.255.255.240 | 16 | A tiny subnet (AWS's minimum subnet size) | | `/24` | 255.255.255.0 | 256 | Home or office LAN, one cloud subnet | | `/16` | 255.255.0.0 | 65,536 | A whole cloud VPC | | `/8` | 255.0.0.0 | 16.7 million | The `10.0.0.0/8` private range | | `/0` | 0.0.0.0 | everything | "Any address", as in `0.0.0.0/0` | Cloud providers reserve a few addresses in every subnet (AWS keeps 5), so a `/24` gives you 251 usable IPs, not 256. ## Private ranges, NAT and IPv6 RFC 1918 sets aside three ranges that are never routed on the public internet. Anyone can use them inside their own network: | Range | Size | Where you see it | |---|---|---| | `10.0.0.0/8` | 16.7 M | Corporate networks, cloud VPCs | | `172.16.0.0/12` | 1 M | Docker's default bridge networks (`172.17.0.0/16`) | | `192.168.0.0/16` | 65 k | Home routers | Other special addresses: `127.0.0.1` is **loopback** (this machine), `0.0.0.0` in a listen address means "all interfaces", and `169.254.0.0/16` is link-local (a machine that got no DHCP answer, and the cloud metadata service at `169.254.169.254`). Because private addresses can't cross the internet, the router does **NAT** (network address translation): it rewrites the source address of outgoing packets to its own public IP and remembers the mapping so replies find their way back. In the cloud, a **NAT gateway** does the same for private subnets. Side effect: everything behind one NAT appears to the outside world as a single IP, which is the address a SaaS vendor has to allow-list. **IPv6** addresses are 128 bits, written in hex (`2001:db8::42`), with so many addresses that NAT is unnecessary. The same prefix idea applies: a typical subnet is a `/64`. ## Ports An IP address gets a packet to a machine; a **port** (0 to 65535) gets it to one program on it. Servers listen on well-known ports; clients use a random high **ephemeral** port for their side. | Port | Service | Port | Service | |---|---|---|---| | 22 | SSH | 5432 | PostgreSQL | | 53 | DNS | 3306 | MySQL | | 80 / 443 | HTTP / HTTPS | 6379 | Redis | | 9092 | Kafka | 8080 | Airflow web UI, many dev servers | A server listening on `127.0.0.1:5432` accepts connections only from the same machine; on `0.0.0.0:5432` it accepts them on every interface. That one setting is behind many "works locally, refused from the container" bugs. ## Why it matters for data engineers - **VPC and subnet design.** You pick the VPC CIDR once (say `10.20.0.0/16`) and carve subnets from it. Too small and a busy Spark or EKS cluster runs out of IPs; overlapping with another VPC or the office network and you can never peer or VPN them together. - **Kubernetes pod and service CIDRs.** Pods get IPs from their own range. If it overlaps your VPC or the database's network, traffic to the database is routed back into the cluster and silently fails. See [Kubernetes]({{< relref "/docs/roadmap/container-orchestration/kubernetes" >}}). - **Security groups and firewalls** are rules over CIDRs and ports: allow `10.20.0.0/16` on 5432, not `0.0.0.0/0`. See [Firewalls]({{< relref "firewalls" >}}). - **Allow-listing.** Snowflake network policies, managed databases and SaaS APIs accept connections only from listed IPs or ranges. Give them the stable public IP of your NAT gateway, not a worker's address that changes on every restart. ## Common mistakes - **Opening `0.0.0.0/0` "just to test".** It is the whole internet, and it tends to stay that way. - **Overlapping CIDRs.** Plan ranges across VPCs, on-prem networks, Docker and Kubernetes before you build, because renumbering later means rebuilding. - **Hard-coding IPs in pipelines.** Managed databases fail over and pods restart with new addresses. Use DNS names. - **Confusing the subnet with the address.** `192.168.1.42/24` is a host; `192.168.1.0/24` is the network it belongs to. ## Try it ```bash ip -br addr # your IPs and prefix lengths ip route # the routing table and default gateway ip route get 1.1.1.1 # which interface and gateway this IP uses ip neigh # ARP cache: IP → MAC of machines on your LAN curl -s https://ifconfig.me; echo # your public IP, as the internet sees it (after NAT) python3 -c "import ipaddress as i; n = i.ip_interface('192.168.1.42/24').network; print(n, n.netmask, n.num_addresses)" ``` ## Learning Resources - [RFC 1918: Address Allocation for Private Internets](https://www.rfc-editor.org/rfc/rfc1918) and [RFC 4632: CIDR](https://www.rfc-editor.org/rfc/rfc4632) - [cidr.xyz](https://cidr.xyz/): an interactive CIDR calculator that shows the bits - [AWS: IP addressing for your VPCs and subnets](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html) - [Python `ipaddress` module](https://docs.python.org/3/library/ipaddress.html): parse and check addresses and networks in code See also: [DHCP]({{< relref "dhcp" >}}), [TCP/IP]({{< relref "/docs/roadmap/networking/protocols/tcp" >}}), [VPN]({{< relref "vpn" >}}).