DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/
$ dig www.example.com # RFC 1034 / 1035, 1987
# DNS The **Domain Name System** is the internet's phone book: it turns a name like `www.example.com` into an IP address like `203.0.113.10`. It is not one database but a **distributed hierarchy**: the root servers know who runs `.com`, the `.com` servers know who runs `example.com`, and that domain's own servers hold its records. Answers are cached everywhere along the way for a time set by the record's **TTL**. ## Following one lookup {{< fig kicker="$ dig +trace www.example.com" title="Resolving a hostname from an empty cache" >}} LAPTOP stub resolver RESOLVER 1.1.1.1 ROOT . TLD .com AUTHORITATIVE ns1.example.com OS cache: miss A? www.example.com A? www.example.com referral → .com A? www.example.com referral → ns1 A? www.example.com A 203.0.113.10 TTL 3600 A 203.0.113.10 cached for 3600 s
NO.STEPFROMTOTYPEINFO
101laptopresolverQUERYA? www.example.com (UDP 53, recursion desired)
202resolverrootQUERYA? www.example.com
302rootresolverREFERRALNS a.gtld-servers.net (+ glue IPs)
403resolvertldQUERYA? www.example.com
503tldresolverREFERRALNS ns1.example.com, ns2.example.com
604resolverauthQUERYA? www.example.com
704authresolverANSWERA 203.0.113.10, TTL 3600, AA flag set
805resolverlaptopANSWERA 203.0.113.10 (now cached)
  1. Your laptop needs the IP of www.example.com. It only knows one server: its recursive resolver.
  2. The OS cache is empty, so the stub resolver sends one query to the recursive resolver and waits.
  3. The resolver starts at a root server. Root doesn't know the answer, it returns a referral to the .com servers.
  4. The .com TLD server doesn't know either. It refers the resolver to the domain's authoritative name servers.
  5. The authoritative server owns the zone and returns the real answer: an A record with a TTL.
  6. The resolver answers the laptop and caches the record. For the next hour, the same question costs one hop.
{{< /fig >}} Four roles take part: - **Stub resolver**: the tiny client inside your OS. It asks one question and expects a final answer. - **Recursive resolver**: your ISP's, your cloud VPC's, or a public one such as 1.1.1.1 or 8.8.8.8. It does the legwork and caches results. - **Root and TLD servers**: they never know the final answer, only who to ask next (a **referral**). - **Authoritative server**: the source of truth for a zone, run by your DNS provider (Route 53, Cloudflare, Cloud DNS). ## Record types you will meet | Type | Maps | Example | |---|---|---| | A / AAAA | name → IPv4 / IPv6 | `db.shop.com A 10.0.1.20` | | CNAME | name → another name | `warehouse.shop.com CNAME abc123.snowflakecomputing.com` | | MX | domain → mail servers | `shop.com MX 10 mail.shop.com` | | TXT | name → text | domain verification, SPF, DKIM | | NS | zone → its name servers | `shop.com NS ns1.provider.net` | | SRV | service → host + port | service discovery, e.g. Kafka or LDAP | ## DNS in data platforms - **Service discovery.** Kubernetes gives every Service a name such as `kafka.data.svc.cluster.local`. Kafka clients bootstrap through DNS and then connect to each broker's advertised hostname. - **Private DNS.** Managed databases and warehouses expose private endpoints that resolve only inside your VPC. "Works on my laptop, fails in the job" is often a resolver difference. - **Failover.** Managed databases fail over by repointing a DNS name. A client that caches the old IP forever keeps talking to a dead primary. ## Common mistakes - **Ignoring TTL when migrating.** Lower the TTL (say to 60 s) a day before you change a record, or clients keep the old IP for hours. - **Clients that cache forever.** The JVM's DNS cache is configurable via `networkaddress.cache.ttl`; long-lived pools must re-resolve to notice a failover. - **Assuming DNS is private.** Classic DNS over UDP port 53 is plain text. **DNS over HTTPS** (DoH) and **DNS over TLS** (DoT) encrypt it. **DNSSEC** signs records so they can't be forged, but doesn't hide them. - **Blaming the network for `ENOTFOUND`.** Check with `dig` against a specific resolver first; the record may simply not exist in that view. ## Try it ```bash dig +short www.example.com # just the answer dig +trace www.example.com # walk root → TLD → authoritative yourself dig @8.8.8.8 db.shop.com AAAA # ask a specific resolver for a specific type resolvectl query db.shop.com # what the OS resolver (systemd) returns ``` ## Learning Resources ### Books - [DNS and BIND, 5th edition](https://www.oreilly.com/library/view/dns-and-bind/0596100574/): Cricket Liu, Paul Albitz ### Articles - [What is DNS?](https://www.cloudflare.com/learning/dns/what-is-dns/): Cloudflare Learning Center - [How DNS works (comic)](https://howdns.works/) - [Mess with DNS](https://messwithdns.net/) and [Julia Evans on DNS](https://jvns.ca/categories/dns/): experiment with real records ### Specifications - [RFC 1034](https://www.rfc-editor.org/rfc/rfc1034) and [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035): the original DNS specs - [RFC 8484: DNS over HTTPS](https://www.rfc-editor.org/rfc/rfc8484) ### Videos - [Everything You Need to Know About DNS](https://www.youtube.com/watch?v=27r4Bzuj5NQ) See also: [UDP]({{< relref "udp" >}}), [HTTPS]({{< relref "https" >}}).