DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ man 7 tcp # RFC 791 (IP) ยท RFC 9293 (TCP)
# TCP/IP **TCP/IP** is the protocol suite the internet runs on. **IP** (Internet Protocol) moves individual packets from one machine to another by address, with no promises: packets can be lost, duplicated or reordered. **TCP** (Transmission Control Protocol) runs on top and turns that unreliable packet service into a **reliable, ordered byte stream** between two programs. Your database driver, HTTP client and Kafka producer all sit on a TCP connection. ## The layers | Layer | Example | Unit | Addresses with | |---|---|---|---| | Application | HTTP, PostgreSQL wire protocol, Kafka | message | URLs, topic names | | Transport | TCP, UDP | segment / datagram | **port** (5432, 443, 9092) | | Internet | IPv4, IPv6 | packet | **IP address** (10.0.1.20, 2001:db8::1) | | Link | Ethernet, Wi-Fi | frame | MAC address | Each layer wraps the one above in its own header. A connection is identified by five values: protocol, source IP, source port, destination IP, destination port. That is how one server can hold thousands of connections on port 5432 at once. The OSI model has seven layers. TCP/IP merges OSI's top three into "application", which matches how real software is built. ## A TCP connection, step by step {{< fig kicker="$ tcpdump -n 'tcp port 5432'" title="TCP connection: handshake, data, teardown" >}} CLIENT 10.0.0.5:52114 SERVER 10.0.1.20:5432 SYN_SENT SYN seq=1000 SYN-ACK seq=5000 ack=1001 ACK ack=5001 ESTABLISHED ESTABLISHED data seq=1001 len=100 ACK ack=1101 FIN seq=1101 ACK ack=1102 FIN seq=5001 ACK ack=5002 TIME_WAIT (2 x MSL)
NO.STEPFROMTOTYPEINFO
101clientserverSYNseq=1000, win=64240, MSS=1460
202serverclientSYN-ACKseq=5000, ack=1001
303clientserverACKack=5001 โ†’ connection established
404clientserverPSH,ACKseq=1001, len=100 (query bytes)
504serverclientACKack=1101 (got bytes 1001โ€“1100)
605clientserverFIN,ACKseq=1101, ack=5001
705serverclientACKack=1102
805serverclientFIN,ACKseq=5001, ack=1102
905clientserverACKack=5002 โ†’ client in TIME_WAIT
  1. A client (say, a Python app) wants to talk to PostgreSQL on port 5432. No connection exists yet.
  2. The client sends SYN with its initial sequence number (1000 here for readability; real ones are random 32-bit values).
  3. The server replies SYN-ACK: its own sequence number, and an acknowledgment of the client's (1000 + 1).
  4. The client sends ACK. Both sides are ESTABLISHED. That cost one full round trip before any data.
  5. Data flows. Every byte is numbered, and the receiver acknowledges what it got. Missing bytes are resent.
  6. Closing takes four segments: each side sends FIN and ACKs the other's. The side that closed first waits in TIME_WAIT.
{{< /fig >}} ## What TCP gives you, and what it costs - **Reliability.** Every byte has a sequence number. Unacknowledged data is retransmitted after a timeout or after duplicate ACKs. - **Ordering.** The receiver reassembles bytes in order before handing them to the app. - **Flow control.** The receiver advertises a **window**: how much it can buffer. A slow consumer slows the sender. - **Congestion control.** The sender probes the network's capacity with an algorithm such as CUBIC (the Linux default) or BBR, starting with slow start and backs off on loss. The costs: one round trip for the handshake before any data, and **head-of-line blocking**: if one segment is lost, everything after it waits, even if it already arrived. That second cost is why HTTP/3 moved to QUIC over UDP. ## Common mistakes - **Opening a new connection per query.** Each one costs a handshake, often plus TLS and database authentication. Use a connection pool (PgBouncer, HikariCP, SQLAlchemy's pool). - **No timeouts.** A peer that vanishes without a FIN leaves your socket waiting forever. Set connect and read timeouts, and enable TCP keepalive for long-idle connections. - **Running out of ephemeral ports.** A client that opens and closes thousands of short connections piles up sockets in `TIME_WAIT`. Reuse connections instead of tuning the kernel. - **Confusing "connection refused" with "timeout".** Refused (an RST) means the host is up but nothing listens on that port. A timeout usually means a firewall silently dropped the SYN. ## Try it ```bash ss -tan state established # open TCP connections on Linux nc -vz db1.internal 5432 # can I reach the port at all? sudo tcpdump -n -i any 'tcp port 5432 and tcp[tcpflags] & (tcp-syn|tcp-fin) != 0' ``` ## Learning Resources ### Books - [TCP/IP Illustrated, Volume 1: The Protocols](https://www.oreilly.com/library/view/tcpip-illustrated-volume/9780132808200/): W. Richard Stevens, Kevin Fall - *Computer Networking: A Top-Down Approach*, Kurose and Ross: chapter 3 (transport layer) - [High Performance Browser Networking: Building Blocks of TCP](https://hpbn.co/building-blocks-of-tcp/): Ilya Grigorik, free online ### Specifications - [RFC 9293: Transmission Control Protocol](https://www.rfc-editor.org/rfc/rfc9293) - [RFC 791: Internet Protocol](https://www.rfc-editor.org/rfc/rfc791) ### Videos - [TCP/IP Tutorial: What Is TCP/IP And How Does It Work?](https://www.youtube.com/watch?v=kE7OLr9zTqk) See also: [UDP]({{< relref "udp" >}}), [HTTP]({{< relref "http" >}}), [Firewalls]({{< relref "../concepts/firewalls" >}}).