DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ curl -s https://api.shop.com/orders/42
# REST **REST** (Representational State Transfer) is an architectural style for APIs over HTTP. You model your domain as **resources** with stable URLs (`/orders/42`), and act on them with the standard HTTP **methods** (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`). Each request is **stateless**, carrying everything the server needs, and responses can be **cached**. Most public APIs you will ingest from (Stripe, GitHub, Salesforce, Shopify) are REST APIs returning JSON. The term comes from Roy Fielding's 2000 PhD thesis. In practice, "RESTful" usually means "resource-oriented JSON over HTTP", not every constraint Fielding described. ## Resources and methods {{< fig kicker="$ curl -X DELETE https://api.shop.com/orders/42" title="One resource, six requests" >}} REQUEST RESPONSE SERVER ยท /orders /orders/41status: paid/orders/42status: paid GET /orders 200 [41, 42] /orders/43status: new POST /orders{"sku":"A7"} 201 Location: /orders/43 GET /orders/42 200 {"status":"paid"} PATCH /orders/42{"status":"shipped"} 200 {"status":"shipped"} DELETE /orders/42 204 No Content GET /orders/42 404 Not Found status: shipped deleted
  1. One collection, /orders, holds two orders. Each order has its own URL. The method says what to do with it.
  2. GET /orders lists the collection: 200 with a JSON array. Safe: nothing changes on the server.
  3. POST /orders creates a new order. The server picks the ID and answers 201 Created with a Location header pointing at it.
  4. GET /orders/42 reads one resource by its URL: 200 and the order as JSON.
  5. PATCH /orders/42 changes only the fields you send. The order is now shipped.
  6. DELETE /orders/42 removes it. 204 No Content: success, with no body to return.
  7. Asking for it again gets 404 Not Found. The URL is the same; the resource is gone.
{{< /fig >}} | Request | Meaning | Success | Idempotent | |---|---|---|---| | `GET /orders?status=shipped&limit=50` | List orders, filtered | 200 | yes | | `GET /orders/42` | Read one order | 200, or 404 | yes | | `POST /orders` | Create an order | 201 + `Location: /orders/43` | no | | `PUT /orders/42` | Replace order 42 | 200 or 204 | yes | | `PATCH /orders/42` | Change some fields | 200 | not necessarily | | `DELETE /orders/42` | Delete it | 204 | yes | | `GET /users/7/orders` | Orders belonging to user 7 | 200 | yes | URLs are **nouns**. The verb is the HTTP method. `/getOrder?id=42` or `POST /deleteOrder` are RPC in disguise. ## Try it [JSONPlaceholder](https://jsonplaceholder.typicode.com/) is a free fake REST API you can practice on. Writes are accepted and echoed back but not stored. ```bash API=https://jsonplaceholder.typicode.com curl -s $API/users/1 | jq '.name, .email' # GET one resource curl -s "$API/posts?userId=1" | jq length # GET a filtered collection curl -s -i -X POST $API/posts \ -H "Content-Type: application/json" \ -d '{"title":"hello","userId":1}' # POST: 201 Created + the new id curl -s -o /dev/null -w "%{http_code}\n" -X DELETE $API/posts/1 # DELETE: 200 ``` ## Design rules that age well - **Use status codes properly.** 400 for bad input with a JSON error body, 401 vs 403, 404, 409 for conflicts, 429 with `Retry-After` for rate limits. A consistent error format such as RFC 9457 "Problem Details" helps clients. - **Paginate every list.** Prefer **cursor** pagination (`?after=ord_123`) over `?page=7`: offsets skip or repeat rows when data changes between requests, which is exactly what happens during an incremental sync. - **Support incremental reads.** A filter like `?updated_since=2026-01-01T00:00:00Z` lets pipelines fetch only what changed. - **Make retries safe.** Accept an `Idempotency-Key` header on `POST` so a retried payment isn't charged twice. - **Version deliberately.** `/v1/...` in the path or a header. Add fields freely; never rename or remove them without a new version. - **Describe it with OpenAPI.** An OpenAPI 3.1 (or 3.2, released 2025) document gives you docs, client generation, mocks and request validation for free. ## REST vs GraphQL vs gRPC | | REST | [GraphQL]({{< relref "../graphql" >}}) | [gRPC]({{< relref "../grpc" >}}) | |---|---|---|---| | Shape | Many resource URLs | One endpoint, client writes a query | Service methods defined in `.proto` | | Payload | JSON (usually) | JSON | Protobuf (binary) | | Transport | HTTP/1.1, 2 or 3 | HTTP | HTTP/2 | | HTTP caching | Natural (GET + ETag) | Hard (mostly POST) | No | | Streaming | SSE, webhooks | Subscriptions | Built in, both directions | | Best for | Public APIs, CRUD, integrations | Front-ends with varied data needs | Internal service-to-service, low latency | ## Common mistakes - **Returning 200 with `{"error": ...}` in the body.** Clients and monitoring can't tell it failed. - **Unbounded list endpoints** that return 2 million rows in one response. - **Chatty APIs** that need 20 calls to render one screen. Offer an `?expand=` option, a dedicated endpoint, or consider GraphQL. - **Breaking changes without a version**, which silently breaks every downstream pipeline. ## Learning Resources ### Books - [Designing Web APIs](https://www.oreilly.com/library/view/designing-web-apis/9781492026914/): Jin, Sahni, Shevat - [API Design Patterns](https://www.manning.com/books/api-design-patterns): JJ Geewax, based on Google's API design rules ### Articles and references - [Fielding's dissertation, chapter 5: REST](https://ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm) - [Microsoft REST API Guidelines](https://github.com/microsoft/api-guidelines) and [Google API Improvement Proposals](https://google.aip.dev/) - [Zalando RESTful API Guidelines](https://opensource.zalando.com/restful-api-guidelines/) - [OpenAPI Specification](https://spec.openapis.org/oas/latest.html) - [RFC 9457: Problem Details for HTTP APIs](https://www.rfc-editor.org/rfc/rfc9457) See also: [HTTP]({{< relref "/docs/roadmap/networking/protocols/http" >}}), [Web frameworks]({{< relref "/docs/roadmap/web-frameworks-and-api-development" >}}).