Web Frameworks and API Development

Data engineers build APIs more often than the job title suggests: a service that serves model features, an internal endpoint over a curated table, a webhook receiver that lands events in Kafka, or an admin tool for a pipeline. A web framework handles the plumbing of HTTP (parsing requests, routing, validation, serialization, errors) so you only write the part that is specific to your data.

What every framework does

$ curl -H 'Authorization: Bearer …' /orders/42
One request through a web framework
CLIENT SERVER uvicorn, Tomcat MIDDLEWARE log, auth, CORS HANDLER get_order(id) DATABASE GET /orders/42 parse → Request Request no token → 401 route /orders/{id} validate id: int SELECT … id=42 row Order → JSON + X-Request-ID 200 OK JSON
  1. Every framework, in any language, runs the same pipeline. The details differ, the shape doesn't.
  2. The server (an ASGI server, a servlet container, Go's net/http) reads bytes off the socket and builds a Request object.
  3. Middleware wraps every request: logging, auth, CORS, rate limits. It can short-circuit, for example 401 when there is no valid token.
  4. The router matches method + path to a handler. The framework parses and validates path, query and body before your code runs.
  5. Your handler does the real work, usually a database call. This is where the time goes.
  6. The return value is serialized to JSON. The response passes back out through the middleware, which can add headers and record latency.

The concepts carry across languages:

Concept Python (FastAPI) Java (Spring Boot) Go (net/http, chi)
Route @app.get("/orders/{id}") @GetMapping("/orders/{id}") mux.HandleFunc("GET /orders/{id}", h)
Validation Pydantic models Bean Validation (@Valid) By hand or a validator library
Middleware ASGI middleware Servlet filters / interceptors func(http.Handler) http.Handler
Server Uvicorn, Granian Embedded Tomcat, Jetty, Netty Built into the standard library
API docs OpenAPI generated automatically springdoc-openapi swaggo, or spec-first with oapi-codegen

Pick by language

Page Frameworks covered
Python FastAPI, Django (+ Django REST Framework / Django Ninja), Flask, Litestar; WSGI vs ASGI
Java Spring Boot, Quarkus, Micronaut; virtual threads and native images
Go The standard library’s net/http, chi, Gin, Echo, Fiber

What makes an API production-ready, in any framework

  • Validation at the edge. Reject bad input with a clear 400 before it reaches your database.
  • Timeouts everywhere. On the server, on database calls, and on every outbound HTTP call.
  • Health and readiness endpoints (/healthz, /readyz) for load balancers and Kubernetes.
  • Structured logs, metrics and traces. OpenTelemetry instrumentation is available for all the frameworks here.
  • Authentication with OAuth and JWT rather than home-made tokens.
  • A contract. An OpenAPI document for REST APIs, or .proto files for gRPC.

Learning Resources

Caching Python