DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/ # Python Python is the most common language in data work, so it is often the natural choice when a data team needs an API: the same models, database drivers and libraries (pandas, Polars, DuckDB, scikit-learn) are already there. Python web frameworks fall into two families, defined by how they talk to the web server: - **WSGI** (synchronous): one request occupies one worker thread or process until it finishes. Flask and classic Django. - **ASGI** (asynchronous): one worker can juggle thousands of requests that are waiting on I/O, and it supports WebSockets and streaming. FastAPI, Litestar, Starlette, and Django's async views. ## The frameworks ### FastAPI **FastAPI** builds an API from ordinary Python **type hints**. It uses **Pydantic** (v2, with a Rust core) to validate and serialize data and **Starlette** for the ASGI web layer, and it generates an **OpenAPI** schema with interactive docs at `/docs` automatically. It is the default choice for new APIs and ML model serving in 2026. ```python from fastapi import FastAPI, HTTPException from pydantic import BaseModel app = FastAPI() class Order(BaseModel): id: int user: str total: float @app.get("/orders/{order_id}") async def get_order(order_id: int) -> Order: # "abc" → 422 automatically # db: an asyncpg pool created at startup; "user" is quoted because it is a reserved word row = await db.fetchrow('SELECT id, "user", total FROM orders WHERE id = $1', order_id) if row is None: raise HTTPException(status_code=404, detail="order not found") return Order(**row) ``` Run it with `fastapi dev main.py` locally, and `fastapi run` or `uvicorn main:app --workers 4` in production. ### Django **Django** is the "batteries included" framework: ORM, migrations, authentication, an auto-generated **admin** interface, forms, templates and security defaults in one package. Django 5.2 is the current LTS release and the 6.x series is current. For APIs, add **Django REST Framework** (mature, huge ecosystem) or **Django Ninja** (FastAPI-style type hints on top of Django). Choose Django when you need a full application with users and an admin panel, for example an internal tool for managing pipeline configuration or data quality rules. ### Flask **Flask** (3.x) is a small WSGI micro-framework: routing, request handling and templates, and you add everything else through extensions (SQLAlchemy, Flask-Login). It is simple to learn and still widely used: Airflow 2's web UI is a Flask app (Airflow 3 moved its API server to FastAPI). For new JSON APIs, FastAPI usually gives you more for the same effort. ### Litestar **Litestar** is an ASGI framework in the same spirit as FastAPI, with built-in dependency injection, DTOs, and support for Pydantic, msgspec and attrs models. A good option if you want more structure than FastAPI provides. ## How to choose | Need | Pick | |---|---| | JSON API, model serving, typed contracts | **FastAPI** (or Litestar) | | Full web app with users, admin, ORM, migrations | **Django** (+ DRF or Django Ninja for the API) | | Small, simple synchronous service or legacy codebase | **Flask** | | Dashboard or data app, not an API | Streamlit, Dash or Panel instead of a web framework | ## Servers | Server | Interface | Notes | |---|---|---| | **Uvicorn** | ASGI | The standard for FastAPI and Starlette | | **Granian** | ASGI, WSGI | Rust-based, fast, one server for both kinds | | **Gunicorn** | WSGI | Process manager for Flask/Django; can also run Uvicorn workers | | **Hypercorn** | ASGI | Supports HTTP/2 and HTTP/3 | ## Common mistakes - **Blocking calls inside `async def`.** `requests.get()`, `time.sleep()` or a sync database driver inside an async endpoint freezes the whole event loop and every other request with it. Use async libraries (`httpx`, `asyncpg`, `psycopg` async) or declare the endpoint with plain `def` so FastAPI runs it in a thread pool. - **Heavy CPU work in the request.** Training a model or crunching a big DataFrame in an endpoint blocks workers. Hand it to a task queue (Celery, Dramatiq, Arq) or a pipeline and return a job ID. - **Loading a model or a big file per request.** Load once at startup with FastAPI's `lifespan` handler. - **Using the development server in production** (`flask run`, `manage.py runserver`). - **Returning whole tables.** Paginate, and stream large exports instead of building them in memory. ## Learning Resources ### Official tutorials - [FastAPI tutorial](https://fastapi.tiangolo.com/tutorial/) - [Django tutorial](https://docs.djangoproject.com/en/stable/intro/) and [Django REST Framework](https://www.django-rest-framework.org/) - [Flask tutorial](https://flask.palletsprojects.com/en/stable/tutorial/) - [Litestar documentation](https://docs.litestar.dev/) - [Pydantic documentation](https://docs.pydantic.dev/latest/) ### Books - [FastAPI: Modern Python Web Development](https://www.oreilly.com/library/view/fastapi/9781098135492/): Bill Lubanovic - [Microservice APIs: Using Python, Flask, FastAPI, OpenAPI and more](https://www.manning.com/books/microservice-apis): José Haro Peralta - [Two Scoops of Django](https://daniel.feldroy.com/books/tech): Daniel and Audrey Feldroy - [Flask Web Development, 2nd edition](https://www.oreilly.com/library/view/flask-web-development/9781491991725/): Miguel Grinberg ### Courses - [Python API Development: Comprehensive Course for Beginners](https://www.youtube.com/watch?v=0sOvCWFmrtA): freeCodeCamp, FastAPI + PostgreSQL - [Django for Everybody](https://www.dj4e.com/): Charles Severance - [The Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-2024-edition-chapter-i-hello-world): Miguel Grinberg See also: [Python language]({{< relref "/docs/roadmap/programming-language/python" >}}), [REST]({{< relref "/docs/roadmap/general-software-engineering-skills/rest" >}}).