DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ terraform plan -out=tfplan && terraform apply tfplan
# Infrastructure Provisioning **Infrastructure provisioning** is creating and changing cloud resources (networks, buckets, databases, clusters, IAM roles) from code. You declare the resources you want; a provisioning tool calls the cloud APIs to make reality match, and remembers what it created so it can update or delete it later. **Terraform** is the most widely used tool, and its open-source fork **OpenTofu** is a drop-in alternative. Both use **HCL** (HashiCorp Configuration Language) and the same **providers**: plugins that translate resources into API calls for AWS, GCP, Azure, Kubernetes, Snowflake, Databricks, Confluent and hundreds more. ## Three views of the same infrastructure Terraform compares three things: your **code** (what you want), the **state file** (what Terraform created last time, with IDs) and the **real cloud** (what actually exists). {{< fig kicker="$ terraform plan" title="Plan and apply: code, state and the real cloud" >}} CODE · main.tf aws_s3_bucket.raw aws_db_instance.dwhclass = small STATE · terraform.tfstate aws_s3_bucket.raw aws_db_instance.dwhclass = small REAL CLOUD · AWS API aws_s3_bucket.raw aws_db_instance.dwhclass = small aws_s3_bucket.curatedclass = medium Plan: 1 to add, 1 to change, 0 to destroy. aws_s3_bucket.curatedclass = medium aws_s3_bucket.curatedclass = medium class = large (console edit)Drift: next plan shows ~ dwh class large → medium
NO.STEPCOMMANDTYPEINFO
102planREFRESHread raw, dwh from AWS: match state
202plan+ CREATEaws_s3_bucket.curated
302plan~ UPDATEaws_db_instance.dwh: class small → medium
403applyLOCKstate lock acquired
503applyAPICreateBucket curated · ModifyDBInstance dwh
604applySTATEstate written (serial 42 → 43), lock released
705consoleDRIFTsomeone resized dwh to large by hand
  1. Code, state and cloud agree: a raw bucket and a small dwh database.
  2. You edit main.tf: add a curated bucket and resize the database to medium. Nothing changes in the cloud yet.
  3. terraform plan refreshes from the cloud, diffs it with your code and prints the changes: 1 to add, 1 to change, 0 to destroy. A reviewer reads this in the pull request.
  4. terraform apply takes a lock on the state and calls the cloud APIs. The bucket is created and the database resized.
  5. Terraform writes the new IDs and attributes into the state and releases the lock. All three views agree again.
  6. Someone resizes the database in the console. Reality has drifted from code; the next plan will propose reverting it, so either import the change into code or stop manual edits.
{{< /fig >}} The code behind the figure: ```hcl terraform { backend "s3" { # remote, shared, locked state bucket = "acme-tf-state" key = "data-platform/prod.tfstate" region = "eu-central-1" use_lockfile = true } } resource "aws_s3_bucket" "raw" { bucket = "acme-lake-raw" } resource "aws_s3_bucket" "curated" { bucket = "acme-lake-curated" } resource "aws_db_instance" "dwh" { identifier = "dwh" engine = "postgres" instance_class = "db.m7g.large" # "medium" in the figure # ... } ``` ## The workflow | Command | What it does | |---|---| | `terraform init` | Download providers and modules, configure the backend | | `terraform fmt` / `validate` | Format and statically check the code | | `terraform plan` | Show the diff between code and reality; changes nothing | | `terraform apply` | Execute a plan, then update state | | `terraform import` or `import {}` blocks | Bring existing resources under management | | `terraform destroy` | Delete everything in this state | In a team, run `plan` automatically on every pull request and `apply` only from CI after merge, with tools such as **Atlantis**, **HCP Terraform**, **Spacelift** or plain [CI/CD]({{< relref "../../cicd" >}}). ## Terraform, OpenTofu and alternatives In 2023 HashiCorp moved Terraform from the open-source MPL license to the source-available **Business Source License**, and IBM completed its acquisition of HashiCorp in 2025. The community forked the last open version as **OpenTofu**, now governed by the Linux Foundation. For most users the two are interchangeable; OpenTofu adds features such as client-side state encryption. | Tool | Language | State | Best for | |---|---|---|---| | **Terraform** / **OpenTofu** | HCL | State file in a backend | Multi-cloud, largest provider ecosystem | | **Pulumi** | Python, TypeScript, Go, C#, Java | Pulumi Cloud or a backend | Teams that want loops, types and tests in a general-purpose language | | **AWS CloudFormation** / **CDK** | YAML/JSON / TypeScript, Python… | Managed by AWS | AWS-only shops | | **Azure Bicep** | Bicep DSL | Managed by Azure | Azure-only shops | | **Crossplane** | Kubernetes YAML | Kubernetes etcd | Platforms that manage everything as Kubernetes resources with continuous reconciliation | ## Common mistakes - **Local state.** A `terraform.tfstate` on a laptop gets lost or overwritten. Use a remote backend with locking and versioning from day one. - **Not reading the plan.** A renamed resource can show up as **destroy and create**. For a database, that means data loss. Look for `-/+` and `destroy` in every plan; use `moved {}` blocks and `prevent_destroy` on stateful resources. - **Hard-coding per environment.** Copy-pasted `prod/` and `dev/` folders drift apart. Use modules with variables, one root per environment. - **Unpinned providers.** A new major provider version can change behaviour. Pin versions and commit `.terraform.lock.hcl`. ## Learning Resources ### Official - [Terraform tutorials (HashiCorp)](https://developer.hashicorp.com/terraform/tutorials) - [Terraform language documentation](https://developer.hashicorp.com/terraform/language) - [OpenTofu documentation](https://opentofu.org/docs/) - [Pulumi documentation](https://www.pulumi.com/docs/) - [Crossplane documentation](https://docs.crossplane.io/) ### Books - *Terraform: Up and Running*, 3rd ed., Yevgeniy Brikman (O'Reilly) - *Terraform in Action*, Scott Winkler (Manning) ### Courses - [Complete Terraform Course: From Beginner to Pro (DevOps Directive)](https://www.youtube.com/watch?v=7xngnjfIlK4) - [Terraform Course: Automate your AWS cloud infrastructure (freeCodeCamp)](https://www.youtube.com/watch?v=SLB_c_ayRMo)