DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ git log --oneline --graph --all
# Git **Git** is a distributed version control system: every clone of a repository holds the complete history, so you can commit, branch and compare versions offline and sync with others later. Linus Torvalds wrote it in 2005 for Linux kernel development; today it is the standard for virtually all software and data projects. ## The mental model Three ideas explain most of Git: 1. **A commit is a snapshot** of the whole project, plus metadata (author, message) and a pointer to its **parent** commit(s). Its ID is a hash of all of that, so changing anything in history creates new commits with new IDs. 2. **A branch is just a movable pointer** to a commit. `main` and `feature` are labels; creating a branch costs a few bytes. 3. **Four places**: your **working tree**, the **staging area**, your **local repository** and a **remote**. Every everyday command moves changes from one place to the next. Git is not GitHub. Git is the tool on your machine and works fully offline; GitHub, GitLab and Bitbucket host a shared copy and add pull requests, review and CI on top. ## Four places your code lives {{< fig kicker="$ git add · git commit · git push · git fetch · git pull" title="One change travels from your editor to the team, and the team's work comes back" >}} your machine GitHub · GitLab WORKING TREE your files on disk STAGING AREA index: the next commit LOCAL REPO .git: every commit REMOTE origin: shared copy report.pyuntracked orders.sqlmodified git add orders.sqlstaged git commit 3f2a1c9main git push 3f2a1c9yours 8b4e0d2teammate git fetch origin/main8b4e0d2 git pull = fetch + merge into your files
  1. You edited orders.sql and created report.py. git status calls the first modified and the second untracked: Git has never stored it.
  2. git add orders.sql copies the change into the staging area. Only staged changes go into the next commit; report.py stays out.
  3. git commit turns the staging area into a permanent snapshot in .git, with an ID like 3f2a1c9. Still only on your laptop.
  4. git push uploads your new commits to the remote, so the team can see them.
  5. A teammate pushed 8b4e0d2. git fetch downloads it and moves origin/main, but your files do not change.
  6. git pull is fetch plus merge: the teammate's change lands in your working tree. Pull before you start, and before you push.
{{< /fig >}} `git status` tells you where each file is: | State | Meaning | Next command | |---|---|---| | Untracked | New file Git has never stored | `git add file`, or list it in `.gitignore` | | Modified | Tracked file changed, not staged | `git add file`, or `git restore file` to drop the edit | | Staged | Will be in the next commit | `git commit`, or `git restore --staged file` to unstage | | Committed (clean) | Working tree matches the last commit | `git push` to share it | ```bash git status git diff # unstaged changes, line by line git add models/orders.sql # or: git add -p to pick hunks git diff --staged # what will be committed git commit -m "Deduplicate orders on order_id" git log --oneline ``` A `.gitignore` file keeps secrets and junk out of the staging area: `.env`, `*.log`, `__pycache__/`, `target/`, local data files. ## One-time setup Tell Git who you are, once per machine. The name and email are written into every commit. ```bash git config --global user.name "Alice Example" git config --global user.email "alice@example.com" git config --global init.defaultBranch main git config --global pull.rebase true # optional: git pull rebases instead of merging ``` Then let GitHub or GitLab recognize your machine with an [SSH]({{< relref "/docs/roadmap/networking/protocols/ssh" >}}) key, so you never type a password to push: ```bash ssh-keygen -t ed25519 -C "alice@example.com" # creates ~/.ssh/id_ed25519 and id_ed25519.pub cat ~/.ssh/id_ed25519.pub # paste into GitHub › Settings › SSH and GPG keys ssh -T git@github.com # "Hi alice! You've successfully authenticated..." ``` Start a repository with `git init` in an existing folder, or copy one with `git clone git@github.com:example/pipelines.git`. ## Branches A branch lets you work on a change without touching `main`. **HEAD** is the pointer to the branch you are on; new commits go on that branch, and the branch label moves forward with them. {{< fig kicker="$ git switch -c feature/login" title="A branch is a label that moves; HEAD says which label you are on" >}} your laptop A B main HEAD feature/login HEAD B C feature/login HEAD D feature/login HEAD origin · GitHub A B C D feature/login main origin/feature/login $ git push -u origin feature/login HEAD
  1. History so far: commits A and B on main. HEAD points at main, so that is the branch you are on.
  2. git switch -c feature/login creates a new label at the same commit and moves HEAD to it. No files are copied.
  3. Edit, git add, git commit: the new commit C goes on feature/login, and the label moves forward with HEAD. main stays at B.
  4. Another commit, D. Each commit points back to its parent, so the branch is simply the chain behind its label.
  5. git push -u origin feature/login uploads C and D and creates the branch on the remote. -u links the two, so later a plain git push is enough.
  6. git switch main moves HEAD back to B and rewrites your files to match it. Your feature work is safe on its own branch, ready for a pull request.
{{< /fig >}} ```bash git branch # list local branches; * marks the current one git switch -c feature/login # create and switch git switch main # go back (commit or stash your changes first) git branch -d feature/login # delete after it has been merged ``` ## Merge vs rebase Sooner or later `main` moves on while you work on a branch. There are two ways to combine the histories. {{< fig kicker="$ git merge feature vs git rebase main" title="Same starting point, two ways to integrate a branch" >}} git merge feature git rebase main ABCDEmainfeature ABCDEmainfeature Mmain M has two parents: C and E.D and E keep their hashes. D′E′featureold D, E: unreferenced mainfast-forward: linear
  1. Both panels start the same: feature branched off at B and added D and E, while main gained C.
  2. Merge: Git creates a new merge commit M that joins both lines, and main moves to it.
  3. M has two parents. Nothing was rewritten: the history shows what really happened, including the parallel work.
  4. Rebase: Git replays D and E on top of C as new commits D′ and E′, with new hashes. The old D and E are left behind.
  5. Now main can fast-forward to E′: a straight, linear history with no merge commit. The cost: D′ and E′ are different commits from what you pushed before.
{{< /fig >}} | | Merge | Rebase | |---|---|---| | History shape | True shape, with merge commits | Linear, easy to read and `bisect` | | Rewrites commits | No | Yes: new hashes | | Safe on shared branches | Yes | No: others have the old commits | | Conflicts | Resolved once, in the merge commit | Possibly once per replayed commit | The rule of thumb: **rebase your own local branch** onto `main` to keep it current and tidy; **never rebase commits others have already pulled**. Many teams avoid the question by using **squash merge** in pull requests: the whole branch lands on `main` as one commit. ## Everyday commands | Command | What it does | |---|---| | `git status`, `git diff`, `git diff --staged` | What changed, staged or not | | `git log --oneline --graph` | Compact history | | `git switch `, `git switch -c ` | Change or create branches | | `git pull --rebase` | Fetch and replay your local commits on top | | `git restore ` | Discard working-tree changes to a file | | `git commit --amend` | Fix the last commit (before pushing) | | `git revert ` | Undo a commit with a new commit: safe on shared branches | | `git reset --soft HEAD~1` | Un-commit, keep the changes staged | | `git stash` | Park uncommitted work temporarily | | `git bisect` | Binary-search history for the commit that broke something | | `git reflog` | Every position HEAD has been: how you recover "lost" commits | ## Your first push From an empty folder to a merged pull request, using the commands above: 1. Create an empty repository on GitHub called `workshop`, with no README, so the first push has nothing to conflict with. 2. Start the local repository and make the first commit: ```bash cd ~/workshop git init git add backup.sh git commit -m "Add backup script" ``` 3. Keep generated files out: `echo "*.tar.gz" > .gitignore`, then add and commit `.gitignore`. 4. Check the SSH key, connect the remote and push `main`: ```bash ssh -T git@github.com git remote add origin git@github.com:/workshop.git git remote -v # origin is just a nickname for that URL git push -u origin main ``` 5. On a new branch, add `README.md`, commit it and push the branch: `git switch -c add-readme`, `git add README.md`, `git commit -m "Add README"`, `git push -u origin add-readme`. 6. Open a pull request on GitHub, merge it, then update your local copy: `git switch main` and `git pull`. 7. Look at the result: `git log --oneline --graph --all`. 8. Bonus: edit the same line on GitHub and locally, pull, and resolve the conflict. Git marks it between `<<<<<<<`, `=======` and `>>>>>>>`; keep what you want, delete the markers, then add and commit. ## A workflow for data repositories - **Trunk-based development**: short-lived branches (hours to days), merged to `main` through pull requests with CI. Avoid long-lived `develop` branches; they drift and merge painfully. - Keep each change small and focused: one model, one DAG, one migration. - Write commit messages that say **why**: "Deduplicate orders on order_id (late CDC events cause doubles)" beats "fix". - Use **pre-commit** hooks for formatting (`ruff`, `sqlfluff`) and secret scanning. - Large files that must live with code (small seed CSVs are fine; models and media are not) belong in **Git LFS**. ## Common mistakes - **Committing secrets.** Deleting the file later does not remove it from history. Rotate the secret immediately; then clean history if needed. - **`git push --force` on a shared branch.** It silently discards others' commits. Use `--force-with-lease`, and only on your own branches. - **Huge, mixed commits.** A commit that changes ten models and the CI config is hard to review and impossible to revert cleanly. - **Committing data and notebooks with outputs.** Repository size explodes and diffs become unreadable. Strip notebook outputs (nbstripout) and keep data elsewhere. ## Learning Resources ### Books - [Pro Git, Scott Chacon and Ben Straub (free online)](https://git-scm.com/book/en/v2) - [Version Control with Git, 3rd ed. (O'Reilly)](https://www.oreilly.com/library/view/version-control-with/9781492091189/) ### Interactive - [Learn Git Branching](https://learngitbranching.js.org/): visual, in-browser exercises for merge, rebase and more - [Oh Shit, Git!?!](https://ohshitgit.com/): how to get out of common messes ### Reference - [Git documentation](https://git-scm.com/docs) - [Git internals: Git objects (Pro Git)](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) - [Trunk-based development](https://trunkbaseddevelopment.com/)