Git
Git is a distributed version control system: every clone of a repository holds the complete history, so you can commit, branch and compare versions offline and sync with others later. Linus Torvalds wrote it in 2005 for Linux kernel development; today it is the standard for virtually all software and data projects.
The mental model
Three ideas explain most of Git:
- A commit is a snapshot of the whole project, plus metadata (author, message) and a pointer to its parent commit(s). Its ID is a hash of all of that, so changing anything in history creates new commits with new IDs.
- A branch is just a movable pointer to a commit.
mainandfeatureare labels; creating a branch costs a few bytes. - Four places: your working tree, the staging area, your local repository and a remote. Every everyday command moves changes from one place to the next.
Git is not GitHub. Git is the tool on your machine and works fully offline; GitHub, GitLab and Bitbucket host a shared copy and add pull requests, review and CI on top.
Four places your code lives
- You edited
orders.sqland createdreport.py.git statuscalls the first modified and the second untracked: Git has never stored it. git add orders.sqlcopies the change into the staging area. Only staged changes go into the next commit;report.pystays out.git committurns the staging area into a permanent snapshot in .git, with an ID like3f2a1c9. Still only on your laptop.git pushuploads your new commits to the remote, so the team can see them.- A teammate pushed
8b4e0d2.git fetchdownloads it and movesorigin/main, but your files do not change. git pullis fetch plus merge: the teammate's change lands in your working tree. Pull before you start, and before you push.
git status tells you where each file is:
| State | Meaning | Next command |
|---|---|---|
| Untracked | New file Git has never stored | git add file, or list it in .gitignore |
| Modified | Tracked file changed, not staged | git add file, or git restore file to drop the edit |
| Staged | Will be in the next commit | git commit, or git restore --staged file to unstage |
| Committed (clean) | Working tree matches the last commit | git push to share it |
git status
git diff # unstaged changes, line by line
git add models/orders.sql # or: git add -p to pick hunks
git diff --staged # what will be committed
git commit -m "Deduplicate orders on order_id"
git log --onelineA .gitignore file keeps secrets and junk out of the staging area: .env, *.log, __pycache__/, target/, local data files.
One-time setup
Tell Git who you are, once per machine. The name and email are written into every commit.
git config --global user.name "Alice Example"
git config --global user.email "[email protected]"
git config --global init.defaultBranch main
git config --global pull.rebase true # optional: git pull rebases instead of mergingThen let GitHub or GitLab recognize your machine with an SSH key, so you never type a password to push:
ssh-keygen -t ed25519 -C "[email protected]" # creates ~/.ssh/id_ed25519 and id_ed25519.pub
cat ~/.ssh/id_ed25519.pub # paste into GitHub › Settings › SSH and GPG keys
ssh -T [email protected] # "Hi alice! You've successfully authenticated..."Start a repository with git init in an existing folder, or copy one with git clone [email protected]:example/pipelines.git.
Branches
A branch lets you work on a change without touching main. HEAD is the pointer to the branch you are on; new commits go on that branch, and the branch label moves forward with them.
- History so far: commits A and B on main. HEAD points at main, so that is the branch you are on.
git switch -c feature/logincreates a new label at the same commit and moves HEAD to it. No files are copied.- Edit,
git add,git commit: the new commit C goes on feature/login, and the label moves forward with HEAD. main stays at B. - Another commit, D. Each commit points back to its parent, so the branch is simply the chain behind its label.
git push -u origin feature/loginuploads C and D and creates the branch on the remote.-ulinks the two, so later a plaingit pushis enough.git switch mainmoves HEAD back to B and rewrites your files to match it. Your feature work is safe on its own branch, ready for a pull request.
git branch # list local branches; * marks the current one
git switch -c feature/login # create and switch
git switch main # go back (commit or stash your changes first)
git branch -d feature/login # delete after it has been mergedMerge vs rebase
Sooner or later main moves on while you work on a branch. There are two ways to combine the histories.
- Both panels start the same: feature branched off at B and added D and E, while main gained C.
- Merge: Git creates a new merge commit M that joins both lines, and main moves to it.
- M has two parents. Nothing was rewritten: the history shows what really happened, including the parallel work.
- Rebase: Git replays D and E on top of C as new commits D′ and E′, with new hashes. The old D and E are left behind.
- Now main can fast-forward to E′: a straight, linear history with no merge commit. The cost: D′ and E′ are different commits from what you pushed before.
| Merge | Rebase | |
|---|---|---|
| History shape | True shape, with merge commits | Linear, easy to read and bisect |
| Rewrites commits | No | Yes: new hashes |
| Safe on shared branches | Yes | No: others have the old commits |
| Conflicts | Resolved once, in the merge commit | Possibly once per replayed commit |
The rule of thumb: rebase your own local branch onto main to keep it current and tidy; never rebase commits others have already pulled. Many teams avoid the question by using squash merge in pull requests: the whole branch lands on main as one commit.
Everyday commands
| Command | What it does |
|---|---|
git status, git diff, git diff --staged |
What changed, staged or not |
git log --oneline --graph |
Compact history |
git switch <branch>, git switch -c <new> |
Change or create branches |
git pull --rebase |
Fetch and replay your local commits on top |
git restore <file> |
Discard working-tree changes to a file |
git commit --amend |
Fix the last commit (before pushing) |
git revert <commit> |
Undo a commit with a new commit: safe on shared branches |
git reset --soft HEAD~1 |
Un-commit, keep the changes staged |
git stash |
Park uncommitted work temporarily |
git bisect |
Binary-search history for the commit that broke something |
git reflog |
Every position HEAD has been: how you recover “lost” commits |
Your first push
From an empty folder to a merged pull request, using the commands above:
- Create an empty repository on GitHub called
workshop, with no README, so the first push has nothing to conflict with. - Start the local repository and make the first commit:
cd ~/workshop git init git add backup.sh git commit -m "Add backup script" - Keep generated files out:
echo "*.tar.gz" > .gitignore, then add and commit.gitignore. - Check the SSH key, connect the remote and push
main:ssh -T [email protected] git remote add origin [email protected]:<you>/workshop.git git remote -v # origin is just a nickname for that URL git push -u origin main - On a new branch, add
README.md, commit it and push the branch:git switch -c add-readme,git add README.md,git commit -m "Add README",git push -u origin add-readme. - Open a pull request on GitHub, merge it, then update your local copy:
git switch mainandgit pull. - Look at the result:
git log --oneline --graph --all. - Bonus: edit the same line on GitHub and locally, pull, and resolve the conflict. Git marks it between
<<<<<<<,=======and>>>>>>>; keep what you want, delete the markers, then add and commit.
A workflow for data repositories
- Trunk-based development: short-lived branches (hours to days), merged to
mainthrough pull requests with CI. Avoid long-liveddevelopbranches; they drift and merge painfully. - Keep each change small and focused: one model, one DAG, one migration.
- Write commit messages that say why: “Deduplicate orders on order_id (late CDC events cause doubles)” beats “fix”.
- Use pre-commit hooks for formatting (
ruff,sqlfluff) and secret scanning. - Large files that must live with code (small seed CSVs are fine; models and media are not) belong in Git LFS.
Common mistakes
- Committing secrets. Deleting the file later does not remove it from history. Rotate the secret immediately; then clean history if needed.
git push --forceon a shared branch. It silently discards others’ commits. Use--force-with-lease, and only on your own branches.- Huge, mixed commits. A commit that changes ten models and the CI config is hard to review and impossible to revert cleanly.
- Committing data and notebooks with outputs. Repository size explodes and diffs become unreadable. Strip notebook outputs (nbstripout) and keep data elsewhere.
Learning Resources
Books
Interactive
- Learn Git Branching: visual, in-browser exercises for merge, rebase and more
- Oh Shit, Git!?!: how to get out of common messes