Coder Legion - Recent posts tagged source-control

Learn Git fast as if your job depends on it

Mon, 16 Feb 2026 19:40:07 +0000

TL;DR

In this post, with a grain of humor, I'm going to show you the minimum number of Git commands that you need to learn in order to start being productive and get that code from your machine to the company Git server. Also, I'll cover a bit of theory and Git installation as well.

Disclaimer

⚠️ Nobody in this story, and no outfit or corporation, thank God, is based upon an actual person or outfit in the real world. But I can tell you this; as my journey through the software jungle progressed, I came to realize that, by comparison with the reality, my story was as tame as a holiday postcard. ~ John le Carré

Setting the stage

So, let's set a scene; it's your first day as a junior developer straight out of college. You get to work, greet your superior that's supposed to be responsible for you, and since he's too busy with putting down some fire that just popped up, he, conversely, while you're trying to keep up walking aside him tells you something like:

I've set the Git access for you to my website project.

You have a ticket in our project management tool assigned to you. No worries, it's just some simple stuff that needs to be changed.

Just check the email details, and you can start immediately.

I'm aware that you may not be familiar with Git. I'm not trying to be an asshole, but just deal with it, learn it, OK? Don't they, like, teach you kids nothing in school these days?

Ah, I gotta run.

Oh, and one more thing, look up branching and do not EVER, and I repeat - EVER, commit to master; do a PR instead! Good luck and see ya later!

OK. You sit down at your cubicle, open up your email client, log in, and there you see the email (without the subject line, of course) with the contents:

Here's a link to the Git repo on Github.

Good luck,
Mr. Superior

Reality sets in

⚠️ This, in and of itself is a poor onboarding if you ask me, but I personally have seen it way too many times, so here are the things you need to do to get productive fast and learn the ropes as fast as you can.

So now you're, in lack of better words, screwed (you know the exact word, but we're not supposed to be using profanity here, ae?).

You're lost. You heard about Git before, but you never used it. Now you're kicking yourself for not learning it properly yourself. But, it is what it is, and since you're really determined to make it in this 'programming' world, you're ready to do what it takes.

I would question the fact that if you are 'so into it', how come you haven't learned it yet, but as Mr. Superior said, I'm not trying to be an asshole

Therefore, buckle up young padawan, you're in for a ride!

What is this Git thing?

First, let me start off with a few problems:

Say you're writing a seminar and you make some changes, save them, go out of the text editor, come back, and you want to see all of your changes that you did last in the file. Or, for example, you want to undo some changes that you did last night (writing a seminar after returning from a party is a noble act, but not a very smart one!).
While writing this seminar you create several versions of the same file and you don't even know what's changed in which version; you would really love to know what was, for example, changed in this current version based on the, say, a version from a few days ago.
You're working on a group project and someone deletes your changes 'by accident'.

VCS (Version Control Systems) tends to solve these problems. And, let me start off by saying that no, you don't need to use VCS only for writing code (thus the seminar example). You can use it for any textual project. One of VCS solutions is Git. One other popular VCS, that's still used to some extent today is Subversion. You can see a detailed list of Version Control Systems here.

Git was made by the ever so slightly genius Linus Torvalds, the father of Linux:

Git is a distributed version control system (VCS).

Version control means that you use it to store versions of your documents. True, it seems it's very popular among developers, but designers and other people (that are a bit more tech savvy) also use it to track versions of their files.

This whole distributed part is one of the things that makes it different from other VCS. Rather than have only one single place for the full version history of the software, as is common in once-popular VCS like CVS or Subversion, in Git every developer's working copy is also a repository that can contain the full history of all changes.

OK, but what is Github?

Something like Dropbox, but customized for Git.

Github is a hosting service for Git projects, with some additional batteries included that we're not going to get into at this point, as you're short on time. Think of it as a place where you store your Git 'stuff'.

Installing Git

To install Git, go to the official website and download the executable for your machine. I'm not going into the installation details as they're really just a bunch of Next, Next, Next, yes to malware, Finish set of steps. I'm kidding about the malware OFC.

To make sure your installation was successful, run the following command in your Terminal/Command prompt:

git --version

You should get something like:

git version 2.10.1

Fetching the repo

OK, so let's finally get some action. In your email, you received an URL to the Git repository. It was this: https://github.com/Hitman666/MrSupervisorWebsite. You click the link, and you see something like this:

Click the Clone or download button, make sure you see the Use SSH in the top right corner of a popup and copy the link.

To get this repository to your computer, you have to execute the following command in your Terminal/Command prompt (I'll use Terminal in the rest of the post):

git clone https://github.com/Hitman666/MrSupervisorWebsite.git

The output of the command should look something like this:

Cloning into 'MrSupervisorWebsite'...
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 7 (delta 0), reused 7 (delta 0), pack-reused 0
Receiving objects: 100% (7/7), done.

Checking out the code

Now, go into the newly created MrSupervisorWebsite folder and open it in your editor.

You should see three files: index.htmnl, style.css, and README.md.

You quickly go and click to open the README.md file, but all you see is this:

# MrSupervisorWebsite

Hmm, so, no help there.

You open up the index.html file and you see:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>MrSupervisorWebsite</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>
    <h1 class="title">Welcome to MrSupervisorWebsite!</h1>
</body>
</html>

OK, finally something that you're familiar with! You also open up the style.css file, and you see:

.title {
    color: blue;
}

Next, you take a look at the ticket assigned to you in the project management tool that Mr.Superior mentioned, and it looks something like:

You take a deep breath, finally something that you know how to handle!

You're a bit taken aback by the 'manly color', so you go and ask Google:

You pick the 4872A2, and you're happy. You think to yourself: OK, now I just need to change this color in the style.css file, somehow add this to this Git thing and 'publish' it.

You're right, but not so fast! You remember that note from Mr.Supervisor that you shouldn't commit to master, EVER?

Creating a new branch

So, what is this master anyway? You go online, and you find that:

Git has branches, and its main one is called master. You can create as many of them as you want, based on any other existing one.

Also, you read a bit about the PR, and you learn that:

PR is short for Pull Request. Pull requests let you tell others about changes you've pushed to a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before the changes are merged into the repository.

So, this is smart, right? You can add your code to the Git repo without breaking any existing stuff on the master branch, and through this PR you can then ask a more experienced developer to do a CR (code review) on your PR and accept it or not.

You're smart; you see the huge benefit of this vs. just committing something to master and potentially breaking something in production.

Now, to circle back a bit and explain how could it be that you would break something!? Well, there's a thing called CI (Continuous Integration) and CD (Continuous Deployment), and you can read more about it, but in essence, it's this:

Every time someone pushes a new commit to master branch, a whole set of deploy scripts are being triggered, and essentially the new code that's pushed is going to appear in production.

What this means is that if you change that color and you commit it to master (you'll learn in a minute how to actually do this commit action), it will 'automagically' appear in the production on the website.

⚠️ Of course, the premise is that the service is set like that in the first place.

So, you get the point and value of additional branches, and you create a new branch by executing the following command in your Terminal:

git checkout -b poppingTitle

The upper command is a shortcut for two commands:

git branch poppingTitle
git checkout poppingTitle

The first one creates a new branch called poppingTitle from the current branch that you're on (in our case that's master). The second one 'checks it out'. Meaning, it places you on that branch.

Adding some code

Now you're finally ready to add some code!

Open up the style.css file, change the color, and save the file. It should look like this now:

.title {
    color: #4872A2;
}

One useful command that you'll use a lot is git status, which will you tell you the status of your git project. If you execute that in your Terminal you will see that the style.css file was modified:

On branch poppingTitle
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   style.css

no changes added to commit (use "git add" and/or "git commit -a")

As you can see, in the instructions from the output, these changes are not yet added to the commit. To do that, execute the command git add style.css. Of course, if you were to have multiple files you would not have to list each and every one of them, you would just use git add . command.

If you run the git status command again, you'll see that we have changes that are ready to be committed:

On branch poppingTitle
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    modified:   style.css

Committing code

To commit (one could use words like save or apply here for better clarity) the added changes to your local repository, just run:

git commit -m "Changed the title color"

Of course, the commit message can be anything, but please do yourself a favor and write meaningful commit messages.

If you run the git status command again, you'll see that we have a clean repository:

On branch poppingTitle
nothing to commit, working tree clean

Pushing to a remote repository

So, great, at this point you have the change locally in your repository, but you'd like to get this back to Github (the so-called remote repository, or origin as it's usually referred to).

For that you have to execute the following command:

git push origin poppingTitle

You should get an output similar to this:

Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 299 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To github.com:Hitman666/MrSupervisorWebsite.git
 * [new branch]      poppingTitle -> poppingTitle

Doing a PR

If you check Github now, you'll see that we've pushed this new branch:

Now, click on the Compare & pull request button.

Here you can add some comment summarizing your pull request:

Finally, click the Create pull request button.

And now (if we're playing by the book) someone else from your team would come and review your code. At this point, you could let someone know about this via your Project Management Tool (JIRA, Trello, Leankit, Kanbanflow, etc...) by putting this task/card in the appropriate column.

Your PR is accepted

When the pull request is approved (you get a notification from Github when someone approves or rejects it), you as the author go back to Github and just click on the Merge pull request button, and that's it:

Finally, click the Delete branch button as it's a good practice to not keep the merged branches in the remote repo:

You get to live another day

Your merge to master goes well, it's auto-deployed, you check the website and all is cool. You update the ticket in your ticketing system by placing it in the 'Deployed' column (or some similarly named column).

It's 17:12, you close your laptop and cheerfully go home to learn more about this awesome Git thingy.

DevLog 20250927: Use Fossil as Binary Asset VCS

Fri, 17 Oct 2025 14:50:27 +0000

It's clean and elegant.

Overview

As discussed in a previous post about binary file version control, keeping binaries well-versioned has always been challenging.

Years ago, when I first discovered SQLite, I also found Fossil. At that time, it didn't meet my needs - Git proved far better for collaboration and text-based workflows (especially with GitHub and GitHub Desktop).

However, Fossil may perform better for binary files under ~2GB, provided some manual bookkeeping and syncing.

Beyond versioning, Fossil offers several useful built-in tools:

Chatroom: Configurable 7-day ephemeral chat for authorized users. Supports HTTP API.
Wiki
Embedded Documentation
Tech Notes
Web UI Server
Direct file views: HEX, Text, Download
Forum: Permanent records; CDN support for images
Bug Tracking / Tickets / Issues
Full-text search

Framework

This framework describes how I'm beginning to use Fossil alongside Git for version control.

Using Fossil + Git

The plan is simple: Use git for text files; fossil for binary assets.

# Create repo
cd "<Repo Folder>"
fossil init ../MyRepo.fossil    # The convention is to create fossil repository outside the repo folder
fossil open -k ../MyRepo.fossil # Creates the _FOSSIL_ tracking file; -k skips checkout

# Create fossil-settings (See commit message below)
mkdir -p .fossil-settings
printf ".git\n.git/*\n.gitignore\n" > .fossil-settings/ignore-glob
fossil add .fossil-settings
fossil commit -m "Add Fossil ignore rules for Git files"

# Manually ignore files already tracked by Git, then commit again if needed
# fossil changes
# fossil commit -m "Update ignore file"

# In Git (e.g. via GitHub Desktop), ignore Fossil's contents, especially:
# *.fossil
# _FOSSIL_

# Treat all files as binary
printf "*" > .fossil-settings/binary-glob
fossil add .fossil-settings/binary-glob
fossil commit -m "Treat all files as binary in Fossil"

# Inspect
# fossil changes
# fossil extras

fossil add .
fossil commit -m "Initial Fossil import alongside Git"

This setup cannot keep Fossil and Git perfectly synchronized. Manual bookkeeping is necessary to restore past states accurately.

Summary

Key conventions:

Keep the .fossil file outside the Git repo folder.
.fossil-settings/binary-glob: mark all files as binary.
.fossil-settings/ignore-glob: define ignore patterns.
_FOSSIL_ and .fossil-settings can be deleted safely.

Quick Reference

Common commands:

fossil init <Repo> [--project-name <Name>] - create repo; edit name later in UI if omitted
fossil add . - add all
fossil add <File> - add file
fossil rm <File> - remove file
fossil commit -m <Message> - commit all local changes
fossil commit file1.c file2.h -m <Message> - commit specific files
fossil open --keep <DB> - open repo without checkout
fossil open --force <DB> - open repo, overwrite with latest
fossil status or fossil info - show repo info
fossil changes - list modified tracked files
fossil extras - list untracked files
fossil diff - show content changes
fossil timeline - history view
fossil ui [<Repo>] - launch web UI
fossil server [<Repo>] - start web server for push/pull

The typical git status is achieved with fossil changes + fossil extras.

Recipes

Ignore Blender versioning files

Use this pattern:

*.blend[0-9]*

Importing Git Repos

Not required here, but reference: https://fossil-scm.org/home/doc/trunk/www/inout.wiki

Stashing and Committing Specific Files

Fossil lacks Git's staging (git add -p) feature. The checkout directory is treated as one working tree snapshot. Still, you can commit selectively.

Commit specific files

You can pass the filenames explicitly to fossil commit:

fossil commit file1.c file2.h

This will commit only the listed files, even if you changed other files in the checkout. The other modified files will stay uncommitted.

Use stash to isolate changes

If you want to set aside changes temporarily:

Stash away all changes: fossil stash
Reapply only the files you want: fossil stash pop file1.c file2.h
Commit those files: fossil commit file1.c file2.h
Optionally reapply remaining changes: fossil stash pop

Revert unwanted files

If you've edited several files but only want to commit some:

fossil revert fileX.c fileY.c

This discards edits to those files, leaving only the desired ones for commit. Use stash first if you want to preserve them.

References

Learn Fossil quickly:

Quickstart Guide

Fixing mistakes with Git

Mon, 11 Aug 2025 19:10:50 +0000

Bob is my uncle…

He dropped his ice cream.

He didn’t know how to git.

Now he has no ice cream.

DONT BE LIKE BOB

Lucky for you I am here to share some ways to fix your mistakes on your project. This is assuming you already know git, if not read these articles:

If you are new to git, maybe start with these first:

If you prefer video: Git and GitHub for Beginners — Crash Course — YouTube

How git can fix my project?

Well, git has a full history(from the day of git init to today) of your project development. All you have to do is view the history and select the day you wanna go back and boom. This all will workout assuming you’ve named your commits properly.

You can not fix the mistakes once you pushed the changes to remote, theoretically of course. I mean you could, but it will mess up everyone else working on the branch so it is not advised not to fix something that is already on the remote unless all the contributors agree to it.

reset

git reset undo’s changes by removing the commits from the branch.

git reset --soft
git reset --mixed
git reset --hard

Each command above moves your current version of your project to the last recent version. You can go back to a specific commit by appending HEAD~n where n is the number of commits to go back.

i.e., git reset --soft HEAD~3 to go back in 3 commits.

--soft flag only moves the HEAD pointer to the commit but doesn’t affect the working directory and staged changes, meaning if you have created some files but that doesn’t exist on the older commit now it will be there because it still exists in the staging area and the working directory.

--mixed flag is the default and this moves the HEAD but it resets the staging area but not the working directory. So, if you have issued agit add . before reset it will be removed from the staging area but it will still be in the working directory.

--hard flag will do a factory reset to the specific commit. Anything and everything and between is vaporized.
Please do not do this in a public branch(a branch that is already pushed to remote) instead use the safer version revert.

You can also use git reset <filename> or git reset <commit-hash> <filename> to unstage a file without removing it from the filesystem.

revert

git revert is a safer version of undoing a commit. Revert works by finding a way to create a new commit without the changes introduced by the specified commit and appends it to the current tree.

Meaning it will not delete any commits instead it will create an inverse version of the specified commit.

git revert HEAD
git revert <commit-hash>

Unlike reset, revert requires a commit hash to act.

You can see in the above image that unlike reset it doesn’t just remove the previous commits instead creates a new commit and preserves the commits and doesn’t create problems with other branch contributors.

If revert and reset do the same which should I use?

Reset:

Don’t use it in a public branch.
Will remove all the commits after the specified commit (i.e., resetting to two days old commit will remove all the commits made in the last two days).
Use it if you are the only one using the branch or if the commits haven’t reached the remote repo.

Revert:

Use it anywhere. Afterall it just gonna look like a new commit…
Edit commit messages
Sometimes you make a silly spelling mistake in your most recent commit message and if you want to edit you can re-commit with the --amend flag. You can also add new files and use amend to update the most recent commit snapshot.

git commit --amend -m "new message that doesn't trigger your OCD"

Again, do not amend a commit that is pushed to a remote.

But the --amend the flag will only work for the most recent commit, what if you want to edit a commit message that is older than the most recent commit?

You can use rebase to change older commit messages. Rebase is also used to integrate changes from other branches just like git merge but rebase gives you finer control on how the history of the base branch should look after integrating.

git rebase -i HEAD~n n is the number of commits to go back.

This will open your default editor and show n number of commits with the work “pick” before it.

Now change the word “pick” to “reword” from the commits that you want to change the message. DO NOT CHANGE THE COMMIT MESSAGE HERE YET… Just change the word “pick” -> “reword”.

Save the file and close the editor.

Now you will get a new editor with the commit message. Change it to your new commit message and save it, then close the editor. If you have are trying to reword multiple commits, then you will get each editor windows one by one. That’s it. Remember, rebase will rewrite the history not just update the message so avoid it in remote branches.

Move a commit to a new branch

Sometimes you wanna work on a specific commit but you don’t want to mess with the main/master branch by performing a reset. For this, you can create a branch that contains that specific commit and work there and merge it later.

git branch -b fix
git checkout fix
git reset --hard <commit-hash>

Or you can do it in one line: git checkout -b fix <commit-hash>.

cherry-pick

git cherry-pick is used to pick a specific commit from a branch and place it in the current branch.

Why in the world would I do that?

You created a commit in the wrong branch. Oops.
You want to include a commit from your branch “fix_contrast” to your main/master branch but you don’t want to merge the whole branch just yet.

Checkout to your base branch where you want the picked commit to be and then issue the command.

git cherry-pick <commit-hash-from-other-branch>

This will copy that commit from the target branch to your current/base branch. This will not delete the commit on the target branch, it just makes a copy. This will have the same commit message as the target branch commit message, if you want to edit it you can use -edit the flag to rename it before cherry-picking finishes. Or if you don’t want to commit the target branch commit, use --no-commit the flag to copy only the changes to the working directory.

Even though the reset did remove the commits it looked like it left a phantom of the commits behind…

If the commits are removed then it shouldn’t have its hash right? YES and NO.

Although you reset your branch history every time you make critical actions like reset or merge git stores these actions in a special record called reference logs (aka reflog).

So, if for some reason you think you made a mistake by hard resetting a branch and if you want the commits back, you can get it back.

Issue the command git reflog and it will return the log of all critical mistakes you’ve done so far.

You can again use the git resetto get back to the previous stage but this time the syntax is a little different, git reset HEAD@{n} where n is the number shown in reflog.

And that’s it. You can fix mistakes and fix the fixed mistakes that turned out to be a mistake by themselves. Also, remember to avoid any of these operations on a public branch with multiple collaborators working on it.

There are some other ways too but I only used the above commands for the most part and I hope this article should give you an intro to cleaning up yours's and your colleges’ mess.

DevLog 20250806: "Change Version" - File Changes History-Only Version Control for Binary Assets

Wed, 06 Aug 2025 20:37:18 +0000

Overview

cv is a command-line utility I originally used for personal Unreal Engine projects, where proper version control for binary assets was lacking - but I still wanted to track what had changed, at least at the file-system level. It served its purpose well, but after switching from Unreal Engine to Godot, I haven't needed it as much.

For Divooka, as our repository grows and more GUI-related work gets involved, we've increasingly run into the issue of syncing assets across different workstations. Perhaps it's time to bring this old tool back to life.

Demand

The core requirements are simple:

View file change history - ideally alongside corresponding Git commit history.
Compare differences - similar to Git commits, but focused on files, not content.
Synchronize files between machines.

Regarding the third point, we’ve made some adjustments to fit our workflow:

Since we generally focus on one PR at a time and aren’t concerned with previous builds, collaborators mainly need the latest version of assets - not their version history.
Everything must remain simple, maintainable, and cost zero.

Reference Workflow

Based on my previous experience at a game company working on AAA titles, our typical flow involved:

Git for source code
Perforce for asset versioning
Azure DevOps (ADO) for task tracking

Each PR included a GitHub pull request, a corresponding Perforce commit, and an ADO task - usually cross-referenced manually. Every Perforce commit needed to mention which GitHub PR it related to.

The Implementation

Change Version (cv) is a CLI tool that provides quick check against a repo's changes without saving any changed contents. It does so by recording only the update time. The outputs is just like git status, without diff.

This is useful for cases when we DO NOT want to do version control yet would still want the capability to see which files has changed, as in the case of multimedia projects (e.g. game projects).

Usage

The utility supports the following key commands:

init
status
log
commit -m <Message>

Output Folders & Files

.cv

cv saves history inside the .cv folder. File update time is stored as utc.

.cvignore

cv uses a .cvignore file, which shares the same .gitignore format as git, this is used to decide which files cv should consider when issuing status command; Notice we are using a new file name instead of using .gitignore directly - this is to allow the same place having a git repo.

Ignore Rules

I've decided to follow Git's philosophy: everything is considered "important" by default, unless explicitly ignored.

Here’s a basic implementation:

private static string[]? ReadIgnoreRules()
{
string[]? ignoreRules = null;
if (File.Exists(IgnoreFilename))
ignoreRules = File.ReadAllLines(IgnoreFilename)
.Where(l => !string.IsNullOrWhiteSpace(l) && !l.Trim().StartsWith('#'))
.ToArray(); // Skip empty lines and comments
return ignoreRules;
}
private static bool ShouldIgnore(string[] rules, string path)
=> rules.Any(path.StartsWith);

To support more robust wildcard patterns, here’s an upgraded version:

public class IgnoreRule
{
    #region Properties
    private readonly Regex _regex;
    public bool IsNegation { get; }
    #endregion

    #region Construction
    public IgnoreRule(string pattern)
    {
        // Support for !foo to "unignore"
        if (pattern.StartsWith('!'))
        {
            IsNegation = true;
            pattern = pattern[1..];
        }

        // Anchor pattern at repo root if it starts with '/'
        bool anchored = pattern.StartsWith('/');
        if (anchored) 
            pattern = pattern[1..];

        // Convert Git-style glob to regex
        _regex = new Regex(
            "^" +
            Regex.Escape(pattern)
                 .Replace(@"\*\*/", "(@@SLUG@@/)")
                 .Replace(@"\*\*", ".*")
                 .Replace(@"\*", @"[^/]*")
                 .Replace(@"\?", @"[^/]")
                 .Replace("@@SLUG@@", ".*") +
            (anchored ? "$" : "(?:$|/)"),
            RegexOptions.Compiled | RegexOptions.IgnoreCase
        );
    }
    #endregion

    #region Methods
    public bool IsMatch(string path, string repoRoot)
    {
        // If anchored, the path is relative to repoRoot already
        return _regex.IsMatch(path);
    }
    #endregion
}

At this stage, creating a unit test feels essential.

Based on what we have so far, it seems plausible to implement a layer of file transfer service to achieve synching of files between workstations with the help of an intermediate file server.

File Transfer

I talked about the need for a simple file-uploading service just yesterday. Today’s use case is a bit different, as it's strictly internal - we don’t need to support multi-tenant scenarios and we have a fairly accurate idea of our storage needs. Scalability isn’t a major concern.

Server Options

We’re considering ASP.NET Core vs raw TCP sockets for the server:

Concern	Raw TCP / FTP	HTTP (ASP.NET Core)
Reliability	Requires custom framing, retries	Built-in streaming, timeouts, retries
Security	Must implement SSH/SSL manually	HTTPS support out-of-the-box, auth middleware
Chunking/Resume	Reinvent the protocol	Use `Range` headers, tus.io, etc.
Deployment	Requires FTP daemon, ports, etc.	ASP.NET + Kestrel or NGINX
Extensibility	Hard to add endpoints	Minimal APIs, MVC, SignalR, gRPC, DI, etc.

Because we plan to expose the server on the internet, authentication is a must, even if it adds some programming overhead.

Endpoints

To start, simple endpoints suffice for basic usages:

GET /files: Returns a JSON list of all tracked paths.
GET /files/{**path}: Streams the latest copy of a single file.
PUT /files/{**path}: Uploads or overwrites a file.
DELETE /files/{**path} Removes a file.

Testing the Server

Example usage:

# List files
curl -H "X-Api-Key: my-super-secret-key" https://localhost:5001/files

# Upload a file
curl -X PUT -H "X-Api-Key: my-super-secret-key" --data-binary @myfile.txt https://localhost:5001/files/docs/myfile.txt

# Download a file
curl -H "X-Api-Key: my-super-secret-key" https://localhost:5001/files/docs/myfile.txt -o downloaded.txt

Summary

Today, I focused on:

Modernizing and refactoring code; upgrading the framework
Making the program AoT-compatible
Drafting the setup for file sync and server

A few TODO items still need to be addressed before safe syncing can be implemented.

Notably, the server doesn’t need to understand cv or RepoStorage - its only job is to store the latest version of each file. Maintaining accurate timestamps isn’t practical; instead, storing an MD5 checksum is a better way to avoid unnecessary uploads/downloads.

In an ideal world, we’d have a tool tailored perfectly to our needs. In reality, we often find ourselves choosing between overengineered tools that don’t quite fit or having no suitable tool at all.

When the need is real, it’s worth rolling up your sleeves and investing some time in building the right tooling.

References

Wiki: https://wiki.methodox.io/en/Utilities/CLI/cv
Source: https://github.com/MethodoxTech/cv

Working with git on large projects with large binary assets.

Wed, 16 Jul 2025 04:55:21 +0000

When I started the now-defunct Steel Hunters (RIP), I wrestled with how to best handle version control for the project. Managing Unreal Engine is easy enough, as Epic has structured its repository to be as git friendly as it can be.

The actual game projects built with Unreal Engine: not so much. Especially when it’s a fairly large-scale, high-fidelity project.

Command Line Work

Before I continue — because it will be critical for the rest of this article — let’s talk about using a command line for work.

Up until about a year or so ago, I rarely ever worked with a command line. It was always a last resort or for dealing with projects that were specifically setup to be managed from the command line. I just like my GUIs. And, as far as coding, I wouldn’t trade Visual Studio for anything (though I now routinely use Visual Studio Code for non-C++ or C# work).

I don’t really know what specifically changed that; maybe it was the OS X terminal or PowerShell that made command line work more palatable. And then I discovered ConEmu which made everything even more user-friendly. And recently, in response to my article about my software recommendations, a reader told me about Cmder which, now, I wouldn’t trade for anything. Unless I found something better.

And now that I’ve gotten used to working within the command line, I couldn’t recommend it strongly enough. And, no, you don’t have to go to an extreme with its use; I would never use a command-line text editor, I don’t use it for all of my tasks and needs, and I certainly would never use it to make an ASCII representation of the Joy Machine logo.

The Joy Machine Logo

He’s so cute.

I also have no memory whatsoever of the gamut of commands that I should know by heart. The variety of flags a given command can take are just things I don’t want to take up space in my brain — especially the PowerShell commands, which follow unconventional standards, are verbose as can be, and, in general, just dumb. The same is true of git commands which, while I always remember the basics, can get so crazy complex that there’s no way I want to — ahem — commit them to memory. But that’s what git aliases are for, but that’s for later. For now, here are some resources on the one of my public GitHub repositories (each subfolder in the scripts root folder has its own README file for details, but I'll TL;DR it here:

OS X Terminal Improvements: An .inputrc file for improving the user experience on OS X terminals. This includes making Ctrl+arrows skip entire words, tab-insensitive command/folder completion, and so on and so forth. The source of this file is a gist from Gregory Nicholas, whom I do not know but greatly appreciate his existence.
PowerShell Scripts — These ones were all me and I adore them. For the most part, they’re a variety of aliases for commands that I’ll never remember but use often. The sample profile, however, also scans the cmdlets subfolder for all self-contained PowerShell scripts (any command with some amount of complexity) and adds it to your toolset whenever you load PowerShell. Yay!

Cmder

We had some good times together old friend.

Steel Hunters Version Control: Iteration One

I like git; it’s a great VCS with an intuitive workflow and command set. That said, Perforce has always been the go-to solution for games as it handles binary files (especially large binary files) incredibly well. I’m not a huge fan of Perforce’s GUI client, but it’s workable. That all said: I like git. And Perforce has Helix4Git, which works with Perforce’s standard version control as well as git. It just maintains two separate databases that it mirrors based on your workflow.

You may have already figured out the issue with this: it mirrors your repository contents on your hard drive. And since I never had the money to upgrade to the desktop computer I spec’d out in January, my storage space on my development laptop is precious.

That quickly made it a no-go.

Steel Hunters Version Control: Iteration Two

Okay, this time we’re going straight-up Perforce. I setup a server on Amazon AWS EC2, installed the necessary Perforce server components and all was right with the world.

Except for the fact that Amazon AWS EC2 server costs add up. Quickly. Especially when you have a project that was, at the time, 100 GB without any local versioning whatsoever. The server-side storage costs increased rapidly.

And since I’m working on Steel Hunters and it brings in negative dollars, I have always had a job (I now have a contract gig or two for moneydollars). I simply did not have time to manage server maintenance on top of everything else. Also: I know nothing about server maintenance, so whenever I needed to do something, I ended up in a google rabbit hole with fifty Chrome tabs open, each of which solved one aspect of a series of problems I was dealing with.

So: no-go.

Steel Hunters Version Control: Iteration Three

git (I refuse to capitalize this for some reason) doesn’t handle binary files incredibly well; especially, it doesn’t handle large binary files well. That’s why git-lfs was introduced, which I started using shortly after it was introduced and found its feature set lacking and its general usability poor.

GitHub, while I love it more than I love most sites on the internet, has a limit on the size of the repositories it supports. It’s rough 2gb; so, I could store source files and key binary content files on it, but it still seemed risky.

So: no-go.

Steel Hunters Version Control: Iteration Four

Someone on Twitter told me about Assembla and it sounded absolutely perfect. git and Perforce support for projects. YES. LIFE WAS COMPLETE. I could store source files on git (which I believe had no size limitations) and content files on Perforce (which had a size limitation, but it worked for me).

When I started adding some contractors to my team — contractors whom, due to some miracle of the universe (or simply being great people), were willing to work for free/deferred payment — this dual-VCS setup was confusing, hard to setup, hard to manage properly, and generally just bad mojo.

This was not aided by some issues I was experiencing with Assembla itself; their Perforce setup was actually fairly difficult to reconcile with how Perforce is generally configured. The server-side git support (including git-LFS which I reluctantly used after being told it was much-improved since I last used it) was not up-to-date with recent releases of git and git-lfs. And this led to some incredibly bizarre errors that were fairly time-consuming and difficult to figure out.

So: no-go.

Steel Hunters Version Control: Iteration Five

I then gave Visual Studio Team Services a shot; it was a git-based VCS and it has no limit on repository size. It is also free unless you want to bolt-on some Azure-based continuous integration or other fanciness. I didn’t believe the lack of size limitations, so I contacted customer support and explained that my project was large. Very large. They said: yup, no limit, don’t worry about. “And it’s free?” “Yes, for your purposes it is completely free of charge.”

I’ve been using VSTS for the last five months without a single issue. Its user management is a complete pain, but that’s such an uncommon task that I just accepted it.

The problem that I ran into a couple days ago was absolutely no fault of VSTS. It was great in every way, minus the variety of tools and utilities and integrations that are entirely reliant on the use of GitHub specifically. I still actively recommend it for mid-sized projects.

The problem was that I kept running out of space on my hard drive. And, since I am not smart, it took me until a couple days ago to finally run a disk drive space analysis tool (WinDirStat) on the disk with my Steel Hunters repository on it.

It totaled 380 GB. Of which 140 GB was the project itself, and 240 GB was the .gitfolder itself.

I spent the next day trying everything I could find on the entire internet to try and prune now-irrelevant large files in the history of the repository using git-based commands, other git-based commands, and even BFG. I think I ended up shaving off about 10–20 GB of data from the .git folder. And when I attempted to push the rewritten history to VSTS it would fail. It would fail every time. And there was no combination of solutions that I could find (including the one on that page) that would cause it not to fail.

Steel Hunters Version Control: Now

I approached this problem from an entirely different perspective yesterday. git and git-lfs remain my favorite form of VCS that doesn’t bring with it the cost of Perforce.

A Quick Aside on git-lfs

git-lfs is now included in the Windows installer for git and that is not made obvious anywhere and can lead to multiple installs of git-lfs on your system.

Where Was I…

I wanted to stick with git and git-lfs and, more than that, I wanted to go back to using GitHub. I bought a few “packs” of git-lfs storage on GitHub, which wasn’t ideal, but it wasn’t going to kill my bank account (on top of another aspect of this plan below) since I’ll just give up a few meals a week. And I came up with a new solution for managing our entire project-base:

Steel Hunters Project — All source code, unique DLLs (those necessary to build aspects of the project), and select content files (game data files and, as of this morning, UMAP files due to the importance of versioning for them despite their large size). Yay, happy Trent. Back on GitHub!
Joy Engine (Unreal Engine 4 Custom Fork) — This too would go on GitHub, because it was never a problem child in the first place.
Steel Hunters Source Asset Files — This repository is for managing the source content files that are imported into the Steel Hunters UE4 project. This includes files like PNGs, TGAs, PSDs, FBXs, and so on and so forth. These files are more critical to the project than any UASSET in the game project, which is why I sprung for the additional git-lfs storage packs, as I want them versioned. They aren’t as numerous and heavy as the game project’s content folder, as that folder contains a number of Unreal Engine asset packages that I don’t generally care about exporting the source files of, save special circumstances.

Steel Hunters Project Content Folder

This was the elephants-stacked-on-elephants-stacked-on-giant-cubes-of-solid-lead of the whole dilemma. In general: most of these content files do not change with any frequency. And, if they do change, it’s generally a fairly trivial alteration of a flag, property, or minor edits. Since I always note what changes I make in fair detail in my git commit messages, if anything ever went bad, I could really just read back in the commit history and re-make the changes if need be (through downloading the original asset package or re-importing the source the file).

The solution I ended up with is this: an Amazon AWS S3 bucket for syncing the Steel Hunters Content directory to whenever any files are changed. This folder is managed through a tool that monitors a folder for any modifications and syncs them to the S3 bucket.

Is this absolutely ideal? No. It is, however, the most time-effective way for me to maintain the project until it gets funded properly and I can bring on a team and someone with more experience than me in managing servers (which would make switching to Perforce a viable option, if the need arises).

Since this whole plan was put into effect yesterday, time will tell how well everything works out.

Working with git Like a Rockstar

I don’t pretend to be a git expert, but at this point I’ve gotten so comfortable with my command line-based workflow and built up a fair amount of shortcuts, tools, and utilities to make my day-to-day work effortless. And I don’t keep those a secret.

But first: git GUI applications are universally bad. All of them lack the full set of functionality that you really want for properly managing a repository, for one. And some of the applications themselves are just… bad.

SourceTree, for instance, should be avoided at all costs. I’ve found it buggy, slow, completely lacking the ability to handle large projects, and just a general disappointment. I’m not even going to link it nor give it bold text. That’s what I think of it.
GitKraken will be good some day. It’s just not there yet. It has issues with large projects, its interface, while pretty, is non-intuitive, and it’s lacking some very necessary features for properly managing a repository. As I said, though: I think it’s promising.
GitHub Desktop is an electron app I just found last night and… I like it? I wouldn’t use it for management of a repository or anything, but if I ever want to quickly analyze all of the various local repositories on my computer, I can open it, see some quick diffs, and then close it.
Tower is thoroughly decent, but I’ve encountered a fair amount of issues with it when working at scale.

That said: GUIs for performing diff and merge operations are absolutely crucial. And while P4Merge gets the job done adequately, what you really want is Beyond Compare. Just… trust me. For one thing, Beyond Compare not only does diff and merge operations on git repositories, but it can also do folder/file diff and merge operations even on things that aren’t managed by any sort of version control. It’s wonderful.

NOTE: This is an archive article that I’m updating; the information here may be out of date. GitHub Desktop, for instance, is pretty good. And I’m looking at giving GitKraken another go in a few minutes.

But Just Work with the Command Line (Except the diff/merge Thing)

I really can’t emphasize how useful the git command line is (by the way, when you install on Windows, ensure you install both bash and standard Windows compatibility). It’s powerful, it gives you all the information you need, it does a better job of informing you what is and isn’t managed by git-lfs, it lets you easily pick-and-choose what you include in a given commit, and so on and so forth.

That doesn’t mean it’ll be instantly obvious how to work with it, but that’s why I have some handy files on my repo’s git folder to help out:

[README](https://github.com/trentpolack/joymachine-public/blob/master/git/README.md) — Some general information about what the templates do as well as some general git information to help out.
[gitconfig](https://github.com/trentpolack/joymachine-public/blob/master/git/gitconfig.template) — A whole lot of aliases that make common operations a bit quicker to execute as well as aliases for operations that are not at all trivial to execute. git state, git lard, git prettylog, and git last are my favorites.
Also: I added output colors for some git operations. It was necessary.
[gitattributes](https://github.com/trentpolack/joymachine-public/blob/master/git/gitattributes.template) — Rules for files that I absolutely want stored as text (CPP/H/CS) and some general rules for files I want managed by git-lfs.
[gitignore](https://github.com/trentpolack/joymachine-public/blob/master/git/gitignore.template) — A list of files and extensions that I commonly ignore (focused on UE4 projects).

Conclusion

And that’s my story about version control. fin