Coder Legion - Recent posts tagged graph

Mapping the Territory

Mon, 16 Mar 2026 02:31:52 +0000

We thought the "Multi-Repo Verse" would save us from spaghetti code. Instead, we traded a tangled web of single files for a labyrinth of repositories, modules, and remote state. The boundaries were supposed to give us clarity, but suddenly, understanding a single depends_on rule became an archaeological dig through half a dozen browser tabs. The code was perfectly structured, but the reality was invisible.

A quick recap

In Part 1 of the "Making Invisible Infrastructure Visible" series, I talked about "The Fog of Code": that overwhelming anxiety of flying blind when making changes to modern, distributed infrastructure. Tab Fatigue is real. The fear of the Butterfly Effect is real.

To stop flying blind, I needed a map. Not just a text search across repositories, but a true semantic map of the territory. I needed to turn static syntax into a dynamic, queryable reality.

Here is exactly how I built the foundation of Infra-Graph to do just that.

The harsh reality of parsing HCL

When I first sat down to solve this, the plan felt straightforward: parse the .tf files, find the references, and draw the lines.

Then I actually looked at the Abstract Syntax Tree (AST).

You open a file and see a seemingly simple string: depends_on = [aws_security_group.web.id]. But to a machine, that’s just text. To build a visualization tool, you have to map every variable, module output, and resource attribute into a rigid, structured schema.

I started using python-hcl2 to turn .tf files into Python dictionaries. For simple files, it worked beautifully. But real-world infrastructure isn't simple. When dealing with nested module calls, interpolations, and dynamic blocks, regex fails spectacularly. The AST is a deeply nested, unforgiving nightmare of dictionaries and lists.

To give you an idea of what I was looking at, a simple 6-line resource block like this:

resource "aws_lb" "main" {
  name               = "${var.project_name}-alb-${var.environment}"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = [aws_subnet.public.id]
}

...becomes this deeply nested list-of-dictionaries format in python-hcl2:

{
  "resource": [
    {
      "aws_lb": {
        "main": {
          "name": "${var.project_name}-alb-${var.environment}",
          "internal": false,
          "load_balancer_type": "application",
          "security_groups": [
            "${aws_security_group.alb.id}"
          ],
          "subnets": [
            "${aws_subnet.public.id}"
          ]
        }
      }
    }
  ]
}

The realization hit hard: translating disjointed, human-written infrastructure code into a rigid database schema wasn't going to be a quick hack. It was going to be technical trench warfare.

Designing the Schema

To map the territory, I needed a database that understood relationships natively. Relational databases, with their rigid tables and expensive JOINs, were the wrong tool for a system defined by its interconnectedness. I chose Neo4j because it is built from the ground up for graph traversal, allowing me to naturally model and query recursive dependencies like depends_on chains without writing thousands of lines of SQL CTEs.

I designed the graph schema around three core Neo4j entities:

Nodes: These represent the blocks of HCL.
- A Resource (e.g., aws_instance) uniquely identified by its file_path and logical_name.
- A Module uniquely identified by its module_id.
- Variable and Provider nodes.
Properties: The metadata attached to nodes, housing the actual configuration (names, files, line numbers, and AWS-specific attributes).
Relationships: This is the wiring:
- [:DEPENDS_ON] connects explicit resource references.
- [:USES_VAR] connects resources and modules to the variables they consume.
- [:CONTAINS] links modules to their internal components.

// The explicit wiring of the Multi-Repo Verse
(:Module {id: "vpc"})-[:CONTAINS]->(:Resource {type: "aws_subnet", name: "main"})
(:Resource {type: "aws_subnet"})-[:USES_VAR]->(:Variable {name: "region"})
(:Resource {type: "aws_instance"})-[:DEPENDS_ON]->(:Resource {type: "aws_security_group"})
(:Resource {type: "aws_instance"})-[:PROVIDED_BY]->(:Provider {name: "aws"})

It looks clean on a whiteboard. In practice, it required an intense normalization logic (an internal mapping script) to flatten the chaotic HCL dictionaries into clean, standardized dataclasses before they ever touched the database.

The Mechanical Grind

Populating the graph brought its own set of friction. It wasn't enough to just CREATE nodes. If you run a scan twice, you don't want a duplicated infrastructure graph.

Writing the Cypher ingestion queries was an exercise in idempotency. I lived and breathed the MERGE clause, ensuring the graph updated existing nodes rather than cloning them. I batched transactions to handle hundreds of resources without locking up the database.

MATCH (source:Resource {id: $sid})
MATCH (target:Resource {type: $ttype, name: $tname})
MERGE (source)-[:DEPENDS_ON]->(target)

It was a mechanical, unglamorous grind. Traversing the ASTs, writing the parsing logic, and meticulously wiring the explicit pointers across files felt farther from "saving DevOps" than I had hoped. I was just moving data from text files into a graph.

Connecting the Dots

And then came the moment that made it all worth it.

After days of wrestling with dictionaries and Cypher queries, I wrapped the Neo4j database with a FastAPI backend and built an interactive Angular frontend using D3.js.

I pointed the parser at a sprawling, complex Terraform directory, the kind that normally requires six open editor windows to mentally model, and initiated the first full infrastructure scan.

When the web interface finally rendered, the result was immediate. The unreadable chaos of HCL text files organized itself into a clear visual model. The nodes snapped into a physics-based, force-directed graph. Solid lines perfectly connected the explicit depends_on rules, variable references, and module outputs.

Selecting a node instantly populated the sidebar with its exact configuration, pulling its properties directly from the local Neo4j instance.

The "Tab Fatigue" vanished. The sprawling "Multi-Repo Verse" was no longer an invisible labyrinth; it was a single, zoomable UI. I could trace an interconnected path from an API Gateway to a subnet, without opening a single text file.

I had mapped the territory.

But as I stared at the perfect structural map, I realized something was missing. The map only showed what was written. It didn't show what it was meant. It didn't connect a security group port 80 to the load balancer that implicitly relied on it. It was strictly explicit.

Next up in Part 3: how I integrated local AI to discover the hidden truth.

Making Invisible Infrastructure Visible series index

DevLog 20250805: Divooka Node Graph File Formats

Wed, 06 Aug 2025 00:57:32 +0000

Preamble

When I first envisioned Divooka, I aimed for open source, open file formats, and strong version control friendliness (especially with Git). JSON is verbose and ugly, YAML is perfect but too complex, and formats like CSV/RTS/OBJ are simple and elegant but lack the flexibility needed for complex structures. A custom line-based format seemed like a good middle ground and necessary starting point.

This led to the intention of creating a simple, human-friendly, and easy-to-parse file format - something like TOML. This approach has proven effective in the past; I've parsed similar formats before, and it works well for INI, config, and even Godot scene files.

Canonical TOML Example

# This is a TOML document

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T07:32:00-08:00

[database]
enabled = true
ports = [ 8000, 8001, 8002 ]
data = [ ["delta", "phi"], [3.14] ]
temp_targets = { cpu = 79.5, case = 72.0 }

[servers]

[servers.alpha]
ip = "10.0.0.1"
role = "frontend"

[servers.beta]
ip = "10.0.0.2"
role = "backend"

The only issue I have with TOML is largely aesthetic: it's just not beautiful enough. Objects are too scattered, and it can feel a bit verbose - especially when incorporating type information, as seen in a Godot scene file:

[gd_scene load_steps=2 format=3 uid="uid://dbmf10vsl4g6q"]

[ext_resource type="Script" uid="uid://dcyr50rclg8m1" path="res://scenes/login.gd" id="1_l7r8x"]

[node name="Login" type="Control"]
layout_mode = 3
anchors_preset = 15
anchor_right = 1.0
anchor_bottom = 1.0
grow_horizontal = 2
grow_vertical = 2
script = ExtResource("1_l7r8x")

[node name="VBoxContainer" type="VBoxContainer" parent="."]
layout_mode = 1
anchors_preset = 8
anchor_left = 0.5
anchor_top = 0.5
anchor_right = 0.5
anchor_bottom = 0.5
offset_left = -80.0
offset_top = -80.5
offset_right = 80.0
offset_bottom = 80.5
grow_horizontal = 2
grow_vertical = 2

[node name="Label" type="Label" parent="VBoxContainer"]
layout_mode = 2
theme_override_font_sizes/font_size = 46
text = "Login"
horizontal_alignment = 1

The Two Approaches

Divooka Open Standard (Plain Text/Binary Hybrid)

We’ll eventually discuss the broader Divooka Open Standard, but for now, let’s focus specifically on file formats. My original vision was for something like Blender’s .blend format - except in plain text: a sequence of self-contained data blocks that allow easy extension and are powerful enough to store arbitrary payloads.

I imagined blocks for: document properties, nodes, graphs (including connections), payloads, revisions, etc. Some attributes - such as node positions and sizes - are considered auxiliary and handled through "decorators" rather than being part of the core schema. Perhaps I went too far here: the flexibility results in many small structures holding trivial bits of data. On the bright side, once implemented, this approach yields a format that is both human-readable and programmatically digestible. Interpreters uninterested in positional data, for instance, can simply ignore those decorators.

One of the early design decisions was to keep all attributes as strings. The idea was to rely on parsing for interpretation and avoid enforcing types directly. Much like the web, this promotes forward compatibility and long-term document stability. I still stand by this: nothing is easier to parse than plain text.

That said, I made a mistake in implementing everything strictly in this format: the string-based format only makes sense during (de)serialization. Once in memory, the data should be converted into strongly typed structures before any meaningful work is done. Otherwise, everything becomes cluttered with unnecessary boilerplate.

PV1 Neo Document Format (Binary)

PV1 Neo takes a more opinionated, object-oriented approach. Documents contain graphs, which in turn contain nodes and connections. Each of these is represented as a strongly typed object. By default, nodes include layout properties like position and size.

This results in a neat OOP hierarchy where everything is properly defined and serialized. The upside is ease of development - you simply call Serialize() at the top level and the rest follows. However, the downside is that parsing becomes nearly impossible without full knowledge of the exact object types. Any misalignment in type expectations results in binary stream reading errors.

Derivations

Since Divooka graphs can already be represented in text via the Open Standard, I’ve proposed two derived, minimalist formats: MiniParcel and MicroParcel. These are lightweight, CLI-friendly formats designed for shell use. The rationale is that since nodes can represent abstract functions or endpoints, users should be able to manipulate them without relying on a GUI.

Default:
    FileList = GetFiles "{tempFolderPath}"
    FileRenameList = Enhance @FileList !#Rename
    Apply @FileRenameList !MoveFile
Rename:
    FileName = GetOnlyName (GetArgument :0)
    Return (Replace @FileName ":File_Sequence_00(.)" ":File-Sequence-00$1")

Conclusion

As we finalize prototyping and start committing to core design decisions, file format is one area requiring careful long-term thinking. There’s an inherent trade-off between programmability and readability/portability - especially since Divooka documents may eventually hold a vast variety of data.

In the foreseeable future, both formats may coexist. In fact, it might make sense to support both for the long term. As Divooka matures and gains better standardization, I’m hopeful we’ll arrive at a robust format that truly supports the long-term vision.

References

Rationale for Godot’s custom INI format: GitHub discussion
Divooka Document Specification: future link