DEV Community

Cover image for The model wrote six sentences about a grandmother. Two were allowed on the altar.
Sanketh Kumar
Sanketh Kumar

Posted on

The model wrote six sentences about a grandmother. Two were allowed on the altar.

The brief this week listed a "digital ofrenda builder" as one of its ideas. I had been circling the same thought all week, so I took it as permission.

An ofrenda is the altar you build for someone who has died: marigolds, candles, a photograph, and the things you want to remember about them. Most of it you arrange by hand. The written part is the part that scares me, because a small language model writing about a real person will invent a detail and hand it to you in exactly the same confident tone as the true ones.

So I built ofrenda. It writes the remembrance offline with a local open-weight model, and it is not allowed to put a sentence on the page that the memories do not support.

ollama pull llama3.2:1b
python3 ofrenda.py --memories examples/memories.json --out altar.html
🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025 🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025

One command, one self-contained HTML file, no API key, no account, no network. The six memories in that example file are invented, which is why I can show them here. The tool is for the ones that aren't.

The failure I designed against

Ask llama3.2:1b for a sentence about a grandmother's marigolds and you may get this:

She kept marigolds in terracotta pots along the kitchen window. [1]

Or this:

She grew up in Oaxaca and kept twelve pots on the sill. [1]

The second one is a false memory about someone's grandmother, formatted identically to the first, on a page you were about to send to your family. Hallucination is a nuisance in a chat window. In a memorial it is the entire failure — the one thing the tool cannot do is be confidently wrong about a person who is gone.

The standard answer is "use a bigger model." I don't think that's an answer. A frontier model invents less often, not never, and it invents with better grammar, which makes it harder to catch. The fix has to be structural.

The rule: a sentence must cite the memory it came from

The prompt numbers the memories and asks for a citation on every line — [1], [2][3]. Then Python checks that the citation actually supports what the sentence claims. It looks for the details that turn a general memory into a specific false one:

def unsupported_claims(sentence, memories, person):
    if not sentence.citations:
        return ["no citation"]                       # uncited, gone

    cited = [by_index[i].text for i in sentence.citations if i in by_index]
    if not cited:
        return [f"cites memories that do not exist: {sentence.citations}"]

    problems = []
    for position, word in enumerate(sentence.text.split()):
        cleaned = word.strip(".,;:!?\"'()[]")

        if re.fullmatch(r"\d+", cleaned):             # "12"
            if not _supported(cleaned, cited):
                problems.append(f"number {cleaned}")
            continue
        if cleaned.lower() in NUMBER_WORDS:           # "twelve"
            if not _supported(cleaned, cited) and not _supported(NUMBER_WORDS[cleaned.lower()], cited):
                problems.append(f"number {cleaned}")
            continue

        if position == 0 or not cleaned[0].isupper(): # not a proper noun
            continue
        if cleaned.lower() in SAFE_CAPITALS:          # "She", "Sunday"...
            continue
        if cleaned.lower() in person.lower():         # her own name
            continue
        if not _supported(cleaned, cited):
            problems.append(f'"{cleaned}"')

    return problems
🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025 🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025

_supported matches case-insensitively as a substring, so a memory that says "marigolds" supports "marigold" and one that says "Sunday" supports "Sundays". Deliberately loose in the direction of not rejecting fair paraphrases, strict about names and numbers.

Anything that fails gets one repair attempt against only the facts it cited, and then it is dropped.

The rule I had to add after watching that one fail

The first version of this passed its own tests and then failed the moment I ran it for real.

The model wrote six sentences. Two of them were fine. One of them was this:

She sang while she worked, even when the task seemed impossible. [4]

Memory 4 is the one about October candles and a photograph on the table. Memory 3 is the one about singing while she worked. The sentence was true and the citation was wrong, and every check above waved it through: there is no invented name in it, no invented number, and the memory it cited obviously exists. Nothing in the sentence was false. It was just attached to the wrong fact, which for a memorial is the same betrayal wearing a different coat.

A citation is not a claim of provenance if nothing checks it against the text. So there is a third rule: the sentence has to share at least one content word with the memory it cites.

claimed = content_words(sentence.text)
claimed -= set(person.lower().split())
supported_words = set()
for source in cited:
    supported_words |= content_words(source)
if claimed and not any(
    words_match(claim, word) for claim in claimed for word in supported_words
):
    problems.append(NO_SHARED_WORDING)
🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025 🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025

content_words drops stopwords, and two words count as the same when one is a prefix of the other, so "baked" matches "bake" and "singing" matches "sing". Irregular verbs like "sang" do not match — that errs towards refusing, which is the direction I want. Re-running the same prompt, the same model, the same memories, the singing sentence is now refused:

it cites [4], but shares no wording with that memory

Two of six sentences survived. The other four are on the page anyway, struck through, with the reason next to each one.

The finished altar: two sentences, the six memories they had to come from, and the four the check refused

The part I care about most: the refusals stay visible

A guard that silently deletes is indistinguishable from a guard that deletes nothing. You cannot audit what you cannot see, and a memorial page is exactly the wrong place to ask for trust.

So the page has a section called What it was not allowed to say. It shows each refused sentence struck through, and under it the memory it cited and what was wrong. That is where a reader learns that the model produced "Elena's love for life was evident in every detail" with no citation at all, and that it cited the tin of buttons for a sentence about October.

What it was not allowed to say: four refused sentences and the exact reason for each

Naming the memory in the reason was a late fix, and it matters more than it sounds. My first version said only "October" was unsupported. The sentence was true — memory 4 does say October — so the refusal looked like a bug in my tool rather than a mistake by the model. The sentence had cited [5], the tin of buttons. Once the reason says which memory, the page is honest about what happened, and the person reading it can tell the difference between a bad model and a bad guard.

And the one it still missed

The second kept sentence is this:

Her kitchen was always filled with the sweet scent of freshly baked goods. [3]

Memory 3 says she sang while she worked, in the kitchen, in the garden, even when the work was hard. It does not say anything about baking. The sentence passed because it shares one word — kitchen — with the memory it cites. The bread is in memory 2, and the model cited the wrong one again.

I could tighten the rule until this is caught: require most content words to match, or the rarest one. But every step in that direction starts refusing fair paraphrases, and a tool that refuses the true sentence about your grandmother because you phrased it differently is a worse tool. So I stopped, wrote the limit down, and left the miss visible on the page — which is the whole reason the page shows you every memory next to the sentence that used it. The guard narrows what the model can get away with. Reading the result is still the last line of defence, and the design should make that easy rather than pretend it isn't needed.

Why local, specifically

Three reasons, in the order they mattered to me.

The memories are the most personal text a person owns. Sending them to a hosted model means someone else's servers hold a copy of what you remember about your grandmother, under terms nobody reads. That is not a trade I want to make on someone's behalf, and it is not one this tool asks them to make.

It works with the internet off. Not as a gimmick — a memorial is something you might want to open in ten years, on a laptop, at a kitchen table, without an account that has since been deleted. The output is one HTML file with no external fonts, scripts, or images. It opens from disk, in ten years, on a machine with nothing installed.

You can inspect the limits of a model you run yourself. llama3.2:1b is 1.3 GB and noticeably worse at prose than a frontier model. That is exactly why this project looks the way it does: the guard exists because I can see how small the model is and I know what it will do. A model behind an API is a model whose failure modes you find out about after you publish.

Small and checked beats large and trusted, for this job.

What I'd tell someone building the same thing

  • Validate provenance, not format. A well-formed citation is a generated token like any other. Mine cited a real memory, in the right syntax, about the wrong thing.
  • The first version of a guard passes its own tests and fails on the first real run. Mine did, twice — once for wrong attribution, once for a reason that didn't name the memory and so read as a bug.
  • Show the refusals. It is the difference between a tool you can audit and a tool you have to believe.
  • Write down what the guard misses. Mine misses generic sentiment that shares a word with the memory it cites. Knowing that is more useful than a claim of correctness.

Try it

git clone https://github.com/sank8-2/ofrenda
cd ofrenda
python3 -m unittest -v test_ofrenda     # 61 tests, no model, no network

python3 ofrenda.py --memories examples/memories.json --out altar.html --dry-run
python3 ofrenda.py --memories examples/memories.json --out altar.html
🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025 🔥 ข่าวด่วน: sank/the model wrote six sentences about a grandmother two were allowed on the altar nn - Kapook Update 2025

--dry-run renders from a canned reply so you can see the layout without waiting, and one sentence in that reply deliberately invents a place and a number so you can watch the verifier refuse a lie.

On this laptop the first run took 90 seconds and a warm one 13. It is a 1B model on a CPU, and that is the honest cost of keeping the memories at home.

Code: github.com/sank8-2/ofrenda — MIT, one file, standard library only. Bring your own memories; the tool is more useful that way.

Top comments (0)