Skip to content

fix(mentions): a corrupt mention_threads.json must not brick server startup - #690

Merged
devikaverma merged 3 commits into
andrewyng:mainfrom
Lesereingrape:fix/mention-store-clean
Sep 30, 2026
Merged

devikaverma merged 3 commits into
andrewyng:mainfrom
Lesereingrape:fix/mention-store-clean

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

Supersedes #682 — that branch was cut from a stale fork base and shows ~5k lines of unrelated
sandbox/ files in its diff. This one is 3 commits off current main (5870585), +56/-2,
two files: coworker/mentions.py and a new test.

Problem

MentionSessionStore._load() is the one JSON state store built by SessionManager.__init__ that
still has no corruption guard:

    def _load(self) -> None:
        if self.path and self.path.is_file():
            data = json.loads(self.path.read_text(encoding="utf-8"))
            self._threads = [MentionThread(**raw) for raw in data.get("threads", [])]

It is constructed unconditionally while the server boots —
coworker/server/run.py:main() → build_app() → SessionManager.__init__ →
manager.py:421 MentionSessionStore(base / "mention_threads.json") — and main() wraps that in
try: … finally: with no except. So one unreadable file raises straight out of the constructor
and openworker-server (also the desktop app's sidecar) never starts again, on every launch, until
the user guesses the filename and deletes it. #204 lists this store as affected; #158 was the same
failure for wakes.json.

Two independent triggers, both re-executed against the real module on 5870585:

$ python -c "…MentionSessionStore(torn_file)…"          # _save() interrupted mid-write
  File "…\coworker\mentions.py", line 37, in __init__
    self._load()
  File "…\coworker\mentions.py", line 41, in _load
    data = json.loads(self.path.read_text(encoding="utf-8"))
json.decoder.JSONDecodeError: Unterminated string starting at: line 2 column 21 (char 34)

$ python -c "…MentionSessionStore(file_without_'channel')…"   # a record predating a field
  File "…\coworker\mentions.py", line 37, in __init__
    self._load()
  File "…\coworker\mentions.py", line 42, in _load
    self._threads = [MentionThread(**raw) for raw in data.get("threads", [])]
TypeError: MentionThread.__init__() missing 1 required positional argument: 'channel'

The first one is self-inflicted by this store's own _save(), which writes in place with
write_text (no temp + os.replace), so a crash, SIGKILL or a full disk mid-write produces
exactly the truncated document that then bricks startup. The second is the ordinary consequence of
adding a field to MentionThread: every existing install's file stops loading.

Fix

Guard the load the way its immediate neighbour in the same constructor block already does — this is
coworker/connectors/parked.py:49-55 verbatim in shape, including the comment:

    def _load(self) -> None:
        if self.path and self.path.is_file():
            try:
                data = json.loads(self.path.read_text(encoding="utf-8"))
                self._threads = [
                    MentionThread(**raw) for raw in data.get("threads", [])
                ]
            except (OSError, ValueError, TypeError):
                self._threads = []  # a corrupt file must never block startup

Deliberately kept to the read side: SubscriptionStore's buffer, the people directory,
ParkedStore and ConversationStore._read_jsonl all made this same call, so the contract in this
repo is already "start, and let the next _save() overwrite the unreadable file". The test below
covers that recovery step.

I did not change _save() to write atomically, even though #204 also asks for that: it is a
separate failure mode (it creates the corruption instead of surviving it), #70 already landed the
tmp-then-os.replace pattern for the path that needed it most, and bundling it here would make this
diff two behaviour changes instead of one. Happy to add it as a follow-up if you would rather have
both.

I also scoped this to one store rather than to #204's whole table on purpose: #205 proposed a shared
coworker/jsonstore.py helper for all of them and was closed unmerged, so this PR follows the
piecemeal shape the repo has actually been using. mentions.py is the entry in that table that is
still unguarded and on the boot path.

Tests

New tests/test_mention_store_robustness.py, in the shape of tests/test_conversation_jsonl_robustness.py:

  • test_load_survives_a_torn_thread_file — truncated document starts empty, and the next set()
    makes the file readable again for a fresh store instance.
  • test_load_survives_a_record_with_a_missing_field — a record predating channel no longer raises.
  • test_load_reads_a_valid_file — control: the normal path is untouched.
before the fix (this test file against `main`'s coworker/mentions.py):
  2 failed, 1 passed in 0.55s        # JSONDecodeError / TypeError, tracebacks above
after the fix:
  3 passed

Adjacent suites on this branch:

  • pytest tests/test_mention_router.py tests/test_mention_store_robustness.py tests/test_conversation_jsonl_robustness.py -q → 17 passed in 10.57s.
  • Every test file that mentions the store (grep -l mention tests/*.py, 19 files):
    10 failed, 301 passed in 244.01s. Those 10 are pre-existing on Windows and not caused by this
    change: re-running exactly those 10 node ids with 5870585's coworker/mentions.py (patch
    reverted) gives 10 failed, 14 passed in 20.48s — the same ids (tests/test_slack_relay.py
    websocket timeouts ×6, tests/test_security_bundles.py ×2, tests/test_configured_spawn.py ×1,
    tests/test_github_installs.py ×1).
  • ruff check and ruff format --check (ruff 0.15.17) are clean on both files; the wrapped
    comprehension above is ruff format's output.

Local env: CPython 3.12.11 on Windows, pytest 9.1.1. CI runs pytest tests -q --cov=coworker on
Linux, which is where those 10 should be green — I could not get the whole suite to finish here (it
stalls past ~84% in the connector/browser modules), so the numbers above are the scoped runs I did
complete, not the full gate.

Per the README's "attach what was broken and how it is fixed now": this is a headless boot-time
crash with no UI surface, so the before/after evidence is the two tracebacks and the red→green test
output rather than screenshots.

MentionSessionStore._load() is the one JSON store built by
SessionManager.__init__ with no corruption guard, so a torn file (the
non-atomic _save leaves one behind after a crash, SIGKILL or a full
disk) raises JSONDecodeError out of the constructor and openworker-server
never starts again until the user finds and deletes the file by hand.
A record written before a field was added raises TypeError the same way.

Use the guard the neighbouring ParkedStore in the same constructor block
already applies.
Covers the valid-file control, a torn document, and a record predating a
field addition, plus recovery: the next set() rewrites the unreadable file.
The wrapped comprehension is what `ruff format` produces for the line; the
sibling store files (connectors/parked.py, subscriptions.py, inbox.py) are
already format-clean.

@devikaverma devikaverma left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for redoing this off current main. Agree with keeping the atomic write out of this one.

Two things to follow up on separately, not here:

  • A file that's valid JSON but not an object ([], null) still crashes on .get with AttributeError, which isn't in the except tuple.
  • This isn't the last unguarded store. inbox, inbox_routing, subscriptions, unrouted, unattended, connections, overrides and selfwake still load without a guard on main, so the "this one was the outlier" line in the test docstring isn't right.

@devikaverma
devikaverma merged commit 3ed35da into andrewyng:main Sep 30, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants