Skip to content

feat(billing): 持久化用量 outbox + 对账,fail-closed(T-8) - #374

Merged
oratis merged 4 commits into
mainfrom
claude/opt-billing
Sep 7, 2026
Merged

oratis merged 4 commits into
mainfrom
claude/opt-billing

Conversation

@oratis

@oratis oratis commented Sep 7, 2026

Copy link
Copy Markdown
Owner

第 6 / 8 条,实现 T-8。基于 #373。

问题

计费此前存在一个无法恢复的窗口:provider 已经收费,但余额提交失败或进程在两次写之间崩溃,就既没有账也没有补偿路径。仓库里 outbox|reconcil 零命中,v0.21 审查把它列为后续项,一直未做。

做法

先写测试。 故障注入覆盖五种情形:outbox 追加失败、追加成功后余额提交失败、标记 committed 失败、追加与提交之间崩溃、同一事件重放。断言:不重复扣费、对账后不丢费、(a) 情形 fail closed、每条失败路径都释放 permit、日志不含密钥。

Outbox。 每次结算前先追加一条不可变用量事件(本地为 JSONL + 索引,Cloud 为以事件 id 为文档 id 的 Firestore 集合,天然幂等)。写事件失败即结算失败:释放 permit、记录不含敏感信息的日志、向调用方返回可重试错误。绝不允许"扣了 provider 的钱但没有记录"。

提交。 支持事务的存储里,标记 committed 与余额变更在同一事务内完成;否则紧接余额写入之后,并以事件 id 作为账本幂等键,使重放无法二次扣费。

对账器。 扫描超过宽限期的 pending 与未超重试上限的 failed,幂等地重放余额提交;超过上限则升级为 needs_human 并打结构化 ERROR。startBillingReconciler() 在 serve 启动后 30 秒首跑、之后每 15 分钟一次(unref 定时器,仅在计费启用时)。运维命令 lisa billing reconcile [--dry-run] [--json] [--uid] [--resolve <id>]。

开关 LISA_BILLING_OUTBOX 默认开启;设为 0 只关闭 outbox 写入,不影响既有的 fail-closed 检查。

docs/RUNBOOK_CLOUD_PROD.md 新增一节:如何读报告、如何处理 needs_human、手工修复流程。

验证

全部测试离线运行,使用内存与临时目录 fake,不接触任何真实 Firestore / GCS / Stripe / Apple 端点。

🤖 Generated with Claude Code

oratis and others added 4 commits September 7, 2026 23:28
Settling a metered turn was two writes with only the second one durable:
"the provider has been paid, here is what it cost" was held in memory while
"take it out of the balance" hit the ledger. A crash, a Firestore blip or a
full disk between them lost the charge with nothing on disk remembering why
the numbers stopped matching — the gap PROJECT_REVIEW_TECH_v0.24.0 T-8 names
(`outbox|reconcil` was 0 hits across src/billing and src/web).

The outbox makes the FIRST write durable and the second one replayable, so
every state the process can die in is either "no charge and no record" or "a
record the reconciler can settle exactly once" (INVARIANTS 计费与交易 §1/§5):

  append(pending) -> debit(balance, eventId) -> update(committed)
    append fails  => FAIL CLOSED: nothing charged, permit released, the caller
                     gets a retryable BillingStateError("outbox_unavailable")
    debit fails   => the event is parked `failed` with a redacted lastError and
                     the error is rethrown; the reconciler retries it
    mark fails    => NOT an error: the money is already right. The record stays
                     open and the reconciler closes it.

Idempotency lives in the LEDGER, not in the outbox. quota.ts `debitTurn` now
takes the event id, checks it against and records it in a bounded `settled`
ring INSIDE the same atomic update as the money (Firestore CAS or the file
lock), and returns whether it actually charged. A replay is therefore refused
by the thing holding the money rather than by a caller who remembered to
check. That is what makes the separate "mark committed" write safe:
firestore.ts exposes CAS only — there is no runTransaction in this client — so
the brief's "same transaction" option does not exist here and its documented
fallback (immediate mark + ledger idempotency key) is what shipped.

A malformed `settled` ring fails the balance CLOSED rather than being dropped:
silently forgetting applied ids is exactly what would let a replay double
charge. Past SETTLED_REPLAY_WINDOW_MS the reconciler escalates instead of
re-applying, because the ring may no longer remember the id and "re-apply" and
"charge again" would be indistinguishable.

Stores sit behind one interface: JSONL under the tenant billing dir for the
local edition (append-only, last line per id wins, appends and compaction under
the tenant billing lock so a torn write costs at most the newest transition),
and Firestore for cloud — the event created id-keyed with exists:false so
creation is idempotent, plus a per-tenant open index because this client has no
query API, and a sticky sharded tenant registry that costs zero writes in the
steady state and is pruned lazily by the reconciler.

The reconciler re-applies pending events past a 5-minute grace period (younger
ones may still be owned by a live settlement elsewhere) and failed ones under a
5-attempt cap, then escalates to needs_human with a structured ERROR carrying
the event id, redacted uid and the amount at stake. It throws rather than
report a clean sweep over a store it could not list.

Wiring is one line each at the two admission seams, so every metered path
(chat, gateway, reflection, birth, voice, autonomy) inherits it without
touching a handler: recordUsage stays the best-effort AUDIT append and
settleUsage owns the money. The permit now carries a reservationId linking the
event back to the admission that ran it.

LISA_BILLING_OUTBOX=0 disables ONLY the outbox write; the pre-T-8 fail-closed
debit path keeps its exact behaviour, asserted in test (h).

Tests were written first and inject every torn state: append throws, debit
throws after a durable append, mark-committed throws, a simulated crash between
append and debit, and a duplicate replay — asserting no double charge, no lost
charge after reconciliation, the permit released on every failure path, and
that neither the logs nor lastError carry a raw uid or a bearer token. The
JSONL adapter is exercised end to end against the real balance ledger in a
temp dir; nothing touches a real Firestore, GCS, Stripe or Apple endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit f2ac9af260f04d9c36d2bdb019e9fd80f406a54f)
The outbox is only half a compensator until something sweeps it. Two minimal
seams, deliberately kept to the smallest possible diff because other streams
are editing both files:

- server.ts: one gated `startBillingReconciler()` beside the other background
  timers. Cloud only — the Mac edition has no accounts, so nothing is ever
  metered there. The timers are unref'd and a pass that throws is logged, not
  fatal: reconciliation must never be why a process refuses to exit, and never
  why serve dies. With Firestore on, each cycle is claimed through a lease so
  N instances do one sweep, not N (safe either way — the ledger key refuses
  the double — but N sweeps would N-plicate every escalation alert).

- cli/account.ts: `billing reconcile` dispatches before the managed-session
  check, because it reads THIS host's ledger rather than calling the cloud
  API — an operator command, not a user one. The implementation lives in
  src/billing/reconcile.ts; this is only the dispatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 55d517f346407d80287df45ebaf2e36d0aefd6c3)
The compensator is only useful if the person paged at 3am knows what the
counters mean and which lever is safe to pull, so the section is written for
that reader rather than as a design summary: what the outbox is and why the
two-write settlement needed one, how to read a reconcile report counter by
counter (with what each SHOULD look like), the two ways an event reaches
needs_human and the ordered recipe for each, and a manual repair recipe.

Two warnings are load-bearing and stated explicitly because both invert into
lost or doubled money: never hand-edit the balance's `settled` ring (it is the
idempotency key — deleting an entry turns a replay into a second charge), and
`--resolve` closes a record WITHOUT debiting, so correcting the balance comes
first if the charge is still owed.

LISA_BILLING_OUTBOX is documented as NOT a kill switch — turning it off
restores the pre-T-8 "crash loses the charge" behaviour, which is the opposite
of what someone reaching for a kill switch wants; LISA_BILLING_KILL is.

Inserted as §7 next to the other monitoring material (§6), renumbering the two
sections after it, plus one smoke-checklist line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 7b714271ff6d246fee7b4bd30138fa5085685379)
`lisa billing reconcile --dry-run` died in the global arg parser with
"unknown flag: --dry-run" before ever reaching the handler: `billing` was
neither a raw nor a passthrough subcommand, so any trailing flag was treated
as a global one. Found by running the command the runbook tells an operator
to run first.

`billing` now matches the existing PASSTHROUGH_SUBCOMMANDS definition exactly
— "subcommands whose handler re-parses all of its trailing args itself" —
which cmdBilling does as of the reconcile dispatch. One token, and the same
mechanism `mail` and `kb` already use.

The regression test spells out why this one matters rather than asserting the
mechanism generically: a swallowed --dry-run silently turns "show me what you
would do to the ledger" into "go do it".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 9f1745618a16d86f490ebf2b493142a7c131a730)
@oratis
oratis force-pushed the claude/opt-billing branch from f7816a6 to fe007d6 Compare September 7, 2026 15:33
@oratis
oratis changed the base branch from claude/opt-server to main September 7, 2026 15:33
@oratis

oratis commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

评审 — 钱的路径逐条验证;无需改动

已 rebase 到 main,base 改为 main,无冲突。这是全栈唯一直接碰钱的 PR,所以我按"能双花吗 / 能丢账吗 / 失败时钱在谁那边"三个问题逐条查,而不是通读。

幂等键:设计与实现都对

settled 环作为幂等键放在 updateBalance 的同一次原子更新内,而不是靠调用方记得先查——这是关键的正确选择:CAS 路径下检查与写入不可分。

环损坏时 fail closed(balance_corrupt)而不是静默丢弃,理由在注释里写明了:丢弃会让对账器把一个它再也认不出的重放当成新事件二次扣费。这个判断是对的。

环的边界与对账窗口的关系是推导出来的,不是硬编码的:

SETTLED_TTL_MS         = 168.0h
SETTLED_REPLAY_WINDOW  = 84.0h   (= SETTLED_TTL_MS / 2)
replay < ttl           ✓

对账器在 now - createdAt > SETTLED_REPLAY_WINDOW_MS 时升级为 needs_human 而不是重放,注释说得很准确:"账本可能已经不记得这个 id,此时'补记'和'二次扣费'无法区分,交给人判断。" 2× 余量足够。

至于 SETTLED_MAX = 1000 这个尺寸上限没有时间保证——理论上单个用户在对账窗口内跑满 1000 个计费 turn 就会把 id 挤出环。但 balance 是按 uid 隔离的(lisa-balances/{uid} / per-uid home),单个用户 15 分钟内 1000 个 turn 不现实,可以接受。

回退验证:幂等守卫是真被测到的

把 settledContains 改成恒返回 false:

--- 幂等守卫已禁用 ---  ℹ pass 91  ℹ fail 1
  ✖ debitTurn(eventId) applies once; a replay returns false and leaves the balance alone
--- 恢复 ---           ℹ pass 92  ℹ fail 0

这条测试不是摆设。

fail-closed 与 permit 释放:逐条走过

settleUsage 在 outbox append 失败时记录脱敏日志(redactId(uid))并抛 outbox_unavailable,未扣费。调用侧我把四条路径都看了——birth / chat / reflect / media——释放都在覆盖 settle 的 finally 里,没有漏网。reflect 还把 BillingStateError 映射成 503 billing_state_unavailable,这是正确的 fail-closed HTTP 语义。

关于 PR 描述里"绝不允许扣了 provider 的钱但没有记录"——我一开始怀疑这句站不住(settle 发生在 provider 调用之后,此时失败已经无法撤销那次调用)。但看清顺序后它是成立的:

settle: async (...) => {
  const record = await recordUsage(...);   // 审计追加,先行,best-effort 不抛
  await settleUsage({...});                 // 才是钱
}

审计账本 usage.jsonl 先落。所以 outbox 不可用时,记录仍在、只是余额没扣——方向是保守的(用户不会被多扣,成本由运营方吸收),且带 ERROR 日志。成立。

一个可记录的观察(非缺陷):outbox 持续不可用时,turn 照常跑完、文本已流式发出,用户实际获得了免费推理。这需要 Firestore/磁盘故障才会触发,不是攻击者可控的,风险很低。

多实例与生命周期

  • 两个定时器都 unref() ✓——不会吊住进程。
  • defaultRunLock 用 Firestore 租约避免多个 Cloud Run 实例并发对账。这正是这类部署最容易翻车的地方(obs(billing): dedupe anomaly alerts across instances #366 刚为异常告警解决过同一类问题),这里一开始就做对了。
  • startBillingReconciler() 只在 isCloud() 下启动——Mac 版没有账户,本就不计费。

运维命令实测

$ lisa billing reconcile --dry-run
dry run — nothing was written across 0 tenant(s)
  scanned: 0  committed: 0  failed: 0 (will retry, under the 5-attempt cap)
  escalated: 0  skipped: 0

输出把"会重试"和"已升级需人工"分开,运维读得懂。RUNBOOK_CLOUD_PROD.md 也补了怎么读报告、怎么处置 needs_human 的章节。

验证

npm run typecheck 通过;src/billing/*.test.ts 92 个全过(含五种故障注入:append 失败、append 成功但余额提交失败、标记 committed 失败、两次写之间崩溃、同一事件重放);npm test 1954 个,1953 通过,0 失败,1 跳过(PTY,Node 26 已知)。

本 PR 我没有改动任何代码——查下来没有需要修的地方。

🤖 Generated with Claude Code

@oratis
oratis merged commit 7a2ea8b into main Sep 7, 2026
1 check passed
oratis added a commit that referenced this pull request Sep 7, 2026
This PR's coverage gate failed on src/billing/ functions: 83.23% against an
85% floor, and the script says to add tests rather than lower it. The drag is
outbox.ts at 61.7% — #374's Firestore adapter, which the local suite cannot
exercise without an emulator.

Most of that class stays uncovered, but five of its helpers are pure and each
encodes an invariant money depends on, so they are exported and tested:

- toDoc/fromDoc round-trip a usage event unchanged. Specifically: Firestore
  rejects an undefined field, so `lastError` is written as "" and dropped on
  read — a round-tripped event must not come back carrying a falsy lastError
  that a later reader treats as "this already failed once".
- tenantShard is deterministic and stays inside 0..7. A shard that drifted
  would strand a tenant's open events where no sweep looks.
- readIds survives a malformed array field instead of throwing mid-sweep.
- isAlreadyExists accepts only a real FirestoreError 409/412. A duck-typed
  look-alike must NOT pass: that classification is what decides whether an
  append failure means "already durable" or "fail the settlement closed".

src/billing/ functions: 83.23% → 87.43%, floors met without moving one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant