Repository navigation
feat(billing): 持久化用量 outbox + 对账,fail-closed(T-8) - #374
Conversation
2b249ce to
3fb60a0
Compare
20df61e to
f7816a6
Compare
e07d56f to
dfdfeea
Compare
Settling a metered turn was two writes with only the second one durable:
"the provider has been paid, here is what it cost" was held in memory while
"take it out of the balance" hit the ledger. A crash, a Firestore blip or a
full disk between them lost the charge with nothing on disk remembering why
the numbers stopped matching — the gap PROJECT_REVIEW_TECH_v0.24.0 T-8 names
(`outbox|reconcil` was 0 hits across src/billing and src/web).
The outbox makes the FIRST write durable and the second one replayable, so
every state the process can die in is either "no charge and no record" or "a
record the reconciler can settle exactly once" (INVARIANTS 计费与交易 §1/§5):
append(pending) -> debit(balance, eventId) -> update(committed)
append fails => FAIL CLOSED: nothing charged, permit released, the caller
gets a retryable BillingStateError("outbox_unavailable")
debit fails => the event is parked `failed` with a redacted lastError and
the error is rethrown; the reconciler retries it
mark fails => NOT an error: the money is already right. The record stays
open and the reconciler closes it.
Idempotency lives in the LEDGER, not in the outbox. quota.ts `debitTurn` now
takes the event id, checks it against and records it in a bounded `settled`
ring INSIDE the same atomic update as the money (Firestore CAS or the file
lock), and returns whether it actually charged. A replay is therefore refused
by the thing holding the money rather than by a caller who remembered to
check. That is what makes the separate "mark committed" write safe:
firestore.ts exposes CAS only — there is no runTransaction in this client — so
the brief's "same transaction" option does not exist here and its documented
fallback (immediate mark + ledger idempotency key) is what shipped.
A malformed `settled` ring fails the balance CLOSED rather than being dropped:
silently forgetting applied ids is exactly what would let a replay double
charge. Past SETTLED_REPLAY_WINDOW_MS the reconciler escalates instead of
re-applying, because the ring may no longer remember the id and "re-apply" and
"charge again" would be indistinguishable.
Stores sit behind one interface: JSONL under the tenant billing dir for the
local edition (append-only, last line per id wins, appends and compaction under
the tenant billing lock so a torn write costs at most the newest transition),
and Firestore for cloud — the event created id-keyed with exists:false so
creation is idempotent, plus a per-tenant open index because this client has no
query API, and a sticky sharded tenant registry that costs zero writes in the
steady state and is pruned lazily by the reconciler.
The reconciler re-applies pending events past a 5-minute grace period (younger
ones may still be owned by a live settlement elsewhere) and failed ones under a
5-attempt cap, then escalates to needs_human with a structured ERROR carrying
the event id, redacted uid and the amount at stake. It throws rather than
report a clean sweep over a store it could not list.
Wiring is one line each at the two admission seams, so every metered path
(chat, gateway, reflection, birth, voice, autonomy) inherits it without
touching a handler: recordUsage stays the best-effort AUDIT append and
settleUsage owns the money. The permit now carries a reservationId linking the
event back to the admission that ran it.
LISA_BILLING_OUTBOX=0 disables ONLY the outbox write; the pre-T-8 fail-closed
debit path keeps its exact behaviour, asserted in test (h).
Tests were written first and inject every torn state: append throws, debit
throws after a durable append, mark-committed throws, a simulated crash between
append and debit, and a duplicate replay — asserting no double charge, no lost
charge after reconciliation, the permit released on every failure path, and
that neither the logs nor lastError carry a raw uid or a bearer token. The
JSONL adapter is exercised end to end against the real balance ledger in a
temp dir; nothing touches a real Firestore, GCS, Stripe or Apple endpoint.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit f2ac9af260f04d9c36d2bdb019e9fd80f406a54f)
The outbox is only half a compensator until something sweeps it. Two minimal seams, deliberately kept to the smallest possible diff because other streams are editing both files: - server.ts: one gated `startBillingReconciler()` beside the other background timers. Cloud only — the Mac edition has no accounts, so nothing is ever metered there. The timers are unref'd and a pass that throws is logged, not fatal: reconciliation must never be why a process refuses to exit, and never why serve dies. With Firestore on, each cycle is claimed through a lease so N instances do one sweep, not N (safe either way — the ledger key refuses the double — but N sweeps would N-plicate every escalation alert). - cli/account.ts: `billing reconcile` dispatches before the managed-session check, because it reads THIS host's ledger rather than calling the cloud API — an operator command, not a user one. The implementation lives in src/billing/reconcile.ts; this is only the dispatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 55d517f346407d80287df45ebaf2e36d0aefd6c3)
The compensator is only useful if the person paged at 3am knows what the counters mean and which lever is safe to pull, so the section is written for that reader rather than as a design summary: what the outbox is and why the two-write settlement needed one, how to read a reconcile report counter by counter (with what each SHOULD look like), the two ways an event reaches needs_human and the ordered recipe for each, and a manual repair recipe. Two warnings are load-bearing and stated explicitly because both invert into lost or doubled money: never hand-edit the balance's `settled` ring (it is the idempotency key — deleting an entry turns a replay into a second charge), and `--resolve` closes a record WITHOUT debiting, so correcting the balance comes first if the charge is still owed. LISA_BILLING_OUTBOX is documented as NOT a kill switch — turning it off restores the pre-T-8 "crash loses the charge" behaviour, which is the opposite of what someone reaching for a kill switch wants; LISA_BILLING_KILL is. Inserted as §7 next to the other monitoring material (§6), renumbering the two sections after it, plus one smoke-checklist line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 7b714271ff6d246fee7b4bd30138fa5085685379)
`lisa billing reconcile --dry-run` died in the global arg parser with "unknown flag: --dry-run" before ever reaching the handler: `billing` was neither a raw nor a passthrough subcommand, so any trailing flag was treated as a global one. Found by running the command the runbook tells an operator to run first. `billing` now matches the existing PASSTHROUGH_SUBCOMMANDS definition exactly — "subcommands whose handler re-parses all of its trailing args itself" — which cmdBilling does as of the reconcile dispatch. One token, and the same mechanism `mail` and `kb` already use. The regression test spells out why this one matters rather than asserting the mechanism generically: a swallowed --dry-run silently turns "show me what you would do to the ledger" into "go do it". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 9f1745618a16d86f490ebf2b493142a7c131a730)
f7816a6 to
fe007d6
Compare
评审 — 钱的路径逐条验证;无需改动已 rebase 到 main,base 改为 幂等键:设计与实现都对
环损坏时 fail closed( 环的边界与对账窗口的关系是推导出来的,不是硬编码的: 对账器在 至于 回退验证:幂等守卫是真被测到的把 这条测试不是摆设。 fail-closed 与 permit 释放:逐条走过
关于 PR 描述里"绝不允许扣了 provider 的钱但没有记录"——我一开始怀疑这句站不住(settle 发生在 provider 调用之后,此时失败已经无法撤销那次调用)。但看清顺序后它是成立的: 审计账本 一个可记录的观察(非缺陷):outbox 持续不可用时,turn 照常跑完、文本已流式发出,用户实际获得了免费推理。这需要 Firestore/磁盘故障才会触发,不是攻击者可控的,风险很低。 多实例与生命周期
运维命令实测输出把"会重试"和"已升级需人工"分开,运维读得懂。 验证
本 PR 我没有改动任何代码——查下来没有需要修的地方。 🤖 Generated with Claude Code |
This PR's coverage gate failed on src/billing/ functions: 83.23% against an 85% floor, and the script says to add tests rather than lower it. The drag is outbox.ts at 61.7% — #374's Firestore adapter, which the local suite cannot exercise without an emulator. Most of that class stays uncovered, but five of its helpers are pure and each encodes an invariant money depends on, so they are exported and tested: - toDoc/fromDoc round-trip a usage event unchanged. Specifically: Firestore rejects an undefined field, so `lastError` is written as "" and dropped on read — a round-tripped event must not come back carrying a falsy lastError that a later reader treats as "this already failed once". - tenantShard is deterministic and stays inside 0..7. A shard that drifted would strand a tenant's open events where no sweep looks. - readIds survives a malformed array field instead of throwing mid-sweep. - isAlreadyExists accepts only a real FirestoreError 409/412. A duck-typed look-alike must NOT pass: that classification is what decides whether an append failure means "already durable" or "fail the settlement closed". src/billing/ functions: 83.23% → 87.43%, floors met without moving one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
第 6 / 8 条,实现 T-8。基于 #373。
问题
计费此前存在一个无法恢复的窗口:provider 已经收费,但余额提交失败或进程在两次写之间崩溃,就既没有账也没有补偿路径。仓库里
outbox|reconcil零命中,v0.21 审查把它列为后续项,一直未做。做法
先写测试。 故障注入覆盖五种情形:outbox 追加失败、追加成功后余额提交失败、标记 committed 失败、追加与提交之间崩溃、同一事件重放。断言:不重复扣费、对账后不丢费、(a) 情形 fail closed、每条失败路径都释放 permit、日志不含密钥。
Outbox。 每次结算前先追加一条不可变用量事件(本地为 JSONL + 索引,Cloud 为以事件 id 为文档 id 的 Firestore 集合,天然幂等)。写事件失败即结算失败:释放 permit、记录不含敏感信息的日志、向调用方返回可重试错误。绝不允许"扣了 provider 的钱但没有记录"。
提交。 支持事务的存储里,标记 committed 与余额变更在同一事务内完成;否则紧接余额写入之后,并以事件 id 作为账本幂等键,使重放无法二次扣费。
对账器。 扫描超过宽限期的 pending 与未超重试上限的 failed,幂等地重放余额提交;超过上限则升级为
needs_human并打结构化 ERROR。startBillingReconciler()在 serve 启动后 30 秒首跑、之后每 15 分钟一次(unref 定时器,仅在计费启用时)。运维命令lisa billing reconcile [--dry-run] [--json] [--uid] [--resolve <id>]。开关
LISA_BILLING_OUTBOX默认开启;设为 0 只关闭 outbox 写入,不影响既有的 fail-closed 检查。docs/RUNBOOK_CLOUD_PROD.md新增一节:如何读报告、如何处理 needs_human、手工修复流程。验证
全部测试离线运行,使用内存与临时目录 fake,不接触任何真实 Firestore / GCS / Stripe / Apple 端点。
🤖 Generated with Claude Code