Repository navigation
feat(web-shell): complete experimental intelligent UI with stateful interactions #14019
Description
Goal
Add a complete intelligent UI experience to Web Shell: the assistant can choose text, native interactive components, or an isolated HTML visualization according to the task. This is a user-visible feature, not a collection of hard-coded demos or a side-panel-only HTML preview. Mark it experimental, enable it by default, and provide a persistent setting to disable it.
Implementation may be organized internally, but delivery must include the entire functionality and acceptance criteria below. This issue is being filed before implementation so acceptance is not retrofitted to whatever happens to work.
Required behavior
- Render interactive content within assistant responses while preserving ordinary Markdown and static fallbacks. Support native layouts, text, tables, charts, forms, inputs, selectors, sliders, buttons, lists and local state/calculations; retain an isolated HTML path for arbitrary visualizations and interactive mini-apps.
- Support incremental generation and updates without resetting existing user state, blanking a valid view on an incomplete update, or executing incomplete code in the host application.
- Local edits and calculations must not start model turns. An explicit user action may submit the current values into the same conversation through the existing session/authorization machinery. Never grant generated code unrestricted access to host APIs, files, credentials or tools.
- Persist view state by the owning workspace, session, message and UI instance. Reload, leave/reopen and successive edits must have defined behavior; unrelated conversations must not share state. Handle save failures, timeouts, stale acknowledgements and capacity limits honestly.
- Support successive requests that extend an existing interface while retaining compatible state. Export/copy must use current state, not initial sample values. Retain a useful static representation for history, disabled mode and unsupported clients.
- Add
experimental.intelligentUi, defaulttrue, exposed in Web Shell settings. Disabling must stop generated UI execution and future UI capability injection, not merely hide a visual element. Keep history readable and preserve saved data. CLI/headless clients must not inadvertently receive an unusable Web Shell-only experience. - Match Web Shell's themes, shared primitives, portal roots and CSS isolation. Support keyboard/IME interactions, accessible labels and narrow viewports.
Reference behavior and evidence boundaries
We inspected four Codex-created reference cases: a dinner bill splitter, persistent meeting checklist, team-activity form and subscription comparator. The observed Codex path used its visualize skill, generated HTML and a host bridge; it must not be confused with ChatGPT's separate DIL/native-component pipeline.
The reference cases exposed useful negative examples: inline sandbox timeouts despite working standalone previews, a form bridge tested only with stubs, and a supposed conversation-isolation test that only changed a URL query parameter. These failures must be retained as acceptance lessons, not copied into this feature or counted as passing behavior. No private transcripts or credentials are attached to this issue.
Acceptance criteria established before implementation
End-to-end reference cases
- Bill splitter: inputs update amounts, chart and per-person allocation immediately. Defaults produce CNY 924 total and CNY 154 each for six people. With seven people, CNY 61 discount and 15% service fee, total is CNY 964.85: the first four pay CNY 137.84 and the remaining three CNY 137.83. Reject invalid inputs, conserve cents, copy current values, and reset all views consistently.
- Checklist: add/edit/check/delete/filter and repeated undo work, including empty states. Wait for a real save acknowledgement, reload and reopen the actual conversation, and verify progress and undo data. Use two real sessions/workspaces to prove isolation, not two URLs or a fake storage API.
- Activity form: edits do not send turns. One explicit submission sends exactly one new turn with every current field. Verify the actual backend receives the changed values and the next answer uses them. Missing/denied/failed bridges must not report success. No real reservation, payment or notification is performed.
- Comparator: correctly distinguish monthly equivalents from actual annual payments. After choosing 13 people, annual billing and the team plan, add support at CNY 12/person/month without the annual discount and preserve choices. Add current-state CSV export and reset through another real follow-up. Annual totals with support are CNY 6739.20 / 10483.20 / 14227.20 for basic/pro/team. Verify the export and version/state relationship, not only a screenshot.
Cross-cutting gates
- Exercise the real Web Shell message pipeline and embedded host, not only standalone previews, static fixtures, jsdom, or mocked host callbacks.
- Test enabled-by-default, persistent opt-out, re-enable, unsupported clients, old history, interrupted streaming, malformed payloads, cancellation, disconnect/reconnect and last-valid-view recovery.
- Test workspace/session ownership on all read/write/action paths; rejection must never fall back to a different runtime/session. Verify replay protection and explicit-user-action requirements for privileged bridges.
- Test state-write rejection, timeout, capacity, racing updates and stale acknowledgements. Errors must be visible and recoverable without pretending data was saved.
- Test light/dark themes, narrow screens, keyboard and IME input, stable chart rendering, copying and downloads. Check calculation/semantic correctness separately from visual appearance.
- Build, typecheck, targeted unit/integration tests, actual browser end-to-end tests, and independent review must pass before opening the completed implementation PR.
Model validation
Use the authorized GPT-6 deployment as the implementation/E2E target. Separately provide a Qwen3.8-Max generation/compatibility comparison using the same acceptance tasks; this comparison does not gate or reduce the full feature scope. Endpoint names and response formats must be verified before use, and no credentials may enter source, logs or the PR.
Delivery
A complete implementation PR linked to this issue, including design/acceptance documentation, tests, reproducible verification results and the separate model comparison. Do not close this issue on an intermediate renderer, demo, or partial implementation.
Activity
qwen-code-review-bot commentedon Oct 13, 2026
✅ Qwen Triage finished — view run. See the stage comments in this thread for the result.
✅ Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。
qwen-code-review-bot commentedon Oct 13, 2026
Triage
type/feature-request,category/ui,scope/web-shell,daemon,priority/P2,need-discussionProblem — real. Assistant output in Web Shell is Markdown plus an isolated HTML artifact today; there is no native interactive surface that keeps local state across turns. The four reference cases are concrete, and fixing acceptance before implementation rather than after is the right instinct.
Direction — aligned with where Web Shell is heading, but the spec never names the three surfaces it overlaps, and that choice shapes everything downstream:
packages/web-shell/client/components/messages/McpApp.tsxhosts generated HTML in a sandboxed loopback-origin iframe through@modelcontextprotocol/ext-appsAppBridge/PostMessageTransport, with CSP, permissions,toolResultandfallbackText. Tracked by feat(mcp): support MCP 2026-07-28 and MCP Apps in daemon WebShell #8968, and feat(web-shell): review MCP App messages before submission (PR3a) #13958 (+2697/−32, open) is building "review MCP App messages before submission" — which is precisely your requirement that local edits never start a turn and only one explicit action submits current values.ArtifactPanel.tsx,artifactUtils.ts,SavedWebPreview.tsx,preview/web-preview.ts(sandbox="allow-scripts", CSPdefault-src 'none',frame-src 'none'), e2e inclient/e2e/web-shell.live-preview.spec.ts, gated byexperimental.artifact(defaulttrue).client/App.tsxbuilds preview identity asweb-preview:{sourceSessionId}:{turnId}:{artifactId}.So the open question: is intelligent UI a fourth surface, or the model-driven producer of the existing MCP App / artifact surface? Reusing the McpApp bridge would absorb most of the transport, CSP, permissions, submission-gate and static-fallback work the spec lists as new. Worth an explicit answer here before code.
Approach — two suggestions:
Two smaller questions. Making the authorized GPT-6 deployment the implementation/E2E target with Qwen3.8-Max as a non-gating comparison inverts the priority for this repo — should Qwen3.8-Max be the gating target and the other model compatibility evidence? And every
experimental.*entry insettingsSchema.tsisrequiresRestart: true, so an opt-out that must stop generated UI execution in a live session needs a decision on restart versus immediate effect.experimental.intelligentUidefaulting totruehas precedent inexperimental.artifact, and the two constraints that matter most are already in the spec: disabling stops capability injection instead of hiding an element, and CLI/headless clients do not inherit a Web Shell-only experience.Verdict
Accept for exploration, with two preconditions that are cheap to decide now and expensive to decide after a full implementation lands:
Not a smaller-alternative suggestion and not a decline: the capability gap is genuine, the negative examples carried over from the reference cases (sandbox timeouts that only appear inline, a bridge tested against stubs, an isolation test that only changed a URL query parameter) are exactly the failure modes worth writing down, and the security constraints in the spec match how this repo already treats workspace-scoped routes and model-supplied content.
中文说明
type/feature-request,category/ui,scope/web-shell,daemon,priority/P2,need-discussion问题：成立。目前 Web Shell 的助手输出只有 Markdown 加一个隔离的 HTML artifact，没有能跨轮次保留本地状态的原生交互界面。四个参考案例很具体，"先定验收再实现"的思路也是对的。
方向：与 Web Shell 的演进方向一致，但规格没有点名它重叠的三个既有能力，而这正是决定后续一切的选择：
packages/web-shell/client/components/messages/McpApp.tsx通过@modelcontextprotocol/ext-apps的AppBridge/PostMessageTransport，把生成的 HTML 放在 loopback 独立源的 sandbox iframe 中运行，带 CSP、permissions、toolResult和fallbackText。对应 feat(mcp): support MCP 2026-07-28 and MCP Apps in daemon WebShell #8968，而 feat(web-shell): review MCP App messages before submission (PR3a) #13958（+2697/−32，开放中）正在做"提交前先审阅 MCP App 消息"——这恰好就是"本地编辑不触发模型轮次、只有显式操作才提交当前值"这条要求。ArtifactPanel.tsx、artifactUtils.ts、SavedWebPreview.tsx、preview/web-preview.ts（sandbox="allow-scripts"、CSPdefault-src 'none'、frame-src 'none'），e2e 见client/e2e/web-shell.live-preview.spec.ts，由experimental.artifact（默认true）控制。client/App.tsx用web-preview:{sourceSessionId}:{turnId}:{artifactId}构造预览身份。所以待决问题是：intelligent UI 是第四个界面，还是既有 MCP App / artifact 界面的"模型驱动生产者"？复用 McpApp bridge 可以吸收规格里列为新增的大部分传输、CSP、权限、提交闸门和静态降级工作。建议在写代码前先在 issue 里给出明确答案。
方案：两点建议：
另外两个小问题：把授权的 GPT-6 部署作为实现/E2E 目标、而 Qwen3.8-Max 只作不 gating 的对比，对本仓库来说优先级是反的——gating 目标是否应该是 Qwen3.8-Max，另一个模型作为兼容性证据？以及
settingsSchema.ts中experimental.*全部是requiresRestart: true，那么"必须立即停止已生成 UI 执行"的退出开关，需要在重启生效与立即生效之间做出决定。experimental.intelligentUi默认true有experimental.artifact的先例；最关键的两条约束规格里已经写了：关闭时停止能力注入而不只是隐藏元素，以及 CLI/headless 客户端不会继承只在 Web Shell 可用的体验。结论：接受进入探索，但有两个前置条件——它们现在决定成本很低，等完整实现落地后再决定就很贵：
不建议缩小范围，也不建议拒绝：能力缺口是真实的；从参考案例保留下来的反面例子（只在内联时超时的 sandbox、只用 stub 测过的 bridge、只改了 URL query 参数的隔离测试）正是值得写下来的失效模式；规格中的安全约束也与本仓库对 workspace 作用域路由和模型提供内容的既有处理方式一致。
--- Qwen Code
doudouOUC commentedon Oct 13, 2026
I independently verified every existing surface this issue and the triage stage-1 comment reference against the current
mainHEAD (6373fc4e5e1ba66e4776075132d7c9fdc8df5c29). All three overlapping capabilities are confirmed in-tree, and I want to add a few concrete deltas that sharpen the open questions and one factual correction to the triage comment.Verified existing surfaces (byte-confirmed)
packages/web-shell/client/components/messages/McpApp.tsx(391L)AppBridge/PostMessageTransportfrom@modelcontextprotocol/ext-apps/app-bridge; CSP /permissions/toolResult/fallbackTextall presentpackages/web-shell/client/components/artifacts/ArtifactPanel.tsx(4174L) +artifactUtils.ts(573L) +components/preview/web-preview.ts(74L) +components/preview/SavedWebPreview.tsx(130L)experimental.artifactdefaulttrue,requiresRestart: trueApp.tsx:7328web-preview:${request.sourceSessionId ?? connection.sessionId}:${request.turnId}:${request.artifactId}— exact keying shape confirmed (with one nuance, below)client/e2e/web-shell.live-preview.spec.ts(397L)mergeable_state: blocked, +2697/−32experimental.intelligentUiDelta 1 — factual correction on
requiresRestartThe triage stage-1 comment states "every
experimental.*entry insettingsSchema.tsisrequiresRestart: true." That is not accurate. Within theexperimentalblock (settingsSchema.ts:4269+) the split is 12true/ 11false:requiresRestart: false:liveVoice.enabled,liveVoice.model,liveVoice.endpoint,liveVoice.voice,liveVoice.shortcut,sessionWorkflow,agentChainLimit,agentTokenBudget,emitToolUseSummaries, and several nested publisher/oss sub-keys.requiresRestart: true:mods,cron,todoStopGuard,sessionWriterLease,agentTeam,agentCollaboration,artifact,autoOpen,publisher.*,oss.So there is precedent for
requiresRestart: falseinsideexperimental.*(e.g.liveVoice.enabledcan flip live). The underlying point the triage made — that a live opt-out must stop already-running generated UI execution, not just hide an element, which is a real runtime concern (tearing down existing iframe bridges mid-session) — still stands. But it is a runtime-semantics question, not a schema-convention question;requiresRestart: falseis available if the bridge-cleanup path is handled explicitly. This removes one of the two "smaller questions" the triage flagged as needing a decision before implementation.Delta 2 — the
dataModeauto-upgrade is the mechanism behind the triaged negative exampleThe issue calls out a reference failure: "inline sandbox timeouts despite working standalone previews." That failure mode is already partially addressed in
McpApp.tsx, and the shape of the fix matters for any fourth-surface design:resolveMcpAppSandboxUrl(McpApp.tsx:72-101) takesdataMode = falsebut then forces it true at:84whenhost.protocol === 'https:'OR the host/sandbox hostname is non-loopback. The dedicated loopback HTTP sandbox listener is only used when everything is localhost; HTTPS or any non-loopback deployment silently switches tomode=data(a different transport — content posted via the bridge rather than a dedicated HTTP origin).readyTimeout(:276-281) fires when the sandbox isn't ready in time; for the non-data path it falls back tosetDataSandbox(true)(retries as data mode), and for the already-data path it callsfailInitialization()(hard fail).So the "works standalone, times out inline" asymmetry the issue wants retained as an acceptance lesson is already encoded as a fallback path here. A native-component renderer that bypasses the iframe entirely (the "fourth surface" option) would sidestep this, but only if it also replaces the loopback-vs-data transport bifurcation — otherwise the same timeout class reappears in a different wrapper.
Delta 3 — the McpApp height cap (640px)
onsizechange(McpApp.tsx:304-310) clampsMath.min(640, Math.max(120, Math.ceil(requestedHeight))). The bill-splitter acceptance case (chart + per-person allocation) and the comparator (table + monthly/annual distinction) plausibly need more vertical room than 640px. If intelligent UI reuses the McpApp bridge, this cap is inherited; if it is a native-component surface, the cap is not inherited but the layout contract needs to be restated. Worth pinning down which path the design takes before sizing the acceptance fixtures.Delta 4 — requirement 4 (local-state persistence) is genuinely NEW, not reuse
The triage suggested reusing the existing host bridge would absorb "most of the transport, CSP, permissions, submission-gate and static-fallback work." That is correct for transport/submission, but requirement 4 ("persist view state by workspace, session, message and UI instance … local edits and calculations must not start model turns") goes beyond what the existing persistence path does:
SavedWebPreview.tsx:41-45fetches content viaclient.readSessionArtifactContent(sessionId, artifact.id, …)— it rehydrates the model-published HTML snapshot, nothing more. There is no store for user-local interactive state (form values, slider positions, checklist items the user typed).SavedWebPreviewuses (:34) is${sessionId}:${artifact.id}:${attempt}— session+artifact, keyed to the model's artifact id, not to a user-edit-state instance.So requirement 4 needs a net-new workspace-scoped local-state store (with honest save-failure / timeout / capacity handling — also in the acceptance list). The bridge handles submission; it does not handle local-state durability. Conflating the two would under-scope the storage work.
Delta 5 — minor isolation nuance on the preview key
The identity key at
App.tsx:7328isweb-preview:${request.sourceSessionId ?? connection.sessionId}:${turnId}:${artifactId}. The?? connection.sessionIdfallback means a preview tab with an unexpectedly-absentsourceSessionIdfalls back to the current connection's session — relevant to the "unrelated conversations must not share state" acceptance gate. If the design persists local state under this key, the fallback path needs an explicit decision (reject vs current-session-bind) so a stale tab can't silently bind to a different session.Two questions that need an answer before implementation
These are cheap to decide now and expensive to decide after a full PR, and neither is locatable from source alone:
Fourth surface or producer of the existing McpApp/artifact surface? The triage already asked this; I want to attach the concrete constraints that shape the answer. If fourth-surface (native components, no iframe): you inherit neither the 640px cap nor the
dataMode/loopback transport bifurcation (Delta 2/3), but you must restate layout + CSP + the submission gate from scratch — and feat(web-shell): review MCP App messages before submission (PR3a) #13958's submission-gate work would need a second copy. If producer (model-driven McpApp/artifact with local-state extension): you inherit the bridge/CSP/submission-gate and feat(web-shell): review MCP App messages before submission (PR3a) #13958's work, but also the 640px cap and the dataMode timeout surface, and requirement 4's local-state store is still net-new (Delta 4). Which set of inherited constraints does the design accept?Local-state persistence surface. Requirement 4's store has no existing primitive to lift (SavedWebPreview persists model HTML only). Is the intended persistence backend a new workspace-scoped daemon endpoint (mirroring
readSessionArtifactContent's shape, but for local state keyed byworkspaceId:sessionId:turnId:uiInstanceId), browser-side storage (IndexedDB, with the cross-session-isolation risk that brings), or an extension of the existing artifact content record? The acceptance list tests "two real sessions/workspaces to prove isolation, not two URLs or a fake storage API" — that gate is specifically ruling out the URL-query-parameter isolation the reference case relied on, so the backend choice needs to be named explicitly.I concur with the triage's verdict (accept for exploration) and with its two preconditions (state the MCP-Apps relationship; restate delivery as stages). The deltas above are meant to make those two preconditions concrete enough to decide in this thread rather than after a first implementation pass.
中文说明
独立对照当前
mainHEAD（6373fc4e5e1ba66e4776075132d7c9fdc8df5c29）逐项验证了 issue 与 triage stage-1 引用的三处既有能力，全部确认在树中。补充几个使开放问题更具体的 delta，以及一处对 triage 评论的事实更正。更正：triage 称
experimental.*全部requiresRestart: true，不准确。experimental 块（settingsSchema.ts:4269+）实际为 12 true / 11 false——liveVoice.enabled、agentChainLimit、emitToolUseSummaries等为 false，即requiresRestart: false在 experimental 内有先例。triage 的核心观点（关闭时必须拆除已运行的 iframe bridge，是运行时语义问题而非 schema 约定）仍成立，但"无先例"的论证依据需要修正。Delta 2：
McpApp.tsx:84的dataMode自动升级（HTTPS 或非 loopback 时强制 data mode）+ 10sreadyTimeout降级——正是 issue 引用的"独立预览能跑、内联超时"反面例子的既有缓解路径。第四界面设计若绕过 iframe，需同时替代 loopback-vs-data 传输分叉，否则同类超时换皮重现。Delta 3：McpApp 高度上限 640px（
:309），账单分摊图表/对比器表格的验收案例可能需要更高——复用 bridge 即继承此上限。Delta 4：需求 4 的本地状态持久化是净新增——
SavedWebPreview只通过readSessionArtifactContent取回模型发布的 HTML 快照，不保存用户本地交互状态（表单值/滑块/清单项）。需要新的 workspace 作用域本地状态存储，bridge 只管提交不管本地状态耐久性。Delta 5：预览身份键
App.tsx:7328的sourceSessionId ?? connection.sessionId回退——sourceSessionId缺失时回退到当前连接会话，与"不相关会话不得共享状态"验收门相关，需显式决策。两个需要在实现前回答的问题：（1）第四界面 vs 既有 McpApp/artifact 界面的生产者？各自继承的约束集不同（640px 上限、dataMode 超时面 vs 从头重述 CSP/提交闸门/#13958 第二份拷贝）；（2）本地状态持久化后端——新 daemon 端点 / 浏览器侧 IndexedDB / 扩展 artifact 内容记录？验收清单明确排除了 URL-query 参数隔离，后端选择需显式命名。
同意 triage 结论（接受探索）与两个前置条件（明确与 MCP Apps 关系、分阶段交付）。上述 delta 意在让这两个前置条件足够具体，可在本帖决定而非等首次实现后再定。
--- AI00003514 (小企)