Skip to content

feat(media): model image, video, external video, and 3D variants - #690

Merged
igrigorik merged 4 commits into
mainfrom
feat/media-sources
Sep 29, 2026
Merged

igrigorik merged 4 commits into
mainfrom
feat/media-sources

Conversation

@igrigorik

@igrigorik igrigorik commented Aug 6, 2026 •

Copy link
Copy Markdown
Contributor

The released Media shape is image-oriented: one URL and one pair of dimensions cannot describe alternate encodings and sizes of an image, alternate renditions or adaptive-streaming manifests of a video, device-specific 3D formats, a poster still, or a video that exists only inside a third-party player. This PR extends the open Media type — one schema, discriminated by type — following the schema.org MediaObject family while preserving the released wire contract.

  • Adds sources on every media item: alternate renditions of the resource at url — image encodings and sizes, video renditions and HLS/DASH manifests, 3D file formats — each with mime_type, format, dimensions, and filesize.
  • Adds preview, a poster or thumbnail still, and name, a title distinct from accessibility-oriented alt_text.
  • Adds duration in seconds for time-based media.
  • Adds external_video as a well-known type for a video presented through a third-party player. Its url is the player; video.url is always a playable rendition.
  • Adds per-type description schemas (media_image, media_video, media_external_video, media_model_3d) selected by open if/then composition. Every field is defined once on media.json; the per-type files refine descriptions only, so generated types are complete.
  • A Platform MAY refuse to load or embed a media URL that does not use the https scheme.
  • A Platform that embeds an external_video player MUST isolate it so that it cannot navigate the top-level document or otherwise act as the Platform; the web mechanism (a sandboxed, credentialless iframe) is illustrated, not mandated.
  • Keeps type open. A Platform MUST NOT reject a product or response for an unrecognized media type; it MAY present the item from preview and name, or omit it. Platforms SHOULD treat the first media item they present as featured.

Image with alternate encodings

{
  "type": "image",
  "url": "https://cdn.example.com/products/runner-pro-blue.jpg",
  "alt_text": "Runner Pro in Blue",
  "width": 1600,
  "height": 1600,
  "sources": [
    { "url": "https://cdn.example.com/products/runner-pro-blue.avif", "mime_type": "image/avif", "width": 1600, "height": 1600 },
    { "url": "https://cdn.example.com/products/runner-pro-blue.jpg", "mime_type": "image/jpeg", "width": 1600, "height": 1600 }
  ]
}

Video

url is a playable rendition and appears in sources; a manifest lets the Platform stream adaptively.

{
  "type": "video",
  "url": "https://cdn.example.com/video-720.mp4",
  "name": "Product demonstration",
  "duration": 24,
  "width": 1280,
  "height": 720,
  "preview": { "url": "https://cdn.example.com/video-poster.jpg", "width": 1280, "height": 720 },
  "sources": [
    { "url": "https://cdn.example.com/video-720.mp4", "mime_type": "video/mp4", "width": 1280, "height": 720 },
    { "url": "https://cdn.example.com/video.m3u8", "mime_type": "application/vnd.apple.mpegurl" }
  ]
}

External video

The video exists only inside a third-party player, so url is the player and there are no renditions. A Platform that does not embed third-party players presents preview or omits the item.

{
  "type": "external_video",
  "url": "https://videos.example.com/embed/123",
  "name": "Runner Pro trail review",
  "duration": 212,
  "preview": { "url": "https://cdn.example.com/review-poster.jpg", "width": 1280, "height": 720 }
}

3D model

{
  "type": "model_3d",
  "url": "https://cdn.example.com/product.glb",
  "preview": { "url": "https://cdn.example.com/product-poster.jpg" },
  "sources": [
    { "url": "https://cdn.example.com/product.glb", "mime_type": "model/gltf-binary" },
    { "url": "https://cdn.example.com/product.usdz", "mime_type": "model/vnd.usdz+zip" }
  ]
}

Backward compatibility

Additive for conforming uses of the released Media contract:

  • The required set remains type and url; no field is removed, renamed, or retyped; all new fields are optional; unknown type values remain valid.
  • url keeps its released meaning — the media resource the Platform renders, now stated per type — and width/height keep their released meaning as intrinsic pixel dimensions.
  • Media URLs carry no scheme constraint, matching every other content URL in UCP today. A Platform MAY refuse non-https media. Requiring https on content URLs is a breaking change and is proposed separately as a protocol-wide sweep of every URL a Platform renders, links, or embeds.

Also in this PR: schema_fields (docs macro) renders a description-only per-type refinement with the type and requirement of the base field it refines, instead of any / Optional; the idiom is documented in schema-authoring.md under Per-Value Field Semantics.


Checklist

  • Capability: New schemas (Discovery, Cart, etc.) or extensions. (Requires Maintainer approval)
  • Documentation: Updates to README, or documentations regarding schema or capabilities.
  • I have followed the Contributing Guide
  • I have updated the documentation (if applicable).
  • My changes pass all local linting and formatting checks.

@igrigorik igrigorik added this to the Working Draft milestone Aug 6, 2026
@igrigorik
igrigorik requested a review from gsmith85 August 6, 2026 16:27
@igrigorik igrigorik self-assigned this Aug 6, 2026
@igrigorik igrigorik added the gov:tc-meeting Proposed for Technical Council live discussion label Aug 6, 2026
@damaz91 damaz91 added status:under-review gov:needs-tc-review Requires review and approval from the Technical Council labels Aug 11, 2026
@damaz91 damaz91 added status:stale-review Applied if a PR is waiting on a reviewer for too long and removed status:under-review labels Sep 1, 2026
Comment thread docs/specification/catalog/index.md Outdated
Comment thread source/schemas/common/types/media_video.json
Comment thread source/schemas/common/types/media_video.json Outdated
@igrigorik

Copy link
Copy Markdown
Contributor Author

@amithanda thanks, great feedback! Addressed, and with benefit of fresh eyes, caught a few additional issues. PTAL.

@igrigorik
igrigorik requested a review from amithanda September 24, 2026 04:43
@damaz91 damaz91 added the area:shopping Issues and pull requests related to the Shopping vertical label Sep 24, 2026
@amithanda

Copy link
Copy Markdown
Contributor

Thanks for the quick turnaround, the updates address all three of my threads (verified locally: http:, javascript:, and data: URLs are now rejected on url, preview.url, and sources[].url, and unknown type values still validate).

One thing to reconcile before merge: the PR description still reflects the first revision. It describes externally hosted video as type: "video" plus embed_url, says "a Platform must ignore media types it does not recognize", and the Backward compatibility section says "existing payloads continue to validate." With the ^https:// pattern on the released url, a payload carrying an http:// media URL that validated against 2026-08-25 will now fail. Since the description is likely to become the squash commit message, could we update it to match the current design (external_video, image sources, https requirement, MUST NOT reject)?

On classification: I agree the practical impact is probably near zero, and the security rationale is sound. But since this tightens validation on an existing field, should we either mark it feat(media)!: per versioning.md, or have TC explicitly record that it's treated as non-breaking security hardening? Either is fine with me, I'd just like the call made explicitly rather than implied by an "additive" description.

The existing Media shape is image-oriented: one URL and one pair of
dimensions cannot describe alternate video resolutions and codecs, adaptive
streaming manifests, device-specific 3D formats, poster images, or an
externally hosted player's embed URL.

Model Media as an open type discriminator with image, video, and model_3d
variants. Keep `url` as the primary resource, add `sources` as the complete
set of negotiable renditions, and represent externally hosted playback as a
video with `embed_url` rather than a separate media type. Optional `preview`
and `name` fields provide a poster and human-readable label; source metadata
covers media type, format, dimensions, and byte size.

The shape follows the schema.org MediaObject family while using scalar seconds
for duration and bytes for file size. REST and MCP examples demonstrate
self-hosted video, 3D models, and embedded video without vendor-specific hosts.

This is additive and preserves generated-source compatibility: the base keeps
`required: [type, url]` plus `width` and `height`, shared constraints are
defined once on the base, and unknown media types remain valid but MUST be
ignored by Platforms.

Follow-up: teach the Schema Reference macro to merge inherited base constraints
when rendering description-only variant refinements.
   Fresh eyes review & amendments to staged update:

   - `url` was overloaded. Folding external video into `video` made
     `video.url` mean "a rendition, or the watch page when `embed_url` is
     present", with `embed_url` presence as an implicit discriminator.
     Restore `external_video` as its own well-known type whose `url` is the
     third-party player; `video.url` is always a playable rendition; drop
     `embed_url`. `required: [type, url]` is untouched.
   - `width`/`height` had been redefined for video as layout size, with two
     SHOULDs patching the resulting confusion. Restore intrinsic pixels,
     drop both rules, and mark them not applicable to `model_3d`.
   - "MUST ignore an unrecognized type" contradicted how `link`, `policy`,
     and `fulfillment_method` treat open vocabularies. Use the house rule:
     MUST NOT reject; MAY present from `preview`/`name`, or omit.

   Review-driven changes:

   - Define every field once on `media.json`. `sources` moves up so images
     can carry AVIF/WebP/JPEG and width variants, and `duration` moves up so
     generated types are complete: JSON Schema code generators do not
     project `if`/`then`, so a field defined only in a per-type file was
     invisible to them. The per-type files now hold description overrides
     only.
   - Require the `https` scheme on `url`, `preview.url`, and
     `sources[].url` via `pattern`, paired with a consumer rule: a Platform
     MUST NOT fetch, embed, or navigate otherwise. This deliberately
     tightens the released `url`; no example or fixture has ever carried
     `http://`, and secure-context Platforms already refuse mixed content.
   - A Platform embedding an `external_video` player MUST isolate it so it
     cannot navigate the top-level document or act as the Platform. The
     mechanism is illustrated, not mandated: a sandboxed, credentialless
     iframe granting only what the player needs.
   - Product/Variant prose says "media item", not "image"; the first item a
     Platform presents is featured.

   main.py: render per-type variants with fields completed from the base.
   A variant property carrying only a `description` refines a field the
   base defines, so it has no `type` or `required` of its own; schema_fields
   rendered such rows as `any` / `Optional` directly under a base table
   saying `string` / `Required`. Find the composing base by scanning
   siblings for an `allOf[].then.$ref` to the file being rendered, copy each
   description-only property's definition from the base while keeping the
   variant's description, and pass the base's `required` through as
   parent_required_list. No call-site parameter: the relationship is read
   from the schema that declares it. Only the four media variants use
   description-only refinement; fulfillment_destination's variants declare
   their own properties and render unchanged. The idiom is documented in
   schema-authoring.md.
   The previous commit added `pattern: ^https://` to `url`, `preview.url`,
   and `sources[].url`, with a Business MUST to match. On `url` that
   tightens a released field: an `http://` media URL that validates against
   2026-08-25 fails, which makes the whole PR breaking and therefore not
   backportable. It also makes media the one place UCP scheme-checks a
   content URL, while `link.url`, `product.url`, `variant.url`,
   `image_url`, `tracking_url`, `continue_url`, and the other `format: uri`
   content fields carry no constraint at all.

   Remove the patterns and the producer MUST so the PR is additive again.
   In their place, a Platform MAY refuse to load or embed a media URL that
   is not `https`: refusing is conformant, and a Business knows an
   `http://` asset may not render. The isolation requirement for
   `external_video` players is unchanged; a player executing in the
   Platform's origin is the player acting as the Platform, which that
   requirement already forbids.

   The scheme requirement can land later as a breaking change. When it
   does, it should be a sweep of every content URL UCP hands a Platform for
   rendering, linking, or embedding — deciding which fields must be `https`
   and which, if any, legitimately carry another scheme — rather than one
   field family at a time.
@igrigorik igrigorik changed the title feat(media): model image, video, and 3D variants feat(media): model image, video, external video, and 3D variants Sep 25, 2026
@igrigorik

Copy link
Copy Markdown
Contributor Author

@amithanda updated PR description. You're right on the ! as it was specced, but I think we should walk that back: added a ~platform may ignore / not load to make it backwards compatible — I'd like to enable this on 08-25, not wait for next release. We can land the stronger and breaking requirement for media and other resources in followup PR.

Comment thread docs/specification/shopping/catalog/index.md Outdated
@maximenajim

Copy link
Copy Markdown

Thanks, Ilya. Thanks for putting this together. I left just one comment for you to consider.

@amithanda amithanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving! Please take a look at the open comment from @maximenajim before merging.

   State the property in terms of what the player must not receive —
   credentials, Platform APIs and agent tools, protocol messaging, and
   navigation of the host application — and add the check that makes the
   example sound: before embedding, decline the item unless `url` is an
   absolute `https` URL without userinfo that is not same-site with the
   Platform. This is the rule Payment Authentication already applies to a
   URL loaded into a browsing context. The example is unchanged; the
   inputs that defeat it are excluded before the iframe exists.
@igrigorik
igrigorik merged commit 4a3aa10 into main Sep 29, 2026
19 checks passed
@igrigorik
igrigorik deleted the feat/media-sources branch September 29, 2026 17:56
@github-actions github-actions Bot added the enhancement New feature or request label Sep 29, 2026
damaz91 added a commit to damaz91/ucp that referenced this pull request Oct 13, 2026
damaz91 added a commit that referenced this pull request Oct 22, 2026
â€Ķ (#941)

Backports #690 (commit 4a3aa10) to the release/2026-08-25 branch.
damaz91 added a commit that referenced this pull request Oct 22, 2026
â€Ķ (#935)

Backports #690 (commit 4a3aa10) to the release/2026-04-08 branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:shopping Issues and pull requests related to the Shopping vertical enhancement New feature or request gov:needs-tc-review Requires review and approval from the Technical Council gov:tc-meeting Proposed for Technical Council live discussion status:stale-review Applied if a PR is waiting on a reviewer for too long

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants