Skip to content

Document the architecture and developer-facing APIs for Embeddings - #1088

Merged
dkotter merged 4 commits into
WordPress:developfrom
theaminulai:docs/embeddings-architecture-and-api
Oct 5, 2026
Merged

dkotter merged 4 commits into
WordPress:developfrom
theaminulai:docs/embeddings-architecture-and-api

Conversation

@theaminulai

@theaminulai theaminulai commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

What?

Closes #1037

Documents the architecture and developer-facing APIs for Embeddings support, scoped specifically to the embeddings foundation from #962 — not a general plugin architecture rewrite (see the note on #1042 below).

Why?

docs/experiments/embeddings.md already covered the storage/CRUD layer (Embedding_Repository, Embedding_Record) and the similarity/ranking utilities (Vector_Math, Vector_Ranker) in real depth, but said nothing about how a vector actually gets generated or how embedding providers and models connect through the PHP AI Client. A developer building semantic search or frontend chat on top of this foundation had no single place that covered the whole pipeline end to end.

How?

Expanded docs/experiments/embeddings.md (kept the existing storage/CRUD and comparison content, which was already accurate) with:

  • Generating embeddings — generate_embeddings(), why a model is always required with no auto-selection fallback (vectors from different models are never comparable), the $args shape, and the EmbeddingBuilder it wraps.
  • How providers and models connect — AiClient::defaultRegistry(), ai_provider connectors (wp_get_connectors() / get_ai_connectors()), and EmbeddingGenerationModelInterface as the contract a model implements to support embeddings.
  • Lifecycle events — BeforeGenerateEmbeddingEvent/AfterGenerateEmbeddingEvent are PSR events dispatched by EmbeddingBuilder through an injected event dispatcher, not WordPress do_action() hooks — and generate_embeddings() does not wire one up, so they only fire for code that builds its own EmbeddingBuilder directly. Verified this by reading includes/helpers.php's generate_embeddings() and confirming no non-vendor code in the plugin constructs an EmbeddingBuilder with a dispatcher.
  • Implementation note: the SDK overlay — includes/SDK_Overlay.php conditionally serves the embeddings classes from this plugin's vendored copy only when the environment's own PHP AI Client SDK doesn't already provide them, so includes/Vendor/AiClient/src/... isn't always what actually runs.
  • Trying it from WP-CLI — a short pointer to the existing wp ai embeddings generate/compare commands as a way to exercise the whole pipeline without writing PHP (including the --dry-run default-true gotcha).

Retitled the doc from "Storing Embeddings" to "Embeddings: Architecture and Developer API" to match the new scope, updated its one inbound link in docs/DEVELOPER_GUIDE.md, and added a link from CONTRIBUTING.md, which had none.

Every class, method, and behavior described was checked directly against the current source — includes/helpers.php, includes/Embeddings/*, includes/CLI/Embeddings_Command.php, includes/SDK_Overlay.php, and the vendored EmbeddingBuilder/EmbeddingResult/event classes — not written from a general impression of the codebase.

Re: #1042

A previous attempt at this (#1042) was closed for being scoped too broadly — it rewrote the whole plugin's architecture overview rather than focusing on embeddings. This PR only touches embeddings-specific documentation and the two existing docs that link to it.

Testing Instructions

Documentation-only change; no code paths affected. Please review for accuracy, and check that the internal anchor links in docs/experiments/embeddings.md resolve, and that the updated links in docs/DEVELOPER_GUIDE.md and CONTRIBUTING.md point at the right file.

Open WordPress Playground Preview

Architecture Overview was a bare directory listing with no explanation
of how the plugin actually works, despite both CONTRIBUTING.md and the
Developer Guide already describing it as a "comprehensive guide to
plugin architecture."

Expands it to cover the bootstrap sequence (ai.php through
Main::initialize_features()), the Feature/Experiment registration flow
and its enabled-state checks, the Abilities registration pattern, how
AI provider integration and connector discovery work, a table of the
cross-cutting modules (Logging, Connector Approval, Key Encryption,
Embeddings, Admin, REST, CLI) and the full ai/v1 REST endpoint
inventory, and a developer-facing API reference table of the
helpers.php functions and filters/actions intended for external use.

Every class, method, hook, and option name cited was verified directly
against the current source rather than summarized from memory. Detail
already covered in depth elsewhere (the Developer Guide's experiment
tutorial, and each experiment's own docs/experiments/*.md page) is
linked rather than repeated.

Closes WordPress#1037.
This commit fixes the indentation in the directory tree shown in docs/ARCHITECTURE_OVERVIEW.md so the architecture overview renders consistently and is easier to read. It is a documentation-only cleanup with no functional changes.
docs/experiments/embeddings.md already covered the storage/CRUD layer
and the similarity/ranking utilities in depth, but said nothing about
how a vector actually gets generated or how providers/models connect
through the PHP AI Client. Expand it to cover the whole pipeline:

- Generating embeddings: generate_embeddings(), why a model is always
  required (no auto-selection, since vectors from different models
  are never comparable), and the EmbeddingBuilder it wraps.
- How providers and models connect: AiClient::defaultRegistry(),
  ai_provider connectors, and EmbeddingGenerationModelInterface.
- Lifecycle events: BeforeGenerateEmbeddingEvent/AfterGenerateEmbeddingEvent
  are PSR events dispatched by EmbeddingBuilder, not WP action hooks,
  and generate_embeddings() does not wire up a dispatcher, so they
  only fire for code that builds its own EmbeddingBuilder.
- Implementation note on the SDK overlay: why the embeddings classes
  developers import may come from this plugin's vendored copy or from
  the environment's own SDK, transparently.
- A short WP-CLI section pointing at the existing `wp ai embeddings
  generate`/`compare` commands as a way to exercise the whole pipeline
  without writing PHP.

Retitles the doc and updates its one inbound link from
docs/DEVELOPER_GUIDE.md, and adds a link from CONTRIBUTING.md, which
had none.

Closes WordPress#1037.
@theaminulai
theaminulai requested a review from a team October 1, 2026 16:01
@theaminulai
theaminulai requested a review from jeffpaul as a code owner October 1, 2026 16:01
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: theaminulai <theaminuldev@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.60%. Comparing base (123ab42) to head (40e6813).
⚠️ Report is 41 commits behind head on develop.

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #1088      +/-   ##
=============================================
+ Coverage      80.23%   81.60%   +1.36%     
- Complexity      2998     3081      +83     
=============================================
  Files            124      129       +5     
  Lines          11978    12283     +305     
=============================================
+ Hits            9611    10023     +412     
+ Misses          2367     2260     -107     
Flag Coverage Δ
unit 81.60% <ø> (+1.36%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jeffpaul

jeffpaul commented Oct 4, 2026

Copy link
Copy Markdown
Member

@dkotter probably worth pulling into 1.4.0?

@jeffpaul jeffpaul added this to the 1.4.0 milestone Oct 4, 2026
Comment thread CONTRIBUTING.md Outdated
- [Testing API Strategy](docs/TESTING_REST_API.md) - Testing philosophy and guidelines
- [Example Experiment](includes/Experiments/Example_Experiment/README.md) - Reference implementation
- [Custom Experiment Reference](docs/experiments/custom-experiment-reference.md) - Documented example for extending the plugin
- [Embeddings: Architecture and Developer API](docs/experiments/embeddings.md) - Generating, storing, and comparing embedding vectors — the shared foundation for features like semantic search

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to link to this from here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason this file has been modified so much? The underlying issue here was around documentation for embeddings, not general plugin architecture. I'd suggest we revert the changes here

@dkotter
dkotter merged commit dfd961e into WordPress:develop Oct 5, 2026
63 of 65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document the architecture and developer-facing APIs for Embeddings support

3 participants