Repository navigation
Document the architecture and developer-facing APIs for Embeddings - #1088
Conversation
Architecture Overview was a bare directory listing with no explanation of how the plugin actually works, despite both CONTRIBUTING.md and the Developer Guide already describing it as a "comprehensive guide to plugin architecture." Expands it to cover the bootstrap sequence (ai.php through Main::initialize_features()), the Feature/Experiment registration flow and its enabled-state checks, the Abilities registration pattern, how AI provider integration and connector discovery work, a table of the cross-cutting modules (Logging, Connector Approval, Key Encryption, Embeddings, Admin, REST, CLI) and the full ai/v1 REST endpoint inventory, and a developer-facing API reference table of the helpers.php functions and filters/actions intended for external use. Every class, method, hook, and option name cited was verified directly against the current source rather than summarized from memory. Detail already covered in depth elsewhere (the Developer Guide's experiment tutorial, and each experiment's own docs/experiments/*.md page) is linked rather than repeated. Closes WordPress#1037.
This commit fixes the indentation in the directory tree shown in docs/ARCHITECTURE_OVERVIEW.md so the architecture overview renders consistently and is easier to read. It is a documentation-only cleanup with no functional changes.
docs/experiments/embeddings.md already covered the storage/CRUD layer and the similarity/ranking utilities in depth, but said nothing about how a vector actually gets generated or how providers/models connect through the PHP AI Client. Expand it to cover the whole pipeline: - Generating embeddings: generate_embeddings(), why a model is always required (no auto-selection, since vectors from different models are never comparable), and the EmbeddingBuilder it wraps. - How providers and models connect: AiClient::defaultRegistry(), ai_provider connectors, and EmbeddingGenerationModelInterface. - Lifecycle events: BeforeGenerateEmbeddingEvent/AfterGenerateEmbeddingEvent are PSR events dispatched by EmbeddingBuilder, not WP action hooks, and generate_embeddings() does not wire up a dispatcher, so they only fire for code that builds its own EmbeddingBuilder. - Implementation note on the SDK overlay: why the embeddings classes developers import may come from this plugin's vendored copy or from the environment's own SDK, transparently. - A short WP-CLI section pointing at the existing `wp ai embeddings generate`/`compare` commands as a way to exercise the whole pipeline without writing PHP. Retitles the doc and updates its one inbound link from docs/DEVELOPER_GUIDE.md, and adds a link from CONTRIBUTING.md, which had none. Closes WordPress#1037.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1088 +/- ##
=============================================
+ Coverage 80.23% 81.60% +1.36%
- Complexity 2998 3081 +83
=============================================
Files 124 129 +5
Lines 11978 12283 +305
=============================================
+ Hits 9611 10023 +412
+ Misses 2367 2260 -107
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@dkotter probably worth pulling into 1.4.0? |
| - [Testing API Strategy](docs/TESTING_REST_API.md) - Testing philosophy and guidelines | ||
| - [Example Experiment](includes/Experiments/Example_Experiment/README.md) - Reference implementation | ||
| - [Custom Experiment Reference](docs/experiments/custom-experiment-reference.md) - Documented example for extending the plugin | ||
| - [Embeddings: Architecture and Developer API](docs/experiments/embeddings.md) - Generating, storing, and comparing embedding vectors — the shared foundation for features like semantic search |
There was a problem hiding this comment.
I don't think we need to link to this from here
There was a problem hiding this comment.
Any particular reason this file has been modified so much? The underlying issue here was around documentation for embeddings, not general plugin architecture. I'd suggest we revert the changes here
What?
Closes #1037
Documents the architecture and developer-facing APIs for Embeddings support, scoped specifically to the embeddings foundation from #962 — not a general plugin architecture rewrite (see the note on #1042 below).
Why?
docs/experiments/embeddings.mdalready covered the storage/CRUD layer (Embedding_Repository,Embedding_Record) and the similarity/ranking utilities (Vector_Math,Vector_Ranker) in real depth, but said nothing about how a vector actually gets generated or how embedding providers and models connect through the PHP AI Client. A developer building semantic search or frontend chat on top of this foundation had no single place that covered the whole pipeline end to end.How?
Expanded
docs/experiments/embeddings.md(kept the existing storage/CRUD and comparison content, which was already accurate) with:generate_embeddings(), why a model is always required with no auto-selection fallback (vectors from different models are never comparable), the$argsshape, and theEmbeddingBuilderit wraps.AiClient::defaultRegistry(),ai_providerconnectors (wp_get_connectors()/get_ai_connectors()), andEmbeddingGenerationModelInterfaceas the contract a model implements to support embeddings.BeforeGenerateEmbeddingEvent/AfterGenerateEmbeddingEventare PSR events dispatched byEmbeddingBuilderthrough an injected event dispatcher, not WordPressdo_action()hooks — andgenerate_embeddings()does not wire one up, so they only fire for code that builds its ownEmbeddingBuilderdirectly. Verified this by readingincludes/helpers.php'sgenerate_embeddings()and confirming no non-vendor code in the plugin constructs anEmbeddingBuilderwith a dispatcher.includes/SDK_Overlay.phpconditionally serves the embeddings classes from this plugin's vendored copy only when the environment's own PHP AI Client SDK doesn't already provide them, soincludes/Vendor/AiClient/src/...isn't always what actually runs.wp ai embeddings generate/comparecommands as a way to exercise the whole pipeline without writing PHP (including the--dry-rundefault-true gotcha).Retitled the doc from "Storing Embeddings" to "Embeddings: Architecture and Developer API" to match the new scope, updated its one inbound link in
docs/DEVELOPER_GUIDE.md, and added a link fromCONTRIBUTING.md, which had none.Every class, method, and behavior described was checked directly against the current source —
includes/helpers.php,includes/Embeddings/*,includes/CLI/Embeddings_Command.php,includes/SDK_Overlay.php, and the vendoredEmbeddingBuilder/EmbeddingResult/event classes — not written from a general impression of the codebase.Re: #1042
A previous attempt at this (#1042) was closed for being scoped too broadly — it rewrote the whole plugin's architecture overview rather than focusing on embeddings. This PR only touches embeddings-specific documentation and the two existing docs that link to it.
Testing Instructions
Documentation-only change; no code paths affected. Please review for accuracy, and check that the internal anchor links in
docs/experiments/embeddings.mdresolve, and that the updated links indocs/DEVELOPER_GUIDE.mdandCONTRIBUTING.mdpoint at the right file.