Skip to content

media-http, vendors: add EmpirioLabs AI - #28

Merged
lihe07 merged 11 commits into
respanai:mainfrom
Leo-rq-yu:feat/empiriolabs-vendor
Oct 25, 2026
Merged

lihe07 merged 11 commits into
respanai:mainfrom
Leo-rq-yu:feat/empiriolabs-vendor

Conversation

@Leo-rq-yu

@Leo-rq-yu Leo-rq-yu commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

EmpirioLabs AI serves chat, Responses, Messages, the Gemini API, images, embeddings, speech and decisions as their specs have them, with no vendor for it: each format is configured by hand, its images answer as jobs, and its speech at a URL. Thus, add an empiriolabs vendor that lists each model by the endpoint that serves it, with its limits, features and price, and lays a dialect over two formats: its images are asked to wait and answer bytes, and its speech is called at /audio/speech:stream, whose events are read whole as the audio. An image of no output_format is now of the type its bytes say, as its answers name none.

Its transcriptions and videos are left out: a transcription is always a job, which no openai-transcriptions client could wait on, and a video is only at a signed URL off its API, which a client of OpenAI Videos could never reach. Its reranks, OCR, search, 3D and music have no kind.

Checked live through the gateway: /v1/models (133 models of 5 kinds); Chat Completions, Responses whole and streamed, Messages and generateContent (mistral-small-3-1); embeddings (text-embedding-v4); decisions (aplomb-1); images whole, streamed and edited (flux-2-klein-4b); speech as audio and as events, and a speed refused (qwen-audio-3-0-tts).

Refs: #21

EmpirioLabs serves chat, Responses, Messages and embeddings as their
specs have them, and video as jobs of its own: `POST
/videos/generations` answers a `job_id`, polled at `/jobs/{id}`. With
no vendor and no format for its video jobs, each format is configured
by hand and its video models cannot be reached. Thus, add an
`empiriolabs` vendor with those formats and an `empiriolabs-videos`
format that creates and polls its jobs; its images, speech and
transcriptions answer as jobs or URLs that no synchronous media format
can relay, so they are left out, and a video's content is refused, as
it sits at a URL on another host.

Refs: respanai#21
@github-actions github-actions Bot added A-kinds Area: the event types per kind (crates/kinds) A-formats Area: a kind over a wire, one spec each (crates/formats) A-vendors Area: the model vendors and their kit (crates/vendors) A-gateway Area: the binary (apps/gateway) A-readme Area: documentation that isn't part of any crate, such as docs/, README.md or CONTRIBUTING.md labels Oct 21, 2026
# Conflicts:
#	README.md
#	apps/gateway/src/compose/tests.rs
#	apps/gateway/src/formats.rs
#	crates/vendors/kit/src/formats.rs
#	crates/vendors/src/lib.rs
#	crates/vendors/yellow/src/lib.rs
#	docs/configuration.md
#	docs/vendors.md
@Adam-Dalloul

Copy link
Copy Markdown

Thanks for adding EmpirioLabs AI, Ruiqing! I checked it against our API: the base URL, EMPIRIOLABS_API_KEY and the model list (read live from /v1/models, so new models appear without a release) are all correct. One small ask: could the user-facing name read "EmpirioLabs AI" in the vendors table, README and docs? Happy to help with the live check.

@Leo-rq-yu Leo-rq-yu changed the title media, media-http, vendors, gateway: add EmpirioLabs media, media-http, vendors, gateway: add EmpirioLabs AI Oct 23, 2026
@Leo-rq-yu

Copy link
Copy Markdown
Contributor Author

Thanks for checking it against your API, Adam! The vendors table, README and docs now read "EmpirioLabs AI". We'd gladly take you up on the live check: a run of the chat, Responses, Messages, embeddings and video formats against your API would let us tick that box.

# Conflicts:
#	README.md
#	crates/vendors/src/lib.rs
#	crates/vendors/yellow/src/lib.rs
#	docs/configuration.md
#	docs/vendors.md
When EmpirioLabs AI was added, only its chat, embeddings and video formats
were declared, its model list took kinds from categories it does not serve,
and its images, speech and transcriptions were left out as unrelayable. Its
images answer at once when asked to wait, its speech streams inline at
`/audio/speech:stream`, and its transcription jobs finish in seconds, and it
also serves the Gemini API and decisions. Thus, add those formats, poll a
transcription job within the call, and list each model by the endpoint that
serves it with its limits, features and price.
@Leo-rq-yu Leo-rq-yu changed the title media, media-http, vendors, gateway: add EmpirioLabs AI media-http, vendors, gateway: add EmpirioLabs AI Oct 24, 2026
@Leo-rq-yu Leo-rq-yu changed the title media-http, vendors, gateway: add EmpirioLabs AI media, media-http, vendors, gateway: add EmpirioLabs AI Oct 24, 2026
# Conflicts:
#	crates/formats/media-http/README.md
#	crates/formats/media-http/src/dialect.rs
#	crates/formats/media-http/src/lib.rs
EmpirioLabs AI speech is a provider format of its own, of OpenAI speech's name, that reads its audio.chunk events as they come. It replaces the dialect that read the whole stream before answering.
@Adam-Dalloul

Adam-Dalloul commented Oct 25, 2026 •

Copy link
Copy Markdown

Thanks, Ruiqing. I ran the live check against our API at 8820fbc, built from the branch's Dockerfile with only EMPIRIOLABS_API_KEY set. Chat Completions, Responses, Messages and generateContent (each whole and streamed), embeddings, decisions, images, speech and transcriptions all answered 200, and /v1/models listed 172 of our models across all seven kinds. Models used: qwen3-8-flash, text-embedding-v4, aplomb-1, flux-2-klein-4b (one 256x256 image), and qwen-audio-3-0-tts for a short clip that whisper-large-v3-turbo transcribed word for word. A hunyuan-video-1-5 job with seconds: 1 and size: 640x360 reached us as 480p, 16:9, 1 s and polled to completed.

Four things to fix:

  1. The finished video can't be fetched: /v1/videos/{id}/content answers 400. Our video URLs are signed and download without the API key, so registering this provider with LocateContent and fetching the job's URL with no key would serve it.
  2. The aspect ratio goes out in lowest terms, so 854x480 becomes 427:240, and models that check the ratio fail the job. Snapping to the nearest standard ratio (16:9, 9:16, 1:1, 4:3, 3:4, 3:2, 2:3, 21:9) would fix it.
  3. A failed job's error.code is the HTTP status as a number, and error_of in videos.rs and failed in transcriptions.rs read it only as a string, so clients see failed.
  4. Speech refuses any speed, but our stream takes it for most speech models (qwen-audio-3-0-tts streamed fine at 1.2) and answers 400 itself where a model can't, so that check can go.

Small one: in the README's Chat row, "EmpirioLabs AI" landed after the sentence, so it reads "whichever of the four formats it speaks EmpirioLabs AI". Happy to run it again after an update.

EmpirioLabs AI's transcriptions and videos needed formats of their own:
transcriptions are always a job, polled within the call for up to ten
minutes, and a video, whose content is only at a signed URL off its API,
could never reach a client of OpenAI Videos. Thus, drop
`empiriolabs-transcriptions` and `empiriolabs-videos`, and list neither
kind's models.

Its speech becomes a dialect of `openai-speech`: called at
`/audio/speech:stream`, its events read whole as the audio. Mistral's and
its own `stream_format` and `stream` stripping share one helper, and it
has a logo.
@github-actions github-actions Bot removed the A-kinds Area: the event types per kind (crates/kinds) label Oct 25, 2026
@lihe07 lihe07 changed the title media, media-http, vendors, gateway: add EmpirioLabs AI media-http, vendors: add EmpirioLabs AI Oct 25, 2026
@lihe07
lihe07 merged commit 57bcbf2 into respanai:main Oct 25, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-formats Area: a kind over a wire, one spec each (crates/formats) A-gateway Area: the binary (apps/gateway) A-readme Area: documentation that isn't part of any crate, such as docs/, README.md or CONTRIBUTING.md A-vendors Area: the model vendors and their kit (crates/vendors)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants