Repository navigation
realtime-ws, vendors, gateway, ui: add Mistral realtime - #35
Merged
Merged
Conversation
2 of 3 tasks
github-actions
Bot
added
A-formats
lihe07
changed the title
A realtime session could not say it wanted the user's speech transcribed and nothing answered, so a transcription client of OpenAI's had no counterpart in the kind, and a model that only transcribes could not be told from one that converses. Thus, give a session its purpose, a conversation or a transcription, read from OpenAI's `type` and, as OpenAI fixes it at connect, from `?intent=transcription`: OpenAI's provider dials that intent with the model routing chose as the one that transcribes, and Gemini, which answers all the same, has its answers dropped and the user's transcript completed at its turn's end, or a transcriber's end of its generation. An OpenAI client is also told each whole transcript is an item of its own. The playground's realtime tab gets a mode, conversation or transcription, listing the models that transcribe first, and says why the gateway closed a session it opened.
Mistral serves realtime transcription on a WebSocket of its own, `/v1/audio/transcriptions/realtime`, whose events no realtime format speaks, so a transcription could not reach Voxtral's realtime models, nor a client of Mistral's SDK any other. Thus, add the `mistral-realtime` format and declare it on the `mistral` vendor. To Mistral it relays a transcription's audio, commits and transcripts, and refuses a conversation, as Mistral only transcribes. A client of it is served too: its session is a transcription from its connect, of the model at its `?model=`, its turns its own, and it is told the transcript, its usage, the session and errors, so it transcribes with OpenAI's or Gemini's models. OpenAI's `/realtime` no longer claims Mistral's path: a request is the spec's whose path it ends in most nearly. Its models, such as `voxtral-mini-transcribe-realtime-2602`, name both transcription and realtime, and are read as realtime. The playground speaks it. Supersedes respanai#29. Refs: respanai#25, respanai#29, respanai#38 Co-authored-by: Leo-rq-yu <leoyuruiqing@gmail.com>
lihe07
force-pushed
the
mistral-realtime
branch
from
October 22, 2026 00:02
4e3a6ae to
5a54f5a
Compare
An OpenAI client whose first frame was not a session opened one of no purpose, and its next, a conversation, was refused by Gemini as a change of it. A later transcription setting without a model gave OpenAI's default transcriber in place of the one routing chose. Gemini left out of a transcript a chunk that carried other fields, and missed a transcriber's end of generation that came with them. Thus, take a session of no purpose for a conversation, name the routed transcriber in every session of a transcription, and complete a Gemini transcript at an answer that completed, the user's next turn, or a transcriber's end of generation, whatever a message carries. Also: OpenAI's intent is read in its own module, the playground keeps its ordered models across renders.
Takes respanai#38's fixes, and fixes what review of Mistral realtime found: - No shared code names Mistral: each spec says whether its sessions transcribe alone and what of its natives another may drop, in one list of the crate's specs, which OpenAI's provider reads to dial a transcription for a client at Mistral's path. - Mistral's writer forgot no audio at a commit, so an OpenAI client's clearing of an empty buffer after one was refused. - A done of no tokens lost them; its usage is taken with the crate's helpers, its total only when it is theirs. - A client's first session is in its audio whether it named one or not; an error without a message is still told.
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #38, which adds transcription sessions; merge it first. Until then this PR shows its commit too.
Mistral serves realtime transcription on a WebSocket of its own,
/v1/audio/transcriptions/realtime, whose events no realtime format speaks, so a transcription could not reach Voxtral's realtime models, nor a client of Mistral's SDK any other. Thus, add themistral-realtimeformat and declare it on themistralvendor. To Mistral it relays a transcription's audio, commits and transcripts, and refuses a conversation, as Mistral only transcribes. A client of it is served too: its session is a transcription from its connect, of the model at its?model=, its turns its own, and it is told the transcript, its usage, the session and errors, so it transcribes with OpenAI's or Gemini's models. OpenAI's/realtimeno longer claims Mistral's path: a request is the spec's whose path it ends in most nearly. Its models, such asvoxtral-mini-transcribe-realtime-2602, name both transcription and realtime, and are read as realtime. The playground speaks it.Supersedes #29.
Refs: #25, #29, #38
Co-authored-by: Leo-rq-yu leoyuruiqing@gmail.com
Checked live through the gateway: a client of Mistral's protocol transcribes with Voxtral (relayed as it came),
gpt-realtime-whisper,gpt-4o-transcribeand Gemini; an OpenAI Realtime client at?intent=transcriptiontranscribes with Voxtral two turns as two items, and a conversation with it is refused (1008).docs/, or the vendor tables)