Bidirectional voice assistant for this machine. Launch it when you want it; quit closes it completely.
- Pipeline (default): Local VAD → voxtype Parakeet (Whisper fallback, then Grok STT) → Grok
grok-4.6text + the same desktop tools →speak. No realtime socket. Reliable for launch / focus / search. - Duplex (Grok live voice): Grok Speech-to-Speech (
grok-voice-think-fast-2.0) over a live socket. Barge-in. Bills while the socket is open. Picked in the bar popover. - Wake word (optional): Wait for
hey dottiebefore a turn. Works in both modes.
cd ~/Projects/dottie-omarchy # wherever you cloned it
make setup
cp .env.example .env
# put XAI_API_KEY in .env (https://console.x.ai)Needs: Rust (cargo), system openssl, PipeWire (pw-cat), XAI_API_KEY, and for local/hybrid: voxtype + ~/.local/bin/speak. The shipped entry is target/release/dottie-omarchy via bin/dottie-omarchy.
Two crates only, both for the mic path: nnnoiseless (RNNoise) and silero-vad-crs (turn detection). HTTP, TLS, WebSocket, and JSON are std plus the system openssl binary.
./bin/dottie-omarchy # pipeline (voxtype + Grok text + tools + speak)
./bin/dottie-omarchy duplex # Grok live voice
./bin/dottie-omarchy wake on # wait for “hey dottie” (needs voxtype)
./bin/dottie-omarchy ask launch terminal # text turn; uses the live session if it is up
./bin/dottie-omarchy mcp # stdio MCP for Cursor / Claude / Codex
./bin/dottie-omarchy --always-stream # billed the whole time — don't leave this onDuplex is $0.08/min of audio ($4.80/hr) while the socket is open. Pipeline only pays Grok text per turn. Paste an xAI key in the bar (or ~/.config/dottie-omarchy/env).
Say goodbye / exit / quit / that's all / stop listening to quit completely (process exits, bar icon goes away). Same as bar Quit or Omarchy menu → Dottie while it is running.
mkdir -p ~/.config/dottie-omarchy
cp .env ~/.config/dottie-omarchy/env # or only the key
make install-usermake install-user installs the unit, plugin, and a Dottie launcher entry (~/.local/share/applications/sd.dottie-omarchy.desktop) but leaves login autostart off. Launch from the app launcher, Omarchy menu → Dottie, or ./bin/dottie-omarchy launch. The bar icon appears only while it is running. Quit / goodbye stops the unit for this session. Optional: dottie-omarchy autostart on if you want it at login.
The unit is WantedBy=graphical-session.target and After=pipewire.socket wireplumber.service. A dead microphone exits non-zero so Restart=on-failure retries while the app is supposed to be up (StartLimitIntervalSec=0, RestartSec=5). Intentional quit exits 0 so it stays down. ExecStartPre clears a stale state file after a SIGKILL and rotates the log past 5MB.
One tree: this repo. Engine is ./bin/dottie-omarchy → target/release/dottie-omarchy. Bar widget is plugin/sd.dottie-omarchy. make install-user rebuilds and symlinks that plugin into ~/.config/omarchy/plugins/sd.dottie-omarchy. Edit here; do not copy into omarchy-plugins.
The bar icon is a menu-bar app: only while Dottie is running. Click for the popover (captions + controls). Middle click or Quit exits. App launcher or Omarchy menu → Dottie starts it (dottie-omarchy launch).
./bin/dottie-omarchy status
./bin/dottie-omarchy stop
./bin/dottie-omarchy toggle
./bin/dottie-omarchy vision # print on/off
./bin/dottie-omarchy vision on # let Grok see the screen (Grok vision only)
./bin/dottie-omarchy wake on # require “hey dottie” before Grok opensDefault service runs pipeline. Bar Start uses ~/.config/dottie-omarchy/mode. The popover's Pipeline / Duplex picker writes that file. Wake word is settings.json, not a separate mode.
| Piece | Value |
|---|---|
| Listen | PipeWire + local energy VAD (400ms silence) on the capture thread. A clip only reaches STT if its loudest frame clears the speech floor and it runs past 350ms, so room noise costs nothing. |
| STT | Streaming Grok STT over wss://api.x.ai/v1/stt, opened on the local VAD onset once the loudest frame so far clears the speech floor. The half second of audio before the onset is sent as pre-roll so the first word is not clipped. interim_results write straight into the caption, so words appear while you are still talking instead of after you stop. The desktop vocabulary and your installed theme names are sent as keyterm biasing (capped at 100 terms, 50 chars each) so Tokyo Night and close_window come back spelled right. If the socket will not open, or it finalizes with nothing, the retained clip goes to the unary /v1/stt call and logs stt socket off; local voxtype Parakeet/Whisper is still the last resort, and DOTTIE_OMARCHY_STT=local forces Parakeet first (cold start ~6s per process). |
| Endpoint | Smart Turn (server-side, confidence 0.7) owns the end of your turn, so a mid-sentence pause to think does not send half a question. The local VAD is only a backstop: 4.5s past its own silence mark, or 60s of one utterance, finalizes the socket anyway. |
| Think | Grok grok-4.6 Responses API with stream: true, so finished clauses reach TTS while the model is still writing. A server that will not stream falls back to the buffered call and logs responses stream off. The current local date and time is refreshed as a system line every turn. Every output item is echoed back, and a repeated tool call in one turn answers {"ok":true,"deduped":true} instead of running twice. If every round calls a tool and none produces words, she says she could not finish rather than going silent. |
| Tools | Client-side function tools (pipeline_tools()), including web_search / x_search, after the transcript is final. Launch detaches. |
| Speak | Streaming Grok TTS over wss://api.x.ai/v1/tts (Eve, 24 kHz PCM straight to the player). The model's text deltas are forwarded as text.delta with no sentence splitting, so audio starts ~250ms after the first words exist rather than a full round trip after the first clause ends; on a long reply she begins speaking about 900ms before the model stops writing. optimize_streaming_latency is left off — levels 0, 1 and 2 all measured ~250ms here, so its quality tradeoff buys nothing. If the socket will not open it falls back to one unary /v1/tts call per clause and logs tts socket off, then to ~/.local/bin/speak. A bare post-tool ack is held back until it is long enough to prove it is a real answer, which keeps talkback=answers honest. |
| Socket | None. A blocking tool cannot kill a voice session. |
| Piece | Value |
|---|---|
| Endpoint | wss://api.x.ai/v1/realtime?model=grok-voice-think-fast-2.0 |
| Transport | binary PCM16, 24 kHz, both directions |
| Gate | Local Silero + energy VAD. Keep 0.6s of onset. Connect after ~500ms speech. Mic stays recorded during handshake. After a live turn, stay up 300s quiet. Empty open hangs at 2.5s. |
| Turns | Once live, the live upload owns the mic and the pre-connect buffer is cleared, so a barge cannot send the same speech twice. server_vad (threshold 0.85, 400ms silence) ends the turn; if it stays quiet for 1.5s after a buffered utterance we send input_audio_buffer.commit + response.create ourselves. Quiet frames are zeroed only while our own speaker is audible — zeroing them otherwise broke the server silence timer. |
| Barge-in | Silero speech held ≥250ms, mic RMS above both a 35 floor and 1.8× the playback-predicted echo, two frames in a row, and ≥1s into her reply. response.cancel is sent only when a reply is actually in flight. |
| Captions | audio.input.transcription.model = grok-transcribe. A turn that spans several responses accumulates its reply text instead of keeping only the last segment. |
| Reconnect | resumption.enabled + conversation_id. One shared backoff (1s → 20s) across every connect path; a 400 drops the conversation_id and starts clean, and the failure is logged once per change instead of per attempt. |
| Tools | Server-side web_search / x_search (session types, not client functions). Desktop: launch, focus, close_window, toggle, remind, workspace, fullscreen, desktop_state, theme, wallpaper, window, media, monitor, volume, brightness, app_catalog. desktop_state reports the clock; volume and brightness read with step=status. Pixels last: see_screen, click, type_text, hotkey. Also go_idle, mute. |
src/engine.rs owns the two pw-cat children so the turn loop can block without losing audio.
| Thread | Job |
|---|---|
| capture | Reads pw-cat --record, mixes AUX0-2 down to mono, runs RNNoise, pushes 100ms frames into a 20-frame ring that drops the oldest and counts the loss |
| player | Drains a queue into pw-cat --playback in 20ms slices; a barge-in clears the queue and respawns the child so the device buffer goes quiet too |
| tts-ws | Owns the streaming TTS socket for one turn: forwards text between reads and republishes decoded audio, so the socket needs no lock |
| main | VAD, turn state machine, tools, websocket |
Capture retries pw-cat with backoff for 30s before it reports a dead mic, which covers both a slow PipeWire at login and a mid-session restart. There is no NLMS echo canceller: it diverged and turned her own voice into a barge-in on every reply. Echo suppression is the predicted-echo gate plus the server's interrupt_response.
DOTTIE_OMARCHY_DEBUG=1 adds the per-frame rms= and live action= lines, one responses stream …B first=…ms round=…ms line per model round, and one stt first_partial=…ms confidence=… line per utterance — that confidence is Smart Turn's end-of-turn score, which is what to read when a turn ends too early or too late. Every pipeline turn logs one turn stt=… think=… audio=… tools=[…] line either way. stt is the wait from the end of your speech to the final text, so streaming reads near zero because the words were already on screen. audio is measured from the start of the turn, so a streamed reply reads lower than think.
The see_screen, click, type_text and hotkey tools need the grok-omarchy-use helper. Dottie looks for it in ~/.local/bin, /usr/local/bin and /usr/bin, never on PATH, and make install-user links it into ~/.local/bin when the repo sits next to this one. To put it elsewhere, set DOTTIE_OMARCHY_USE_BIN=/absolute/path (a leading ~/ also works) in ~/.config/dottie-omarchy/env, which the systemd unit reads. A bad override is not ignored: the tools stay off and the log says why. Each start logs use_bin <path> or use_bin off: <reason>, and when the helper is missing the model is not offered the screen tools.
| Path | What |
|---|---|
src/ |
Rust engine (std + nnnoiseless + silero-vad-crs) |
src/engine.rs |
Capture and playback threads |
bin/dottie-omarchy |
Wrapper → target/release/dottie-omarchy |
plugin/sd.dottie-omarchy |
Omarchy bar widget + overlay |
skill/SKILL.md |
Voice desktop instructions |
omarchy/ |
Omarchy menu item (trigger.dottie-omarchy) |
contrib/ |
systemd user unit template (@ROOT@ filled by make install-user) |
make install-user rebuilds and symlinks plugin/sd.dottie-omarchy into ~/.config/omarchy/plugins/sd.dottie-omarchy. Do not copy widget files into omarchy-plugins.
Other agents drive the same desktop tools over stdio MCP. Chat turns persist in ~/.local/state/dottie-omarchy/history.json (last 30). A live pipeline listens on $XDG_RUNTIME_DIR/dottie-omarchy/ask.sock, so dottie-omarchy ask … from SSH joins that session after a restart.
# Cursor / Claude — command is the wrapper
# ~/.cursor/mcp.json (or project .cursor/mcp.json)
# { "mcpServers": { "dottie-omarchy": { "command": "dottie-omarchy", "args": ["mcp"] } } }
ssh omarchy-16.local dottie-omarchy ask "what windows are open"Parakeet model: /usr/lib/voxtype/voxtype-onnx-avx2 setup --download --model parakeet-tdt-0.6b-v3-int8. Dottie uses ~/.config/dottie-omarchy/voxtype-parakeet.toml and does not change the voxtype daemon. Each voxtype transcribe reloads the ONNX graph (~6s), so Grok STT stays first.
