Voice
Everything that controls how ORBIS hears you and how it speaks back. These live in Settings → Voice (mic, speech-to-text, text-to-speech), and the same options can be set in orbis.yaml for headless setups.
For the why behind the pipeline, see The voice loop.
Microphone
The Voice → Microphone panel manages audio input.
| Control | What it does |
|---|---|
| Permission | macOS mic access. If it's not granted, ORBIS shows a button to request it (or to open System Settings if it was previously denied). |
| Input device | Choose which microphone to use. Shown only when the audio mode uses a selectable device; in voice-processing mode ORBIS follows the macOS system input instead. |
| Level meter | Confirms microphone activity. Moving bars alone do not prove audio reaches speech recognition or that the voice pipeline is ready. |
M-series internal mic
The built-in mic on M-series Macs is quiet without hardware AGC. ORBIS applies input gain so normal speech registers; if you use an external mic and it's clipping, prefer a quieter input or external device.
Speech-to-text (STT)
How spoken audio becomes text. Set the backend, plus per-backend options. Config lives under stt: in orbis.yaml; the STT_BACKEND env var overrides stt.backend.
| Backend | What it is | When to use |
|---|---|---|
local (Whisper) | In-process Whisper on Apple-Silicon (MPS). Segmented per turn — it transcribes after you stop talking, not while. Default for direct Python runs. | An on-device alternative to Parakeet. |
parakeet | NVIDIA Parakeet-TDT via Apple MLX. The packaged Mac app selects this for on-device speech. ~600 MB model on first use; restart to apply. Source installations need the [parakeet] extra. | The desktop on-device default. |
openai | An OpenAI-compatible Whisper endpoint. | You want a hosted transcriber. |
| protoLabs | faster-whisper on the protoLabs gateway (same key as the LLM). | protoLabs-hosted setups. |
Keys (stt:):
| Key | Backend | Meaning |
|---|---|---|
backend | all | local · parakeet · openai |
whisper_model | local | Whisper model id (takes effect on restart). |
model | openai | Remote model id (e.g. whisper-1). |
url / api_key | openai | Endpoint + key (take effect on the next session). |
Text-to-speech (TTS)
How ORBIS's replies become the orb's voice. Set the backend and a voice. Config lives under voice: in orbis.yaml; TTS_BACKEND and KOKORO_VOICE override voice.tts_backend and voice.voice.
| Backend | What it is | Voice selection |
|---|---|---|
kokoro | The default. Runs on CPU, fully local. | A fixed catalogue (e.g. af_heart) — pick from the dropdown. |
openai | OpenAI-compatible /v1/audio/speech. | Type any voice id the endpoint supports. |
| protoLabs (Fish) | Fish S2-Pro via the protoLabs gateway (same key as the LLM). | protolabs/fish. |
fish | Opt-in local sidecar — cloneable voices, needs a GPU. | Custom. |
Keys (voice:):
| Key | Meaning |
|---|---|
tts_backend | kokoro · openai · fish · gateway. |
voice | Voice id (Kokoro voice, OpenAI voice, ElevenLabs voice_id, …). |
tts_url / tts_model / tts_api_key | For OpenAI-compatible endpoints. |
Voice cache
Kokoro voices are cached after first use; the TTS panel shows which voices are already cached so you know which will start instantly.
Startup readiness
In the desktop app, opening the microphone or connecting the native audio socket is not enough to make voice usable. ORBIS warms required local speech engines first, then starts Pipecat. The current state is available at /healthz as voice.lifecycle and as the retained voice-lifecycle event on /api/events:
| State | Meaning |
|---|---|
warming | Required local speech engines are loading off the API event loop. |
starting | Models are ready and Pipecat is completing pipeline setup. |
running | Pipecat confirmed the complete pipeline can consume audio. |
failed | Warmup, socket connection, or pipeline setup stopped. Follow the payload's recovery action. |
Direct python app.py runs without the native audio socket and reports a null lifecycle. This is expected for API-only and A2A-only deployments.
Public failure details and codes are deliberately stable and sanitized. The full exception remains in the private sidecar log. action: retry is emitted only for startup failures that can safely be retried before the native audio session starts, such as model warmup or initial connection failure. A later pipeline or backend loss requires action: relaunch_required; accepting a new socket connection does not rebuild the voice pipeline or restart the sidecar. Cancelling startup prevents Pipecat from starting, but Python cannot forcibly stop synchronous native model code already running inside asyncio.to_thread; process shutdown may therefore wait for that call to return.
The desktop loading gate releases at app-ready, so Settings and first-run setup remain available while local speech models warm. Until lifecycle reaches running, the status line reports the current warmup/startup detail and the orb and microphone controls cannot begin a voice turn. A failed state remains visible rather than treating an acquired Mac microphone as a working voice pipeline. Correct the reported cause and use Settings → Quick → Retry voice only when the lifecycle advertises action: retry. For failures that require a new native session, Settings instead offers Relaunch ORBIS for action: relaunch_required; it never sends a retry that the backend cannot honor. Operators can trigger the same explicit, idempotent retry with POST /api/voice/retry; ORBIS never runs a blind retry loop.
The v0.2.172 public beta monitors complete microphone-frame delivery and allows the native socket to accept another client after a disconnect, discarding stale audio. Automated socket and capture-loss tests pass; live microphone, device changes, and sleep/wake acceptance remain outstanding. ORBIS does not automatically restart a failed hardware stream or sidecar; use the recovery action shown by the app.
See also
- The voice loop — how a turn flows end to end.
- Voice isn't working — troubleshooting.
- Config reference — the full
orbis.yaml.