privateer-speak

Voice for Pi coding agents, both directions. /speak on, and answers are read aloud as they are written — sentence by sentence, so the reply starts while Pi is still typing it, with code blocks, tables and URLs stripped. Press alt+t and talk; the mic stops when you stop talking, and the transcript lands in the composer for you to review before Enter sends it.

• Speech works out of the box with your OS's local voice (say on macOS, espeak-ng on Linux, System.Speech on Windows). Nothing leaves the machine.
• Any OpenAI-compatible endpoint: TTS via POST {baseUrl}/audio/speech, STT via POST {baseUrl}/audio/transcriptions — OpenAI, Tinfoil, a local Kokoro/whisper.cpp/LocalAI server, or a proxy, from one config block.
• Hands-free when you want it: /talk loop on reopens the mic after each answer, so you can keep your hands off the keyboard and your eyes off the screen.
• Several agents, one voice: only the window in front listens, and when you switch windows the conversation goes with you, into a new agent too.
• Pluggable: hosts register their own providers (e.g. privateer-agent registers its account's confidential-compute TTS and STT) through tiny structural interfaces.

Install

Add privateer-speak to your Pi packages, or load it from an extension factory:

import { makePiSpeakExtension } from "privateer-speak";
factories.push(makePiSpeakExtension());

Use

/speak on            start speaking answers
/speak off           stop
/speak stop          cut off the current utterance
/speak test          say a sample line now
/speak provider      pick a provider, then one of its voices, from a list
/speak provider list list providers as text (→ marks the active one)
/speak provider <id> pick one deliberately ("auto" un-picks)
/speak voice         pick a voice for the active provider from its list
/speak voice <name>  pick a voice (stored per provider)
/speak rate <n>      speaking pace, 0.5–3× normal (stored per provider)
/speak length <n>    characters spoken per answer, or "full" for all of it
/speak stream on|off speak sentences as they arrive vs once the turn ends
/speak announce on|off  say what tools are running ("Running npm.")
/speak scope all|last   speak every message, or only the one the turn ends on
/speak chime         pick an end-of-turn sound (off, ping, chime, knock, or your file)
/speak chime <name|path|off|test>  set it directly, or replay the current one
/speak unfocused speak|quiet  keep speaking in a background window, or leave it to the focused one

alt+t                push to talk — press to listen, press again to send
/talk                the same thing, typed
/talk loop on|off    conversation mode: reopen the mic after each answer
/talk silence <ms>   silence that ends an utterance (default 1200, "off" disables)
/talk vocab <words>  vocabulary the transcriber should expect
/talk provider [id]  list / pick the transcription provider
/talk send on|off    submit transcripts immediately vs prefill the composer
/talk lang [code]    language hint for the engine ("auto" clears)
/talk key <binding>  rebind push-to-talk ("none" unbinds; /reload applies it)
/talk commands       list the spoken commands
/mood                pick a voice mood (see Moods)

While recording, a line above the composer shows what the mic hears:

● ▁▁▂▅▇▆▄▆▇▅▃▂ hearing you    ↵ send · esc discard

The red dot means the mic is open, and it turns hollow (○) once the mic closes to transcribe. The trace only moves for a voice, so it stays flat in a quiet room. The words change from listening to hearing you to finishing… (you've paused, and the utterance ends if the pause lasts) to transcribing…. A countdown appears 10 seconds before the recording cap. Enter sends early and escape discards. Hosts without widgets get the same line, compacted, in the footer. In conversation mode, touching the keyboard while Pi is talking stops it — you have the floor back.

Speech is off by default and only ever runs in interactive UI sessions — headless daemons never speak. The microphone opens only on something you did: /talk, the push-to-talk key, a conversation turn you switched on, or bringing a window to the front while dictation or conversation mode is on (see Several windows). There is no wake word. Capture is hard-capped at input.maxSeconds (default 60), audio is held in memory and never written to disk, and discarding drops it unheard. Transcripts prefill the composer so you read what was heard before it sends — send on and loop on are the deliberate hands-free opt-ins, and a mis-heard transcript is still just a prompt: tool calls stay behind the same permission gate as anything you type.

Recording needs a capture tool on PATH: sox anywhere, or arecord/parecord/ffmpeg on Linux, ffmpeg on macOS.

Conversation mode

/speak on
/talk loop on

Now it goes back and forth on its own: you speak, the mic closes when you stop, the transcript sends, the answer is spoken as it is written, and the mic reopens. Tool cues ("Running npm.") come on with it, because with your eyes off the screen a silent two-minute tool run is indistinguishable from a crash.

Three things end a turn without ceremony: say nothing and the mic closes by itself, press escape to discard what you just said, or type — which stops the voice mid-sentence and leaves you at a normal composer. /talk loop off ends the mode.

Dictation

/mood dictate

The mic stays open, and everything you say is added to the composer. Nothing is sent, so you can talk a prompt through in pieces, look it over, and fix a word by hand. When it reads right, say "send it" (or press enter). Once the agent has answered, the mic opens again for the next prompt.

"Send it" only counts as the send command while you're dictating. It works on its own or at the end of what you're saying ("…rename the helper, send it"), but not mid-sentence ("send it to staging") and not after a negation ("don't send it"). Outside dictation it's just words. Staying quiet doesn't close the mic. Escape pauses dictation, /talk or the push-to-talk key resumes it, and picking any other mood ends it.

Dialogue: dictate, and hear the answer

/mood dialogue

The same as dictate, but the agent answers out loud. You talk the prompt into the composer and say "send it". The mic stays shut while the agent works and speaks, so it never transcribes its own voice. When the answer is done it chimes (if a chime is set), and the mic opens for your reply. Only the final answer is spoken, not a narration of each tool step. Press any key to cut the voice short and take your turn. /mood dictate goes back to silent dictation.

Open mic: keep talking while the agent works

/mood open-mic

Dictation that doesn't stop for the agent. The mic stays open while the agent is thinking, running tools or answering, so you can carry on talking the next thing into the composer. Then pick how it goes out:

• "send it" (or enter) queues it. It goes as a follow-up once the current turn is done, and nothing running is disturbed. With the agent idle, it just sends.
• "interrupt" sends it now. The running turn is stopped and your message starts a new one. Anything you'd already queued with "send it" goes along with it, as one message. A bare "interrupt" with nothing dictated just stops the agent.

"Interrupt" follows the same rules as "send it": on its own or as the last words, never after a negation. As the last words it needs a pause or an "and" before it ("…use the other file, interrupt"), so "I didn't mean to interrupt" stays dictated. It works in dictate and dialogue too, when you've opened the mic while the agent is busy.

While the agent works, escape is Pi's again: it stops the turn and leaves the mic open. With the agent idle, escape pauses the mic as usual. open-mic is silent. If you turn speech on with it, use headphones: the mic is open while the agent talks, so a speaker would be heard and transcribed. Your own voice cuts the agent off mid-sentence.

Several windows

Run two agents in two windows and only the one you're looking at listens. The terminal reports when its window gains and loses focus, and a mic that opens on its own (dictation, conversation mode) does so only in the focused window. Switch away and that window's mic pauses, with a note in the footer. Words you'd already dictated stay in the composer, and an unfinished conversation turn is dropped rather than sent. Switch back and it listens again. A mic you opened yourself with /talk or the push-to-talk key isn't cut off.

A hands-free mode follows you. Settings live in one file, so every window has the same mode on. With dictation or conversation mode on, whichever window you bring to the front picks the conversation up and listens, with no keypress needed. That includes a new agent you've just opened in another window. A host that can ask the terminal app which window is in front (Privateer does, for Terminal and iTerm2) lets a just-started window know straight away; otherwise it starts listening at its first focus report. It waits while its agent is still working or speaking. Escape in a window keeps its mic shut, even when you come back to it, until you open it again with /talk, the key, or a mood.

Answers are still spoken from a background window, because hearing a long task finish is half the point. /speak unfocused quiet leaves speaking to the focused window, so two agents never talk over each other. The end-of-turn chime plays either way.

This works in any terminal that reports focus: Terminal.app, iTerm2, kitty, WezTerm, Ghostty, Windows Terminal and VS Code. In tmux, turn on set -g focus-events on. Where nothing is reported, every window counts as focused, which is how it behaved before.

Moods

A mood is a named voice setup you switch to in one step: short and quick while you're heads-down, whole answers read slowly when you're reviewing, talked through tool runs when your eyes are off the screen.

/mood                     pick one from a list (the active one is marked)
/mood focused             apply one by name
/mood list                every mood and what it changes
/mood show <name>         one mood in full
/mood save <name> [desc]  keep your current setup as a mood (overwrites that name)
/mood delete <name>       remove one of yours, or hide a built-in

Eight built-in moods, none of which depends on a particular engine:

mood	what it does
focused	rate 1.25×, first 200 characters, no tool cues
relaxed	rate 0.9×, whole answers
hands-free	tool cues on, conversation mode on
final	only the last message of each response, in full, no tool cues
dictate	mic stays open, speech gathers in the composer, "send it" sends (Dictation)
dialogue	dictate, plus the answer is spoken back before the mic reopens (Dialogue)
open-mic	dictate, but the mic stays open while the agent works: "send it" queues, "interrupt" sends now (Open mic)
quiet	speech off, conversation mode off

A mood is an overlay: it only sets the fields it names, and applying it writes those fields into your live config. Changing something with /speak afterwards is fine, and /mood list then marks the mood "(changed since)". /speak shows the active mood.

Define your own in speak.json under moods. They are added to the built-ins, the same name overrides a built-in, and null hides one:

{
  "moods": {
    "calm": { "description": "slow and soft", "rate": 0.85, "maxChars": 0 },
    "night": { "providers": { "local": { "voice": "Samantha", "rate": 0.9 } } },
    "quiet": null
  }
}

The fields are description, enabled, provider, rate, maxChars, stream, announce, conversation, dictation, continuous (open mic), and providers (voice and rate per engine id). dictation and continuous are the exceptions to the overlay rule: they last only while their mood is active, and picking any other mood turns them off. Voices are kept per engine because voice names don't carry over between engines. That's also why a saved mood never pins your provider unless you picked it yourself: a mood saved while on "auto" stays on auto. A top-level rate applies to whichever engine is speaking.

Hosts can ship their own set with makePiSpeakExtension({ moods }), which replaces the built-ins. Your moods still layer on top. A spoken command can switch moods too: "input": { "commands": { "focus mode": "/mood focused" } }.

Spoken commands

Some things you say are actions, not prompts. A spoken command is a phrase that runs a slash command instead of being sent to the model. For example, Privateer maps "fresh start" to its /fresh:

{ "input": { "commands": { "fresh start": "/fresh", "new agent": "" } } }

A phrase matches only when it is the whole utterance. Case, punctuation and extra spaces don't matter ("Fresh start." matches), but a sentence that contains it ("why did the fresh start fail") is an ordinary prompt. A matched command runs immediately, even when transcripts normally prefill the composer, and it never reaches the model. If the command isn't registered, you get a warning instead.

Hosts ship defaults with makePiSpeakExtension({ voiceCommands }). Your input.commands adds to them or overrides them, and "" switches a phrase off. Only extension commands can run this way, because that's what Pi dispatches from a sent message. Its built-ins (/new, /quit) belong to the editor.

Config

~/.pi/speak.json (hosts can relocate it):

{
  "enabled": true,
  "provider": "auto",
  "maxChars": 400,
  "stream": true,
  "announce": false,
  "scope": "all",
  "chime": "ping",
  "unfocused": "speak",
  "providers": {
    "local": { "voice": "Samantha", "rate": 1.3 },
    "openai-compatible": {
      "baseUrl": "https://api.openai.com/v1",
      "apiKeyEnv": "OPENAI_API_KEY",
      "model": "tts-1",
      "sttModel": "whisper-1",
      "voice": "alloy",
      "rate": 1.2
    }
  },
  "input": {
    "provider": "auto",
    "language": "en",
    "autoSend": false,
    "conversation": false,
    "maxSeconds": 60,
    "silenceMs": 1200,
    "threshold": 0.02,
    "shortcut": "alt+t",
    "hints": "privateer-speak, Pi, espeak-ng, whisper.cpp, Tinfoil",
    "commands": { "fresh start": "/fresh" }
  }
}

apiKeyEnv names an environment variable so the key stays out of the file; a literal apiKey also works. Voice and rate are stored per provider because neither ports across engines. maxChars: 0 speaks answers in full. silenceMs is how long you have to stop talking before the utterance is considered finished — lower it if you speak in short bursts, raise it if you think mid-sentence. Speech is detected by pitch as well as loudness: hiss, static, key clicks and bumps have no pitch and never count, and a steady hum or fan is learned as the room's noise floor. The level trace only moves when it hears a voice. threshold is the quietest speech it accepts (RMS, default 0.02). Lower it if it misses a soft voice, and raise it if voices in the background, like a TV or other people talking, get picked up. hints is passed to the transcriber as a vocabulary bias, which is what keeps project nouns and command names from coming back as prose.

scope: "last" skips the running commentary ("let me check the tests…") and speaks only the message the turn ends on, once the turn is over. chime plays when the agent finishes a turn, whether or not speech is on, and after any speech so it never talks over the answer: "off", one of the built-in sounds (ping, chime, knock, synthesized on the fly so they work on every OS), or a path to your own .wav/.mp3/.aiff/.ogg. In conversation mode it's also your cue that the mic is about to open. unfocused is what a window in the background does with its answers: "speak" (the default) or "quiet" (see Several windows).

Voices and providers

/speak voice      browse every voice from every provider
/speak add        add a provider: OpenAI, ElevenLabs, a local Kokoro, or any endpoint

/speak voice opens one list of every voice you can use, grouped by provider and model, with where your text goes and what each voice sounds like:

Voice — type to filter
Privateer · Qwen3 TTS  🛡 TEE · attested by Privateer
› serena        female · young · warm, gentle · Chinese              current
  aiden         male · American · sunny, clear · English
Privateer · Deepgram: Aura-2  🛡 ZDR · provider policy
  Thalia        female · adult · American · clear, confident · English
Privateer · ElevenLabs v3  • Standard
  Rachel        female · young · American · calm · for narration
Local OS voice · macOS say  🛡 On-device
  Samantha      English (US)
+ Add a provider…
↑↓ move · type to filter · space hear it · enter choose · esc cancel

The shield uses pi-privacy's symbols, but it always says who vouches for the claim, because the terminal checks none of it itself:

shield	means
🛡 On-device	the voice runs on this machine; the text never leaves it
🛡 TEE · attested by X	synthesis runs in a confidential enclave, and X attests it (Privateer's server checks the enclave, then forwards your text)
🛡 TEE · provider's claim	an enclave the provider asserts, with no attester named
🛡 ZDR · provider policy	the provider promises zero data retention
• Standard	an ordinary cloud API

Type tee, zdr or on-device to show only those voices.

Type to narrow it ("warm", "british", "kokoro"), press space to hear the highlighted voice, and press enter to switch to it. Picking a voice picks its provider and model too. Samples use the provider's own published clip when there is one (ElevenLabs, the Privateer account), which costs nothing. Otherwise a short sentence is synthesized once and replayed from memory after that.

Descriptions are always the voice maker's own words (ElevenLabs' labels, Kokoro's quality grades, Qwen's one-line descriptions) and never guessed from a name. A voice whose maker publishes nothing shows its language, or nothing, and the sample tells you the rest.

Adding a provider

/speak add (or "+ Add a provider…" at the bottom of the list) asks only what the provider needs:

provider	what it asks	what you get
OpenAI	the env var holding your key (OPENAI_API_KEY if it's set)	gpt-4o-mini-tts, 11 voices
ElevenLabs	the env var holding your key (ELEVENLABS_API_KEY)	your voice library, described, with free previews
Kokoro on this machine	the server's URL (default http://localhost:8880/v1)	54 voices, graded, nothing leaves the machine
another OpenAI-compatible endpoint	a name, URL, model, and whether it's local	anything serving POST /audio/speech

Give it the name of an environment variable and the key never touches disk. Pasting the key itself works too, but it's then stored in speak.json. What gets saved is a plain entry under providers, so you can also write one by hand:

{
  "providers": {
    "kokoro": {
      "type": "openai-compatible",
      "label": "Kokoro (local)",
      "baseUrl": "http://localhost:8880/v1",
      "model": "kokoro",
      "privacy": "local"
    },
    "studio": {
      "type": "openai-compatible",
      "label": "Studio TTS",
      "baseUrl": "https://tts.example.com/v1",
      "apiKeyEnv": "STUDIO_TTS_KEY",
      "model": "studio-2",
      "voices": ["ava", { "id": "leo", "description": "male · calm · for narration" }]
    },
    "elevenlabs": { "type": "elevenlabs", "apiKeyEnv": "ELEVENLABS_API_KEY", "model": "eleven_multilingual_v2" }
  }
}

type is openai-compatible or elevenlabs. For an OpenAI-compatible server, voices come from your voices list if you give one, from OpenAI's list for api.openai.com, or from the server's own GET /audio/voices (Kokoro-FastAPI has one). A key is needed only if you name one. Edits to the file apply the next time a picker opens.

Transcription models

/talk model       browse every transcription model from every provider
/talk add         add one: OpenAI, Groq, ElevenLabs Scribe, a local Whisper server, or any endpoint

/talk model is the same list as /speak voice, with one row per model, grouped by provider and the same privacy shields:

Transcription model — type to filter
Privateer  🛡 TEE · attested by Privateer
› Whisper Large v3 Turbo   Fast confidential speech-to-text transcription      current
  Voxtral Small 24B        Transcription, audio Q&A, summarization, and translation
Privateer  🛡 ZDR · provider policy
  Deepgram: Nova-3         Deepgram Nova-3 general-purpose speech-to-text model…
Groq  • Standard
  Whisper Large v3 Turbo
+ Add a provider…

Adding one asks only what it needs: the key's environment variable for OpenAI, Groq and ElevenLabs, and the URL and model id for a local Whisper server (speaches, faster-whisper-server, LocalAI). Adding transcription to an engine you already use for speech extends that entry instead of creating a second one. In speak.json an engine transcribes when it names an sttModel, and "speech": false makes it listen only:

{
  "providers": {
    "groq": {
      "type": "openai-compatible",
      "baseUrl": "https://api.groq.com/openai/v1",
      "apiKeyEnv": "GROQ_API_KEY",
      "sttModel": "whisper-large-v3-turbo",
      "speech": false
    },
    "whisper-local": {
      "type": "openai-compatible",
      "label": "Whisper (local)",
      "baseUrl": "http://localhost:8000/v1",
      "sttModel": "Systran/faster-whisper-small",
      "speech": false,
      "privacy": "local"
    }
  }
}

Hosted endpoints come with their model lists. A local server is asked (GET /models), and sttModels in the entry overrides both.

Provider resolution

1. Your explicit /speak provider pick, if it's available.
2. The most recently registered provider whose preferWhen() is true (how a host makes "signed in → use the account's TTS" automatic without stomping your pick).
3. The first available provider — the local engine, so an unconfigured install talks.

Registering a provider

import { registerSpeechProvider } from "privateer-speak";

registerSpeechProvider(
  {
    id: "my-tts",
    label: "My TTS",
    privacy: "standard", // "local" | "confidential" | "zdr" | "standard" — shown in the picker
    defaultVoice: "ada",
    model: () => "my-model-1",
    // Plain ids work; VoiceInfo lets the picker say what each one sounds like.
    voices: async () => [
      { id: "ada", description: "female · warm · for narration", language: "English (US)" },
      { id: "bo", model: "my-model-2", modelName: "My Model 2", preview: async () => fetchSample("bo") },
    ],
    available: () => haveCredentials(),
    fetchSpeech: async (text, { voice, model, rate, signal }) => ({
      data: await synthesize(text, voice, model, rate, signal), // Uint8Array
      format: "mp3",
    }),
  },
  { preferWhen: () => haveCredentials() },
);

Providers implement either fetchSpeech (return bytes; pi-speak plays them) or speak (own the utterance; return a stoppable handle — how the local engines work). A voice's model is passed back as model when that voice speaks, so one provider can serve several models; model() names the default. preview returns a ready-made sample; without one, the picker synthesizes a sentence. rate is a multiple of the engine's normal pace; map it to whatever dial the engine actually has.

A streaming answer arrives as several short utterances rather than one long one, so fetchSpeech is called repeatedly and its signal matters: pi-speak synthesizes up to two clips ahead of the voice and aborts the rest the moment the user interrupts.

Transcription is the mirror image, in its own registry with the same resolution rules:

import { registerTranscriptionProvider } from "privateer-speak";

registerTranscriptionProvider(
  {
    id: "my-stt",
    label: "My STT",
    privacy: "standard",
    available: () => haveCredentials(),
    transcribe: async (audio, { language, prompt, signal }) =>
      await recognize(audio.data, audio.format, language, prompt, signal), // → string
  },
  { preferWhen: () => haveCredentials() },
);

prompt is the vocabulary hint (project name plus input.hints). Whisper-style engines take it as preceding context and bias decoding toward it; engines without the concept can ignore it. audio is 16 kHz mono WAV.

Telling it which window is in front

A terminal reports focus only when it changes, so a window that has just started can't tell whether it's the one in front. Until something says so, pi-speak won't open a mic on its own there. A host that can ask the terminal app passes isFocused:

makePiSpeakExtension({
  // true: in front, false: behind, undefined: can't tell (wait for a focus report).
  isFocused: async () => askTerminalAppIfThisWindowIsFrontmost(),
});

It's asked once, at session start, and only when dictation or conversation mode is on. Privateer answers it through AppleScript for Terminal and iTerm2, matching the window by its tty.

Some terminals never report focus to the program inside: on Windows, a program in Git Bash's mintty runs behind a pseudo-console that drops the reports. A host that can watch the window system passes watchFocus, which is started with the session and stopped at its end:

makePiSpeakExtension({
  watchFocus: (report) => {
    const watcher = watchForegroundWindow((front) => report(front)); // call on every change
    return () => watcher.stop();
  },
});

Privateer does this for mintty by asking Windows which window is in front.

License

MIT