pleeplydocs

Learn

How a local voice assistant works

Updated 4 min read View .md

An offline voice assistant turns your speech into a spoken answer in four stages that all run on one machine: a wake word detector, speech-to-text, a language model and text-to-speech. On a Raspberry Pi 5 the whole loop takes several seconds, and most of that time is the language model writing its answer.

The pipeline#

StageWhat it doesCommon open-source choice
1. Wake wordListens for a trigger phrase and ignores everything elseopenWakeWord, microWakeWord
2. Speech-to-textTurns your sentence into textWhisper (faster-whisper, whisper.cpp)
3. Language modelUnderstands the request and writes the answerLlama, Qwen or Gemma through Ollama or llama.cpp
4. Text-to-speechSpeaks the answerPiper

In front of all four sits the audio hardware: a microphone array with echo cancellation and noise suppression, so the assistant can hear you over its own speaker and the room. Home Assistant's Voice Preview Edition, for example, uses an XMOS XU316 chip for this.

Where the time goes#

Timings reported by third-party tests and guides on a Raspberry Pi 5 (not measured by pleeply):

  • Speech-to-text: Whisper "small" in int8 takes about 2–4 seconds, depending on sentence length.
  • Language model: with 4-bit models in Ollama, a 1B model generates about 11 tokens per second and a 3B model about 4.7. A one-sentence answer of about 25 tokens therefore takes roughly 2–5 seconds, plus the time to read the prompt.
  • Text-to-speech: Piper synthesizes faster than real time on the Pi 5's CPU.

End to end, guides report about 5–8 seconds per request on a Pi 5 with a small model. That works for hands-free questions but feels slow in conversation. Faster hardware mostly shortens the language-model step — see Choose your hardware.

How to make it feel faster#

  • Stream the answer. Start speaking the first sentence while the model writes the rest.
  • Send simple commands around the model. "Turn off the kitchen lights" can go to a fast intent matcher; only open questions need the language model. Home Assistant supports this pattern.
  • Keep the model loaded. Loading a model from disk adds seconds; Ollama's keep-alive setting keeps it in memory.
  • Use the smallest model that does the job. 1–3B for commands and short answers, 7–8B and up for conversation.
  • Cool the board. A Raspberry Pi 5 throttles under sustained load without active cooling, which can roughly halve language-model throughput.

What works offline#

Timers and alarms, conversions, questions the model can answer from what it learned in training, your notes and memory, and control of smart-home devices on your network. Anything that needs fresh data — weather, news, sports scores — needs an internet lookup, which a local assistant can make on request without sending your audio anywhere.

FAQ#

Can Whisper run on a Raspberry Pi?#

Yes. Small Whisper models run on a Raspberry Pi 5: guides report the "small" model in int8 takes about 2–4 seconds per sentence. Larger models such as "medium" use too much of the Pi's memory to be practical.

What is the best offline text-to-speech?#

Piper is the most widely used offline text-to-speech for home assistants. It runs faster than real time on a Raspberry Pi 5 CPU and has voices in many languages. Its development moved to the OHF-Voice/piper1-gpl repository under the GPL-3.0 license, which matters if you ship it in a product.

Why are local voice assistants slower than Alexa?#

Because the language model runs on small hardware in your home instead of on data-center GPUs. Published guides report about 5–8 seconds per request end to end on a Raspberry Pi 5; a board with a GPU or an Apple Silicon Mac generates short answers several times faster.

Does the wake word detector record everything?#

In a typical design, no. The wake word model checks a short rolling audio buffer on the device and discards it; transcription starts only after it hears the trigger phrase. In a local assistant even the audio after the wake word stays on the device.

Sources#

Want this without building a server?

pleeply is an early project: a speaker with a built-in computer that runs open models at home. There's no prototype yet, and your answers decide what we build first.

Answer 5 quick questions →30 seconds · only email is required · no payment

pleeply Community

Keep going with the community

Running AI at home is easier with people who've already done it. On the pleeply forum you can:

  • ask about local models, hardware and Home Assistant, and get answers from people running them;
  • vote on what pleeply should do first;
  • show your own local-AI setup and get feedback;
  • hear first about test builds and reservations.

Free · no email needed

This page is also available as Markdown. For the full docs index, see llms.txt.