pleeplydocs

Learn

What is a local AI assistant?

Updated 3 min read View .md

A local AI assistant is a voice or chat assistant whose AI runs on a computer in your home instead of on a company's servers. It hears you, works out an answer with an open-weight language model and replies — all on the device — so your voice, your questions and its memory of you stay on hardware you own.

Local vs cloud#

Local AI assistantCloud assistant (for example Alexa+)
Where your voice is processedOn a device in your homeOn the company's servers
Without internetCore features keep workingStops working
Model sizeLimited by your hardware: about 1–3B parameters on a Raspberry Pi 5, 7–8B with 16 GB of memoryVery large models in data centers
Answer qualityGood for everyday questions, timers, notes and home control; weaker on complex reasoningStrongest on hard questions
SpeedDepends on hardware — several seconds on small boardsUsually quick, depends on your connection
Memory of youStored on the device, where you can inspect and delete itStored in the provider's account
CostHardware up front, no per-request feesOften a subscription — Alexa+ is $19.99 a month without Prime

What runs on the device#

A local assistant chains four components:

  1. Wake word — a tiny model listens for a trigger phrase, for example openWakeWord.
  2. Speech-to-text — turns your sentence into text, for example Whisper.
  3. Language model — an open-weight model such as Llama, Qwen, Gemma or Mistral writes the answer, usually through a runtime like Ollama or llama.cpp.
  4. Text-to-speech — speaks the answer, for example Piper.

Timings and tuning are in How a local voice assistant works.

What "open-weight model" means#

An open-weight model is a language model whose trained weights are published, so anyone can download it and run it on their own hardware. Meta's Llama, Alibaba's Qwen, Google's Gemma and Mistral's models are common examples. Because the weights sit on your disk, the model works without contacting its publisher, and you can swap it for a better one when it comes out. Licenses differ, so check each model's terms before using it in a product.

Is it as smart as ChatGPT?#

Not for every task. Small models that fit on home hardware handle everyday requests well — reminders, short answers, summaries of your notes, controlling devices — but they make more mistakes than frontier cloud models on multi-step reasoning and niche facts. The gap narrows as memory grows: 16 GB runs 7–8B models, and 24 GB or more opens 14–32B models. See How much RAM a local LLM needs.

What stays at home#

With a local assistant the audio, the transcript, the answer and the long-term memory can all stay on the device. That is a property of the design rather than a setting: data that is never sent can't be stored somewhere else. A local assistant can still go online for things that need fresh data — weather, news, music services — but only for those requests, and only if you connect them.

FAQ#

Does a local AI assistant need internet?#

Not for its core loop. The wake word, speech recognition, the language model and speech output all run on the device. You need internet to download models and updates, and for online services you choose to connect, such as weather or music.

What is the difference between on-device AI and cloud AI?#

On-device AI runs the model on hardware you own, so data stays local and it works offline, but the model's size is limited by your hardware. Cloud AI runs large models in a provider's data center, which is more capable but means sending your data there.

Which models can a local assistant run?#

Open-weight models such as Llama, Qwen, Gemma and Mistral, in sizes that fit the device's memory. Small boards run 1–3B models; 16 GB of memory fits 7–8B models at 4-bit quantization.

Can a local assistant remember things about me?#

Yes. Memory is data stored on the device — notes, preferences, past conversations — that the assistant looks up when it answers. Because it lives locally, you can see, edit or delete it.

Sources#

Want this without building a server?

pleeply is an early project: a speaker with a built-in computer that runs open models at home. There's no prototype yet, and your answers decide what we build first.

Answer 5 quick questions →30 seconds · only email is required · no payment

pleeply Community

Keep going with the community

Running AI at home is easier with people who've already done it. On the pleeply forum you can:

  • ask about local models, hardware and Home Assistant, and get answers from people running them;
  • vote on what pleeply should do first;
  • show your own local-AI setup and get feedback;
  • hear first about test builds and reservations.

Free · no email needed

This page is also available as Markdown. For the full docs index, see llms.txt.