pleeplydocs

Run it yourself

Run an LLM on a Raspberry Pi 5

Updated 3 min read View .md

A Raspberry Pi 5 can run small language models locally. In published third-party Ollama tests with 4-bit models it generates about 11 tokens per second with 1B models and about 5 with 3B models — enough for short voice answers, too slow for long conversation. Use the 8 GB or 16 GB board and an active cooler.

Published speeds (third-party tests)#

ModelQuantizationTokens per second
Gemma 3 1BQ4_K_M11.5
Llama 3.2 1BQ4_K_M11.2
Qwen 2.5 1.5BQ4_K_M10.0
SmolLM2 1.7BQ4_K_M8.2
Granite 3.1 2BQ4_K_M5.8
Qwen 2.5 3BQ4_K_M5.2
Llama 3.2 3BQ4_K_M4.7

Measured with Ollama on a Raspberry Pi 5 by the Stratosphere Linux IPS project, not by pleeply. Larger models are much slower: bestllmfor reports about 1.5–2.5 tokens per second for 7B models on the 16 GB board.

Which model to pick#

  • For a voice assistant: a 1–1.5B model (Gemma 3 1B, Llama 3.2 1B, Qwen 2.5 1.5B) keeps answers to a few seconds.
  • For better answers: a 3B model (Qwen 2.5 3B, Llama 3.2 3B) is about half as fast but follows instructions better.
  • 7B and up: it fits on the 16 GB board, but at 1.5–2.5 tokens per second a one-paragraph answer takes close to a minute.

Install Ollama#

  1. Install 64-bit Raspberry Pi OS and update it.
  2. Install Ollama with its official script:
curl -fsSL https://ollama.com/install.sh | sh
  1. Download and start a small model:
ollama run llama3.2:1b
  1. Try a larger one with ollama run llama3.2:3b. Type /bye to exit.

Ollama also serves an API on port 11434, which Home Assistant and other apps can use — see Use a local LLM with Home Assistant.

Tips for speed#

  • Cooling. Without active cooling the chip throttles under sustained load, which can roughly halve throughput. The official Active Cooler is enough.
  • AI HAT accelerators. Hailo-based AI HATs speed up vision and some speech workloads, but independent tests found they don't speed up token generation for language models; the Pi's CPU is faster at it.
  • Keep the model loaded between requests so it isn't read from storage each time.
  • Storage. An NVMe SSD loads models faster than a microSD card.

Is a Pi 5 enough for a voice assistant?#

For commands and short answers, yes: guides report about 5–8 seconds per request with a small model, Whisper and Piper. For natural back-and-forth it's slow; faster hardware helps most at the language-model step. Compare options in Choose your hardware.

FAQ#

Can a Raspberry Pi 5 run Llama?#

Yes. In third-party Ollama tests (Stratosphere Linux IPS), Llama 3.2 1B ran at about 11 tokens per second and Llama 3.2 3B at about 4.7 with 4-bit quantization.

What is the best LLM for a Raspberry Pi 5?#

For speed, Gemma 3 1B or Llama 3.2 1B — about 11 tokens per second in third-party tests. For better answers at a usable speed, Qwen 2.5 3B or Llama 3.2 3B — about 5 tokens per second in the same tests.

Can a Raspberry Pi 5 run a 7B model?#

On the 16 GB board, yes, but slowly — about 1.5–2.5 tokens per second, according to third-party reports.

Does the Raspberry Pi AI HAT speed up LLMs?#

Independent tests found it doesn't speed up token generation for language models. It helps with vision and some speech workloads.

Sources#

Want this without building a server?

pleeply is an early project: a speaker with a built-in computer that runs open models at home. There's no prototype yet, and your answers decide what we build first.

Answer 5 quick questions →30 seconds · only email is required · no payment

pleeply Community

Keep going with the community

Running AI at home is easier with people who've already done it. On the pleeply forum you can:

  • ask about local models, hardware and Home Assistant, and get answers from people running them;
  • vote on what pleeply should do first;
  • show your own local-AI setup and get feedback;
  • hear first about test builds and reservations.

Free · no email needed

This page is also available as Markdown. For the full docs index, see llms.txt.