Ollama is an inference engine, which is a tool for managing and running local LLMs. It also provides a Python library for interacting with local LLMs, allowing them to be integrated into projects. Ollama runs on different hardware and operating systems (OS) including Mac, Windows, and Linux.

Ollama supports many different models, but the majority of them are made for running on fancy GPUs and higher powered computers than the Raspberry Pi. However, there are some that are small enough to work on the Raspberry Pi 4 and 5. This guide will document a few we've tested successfully on the Pi 4 & 5 with 8 GB of RAM.

Raspberry Pi OS Setup

First, get the latest release of Raspberry Pi OS installed on your Pi and update all of the built-in software with the apt tool. If you are comfortable with the Raspberry Pi imaging and setup process, you can follow the steps listed here under the quick start prerequisites. If you'd like more details, a more thorough guide page can be found here.

Install Ollama

To install Ollama, download and run the shell script from ollama.com/download.

On a Raspberry Pi 5, the process takes 3-5 minutes or longer for slower networks. During installation, it will print messages regarding the steps being taken and progress as well as a warning about not finding a GPU.

After it is finished, verify that that Ollama installed successfully by running the help command:

Now you're ready to install and run individual models. Follow the instructions on model specific pages.