Ollama & SmolLM3 Setup
Ollama is an inference engine, which is a tool for managing and running local LLMs. It also provides a Python library for interacting with local LLMs, allowing them to be integrated into projects. Ollama runs on different hardware and operating systems (OS) including Mac, Windows, and Linux.
Ollama supports many different models, but the majority of them are made for running on fancy GPUs and higher powered computers than the Raspberry Pi. However, there are some that are small enough to work on the Raspberry Pi 5.
Install Ollama
To install Ollama, download and run the shell script from ollama.com/download.
curl -fsSL https://ollama.com/install.sh | sh
On a Raspberry Pi 5, the process takes 3-5 minutes or longer for slower networks. During installation, it will print messages regarding the steps being taken and progress as well as a warning about not finding a GPU.
After it is finished, verify that that Ollama installed successfully by running the help command:
ollama -h
Create SmolLM3 Model Instance
Many different models are available through a central hub hosted by Ollama simply by specifying a model name and version i.e. qwen3:0.6b in the code. The SmolLM3 model that this project uses is not hosted directly in that central Ollama hub. Not to worry though, the model is hosted in HuggingFace and it's easy to download the model and create a local instance with one one additional step.
In the project files there is a SmolLM3.modelfile. This file contains a template that is used as the basis for the model interaction. It can be customized and fine tuned based on different use-cases. For this project, it's a fairly standard chat "assistant" style template which supports short conversations with a user. The contents of this file are shown below along with a download project bundle button.
FROM hf.co/unsloth/SmolLM3-3B-128K-GGUF
TEMPLATE "
{{- $lastUserIdx := -1 }}
{{- range $i, $_ := .Messages }}
{{- if eq .Role "user" }}{{- $lastUserIdx = $i }}{{ end }}
{{- end -}}
<|im_start|>system
## Metadata
Knowledge Cutoff Date: June 2025
Today Date: {{ currentDate }}
Reasoning Mode: {{ if $.IsThinkSet }}{{ if $.Think }}/think{{ else }}/no_think{{ end }}{{ else }}/think{{ end }}
{{ if .System }}
## Custom Instructions
{{ .System }}
{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{- else if eq .Role "assistant" }}<|im_start|>assistant
{{- if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}
<think>{{ .Thinking }}</think>
{{- end }}
{{ .Content }}
{{- end }}
{{ if and (ne .Role "assistant") $last }}<|im_start|>assistant
{{- if and $.IsThinkSet (not $.Think) -}}
<think>
</think>
{{ end }}
{{ end }}
{{- end -}}
"
PARAMETER temperature 0.3
PARAMETER top_p 0.9
To create the local instance of the model run this command.
ollama create translator-smollm3 -f SmolLM3.modelfile
Within that command, translator-smollm3 is the name I assigned the local model instance to match the name on the venv used by the project. You can choose a different name if you want, just be sure to update the reference to it in the code as well.
The command takes a few minutes to complete depending on the network connection speed as it needs to download about 2gb of data. Once complete, you can confirm it worked by running this command.
ollama list
The local instance should appear in the list as in the screenshot below.
Page last edited November 03, 2025
Text editor powered by tinymce.
