Most people’s experience of AI involves sending their data to a server run by a big company. That’s fine for many use cases, but there’s a compelling alternative: running AI models entirely on your own machine. Ollama has made this dramatically easier than it used to be, and I think more people should know about it.
What Is Ollama?
Ollama is an open-source tool that lets you download and run large language models locally on Mac, Windows, or Linux. It handles all the complexity of running models — quantisation, memory management, GPU acceleration — behind a clean command-line interface and a local API endpoint. Running a model is as simple as ollama run llama3.
Why Bother?
Privacy. When you run a model locally, your data never leaves your machine. For anything involving client information, confidential business data, or personal details you’d rather not send to a US server, local models are the answer.
Cost. Once you have the hardware, running local models is free. No API costs, no subscription fees, no usage limits.
Offline capability. Local models work without internet. Useful on a plane, in areas with poor connectivity, or just when the cloud is having a bad day.
What Hardware Do You Need?
A modern Mac with Apple Silicon (M1 or later) is actually excellent for local AI. The unified memory architecture means even the base M1 MacBook Air can run 7B and 13B parameter models respectably. For Windows/Linux, a GPU with at least 8GB VRAM will serve you well.
Best Local Models Right Now
- Mistral 7B — fast, capable, great for general tasks
- LLaMA 3.1 8B — Meta’s latest, excellent quality for the size
- Qwen2.5-Coder — outstanding for coding tasks
- Phi-3 Mini — surprisingly capable at very small size
Honest Limitations
Local models are a generation or two behind the frontier cloud models. Claude Sonnet or GPT-4o will outperform any 7B local model on complex reasoning tasks. For everyday writing, summarisation, and coding help though, the gap is smaller than you’d think.
If you value privacy or want to experiment without spending money, give Ollama a try. The barrier to entry is lower than ever.
— Chris
