Install One Tool. Run Any LLM Locally in 5 Minutes.

Ollama is the fastest way to run large language models on your own hardware. One command installs it. One command downloads a model. One command starts a conversation.

No API keys. No monthly bills. No data sent to anyone's servers. Your conversations stay on your machine.

This guide covers everything: installation on all platforms, pulling and managing models, which models to use for different tasks, advanced configuration with Modelfiles, API mode for building apps, and how to fix the most common errors.

---

1. Installation

Ollama supports macOS, Linux, and Windows. Pick your platform:

macOS

Download the macOS app from ollama.com/download and drag it to your Applications folder. Or install via Homebrew:

brew install ollama

After installing, Ollama runs as a background service automatically. You'll see the llama icon in your menu bar. Apple Silicon (M1/M2/M3/M4): Ollama uses Metal GPU acceleration automatically. M-series chips share CPU and GPU memory, so your full RAM is available for models — an M3 Max with 48GB RAM can run 32B parameter models without breaking a sweat.

Linux

One command installs Ollama and sets it up as a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

Verify the service is running:

systemctl status ollama

# Check GPU is detected
ollama run llama3.2 --verbose
Look for "GPU layers: X/Y" in the output

Windows

Download the installer from ollama.com/download. Run it, follow the prompts. Ollama starts automatically and runs in the system tray. Windows GPU support: NVIDIA CUDA works out of the box if your drivers are current. AMD GPUs are supported via DirectML. Apple Silicon is macOS-only. Verify installation works on any platform:

ollama --version
ollama version 0.6.x

---

2. Your First Model: The "Hello World" Moment

Pull Llama 3.2 — Meta's fast, capable 3B parameter model:

ollama pull llama3.2

You'll see a progress bar as the model downloads (~2GB for the default Q4 quantization). Once it's done:

ollama run llama3.2

That's it. You're now talking to a local LLM:

>>> Hello! What can you do?
I'm a large language model trained by Meta. I can help with writing, coding,
analysis, answering questions, brainstorming, and much more. What would you
like to work on?

>>> Send a message (/? for help)

Type /bye or press Ctrl+D to exit the conversation. Run a one-shot prompt (no interactive mode):

ollama run llama3.2 "Explain recursion in one paragraph"

---

3. Model Management

Ollama stores models locally in ~/.ollama/models. Here are the commands you'll use constantly:

Pull a model

ollama pull llama3.2          # Latest Llama 3.2 3B (default)
ollama pull llama3.2:1b       # Smallest, fastest version
ollama pull llama4:scout      # Llama 4 Scout — best chat model
ollama pull qwen2.5-coder:32b # Best coding model
ollama pull deepseek-r1:14b   # Strong reasoning model

List downloaded models

ollama list
NAME                 ID              SIZE    MODIFIED
llama3.2:latest      a80c4f17acd5    2.0 GB  2 days ago
llama4:scout         c8c1253f3f28    9.8 GB  1 day ago
qwen2.5-coder:32b    40b4eb523a7d    19 GB   3 hours ago

Inspect a model

ollama show llama3.2
Shows model details: parameters, quantization, context length, template

Remove a model

ollama rm llama3.2
Frees up the disk space

Copy a model (to create a custom version)

ollama cp llama3.2 my-assistant
Creates a copy you can customize with a Modelfile

Check what's running

ollama ps
Shows models currently loaded in memory
NAME           ID      SIZE    PROCESSOR  UNTIL
llama3.2:latest  ...   3.8 GB  GPU        4 minutes from now

Ollama unloads a model from memory after 5 minutes of inactivity by default. This frees VRAM for other models.

---

4. Best Models by Use Case

Not every model is good at everything. Here's what to use for each task:

General Chat & Writing

ollama pull llama4:scout
ollama run llama4:scout

Coding

ollama pull qwen2.5-coder:32b  # Best quality, needs 20GB+ VRAM
ollama pull qwen2.5-coder:14b  # Great balance, needs 10GB VRAM
ollama pull qwen2.5-coder:7b   # Budget option, needs 5GB VRAM

ollama pull deepseek-coder-v2:16b  # 10GB VRAM

Writing & Long-Form Content

ollama pull glm4:9b   # 6GB VRAM
ollama pull glm4:27b  # 16GB VRAM

ollama pull mistral:7b

Small & Fast (Low VRAM / CPU)

If you're on older hardware, a laptop, or want instant responses:

ollama pull phi4:mini         # Microsoft Phi-4 Mini, 3.8B, ~2.5GB VRAM
ollama pull gemma3:4b         # Google Gemma 3, strong reasoning, ~3GB VRAM
ollama pull llama3.2:1b       # Fastest, bare minimum 1GB VRAM
ollama pull qwen2.5:3b        # Good all-rounder, ~2GB VRAM

Reasoning & Math

ollama pull deepseek-r1:7b    # 5GB VRAM — great value
ollama pull deepseek-r1:14b   # 9GB VRAM — noticeably better
ollama pull deepseek-r1:32b   # 20GB VRAM — near frontier quality

---

5. VRAM Requirements

The single biggest factor in what models you can run is your GPU's VRAM. Here's a quick reference:

The calculator accounts for quantization level, context length, and model architecture — far more accurate than the simple table above.

GPU offloading (not enough VRAM?)

If a model is larger than your VRAM, Ollama automatically offloads layers to CPU RAM. It's slower but it works:

# Check how many layers are on GPU vs CPU
ollama run llama3.3:70b --verbose
"GPU layers: 40/81" means 40 layers on GPU, 41 on CPU

For CPU-only inference, Ollama uses llama.cpp and runs entirely in RAM. Expect 1–5 tokens/second on modern CPUs for 7B models, compared to 30–80 tokens/second on a decent GPU.

> Want to know which GPU to buy? Read our best GPU for local LLMs guide for 2026 recommendations at every price point.

---

6. Using the Ollama REST API

Ollama runs a local REST API server at http://localhost:11434. This is what makes it useful for building apps — any program can talk to your local LLM the same way it would talk to OpenAI's API.

Generate a completion

curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "prompt": "Why is the sky blue?", "stream": false}'

Chat with message history

curl http://localhost:11434/api/chat -d '
{
  "model": "llama3.2",
  "messages": [
    {"role": "user", "content": "What is machine learning?"},
    {"role": "assistant", "content": "Machine learning is..."},
    {"role": "user", "content": "Give me a simple example"}
  ]
}'

OpenAI-compatible endpoint

Ollama exposes an OpenAI-compatible API at /v1/chat/completions. Any app built for OpenAI can be pointed at Ollama with zero code changes:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama", // Required but ignored
});

const response = await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Hello!" }],
});

console.log(response.choices[0].message.content);

Python with the official Ollama library

pip install ollama

import ollama

response = ollama.chat(
  model="llama3.2",
  messages=[{"role": "user", "content": "What's the capital of France?"}]
)

print(response["message"]["content"])

Stream responses

import ollama

for chunk in ollama.chat(
  model="llama3.2",
  messages=[{"role": "user", "content": "Tell me a story"}],
  stream=True
):
  print(chunk["message"]["content"], end="", flush=True)

---

7. Custom Modelfiles

A Modelfile lets you create a custom version of any model with a persistent system prompt, specific parameters, and even a custom name. This is how you build specialized assistants from base models.

Basic Modelfile

Create a file called Modelfile (no extension):

FROM llama3.2

SYSTEM """
You are a senior software engineer specializing in Python and system design.
You give direct, precise answers with working code examples.
When reviewing code, you identify bugs and suggest improvements.
You never say "Great question!" or add unnecessary filler.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run it:

ollama create my-engineer -f ./Modelfile
ollama run my-engineer

Key Modelfile parameters

System prompt best practices

The system prompt is the most powerful customization. Short, specific prompts beat long generic ones:

# Good — specific role with clear constraints
SYSTEM "You are a terse technical writer. Output markdown only. No preamble."

Bad — vague and generic
SYSTEM "You are a helpful AI assistant that helps users with various tasks."

Pre-loading conversation history

You can inject example conversations to prime the model's behavior:

FROM llama3.2

SYSTEM "You answer questions about SQL databases concisely."

MESSAGE user "How do I get the 5 most recent rows?"
MESSAGE assistant "SELECT * FROM table ORDER BY id DESC LIMIT 5;"

MESSAGE user "How do I delete duplicates?"
MESSAGE assistant "DELETE FROM t WHERE id NOT IN (SELECT MIN(id) FROM t GROUP BY col);"

---

8. Advanced Configuration

Increase context window

By default, most models use a 2048–4096 token context. For longer documents:

# Set context to 32K for this session
ollama run llama3.2 --ctx-size 32768

Or set permanently in Modelfile
PARAMETER num_ctx 32768

Run Ollama on a specific IP/port

By default, Ollama only listens on 127.0.0.1:11434. To share it on your local network:

# macOS / Linux
OLLAMA_HOST=0.0.0.0:11434 ollama serve

Or set permanently via environment variable
export OLLAMA_HOST=0.0.0.0:11434

Keep models loaded in memory

Ollama unloads models after 5 minutes by default. Override this:

# Keep loaded indefinitely
curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "keep_alive": -1}'

Unload immediately after use
curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "keep_alive": 0}'

Or set globally
export OLLAMA_KEEP_ALIVE=10m

Environment variables reference

Enable Flash Attention

Flash Attention reduces memory usage with minimal quality impact — recommended for most setups:

export OLLAMA_FLASH_ATTENTION=1
ollama serve

---

9. Troubleshooting Common Errors

"Error: model not found"

# Check available models
ollama list

Pull the missing model
ollama pull llama3.2

Check exact model name on ollama.com/library
ollama pull llama3.2:latest

Also check spelling — llama3.2 is correct, not llama-3.2 or llama_3_2.

VRAM out of memory (OOM)

Error: GPU out of memory. Requested X bytes but only Y bytes available

# Check VRAM usage on NVIDIA
nvidia-smi

Check on Mac
sudo powermetrics --samplers gpu_power -n 1

Slow inference (low tokens/second)

Expected speeds for reference:

# Check if GPU is being used (look for GPU layers count)
ollama run llama3.2 --verbose

Check if model is in VRAM or RAM
ollama ps

Common causes of unexpectedly slow inference:

• Model not using GPU: Drivers not installed, CUDA not found → reinstall CUDA
• Model in CPU RAM: Not enough VRAM for full GPU load → use smaller model or enable Flash Attention
• Wrong model format: Some models are fp16 by default, much larger than Q4

Ollama not starting

# Linux: check service status
systemctl status ollama
journalctl -u ollama -n 50

macOS: start manually
ollama serve

Check if port is in use
lsof -i :11434

Connection refused (API not responding)

# Start the Ollama server manually
ollama serve

Test if it's running
curl http://localhost:11434
Should return: "Ollama is running"

If ollama serve throws an error, Ollama is probably already running in the background (system tray on macOS/Windows). Kill and restart it.

Model download stuck / corrupted

# Force re-download
ollama rm llama3.2
ollama pull llama3.2

Check disk space — large models need 4–50GB free
df -h ~/.ollama

---

What to Try Next

You're set up and running. Here's where to go from here:

• Find the right model for your hardware: VRAM Calculator — enter your GPU, see exactly which models fit
• Picking a GPU: Best GPU for Local LLMs — 2026 buying guide at every price point
• Coding assistant: Best LLMs for Coding — set up a local GitHub Copilot alternative
• Understand quantization: LLM Quantization Explained — why Q4 vs Q8 matters for your hardware
• Compare tools: Ollama vs LM Studio vs Jan — when to use which