Running AI Locally: Ollama, LM Studio and llama.cpp Complete Setup Guide

Running AI on your own machine costs nothing after setup, keeps your data off third-party servers, and works without an internet connection. This guide covers Ollama, LM Studio, and llama.cpp with real installation commands, model specs matched to specific hardware, and a complete code review example you can run today.

Hardware You Need Before You Start

Local AI performance depends on RAM and GPU VRAM. Here is what each configuration can handle:

Apple Silicon Macs use unified memory shared between CPU and GPU. A 16 GB M2 MacBook Air runs Llama 3.1 8B at Q8 quality at 20 to 35 tokens per second because the whole model fits in high-bandwidth memory with no VRAM bottleneck.

Ollama: Install and Run in Under Five Minutes

Ollama handles model downloads, quantization selection, and the REST API server automatically. Install with one command on Mac or Linux:

# macOS or Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows: download the installer from https://ollama.ai
# Verify installation
ollama --version

Pull and run Llama 3.1 8B immediately after installation:

ollama pull llama3.1:8b
ollama run llama3.1:8b

The pull downloads about 4.7 GB. After it finishes you get a chat prompt in your terminal. Type your message and press Enter. To exit, type /bye.

Ollama runs a REST API on port 11434. Query it from any script:

curl http://localhost:11434/api/generate   -d '{
    "model": "llama3.1:8b",
    "prompt": "Explain transformer architecture in two sentences.",
    "stream": false
  }'

Essential Ollama Commands

• ollama list - see all downloaded models
• ollama pull qwen2.5:14b - download Qwen2.5 14B
• ollama rm llama3.1:8b - delete a model to free disk space
• ollama serve - start the API server without opening a chat session
• ollama ps - check which model is currently loaded in memory

To make Ollama accessible from other devices on your local network, set the host before starting:

OLLAMA_HOST=0.0.0.0 ollama serve

Any device on your home network can now reach the API at http://YOUR_IP:11434.

Model Comparison Table

Model names follow the pattern name:size or name:size-quantization. Q4_K_M is the best default: it uses mixed precision for key weight layers and gives better output than plain Q4 at nearly the same file size.

LM Studio: Visual Interface

Download LM Studio from lmstudio.ai. It provides a graphical model browser, a ChatGPT-style chat interface, and a local API server. The Discover tab shows models with green checkmarks next to ones that fit your detected RAM.

After downloading a model, click the chat icon in the left sidebar. Adjust the GPU Layers slider to offload layers to VRAM: more layers means faster generation when you have VRAM headroom. The local server mode at the bottom of the sidebar mimics the OpenAI API format, so apps built for GPT-3.5 or GPT-4 often work after just changing the base URL to http://localhost:1234/v1.

llama.cpp: Direct Inference for Developers

llama.cpp is the inference engine that powers Ollama and LM Studio internally. Using it directly gives maximum control over batch sizes, memory mapping, and GPU layer counts.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j4

./llama-cli   -m /path/to/llama-3.1-8b-instruct.Q4_K_M.gguf   -p "You are a helpful assistant."   -n 300   --gpu-layers 35

Start --gpu-layers at 20 and increase until you run out of VRAM. Each additional layer offloaded to GPU improves generation speed proportionally. Use llama-server instead of llama-cli to get an OpenAI-compatible REST endpoint.

Worked Example: Automated Code Review

This Python script reads a source file and sends it to Llama 3.1 8B for a bug review. Start Ollama first with ollama serve, then run this:

import requests, sys

def review_code(filepath):
    with open(filepath) as f:
        code = f.read()

    payload = {
        "model": "llama3.1:8b",
        "prompt": (
            "Review this Python code for bugs, edge cases, and improvements. "
            "Be specific and concise.

Code:
" + code
        ),
        "stream": False
    }

    resp = requests.post(
        "http://localhost:11434/api/generate",
        json=payload
    )
    return resp.json()["response"]

if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "example.py"
    print(review_code(path))

Test it with a function that divides by list length without an empty-list guard. Llama 3.1 8B spots the zero-division bug, suggests a guard clause, and recommends replacing the manual accumulation loop with Python built-ins. On a 6 GB GPU the response takes 3 to 5 seconds. No API key, no cloud cost, no data sent anywhere.

Performance Tips

• Use Q4_K_M quantization as your default. It consistently outperforms plain Q4 at nearly the same file size by using higher precision for the model layers that matter most.
• Set OLLAMA_GPU_LAYERS=99 and let Ollama auto-detect the maximum layers that fit your VRAM. This avoids the trial-and-error of manual tuning.
• Reduce context length if you hit memory limits. A 4096-token context uses roughly half the memory of 8192 for the same model.
• Load one model at a time. Swapping models forces a full reload from disk. Check what is loaded with ollama ps before pulling a second model into memory.

Frequently Asked Questions

Can I run local AI without a GPU?

Yes. CPU-only inference works for 7B models with 16 GB RAM at 5 to 12 tokens per second. That is slow enough to notice during conversation but fine for batch tasks like summarizing documents or reviewing code overnight. Phi-3.5 Mini and Gemma 2 2B were designed specifically for CPU use and run noticeably faster than 7B models on the same hardware.

What is quantization and why does it matter?

Quantization reduces weight precision from 16-bit floats to 4-bit or 8-bit integers. A Q4 model uses roughly four times less memory than the original with minimal quality loss. The Q4_K_M format improves on plain Q4 by using higher precision for the most sensitive weight layers, giving noticeably better output for math, reasoning, and code tasks.

How do I choose between Ollama and LM Studio?

Use Ollama for command-line workflows, REST API integrations, and scripted automation. Use LM Studio for visual model browsing, side-by-side comparisons, and interactive chat. Both pull from the same GGUF model repositories and produce identical output for the same model. Many users install both and switch depending on the task.

Will local models ever match ChatGPT quality?

Smaller 7B to 8B models are roughly equivalent to GPT-3.5 on most writing and summarization tasks. The 70B models approach GPT-4 quality on coding and reasoning benchmarks. The gap closes every few months as training methods improve. For practical everyday tasks, a local 8B model already handles the majority of what most people use cloud AI for.

How much storage do I need for multiple models?

A Q4-quantized 7B model uses 4 to 5 GB on disk. A 13B model uses 8 to 10 GB. A 70B model uses 38 to 42 GB. Models load directly from GGUF files without decompression. Budget at least 50 GB if you want three or four models available without constantly deleting and re-downloading.

Connecting Local AI to Your Existing Tools

Running Ollama or LM Studio in isolation is useful. Connecting them to your code editor, browser, or automation scripts multiplies that value considerably.

Continue.dev in VS Code and JetBrains: Continue is a free open-source AI coding assistant that plugs directly into VS Code and JetBrains IDEs. Point it at your local Ollama server and you get inline code completions, refactoring suggestions, and a chat panel, all running locally. Add this to your Continue config file at ~/.continue/config.json:

{
  "models": [
    {
      "title": "Llama 3.1 8B (local)",
      "provider": "ollama",
      "model": "llama3.1:8b",
      "apiBase": "http://localhost:11434"
    }
  ]
}

After saving, reload VS Code. The Continue sidebar connects to Ollama and you can highlight any function, press Ctrl+L, and ask for an explanation or improvement.

Open WebUI as a ChatGPT replacement: Install Open WebUI via Docker and point it at your Ollama backend. You get a browser interface with conversation history, file uploads, and multiple user accounts. Your team can share one machine running Ollama while each person has their own private chat history.

Python scripts and automation: The Ollama REST API works with any HTTP client. Use it in Zapier alternatives like n8n or Activepieces for local AI-powered automation. Processing a folder of documents, generating email drafts from data, or building a private Slack bot all follow the same pattern: POST to http://localhost:11434/api/generate with a model name and prompt, parse the response JSON.

The key point is that the Ollama API follows the OpenAI format closely enough that many tools designed for GPT-3.5 work with local models after changing only the base URL and removing the API key requirement.