Axel Verne → Junk Drawer kit → Chapter 13

Chapter 13 — The Oracle I: Local AI on Old Hardware

23 blocks to copy, in the order they appear in the chapter. Nothing here is required to be typed by hand.

Step 1: Install Ollama

13.1 · bash

curl -fsSL https://ollama.ai/install.sh | sh
ollama --version

13.2 · bash

curl -fsSL https://ollama.ai/install.sh | sh

13.3 · bash

ollama serve &    # starts the daemon in background

Step 2: Pull a model

13.4 · bash

ollama pull qwen2.5:7b

13.5 · bash

ollama list

Step 3: Run the model locally

13.6 · bash

ollama run qwen2.5:7b "What is the capital of France?"

13.7 · text

The capital of France is Paris.

13.8 · bash

ollama run qwen2.5:7b "Explain Docker in one paragraph for someone who's never heard of it."

Step 4: Open WebUI (the nice interface)

13.9 · bash

docker pull ghcr.io/open-webui/open-webui:latest

docker run -d \
  --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v /srv/lab/data/open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

13.10 · yaml

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"  # Linux: http://172.17.0.1:11434
      TZ: "Pacific/Honolulu"
    volumes:
      - /srv/lab/data/open-webui:/app/backend/data
    restart: unless-stopped

13.11 · bash

DOCKER_HOST=$(ip route | grep docker | awk '{print $3}')
# or hardcode it: usually 172.17.0.1

13.12 · bash

cd /srv/lab/compose/open-webui
docker compose up -d
docker compose logs -f

Step 5: Connect to Ollama

13.13 · yaml

# If that doesn't work, try this in compose.yaml:
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    # ... rest of config ...
    network_mode: "host"  # lets it reach localhost:11434 directly

Step 6: Add RAG (your documents)

13.14 · text

@<filename> Summarize this document.

When it breaks

13.15 · bash

docker network inspect bridge | grep Gateway    # should show something like 172.17.0.1

13.16 · bash

ollama show qwen2.5:7b

13.17 · bash

ollama serve --gpu true &

13.18 · bash

sudo lsof -i :3000
mkdir -p /srv/lab/data/open-webui
docker compose up -d

13.19 · bash

ollama pull qwen2.5:7b && ollama run qwen2.5:7b "test prompt"
ollama pull llama3.2:3b && ollama run llama3.2:3b "test prompt"

Make it good

13.20 · bash

export OLLAMA_KEEP_ALIVE="24h"   # keep the model in memory for 24 hours
ollama run qwen2.5:7b "hello"

13.21 · yaml

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    environment:
      OLLAMA_KEEP_ALIVE: "24h"

13.22 · bash

ollama pull qwen2.5:7b:latest    # highest quality, biggest

13.23 · bash

ollama pull qwen2.5:3b-q2_K      # tiny, very fast, dumb

← Chapter 12Chapter 14 →