Linux
2 posts
Running Local LLMs with Ollama: A Hands-On Guide
Ollama runs open-weight models on your own machine behind a small HTTP server. No API key, no per-token bill, and your prompts never leave the laptop. It's the model backend for my OpenClaw Telegram assistant, and the easiest way I know to learn how LLMs behave. Every screenshot here is real terminal output, captured while writing this guide. 1. Install it On Omarchy, use Install AI Ollama in the menu (Super + Alt + Space), or install from the repos. Pick the package that matches your GPU: bash yay -S ollama CPU only yay -S ollama-cuda NVIDIA yay -S ollama-rocm AMD sudo systemctl enable --now ollama On any other Linux, the official script does the same: curl -fsSL https://ollama.com/install.sh | sh. 2. Pull and run a model bash ollama pull qwen3:1.7b ollama run qwen3:1.7b --think=false "In one sentence: what does Ollama do?" Look at that answer. Ollama is not a model and has nothing to do with OpenAI, but a small model said so with total confidence. Small local models are great for classification, extraction, formatting and grounded answers. Don't trust them on facts from memory. Give them the facts (that's what RAG is for). ollama ps is the other half of that screenshot. It shows what's loaded, how much memory it takes, whether it's on GPU or CPU, and the context size it was loaded with. 3. Pick a model that fits your hardware Rules of thumb that have held up for me: - Memory: a 4-bit model needs roughly 0.6 to 0.7 GB per billion parameters, plus room for the context. A 7–8B model wants about 6 GB of VRAM; a 14B model about 10 GB. - Context costs memory too. The KV cache grows with numctx. On a test box with 8 GB of RAM, a 32k context on even a tiny model got the runner killed and every request returned HTTP 500. Dropping to 8k fixed it. - Start small, then go up. Try a 1.7B–4B model first to check your pipeline works, then move to the largest model that stays fast enough. bash ollama show qwen3:1.7b architecture, context length, capabilities ollama list what's on disk ollama rm <model free the space 4. Call it from code The CLI is a client for a REST API on localhost:11434. There's a native API and an OpenAI-compatible one, so most SDKs work by changing the base URL. In Python with the OpenAI SDK: python from openai import OpenAI client = OpenAI(baseurl="http://localhost:11434/v1", apikey="ollama") key is ignored r = client.chat.completions.create(model="qwen3:1.7b", messages=[{"role": "user", "content": "Summarise this log line: ..."}]) print(r.choices[0].message.content) 5. Make your own model with a Modelfile A Modelfile bakes a system prompt and parameters into a named model, so every client gets the same behaviour: Notice it answered with netstat, which is fine, but on Omarchy I'd use ss -ltnp | grep 11434. Small models give the most common answer, not always the best one. 6. Embeddings for search and RAG Ollama also serves embedding models. nomic-embed-text is small and good enough to build real retrieval on: The RAG guide uses exactly this to build a 30-line retrieval demo. 7. Settings worth changing Set these on the service so every client gets them: bash sudo systemctl edit ollama ini [Service] Environment="OLLAMACONTEXTLENGTH=65536" agents like OpenClaw need a big window Environment="OLLAMAKEEPALIVE=30m" keep the model warm between messages Environment="OLLAMAHOST=127.0.0.1:11434" don't expose it to the network bash sudo systemctl restart ollama Keep the server on 127.0.0.1. The API has no authentication, so anyone who can reach the port can use your GPU. Cheat sheet | Command | What it does | | --- | --- | | ollama pull <model | Download a model | | ollama run <model | Chat, or answer one prompt | | ollama ps | What's loaded, on GPU or CPU, and its context | | ollama show <model | Capabilities and parameters | | ollama create <name -f Modelfile | Build a custom model | | ollama launch openclaw | Set up OpenClaw on top of Ollama | Next: hooking Ollama up to Telegram with OpenClaw.
Setting Up Omarchy: My Arch + Hyprland Dev Laptop
Omarchy is DHH's opinionated Arch Linux setup: Hyprland as a tiling window manager, a themed terminal and Neovim, and every keyboard shortcut already decided for you. It's what my laptop runs, and it's where my local AI stack (Ollama and OpenClaw over Telegram) lives. This is the setup as I'd do it again, in order. Before you start - Use a dedicated drive. The installer wipes the disk you pick and sets up full-disk encryption. Dual-booting on one drive needs the manual install route. - Turn off Secure Boot (and TPM if your BIOS ties them together). The installer won't boot otherwise. - Have a wired or 2.4 GHz keyboard. You type the disk password before Bluetooth exists, so a Bluetooth-only keyboard can't unlock the machine. - Back up anything on that drive. It will be gone. 1. Flash the ISO and install Download the ISO from omarchy.org, write it to a USB stick (balenaEtcher on Windows/Mac, caligula on Linux), and boot from it. The installer asks a handful of questions: keyboard layout, username, password, hostname, timezone. Confirm them, pick the drive, and wait 2 to 10 minutes. 2. Learn these shortcuts on day one Everything in Omarchy is a keystroke. You don't need all of them; you need these, and Super + K shows the rest whenever you forget. The two that change how you work: Super + Alt + Space opens the Omarchy menu (install apps, change themes, update), and Super + 1–4 jumps between workspaces. I keep the editor on 1, browser on 2, terminals on 3 and chat on 4. 3. Update the Omarchy way Don't reach for pacman -Syu out of habit. Use Update Omarchy in the menu (or omarchy update in a terminal). It pulls new configs and runs migrations together with the package upgrade, so a new library version doesn't arrive without the config change it needs. Every update takes a Btrfs snapshot first. If something breaks, reboot, pick the pre-update snapshot in the Limine boot menu, and restore. One catch: snapshots restore the system, not /home. They won't bring back a deleted file, so keep real backups too. You can take one by hand before a risky change: bash omarchy-snapshot create 4. Set up the dev tools Omarchy already ships Neovim (LazyVim), Docker, mise for language runtimes, and AI coding CLIs (c opens OpenCode, cx opens Claude Code). What I add on top: bash Language runtimes through mise mise use -g node@24 python@3.12 Git identity git config --global user.name "Yashank Kothari" git config --global user.email "you@example.com" Anything else from the Arch repos or the AUR yay -S --needed ollama The omarchy command is worth a look too. Run it with no arguments to see every helper the menu uses (themes, fonts, screenshots, debug info). It's also what makes Omarchy easy for an AI agent to configure. 5. Make it yours, safely - Configs live in ~/.config (Hyprland, Waybar, terminal, Neovim). Edit there, not in the Omarchy source. - Try themes with Super + Ctrl + Shift + Space before you start editing colours by hand. - If you wreck your configs, omarchy reinstall resets them to the current release. It overwrites your changes, so commit ~/.config to a dotfiles repo first. What I'd tell myself before installing 1. Give it a week. Tiling feels slow for two days and then you can't go back. 2. Learn Super + K. It answers most questions. 3. Update through the menu, and trust the snapshots. 4. Put local AI on it. Next up: running models with Ollama.