Ollama
2 posts
Running Local LLMs with Ollama: A Hands-On Guide
Ollama runs open-weight models on your own machine behind a small HTTP server. No API key, no per-token bill, and your prompts never leave the laptop. It's the model backend for my OpenClaw Telegram assistant, and the easiest way I know to learn how LLMs behave. Every screenshot here is real terminal output, captured while writing this guide. 1. Install it On Omarchy, use Install AI Ollama in the menu (Super + Alt + Space), or install from the repos. Pick the package that matches your GPU: bash yay -S ollama CPU only yay -S ollama-cuda NVIDIA yay -S ollama-rocm AMD sudo systemctl enable --now ollama On any other Linux, the official script does the same: curl -fsSL https://ollama.com/install.sh | sh. 2. Pull and run a model bash ollama pull qwen3:1.7b ollama run qwen3:1.7b --think=false "In one sentence: what does Ollama do?" Look at that answer. Ollama is not a model and has nothing to do with OpenAI, but a small model said so with total confidence. Small local models are great for classification, extraction, formatting and grounded answers. Don't trust them on facts from memory. Give them the facts (that's what RAG is for). ollama ps is the other half of that screenshot. It shows what's loaded, how much memory it takes, whether it's on GPU or CPU, and the context size it was loaded with. 3. Pick a model that fits your hardware Rules of thumb that have held up for me: - Memory: a 4-bit model needs roughly 0.6 to 0.7 GB per billion parameters, plus room for the context. A 7–8B model wants about 6 GB of VRAM; a 14B model about 10 GB. - Context costs memory too. The KV cache grows with numctx. On a test box with 8 GB of RAM, a 32k context on even a tiny model got the runner killed and every request returned HTTP 500. Dropping to 8k fixed it. - Start small, then go up. Try a 1.7B–4B model first to check your pipeline works, then move to the largest model that stays fast enough. bash ollama show qwen3:1.7b architecture, context length, capabilities ollama list what's on disk ollama rm <model free the space 4. Call it from code The CLI is a client for a REST API on localhost:11434. There's a native API and an OpenAI-compatible one, so most SDKs work by changing the base URL. In Python with the OpenAI SDK: python from openai import OpenAI client = OpenAI(baseurl="http://localhost:11434/v1", apikey="ollama") key is ignored r = client.chat.completions.create(model="qwen3:1.7b", messages=[{"role": "user", "content": "Summarise this log line: ..."}]) print(r.choices[0].message.content) 5. Make your own model with a Modelfile A Modelfile bakes a system prompt and parameters into a named model, so every client gets the same behaviour: Notice it answered with netstat, which is fine, but on Omarchy I'd use ss -ltnp | grep 11434. Small models give the most common answer, not always the best one. 6. Embeddings for search and RAG Ollama also serves embedding models. nomic-embed-text is small and good enough to build real retrieval on: The RAG guide uses exactly this to build a 30-line retrieval demo. 7. Settings worth changing Set these on the service so every client gets them: bash sudo systemctl edit ollama ini [Service] Environment="OLLAMACONTEXTLENGTH=65536" agents like OpenClaw need a big window Environment="OLLAMAKEEPALIVE=30m" keep the model warm between messages Environment="OLLAMAHOST=127.0.0.1:11434" don't expose it to the network bash sudo systemctl restart ollama Keep the server on 127.0.0.1. The API has no authentication, so anyone who can reach the port can use your GPU. Cheat sheet | Command | What it does | | --- | --- | | ollama pull <model | Download a model | | ollama run <model | Chat, or answer one prompt | | ollama ps | What's loaded, on GPU or CPU, and its context | | ollama show <model | Capabilities and parameters | | ollama create <name -f Modelfile | Build a custom model | | ollama launch openclaw | Set up OpenClaw on top of Ollama | Next: hooking Ollama up to Telegram with OpenClaw.
My Own AI Assistant on Telegram: OpenClaw + Ollama on Omarchy
I message a Telegram bot from my phone, and an agent running on my Omarchy laptop answers, using a model served locally by Ollama. No cloud model, no open ports, and the agent can use tools on the machine I gave it. The piece in the middle is OpenClaw, an open-source assistant that connects chat apps to AI agents through a local gateway. Here's the exact setup, with real terminal output from each step. What you need - Ollama running with a model that supports tool calling (ollama show <model lists tools under capabilities). - Node 24.16+ (OpenClaw won't install on Node 22). On Omarchy: mise use -g node@24. - A Telegram account. 1. Install OpenClaw bash npm install -g openclaw@latest --allow-scripts=openclaw openclaw --version There's also a one-line installer (curl -fsSL https://openclaw.ai/install.sh | bash) and, if Ollama is already set up, ollama launch openclaw does the install, model pick and onboarding in one go. 2. Onboard it with Ollama as the model bash openclaw onboard --install-daemon The wizard asks for a model provider (choose Ollama), detects your local models, writes ~/.openclaw/openclaw.json, and installs the gateway as a systemd user service so it starts on login. The scriptable version of the same thing: The relevant part of the config it writes: json { "agents": { "defaults": { "model": { "primary": "ollama/qwen3:0.6b" } } }, "models": { "providers": { "ollama": { "baseUrl": "http://127.0.0.1:11434", "api": "ollama" } } }, "gateway": { "mode": "local", "port": 18789, "bind": "loopback" } } Keep bind on loopback. The gateway can run tools on your machine, so only local clients should reach it. 3. Check the gateway is up bash openclaw gateway status openclaw status 4. Create the Telegram bot 1. In Telegram, open a chat with @BotFather (check the handle exactly). 2. Send /newbot, pick a display name and a username ending in bot. 3. Copy the token it gives you. Treat it like a password. 5. Connect the bot to OpenClaw bash openclaw channels add --channel telegram --token <your-bot-token openclaw channels status --probe Or put it in ~/.openclaw/openclaw.json yourself: json { "channels": { "telegram": { "enabled": true, "botToken": "123456:ABC...", "dmPolicy": "pairing", "groups": { "": { "requireMention": true } } } } } The running gateway hot-reloads the change. dmPolicy: "pairing" is the important line: strangers who find your bot can't talk to your agent until you approve them. 6. Pair your own account Send your bot any message from your phone. That creates a pairing request. Approve it on the laptop: bash openclaw pairing list telegram openclaw pairing approve telegram <CODE Codes expire after an hour. Once approved, every message you send the bot goes to the agent, and its reply comes back in the chat. 7. Make the local model good enough This is where most local setups fall over: - Context window. OpenClaw's system prompt, tools and memory need room. Ollama's docs recommend at least 64k tokens for local models. Set OLLAMACONTEXTLENGTH=65536 on the Ollama service (how). - But context costs RAM. On an 8 GB test box, a 32k context got the model killed mid-load, and the agent just reported provider internal error, HTTP 500. If you see that, check journalctl -u ollama for a killed runner and lower the context or use a smaller model. - Model size. Tiny models can call tools but make poor decisions. Use the biggest tool-capable model your GPU runs comfortably, and keep a small one for quick tests. - Keep it warm. OLLAMAKEEPALIVE=30m stops the first message after a break from waiting for a model load. 8. Day-to-day commands | Command | Use | | --- | --- | | openclaw status | Health, sessions, channels at a glance | | openclaw logs --follow | Watch messages and tool calls live | | openclaw models list | Which model the agent uses | | openclaw doctor | Find and fix config problems | | openclaw gateway stop | Stop the gateway | | openclaw security audit | Check what the agent is allowed to do | Guardrails I'd keep - Pairing on, and only your Telegram user ID allowed. - Gateway on loopback, no port forwarding. - Run openclaw security audit after changing tools or skills, and keep risky tools behind approvals. - Keep the bot token and gateway token out of screenshots and git. None of the ones here show either. The agent loop running behind all this is the one from my AI agents guide, just with a phone on the other end.