GenAI
12 posts
I Gave an AI My Whole Life for a Month. Hark Gets a 10/10.
Rating: 10/10 Not an ad: nobody paid me, sponsored this, or asked me to write it. Full disclosure: I signed up early as an interested user, and Hark gave me their $100-a-month plan free for a month. That's how I got to push it this hard. Everything below is my honest take. I've used every AI chatbot you can name. You type a question, you get a nicely formatted paragraph, you copy it somewhere, and then you do the actual work yourself. Hark is the first one that broke that loop for me, and after a month I'm convinced it's what the next generation of AI looks like. The short version: it doesn't just answer. It does. It actually uses a computer Hark runs real browsers on its own machines. It signs in to sites, fills in forms, clicks through dashboards, and comes back with the result. Things I used to put off for weeks because they meant forty tabs and three logins just quietly get done while I'm at work. When something needs me (a code, an approval, a yes or no), it asks, and only then. It remembers me There's no "new chat" button, and that turns out to be the whole point. It's one long conversation with something that keeps track of my preferences, my projects, my routines and the things I said I'd do. I never re-explain context. It knows what I'm working toward and quietly gets the details right. It lives in my accounts, carefully With my mail, calendar, code and notes connected, it can triage an inbox, reshape a week, review a pull request, or file research straight into my notes. And it asks before anything that matters: spending money, sending something in my name, or anything public. I've never had it go rogue on me. It works in parallel This is the absurd part. Ask it for a big piece of research and it spins up several browsers at once, each chasing a different angle, then hands back one clean comparison. Deep market research, legal precedent digs, competitor teardowns, product audits: work that would eat my weekend comes back in the time it takes to make dinner. It builds things It writes and ships code, builds web pages, makes documents, slides and study guides, and has even put live panels on my home screen that keep themselves updated. I've had it test software by playing through thousands of rounds and report back what broke. It's proactive without being annoying Morning rundowns, reminders that land at the right time, watches on things I care about, and the occasional nudge about something I forgot. It feels less like an app and more like a sharp friend who has my back. The ADHD angle I have ADHD, and the hardest part of my day is never the work itself. It's starting, switching and remembering. Hark takes the switching and remembering off my plate. That alone has changed how much I get done. Anything I'd change? Honestly, not much. It's early, sometimes a website fights back, and big jobs take a few minutes. But it tells you straight when something's blocked, and it tries another way before giving up. Verdict Chatbots gave us answers. Hark gives you time back. This is the next generation of AI use: not a smarter search box, but something that actually takes things off your plate. 10/10, and again, this isn't sponsored. I'm just not going back. Try it at hark.com.
Evaluating LLM Apps: Treat Prompts Like Pipelines
A data pipeline without tests breaks quietly. An LLM feature without evaluations breaks quietly and changes behaviour whenever someone edits a prompt or the provider ships a new model version. The fix is the same discipline data engineers already use: a fixed dataset, automated checks and a number you watch. 1. Build a golden dataset Collect 50–200 real inputs: actual user questions, real documents, messy edge cases. For each, write down what a good output looks like. That could be an exact label, a required fact, or a short rubric. Version it in git next to the prompts. Start small. Thirty well-chosen examples beat none. 2. Use the cheapest check that works In order of preference: 1. Deterministic checks. Valid JSON? Matches the schema? Contains the required field? Correct label? These are fast, free and unambiguous. 2. Reference comparisons. Does the extracted amount equal the labelled amount? Did retrieval return the labelled document? 3. LLM-as-judge. A second model grades the output against a rubric ("Is every claim supported by the sources? yes/no, and why"). Useful for open-ended text, but spot-check the judge against human ratings before you trust it. 3. Score the parts separately For a RAG app, a single "quality" number hides the cause. Track retrieval (did the right chunk come back?), faithfulness (does the answer stick to the sources?) and answer quality on their own. For an agent, track task success, steps and cost per task. 4. Run evals like CI text prompt or model change │ ▼ run the golden set ──► scores vs. last release │ ▼ regression? ── yes ──► block the change │ no ▼ ship it Every prompt edit, model upgrade or retrieval change runs the suite. A drop in any metric beyond a small tolerance blocks the change, just as a failing dbt test blocks a deploy. 5. Keep watching in production - Log inputs, outputs, latency, token cost and the model version for every call. - Sample production traffic for review each week and add interesting failures to the golden set. - Collect cheap user signals such as thumbs up or down, and whether the user retried or edited the answer. 6. Don't forget cost and latency Quality is one axis. A change that improves accuracy by two points but doubles latency or cost per request is a trade-off, not a win. Put all three on the same dashboard. The teams that ship reliable GenAI features aren't the ones with the cleverest prompts. They're the ones that can tell, within minutes, whether a change made things better or worse. Try it: a five-question eval that fails Here's a tiny golden set with cheap graders (substring, exact match, regex) run against a 1.7B local model: bash curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/evaldemo.py python evaldemo.py This is the eval working, not breaking. It caught a model inventing a port number, contradicting a basic SQL fact, and running past the length budget. Swap in a bigger model or a better prompt, rerun, and you have a number to compare instead of a feeling.
Running Local LLMs with Ollama: A Hands-On Guide
Ollama runs open-weight models on your own machine behind a small HTTP server. No API key, no per-token bill, and your prompts never leave the laptop. It's the model backend for my OpenClaw Telegram assistant, and the easiest way I know to learn how LLMs behave. Every screenshot here is real terminal output, captured while writing this guide. 1. Install it On Omarchy, use Install AI Ollama in the menu (Super + Alt + Space), or install from the repos. Pick the package that matches your GPU: bash yay -S ollama CPU only yay -S ollama-cuda NVIDIA yay -S ollama-rocm AMD sudo systemctl enable --now ollama On any other Linux, the official script does the same: curl -fsSL https://ollama.com/install.sh | sh. 2. Pull and run a model bash ollama pull qwen3:1.7b ollama run qwen3:1.7b --think=false "In one sentence: what does Ollama do?" Look at that answer. Ollama is not a model and has nothing to do with OpenAI, but a small model said so with total confidence. Small local models are great for classification, extraction, formatting and grounded answers. Don't trust them on facts from memory. Give them the facts (that's what RAG is for). ollama ps is the other half of that screenshot. It shows what's loaded, how much memory it takes, whether it's on GPU or CPU, and the context size it was loaded with. 3. Pick a model that fits your hardware Rules of thumb that have held up for me: - Memory: a 4-bit model needs roughly 0.6 to 0.7 GB per billion parameters, plus room for the context. A 7–8B model wants about 6 GB of VRAM; a 14B model about 10 GB. - Context costs memory too. The KV cache grows with numctx. On a test box with 8 GB of RAM, a 32k context on even a tiny model got the runner killed and every request returned HTTP 500. Dropping to 8k fixed it. - Start small, then go up. Try a 1.7B–4B model first to check your pipeline works, then move to the largest model that stays fast enough. bash ollama show qwen3:1.7b architecture, context length, capabilities ollama list what's on disk ollama rm <model free the space 4. Call it from code The CLI is a client for a REST API on localhost:11434. There's a native API and an OpenAI-compatible one, so most SDKs work by changing the base URL. In Python with the OpenAI SDK: python from openai import OpenAI client = OpenAI(baseurl="http://localhost:11434/v1", apikey="ollama") key is ignored r = client.chat.completions.create(model="qwen3:1.7b", messages=[{"role": "user", "content": "Summarise this log line: ..."}]) print(r.choices[0].message.content) 5. Make your own model with a Modelfile A Modelfile bakes a system prompt and parameters into a named model, so every client gets the same behaviour: Notice it answered with netstat, which is fine, but on Omarchy I'd use ss -ltnp | grep 11434. Small models give the most common answer, not always the best one. 6. Embeddings for search and RAG Ollama also serves embedding models. nomic-embed-text is small and good enough to build real retrieval on: The RAG guide uses exactly this to build a 30-line retrieval demo. 7. Settings worth changing Set these on the service so every client gets them: bash sudo systemctl edit ollama ini [Service] Environment="OLLAMACONTEXTLENGTH=65536" agents like OpenClaw need a big window Environment="OLLAMAKEEPALIVE=30m" keep the model warm between messages Environment="OLLAMAHOST=127.0.0.1:11434" don't expose it to the network bash sudo systemctl restart ollama Keep the server on 127.0.0.1. The API has no authentication, so anyone who can reach the port can use your GPU. Cheat sheet | Command | What it does | | --- | --- | | ollama pull <model | Download a model | | ollama run <model | Chat, or answer one prompt | | ollama ps | What's loaded, on GPU or CPU, and its context | | ollama show <model | Capabilities and parameters | | ollama create <name -f Modelfile | Build a custom model | | ollama launch openclaw | Set up OpenClaw on top of Ollama | Next: hooking Ollama up to Telegram with OpenClaw.
My Own AI Assistant on Telegram: OpenClaw + Ollama on Omarchy
I message a Telegram bot from my phone, and an agent running on my Omarchy laptop answers, using a model served locally by Ollama. No cloud model, no open ports, and the agent can use tools on the machine I gave it. The piece in the middle is OpenClaw, an open-source assistant that connects chat apps to AI agents through a local gateway. Here's the exact setup, with real terminal output from each step. What you need - Ollama running with a model that supports tool calling (ollama show <model lists tools under capabilities). - Node 24.16+ (OpenClaw won't install on Node 22). On Omarchy: mise use -g node@24. - A Telegram account. 1. Install OpenClaw bash npm install -g openclaw@latest --allow-scripts=openclaw openclaw --version There's also a one-line installer (curl -fsSL https://openclaw.ai/install.sh | bash) and, if Ollama is already set up, ollama launch openclaw does the install, model pick and onboarding in one go. 2. Onboard it with Ollama as the model bash openclaw onboard --install-daemon The wizard asks for a model provider (choose Ollama), detects your local models, writes ~/.openclaw/openclaw.json, and installs the gateway as a systemd user service so it starts on login. The scriptable version of the same thing: The relevant part of the config it writes: json { "agents": { "defaults": { "model": { "primary": "ollama/qwen3:0.6b" } } }, "models": { "providers": { "ollama": { "baseUrl": "http://127.0.0.1:11434", "api": "ollama" } } }, "gateway": { "mode": "local", "port": 18789, "bind": "loopback" } } Keep bind on loopback. The gateway can run tools on your machine, so only local clients should reach it. 3. Check the gateway is up bash openclaw gateway status openclaw status 4. Create the Telegram bot 1. In Telegram, open a chat with @BotFather (check the handle exactly). 2. Send /newbot, pick a display name and a username ending in bot. 3. Copy the token it gives you. Treat it like a password. 5. Connect the bot to OpenClaw bash openclaw channels add --channel telegram --token <your-bot-token openclaw channels status --probe Or put it in ~/.openclaw/openclaw.json yourself: json { "channels": { "telegram": { "enabled": true, "botToken": "123456:ABC...", "dmPolicy": "pairing", "groups": { "": { "requireMention": true } } } } } The running gateway hot-reloads the change. dmPolicy: "pairing" is the important line: strangers who find your bot can't talk to your agent until you approve them. 6. Pair your own account Send your bot any message from your phone. That creates a pairing request. Approve it on the laptop: bash openclaw pairing list telegram openclaw pairing approve telegram <CODE Codes expire after an hour. Once approved, every message you send the bot goes to the agent, and its reply comes back in the chat. 7. Make the local model good enough This is where most local setups fall over: - Context window. OpenClaw's system prompt, tools and memory need room. Ollama's docs recommend at least 64k tokens for local models. Set OLLAMACONTEXTLENGTH=65536 on the Ollama service (how). - But context costs RAM. On an 8 GB test box, a 32k context got the model killed mid-load, and the agent just reported provider internal error, HTTP 500. If you see that, check journalctl -u ollama for a killed runner and lower the context or use a smaller model. - Model size. Tiny models can call tools but make poor decisions. Use the biggest tool-capable model your GPU runs comfortably, and keep a small one for quick tests. - Keep it warm. OLLAMAKEEPALIVE=30m stops the first message after a break from waiting for a model load. 8. Day-to-day commands | Command | Use | | --- | --- | | openclaw status | Health, sessions, channels at a glance | | openclaw logs --follow | Watch messages and tool calls live | | openclaw models list | Which model the agent uses | | openclaw doctor | Find and fix config problems | | openclaw gateway stop | Stop the gateway | | openclaw security audit | Check what the agent is allowed to do | Guardrails I'd keep - Pairing on, and only your Telegram user ID allowed. - Gateway on loopback, no port forwarding. - Run openclaw security audit after changing tools or skills, and keep risky tools behind approvals. - Keep the bot token and gateway token out of screenshots and git. None of the ones here show either. The agent loop running behind all this is the one from my AI agents guide, just with a phone on the other end.
AI Agents Without the Hype: Tools, Loops and Guardrails
An agent is an LLM running in a loop that can call tools. Instead of answering in one shot, it decides on an action, sees the result, and decides again until the task is done or it gives up. That's all it is. The engineering is in the tools, the loop's limits and what happens when it goes wrong. The loop python messages = [systemprompt, usertask] for step in range(MAXSTEPS): always cap it reply = llm(messages, tools=TOOLS) if reply.toolcall is None: return reply.text done result = runtool(reply.toolcall) validated, sandboxed messages += [reply, toolresult(result)] raise RuntimeError("step limit reached") Model providers support tool calling (also called function calling): you describe each tool with a name, a description and a JSON schema, and the model returns a structured call instead of prose. Design tools like an API for a junior engineer - Few, clear tools beat many vague ones. getpolicy(policyid) is better than runsql(query). - Write the description for the model. Say when to use the tool, what it returns and what it can't do. - Return compact, useful results. Twenty relevant rows, not a 10,000-row dump that fills the context window. - Return errors as text the model can act on: "policyid must look like POL-123456", not a stack trace. Start with a workflow, not an agent If the steps are known in advance (extract, then validate, then load), write them as a fixed pipeline with an LLM call in one or two steps. It's cheaper, faster and easier to test. Reach for a free-running agent only when the path genuinely depends on what it finds along the way. Guardrails that matter 1. Step and cost limits on every run. 2. Least privilege. Read-only tools by default; separate, narrow tools for writes. 3. A human approves anything irreversible: sending an email, paying, deleting, deploying. 4. Treat tool output as untrusted data. A web page or document can contain text that tries to give the agent instructions (prompt injection). Never let retrieved content grant new permissions. 5. Validate tool arguments against the schema before running anything. Make it observable Log every step: the prompt, the tool call, its arguments, the result, tokens used and time taken. When an agent fails, the trace is the only way to tell whether the model chose the wrong tool, the tool returned something confusing or the task was underspecified. Evaluate on tasks, not vibes Keep a set of realistic tasks with a known correct outcome and run the agent against them after every change to the prompt, tools or model. Track success rate, steps and cost per task. An agent that's 5% more accurate but three times as expensive may not be an improvement. Try it: a tool-calling agent in 25 lines This runs the loop above against a local model through Ollama's tool calling. One tool, an allowlist of paths, and a four-step budget: bash curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/agentdemo.py python agentdemo.py Try asking it about /etc and watch the allowlist return an error the model has to deal with. That's the guardrail doing its job. For a full agent with memory, skills and a chat front end, see how I run OpenClaw over Telegram.
Building a RAG Pipeline That Actually Retrieves
Retrieval-augmented generation (RAG) means: find the passages that answer a question, put them in the prompt, and ask the model to answer from those passages. It's the standard way to give an LLM knowledge it wasn't trained on, without fine-tuning. When a RAG system gives a bad answer, the generation step usually isn't the culprit. Retrieval handed the model the wrong text. So treat RAG as a data pipeline first. The two halves Indexing (offline, like any ETL job) 1. Load documents and clean them: strip boilerplate, keep titles and section headings. 2. Split them into chunks. 3. Embed each chunk and store the vector with its text and metadata (source, section, date, permissions). Query time 1. Embed the user's question. 2. Retrieve the top candidate chunks. 3. Optionally rerank them. 4. Build a prompt with the best few chunks and generate an answer with citations. Chunking decides most of your quality - Split on structure (headings, paragraphs) before falling back to a fixed size. - A few hundred tokens per chunk with a small overlap is a sensible starting point. Tune it against real questions. - Prepend the document title and section heading to each chunk before embedding, so "Section 4.2: Exclusions" still knows which policy it belongs to. Retrieve with more than one signal Pure vector search misses exact terms such as policy numbers, error codes and product names. Hybrid search combines keyword search (BM25) with vector search and merges the two result lists. Then a reranker (a cross-encoder that scores each question-chunk pair) reorders the top 20–50 candidates so the best five go in the prompt. Filter on metadata before ranking: date ranges, document type and, above all, the user's access permissions. Prompt for grounded answers text Answer the question using only the sources below. Cite sources as [1], [2]. If the sources don't contain the answer, say you don't know. [1] {chunk text, with title} [2] {chunk text, with title} Question: {user question} "Say you don't know" plus visible citations does more against hallucination than any amount of clever wording. Measure retrieval and answers separately Build a small evaluation set of 30–50 real questions, each labelled with the chunk or document that answers it. Then track: - Retrieval: is the right chunk in the top k (recall@k)? - Faithfulness: does the answer only claim what the sources say? - Answer quality: is it correct and does it actually answer the question? If recall@k is low, fix chunking, hybrid search or reranking. Tweaking the prompt won't help. Keep the index fresh Treat the index like a warehouse table: load incrementally, re-embed changed documents, delete removed ones, and re-embed everything when you change the embedding model, because vectors from different models aren't comparable. Try it: a 30-line RAG on your laptop No vector database needed to learn the moving parts. This uses Ollama's nomic-embed-text for embeddings, plain cosine similarity, and qwen3:1.7b to answer: bash ollama pull nomic-embed-text && ollama pull qwen3:1.7b curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/ragdemo.py python ragdemo.py That last part is the lesson. The answer is correct and cited, but it also rambles about warehouse auto-suspend because a weak second chunk made the cut. Two fixes: drop chunks below a score threshold, and tell the model to ignore sources that don't answer the question. Then measure retrieval on its own, as described above.
How LLMs Work: A Mental Model for Engineers
You don't need to train a large language model to build good things with one. You do need an accurate picture of what it does, because most bad LLM features come from expecting it to behave like a database or a search engine. It predicts the next token An LLM reads a sequence of tokens (chunks of text, often parts of words) and predicts a probability for every possible next token. It picks one, appends it, and repeats. Everything else (answering, summarising, writing SQL) emerges from doing that very well. Two consequences: - It generates plausible text, not retrieved facts. If the right answer wasn't in its training data or in the prompt, it will still produce something that sounds right. That's a hallucination, and it's the default behaviour, not a bug. - Cost and speed scale with tokens. You pay for input and output tokens, and long outputs are slow because they're generated one token at a time. The context window is its only working memory The model knows nothing about your conversation, your company or your data except what's in the context window for this request: the system prompt, the chat history, any documents you pasted. Close the request and it's gone. That's why "chat with your docs" products don't retrain anything. They find the relevant text and put it in the context. That pattern has a name: retrieval-augmented generation, or RAG. Sampling controls variety - Temperature near 0 makes the output close to deterministic. Use it for extraction, classification and SQL. - Higher temperature gives more varied wording. Use it for brainstorming and creative text. Even at temperature 0, don't assume byte-identical output across runs or model versions. Validate the result instead. Embeddings: meaning as numbers An embedding model turns text into a vector, a list of numbers, where texts with similar meaning land close together. Search "getting fit" and you can find a post about a workout routine that never uses those words. Embeddings power semantic search, deduplication, clustering and the retrieval step of RAG. Ask for structure When the output feeds code, ask for JSON that matches a schema, and validate it: python from pydantic import BaseModel class Claim(BaseModel): claimid: str amountinr: float category: str raw = llmcall(prompt) your provider's SDK claim = Claim.modelvalidatejson(raw) fails loudly on bad output Most providers now offer a structured-output or JSON mode. Use it, and still validate. Where LLMs fit in a data stack They're good at messy-text-to-structure jobs: classifying free-text claim notes, extracting fields from PDFs, writing a first draft of a SQL query, summarising incident logs. They're bad at arithmetic over large tables and at being the source of truth. Let the warehouse compute; let the model read and write language. Try it with a local model Everything above is easy to see on a laptop with Ollama. This script asks qwen3:1.7b for a product name four times at two temperatures, then asks for a log classification that has to match a JSON schema: bash ollama pull qwen3:1.7b curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/samplingdemo.py curl -O https://www.yashank.site/files/demos/structureddemo.py python samplingdemo.py && python structureddemo.py The token counts are worth a look too: a 20-word prompt came to 32 tokens once the chat template was added. That's the number you pay for and the number that fills the context window.
Embeddings Explained: Meaning as Vectors
Most people meet embeddings as the retrieval step in a RAG pipeline and never look further. That's a shame, because they're one of the most useful tools in applied machine learning, and they're far simpler than the models that generate text. What an embedding is An embedding is a list of numbers (a vector) that represents a piece of input, such as a sentence, a product description or an image. An embedding model is trained so that inputs with similar meaning end up with vectors that point in similar directions. "How do I reset my password?" and "I forgot my login" share almost no words, but a good text embedding model places them close together. "How do I reset my router?" lands somewhere nearby but noticeably further away. That's the whole trick. Once meaning is geometry, you can compare, group and search things using basic linear algebra. Cosine similarity The standard way to compare two embeddings is cosine similarity: the cosine of the angle between the vectors. text cos(a, b) = (a · b) / (|a| |b|) It ranges from -1 to 1. Close to 1 means the vectors point the same way, close to 0 means they're unrelated. In practice, with most text embedding models, scores cluster in a narrower band, so "0.8" doesn't have a universal meaning. You learn what a good score looks like for your model and your data. Cosine similarity ignores vector length and only looks at direction. That's usually what you want, since length often reflects things like input size rather than meaning. Normalisation If you normalise every vector to unit length (divide by its norm), cosine similarity becomes a plain dot product. That's faster to compute and it's what many vector databases assume. Some embedding APIs return normalised vectors already. Check the documentation rather than assuming. Normalising an already normalised vector is harmless, so when in doubt, do it once at write time and store the result. A small runnable example Real embeddings have hundreds or thousands of dimensions. These toy vectors have four, hand-made so the dimensions loosely mean "account", "networking", "billing" and "food". The maths is identical. python import numpy as np docs = { "reset my password": [0.9, 0.1, 0.0, 0.0], "forgot my login": [0.8, 0.2, 0.1, 0.0], "router keeps disconnecting": [0.1, 0.9, 0.0, 0.0], "wifi is very slow": [0.0, 0.8, 0.1, 0.1], "charged twice this month": [0.1, 0.0, 0.9, 0.0], "refund for my invoice": [0.2, 0.0, 0.8, 0.1], "best biryani nearby": [0.0, 0.0, 0.1, 0.9], } names = list(docs) X = np.array([docs[n] for n in names], dtype=float) X = X / np.linalg.norm(X, axis=1, keepdims=True) normalise once def search(queryvec, k=3): q = np.array(queryvec, dtype=float) q = q / np.linalg.norm(q) scores = X @ q cosine via dot product top = np.argsort(-scores)[:k] return [(names[i], round(float(scores[i]), 3)) for i in top] print("Search 'can't sign in':") for name, score in search([0.85, 0.15, 0.05, 0.0]): print(f" {score:.3f} {name}") Near-duplicate detection: pairs above a threshold sim = X @ X.T print("\nLikely duplicates ( 0.95):") for i in range(len(names)): for j in range(i + 1, len(names)): if sim[i, j] 0.95: print(f" {sim[i, j]:.3f} {names[i]!r} ~ {names[j]!r}") Classification by nearest labelled centroid labels = {"account": [0, 1], "network": [2, 3], "billing": [4, 5]} centroids = {k: X[idx].mean(axis=0) for k, idx in labels.items()} centroids = {k: v / np.linalg.norm(v) for k, v in centroids.items()} ticket = np.array([0.15, 0.05, 0.85, 0.05]) ticket = ticket / np.linalg.norm(ticket) best = max(centroids, key=lambda k: centroids[k] @ ticket) print(f"\nNew ticket classified as: {best}") Swap the toy vectors for output from a real embedding model and the same code works for search, deduplication and classification. Uses beyond RAG - Semantic search. Find documents by meaning rather than keywords. Often best combined with keyword search (hybrid search), because embeddings can miss exact terms like product codes. - Deduplication. Near-identical support tickets, product listings or records from different sources score very high against each other. A threshold plus human review of the borderline band goes a long way. - Clustering. Run k-means or a density-based method on embeddings to find themes in thousands of free-text responses without labelling anything first. - Classification. Train a simple logistic regression on embeddings, or use nearest centroid as above. With a few hundred labelled examples this is often competitive with much heavier approaches, and far cheaper to run. - Anomaly detection. Items far from every cluster are worth a look. Choosing dimensions More dimensions can capture more nuance, but cost more to store and compare. A million 1,536-dimensional float32 vectors is roughly 6 GB before any index overhead. Some newer models are trained so you can truncate vectors to fewer dimensions with modest quality loss. If yours supports that, test a smaller size on your own retrieval eval before defaulting to the largest. Re-normalise after truncating. Pitfalls - Mixing models. Vectors from different embedding models (or different versions of the same one) are not comparable. Changing models means re-embedding everything. Store the model name alongside each vector. - Assuming a universal threshold. A similarity cut-off that works for one model and domain won't transfer. Calibrate on labelled pairs. - Long inputs. Models have input limits and may truncate silently. Embedding a whole document into one vector also blurs its topics together. Chunk sensibly. - Domain mismatch. A general model may not separate terms that matter in your field, like insurance policy types. Test on your own data. - Exact matches. Embeddings are weak at IDs, numbers and rare names. Keep keyword search around. Where to start - Pick one embedding model, record its name with every vector, and normalise at write time. - Build a small labelled set of similar and dissimilar pairs from your own data, and use it to choose thresholds and dimensions. - Try deduplication or clustering before RAG. They're simpler, easier to evaluate and often more immediately useful. - Use hybrid search when exact terms matter.
Prompting, RAG or Fine-Tuning? Picking the Right Lever
When an LLM doesn't do what you want, there are three main levers: change the prompt, give it better context with retrieval, or change the model itself with fine-tuning. They get discussed as if they compete. They don't. They fix different problems, and picking the wrong one is an expensive way to not solve yours. The one distinction that matters Before anything else, ask whether the problem is about knowledge or behaviour. - Knowledge: the model doesn't know something. Your internal docs, last week's policy change, a customer's order history. - Behaviour: the model knows enough but responds the wrong way. Wrong format, wrong tone, ignores a rule, too verbose, inconsistent at a narrow task. Retrieval mostly fixes knowledge. Fine-tuning mostly fixes behaviour. Prompting can help with both, a bit, and it's nearly free, so it always comes first. Prompting Prompting changes nothing about the model. You're just giving it better instructions for this request. What it's good at: - clear task definitions and constraints - output formats, especially combined with structured-output or JSON modes - a few worked examples (few-shot prompting) to show the pattern you want - small amounts of reference information pasted directly in Cost: basically your time, plus the extra input tokens. Data needed: a handful of examples. You can iterate in minutes. Where it runs out: when the information you need is too large or changes too often to paste in, or when you've written a page of rules and the model still drifts on some of them. RAG Retrieval-augmented generation keeps the model unchanged and adds a search step. At query time you find the most relevant chunks of your data and put them in the context window alongside the question. python def answer(question: str) - str: chunks = vectorstore.search(embed(question), k=5) retrieve context = "\n\n".join(c.text for c in chunks) prompt = ( "Answer using only the context below. " "If the answer isn't there, say so.\n\n" f"Context:\n{context}\n\nQuestion: {question}" ) return llmcall(prompt) generate What it changes: what the model knows for this request. Nothing is learned permanently. What it's good at: - large or frequently changing knowledge bases - answers that need citations, since you know exactly which chunks were used - access control, because you can filter retrieval by what the user is allowed to see - keeping information current: update the index, and the next answer reflects it Cost: an embedding model, a vector store (or a search engine you already run), and the engineering to chunk, index and keep data fresh. More input tokens per call. Data needed: your documents. No labelled examples required to get started, though you'll want some question–answer pairs to evaluate retrieval quality. Where it struggles: most RAG failures are retrieval failures. If the right chunk isn't found, the model can't use it. Bad chunking, weak queries and missing metadata cause more trouble than the choice of model. Hybrid search (keyword plus embeddings) and reranking help a lot. Fine-tuning Fine-tuning continues training the model on your own examples, so the weights change. What it changes: behaviour. How the model responds by default: format, style, terminology, how it handles a narrow task. It's a poor way to add facts. The model may pick some up, but unreliably, they go stale, and you can't cite them. Use retrieval for facts. What it's good at: - consistent output format or tone without a long prompt every time - narrow, repetitive tasks such as classification, extraction and routing, where a smaller tuned model can match a larger general one at lower cost and latency - shortening prompts, which cuts tokens on every call LoRA Full fine-tuning updates every weight, which needs a lot of GPU memory. LoRA (low-rank adaptation) freezes the original weights and trains small extra matrices added to certain layers. You end up with an adapter that's a small fraction of the model's size. Why it matters in practice: - far less memory to train, so it fits on modest hardware (QLoRA goes further by quantising the base model) - you can keep several adapters for different tasks on one base model - results are often close to full fine-tuning for task adaptation Cost: preparing data is usually the biggest cost, followed by training runs, evaluation and hosting the tuned model. Data needed: hundreds to thousands of good input–output examples, consistent and representative. Fifty sloppy examples will teach the model to be sloppy. The hidden cost is maintenance. When a better base model comes out, you retrain. When requirements change, you rebuild the dataset. They combine These aren't exclusive. A common mature setup is a model fine-tuned for format and domain behaviour, fed by RAG for current facts, with a solid prompt holding it together. But each layer should earn its place. A decision checklist Work through these in order: 1. Have I written a clear prompt with a few examples and tested it on real inputs? If not, do that first. 2. Is the failure about missing or changing information? Use RAG. 3. Do I need citations, freshness or per-user access control? Use RAG. 4. Is the failure about format, tone or consistency, after good prompting? Consider fine-tuning. 5. Do I have hundreds of high-quality examples, or can I build them? If not, fine-tuning isn't ready yet. 6. Is cost or latency the problem at high volume on a narrow task? A small model fine-tuned with LoRA may beat a large one. 7. Do I have an evaluation set to prove each change helped? Without it, none of these decisions are measurable. In short - Prompt first. It's cheap and fast, and often enough. - Knowledge problem: retrieve. Behaviour problem: fine-tune. - Spend effort on retrieval quality before blaming the model. - Fine-tune only with good data and an eval set, and prefer LoRA unless you have a reason not to.
Open-Weight vs Closed Models: How I Choose
Every LLM project I start hits the same fork early: call a hosted API, or run an open-weight model myself. It's a trade-off across a handful of dimensions, and once you list them the answer for a given project is usually obvious. What the two camps actually are Closed models are the ones you reach through a provider's API: the GPT, Claude and Gemini families. You send a request, you get tokens back, and you never see the weights. Open-weight models publish their weights so you can download and run them: the Llama, Qwen, Mistral and Gemma families, among others. "Open-weight" is the honest term. Most of them don't release training data or the full training recipe, so they aren't open source in the sense the term usually carries for software. Neither group is uniformly better. The best closed models tend to lead on the hardest reasoning tasks, but the gap on everyday work like classification, extraction and summarisation is often small, and a well-chosen open model can be the better tool. Privacy and data residency This is usually the first filter, and sometimes the only one that matters. With a closed API, your prompts leave your infrastructure. Providers offer enterprise terms, zero-retention options and regional hosting, and for many teams that's enough. For others, particularly anything touching financial, health or customer records, the policy answer is simply "data does not leave our network". In that case an open-weight model running on your own hardware or a private cloud deployment isn't a preference, it's the requirement. Cost The cost models are different in shape, not just size. - Closed APIs charge per token. Zero cost when idle, cost grows linearly with use. - Self-hosted open models cost roughly the same whether you send one request or a million, because you're paying for GPUs by the hour. Cheap per request at high, steady volume; expensive if the machine sits idle. A rough way to think about it: estimate monthly tokens, multiply by the API price, and compare that with the cost of the smallest GPU setup that can serve the model at the latency you need. At low volume the API almost always wins. At high, predictable volume, self-hosting starts to look attractive, provided someone is going to run it. Latency Hosted APIs are fast but you don't control them. You share capacity with everyone else, and response times can vary through the day. Self-hosting gives you control: you can put a small model next to your application, keep it warm, and get consistent latency. A small open model that answers in a fraction of a second can beat a much stronger remote model for anything interactive, like autocomplete or routing a request to the right handler. Quality Measure it on your task, not on a leaderboard. Build a small evaluation set (fifty to a few hundred real examples with known good answers) and run every candidate against it. What I usually find: - For open-ended reasoning, long multi-step instructions and tricky code, the top closed models are still the safest bet. - For narrow, well-defined jobs, a mid-sized open model often lands close enough, especially with a good prompt and a few examples. "Close enough" is a product decision. A 2% drop in accuracy may be fine for tagging support tickets and unacceptable for anything a customer reads directly. Licensing Open-weight doesn't mean do whatever you like. Licences vary across families: some are permissive (Apache 2.0 style), others are custom licences with acceptable-use policies, attribution rules or restrictions above a certain user count. Read the licence for the specific model before building a product on it, and check it again when you upgrade, because terms can change between releases. Closed APIs have their own terms too, usually around what you can use outputs for. Read those as well. Fine-tunability With open weights you can fine-tune however you want: full fine-tuning, or parameter-efficient methods like LoRA that train a small set of extra weights. You own the result and can run it anywhere. Some closed providers offer fine-tuning through their API, but on their terms: limited model choice, limited control over the process, and the tuned model lives on their platform. If adapting the model's behaviour deeply is central to the project, open weights give you far more room. Ops burden This is the dimension people underestimate. Running a model means: - provisioning and paying for GPUs - choosing and maintaining a serving stack (vLLM, TGI, llama.cpp or similar) - handling scaling, monitoring, failover and upgrades - managing quantisation trade-offs between memory and quality An API call is one line of code and someone else's pager. If there's no one on the team who wants to own inference infrastructure, that's a strong vote for closed. How it plays out A simplified version of my decision: text Must data stay in-house? - open-weight, self-hosted Prototype or low/spiky volume? - closed API Hardest reasoning, top quality? - closed API (check with evals) Narrow task, high steady volume? - open-weight, possibly fine-tuned Need deep behaviour changes? - open-weight + LoRA No one to run infra? - closed API, or a managed host for open models The last line matters: there's a middle path. Several providers host open-weight models behind an API, so you get the model choice and licence of open weights with per-token pricing and no GPUs to babysit. The practical bit - Start with a closed API to prove the feature is worth building. It's the fastest way to learn what "good" looks like. - Build an evaluation set from day one, so switching models is a measurement rather than a debate. - Keep the model behind a thin interface in your code. Swapping providers or moving to a self-hosted model should be a config change, not a rewrite. - Revisit the choice when volume, data sensitivity or the model landscape changes. It isn't a decision you make once.
Prompt Engineering Is Mostly Specification Writing
"Prompt engineering" sounds like a bag of tricks: magic phrases, threats, tipping the model. Some of those tricks worked once, on some model, for some task. What keeps working is far more boring. A good prompt is a specification: it says what the input is, what the output must look like, what's out of scope, and how you'll know it's right. If you've ever written a ticket that a stranger could pick up and finish without asking you anything, you already know how to write most of a good prompt. Treat the prompt as a contract The model has no idea what you're building. It sees text and predicts more text. Every gap in your instructions gets filled with whatever is most plausible, and plausible is not the same as what you wanted. So write the prompt the way you'd write an interface contract: - Inputs: what the model will receive and in what form. - Outputs: the exact shape of the response. - Constraints: what it must not do. - Edge cases: what to do when the input is empty, ambiguous or off-topic. Vague prompt: text Summarise this customer complaint. Specified prompt: text You will receive one customer complaint email for a food delivery app. Return a JSON object with: - "category": one of "late", "wrongitem", "refund", "appbug", "other" - "summary": one sentence, under 25 words, no customer names - "needshuman": true if the customer mentions legal action or a health issue If the email is not a complaint, return {"category": "other", "summary": "", "needshuman": false}. The second one is longer, but every extra line removes a decision the model would otherwise make for you. Roles are context, not magic "You are a senior data engineer" doesn't make a model smarter. What a role line does is set context: vocabulary, assumed audience, level of detail. That's useful, but it's a small lever. A role with no task description is still a vague prompt. I find it more reliable to describe the situation directly: who will read the output, what they already know, and what they'll do with it. "The reader is a support agent who has 30 seconds to decide whether to escalate" tells the model more than any job title. Constraints do the heavy lifting Most bad outputs aren't wrong, they're unbounded: too long, too chatty, padded with caveats, or inventing details. Constraints fix that. - Length: "under 25 words", "at most three bullet points". - Scope: "use only the information in the provided text". - Refusal path: "if the answer isn't in the text, say NOTFOUND". - Tone: "plain language, no marketing words". Giving the model an explicit way to say "I don't know" matters more than people expect. Without one, the most plausible continuation is usually a confident guess. Examples beat adjectives "Write a concise, professional summary" means different things to different people, and to models. One or two examples of input and expected output pin down what you mean far better than adjectives. A few things I've learned about examples: - Make them representative, including at least one awkward case. - Vary them. If every example has three bullet points, every output will too. - Keep them consistent with your stated rules. If the instructions say 25 words and the example has 40, you've written a contradiction and the model will pick one. Ask for a schema when code reads the output If the response goes into a program, don't parse prose. Ask for JSON with named fields and allowed values, and validate it on the way in: python from typing import Literal from pydantic import BaseModel, Field class Complaint(BaseModel): category: Literal["late", "wrongitem", "refund", "appbug", "other"] summary: str = Field(maxlength=200) needshuman: bool raw = callmodel(prompt, emailtext) result = Complaint.modelvalidatejson(raw) raises on bad output Many APIs now have a JSON or structured-output mode. Use it, and still validate. A schema in the prompt plus a schema in code is the same contract written down twice, which is exactly what you want. Specs need tests A specification you never check is a wish. The step most people skip is evaluation: running the prompt against a fixed set of inputs and checking the outputs, every time you change something. It doesn't need to be fancy: python cases = [ ("My order came 2 hours late and cold.", "late"), ("I got paneer instead of chicken.", "wrongitem"), ("Hi, what are your opening hours?", "other"), ] failures = [] for text, expected in cases: out = Complaint.modelvalidatejson(callmodel(prompt, text)) if out.category != expected: failures.append((text, expected, out.category)) print(f"{len(cases) - len(failures)}/{len(cases)} passed") Twenty or thirty hand-picked cases catch most regressions. When you find a new failure in the wild, add it to the set. This is ordinary test-driven thinking applied to a component that happens to be probabilistic. For outputs that can't be checked with equality, such as summaries, you can check properties instead: length limits, required fields, banned words, whether named entities in the output actually appear in the input. Iterate like you'd debug When a prompt fails, resist the urge to add another sentence in capital letters. Look at the failing case and ask what the spec left open. Usually it's one of these: - An edge case nobody described. - Two instructions that conflict. - An example that teaches the wrong pattern. - A task that's really two tasks and should be two calls. Change one thing, rerun the eval set, compare. It's slower than vibes but you'll know whether you made it better. What to keep in mind - Write the prompt so a capable stranger could do the task from it alone. - Define the output shape exactly, and validate it in code. - Give the model an explicit way to say it doesn't know. - Use a couple of varied examples instead of adjectives. - Keep a small test set and rerun it on every change. Most of what gets called prompt engineering is just being precise about what you want. That skill was useful before LLMs and it'll stay useful after the current tricks stop working.
Attention, Explained for Engineers
Every explanation of transformers I found while studying either stopped at "attention lets the model focus on relevant words" or jumped straight into a wall of matrix notation. The first is too vague to reason with. The second hides a fairly simple idea. This is the middle version: enough to understand why models behave the way they do, and why long prompts cost what they cost, with code you can run. Text becomes tokens A model never sees characters or words directly. A tokenizer splits text into tokens, chunks from a fixed vocabulary that are often whole common words, word pieces, or punctuation. "Unbelievably" might become un, believ, ably. Each token maps to an integer ID. This is why models are oddly bad at counting letters in a word: they never saw the letters, only the chunks. Tokens become embeddings Each token ID looks up a row in an embedding matrix, giving a vector of a few hundred to a few thousand numbers. These vectors are learned during training, and tokens used in similar contexts end up with similar vectors. So a sentence of n tokens becomes a matrix of shape (n, dmodel). Everything after this is arithmetic on that matrix. The problem attention solves A token's meaning depends on its neighbours. "Bank" in "river bank" and "bank account" should end up represented differently. Older recurrent networks passed information along the sequence one step at a time, which was slow to train and tended to lose track of distant words. Attention lets every token look at every other token directly, in one step, and decide how much each one matters to it. Queries, keys and values For each token, the model computes three vectors by multiplying its embedding with three learned weight matrices: - Query (Q): what this token is looking for. - Key (K): what this token offers to others looking. - Value (V): the information it actually passes along if chosen. The analogy I find useful is a fuzzy dictionary lookup. A normal dictionary matches a query to exactly one key and returns its value. Attention compares a query against every key, scores how well each matches, and returns a weighted blend of all the values. Scaled dot-product attention The scoring is a dot product: Q · K is large when the two vectors point in similar directions. For a whole sequence at once: Attention(Q, K, V) = softmax(Q Kᵀ / √dk) V Step by step: 1. Q Kᵀ gives an (n, n) matrix of scores: how much each token cares about each other token. 2. Divide by √dk (the key dimension). Without this, dot products grow with dimension, softmax saturates, and gradients become tiny. 3. Softmax each row, turning scores into weights that are positive and sum to 1. 4. Multiply by V, so each token's output is a weighted average of the value vectors. Here it is in NumPy: python import numpy as np def softmax(x, axis=-1): x = x - x.max(axis=axis, keepdims=True) for numerical stability e = np.exp(x) return e / e.sum(axis=axis, keepdims=True) def attention(Q, K, V, causal=False): dk = Q.shape[-1] scores = Q @ K.T / np.sqrt(dk) (seq, seq) if causal: mask = np.triu(np.oneslike(scores, dtype=bool), k=1) scores = np.where(mask, -np.inf, scores) weights = softmax(scores, axis=-1) each row sums to 1 return weights @ V, weights rng = np.random.defaultrng(0) seqlen, dmodel, dk = 4, 8, 8 X = rng.normal(size=(seqlen, dmodel)) 4 token embeddings Wq = rng.normal(size=(dmodel, dk)) Wk = rng.normal(size=(dmodel, dk)) Wv = rng.normal(size=(dmodel, dk)) out, w = attention(X @ Wq, X @ Wk, X @ Wv, causal=True) print(out.shape) (4, 8) print(w.round(2)) lower-triangular: no token sees the future The causal flag is what text-generating models use. Setting future positions to negative infinity before softmax makes their weights zero, so token 3 can attend to tokens 1 to 3 but not 4. That's what lets the model be trained to predict the next token without cheating. In a real model the weight matrices are learned, not random. Multi-head attention One attention pattern can only capture one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own smaller Q, K and V projections. One head might learn to track which noun a pronoun refers to; another might focus on the previous token. The heads' outputs are concatenated and projected back to dmodel. A transformer block is multi-head attention followed by a small feed-forward network applied to each token separately, with residual connections and normalisation around both. A model stacks dozens of these blocks. Where does word order come from? Look at the formula again: nothing in it knows about position. Shuffle the input tokens and each token's output is the same, just shuffled. Attention on its own treats a sentence as a bag of tokens. So position has to be injected. The original transformer paper added fixed sine and cosine patterns to the embeddings. Some models learn a position embedding per slot. Many recent models use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position, so their dot product reflects relative distance. The details vary; the point is that order is added information, not something attention gets for free. Why context windows cost what they do The score matrix is n × n. Double the prompt length and you quadruple the number of query-key comparisons, per head, per layer. That quadratic cost is the main reason context windows were small for so long and why long prompts are slow and expensive. There's a second cost during generation. To avoid recomputing everything for each new token, models keep a KV cache: the keys and values for every previous token, at every layer. It grows linearly with context length and can eat a lot of GPU memory, which limits how many requests a server can handle at once. Plenty of engineering goes into softening this, from memory-efficient attention kernels to sparse and sliding-window attention patterns. But the basic shape holds: more context means more compute and memory, and providers price accordingly. The short version - Text becomes tokens, tokens become vectors. - Attention scores every token against every other with Q · K, softmaxes the scores, and blends the values. - Multiple heads learn different relationships; position has to be added separately. - The n × n score matrix and the KV cache are why long contexts are expensive. If you remember one thing, make it the fuzzy dictionary lookup. Most transformer behaviour, good and odd, makes more sense once you picture every token asking every other token "are you relevant to me?" at every layer.