Guides
22 posts
How I debloated my slow TCL Android TV (and what went wrong along the way)
My TCL TV had become painfully slow. It's an Android 11 set with only about 940 MB of RAM, and when I started, around 300 MB of that was free. Menus lagged, apps took ages to open, and the home screen was full of sponsored rows and ads. I fixed most of it in one evening using an open-source project called tv-detox and Claude Code to drive the whole process from my laptop. Here's what I did, what worked, and the mistakes worth knowing about before you try it. Disclaimer: This is what worked on my TV, not professional advice. Everything below changes system settings on your device. Use it at your own risk, and read the "undo" section before you start. The idea Android TVs ship with a lot of apps you never asked for: manufacturer account apps, analytics services, ad feeds, demo-mode helpers, and Google's own sponsored home screen row. You can't uninstall most of them, but you can turn them off over adb, Android's debugging bridge. Turned-off apps stop using RAM and CPU, and they stay on the device, so everything is reversible. tv-detox packages that approach: a planner script that classifies every installed app, a tiny replacement home screen, and an undo script. What you need - A laptop on the same network as the TV - adb installed (android-platform-tools) - Developer mode on the TV: Settings → Device Preferences → About, press Build seven times, then enable USB debugging (or network debugging) in Developer options Step 1: Connect I found the TV's IP address in its network settings and ran: sh adb connect <TVIP:5555 The first connection showed as unauthorized. The TV pops up an "Allow USB debugging?" prompt that disappears after a short time. I had to reconnect and approve it quickly, ticking "Always allow". Step 2: Look before touching anything tv-detox includes a read-only planner: sh sh candidates.sh <TVIP It labelled every app as one of: - never: the TV needs it (System UI, Google Play services, Bluetooth, the HDMI input framework, TCL's settings store) - keep: carries a feature you might use (casting, voice search, the phone remote) - safe: common preinstalled junk - review: unknown, so leave it alone unless you know it On my TV that was 54 never, 10 keep, 22 safe and 55 to review. Step 3: Turn off the junk I disabled all 22 "safe" apps. Nothing was uninstalled. Each package name was written to a log on the TV first, so the undo script knows what to switch back on. The list covered: - Google ads and extras: the home-screen backdrop ads, the sponsored recommendations row, Google TV (Play Movies), Play Games, YouTube Music, and the feedback app - TCL bloat: user centre, an ads feed service, analytics, message box and promos, built-in Alexa, store-demo sticker, phone-migration tool, key help, dashboard, launcher customizer, and the audio and picture players - Android extras: the basic screensaver, an HTML viewer, the print spooler and the sound picker The command for each one is simply: sh adb shell pm disable-user --user 0 <package Step 4: Replace the home screen tv-detox ships a small replacement launcher: one screen with your apps, your HDMI inputs and a clock. It uses around 20-30 MB of RAM, which is a big saving compared to Google TV's launcher. sh sh install.sh <TVIP The installer also turns off Google's stock launcher so that pressing Home opens the new one. It has a safety net: if Home doesn't open the new launcher after the change, it automatically turns everything back on, so you can't end up with a TV that has no home screen. Hiccup 1: "not enough space" The first install failed with Requested internal only, but not enough space. The TV had 3.4 GB of storage and only about 450 MB free. Android needs more headroom than the size of the app. Clearing the app caches (pm trim-caches 2G) freed enough to get it through. This turned out to be the theme of the whole evening. Speed tweaks and a reboot test After the home screen, I applied a few more settings: - Window and transition animations set to 0 - The screensaver turned off - A cap on background processes: deviceconfig put activitymanager maxcachedprocesses 4 (this survived a reboot) Then I waited 30 seconds for Android to flush its settings to disk, rebooted, and confirmed the TV booted normally and Home still opened the new launcher. Don't skip the reboot test. It's the safest way to catch a bad change while you can still undo it. Finding what else was eating resources I used dumpsys meminfo to see actual memory use instead of guessing. Most of the RAM was taken by things you shouldn't touch: the Android system, Google Play services and the Play Store. The one optional hog was Google's voice-search service (about 60 MB), which I disabled because I don't use voice search. I also removed apps I'd sideloaded and no longer used, disabled TalkBack and Google Assistant, and deleted a leftover APK in the Downloads folder. Free memory went from about 306 MB at the start to about 430 MB after the first reboot. Part of that gain is just from restarting a TV that had been on for a long time, so don't treat the number as pure savings. Hiccup 2: the Play Store silently failing Later I tried to install Surfshark VPN from the Play Store. It "installed" and then did nothing. Crunchyroll also refused to update. Nothing showed on screen, so I looked at the logs: sh adb logcat -d | grep -i "<app name" Both failures were INSTALLFAILEDINSUFFICIENTSTORAGE (error 6005) again. With around 650 MB free, the Play Store would download the app but Android refused to install it. I wanted to keep the two biggest apps I use (Stremio and Nuvio, about 270 MB and 190 MB), so I looked at where the space was going with dumpsys diskstats. The surprise was that the apps I'd disabled were still holding on to their data, cache and downloaded updates. The stock Google TV launcher alone held about 140 MB while turned off. I cleared them with pm clear and reverted their updates with pm uninstall-system-updates. That freed about 190 MB. Hiccup 3: a cleanup command that undid my work This one is worth highlighting. pm uninstall-system-updates re-enabled most of the apps I'd disabled, including Google's stock launcher, and Home went back to opening it. I had to disable them all again and re-set my launcher as the default. If you try this: run the cleanup, then check pm list packages -d and which app currently handles Home, and fix anything that changed. A lighter VPN: WireGuard instead of the Surfshark app The Surfshark app was too big for the storage I had. But Surfshark supports plain WireGuard, and the WireGuard app is only a few MB. 1. On the laptop: Surfshark account → VPN → Manual setup → Android → WireGuard. Generate a key pair and download a .conf file per country. 2. Install WireGuard from the Play Store on the TV. 3. Push the config files over adb: adb push us-sfo.conf /sdcard/Download/ 4. In the WireGuard app: + → Import from file or archive, then pick the file. A zip of several configs imports them all in one go. You lose Surfshark's server list, kill switch and extras, but you get a working VPN in a few megabytes. Treat these config files as passwords. They contain your private key, so delete them from the TV's Download folder after importing. Keeping the TV awake I also wanted the TV never to sleep or start the screensaver: sh adb shell settings put secure sleeptimeout 0 adb shell settings put system screenofftimeout 2147483647 adb shell settings put secure screensaverenabled 0 Manufacturers often add their own auto power-off option inside their own settings menu, so check there too. Leaving a TV on all the time is fine on an LCD, but a static image can burn into an OLED. Undoing everything I saved the undo script and a plain-text list of every change (disabled apps, changed settings, their old values) in my Documents folder. Undoing the home screen and re-enabling every app on the log is one command: sh sh undo.sh <TVIP If the TV ever gets stuck restarting, the project's README documents how to reconnect during the first seconds of boot and re-enable everything: sh for p in $(adb shell pm list packages -d | tr -d '\r' | sed 's/package://'); do adb shell pm enable "$p"; done What I'd tell you before you start 1. Look at free storage first. Under about 15% free, installs and updates quietly fail. Free space before you start, not after. 2. Disable, don't uninstall, wherever you can. It's the difference between a mistake and a disaster. 3. Keep a written list of every change and its old value. I kept one in a text file the whole time. 4. Reboot and test after the big changes. 5. Check the logs when something silently doesn't work. adb logcat told me exactly what was wrong in seconds. 6. Don't touch Google Play services' data. It's the biggest app on the TV (700 MB), but clearing it can sign you out and break things. Result The TV is noticeably snappier. The home screen is clean, with no ads, sponsored rows or manufacturer promos. Animations are instant, around 125 MB more RAM is free after a fresh boot, and the disk went from 88% full to 76% full. Most of that came from about an hour of careful, reversible changes. Thanks to Ansub Khan for the tv-detox project that made this safe and simple to try.
Evaluating LLM Apps: Treat Prompts Like Pipelines
A data pipeline without tests breaks quietly. An LLM feature without evaluations breaks quietly and changes behaviour whenever someone edits a prompt or the provider ships a new model version. The fix is the same discipline data engineers already use: a fixed dataset, automated checks and a number you watch. 1. Build a golden dataset Collect 50–200 real inputs: actual user questions, real documents, messy edge cases. For each, write down what a good output looks like. That could be an exact label, a required fact, or a short rubric. Version it in git next to the prompts. Start small. Thirty well-chosen examples beat none. 2. Use the cheapest check that works In order of preference: 1. Deterministic checks. Valid JSON? Matches the schema? Contains the required field? Correct label? These are fast, free and unambiguous. 2. Reference comparisons. Does the extracted amount equal the labelled amount? Did retrieval return the labelled document? 3. LLM-as-judge. A second model grades the output against a rubric ("Is every claim supported by the sources? yes/no, and why"). Useful for open-ended text, but spot-check the judge against human ratings before you trust it. 3. Score the parts separately For a RAG app, a single "quality" number hides the cause. Track retrieval (did the right chunk come back?), faithfulness (does the answer stick to the sources?) and answer quality on their own. For an agent, track task success, steps and cost per task. 4. Run evals like CI text prompt or model change │ ▼ run the golden set ──► scores vs. last release │ ▼ regression? ── yes ──► block the change │ no ▼ ship it Every prompt edit, model upgrade or retrieval change runs the suite. A drop in any metric beyond a small tolerance blocks the change, just as a failing dbt test blocks a deploy. 5. Keep watching in production - Log inputs, outputs, latency, token cost and the model version for every call. - Sample production traffic for review each week and add interesting failures to the golden set. - Collect cheap user signals such as thumbs up or down, and whether the user retried or edited the answer. 6. Don't forget cost and latency Quality is one axis. A change that improves accuracy by two points but doubles latency or cost per request is a trade-off, not a win. Put all three on the same dashboard. The teams that ship reliable GenAI features aren't the ones with the cleverest prompts. They're the ones that can tell, within minutes, whether a change made things better or worse. Try it: a five-question eval that fails Here's a tiny golden set with cheap graders (substring, exact match, regex) run against a 1.7B local model: bash curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/evaldemo.py python evaldemo.py This is the eval working, not breaking. It caught a model inventing a port number, contradicting a basic SQL fact, and running past the length budget. Swap in a bigger model or a better prompt, rerun, and you have a number to compare instead of a feeling.
Running Local LLMs with Ollama: A Hands-On Guide
Ollama runs open-weight models on your own machine behind a small HTTP server. No API key, no per-token bill, and your prompts never leave the laptop. It's the model backend for my OpenClaw Telegram assistant, and the easiest way I know to learn how LLMs behave. Every screenshot here is real terminal output, captured while writing this guide. 1. Install it On Omarchy, use Install AI Ollama in the menu (Super + Alt + Space), or install from the repos. Pick the package that matches your GPU: bash yay -S ollama CPU only yay -S ollama-cuda NVIDIA yay -S ollama-rocm AMD sudo systemctl enable --now ollama On any other Linux, the official script does the same: curl -fsSL https://ollama.com/install.sh | sh. 2. Pull and run a model bash ollama pull qwen3:1.7b ollama run qwen3:1.7b --think=false "In one sentence: what does Ollama do?" Look at that answer. Ollama is not a model and has nothing to do with OpenAI, but a small model said so with total confidence. Small local models are great for classification, extraction, formatting and grounded answers. Don't trust them on facts from memory. Give them the facts (that's what RAG is for). ollama ps is the other half of that screenshot. It shows what's loaded, how much memory it takes, whether it's on GPU or CPU, and the context size it was loaded with. 3. Pick a model that fits your hardware Rules of thumb that have held up for me: - Memory: a 4-bit model needs roughly 0.6 to 0.7 GB per billion parameters, plus room for the context. A 7–8B model wants about 6 GB of VRAM; a 14B model about 10 GB. - Context costs memory too. The KV cache grows with numctx. On a test box with 8 GB of RAM, a 32k context on even a tiny model got the runner killed and every request returned HTTP 500. Dropping to 8k fixed it. - Start small, then go up. Try a 1.7B–4B model first to check your pipeline works, then move to the largest model that stays fast enough. bash ollama show qwen3:1.7b architecture, context length, capabilities ollama list what's on disk ollama rm <model free the space 4. Call it from code The CLI is a client for a REST API on localhost:11434. There's a native API and an OpenAI-compatible one, so most SDKs work by changing the base URL. In Python with the OpenAI SDK: python from openai import OpenAI client = OpenAI(baseurl="http://localhost:11434/v1", apikey="ollama") key is ignored r = client.chat.completions.create(model="qwen3:1.7b", messages=[{"role": "user", "content": "Summarise this log line: ..."}]) print(r.choices[0].message.content) 5. Make your own model with a Modelfile A Modelfile bakes a system prompt and parameters into a named model, so every client gets the same behaviour: Notice it answered with netstat, which is fine, but on Omarchy I'd use ss -ltnp | grep 11434. Small models give the most common answer, not always the best one. 6. Embeddings for search and RAG Ollama also serves embedding models. nomic-embed-text is small and good enough to build real retrieval on: The RAG guide uses exactly this to build a 30-line retrieval demo. 7. Settings worth changing Set these on the service so every client gets them: bash sudo systemctl edit ollama ini [Service] Environment="OLLAMACONTEXTLENGTH=65536" agents like OpenClaw need a big window Environment="OLLAMAKEEPALIVE=30m" keep the model warm between messages Environment="OLLAMAHOST=127.0.0.1:11434" don't expose it to the network bash sudo systemctl restart ollama Keep the server on 127.0.0.1. The API has no authentication, so anyone who can reach the port can use your GPU. Cheat sheet | Command | What it does | | --- | --- | | ollama pull <model | Download a model | | ollama run <model | Chat, or answer one prompt | | ollama ps | What's loaded, on GPU or CPU, and its context | | ollama show <model | Capabilities and parameters | | ollama create <name -f Modelfile | Build a custom model | | ollama launch openclaw | Set up OpenClaw on top of Ollama | Next: hooking Ollama up to Telegram with OpenClaw.
Setting Up Omarchy: My Arch + Hyprland Dev Laptop
Omarchy is DHH's opinionated Arch Linux setup: Hyprland as a tiling window manager, a themed terminal and Neovim, and every keyboard shortcut already decided for you. It's what my laptop runs, and it's where my local AI stack (Ollama and OpenClaw over Telegram) lives. This is the setup as I'd do it again, in order. Before you start - Use a dedicated drive. The installer wipes the disk you pick and sets up full-disk encryption. Dual-booting on one drive needs the manual install route. - Turn off Secure Boot (and TPM if your BIOS ties them together). The installer won't boot otherwise. - Have a wired or 2.4 GHz keyboard. You type the disk password before Bluetooth exists, so a Bluetooth-only keyboard can't unlock the machine. - Back up anything on that drive. It will be gone. 1. Flash the ISO and install Download the ISO from omarchy.org, write it to a USB stick (balenaEtcher on Windows/Mac, caligula on Linux), and boot from it. The installer asks a handful of questions: keyboard layout, username, password, hostname, timezone. Confirm them, pick the drive, and wait 2 to 10 minutes. 2. Learn these shortcuts on day one Everything in Omarchy is a keystroke. You don't need all of them; you need these, and Super + K shows the rest whenever you forget. The two that change how you work: Super + Alt + Space opens the Omarchy menu (install apps, change themes, update), and Super + 1–4 jumps between workspaces. I keep the editor on 1, browser on 2, terminals on 3 and chat on 4. 3. Update the Omarchy way Don't reach for pacman -Syu out of habit. Use Update Omarchy in the menu (or omarchy update in a terminal). It pulls new configs and runs migrations together with the package upgrade, so a new library version doesn't arrive without the config change it needs. Every update takes a Btrfs snapshot first. If something breaks, reboot, pick the pre-update snapshot in the Limine boot menu, and restore. One catch: snapshots restore the system, not /home. They won't bring back a deleted file, so keep real backups too. You can take one by hand before a risky change: bash omarchy-snapshot create 4. Set up the dev tools Omarchy already ships Neovim (LazyVim), Docker, mise for language runtimes, and AI coding CLIs (c opens OpenCode, cx opens Claude Code). What I add on top: bash Language runtimes through mise mise use -g node@24 python@3.12 Git identity git config --global user.name "Yashank Kothari" git config --global user.email "you@example.com" Anything else from the Arch repos or the AUR yay -S --needed ollama The omarchy command is worth a look too. Run it with no arguments to see every helper the menu uses (themes, fonts, screenshots, debug info). It's also what makes Omarchy easy for an AI agent to configure. 5. Make it yours, safely - Configs live in ~/.config (Hyprland, Waybar, terminal, Neovim). Edit there, not in the Omarchy source. - Try themes with Super + Ctrl + Shift + Space before you start editing colours by hand. - If you wreck your configs, omarchy reinstall resets them to the current release. It overwrites your changes, so commit ~/.config to a dotfiles repo first. What I'd tell myself before installing 1. Give it a week. Tiling feels slow for two days and then you can't go back. 2. Learn Super + K. It answers most questions. 3. Update through the menu, and trust the snapshots. 4. Put local AI on it. Next up: running models with Ollama.
My Own AI Assistant on Telegram: OpenClaw + Ollama on Omarchy
I message a Telegram bot from my phone, and an agent running on my Omarchy laptop answers, using a model served locally by Ollama. No cloud model, no open ports, and the agent can use tools on the machine I gave it. The piece in the middle is OpenClaw, an open-source assistant that connects chat apps to AI agents through a local gateway. Here's the exact setup, with real terminal output from each step. What you need - Ollama running with a model that supports tool calling (ollama show <model lists tools under capabilities). - Node 24.16+ (OpenClaw won't install on Node 22). On Omarchy: mise use -g node@24. - A Telegram account. 1. Install OpenClaw bash npm install -g openclaw@latest --allow-scripts=openclaw openclaw --version There's also a one-line installer (curl -fsSL https://openclaw.ai/install.sh | bash) and, if Ollama is already set up, ollama launch openclaw does the install, model pick and onboarding in one go. 2. Onboard it with Ollama as the model bash openclaw onboard --install-daemon The wizard asks for a model provider (choose Ollama), detects your local models, writes ~/.openclaw/openclaw.json, and installs the gateway as a systemd user service so it starts on login. The scriptable version of the same thing: The relevant part of the config it writes: json { "agents": { "defaults": { "model": { "primary": "ollama/qwen3:0.6b" } } }, "models": { "providers": { "ollama": { "baseUrl": "http://127.0.0.1:11434", "api": "ollama" } } }, "gateway": { "mode": "local", "port": 18789, "bind": "loopback" } } Keep bind on loopback. The gateway can run tools on your machine, so only local clients should reach it. 3. Check the gateway is up bash openclaw gateway status openclaw status 4. Create the Telegram bot 1. In Telegram, open a chat with @BotFather (check the handle exactly). 2. Send /newbot, pick a display name and a username ending in bot. 3. Copy the token it gives you. Treat it like a password. 5. Connect the bot to OpenClaw bash openclaw channels add --channel telegram --token <your-bot-token openclaw channels status --probe Or put it in ~/.openclaw/openclaw.json yourself: json { "channels": { "telegram": { "enabled": true, "botToken": "123456:ABC...", "dmPolicy": "pairing", "groups": { "": { "requireMention": true } } } } } The running gateway hot-reloads the change. dmPolicy: "pairing" is the important line: strangers who find your bot can't talk to your agent until you approve them. 6. Pair your own account Send your bot any message from your phone. That creates a pairing request. Approve it on the laptop: bash openclaw pairing list telegram openclaw pairing approve telegram <CODE Codes expire after an hour. Once approved, every message you send the bot goes to the agent, and its reply comes back in the chat. 7. Make the local model good enough This is where most local setups fall over: - Context window. OpenClaw's system prompt, tools and memory need room. Ollama's docs recommend at least 64k tokens for local models. Set OLLAMACONTEXTLENGTH=65536 on the Ollama service (how). - But context costs RAM. On an 8 GB test box, a 32k context got the model killed mid-load, and the agent just reported provider internal error, HTTP 500. If you see that, check journalctl -u ollama for a killed runner and lower the context or use a smaller model. - Model size. Tiny models can call tools but make poor decisions. Use the biggest tool-capable model your GPU runs comfortably, and keep a small one for quick tests. - Keep it warm. OLLAMAKEEPALIVE=30m stops the first message after a break from waiting for a model load. 8. Day-to-day commands | Command | Use | | --- | --- | | openclaw status | Health, sessions, channels at a glance | | openclaw logs --follow | Watch messages and tool calls live | | openclaw models list | Which model the agent uses | | openclaw doctor | Find and fix config problems | | openclaw gateway stop | Stop the gateway | | openclaw security audit | Check what the agent is allowed to do | Guardrails I'd keep - Pairing on, and only your Telegram user ID allowed. - Gateway on loopback, no port forwarding. - Run openclaw security audit after changing tools or skills, and keep risky tools behind approvals. - Keep the bot token and gateway token out of screenshots and git. None of the ones here show either. The agent loop running behind all this is the one from my AI agents guide, just with a phone on the other end.
Snowflake Cost and Performance: The Basics That Matter Most
In Snowflake you pay for storage and for compute, and compute is almost always the bigger bill. Compute is billed per second while a virtual warehouse is running, with a 60-second minimum each time it starts. Most savings come from keeping warehouses the right size and off when idle. Warehouses Auto-suspend and auto-resume. Set a short auto-suspend for most workloads: sql ALTER WAREHOUSE transformwh SET AUTOSUSPEND = 60 -- seconds AUTORESUME = TRUE; A warehouse left at a 10-minute suspend after every two-minute job spends most of its credits doing nothing. Size up for speed, not as a habit. Each size step roughly doubles both the compute and the credits per hour. A query that scales well can finish in half the time on the next size up for about the same cost. One that doesn't scale just costs double, so check before you upsize. Separate warehouses by workload. Keep ELT, BI dashboards and ad-hoc analysis on different warehouses, so one heavy job doesn't slow the CEO's dashboard and each cost is visible on its own. Queries 1. Select only the columns you need. Snowflake stores data by column, so SELECT on a wide table reads far more than it has to. 2. Filter on columns that prune well. Snowflake skips micro-partitions whose min/max range can't match your filter. Filtering on a date column that follows load order usually prunes well; wrapping it in a function (WHERE TOCHAR(createdat, 'YYYY') = '2026') can defeat pruning. 3. Read the Query Profile. Compare "Partitions scanned" with "Partitions total", and look for "Bytes spilled to local/remote storage". Spilling means the warehouse ran out of memory for that query. 4. Watch for exploding joins. A join on a non-unique key multiplies rows. If the output is much larger than both inputs, the join keys are probably wrong. Caching - Result cache. An identical query on unchanged data returns the stored result without using a warehouse, for up to 24 hours. - Warehouse cache. A running warehouse keeps recently read data on local disk. Suspending clears it, which is a fair trade for most batch jobs. Clustering keys Only for very large tables (multi-terabyte) where queries filter on the same columns and pruning is poor. Automatic clustering costs credits to maintain, so measure before and after. Keep an eye on spend sql SELECT warehousename, SUM(creditsused) AS credits FROM snowflake.accountusage.warehousemeteringhistory WHERE starttime = DATEADD('day', -30, CURRENTTIMESTAMP()) GROUP BY 1 ORDER BY 2 DESC; Pair it with a resource monitor on each warehouse, so a runaway query hits a credit limit instead of the monthly bill. Try it: three queries to run this week sql -- 1. Which warehouses burned the most credits in the last 7 days? select warehousename, sum(creditsused) as credits from snowflake.accountusage.warehousemeteringhistory where starttime = dateadd(day, -7, currenttimestamp()) group by 1 order by 2 desc; -- 2. The 10 slowest queries, and how much of each table they scanned select queryid, warehousename, totalelapsedtime / 1000 as seconds, partitionsscanned, partitionstotal from snowflake.accountusage.queryhistory where starttime = dateadd(day, -7, currenttimestamp()) order by totalelapsedtime desc limit 10; -- 3. A guardrail: suspend at the monthly budget create resource monitor monthlybudget with creditquota = 400 triggers on 90 percent do notify on 100 percent do suspend; alter warehouse etlwh set resourcemonitor = monthlybudget; When partitionsscanned is close to partitionstotal on a big table, the filter isn't pruning. That's your cue to check the where clause or think about a clustering key.
AI Agents Without the Hype: Tools, Loops and Guardrails
An agent is an LLM running in a loop that can call tools. Instead of answering in one shot, it decides on an action, sees the result, and decides again until the task is done or it gives up. That's all it is. The engineering is in the tools, the loop's limits and what happens when it goes wrong. The loop python messages = [systemprompt, usertask] for step in range(MAXSTEPS): always cap it reply = llm(messages, tools=TOOLS) if reply.toolcall is None: return reply.text done result = runtool(reply.toolcall) validated, sandboxed messages += [reply, toolresult(result)] raise RuntimeError("step limit reached") Model providers support tool calling (also called function calling): you describe each tool with a name, a description and a JSON schema, and the model returns a structured call instead of prose. Design tools like an API for a junior engineer - Few, clear tools beat many vague ones. getpolicy(policyid) is better than runsql(query). - Write the description for the model. Say when to use the tool, what it returns and what it can't do. - Return compact, useful results. Twenty relevant rows, not a 10,000-row dump that fills the context window. - Return errors as text the model can act on: "policyid must look like POL-123456", not a stack trace. Start with a workflow, not an agent If the steps are known in advance (extract, then validate, then load), write them as a fixed pipeline with an LLM call in one or two steps. It's cheaper, faster and easier to test. Reach for a free-running agent only when the path genuinely depends on what it finds along the way. Guardrails that matter 1. Step and cost limits on every run. 2. Least privilege. Read-only tools by default; separate, narrow tools for writes. 3. A human approves anything irreversible: sending an email, paying, deleting, deploying. 4. Treat tool output as untrusted data. A web page or document can contain text that tries to give the agent instructions (prompt injection). Never let retrieved content grant new permissions. 5. Validate tool arguments against the schema before running anything. Make it observable Log every step: the prompt, the tool call, its arguments, the result, tokens used and time taken. When an agent fails, the trace is the only way to tell whether the model chose the wrong tool, the tool returned something confusing or the task was underspecified. Evaluate on tasks, not vibes Keep a set of realistic tasks with a known correct outcome and run the agent against them after every change to the prompt, tools or model. Track success rate, steps and cost per task. An agent that's 5% more accurate but three times as expensive may not be an improvement. Try it: a tool-calling agent in 25 lines This runs the loop above against a local model through Ollama's tool calling. One tool, an allowlist of paths, and a four-step budget: bash curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/agentdemo.py python agentdemo.py Try asking it about /etc and watch the allowlist return an error the model has to deal with. That's the guardrail doing its job. For a full agent with memory, skills and a chat front end, see how I run OpenClaw over Telegram.
Building a RAG Pipeline That Actually Retrieves
Retrieval-augmented generation (RAG) means: find the passages that answer a question, put them in the prompt, and ask the model to answer from those passages. It's the standard way to give an LLM knowledge it wasn't trained on, without fine-tuning. When a RAG system gives a bad answer, the generation step usually isn't the culprit. Retrieval handed the model the wrong text. So treat RAG as a data pipeline first. The two halves Indexing (offline, like any ETL job) 1. Load documents and clean them: strip boilerplate, keep titles and section headings. 2. Split them into chunks. 3. Embed each chunk and store the vector with its text and metadata (source, section, date, permissions). Query time 1. Embed the user's question. 2. Retrieve the top candidate chunks. 3. Optionally rerank them. 4. Build a prompt with the best few chunks and generate an answer with citations. Chunking decides most of your quality - Split on structure (headings, paragraphs) before falling back to a fixed size. - A few hundred tokens per chunk with a small overlap is a sensible starting point. Tune it against real questions. - Prepend the document title and section heading to each chunk before embedding, so "Section 4.2: Exclusions" still knows which policy it belongs to. Retrieve with more than one signal Pure vector search misses exact terms such as policy numbers, error codes and product names. Hybrid search combines keyword search (BM25) with vector search and merges the two result lists. Then a reranker (a cross-encoder that scores each question-chunk pair) reorders the top 20–50 candidates so the best five go in the prompt. Filter on metadata before ranking: date ranges, document type and, above all, the user's access permissions. Prompt for grounded answers text Answer the question using only the sources below. Cite sources as [1], [2]. If the sources don't contain the answer, say you don't know. [1] {chunk text, with title} [2] {chunk text, with title} Question: {user question} "Say you don't know" plus visible citations does more against hallucination than any amount of clever wording. Measure retrieval and answers separately Build a small evaluation set of 30–50 real questions, each labelled with the chunk or document that answers it. Then track: - Retrieval: is the right chunk in the top k (recall@k)? - Faithfulness: does the answer only claim what the sources say? - Answer quality: is it correct and does it actually answer the question? If recall@k is low, fix chunking, hybrid search or reranking. Tweaking the prompt won't help. Keep the index fresh Treat the index like a warehouse table: load incrementally, re-embed changed documents, delete removed ones, and re-embed everything when you change the embedding model, because vectors from different models aren't comparable. Try it: a 30-line RAG on your laptop No vector database needed to learn the moving parts. This uses Ollama's nomic-embed-text for embeddings, plain cosine similarity, and qwen3:1.7b to answer: bash ollama pull nomic-embed-text && ollama pull qwen3:1.7b curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/ragdemo.py python ragdemo.py That last part is the lesson. The answer is correct and cited, but it also rambles about warehouse auto-suspend because a weak second chunk made the cut. Two fixes: drop chunks below a score threshold, and tell the model to ignore sources that don't answer the question. Then measure retrieval on its own, as described above.
Designing Airflow DAGs That Don't Wake You Up
Airflow is a scheduler and orchestrator, not a processing engine. The DAGs that cause the fewest incidents keep the heavy lifting in the warehouse and use Airflow to decide what runs, when, and in what order. 1. Keep top-level code cheap The scheduler re-parses every DAG file every few seconds. Anything at module level runs on every parse: python Bad: hits the database on every parse tables = gettablesfromdb() Good: do it inside a task @task def listtables(): return gettablesfromdb() Slow parsing is the most common reason a scheduler "feels stuck". 2. One task, one job, safe to retry Each task should do one thing and be idempotent (same window in, same result out), so retries are harmless: python from datetime import datetime, timedelta from airflow.decorators import dag, task @dag( schedule="@daily", startdate=datetime(2026, 1, 1), catchup=False, maxactiveruns=1, defaultargs={"retries": 2, "retrydelay": timedelta(minutes=5)}, tags=["claims"], ) def claimsdaily(): @task def extract(dataintervalstart=None, dataintervalend=None): pull exactly this run's window ... @task def load(path: str): ... load(extract()) claimsdaily() 3. Use the run's window, never now() Every DAG run has a dataintervalstart and dataintervalend. Use them, and reruns and backfills just work. datetime.now() inside a task means a rerun on Tuesday processes Tuesday, not the Monday it was meant for. 4. Decide on catchup on purpose catchup=True with an old startdate schedules every missed interval the moment you unpause. Leave it False unless you actually want a backfill, and run backfills deliberately. 5. Pass references, not data XCom is for small values: a file path, a row count, a table name. Don't push DataFrames through it. Write to object storage or a staging table and pass the location. 6. Make waiting cheap Sensors that poke in mode="poke" hold a worker slot the whole time. Use mode="reschedule", or a deferrable sensor, so waiting costs nothing. 7. Let the warehouse do the work Pulling a million rows into a Python task to transform them is slower and more fragile than one SQL statement. Trigger the transform (a stored procedure, a dbt build, a MERGE) and let Snowflake run it. A pre-merge checklist - No database or API calls at module level. - catchup, retries, and maxactiveruns set explicitly. - Every task uses the data interval, not wall-clock time. - Only small values in XCom. - Alerts (onfailurecallback or email) wired to someone who will read them. Try it: lint your DAGs before they hit the scheduler Two commands catch most of the problems above before they reach production: bash Fails fast on import errors and slow top-level code time python dags/loadorders.py Runs one logical date end to end, no scheduler needed airflow dags test loadorders 2026-09-21 If the first command takes more than a second or two, something at the top of the file is calling a database or an API. Move it inside a task.
How LLMs Work: A Mental Model for Engineers
You don't need to train a large language model to build good things with one. You do need an accurate picture of what it does, because most bad LLM features come from expecting it to behave like a database or a search engine. It predicts the next token An LLM reads a sequence of tokens (chunks of text, often parts of words) and predicts a probability for every possible next token. It picks one, appends it, and repeats. Everything else (answering, summarising, writing SQL) emerges from doing that very well. Two consequences: - It generates plausible text, not retrieved facts. If the right answer wasn't in its training data or in the prompt, it will still produce something that sounds right. That's a hallucination, and it's the default behaviour, not a bug. - Cost and speed scale with tokens. You pay for input and output tokens, and long outputs are slow because they're generated one token at a time. The context window is its only working memory The model knows nothing about your conversation, your company or your data except what's in the context window for this request: the system prompt, the chat history, any documents you pasted. Close the request and it's gone. That's why "chat with your docs" products don't retrain anything. They find the relevant text and put it in the context. That pattern has a name: retrieval-augmented generation, or RAG. Sampling controls variety - Temperature near 0 makes the output close to deterministic. Use it for extraction, classification and SQL. - Higher temperature gives more varied wording. Use it for brainstorming and creative text. Even at temperature 0, don't assume byte-identical output across runs or model versions. Validate the result instead. Embeddings: meaning as numbers An embedding model turns text into a vector, a list of numbers, where texts with similar meaning land close together. Search "getting fit" and you can find a post about a workout routine that never uses those words. Embeddings power semantic search, deduplication, clustering and the retrieval step of RAG. Ask for structure When the output feeds code, ask for JSON that matches a schema, and validate it: python from pydantic import BaseModel class Claim(BaseModel): claimid: str amountinr: float category: str raw = llmcall(prompt) your provider's SDK claim = Claim.modelvalidatejson(raw) fails loudly on bad output Most providers now offer a structured-output or JSON mode. Use it, and still validate. Where LLMs fit in a data stack They're good at messy-text-to-structure jobs: classifying free-text claim notes, extracting fields from PDFs, writing a first draft of a SQL query, summarising incident logs. They're bad at arithmetic over large tables and at being the source of truth. Let the warehouse compute; let the model read and write language. Try it with a local model Everything above is easy to see on a laptop with Ollama. This script asks qwen3:1.7b for a product name four times at two temperatures, then asks for a log classification that has to match a JSON schema: bash ollama pull qwen3:1.7b curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/samplingdemo.py curl -O https://www.yashank.site/files/demos/structureddemo.py python samplingdemo.py && python structureddemo.py The token counts are worth a look too: a 20-word prompt came to 32 tokens once the chat template was added. That's the number you pay for and the number that fills the context window.
dbt Incremental Models on Snowflake, Without the Surprises
A table model rebuilds everything on every run. That's simple and correct, and it's the right default until the table gets big. An incremental model processes only new or changed rows and merges them into what's already there. It's faster and cheaper, and it's also where most dbt bugs live. The minimal correct version {% raw %} sql {{ config( materialized = 'incremental', uniquekey = 'claimid', incrementalstrategy = 'merge', onschemachange = 'appendnewcolumns' ) }} select claimid, policyid, claimamount, status, updatedat from {{ ref('stgclaims') }} {% if isincremental() %} where updatedat ( select dateadd('day', -3, max(updatedat)) from {{ this }} ) {% endif %} {% endraw %} What each piece does: - uniquekey tells dbt how to match incoming rows to existing ones. Without it, the merge becomes a plain append and reruns create duplicates. - incrementalstrategy = 'merge' is the default on Snowflake. It's written out here so nobody has to wonder. - isincremental() is false on the first run and on --full-refresh, so the filter only applies once the table exists. - The 3-day lookback catches late-arriving rows. Filtering on exactly max(updatedat) silently drops any record that lands after a later one has already been loaded. Things that bite 1. Duplicate keys in the source. If stgclaims has two rows for one claimid, the merge errors out. Deduplicate in staging with qualify rownumber() over (partition by claimid order by updatedat desc) = 1. 2. Logic changes don't backfill themselves. Change a calculation and only new rows get it. Run dbt run --select mymodel --full-refresh after changing business logic. 3. Schema changes. The default onschemachange is ignore, so new columns just don't appear. appendnewcolumns or syncallcolumns is usually what you want. 4. Hard deletes are invisible. Rows deleted upstream stay in your table forever. Use a soft-delete flag from the source, or a snapshot. Test it like you mean it yaml models: - name: fctclaims columns: - name: claimid tests: - unique - notnull A unique test on the uniquekey is the cheapest insurance you can buy for an incremental model. It fails the build the first time a duplicate slips through. When to go incremental Stay on table until a full rebuild is noticeably slow or expensive. Incremental models trade simplicity for speed, so make the trade only when you need to. Try it locally with DuckDB You don't need a Snowflake account to see the behaviour. dbt-duckdb runs the same Jinja and the same merge strategy on your laptop. bash pip install dbt-duckdb mkdir -p dbtdemo/models && cd dbtdemo curl -o models/fctevents.sql https://www.yashank.site/files/demos/fctevents.sql Add a two-line dbtproject.yml (name and profile) and a profiles.yml with type: duckdb and path: demo.duckdb. Create a rawevents table, run dbt run, then update 10 rows and insert 200 new ones upstream and run again: Open target/compiled/.../fctevents.sql after the second run and you'll see the where updatedat (select max(updatedat) - interval 3 day ...) filter that isincremental() switched on. When you move to Snowflake, only the profile changes.
Embeddings Explained: Meaning as Vectors
Most people meet embeddings as the retrieval step in a RAG pipeline and never look further. That's a shame, because they're one of the most useful tools in applied machine learning, and they're far simpler than the models that generate text. What an embedding is An embedding is a list of numbers (a vector) that represents a piece of input, such as a sentence, a product description or an image. An embedding model is trained so that inputs with similar meaning end up with vectors that point in similar directions. "How do I reset my password?" and "I forgot my login" share almost no words, but a good text embedding model places them close together. "How do I reset my router?" lands somewhere nearby but noticeably further away. That's the whole trick. Once meaning is geometry, you can compare, group and search things using basic linear algebra. Cosine similarity The standard way to compare two embeddings is cosine similarity: the cosine of the angle between the vectors. text cos(a, b) = (a · b) / (|a| |b|) It ranges from -1 to 1. Close to 1 means the vectors point the same way, close to 0 means they're unrelated. In practice, with most text embedding models, scores cluster in a narrower band, so "0.8" doesn't have a universal meaning. You learn what a good score looks like for your model and your data. Cosine similarity ignores vector length and only looks at direction. That's usually what you want, since length often reflects things like input size rather than meaning. Normalisation If you normalise every vector to unit length (divide by its norm), cosine similarity becomes a plain dot product. That's faster to compute and it's what many vector databases assume. Some embedding APIs return normalised vectors already. Check the documentation rather than assuming. Normalising an already normalised vector is harmless, so when in doubt, do it once at write time and store the result. A small runnable example Real embeddings have hundreds or thousands of dimensions. These toy vectors have four, hand-made so the dimensions loosely mean "account", "networking", "billing" and "food". The maths is identical. python import numpy as np docs = { "reset my password": [0.9, 0.1, 0.0, 0.0], "forgot my login": [0.8, 0.2, 0.1, 0.0], "router keeps disconnecting": [0.1, 0.9, 0.0, 0.0], "wifi is very slow": [0.0, 0.8, 0.1, 0.1], "charged twice this month": [0.1, 0.0, 0.9, 0.0], "refund for my invoice": [0.2, 0.0, 0.8, 0.1], "best biryani nearby": [0.0, 0.0, 0.1, 0.9], } names = list(docs) X = np.array([docs[n] for n in names], dtype=float) X = X / np.linalg.norm(X, axis=1, keepdims=True) normalise once def search(queryvec, k=3): q = np.array(queryvec, dtype=float) q = q / np.linalg.norm(q) scores = X @ q cosine via dot product top = np.argsort(-scores)[:k] return [(names[i], round(float(scores[i]), 3)) for i in top] print("Search 'can't sign in':") for name, score in search([0.85, 0.15, 0.05, 0.0]): print(f" {score:.3f} {name}") Near-duplicate detection: pairs above a threshold sim = X @ X.T print("\nLikely duplicates ( 0.95):") for i in range(len(names)): for j in range(i + 1, len(names)): if sim[i, j] 0.95: print(f" {sim[i, j]:.3f} {names[i]!r} ~ {names[j]!r}") Classification by nearest labelled centroid labels = {"account": [0, 1], "network": [2, 3], "billing": [4, 5]} centroids = {k: X[idx].mean(axis=0) for k, idx in labels.items()} centroids = {k: v / np.linalg.norm(v) for k, v in centroids.items()} ticket = np.array([0.15, 0.05, 0.85, 0.05]) ticket = ticket / np.linalg.norm(ticket) best = max(centroids, key=lambda k: centroids[k] @ ticket) print(f"\nNew ticket classified as: {best}") Swap the toy vectors for output from a real embedding model and the same code works for search, deduplication and classification. Uses beyond RAG - Semantic search. Find documents by meaning rather than keywords. Often best combined with keyword search (hybrid search), because embeddings can miss exact terms like product codes. - Deduplication. Near-identical support tickets, product listings or records from different sources score very high against each other. A threshold plus human review of the borderline band goes a long way. - Clustering. Run k-means or a density-based method on embeddings to find themes in thousands of free-text responses without labelling anything first. - Classification. Train a simple logistic regression on embeddings, or use nearest centroid as above. With a few hundred labelled examples this is often competitive with much heavier approaches, and far cheaper to run. - Anomaly detection. Items far from every cluster are worth a look. Choosing dimensions More dimensions can capture more nuance, but cost more to store and compare. A million 1,536-dimensional float32 vectors is roughly 6 GB before any index overhead. Some newer models are trained so you can truncate vectors to fewer dimensions with modest quality loss. If yours supports that, test a smaller size on your own retrieval eval before defaulting to the largest. Re-normalise after truncating. Pitfalls - Mixing models. Vectors from different embedding models (or different versions of the same one) are not comparable. Changing models means re-embedding everything. Store the model name alongside each vector. - Assuming a universal threshold. A similarity cut-off that works for one model and domain won't transfer. Calibrate on labelled pairs. - Long inputs. Models have input limits and may truncate silently. Embedding a whole document into one vector also blurs its topics together. Chunk sensibly. - Domain mismatch. A general model may not separate terms that matter in your field, like insurance policy types. Test on your own data. - Exact matches. Embeddings are weak at IDs, numbers and rare names. Keep keyword search around. Where to start - Pick one embedding model, record its name with every vector, and normalise at write time. - Build a small labelled set of similar and dissimilar pairs from your own data, and use it to choose thresholds and dimensions. - Try deduplication or clustering before RAG. They're simpler, easier to evaluate and often more immediately useful. - Use hybrid search when exact terms matter.
Idempotent Pipelines: Making the 3 a.m. Rerun Safe
A pipeline is idempotent when running it once or five times for the same input gives the same result. It is the single most useful property a batch job can have, because every job eventually gets rerun: a retry after a timeout, a backfill after a bug fix, someone clicking "Clear" in Airflow at 3 a.m. If a rerun doubles your revenue numbers, the pipeline is broken even when every task is green. The three patterns that get you there 1. Process a window, not "whatever is new" Bad: WHERE loadedat (SELECT MAX(loadedat) FROM target). The answer changes every time you run it. Good: every run owns a fixed, explicit window, such as one day, passed in from the scheduler. In Airflow that is the run's dataintervalstart and dataintervalend. Rerunning the 7 October run always processes 7 October. 2. Overwrite the window, don't append to it Two safe ways to write: Delete-then-insert inside one transaction sql BEGIN; DELETE FROM analytics.dailypolicies WHERE businessdate = '2026-10-07'; INSERT INTO analytics.dailypolicies SELECT FROM staging.dailypolicies WHERE businessdate = '2026-10-07'; COMMIT; MERGE on a real business key sql MERGE INTO analytics.policies t USING staging.policies s ON t.policyid = s.policyid WHEN MATCHED THEN UPDATE SET t.status = s.status, t.premium = s.premium, t.updatedat = s.updatedat WHEN NOT MATCHED THEN INSERT (policyid, status, premium, updatedat) VALUES (s.policyid, s.status, s.premium, s.updatedat); INSERT INTO ... SELECT with no delete or key is the classic source of duplicates. Avoid it for anything that can be rerun. 3. Deduplicate the source before you merge MERGE fails (or behaves unpredictably) when the source has two rows for the same key. Keep only the latest: sql SELECT FROM raw.policies QUALIFY ROWNUMBER() OVER ( PARTITION BY policyid ORDER BY updatedat DESC ) = 1; A quick checklist - Every run is parameterised by a date window from the scheduler, never by CURRENTDATE inside the SQL. - Writes are overwrite-by-partition or MERGE on a key, never blind appends. - The source is deduplicated on that key first. - Side effects (emails, API calls, file drops) live in their own task, so retrying the load doesn't resend them. - You've actually tested it: run the same day twice and compare row counts. Make this the default and backfills stop being scary. Try it: break it, then fix it Here's the whole idea in 25 lines of Python and SQLite. The script loads the same 1,000 orders twice, once with a blind append and once with delete-then-insert inside a single transaction. bash curl -O https://www.yashank.site/files/demos/idempotentdemo.py python idempotentdemo.py append python idempotentdemo.py idempotent Swap SQLite for Snowflake and the pattern is the same: DELETE ... WHERE orderdate = :ds and INSERT inside one transaction, or a MERGE on the business key.
Caching Without Regret: A Practical Guide to Redis
Caching is the fastest way to make a slow system feel quick, and one of the fastest ways to make a correct system wrong. The speed-up is immediate. The bugs show up weeks later, when someone asks why the dashboard still shows yesterday's number. This is the set of patterns I reach for with Redis, and the questions I try to answer before adding a cache at all. Start with the question Before writing any code, answer three things: - How stale can this data be? Seconds, minutes, a day? If the answer is "never", don't cache it, or cache it with a very deliberate invalidation path. - How expensive is a miss? A cache in front of a 5 ms query buys you little. A cache in front of a heavy aggregation or a slow third-party API buys a lot. - How is it read? Lots of reads of the same keys is ideal. Every request hitting a unique key means your hit rate will be poor. If the answers aren't good, a cache adds complexity without much benefit. Cache-aside: the default pattern Cache-aside (or lazy loading) is the pattern most applications use. The application checks the cache, and on a miss it loads from the source and writes the result back. python import json import redis r = redis.Redis(host="localhost", port=6379, decoderesponses=True) def getuser(userid: int) - dict: key = f"user:{userid}" cached = r.get(key) if cached is not None: return json.loads(cached) user = dbfetchuser(userid) your database call r.set(key, json.dumps(user), ex=300) 5-minute TTL return user It's simple, the cache only holds data that's actually requested, and if Redis goes down the application can still fall back to the database. The downside is that the first request after expiry always pays the full cost. TTLs are your safety net Always set a TTL (time to live). Even if you have an explicit invalidation path, a TTL bounds how long a bug can keep wrong data alive. A missed invalidation with a 10-minute TTL is an annoyance. A missed invalidation with no TTL is a support ticket next month. Invalidation On writes, you have two common options: - Delete the key after updating the source. The next read repopulates it. - Update the key with the new value. Deleting is usually safer. Updating risks a race: two writers update the database in one order and the cache in the other, and the cache ends up holding the older value until the TTL expires. python def updateuser(userid: int, fields: dict) - None: dbupdateuser(userid, fields) r.delete(f"user:{userid}") Update the database first, then delete. If you delete first, a concurrent read can repopulate the cache with the old value before your write lands. The hard part of invalidation is knowing which keys depend on a piece of data. A user's profile might be cached under user:42, inside a team listing, and inside a search result. Keep key naming consistent and the dependency map written down, or you'll miss one. Stampedes A cache stampede happens when a popular key expires and many requests miss at once, all hitting the database together. Three ways to handle it: Jittered TTLs If you cache lots of keys at the same time (on deploy, say), they'll also expire at the same time. Add randomness: python import random def ttlwithjitter(base: int, spread: float = 0.1) - int: return int(base random.uniform(1 - spread, 1 + spread)) r.set(key, value, ex=ttlwithjitter(300)) This spreads expiries out and avoids synchronised misses. A lock around recomputation Let one request rebuild the value while the others wait briefly or use a fallback. Redis SET with NX gives a simple lock: python import time def getreport(reportid: str) - dict: key = f"report:{reportid}" lockkey = f"lock:{key}" for in range(50): cached = r.get(key) if cached is not None: return json.loads(cached) if r.set(lockkey, "1", nx=True, ex=30): try: report = buildreport(reportid) expensive r.set(key, json.dumps(report), ex=ttlwithjitter(600)) return report finally: r.delete(lockkey) time.sleep(0.1) return buildreport(reportid) give up waiting The lock's own TTL matters: if the process holding it crashes, the lock frees itself. For anything critical, use a token so a process only deletes a lock it owns. Stale-while-revalidate Store the value with a soft expiry inside it, and a longer hard TTL on the key. When the soft expiry passes, serve the stale value and trigger a refresh in the background. Users never wait on the rebuild, and a slightly old value is often fine. python def getwithswr(key: str, loader, softttl: int = 300, hardttl: int = 3600): raw = r.get(key) if raw is not None: entry = json.loads(raw) if time.time() entry["softexpiry"]: schedulerefresh(key, loader, softttl, hardttl) e.g. a task queue return entry["value"] value = loader() entry = {"value": value, "softexpiry": time.time() + softttl} r.set(key, json.dumps(entry), ex=hardttl) return value Combine this with the lock so only one background refresh runs per key. What not to cache - Data that must be exact right now: balances, inventory at checkout, permissions. Stale permissions are a security bug. - Per-user data with low reuse, unless the source is slow. You'll fill memory with keys read once. - Huge values. Large blobs slow down Redis and the network. Cache an ID or a summary instead. - Errors, unless deliberately. Caching a failed lookup for a few seconds can protect a struggling dependency, but caching it for an hour hides recovery. Also set a memory limit and an eviction policy (maxmemory and maxmemory-policy, such as allkeys-lru), so Redis degrades predictably when it fills up. Before you ship the cache - Write down the acceptable staleness for each cached thing, and set TTLs from it. - Update the source first, then delete the key. - Add jitter to TTLs and a lock or stale-while-revalidate for expensive, popular keys. - Track hit rate and miss latency. If the hit rate is low, the cache is probably costing more than it saves.
Prompting, RAG or Fine-Tuning? Picking the Right Lever
When an LLM doesn't do what you want, there are three main levers: change the prompt, give it better context with retrieval, or change the model itself with fine-tuning. They get discussed as if they compete. They don't. They fix different problems, and picking the wrong one is an expensive way to not solve yours. The one distinction that matters Before anything else, ask whether the problem is about knowledge or behaviour. - Knowledge: the model doesn't know something. Your internal docs, last week's policy change, a customer's order history. - Behaviour: the model knows enough but responds the wrong way. Wrong format, wrong tone, ignores a rule, too verbose, inconsistent at a narrow task. Retrieval mostly fixes knowledge. Fine-tuning mostly fixes behaviour. Prompting can help with both, a bit, and it's nearly free, so it always comes first. Prompting Prompting changes nothing about the model. You're just giving it better instructions for this request. What it's good at: - clear task definitions and constraints - output formats, especially combined with structured-output or JSON modes - a few worked examples (few-shot prompting) to show the pattern you want - small amounts of reference information pasted directly in Cost: basically your time, plus the extra input tokens. Data needed: a handful of examples. You can iterate in minutes. Where it runs out: when the information you need is too large or changes too often to paste in, or when you've written a page of rules and the model still drifts on some of them. RAG Retrieval-augmented generation keeps the model unchanged and adds a search step. At query time you find the most relevant chunks of your data and put them in the context window alongside the question. python def answer(question: str) - str: chunks = vectorstore.search(embed(question), k=5) retrieve context = "\n\n".join(c.text for c in chunks) prompt = ( "Answer using only the context below. " "If the answer isn't there, say so.\n\n" f"Context:\n{context}\n\nQuestion: {question}" ) return llmcall(prompt) generate What it changes: what the model knows for this request. Nothing is learned permanently. What it's good at: - large or frequently changing knowledge bases - answers that need citations, since you know exactly which chunks were used - access control, because you can filter retrieval by what the user is allowed to see - keeping information current: update the index, and the next answer reflects it Cost: an embedding model, a vector store (or a search engine you already run), and the engineering to chunk, index and keep data fresh. More input tokens per call. Data needed: your documents. No labelled examples required to get started, though you'll want some question–answer pairs to evaluate retrieval quality. Where it struggles: most RAG failures are retrieval failures. If the right chunk isn't found, the model can't use it. Bad chunking, weak queries and missing metadata cause more trouble than the choice of model. Hybrid search (keyword plus embeddings) and reranking help a lot. Fine-tuning Fine-tuning continues training the model on your own examples, so the weights change. What it changes: behaviour. How the model responds by default: format, style, terminology, how it handles a narrow task. It's a poor way to add facts. The model may pick some up, but unreliably, they go stale, and you can't cite them. Use retrieval for facts. What it's good at: - consistent output format or tone without a long prompt every time - narrow, repetitive tasks such as classification, extraction and routing, where a smaller tuned model can match a larger general one at lower cost and latency - shortening prompts, which cuts tokens on every call LoRA Full fine-tuning updates every weight, which needs a lot of GPU memory. LoRA (low-rank adaptation) freezes the original weights and trains small extra matrices added to certain layers. You end up with an adapter that's a small fraction of the model's size. Why it matters in practice: - far less memory to train, so it fits on modest hardware (QLoRA goes further by quantising the base model) - you can keep several adapters for different tasks on one base model - results are often close to full fine-tuning for task adaptation Cost: preparing data is usually the biggest cost, followed by training runs, evaluation and hosting the tuned model. Data needed: hundreds to thousands of good input–output examples, consistent and representative. Fifty sloppy examples will teach the model to be sloppy. The hidden cost is maintenance. When a better base model comes out, you retrain. When requirements change, you rebuild the dataset. They combine These aren't exclusive. A common mature setup is a model fine-tuned for format and domain behaviour, fed by RAG for current facts, with a solid prompt holding it together. But each layer should earn its place. A decision checklist Work through these in order: 1. Have I written a clear prompt with a few examples and tested it on real inputs? If not, do that first. 2. Is the failure about missing or changing information? Use RAG. 3. Do I need citations, freshness or per-user access control? Use RAG. 4. Is the failure about format, tone or consistency, after good prompting? Consider fine-tuning. 5. Do I have hundreds of high-quality examples, or can I build them? If not, fine-tuning isn't ready yet. 6. Is cost or latency the problem at high volume on a narrow task? A small model fine-tuned with LoRA may beat a large one. 7. Do I have an evaluation set to prove each change helped? Without it, none of these decisions are measurable. In short - Prompt first. It's cheap and fast, and often enough. - Knowledge problem: retrieve. Behaviour problem: fine-tune. - Spend effort on retrieval quality before blaming the model. - Fine-tune only with good data and an eval set, and prefer LoRA unless you have a reason not to.
Postgres Indexes: A Practical Guide
Most slow Postgres queries I've debugged came down to one of two things: a missing index, or an index that existed but couldn't be used. This is the mental model and the handful of patterns that cover most real cases. What a B-tree index is When you run CREATE INDEX without specifying a type, Postgres builds a B-tree. It's a sorted, balanced tree of the indexed values, each pointing to the row's location in the table (the heap). Because it's sorted, a B-tree is good at: - equality: WHERE email = 'a@b.com' - ranges: WHERE createdat = '2025-12-01' - sorting: ORDER BY createdat - prefix matches: WHERE name LIKE 'Yas%' (with the right collation or operator class) It does not help LIKE '%shank', because there's no sorted prefix to search on. sql CREATE TABLE orders ( id bigserial PRIMARY KEY, customerid bigint NOT NULL, status text NOT NULL, total numeric(12,2), createdat timestamptz NOT NULL DEFAULT now() ); CREATE INDEX idxorderscustomer ON orders (customerid); The primary key already gets a unique B-tree index automatically. Foreign keys do not, which catches people out: if you join or delete on customerid, index it yourself. Composite indexes: column order matters A composite index sorts by the first column, then by the second within each value of the first, and so on. Think of a phone book sorted by surname, then first name. sql CREATE INDEX idxorderscustcreated ON orders (customerid, createdat); This index serves: sql -- uses both columns SELECT FROM orders WHERE customerid = 42 AND createdat = '2025-11-01'; -- uses the leading column SELECT FROM orders WHERE customerid = 42; -- equality on the first, sort on the second: no separate sort step SELECT FROM orders WHERE customerid = 42 ORDER BY createdat DESC LIMIT 20; It is a poor fit for WHERE createdat = ... alone, because the dates are scattered across every customer. The rule of thumb: equality columns first, then the range or sort column. Put the column you filter on with = at the front, and the one you filter with or ORDER BY after it. Partial indexes If your queries only ever care about a subset of rows, index only that subset. sql CREATE INDEX idxorderspending ON orders (createdat) WHERE status = 'pending'; If 2% of orders are pending, this index is a fraction of the size of a full one, cheaper to maintain and more likely to stay in memory. The query has to include a matching condition for the planner to use it: sql SELECT FROM orders WHERE status = 'pending' ORDER BY createdat LIMIT 50; Partial indexes are also the clean way to enforce conditional uniqueness: sql CREATE UNIQUE INDEX oneactivecart ON carts (userid) WHERE active; Expression indexes An index on email doesn't help WHERE lower(email) = ..., because the indexed values aren't lowercased. Index the expression instead: sql CREATE INDEX idxusersemaillower ON users (lower(email)); SELECT FROM users WHERE lower(email) = 'yashank@example.com'; The query must use the same expression. The same applies to things like datetrunc('day', createdat), though often it's better to rewrite the query as a range on the raw column so a plain index works. Covering indexes with INCLUDE Normally an index lookup finds matching entries, then visits the table to fetch the other columns. If the index holds every column the query needs, Postgres can do an index-only scan and skip the table (as long as the visibility map says the pages are all-visible, which regular vacuuming keeps true). INCLUDE adds columns to the index leaf entries without making them part of the sort key: sql CREATE INDEX idxorderscustcov ON orders (customerid, createdat) INCLUDE (status, total); SELECT createdat, status, total FROM orders WHERE customerid = 42 ORDER BY createdat DESC; Use it for hot queries that read a few small columns. Don't stuff wide text columns in; the index gets large and the benefit disappears. Reading EXPLAIN ANALYZE Never guess whether an index is used. Ask: sql EXPLAIN (ANALYZE, BUFFERS) SELECT FROM orders WHERE customerid = 42 ORDER BY createdat DESC LIMIT 20; ANALYZE actually runs the query, so be careful with writes (wrap them in a transaction and roll back). What I look for: - Node types. Seq Scan reads the whole table. Index Scan and Index Only Scan use an index. Bitmap Heap Scan collects matches from an index first, then reads table pages in order, which is common for medium selectivity. - Estimated vs actual rows. rows=10 estimated against rows=50000 actual means the planner's statistics are off. Run ANALYZE orders; and see if the plan changes. - Where the time goes. Each node shows actual time=start..end. Read from the innermost node outwards and find the expensive one. - Sort nodes. A Sort with external merge spilled to disk. An index matching the ORDER BY can remove the sort entirely. - Buffers. shared hit came from memory, read came from disk. A sequential scan isn't automatically wrong. On a small table, or when the query returns a large share of rows, it's often the fastest plan, and the planner knows that. When indexes hurt Every index is a cost on writes. Each INSERT updates every index on the table; an UPDATE that changes an indexed column does too, and it can prevent Postgres's HOT optimisation, which avoids index updates when no indexed column changes. Indexes also take disk space, compete for memory and add vacuum work. So: - Don't index every column "just in case". - Find unused indexes with pgstatuserindexes (look for idxscan = 0 over a meaningful period) and drop them. - Watch for redundant indexes: one on (customerid) is mostly covered by one on (customerid, createdat). - On a live table, use CREATE INDEX CONCURRENTLY to avoid blocking writes while it builds. It's slower and can't run inside a transaction. What to remember - Start from the slow query, run EXPLAIN (ANALYZE, BUFFERS), and let the plan tell you what's missing. - Composite index: equality columns first, then range or sort. - Use partial indexes for hot subsets, expression indexes for transformed lookups, INCLUDE for index-only scans. - Index foreign keys yourself. - Treat every index as a write cost and remove the ones nobody uses.
FastAPI in Production: The Things the Tutorial Skips
FastAPI makes the first version of an API very pleasant. You write a typed function, add a decorator, and get validation and interactive docs for free. The trouble starts later, when real traffic arrives and something that worked fine on your laptop starts timing out. Most of those problems come from a handful of things the getting-started guide doesn't dwell on. Here's what I'd check before calling a FastAPI service production-ready. async def vs def, and blocking calls FastAPI lets you write endpoints as either async def or plain def, and the difference matters a lot. - An async def endpoint runs on the event loop. While it's awaiting something, other requests can run. But if it does anything blocking, such as a synchronous database driver, requests.get, or heavy CPU work, it blocks the entire loop and every other request on that worker waits. - A plain def endpoint is run in a thread pool, so blocking calls inside it don't freeze the loop. The classic mistake: python import requests @app.get("/rates") async def getrates(): r = requests.get("https://example.com/rates") blocks the event loop return r.json() Either make it fully async with an async client: python import httpx client = httpx.AsyncClient(timeout=5.0) @app.get("/rates") async def getrates(): r = await client.get("https://example.com/rates") return r.json() or keep it synchronous and declare it with def, letting FastAPI use the thread pool. The rule of thumb: use async def only if everything slow inside it is awaited. If in doubt, def is the safer default. For CPU-heavy work, neither helps much because of the GIL. Move it to a separate process or a task queue. Let Pydantic do the validation Request and response models are where FastAPI earns its keep. Be strict about them. python from pydantic import BaseModel, Field, fieldvalidator class PaymentIn(BaseModel): accountid: str = Field(minlength=1, maxlength=64) amount: float = Field(gt=0, le=1000000) currency: str = Field(pattern=r"^[A-Z]{3}$") @fieldvalidator("accountid") @classmethod def nospaces(cls, v: str) - str: if " " in v: raise ValueError("accountid must not contain spaces") return v class PaymentOut(BaseModel): paymentid: str status: str @app.post("/payments", responsemodel=PaymentOut) def createpayment(body: PaymentIn) - PaymentOut: ... Setting a responsemodel isn't just for docs. It filters the output, so an internal field you accidentally return doesn't leak to the client. Bad input gets a 422 with a clear error before your code runs, which keeps endpoint bodies free of defensive checks. Dependency injection for shared resources Depends is FastAPI's way of providing things an endpoint needs: a database session, the current user, settings. Using it consistently makes testing much easier, because you can override any dependency. python from fastapi import Depends def getdb(): db = SessionLocal() try: yield db finally: db.close() @app.get("/accounts/{accountid}") def readaccount(accountid: str, db=Depends(getdb)): return db.get(Account, accountid) in tests app.dependencyoverrides[getdb] = lambda: fakedb The yield form guarantees cleanup runs after the response. For things that should exist once per process, like an HTTP client or a connection pool, create them in the app's lifespan handler rather than per request. Background tasks are not a queue BackgroundTasks runs a function after the response is sent. It's handy for small things like writing an audit log entry or sending a notification. python from fastapi import BackgroundTasks @app.post("/signup") def signup(user: UserIn, tasks: BackgroundTasks): createuser(user) tasks.addtask(sendwelcomeemail, user.email) return {"ok": True} But it runs inside the same worker process. If the worker restarts, the task is lost. There are no retries, no visibility, and long tasks tie up the worker. For anything that must happen, takes a while, or needs retrying, use a real task queue (Celery, RQ, Arq, or a managed queue) with separate workers. My rough test: if losing the task would mean a support ticket, it doesn't belong in BackgroundTasks. Workers: uvicorn and gunicorn uvicorn main:app --reload is for development. In production you want several worker processes so one slow request or crash doesn't take down everything, and so you use more than one CPU core. A common setup runs gunicorn as the process manager with uvicorn workers: bash gunicorn main:app \ -k uvicorn.workers.UvicornWorker \ --workers 4 \ --bind 0.0.0.0:8000 \ --timeout 30 \ --graceful-timeout 20 Uvicorn can also run multiple workers on its own with --workers. In containers on Kubernetes, many teams run one uvicorn process per container and scale by adding pods instead. Either way, start with roughly one worker per core and measure. Timeouts everywhere A service without timeouts will eventually hang on a slow dependency and pile up requests until it falls over. Set them at every layer: - Outbound HTTP: always pass a timeout to your client. httpx has a default; requests waits forever unless you set one. - Database: set connection and statement timeouts in the driver or pool. - The worker: gunicorn's --timeout kills stuck workers. - The proxy or load balancer in front of the app. For async code, asyncio.timeout (Python 3.11+) or asyncio.waitfor bounds a whole operation, not just one call. Structured logging Plain text logs are fine until you need to search them. Log JSON with consistent fields, and include a request ID so you can follow one request through everything it touched. python import logging, time, uuid, json logger = logging.getLogger("api") @app.middleware("http") async def logrequests(request, callnext): requestid = request.headers.get("x-request-id", str(uuid.uuid4())) start = time.perfcounter() response = await callnext(request) logger.info(json.dumps({ "requestid": requestid, "method": request.method, "path": request.url.path, "status": response.statuscode, "durationms": round((time.perfcounter() - start) 1000, 1), })) response.headers["x-request-id"] = requestid return response Libraries like structlog make this neater. Don't log request bodies blindly; they often contain personal or payment data. Before you ship - Use async def only when every slow call inside is awaited; otherwise use def. - Validate inputs and set responsemodel on every endpoint. - Provide shared resources through Depends and the lifespan handler. - Keep BackgroundTasks for small, losable work; use a queue for the rest. - Run multiple workers, set timeouts at every layer, and log JSON with request IDs. None of this is exotic. It's just the difference between an API that works and one that keeps working.
Why Rule Engines Still Matter in Fraud Detection
When people hear "fraud detection" they usually picture a machine learning model scoring every transaction. Models are a big part of it. But in most real payment systems there's also a rule engine sitting in the path of every transaction, and it isn't going away. I spent this summer as an AI/ML intern at Card91, where I researched fraud detection and evaluated the open-source Tazama fraud engine. Reading through its rule logic made it much clearer to me why rules and models end up working side by side. What a rule engine does A rule engine evaluates each transaction against a set of explicit conditions: "more than five transactions from this card in ten minutes", "amount far above this customer's usual spend", "card used in two distant cities within an hour". Each rule produces a result, the results are combined, and the system decides whether to allow, block, or flag the transaction for review. The logic is written by people, readable by people, and changed by people. That's both the point and the limitation. Why rules haven't been replaced by ML Explainability. When a transaction is declined, someone will ask why: the customer, a support agent, an auditor, a regulator. "Rule 14: transaction amount exceeded the per-day limit for a new account" is an answer. "The model scored it 0.91" usually isn't, at least not without extra tooling. Latency. Payment authorisation has a tight time budget. Many rules are simple lookups and comparisons that run in microseconds once the right counters are in memory. They don't need a feature pipeline or a model server. Speed of response. When a new fraud pattern appears, a team can write and deploy a rule the same day. Retraining a model needs labelled examples of the new pattern, which by definition you don't have much of yet. Hard requirements. Some checks aren't predictions at all. Sanctions lists, blocked merchants, regulatory limits: these are policies, and policies should be code, not probabilities. Models are better at catching subtle patterns across many weak signals. Rules are better at being certain, fast and auditable. Most systems use rules as a first line and a model score as one more input, sometimes literally as a rule: "block if model score threshold and amount X". Common rule types Most transaction rules fall into a few families. - Velocity rules count events in a time window: transactions per card per minute, new payees per account per day, failed PIN attempts per hour. Fraudsters testing stolen cards tend to move fast. - Geolocation rules look at where the transaction happens: a country the customer has never used, or "impossible travel" where two transactions are too far apart for the time between them. - Amount threshold rules compare the amount to fixed limits or to the customer's own history: a first transaction far above the account's average, or many transactions just under a reporting threshold. - Behavioural rules look at changes: a new device followed immediately by a password reset and a large transfer. A simple rule in Python Here's a generic sketch. It isn't how any particular engine is built, but it shows the shape: each rule is a small, pure function that takes a transaction plus some context and returns a result with a reason. python from dataclasses import dataclass from datetime import datetime, timedelta @dataclass class Txn: cardid: str amount: float country: str ts: datetime @dataclass class RuleResult: ruleid: str triggered: bool reason: str = "" def velocityrule(txn: Txn, recent: list[Txn], window=timedelta(minutes=10), limit=5) - RuleResult: count = sum(1 for t in recent if t.cardid == txn.cardid and txn.ts - t.ts <= window) if count = limit: return RuleResult("VEL-01", True, f"{count} txns in {window}, limit {limit}") return RuleResult("VEL-01", False) def amountrule(txn: Txn, avgamount: float, factor=10) - RuleResult: if avgamount 0 and txn.amount avgamount factor: return RuleResult("AMT-01", True, f"amount {txn.amount} {factor}x avg {avgamount:.0f}") return RuleResult("AMT-01", False) Keeping rules pure (no database calls inside, context passed in) makes them easy to test and easy to reason about. The engine around them handles fetching history and combining the results. False positives are the real cost A rule that blocks all fraud is easy: decline everything. The hard part is false positives, legitimate customers getting blocked. Each one is a frustrated customer, a support call, and sometimes a lost customer. Some things that help: - Graduated outcomes. Not every triggered rule needs a block. Many should flag for review or ask for extra authentication. - Combine signals. One weak rule firing alone may mean little; three firing together means more. Weighted scoring across rules is common. - Personal baselines. "Ten times this customer's average" is usually better than a single fixed amount for everyone. - Watch the rates. Track how often each rule fires and how often its alerts turn out to be real fraud. A rule that fires constantly and is almost never right is noise. Testing rules properly Rules look simple, which is why they're easy to get subtly wrong. Off-by-one errors at window boundaries, time zones, currency conversions, and versus = all matter. Treat rules like any other code: python from datetime import datetime, timedelta def testvelocitytriggersatlimit(): now = datetime(2025, 7, 1, 12, 0) recent = [Txn("c1", 100, "IN", now - timedelta(minutes=i)) for i in range(5)] txn = Txn("c1", 100, "IN", now) assert velocityrule(txn, recent).triggered def testvelocityignoresoldtxns(): now = datetime(2025, 7, 1, 12, 0) recent = [Txn("c1", 100, "IN", now - timedelta(minutes=30))] 5 assert not velocityrule(Txn("c1", 100, "IN", now), recent).triggered Beyond unit tests, the useful habit is backtesting: run a new or changed rule over historical transactions before deploying it, and see how many it would have flagged. When porting rules between languages, comparing outputs from the old and new implementations on the same inputs is the quickest way to catch behaviour that drifted in translation. The short version - Use rules for what must be certain, fast and explainable; use models for subtle patterns. - Write each rule as a small pure function that returns a reason, not just a boolean. - Prefer graduated outcomes and combined signals over single hard blocks. - Test boundaries explicitly, and backtest before every rule change. Rules aren't the old way of doing fraud detection. They're the part of the system you can read, and that matters more than it sounds.
Prompt Engineering Is Mostly Specification Writing
"Prompt engineering" sounds like a bag of tricks: magic phrases, threats, tipping the model. Some of those tricks worked once, on some model, for some task. What keeps working is far more boring. A good prompt is a specification: it says what the input is, what the output must look like, what's out of scope, and how you'll know it's right. If you've ever written a ticket that a stranger could pick up and finish without asking you anything, you already know how to write most of a good prompt. Treat the prompt as a contract The model has no idea what you're building. It sees text and predicts more text. Every gap in your instructions gets filled with whatever is most plausible, and plausible is not the same as what you wanted. So write the prompt the way you'd write an interface contract: - Inputs: what the model will receive and in what form. - Outputs: the exact shape of the response. - Constraints: what it must not do. - Edge cases: what to do when the input is empty, ambiguous or off-topic. Vague prompt: text Summarise this customer complaint. Specified prompt: text You will receive one customer complaint email for a food delivery app. Return a JSON object with: - "category": one of "late", "wrongitem", "refund", "appbug", "other" - "summary": one sentence, under 25 words, no customer names - "needshuman": true if the customer mentions legal action or a health issue If the email is not a complaint, return {"category": "other", "summary": "", "needshuman": false}. The second one is longer, but every extra line removes a decision the model would otherwise make for you. Roles are context, not magic "You are a senior data engineer" doesn't make a model smarter. What a role line does is set context: vocabulary, assumed audience, level of detail. That's useful, but it's a small lever. A role with no task description is still a vague prompt. I find it more reliable to describe the situation directly: who will read the output, what they already know, and what they'll do with it. "The reader is a support agent who has 30 seconds to decide whether to escalate" tells the model more than any job title. Constraints do the heavy lifting Most bad outputs aren't wrong, they're unbounded: too long, too chatty, padded with caveats, or inventing details. Constraints fix that. - Length: "under 25 words", "at most three bullet points". - Scope: "use only the information in the provided text". - Refusal path: "if the answer isn't in the text, say NOTFOUND". - Tone: "plain language, no marketing words". Giving the model an explicit way to say "I don't know" matters more than people expect. Without one, the most plausible continuation is usually a confident guess. Examples beat adjectives "Write a concise, professional summary" means different things to different people, and to models. One or two examples of input and expected output pin down what you mean far better than adjectives. A few things I've learned about examples: - Make them representative, including at least one awkward case. - Vary them. If every example has three bullet points, every output will too. - Keep them consistent with your stated rules. If the instructions say 25 words and the example has 40, you've written a contradiction and the model will pick one. Ask for a schema when code reads the output If the response goes into a program, don't parse prose. Ask for JSON with named fields and allowed values, and validate it on the way in: python from typing import Literal from pydantic import BaseModel, Field class Complaint(BaseModel): category: Literal["late", "wrongitem", "refund", "appbug", "other"] summary: str = Field(maxlength=200) needshuman: bool raw = callmodel(prompt, emailtext) result = Complaint.modelvalidatejson(raw) raises on bad output Many APIs now have a JSON or structured-output mode. Use it, and still validate. A schema in the prompt plus a schema in code is the same contract written down twice, which is exactly what you want. Specs need tests A specification you never check is a wish. The step most people skip is evaluation: running the prompt against a fixed set of inputs and checking the outputs, every time you change something. It doesn't need to be fancy: python cases = [ ("My order came 2 hours late and cold.", "late"), ("I got paneer instead of chicken.", "wrongitem"), ("Hi, what are your opening hours?", "other"), ] failures = [] for text, expected in cases: out = Complaint.modelvalidatejson(callmodel(prompt, text)) if out.category != expected: failures.append((text, expected, out.category)) print(f"{len(cases) - len(failures)}/{len(cases)} passed") Twenty or thirty hand-picked cases catch most regressions. When you find a new failure in the wild, add it to the set. This is ordinary test-driven thinking applied to a component that happens to be probabilistic. For outputs that can't be checked with equality, such as summaries, you can check properties instead: length limits, required fields, banned words, whether named entities in the output actually appear in the input. Iterate like you'd debug When a prompt fails, resist the urge to add another sentence in capital letters. Look at the failing case and ask what the spec left open. Usually it's one of these: - An edge case nobody described. - Two instructions that conflict. - An example that teaches the wrong pattern. - A task that's really two tasks and should be two calls. Change one thing, rerun the eval set, compare. It's slower than vibes but you'll know whether you made it better. What to keep in mind - Write the prompt so a capable stranger could do the task from it alone. - Define the output shape exactly, and validate it in code. - Give the model an explicit way to say it doesn't know. - Use a couple of varied examples instead of adjectives. - Keep a small test set and rerun it on every change. Most of what gets called prompt engineering is just being precise about what you want. That skill was useful before LLMs and it'll stay useful after the current tricks stop working.
Building a PWA That Works on a Crowded Bus
The real test for a study app isn't a fast laptop on college Wi-Fi. It's someone on a packed bus, one bar of signal, trying to open their notes ten minutes before an exam. If the app shows a spinner and then a dinosaur, it has failed at the one moment it mattered. That's the problem I've been working on for NotesAid, a notes app for my classmates that's launching soon. It's a progressive web app (PWA), and most of the effort has gone into making it behave well with bad or no network. Here's what I've learned. What makes something a PWA A PWA is a website that can be installed and can work offline. Practically, that means three things: 1. Served over HTTPS (localhost is allowed during development). 2. A web app manifest describing how it looks when installed. 3. A service worker that intercepts network requests and can answer them from a cache. The manifest The manifest is a JSON file linked from your HTML with <link rel="manifest" href="/manifest.json": json { "name": "NotesAid", "shortname": "NotesAid", "starturl": "/", "display": "standalone", "backgroundcolor": "ffffff", "themecolor": "1f2937", "icons": [ { "src": "/icons/192.png", "sizes": "192x192", "type": "image/png" }, { "src": "/icons/512.png", "sizes": "512x512", "type": "image/png" }, { "src": "/icons/maskable-512.png", "sizes": "512x512", "type": "image/png", "purpose": "maskable" } ] } display: standalone hides the browser UI so it feels like an app. Provide a maskable icon, or Android will put your logo in an awkward white circle. iOS supports home screen installation too, but has historically been more limited and still reads some Apple-specific meta tags, so test on an actual iPhone. The service worker A service worker is a script that runs separately from your page and sits between it and the network. You register it from the page: javascript if ('serviceWorker' in navigator) { navigator.serviceWorker.register('/sw.js'); } Its lifecycle has three events that matter: - install: fires once for a new version. Pre-cache your app shell here. - activate: fires when this version takes control. Clean up old caches here. - fetch: fires for every request in scope. This is where caching strategies live. javascript const SHELLCACHE = 'shell-v3'; const SHELL = ['/', '/offline.html', '/app.css', '/app.js']; self.addEventListener('install', (event) = { event.waitUntil(caches.open(SHELLCACHE).then((c) = c.addAll(SHELL))); }); self.addEventListener('activate', (event) = { event.waitUntil( caches.keys().then((keys) = Promise.all(keys.filter((k) = k !== SHELLCACHE).map((k) = caches.delete(k))) ) ); }); Pick a caching strategy per kind of request There's no single right strategy. The trick is matching each type of request to the one that fits. Cache-first: check the cache, only go to the network on a miss. Best for things that never change once built, like versioned JS bundles, fonts and icons. javascript async function cacheFirst(req) { const cached = await caches.match(req); return cached || fetch(req); } Network-first: try the network, fall back to cache if it fails. Best for content that must be fresh when possible, like an API response for a user's notes list. Add a timeout, otherwise a weak connection means waiting a long time before the fallback kicks in. On a bus, a request that hangs is worse than one that fails fast. javascript async function networkFirst(req, cacheName, ms = 3000) { const cache = await caches.open(cacheName); try { const res = await Promise.race([ fetch(req), new Promise((, reject) = setTimeout(() = reject(new Error('timeout')), ms)), ]); cache.put(req, res.clone()); return res; } catch { return (await cache.match(req)) || caches.match('/offline.html'); } } Stale-while-revalidate: return the cached copy immediately, and fetch a fresh one in the background for next time. Best for content where slightly old is fine but speed matters, like a subject index or a shared note someone opened yesterday. javascript async function staleWhileRevalidate(req, cacheName) { const cache = await caches.open(cacheName); const cached = await cache.match(req); const network = fetch(req).then((res) = { cache.put(req, res.clone()); return res; }).catch(() = cached); return cached || network; } Then route in the fetch handler. Only handle GET requests; the Cache API won't store anything else. javascript self.addEventListener('fetch', (event) = { const { request } = event; if (request.method !== 'GET') return; const url = new URL(request.url); if (url.pathname.startsWith('/api/')) { event.respondWith(networkFirst(request, 'api-v1')); } else if (url.pathname.startsWith('/next/static/')) { event.respondWith(cacheFirst(request)); } else { event.respondWith(staleWhileRevalidate(request, 'pages-v1')); } }); In practice I'd use Workbox rather than hand-writing all of this; it implements these strategies with expiry and edge cases handled. But writing them once by hand is the best way to understand what Workbox is doing. The update flow This is the part that surprised me most. When you deploy a new service worker, the browser installs it, but it sits in a waiting state until every tab using the old one is closed. Users with the app open all day may never get the update. You can call self.skipWaiting() to activate immediately, but then the new worker may serve new assets to a page that loaded old ones, which can break things. The safer pattern is to detect the waiting worker, show a small "New version available, refresh" prompt, and only then tell it to skip waiting and reload the page. Also: serve sw.js itself with no long-lived caching headers, so the browser can actually see new versions. Pitfalls I ran into, or nearly did - Caching opaque or error responses. Check res.ok before putting a response in the cache, or you'll happily serve a cached 500 forever. - Unbounded caches. Notes and images add up. Cap entries or age, and remember the browser can evict your storage under pressure. - Caching authenticated data on shared devices. Clear user-specific caches on logout. - Writes while offline. Reads are the easy half. For edits, queue them locally (IndexedDB) and sync when back online. Background Sync helps where supported, but don't rely on it everywhere; retry on app open as well. - Testing only on fast Wi-Fi. Chrome DevTools has offline and slow-network throttling. Use them constantly, and test on a real low-end phone. What I'd tell my past self Decide upfront which content must work offline, then pick a strategy per request type: cache-first for build assets, network-first with a short timeout for live data, stale-while-revalidate for everything that can be a little old. Version your caches, handle the waiting worker deliberately, and test with the network switched off. The bus is the real user.
Attention, Explained for Engineers
Every explanation of transformers I found while studying either stopped at "attention lets the model focus on relevant words" or jumped straight into a wall of matrix notation. The first is too vague to reason with. The second hides a fairly simple idea. This is the middle version: enough to understand why models behave the way they do, and why long prompts cost what they cost, with code you can run. Text becomes tokens A model never sees characters or words directly. A tokenizer splits text into tokens, chunks from a fixed vocabulary that are often whole common words, word pieces, or punctuation. "Unbelievably" might become un, believ, ably. Each token maps to an integer ID. This is why models are oddly bad at counting letters in a word: they never saw the letters, only the chunks. Tokens become embeddings Each token ID looks up a row in an embedding matrix, giving a vector of a few hundred to a few thousand numbers. These vectors are learned during training, and tokens used in similar contexts end up with similar vectors. So a sentence of n tokens becomes a matrix of shape (n, dmodel). Everything after this is arithmetic on that matrix. The problem attention solves A token's meaning depends on its neighbours. "Bank" in "river bank" and "bank account" should end up represented differently. Older recurrent networks passed information along the sequence one step at a time, which was slow to train and tended to lose track of distant words. Attention lets every token look at every other token directly, in one step, and decide how much each one matters to it. Queries, keys and values For each token, the model computes three vectors by multiplying its embedding with three learned weight matrices: - Query (Q): what this token is looking for. - Key (K): what this token offers to others looking. - Value (V): the information it actually passes along if chosen. The analogy I find useful is a fuzzy dictionary lookup. A normal dictionary matches a query to exactly one key and returns its value. Attention compares a query against every key, scores how well each matches, and returns a weighted blend of all the values. Scaled dot-product attention The scoring is a dot product: Q · K is large when the two vectors point in similar directions. For a whole sequence at once: Attention(Q, K, V) = softmax(Q Kᵀ / √dk) V Step by step: 1. Q Kᵀ gives an (n, n) matrix of scores: how much each token cares about each other token. 2. Divide by √dk (the key dimension). Without this, dot products grow with dimension, softmax saturates, and gradients become tiny. 3. Softmax each row, turning scores into weights that are positive and sum to 1. 4. Multiply by V, so each token's output is a weighted average of the value vectors. Here it is in NumPy: python import numpy as np def softmax(x, axis=-1): x = x - x.max(axis=axis, keepdims=True) for numerical stability e = np.exp(x) return e / e.sum(axis=axis, keepdims=True) def attention(Q, K, V, causal=False): dk = Q.shape[-1] scores = Q @ K.T / np.sqrt(dk) (seq, seq) if causal: mask = np.triu(np.oneslike(scores, dtype=bool), k=1) scores = np.where(mask, -np.inf, scores) weights = softmax(scores, axis=-1) each row sums to 1 return weights @ V, weights rng = np.random.defaultrng(0) seqlen, dmodel, dk = 4, 8, 8 X = rng.normal(size=(seqlen, dmodel)) 4 token embeddings Wq = rng.normal(size=(dmodel, dk)) Wk = rng.normal(size=(dmodel, dk)) Wv = rng.normal(size=(dmodel, dk)) out, w = attention(X @ Wq, X @ Wk, X @ Wv, causal=True) print(out.shape) (4, 8) print(w.round(2)) lower-triangular: no token sees the future The causal flag is what text-generating models use. Setting future positions to negative infinity before softmax makes their weights zero, so token 3 can attend to tokens 1 to 3 but not 4. That's what lets the model be trained to predict the next token without cheating. In a real model the weight matrices are learned, not random. Multi-head attention One attention pattern can only capture one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own smaller Q, K and V projections. One head might learn to track which noun a pronoun refers to; another might focus on the previous token. The heads' outputs are concatenated and projected back to dmodel. A transformer block is multi-head attention followed by a small feed-forward network applied to each token separately, with residual connections and normalisation around both. A model stacks dozens of these blocks. Where does word order come from? Look at the formula again: nothing in it knows about position. Shuffle the input tokens and each token's output is the same, just shuffled. Attention on its own treats a sentence as a bag of tokens. So position has to be injected. The original transformer paper added fixed sine and cosine patterns to the embeddings. Some models learn a position embedding per slot. Many recent models use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position, so their dot product reflects relative distance. The details vary; the point is that order is added information, not something attention gets for free. Why context windows cost what they do The score matrix is n × n. Double the prompt length and you quadruple the number of query-key comparisons, per head, per layer. That quadratic cost is the main reason context windows were small for so long and why long prompts are slow and expensive. There's a second cost during generation. To avoid recomputing everything for each new token, models keep a KV cache: the keys and values for every previous token, at every layer. It grows linearly with context length and can eat a lot of GPU memory, which limits how many requests a server can handle at once. Plenty of engineering goes into softening this, from memory-efficient attention kernels to sparse and sliding-window attention patterns. But the basic shape holds: more context means more compute and memory, and providers price accordingly. The short version - Text becomes tokens, tokens become vectors. - Attention scores every token against every other with Q · K, softmaxes the scores, and blends the values. - Multiple heads learn different relationships; position has to be added separately. - The n × n score matrix and the KV cache are why long contexts are expensive. If you remember one thing, make it the fuzzy dictionary lookup. Most transformer behaviour, good and odd, makes more sense once you picture every token asking every other token "are you relevant to me?" at every layer.
REST API Design Basics I Wish I'd Learned Earlier
Most APIs I've written for college projects worked. Few of them were pleasant to use. The difference is a handful of conventions that cost nothing to follow once you know them, and a lot to retrofit once a frontend depends on your mistakes. This is the checklist I now run through before writing a single route. Model resources, not actions A REST API is organised around resources: nouns your clients care about. The HTTP method carries the verb. http GET /notes list notes POST /notes create a note GET /notes/42 fetch one note PATCH /notes/42 update part of it DELETE /notes/42 delete it GET /notes/42/comments comments belonging to note 42 Avoid /getNotes, /createNote, /deleteNoteById. They duplicate what the method already says and make the surface area grow with every feature. Some operations genuinely aren't CRUD, like "publish this note" or "send a password reset". It's fine to model those as a sub-resource or an action endpoint (POST /notes/42/publish). Just keep them the exception. Use plural nouns, lowercase, and hyphens in paths. Keep nesting shallow; past two levels, a filter usually reads better (/comments?noteid=42). Use status codes properly Status codes let clients handle results without parsing your message strings. The ones you'll use most: | Code | Meaning | |------|---------| | 200 OK | Success with a body | | 201 Created | Resource created; include a Location header | | 204 No Content | Success, nothing to return (common for DELETE) | | 400 Bad Request | Malformed request or invalid input | | 401 Unauthorized | Not authenticated | | 403 Forbidden | Authenticated, but not allowed | | 404 Not Found | Resource doesn't exist (or you won't admit it does) | | 409 Conflict | State conflict, e.g. duplicate or version mismatch | | 422 Unprocessable Content | Well-formed but fails validation (some APIs use 400 for this) | | 429 Too Many Requests | Rate limited; send Retry-After | | 500 Internal Server Error | Your bug, not theirs | The one rule I'd enforce above all: never return 200 with "success": false in the body. It breaks every HTTP client, proxy and monitoring tool that trusts the status line. Make error bodies consistent Pick one error shape and use it everywhere. Clients will write one handler for it. There's a standard worth borrowing, Problem Details (RFC 7807, updated as RFC 9457), served as application/problem+json: json { "type": "https://api.example.com/errors/validation", "title": "Validation failed", "status": 422, "detail": "One or more fields are invalid.", "errors": [ { "field": "title", "message": "must not be empty" }, { "field": "subject", "message": "unknown subject code" } ] } Include a machine-readable identifier (type or a code field) so clients branch on that, not on the human message. Never leak stack traces or SQL errors in production responses. Paginate from day one Any list endpoint will eventually be too large to return in one go. There are two common approaches. Offset pagination is the familiar one: http GET /notes?limit=20&offset=40 It's easy to implement and lets you jump to page 7. But it has two problems. Large offsets get slow, because the database still walks past every skipped row. And if items are inserted or deleted while someone pages through, they'll see duplicates or miss rows. Cursor pagination returns an opaque token pointing at the last item seen: http GET /notes?limit=20&cursor=eyJpZCI6MTIzNH0 json { "data": [ { "id": 1235, "title": "DBMS unit 3" } ], "nextcursor": "eyJpZCI6MTI1NH0", "hasmore": true } Under the hood the cursor encodes something like "id greater than 1234", which turns into an indexed WHERE id ? ORDER BY id LIMIT 20. It stays fast and stable under writes. The trade-off is no random page access. My default: cursor pagination for feeds and anything that grows, offset for small admin tables where "go to page 5" matters. Either way, set a sensible default limit and a hard maximum. Idempotency keys for unsafe retries Networks fail. A client sends POST /payments, the connection drops, and it has no idea whether the payment went through. Retrying might charge twice. GET, PUT and DELETE are defined as idempotent: doing them twice has the same effect as doing them once. POST isn't. The fix is an idempotency key: the client generates a unique value and sends it with the request. http POST /payments Idempotency-Key: 7b4c1f0e-2a9d-4c55-9e1a-3f2d8b6c0a11 Content-Type: application/json { "amount": 49900, "currency": "INR" } The server stores the key with the result. If the same key arrives again, it returns the stored response instead of doing the work twice. A few details matter: scope keys per user, expire them after a reasonable window, and reject a reused key that comes with a different body. Version deliberately You will need to make breaking changes. Plan for it. - URL versioning (/v1/notes) is the most visible and easiest to route and cache. Most public APIs do this. - Header versioning (Accept: application/vnd.example.v2+json or a custom header) keeps URLs clean but is harder to test in a browser. Whichever you choose, the bigger discipline is not breaking things inside a version. Adding a field is fine. Renaming, removing, or changing a field's type is not. Clients should ignore fields they don't recognise, and you should document that expectation. What I'd tell my past self - Nouns in the path, verbs in the method. - Honest status codes, one error format, no 200 failures. - Paginate every list, and prefer cursors for anything that grows. - Accept an Idempotency-Key on any POST that moves money or sends messages. - Put /v1 in the URL now, and treat every field you ship as a promise. None of this is clever. That's the point: a boring, predictable API is the one people enjoy integrating with.