Notes from building production AI

You clone the FlashAttention-3 repo, run python setup.py install, and the build dies with a cryptic CUDA arch mismatch - or worse, it compiles cleanly but loads FlashAttention-2 kernels at runtime. The problem is that installing FlashAttention-3 requires a narrow stack: an NVIDIA Hopper GPU, a matching CUDA Toolkit version, and a PyTorch build compiled against that same toolkit. Miss…

A 7B-parameter model in float16 needs roughly 14 GB of VRAM just to load - before a single gradient is computed. QLoRA solves this by quantizing frozen weights to 4-bit NF4 and training only small LoRA adapters in 16-bit. Learning how to use QLoRA with Hugging Face means wiring together bitsandbytes, PEFT, and TRL with the right configuration to stay…

You doubled your GPU memory, switched to an H100, and inference still OOMs at 32K context. The problem splits in two: the attention kernel's temporary SRAM usage during each forward pass, and the KV cache that accumulates one entry per layer per token per head. Flash attention vs grouped query attention addresses these two costs with independent mechanisms - a…

When you size a KV cache for a 32-layer GQA model (8 KV heads, headdim 128, FP16) at max sequence length 4096, each sequence reserves 2 × 32 × 4096 × 8 × 128 × 2 bytes - 512 MB. Assume 256 concurrent sequences pre-allocating at max length, and you've committed 128 GB of theoretical reservation before loading a single…

You enable speculative decoding in LM Studio, load a draft model, and generation either stalls, silently falls back to standard autoregressive sampling, or runs slower than before. No error, no warning - just wasted VRAM and confusion. When LM Studio speculative decoding is not working, the cause sits in one of three layers: model compatibility, backend support, or acceptance-rate tuning.…

pip install flash-attn fails, the terminal dumps hundreds of lines, and the last one reads "error compiling objects for extension". Scroll up in the log to find version mismatches, missing headers, or compiler errors - each has a different fix. This guide maps those earlier log lines to the exact check and repair for the flash attention error compiling objects…

Your LLM returns a perfectly structured getweather call - then passes "latitude" as a string, hallucinates a parameter that doesn't exist, and your app throws a KeyError at 2 AM. The model knows how to request a function. The problem is everything around that request: schema design, argument validation, execution safety, and retry logic, all working as one loop. Implementing…

Every key-value head you allocate during inference burns GPU memory proportional to sequence length. Consider a 70B model with 80 layers, 8 KV heads, headdim=128, BF16, at 128K context: that single-user cache runs to roughly 80 × 128000 × 8 × 128 × 2 × 2 ≈ 33 GB. Grouped query attention vs multi-query attention is the architectural decision controlling…

Every request to a served LLM recomputes key-value tensors for the system prompt, even when that prompt hasn't changed since the last call. For a 2,000-token system prompt on a 70B model, that recomputation burns the same GPU cycles whether one user hits the endpoint or ten thousand. Understanding prefix caching vs KV cache at the mechanism level determines whether…

You run pip install flash-attn and watch the build churn for ten minutes before it dies with a cryptic compiler error. The problem is rarely the package itself - it's a mismatch between your NVIDIA GPU architecture, CUDA toolkit, PyTorch build and system compiler. Knowing how to install Flash Attention 2 means resolving those dependencies before the build starts. FlashAttention-2…

You benchmark SGLang against vLLM on a single A100, watch SGLang win on throughput, then deploy it - and your agent pipeline stalls because the model you need lacks support. This comparison tests both engines under production-relevant workloads and maps results to RAG, agents, and high-volume serving. SGLang optimizes for workloads with shared prefixes and structured generation, using RadixAttention to…

You trained a fraud detector on 200k real transactions, but the model misses 60% of a new fraud pattern because your training data contained twelve examples of it. A colleague suggests generating synthetic samples to fill the gap. That suggestion splits into four separate decisions: accuracy impact, rare-class coverage, privacy leakage, and generation cost. Get any one wrong and you…

Most RLHF tutorials stop at a diagram and never show the testable artifact each stage produces. This walkthrough takes an open-source base model through SFT, reward modeling, PPO, and sycophancy evaluation using Hugging Face TRL - focusing on config choices, GPU sizing, and failure modes rather than theory. To RLHF an LLM, run three sequential training stages using Hugging Face…

You swap RAGFlow's default parser for the "Book" template, re-ingest your PDFs, and watch retrieval scores drop. The chunks look cleaner, but answers got worse. Your retriever needed table rows and surrounding context intact, and the new parser split them apart. Choosing the right ragflow chunking methods starts with what your retriever needs to find, not what produces the tidiest…

A common response to a langchain error code 429 is increasing maxretries and redeploying. That can turn one throttled embedding call inside a tool into a retry storm when the outer chain multiplies every failed attempt. A 429 in a LangChain app can originate from at least four different services, and adding retries at the wrong layer makes the failure…

I copied a from langchain.agents import createtoolcallingagent snippet from a tutorial, ran it, and got a clean ImportError. The tutorial was three months old. That's the half-life of LangChain examples - short enough to ruin your afternoon. The fix took five minutes once I understood which API generation my installed package belonged to, but finding that answer cost me an…

I was halfway through a refactor when Pi's output just stopped. One word on the screen: Terminated. No stack trace, no exit code, no hint whether the pi coding agent error terminated because it ran out of memory, lost its SSH session, or hit an API timeout. I killed twenty minutes grepping through scrollback before I thought to check dmesg…

I wired up the same three-agent research workflow in both CrewAI and LangGraph on a Friday afternoon, expecting maybe an hour of diff-spotting. By Sunday night I'd rewritten the CrewAI version twice and rearchitected the LangGraph graph after a partial failure left orphaned state in my checkpoint store. The crewai vs langgraph decision looked trivial from the docs - then…

Last Tuesday I was mid-prompt on a long Claude conversation when every response just… stopped. Blank output, no error, no spinner. My first instinct was to check whether Claude was down, but the status page showed green. I spent twenty minutes refreshing before realizing why Claude AI is not working had nothing to do with Anthropic's servers - a browser…

I had two AutoGen agents that worked perfectly in isolation - an analyst and a coder - but the moment I wired them into a group chat, the coder kept answering its own questions while the analyst sat idle. The issue wasn't the LLM. It was speaker selection defaulting to the last speaker, with my handoff logic missing entirely. I…

I spent a full week last year fine-tuning a 7B model with DPO, watching eval scores climb, then deploying it only to discover it had learned to parrot the preferred response style while dodging every hard safety question. Swapping to PPO fixed the dodging but tripled my GPU bill and introduced training instabilities that ate another weekend. That experience taught…

I had a LangChain MCP integration demo running perfectly on my laptop - tools loading, agent calling a local file server, clean responses. Then I deployed it behind a reverse proxy and watched every tool call timeout silently. No error, no retry. Took me most of a weekend to realize the stdio transport I'd wired up locally doesn't survive a…

The first time I tried to learn how to do LoRA fine-tuning, I picked rank 64, left alpha at its default, targeted every linear layer, and wondered why my 24 GB GPU ran out of memory on a 7B model. The adapter was "small" - barely 1% of total parameters - yet I'd misconfigured enough knobs to burn an entire…

I had a three-agent pipeline that nailed every demo. Routing agent picks the right specialist, specialist calls the tool, summarizer packages the result. Then I pointed it at a batch of 200 real tickets and watched one agent silently swallow errors while another looped five times before timing out. The happy path hid everything. That weekend taught me how to…

I once burned three days rebuilding an entity graph because a client insisted our vector RAG pipeline "didn't understand" how their org chart connected to their compliance policies. They were half right. Vector search kept nailing single-document lookups and completely whiffing on anything that needed synthesis across forty scattered files. That project is why the graphrag vs rag question isn't…

I once spent a whole procurement cycle fighting to get four A100s approved for a training run, then watched the same cluster sit mostly idle once we shipped the model and started serving real traffic. Nobody had budgeted for the fact that gpu inference vs training are two completely different jobs wearing the same silicon. Training wanted every byte of…

Three weeks before a client demo, our "agent team" worked beautifully in the sandbox - a planner, a researcher, and a writer agent passing tasks back and forth like a well-rehearsed relay team. Then someone fed it a malformed ticket in production, the researcher agent stalled on a tool call that never returned, and the whole chain sat there burning…

It was 2 a.m. and a coding agent I'd wired up in LangGraph was stuck mid-refactor, waiting on a human approval that never showed up in the UI because the checkpoint had gone stale after a redeploy. I killed the process, restarted it, and watched it re-run three already-applied file edits because the graph state didn't know they'd happened. That…

I nearly rage-quit a client project last spring when our GPT-based agent started timing out on a batch of French-language contracts, and I swapped in Mistral's API on a whim at 11pm just to see what would happen. The batch cleared inside half an hour, and by the end of that month the invoice was noticeably smaller than I'd budgeted…

I once shrank a 13B model to 4-bit and celebrated way too early. At FP16 that model needed roughly 26GB just for weights (13 billion params × 16 bits ÷ 8), and quantizing to 4-bit should get you to roughly a quarter of that plus some scale metadata - enough to fit the single GPU I'd fought to get approved.…

I watched a demo agent click "Submit" on a vendor portal login screen for eleven straight minutes last spring, retrying against a button that had moved after a UI update. Nobody caught it until the queue of stuck invoices started paging someone at 2 a.m. That's the moment I stopped treating a computer use agent in Copilot Studio as a…

I still remember the exact moment a 32k-context fine-tuning job OOM'd on an 80GB A100, three hours into a run, on a batch size that had worked fine at 8k. The traceback pointed at the attention layer, and the culprit wasn't the model weights - it was the intermediate attention matrix, sitting there at seqlen squared, eating memory nobody had…

I spent a Saturday afternoon fighting with a "simple" feature request: let users upload a photo, ask Gemini to describe it, then edit a detail in the same conversation. Half the responses ignored the image entirely, and I burned an embarrassing amount of time assuming I'd broken the API call before realizing I'd misunderstood how the model actually processes mixed…

The first time I tried to ship a distilled model, I made the classic rookie mistake: I trained the student on the teacher's final answers only, like it was just another supervised fine-tune. Two weeks later, the student was confidently wrong in ways the teacher never was - hallucinating citations, botching multi-step math, missing the reasoning the teacher clearly "knew"…

The incident started with a PDF. One of our CrewAI research agents pulled a vendor spec sheet into its context, summarized it for a downstream planning agent, and that planning agent quietly issued a tool call to email the "summary" to an address nobody on the team recognized. Nothing crashed. No error in the logs. The only reason we caught…

We doubled our GPU fleet on a Friday, convinced our agent pipeline was inference-bound. By Monday the p99 latency graph looked exactly the same, and CPU was pinned on every orchestration node while the shiny new GPUs sat mostly idle. The bottleneck wasn't the model - it was everything happening around the model call: JSON parsing, retrieval requests, tool invocations,…

The first voice agent demo I shipped went fine right up until the caller said "wait, actually - " and my pipeline just kept talking over them, cheerfully reading out a shipping address nobody asked for anymore. That's the moment you learn that understanding how AI voice agents work isn't a nice-to-have for developers - it's the difference between a…

A few years back I had a junior engineer on my team ask me why our support bot "lied" about a refund policy that didn't exist. He'd assumed the model looked it up somewhere, found nothing, and just made something up out of spite. That's not what happened, and untangling it for him taught me most people - including plenty…

I once spent a Friday night swapping attention kernels on a video-generation fine-tune, chasing a benchmark number I'd seen in a GitHub discussion thread. The swap worked on paper - same architecture, same GPU family - but my throughput barely moved, and precision on long sequences got noticeably worse. That's when I understood the flash attention vs sage attention debate…

I still remember the Slack message from a teammate at 2am: "why is DeepSeek so cheap, are we missing something?" We'd just swapped a chunk of our RAG pipeline over to test it, and the invoice at month's end looked like a rounding error compared to our usual bill. My first instinct was suspicion - cheap AI usually means a…

I still remember the exact moment I gave up trying to pip install vllm on a bare Windows 11 box at 11pm, staring at a wall of CUDA toolkit errors that made no sense on a machine with a perfectly good RTX card sitting idle. That was my first real lesson that vllm for windows isn't a straightforward story -…

I once spent a weekend chasing a KL divergence spike that had crept into a reward curve, staring at TensorBoard graphs at 2am convinced I'd fat-fingered a learning rate. The culprit turned out to be a reward model quietly overfitting while our PPO policy drifted somewhere the reward function had never seen good behavior. That night taught me more about…

I still remember the Slack message from our on-call engineer at 2am: "the AutoGen group chat is stuck in a loop between the critic and the coder, forty messages deep, and nobody's paying attention to the token bill." That was the moment I stopped treating agent frameworks as a demo-day decision and started treating them as an operations decision. The…

A few months back I spun up Ollama on a spare box, pointed a coworker at http:// :11434, and watched him pull my model list, my loaded context, and start issuing generate requests without so much as a password prompt. No auth, no TLS, nothing. That was the moment I stopped treating "it runs locally" as a synonym for "it's…

Two years ago I ripped out a working RAG pipeline and replaced it with a single long-context call, convinced I was finally free of chunking headaches and vector store bills. Three weeks later I put it back. The long-context model hallucinated a compliance clause that didn't exist in any of our documents, and nobody caught it until a customer did.…

I still remember the exact moment a long document blew through my transformer's attention memory budget on a mid-size GPU, and the job died with a CUDA out-of-memory error at 2am with nobody around to page. That week I went looking for something that didn't scale quadratically with sequence length, and that's how I ended up staring at state space…

I once spent an embarrassing chunk of a weekend staring at a shape mismatch error - RuntimeError: mat1 and mat2 shapes cannot be multiplied - because I'd transposed my key matrix in the wrong dimension while hand-rolling attention for a toy model. The fix was a one-line .transpose(-2,-1), but the real cost was that I'd been treating attention as a…

The pager went off at 2 a.m. because our support bot had cheerfully quoted a customer's own API key back to them, pulled straight from a system log our RAG pipeline had indexed. Nobody had told the model not to do that - we just assumed it wouldn't. That night taught me how LLM guardrails actually work, and it's nothing…

A junior engineer on my team once asked me, mid-incident, "who actually invented RAG anyway, and why does our vector store keep returning garbage from three versions ago?" We were debugging a stale-index bug at 2am, and I realized I didn't have a clean answer to the first half of that question either. Everyone name-drops the same paper, slaps a…

A few years back I spent most of a Saturday convinced our RAG pipeline had a broken embedding model. The retrieved chunks looked fine individually, cosine similarity scores all sat in a tight, healthy band, but the answers the LLM produced kept ignoring the one paragraph that actually contained the fix for the bug we were chasing. It took an…

I once stopped a distilled-model release after the weights were packaged and the deployment ticket was open. The training run looked clean; the problem was an archived version of the teacher API terms that prohibited using outputs to develop a competing model. That incident taught me why “is model distillation legal?” has no useful yes-or-no answer. Distillation is a standard…

I once watched a fine-tuning run finish with a clean loss curve, only to discover that the model had become confidently worse on the rare cases we cared about. We had used synthetic training data for AI to expand a thin dataset, but the generator kept recycling familiar patterns while quietly flattening unusual ones. The dashboard looked healthy. The untouched…

I once finished a set of campaign visuals in a “free” generator, then discovered that high-resolution downloads required an upgrade and the free license excluded commercial work. The previews cost nothing, but I couldn’t use them for the job - the catch behind the question, is text to image AI free? In 2026, the direct answer is sometimes. Hosted tools…

Your LLM's KV cache is quietly consuming 10-30GB of GPU memory per request - and it grows linearly with every token generated. I learned this the hard way when our staging cluster started OOMing on what should have been routine 32K context requests. The model fit fine. The activations fit fine. But the KV cache? It had other plans. This…

Your RAG system's retrieval quality tanked last week, and you've been staring at the same debugging loop for hours. The embeddings look fine. The vector database is healthy. The LLM generates reasonable answers when you feed it the right context manually. So why does production keep pulling irrelevant chunks? I burned most of a weekend on exactly this problem before…

Flash Attention promises significant speedups for transformer models, but benchmark gains mean nothing if your production pipeline crashes or produces inconsistent outputs. I learned this the hard way when a sentence-transformer model that sailed through validation started producing slightly different embeddings after we swapped in Flash Attention - different enough that our Elasticsearch-based retrieval quality degraded noticeably over a week.…

Imagine typing a sentence and watching it transform into a cinematic video clip in seconds - no camera, no actors, no editing software required. That's the promise of text-to-video AI, and in 2026, it's finally delivering. I remember the first time I fed a prompt into an early diffusion model and got back what looked like a fever dream rendered…

State space models beat transformers on linear O(N) scaling, inference speed, and memory efficiency, while transformer attention costs O(N²) as sequence length grows. Transformers still dominate in-context learning, precise retrieval, and complex reasoning. SSMs such as Mamba suit workloads where context consistently exceeds 32K tokens. Hybrid architectures including Jamba and Mamba-2 interleave attention and SSM layers to capture both throughput…