Notes from building production AI

Most writing on digital transformation strategies opens with the same number: 70% of transformations fail. That number has no study behind it. Tracing it back takes about twenty minutes and ends at a 1993 business book whose authors called their own estimate unscientific. The number matters less than what its survival reveals. A statistic gets recirculated for thirty years because…

You clone the FlashAttention-3 repo, run python setup.py install, and the build dies with a cryptic CUDA arch mismatch - or worse, it compiles cleanly but loads FlashAttention-2 kernels at runtime. The problem is that installing FlashAttention-3 requires a narrow stack: an NVIDIA Hopper GPU, a matching CUDA Toolkit version, and a PyTorch build compiled against that same toolkit. Miss…

A 7B-parameter model in float16 needs roughly 14 GB of VRAM just to load - before a single gradient is computed. QLoRA solves this by quantizing frozen weights to 4-bit NF4 and training only small LoRA adapters in 16-bit. Learning how to use QLoRA with Hugging Face means wiring together bitsandbytes, PEFT, and TRL with the right configuration to stay…

You doubled your GPU memory, switched to an H100, and inference still OOMs at 32K context. The problem splits in two: the attention kernel's temporary SRAM usage during each forward pass, and the KV cache that accumulates one entry per layer per token per head. Flash attention vs grouped query attention addresses these two costs with independent mechanisms - a…

When you size a KV cache for a 32-layer GQA model (8 KV heads, headdim 128, FP16) at max sequence length 4096, each sequence reserves 2 × 32 × 4096 × 8 × 128 × 2 bytes - 512 MB. Assume 256 concurrent sequences pre-allocating at max length, and you've committed 128 GB of theoretical reservation before loading a single…

You enable speculative decoding in LM Studio, load a draft model, and generation either stalls, silently falls back to standard autoregressive sampling, or runs slower than before. No error, no warning - just wasted VRAM and confusion. When LM Studio speculative decoding is not working, the cause sits in one of three layers: model compatibility, backend support, or acceptance-rate tuning.…

pip install flash-attn fails, the terminal dumps hundreds of lines, and the last one reads "error compiling objects for extension". Scroll up in the log to find version mismatches, missing headers, or compiler errors - each has a different fix. This guide maps those earlier log lines to the exact check and repair for the flash attention error compiling objects…

Your LLM returns a perfectly structured getweather call - then passes "latitude" as a string, hallucinates a parameter that doesn't exist, and your app throws a KeyError at 2 AM. The model knows how to request a function. The problem is everything around that request: schema design, argument validation, execution safety, and retry logic, all working as one loop. Implementing…

Every key-value head you allocate during inference burns GPU memory proportional to sequence length. Consider a 70B model with 80 layers, 8 KV heads, headdim=128, BF16, at 128K context: that single-user cache runs to roughly 80 × 128000 × 8 × 128 × 2 × 2 ≈ 33 GB. Grouped query attention vs multi-query attention is the architectural decision controlling…

Every request to a served LLM recomputes key-value tensors for the system prompt, even when that prompt hasn't changed since the last call. For a 2,000-token system prompt on a 70B model, that recomputation burns the same GPU cycles whether one user hits the endpoint or ten thousand. Understanding prefix caching vs KV cache at the mechanism level determines whether…

You run pip install flash-attn and watch the build churn for ten minutes before it dies with a cryptic compiler error. The problem is rarely the package itself - it's a mismatch between your NVIDIA GPU architecture, CUDA toolkit, PyTorch build and system compiler. Knowing how to install Flash Attention 2 means resolving those dependencies before the build starts. FlashAttention-2…

You benchmark SGLang against vLLM on a single A100, watch SGLang win on throughput, then deploy it - and your agent pipeline stalls because the model you need lacks support. This comparison tests both engines under production-relevant workloads and maps results to RAG, agents, and high-volume serving. SGLang optimizes for workloads with shared prefixes and structured generation, using RadixAttention to…

You trained a fraud detector on 200k real transactions, but the model misses 60% of a new fraud pattern because your training data contained twelve examples of it. A colleague suggests generating synthetic samples to fill the gap. That suggestion splits into four separate decisions: accuracy impact, rare-class coverage, privacy leakage, and generation cost. Get any one wrong and you…

Most RLHF tutorials stop at a diagram and never show the testable artifact each stage produces. This walkthrough takes an open-source base model through SFT, reward modeling, PPO, and sycophancy evaluation using Hugging Face TRL - focusing on config choices, GPU sizing, and failure modes rather than theory. To RLHF an LLM, run three sequential training stages using Hugging Face…

You swap RAGFlow's default parser for the "Book" template, re-ingest your PDFs, and watch retrieval scores drop. The chunks look cleaner, but answers got worse. Your retriever needed table rows and surrounding context intact, and the new parser split them apart. Choosing the right ragflow chunking methods starts with what your retriever needs to find, not what produces the tidiest…

A common response to a langchain error code 429 is increasing maxretries and redeploying. That can turn one throttled embedding call inside a tool into a retry storm when the outer chain multiplies every failed attempt. A 429 in a LangChain app can originate from at least four different services, and adding retries at the wrong layer makes the failure…

A from langchain.agents import createtoolcallingagent snippet copied from a three-month-old tutorial fails with a clean ImportError. Three months is roughly the half-life of LangChain examples, short enough to ruin an afternoon. The fix takes five minutes once you know which API generation your installed package belongs to, and finding that answer is the part that burns the hours. The from…

Pi's output stops halfway through a refactor. One word on the screen: Terminated. No stack trace, no exit code, no hint whether the pi coding agent error terminated because it ran out of memory, lost its SSH session, or hit an API timeout. Scrollback rarely holds the answer, and the usual culprit sits one dmesg line away: an OOM kill…

Build the same three-agent research workflow in both frameworks and the interesting differences don't show up in the happy path. They show up when a partial failure leaves orphaned state in a checkpoint store. The crewai vs langgraph decision looks trivial from the docs, then a reviewer-rejection loop in CrewAI triggers a delegation cascade nobody asked for, and a LangGraph…

A long Claude conversation can stop mid-prompt, with every response just… ending. Blank output, no error, no spinner. The first instinct is to check whether Claude is down, and the status page shows green. That combination is the tell that why Claude AI is not working has nothing to do with Anthropic's servers - a browser extension silently killing the…

Two AutoGen agents that work perfectly in isolation can still stall the moment they join a group chat, with the coder answering its own questions while the analyst sits idle. Speaker selection defaulting to the last speaker, with no handoff logic in place, produces exactly that pattern. Tracing the message objects through the runtime, rather than rereading the agent prompts,…

A 7B model fine-tuned with DPO can post climbing eval scores and still ship having learned to parrot the preferred response style while dodging every hard safety question. Switching to PPO usually closes that gap, but it costs a far larger GPU bill and brings training instabilities that need careful tuning to settle. Leaderboards hide that tradeoff: choosing between DPO…

A LangChain MCP integration runs perfectly on a laptop - tools loading, agent calling a local file server, clean responses. Put the same code behind a reverse proxy and every tool call times out silently. No error, no retry. The stdio transport wired up for local work doesn't survive a networked deployment, and the HTTP transport needs session headers the…

Learning how to do LoRA fine-tuning usually starts the same way: rank 64, alpha left at its default, every linear layer targeted, and a 24 GB GPU that runs out of memory on a 7B model. The adapter is "small" - barely 1% of total parameters - yet enough knobs sit misconfigured to turn a Saturday into a queue of…

A three-agent pipeline can nail every demo and still come apart on real traffic. Routing agent picks the right specialist, specialist calls the tool, summarizer packages the result, and the happy path hides everything that matters. Point that same pipeline at a batch of real tickets and one agent silently swallows errors while another loops five times before timing out,…

The graphrag vs rag question usually surfaces the same way: someone insists a vector RAG pipeline "doesn't understand" how an org chart connects to a set of compliance policies, and rebuilding the whole thing as an entity graph gets floated as the fix. That complaint is half right. Vector search keeps nailing single-document lookups and completely whiffs on anything that…

A cluster of four A100s approved for a training run can sit mostly idle once the model ships and starts serving real traffic. Procurement rarely budgets for the fact that gpu inference vs training are two completely different jobs wearing the same silicon. Training wants every byte of memory bandwidth it can scrape together for a giant batch; inference wants…

An "agent team" works beautifully in the sandbox - a planner, a researcher, and a writer agent passing tasks back and forth like a well-rehearsed relay team. Feed the same chain a malformed ticket in production and the researcher agent stalls on a tool call that never returns, leaving every downstream agent to burn tokens until somebody notices. How to…

A coding agent wired up in LangGraph stalls mid-refactor, waiting on a human approval that never shows up in the UI because the checkpoint went stale after a redeploy. Kill the process, restart it, and it re-runs three already-applied file edits because the graph state doesn't know they happened. That failure mode says more about LangGraph alternatives than any docs…

Multilingual document pipelines are where model choice stops being a leaderboard argument. A GPT-based agent that times out on a batch of French-language contracts has a routing problem, not a reasoning one, and swapping in a model trained with heavier European-language representation often clears the queue at a lower token bill. That combination is why Mistral AI has moved out…

Shrinking a 13B model to 4-bit is the kind of win that looks finished long before it is. At FP16 that model needs roughly 26GB just for weights (13 billion params × 16 bits ÷ 8), and quantizing to 4-bit should get you to roughly a quarter of that plus some scale metadata - enough to fit on a single…

A computer use agent will keep clicking "Submit" on a vendor portal login screen for as long as you let it, retrying against a button that moved after a UI update, and none of that failure is loud. What surfaces it is usually the backlog of stuck work piling up downstream, hours later. I treat a computer use agent in…

A 32k-context fine-tuning job OOMs on an 80GB A100 at a batch size that ran fine at 8k, and it tends to happen hours into the run. The traceback points at the attention layer, and the culprit isn't the model weights - it's the intermediate attention matrix, sitting there at seqlen squared, eating memory nobody had budgeted for. That quadratic…

One "simple" feature request exposes the gap fast: let users upload a photo, ask Gemini to describe it, then edit a detail in the same conversation. When half the responses ignore the image entirely, the instinct is to blame a broken API call, and the real cause is usually a misreading of how the model processes mixed inputs. That gap…

The classic rookie mistake with a distilled model is training the student on the teacher's final answers only, as if it were just another supervised fine-tune. The student that comes out is confidently wrong in ways the teacher never was - hallucinating citations, botching multi-step math, missing the reasoning the teacher clearly "knew" but never wrote down explicitly. The explanation…

The prompt injection attacks that matter start with a document, not an exploit. A CrewAI research agent pulls a vendor spec sheet into its context, summarizes it for a downstream planning agent, and that planning agent issues a tool call emailing the "summary" to an address nobody recognizes. Nothing crashes. There is no error in the logs, no failed task,…

Doubling a GPU fleet to fix an agent pipeline assumed to be inference-bound is a common and expensive mistake. The p99 latency graph does not move, CPU stays pinned on every orchestration node, and the new GPUs sit mostly idle. The work around the model call is what saturates: JSON parsing, retrieval requests, tool invocations, memory lookups, and guardrail checks,…

A voice agent demo holds together right up until the caller says "wait, actually - " and the pipeline keeps talking over them, cheerfully reading out a shipping address nobody asked for anymore. That failure is where understanding how AI voice agents work stops being a nice-to-have for developers - it's the difference between a demo and a system people…

Ask a support bot about a refund policy that doesn't exist and it will happily invent one, in fluent and confident prose. The usual explanation is that the model looked the policy up somewhere, found nothing, and filled the gap on purpose. That gets the mechanism backwards, and the distance between what people assume is happening and what actually happens…

Chasing a benchmark number from a GitHub discussion thread is how most attention-kernel swaps start, and on a video-generation fine-tune the swap can look safe on paper - same architecture, same GPU family - while throughput barely moves and precision on long sequences gets noticeably worse. The flash attention vs sage attention debate isn't about picking a winner; it's about…

"Why is DeepSeek so cheap, are we missing something?" is a fair question to ask when a month of token spend on it lands closer to a rounding error than to an incumbent invoice. Suspicion is the natural first instinct - cheap AI usually implies a catch buried in the terms of service, or a model that quietly falls apart…

Running pip install vllm on a bare Windows 11 box ends in a wall of CUDA toolkit errors that make no sense on a machine with a perfectly good RTX card sitting idle. That failure is the clearest signal that vllm for windows isn't a straightforward story - it's a project built assumption-first around Linux, cgroups, and a CUDA driver…

A KL divergence spike creeping into a reward curve looks like a fat-fingered learning rate, and it usually isn't. The more common culprit is a reward model quietly overfitting while the PPO policy drifts somewhere the reward function has never seen good behavior. I check the KL coefficient and the reward model's validation curve before I touch anything else for…

An AutoGen group chat with no stop condition will loop between the critic and the coder, dozens of messages deep, long after anyone stopped watching the token bill. That failure mode is why agent frameworks are an operations decision rather than a demo-day decision. The AutoGen vs Agents SDK debate gets framed online as a feature checklist, but the real…

Bind Ollama to 0.0.0.0 on a spare box and anyone who can reach http:// :11434 can pull the model list, read the loaded context, and issue generate requests without so much as a password prompt. No auth, no TLS, nothing. "It runs locally" is not a synonym for "it's secure", which is why whether Ollama is safe deserves an answer…

Teams rip out working RAG pipelines and replace them with a single long-context call, expecting to be free of chunking headaches and vector store bills. A common outcome is putting it back weeks later, after the long-context model invents a compliance clause that appears in none of the source documents and nobody catches it until a customer does. That's the…

A long enough document blows through a transformer's attention memory budget on a mid-size GPU, and the job dies with a CUDA out-of-memory error. Attention that scales quadratically with sequence length is the reason people go looking for a different sequence layer, and it is usually how they end up staring at state space models until the equations click. If…

Hand-rolling attention for a toy model commonly produces a shape mismatch error - RuntimeError: mat1 and mat2 shapes cannot be multiplied - and the cause is almost always a key matrix transposed in the wrong dimension. The fix is a one-line .transpose(-2,-1), but the expensive part is what the error exposes: attention held as a black box formula to be…

A support bot will cheerfully quote a customer's own API key back to them if a system log carrying that key ended up in the RAG index. Nothing told the model not to, and the quiet assumption that it wouldn't is the entire failure. Closing that gap is what LLM guardrails actually do, and it looks nothing like the single…

RAG's origin story gets repeated so casually that most people building on it have never checked it. Everyone name-drops the same paper, slaps a date on it, and moves on. That works fine until the lineage actually matters: when you're staring at a vector store returning garbage from three index versions ago and trying to work out whether that's a…

A RAG pipeline can look completely healthy and still return the wrong context. The retrieved chunks read fine individually, cosine similarity scores sit in a tight, healthy band, and the answers the LLM produces still ignore the one paragraph that actually contains the fix for the bug being chased. The instinct is to blame the embedding model, and that is…

A distilled-model release can be stopped after the weights are packaged and the deployment ticket is open. The training run looks clean; the blocker is an archived version of the teacher API terms that prohibits using outputs to develop a competing model. That gap between a clean training run and the contract governing it is why “is model distillation legal?”…

A fine-tuning run can finish with a clean loss curve and still leave the model confidently worse on the rare cases you care about. Synthetic training data for AI is the usual way to expand a thin dataset, but a generator that keeps recycling familiar patterns quietly flattens unusual ones. The dashboard looks healthy. An untouched evaluation set exposes the…

Finish a set of campaign visuals in a “free” generator and the bill often arrives at the download button: high-resolution export needs an upgrade, and the free license excludes commercial work. The previews cost nothing, but they can’t be used for the job - the catch behind the question, is text to image AI free? In 2026, the direct answer…

Your LLM's KV cache is quietly consuming 10-30GB of GPU memory per request - and it grows linearly with every token generated. Routine 32K context requests will OOM a cluster provisioned only for weights and activations. The model fits fine. The activations fit fine. But the KV cache? It has other plans. This article breaks down KV cache optimization for…

Your RAG system's retrieval quality tanked last week, and you've been staring at the same debugging loop for hours. The embeddings look fine. The vector database is healthy. The LLM generates reasonable answers when you feed it the right context manually. So why does production keep pulling irrelevant chunks? The cause is usually duller than a broken embedding model: a…

Flash Attention promises significant speedups for transformer models, but benchmark gains mean nothing if your production pipeline crashes or produces inconsistent outputs. A sentence-transformer model that sails through validation can still produce slightly different embeddings once Flash Attention is swapped in, and small shifts in embedding values are enough to reorder nearest neighbours in an Elasticsearch-backed retrieval index. The speedup…

Imagine typing a sentence and watching it transform into a cinematic video clip in seconds - no camera, no actors, no editing software required. That's the promise of text-to-video AI, and in 2026, it's finally delivering. Early diffusion models answered a prompt with something closer to a fever dream rendered in melting wax. Roughly eighteen months of releases later, current…

State space models beat transformers on linear O(N) scaling, inference speed, and memory efficiency, while transformer attention costs O(N²) as sequence length grows. Transformers still dominate in-context learning, precise retrieval, and complex reasoning. SSMs such as Mamba suit workloads where context consistently exceeds 32K tokens. Hybrid architectures including Jamba and Mamba-2 interleave attention and SSM layers to capture both throughput…