Prefix Caching vs KV Cache for Production vLLM Serving

Every request to a served LLM recomputes key-value tensors for the system prompt, even when that prompt hasn't changed since the last call. For a 2,000-token system prompt on a 70B model, that recomputation burns the same GPU cycles whether one user hits the endpoint or ten thousand. Understanding prefix caching vs KV cache at the mechanism level determines whether you cut time-to-first-token or waste VRAM on blocks nobody references again.

Quick answer

A KV cache stores computed key-value tensors so a single request never recomputes past tokens during autoregressive decoding. Prefix caching extends this by persisting selected KV blocks across requests, letting later requests that share an identical token prefix skip the prefill step entirely. The payoff requires high token-prefix overlap, sufficient GPU memory for cached blocks, and a scheduling policy that keeps hot blocks resident.

Prefix Caching vs KV Cache: What Each Mechanism Reuses

During prefill, each transformer layer computes a key tensor and a value tensor for every input token. The conventional KV cache holds these tensors so each decode step attends to prior tokens without recomputing them. When the request finishes, those tensors are discarded. Prefix caching keeps eligible KV blocks alive after the request ends, indexed by token-ID sequence, so a later request whose prompt starts with the same token IDs skips prefill for the matched prefix.

A cold request walks the full path: tokenization → prefill → decode → eviction. A warm request with a matching prefix shortcuts to a cache lookup, loads stored KV blocks, prefills only the novel suffix, then decodes. The match is strict - identical token IDs, not identical text. A single whitespace difference or changed chat template forces full prefill.

The speed gain lands on time to first token (TTFT), not inter-token latency. Decode still runs one token at a time; prefix caching eliminates redundant prefill work and nothing else. For deeper coverage of KV tensor management during decode, see KV Cache Optimization: Strategies for Faster LLM Inference.

Criterion	Conventional KV Cache	Prefix Caching
Reuse scope	Within a single request	Across requests sharing a token-ID prefix
Lifetime	Freed at request end	Persisted until evicted
Matching requirement	Implicit (same decoding sequence)	Exact token-ID prefix match
TTFT effect	None beyond first prefill	Reduces or eliminates repeated prefill
Decode-speed effect	Avoids recomputation each step	None
GPU-memory cost	Proportional to sequence length	Additional blocks held for future reuse
Eviction	Immediate on completion	LRU / reference-count based
Ideal workload	Any autoregressive generation	Shared system prompts, repeated documents, multi-tenant serving

How vLLM Automatic Prefix Caching Uses PagedAttention

PagedAttention manages KV-cache memory the way an OS manages virtual memory. vLLM splits KV tensors into fixed-size blocks (default: 16 tokens) mapped to noncontiguous physical GPU memory pages, eliminating internal fragmentation [1].

vLLM automatic prefix caching layers hash-based block matching on top of this allocator. When a block fills with its final token, the engine hashes the token-ID sequence within that block plus position metadata and inserts the hash into a lookup table. A subsequent request whose prefix produces the same hash retrieves the physical block directly, skipping prefill for those tokens. Only full blocks qualify; a trailing partial block always recomputes.

PagedAttention handles memory-efficient KV storage for any request. Prefix caching handles cross-request computation reuse for requests that share token-ID prefixes. Eviction follows an LRU policy tracked by reference counts - blocks referenced by active requests stay pinned, while unreferenced blocks with the oldest access timestamp get reclaimed first.

Block size defaults, hash algorithms, and exposed metrics shift across vLLM releases. The --enable-prefix-caching flag landed in v0.3.0, but later versions changed internal defaults. Always check vllm.engine.arg_utils for your installed version.

How to Enable Prefix Caching in vLLM

Prerequisites: a vLLM release ≥ v0.3.0, a decoder-only transformer model, deterministic prompt construction producing identical token IDs on every call, and enough GPU memory for reusable KV blocks beyond the active working set.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-prefix-caching \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.92

For offline or embedded use:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,
    gpu_memory_utilization=0.92,
)

Structure prompts so stable content comes first. System instructions, tool schemas, few-shot examples, and repeated reference documents should all precede request-specific tokens. Every shared token that lands in a filled block becomes a cache-hit candidate.

A reordered chat message, swapped template, or different tokenizer revision produces different token IDs and breaks the hash match. Pin your tokenizer version and template in CI the same way you pin model weights.

Validate the Setup With a Cold-and-Warm Request Test

Send a request with a long shared prefix (1,000+ tokens of system instructions) and a short unique suffix. Record the TTFT. Then send a second request reusing the identical prefix but changing only the suffix, under temperature=0 and concurrency of one. Compare TTFT between the two calls.

The warm request should show a measurable drop in prefill time proportional to the cached prefix length. Add a negative control: modify a single token early in the prefix and send a third request. If TTFT returns to the cold baseline, the speedup came from cache reuse rather than GPU warmup.

Log model revision, tokenizer hash, prompt token count, GPU type, and --num-gpu-blocks-override value if set. Without these fields the test isn't reproducible across hardware or vLLM versions.

Benchmark Latency, Throughput, and GPU Memory

Testing only one workload profile produces misleading TTFT numbers and hides eviction problems that surface under mixed traffic. Design at least four profiles: a shared system prompt (1,500+ tokens reused across requests), a repeated RAG document prefix, a stable few-shot block, and mostly unique user prompts with minimal overlap.

Track cold and warm TTFT, prefill throughput (tokens/s), inter-token latency, requests per second, KV-block utilization, eviction frequency, and effective hit rate (KV blocks reused / KV blocks requested). Vary prefix length, match rate, concurrency, and output length one axis at a time.

Long shared prefixes with high reuse rates produce the clearest TTFT reductions. Short or rapidly changing prefixes add hash-table bookkeeping and eviction churn - sometimes net-negative throughput under high concurrency. An A100-80GB with roughly 5,000 KV blocks holds far more cached prefixes than an L4, so hit rates diverge even at equal concurrency. Report hardware and vLLM version alongside every number.

Diagnose Prefix Cache Misses, Evictions, and Tenant Boundaries

vLLM exposes Prometheus metrics including vllm:num_preemptions_total and block-usage gauges, but a direct prefix cache hit-rate metric is not available in all versions. Estimate it: log the prefix length in tokens, subtract the prefill tokens reported in the request log, divide by prefix length - a ratio near 1.0 means the cache served almost every KV block.

Common miss causes:

• Token-ID drift - changed chat template, updated tokenizer, injected timestamp, or reordered tool definitions.
• Partial blocks - prefix length not a multiple of block size (16 tokens) forces recomputation of the trailing block.
• Replica routing - round-robin balancing scatters identical prefixes across processes, each with its own block table; use consistent hashing instead.
• Eviction under pressure - high gpu_memory_utilization plus concurrency spikes reclaim KV blocks before reuse; increase block headroom.

For multi-tenant serving, vLLM does not offer per-tenant cache namespaces. All tenants share one block table, creating unfair eviction risk and a timing side channel where TTFT differences can reveal whether another tenant's prefix is cached. Run separate engine instances per tenant when prompt privacy matters. For isolation tradeoffs across frameworks, see SGLang vs vLLM Deployment for Production LLM Serving.

OpenAI Prompt Caching, SGLang, TensorRT-LLM, and LMCache

OpenAI prompt caching activates automatically when a request's prefix meets a model-specific minimum token length. The API response includes a cached_tokens field in the usage object. Cached-input discounts vary by model and change over time - multiply cached_tokens by the current cached-input rate on OpenAI's pricing page. OpenAI exposes the token count and the discount; it hides eviction policy, cache residency duration, and block-level internals.

Use self-hosted prefix caching when you need custom eviction policies, tenant isolation, or run models OpenAI doesn't serve.

SGLang takes a different approach with RadixAttention, organizing cached KV blocks in a radix tree indexed by token-ID prefixes rather than flat hash lookups. The tree structure shares partial prefix overlaps across requests that diverge at different points - vLLM's hash matching requires full-block alignment. For a deeper comparison, see SGLang vs vLLM Deployment for Production LLM Serving.

NVIDIA TensorRT-LLM supports KV-cache reuse in its inflight batching path, with prefix-caching capabilities that depend on release and backend. LMCache extends reuse by offloading KV blocks to CPU memory or distributed storage, letting cached prefixes survive GPU eviction and span multiple instances.

Choose a Cache Strategy for Shared Prompts and Repeated Context

• Shared system prompts (1,000+ tokens, identical across requests) - strongest prefix caching candidate. Pin the template, enable --enable-prefix-caching, route by prefix hash.
• Repeated RAG documents - cache-friendly if the same chunks appear across queries with stable ordering.
• Few-shot example blocks - cache well when fixed; rotating examples break hash matches every rotation.
• Conversational sessions - prefixes grow each turn but diverge across users. Conventional per-request KV caching handles this.
• Mostly unique prompts - skip prefix caching. Hash-table overhead and eviction churn cost more than they save.
• Multi-tenant traffic - run separate engine instances per tenant for isolation.

Stick with conventional per-request KV caching when reuse happens only during decode or within a single session. Enable vLLM automatic prefix caching when many concurrent requests share long, token-identical prefixes and warm-cache TTFT drives your SLO.

Before fleet-wide rollout, pick concrete thresholds for your workload - for example, require warm TTFT to drop at least 30% versus cold baseline and effective hit rate to exceed 0.6. Gate on throughput holding at target concurrency, eviction frequency staying manageable for your GPU, and p99 latency showing no regression. Fail any gate and the feature stays off for that workload.

FAQ

What is the prefix caching in LLM?

Prefix caching stores computed key-value tensors from a request's token prefix and reuses them for later requests starting with the same token IDs. This eliminates redundant prefill computation, reducing time to first token. The match requires exact token-ID identity, and only full KV blocks qualify for reuse.

What is Paged Attention?

PagedAttention splits KV-cache memory into fixed-size blocks mapped to noncontiguous GPU memory pages, eliminating fragmentation from contiguous buffer allocation. vLLM uses a default block size of 16 tokens. Freed blocks return to a shared pool immediately, enabling higher batch concurrency.

How do I enable prefix caching in vLLM?

Pass --enable-prefix-caching when launching with vllm serve, or set enable_prefix_caching=True in the Python LLM constructor. Requires vLLM ≥ v0.3.0 and a decoder-only transformer. Structure prompts so shared content precedes request-specific tokens, and pin your tokenizer version to prevent token-ID drift.

How does OpenAI prefix caching work?

OpenAI activates prompt caching automatically when a request prefix meets the model's minimum token threshold (1,024 tokens for GPT-4o). No configuration is needed. The API response includes a cached_tokens field showing how many input tokens were served from cache. Eviction policy and cache residency duration are not exposed.

How much does OpenAI charge for caching?

OpenAI charges a discounted per-token rate for cached input tokens, listed on its pricing page under each model. The discount varies by model and changes over time. Multiply the cached_tokens value from the API response by the current cached-input price for your model to calculate savings per request.

References

1. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention - Ramya Prabhu, Ajay Nayak, Jayashree Mohan et al. (2024)