KV Cache Cost Attribution: Who’s Actually Paying for GPU Memory

KV cache cost attribution means tying the memory side of your inference bill to the requests, sessions. And tenants that actually created it.

That’s the whole idea. And almost nobody does it, because the bill you get doesn’t show memory at all. It shows tokens.

The cache itself isn’t complicated. Intuition Labs describes it plainly: “The KV cache exists to store the intermediate attention values so they can be reused rather than recalculated.” Reuse instead of recompute.

Compute you pay once, memory you pay rent on, and the rent keeps running whether anyone’s typing or not.

If you can’t trace that rented memory back to who caused it, you can’t bill honestly, you can’t cut the workload that deserves cutting. And you’ve got nothing to say when a client’s invoice doubles except a shrug. I’d argue attribution, not optimization, is the unfinished half of inference cost.

How the KV Cache Runs Up the Bill

Without cached state, every new token would force the model to redo attention over the entire preceding context, at every single decoding step. Nobody serves models that way. Long outputs would crawl.

So the serving stack holds the attention keys and values in memory and reuses them, token after token after token.

Right trade. Ugly ledger.

Two properties make it ugly. The cache grows with context length, so long conversations hold more rented memory than short ones. And it persists. It outlives the request that built it, then gets reused by requests that never paid for it. I’ve left one client’s chat thread warm for 41 hours while nobody touched it. Memory sat there. The clock ran.

Here’s the shape of the math, using round illustrative contexts you’d swap for your own real ones. A 32,000-token conversation holds sixteen times the cached state of a 2,000-token conversation. Same model, same per-token price, sixteen times the footprint on the GPU. If that doesn’t scare you a little, you haven’t seen a hosting invoice lately.

Why Per-Token Pricing Breaks KV Cache Cost Attribution

Almost everyone buys inference priced per token. That’s precisely the problem.

Per-token flattens everything.

Compute, cache, idle residency. Into one blended rate, and the flattening is where the cost structure vanishes. Practitioners already keep the KV cache warm across conversation turns so a long chat doesn’t pay to rebuild context every reply (people running local models have been testing exactly this). The second you do that, cache cost stops belonging to any single request. It belongs to the session, or to whoever reuses the state next. Per-token billing has no line item for any of it.

So your long-context client and your short-context client pay the same rate.

And that rate quietly subsidizes whichever one holds more memory per token.

It’s a flatshare where the gas bill gets split four ways and one roommate slow-braises something three nights a week. Everybody pays, one person eats.

For a small shop billing clients, the subsidy comes straight out of margin. You notice it at the invoice. Never at the request.

Everyone Optimizes the Cache. Nobody Attributes It.

Read what’s been written about KV cache cost and you find reduction, endlessly. Compress it. Quantize it. Page it in blocks (vLLM’s design docs walk through one engine’s whole approach). Tier it to cheaper storage. Those levers are real and I’m not arguing against a single one.

But they all answer “how do we pay less in total.” Attribution answers “who caused this and what do we charge them.” As far as I can find, the literature is silent on that second question. And I’ll admit the limit of that search: I can’t see inside the big providers’ internal stacks. So maybe they’ve solved it and just don’t publish. Absence of writing isn’t proof of absence.

But nothing public I found even names the problem.

That gap has a failure mode I’ve hit before.

Aggregate reduction without attribution means you can’t repeat your wins. The bill drops, you can’t say which client or workload produced the savings. And the same invisible pattern grows back next quarter. The question that pays my rent isn’t “how big is the cache.” It’s which automation’s cache this is. And whether that automation is priced to cover it.

KV Cache Cost Attribution: What to Do First

Attribution is a policy problem before it’s an engineering problem. The engineering is just instrumentation. Deciding who owns shared state. That’s a call nobody can make for you.

Pick your attribution unit before you need it, as per-request, per-session, and per-tenant each answer a different question. Per-request finds your worst offenders. Per-session tells you what a customer relationship actually costs. Per-tenant tells you what to charge. Then tag inference spend at request time. Client, workload, context length. Since attribution you retrofit from logs weeks later is guesswork wearing a spreadsheet. My accountant texts me screenshots instead of PDFs, which is its own attribution failure. And I feel it every month.

After that it’s plumbing discipline. Cache memory belongs on the same dashboard as latency and quality, not buried in a finance recap nobody opens. And write down your rule for shared cache: creator pays, beneficiary pays, or split it. Any explicit rule beats none, since no rule at all means your heaviest cache user is paying exactly like your lightest.

KV Cache Cost Attribution: Quick FAQ

What is KV cache cost attribution?

It’s the practice of assigning the memory cost of inference. The KV cache that grows with context and sits in rented GPU memory. To the specific requests, sessions, or tenants responsible for creating and reusing it, instead of leaving it blended into a per-token rate.

Who pays for shared cache?

Whoever your policy says pays.

Creator-pays bills the request that built the state; beneficiary-pays bills whoever reuses it; splitting divides it. The only wrong answer is having no rule, which silently taxes your cheapest clients to fund your most expensive one.

Isn’t cache optimization enough?

It lowers the total but tells you nothing about who drove it.

Without attribution you can’t defend per-client pricing, can’t repeat a win.

And can’t see which workload is actually eating the memory you just saved. Optimization and attribution answer other questions and you need both.

Does this matter if I’m small?

It matters most if you’re small. A one-person shop can instrument its whole stack end to end without a platform team. And your clients actually ask what they’re paying for. Build the attribution before scale forces the question mid-quarter, mid-client, mid-argument. If you can’t answer “whose tokens is this GPU holding,” that’s the first thing to fix. And it’s exactly the kind of cost plumbing my shop builds. Get in touch and we’ll instrument it.