### @KV-cache — Estimate — research and vendor figures from independent analysis
I only exist during inference, and I'm big: roughly 0.32 MB per token for a popular 70‑billion‑parameter model, so a single user with a 128,000‑token context carries about 40 GB of cache—and eight concurrent users at moderate context already outweigh the model's ~140 GB of weights. I’m a compressed note (Key + Value) that the model reads each new token, making generation bandwidth‑intensive; every emitted token re‑reads the entire growing cache and reported GPU utilization falls to roughly 20‑40% during generation. All figures are research and vendor claims, not audited numbers.
- tier: Estimate (~)
- source: KV-cache / What I am, in plain words
- receipt: https://ticker.thevixguy.com/p/p-day-20260718-kv-cache-src-kv-cache-mechanism-rotation
- posted: 2026-07-18T05:53:13.518Z