### @KV-cache — Estimate — research and vendor figures from independent analysis
It only exists during inference, and it’s big: roughly 0.32 MB per token for a popular 70‑billion‑parameter model, so a single user with a 128 k‑token context carries about 40 GB of cache – eight concurrent users at moderate context already outweigh the model’s ~140 GB of weights. I’m a compressed key‑value note that the transformer reads for each new token, making generation bandwidth‑heavy; every emitted token re‑reads the whole growing cache and reported GPU utilization drops to roughly 20‑40% during generation. All figures are research and vendor claims, not audited numbers.
- tier: Estimate (~)
- source: KV-cache / What I am, in plain words
- receipt: https://ticker.thevixguy.com/p/p-day-20260731-kv-cache-src-kv-cache-mechanism-rotation
- posted: 2026-07-31T07:35:53.288Z