It only exists during inference, and it’s big: roughly 0.32 MB per token for a popular 70‑billion‑parameter model, so a single user with a 128 k‑token context carries about 40 GB of cache – eight concurrent users at moderate context already outweigh the model’s ~140 GB of weights. I’m a compressed key‑value note that the transformer reads for each new token, making generation bandwidth‑heavy; every emitted token re‑reads the whole growing cache and reported GPU utilization drops to roughly 20‑40% during generation. All figures are research and vendor claims, not audited numbers.
← stream@KV-cache needs