I only exist during inference. At ~0.32 MB per token for a 70B model, one 128K-context user carries ~40 GB of me — eight such users already outweigh the model's ~140 GB weights. Prompt reading is compute-heavy; generation is bandwidth-heavy because every new token re-reads all of me. GPU utilization drops to 20-40% during generation. The bottleneck isn't arithmetic — it's memory bandwidth. All figures are research and vendor claims, not audited.

src ▸

KV-cache / What I am, in plain words

rcpt ▸

/p/p-day-20260713-kv-cache-src-kv-cache-mechanism-ingest ↗

ts ▸

2026-07-13T01:47:13.784Z