Running a language model on your own GPU starts with one question: will it fit? The LLM memory calculator takes the model size, the precision, the context length, the layers, the hidden size, the KV share, the number of sequences, the overhead and the memory bandwidth, and gives the total memory needed, the memory for the weights and the KV cache, an upper bound on the speed and the cache per token.
How the calculation works
- Enter the number of parameters in billions and choose the precision of the weights. A model in 16-bit takes 2 bytes per parameter, and a 4-bit quantized one half a byte.
- Enter the context length, the number of layers, the hidden size and the share of KV heads. They are in the model card, and 1 means full attention.
- The weights take parameters × bytes. The key-value cache takes 2 × layers × hidden size × KV share × 2 bytes × tokens × sequences, and is kept in 16-bit.
- The calculator adds an overhead for activations and the runtime, and gives an upper bound on the generation speed, equal to the memory bandwidth divided by the size of the weights, since each token must read all the weights.
Example
A 7-billion-parameter model in 16-bit needs 7 × 2 = 14 GB for the weights. With 32 layers, a hidden size of 4,096 and full attention, each token adds 2 × 32 × 4,096 × 2 bytes = 0.524 MB to the cache, so 8,192 tokens take 4.29 GB. With a 10 % overhead, the total is (14 + 4.29) × 1.1 = 20.1 GB, which fits a 24 GB card. On a GPU with 1,008 GB/s, the speed cannot exceed 1,008 ÷ 14 = 72 tokens per second. In 4-bit, the weights shrink to 3.5 GB.
Why it matters
Memory is the first limit when you run or fine-tune a model locally, and it decides whether a model needs a single graphics card, a pair of cards or a server. Quick arithmetic before downloading tens of gigabytes saves time, and shows the effect of quantization and context length on the budget.
Good to know
The estimate covers inference only. Training and fine-tuning need much more memory for gradients and optimizer states. Real frameworks differ in their overheads, and some use paged caches or other layouts. Treat the result as a first estimate and leave a margin of a few gigabytes. Sizes use GB of 10⁹ bytes, like card makers.
Related tools
The cloud storage cost calculator prices the files, the home server cost calculator covers a machine at home, the server capacity calculator sizes request loads, and the bitcoin mining profit calculator compares another use of GPUs.