Skip to content
Omnikalc

LLM memory calculator: how much GPU memory does a language model need?

Enter the model size, the precision, the context length and the architecture to get the memory for weights, the KV cache and the total, with a speed limit.

Updated on
Your inputs

Value out of range

tokens

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

%

Value out of range

GB/s

Value out of range

Results

Total memory needed

20.1 GB

Fits a 24 GB consumer graphics card
Memory for the weights
14.0 GB
Memory for the KV cache
4.29 GB
Upper bound on generation speed (one sequence)
72.0 tokens/s
KV cache per token
0.524 MB

Running a language model on your own GPU starts with one question: will it fit? The LLM memory calculator takes the model size, the precision, the context length, the layers, the hidden size, the KV share, the number of sequences, the overhead and the memory bandwidth, and gives the total memory needed, the memory for the weights and the KV cache, an upper bound on the speed and the cache per token.

How the calculation works

  • Enter the number of parameters in billions and choose the precision of the weights. A model in 16-bit takes 2 bytes per parameter, and a 4-bit quantized one half a byte.
  • Enter the context length, the number of layers, the hidden size and the share of KV heads. They are in the model card, and 1 means full attention.
  • The weights take parameters × bytes. The key-value cache takes 2 × layers × hidden size × KV share × 2 bytes × tokens × sequences, and is kept in 16-bit.
  • The calculator adds an overhead for activations and the runtime, and gives an upper bound on the generation speed, equal to the memory bandwidth divided by the size of the weights, since each token must read all the weights.

Example

A 7-billion-parameter model in 16-bit needs 7 × 2 = 14 GB for the weights. With 32 layers, a hidden size of 4,096 and full attention, each token adds 2 × 32 × 4,096 × 2 bytes = 0.524 MB to the cache, so 8,192 tokens take 4.29 GB. With a 10 % overhead, the total is (14 + 4.29) × 1.1 = 20.1 GB, which fits a 24 GB card. On a GPU with 1,008 GB/s, the speed cannot exceed 1,008 ÷ 14 = 72 tokens per second. In 4-bit, the weights shrink to 3.5 GB.

Why it matters

Memory is the first limit when you run or fine-tune a model locally, and it decides whether a model needs a single graphics card, a pair of cards or a server. Quick arithmetic before downloading tens of gigabytes saves time, and shows the effect of quantization and context length on the budget.

Good to know

The estimate covers inference only. Training and fine-tuning need much more memory for gradients and optimizer states. Real frameworks differ in their overheads, and some use paged caches or other layouts. Treat the result as a first estimate and leave a margin of a few gigabytes. Sizes use GB of 10⁹ bytes, like card makers.

The cloud storage cost calculator prices the files, the home server cost calculator covers a machine at home, the server capacity calculator sizes request loads, and the bitcoin mining profit calculator compares another use of GPUs.

Frequently asked questions

What takes memory when running a language model?

Three things: the weights, which are fixed by the number of parameters and their precision, the key-value cache, which grows with the length of the text and the number of sequences, and the working memory for activations and the runtime. The weights usually dominate for short contexts, and the cache for long ones.

What is quantization?

Quantization stores the weights with fewer bits, such as 8 or 4, instead of 16. It cuts the memory by half or by three quarters and also speeds up generation, since less data is read per token, at a small cost in quality. The cache is often kept in 16-bit, which this calculator assumes.

What is grouped-query attention?

In standard attention, every attention head has its own keys and values. Grouped-query attention shares them among several heads, so the cache is smaller by the ratio of KV heads to attention heads. A model with 8 KV heads and 32 attention heads has a ratio of 0.25 and a cache four times smaller.

How is the speed limit found?

To produce each token, a GPU must read all the weights from memory. The number of tokens per second cannot exceed the memory bandwidth divided by the size of the weights. Real speeds are lower because of compute, overheads and the cache, but the bound shows why bandwidth matters more than raw compute for generation.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.