The embedding table size calculator gives the parameters and the memory of the token embedding table of a language model, with the position table and the share of the whole model.
How the calculation works
- Enter the vocabulary size, the embedding dimension and the context length.
- Pick the precision and whether the output layer is tied to the table.
- Enter the total model size.
- The calculator gives the token table parameters and memory, the position parameters, the total and the share of the model.
Example
A vocabulary of 50,257 tokens with 768 dimensions has 50,257 × 768 = 38,597,376 parameters, or 154.4 MB in 32-bit. A position table for 1,024 positions adds 786,432 parameters. The embeddings total 39,383,808, which is 31.8 % of a 124 million parameter model. Each token costs 3,072 bytes.
Why it matters
The calculator helps machine learning students, engineers who train or compress small language models and anyone who wants to know why vocabulary size matters for model size.
Good to know
In small models the embedding table is a large part of the parameters, which is why techniques such as tying, factorized embeddings or smaller vocabularies are used. Models with rotary positions do not have a learned position table, so set the context length to 0 for them. Multilingual models need big vocabularies to cover scripts, which raises the table to hundreds of millions of parameters. Quantizing the table is possible but it affects rare tokens more. On a device with little memory, the embedding table can sometimes stay on storage and be read row by row, since only a few rows are used per token. The memory shown is for the weights only.
Related tools
The neural network parameter count calculator handles dense layers, the LLM memory calculator sizes complete models, the type-token ratio calculator measures vocabulary richness and the Shannon entropy calculator measures information in text.