Skip to content
Omnikalc

Embedding table size calculator

Enter the vocabulary size, the embedding dimension, the context length and the precision to get the parameters and the memory of the token and position tables and their share of the model.

Updated on
Your inputs
tokens

Value out of range

Value out of range

Value out of range

million parameters

Value out of range

Results

Parameters of the token table(s)

38,597,376

Memory of the token table(s)
154.4 MB
Parameters of the position table
786,432
Total embedding parameters
39,383,808
Share of the whole model
31.8 %
Embeddings are a minor part of the model
Memory per token (one row)
3,072 bytes

The embedding table size calculator gives the parameters and the memory of the token embedding table of a language model, with the position table and the share of the whole model.

How the calculation works

  • Enter the vocabulary size, the embedding dimension and the context length.
  • Pick the precision and whether the output layer is tied to the table.
  • Enter the total model size.
  • The calculator gives the token table parameters and memory, the position parameters, the total and the share of the model.

Example

A vocabulary of 50,257 tokens with 768 dimensions has 50,257 × 768 = 38,597,376 parameters, or 154.4 MB in 32-bit. A position table for 1,024 positions adds 786,432 parameters. The embeddings total 39,383,808, which is 31.8 % of a 124 million parameter model. Each token costs 3,072 bytes.

Why it matters

The calculator helps machine learning students, engineers who train or compress small language models and anyone who wants to know why vocabulary size matters for model size.

Good to know

In small models the embedding table is a large part of the parameters, which is why techniques such as tying, factorized embeddings or smaller vocabularies are used. Models with rotary positions do not have a learned position table, so set the context length to 0 for them. Multilingual models need big vocabularies to cover scripts, which raises the table to hundreds of millions of parameters. Quantizing the table is possible but it affects rare tokens more. On a device with little memory, the embedding table can sometimes stay on storage and be read row by row, since only a few rows are used per token. The memory shown is for the weights only.

The neural network parameter count calculator handles dense layers, the LLM memory calculator sizes complete models, the type-token ratio calculator measures vocabulary richness and the Shannon entropy calculator measures information in text.

Frequently asked questions

How big is an embedding table?

It has one row per token of the vocabulary and one column per dimension, so vocabulary × dimension parameters. A vocabulary of 50,257 tokens with 768 dimensions gives 38.6 million parameters, about 154 MB in 32-bit.

What does tying the output layer mean?

The matrix that turns the final hidden state into a score for each token has the same shape as the embedding table. Many models reuse the same weights for both, which saves vocabulary × dimension parameters. Choose separate to count both matrices.

Why does the vocabulary size matter?

A larger vocabulary shortens sequences, since common words become a single token, but it makes the table bigger and rare tokens poorly trained. For small models the table can be a third or more of all parameters.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.