Skip to content
Omnikalc

Training steps per epoch calculator

Enter the data set size, the batch size per device, the number of devices, the gradient accumulation and the epochs to get the effective batch size, the steps per epoch, the total steps and the training time.

Updated on
Your inputs

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

samples/s

Value out of range

Results

Effective batch size

128

Optimizer steps per epoch
391
Total optimizer steps
35,190
Samples processed in total
4,500,000
Size of the last batch of an epoch
80
Training time
0.83 hours

The training steps per epoch calculator gives the effective batch size, the optimizer steps per epoch, the total steps, the samples processed and the training time of a machine learning run.

How the calculation works

  • Enter the number of training examples and the batch size per device.
  • Enter the number of devices, the gradient accumulation and the epochs.
  • Enter the throughput in samples per second.
  • The calculator gives the effective batch, the steps per epoch, the total steps, the samples, the last batch and the time.

Example

A data set of 50,000 examples with a batch of 128 on one device has an effective batch of 128. One epoch takes ceil(50,000 / 128) = 391 steps, the last batch holding 80 examples. Over 90 epochs, that makes 35,190 steps and 4,500,000 samples. At 1,500 samples per second, training takes 0.83 hours.

Why it matters

The calculator helps students and engineers who schedule training runs, set learning rate schedules and warm-up lengths in steps, or estimate how long and how costly a training job will be.

Good to know

Learning rate schedules and checkpoints are usually defined in steps, so the total steps are the number to give. If the last incomplete batch is dropped, which some data loaders do by default, the steps per epoch are rounded down instead. With several devices, each processes its own share of the batch and the throughput entered should be the total over all devices. Real throughput varies with data loading, communication between devices and the length of the examples, so measure it on a few steps. Larger batches use the hardware better but may need more epochs to reach the same accuracy.

The neural network parameter count calculator sizes the model, the learning curve calculator relates experience and speed, the LLM memory calculator estimates memory and the embedding table calculator sizes vocabularies.

Frequently asked questions

How are the steps per epoch computed?

An epoch is one pass over the training set. The optimizer takes one step per effective batch, which is the batch size per device times the number of devices times the accumulation steps. The steps per epoch are the data set size divided by that, rounded up when the last batch is kept.

What is gradient accumulation?

The gradients of several small batches are added up before the optimizer takes a step. It gives the effect of a large batch when the memory of the device only allows a small one, at the cost of more forward and backward passes per step.

Does the batch size change the learning rate?

Often yes. A common rule scales the learning rate in proportion to the effective batch size, with a warm-up at the start, so that a larger batch does not simply mean fewer, cautious steps. It is a starting point, and very large batches need care.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.