Batch Size

Batch size is a training setting that defines how many samples are fed through a model before the optimiser computes gradients and updates the parameters. Using the whole dataset at once is batch gradient descent, a single sample is stochastic gradient descent, and anything between is mini-batch training, which is the norm.

Why it matters

Larger batches use hardware such as GPUs more efficiently and give smoother gradient estimates, but they need more memory and can generalise slightly worse or need a retuned learning rate. Smaller batches add noise, which can help escape poor solutions, and fit into limited memory but run slower per epoch. Typical values are powers of two such as 16, 32 or 256, though the best choice depends on the model and hardware.

Example

dataset = 50,000 images
batch_size = 250  -> 200 updates per epoch
batch_size = 50   -> 1,000 updates per epoch

When memory is limited, gradient accumulation sums gradients over several small batches to mimic a bigger one.

Common mistakes

• Increasing batch size without adjusting the learning rate.
• Ignoring out-of-memory errors that could be solved by accumulation.
• Confusing this training batch size with the batching of requests at inference time.

Related

• Epoch
• Gradient descent
• Loss function
• Neural network

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.