Gradient Descent

Gradient descent is the core method for training most machine learning models. It computes the gradient of the loss function, the direction in which the loss increases fastest, and moves the parameters a small step in the opposite direction, repeating until the loss stops improving.

Why it matters

Neural networks have millions or billions of parameters, and gradient descent is how they are tuned. The step size is the learning rate: too large and training overshoots or diverges, too small and it crawls or stalls. Variants such as stochastic gradient descent, momentum and Adam compute gradients on mini-batches and adapt the step, making training practical on large datasets.

Example

loss = (w - 3)^2        gradient = 2(w - 3)
w = 0, learning_rate = 0.25

step 1: w = 0 - 0.25 * (-6)   = 1.5
step 2: w = 1.5 - 0.25 * (-3) = 2.25
step 3: w = 2.25 - 0.25*(-1.5) = 2.625   -> approaching 3

Common mistakes

• Using one learning rate without checking the loss curve.
• Assuming it finds the global minimum. On complex models it may settle in a local minimum or a flat region.
• Forgetting to scale inputs, which slows convergence.

Related

• Loss function
• Backpropagation
• Epoch
• Neural network

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.