Gradient Descent — ML Technical Interview Questions & Answers

The core optimization algorithm behind most ML training.

Gradient Descent

  • Simple meaning: Training often means “walk downhill on the error hill”. Gradient descent looks at the slope of the loss and takes a small step the other way: weight = weight − learning_rate × slope.
  • Picture: You are in fog on a hill (the loss). Feel the ground: which way is down? Take a small step. Repeat. Too big a step (huge learning rate) — you jump over the valley. Too small — you crawl forever.
  • Batch = use all rows each step (stable, slow). Stochastic = one row (noisy, fast). Mini-batch = a small group (usual in deep learning).
  • Trap: saying gradient descent is the model. It is the training method. Linear regression can also be solved with a formula; GD is how neural nets learn.

Technical Interview Tip: Loss hill. Step opposite the slope. Learning rate matters.

Code Examples for Gradient Descent

Gradient Descent — working example

# One gradient step for y ≈ w*x  (MSE loss)
xs = [1.0, 2.0, 3.0]
ys = [2.0, 4.0, 6.0]
w = 0.0
lr = 0.1
grad = sum(2 * (w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
w = w - lr * grad
print(round(grad, 4), round(w, 4))

Gradient descent updates parameters opposite the loss gradient: w ← w − η∇L.

How to Explain Gradient Descent to an Interviewer

What does gradient descent do?

Reduces loss by stepping opposite the slope

What if learning rate is too high?

Overshoot / diverge