Understanding Backpropagation: Calculating Gradients for Hidden Layer Weights and Biases

Tue, 30 Jun 2026 20:23:01 +0000

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.

In the previous article, we derived formulas for updating the output layer weights w3, w4, and bias b3. Now, we will understand how to calculate the gradients for the hidden layer parameters: w1, b1, w2, and b2.

How are w1, b1, w2, and b2 connected to the prediction?

To find the gradients of the parameters in the hidden layer, we need to trace how changing these values affects the final prediction and the error (SSR).

Let's recall the structure of our neural network:

For the top neuron:

x1 = input * w1 + b1

y1 = f(x1) = log(1 + e^x1) (using the softplus function)

For the bottom neuron:

x2 = input * w2 + b2

y2 = f(x2) = log(1 + e^x2) (using the softplus function)

Finally, the prediction:

Predicted = y1 * w3 + y2 * w4 + b3

And the prediction error:

SSR = Σ (observed − predicted)²

Since w1, b1, w2, and b2 are not directly connected to the output prediction, we must use the chain rule to backpropagate the error from the output layer back to the hidden layer.

Applying the Chain Rule to the Hidden Layer

Let's calculate the gradient for the top neuron's weight w1 first.

A change in w1 affects x1, which affects the output y1, which affects the predicted value, which finally affects the SSR.

So, by the chain rule:

dSSR/dw1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/dw1

Let's calculate each of these values:

1. dSSR/d(predicted)

As we saw in the previous articles, this is the derivative of SSR with respect to the predicted value:

dSSR/d(predicted) = -2 * (Observed - Predicted)

2. d(predicted)/dy1

Since Predicted = y1 * w3 + y2 * w4 + b3, and all other terms are treated as constants w.r.t y1:

d(predicted)/dy1 = w3

3. dy1/dx1

Since y1 = log(1 + e^x1), the derivative of the softplus function is the logistic sigmoid function:

dy1/dx1 = e^x1 / (1 + e^x1)

4. dx1/dw1

Since x1 = input * w1 + b1, differentiating w.r.t w1 gives:

dx1/dw1 = input

Final formula for dSSR/dw1:

Multiplying these parts together, we get:

dSSR/dw1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * input

Deriving the Gradient for Bias b1

Similarly, for the top neuron's bias b1:

dSSR/db1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/db1

The only term that changes here is the last one:

dx1/db1 = 1 (since x1 = input * w1 + b1, derivative w.r.t b1 is 1)

So:

dSSR/db1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * 1

Deriving the Gradients for the Bottom Neuron (w2 and b2)

Following the same logic, we can find the gradients for the bottom neuron's parameters:

For weight w2:

dSSR/dw2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/dw2

dSSR/dw2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * input

For bias b2:

dSSR/db2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/db2

dSSR/db2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * 1

Improving Prediction with self Learning

Once we calculate all these derivatives (dSSR/dw1, dSSR/db1, dSSR/dw2, dSSR/db2), we can update the hidden layer weights and biases using gradient descent:

Step size w1 = derivation w1 * Learning rate
New w1 = old w1 - Step size w1

Step size b1 = derivation b1 * Learning rate
New b1 = old b1 - Step size b1

Step size w2 = derivation w2 * Learning rate
New w2 = old w2 - Step size w2

Step size b2 = derivation b2 * Learning rate
New b2 = old b2 - Step size b2

By doing this repeatedly, the model minimizes the error and converges to the optimal values for all weights and biases.

Conclusion

We have successfully derived the formulas to calculate the gradients for w1, b1, w2, and b2. Combined with the output layer derivations, we now have the math for the entire neural network's backpropagation!

In the next article, we will see how to implement this in code.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

Coder Legion - Recent posts tagged affects