Coder Legion - Recent posts tagged gradient

Understanding Backpropagation: How Neural Networks Learn from Their Mistakes

Sun, 05 Jul 2026 07:23:45 +0000

From Linear Regression to Gradient Descent

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.

In previous parts we discussed about linear regression, gradient descent and how to calculate the optimal slope and intercept using Gradient Descent.

In this article, we'll build an intuition for backpropagation, understand why it is necessary, and explore how the chain rule and gradient descent work together to enable neural network learning.

What is Backpropagation?

Backpropagation is an algorithm used to train neural networks.

It is a way to adjust the weights and biases of a neural network to reduce the error between the predicted output and the actual output.

In simple terms, it is a way for the neural network to learn from its mistakes.

This learning process is powered by an algorithm called backpropagation, one of the most important concepts in machine learning. Backpropagation provides a systematic way for a neural network to determine which internal parameters contributed to an error and how those parameters should be updated to reduce future mistakes.

Imagine teaching a student to solve math problems.

After each attempt, you compare the student's answer with the correct one, identify where mistakes occurred, and provide feedback. Over time, the student adjusts their approach and improves. Backpropagation works in a similar way: the network makes a prediction, calculates the error, traces that error backward through the network, and updates its parameters accordingly.

At its core, backpropagation combines two fundamental mathematical ideas:

The Chain Rule from calculus, which helps determine how changes in one part of the network affect the final error.
Gradient Descent, an optimization technique that uses those calculations to update the network's parameters in the direction that reduces error.

By repeatedly answering these questions across thousands or millions of training examples, neural networks gradually learn patterns hidden within data and become increasingly accurate.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

Understanding Idea behind Full Backpropogation

Thu, 02 Jul 2026 18:14:45 +0000

In the previous article, we learned how to calculate the gradient of the last bias in a neural network.

Now we will explore how gradients flow through the entire network and how to calculate the weights of previous layers.

How to calculate weights

Now we will calculate weights of the previous layer

The challenge is that the loss function does not directly depend on these earlier weights.

For example, consider a weight (w1 and w2 are already calculated ) in a hidden layer.

Changing (w3 and w4):

Changes the hidden neuron output.
Changes the output neuron input.
Changes the final prediction.
Changes the loss.

So there is an indirect relationship between the weight and the loss.

This is exactly why we need the chain rule.

Conclusion

Similar to previous calculation we should also calculate for all weights and biases using chain rule and gradient descent.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

How to calculate weights using gradient descent

Thu, 02 Jul 2026 02:57:30 +0000

In the previous article, I explained the requirements for finding the weights of the last layer. Now let's see how we actually assign and optimize those weights using gradient descent.

Selecting Random Weights From a Normal Distribution

First, we assign random numbers (drawn from a normal distribution) to the weights w3 and w4.

Then we sum up both results along with the bias b3 = 0:

predicted = (output of top neuron × w3) + (output of bottom neuron × w4) + b3

This gives us our initial prediction, and we can plot the final graph:

With these initial random weights, the SSR (Sum of Squared Residuals) is calculated again to measure how far off our predictions are.

Gradient Descent Algorithm For Optimal Values

Now we need to find the derivative of SSR with respect to b3 so we can update it.

Recall our loss function:

SSR = Σ (observed − predicted)²

And our predicted value is:

predicted = (output of top neuron × w3) + (output of bottom neuron × w4) + b3

This is the same chain rule approach we used for backpropagation when optimizing only b3.

The key insight here is that the products of w3 and w4 with their respective neuron outputs are treated as constants for a single gradient calculation with respect to b3. Since only b3 is the variable in this expression, the derivative simplifies cleanly — just as we saw in the previous articles.

Conclusion

By assigning random weights from a normal distribution and then applying gradient descent with the chain rule, we can iteratively optimize each weight and bias in the network. The same process that worked for b3 alone now extends to w3 and w4 — we just need to carefully apply the chain rule at each step to compute the correct gradients.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

From Linear Regression to Gradient Descent

Tue, 30 Jun 2026 20:23:06 +0000

In the previous section, we learned that linear regression finds the best-fitting line by determining the optimal slope and intercept.

In this article, we will discuss how to calculate the optimal slope and intercept using Gradient Descent.

How to calculate the optimal slope and intercept using Gradient Descent

The quality of that line is measured using the Sum of Squared Residuals (SSR), which represents the total prediction error.

SSR = sum( (y_observed - y_predicted)^2 )

The best regression line is simply the line that produces the smallest SSR.

When studying linear regression, it's easy to think that the slope and intercept magically appear from a formula. In reality, they are the values that minimize the prediction error. This is where Gradient Descent comes in.

Instead of calculating the optimal slope and intercept directly using a closed-form equation, Gradient Descent starts with arbitrary values and gradually improves them. After each step, it measures how the SSR changes and adjusts the parameters in the direction that reduces the error.

Step-by-Step Gradient Descent Example

Let's illustrate how Gradient Descent works using the exact same dataset of 4 points from Part 10:

1. The Dataset

Point 1: (1, 2)
Point 2: (2, 3)
Point 3: (3, 5)
Point 4: (4, 4)

2. Simplifying the Problem

To make the math easy to trace, we will hold the Slope (m) constant at its optimal value of 0.8 and focus purely on finding the optimal Intercept (b).

Our prediction equation is:

y_predicted = 0.8 * x + b

We start with an initial guess for the intercept: b = 0.

3. Calculating the Initial SSR (at b = 0)

Let's find the predicted values and calculate the residuals (observed - predicted):

For Point 1 (1, 2):
- y_predicted = 0.8 * 1 + 0 = 0.8
- Residual_1 = 2 - 0.8 = 1.2
For Point 2 (2, 3):
- y_predicted = 0.8 * 2 + 0 = 1.6
- Residual_2 = 3 - 1.6 = 1.4
For Point 3 (3, 5):
- y_predicted = 0.8 * 3 + 0 = 2.4
- Residual_3 = 5 - 2.4 = 2.6
For Point 4 (4, 4):
- y_predicted = 0.8 * 4 + 0 = 3.2
- Residual_4 = 4 - 3.2 = 0.8

Now, sum the squared residuals:

SSR = 1.2^2 + 1.4^2 + 2.6^2 + 0.8^2
    = 1.44 + 1.96 + 6.76 + 0.64
    = 10.8

4. Derivation of the Gradient (d(SSR)/db)

To know which direction to move the intercept b and by how much, we take the derivative of SSR with respect to b:

SSR = sum( (y_observed - (0.8 * x_observed + b))^2 )

Applying the chain rule:

d(SSR)/db = sum( 2 * (y_observed - (0.8 * x_observed + b)) * (-1) )
          = -2 * sum( y_observed - y_predicted )
          = -2 * sum( Residuals )

The gradient is simply -2 times the sum of the residuals.

5. Updating the Intercept

The update rule is:

b_new = b_old - (Learning Rate * Gradient)

Let's choose a Learning Rate (LR) of 0.1.

Step 1:
- Gradient: d(SSR)/db = -2 * (1.2 + 1.4 + 2.6 + 0.8) = -2 * 6.0 = -12.0
- Step Size: Gradient * LR = -12.0 * 0.1 = -1.2
- New Intercept: b_new = 0 - (-1.2) = 1.2
Step 2:
- With b = 1.2, the predictions are closer to the actual values.
- The new residuals are: 0.0, 0.2, 1.4, and -0.4.
- SSR: 0.0^2 + 0.2^2 + 1.4^2 + (-0.4)^2 = 2.16
- Gradient: -2 * (0.0 + 0.2 + 1.4 - 0.4) = -2.4
- Step Size: -2.4 * 0.1 = -0.24
- New Intercept: b_new = 1.2 - (-0.24) = 1.44
Step 3 (Convergence):
- With b = 1.44, the new residuals are: -0.24, -0.04, 1.16, and -0.64.
- Gradient: -2 * (-0.24 - 0.04 + 1.16 - 0.64) = -0.48
- Step Size: -0.48 * 0.1 = -0.048
- New Intercept: b_new = 1.44 - (-0.048) = 1.488
- We repeat this loop. As we approach the optimal intercept, the residuals sum up closer to 0, which shrinks the gradient and steps.
- After several iterations, the gradient becomes 0, and the intercept converges to the exact optimal value of 1.5 (where SSR reaches its minimum value of 1.8).

Conclusion

We started with an arbitrary intercept of 0 and adjusted it step-by-step. Each step was guided by the gradient, which told us exactly how much to change the intercept to reduce the prediction error (SSR). We repeated this process until the error reached its minimum.

While this example focused on a simple linear regression with a single variable, this same principle applies to deep neural networks with millions of parameters. Gradient descent is the engine that drives learning in machine learning.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub