Coder Legion - Recent posts tagged layer

Understanding Multiple Input and Output Neural Network

Wed, 15 Jul 2026 13:17:23 +0000

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.

In the previous article, we discussed how ReLU activation function works. Now let's see how multiple input and multiple output neural networks work.

How Multiple Input and Output Neural Network works

Until now, we worked with a neural network that had a single input and a single output. In real-world problems, we usually have multiple inputs and multiple outputs.

In this article, we will build a neural network with:

2 inputs (input1 and input2)
1 hidden layer with 2 neurons
3 outputs (output1, output2, output3)
ReLU as the activation function

Network Structure

input1 ──┐
         ├──► hidden_neuron1 ──┬──► output1
input2 ──┤                     ├──► output2
         └──► hidden_neuron2 ──┴──► output3

Each input is connected to each hidden neuron, and each hidden neuron is connected to each output neuron.

Forward Pass Equations

All weights and biases are assigned based on normal distribution.
Hidden Layer Calculation
For hidden neuron 1:

x1 = (input1 * w1) + (input2 * w2) + b1
y1 = ReLU(x1) = max(0, x1)

For hidden neuron 2:

x2 = (input1 * w3) + (input2 * w4) + b2
y2 = ReLU(x2) = max(0, x2)

Output Layer Calculation

For output1:

output1 = (y1 * w5) + (y2 * w6) + b3

For output2:

output2 = (y1 * w7) + (y2 * w8) + b4

For output3:

output3 = (y1 * w9) + (y2 * w10) + b5

Example with Numbers

Let's work through a concrete example.

Given inputs:

input1 = 2
input2 = 3

Assume the following initial weights and biases:

Hidden layer weights:
  w1 = 0.5,  w2 = -0.3,  b1 = 0.1
  w3 = -0.4, w4 = 0.8,   b2 = 0.2

Output layer weights:
  w5 = 0.6,  w6 = 0.7,  b3 = 0.1
  w7 = -0.5, w8 = 0.4,  b4 = 0.2
  w9 = 0.3,  w10 = -0.6, b5 = 0.0

Calculate hidden neuron values

Hidden neuron 1:

x1 = (2 * 0.5) + (3 * -0.3) + 0.1
   = 1.0 - 0.9 + 0.1
   = 0.2

y1 = ReLU(0.2) = max(0, 0.2) = 0.2

Hidden neuron 2:

x2 = (2 * -0.4) + (3 * 0.8) + 0.2
   = -0.8 + 2.4 + 0.2
   = 1.8

y2 = ReLU(1.8) = max(0, 1.8) = 1.8

Calculate outputs

Output 1:

output1 = (0.2 * 0.6) + (1.8 * 0.7) + 0.1
        = 0.12 + 1.26 + 0.1
        = 1.48

Output 2:

output2 = (0.2 * -0.5) + (1.8 * 0.4) + 0.2
        = -0.10 + 0.72 + 0.2
        = 0.82

Output 3:

output3 = (0.2 * 0.3) + (1.8 * -0.6) + 0.0
        = 0.06 - 1.08 + 0.0
        = -1.02

Final Results

output1 = 1.48
output2 = 0.82
output3 = -1.02

Why ReLU works here

Notice that x1 = 0.2 and x2 = 1.8 — both are positive, so ReLU passes them through unchanged.

If any x value were negative (say x1 = -0.5), then:

y1 = ReLU(-0.5) = max(0, -0.5) = 0

That neuron would contribute nothing to the outputs, effectively "turning off" and making the network sparse and efficient.

Matrix Representation

You can think of all the weights as a matrix of connections:

Hidden Layer (2x2 weight matrix + 2 biases):

  w1   w2   b1        w3   w4   b2
[ 0.5  -0.3  0.1 ]  [ -0.4  0.8  0.2 ]


Output Layer (3x2 weight matrix + 3 biases):

  w5   w6   b3
[ 0.6  0.7  0.1 ]   → output1

  w7   w8   b4
[-0.5  0.4  0.2 ]   → output2

  w9  w10   b5
[ 0.3 -0.6  0.0 ]   → output3

Each output neuron learns a different combination of the hidden neuron outputs, allowing the network to produce multiple independent predictions.

Conclusion

We now understand how a neural network with 2 inputs and 3 outputs works step by step using the ReLU activation function:

Each hidden neuron receives all inputs, computes a weighted sum plus bias, and applies ReLU.
Each output neuron receives all hidden neuron outputs and computes its own weighted sum plus bias.
The network can produce multiple distinct outputs simultaneously from the same inputs.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

Why Do Neural Networks Need the Chain Rule? How do we apply it?

Fri, 03 Jul 2026 09:32:00 +0000

In the previous article, we introduced backpropagation and learned that neural networks improve by reducing prediction errors.

We also saw that backpropagation relies on two fundamental ideas:

The Chain Rule
Gradient Descent

But we haven't yet answered an important question:

How does we calculate wieghts and biases to decrease the error?

To answer that, let's look at a very small neural network.

A Simple Neural Network

Imagine a neural network with:
Similar to the previous example.

One input neuron
Two hidden neurons
One output neuron

Calculating Last Bias In the last layer

Let's asssume we have wieght and bias of all hidden layer and we only want to find last bias b3

Now from gradient descent, we can update the last bias b3 using the partial derivative of loss with respect to b3

The Error rate is done with Residuals.
Residual = Observed - Predicted

SSR = Sum of (Observed - Predicted)^2

So, We take 3 samples for training

Starting, Ending and middle values.

Finaly By calculating SSR.

Use of Chain Rule

We actually calculated b3 only using gradient descent.

Now Using chain Value generated from the weight and bias of previous layers

Predicted = Top Layer + Bottom Layer + Bias (b3)

Using Chain Rule we can write Dirivative of SSR with

dssr/db3 = dssr/dpredicted * dpredicted/db3

dssr/dpredicted = (Observed - Predicted)^2

As predicted, it is not constant and we are dirving it.

dssr/dpredicted = 2*(Observed - Predicted)*(d(Observed - Predicted))/dpredicted)

dssr/dpredicted = 2*(Observed - Predicted)(-1)
dssr/dpredicted = -2(Observed - Predicted)

For dpredicted/db3

dpredicted = Top Layer + Bottom Layer + Bias (b3)
Both Top Layer and Bottom Layer is constant for this calculation
dpredicted/db3 = 1

Finaly dssr/db3 = -2*(Observed - Predicted) * 1

Slop Calculation and Learning

Now we have 3 values of predicted for 3 samples

dssr/db3 = Σ(-2*(Observed-Predicted))

dssr/db3 = -2 * [(Observed1 - Predicted1) * 1 + (Observed2 - Predicted2) * 1 + (Observed3 - Predicted3) * 1]

dssr/db3 = -2 * [(Residual1) + (Residual2) + (Residual3)]

dssr/db3 = -2 * (ResidualSum)

For our training data I got slope = -15.7

step size = slope x learning rate

step size = -15.7 x 0.1 = -1.57

new b3 = old b3 + step size

new b3 = 0 + (-1.57) = -1.57

Then again, recalculating SSR with new b3 we got slop.

slop = -6.26

step size = -6.26 x 0.1 = -0.626

new b3 = -1.57 + (-0.626) = -2.196

Similarly after calculatinng multiple times utile we get step size close to 0.

Final Result
We found the optimal
b3 = 2.21

Conclusion

We could able to apply these chain rule, gradient descent and backpropagation in a very small neural network.

In next article we will discuss how to calculate wieghts and biases in same neural network.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub