Coder Legion - Recent posts tagged head

Attention Is All You Need - Part 5

Tue, 02 Jun 2026 00:46:42 +0000

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on GitHub. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

In previous article we discussed about step 2 of transformer model, i.e. position encoding.

In this article we will discuss step 3 of transformer model, i.e. Multi-Head Attention.

Why Traditional RNN model didn't work for long sentences?

Before 2017, we were using LSTM and RNN models for NLP tasks.

Basicaly as the input of words and processing and context was very less.

For Example let's assume there are 3 words model processes words 1 by 1.

So, first sentence it was taking about river bank.

The river bank.
The United Bank

Next it is about united bank which is has no related data but as we did embeding and positonal encodings we have very low probablity of understanding the context.

Here is the example of how it vector might look like.

How Single Attention Head works?

A single attention head works by determining how much focus a specific token (word) in a sequence should place on other tokens to better understand its own context.

Let's take example: "The cat sat on the mat."

For the token "sat", the attention head might learn to pay high attention to "cat" and "mat" because they are directly related to "sat".

Let's get understanding these in details in next article by actual implementing it.

Reference: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.
⭐ Star it on GitHub: https://github.com/HexmosTech/git-lrc

Attention Is All You Need - Part 6

Mon, 01 Jun 2026 09:29:27 +0000

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

In previous article we discussed about why traditional RNN model didn't work for long sentences.

In this article we will discuss about, how single attention head works?

Why Single Attention Head is not enough?

In single attention head, context of word is maintained and relates word/ token to each other.

But this also comes with a limitation of contextual understanding.

For Example:

If the sentence had multiple meanings, single attention head will focus on one.

Example:

In 5 Coach long train, in first coach the man was sleeping in 2nd coach the man was standing in 3rd coach the man was playing and in 4th coach the man was eating food.

So, for contextual understanding of man in above example needed and that is where single attention head fails.

To Solve this, multi head attention is introduced.

Multi Attention Head

Multi Attention Head works parallelly and independently. So, it captures contextual understanding of word/ token.

What next?

Finaly we got some idea on what end all happening under this.

I just want to hold upcomming exploration.

Just by getting into surface level I learned many things. But detailing i couldn't learn.

I think by 2 weeks I will explore those field and share my learning.

Reference: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.
⭐ Star it on GitHub: https://github.com/HexmosTech/git-lrc