Reading Google's "Attention Is All You Need" Paper

Today I sat down to actually understand what's happening when I type something into ChatGPT and it talks back. I'd used LLMs from over year without really getting them and that bugged me enough to dig in properly. This Blog is basically my brain dump after going through it piece by piece tokenization, embeddings, positional encoding, attention all of it in the order it finally clicked for me not necessarily the "correct" academic order.

If you've ever wondered what's actually happening between "you type a sentence" and "the model replies," this is for you.

Lets start with basics: LLM, GPT and ChatGPT are not the same thing

This tripped me up early on so let's get it out of the way.

• LLM = Large Language Model. The general category of all.

• GPT = Generative Pre-trained Transformer. This is the actual engine/architecture doing the work.

• ChatGPT/Gemini = an application built on top of GPT models the interface you interact with.

So when you chat with any AI model you're really talking to a GPT model wrapped in a product. The simplest way I can describe what's happening under the hood:

[Your Text] ->[Tokenizer] -> [GPT/LLM] -> [Next Token Prediction]

GPT's entire job at its core is next Token prediction. It looks at everything that came before and calculates which token is most likely to come next.
Type "Write a story about me" and here's roughly the pipeline:

1. Your text gets broken into tokens

2. Tokens get converted into numbers

3. The transformer processes those numbers

4. The model spits out a probability distribution over what comes next

5. You get your output, one token at a time

That's it. There's no understanding in the human sense it's pattern prediction at a scale large enough that it feels like understanding.

Why we even need tokenization

Computers don't understand natural language(English,Hindi etc). They only understand numbers. So before any processing happens we have to convert the text we send into something numeric. That conversion step is tokenization.

In GPT there's a concept called a vocabulary basically a dictionary of every token the model knows how to read or generate (things like "the", "cat", "ing" and so on).

I didn't want to just trust this conceptually so I tried it myself using tiktoken js npm package (the actual tokenizer library OpenAI uses for GPT models):

import { get_encoding } from "tiktoken";

const MyEncoder = get_encoding("gpt2");

const EncodedTokens = MyEncoder.encode("Hey, this is a quick test");
console.log(EncodedTokens);

const DecodedTokens = MyEncoder.decode(EncodedTokens);
console.log(new TextDecoder().decode(DecodedTokens));

Running this "Hey, this is a quick test" turns into a list of integers and decoding gives you the original sentence right back.
That round trip is the whole point: tokenization just needs to be reversible
and consistent.

While I was at it I also built a much simpler word level tokenizer myself (not the subword/BPE approach the real models use but easier to actually see what's happening):

• Go through all your training text

• Pull out every unique word

• Sort them

• Assign each one an ID number

• Reserve the first few IDs for special tokens: PAD (padding shorter sequences), UNK (unknown words), BOS/EOS (start and end markers)

The part that actually made me stop and think was UNK. If a word like "dog" never showed up in your training vocabulary, it doesn't get its own ID — it just gets dumped into the generic UNK slot. So the sentence "the dog sat" effectively becomes "the [unknown] sat" to the model, and the model has no idea anything is even missing. It just silently loses that information.

This is exactly why production tokenizers use subword splitting (BPE) instead of whole words it drastically cuts down on how often you hit unknown tokens, since a word can be broken into smaller known chunks even if the whole word wasn't seen before.

Embeddings : Giving numbers actual meaning

Okay so now every word is just an integer. But here's the problem token ID 4 and token ID 9 tell you nothing about how those two words relate to each other. They're arbitrary labels.

This is where embeddings come in. Embeddings map tokens to vectors that capture real world meaning essentially placing tokens in a high dimensional space where similar concepts sit closer together.

The example that made this click : imagine "pizza" and "burger" sitting in this space. Both are food items so they'd land reasonably close to each other geometrically.
Meanwhile "Italy" and "India" would sit near their respective foods too because they share a similar kind of relationship a country and a dish associated with it.

Under the hood this is implemented as a big lookup matrix sized (vocab_size × d_model)one row per word in your vocabulary and d_model defines how wide each token's vector is (512 in the original Transformer paper).
Token ID 4 comes in the model literally just grabs row 4 from that matrix. During training these rows shift around so that tokens used in similar contexts end up near each other in this space.

One detail I almost glossed over: right before adding positional information (next section), each embedding gets multiplied by the square root of d_model. It's not explained in depth anywhere, but the way I understand it it's purely about scale. If the embedding values and the positional values you're about to add are on wildly different numeric scales, one ends up drowning out the other. This multiplication keeps both in a comparable range.

Positional Encoding : Teaching the model about order

Here's the gap that becomes obvious once embeddings are sorted: The transformer processes the entire sequence at once in parallel. It has zero built in sense of order.

Take these two sentences:

The cat chased the mouse
The mouse chased the cat

Same exact set of tokens. Completely different meaning. Without some way to encode position the model would treat these as identical because right now it's just looking at a bag of word vectors with no sequence information attached.

So positional encoding injects extra information about where each token sits in the sequence added directly onto that token's embedding. The original paper does this with sine and cosine waves even dimensions get sine odd dimensions get cosine each at a different frequency.
Early dimensions oscillate fast and pick up fine grained differences between nearby positions later dimensions move slowly and capture broader structure across the sequence. Stack enough of these frequencies and every position ends up with its own unique "fingerprint."

One practical gotcha if you ever implement this yourself: don't compute 10000^(2i/d_model) directly it explodes for larger dimension indices.
Do the computation in log space instead you get the same numbers without the overflow. I learned this one the hard way by not doing it that way first.

After this step:

Final Input = Token Embeddings + Positional Encoding

Now every token knows both what it is and where it sits. It still doesn't know anything about its neighbors though each token vector is sitting alone in its own lane. That gap is exactly what attention exists to solve.

Self-Attention : Letting tokens actually talk to each other

This is the part where things stopped feeling like setup and started feeling like the real mechanism.

Take a classic ambiguous word: "match." Compare:

Cricket match Match the colors

"Match" sits in roughly the same kind of position relative to the other word in both phrases but it means something completely different in each a competition versus a verb about pairing things up. Self-attention is what lets a token look at the other tokens around it and figure out which ones are actually relevant to its meaning in this specific context it lets tokens talk to each other and decide who matters most.

Mechanically, every token gets transformed into three vectors:

• Query (Q) — what this token is looking for

• Key (K) — what every token is advertising about itself

• Value (V) — the actual information handed over if a token gets selected

The process take the dot product of every Query against every Key to get a grid of raw scores, run soft-max across that grid so the scores become weights that sum to 1 then use those weights to compute a weighted sum over the Value vectors. That weighted sum is the attention output for that token.

There's a scaling step in there too dividing by the square root of d_k (the dimensionality of the Q/K vectors) before the softmax. Without it, as d_k grows the dot products get large and large values pushed through softmax cause it to saturate nearly all the weight piles onto a single token and gradients basically flatten out, which kills training. The square root scaling keeps the variance stable regardless of how big the vectors are.

Multi-Head Attention : Asking more than one question at once

A single attention pass can only really capture one kind of relationship between tokens at a time and that's rarely enough. Take a word like "ate" in a sentence to represent it well you'd want to simultaneously know who is eating what's being eaten and the surrounding context. One head can't hold all of that at once.

So instead, multiple attention "heads" run in parallel each with its own learned projections of Q, K, and V, each focusing on a different slice of the dimension space. Their outputs get concatenated back together and passed through one final linear layer that merges everything into a single combined representation. Nobody explicitly tells each head what to specialize in they just naturally drift toward different patterns during training.

Putting numbers into probabilities

After attention does its thing GPT assigns each candidate next token a raw score. These scores on their own aren't very usable, so they get passed through a softmax function which converts them into a proper probability distribution every token gets some probability all of them summing to 1.

This is also where temperature comes in it controls how much randomness you want in the output. Lower temperature makes the model lean hard toward the most likely token (more deterministic, more repetitive) higher temperature flattens the distribution out and lets less likely tokens have a real shot at being picked (more varied, sometimes less coherent).

And during training the model needs a way to measure how wrong its predictions were compared to the actual next token. That's what cross entropy loss does it's a way to quantify the gap between the model's predicted distribution and the correct answer and that gap is what gets used to adjust the model's weights.

The Feed Forward Network : Where actual transformation happens

Here's something that took me a while to fully appreciate: attention no matter how many heads you run is fundamentally linear.
Every output is just a weighted sum of value vectors. Weighted sums alone can't capture non linear relationships so up to this point the model has been moving information around really well but transforming none of it.

That's the job of the feed forward network (FFN) which runs after every attention layer. Each token's representation gets pushed through a small neural network independently with no interaction between tokens at this stage (mixing tokens was attention's job, already done).

It's just two linear layers with a ReLU activation in between:

1. Expand the token's vector from d_model up to a wider d_ff (commonly 4x wider)

2. Apply ReLU which zeroes out anything negative

3. Compress back down to d_model

The ReLU is the entire reason this block matters. Without it those two linear layers would mathematically collapse into a single linear operation meaning the whole block would add nothing.
That one non-linear step is what actually lets the network reshape features instead of just shuffling them around.

Layer Normalization : Keeping things stable as you stack layers

Here's something I didn't appreciate until I tried stacking multiple blocks together the deeper a network gets, the more unstable the activations can become with each layer inheriting whatever mess the previous one left behind. Layer normalization is what keeps things in a sane range so each layer starts from a clean baseline instead of compounding chaos from the one before it.

The mechanics are straightforward. For each token you calculate the mean and variance across its own features, subtract the mean divide by the standard deviation and then scale and shift the result using two learnable parameters (commonly called gamma and beta). The important part is that each token only ever looks at its own features for this it doesn't need any information from other tokens in the batch or sequence.

That's actually the key difference from batch normalization. Batch norm needs statistics computed across an entire batch which breaks down the moment you're generating tokens one at a time or working with sequences of different lengths. Layer norm sidesteps that completely since it only cares about the current token.

One small detail I liked gamma and beta start out as 1 and 0 respectively so at the very beginning the layer is just doing plain normalization and nothing more. As training progresses the model learns to adjust these values and if normalizing turns out to be hurting performance somewhere the model can effectively learn to dial that back. It has the freedom to undo itself if needed.

At this point, every individual piece exists on its own: tokenizer, embeddings, positional encoding, self-attention, multi-head attention, the feed-forward network, and now layer normalization. The only thing left isn't a new concept it's just wiring these pieces together in the right sequence.

The Encoder Block : Wiring it all together

The encoder block sounds intimidating until you realize it's just the pieces above connected in a specific order.

Here's the flow : run multi head self-attention on the input add the original input back on top of the result, then normalize. Then take that output, push it through the feed-forward network add the input to that sublayer back on top again and normalize once more. Two sublayers total each one wrapped in a residual connection plus a layer norm. That's the entire block.

The residual connections deserve a closer look. Instead of letting each sublayer's output simply replace what came before you add the original input back on top before moving forward.
This gives gradients a shorter, more direct path back to earlier layers during training rather than forcing them through every single matrix multiplication in the stack. Skip this step, and a deep stack of these blocks runs straight into the vanishing gradient problem early layers barely update at all and the whole network stops learning effectively.

The nice part is that since every genuinely hard concept was already solved in the earlier pieces the encoder block itself ends up being a small amount of code it's really just calling the pieces you already built in order with the residual additions in between.

Seeing it actually run

Once you understand the pipeline conceptually actually calling a model is almost anticlimactic. Here's a basic example of sending a message to a GPT model via the API:

import { OpenAI } from "openai";

const aiClient = new OpenAI({
  apiKey: "YOUR_API_KEY",
});

aiClient.chat.completions
  .create({
    model: "gpt-4",
    messages: [{ role: "user", content: "Hello, how are you?" }],
  })
  .then((apiResponse) => {
    console.log(apiResponse.choices[0].message.content);
  });

Behind that one API call everything above is happening : your message gets tokenized, converted to embeddings, combined with positional encoding passed through layers of self-attention and feed-forward networks and the model predicts its response one token at a time using the probability distribution from softmax.

Tying it all together

If I had to compress the whole pipeline into one line it'd be this:

Text → Tokens → Token IDs → Embeddings (Positional Encoding) → Self-Attention (multi-head) → Add & Normalize → Feed-Forward Network → Add & Normalize → Probability Distribution → Next Token → Repeat

That "Add & Normalize" step happening twice per block is the residual connection plus layer norm pattern and stacking several of these blocks back to back is essentially what an encoder is.

None of these pieces are individually that complicated tokenization is just a lookup embeddings are just a matrix attention is a handful of matrix multiplications the FFN is two linear layers with an activation function squeezed in between. What makes it feel intimidating is seeing all of it stacked together for the first time without knowing how the pieces connect.

Building small pieces of this myself instead of just reading about it was honestly what made it stick. If you're trying to actually understand transformers rather than just use them I'd genuinely recommend doing the same pick one piece build a tiny version of it and only then move to the next.