Attention Is All You Need

How Google Accidentally Invented the Transformer

In 2017, a small team at Google Brain working on Google Translate faced a scaling wall. Recurrent Neural Networks (RNNs) and LSTMs — the workhorses of sequence models struggled to handle long sentences and parallel training. Training large translation models took weeks, and inference was painfully sequential.

So the Google Translate team asked themselves a radical question:

What if we throw away recurrence altogether?

That single question led to the Transformer, the architecture that ignited today’s entire LLM revolution. And in a plot twist worthy of Silicon Valley legend, it would soon challenge its own creator. A paper born inside Google's campus, would seed the then no name start up OpenAI, goes on to reshape the global AI landscape. Today, OpenAI poses a formidable challenge to the dominance of Google and Microsoft quite ironically using the very tech and resources that originated from those giants, challenging the lion in its own den.

Core Idea

This paper introduced the Transformer architecture, a radical departure from recurrent and convolutional models that dominated sequence modeling. Instead of processing tokens sequentially, Transformers use self-attention to let every token attend to every other token in parallel. This unlocks massive scalability and enables training on unprecedented volumes of text.

Technical Details

1. Self-Attention Mechanism

Each token in the input can attend to every other token directly, capturing long-range dependencies without relying on sequential processing. This is the core reason Transformers can scale efficiently.

2. Multi-Head Attention

Instead of performing a single attention function, the Transformer uses multiple attention heads in parallel. By projecting inputs into multiple subspaces, the model can learn different relationships simultaneously, allowing richer contextual understanding.

3. Positional Encoding

Since the Transformer contains no recurrence or convolution, positional information must be injected to give the model some sense of the order of the sequence. The paper uses sinusoidal positional encodings.

4. Encoder-Decoder Architecture

The Transformer follows an encoder-decoder structure:

• Encoder: Maps an input sequence to a continuous representation
• Decoder: Generates an output sequence one element at a time

5. Feed-Forward Layers & Residual Connections

Deep stacks of layers become trainable and stable, enabling very deep architectures without vanishing gradients.

6. Fully Parallelizable Architecture

Eliminates sequential bottlenecks of RNNs, allowing distributed training on GPUs and TPUs at unprecedented scale