Transformers

Overview

The transformer architecture, introduced by Vaswani et al. in 2017 (Vaswani et al. 2017), revolutionised natural language processing by eliminating the sequential processing constraints of recurrent networks. This architecture enables fully parallel computation and superior handling of long-distance dependencies through its self-attention mechanism, becoming the foundation for all subsequent large language models.

Key Components

Input Processing and Embeddings

Transformers process sequences of tokens, converting words into numerical representations through encoding and embedding. Words are first encoded into one-hot vectors, then transformed into dense embeddings that capture semantic information. For GPT-3, the vocabulary contains 50,257 words, which are embedded into 12,288-dimensional vectors.

corpus = """the sun rose over the mountain
the rain fell on the green valley
the wind blew through the tall trees
a cold wind came from the north
the bright sun warmed the cold earth
a heavy rain fell on the dry land
the wind blew and the rain fell"""

token_list = corpus.split()
vocabulary = sorted(set(token_list))
token_to_idx = {t: i for i, t in enumerate(vocabulary)}
idx_to_token = {i: t for i, t in enumerate(vocabulary)}

print(f"Corpus: {len(token_list)} words")
print(f"Vocabulary: {len(vocabulary)} unique tokens")
print(f"Tokens: {vocabulary}")
print(f"\nSample mapping: 'sun' → {token_to_idx['sun']}, 'rain' → {token_to_idx['rain']}")

import numpy as np
import matplotlib.pyplot as plt

bigram_counts = np.zeros((len(vocabulary), len(vocabulary)))
for line in corpus.strip().split('\n'):
    tokens = line.split()
    for i in range(len(tokens) - 1):
        bigram_counts[token_to_idx[tokens[i]], token_to_idx[tokens[i+1]]] += 1

bigram_probs = bigram_counts / bigram_counts.sum(axis=1, keepdims=True)

A simple word-level tokenizer converts text to integer IDs and back:

class Tokenizer:
    """Word-level tokenizer with encode/decode methods."""

    def __init__(self, text_corpus):
        self.token_to_idx, self.idx_to_token = self._build_vocab(text_corpus)
        self.vocab_size = len(self.token_to_idx)

    def _build_vocab(self, text):
        tokens = text.split()
        unique_tokens = sorted(set(tokens))
        to_idx = {t: i for i, t in enumerate(unique_tokens)}
        to_token = {i: t for t, i in to_idx.items()}
        return to_idx, to_token

    def encode(self, text):
        """Convert text to token IDs."""
        return [self.token_to_idx[t] for t in text.split() if t in self.token_to_idx]

    def decode(self, ids):
        """Convert token IDs back to text."""
        return " ".join(self.idx_to_token[i] for i in ids)

tokenizer = Tokenizer(corpus)
print(f"Vocabulary size: {tokenizer.vocab_size}")

sample = "the rain fell on the valley"
encoded = tokenizer.encode(sample)
decoded = tokenizer.decode(encoded)
print(f"\nEncode: '{sample}'")
print(f"  → IDs: {encoded}")
print(f"Decode: {encoded}")
print(f"  → '{decoded}'")
print(f"\nRoundtrip OK: {sample == decoded}")

The embedding layer converts token IDs into dense vectors of fixed dimensionality:

import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)

class TokenEmbedding:
    """Maps token IDs to dense vectors of fixed dimensionality."""

    def __init__(self, vocab_size, embed_dim):
        self.weights = np.random.randn(vocab_size, embed_dim) * 0.1
        self.embed_dim = embed_dim

    def forward(self, token_ids):
        """Look up embeddings for a list of token IDs."""
        return self.weights[token_ids]

    def __call__(self, token_ids):
        return self.forward(token_ids)

vocab_size = len(vocabulary)
embed_dim = 8
embedding_layer = TokenEmbedding(vocab_size, embed_dim)

sequence = ["the", "rain", "fell", "on", "the", "valley"]
token_ids = [token_to_idx[t] for t in sequence]
vectors = embedding_layer.forward(np.array(token_ids))

print(f"Input: {len(token_ids)} token IDs")
print(f"Output: {vectors.shape} (seq_len × embed_dim)")
print(f"\nSame word, same vector:")
print(f"  'the' at pos 0: {vectors[0][:4].round(3)}...")
print(f"  'the' at pos 4: {vectors[4][:4].round(3)}...")
print(f"  Match: {np.allclose(vectors[0], vectors[4])}")

Positional Encoding

Since transformers process all positions in parallel, they require explicit positional information to understand word order. Positional encoding adds this information to the embeddings, using trigonometric functions to create position-dependent encodings that can generalise to sequences longer than those seen during training (Vaswani et al. 2017). The input matrix combines word embeddings and positional encodings: $X = X_{\text{WordEmbedding}} + X_{\text{PositionalEncoding}}$.

Self-Attention

Self-attention is the core innovation of transformers, allowing the model to weigh the importance of different words when processing each position. The mechanism uses three matrices—Query (Q), Key (K), and Value (V)—derived from the input through learned projections. Attention scores are computed as:

0

This enables the model to focus on relevant parts of the input, similar to how humans concentrate on significant aspects of complex information.

import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)

class AttentionHead:
    """Single head of scaled dot-product attention."""

    def __init__(self, input_dim, head_dim):
        self.head_dim = head_dim
        self.W_q = np.random.randn(input_dim, head_dim) * 0.1
        self.W_k = np.random.randn(input_dim, head_dim) * 0.1
        self.W_v = np.random.randn(input_dim, head_dim) * 0.1

    def forward(self, X):
        """Compute attention: X → (output, weights)."""
        Q = X @ self.W_q
        K = X @ self.W_k
        V = X @ self.W_v

        scores = Q @ K.T / np.sqrt(self.head_dim)
        exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
        weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)

        output = weights @ V
        return output, weights

input_dim, head_dim = 8, 4
X = np.random.randn(4, input_dim)
tokens = ["the", "rain", "fell", "softly"]

attention = AttentionHead(input_dim, head_dim)
output, weights = attention.forward(X)

print(f"Input:  {X.shape}")
print(f"Output: {output.shape}")
print(f"\nAttention weights (who attends to whom):")
for i, tok in enumerate(tokens):
    top_j = np.argmax(weights[i])
    print(f"  {tok:8s} → {tokens[top_j]:8s} ({weights[i].round(2)})")

Multi-Head Attention

Multi-head attention runs multiple attention operations in parallel, each with different learned projections. This allows the model to capture various types of relationships simultaneously—syntactic, semantic, and long-range dependencies. GPT-3, for example, uses 96 attention heads, with each head focusing on different aspects of the sequence.

import numpy as np
np.random.seed(42)

class MultiHeadAttention:
    """Parallel attention heads with output projection."""

    def __init__(self, model_dim, num_heads):
        self.num_heads = num_heads
        self.head_dim = model_dim // num_heads
        self.heads = []
        for _ in range(num_heads):
            head = {
                'W_q': np.random.randn(model_dim, self.head_dim) * 0.1,
                'W_k': np.random.randn(model_dim, self.head_dim) * 0.1,
                'W_v': np.random.randn(model_dim, self.head_dim) * 0.1,
            }
            self.heads.append(head)
        self.W_out = np.random.randn(model_dim, model_dim) * 0.1

    def _scaled_dot_product(self, Q, K, V):
        """Compute attention for a single head."""
        d_k = Q.shape[-1]
        scores = Q @ K.T / np.sqrt(d_k)
        exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
        weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)
        return weights @ V

    def forward(self, X):
        """Run all heads in parallel, concatenate, project."""
        head_outputs = []
        for head in self.heads:
            Q = X @ head['W_q']
            K = X @ head['W_k']
            V = X @ head['W_v']
            head_outputs.append(self._scaled_dot_product(Q, K, V))
        concatenated = np.concatenate(head_outputs, axis=-1)
        output = concatenated @ self.W_out
        return output

model_dim, num_heads = 8, 2
X = np.random.randn(4, model_dim)

mha = MultiHeadAttention(model_dim, num_heads)
output = mha.forward(X)

print(f"Input:       {X.shape}")
print(f"Heads:       {num_heads} (each {model_dim // num_heads} dims)")
print(f"Concatenated: ({X.shape[0]}, {model_dim})")
print(f"Output:      {output.shape}")
print(f"\nOutput preserved model_dim={model_dim} - ready to stack!")

Masked Self-Attention

For autoregressive models like GPT, masked self-attention prevents access to future words during training. This maintains the autoregressive property where each word is predicted based only on previous words, while still allowing parallel processing. This differs from bidirectional models like BERT (Devlin et al. 2018), which can attend to all positions in both directions.

Feedforward Networks and Normalisation

Each transformer layer includes a feedforward neural network that non-linearly transforms attention outputs, enhancing the model’s expressive capabilities. Residual connections and layer normalisation (Ba et al. 2016) stabilise training and enable deep stacking—GPT-3 uses 96 layers—by alleviating vanishing gradients and normalising layer inputs.

import numpy as np
np.random.seed(42)

def layer_normalisation(x, epsilon=1e-5):
    """Apply layer normalisation across the last dimension."""
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + epsilon)

class TransformerLayer:
    """Single transformer layer: attention + feedforward with residuals."""

    def __init__(self, model_dim, num_heads, ff_dim):
        self.model_dim = model_dim
        d_head = model_dim // num_heads

        self.W_q = np.random.randn(model_dim, model_dim) * 0.1
        self.W_k = np.random.randn(model_dim, model_dim) * 0.1
        self.W_v = np.random.randn(model_dim, model_dim) * 0.1

        self.ff_up = np.random.randn(model_dim, ff_dim) * 0.1
        self.ff_bias = np.zeros(ff_dim)
        self.ff_down = np.random.randn(ff_dim, model_dim) * 0.1
        self.out_bias = np.zeros(model_dim)

    def _self_attention(self, X):
        Q, K, V = X @ self.W_q, X @ self.W_k, X @ self.W_v
        scores = Q @ K.T / np.sqrt(self.model_dim)
        exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
        weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)
        return weights @ V

    def _feedforward(self, X):
        hidden = np.maximum(0, X @ self.ff_up + self.ff_bias)
        return hidden @ self.ff_down + self.out_bias

    def forward(self, X):
        """Attention → residual → FFN → residual."""
        normed = layer_normalisation(X)
        X = X + self._self_attention(normed)
        normed = layer_normalisation(X)
        X = X + self._feedforward(normed)
        return X

model_dim, num_heads, ff_dim = 8, 2, 16
layers = [TransformerLayer(model_dim, num_heads, ff_dim) for _ in range(3)]

X = np.random.randn(4, model_dim)
print(f"Input: {X.shape}")

for i, layer in enumerate(layers):
    X = layer.forward(X)
    avg_norm = np.linalg.norm(X, axis=-1).mean()
    print(f"After layer {i+1}: {X.shape}, avg norm: {avg_norm:.3f}")

print(f"\nFinal output: {X.shape}")
print("Shape preserved through all layers!")

Text Generation

Autoregressive generation predicts one token at a time, feeding each output back as input for the next step.

import numpy as np
np.random.seed(42)

vocab = ["the", "sun", "rose", "over", "mountain", "rain", "fell", "valley", "wind", "trees"]
vocab_size = len(vocab)
token_to_idx = {w: i for i, w in enumerate(vocab)}
idx_to_token = {i: w for i, w in enumerate(vocab)}

bigram_counts = np.zeros((vocab_size, vocab_size))
training_corpus = """the sun rose over the mountain
the rain fell on the green valley
the wind blew through the tall trees
a cold wind came from the north
the bright sun warmed the cold earth
a heavy rain fell on the dry land
the wind blew and the rain fell"""

for line in training_corpus.strip().split('\n'):
    tokens = line.split()
    for i in range(len(tokens) - 1):
        if tokens[i] in token_to_idx and tokens[i+1] in token_to_idx:
            bigram_counts[token_to_idx[tokens[i]], token_to_idx[tokens[i+1]]] += 1

bigram_probs = bigram_counts / bigram_counts.sum(axis=1, keepdims=True)
log_probs = np.log(bigram_probs + 1e-10)

def compute_logits(token_ids):
    """Return log-probabilities for next token based on bigram model."""
    return log_probs[token_ids[-1]]

def softmax(logits):
    exp_x = np.exp(logits - logits.max())
    return exp_x / exp_x.sum()

def greedy_generate(prompt_ids, num_tokens=5):
    """Autoregressive generation with greedy decoding."""
    ids = list(prompt_ids)
    for _ in range(num_tokens):
        logits = compute_logits(np.array(ids))
        probabilities = softmax(logits)
        next_id = np.argmax(probabilities)
        ids.append(next_id)
    return ids

prompt = [token_to_idx["the"], token_to_idx["sun"]]
result = greedy_generate(prompt, num_tokens=5)

prompt_text = " ".join(idx_to_token[i] for i in prompt)
generated_text = " ".join(idx_to_token[i] for i in result)
print(f"Prompt:    '{prompt_text}'")
print(f"Generated: '{generated_text}'")
print(f"\nToken IDs: {result}")

Temperature and nucleus (top-p) sampling control the randomness of generation:

import matplotlib.pyplot as plt
np.random.seed(42)

def temperature_sample(logits, temperature=1.0, top_p=0.9):
    """Sample with temperature scaling and nucleus filtering."""
    if temperature == 0:
        return np.argmax(logits)

    scaled = logits / temperature
    probs = np.exp(scaled - scaled.max())
    probs = probs / probs.sum()

    sorted_idx = np.argsort(-probs)
    sorted_probs = probs[sorted_idx]
    cumsum = np.cumsum(sorted_probs)
    cutoff = np.searchsorted(cumsum, top_p) + 1

    filtered = np.zeros_like(probs)
    filtered[sorted_idx[:cutoff]] = probs[sorted_idx[:cutoff]]
    filtered = filtered / filtered.sum()

    return np.random.choice(len(filtered), p=filtered)

def sample_generate(prompt_ids, num_tokens=6, temperature=1.0, top_p=0.9):
    """Autoregressive generation with temperature sampling."""
    ids = list(prompt_ids)
    for _ in range(num_tokens):
        logits = compute_logits(np.array(ids))
        next_id = temperature_sample(logits, temperature, top_p)
        ids.append(next_id)
    return ids

prompt = [token_to_idx["the"], token_to_idx["sun"]]

print("Same prompt, different sampling strategies:\n")
for temp, p in [(0, 1.0), (0.5, 0.9), (1.0, 0.9), (1.5, 0.95)]:
    np.random.seed(42)
    result = sample_generate(prompt, num_tokens=4, temperature=max(temp, 0.01), top_p=p)
    text = " ".join(idx_to_token[i] for i in result)
    label = f"temp={temp}, top_p={p}"
    print(f"  {label:25s} → {text}")

print("\nLower temperature = more repetitive but focused")
print("Higher temperature = more diverse but less coherent")

Advantages

The transformer architecture offers several key advantages:

• Parallelisation: All sequence positions can be processed simultaneously, significantly speeding up training and inference compared to sequential RNNs
• Long-range Dependencies: Self-attention directly models relationships between any two positions, regardless of distance
• Computational Efficiency: Despite quadratic complexity with sequence length, parallelisation and fewer required layers make transformers efficient for long sequences
• Interpretability: Attention weights can be visualised to understand which parts of the input the model focuses on

Variants and Extensions

Numerous transformer variants have been developed to address specific challenges:

• Transformer-XL (Dai et al. 2019): Handles longer sequences through segment-level recurrence
• ALBERT (Lan et al. 2019): Reduces parameters through sharing while maintaining performance
• XLNet (Yang et al. 2019): Combines benefits of autoregressive and bidirectional approaches

These innovations continue to advance transformer-based models, enabling increasingly capable language models (Zhao et al. 2023).

KV Cache and Long-Context Costs

During autoregressive generation, transformers avoid recomputing attention for every previously seen token by caching the key and value matrices for each layer — the KV cache. This trades memory for speed: once a token’s keys and values are cached, subsequent forward passes only need to compute attention for the new token against the cached history.

The KV cache has an important implication for any system that modifies past context: changing a token at position $k$ invalidates the cache for every token after it, forcing a full recomputation from that point onwards. This is why approaches that compress or rewrite earlier parts of a conversation must do so carefully — the timing of the modification determines how much cache can be preserved (Xiao et al. 2026).

A separate, more fundamental concern is that long contexts degrade model quality, not just throughput. Empirical studies of transformer-based LLMs show that performance on tasks requiring retrieval or reasoning from context tends to decline as context length grows, with a characteristic “lost in the middle” effect where information in the middle of a long sequence is attended to less reliably than information at the edges (Liu et al. 2024). In agentic settings — where a model accumulates tool outputs across dozens of steps — this means that an agent reading a 200K-token trajectory may reason less accurately than one reading a 20K-token trajectory containing the same essential information (Xiao et al. 2026).

import numpy as np
import time

class AttentionWithCache:
    """Demonstrates KV cache for autoregressive attention."""

    def __init__(self, dim):
        self.W_q = np.random.randn(dim, dim) * 0.1
        self.W_k = np.random.randn(dim, dim) * 0.1
        self.W_v = np.random.randn(dim, dim) * 0.1

    def forward_without_cache(self, X):
        """Recompute K, V for all tokens every step."""
        Q = X @ self.W_q
        K = X @ self.W_k
        V = X @ self.W_v
        scores = Q @ K.T / np.sqrt(X.shape[-1])
        weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)
        return weights @ V

    def forward_with_cache(self, x_step, kv_cache):
        """Compute only for new token, use cached K/V for previous."""
        q = x_step @ self.W_q
        k_new = x_step @ self.W_k
        v_new = x_step @ self.W_v

        if kv_cache is not None:
            k = np.concatenate([kv_cache['k'], k_new])
            v = np.concatenate([kv_cache['v'], v_new])
        else:
            k, v = k_new, v_new

        scores = q @ k.T / np.sqrt(x_step.shape[-1])
        weights = np.exp(scores) / np.exp(scores).sum()
        output = weights @ v

        return output, {'k': k, 'v': v}

dim = 64
attn = AttentionWithCache(dim)

# Generate random sequence
seq_len = 100
X = np.random.randn(seq_len, dim)

# Without cache: recompute everything each step
start = time.time()
for i in range(1, seq_len + 1):
    attn.forward_without_cache(X[:i])
time_no_cache = time.time() - start

# With cache: only compute for new token
start = time.time()
cache = None
for i in range(seq_len):
    _, cache = attn.forward_with_cache(X[i:i+1], cache)
time_with_cache = time.time() - start

print(f"Sequence length: {seq_len} tokens")
print(f"Without cache: {time_no_cache:.4f}s")
print(f"With cache:    {time_with_cache:.4f}s")
print(f"Speedup:       {time_no_cache / time_with_cache:.1f}x")

seq_lengths = [10, 20, 50, 100, 200, 500]
times_no_cache = []
times_with_cache = []

for n in seq_lengths:
    X = np.random.randn(n, dim)
    attn = AttentionWithCache(dim)

    start = time.time()
    for i in range(1, n + 1):
        attn.forward_without_cache(X[:i])
    times_no_cache.append(time.time() - start)

    start = time.time()
    cache = None
    for i in range(n):
        _, cache = attn.forward_with_cache(X[i:i+1], cache)
    times_with_cache.append(time.time() - start)

seq_len = 20
dim = 32

# Create synthetic attention weights showing "lost in the middle"
np.random.seed(42)
attention_matrix = np.zeros((seq_len, seq_len))

for i in range(seq_len):
    for j in range(seq_len):
        # Strong attention to first 3 tokens
        if j < 3:
            attention_matrix[i, j] = 0.3 + 0.1 * np.random.rand()
        # Weak attention to middle tokens
        elif 3 <= j <= seq_len - 4:
            attention_matrix[i, j] = 0.02 + 0.03 * np.random.rand()
        # Strong attention to last 3 tokens
        else:
            attention_matrix[i, j] = 0.25 + 0.1 * np.random.rand()

# Normalize rows
attention_matrix = attention_matrix / attention_matrix.sum(axis=1, keepdims=True)

num_layers = 12
model_dim = 128
bytes_per_param = 4  # float32

context_lengths = [64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072]

# KV cache memory: 2 (K+V) × layers × seq_len × dim × bytes
kv_cache_memory = [
    2 * num_layers * seq_len * model_dim * bytes_per_param / (1024**3)
    for seq_len in context_lengths
]

Impact

The transformer architecture has become the universal foundation for modern large language models. All state-of-the-art LLMs, including GPT-3 (Brown et al. 2020), GPT-4 (Achiam et al. 2023), PaLM (Chowdhery et al. 2023), and LLaMA (Touvron et al. 2023), are built on transformer architectures. Combined with advances in computational resources and training techniques, this has enabled models with hundreds of billions of parameters that demonstrate remarkable capabilities across diverse tasks.