Machine Learning

Imperial College London

Nuri Cingillioglu

Part 4: Connectionist Renaissance

The return of the Jedi Neural Networks

Why did linear models hit a ceiling?

Part 2, glorified curve-fitting. But the reality of perceptual data:

  • For a modest image (), we cannnot just hand-craft features. Polynomial expansion goes BOOM...

The Fundamental Question

Instead of manually engineering fixed non-linear functions or features , can we learn end-to-end directly from raw data?

This is the central thesis of Deep Learning.

Key Insight

Learn Intermediate Representations

Each layer transforms data into a new representation

Early Layers

Low-level sensory primitives (edges, gradients, frequencies)

Middle Layers

Combinatorial motifs (textures, contours, phonemes)

Deep Layers

High-level semantic parts (faces, wheels, words, concepts)

Multi-Layer Perceptrons (MLPs)

A Multi-Layer Perceptron (MLP) stacks affine transformations interleaved with element-wise non-linear activation functions :

For layer :

  • (Input layer)
  • (Synaptic weight matrix)
  • (Bias vector)
  • (Pre-activation vector)
  • (Post-activation representation)

Also known as feedforward neural networks (FFNNs).

Why the Non-Linearity

If (linear identity), the entire -layer network collapses into a single matrix product:

Composition of linear operators is strictly linear.

Please note the mathematical nature of this construction. While the biological inspiration is admirable, we are in not in Kansas anymore.

MLP - Feedforward Neural Network (FFNN)

Resolving the XOR Curse

Consider a 2-layer network with step activation :

Finally, XOR is solved...

XOR: Hidden Layer

Batch Computation

In practice, we stack input vectors into a data matrix :

Shape check:

Computational Efficiency

  • The forward pass is converted into a General Matrix Multiplication (GEMM).
  • Modern GPUs (tensor cores) are hardware-optimised for dense matrix multiplication.
  • Give me more GPUs! More matrix multiplication MOAR AI!

We'll use both notation going foward depending on whether is batched or not.

PyTorch Parameter Level

import torch
import torch.nn as nn
import torch.nn.functional as F

class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.W_1 = nn.Parameter(torch.randn(hidden_size, input_size))
        self.b_1 = nn.Parameter(torch.randn(hidden_size))
        self.W_2 = nn.Parameter(torch.randn(output_size, hidden_size))
        self.b_2 = nn.Parameter(torch.randn(output_size))

    def forward(self, x):
        h = torch.sigmoid(x @ self.W_1.T + self.b_1)
        y = h @ self.W_2.T + self.b_2
        return y

PyTorch Module Level

import torch.nn as nn
import torch.nn.functional as F

class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        # Notice how a "linear" layer is abstracted away
        self.l1 = nn.Linear(input_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        h = F.sigmoid(self.l1(x))
        y = self.l2(h)
        return y

PyTorch Sequential

import torch.nn as nn
import torch.nn.functional as F

model = nn.Sequential(
    nn.Linear(input_size, hidden_size),
    nn.Sigmoid(),
    nn.Linear(hidden_size, output_size),
)

My advice is to not write this code by hand anymore. Connect the practice and theory yourself but let this plumbing be handled by LLMs. Back then, I used to have 50 tabs open just to get a new research model working, now I just ask a LLM. Why does this work so well in this case?

There are other abstractions but these are the basics. The library is packed with stuff for almost any deep learning architecture. Take a look at PyTorch documentation for more.

Bio vs. ANN

Property Biological Human Brain Artificial Neural Network (GPU)
Scale neurons, synapses to parameters
Power (a dim lightbulb!) Megawatts (data centre clusters)
Signalling Mode Asynchronous, discrete analogue action potentials (spikes) Synchronous, dense floating-point tensors (FP16/FP8)
Synaptic Plasticity Local chemical rules (STDP, Hebbian: "fire together, wire together") Global synchronous backpropagation (Reverse-mode autodiff)
Connectivity Highly sparse, 3D anatomical goop Dense matrix multiplication blocks

Spoiler: ANNs are not biological simulations, they are differentiable mathematical function approximators optimised by numerical calculus.

Okay Steve, how do we train it?

Recall our emerging update rule:

  1. Without any hidden layers, everything is the same. Loss functions, softmax, etc.
  2. The idea of descending in the direction of the gradient lowers the loss.
  3. AI happens.

This time there are thousands of parameters all chained together and it spits out some numbers.

Who is to blame to the high loss?

Backpropagation

Follow the money signal.

Neural networks are simply a chain of elementary differentiable operations ().

Forward Pass

  • Feed inputs forward through elementary operations.
  • Compute and cache intermediate values at every step:

Backward Pass

  • Start at the scalar loss with seed .
  • Propagate raw partial derivatives right-to-left using the chain rule.
  • Evaluates gradients for all parameters in a single backward pass!

The pinnacle of calculus

And then we ask:

then:

Magic... 🪄

The Scalar Chain of Computation

To build intuition, consider a single neuron with scalar input , weight , bias , and target :

4 Elementary Operations

  1. Multiply Node:
  2. Addition Node:
  3. Activation Node:
  4. Loss Node:

(All tensors here are scalars of shape )

Concrete Forward Example

:

Backpropagation Demo

The full soup

I won't hold back here, we will do the full derivation for a single layer, no shortcuts. Batch mode.

where

  • (batch of inputs, features)
  • (weights)
  • (stacked bias , where )
  • (pre-activations)
  • (post-activation hidden representations)

Why instead of ? In deep learning libraries (PyTorch, JAX), inputs are matrices of samples. Multiplying on the right keeps batch size as the leading dimension without transpose overhead. .

Single Layer: Concrete Setup

Let :

Multiplying :

The Upstream Gradient Matrix:

The upper layer passes us the incoming upstream gradient matrix w.r.t our outputs :

We need to compute three partial derivative matrices:

  1. — to pass backward to earlier layers.
  2. — to update weights via gradient descent.
  3. — to update bias via gradient descent.

Shape Rule: Because scalar loss is differentiated w.r.t each entry, every gradient matrix has the exact same shape as the variable itself.

Journey of a Single Input:

How does affect the scalar loss ? Look at where appears in :

appears only in the first row of , affecting , and :

By the multivariable chain rule, sum along all outgoing paths:

Intuition: How affects loss depends on the weights it multiplies with and upstream error signals!

Input Gradient Matrix:

Repeating this for all 4 entries in gives the complete matrix:

Extracting the incoming upstream gradient matrix:

Journey of a Single Weight:

Now consider weight . Where does appear in ?

appears in column 1 for both data points ( and ):

By chain rule, sum contributions across all samples in the batch:

Intuition: A shared parameter affects every sample in the batch; its gradient accumulates across all batch samples!

Weight Gradient Matrix:

Writing this out for all 6 weights:

Extracting matrices reveals the exact matrix product:

Shape check: , exactly matching !

Bias Gradient & Activation

Bias Gradient:

The bias is added to each row:

In matrix form:

If this isn't clear, follow where etc appears.

Activation:

For elementwise activation :
Each affects only with local derivative :

In full matrix form:

No shortcuts, no Greek letter soups: Every matrix equation follows directly from the scalar multivariable chain rule on concrete entries!

I first wrote these notes back in 2019...

Burn Learn baby Learn

Parameter update via Gradient Descent:

All your hopes and dreams will fill the blank space below in this slide...

Curious about the slide title, click here

David Rumelhart (1942–2011)

In 1986, a landmark paper appeared in Nature that ignited the modern connectionist era:

"Learning representations by back-propagating errors"

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams (Nature, 1986)

Historical context: Reverse-mode automatic differentiation was introduced earlier by Seppo Linnainmaa (1970) and adapted for neural networks by Paul Werbos (1974, 1982). Rumelhart came up with it independently and shared it with Hinton.

Geoffrey Hinton

"Dave Rumelhart came up with this idea of backpropagation. He wasn't the
first to come up with it, but ...
I first of all explained to him why it wouldn't work, based on an argument in Rosenblatt's book, which showed that essentially it was an algorithm that couldn't break symmetry. If you had a deterministic learning rule, and you started off with two identical hidden units, they would stay identical, so the deterministic learning rule was no good.

Now, as you know, the standard way of doing learning is to have random initial weights, or you could just put a little bit of noise in the learning rule -either would be fine. So Dave Rumelhart pointed out that we didn't need to worry about symmetry.

The next argument I gave him was that it would get stuck in local minima. There was no guarantee you'd find a global optimum. Since you're bound to get stuck in local minima, it wasn't really worth investigating. Nevertheless, Dave programmed it up and did a few things with it. I think he showed it could learn X-OR [exclusive OR]."

Talking Nets: An Oral History of Neural Networks, Chapter 16 page 376.

David Rumelhart

Science is human.

  • In the late 1990s, Rumelhart began experiencing progressive memory and speech difficulties.
  • Diagnosed with Pick's disease (frontotemporal lobar degeneration).
  • The scientist who unlocked how to train artificial neural networks lost his own cognitive abilities, passing away in 2011.

The prestigious Rumelhart Prize is awarded annually today to honour lifetime contributions to the theoretical foundations of human cognition.

Bio Backprop?

Neuroscientists widely agree that exact backpropagation is not how biological brains learn:

Backwards

Biological synapses are strictly unidirectional chemical junctions.

Global Signals

Biological neurons update synapses based on local signals (pre-synaptic firing, post-synaptic voltage, and neuromodulators like dopamine).

Synchrony

The brain operates as an asynchronous, continuous-time dynamical system with no master clock.

Modern computational neuroscience alternatives: Feedback Alignment (Lillicrap et al., 2016), Equilibrium Propagation (Scellier & Bengio, 2017), and Predictive Coding (Friston, 2005).

Vanishing & Exploding Gradients

Networks deeper than 3-4 layers may fail to train. Why?

  1. Weight matrices : Depends on initialisation scale. If we initialise with , every step we multiply by small numbers.
  2. Activation slopes :
    • Sigmoid:
    • Tanh: (and drops to in saturation)

Vanishing & Exploding Gradients

The gradient norm depends on the singular values of the product

Symptom Condition Consequence
Vanishing Gradients Singular values -
- Early layers receive zero updates and freeze.
- Deep net acts like a shallow net.
Exploding Gradients Singular values -
- Parameter updates oscillate wildly.
- Numerical overflow (NaN loss).

What were singular values again?

Singular values measure how much the matrix stretches or shrinks vectors in different directions.

Visualising the Gradient Collapse

Fix this Steve

  1. Non-saturating Activations:
    • Swap Sigmoid/Tanh for ReLU / Leaky ReLU ( for ).
  2. Variance-Preserving Initialisation:
    • He / Xavier Initialisation scales weights by layer fan-in/fan-out to ensure variance stays .
  3. Architectural Highway (Residual Connections):
    • ResNets: .
    • Gradient flow includes an additive identity term , preventing exponential decay.
  4. Normalisation & Safeguards:
    • LayerNorm / BatchNorm + Gradient Clipping.

Activation Functions

The traditional Sigmoid function was the primary culprit behind vanishing gradients:

Every single sigmoid layer shrinks the gradient by at least ()! Ouch...

Activation Formula Derivative Range Properties
Sigmoid Non-zero centred, severe vanishing gradient
Tanh Zero-centred, but still saturates at
ReLU Constant gradient 1 for ; dying ReLU risk
Leaky ReLU Prevents dying neurons by keeping slope

Activation Functions

Weight Initialisation

If all weights are initialised to zero, every neuron in a layer computes the exact same gradient symmetry Hinton was talking about.

To break symmetry, we sample weights randomly with zero mean:

However, if the variance is chosen poorly, our network values will either exponentially explode or vanish to zero as depth increases.

Variance Propagation Derivation (Linear Forward Pass)

For a single pre-activation unit at layer receiving inputs from layer :

Assumptions:

  1. Weights and inputs are independent.
  2. Zero-mean distributions: and .

Summing over all independent inputs:

For near zero, , so :

What we want (layer-to-layer variance preservation):

  • by definition of variance.
  • For independent random variables and , and .
  • Since and , and .

Backward Pass & The Glorot / Xavier Solution

During backpropagation, gradients flow backward from layer to layer :

  • Preserving forward activation variance requires:
  • Preserving backward gradient variance requires:

Glorot & Bengio (2010) took the harmonic mean of both constraints:

Practical Implementation:

  • Normal Distribution:

  • Uniform Distribution (since ):

  • Biases: Initialised to zeros ().
  • Best for: Symmetric activations with unit derivative at zero (e.g., ).

The He / Kaiming Solution (ReLU)

Consider the signal propagation through a layer hop with ReLU:

ReLU acts as a half-wave rectifier: . By definition of expectation:

Revisit:

By the Central Limit Theorem, previous-layer pre-activation is normally distributed and symmetric around zero ( and ).

Substitute back into the layer:

Practical Implementation:

  • Normal Distribution (he_normal):

  • Uniform Distribution (he_uniform):

  • Biases: Initialised to zeros ().
  • Best for: ReLU, Leaky ReLU (scaled with gain), and GELU variants.

It works

Beyond Vanilla SGD

Vanilla Stochastic Gradient Descent updates parameters along the negative gradient:

In deep networks, vanilla SGD struggles with three pervasive geometric landscapes:

1. U-shaped Ravines

Curvature along one axis is vastly higher than along another. SGD oscillates wildly across the walls while crawling along the base.

2. Saddle Points & Plateaus

Points where . SGD stalls indefinitely in zero-gradient zones.

3. Noisy Mini-Batches

Variance across mini-batch gradients can lead to erratic, wandering optimisation trajectories.

Optimisers

Optimisers

Momentum

Intuition: Model optimisation as a heavy physical ball rolling down a potential well with inertia.

Polyak Momentum (1964)

Maintain an Exponential Moving Average of gradients (momentum):

  • (momentum coefficient).
  • Oscillations across steep walls cancel out ().
  • Consistent gradients along the valley accumulate momentum we start rolling down!

Adaptive Learning Rates: RMSProp & Adam

Instead of a single scalar learning rate for all parameters, can we adapt per coordinate?

RMSProp (Hinton, 2012)

Scale updates by the root mean square of recent gradients (usually ):

  • Large gradients large smaller step (damps oscillations).
  • Tiny gradients small larger step (speeds up flat valleys).

Where is RMS? is the square, is the exponentially weighted mean square (variance estimate), and is the root.

Adam (Kingma & Ba, 2014)

Combines momentum (, 1st moment) and RMSProp (, 2nd moment):

Standard defaults: . The division by is a correction for unbiased estimate of the moments. Notice that as , .

What is the moment? The -th moment of a random variable is defined as .

AdamW (Loshchilov & Hutter, 2017)

In Adam, adding regularisation comes out as (ignoring the momentum term):

Weights with large gradients receive less weight decay than weights with small gradients! This is of course a problem.

Instead, decouple weight decay completely from the adaptive gradient moment updates:

This decoupled formulation is the universal standard for training modern deep learning (Transformers, LLMs) and our cheap MLPs.

Regularisation

Explicit Penalty

The intuition works the same from our curve-fitting days.

  • Weight Decay (): Penalises large weights, favouring smooth decision surfaces:

  • Sparsity (): Drives non-essential weights to exactly zero:

Early Stopping

Halting training when validation loss begins to rise limits effective capacity.

Dropout

Introduced by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov (2014):

Training Phase

At each training step, randomly zero out each hidden unit with probability (typically ):

(Inverted dropout scales by during training so inference requires zero modifications as in torch.nn.Dropout).

Why Dropout? Intuition:

  1. Breaks Co-Adaptation: Neurons cannot rely on specific neighbouring units to fix their mistakes; each unit must learn independently useful features.
  2. Implicit Ensemble: Training with dropout samples 1 of possible subnetworks at every step. At test time, using all neurons acts as an ensemble over models.

Internal Shift

What the layer is learning changes as the previous layers are learning!

Deeper layers are forced to constantly adapt to changing input distributions, slowing convergence and requiring delicate learning rates.

Normalisation

Transform pre-activations to have zero mean and unit variance:

are learnable parameters that allow the network to recover the original identity mapping if needed.

Batch vs. Layer Normalisation

Where do we compute the statistics and ?

Batch Normalisation

(Ioffe & Szegedy, 2015)

  • Normalises across the batch dimension for each feature independently.
  • Computes mini-batch stats during training; uses running exponential averages at inference.
  • Drawback: Breaks when batch size is small; impossible for variable-length sequences with zero padding.

Layer Normalisation

(Ba, Kiros, Hinton, 2016)

  • Normalises across all features for each individual sample independently.
  • Exactly identical computation during training and inference!
  • Fully independent of batch size; the universal standard in Transformers and LLMs.

BatchNorm vs. LayerNorm

Okay Steve, why now?

Neural networks were theorised in the 1950s and backprop was popularised in 1986. Why did the deep learning revolution only go BOOM in 2012?

1. Compute

  • NVIDIA CUDA (2006) repurposed GPUs from 3D gaming to general-purpose matrix engines.
  • speedup over CPUs made multi-day experiments feasible.

2. Big Data

  • High-capacity neural networks overfit on small datasets.
  • The web enabled ImageNet (Fei-Fei Li, 2009): 14 million annotated images across 20,000 categories.

3. Soup

  • Non-saturating ReLU activations.
  • Principled initialisation.
  • Adam / AdamW optimisers.
  • Dropout & LayerNorm.

PyTorch: Does all this for you

PyTorch bridges mathematical formulation and hardware execution for you:

  1. Autograd Engine (Reverse-Mode AD)

    • Constructs a dynamic computational graph (define-by-run) on the fly during the forward pass.
    • Automatically computes gradients via .backward().

  2. Hardware Acceleration via Tensor Abstraction

    • High-dimensional arrays (torch.Tensor) backed by CUDA/ROCm C++ kernels.
    • Maps batched forward/backward passes directly to GPU GEMM operations.

torch.Tensor

N-dimensional array with gradient tracking

torch.autograd

Automatic differentiation backpropagation

torch.nn

Neural networks (Module, Parameter, layers, losses)

MNIST Example

1. Model Definition

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.net(x)

2. Training Loop

model = MLP().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
lossfn = nn.CrossEntropyLoss()

for x, y in train_loader:
    ...
    optimizer.zero_grad()
    out = model(x)
    loss = lossfn(out, y)
    loss.backward()
    optimizer.step()
    ...

Questions?

That wraps up the core of neural networks.

Inductive Biases & The Architecture Zoo

An inductive bias is the set of prior assumptions an algorithm encodes regarding the geometry and structure of the target function.

  • Fully-Connected (MLP) All to all, no assumptions
  • Convolutional (CNN) Spatial Locality & Translation Equivariance
  • Recurrent Sequential, Temporal
  • Attention Relational

We can mix and match, e.g. CNN -> RNN -> Attention to process videos.

Inductive Biases: Architectures

Convolutional Neural Networks

Two problems with fully-connected neural networks for computer vision:

1. Parameter Explosion

A high-resolution image () has input dimensions. Connecting this to 1,000 hidden units requires:

A single dense layer (MLP) = gigabytes of memory.

2. Destruction of 2D Topology

Flattening an image into a 1D vector destroys spatial neighbourhood relationships. Neighbouring pixels and are treated independently.

Biological Inspiration: Hubel & Wiesel (1962, 1981 Nobel Prize)

David Hubel and Torsten Wiesel discovered that neurons in the mammalian primary visual cortex (V1) have local receptive fields sensitive to oriented edges, organised into hierarchical simple and complex cell columns.

The Hierarchy of Learned Representations

Source: https://twopointseven.github.io/2017-10-29/cnn/

The 2D Convolution Operation

Instead of all-to-all dense matrix multiplication, a convolutional layer slides a small parameter kernel across the input image:

1. Local Connectivity

Each neuron connects only to a tiny spatial patch (e.g. or ), mimicking biological receptive fields.

2. Weight Sharing

Kernel weights are reused across every single pixel location in the image.

3. Translation Equivariance

If an object shifts in the input by , its activation representation shifts by the exact same amount:

2D Convolution & Max Pooling

No Stride, Padding

Stride and Padding

The same sliding window business happens for convolutional and pooling layers.

Typical CNN Architecture

Hyperparameters of the Convolutional Layer

Given input spatial dimension , kernel size , padding , and stride :

  • Stride : Number of pixels the kernel jumps per step (downsamples when ).
  • Padding : Border pixels added to preserve spatial boundaries:
    • Valid padding: (output shrinks).
    • Same padding: (for odd and , ).

AlexNet & The 2012 ImageNet Breakthrough

The modern era of deep learning officially began in October 2012 at the ImageNet Large Scale Visual Recognition Challenge (ILSVRC):

Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever
Credit: Computer History Museum

The 10.9% Error Drop

  • Traditional Computer Vision: SIFT / HOG features + SVM achieved 26.2% error.
  • AlexNet (CNN): 15.3% error!
  • Architecture:
    • 5 convolutional layers + 3 dense layers (60M parameters).
    • First large-scale use of ReLU activations.
    • Dropout (0.5) in dense layers.

Residual Networks (ResNet)

After AlexNet (8 layers) and VGG (19 layers), researchers attempted to train 30, 50, and 100 layer plain networks. They went BOOM (in a bad way):

The Degradation Barrier

(He et al., 2015)
A 56-layer plain network suffered higher training error than a 20-layer network.

  • This was not overfitting (both training and test error were worse).
  • Worst case idea: identity mapping . But unable to learn identity functions via gradient descent.

The Residual Solution

Instead of forcing weight layers to fit target mapping , parameterise the residual:

If identity is optimal, optimiser simply drives .

Residual Block

Why ResNets Can Train 100+ Layers

Consider the chain rule gradient of a residual block with loss :

Even if completely vanishes, the gradient flows directly backwards unimpeded to layer 0!

ResNet-152 (ILSVRC 2015)

  • Trained a 152-layer network cleanly.
  • Achieved 3.57% top-5 error, surpassing human-level performance (5.1%) on ImageNet.
  • Residual skip connections are now an indispensable architectural primitive across all of deep learning (including Transformers!).

Recurrent Neural Networks (RNNs)

Images are static spatial grids. But language, audio, DNA, and financial time-series are sequential and variable-length: .

Maintain an internal hidden memory state :

  • (Hidden-to-hidden recurrent transitions).
  • (Input-to-hidden projections).
  • Parameter Sharing: The exact same weight matrices are reused at every time step !

RNN Computational Graph

Backpropagation Through Time (BPTT)

To train an RNN, we unroll the computation graph across steps and accumulate gradients:

Follow the money gradient

This time involves repeated applications of and derivatives of .

Vanishing & Exploding Gradients

  • Gradients can vanish like before difficult to learn long-range dependencies.
  • Gradients can explode, like before must use something like gradient clipping.
  • Gated units (LSTM / GRU) solved vanishing gradients via additive linear cell states:

GRU (slightly simpler of two):

where:

  • is the update gate (how much of the past information to keep)
  • is the reset gate (how much of previous state to retain for candidate)

Gated Recurrent Unit (GRU)

Two problems with RNNs

  • The Information Bottleneck
    Compressing arbitrary long-range semantics into a fixed vector causes catastrophic forgetting of early sentence context.
  • Sequential Computation
    Since each hidden state depends on the previous one, the computation cannot be parallelised across the time dimension. This prevents RNNs from taking full advantage of modern parallel hardware like GPUs.

The Attention Mechanism

Let the model learn to access and combine only the relevant information.

In sequence-to-sequence translation, an RNN encoder compresses an entire 50-word sentence into a single fixed-size vector :

Bahdanau et al. (2014) Soft Alignment

Instead of a static bottleneck , allow the decoder to look back and attend dynamically to all encoder hidden states via content-based addressing!

Seq2Seq Example: English French

Task: translate "The black cat slept quietly" "Le chat noir dormait paisiblement".

1. Encoder (Source Sentence)

Input sequence of tokens ():

At each step , the encoder RNN computes:

  • Produces hidden states .
  • Each encodes word in context.
  • Bottleneck: Vanilla Seq2Seq keeps only !

2. Decoder (Target Translation)

Target sequence of tokens ():

At step , given state and context :

  • Word reordering: "black cat" "chat noir".
  • Generating needs ("black")!
  • How does decoder get ? Attention

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014).

Transformers: "Attention Is All You Need"

In 2017, Vaswani et al. (Google Brain / Google Research) posed a radical question:

If attention provides dynamic information routing, why keep the recurrent sequential layers at all?

The model is like a database:

  • Query — What am I looking for?
  • Key — What information do I have?
  • Value — What information do I want to retrieve?

Where does come from?

We learn them of course... Given token representations :

where and learnable weights.

Where does come from?

We use learned token embeddings: .

And positional encodings: :

The deep dive is left for the NLP course!

Example

Input sequence: .

  • Step 1 (Projections): Compute and .
  • Step 2 (Querying): Word ("bank") asks "What kind of bank am I?" via query :

  • Step 3 (Softmax): Scaled attention weights:

  • Step 4 (Value Mixing):

now represents river bank rather than a financial bank!

Why Scale by ?

Assume and for vectors :

Dividing by preserves unit variance and healthy gradient flow.

Multi-Head Self-Attention

Rather than a single attention map, project into distinct representation subspaces in parallel:

Allows the model to attend simultaneously to syntactic relations, coreferences, and semantic attributes!

Scaled Dot-Product Attention

The Complete Transformer Block

Recurrent vs. Transformer Inductive Biases

Dimension Recurrent Neural Network Transformer (Self-Attention)
Computational Dependency Strictly sequential: must compute before Fully parallel: all tokens processed simultaneously
Training Complexity sequential steps (cannot saturate GPU) sequential operations (pure GEMM matrix products!)
Maximum Path Length forward and backward steps across time direct connection between any two arbitrary tokens
Memory / Compute Scaling computation, memory pairwise attention matrix
Positional Awareness Implicit in sequential transition order Permutation equivariant! Needs Positional Encodings (RoPE / Sinusoidal)

Show me the code

class ResidualMLPBlock(nn.Module):
    def __init__(self, dim: int):
        super().__init__()
        self.norm = nn.LayerNorm(dim)
        self.linear1 = nn.Linear(dim, dim * 4)
        self.act = nn.GELU()
        self.linear2 = nn.Linear(dim * 4, dim)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Pre-LN Residual Highway: x + F(Norm(x))
        residual = x
        out = self.linear2(self.dropout(self.act(self.linear1(self.norm(x)))))
        return residual + out

Training loop

import torch.optim as optim

model = ResidualMLPBlock(dim=256)
# Decoupled AdamW optimiser with cosine learning rate schedule
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
criterion = nn.MSELoss()

for epoch in range(100):
    optimizer.zero_grad(set_to_none=True)  # Faster than setting to 0
    
    inputs = torch.randn(64, 256)
    targets = torch.randn(64, 256)
    
    predictions = model(inputs)
    loss = criterion(predictions, targets)
    
    loss.backward()                       # Reverse-mode autodiff (Backpropagation)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # Gradient clipping
    
    optimizer.step()                      # AdamW parameter update
    scheduler.step()                      # Learning rate decay

Where is my artificial brain?

I think we started really well this time, more neurons, layers etc. but went all mechanical, fixing things until they work...

AI ML Numbers game

Next Part: What happens when we abandon continuous differentiable hyperplanes entirely for orthogonal, rule-based space partitioning? Welcome to Decision Trees, Ensembles, and Collective Intelligence!