Machine Learning

Imperial College London

Nuri Cingillioglu

Part 2: Statistical Foundations of ML

The glory of curve-fitting awaits you all...

One step back

Recall the artificial neuron (perceptron) model:

now let's remove the activation:

oh gosh, this looks like a line, doesn't it??? We have effectively come back to the gooooood old

Let's machine learn this then

Fitting a plane in action

Linear Regression

For some given inputs we have an associated output modelled as:

For convention, we'll let and . Then, our model becomes:

So how do we find the best parameters? What does best mean?

Notation abuse

You'll see, even in this course, that the notation may change as we go along. There is no right or wrong, just different ways people write things down. I've seen all these:

  • It's all the same thing, some parameterised function that we are going to optimise.

  • There is some baseline similarity which is etc are used for functions, etc are used for parameters and is input, is output.

Least Squares

We need a way to measure how well our model is doing. Consider the distance between the predictions and true values :

Now, we can minimise the error by minimising :

Ordinary Least Squares (OLS)

From calculus we know that the minimum of occurs when its derivative is zero. Given training samples with features and outputs (assuming includes the bias parameter and ):

  • This is a system of equations in unknowns written in matrix form.

In vector calculus, the gradient of a scalar function with respect to a vector or matrix is the vector or matrix of the partial derivatives.

In matrix form

Now, since for a single input vector, we write in matrix form:

Shape check: .

Using the fact that for any vector we have , we can write our cost function as:

Normal equations

Recall that: , , and for symmetric matrix .

Normal equations

The closed-form solution requires that is invertible. That inversion breaks down whenever lacks sufficient independent features:

  • Too few samples (): There are fewer training examples than parameters to learn.
  • Redundant features: One or more features can be written as a linear combination of others.

In either case, , rendering singular (non-invertible) and leaving infinitely many parameter vectors that minimise the loss.

Why call it normal equations?

This equation is called the normal equation because it represents the condition that the residual is orthogonal (or “normal”) to the column space of .

Mini-break

When we removed the activation function from the perceptron, we ended up with linear regression. Doesn't look like our artificial brain...

  • This is an equation / problem with a rich history.
  • We are now at the mercy of statistics, linear algebra, calculus and probability theory...

Please hang in there.

Gradient descent

I don't want to invert matrices, I want to crank the handle.

We know that the minimum occurs when the derivative is zero. So, we can start with an initial guess for and iteratively update it in the direction of the negative gradient:

where is the learning rate. This update occurs simultaneously for all :

Intuition

  1. The gradient tells us how each parameter changes the cost function .
  2. If we go in the direction of the gradient, we increase the cost function. (Gradient Ascent)
  3. If we go in the direction of the negative gradient, we decrease the cost function. (Gradient Descent)

This is like walking on a hill blind folded. You take a step in the direction where you feel has the biggest drop until you reach the bottom.

Gradient descent

Least Mean Squares

Thus the LMS update rule with gradient descent is:

This is also known as the Widrow-Hoff learning rule (1960).

LMS: Vectorised Form

Given design matrix , targets , and parameters :

Gradient Derivation

Shape check:

Batch Update Rule

Probabilistic Interpretation

Is there any other way to look at this? That perhaps gives us why the data, the cost function etc work together?

Let us assume that for a single data point we have:

where is an error term that captures the noise in the model. Furthermore, let's assume that these values are independent and identically distributed with zero mean and variance Gaussian distribution: :

Probabilistic Interpretation cont.

If we replace with in the equation, we get:

Take a very good look at this, what does it mean?

  • It looks like our predictions are the mean of a Gaussian distribution.

Likelihood

Surely, we would like to find the parameters that maximise the probability of observing given and the parameters :

This is our likelihood function. Notice how it is a function of now. Using IID assumption:

Here denotes the -th feature vector and its target scalar.

Maximum Likelihood Estimation (MLE)

Maximum likelihood estimation is about finding parameters that maximise .

What are the best parameters that maximise the probability of observing given ?

To avoid dealing with a huge product of terms that can underflow (multiply to zero), we can instead maximise the logarithm of the likelihood function:

since logarithm is a monotonically increasing function.

This is really it. Behind the scenes of AI, we maximise some likelihood.

MLE Cont.

Connecting MLE to Least Squares

Maximising the log-likelihood with respect to :

which is our original least squares cost function. Also notice how was not important in the final result...

Connecting MLE to Least Squares

Under the assumptions we made, least squares regression corresponds to maximum likelihood estimation.

  • This sort of jiggery pockery is not necessary although you will come across it a fair amount...
  • A geometric interpretation from our visuals is an equally valid, good and rational procedure and there may be indeed other assumptions and / or interpretations.

But the Bayesian conjugate prior of the normal posterior estimation is another normal... This is proof by intimidation. Don't be scared.

Where we are so far?

  • We started with a biological neuron and how we can model it computationally.
  • We looked at the Perceptron and struggled with XOR
  • We then briefly covered k-nearest neighbours
  • We removed the activation function to discover linear regression and saw how it related to maximum likelihood estimation.

Questions?

Linear Regression for Classification?

Suppose our task is binary classification: . Why can't we simply fit a linear regression model and threshold predictions at ?

  1. Unbounded Predictions: A linear model produces . Outputs like or cannot represent valid probabilities.
  2. Extreme Sensitivity to Outliers: An observation with a large feature value (even if correctly classified) pulls the linear regression line, shifting the entire decision boundary.
  3. Violates Error Assumptions: Binary targets mean errors cannot be normally distributed with constant variance .

We need a smooth, non-linear activation that squashes into valid probabilities !

Bring back the activation

Stone T, Webb B, Adden A, et al.
An Anatomically Constrained Model for Path Integration in the Bee Brain
Current Biology, 2017; 27, 3069-3085.e11

Logistic function

It seems like some of these are modelling on or off like behaviour following S shape?

Let's plug this into our previous linear model:

we're back in stats land with logistic regression!

One day, we will have a discussion about artificial brains; but today is not that day.

Logistic Regression

Logistic Regression

The aim is now to classify data rather than predict continuous values. The regression in "logistic regression" is a historical naming quirk - it is a classification algorithm.

  • We now have the tools to make sense of this setup and try and optimise it using gradient descent.

Let us assume:

we can rewrite this more compactly:

MLE again

The last equation looks like a Bernoulli distribution. So our assumption is that our model produces probabilities! Bring in MLE to the rescue:

take the log:

Negative Log Likelihood

Maximising log-likelihood is equivalent to minimising the negative log-likelihood:

Gradient of Sigmoid

The gradient / derivative of the sigmoid function has a really nice property:

we can use this and the chain rule of calculus to find the gradient of the cost function with respect to the parameters .

Deriving the Gradient: Chain Rule Setup

Let and .

For a single data point , the negative log-likelihood is:

Using the chain rule, we decompose the gradient with respect to :

Evaluating the Derivatives

We evaluate each of the three terms separately:

  1. Loss with respect to prediction (deriv. of log, don't panic):

  1. Prediction with respect to (Sigmoid derivative):

  1. Linear combination with respect to :

The Chained Derivative

Multiplying the three factors together, the terms cancel out cleanly:

Summing across all data points, the total gradient of is:

I asked the best LLM I had access to make a diagram for you, after thinking for a while it gave me this. Please ignore it. The LLM just took my beautiful hand written equations and copied it. Where is my PhD level AI assistant now?

Gradient Descent Update

To minimise the cost , we take steps in the direction opposite to the gradient:

Substituting our derived gradient:

where is the learning rate.

(Note the exact same functional form as the Least Mean Squares update in linear regression, but here is non-linear!)

Why No Closed-Form Solution?

In linear regression (OLS), setting the gradient to zero gives the normal equations:

For logistic regression, we attempt the same:

The Problem: Non-Linearity

  • The parameter vector is trapped inside the non-linear sigmoid term .
  • This forms a system of coupled non-linear equations across all data points.
  • You cannot factor out or isolate algebraically via matrix inversion.

The Solution: Numerical Optimisation

Because no closed-form solution exists, we must rely on iterative numerical methods:

  • First-order: Gradient Descent / Stochastic Gradient Descent (SGD)

(Fortunately, in this simple case is convex, so numerical methods are guaranteed to converge to the global minimum.)

Turns out, this optimisation via gradient descent is decent enough! You crank it up to large neural networks, with tricks and tweaks, it's what we will be doing in later parts of this course 🫠

Beyond Binary: Multi-Class

So far, our classification world has been strictly binary: . But the real world rarely offers only two choices:

  • Classifying handwritten digits: (MNIST)
  • Medical diagnoses: multiple distinct disease categories
  • Object recognition: cats, dogs, birds, cars...

Why Not Just Train Multiple Binary Classifiers?

  • One-vs-Rest (OvR): Train independent binary classifiers separating class from all other classes.
  • Problem: Probabilities are uncalibrated and don't sum to 1. What happens if multiple classifiers output , or all output ?

Multi-class Classification

The Goal: A single unified probabilistic model producing a valid categorical distribution:

We want our model to output this vector :

From Linear Scores to Logits

For classes, instead of a single parameter vector , each class gets its own weight vector and bias :

In compact matrix notation:

We call the logits (unconstrained linear scores).

The Challenge

  • Each logit can be any real number: .
  • Simple division fails: logits can be negative, and the sum can be zero!
  • How do we map an arbitrary vector into a valid probability distribution?

The Softmax Function

To convert unbounded logits into valid probabilities, we take exponents and normalise:

Essential Properties

  1. Strictly Positive: for all .
  2. Sums to One: .
  3. Monotonic: If , then . Larger logits receive exponentially more probability.

What's up with this softmax?

"Soft" Maximum: A smooth, differentiable approximation to . Keeps gradient descent happy.

If we scale logits by a temperature hyperparameter (computing ): as , the distribution sharpens into a hard one-hot argmax; as , it flattens into a uniform distribution ().

Have you heard of temperature before? Where? We will revisit it later.

Sanity check for : When there are only two classes, . Softmax is the exact multi-class generalisation of sigmoid.

Softmax Classification

Multi-Class Likelihood

To represent targets across classes, we use one-hot encoding:
If sample belongs to class , then with and for .

Under the Categorical distribution (generalised Bernoulli), the probability of observing is:

Because only the true class index has , all other terms are . It cleanly selects the probability of the true label.

Assuming independent observations, the total likelihood is:

Multi-Class Likelihood Example

Consider our 3-class animal classifier (): , , .

For an input , our model outputs predicted probabilities:

Suppose the true label is a cat (), so the one-hot target is :

Categorical Cross-Entropy

Taking the negative log-likelihood gives our cost function:

For a single data point (omitting the sample index ):

because only the true label has . This is the famous Categorical Cross-Entropy Loss.

Intuition

  • If the model is confident and correct (): Loss .
  • If the model is wrong (): Loss (severely penalised!).

Multi-Class Gradient Update

In vector form, the gradient with respect to the logits is (see appendix for gory details):

And applying the chain rule to the weight vector for class :

Summing over all data points gives the gradient descent update:

Look Familiar?

  • Linear Regression (OLS):
  • Logistic Regression:
  • Softmax Regression:

The gradient has this general pattern: (Prediction Target) Input! It is not coincidence this happened.

Hold on to this thought until we get to neural networks.

Information Theory: Surprise

Why is this called "Cross-Entropy"?

This brings us to information theory and how we interpret our model outputs. Similarly to the probabilistic framework, this will give us a new perspective.

A Mathematical Theory of Communication

Claude E. Shannon. The Bell System Technical Journal 1948-07: Vol 27 Iss 3. AT & T Bell Laboratories. 1948-07-01.

How Much Information Does an Event Carry?

Imagine hearing two news reports:

  1. "The sun rose this morning." Completely expected. Zero surprise.
  2. "A massive meteor landed in Hyde Park!" Highly improbable! Enormous surprise!

Shannon formalised this intuition as Information Content (or Surprise):

  • If bits (no new information).
  • If bits (infinite information / shock).

Shannon Entropy

What is the average surprise you expect when sampling from a distribution ?

This expected surprise is the Shannon Entropy :

Physical & Computational Meaning

  • Measure of Uncertainty: High entropy means outcomes are uniform and unpredictable; zero entropy means the outcome is completely deterministic.
  • Theoretical Compression Limit: Shannon's source coding theorem shows that is the absolute minimum number of bits needed, on average, to encode outcomes from without loss.

Entropy in Action: Binary Entropy Curve

Fair vs. Biased Coin

  • Fair Coin (): Maximum Uncertainty

    • Surprise: , .
    • Expected Entropy: . Unpredictable and incompressible: strictly requires 1 full bit per flip to store or transmit.

  • Biased Coin (): Low Uncertainty

    • Common outcome (): (low surprise, little new information).
    • Rare outcome (): (high surprise!).
    • Expected Entropy: .
    • Highly compressible: on average, takes less than half a bit per flip!

  • Deterministic Coin (): Zero Uncertainty

    • Surprise: .

Visual and modern deep dive by 3B1B

There is a fantastic playlist by 3B1B on Information Theory. I would recommend you watch it! Especially for visual learners who find the mathematical derivations challenging.

A Note on Logarithms: Bits vs Nats

You will see both base and base used across literature:

  • Information Theory (Bits): Shannon originally defined entropy using :

  • Machine Learning (Nats): In ML, we almost always use the natural logarithm (, base ):

    Because , base is preferred.
  • Equivalence: Since , the two differ only by a constant positive multiplier. Minimising nats is strictly identical to minimising bits.

Cross-Entropy & KL Divergence

Suppose nature generates data from the true distribution , but we believe nature follows our model's distribution denoted as .

  • Cross-Entropy : The average code length needed to transmit events from using an optimal code designed for :

  • KL Divergence : The "wasted bits" (inefficiency) incurred by using distribution instead of true :

The Fundamental Link

Since the training labels are fixed, the true entropy is a constant! Therefore:

Minimising Cross-Entropy Minimising KL Divergence Maximum Likelihood Estimation.

The Grand Unification

We have now journeyed across the statistical foundations of supervised learning:

Model Target / Noise Loss Function Information / Likelihood View
OLS , Gaussian Noise Mean Squared Error MLE under
Logistic , Bernoulli Binary Cross-Entropy MLE under
Softmax , Categorical Categorical Cross-Entropy Min / Categorical MLE

There are bigger unifications than this but it is out of scope for this course and depends on how you want to specialise across statistics, information theory or probabilistic views.

But wait... Remember XOR?

We have built linear regression, logistic regression, and softmax regression. All mathematically sound, all derived from first statistical principles.

Now, take a careful look at the decision boundary of logistic regression:

The decision boundary is strictly a flat hyperplane:

  • In 2D: a straight line ().
  • In 3D: a flat two-dimensional plane.
  • In -D: a -dimensional hyperplane.

Can this separate the XOR function? No! We are trapped at the exact same geometric wall that Minsky and Papert raised against the Perceptron in 1969.

Playground (Remove Hidden Layers)

The Escape: Basis Expansion

If our statistical model is linear in its parameters, how can we possibly learn non-linear boundaries?

  • The model must be linear in its parameters , but it does not have to be linear in the original input features .

Let us define an explicit non-linear feature mapping:

Instead of feeding directly, we feed the transformed feature vector :

We project the data into a higher-dimensional space where previously entangled points become linearly separable.

Cracking XOR (finally)

Let the input be :

Now, augment the input with a simple polynomial interaction term:

Notice what happens to the third feature value for :

  • For
  • For
  • For
  • For lifted off the base plane!

Cracking XOR: The Solution

With this new feature space, let our parameter vector be:

Let us evaluate the linear score :

  • Sample : (Correct)
  • Sample : (Correct)
  • Sample : (Correct)
  • Sample : (Correct)

The non-linear interaction term pulled the point into an orthogonal dimension, allowing a flat plane to slice cleanly between the classes.

General Polynomial Basis Expansion

We can extend this principle to arbitrary degree . For a scalar input :

Our regression model becomes:

The Crucial Property

Because remains strictly linear with respect to the weights :

  • All previous least-squares derivations apply without modification.
  • We simply replace the design matrix with the expanded matrix :

And for logistic regression, the loss function works like before.

The Curse of Combinatorics

If polynomial expansions give us non-linear power, why not expand everything to degree 20?

For an input vector expanded to polynomial degree , the number of features is:

Consider a modest feature space:

  • A small greyscale icon of size pixels has features.
  • If we want pairwise interactions ():

Hand-crafting basis functions goes BOOM 💥. Forget vision, audio, or language...

SVMs & The Kernel Trick

Before deep learning, how did machine learning deal with the combinatorial explosion of polynomial expansions?

In the 1990s and 2000s, the dominant paradigm was the Support Vector Machine (SVM) and the Kernel Trick:

  • The Idea: Many algorithms (including linear regression and SVMs) can be expressed purely in terms of inner products between data points: .
  • The Kernel Trick: Compute this inner product implicitly using a kernel function in original input space, without ever calculating the massive vector directly.

Kernel methods scale with number of samples . For modern datasets with millions of samples, this is intractable. Deep learning eventually destroyed all this...

The Danger of High Capacity

Even when we can compute the expansion, should we?

Suppose we collect noisy observations from an underlying natural phenomenon:

Let us fit polynomials of varying degree to this data:

  1. Degree (Underfitting): Too rigid. Cannot capture curvature.
  2. Degree (Appropriate Fit): Balances flexibility with simplicity. Captures the true generative pattern.
  3. Degree (Overfitting): Excess capacity. Snaking through every single noisy sample with zero training error, but exploding into wild oscillations between points.

Memorisation vs Generalisation

This brings us to the foundational objective of machine learning:

  • Pure Optimisation: Minimise the empirical risk / loss on observed data:

  • Machine Learning: Minimise the expected risk / loss on unseen future data drawn from the underlying distribution :

A model with infinite capacity (e.g. high degree polynomial or kNN) can achieve by memorising the training data and any noise in it, while going BOOM 💥 in the real world.

All Possible Datasets

Assume the true data-generating process is:

Suppose we collect a training set and train an estimator .

What is the expected squared test error at a fixed query point , averaged across all possible training sets and test noise ?

A Useful Probability Identity

Recall from introductory probability that for any random variable :

Rearranging this gives a clean shortcut for the expected squared value:

Key Idea: Let our random error at query point be .

Step 1: Mean of the Error

Substitute and apply linearity of expectation:

Since is deterministic at and the noise has zero mean ():

Step 2: Variance of the Error

Now find the variance of :

  1. is a constant, so .
  2. Test noise is independent of the training set (and thus independent of ).
  3. Using for independent .

Step 3: Putting It All Together

Substitute the Mean (Step 1) and Variance (Step 2) back into our identity:

  • : How far the average model is from the true function .
  • : How sensitive is to the specific choice of training set .
  • : Inherent noise in the target that no model can predict.

The Bias-Variance Decomposition

You'll also see it written with explicitly:

Let's try again

  • (Systematic Error): How much the average model over all possible datasets deviates from the true underlying function.
  • (Instability): How much the model's predictions fluctuate when trained on different random samples from the same population.
  • (Irreducible Noise): Unavoidable variance in the measurement or universe. No model can eliminate this.

Overfitting & Underfitting

Regime Train Error Test Error What's Happening? Concrete Example
Underfitting High Low High High Model is too simple to capture ; insensitive to the choice of . Fitting a straight line () to a sine wave.
Overfitting Low High Very Low High Model captures on average, but chases random noise in each specific . Unregularized degree-20 polynomial on noisy points.
Sweet Spot Low Low Low Low Model capacity matches the complexity of and dataset size . Degree-3 polynomial or Ridge regression with tuned .

How to reduce variance?

Our foundational goal: minimise total expected test risk.

When models have high capacity (e.g. high-degree polynomials or huge neural networks), variance dominates. How can we reduce variance?

  1. Collect more training data (): As , variance shrinks towards zero. But acquiring more data is often expensive or impossible.
  2. Remove features / reduce degree (): Eliminates variance, but risks underfitting (causing bias to spike).
  3. Constrain parameter magnitudes (Regularisation): Retain all expressive features, but penalise large weights.

Taming Overfitting: Regularisation

When an polynomial oscillates violently to pass through noisy samples, its weight vector behaves in a telltale way:

The coefficients explode into massive positive and negative values that cancel each other out precisely on the training points, but diverge violently elsewhere.

Regularisation

Penalise the model for having large weights:

where is the regularisation hyperparameter controlling the trade-off between fitting the training data and keeping model capacity restricted.

Regularisation

In Ridge Regression, we penalise the squared Euclidean norm ( norm) of the weight vector:

The total cost function is:

(Convention: We regularise the weights , but typically leave the bias term unpenalised so the model can freely shift the baseline.)

Ridge: Closed-Form Solution

Let us take the gradient of with respect to and set it to zero:

Solving for :

A small note

Recall that OLS broke down when was non-invertible ( or collinear features).

For any , the matrix shifts all eigenvalues by . It is strictly positive definite and guaranteed to be invertible. But who cares...

Regularisation

What if we penalise the sum of absolute values ( norm) instead?

This gives Lasso Regression (Least Absolute Shrinkage and Selection Operator):

  • Ridge (): Shrinks all weights smoothly towards zero, but rarely become exactly zero.
  • Lasso (): Drives many weights to exactly zero ().

Lasso performs automatic feature selection, yielding sparse models that retain only the most explanatory inputs. In 2026, it is used in mechanistic interpretability of LLMs.

Ridge () vs. Lasso ()

Why does produce exact zeros?

We effectively have a constrained optimisation problem with some budget :

  • Ridge ( Circle: ): Smooth boundary tangential contact happens off-axis () with best solution .
  • Lasso ( Diamond: ): Sharp corners sit directly on the axes (e.g., ) the ellipse hits a corner first, zeroing out .

The way I think about it

How hard does each penalty push a small weight (say, ) towards zero?

Penalty Type Penalty Term Derivative (Push towards ) At Behaviour near
Ridge () Push weakens as
Lasso () Constant push all the way to exactly

The Bayesian MAP Perspective

Can we understand Ridge and Lasso from a probabilistic perspective? Because we're bored.

Recall Bayes' Theorem, and apply to our model parameters :

  • MLE (Frequentist): Finds the single that maximises data likelihood:

  • MAP (Bayesian): Finds the mode of the posterior distribution, incorporating prior beliefs about :

Ridge: MAP with a Gaussian Prior

Assume the standard Gaussian likelihood from linear regression:

Now place an independent zero-mean Gaussian prior on the weights: :

Take the almighty log:

Maximising the posterior is equivalent to minimising:

Multiplying through by gives Ridge Regression with 😯

Wait da hell is ?

  • High data noise () or tight prior () Large (trust the prior, shrink ).
  • Low data noise () or broad prior () Small (trust the data).

Lasso: MAP with a Laplace Prior

I believe that most features are irrelevant, and weights should be concentrated at zero.

Laplace prior on the weights: :

Take the almighty log:

Maximising the log-posterior corresponds to minimising:

Multiplying through by gives Lasso Regression with .

The Regularisation Summary

Property Ordinary Least Squares Ridge Regression () Lasso Regression ()
Penalty None ()
Bayesian Prior Uniform (No prior) Gaussian Laplace
Weight Effect Unconstrained, can explode Shrunk smoothly towards 0 Sparse: driven to exact 0
Feature Selection None Retains all features Automatic selection
Solution Method Gradient descent
Collinearity Handling Fails (Singular ) Robust (Always invertible) Selects one feature arbitrarily

"Double Descent"

For a long time, the bias-variance tradeoff was king:

But modern deep neural networks have hundreds of billions of parameters, way more than training samples (), and they generalise remarkably well.

Double Descent (Belkin et al., 2019)

  • Classical Regime (): Test error follows the traditional U-shaped curve.
  • Interpolation Threshold (): The model has just enough capacity to interpolate all training points exactly.
  • Overparameterised Regime (): Test error descends a second time!

We do not formally examine double descent in this course, but it covers why the classical fear of overparameterisation does not destroy modern deep learning.

Belkin et al., Reconciling modern machine-learning practice and the classical bias–variance trade-off, PNAS, https://doi.org/10.1073/pnas.1903070116 (2019).

Where is our artificial brain?

Phew! I'm tired, you're tired of playing around with all this. Where is our artificial brain?

But

We are now only 1 step away from deep learning. So close, yet so far...

I personally prefer things that are elegant rather than wrangled together to make it work. What did we experience so far?

Here is what elegant looks looks like:

Appendix

AKA The Cabinet of Mysteries

Deriving the Gradient: Softmax

To optimise via gradient descent, we need . But unlike sigmoid, every logit affects every output probability via the denominator .

We need to use the quotient rule:

to handle the denominator:

Let:

Case 1:

Case 2:

Softmax Derivative Cont.

We can express both cases concisely using the Kronecker delta :

where is:

  • If : (Case 1)
  • If : (Case 2)

It unifies both terms into a single expression in a fancy way.

Now, using the multivariable chain rule to find :

Counting Features

Why is the number of features up to degree given by ?

Introduce bias variable as . Now every term of degree can be written as a product of exact degree :

Finding all features is equivalent to distributing degree units into buckets:

  • Example ():
  • Total slots to fill: dividers + units = slots.
  • Choosing which out of slots: