Sometimes it does seem like my blog is just increasingly complex applications of the Fourier Transform. In the previous post we applied the Fourier Transform to graphs, drawing connections between frequency (which is the usual Fourier transform) and properties of the graph. There is yet another interesting, if abstract, application of the Fourier transform that is used in Quantum computers. Somewhat surprisingly, it is called the “Quantum Fourier Transform”. More specifically, we will study how the Fourier Transform appears as a unitary linear operator acting on quantum states.

At the end of the day this is all just linear algebra, requiring no knowledge of actual quantum physics. Because the Quantum Fourier Transform can be somewhat mathematically abstract and also because the Fourier Transform is so easily visualized as a decomposition into various sines and cosines, I thought of coming up with a similar visualization for the Quantum Fourier Transform case (spoiler: it involves clocks).

Motivation

Before discussing in detail what the QFT is mathematically, it is useful to recap what the Fourier transform is in general. The Fourier transform is a way of transforming information from one domain to another domain. Why? Because certain operations become simpler in the transformed domain. For example, in classical signal processing, convolution of a signal (the mathematical definition of filtering) in the time domain corresponds to simple multiplication in the frequency domain.

In the graph setting, we saw that potentially complex behaviors in the edge-node representation of the graph were far more mathematically tractable when looking at the “frequency” equivalent of the graph. Eigenvectors of the graph Laplacian isolate modes of variation: low-frequency components capture global structure, while high-frequency components capture local fluctuations.

Similarly, for the Quantum Fourier Transform, we move from a bit representation of a number to a cyclical or phase representation. In the computational basis, information is stored as binary digits, essentially a sequence of ON/OFF switches taking values in $\{0,1\}$.

In this form, the data is linear and rigid. Any underlying periodic structure is hidden inside the positional encoding. Phases, however, live on the circle and are inherently cyclical. If we want to detect periodicity or modular structure, it is more natural to encode information as rotations rather than switches.

The QFT therefore plays the same conceptual role as the classical Fourier transform: it changes coordinates to a representation in which the problem’s hidden structure becomes easier to manipulate.

I might do a post later on why this is true on so many different problems. But it is not true for some problems such as when you need convolution to learn a local filter.

Useful Intuition

One of the reasons the Fourier transform in its simplest form is so
interesting is that it is so visual. In this blog post I will try to provide a nice visual explanation for
the QFT. Essentially we want to draw a connection between the binary
representation of a number and the cyclical nature of the QFT.
Fortunately, there is a nice visual representation for a binary
representation of a number on a computer, called a qubit. This
representation of a number is called a qubit.

A Useful Visualization

Convolution sits at the heart of modern machine learning—especially convolutional neural networks (CNNs)—yet the underlying mathematics is often hidden behind highly optimised implementations in PyTorch, TensorFlow, and other frameworks. As a result, many of the properties that make convolution such a powerful building block for deep learning become obscured, particularly when we try to reason about model behaviour or debug a failing architecture.

If you know the convolution theorem, a natural question arises:

Why don’t CNNs simply compute a Fourier transform of the input and kernel, multiply them in the frequency domain, and invert the result? Wouldn’t that be simpler and faster?

This blog post addresses exactly that question. We will see that:

1. FFT-based convolution is not local.
   In the Fourier domain every coefficient depends on every input pixel. This destroys the locality structure that CNNs rely on to learn hierarchical, spatially meaningful features. As a result, it breaks the very inductive bias that makes CNNs effective.

2. FFT-based convolution is not computationally cheaper in neural networks.
   Although FFTs are asymptotically efficient, they must be recomputed on every forward and backward pass—and the cost of repeatedly transforming inputs, kernels, and gradients outweighs any benefit from spectral multiplication.

By the end of this post, we’ll have a clear, explicit comparison—both in matrix form and via backpropagation—showing why CNNs deliberately perform convolution in the spatial domain. Any practioner of signal processing should also be interested in knowing when the “locality” property is useful and when it is not!

1-D Convolution

Let us start with the most basic form of convolution, the 1D convolution. In this case you have a filter (which is nothing but a sequence of numbers) that you want to multiply with your signal in order to produce another signal which is hopefully more interesting to you. For example, in your headphones, you want to multiply a set of numbers with the music signal such that the resulting signal is more music than the wailing baby 1 row behind you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import numpy as np

def conv1d_direct(x, h):
    nx, nh = len(x), len(h)
    y = np.zeros(nx+nh-1)
    for n in range(len(y)):
        for m in range(nx):
            k = n - m
            if 0 <= k < nh:
                y[n] += x[m] * h[k]
    return y

x = np.array([1.,2.,0.,-1.]) # this is the signal of music + baby wailing
h = np.array([0.5,1.,0.5]) # this is a filter that when multiplied with x makes it more music
conv1d_direct(x,h)

Convolution Theorem

This brings us to the convolution theorem wherein we can prove that the process of convolution i.e. multiplying window-wise h and x is mathematically equivalent to a simple multiplication between the fft of h and the fft of x.

1
2
3
4
5
6
7
8
9

def conv_via_fft(x,h):
    N = len(x)+len(h)-1
    X = np.fft.rfft(x,n=N)
    H = np.fft.rfft(h,n=N)
    return np.fft.irfft(X*H,n=N)

np.max(np.abs(conv1d_direct(x,h) - conv_via_fft(x,h)))
print(conv1d_direct(x,h))
print(conv_via_fft(x,h))

2-D Convolution

Just like before before we will convolve a 2D filter with a 2D signal in the spatial domain. We will then, try to do it using the FFT. We will verify that the convolution theorem does indeed work in the 2D space as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def conv2d_direct(img, ker):
    ih, iw = img.shape
    kh, kw = ker.shape
    out = np.zeros((ih+kh-1, iw+kw-1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            for m in range(ih):
                for n in range(iw):
                    km, kn = i-m, j-n
                    if 0 <= km < kh and 0 <= kn < kw:
                        out[i,j] += img[m,n] * ker[km,kn]
    return out

img = np.array([[0,0,0,0],[0,1,2,0],[0,3,4,0],[0,0,0,0]])
ker = np.array([[1,2,1],[2,4,2],[1,2,1]])/16
conv2d_direct(img,ker)

Convolution Theorem 2D

In a similar way to the 1D case instead of windowing and multiplying, we can take the fft of the signal and the kernel and simply multiply.

1
2
3
4
5
6
7
8
9
10
11

def conv2d_fft(img,ker):
    H,W = img.shape
    Kh,Kw = ker.shape
    OH,OW = H+Kh-1, W+Kw-1
    IMG = np.fft.rfft2(img, s=(OH,OW))
    KER = np.fft.rfft2(ker, s=(OH,OW))
    return np.fft.irfft2(IMG*KER, s=(OH,OW))

out_d = conv2d_direct(img,ker)
out_f = conv2d_fft(img,ker)
np.max(np.abs(out_d - out_f))

So why do NNs not use the FFT?

In a neural network, convolution is used to generate feature maps that feed into the next layer. At first glance, the convolution theorem suggests a tempting shortcut: instead of sliding a kernel spatially, we could transform both the image and kernel into the frequency domain, multiply them element-wise, and transform the result back. The output would be mathematically equivalent—so why not do this inside CNNs?

It turns out there are two fundamental reasons:

1. Neural networks care about more than just the output—they care about how the output is produced.
   During backpropagation, each filter weight is updated using gradients derived from local spatial features. This locality enables CNNs to learn hierarchies of edges, textures, shapes, and patterns.
   In the Fourier domain, however, gradients flow through global Fourier coefficients. Every frequency component depends on every pixel, so the update for a single weight depends on the entire image. This destroys the spatial locality that CNNs rely on and eliminates the inductive bias that makes them effective.

2. The FFT is not “simpler” computationally for neural networks.
   While FFTs are efficient in isolation, a CNN would need to repeatedly compute forward FFTs, spectral multiplications, and inverse FFTs—not just for the forward pass, but also for backpropagation.
   When you count actual multiplications and transforms, the FFT approach is often more expensive, especially for small kernels (e.g., 3×3, 5×5), which dominate modern architectures.

In short: CNNs avoid the Fourier domain because it removes locality and adds computational overhead—both of which undermine the very reasons convolution works so well in deep learning.

2D Spatial Convolution as a Matrix Multiply

For our next trick we will show the exact way in which your hardware actually computes convolutions. Spoiler: it will be some kind of matrix multiplication. This is quite different from the way convolution is taught in the classroom where you usually convolve with a patch of pixels in the spatial domain and roll the kernel onto the next patch nearby. In reality, this whole process is just represented as one huge matrix multiply. It is very important to think about convolution in this way, as it makes approaching complex questions easier. Since looping over pixels is not a coherent mathematical approach whose complexity is easy to compute. Once it is expressed as a matrix multiply between to matrices we can directly use a formula to compute complexity. More importantly, GPUs work fast precisely because they can parallelize this matrix multiply (as opposed to parallizing various kinds of for-loop structures).

In this section, $X$ denotes the input image. It’s worth noting that most deep-learning libraries treat the 2D and 1D cases in essentially the same way: the very first step is to reshape the image into a long vector, commonly written as $\mathrm{vec}(X)$. This operation—often implemented as im2col in the source code—unrolls local patches of the image so that convolution can be expressed as a matrix–vector multiplication.

Let the $3\times 3$ kernel we are interested in convolving be:

The valid convolution output (size $2\times 2$) is (again im2col outputs a long vector that can be then transformed to an image on the other end):

We can express the convolution as a matrix multiply:

where $T(W)$ is the Block-Toeplitz with Toeplitz Blocks (BTTB) matrix.

Expanded, the output entries are:

Loss Backpropagation in Convolution

1D Convolution Example

Let the 1D convolution be:

where:

• ($x \in \mathbb{R}^6$) is the input
• ($w \in \mathbb{R}^3$) is the kernel
• ($y \in \mathbb{R}^4$) is the output (valid convolution)

Assume a scalar loss ($L(y)$).

Step 1: Gradient w.r.t Output

Step 2: Gradient w.r.t Kernel

Construct the input Toeplitz matrix:

Then the gradient w.r.t the kernel is:

Observation: Each kernel weight sees only the local patches of the input it touches, preserving locality.

Step 3: Gradient w.r.t Input

Again, each input element only receives gradient from the outputs it contributed to.

2D Convolution Example

Only for completeness, it should be clear that 1D and 2D is handled the same way using im2col

For 2D BTTB convolution:

with scalar loss ($L(Y)$):

• Gradient w.r.t kernel:

• Gradient w.r.t input:

Observation

• Each kernel weight is influenced only by the input pixels in the patch it was applied to
• Each input pixel receives gradients only from outputs it contributed to
• This is why CNNs learn localized features efficiently.

2D Fourier Transform Convolution as Matrix Multiplies

Similar to the spatial convolution case we will represent the Fourier transform as a sequence of matrix multiplies. The recipe is as follows,

1. Fourier Transform of Kernel
2. Fourier Transform of 2D Image
3. Elementwise Multiply in the Frequency Domain
4. Inverse Fourier Transform

These matrices can get quite huge, but I thought we need to see them explicitly to make understanding them a bit easier.

We assume:

Flatten row-major:

The 2D DFT matrix for a 4×4 image (flattened row-major) is:

Where

1. Fourier Transform of the Kernel**

where $W_{padded}$ is the 3×3 kernel zero-padded to 4×4. Explicitly:

Then:

Take the first row,

2. Fourier Transform of the Image

Take the first row,

3. Multiply (Elementwise) in Frequency Space

Define the frequency-domain product:

Written explicitly:

4. Inverse Fourier Transform

To return to spatial domain:

Explicitly:

Thus the first row of the output looks like (the subscript is 11 because it will eventually be recast to an image),

We will try to focus on that first term on the RHS, $\hat{W}_1$, $\hat{X}_1$,

Compare this to $y_{11}$ from the spatial case, notice that the term $\textcolor{red}{x_{44}}$ is missing in the below expression,

Eventually these two values will be numerically the same! We know this from the convolution theorem. In the next section we will see that the contributing values matter to the gradient back propagation and that is where the two approaches will differ.

Gradient Comparison

FFT Gradient

Notice that every input pixel contributes to the gradient of $w_{11}$.

Similarly for other weights, EVERY pixel contributes to the gradient.

Gradient in the Spatial Convolution Case

Notice that each update depends only on the pixel patch that it touches!

Gradient update for scalar loss L

Computational Comparison

Spatial Convolution

Suppose:

• Input image: $X$ of size $N \times N$
• Kernel: $W$ of size $K \times K$
• Output: $Y$ of size $(N-K+1) \times (N-K+1)$

Number of multiplications

Each output pixel requires $K^2$ multiplications:

• Linear in number of pixels and kernel size.
• Memory access is local, cache-friendly.

FFT-based Convolution

Forward pass:

1. Zero-pad kernel to size $N \times N$
2. Compute 2D FFT of input and kernel: $O(N^2 \log N)$ each
3. Elementwise multiplication in Fourier domain: $O(N^2)$
4. Inverse FFT: $O(N^2 \log N)$

Total computational cost

• For small kernels $K \ll N$ $K^2 \ll \log N$, so:

• Spatial convolution is cheaper for small kernels, which is why CNNs prefer it.
• FFT becomes advantageous only for very large kernels or very large images.

TL;DR

1. Spatial convolution is efficient for small kernels and preserves locality which is crucial for CNNs to learn hierarchies.
2. FFT convolution has global interactions, destroys the local inductive bias, and is only computationally advantageous for very large kernels.

Conclusion

We have seen that spatial convolution is not only computationally more efficient but also better suited to capturing the hierarchical structure inherent in most images. For instance, a face detection algorithm may rely on local patterns such as the triangle formed by the eyes and the nose. A kernel that focuses specifically on this local arrangement is highly effective because it preserves locality.

Conversely, in domains like recommendation systems, where data may be represented as a sparse matrix of product–user interactions, capturing global patterns can be more important. Here, the “local” interactions often correspond to users with strong connections, whereas broader, global patterns reveal trends across the entire system. In such contexts, FFT-based approaches—or methods that leverage global connectivity, like graph convolutional networks—can be more appropriate.

This contrast explains why spatial CNNs excel in image-based tasks, while GCNs or FFT-based methods are more suitable for graphs representing global interactions, such as those between users and products.

References & Further Reading

• Spatial Convoluttions visualized

• “A Beginner’s Guide to Convolutions” (Colah’s Blog) – A visual, intuitive introduction to convolution and receptive fields.
  https://colah.github.io/posts/2014-07-Understanding-Convolutions/

• “The Fast Fourier Transform (FFT): Most Ingenious Algorithm Ever?” (3Blue1Brown video) – A beautiful geometric explanation of the FFT.
  https://www.youtube.com/watch?v=h7apO7q16V0&utm_source=chatgpt.com

• “Convolutional Neural Networks for Visual Recognition” (Stanford CS231n) – Gold-standard material on spatial convolution.
  https://cs231n.github.io/convolutional-networks/

Visualization & Signal Processing

• Khan Academy – Fourier Series & Fourier Transform – Visual and interactive explanations of frequency-domain thinking.
  https://www.khanacademy.org/math/differential-equations/fourier-series

• DSP Guide (Free Online Book) – Clear, practical engineering-focused intuition on convolution and transforms.
  https://www.dspguide.com/

Implementing FFT-based Convolution

• PyTorch FFT Tutorial – How PyTorch performs FFT-based convolution behind the scenes.
  https://pytorch.org/docs/stable/fft.html

• SciPy signal.fftconvolve – Practical tool frequently used for 2D FFT convolution.
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.fftconvolve.html

Graph Neural Networks & Spectral Methods

• “A Friendly Introduction to Graph Neural Networks” (Stanford) – Excellent intuition about GCNs and why they differ from CNNs.
  https://web.stanford.edu/class/cs224w/

• “Spectral Graph Convolution Explained” (Medium) – Gentle intro to graph Laplacians and filtering.
  https://medium.com/towards-data-science/spectral-graph-convolution-explained-6dddb6c1c2b0

Practical Engineering Notes

• “Why FFT Convolution is Faster” (StackOverflow discussion) – Short, practical engineering explanation.
  https://stackoverflow.com/questions/12665249/why-is-fft-convolution-faster

• “im2col and GEMM: How CNNs Are Really Implemented” (DeepLearning.ai forums) – Helps connect the maths to real-world kernels.
  https://community.deeplearning.ai/t/how-im2col-really-works/27659

This is Part 2 of the series. In Part 1 we derived the Graph Fourier Transform from the Laplacian eigenbasis and built up the one-layer Spectral GCN formulation. Here we put it to work.

Repo:
https://github.com/FranciscoRMendes/graph-networks/tree/main

Notebooks:

• GCN.ipynb — end-to-end experiment on a 3-D torus
• foundations.ipynb — mathematical derivations from DFT to irregular graphs

Application of Spectral GCN: Heat Propagation

In this section, we investigate a simple setting where a Spectral Graph Convolutional Network (GCN) performs surprisingly well: predicting heat diffusion across a toroidal mesh. Although the spectral approach is elegant and effective in the right circumstances, it also highlights several structural limitations inherent to spectral methods.

Graph Model of Heat Propagation

When we zoom into a small patch of the torus and add the connecting edges, the mesh suddenly looks like a familiar graph. This makes the role of the graph Laplacian immediately intuitive.

We simulate heat diffusion on the graph using the discrete heat equation:

where $x \in \mathbb{R}^N$ is the heat at each node and $L$ is the graph Laplacian. Starting from two random vertices with initial heat, we update the heat iteratively using a simple forward Euler scheme:

storing the state at each timestep to visualize how heat spreads across the mesh. Low-frequency modes of $L$ correspond to smooth, global patterns of heat, while high-frequency modes produce rapid, local variations.

The wraparound plot below shows this dramatically: starting from a single point source, the heat arc expands symmetrically around the ring until it meets itself on the far side.

Graph Fourier Transform of Heat Propagation

In order to get intuition for how the Fourier transform behaves on a graph, consider the distribution of heat on the graph surface.

• The heat on the graph is represented by a real number for each node (temperature or heat energy in joules), so the signal is a vector $$x \in \mathbb{R}^{N},$$ where $N$ is the number of nodes.

• If there are $N$ nodes in the graph the (combinatorial or normalized) Laplacian is an $N\times N$ matrix $$L \in \mathbb{R}^{N\times N}$$.

We use the eigendecomposition of the Laplacian to move between the vertex domain and the spectral (frequency) domain: $$L = U \Lambda U^{\top}, \qquad\Lambda = \operatorname{diag}(\lambda_1,\ldots,\lambda_N), \qquadU = [U_1\; U_2\; \cdots\; U_N],$$ with the eigenvalues ordered $0=\lambda_1 \le \lambda_2 \le \cdots \le \lambda_N$. The graph Fourier transform (GFT) and inverse GFT are $$\widehat{x} = U^{\top} x, \qquad x = U \widehat{x}$$.

To visualise single-frequency modes we simply pick individual eigenvectors $U_k$: $$\text{low-frequency mode: } x_{\text{low}} = U_{k_{\text{low}}}, \qquad\text{high-frequency mode: } x_{\text{high}} = U_{k_{\text{high}}},$$ where a natural choice is $k_{\text{low}}=2$ (the first nontrivial eigenvector) and $k_{\text{high}}=N$ (one of the largest-eigenvalue modes). Each vector $U_k$ assigns one scalar value to every vertex; plotting those values on the torus surface gives the heat-colour visualisation.

Practical steps used to create the figure

1. Build a uniform torus mesh and assemble adjacency and Laplacian $L$.

2. Compute the eigendecomposition $L=U\Lambda U^\top$ (for small / moderate meshes) or compute a selection of eigenpairs (Lanczos) for large meshes.

3. Select a low-frequency eigenvector $U_{k_{\text{low}}}$ and a high-frequency eigenvector $U_{k_{\text{high}}}$.

4. [Optional; not done here; to show smaller values in absolute terms]Normalize each eigenvector for display: $$\tilde{x} = \frac{x - \min(x)}{\max(x)-\min(x)} \quad\text{or}\quad \tilde{x} = \frac{x}{\max(|x|)},$$ so colours are comparable across panels.

5. Render the torus surface and colour each vertex by the value $\tilde{x}$ using a diverging colormap (e.g. heat) and add a colourbar showing the mapping from value to colour.

Interpreting the GFT on the torus

• Low-frequency mode. The plotted heat corresponds to $U_{k_{\text{low}}}$ (small eigenvalue). The signal varies smoothly over the torus: neighbouring vertices have similar values, representing broad, global patterns of heat.

• High-frequency mode. The plotted heat corresponds to $U_{k_{\text{high}}}$ (large eigenvalue). The signal alternates rapidly across nearby vertices, producing fine-scale oscillations around the torus that represent high-frequency, localised variations.

Spectral intuition

Recall, we expressed discrete heat propagation on a graph as,

where $L$ is the graph Laplacian and $\alpha$ is a small step size.

Using the eigendecomposition of $L$,

we can rewrite the propagation as

Comparing with the spectral graph filtering form,

we can identify the corresponding filter as

Applying a spectral filter $g(\Lambda)$ to a heat signal $x$ acts by scaling each mode:

so a low-pass filter suppresses the high-frequency panel patterns and produces smoother heat distributions, while a high-pass filter accentuates the oscillatory features visible in the high-frequency panel.

The Heat Kernel: The Analytically Correct Filter

One particularly revealing special case: the heat equation $\frac{dx}{dt} = -Lx$ has a closed-form solution. Starting from initial condition $x(0)$ and evolving for time $t$:

This means the exact spectral filter for heat diffusion at time $t$ is the heat kernel:

Smooth modes (small $\lambda$) survive nearly unchanged; oscillatory modes (large $\lambda$) are exponentially suppressed. In code this is a single line:

1
2

def heat_kernel_filter(Lambda: torch.Tensor, t: float) -> torch.Tensor:
    return torch.exp(-Lambda * t)

This provides a physics-informed baseline: after training on heat-diffusion data, the learned weights $\theta$ should approximately recover this exponential shape. Heat diffusion is therefore an ideal test case for spectral GCNs—the ground-truth spectral filter has a known closed form, so we can verify that the network has learned something physically meaningful rather than a coincidental fit.

Neural Network To Learn $g_{\theta}(\Lambda)$

We can write a spectral graph convolution / filter with learnable parameters $\theta$ as

where $U$ is the eigenvector matrix of the Laplacian, $\Lambda$ is the diagonal eigenvalue matrix, and $g_\theta(\Lambda)$ is a diagonal matrix of learnable weights acting on each eigenmode.

Fully expanding the diagonal $g_\theta(\Lambda)$:

and the Laplacian eigenvectors as column vectors $U = [U_1 \; U_2 \; \cdots \; U_n]$, $U^\top = \begin{bmatrix} U_1^\top \\ U_2^\top \\ \vdots \\ U_n^\top \end{bmatrix}$, we have

This makes it explicit that each column vector $U_i$ (the $i$-th eigenvector) is scaled by the learnable weight $\theta_i$ in the spectral domain, and then transformed back to the original node space via $U$ to produce the predicted signal $x_{t+1}$.

PyTorch Implementation

The three core operations—GFT, elementwise filtering, and inverse GFT—translate directly to PyTorch matrix operations (from graph_fourier.py):

1
2
3
4
5
6
7
8
9
10
11

def gft(x: torch.Tensor, U: torch.Tensor) -> torch.Tensor:
    """Graph Fourier Transform: x̂ = Uᵀ x"""
    return U.T @ x

def igft(x_hat: torch.Tensor, U: torch.Tensor) -> torch.Tensor:
    """Inverse GFT: x = U x̂"""
    return U @ x_hat

def spectral_filter(x, U, h):
    """Graph convolution: y = U (h ⊙ Uᵀ x)"""
    return igft(h * gft(x, U), U)

The SpectralGCN module wraps these into a learnable layer. The only trainable parameter is theta—one weight per eigenvector:

1
2
3
4
5
6
7
8
9
10
11

class SpectralGCN(nn.Module):
    def __init__(self, U, Lambda):
        super().__init__()
        self.U = U                # eigenvectors (fixed)
        self.Lambda = Lambda      # eigenvalues (fixed)
        self.theta = nn.Parameter(torch.ones(U.shape[0]))

    def forward(self, x):
        x_hat = self.U.T @ x                          # GFT
        filtered = torch.sigmoid(self.theta) * x_hat  # learned filter
        return self.U @ filtered                       # inverse GFT

The graph Laplacian is assembled from the mesh topology in build_graph.py. Each triangular face contributes three undirected edges, self-loops are added for stability, and the degree-normalised form is computed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

def create_adjacency_matrix(mesh):
    vertices, faces = mesh.vertices, mesh.faces
    num_nodes = len(vertices)
    adj = np.zeros((num_nodes, num_nodes))
    for face in faces:
        for i in range(3):
            for j in range(i + 1, 3):
                adj[face[i], face[j]] = 1
                adj[face[j], face[i]] = 1
    adj = torch.tensor(adj, dtype=torch.float32) + torch.eye(num_nodes)
    deg = adj.sum(dim=1)
    D_inv_sqrt = torch.diag(1.0 / torch.sqrt(deg))
    L = torch.eye(num_nodes) - D_inv_sqrt @ adj @ D_inv_sqrt
    return adj, L

Why Use A Neural Network?

Two motivating examples illustrate the practical usefulness of such a model:

• Partial Observations from Sensors
  In many real-world systems, heat or pressure sensors are only available at a small subset of points. The experiment uses 750 sensors placed on the torus using farthest-point sampling (FPS)—an algorithm that greedily picks each next sensor as the vertex farthest from all already-chosen sensors, ensuring uniform coverage of the surface rather than random clustering:

1
2
3
4
5
6
7
8
9
10

def farthest_point_sampling(vertices, k):
    dist = torch.full((len(vertices),), float('inf'))
    sampled = [torch.randint(0, len(vertices), (1,)).item()]
    for _ in range(1, k):
        dist = torch.minimum(dist,
                             torch.norm(vertices - vertices[sampled[-1]], dim=1))
        sampled.append(torch.argmax(dist).item())
    return sampled

sensor_indices = farthest_point_sampling(mesh.vertices, num_sensors=750)

All non-sensor nodes are zeroed during training over 40 timesteps of diffusion (step size $\alpha = 0.4$). We train the Spectral GCN using only these sparse observations, yet the learned model reconstructs and predicts the heat field across all vertices on the mesh—effectively turning sparse sensor readings into a full-field prediction.

• Generalization to a New Geometry
  One might hope that a model trained on one torus could be applied to a slightly different torus. Unfortunately, this is generally not possible in the GCN setting. The eigenvectors of the Laplacian form the coordinate system in which the model operates, and even small geometric changes produce different Laplacian spectra. As a result, the learned spectral filters are not transferable across meshes. This is a fundamental drawback of spectral GCNs. However, we shall see that the GCN framework inspires frameworks that do not suffer from this drawback.

Stability Issues And Normalization

While the Spectral GCN learns the qualitative behaviour of heat diffusion, raw training often leads to unstable predictions. After several steps, the overall temperature of the mesh may drift upward or downward, even though heat diffusion is energy-conserving. This is because the neural network makes predictions locally without obeying the laws of physics such as the law of conservation of energy. Which is why our predictions are on average “hotter” than the actual.

Two practical fixes alleviate this:

• Eigenvalue Normalization. Applying a sigmoid or similar squashing function to the learned spectral filter ensures that each frequency component is damped in a physically plausible range. This prevents the model from amplifying high-frequency modes, which would otherwise cause heat values to explode.

• Energy Conservation. After each predicted step, the total heat can be renormalized to match the physical energy of the system. This ensures that although the shape of the prediction is learned by the model, the magnitude remains consistent with diffusion dynamics. Empirically, this correction dramatically improves long-horizon stability.

Training Results

Training the SpectralGCN for 300 epochs with the Adam optimizer on 40 diffusion steps yields rapid convergence:

The model achieves roughly a 10× loss reduction in the first 50 epochs, then plateaus near $1.5 \times 10^{-5}$. The steep initial descent reflects the fact that most heat-diffusion structure is captured by the lowest few eigenmodes — the tail of the training curve squeezes out residual error from higher-frequency components.

Overall, the Spectral GCN provides a compact and interpretable model for heat propagation on a fixed mesh and performs remarkably well given its simplicity. However, its reliance on the Laplacian eigenbasis also limits its ability to generalize across geometries, motivating the need for more flexible spatial or message-passing approaches in applications where the underlying mesh may change.

Efficient Spectral Filtering: Chebyshev Approximation

The full spectral GCN has a critical bottleneck: computing $U$ costs $O(N^3)$ and must be recomputed whenever the graph changes. An elegant fix avoids the eigendecomposition entirely by approximating the spectral filter as a truncated Chebyshev polynomial:

where $T_k$ is the $k$-th Chebyshev polynomial, $\tilde{L} = \frac{2}{\lambda_{\max}} L - I$ is a scaled Laplacian with spectrum in $[-1, 1]$, and $K$ is the polynomial order (typically 2–5). Two key advantages over the full eigendecomposition:

• Complexity: $O(K \cdot |E|)$ per forward pass instead of $O(N^2)$, since applying $\tilde{L}$ is a sparse matrix–vector multiply.
• Locality: A degree-$K$ polynomial only aggregates $K$-hop neighbourhoods—spatial support is finite and interpretable.

The scaled Laplacian is assembled with a numerical stability term (from build_graph.py):

1
2
3
4
5

def create_adjacency_matrix_tilde(mesh):
    # ... build normalized Laplacian L as above ...
    lambda_max = torch.linalg.eigvals(L).real.max()
    L_tilde = (2.0 / lambda_max) * L - torch.eye(num_nodes)
    return adj, L_tilde

This is the computational insight that made GCNs practical at scale. It directly leads to the Kipf & Welling (2017) formulation, which further simplifies to $K=1$ with $\lambda_{\max} \approx 2$, collapsing the polynomial to a single propagation step: $g_\theta(L) \approx \theta (I + D^{-1/2} A D^{-1/2})$. The tradeoff is expressiveness: the polynomial can only represent $K$-local filters, whereas the full spectral GCN can learn an arbitrary per-eigenvalue response.

Label Propagation: Pinning the Source

Heat diffusion has a fundamental property that makes it unsuitable for certain tasks: it is energy-conserving. The source node loses heat as it spreads — after enough steps, the torus temperature equilibrates and all memory of the original source is lost. This is physically correct, but it is the wrong model when you want to say “this node is permanently important.”

Label propagation fixes this by re-injecting the source label at every step:

where $\tilde{A} = D^{-1} A$ is the row-normalised adjacency (each row sums to 1), $Y$ is the initial label vector (1 at source nodes, 0 elsewhere), and $\alpha \in (0,1)$ controls the trade-off between neighbour averaging and staying close to the known labels. Because $(1-\alpha)Y$ is added back every iteration, the source node never loses its value — the label is clamped at 1 throughout diffusion.

1
2
3
4
5
6
7
8

def simulate_label_propagation(A_norm, source_idx, steps=100, alpha=0.9):
    n = A_norm.shape[0]
    Y = torch.zeros(n, 1)
    Y[source_idx] = 1.0
    x = Y.clone()
    for _ in range(steps):
        x = alpha * (A_norm @ x) + (1.0 - alpha) * Y
    return x

The figure below shows both algorithms running from the same source on the torus. The top two metric panels tell the story clearly:

• Heat diffusion (orange): peak node value decays to near zero as energy spreads out — the source cools.
• Label propagation (purple): peak node value stays pinned at 1 throughout — the source stays “hot”.

Both methods spread information, but they answer different questions. Heat diffusion asks “where does energy go?” — the answer changes over time and the source eventually forgets it was special. Label propagation asks “how influential is this source?” — the source retains its identity and neighbouring nodes accumulate influence proportional to their graph proximity.

Cold Start: Recommender Systems

What does spectral graph theory have to do with recommender systems? Once we view user–item behaviour as a graph, the connection becomes natural. In the spectral domain, low-frequency Laplacian eigenvectors capture broad, mainstream purchasing patterns, while high-frequency components represent niche tastes and micro-segments. Matrix Factorisation (MF) implicitly applies a low-pass filter: embeddings vary smoothly across the item–item graph, meaning MF emphasises low-frequency structure. But MF breaks down for cold-start items because an isolated item contributes no coliorative signal.

In contrast, a spectral GCN applies a learned filter $$T x = g(L)x = U\ g(\Lambda) U^\top x$$

In general, we represent user-item interactions as a bipartite graph i.e. edges do not exist between products. In this scenario, even the GCN cannot help since very clearly for a node to get assigned a value it must be connected to at least one other node. However, the graph formulation provides a very intuitive way to fix this issue! Simply add edges between products that are similar to each other. Then low frequency patterns are bound to filter into the node even if high frequency niche patterns will not.

Matrix factorization resolves this issue by using side information (such as product attributes), which asserts similarity from external data. In my previous post I argued that you can achieve something similar through an intuitive edge-addition approach—even though it amounts to inserting 1’s into a fairly unintuitive matrix and factorizing it.

Conclusion

In this post, we’ve put the spectral GCN machinery to work: simulating heat diffusion on a toroidal mesh, training the model from sparse sensor readings, and verifying that the learned filter converges to the analytically correct heat kernel $h(\lambda,t) = e^{-\lambda t}$. We contrasted heat diffusion with label propagation — the pinned-source variant that is better suited to spreading known labels through a graph. We also covered the Chebyshev approximation that eliminates the $O(N^3)$ eigendecomposition bottleneck, and connected the whole framework to the cold-start problem in recommender systems.

While spectral GCNs shine on fixed graphs and structured problems, they also come with caveats: eigen-decompositions can be expensive, and filters are not always transferable across different geometries. Nevertheless, the framework provides intuition and a foundation for more flexible spatial or message-passing approaches.

So, whether you’re modeling heat flowing across a mesh or figuring out what obscure sock a new customer might want next, spectral graph theory shows that Fourier Transforms can take you a long way.

In my next post, I will deal with the remaining fundamental limitation of spectral GCNs:

• Adding a new node / transferring information to a similar graph (spatial and message-passing approaches)

Although I’ve focused much more on the ML side of consulting projects—and I really enjoy it—I’ve often had to dust off my statistician hat to measure how well the algorithms I build actually perform. Most of my experience in this area has been in verifying that recommendation engines, once deployed, truly deliver value. In this article, I’ll explore some key themes in AB Testing. While, I tried to be as general as possible, I did drill down on specific concepts that are particularly salient to recommender systems.

I thoroughly enjoy the “measurement science” behind these challenges; it’s a great reminder that classic statistics is far from obsolete. In practice, it also lets us make informed claims based on simulations, even if formal proofs aren’t immediately available. I’ve also included some helpful simulations.

Basic Structure of AB Testing

AB Testing begins on day zero, often in a room full of stakeholders, where your task is to prove that your recommendation engine, feature (like a new button), or pricing algorithm really works. Here, the focus shifts from the predictive power of machine learning to the causal inference side of statistics. (Toward the end of this article, I’ll also touch briefly on causal inference within the context of ML.)

Phase 1: Experimental Context

• Define the feature under analysis and evaluate whether AB testing is necessary. Sometimes, if a competitor is already implementing the feature, testing may not be essential; you may simply need to keep pace.

• Establish a primary metric of interest. In consulting projects, this metric often aligns closely with engagement fees, so it’s critical to define it well.

• Identify guardrail metrics—these are typically independent of the experiment (e.g., revenue, profit, total rides, wait time) and represent key business metrics that should not be negatively impacted by the test.

• Set a null hypothesis, $H_0$ (usually representing a zero effect size on the main metric). Consider what would happen without the experiment, which may involve using non-ML recommendations or an existing ML recommendation in recommendation engine contexts.

• Specify a significance level, $\alpha$, which is the maximum probability of rejecting the null hypothesis when it is true, commonly set at 0.05. This value is conventional but somewhat arbitrary, and it’s challenging to justify because humans often struggle to assign accurate probabilities to risk.

• Define the alternative hypothesis, $H_1$, indicating the minimum effect size you hope to observe. For example, in a PrimeTime pricing experiment, you’d specify the smallest expected change in your chosen metric, such as whether rides will increase by hundreds or by 1%. This effect size is generally informed by prior knowledge and reflects the threshold at which the feature becomes worthwhile.

• Choose a power level, $1 - \beta$, usually set to 0.8. This means there is at least an 80% chance of rejecting the null hypothesis when $H_1$ is true.

• Select a test statistic with a known distribution under both hypotheses. The sample average of the metric of interest is often a good choice.

• Determine the minimum sample size required to achieve the desired power level $1 - \beta$ with all given parameters.

Before proceeding, it’s crucial to recognize that many choices, like those for $\alpha$ and $\beta$, are inherently subjective. Often, these parameters are predefined by an existing statistics or measurement science team, and a “Risk” team may also weigh in to ensure the company’s risk profile remains stable. For instance, if you’re testing a recommendation engine, implementing a new pricing algorithm, and cutting costs simultaneously, the risk team might have input on how much overall risk the company can afford. This subjectivity often makes Bayesian approaches appealing, driving interest in a Bayesian perspective for AB Testing.

Phase 2: Experiment Design

With the treatment, hypothesis, and metrics established, the next step is to define the unit of randomization for the experiment and determine when each unit will participate. The chosen unit of randomization should allow accurate measurement of the specified metrics, minimize interference and network effects, and account for user experience considerations.The next couple of sections will dive deeper into certain considerations when designing an experiment, and how to statistically overcome them. In a recommendation engine context, this can be quite complex, since both treatment and control groups share the pool of products, it is possible that increased purchases from the online recommendation can cause the stock to run out for people who physically visit the store. So if we see the control group (i.e. the group not exposed to the new recommender system) buying more competitor products (competitors to the products you are recommending) this could simply be because the product was not available and the treatment was much more effective than it seemed!

Unit of Randomization and Interference

Now that you have approval to run your experiment, you need to define the unit of randomization. This can be tricky because often there are multiple levels at which randomization can be carried out for example, you can randomize your app experience by session, you could also randomize it by user. This leads to our first big problem in AB testing. What is the best unit of randomization? And what are the pitfalls of picking the wrong unit? Sometimes, the unit is picked for you, you simply may not have recommendation engine data at the exact level you want. A unit is often hard to conceptualize, it is easy to think that it is one user. But one user at different points in their journey through the app can be treated as different units.

Example of Interference

Interference is a huge problem in recommendation engines for most retail problems. Let me walk you through an interesting example we saw for a large US retailer. We were testing whether a certain product (high margin obviously!) was being recommended to users. The treatment group was shown the product and the control group was not. The metric of interest was the number of purchases of a basket of high margin products. The control group purchased the product at a rate of $\tau_0\%$ and the treatment group purchased the product at a rate of $\tau_t\%$. The experiment was significant at the $0.05$ level. However, after the experiment we noticed that the difference in sales closed up to $\tau_t - \tau_0 = \delta\%$. This was because the treatment group was buying up the stock of the product and the control group was not because they could not. Sometimes the act of being recommended a product was a kind of treatment in itself. This is a non-classical example of interference. This is a good reason to use a formal causal inference framework to measure the effect of the treatment. One way to do this is DAGs, which I will discuss later. The best way to run an experiment like this is to randomize by region. However, this is not always possible since regions share the same stock. But I think you get the idea.

Robust Standard Errors in AB Tests

You can fix interference by clustering at the region level but very often this leads to another problem of its own. The unit of treatment allocation is now fundamentally bigger than the unit at which you are conducting the analysis. We do not really recommend products at the store level, we recommend products at the user level. So while we assign treatment and control at the store level we are analyzing effects at the user level. As a consequence we need to adjust our standard errors to account for this. This is where robust standard errors come in. In such a case, the standard errors you calculate for the average treatment effect are
lower than what they truly are. And this has far-reaching effects for power, effect size and the like.

Recall, the variance of the OLS estimator

You can analyze the variance matrix under various assumptions to estimate, $$\epsilon \epsilon’ = \Omega$$

Under homoscedasticity,

Under heteroscedasticity (Heteroscedastic robust standard errors),

And finally under clustering, $$\Omega = \begin{bmatrix} \epsilon_1^2 & \epsilon_1 \epsilon_2 & 0 & 0 & \dots & 0 & 0 \\ \epsilon_1 \epsilon_2 & \epsilon_2^2 & 0 & 0 & & 0 & 0 \\ 0 & 0 & \epsilon_3^2 & \sigma^2_{34} & & 0 & 0 \\ 0 & 0 & \sigma^2_{34} & \epsilon_3^2 & & 0 & 0 \\ \vdots & & & & \ddots & & \vdots \\ 0 & 0 & 0 & 0 & & \epsilon_{n-1}^2 & \sigma^2_{n-1,n} \\ 0 & 0 & 0 & 0 & \dots & \sigma^2_{n-1,n} & \epsilon_n^2 \\ \end{bmatrix}$$

The cookbook, for estimating $\Omega$ is therefore multiplying your matrix $\epsilon\epsilon'$ with some kind of banded matrix that represents your assumption $C$,

Range of Clustered Standard Errors

Where the left boundary is where no clustering occurs and all errors are independent and the right boundary is where the clustering is very strong but variance between clusters is zero. It is fair to ask, why we need to multiply by a matrix of assumptions $C$ at all, the answer is that the assumptions scale the error to tolerable levels, such that the error is not too large or too small. By pure coincidence, it is possible to have high covariance between any two observations, whether to include it or not is predicated by your assumption matrix $C$.

Power Analysis

I have found that power analysis is an overlooked part of AB Testing, in Consulting you will probably have to work with the existing experimentation team to make sure the experiment is powered correctly. There is usually some amount of haggling and your tests are likely to be underpowered. There is a good argument to be made about overpowering your tests (such a term does not exist in statistics, who would complain about that), but this usually comes with some risk to guardrail metrics, thus you are likely to under power your tests when considering a guardrail metric. This is OKAY, because remember the $0.05$ level is a convention, and the $0.8$ power level is also a convention that by definition err on the side of NOT rejecting the null. So if you see an effect with an underpowered test you do have some latitude to make a claim while reducing the significance level of your test.

Power analysis focuses on reducing the probability of accepting the null hypothesis when the alternative is true. To increase the power of an A/B test and reduce false negatives, three key strategies can be applied:

• Effect Size: Larger effect sizes are easier to detect. This can be achieved by testing bold, high-impact changes or trying new product areas with greater potential for improvement. Larger deviations from the baseline make it easier for the experiment to reveal significant effects.

• Sample Size: Increasing sample size boosts the test’s accuracy and ability to detect smaller effects. With more data, the observed metric tends to be closer to its true value, enhancing the likelihood of detecting genuine effects. Adding more participants or reducing the number of test groups can improve power, though there’s a balance to strike between test size and the number of concurrent tests.

• Reducing Metric Variability: Less variability in the test metric across the sample makes it easier to spot genuine effects. Targeting a more homogeneous sample or employing models that account for population variability helps reduce noise, making subtle signals easier to detect.

Finally, experiments are often powered at 80% for a postulated effect size — enough to detect meaningful changes that justify the new feature’s costs or improvements. Meaningful effect sizes depend on context, domain knowledge, and historical data on expected impacts, and this understanding helps allocate testing resources efficiently.

Power as a function of effect size and sample size

In an A/B test, the power of a test (the probability of correctly detecting a true effect) is influenced by the effect size, sample size, significance level, and pooled variance. The formula for power,

Where,

• $\Delta$ is the **Minimum Detectable Effect (MDE)**, representing the smallest effect size we aim to detect.
• $z_{1-\alpha/2}$ is the critical z-score for a significance level$\alpha$ (e.g., 1.96 for a 95% confidence level).
• $\sigma_{\text{pooled}}$ is the **pooled standard deviation

  ** of the metric across groups, representing the combined variability.

• $n$ is the **sample size per group**.
• $\Phi$ is the **cumulative distribution function

  ** (CDF) of the standard normal distribution, which gives the probability that a value is below a given z-score.

Understanding the Role of Pooled Variance

• Power decreases as the pooled variance ($\sigma_{\text{pooled}}^2$) increases. Higher variance increases the "noise" in the data, making it more challenging to detect the effect (MDE) relative to the variation.

• When pooled variance is low, the test statistic (difference between groups) is less likely to be drowned out by noise, so the test is more likely to detect even smaller differences. This results in higher power for a given sample size and effect size.

Practical Implications

In experimental design:

• Reducing $\sigma_{\text{pooled}}$ (e.g., by choosing a more homogeneous sample) improves power without increasing sample size.

• If $\sigma_{\text{pooled}}$ is high due to natural variability, increasing the sample size $n$ compensates by lowering the standard error $\left(\frac{\sigma_{\text{pooled}}}{\sqrt{n}}\right)$, thereby maintaining power.

Difference in Difference

Randomizing by region to solve interference can create a new issue: regional trends may bias results. If, for example, a fast-growing region is assigned to the treatment, any observed gains may simply reflect that region’s natural growth rather than the treatment’s effect.

In recommender system tests aiming to boost sales, retention, or engagement, this issue can be problematic. Assigning a growing region to control and a mature one to treatment will almost certainly make the treatment group appear more effective, potentially masking the true impact of the recommendations.

Linear Regression Example of DiD

To understand the impact of a new treatment on a group, let’s consider an example where everyone in group $G$ receives a treatment at time $t_e$. Our goal is to measure how this treatment affects outcomes over time.

First, we’ll introduce some notation:

Define $\mathbb{1}_A(x)$, which indicates if $x$ belongs to a specific set $A$:

Let $T = \{t : t > t_e\}$, which represents the period after treatment. We can use this to set up a few key indicators:

• $\mathbb{1}_{T(t)} = 1$ if the time $t$ is after the treatment, and $0$ otherwise.
• $\mathbb{1}_{G(i)} = 1$ if an individual $i$ is in group $G$, meaning they received the treatment.
• if they are both $1$ then they refer to those in the treatment group during the post-treatment period.

Using these indicators, we can build a simple linear regression model:

$$y_{it} = \beta_0 + \beta_1 \mathbb{1}_{T(t)} + \beta_2 \mathbb{1}_{G(i)} + \beta_3 \mathbb{1}_{T(t)} \mathbb{1}_{G(i)}+ \epsilon_{it}$$

In this model, the coefficient $\beta_3$ is the term we’re most interested in. It represents the Difference-in-Differences (DiD) effect: how much the treatment group’s outcome changes after treatment compared to the control group’s change in the same period. In other words, $\beta_3$ provides a clearer picture of the treatment’s direct impact, isolating it from other factors.

For this model to work reliably, we rely on the parallel trends assumption: the control and treatment groups would have followed similar paths over time if there had been no treatment. Although the initial levels of $y_{it}$ can differ between groups, they should trend together in the absence of intervention.

Testing the Parallel Trends Assumption

You can always test whether your data satisfies the parallel trends assumption by looking at it. In a practical environment, I have never really tested this assumption, for two big reasons (it is also why I personally think DiD is not a great method):

• If you need to test an assumption in your data, you are likely to have a problem with your data. If it is not obvious from some non-statistical argument or plot etc you are unlikely to be able to convince a stakeholder that it is a good assumption.
• The data required to test this assumption, usually invalidates its need. If you have data to test this assumption, you likely have enough data to run a more sophisticated model than DiD (like CUPED).

Having said all that, here are some ways you can test the parallel trends assumption:

• Visual Inspection:

  • Plot the average outcome variable over time for both the treatment and control groups, focusing on the pre-treatment period. If the trends appear roughly parallel before the intervention, this provides visual evidence supporting the parallel trends assumption.

  • Make sure any divergence between the groups only occurs after the treatment.

• Placebo Test:

  • Pretend the treatment occurred at a time prior to the actual intervention and re-run the DiD analysis. If you find a significant “effect” before the true treatment, this suggests that the parallel trends assumption may not hold.

  • Use a range of pre-treatment cutoff points and check if similar differences are estimated. Consistent non-zero results may indicate underlying trend differences unrelated to the actual treatment.

• Event Study Analysis (Dynamic DiD):

  • Extend the DiD model by including lead and lag indicators for the treatment.

  • If pre-treatment coefficients (leads) are close to zero and non-significant, it supports the parallel trends assumption. Large or statistically significant leads could indicate violations of the assumption.

• Formal Statistical Tests:

  • Run a regression on only the pre-treatment period, introducing an interaction term between time and group to test for significant differences in trends:

  • If the coefficient $\alpha_3$ on the interaction term is close to zero and statistically insignificant, this supports the parallel trends assumption. A significant $\alpha_3$ would indicate a pre-treatment trend difference, which would challenge the assumption.

• Covariate Adjustment (Conditional Parallel Trends):

  • If parallel trends don’t hold unconditionally, you might adjust for observable characteristics that vary between groups and influence the outcome. This is a more relaxed “conditional parallel trends” assumption, and you could check if trends are parallel after including covariates in the model.

If you can make all this work for you, great, I never have. In the dynamic world of recommendation engines (especially always ‘’online’’ recommendation engines) it is very difficult to find a reasonably good cut-off point for the placebo test. And the event study analysis is usually not very useful since the treatment is usually ongoing.

Peeking and Early Stopping

Your test is running, and you’re getting results—some look good, some look bad. Let’s say you decide to stop early and reject the null hypothesis because the data looked good. What could happen? Well, you shouldn’t. In short, you’re changing the power of the test. A quick simulation can show the difference: with early stopping or peeking, your rejection rate of the null hypothesis is much higher than the 0.05 you intended. This isn’t surprising since increasing the sample size raises the chance of rejecting the null when it’s true.

The benefits of early stopping aren’t just about self-control. It can also help prevent a bad experiment from affecting critical guardrail metrics, letting you limit the impact while still gathering needed information. Another example is when testing expendable items. Think about a magazine of bullets: if you test by firing each bullet, you’re guaranteed they all work—but now you have no bullets left. So you might rephrase the experiment as, How many bullets do I need to fire to know this magazine works?

In consulting you are going to peek early, you have to live with it. For one reason or another, a bug in production, an eager client whatever the case, you are going to peek, so you better prepare accordingly.

Simulated Effect of Peeking on Experiment Outcomes

(a) Without Peeking: $\frac{3}{100}$ reject null, $\alpha=0.05$

(b) With Peeking: $\frac{29}{100}$ reject null, $\alpha=0.05$

Under a given null hypothesis, we run 100 simulations of experiments and record the z-statistic for each. We do this once without peeking and let the experiments run for $1000$ observations. In the peeking case, we stop whenever the z-statistic crosses the boundary but only after $100$th observation.

Sequential Testing for Peeking

The Sequential Probability Ratio Test (SPRT) compares the likelihood ratio at the $n$-th observation, given by:

$$\Lambda_n = \frac{L(H_1 \mid x_1, x_2, \dots, x_n)}{L(H_0 \mid x_1, x_2, \dots, x_n)}$$

where $L(H_0 \mid x_1, x_2, \dots, x_n)$ and $L(H_1 \mid x_1, x_2, \dots, x_n)$ are the likelihood functions under the null hypothesis $H_0$ and the alternative hypothesis $H_1$, respectively.

The test compares the likelihood ratio to two thresholds, $A$ and $B$, and the decision rule is:

$$\text{If } \Lambda_n \geq A, \text{ accept } H_1,$$ $$\text{If } \Lambda_n \leq B, \text{ accept } H_0,$$ $$\text{If } B < \Lambda_n < A, \text{ continue sampling}.$$

The thresholds $A$ and $B$ are determined based on the desired error probabilities. For a significance level $\alpha$ (probability of a Type I error) and power $1 - \beta$ (probability of detecting a true effect when $H_1$ is true), the thresholds are given by:

$$A = \frac{1 - \beta}{\alpha}, \quad B = \frac{\beta}{1 - \alpha}.$$

Normal Distribution

This test is in practice a lot easier to carry out for certain distributions like the normal distribution, assume an unknown mean $\mu$ and known variance $\sigma^2$

$$\begin{aligned} H_0: \quad & \mu = 0 , \\ H_1: \quad & \mu = 0.1 \end{aligned}$$$$\mathcal L(\mu) = \left( \frac{1}{\sqrt{2 \pi} \sigma } \right)^n e^{- \sum_{i=1}^{n} \frac{(X_i - \mu)^2}{2 \sigma^2}}$$$$\Lambda(X) = \frac{\mathcal L (0.1, \sigma^2)}{\mathcal L (0, \sigma^2)} = \frac{e^{- \sum_{i=1}^{n} \frac{(X_i - 0.1)^2}{2 \sigma^2}}}{e^{- \sum_{i=1}^{n} \frac{(X_i)^2}{2 \sigma^2}}}$$

The sequential rule becomes the recurrent sum, $S_i$ (with $S_0=0$) $$S_{i} = S_{i-1} + \log(\Lambda_{i})$$

With the stopping rule

• $S_i \geq b$ : Accept $H_1$
• $S_i\geq a$ : Accept $H_0$
• $a

$a \approx \log {\frac {\beta }{1-\alpha }} \quad \text{and} \quad b \approx \log {\frac {1-\beta }{\alpha }}$

There is another elegant method outlined in Evan Miller’s blog post, which I will not go into here but just state it for brevity (it is also used at Etsy, so there is certainly some benefit to it). It is a very good read and I highly recommend it.

• At the beginning of the experiment, choose a sample size $N$.
• Assign subjects randomly to the treatment and control, with 50% probability each.
• Track the number of incoming successes from the treatment group. Call this number $T$.
• Track the number of incoming successes from the control group. Call this number $C$.
• If $T−C$ reaches $2\sqrt{N}$, stop the test. Declare the treatment to be the winner.
• If $T+C$ reaches $N$, stop the test. Declare no winner.
• If neither of the above conditions is met, continue the test.

Using these techniques you can “peek” at the test data as it comes in and decide to stop as per your requirement. This is very useful as the following simulation using this more complex criteria shows. Note that what you want to verify is two things,

• Does early stopping under the null hypothesis, accept the null in approximately $\alpha$ fraction of simulations once the stopping criteria is reached and does it do so
  fast.

• Does early stopping under the alternative reject the null hypothesis in $1-\beta$ fraction of simulations and does it do so
  fast.

The answer to these two questions is not always symmetrical, and it seems that we need more samples to reject the null (case 2) versus accept it case 1. Which is as it should be! But in both cases, as the simulations below show, you need a significantly fewer number of samples than before.

CUPED and Other Similar Techniques

Recall, our diff-in-diff equation, $$Y_{i,t} = \alpha + \beta D_i + \gamma \mathbb I (t=1) + \delta D_i * \mathbb I (t=1) + \varepsilon_{i,t}$$

Diff in Diff is nothing but CUPED for $\theta=1$. I state this without proof. I was not able to find a clear one any where.

Consider the auto-regression with control variates regression equation, $$Y_{i, t=1} = \alpha + \beta D_i + \gamma Y_{i, t=0} + \varepsilon_i$$ This is also NOT equivalent to CUPED, nor is it a special case. Again, I was not able to find a good proof anywhere.

Multiple Hypotheses

In most of the introduction, we set the scene by considering only one hypotheses. However, in real life you may want to test multiple hypotheses at the same time.

• You may be testing multiple hypotheses even if you did not realize it, such as over time. In the example of early stopping you are actually checking multiple hypotheses. One at every time point.

• You truly want to test multiple features of your product at the same time and want to run one test to see if the results got better.

Regression Model Setup

We consider a regression model with three treatments, $D_1$, $D_2$, and $D_3$, to study their effects on a continuous outcome variable, $Y$. The model is specified as: $$Y = \beta_0 + \beta_1 D_1 + \beta_2 D_2 + \beta_3 D_3 + \epsilon$$ where:

• $Y$ is the outcome variable,
• $D_1$, $D_2$, and $D_3$ are binary treatment indicators (1 if the treatment is applied, 0 otherwise),
• $\beta_0$ is the intercept,
• $\beta_1$, $\beta_2$, and $\beta_3$ are the coefficients representing the effects of treatments $D_1$, $D_2$, and $D_3$, respectively,
• $\epsilon$ is the error term, assumed to be normally distributed with mean 0 and variance $\sigma^2$.

Hypotheses Setup

We aim to test whether each treatment has a significant effect on the outcome variable $Y$. This involves testing the null hypothesis that each treatment coefficient is zero.

The null hypotheses are formulated as follows: $$H_0^{(1)}: \beta_1 = 0$$ $$H_0^{(2)}: \beta_2 = 0$$ $$H_0^{(3)}: \beta_3 = 0$$

Each null hypothesis represents the assumption that a particular treatment (either $D_1$, $D_2$, or $D_3$) has no effect on the outcome variable $Y$, implying that the treatment coefficient $\beta_i$ for that treatment is zero.

Multiple Hypothesis Testing

Since we are testing three hypotheses simultaneously, we need to control for the potential increase in false positives. We can use a multiple hypothesis testing correction method, such as the
Bonferroni correction or the Benjamini-Hochberg procedure.

Bonferroni Correction

With the Bonferroni correction, we adjust the significance level $\alpha$ for each hypothesis test by dividing it by the number of tests $m = 3$. If we want an overall significance level of $\alpha = 0.05$, then each individual hypothesis would be tested at: $$\alpha_{\text{adjusted}} = \frac{\alpha}{m} = \frac{0.05}{3} = 0.0167$$

Benjamini-Hochberg Procedure

Alternatively, we could apply the Benjamini-Hochberg procedure to control the False Discovery Rate (FDR). The procedure involves sorting the p-values from smallest to largest and comparing each p-value $p_i$ with the threshold: $$p_i \leq \frac{i}{m} \cdot \alpha$$ where $i$ is the rank of the p-value and $m$ is the total number of tests. We declare all hypotheses with p-values meeting this criterion as significant. This framework allows us to assess the individual effects of $D_1$, $D_2$, and $D_3$ while properly accounting for multiple hypothesis testing.

Variance Reduction: CUPED

When analyzing the effectiveness of a recommender system, sometimes your metrics are skewed by high variance in the metric i.e. $Y_i$. One easy way to fix this is by using the usual outlier removal suite of techniques. However, outlier removal is a difficult thing to statistically define, and very often you may be losing “whales”. Customers who are truly large consumers of a product. One easy way to do this would be to normalize the metric by its mean, i.e. $Y_i = \frac{Y_i}{\bar Y}$. Any even better way to do this would be to normalize the metric by that users own mean, i.e. $Y_i = \frac{Y_i}{\bar Y_i}$. This is the idea behind CUPED.

Consider, the regression form of the treatment equation,

$$Y_{i, t=1} = \alpha + \beta D_i + \varepsilon_i$$

Assume you have data about the metric from before, and have values $Y_{i,t=0}$. Where the subscript denoted the $i$ individuals outcome, before the experiment was even run, $t=1$.

$$\hat Y^{cuped}\_{t=1} = \theta\bar Y_{t=0} + \theta \mathbb E [Y_{t=0} ]$$

This is like running a regression of $Y_{t=1}$ on $Y_{t=0}$.

$$Y\_{i, t=1} = \theta Y_{i, t=0} + \hat Y^{cuped}_{i, t=1}$$

Now, use those residuals in the treatment equation above,

$$\hat Y^{cuped}_{i, t=1} = \alpha + \beta D_i + \varepsilon_i$$

And then estimate the treatment effect.

The statistical theory behind CUPED is fairly simple and setting up the regression equation is not difficult. However, in my experience, choosing the right window for pre-treatment covariates is extremely difficult, choose the right window and you reduce your variance by a lot. The right window depends a lot on your business. Some key considerations,

• Sustained purchasing behavior is a key requirement. If the $Y_{t=0}$ is not a good predictor of $Y_{t=1}$ for the interval $t=0$ to $t=1$ then the variance of $Y^{cuped}$ will be high. Defeating the purpose.
• Longer windows come with computational costs.
• In practice, because companies are testing things all the time you could have noise left over from a previous experiment that you need to randomize/ control for.

Simulating CUPED

One way you can guess a good pre-treatment window is by simulating the treatment effect for various levels of MDEs (the change you expect to see in $Y_i$) and plot the probability of rejecting the alternative hypothesis if it is true i.e. Power.

MDE vs Power for 2 Different Metrics

So you read off your hypothesized MDE and Power, and then every point to the left of that is a good window. As an example, lets say you know your MDE to be $3\%$ and you want a power of $0.8$, then your only option is the 16 week window. Analogously, if you have an MDE of $5\%$ and you want a power of $0.8$, then the conventional method (with no CUPED) is fine as you can attain an MDE of $4\%$ with a power of $0.8$. Finally, if you have an MDE of $4\%$ and you want a power of $0.8$ then a 1 week window is fine.

Finally, you can check that you have made the right choice by plotting the variance reduction factor against the pre-period (weeks) and see if the variance reduction factor is high.

CUPED is a very powerful technique, but if I could give one word of advice to anyone trying to do it, it would be this:
get the pre-treatment window
right. This has more to do with business intelligence than with statistics. In this specific example longer windows gave higher variance reduction, but I have seen cases where a “sweet spot” exists.

Variance Reduction: CUPAC

As it turns out we can control variance, by other means using the same principle as CUPED. The idea is to use a control variate that is not a function of the treatment. Recall, the regression equation we ran for CUPED, $$Y\_{i, t=1} = \theta Y_{i, t=0} + \hat Y^{cuped}_{i, t=1}$$ Generally speaking, this is often posed as finding some $X$ that is uncorrelated with the treatment but correlated with $Y$.

$$Y\_{i, t=1} = \theta X_{i, t=0} + \hat Y^{cuped}_{i, t=1}$$

You could use
any $X$ that is uncorrelated with the treatment but correlated with $Y$. An interesting thing to try would be to fit a highly non-linear machine learning model to $Y_t$ (such as random forest, XGBoost) using a set of observable variables $Z_t$, call it $f(Z_t)$. Then use $f(Z_t)$ as your $X$.

$$Y\_{i, t=1} = \theta f(Z_{i,t=1}) + \hat Y^{cuped}_{i, t=1}$$

Notice here two things, - that $f(Y)$ is not a function of $D_i$ but is a function of $Y_i$. - that $f(Z)$ does not (necessarily) need any data from $t=0$ to be calculated, so it is okay, if
no pre-treatment data
exists! - if pre-treatment data exists then you can use it to fit $f(Z)$ and then use it to predict $Y$ at $t=1$ as well, so it can only enhance the performance of your fit and thereby reduce variance even more.

If you really think about it, any process to create pre-treatment covariates inevitably involves finding some $X$ highly correlated with outcome and uncorrelated with treatment and controlling for that. In CUPAC we just dump all of that into one ML model and let the model figure out the best way to control for variance using all the variables we threw in it.

I highly recommend CUPAC over CUPED, it is a more general technique and can be used in a wider variety of situations. If you really want to, you can throw $Y_{t=0}$ into the mix as well!

A Key Insight: Recommendation Engines and CUPAC/ CUPED

Take a step back and think about what $f(Z)$ is
really saying in context of a recommender system, it is saying given some $Z$ can I predict my outcome metric. Let us say the outcome metric is some $G(Y)$, where $Y$ is sales.

$$G(Y) = G(f(Z)) + \varepsilon$$

What is a recommender system? It takes some $Z$ and predicts $Y$.

$$\hat Y = r(Z) + \varepsilon'$$$$G(\hat Y) = G(r(Z)) + \varepsilon''$$

This basically means that a pretty good function to control for variance is a recommender system itself! Now you can see why CUPAC is so powerful, it is a way to control for variance using a recommender system itself. You have all the pieces ready for you. HOWEVER! You cannot use the recommender system you are currently testing as your $f(Z)$, that would be mean that $D_i$ is correlated with $f(Z)$ and that would violate the assumption of uncorrelatedness. Usually, the existing recommender system (the pre-treatment one) can be used for this purpose. The finally variable $Y^{cupac}$ then has a nice interpretation it is not the difference between what people
truly did and the recommended value, but rather the difference between the two recommender systems! Any model is a variance reduction model, it is just a question of how much variance it reduces. Since the existing recommender system is good enough it is likely to reduce a lot of variance. If it is terrible (which is why they hired you in the first place) then this approach is unlikely to work. But in my experience, existing recommendations are always pretty good in the industry it is a question of finding those last few drops of performance increase.

Conclusion

The above are pretty much all you can expect to find in terms of evaluating models in Consulting. In my experience considering all the possibilities that would undermine your test are worth thinking about before embarking on the AB test.

]]> https://franciscormendes.com/2024/11/08/consulting-ab-testing/ 2024-11-08T05:00:00.000Z

A/B testing in management consulting: frequentist and Bayesian methods, multiple testing corrections, and the specific challenges of testing recommender systems in a live production environment.

2026-05-20T16:32:39.866Z Francisco Romaldo Fernandes Mendes Introduction

It is often difficult to speak about things like singularities because of their prevalence in pop culture. Oftentimes a concept like this takes a life of its own, forever ingrained in ones imagination as a still from a movie (for me this is that scene from Inception where they encounter Gargantua for the first time). Like many concepts in theoretical physics, popular culture is often better at bringing them into light than it is at bringing them into focus. In this article I will try to explain in simple terms what a singularity is and how that relates to physical reality. As always, I will give an exact example of the singularity by means of an equation. At the end, once the mathematics is clear, I will try to explain what the physical reality of the singularity is.

Mathematical Singularities

Singularity of $f(x) = \frac{1}{x}$

1. Behavior of the Function:

$$f(x) = \frac{1}{x}$$

- As $x \to 0^+$ (approaching from the positive side): $$f(x) \to +\infty$$ - As $x \to 0^-$ (approaching from the negative side): $$f(x) \to -\infty$$

At $x = 0$, the function becomes infinitely large (or small), making $x = 0$ a singularity. This is a pole of the function where the value tends to infinity.

2. Undefined at the Singularity:

The function $f(x) = \frac{1}{x}$ is undefined at $x = 0$, which is the point of discontinuity.

In mathematics, singularities are not a problem.

Physics Singularities

The singularity of a black hole can be described by the Schwarzschild metric, which is the solution to Einstein’s field equations for a non-rotating, uncharged black hole. The Schwarzschild metric is given by:

$$ds^2 = - \left( 1 - \frac{2GM}{r c^2} \right) c^2 dt^2 + \left( 1 - \frac{2GM}{r c^2} \right)^{-1} dr^2 + r^2 \left( d\theta^2 + \sin^2 \theta \, d\phi^2 \right)$$

Where:

• $ds^2$ is the spacetime interval,
• $c$ is the speed of light,
• $G$ is the gravitational constant,
• $M$ is the mass of the black hole,
• $r$ is the radial coordinate,
• $\theta$ and $\phi$ are angular coordinates.

Be careful though these are not polar co-ordinates, these are coordinates for the Schwarzschild metric. They are a kind of nested spherical coordinate system, this does not seem to affect the solution but helpful to know.

The singularity occurs at $r = 0$. As $r \to 0$, the term $\frac{2GM}{r c^2}$ grows without bound, leading to an infinite curvature of spacetime. This represents the physical singularity of the black hole.

Additionally, the $g_{tt}$ component of the Schwarzschild metric, which is the time-time component, becomes singular as $r \to 0$:

$$g_{tt} = - \left( 1 - \frac{2GM}{r c^2} \right)$$

As $r \to 0$, $g_{tt} \to -\infty$, indicating the breakdown of spacetime and the presence of a singularity.

You can create another singularity by setting $r = 2GM/c^2$ in the metric, this is the event horizon of the black hole. This is the point at which light can no longer escape the black hole. However, this is solely a mathematical singularity, since you can still define the metric at this point by a change of coordinates. One such set of coordinates is the Kruskal-Szekeres coordinates, which are used to describe the Schwarzschild metric in a way that is regular across the event horizon.

The Schwarzschild metric in Kruskal-Szekeres coordinates is given by:

$$ds^2 = \frac{32 G^3 M^3}{r c^6} e^{-r/2GM/c^2} \left( -dU dV \right) + r^2 \left( d\theta^2 + \sin^2 \theta \, d\phi^2 \right)$$

where $r$ is a function of $U$ and $V$, implicitly determined by:

$$U V = \left( \frac{r}{r_s} - 1 \right) e^{r / r_s}$$

Here, $r_s$ is the Schwarzschild radius:

$$r_s = \frac{2GM}{c^2}$$

The coordinate singularity at $r = r_s$ in the Schwarzschild metric is removed by transforming to Kruskal-Szekeres coordinates, and the metric remains regular across the event horizon.

Another Physics Singularity

Again, starting from yet another solution for the field equations we can derive FLRW metric (Friedmann-Lemaître-Robertson-Walker metric) which describes the universe as a whole. The words homogenous and isotropic, effectively mean that instead of considering each individual planet in the universe as an actual individual body, we consider them to be individual particles in a fluid (in fact, the FLRW metric considers each galaxy to be a particle). We do this so that we can use equations for fluids to simplify the stress energy tensor $T$ in the Field Equations. Our strategy to solve the field equations is as follows,

1. Assume the universe is some kind of fluid (so basically zoom out till all the galaxies look like a fluid)
2. From 1, you can write down the stress energy tensor $T_{\mu\nu}$ for the fluid, this is a simple equation (This is $0$ for the Schwarzschild metric, and for many other useful metrics, so we never really had this problem before, but when you zoom out you need it)

The FLRW metric, which describes a homogeneous and isotropic universe, is given by:

$$ds^2 = - c^2 dt^2 + a(t)^2 \left( \frac{dr^2}{1 - k r^2} + r^2 d\theta^2 + r^2 \sin^2 \theta \, d\phi^2 \right)$$

Where:

• $ds^2$ is the spacetime interval,
• $c$ is the speed of light,
• $t$ is the cosmic time,
• $a(t)$ is the scale factor of the universe,
• $r$ is the radial coordinate,
• $\theta$ and $\phi$ are angular coordinates,
• $k$ is the curvature of space, which can be $-1$, $0$, or $1$.
• The scale factor $a(t)$ describes how the universe expands or contracts with time.
• The curvature parameter $k$ determines the geometry of space: negative curvature for $k = -1$, flat curvature for $k = 0$, and positive curvature for $k = 1$.

Friedmann Equations Recap

The Big Bang is represented in the Friedmann equations as a singularity at the beginning of time when the scale factor $a(t) $approaches zero. This signifies an initial state of infinite density, temperature, and curvature.

The Friedmann equations in cosmology are derived from Einstein’s field equations for a homogeneous and isotropic universe. Assuming zero cosmological constant ($\lambda = 0$), they are:

1. First Friedmann Equation: $$ \left( \frac{\dot{a}}{a} \right)^2 = \frac{8 \pi G}{3} \rho - \frac{k}{a^2} $$

2. Second Friedmann Equation (acceleration equation): $$ \frac{\ddot{a}}{a} = - \frac{4 \pi G}{3} \left( \rho + \frac{3p}{c^2} \right) $$

3. Continuity Equation (conservation of energy): $$ \dot{\rho} + 3 \frac{\dot{a}}{a} \left( \rho + \frac{p}{c^2} \right) = 0 $$

where: - $a(t) $is the scale factor (the “size” of the universe at a given time $t $), - $\rho $is the energy density, - $p $is the pressure, - $G $is the gravitational constant, - $k $is the curvature parameter ($k = 0 $for a flat universe, $k = +1 $for closed, and $k = -1 $for open).

Representation of the Big Bang Singularity

In the context of the Friedmann equations, the Big Bang is identified by the conditions: - $a(t) \to 0$as $t \to 0$, - $\rho \to \infty $as $a(t) \to 0$(implying infinite density and temperature), - Curvature becomes infinite, signaling a physical singularity.

Explanation Using the First Friedmann Equation

In the first Friedmann equation: $$\left( \frac{\dot{a}}{a} \right)^2 = \frac{8 \pi G}{3} \rho - \frac{k}{a^2}$$

As $t \to 0 $: - The scale factor $a(t) $approaches zero. - For a positive energy density $\rho$, the term $\frac{\dot{a}}{a}$(known as the Hubble parameter) goes to infinity, meaning the rate of expansion is initially unbounded. - If $a \to 0$, then the energy density $\rho \to \infty $since $\rho $is inversely related to the volume of the universe.

Thus, at $a = 0 $, the universe is in a state of infinite density and infinite curvature, which we identify as the Big Bang singularity.

Continuity Equation and Energy Conservation

The continuity equation: $$\dot{\rho} + 3 \frac{\dot{a}}{a} \left( \rho + \frac{p}{c^2} \right) = 0$$

implies that as $a(t) $approaches zero, the rapid change in the scale factor causes the energy density $\rho $to increase sharply, reinforcing the singularity concept.

Physical Interpretation

At $t = 0 $, when the scale factor $a(t) = 0 $, the energy density $\rho $theoretically becomes infinite, meaning all mass, energy, and curvature are compressed into a single point. This condition marks the beginning of the universe, as described by the Big Bang theory, before which classical descriptions of time and space may no longer apply due to quantum gravitational effects.

In short, the Big Bang singularity in the Friedmann equations marks the initial state of the universe at $t = 0 $, where $a = 0 $, density and temperature are infinite, and classical general relativity predicts a breakdown in spacetime structure.

Connection to Reality

While all of the above can be found in a basic undergraduate textbook, I think the goal of me writing this post was to have a collection of examples of singularities both from mathematics and physics to reinforce the idea of reality. While in the mathematical examples, the $x=0$ does not represent an actual place that we can go and take measurements of $y$, but what if we did? What if we indeed knew a physical place in the world, where the function $\frac{1}{x}$ really described the behavior of the world. This is not hard, you could imagine this as the share that each person gets (of a cake or similarly sweet treat) if there are $x$ people. If there are $3$ people, each person gets $\frac{1}{3}$ of a share. What does it mean to have $0$ people? This is the kind of question that the mathematical singularity is trying to answer. But it is physically impossible to have $0$ people, so the singularity is not a real place. If you had $0$ people and a cake, the question of dividing it does not make sense. In much the same way, the singularity of the Schwarzschild metric is not a real place, it is a place where the equations break down. This does not mean that some wild stuff happens at the singularity, it means that the equations we are using to describe the world are not valid at that point. This is the same as saying that the function $\frac{1}{x}$ is not defined at $x=0$. Very often in movies, the singularity is portrayed as a place where the laws of physics break down. This is not true it is just that the laws of physics defined by the equations work everywhere else but not at that point. This could mean one of two things, 1. The equations are not valid at that point, so we need to find new equations that are valid at that point. 2. Some wild stuff happens at that point, and we need to find out what that is. And rework our equations to include that.

But simply by looking at the equations, we cannot say which of the two is true. We need to go out and measure the world to find out.

References

https://diposit.ub.edu/dspace/bitstream/2445/59759/1/TFG-Arnau-Romeu-Joan.pdf

]]> https://franciscormendes.com/2024/10/22/singularities/ 2024-10-22T04:00:00.000Z

From 1/x to the Schwarzschild radius: what singularities mean in analysis and general relativity, and whether they correspond to anything physically real.

2026-04-18T15:56:33.610Z Francisco Romaldo Fernandes Mendes Introduction

When do I use “old-school” ML models like matrix factorization and when do I use graph neural networks?

Can we do something better than matrix factorization?

Why can’t we use neural networks? What is matrix factorization anyway?

These are just some of the questions, I get asked whenever I start a recommendation engine project. Answering these questions requires a good understanding of both algorithms, which I will try to outline here. The usual way to understand the benefit of one algorithm over the other is by trying to prove that one is a special case of the other.

While it can be shown that a Graph Neural Network can be expressed as a matrix factorization problem. This matrix is not easy to interpret in the usual sense. Contrary to popular belief, matrix factorization (MF) is not “simpler” than a Graph Neural Network (nor is the opposite true). To make matters worse, the GCN is actually more expensive to train since it takes far more cloud compute than does MF. The goal of this article is to provide some intuition as to when a GCN might be worthwhile to try out.

This article is primarily aimed at data science managers with some background in linear algebra (or not, see next sentence) who may or may not have used a recommendation engine package before. Having said that, if you are not comfortable with some proofs I have a key takeaways subsection in each section that should form a good basis for decision making that perhaps other team members can dig deep into.

Key Tenets of Linear Algebra and Graphs in Recommendation Engine design

The key tenets of design come down to the difference between a graph and a matrix. The linking between graph theory and linear algebra comes from the fact that ALL graphs come with an adjacency matrix. More complex versions of this matrix (degree matrix, random walk graphs) capture more complex properties of the graph. Thus you can usually express any theorem in graph theory in matrix form by use of the appropriate matrix.

1. The Matrix Factorization of the interaction matrix (defined below) is the most commonly used form of matrix factorization. Since this matrix is the easiest to interpret.
2. Any Graph Convolutional Neural Network can be expressed as the factorization of some matrix, this matrix is usually far removed from the interaction matrix and is complex to interpret.
3. For a given matrix to be factorized, matrix factorization requires fewer parameters and is therefore easier to train.
4. Graphical structures are easily interpretable even if matrices expressing their behavior are not.

Tensor Based Methods

In this section, I will formulate the recommendation engine problem as a large tensor or matrix that needs to be “factorized”.
In one of my largest projects in Consulting, I spearheaded the creation of a recommendation engine for a top 5 US retailer. This project presented a unique challenge: the scale of the data we were working with was staggering. The recommendation engine had to operate on a 3D tensor, made up of products × users × time. The sheer size of this tensor required us to think creatively about how to scale and optimize the algorithms.

Let us start with some definitions, assume we have $n_u, n_v$ and $n_t$, users, products and time points respectively.

1. User latent features, given by matrix $U$ of dimension $n_u \times r$ and each index of this matrix is $u_i$

2. Products latent features, given by matrix $V$, of dimensions $n_v \times r$ and each index of this matrix is $v_j$

3. Time latent features given by Matrix $T$, of dimensions $n_t \times r$ and each index of this matrix is $t_k$

4. Interaction given by $y_{ijk}$ in the tensor case, and $y_{ij}$ in the matrix case. Usually this represents either purchasing decision, or a rating (which is why it is common to name this $r_{ijk}$) or a search term. I will use the generic term “interaction” to denote any of the above.

In the absence of a third dimension one could look at it as a matrix factorization problem, as shown in the image below,

Increasingly, however, it is important to take other factors into account when designing a recommendation system, such as context and time. This has led to the tensor case being the more usual case.

This means that for the $i$th user, $j$th product at the $k$th moment in time, the interaction $y_{ijk}$ is functionally represented by the dot product of these $3$ matrices, $$y_{ijk} \approx u_i\cdot v_j\cdot t_k$$ An interaction $y_{ijk}$ can take a variety of forms, the most common approach, which we follow here will be, $y_{ijk} = 1$, if the $i$th user interacted with the $j$th product at that $k$th instance. Else, $0$. But other more complex functional forms can exist, where we can use the rating of an experience at that moment, where instead of $y \in {0,1}$ we can have a more general form $y \in \mathcal{R}$. Thus this framework is able to handle a variety of interaction functions. A question we often get is that this function is inherently linear since it is the dot product of multiple matrices. We can handle non-linearity in this framework as well, via the use of non-linear function (a.k.a an activation function). $$y_{ijk} \approx {1- \exp^{u_i\cdot v_j\cdot t_k }}$$ Or something along those lines. However, one of the attractions of this approach is that it is absurdly simply to set up.

Side Information

Very often in a real word use case, our clients often have information that they are eager to use in a recommendation system. These range from user demographic data that they know from experience is important, to certain product attribute data that has been generated from a different machine learning algorithm. In such a case we can integrate that into the equation given above,

$$y_{ijk} \approx u_i\cdot v_j\cdot t_k + v_j \cdot v'_j + u_i \cdot u'_i$$

Where, $u'_i, v'_i$ are attributes for users and products that are known beforehand. Each of these vectors are rows in $U', V'$, that are called “side-information" matrices.

Optimization

We can then set up the following loss function,

$$\mathcal{L}(X, U, V, W_t, U', V') = \| X - (U \cdot V \cdot W_t) \|^2 + \lambda_1 \| U \cdot U' - X_u \|^2 + \lambda_2 \| V \cdot V' - X_p \|^2 + \lambda_3 (\| U \|^2 + \| V \|^2 + \| W_t \|^2)$$

Where:

• $\lambda_1$ and $\lambda_2$ are regularization terms for the alignment with side information.
• $\lambda_3$ controls the regularization of the latent matrices $U$, $V$, and $W_t$.

• The first term is the reconstruction loss of the tensor, ensuring that the interaction between users, products, and time is well-represented.

• The second and third terms align the latent factors with the side information for users and products, respectively.

Tensor Factorization Loop

For each iteration:

1. Compute the predicted tensor using the factorization: $$\hat{X} = U \cdot V \cdot W_t$$

2. Compute the loss using the updated loss function.

3. Perform gradient updates for $U$, $V$, and $W_t$.

4. Regularize the alignment between $U$, $V$ with $U'$ and $V'$

5. Repeat until convergence.

Key Takeaway

Matrix factorization allows us to decompose a matrix into two low-rank matrices, which provide insights into the properties of users and items. These matrices, often called embeddings, either embed given side information or reveal latent information about users and items based on their interaction data. This is powerful because it creates a representation of user-item relationships from behavior alone.

In practice, these embeddings can be valuable beyond prediction. For example, clients often compare the user embedding matrix

$U$ with their side information to see how it aligns. Interestingly, clustering users based on$U$ can reveal new patterns that fine-tune existing segments. Rather than being entirely counter-intuitive, these new clusters may separate users with subtle preferences, such as distinguishing between those who enjoy less intense thrillers from those who lean toward horror. This fine-tuning enhances personalization, as users in large segments often miss out on having their niche behaviors recognized.\

Mathematically, the key takeaway is the following equation (at the risk of overusing a cliche, this is the $e=mc^2$ of the recommendation engines world)

$$y_{ij} = u_i'v_j + \text{possibly other regularization terms}$$

Multiplying the lower dimensional representation of the $i$th user and the $j$th item together yields a real number that represents the magnitude of the interaction. Very low and its not going to happen, and very high means that it is. These two vectors are the “deliverable”! How we got there is irrelevant. Turns out there are multiple ways of getting there. One of them is the Graph Convolutional Network. In recommendation engine literature (particularly for neural networks) embeddings are given by $H$, in the case of matrix factorization, $H$ is obtained by stacking $U$ and $V$,

$$H = [U \hspace{5 pt} V]$$

Extensions

You do not need to stick to the simple multiplication in the objective function, you can do something more complex,

$$\min \sum_{(i,j) \in E} y_{ij} \left( \log \sigma(U_i^T V_j) + (1 - y_{ij}) \log (1 - \sigma(U_i^T V_j)) \right)$$

The above objective function is the LINE embedding. Where $\sigma$ is some non-linear function.

Interaction Tensors as Graphs

One can immediately view a the interactions between users and items as a bipartite graph, where an edge is present only if the user interacts with that item. It is immediately obvious that we can embed the interactions matrix inside the adjacency matrix, noting that there are no edges between users and there are no edges between items.

The adjacency matrix $A$ can be represented as:

$$A = \begin{bmatrix}0 & R \\R^T & 0\end{bmatrix}$$

Recall, the matrix factorization $R = UV^T$,

$$A \approx\begin{bmatrix}0 & UV^T \\VU^T & 0\end{bmatrix}$$

where:

• $R$ is the user-item interaction matrix (binary values: 1 if a user has interacted with an item, 0 otherwise),
• $R^T$ is the transpose of $R$, representing item-user interactions.

For example, if $R$ is the following binary interaction matrix:

$$R = \begin{bmatrix}1 & 0 & 1 \\1 & 1 & 0\end{bmatrix}$$

Note, here that $R$ could have contained real numbers (such as ratings etc.) but the adjacency matrix is strictly binary. Using the weighted adjacency matrix is perfectly “legal”, but has mathematical implications that we will discuss later. Thus, the adjacency matrix $A$ becomes:

$$A = \begin{bmatrix}0 & 0 & 0 & 1 & 0 & 1 \\0 & 0 & 0 & 1 & 1 & 0 \\1 & 1 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 & 0 \\1 & 0 & 0 & 0 & 0 & 0\end{bmatrix}$$

Matrix Factorization of Adjacency Matrix

Now you could use factorize, $$A \approx LM^T$$ And then use the embeddings $L$ and $M$, but now $L$ represents embeddings both for users and items (as does $M$). However, this matrix is much bigger than $R$ since the top left and bottom right block matrix are $0$. You are much better off using the $R = UV^T$ formulation to quickly converge on the optimal embeddings. The key here is that factorizing this matrix is roughly equivalent to factorizing the $R$ matrix. This is important because the adjacency matrix plays a key role in the graphical convolutional network.

What are the REAL Cons of Matrix Factorization

Matrix factorization offers key advantages in a consulting setting by quickly assessing the potential of more advanced methods on a dataset. If the user-item matrix performs well, it indicates useful latent user and item embeddings for predicting interactions. Additionally, regularization terms help estimate the impact of any side information provided by the client. The resulting embeddings, which include both interaction and side information, can be used by marketing teams for tasks like customer segmentation and churn reduction.
First, let me clarify some oft quoted misconceptions about matrix factorization disadvantages versus GCNs,

1. User item interactions are a simple dot product ($\hat y_{ij} = u_i'v_j$) and is therefore not linear. This is not true, even in the case of a GCN the final prediction is given by a simple dot product between the embeddings.

2. Matrix factorization cannot use existing features . This is probably due to the fact that matrix factorization was popularized by the simple Netflix case, where only user-item matrix was specified. But in reality, very early in the development of matrix factorization, all kinds of additional regularization terms such as bias, side information etc. were introduced. The side information matrices are where you can specify existing features (recall, $y_{ij} = u_i'v_j + \text{possibly other regularization terms}$).

3. Cannot handle cold start Neither matrix factorization nor neural networks can handle the cold start problem very well. However, this is not an unfair criticism as the neural network is better, but this is more as a consequence of its truly revolutionary feature, which I will discuss under its true advantage.

4. Higher order interactions this is also false, but it is hard to see it mathematically. Let me outline a simple approach to integrate side information. Consider the matrix adjacency matrix $A$, $A^2$ gives you all edges with length $2$, such that $A + A^2$ represents all nodes that are at most $2$ edges away. You can then factorize this matrix to get what you want. This is not an unfair criticism either as multiplying such a huge matrix together is not advised and neither is it the most intuitive method.

The biggest problem with MF is that a matrix is simply not a good representation of how people interact with products and each other. Finding a good mathematical representation of the problem is sometimes the first step in solving it. Most of the benefits of a graph convolutional neural network come as a direct consequence of using a graph structure not from the neural network architecture. The graph structure of a user-item behavior is the most general form of representation of the problem.

1. Complex Interactions - In this structure one can easily add edges between users and between products. Note in the matrix factorization case, this is not possible since $R$ is only users x items. To include more complex interactions you pay the price with a larger and larger matrix.

2. Graph Structure - Perhaps the most visually striking feature of graph neural networks is that they can leverage graph structure itself (see Figure 4). Matrix factorization cannot do so easily

3. Higher order interactions can be captured more intuitively than in the case of matrix factorization

Before implementing a GCN, it’s important to understand its potential benefits. In my experience, matrix factorization often provides good results quickly, and moving to GCNs makes sense only if matrix factorization has already shown promise. Another key factor is the size and richness of interactions. If the graph representation is primarily bipartite, adding user edges may not significantly enhance the recommender system. In retail, edges sometimes represented families, but these structures were often too small to be useful—giving different recommendations to family members like $11$ and $1$ is acceptable since family ties alone don’t imply similar consumption patterns. However, identifying influencers, such as nodes with high degrees connected to isolated nodes, could guide targeted discounts for products they might promote.

I would be remiss, if I did not add that ALL of these issues with matrix factorization can be fixed by tweaking the factorization in some way. In fact, a recent paper Unifying Graph Convolutional Networks as Matrix Factorization by Liu et. al. does exactly this and shows that this approach is even better than a GCN. Which is why I think that the biggest advantage of the GCN is not that it is “better” in some sense, but rather the richness of the graphical structure lends itself naturally to the problem of recommending products, even if that graphical structure can then be shown to be equivalent to some rather more complex and less intuitive matrix structure. I recommend the following experiment flow :

A Simple GCN model

Let us continue on from our adjacency matrix $A$ and try to build a simple ML model of an embedding, we could hypothesize that an embedding is linearly dependent on the adjacency matrix.

$$H = f(AWX + I_nWX)$$

The second additive term bears a bit of explaining. Since the adjacency matrix has a $0$ diagonal, a value of $0$ get multiplied with the node’s own features $x\in X$. To avoid this we add the node’s own feature matrix $X$ using the diagonal matrix.

We need to make another important adjustment to $A$, we need to divide each term in the adjacency matrix by the degree of each node. $$\tilde{A} = A + I_n$$ $$A \equiv \tilde{D}^\frac{1}{2}\tilde{A}\tilde{D}^\frac{1}{2}$$ At the risk of abusing notation, we redefine $A$ as some normalized form of the adjacency matrix after edges connecting each node with itself have been added to the graph. I like this notation because it emphasizes the fact that you do not need to do this, if you suspect that normalizing your nodes by their degree of connectivity is not important then you do not need to do this step (though it costs you nothing to do so). In retail, the degree of a user node refers to the number of products they consume, while the degree of a product node reflects the number of customers it reaches. A product may have millions of consumers, but even the most avid user node typically consumes far fewer, perhaps only hundreds of products.

Here $X = [X_{u}, X_{i}$]. $$H = [U V]$$

Here we can split the equations by the subgraphs for which they apply to,

$$H_u = f(A_u W_u X_u)$$ $$H_v = f(A_v W_v X_v)$$

Note the equivalence the matrix case, in the matrix case we have to stack it ourselves because of the way we set up the matrix, but in the case of a GCN $H$ is already $m\times n$ and represents embeddings of both users and items.

The likelihood of an interaction is,

$$\hat y_{ij} = H_u^T H_v$$

The loss function is,

$$L = \sum_{(u, i) \in \mathcal{I}} \left( y_{ui} - \hat{y}_{ui} \right)^2$$

We can substitute the components of $H$ to get a tight expression for optimizing loss,

$$L = \sum_{(u, i) \in \mathcal{I}} \left( y_{ui} - f(A_u W_u X_u)^T f(A_v W_v X_v)\right)^2$$

This is the main “result” of this blog post that you can equally look at this one layer GCN as a matrix factorization problem of the user-item interaction matrix but with the more complex looking low rank matrices on the right. In this sense, you can always create a matrix factorization that equates to the loss function of a GCN.

You can update parameters using SGD or some other technique. I will not get into that too much in this post.

Understanding the GCN equation

Equations 1 and 2 are the most important equations in the GCN framework. $W$ is some $(m+n) \times d$ set of weights that learn how to embed or encode the information contained in $X$ into $H$. For this one layer model, we are only considering values from the nodes that are one edge away, since the value of $h_i$ is only dependent on all the $x_j$‘s that are directly connected to it and its own $x_i$. However, if you then apply this operation again, $H$ now has all the information contained in all the nodes connected to it in its own $h_i$ but also so does every other nodes $h_k$.

$$H^0 = f(AW^0X + I_nW^0X)$$$$H^1 = f(AW^1H^0 + I_nW^1H^0)$$

More succinctly, $$H^1 = f(AW^1 f(AW^0X + I_nW^0X)+ I_nW^1H^0)$$

Equivalence to Matrix Factorization for a one layer GCN

You could just as easily have started with two random matrices $U$ and $V$ and optimize them using your favorite optimization algorithm and end up with the likelihood for interaction function,

$$\hat y_{ij} = U^T V \equiv H_u^T H_v$$

So you get the same outcome for a one layer GCN as you would from matrix factorization. Note that, it has been proved that even multi-layer GCNs are equivalent to matrix factorization but the matrix being factorized is not that easy to interpret.

Key Takeaways

The differences between MF and GCN really begin to take form when we go into multi-layerd GCNs. In the case of the one layer GCN the embeddings of $H^0$ are only influenced by the nodes connected to it. Thus the features of a customer node will be only influenced by the products that they buy, similarly, the product node will be only influenced by the customers who by them. However, for deeper neural networks :

1. 2 layer: every customer’s embedding is influenced by the embeddings of the products they consume and the embeddings of other customers of the products they consume. Similarly, every product is influenced by the customers who consume that product as well as by the products of the customers who consume that product.

2. 3 layer: every customers embedding is influenced by the products they consume, other customers of the products they consume and products consumed by other customers of the products they consume. Similarly, every product is influenced by the consumers of that product, as well as products of consumers of that product as well as products consumed by consumers of that product.

You can see where this is going, in most practical applications, there are only so many levels you need to go to get a good result. In my experience $2$ is the bare minimum (because $1$ is unlikely to do better than an MF, in fact they are equivalent) and $3$ is about how deep you can feasibly go without exploding the number of training parameters.

That leads to another critical point when considering GCNs, you really pay a price (in blood, mind you) for every layer deep you go. Consider the one layer case, you really have $n\times d$ and $n\times d'$ parameters to learn, because you have to learn both the weight matrix $W$ and the matrix of embeddings $H$. But the MF case you directly learn $H$. So if you were only going to go one layer deep you might as well use matrix factorization.

Going the other way, if you are considering more than $3$ layers the reality of the problem (in my usual signal processing problems this would be “physical” laws) i.e. the behavioral constraints mean that more than 3 degrees deep of influence (think about what point 3 would mean for a $5$ layer network) is unlikely to be supported by any theoretical evidence of consumer behavior.

Final Prayer and Blessing

I would like for the reader of this to leave with a better sense of the relationship between matrix factorization and GCNs. Like most neural network based models we tend to think of them as a black box and a black box that is “better”. However, in the one layer GCN case we can see that they are equal, with the GCN in fact having more learnable parameters (therefore more cost to train).
Therefore, it makes sense to use $2$ layers or more. But when using more, we need to justify them either behaviorally or with expert advice.

How to go from MF to GCNs

1. Start with matrix factorization of the user-item matrix, maybe add in context or time. If it performs well and recommendations line up with non-ML recommendations (using base segmentation analysis), that means the model is at least somewhat sensible.

2. Consider doing a GCN next if the performance of MF is decent but not great. Additionally, definitely try GCN if you know (from marketing etc) that the richness of the graph structure actually plays a role in the prediction. For example, in the sale of Milwaukee tools a graph structure is probably not that useful. However, for selling Thursday Boots which is heavily influenced by social media clusters, the graph structure might be much more useful.

3. Interestingly, the MF matrices tend to be very long and narrow (there are usually thousands of users and most companies have far more users than they have products. This is not true for a company like Amazon (300 million users and 300 million products). But if you have a long narrow matrix that is sparse you are not too concerned with computation since at worst you have $m\times n \approx O(n), m<

It is worthwhile in a consulting environment to always start with a simple matrix factorization, the GCN for simplicity of use and understanding but then find a matrix structure that approximates only the most interesting and rich aspects of the graph structure that actually influence the final recommendations.

References

https://jonathan-hui.medium.com/graph-convolutional-networks-gcn-pooling-839184205692
https://tkipf.github.io/graph-convolutional-networks/ https://openreview.net/forum?id=HJxf53EtDr
https://distill.pub/2021/gnn-intro/ https://jonathan-hui.medium.com/graph-convolutional-networks-gcn-pooling-839184205692

]]> https://franciscormendes.com/2024/09/28/graph-convolutional-neural-network-and-matrix-factorization/ 2024-09-28T04:00:00.000Z

The mathematical essentials for recommender systems: from matrix factorization via SVD to graph convolutional networks, and why the spectral perspective unifies both approaches.

2026-05-21T15:47:44.765Z Francisco Romaldo Fernandes Mendes Decomposition of a Convolutional layer

In Part I I described (in some detail) what it means to decompose a matrix multiply into a sequence of low rank matrix multiplies, and Part II extended that to convolutional kernels and rank selection. We can go further still for general tensors, though this is somewhat less easy to see since tensors in higher dimensions are quite hard to visualize.
Recall, the matrix formulation,

$$Y = XW + b = XUSV' + b$$

Where $U$ and $V$ are the left and right singular vectors of $W$ respectively. The idea is to approximate $W$ as a sum of outer products of $U$ and $V$ of lower rank.
Now instead of a weight matrix multiplication $y = WX + b$ we have a kernel operation, $y = K\circledast X + b$ where $\circledast$ is the convolution operation. The idea is to approximate $K$ as a sum of outer products of $U$ and $V$ of lower rank.
Interestingly, you can also think about this as a matrix multiplication, by creating a Toplitz matrix version of $K$ , call it $K'$ and then doing $y = K'X + b$. But this comes with issues as $K'$ is much much bigger than $K$. So we just approach it as a convolution operation for now.

Convolution Operation

At the heart of it, a convolution operation takes a smaller cube subset of a “cube” of numbers (also known as the map stack) multiplies each of those numbers by a fixed set of numbers (also known as the kernel) and gives a single scalar output. Let us start with what each “slice” of the cube really represents.

Now that we have a working example of the representation, let us try to visualize what a convolution is.

A convolution operation takes a subset of the RGB image across all channels and maps it to one number (a scalar), by multiplying the cube of numbers with a fixed set of numbers (a.k.a kernel, not pictured here) and adding them together.A convolution operation multiplies each pixel in the image across all $3$ channels with a fixed number and add it all up.

Low Rank Approximation of Convolution

Now that we have a good idea of what a convolution looks like, we can now try to visualize what a low rank approximation to a convolution might look like. The particular kind of approximation we have chosen here does the following 4 operations to approximate the one convolution operation being done.

Painful Example of Convolution by hand

Consider the input matrix :

$$X = \begin{bmatrix}1 & 2 & 3 & 0 & 1 \\0 & 1 & 2 & 3 & 0 \\3 & 0 & 1 & 2 & 3 \\2 & 3 & 0 & 1 & 2 \\1 & 2 & 3 & 0 & 1 \\\end{bmatrix}$$ Input slice: $$\begin{bmatrix}1 & 2 & 3 \\0 & 1 & 2 \\3 & 0 & 1 \\\end{bmatrix}$$

Kernel: $$\begin{bmatrix}1 & 0 & -1 \\1 & 0 & -1 \\1 & 0 & -1 \\\end{bmatrix}$$

Element-wise multiplication and sum: $$(1 \cdot 1) + (2 \cdot 0) + (3 \cdot -1) + \\(0 \cdot 1) + (1 \cdot 0) + (2 \cdot -1) + \\(3 \cdot 1) + (0 \cdot 0) + (1 \cdot -1)$$

$$\implies1 + 0 - 3 + \\0 + 0 - 2 + \\3 + 0 - 1 = -2$$ Now repeat that by moving the kernel one step over (you can in fact change this with the stride argument for convolution).

Low Rank Approximation of convolution

Now we will painfully do a low rank decomposition of the convolution kernel above. There is a theorem that says that a $2D$ matrix can be approximated by a sum of 2 outer products of two vectors. Say we can express $K$ as, $$K \approx a_1 \times b_1 + a_2\times b_2$$

We can easily guess $a_i, b_i$. Consider, $$a_1 = \begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix}$$ $$b_1 = \begin{bmatrix} 1\\ 0\\ -1\\ \end{bmatrix}$$ $$a_2 = \begin{bmatrix} 0\\ 0\\ 0\\ \end{bmatrix}$$ $$b_2 = \begin{bmatrix} 0\\ 0\\ 0\\ \end{bmatrix}$$

This is easy because I chose values for the kernel that were easy to break down. How to perform this breakdown is the subject of the later sections.

$$K = a_1\times b_1 + a_2 \times b_2 = \begin{bmatrix}1 & 0& -1 \\1 & 0 & -1 \\1 & 0 & -1 \\\end{bmatrix} +\begin{bmatrix}0 & 0 & 0 \\0 & 0 & 0 \\0 & 0 & 0 \\\end{bmatrix} = \begin{bmatrix}1 & 0 & -1 \\1 & 0 & -1 \\1 & 0 & -1 \\\end{bmatrix}$$

Consider the original kernel matrix $K$ and the low-rank vectors:

$$K = \begin{bmatrix}1 & 0 & -1 \\1 & 0 & -1 \\1 & 0 & -1\end{bmatrix}$$$$a_1 = \begin{bmatrix}1 \\1 \\1\end{bmatrix}, \quadb_1 = \begin{bmatrix}1 \\0 \\-1\end{bmatrix}$$

The input matrix $M$ is:

$$M = \begin{bmatrix}1 & 2 & 3 & 0 & 1 \\0 & 1 & 2 & 3 & 0 \\3 & 0 & 1 & 2 & 3 \\2 & 3 & 0 & 1 & 2 \\1 & 2 & 3 & 0 & 1\end{bmatrix}$$

Convolution with Original Kernel

Perform the convolution at the top-left corner of the input matrix:

$$\text{Input slice} = \begin{bmatrix}1 & 2 & 3 \\0 & 1 & 2 \\3 & 0 & 1\end{bmatrix}$$$$\text{Element-wise multiplication and sum:}$$$$\begin{aligned}(1 \times 1) + (2 \times 0) + (3 \times -1) + \\(0 \times 1) + (1 \times 0) + (2 \times -1) + \\(3 \times 1) + (0 \times 0) + (1 \times -1) &= \\1 + 0 - 3 + 0 + 0 - 2 + 3 + 0 - 1 &= -2\end{aligned}$$

Convolution with Low-Rank Vectors

Using the low-rank vectors:

$$a_1 = \begin{bmatrix}1 \\1 \\1\end{bmatrix}, \quadb_1 = \begin{bmatrix}1 \\0 \\-1\end{bmatrix}$$

Step 1: Apply $b_1$ (filter along the columns):**

$$\text{Column-wise operation:}$$$$\begin{aligned}1 \cdot \begin{bmatrix}1 \\0 \\-1\end{bmatrix} &= \begin{bmatrix}1 \\0 \\-1\end{bmatrix} \\2 \cdot \begin{bmatrix}1 \\0 \\-1\end{bmatrix} &= \begin{bmatrix}2 \\0 \\-2\end{bmatrix} \\3 \cdot \begin{bmatrix}1 \\0 \\-1\end{bmatrix} &= \begin{bmatrix}3 \\0 \\-3\end{bmatrix}\end{aligned}$$$$\text{Summed result for each column:}$$$$\begin{bmatrix}1 \\0 \\-1\end{bmatrix} +\begin{bmatrix}2 \\0 \\-2\end{bmatrix} +\begin{bmatrix}3 \\0 \\-3\end{bmatrix} =\begin{bmatrix}6 \\0 \\-6\end{bmatrix}$$

Step 2: Apply $a_1$ (sum along the rows):**

$$\text{Row-wise operation:}$$$$1 \cdot (6) + 1 \cdot (0) + 1 \cdot (-6) = 6 + 0 - 6 = 0$$

Comparison

• Convolution with Original Kernel: -2

• Convolution with Low-Rank Vectors: 0

The results are different due to the simplifications made by the low-rank approximation. But this is part of the problem that we need to optimize for when picking low rank approximations. In practice, we will ALWAYS lose some accuracy

PyTorch Implementation

Below you can find the original definition of AlexNet.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

class Net(nn.Module): def __init__(self): super().__init__() self.layers = nn.ModuleDict() self.layers['conv1'] = nn.Conv2d(3, 6, 5) self.layers['pool'] = nn.MaxPool2d(2, 2) self.layers['conv2'] = nn.Conv2d(6, 16, 5) self.layers['fc1'] = nn.Linear(16 * 5 * 5, 120) self.layers['fc2'] = nn.Linear(120, 84) self.layers['fc3'] = nn.Linear(84, 10) def forward(self,x): x = self.layers['pool'](F.relu(self.layers['conv1'](x))) x = self.layers['pool'](F.relu(self.layers['conv2'](x))) x = torch.flatten(x, 1) x = F.relu(self.layers['fc1'](x)) x = F.relu(self.layers['fc2'](x)) x = self.layers['fc3'](x) return x def evaluate_model(net): import torchvision.transforms as transforms batch_size = 4 # [4, 3, 32, 32] transform = transforms.Compose( [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') trainset = torchvision.datasets.CIFAR10(root='../data', train=True, download=True, transform=transform) trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=2) testset = torchvision.datasets.CIFAR10(root='../data', train=False, download=True, transform=transform) testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=2) # prepare to count predictions for each class correct_pred = {classname: 0 for classname in classes} total_pred = {classname: 0 for classname in classes} # again no gradients needed with torch.no_grad(): for data in testloader: images, labels = data outputs = net(images) _, predictions = torch.max(outputs, 1) # collect the correct predictions for each class for label, prediction in zip(labels, predictions): if label == prediction: correct_pred[classes[label]] += 1 total_pred[classes[label]] += 1 # print accuracy for each class for classname, correct_count in correct_pred.items(): accuracy = 100 * float(correct_count) / total_pred[classname] print(f'Original Accuracy for class: {classname:5s} is {accuracy:.1f} %')

Now let us decompose the first convolutional layer into 3 simpler layers using SVD

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95

def slice_wise_svd(tensor,rank): # tensor is a 4D tensor # rank is the target rank # returns a list of 4D tensors # each tensor is a slice of the input tensor # each slice is decomposed using SVD # and the decomposition is used to approximate the slice # the approximated slice is returned as a 4D tensor # the list of approximated slices is returned num_filters, input_channels, kernel_width, kernel_height = tensor.shape kernel_U = torch.zeros((num_filters, input_channels,kernel_height,rank)) kernel_S = torch.zeros((input_channels,num_filters,rank,rank)) kernel_V = torch.zeros((num_filters,input_channels,rank,kernel_width)) approximated_slices = [] reconstructed_tensor = torch.zeros_like(tensor) for i in range(num_filters): for j in range(input_channels): U, S, V = torch.svd(tensor[i, j,:,:]) U = U[:,:rank] S = S[:rank] V = V[:,:rank] kernel_U[i,j,:,:] = U kernel_S[j,i,:,:] = torch.diag(S) kernel_V[i,j,:,:] = torch.transpose(V,0,1) # print the reconstruction error print("Reconstruction error: ",torch.norm(reconstructed_tensor-tensor).item()) return kernel_U, kernel_S, kernel_V def svd_decomposition_conv_layer(layer, rank): """ Gets a conv layer and a target rank, returns a nn.Sequential object with the decomposition """ # Perform SVD decomposition on the layer weight tensorly. layer_weight = layer.weight.data kernel_U, kernel_S, kernel_V = slice_wise_svd(layer_weight,rank) U_layer = nn.Conv2d(in_channels=kernel_U.shape[1], out_channels=kernel_U.shape[0], kernel_size=(kernel_U.shape[2], 1), padding=0, stride = 1, dilation=layer.dilation, bias=True) S_layer = nn.Conv2d(in_channels=kernel_S.shape[1], out_channels=kernel_S.shape[0], kernel_size=1, padding=0, stride = 1, dilation=layer.dilation, bias=False) V_layer = nn.Conv2d(in_channels=kernel_V.shape[1], out_channels=kernel_V.shape[0], kernel_size=(1, kernel_V.shape[3]), padding=0, stride = 1, dilation=layer.dilation, bias=False) # store the bias in U_layer from layer U_layer.bias = layer.bias # set weights as the svd decomposition U_layer.weight.data = kernel_U S_layer.weight.data = kernel_S V_layer.weight.data = kernel_V return [U_layer, S_layer, V_layer] class lowRankNetSVD(Net): def __init__(self, original_network): super().__init__() self.layers = nn.ModuleDict() self.initialize_layers(original_network) def initialize_layers(self, original_network): # Make deep copy of the original network so that it doesn't get modified og_network = copy.deepcopy(original_network) # Getting first layer from the original network layer_to_replace = "conv1" # Remove the first layer for i, layer in enumerate(og_network.layers): if layer == layer_to_replace: # decompose that layer rank = 1 kernel = og_network.layers[layer].weight.data decomp_layers = svd_decomposition_conv_layer(og_network.layers[layer], rank) for j, decomp_layer in enumerate(decomp_layers): self.layers[layer + f"_{j}"] = decomp_layer else: self.layers[layer] = og_network.layers[layer] def forward(self, x): x = self.layers['conv1_0'](x) x = self.layers['conv1_1'](x) x = self.layers['conv1_2'](x) x = self.layers['pool'](F.relu(x)) x = self.layers['pool'](F.relu(self.layers['conv2'](x))) x = torch.flatten(x, 1) x = F.relu(self.layers['fc1'](x)) x = F.relu(self.layers['fc2'](x)) x = self.layers['fc3'](x) return x

Decomposition into a list of simpler operations

The examples above are quite simple and are perfectly good for simplifying neural networks. This is still an active area of research. One of the things that researchers try to do is try to further simplify each already simplified operation, of course you pay the price of more operations. The one we will use for this example is one where the operations is broken down into four simpler operations.

• (Green) Takes one pixel from the image across all $3$ channels and maps it to one value

• (Red) Takes one long set of pixels from one channel and maps it to one value

• (Blue) Takes one wide set of pixels from one channel and maps it to one value

• (Green) takes one pixel from all $3$ channels and maps it to one value

Intuitively, we are still taking the subset “cube” but we have broken it down so that in any given operation only $1$ dimension is not $1$. This is really the key to reducing the complexity of the initial convolution operation, because even though there are more such operations each operations is more complex.

PyTorch Implementation

In this section, we will take AlexNet (Net), evaluate (evaluate_model) it on some data and then decompose the convolutional layers.

Declaring both the original and low rank network

Here we will decompose the second convolutional layer, given by the layer_to_replace argument. The two important lines to pay attention to are est_rank and cp_decomposition_conv_layer. The first function estimates the rank of the convolutional layer and the second function decomposes the convolutional layer into a list of simpler operations.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

class lowRankNet(Net): def __init__(self, original_network): super().__init__() self.layers = nn.ModuleDict() self.initialize_layers(original_network) def initialize_layers(self, original_network): # Make deep copy of the original network so that it doesn't get modified og_network = copy.deepcopy(original_network) # Getting first layer from the original network layer_to_replace = "conv2" # Remove the first layer for i, layer in enumerate(og_network.layers): if layer == layer_to_replace: # decompose that layer rank = est_rank(og_network.layers[layer]) decomp_layers = cp_decomposition_conv_layer(og_network.layers[layer], rank) for j, decomp_layer in enumerate(decomp_layers): self.layers[layer + f"_{j}"] = decomp_layer else: self.layers[layer] = og_network.layers[layer] # Add the decomposed layers at the position of the deleted layer def forward(self, x, layer_to_replace="conv2"): x = self.layers['pool'](F.relu(self.layers['conv1'](x))) # x = self.layers['pool'](F.relu(self.laye['conv2'](x) x = self.layers['conv2_0'](x) x = self.layers['conv2_1'](x) x = self.layers['conv2_2'](x) x = self.layers['pool'](F.relu(self.layers['conv2_3'](x))) x = torch.flatten(x, 1) x = F.relu(self.layers['fc1'](x)) x = F.relu(self.layers['fc2'](x)) x = self.layers['fc3'](x) return x

Evaluate the Model

You can evaluate the model by running the following code. This will print the accuracy of the original model and the low rank model.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

decomp_alexnet = lowRankNetSVD(net) # replicate with original model correct_pred = {classname: 0 for classname in classes} total_pred = {classname: 0 for classname in classes} # again no gradients needed with torch.no_grad(): for data in testloader: images, labels = data outputs = decomp_alexnet(images) _, predictions = torch.max(outputs, 1) # collect the correct predictions for each class for label, prediction in zip(labels, predictions): if label == prediction: correct_pred[classes[label]] += 1 total_pred[classes[label]] += 1 # print accuracy for each class for classname, correct_count in correct_pred.items(): accuracy = 100 * float(correct_count) / total_pred[classname] print(f'Lite Accuracy for class: {classname:5s} is {accuracy:.1f} %')

Let us first discuss estimate rank. For a complete discussion see the the references by Nakajima and Shinchi. The basic idea is that we take the tensor, “unfold” it along one axis (basically reduce the tensor into a matrix by collapsing around other axes) and estimate the rank of that matrix.
You can find est_rank below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114

from __future__ import division import torch import numpy as np # from scipy.sparse.linalg import svds from scipy.optimize import minimize_scalar import tensorly as tl def est_rank(layer): W = layer.weight.data # W = W.detach().numpy() #the weight has to be a numpy array for tl but needs to be a torch tensor for EVBMF mode3 = tl.base.unfold(W.detach().numpy(), 0) mode4 = tl.base.unfold(W.detach().numpy(), 1) diag_0 = EVBMF(torch.tensor(mode3)) diag_1 = EVBMF(torch.tensor(mode4)) # round to multiples of 16 multiples_of = 8 # this is done mostly to standardize the rank to a standard set of numbers, so that # you do not end up with ranks 7, 9 etc. those would both be approximated to 8. # that way you get a sense of the magnitude of ranks across multiple runs and neural networks # return int(np.ceil(max([diag_0.shape[0], diag_1.shape[0]]) / 16) * 16) return int(np.ceil(max([diag_0.shape[0], diag_1.shape[0]]) / multiples_of) * multiples_of) def EVBMF(Y, sigma2=None, H=None): """Implementation of the analytical solution to Empirical Variational Bayes Matrix Factorization. This function can be used to calculate the analytical solution to empirical VBMF. This is based on the paper and MatLab code by Nakajima et al.: "Global analytic solution of fully-observed variational Bayesian matrix factorization." Notes ----- If sigma2 is unspecified, it is estimated by minimizing the free energy. If H is unspecified, it is set to the smallest of the sides of the input Y. Attributes ---------- Y : numpy-array Input matrix that is to be factorized. Y has shape (L,M), where L<=M. sigma2 : int or None (default=None) Variance of the noise on Y. H : int or None (default = None) Maximum rank of the factorized matrices. Returns ------- U : numpy-array Left-singular vectors. S : numpy-array Diagonal matrix of singular values. V : numpy-array Right-singular vectors. post : dictionary Dictionary containing the computed posterior values. References ---------- .. [1] Nakajima, Shinichi, et al. "Global analytic solution of fully-observed variational Bayesian matrix factorization." Journal of Machine Learning Research 14.Jan (2013): 1-37. .. [2] Nakajima, Shinichi, et al. "Perfect dimensionality recovery by variational Bayesian PCA." Advances in Neural Information Processing Systems. 2012. """ L, M = Y.shape # has to be L<=M if H is None: H = L alpha = L / M tauubar = 2.5129 * np.sqrt(alpha) # SVD of the input matrix, max rank of H U, s, V = torch.svd(Y) U = U[:, :H] s = s[:H] V[:H].t_() # Calculate residual residual = 0. if H < L: residual = torch.sum(torch.sum(Y ** 2) - torch.sum(s ** 2)) # Estimation of the variance when sigma2 is unspecified if sigma2 is None: xubar = (1 + tauubar) * (1 + alpha / tauubar) eH_ub = int(np.min([np.ceil(L / (1 + alpha)) - 1, H])) - 1 upper_bound = (torch.sum(s ** 2) + residual) / (L * M) lower_bound = np.max([s[eH_ub + 1] ** 2 / (M * xubar), torch.mean(s[eH_ub + 1:] ** 2) / M]) scale = 1. # /lower_bound s = s * np.sqrt(scale) residual = residual * scale lower_bound = float(lower_bound * scale) upper_bound = float(upper_bound * scale) sigma2_opt = minimize_scalar(EVBsigma2, args=(L, M, s, residual, xubar), bounds=[lower_bound, upper_bound], method='Bounded') sigma2 = sigma2_opt.x # Threshold gamma term threshold = np.sqrt(M * sigma2 * (1 + tauubar) * (1 + alpha / tauubar)) pos = torch.sum(s > threshold) if pos == 0: return np.array([]) # Formula (15) from [2] d = torch.mul(s[:pos] / 2, 1 - (L + M) * sigma2 / s[:pos] ** 2 + torch.sqrt( (1 - ((L + M) * sigma2) / s[:pos] ** 2) ** 2 - \ (4 * L * M * sigma2 ** 2) / s[:pos] ** 4)) return torch.diag(d)

You can find the EVBMF code on my github page. I do not go into it in detail here. Jacob Gildenblatt’s code is a great resource for an in-depth look at this algorithm.

Conclusion

So why is all this needed? The main reason is that we can reduce the number of operations needed to perform a convolution. This is particularly important in embedded systems where the number of operations is a hard constraint. The other reason is that we can reduce the number of parameters in a neural network, which can help with overfitting. The final reason is that we can reduce the amount of memory needed to store the neural network. This is particularly important in mobile devices where memory is a hard constraint.
What does this mean mathematically? Fundamentally it means that neural networks are over parameterized i.e. they have far more parameters than the information that they represent. By reducing the rank of the matrices needed carry out a convolution, we are representing the same operation (as closely as possible) with a lot less information.

References

• [Low Rank approximation of CNNs] (https://arxiv.org/pdf/1511.06067)
• [CP Decomposition] (https://arxiv.org/pdf/1412.6553)
• Kolda & Bader “Tensor Decompositions and Applications”in SIAM REVIEW, 2009
• [1] Nakajima, Shinichi, et al. “Global analytic solution of fully-observed variational Bayesian matrix factorization.” Journal of Machine Learning Research 14.Jan (2013): 1-37.
• [2] Nakajima, Shinichi, et al. “Perfect dimensionality recovery by variational Bayesian PCA.”
• [Python implementation of EVBMF] (https://github.com/CasvandenBogaard/VBMF)
• [Accelerating Deep Neural Networks with Tensor Decompositions - Jacob Gildenblat] (https://jacobgil.github.io/deeplearning/tensor-decompositions-deep-learning)
• [Python Implementatioon of VBMF] (https://github.com/CasvandenBogaard/VBMF)
• [Similar article that is more high level] (https://medium.com/@anishhilary97/low-rank-approximation-for-4d-kernels-in-convolutional-neural-networks-through-svd-65b30dc55f6b)

]]> https://franciscormendes.com/2024/09/13/lora-3/ 2024-09-13T04:00:00.000Z

In this post, we will explore the Low Rank Approximation (LoRA) technique for shrinking neural networks for embedded systems. We will focus on the Convolutional Neural Network (CNN) case and discuss the rank selection process.

2026-04-18T15:56:33.537Z