Inside the Autoencoder

It has two parts. An encoder turns the input into a new set of numbers. A decoder tries to turn those numbers back into the input.

We call the input $x$, the numbers in the middle $z$, and the attempt to rebuild the input $\hat{x}$. The vector $z$ is the latent representation. It is latent because nobody observes or labels it.

Training compares $\hat{x}$ with $x$ and adjusts the weights to bring them closer. That comparison is the only feedback the model gets. Whatever $z$ ends up containing is whatever helps the decoder rebuild $x$.

To see every computation, we use the smallest example that still means something. The input has two numbers. The latent has one.

Here the latent is smaller than the input, so the model has to compress. That is a common choice, not a requirement. The last chapter uses a latent that is much wider than the input.

This one line of NumPy is a complete encoder. It starts from the input vector $x$, which has shape (2,).

The weight matrix $\Wenc$ holds one weight per input dimension. Its shape (2, 1) says that it maps two numbers to one.

The @ operator pairs each entry of $x$ with the matching weight. The first input meets the first weight, and the second meets the second.

Each pair is multiplied.

The products are added, together with the bias $\benc$. The bias shifts the result by the same amount for every input.

The whole line reduces to one number, the latent representation of $x$.

This is exactly what one neuron without an activation function computes. A layer is several neurons side by side, which is why $W$ is a matrix.

An input with two numbers is a point in a plane. Here are seven inputs, including $x$. The encoder gives each of them one number.

Inputs that share a value of $z$ lie on a straight line at a right angle to the weight vector $\Wenc$. Along each of these lines, $z$ is constant.

So the encoder slides every point along its line until it meets the direction of $\Wenc$. Where the point sat along its line is forgotten.

A point's position along that direction, scaled by the length of $\Wenc$ and shifted by the bias, is its value of $z$. Seven points in two dimensions are now seven numbers.

This is not the same as deleting a coordinate. Dropping $x_2$ would be one particular choice, the direction at 0°. Training can choose any direction.

Try a few with the slider. Some directions keep the points spread apart. Others squeeze them together, which leaves the decoder little to work with.

It has the same form as the encoder, in reverse. It starts from $z = 0.8$.

$\Wdec$ has shape (1, 2). Multiplying by $z$ scales its single row, so the result always lies along the same direction. Only its length, and for negative $z$ its sign, depends on $z$.

The bias moves the result to its final place.

Because $z$ is a single number, every possible output lies on one line. Move $z$ to trace it. The other six inputs are decoded onto the same line.

Now compare. The input was $x = [2.0,\ 1.0]$. The reconstruction is $\hat{x} = [1.4,\ 0.6]$.

Nothing has been learned yet. The weights are numbers we picked by hand, so every reconstruction misses its input.

Start with the difference between the reconstruction and the input.

Square each entry, so that misses in either direction count and large misses count more. Then take the mean over the two coordinates. In the figure, each squared error is the area of a square.

The loss depends only on the distance between $\hat{x}$ and $x$. Every point on a circle around $x$ has the same loss. These circles are the contour lines of the loss.

The gradient tells us how the loss changes as $\hat{x}$ moves. For the mean over $n$ coordinates it is

It points straight away from $x$, across the contour lines. That is the direction in which the loss grows fastest.

Its negative points toward $x$. A small step in that direction lowers the loss. Gradient descent repeats such small steps.

Drag $\hat{x}$, or focus it and use the arrow keys. The gradient grows as $\hat{x}$ moves away from $x$, and it always points away from it.

But we cannot move $\hat{x}$ directly. It is computed from the weights. What we need is the gradient for each weight.

The forward pass computed every value in this graph. Each operation knows only its own inputs and outputs.

Backpropagation lets every operation answer one local question. How much did my input contribute to the final error?

It starts at the loss, where we already know the answer.

The decoder computed $\hat{x} = z\,\Wdec + \bdec$. A weight's gradient is the input it multiplied, times the gradient arriving from above. The bias passes that gradient through unchanged.

The gradient also passes through the decoder to $z$. This is the chain rule. The slope of $\hat{x}$ with respect to $z$ is $\Wdec$, so

The sign is informative. Increasing $z$ would lower the loss, because it moves $\hat{x}$ along the decoder line toward $x$.

The same rule reaches the encoder. Its weights multiplied $x$, so their gradient is $x$ times the gradient at $z$.

Every weight now has a gradient. Each takes a small step against it, scaled by a learning rate $\eta$.

With $\eta = 0.1$, one step lowers the loss from 0.26 to 0.14. Repeating this step is training.

A dataset is a matrix with one row per observation. Here $X$ has 100 rows and 2 columns. Its first row is our $x$.

The model does not change. The same weights are applied to every row, in a single matrix multiplication. The bias is added to each row, which NumPy calls broadcasting.

Before each pass over the data, the rows are shuffled. That way the updates do not see the data in the same order every time.

The shuffled rows are cut into mini-batches, here 5 batches of 20 rows.

Each batch produces one update. The loss is averaged over the rows of the batch, so the gradient is an average as well. It is noisier than the gradient over all the data, and much cheaper when the data is large.

One pass through all batches is an epoch. Here an epoch makes 5 updates.

After 40 epochs, 200 updates in all, the loss over the dataset has dropped from 0.60 to 0.011. In the code, gradients is the backward pass from the previous section, averaged over the batch.

This data lies close to a line, with some scatter around it. It has two coordinates, but most of its variation runs in one direction.

The trained encoder gives each point one number. Points that are close together along the line get similar values of $z$.

The decoder sends each $z$ back to a line in the plane. Training has turned and shifted that line until it runs through the data.

Hover over a point, or use the slider, to follow one input through the model. What survives in $z$ is the position along the line. What is lost is how far the point sits from the line, and that offset is the reconstruction error.

The model found the direction that best describes the data, without being told that it existed.

Take a latent space with two dimensions and three inputs. We use two dimensions so that we can draw them. Real models often use many more.

An ordinary autoencoder places each input at one point, $z^{(1)}, z^{(2)}, z^{(3)}$. Its decoder is trained on those points only. Nothing in training constrains what it produces for a $z$ in between.

A variational autoencoder encodes each input as a region instead. Every point in the region should decode to something close to the input. That pushes nearby codes to mean similar things.

The encoder now ends in two outputs. For each latent dimension it predicts a mean and a log variance. With a two-dimensional latent, that is four numbers.

These are not extra latent dimensions. They are the parameters of a Gaussian distribution over $z$, and $z$ still has two dimensions. The mean $\mu$ sets the center. The standard deviation $\sigma$ sets the width along each axis.

A standard VAE assumes a diagonal covariance. The dimensions do not co-vary, which is why the region is an ellipse aligned with the axes.

The encoder predicts $\log \sigma^2$ rather than the variance itself. A variance must be positive, but a linear layer can output any number. The exponential of any number is positive, and the log scale keeps very small and very large variances numerically stable.

The two are easy to confuse. $\sigma^2$ is the variance and $\sigma$ is the standard deviation.

To decode, we need one concrete $z$ from the region. The obvious way is to sample it directly.

But a random draw has no derivative. Backpropagation cannot tell how $\mu$ or $\sigma$ should change to improve the sample.

The trick moves the randomness out of the way. First, turn the predicted log variance into a standard deviation.

Then draw noise from a fixed standard normal distribution. This draw does not depend on any weight.

Scale the noise by $\sigma$, one dimension at a time. The unit circle stretches into the shape of the ellipse.

Then shift it by the mean.

The result has exactly the distribution we wanted, $\N(\mu, \sigma^2)$.

Now $z$ is an ordinary function of $\mu$ and $\sigma$, with $\varepsilon$ as a fixed input. Its derivatives are simple, so the gradient reaches the encoder.

The randomness is still there. It has moved to an input that needs no gradient.

Draw more samples. Each lands somewhere in the ellipse, more often near $\mu$. Over many draws they fill in $\N(\mu, \sigma^2)$.

The reconstruction term asks the model to keep enough about $x$ to rebuild it. For real-valued data it is often the squared error.

On its own it favors small regions far apart. Precise, separate codes are the easiest to decode.

The KL divergence measures how far each encoded distribution is from a shared prior, the standard normal $\N(0, I)$. It keeps every input's code compatible with one common space.

The $\mu_j^2$ term pulls each center toward the origin. The other terms are smallest when $\sigma_j = 1$.

With the KL term alone, the best solution is to encode every input as $\N(0, I)$ itself. The KL is zero, but $z$ now carries no information about $x$. The decoder cannot tell one input from another.

With both terms, the distributions move toward the origin and widen, but stay apart enough to identify their input. The gaps between them close, so points there decode to something plausible.

The standard VAE uses $\beta = 1$. Larger values trade reconstruction quality for a more regular latent space.

Change $\beta$ to see the tradeoff. As it grows, the distributions crowd toward the prior. With a very large $\beta$, real models can stop using some latent dimensions altogether.

The arrangement in this figure is drawn by hand to show the trend. It is not the result of training.

During training, every $z$ comes from an encoded input and is decoded back into that input. That is reconstruction.

The KL term has arranged the encoded regions around $\N(0, I)$. So we can draw $z$ from that prior directly and decode it, without any input at all. That is generation.

Decoding a grid of latent points shows why this works. Nearby points decode to similar shapes, and the shapes change smoothly across the space. Far outside the prior, where training rarely placed any codes, there is less reason to trust the decoder.

The decoder in these figures is a small hand-written function standing in for a trained network, so that the geometry stays visible.

Reconstruction starts from an input.

Generation starts from the prior.

Reconstruction describes an input we already have. Generation produces one we have never seen.

Our first latent was a single number. A sparse autoencoder instead uses a latent that is wider than its input. Here, eight units encode a two-dimensional input.

For any one input, most units are exactly zero. This input uses two of the eight.

The encoder ends in a ReLU, which turns negative values into zero. An inactive unit contributes nothing at all.

The decoder shows what a unit stands for. Unit $j$ owns a row of $\Wdec$, a direction $d_j$in input space. The reconstruction adds up these directions, each scaled by its unit's activation.

The rows form a dictionary of directions. In this figure $\bdec = 0$.

With eight directions in a two-dimensional space, there are endless ways to reach the same point. This code uses all eight units and still reconstructs $x$ exactly.

The eight directions are evenly spread, so they cancel out when added in equal amounts. Extra activity can hide in the code without changing the output. The reconstruction loss cannot tell the dense code from the sparse one.

A sparsity penalty breaks the tie. A common choice adds the L1 norm of $z$ to the loss. Among codes that reconstruct equally well, it prefers the one with the least total activity.

Other implementations keep only the $k$ largest activations and zero the rest, or constrain how often each unit may be active. The goal is the same.

Drag $x$, or focus it and use the arrow keys. The sparse code always uses the two directions on either side of it.

The figure computes the exact code with the smallest L1 norm. A trained encoder has to approximate that code in one forward pass, and does so imperfectly.

Since each unit is active for only a few inputs, we can ask a new question of it. Which inputs make it fire?

Below each shape is its code over sixteen units. Most are zero. The shapes and codes here are constructed for illustration.

Collect the inputs where one unit is most active. For $z_3$, they all look long and thin.

Nobody told the model about elongation. The unit was useful for reconstruction, and it happens to track a property we can name.

$z_7$ responds most strongly to shapes with three lobes.

$z_{12}$ is harder. Its strongest inputs share no property we can see.

This is common in practice. Some learned features are clear, some are partly clear, and some resist any short description.

A sparse feature is a learned direction, not a concept someone assigned.