Steering the Latent Space

A generative model is a function. It takes a short list of numbers $z$ and turns it into something: an image, a sound, a sentence. The list is called a latent vector, and the set of every list the model accepts is its latent space.

Our model is as small as it gets. Its latent vector has two numbers, so every $z$ is a point in a plane, and its decoder $g$ draws a face. The red point is $z$, and the face on the right is $g(z)$.

Real models work the same way with more numbers: 512 for a face GAN, and 16,384 for the noise that Stable Diffusion turns into an image. Nobody chose what these numbers mean. The model invented them while learning to rebuild its training data, which is why they are called latent, or hidden. The autoencoder essay shows how such a space is learned.

Draw the face at every point of a seven by seven grid and you get an atlas of everything this model can make. Frowns are in the lower left, grins in the upper right, and the faces in between change a little at a time.

That smoothness is what makes control possible at all. A small step in $z$ gives a small change in the output, so getting the output you want comes down to getting to the right place in the space.

The axes, however, are no help. Walk to the right along $z_1$ and the face starts smiling but also closes its eyes. Walk up along $z_2$ and the eyes open while the smile grows a little. Each axis mixes both traits, and researchers call such a space entangled.

The model has no reason to do better. Training only asks it to reproduce the data, and a rotated copy of the plane, with the decoder rotated to match, would reproduce the data just as well. Nothing pushes the axes to line up with the words a person would use.

Try it. Drag the point around the square and watch which way you have to move to change only the smile. It is a diagonal, and you have to find it by feel.

The model has its own coordinates. To control it, we have to find our concepts inside them.

Start with examples. Here are 40 faces from the model, each a point in the latent space, and a person has labelled each one: 23 smiling, filled, and 17 not smiling, hollow.

The two groups overlap, but the smiling ones lean to the right. With a real image model you would get these points by encoding labelled photos, or by keeping the $z$ behind each generated image.

Average each group. The two crosses are the means, and the arrow from one to the other is the smile direction:

That is the whole method. There is nothing to train, just two averages and a subtraction. This difference of means turns up everywhere, from word vectors to steering language models.

To edit a face, add the direction to its latent vector and decode again:

The hollow red point is a frowning face with smile −0.62. One step of $v$ takes it to 0.79. The same $v$ works on every face, which is the real claim: a concept is a single direction, the same everywhere in the space.

This is the old word vector result, $\text{king} - \text{man} + \text{woman} \approx \text{queen}$, and the DCGAN demo where a smiling woman, minus a neutral woman, plus a neutral man gave a smiling man.

Now $\alpha$ is a dial. Turn it up and the face smiles more, turn it below zero and it frowns. Many sliders in face editing tools are a vector like this one with its own $\alpha$.

Watch the eyes, though. As the smile grows they close, from 0.75 to 0.30 at $\alpha = 1$. We asked for a smile and got a squint as well. Chapter 4 is about why.

A direction can also be read. Project any point onto the line through $v$, by dropping a perpendicular, and the position along the line is a smile score:

On the right, every face is placed by its score. Smiling faces score high and the rest score low, with a little overlap in the middle. A single dot product says how much smile a point contains.

One vector does two jobs. Adding it changes the concept, and projecting onto it measures the concept.

The simplest path between two latent vectors $a$ and $b$ is the straight one, linear interpolation, or lerp:

In two dimensions it works. The faces along the bottom morph smoothly from a wide-eyed frown into a squinting grin. The chart tracks the length of $z(t)$, which dips in the middle but stays well inside the shaded band where samples usually fall.

Most generative models draw their latent vectors from a standard Gaussian, $z \sim \mathcal{N}(0, I)$. Its squared length is a sum of $d$ independent squares, each about 1 on average, so

and the spread around $\sqrt{d}$ stays near $1/\sqrt{2}$ no matter how large $d$ gets. With $d = 512$, almost every sample has a length within 6 percent of 22.6. The samples live on a thin shell, and the center of the Gaussian, its single most likely point, is never drawn.

Two random samples are also nearly at right angles, so the midpoint of the straight line has length $0.71\sqrt{d}$, far inside the shell. The decoder has never seen anything like it. Real models give blurry, washed-out images there, drawn here as faded faces.

The fix is to travel along the shell instead of through it. Spherical interpolation, or slerp, rotates from $a$ to $b$ along the arc between them, where $\Omega$ is the angle between the two vectors:

The length stays the same all the way, and every face stays sharp. Many image pipelines interpolate their starting noise this way.

The same goes for edits. A large $\alpha$ in $z + \alpha v$ pushes a point off the shell, so a common trick is to rescale it back to length $\sqrt{d}$ afterward.

Change $d$ yourself. At $d = 2$ the shell is a thick ring, and nearly any point is a plausible input. Each step up makes the shell thinner relative to its radius, and the straight line falls off it sooner.

A latent space is not a uniform box. Good control stays where the model has seen data.

The edit from chapter 2 raises the smile from −0.62 to 0.79, but it also drops the eyes from 0.75 to 0.30.

The cause is the data, not the model. People who smile tend to squint, so the smiling examples also had narrower eyes. A difference of means captures every way the two groups differ, the correlations included.

Real models have the same problem on a larger scale. In face GANs, the direction that makes a face older also tends to add glasses, because older people in the training photos wear glasses more often.

To keep the eyes fixed, we first need a direction for eyes. Here it comes from a linear probe, a logistic regression trained to tell open eyes from closed ones. Its weight vector, scaled to length 1, is $n$. It is perpendicular to the probe's decision boundary and points toward wider eyes.

The grey lines are perpendicular to $n$. Along each line, eye openness is constant: these are its level sets. Moving along a line leaves the eyes alone, and crossing lines changes them. The smile vector $v$ crosses several.

Split $v$ into a part along $n$ and a part perpendicular to it:

Here $v \cdot n = -1.15$, so the smile vector was hiding a component that says “close your eyes”. Removing it is a projection, the dashed red segment in the figure:

This is one step of Gram–Schmidt, the procedure that makes vectors orthogonal to each other.

Editing with $v_{\perp}$ moves the face along its own level line. The smile goes to 0.80, and the eyes stay at 0.73.

To hold several traits fixed at once, put their unit directions in the columns of a matrix $N$, make them orthonormal, and project out all of them together:

InterFaceGAN calls this conditional manipulation.

It only works as well as the space is flat. If the level sets curve, a direction is only right locally. That is why some methods look for better coordinates after training, and others, such as β-VAE, try to untangle the axes during training.

To change one thing, move along the lines where everything else stays the same.

A diffusion model generates by removing noise a little at a time. At each step it estimates which way the data lies, the gradient of the log density, called the score:

Following the score is like walking uphill on the density. In our plane the data has two clusters, frowning and smiling faces. Each grey dot is a starting point of pure noise, each line is its path, and the green dots are where the paths end. Without guidance, 9 of 14 samples smile and the rest frown, a mix of both clusters like the data itself.

Train the same network with a prompt $c$ as an extra input, and it learns the score of $p(x \mid c)$. With the prompt “a smiling face” every path ends in the smiling cluster.

Bayes' rule shows what the prompt adds:

The last term is the gradient of a classifier, the direction in which $x$ looks most like $c$. It plays the same role as the smile vector of chapter 2, except that it is recomputed at every point and every step. Using a separately trained classifier for it is called classifier guidance.

Classifier-free guidance gets that direction without a classifier. The network is trained with the prompt sometimes dropped, so it can predict both with and without it. Their difference is the prompt direction, and the guidance weight $w$ plays the role of $\alpha$:

Here $\epsilon_\theta$ is the predicted noise, which is the score times $-\sigma_t$, the noise level at step $t$. With $w = 1$ this is just the conditional model. With $w > 1$ it overshoots, sampling roughly from $p(x)\, p(c \mid x)^{w}$, which favors the points the classifier is most sure about.

At $w = 4$ the mean smile rises from 0.79 to 0.93, and the spread between samples falls from 0.16 to 0.03. Every face becomes the same broad grin.

Try other weights. Image models usually run around $w = 7$, where pictures look sharp and on prompt but start to look alike. Much higher, and colors saturate and textures go strange.

Negative prompts use the same formula with the unconditional prediction replaced by a prediction for the thing you do not want, so the sample is pushed away from it.

Guidance trades diversity for fidelity, by pushing every step along the difference between with the prompt and without it.

A Transformer keeps one vector per token, the residual stream, and every layer adds its output to it. Take the vector at the last token after layer 12, $h \in \mathbb{R}^{d}$, for two sets of prompts: harmful requests the model refuses, and harmless ones it answers.

Projected onto a plane, the two sets form two clusters. The points here are a toy, but the method is the one used on real chat models.

The recipe from chapter 2 carries over unchanged. The difference of the means is a refusal direction:

Reading it works as before. The projection $h \cdot \hat{r}$ says how strongly a prompt activates refusal, and past a threshold, the shaded side, the model refuses. A readout like this is a probe, and probes are being studied as monitors for things a model will not say out loud.

Arditi and colleagues found that in many open chat models, refusal is carried largely by one such direction.

Writing works as before too. Add $\alpha r$ to the residual stream at every token while the model generates, and a harmless prompt is pushed over the threshold. The model refuses to give a recipe for pancakes.

This is called activation steering. The same trick makes models more honest, less flattering, more cheerful, or obsessed with a single topic. It needs no retraining, and it can be switched off again at any moment. Sparse autoencoders, from the autoencoder essay, find thousands of such directions at once.

Now remove the direction instead. Subtract each vector's component along $\hat{r}$, exactly the projection from chapter 4:

Every point slides onto the line through the origin perpendicular to $r$. The model can no longer represent “this should be refused”, and it refuses nothing at all.

For alignment this cuts both ways. Safety training can end up as a thin layer, a single direction that anyone with the weights can remove. The same geometry also gives us handles that are easier to read and more precise to adjust than retraining a whole model.

A direction that lets you steer a model also lets someone else steer it the other way.

Someone wants faces that smile with their eyes open. They could not write that down as a vector $w$, but they can look at two faces and pick one.

Assume their taste is linear, a score $r(z) = w \cdot z$. Then picking A over B says

That is a half-plane. Every direction $w$ on the wrong side of the line through the origin perpendicular to $z_A - z_B$ is ruled out. On the dial of all possible directions, one answer removes half the circle.

The next pair is chosen so that its line splits what is left exactly in half. After 5 answers the uncertainty has shrunk from ±180° to ±6°, a factor of 32. The face in the middle of the dial is the model's current best guess.

This is binary search over directions. Choosing the most informative question is called active learning. Real answers are noisy and real spaces have many more dimensions, so it takes more questions, but each answer still removes a half-space.

Your turn. Think of the face you would like to see, then keep pressing A or B for the one that is closer. The face in the middle comes to match what you have in mind, although you never said what that was.

This is human–AI interaction at its simplest. The person only has to recognise what they like, and the geometry turns their choices into a direction.

People are not perfectly consistent, so real systems use a soft version, the Bradley–Terry model:

Fitting $w$ to many comparisons with this likelihood is the code above. Replace $z$with a language model's last hidden state and you have the reward model of RLHF. Reinforcement learning then shifts the model so its outputs score higher, like guidance that is built into the weights.

Push too far and the outputs leave the region where the reward model was fitted, the same mistake as leaving the shell in chapter 3. The reward keeps rising while real quality falls, a failure called reward hacking. RLHF therefore keeps the new model close to the original:

Comparisons reveal a direction, and alignment is the craft of following it without going too far.