Inside the Model: Visual Control

Stable Diffusion redraws a scene from noise. SAM, inpainting, and latent blending are how you decide which object is allowed to move.

A scene is a latent

Rombach, Blattmann, Lorenz, Esser, and Ommer (2022) train Stable Diffusion as a latent diffusion model. A variational autoencoder compresses the image before any denoising happens. The Stable Diffusion 1.5 model card states the factor: the autoencoder downsamples by 8 and maps an H × W × 3 image to a latent of shape H/8 × W/8 × 4. A 512 by 512 picture becomes a 64 by 64 grid with 4 channels. A network conditioned on a text embedding learns to remove noise from that grid. A decoder turns the cleaned grid back into pixels.

The compression is why generation is affordable, and why control is subtle. The model moves a few thousand latent numbers. The decoder invents the pixels those numbers imply. A face survives only if you protect it in the latent, which is where the edit actually happens.

Sampling starts from noise, or from a noised copy of your image, and walks toward a clean latent over a few dozen steps. Early steps commit to layout: where the person stands, where the horizon sits. Later steps commit to texture: grain, type, the edge of a collar. Control is a schedule on top of that walk. The training walk, and why predicting noise teaches a model anything, is part 3.

The control problem

A prompt is a global condition. "Evening, warm lamps, the same room" still licenses the sampler to redraw the person. Nothing in the text names the pixels that are already finished.

Visual control adds a region and a budget. The region is a mask. The budget is how much noise you reintroduce, and how much of the original latent you paste back at each step. A large budget and a large open mask give you a new scene. A small budget, with the mask closed over the person, changes the room and leaves the person in place.

SAM names the region. Inpainting and latent blending spend the budget.

SAM

Kirillov et al. (2023) train Segment Anything as a promptable segmenter. An image encoder, a vision transformer, reads the picture once. A prompt encoder takes the pointer you give it: a click, a box, or a rough mask. A light decoder returns a segmentation, and further clicks are cheap because the image encoding is reused. A click is ambiguous. The point on a shoulder can mean the person, the shirt, or the arm. The model is built to return three masks for one prompt, each with a predicted score. Keep the mask that matches the object you are holding still, and add a negative click where it leaked onto the couch or a second person.

Ravi et al. (2024) extend this to video in SAM 2. A memory follows the object across frames, so one click can track a person through a shot. That memory names the object over time. The diffusion sampler still decides what may change underneath the tracked mask.

Inpainting

Inpainting fills a region and leaves the rest of the picture in place. Latent diffusion keeps that promise in two ways, and you usually want both.

The first is conditioning. An inpainting model sees the mask and the known part of the image as extra inputs. The Stable Diffusion 1.5 inpainting model card states the shape. The U-Net gains five channels beyond the four noisy latent channels: four for the encoded masked image, and one for the mask. Nine in total. The extra weights start at zero, and the checkpoint is then trained to invent the hole and to continue the surroundings.

The second is a composite during sampling. At each step, keep the edited latent where the mask is open. Where the mask is closed, write the source latent noised to that same step.

latent = mask · latent_edit + (1 − mask) · latent_source

Conditioning helps the invention agree with the border of the hole. The paste makes the closed region match the original at that noise level, so the decoder has nothing new to invent there. The source side of the paste has to carry the same amount of noise as the edit. Mixing a clean latent into a half-noised one paints a scar.

Which side is open depends on the job. To change the room and keep the person, open the mask on the room and close it on the person. To change the jacket and keep the room, open the mask on the jacket. The closed side is the side whose latent you refuse to replace.

How far you noise the source is the budget. Meng et al. (2021) measure it as t₀ in SDEdit, the noise level where the reverse walk begins. A higher t₀ makes the result more realistic and less faithful to the guide. A walk that starts near the photograph keeps the room. A walk that starts near pure noise is free to move the furniture and the posture.

Latent blending

Avrahami, Fried, and Lischinski (2022) make that composite a method, blended latent diffusion. One branch denoises under the text prompt. The other is the original latent, noised to the same step. The mask is downsampled by the autoencoder's factor of 8, and the two latents are blended with it at every step. Where the mask is closed, the source latent is what continues.

The same paper treats a mask that becomes too thin after that downsample. The schedule starts dilated, so the edit still lands, and shrinks back to the original boundary as the steps proceed. They also show that the autoencoder's own reconstruction is already lossy, before any diffusion. The encoder and the decoder have to be the pair the checkpoint was trained with.

Ho and Salimans (2022) define classifier-free guidance as a scale on the gap between a text-conditioned prediction and an unconditioned one. A larger scale follows the prompt more strongly. The blend, not the guidance weight, is what writes the closed region back.

When the closed region is a face or a product and the scene change is large, a reference image can hold identity through what the edge lets in. Ye et al. (2023) do this with IP-Adapter, a picture fed into the same cross-attention path the text uses. The prompt asks for evening light. The adapter asks for this person. Hertz et al. (2022) do a related job inside one sampling run. Prompt-to-Prompt reuses the cross-attention maps of a source generation, so layout survives a change of caption. Use attention reuse when you are generating both versions. Use the latent composite when the photograph already exists.

A working order

1
Encode with the checkpoint's own VAE Avrahami et al. show the reconstruction is already lossy. The encoder and decoder are the pair that was trained together.
2
Ask SAM for the object Kirillov et al. return three masks for one click. Keep the one that matches the object, and add a negative click where it leaked.
3
Choose the closed side Close it over the object when the scene is what should change.
4
Spend a small noise budget Start near the photograph when the room should stay recognizable. Meng et al. show that a higher noise level spends faithfulness.
5
Composite on every step Paste the noised source into the closed side. Dilate the open side early. Tighten it late.
6
Judge the boundary first A bad edge is a mask problem. A wrong mood with a clean edge is a prompt or a strength problem.

Holding an object still

A frame stays still where Avrahami et al. write the source latent back. A shot stays still when that paste happens on every frame under the same mask.

Ravi et al. supply the mask across frames. Meng et al. supply the reason to keep the starting noise low: a higher noise level spends faithfulness to the source. The paste still runs at every step. An edge that flickers is a mask that moved between frames, or a noise level high enough that the open region is free to invent a new texture.

What to take with you

The prompt proposes. The mask decides the region. SAM is how you name the object. The inpainting network is how a hole learns to agree with its border. Latent blending is how the original numbers occupy the closed region for the whole walk, loose while layout is still open, tight while texture is being written.

If the object moved and you wanted it still, one of three things happened. The closed latent was updated. The mask was open on the object. Or the noise budget was large enough to redraw it.

Sources

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752, 2022.
Runway. Stable Diffusion v1.5 model card. Autoencoder downsample factor 8, latent channels 4.
Runway. Stable Diffusion inpainting model card. Five extra U-Net channels: four for the encoded masked image and one for the mask.
Alexander Kirillov and colleagues. Segment Anything. arXiv:2304.02643, 2023.
Nikhila Ravi and colleagues. SAM 2: Segment Anything in Images and Videos. arXiv:2408.00714, 2024.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended Latent Diffusion. arXiv:2206.02779, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. arXiv:2108.01073, 2021.
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv:2207.12598, 2022.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. IP-Adapter: Text Compatible Image Prompt Adapter. arXiv:2308.06721, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompt Image Editing with Cross Attention Control. arXiv:2208.01626, 2022.