Image and video generators start with noise and repeatedly refine it into a coherent result. Latent diffusion models reduce this work by refining a compact representation, called a latent, instead of working directly on every image pixel. A pretrained autoencoder compresses the image into a latent and later decodes the finished latent into an image.
At each step, the diffusion model processes less data, so it uses less computation and memory than a comparable model that works in pixel space. Video models apply the same idea to frames. The savings depend on the model’s design. Compression also has a cost: decoding cannot always recover details lost during encoding.
How compression reduces image-generation work
A latent is a compact grid of numbers that represents an image’s visual content. A variational autoencoder, or VAE, uses an encoder to create the grid and a decoder to turn it back into an image. During generation, a separate neural network, often a U-Net, gradually removes noise from the latent. The decoder then turns the refined latent into pixels.
The main savings come from repeatedly removing noise from a smaller representation. A pixel-space model processes the full pixel grid throughout generation. A latent diffusion model processes the compressed grid instead. The latent diffusion research describes this approach as a way to reduce computation while maintaining image quality.
The model can also use cross-attention, which connects the image representation to information that guides generation, such as a text prompt, an image, or a layout. This supports tasks such as text-to-image generation, inpainting, super-resolution, and text-based editing. For example, an inpainting model can use a prompt and a mask marking the area to change. The diffusion process then fills that area in the latent representation. Research on latent diffusion describes cross-attention as a way to support these conditional tasks.
How video models apply the same idea
A video generator can encode each frame as a latent and run diffusion on these smaller representations. It must also model time so objects move consistently and details do not shift arbitrarily between frames. One approach adds temporal layers to the U-Net and fine-tunes the model on video data.
The video diffusion work “Align Your Latents” builds on pretrained image latent diffusion models. It trains added temporal components to model relationships between frames. The image model provides a starting point for appearance, while the temporal layers model how that appearance changes over time. The model works on frame latents and decodes them into images.
This design helps reduce the cost of generating or editing sequences that contain many full-resolution frames. The model still has to represent motion, and its total computational demand depends on the video’s size and length.
Where the savings have limits
Compression reduces the representation’s size but can lose detail. If the encoder discards fine textures or small features, the decoder cannot reliably restore them. The VAE’s quality therefore affects the final image and the model’s efficiency. Recent work on limits of latent diffusion also identifies VAE reconstruction quality as a potential bottleneck.
Choose a model that preserves the details your output needs while reducing computation. If the work depends on subtle textures or precise small features, inspect decoded outputs and compare them with the source or reference. For video, check individual frames and continuity between them. A plausible still image does not guarantee stable motion.
Latent diffusion reduces the amount of data a model repeatedly processes. There is no universal percentage for the time or cost saved. Results depend on the model and generation setup, and the available evidence does not establish a general figure.