Skip to content
explainroo
AI models explained

Diffusion Models Explained

Updated 3 min read

With explainroo Text-to-Video Models vs. Code-Rendered Video for Explainers
On this page
  1. How a diffusion model learns to remove noise
  2. How generation turns noise into an image
  3. Why many image generators use latent diffusion
  4. What diffusion models do well, and where they struggle

When an image generator turns a text prompt into a picture, it needs a way to create an image from scratch. A diffusion model does this by starting with random noise and gradually shaping it into an image. It learns that process by training on images that have been progressively corrupted with noise, then learning how to reverse the corruption.

Diffusion models are generative AI models used to create images, video, and other kinds of data. During training, noise is added to examples, and a neural network learns to predict it. The model then repeatedly uses the network to create new examples. This gradual process produces effective results but requires substantial computation.

How a diffusion model learns to remove noise

Training has two parts. First, a predefined process adds small amounts of Gaussian noise, which is random noise with a bell-shaped distribution, to an image over a sequence of steps. After enough steps, the original image becomes nearly indistinguishable from noise. This forward process is designed in advance; the model does not learn how to corrupt the image.

The neural network learns to reverse the process. It receives a noisy image and information about how much noise was added, then estimates the noise in the image. Training commonly compares the estimated noise with the noise that was actually added, using mean squared error, a measure of the average squared difference. A MindSpore diffusion tutorial describes this forward-and-reverse setup.

During training, the model does not have to create every intermediate noisy image. A formula lets it sample an image at a chosen noise level directly from the original image. This gives the network examples from many stages of the corruption process without replaying every earlier step.

For image generation, a common network design is a U-Net. It processes an image at different levels of detail and uses skip connections to carry information between those levels. This helps it account for both broad structure and fine details when estimating noise.

How generation turns noise into an image

To generate an image, the model starts with random noise and repeatedly uses its learned operation to remove noise. At each step, it estimates the noise and uses that estimate to produce a slightly less noisy image. After enough steps, the result contains recognizable shapes, textures, and other image features.

The number of steps affects the trade-off between generation time and image quality. More steps require more model computations. Too few steps can prevent gradual denoising from producing a good result. The exact number depends on the model and sampling method; diffusion designs range from dozens of steps to far more.

Text-to-image models also need a way to connect words with visual content. A text encoder maps the prompt to numerical representations, or vectors. The image model uses those vectors as guidance while denoising, so the generated image reflects the prompt. Interactive research on Stable Diffusion illustrates how text and image components work together in that kind of generation.

Why many image generators use latent diffusion

Some image generators remove noise directly from pixels. Latent diffusion models first use a variational autoencoder, a neural network that compresses images into a smaller representation called a latent space. The diffusion process runs on that compressed representation, and a decoder turns the result back into an image.

Working in a smaller space reduces the amount of data the diffusion model must process, which helps make image generation more efficient. Stable Diffusion is a well-known example of a latent diffusion model. The latent diffusion overview describes this compression-and-generation approach.

What diffusion models do well, and where they struggle

Diffusion models support more than text-to-image generation. Researchers have applied them to video generation and molecule design, among other uses, as summarized in a survey of diffusion methods and applications. The basic recipe also gives developers several points of control: they can condition generation on text or other inputs and choose how the model moves from noise toward a finished sample.

The repeated denoising steps require substantial computation, especially when a model works with large images or generates many samples. Research on efficient diffusion models focuses on reducing those costs. The reverse process also relies on an imperfect neural-network estimate, so individual denoising steps do not guarantee a flawless result.

Training data raises concerns about artists’ styles and the creation of harmful material. These concerns relate to how developers collect and use training data, and how people use generated outputs. The Stable Diffusion learning study discusses these social and ethical concerns.

Free and open source

Let your AI agent make the video

explainroo lets a coding agent like Claude Code or Codex make narrated explainer videos and product demos. The voice, the word timing and the rendering run on your own computer, with no API key and no cost per video. Optional outside services use your own key.

Copy this into your AI agent

Make me a short explainer video about [your topic]. Use explainroo for it: run npx --yes explainroo docs and follow what it prints.

Getting started Example videos GitHub

Made with explainroo: How explainroo makes a video

More on ai models explained

All guides on ai models explained