Diffusion Models: How AI Learns to Create by Practicing Destruction

Take a photograph. Now imagine a machine that spends a thousand tiny steps smearing it into pure static, like a television losing its signal in slow motion. That's easy — entropy does that for free. Now imagine running the tape backward: starting from pure static and, one tiny denoising step at a time, having a machine guess its way back to a photograph that never existed. That's not easy. That's a diffusion model, and it's the technology quietly running Stable Diffusion, DALL-E, Midjourney, Sora, and — more surprisingly — some of the best protein engineering and weather forecasting tools on Earth.

The strange part is the destruction isn't a metaphor. The researchers who invented this approach really did borrow the physics of how gas molecules diffuse into equilibrium, then asked whether a neural network could learn to run that physical process in reverse.

The Concept

A diffusion model learns to generate new data — images, audio, protein structures, weather maps — by first learning to undo noise.

Training happens in two passes. The forward process takes a real image and adds a small amount of Gaussian (random) noise to it, repeatedly, over many steps — a few hundred to a couple thousand, depending on the model. After enough steps, the "image" is indistinguishable from television static. There's no skill involved in this half; it's just a fixed mathematical recipe for destroying information, step by step, at a known rate.

The reverse process is where the learning happens. A neural network is trained to look at a noisy image at step t and predict what noise was just added, so it can be subtracted back out to recover step t-1. Do that once and you've removed a little noise. Do it a thousand times in sequence — each step conditioned on the output of the last — and you've walked all the way from pure static back to a coherent image.

The trick is that the network never needs to understand "what a photograph is" in one leap. It only ever has to solve a much narrower problem: given this noisy thing, what noise should I subtract to make it slightly less noisy? That's a far easier function to learn than "map random numbers directly to a realistic photo," which is roughly what earlier generative approaches like GANs were asking a network to do in a single shot. Diffusion breaks an impossibly hard problem into thousands of very easy ones, and that reframing turned out to be the key.

Once trained, you don't need a real image to start the reverse process — you can start from pure random noise and let the network denoise its way to a brand-new image that resembles the training data but has never existed before. Guide that process with a text embedding (as in Stable Diffusion or DALL-E 2) and you get text-to-image generation: the noise is nudged, at every step, toward whatever the text describes.

Why It Matters

The idea has a specific, documented origin. In 2015, Jascha Sohl-Dickstein and colleagues at Stanford published "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," which first proposed destroying data with a diffusion process and training a model to reverse it — explicitly modeled on nonequilibrium statistical physics. The paper barely registered at the time; generative modeling in 2015 was dominated by other approaches, and this one was slow and mathematically unwieldy. For five years it sat as a largely unnoticed curiosity.

The breakthrough that made it practical came in 2020, when Jonathan Ho, Ajay Jain, and Pieter Abbeel at UC Berkeley published "Denoising Diffusion Probabilistic Models" (DDPM). They simplified the training objective and showed that diffusion models could match the image quality of state-of-the-art GANs — a genuine surprise, since GANs had dominated image generation for years. Shortly after, Yang Song, Stefano Ermon, and collaborators (including Sohl-Dickstein again) reframed the whole approach through the lens of stochastic differential equations, unifying diffusion models with an older technique called score matching and giving researchers a cleaner mathematical toolkit to improve them.

From there it moved fast. OpenAI's DALL-E 2 (April 2022) and Stability AI's open-source Stable Diffusion (August 2022) brought diffusion-based text-to-image generation to the public, and within months "type a sentence, get a picture" went from research-lab demo to something millions of people used daily. Stable Diffusion's key efficiency trick — running the diffusion process in a compressed "latent space" rather than on full-resolution pixels — is also why it could run on an ordinary consumer GPU instead of a data center.

What's less widely known is that the same mathematics generalizes far beyond pictures, because "noise in, structure out" is a description of almost any generative task.

Protein design. The University of Washington's Baker Lab adapted diffusion to protein structures with RFdiffusion, treating a protein's 3D backbone the way an image model treats pixels — start from noise, denoise toward a viable structure. Older protein-design methods sometimes required testing tens of thousands of candidate molecules in the lab to find one that worked; with RFdiffusion, researchers have in some cases needed to test as few as one design per problem. David Baker shared the 2024 Nobel Prize in Chemistry for computational protein design, the broader field RFdiffusion extended.

Weather forecasting. Google DeepMind's GenCast treats a future atmospheric state as a "denoising" target, generating 15-day global forecasts by iteratively refining noise conditioned on the last two days of real weather. It's reported to beat the European Centre for Medium-Range Weather Forecasts' operational ensemble system (ENS) — the long-standing gold standard — on accuracy, while running far faster than traditional physics-based simulation.

Robotics. "Diffusion policies" let robot arms learn complex, multi-step manipulation tasks by treating a sequence of future actions as data to be denoised from noise, rather than predicted directly — which turns out to handle the messy, multi-modal nature of real-world robot motion (there's often more than one valid way to pick something up) better than earlier approaches.

The Details

The forward process is deliberately simple and requires no learning at all. At each step t, you take the current (slightly noisy) image and blend in a small, fixed amount of Gaussian noise according to a predetermined schedule. After enough steps — DDPM used 1,000 — the result converges to something statistically indistinguishable from pure noise, regardless of what the original image was. This is the "diffusion" in the name: it's the same math used to describe a drop of dye spreading through still water until the color is uniform, or heat spreading through a metal bar until the temperature evens out. Push any structured starting state through enough random perturbation, and it converges to the same featureless equilibrium.

The reverse process is where it gets interesting. At each step, the network doesn't literally see "noise" as a label — it's trained (via a surprisingly simple loss function) to predict the specific noise pattern that was added at that step, so that subtracting the prediction recovers a cleaner image. Chain thousands of these small, confident guesses together and you get long-range coherence that no single step "knows about": the network predicting noise on step 850 has no explicit memory of step 1, yet the final image comes out with consistent lighting, anatomy, and perspective. That's the model implicitly encoding a huge amount of structure about what real images look like into its noise predictions, learned purely from exposure to millions of training images going through the forward process.

Visually, you can picture the whole thing as a narrow, winding path through an astronomically high-dimensional space — the space of "all possible images" — where random noise is a point drifting aimlessly and a realistic photo is an infinitesimally rare, specific point. The forward process is a controlled walk away from that rare point toward the vast, featureless sea of noise. The reverse process is a trained guide finding its way back, step by tiny step, along a route it never explicitly memorized but has learned to infer from the local "slope" of realism at every point it passes through — which is exactly what the score-based reformulation made mathematically precise: the gradient of a probability landscape, pointing uphill toward "more like real data," estimated at every noise level.

This also explains why diffusion models are slow compared to GANs: generating one image means running the network hundreds or thousands of times in sequence, whereas a GAN generates in a single pass. Much of the engineering effort since 2020 — including Stable Diffusion's latent-space trick — has gone into cutting that step count down (some modern variants generate usable images in under 10 steps) without losing the quality that made the slow version worth using in the first place.

Takeaways

• Diffusion models generate data by learning to reverse a simple, fixed process of adding random noise — turning one hard problem ("create a realistic image from nothing") into thousands of easy ones ("remove a little noise from this").
• The idea traces to 2015 (Sohl-Dickstein et al.'s nonequilibrium thermodynamics paper) but only became practical with 2020's DDPM, after which it rapidly overtook GANs as the dominant approach to image generation.
• The same math — noise in, structure out — generalizes beyond images: it drives Stable Diffusion and DALL-E, but also RFdiffusion's protein design, DeepMind's GenCast weather forecasts, and diffusion-based robot control policies.
• The physics analogy is literal, not decorative: the forward process is mathematically the same diffusion that spreads dye through water or heat through metal, just run on pixel values instead of molecules.
• The main cost of the approach — needing hundreds to thousands of sequential steps to generate one sample — is also an active area of research, with newer techniques pushing generation down to single-digit step counts.

Resources: - Sohl-Dickstein et al., "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (2015) - Ho, Jain, Abbeel, "Denoising Diffusion Probabilistic Models" (2020) - Song et al., "Score-Based Generative Modeling through Stochastic Differential Equations" (2021) - Watson et al., "De novo design of protein structure and function with RFdiffusion", Baker Lab - DeepMind, "GenCast: Diffusion-based ensemble forecasting for medium-range weather"