How Do AI Image Generators Work? Diffusion, Prompts and Noise

How do AI image generators work? Many start from random noise and clean it up step by step, guided by your prompt. The best-known of these are called diffusion models: they begin with an image of pure noise and steer each stage toward the words you typed. Some newer systems work differently and build an image piece by piece, the way a language model writes text. Both approaches learn from very large collections of images paired with text.

This guide explains the main ideas using the research papers and documentation the systems are built on.

The core idea: learning to remove noise

The method behind many image generators was set out in the 2020 paper Denoising Diffusion Probabilistic Models by Jonathan Ho, Ajay Jain and Pieter Abbeel at UC Berkeley. It describes two processes:

• A forward process that gradually adds small amounts of noise to a real image until the original signal is destroyed.
• A reverse process, learned by a neural network, that undoes that corruption one step at a time.

During training, the model sees real images at many noise levels and learns to predict how to make each one slightly cleaner. Once trained, it can start from noise that never came from a real image and run the reverse process. After enough steps, what is left looks like a plausible picture.

That starting noise is random, which is why the same prompt gives a different image each time. Hugging Face’s Diffusers documentation explains that its pipelines create the initial noise with a random seed. Fixing the seed is how developers get the same image again.

How the prompt steers the image

A model that only removes noise would produce random pictures. To follow a prompt, it needs a way to connect words and images.

A widely used connector is CLIP, which OpenAI described in a 2021 paper. CLIP was trained on 400 million image-and-text pairs collected from the internet, with a simple task: predict which caption goes with which image. The result is a pair of encoders that map text and images into a shared representation, so a sentence and a matching picture end up close together.

Image generators then use that text representation as a condition on each denoising step. In the latent diffusion paper that Stable Diffusion is built on, Robin Rombach and colleagues added cross-attention layers so the denoising network can draw on text, or other inputs such as bounding boxes, while it works. OpenAI’s DALL·E 2 paper used a two-stage design: a “prior” turns the caption into a CLIP image embedding, and a diffusion decoder turns that embedding into the final picture.

A further technique controls how strictly the image follows the prompt. In classifier-free guidance, Jonathan Ho and Tim Salimans trained one model to work both with and without the text condition, then combined the two predictions at each step. Turning that guidance up trades variety for closer adherence to the prompt.

Why most models work in a compressed space

Running diffusion directly on millions of pixels is expensive. The latent diffusion authors wrote that training powerful pixel-space diffusion models often consumes hundreds of GPU days, and that generating images is costly because the network must run many times in sequence.

Their solution was to train an autoencoder first. It compresses an image into a smaller “latent” representation and can decode it back. The diffusion model then does all its denoising in that compact latent space, and the decoder turns the finished latent into pixels at the end. The paper reports that this keeps image detail while sharply reducing the computing needed.

Stable Diffusion is a well-documented example. According to the Diffusers documentation, the original model was created by researchers and engineers from CompVis, Stability AI, Runway and LAION, trained on 512x512 images from a subset of the LAION-5B dataset, and uses a frozen CLIP ViT-L/14 text encoder. It has an 860-million-parameter denoising network and a 123-million-parameter text encoder, small enough to run on consumer GPUs.

Not every generator uses diffusion

Diffusion is not the only approach. When OpenAI introduced image generation inside GPT-4o in March 2025, its system card addendum said that, unlike DALL·E, which operates as a diffusion model, 4o image generation is an autoregressive model built natively into the same model that handles text. Autoregressive generation produces output in sequence, predicting each next piece from what came before, the same basic principle a large language model uses for words.

OpenAI said that design let the model take images as input and transform them, follow detailed instructions, and render text inside images more reliably. Company product pages rarely say which architecture a given tool uses, so treat claims about a specific product’s internals with care unless the company has published them.

What the output can and cannot tell you

Generated images have no camera behind them. The model produces a picture that fits its training and your prompt, not a record of a real scene, so a detail can look convincing and still be invented, much like AI hallucination in chatbots.

Because results can look like photographs, companies have added ways to mark them:

• Invisible watermarks. Google DeepMind’s SynthID embeds a watermark directly in the pixels of images made with Google’s AI tools. Google says it is imperceptible to people, is designed to survive cropping, filters and lossy compression, and can be checked by uploading an image to Gemini.
• Content Credentials. The C2PA standard attaches signed information about how a file was made and edited. Our guide to C2PA explains what those credentials can show and why they can be stripped.

Neither method covers every generator, and an image without a watermark or credentials is not proof that it is real.