# Why is the loss of Diffusion model calculated between "RANDOM noise" and "model predicted noise"? Not between "Actual added noise" and "model predicted noise"?

**URL:** <https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198>\
**Category:** 🧨 Diffusers\
**Created:** [May 28, 2023, 8:14am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198 "2023-05-28T08:14:56Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![egshkim](https://avatars.discourse-cdn.com/v4/letter/e/d9b06d/32.png) [@egshkim](https://discuss.huggingface.co/u/egshkim)\
**Post date:** [May 28, 2023, 8:14am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/1 "2023-05-28T08:14:56Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/1/41c0bc265648045cd2b603930b9f3a98f06bb72d.jpeg)

Why is the loss of Diffusion model calculated between **“RANDOM noise” and “model predicted noise”**? Not between “Actual added noise” and “model predicted noise”?

When i trained the U-Net with loss between “the Actual added noise” and “model predicted noise”,  
it seems the model be optimized much much faster on my training dataset.  
May I use this loss ?

Anybody has insight ?

---

<div class="post-metadata">

**Author:** ![williamberman](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/williamberman/32/14112_2.png) [@williamberman](https://discuss.huggingface.co/u/williamberman)\
**Post date:** [May 29, 2023, 5:24pm UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/2 "2023-05-29T17:24:35Z")

</div>

sorry I’m not sure I understand. what is the alternative noise you’re talking about?

---

<div class="post-metadata">

**Author:** ![egshkim](https://avatars.discourse-cdn.com/v4/letter/e/d9b06d/32.png) [@egshkim](https://discuss.huggingface.co/u/egshkim)\
**Post date:** [May 30, 2023, 12:35am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/3 "2023-05-30T00:35:50Z")

</div>

In the above picture, **“noise” is randomly sampled noise** by **“noise = torch.randn(sample\_image.shape)”** and is used for loss calculation.

But I think “Actually added noise” should be used for loss calculation.  
the noise added between the “t-1” step and “t” step.

Why we are using random noise for loss calculation?

---

<div class="post-metadata">

**Author:** ![egshkim](https://avatars.discourse-cdn.com/v4/letter/e/d9b06d/32.png) [@egshkim](https://discuss.huggingface.co/u/egshkim)\
**Post date:** [May 30, 2023, 12:50am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/4 "2023-05-30T00:50:23Z")

</div>

I trained U-Net using the loss with **“the actual added noise”** that is, **the noise added between “t-1” step and “t” step** , NOT “random noise”.  
then U-Net seems to be optimized faster.

**Why** we are using **“random noise”** for diffusion loss calculation and **how this be possible?**

---

<div class="post-metadata">

**Author:** ![offchan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/offchan/32/21567_2.png) [@offchan](https://discuss.huggingface.co/u/offchan)\
**Post date:** [May 30, 2023, 5:23am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/5 "2023-05-30T05:23:52Z")

</div>

The training process is like this:

1. we generate a random noise vector with std=1
2. we scale it down to have std\<1
3. we add the scaled down noise to the image
4. we predict the noise with std=1

If I understand correctly, you are asking why we are predicting the noise with std=1 instead of the one with std\<1, right?

There are many ways to formulate the output of the model:

1. predict the noise with std=1 (normalized noise)
2. predict the noise with std\<1 (actual added noise)
3. predict the original clean image itself

You can actually try all these formulations. In fact, someone has tried it using the 3rd formulation on a toy project and it also works. Check the 2nd notebook in this repo here: [diffusion-models-class/unit1 at main · huggingface/diffusion-models-class · GitHub](https://github.com/huggingface/diffusion-models-class/tree/main/unit1)

But I think the 1st formulation has advantage that it’s making the low noise prediction more important than 2nd formulation.  
Even if the actual added noise has std only 0.01, the 1st approach would still predict it as std=1, therefore making it 100x more important in the loss. This should result in a model that cares a lot about denoising correctly at the last few inference steps.

Another advantange is probably about forcing the model to always predict noise that has std=1, this might help the model stabilize? I’m not sure about this reasoning though.

In short, it’s all about the trade-offs on the model performance. Research is still going on about which formulation is the best.

---

<div class="post-metadata">

**Author:** ![egshkim](https://avatars.discourse-cdn.com/v4/letter/e/d9b06d/32.png) [@egshkim](https://discuss.huggingface.co/u/egshkim)\
**Post date:** [June 1, 2023, 2:09am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/6 "2023-06-01T02:09:54Z")

</div>

You may have no idea how much you’ve helped me.  
and I bet there will be many who wonder this one as well.  
Hope they find the repo you mentioned.  
Thanks a lot, @offchan  
Good luck !

---

<div class="post-metadata">

**Author:** ![fortmeier](https://avatars.discourse-cdn.com/v4/letter/f/ecae2f/32.png) [@fortmeier](https://discuss.huggingface.co/u/fortmeier)\
**Post date:** [June 2, 2023, 1:19pm UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/7 "2023-06-02T13:19:21Z")

</div>

> Another advantange is probably about forcing the model to always predict noise that has std=1, this might help the model stabilize? I’m not sure about this reasoning though.

Intuitively, this makes a lot of sense. It makes the model output scaling independent of the timestep, right?

---

<div class="post-metadata">

**Author:** ![offchan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/offchan/32/21567_2.png) [@offchan](https://discuss.huggingface.co/u/offchan)\
**Post date:** [June 17, 2023, 4:19pm UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/8 "2023-06-17T16:19:54Z")

</div>

Yeah, that’s what I think.

---

<div class="post-metadata">

**Author:** ![nongnongzi](https://avatars.discourse-cdn.com/v4/letter/n/ec9cab/32.png) [@nongnongzi](https://discuss.huggingface.co/u/nongnongzi)\
**Post date:** [November 22, 2023, 2:51am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/9 "2023-11-22T02:51:20Z")

</div>

Hi I have read the solutions and I am still confused about it. Why we want to use unet to predict the random noise, which is irrelevant with timesteps. If we need the prediction of random noise, why can’t we directly generate a random noise in the reverse process?

---

<div class="post-metadata">

**Author:** ![Errenn](https://avatars.discourse-cdn.com/v4/letter/e/d9b06d/32.png) [@Errenn](https://discuss.huggingface.co/u/Errenn)\
**Post date:** [November 23, 2023, 7:35am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/11 "2023-11-23T07:35:56Z")

</div>

Same question. Have you found the explanation yet?

---

<div class="post-metadata">

**Author:** ![nongnongzi](https://avatars.discourse-cdn.com/v4/letter/n/ec9cab/32.png) [@nongnongzi](https://discuss.huggingface.co/u/nongnongzi)\
**Post date:** [November 27, 2023, 3:30am UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/12 "2023-11-27T03:30:33Z")

</div>

@offchan Hi sorry to bother, I am appreciate your answer for this topic. However, I am still confused about it. Why we want to use unet to predict the random noise, which is irrelevant with timesteps. If we need the prediction of random noise, why can’t we directly generate a random noise in the reverse process?

---

<div class="post-metadata">

**Author:** ![offchan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/offchan/32/21567_2.png) [@offchan](https://discuss.huggingface.co/u/offchan)\
**Post date:** [November 27, 2023, 1:31pm UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/13 "2023-11-27T13:31:55Z")

</div>

Each training step is like this:

1. we have a clean image from the training data.
2. we generate a random gaussian noise with std=1. let’s call it **big noise**
3. we scale down the big noise based on timestep. For early timesteps, we’ll scale the big noise down just a little bit. For late timesteps, we’ll scale the big noise down a lot. Now the std will be less than 1. Let’s call the scaled noise as **small noise**.
4. we add small noise to the clean image, resulting in a **noisy image**
5. we train the unet to predict **big noise** given **noisy image** and **timestep** as input.

Notice in the last step, we are asking the model to predict the big noise we have calculated in step 2. We are not asking it to predict a new random noise.  
And timestep is indeed relevant in step 3. One crucial detail here is that timestep is randomly sampled. It’s not sampled sequentially in a for loop like our intuition might suggest. You can read more about the logic behind random timestep in my [other answer here](https://discuss.huggingface.co/t/does-controlnet-and-other-diffusers-only-include-1-noise-injection-per-iteration-in-training-loop/40402/2)

I urge you to look at the training code to gain deeper understanding.  
Here’s the code from the text-to-image training script by diffusers:

> <https://github.com/huggingface/diffusers/blob/ebf581e85f3aad7faa30ceb4678148ee87375446/examples/text_to_image/train_text_to_image.py#L893-L912>

The clean image is named `latents` and the big noise is named `noise` in the code above. But the small noise is not shown here. The line that calls `add_noise()` function is internally computing the small noise and produces noisy image which is named `noisy_latents` above.

---

<div class="post-metadata">

**Author:** ![offchan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/offchan/32/21567_2.png) [@offchan](https://discuss.huggingface.co/u/offchan)\
**Post date:** [November 27, 2023, 1:39pm UTC](https://discuss.huggingface.co/t/why-is-the-loss-of-diffusion-model-calculated-between-random-noise-and-model-predicted-noise-not-between-actual-added-noise-and-model-predicted-noise/41198/14 "2023-11-27T13:39:55Z")

</div>

Check the answer above.