# Stable Diffusion XL much slower with candle than with diffusers

**URL:** <https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649>\
**Category:** Beginners\
**Created:** [March 26, 2026, 11:45am UTC](https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649 "2026-03-26T11:45:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jthulhu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jthulhu/32/30331_2.png) [@jthulhu](https://discuss.huggingface.co/u/jthulhu)\
**Post date:** [March 26, 2026, 11:45am UTC](https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649/1 "2026-03-26T11:45:48Z")

</div>

I am trying to embed a text-to-image procedure in a Rust application and, to do so, I’ve tried the candle library. However, I have the impression that it is slower, and sometimes of lesser quality, than its diffusers counterpart.

Consider, for instance, as follows

```python
from diffusers import StableDiffusionXLPipeline
import torch
pipe = StableDiffusionXLPipeline("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True).to("cuda")
image = pipe(prompt="A rusty robot holding a torch in its hand, photorealistic", num_inference_steps=50, guidance_scale=5).images[0]
image.save("hello.png")

```

This takes in total (without counting the `import` lines) about 18 seconds, and the iterative inference steps have a speed of 2.83 steps per second.

Compare this with the following command (running the official example of stable diffusion of the candle repository)

```shell
$ cargo run --release --example stable-diffusion --features=cuda,cudnn,flash-attn -- --prompt "A rusty robot holding a torch in its hand, photorealistic." --sd-version xl --use-flash-attn --use-f16 --guidance-scale 5 --n-steps 50

```

which takes, in total, about 45 seconds, and the inference part has a speed of 1.2 step per second. I’ve tried use flash attention, which is supposed to speed up things and reduce memory consumption, and indeed it did so but it’s still almost twice slower than its diffuers counterpart.

Also, the diffusers version takes up more VRAM (~9GB during the inference, and 16GB when decoding the image) than the candle version (~8GB during the inference, and 13GB when decoding the image).

Is there a way to speed things up?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 27, 2026, 1:28am UTC](https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649/2 "2026-03-27T01:28:07Z")

</div>

Seems known mismatch…?

* * *

The simplest explanation is:

**You are not benchmarking equivalent SDXL pipelines.** Candle’s example is still on a weaker-looking SDXL configuration path, while Diffusers on modern PyTorch gets several runtime optimizations “for free” that Candle’s example does not obviously match. Your forum post’s numbers, about **18s / 2.83 steps/s** in Diffusers versus **45s / 1.2 steps/s** in Candle, line up with known public reports and with the current example code.

## What is most likely going on

### 1. Candle’s SDXL example is still using DDIM

Candle’s current Stable Diffusion example README says that for **v1.5, v2.1, and XL 1.0** , the default scheduler is **DDIM**. That is already a warning sign for SDXL. Candle issue `#1508` says the example should not use DDIM for SDXL, and PR `#1635` changes the SDXL path from `DDIMSchedulerConfig` to `EulerAncestralDiscreteSchedulerConfig`. So even Candle contributors recognized this as a mismatch. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/README.md "raw.githubusercontent.com"))

There is one important nuance. The official `stabilityai/stable-diffusion-xl-base-1.0` scheduler config currently shows `_class_name: "EulerDiscreteScheduler"`, not Euler Ancestral. So the safest statement is not “SDXL must be Euler A.” The safe statement is: **the official SDXL repo advertises an Euler-family scheduler, while Candle’s example still uses DDIM.** That is enough to make the comparison non-apples-to-apples. ([Hugging Face](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/scheduler/scheduler_config.json "scheduler/scheduler\_config.json · stabilityai/stable-diffusion-xl-base-1.0 at main"))

### 2. DDIM is a plausible reason for the quality drop

This is not just theory. Diffusers issue `#6068` reports that **DDIM on SDXL** can leave **residual noise** and a **“smudgy”** look because of timestep alignment problems. That matches your “sometimes lesser quality” observation very closely. So one likely story is:

- Candle SDXL example uses DDIM.
- DDIM has had SDXL-specific quality issues.
- Your worse quality is at least partly scheduler-driven. ([GitHub](https://github.com/huggingface/diffusers/issues/6068 "DDIM produces incorrect samples with SDXL (epsilon or v- ..."))

## Why Diffusers can still be much faster even on the same GPU

### 3. Diffusers is standing on PyTorch 2 runtime optimizations

Hugging Face’s Diffusers optimization guide says **SDPA** is used by default in Diffusers, and in their SDXL benchmark it improves latency from **4.63s to 3.31s**. The same guide says compiling the **UNet and VAE** with `torch.compile` cuts it further from **3.31s to 2.54s** , and notes that the “max-autotune” mode uses **CUDA graphs** to reduce GPU launch overhead. PyTorch’s own blog summarizes the same stack as **bfloat16 + SDPA + torch.compile + combined q/k/v projections** , showing about **3×** SDXL latency improvement. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

That matters because Candle’s README only promises a narrower optimization in the example: **flash attention makes image generation a lot faster and uses less memory**. That helps, but it is not the same as having the full PyTorch 2 stack of SDPA, graph-friendly compilation, CUDA graphs, and related kernel fusion. So “both are using fast attention” does **not** mean “both are equally optimized.” ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/README.md "raw.githubusercontent.com"))

### 4. Diffusion models amplify small runtime overheads

Diffusion inference is iterative. The denoiser runs over and over across many timesteps. PyTorch’s guidance explicitly warns that **graph breaks** reduce the benefit of `torch.compile`, and Diffusers’ optimization guide says `fullgraph=True` helps avoid those breaks. Their guide also talks about removing **GPU sync after compilation**. In plain terms, once you pay a little overhead every step, the total gap grows quickly over 30 or 50 steps. That is why a runtime-execution advantage can show up as a large end-to-end difference even when both implementations use the same model weights. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

## Why Candle can use less VRAM but still be slower

That part is not contradictory. Your post reports Candle using less VRAM than Diffusers during inference and decode. Lower memory use does **not** imply higher throughput. It can simply mean the runtime is making different tradeoffs. In your case, the public evidence points to exactly that: Candle’s example can be more memory-conservative while Diffusers is faster because Diffusers is using a more mature optimized execution stack.

## Two secondary details in Candle’s example

These are not the main cause, but they matter.

First, Candle’s `main.rs` now has a special fp16 SDXL VAE override. When SDXL or Turbo is used with `--use-f16`, it switches to **`madebyollin/sdxl-vae-fp16-fix`** , with a comment pointing to Candle issue `#1060`. That means SDXL fp16 behavior has already needed a workaround in Candle. If someone overrides the VAE path, quality or stability can move around for reasons unrelated to the scheduler. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/main.rs "raw.githubusercontent.com"))

Second, the same file builds the text encoder with **`DType::F32`** in `build_clip_transformer(..., DType::F32)`. That will not dominate the repeated denoising loop, but it can still add wall-clock cost compared with a more uniformly half-precision path. I would treat this as a secondary factor, not the main reason for the 2×+ gap you saw. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/main.rs "raw.githubusercontent.com"))

## What I think is the real diagnosis

In order of likelihood:

1. **Scheduler mismatch** is probably the main reason for the quality difference. Candle’s SDXL example is on DDIM, while the official SDXL repo is Euler-family, and DDIM had a documented SDXL quality bug in Diffusers. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/README.md "raw.githubusercontent.com"))

2. **Runtime maturity** is probably the main reason for the speed difference. Diffusers on PyTorch 2 gets SDPA, compile, CUDA-graph-assisted execution, and related optimizations that Candle’s example does not obviously replicate. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

3. **Secondary implementation details** in Candle, such as the fp16 VAE workaround and F32 text encoder path, may add some extra overhead or quality variability, but they are unlikely to be the whole story. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/main.rs "raw.githubusercontent.com"))

## What to try next

### Best first test

Patch Candle so SDXL uses an **Euler-family scheduler** instead of DDIM, then rerun the exact same prompt, seed, image size, guidance, and step count. This is the cleanest test because it isolates the most suspicious difference first. The open Candle PR already shows the exact scheduler change contributors had in mind. ([GitHub](https://github.com/huggingface/candle/pull/1635/files "Use the Euler Ancestral scheduler by default with SDXL by freespirit · Pull Request #1635 · huggingface/candle · GitHub"))

### Best second test

Benchmark only the **denoising loop** , separate from text encoding and VAE decode. Your post already distinguishes total time from step speed, which is the right instinct. If Candle loses mostly in step speed, that points directly to runtime execution rather than one-time setup costs.

### Best third test

Make the Diffusers baseline explicit:

- plain Diffusers
- Diffusers with default SDPA
- Diffusers with compiled UNet/VAE

That tells you whether Candle is behind a basic baseline or only behind the fully optimized PyTorch 2 path. The docs show that these modes can differ materially. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

## What I would conclude today

I would **not** conclude that Rust is the problem. I would also **not** conclude that Candle is “bad.” I would conclude that, for **SDXL inference right now** , Candle’s public example is not yet a fair throughput match for a modern Diffusers-on-PyTorch stack, and its SDXL scheduler choice makes the quality comparison worse than it needs to be. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/README.md "raw.githubusercontent.com"))

If you want raw speed in Rust today, the strongest near-term workaround is often a **Torch-backed Rust path** such as `diffusers-rs` or `tch-rs`, because Candle’s own SD example is described as a conversion of `diffusers-rs` “using candle rather than libtorch.” That suggests the main tradeoff here is the execution stack, not the language. ([GitHub](https://raw.githubusercontent.com/huggingface/candle/main/candle-examples/examples/stable-diffusion/README.md "raw.githubusercontent.com"))

The shortest practical summary is:

- **Worse quality:** likely **DDIM on SDXL**.
- **Worse speed:** likely **runtime optimization gap** versus modern Diffusers/PyTorch.
- **Less VRAM:** real, but not evidence of higher overall optimization.

* * *

Here is the concrete checklist.

## Goal

Answer three questions cleanly:

1. Is the image-quality drop mostly a **scheduler** problem?
2. Is some of the wall-clock cost coming from Candle’s **text encoder dtype** choice?
3. After those fixes, is the remaining gap mostly just **runtime-stack maturity** versus Diffusers on PyTorch 2? ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/README.md "candle/candle-examples/examples/stable-diffusion/README.md at main · huggingface/candle · GitHub"))

## 1. Keep the comparison fixed

Use the same prompt, seed, size, steps, guidance scale, and model each time. Candle’s SD example still defaults SDXL 1.0 to **DDIM** , and `--use-flash-attn` is the intended fast-attention path. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/README.md "candle/candle-examples/examples/stable-diffusion/README.md at main · huggingface/candle · GitHub"))

Base command:

```bash
cargo run --release --example stable-diffusion --features=cuda,cudnn,flash-attn -- \
  --prompt "A rusty robot holding a torch in its hand, photorealistic." \
  --sd-version xl \
  --use-flash-attn \
  --use-f16 \
  --guidance-scale 5 \
  --n-steps 50 \
  --height 1024 \
  --width 1024 \
  --seed 42

```

## 2. First code change: switch the SDXL scheduler

Why first: Candle’s README says SDXL uses **DDIM** by default, Candle issue `#1508` says SDXL should use an Euler-family scheduler instead, and PR `#1635` changes the code to **Euler Ancestral**. Separately, the official SDXL scheduler config currently says **EulerDiscreteScheduler** , which is still Euler-family and still different from DDIM. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/README.md "candle/candle-examples/examples/stable-diffusion/README.md at main · huggingface/candle · GitHub"))

Change this in:

```plaintext
candle-transformers/src/models/stable_diffusion/mod.rs

```

From:

```rust
let scheduler = Arc::new(ddim::DDIMSchedulerConfig {
    prediction_type,
    ..Default::default()
});

```

To:

```rust
let scheduler = Arc::new(
    euler_ancestral_discrete::EulerAncestralDiscreteSchedulerConfig {
        prediction_type,
        ..Default::default()
    },
);

```

That is the exact replacement shown in PR `#1635`. ([GitHub](https://github.com/huggingface/candle/pull/1635/files "Use the Euler Ancestral scheduler by default with SDXL by freespirit · Pull Request #1635 · huggingface/candle · GitHub"))

## 3. Second code change: stop forcing the text encoder to F32

In current `main.rs`, Candle builds the CLIP text encoder with `DType::F32` even though `dtype` is already set from `--use-f16`. That is probably not the main reason for slow **step speed** , but it can add one-time overhead to total runtime. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/main.rs "candle/candle-examples/examples/stable-diffusion/main.rs at main · huggingface/candle · GitHub"))

Change this in:

```plaintext
candle-examples/examples/stable-diffusion/main.rs

```

From:

```rust
stable_diffusion::build_clip_transformer(clip_config, clip_weights, device, DType::F32)?;

```

To:

```rust
stable_diffusion::build_clip_transformer(clip_config, clip_weights, device, dtype)?;

```

## 4. Add two tiny timers

Candle already prints sampling progress starting at `println!("starting sampling");`, and `save_image()` does the final VAE decode and write. Add two coarse timers so you can split:

- text encode
- sampling loop
- final decode/save ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/main.rs "candle/candle-examples/examples/stable-diffusion/main.rs at main · huggingface/candle · GitHub"))

Minimal idea:

```rust
let encode_t0 = std::time::Instant::now();
// text/token/CLIP work
println!("text encode total {:.2}s", encode_t0.elapsed().as_secs_f32());

let sampling_t0 = std::time::Instant::now();
println!("starting sampling");
// denoising loop
println!("sampling total {:.2}s", sampling_t0.elapsed().as_secs_f32());

let decode_t0 = std::time::Instant::now();
save_image(...)?;
println!("final decode+save {:.2}s", decode_t0.elapsed().as_secs_f32());

```

## 5. Run these 3 benchmarks

### Benchmark 1. Current baseline

No scheduler patch. No dtype patch. Just run the command.

Record:

- total time
- `text encode total`
- `sampling total`
- `final decode+save`
- output image

### Benchmark 2. Scheduler-only

Apply the scheduler patch. Leave the text encoder line alone.

Run the same command again.

This isolates the “DDIM vs Euler-family” effect. That matters because Diffusers has an SDXL issue showing **DDIM can produce smudgy / noisy results** on SDXL. ([GitHub](https://github.com/huggingface/candle/pull/1635/files "Use the Euler Ancestral scheduler by default with SDXL by freespirit · Pull Request #1635 · huggingface/candle · GitHub"))

### Benchmark 3. Scheduler + dtype

Keep the scheduler patch. Also replace `DType::F32` with `dtype`.

Run the same command again.

This tells you whether one-time prompt encoding was adding avoidable wall-clock cost. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/main.rs "candle/candle-examples/examples/stable-diffusion/main.rs at main · huggingface/candle · GitHub"))

## 6. Interpret the results like this

If **Benchmark 2 fixes image quality a lot** , the scheduler was a major quality problem. That is the outcome I would expect first. Candle’s published SDXL path is DDIM, while the official SDXL config is Euler-family. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/README.md "candle/candle-examples/examples/stable-diffusion/README.md at main · huggingface/candle · GitHub"))

If **Benchmark 2 also improves sampling time a lot** , then DDIM was hurting both quality and throughput in this setup. That would make the original comparison less apples-to-apples than it looked. ([GitHub](https://github.com/huggingface/candle/issues/1508 "[Examples] SDXL should also use Euler Ancestral · Issue #1508 · huggingface/candle · GitHub"))

If **Benchmark 3 lowers total time but barely changes sampling time** , then the F32 text encoder was only a front-end tax. The main bottleneck is still the denoising loop. ([GitHub](https://github.com/huggingface/candle/blob/main/candle-examples/examples/stable-diffusion/main.rs "candle/candle-examples/examples/stable-diffusion/main.rs at main · huggingface/candle · GitHub"))

If **Benchmark 3 is still clearly behind Diffusers** , the remaining gap is probably mostly the runtime stack. Diffusers’ optimization docs say **SDPA is used by default** , and PyTorch 2 can improve text-to-image inference latency by **up to 3x**. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

## 7. What I expect

Most likely:

- **Scheduler patch** helps quality the most.
- **Text encoder dtype patch** helps total time a bit.
- A meaningful gap may still remain, because Diffusers on PyTorch 2 gets SDPA and broader compiler/runtime optimizations that Candle’s public SDXL example does not obviously match. ([Hugging Face](https://huggingface.co/docs/diffusers/v0.26.1/tutorials/fast_diffusion "Accelerate inference of text-to-image diffusion models · Hugging Face"))

The shortest decision rule is:

- quality improves after Benchmark 2 → blame the scheduler first
- total time improves after Benchmark 3 but sampling does not → blame the text encoder only for setup overhead
- large gap still remains after both → blame the runtime stack first ([GitHub](https://github.com/huggingface/candle/pull/1635/files "Use the Euler Ancestral scheduler by default with SDXL by freespirit · Pull Request #1635 · huggingface/candle · GitHub"))

---

<div class="post-metadata">

**Author:** ![jthulhu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jthulhu/32/30331_2.png) [@jthulhu](https://discuss.huggingface.co/u/jthulhu)\
**Post date:** [March 28, 2026, 9:57pm UTC](https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649/3 "2026-03-28T21:57:02Z")

</div>

I already tried to change the float type to f16 for the decover, the change is barely noticeable.

As suggested, I’ve tried patching the scheduler, using the same configuration as found in the `scheduler.json` in the repository of the model, but that changes nothing: the speed is exactly the same as before, and I still get weirdly off looking images that I don’t get with `diffusers`.

I think the next step will be to try to use the `diffusers-rs` library, which seems to be backed by Torch rather than candle.

Incidentally, I tried to use Torch compilation with the `diffusers` library and that resulted in a huge speedup. With it, the diffusion steps can reach the speed of 4 steps per second.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 28, 2026, 10:39pm UTC](https://discuss.huggingface.co/t/stable-diffusion-xl-much-slower-with-candle-than-with-diffusers/174649/4 "2026-03-28T22:39:37Z")

</div>

Hmm… It’s surprising that changing the scheduler doesn’t change the results… I’d think that in Rust, dependencies are less likely to break over time compared to Python…

If you’re set on a native Rust implementation, you might want to try `tch-rs` first. If PyTorch’s optimizations are the main contributor to the speed, this approach is more direct.

* * *

Based on what you already tested, the best next step is **not** another small Candle-side tweak.

## The short answer

For **SDXL today** , the best practical move is:

1. **Use Python Diffusers with `torch.compile` for image generation now.**
2. **Keep Rust around it** , not inside the diffusion core yet.
3. If Rust for the core is mandatory, **evaluate `tch-rs` before `diffusers-rs` for SDXL**.
4. Only keep digging into Candle if your goal is **debugging or contributing upstream** , not just getting a fast, correct SDXL pipeline. ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

## Why this is the right move

Your recent tests already ruled out the two easiest explanations:

- changing float type barely helped
- changing scheduler did not fix speed or image correctness

That means the problem is probably **not** “one wrong config value.” It is more likely a combination of **runtime-stack gap** and **implementation-parity gap** for SDXL in Candle. Diffusers on PyTorch 2 gets optimized attention by default through **SDPA** , and Hugging Face documents `torch.compile` as an extra performance layer on top. PyTorch’s own SDXL acceleration post says those native optimizations can push diffusion inference **up to 3× faster**. Candle, by contrast, has an active RFC proposing a future tracing/compilation layer with **JIT, kernel fusion, and static buffer planning** , which strongly suggests that this kind of optimization layer is still emerging there rather than already mature in the current path. ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

## What I would do next, in order

### Option A. Best overall: keep Diffusers for generation

If the goal is **good SDXL images plus speed now** , stay with **Diffusers + PyTorch 2 + `torch.compile`**. That path is already working for you, and it matches the official optimization guidance: Diffusers supports **SDPA by default** on PyTorch 2, and recommends `torch.compile` for extra speed. PyTorch’s SDXL acceleration writeup is basically the background explanation for the speedup you just saw. ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

The clean architecture is:

- Rust app for orchestration, UI, job queue, prompt handling, file management
- Python worker or microservice for SDXL generation
- communicate over CLI, HTTP, gRPC, or a local queue

That is not elegant in the “single-language purity” sense, but it is the fastest route to a system that is **fast, correct, and maintainable**. This recommendation is an inference from the facts above and from your own benchmark outcome. ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

### Option B. If Rust is mandatory: try `tch-rs` first

If you truly need the generation core in Rust, I would look at **`tch-rs` directly** before betting on `diffusers-rs` for SDXL.

Why:

- `tch-rs` is the Rust binding to **libtorch** , so you stay closer to the PyTorch execution stack. ([GitHub](https://github.com/LaurentMazare/tch-rs/blob/main/Cargo.toml "Cargo.toml - LaurentMazare/tch-rs"))
- `tch-rs` explicitly mentions a **Stable Diffusion implementation following Diffusers-style code**. ([GitHub](https://github.com/LaurentMazare/tch-rs "GitHub - LaurentMazare/tch-rs: Rust bindings for the C++ api of PyTorch. · GitHub"))
- `diffusers-rs` is Torch-backed too, but its README currently says it supports **Stable Diffusion v1.5 and v2.1**. I could not verify a clearly documented, ready-to-use **SDXL** path from its public docs. ([GitHub](https://github.com/LaurentMazare/diffusers-rs "GitHub - LaurentMazare/diffusers-rs: An implementation of the diffusers api in Rust · GitHub"))

So the practical reading is:

- `diffusers-rs` is promising
- but for **SDXL specifically** , the README does not yet present it as the obvious drop-in answer ([GitHub](https://github.com/LaurentMazare/diffusers-rs "GitHub - LaurentMazare/diffusers-rs: An implementation of the diffusers api in Rust · GitHub"))

### Option C. Only if you want to debug Candle itself

If your goal is to help Candle improve, the next useful step is **not more top-level flags**. It is a **numerical comparison**.

Do this:

- fix one prompt, seed, size, guidance, steps
- capture intermediate tensors at the same timestep in Diffusers and Candle
- compare latents, scheduler inputs/outputs, and UNet outputs step by step

At this point, that is the only route likely to reveal why images still look wrong after matching the obvious settings. This is also consistent with Candle’s current state: there is ongoing interest in more advanced compilation and execution infrastructure, so parity bugs or performance gaps at the model-runtime boundary are plausible areas for real investigation. ([GitHub](https://github.com/huggingface/candle/issues/3294 "[RFC] Candelabra: JAX-style tracing and compilation for Candle · Issue #3294 · huggingface/candle · GitHub"))

## What I would _not_ do next

I would not spend more time on:

- more scheduler swaps
- more fp16/f32 toggles
- minor decode-side changes

You already tested the high-probability knobs. The evidence now points away from “small local fix” and toward “different execution stack” or “deeper implementation difference.” ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

## My recommendation

If I had to choose one path:

### Use compiled Diffusers now, and wrap it from Rust if needed.

That gives you:

- the speedup you already observed
- the image quality you already trust
- the least engineering risk

Then, in parallel:

- evaluate `tch-rs` if Rust-only execution is a hard requirement
- treat Candle SDXL as an experiment or contribution target, not the shortest path to a production-quality result today ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))

## Simple decision rule

Use this:

- **Need SDXL working now** → Diffusers + `torch.compile` ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))
- **Need Rust around the system** → keep Rust at the orchestration layer, Python for generation ([Hugging Face](https://huggingface.co/docs/diffusers/main/optimization/torch2.0 "PyTorch 2.0 · Hugging Face"))
- **Need Rust in the generation core** → investigate `tch-rs` first, then `diffusers-rs` only if you confirm SDXL support yourself ([GitHub](https://github.com/LaurentMazare/tch-rs "GitHub - LaurentMazare/tch-rs: Rust bindings for the C++ api of PyTorch. · GitHub"))
- **Need to understand Candle’s bug/perf gap** → compare intermediate tensors and file an upstream issue with a minimal repro ([GitHub](https://github.com/huggingface/candle/issues/3294 "[RFC] Candelabra: JAX-style tracing and compilation for Candle · Issue #3294 · huggingface/candle · GitHub"))