# How do you train a LoRA?

**URL:** <https://discuss.huggingface.co/t/how-do-you-train-a-lora/179814>\
**Category:** Beginners\
**Created:** [September 3, 2026, 1:18pm UTC](https://discuss.huggingface.co/t/how-do-you-train-a-lora/179814 "2026-09-03T13:18:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuil12](https://avatars.discourse-cdn.com/v4/letter/y/e19b73/32.png) [@yuil12](https://discuss.huggingface.co/u/yuil12)\
**Post date:** [September 3, 2026, 1:18pm UTC](https://discuss.huggingface.co/t/how-do-you-train-a-lora/179814/1 "2026-09-03T13:18:46Z")

</div>

I’ve started creating AI models, but I’m currently stuck at the LoRA training stage; my videos are turning out blurry and unrealistic. I’d like to know how the pros handle this and what mistakes I might be making, as I haven’t been able to find any solid information in open sources.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 4, 2026, 2:48am UTC](https://discuss.huggingface.co/t/how-do-you-train-a-lora/179814/2 "2026-09-04T02:48:02Z")

</div>

Hmm… I’m not very familiar with T2V/I2V, but it seems there are quite a few different ways blur can show up here…:

* * *

I cannot really speak for how “the pros” do it, but I would **not** start by assuming that blurry output means you picked the wrong LoRA rank or learning rate.

With video LoRAs, “blurry and unrealistic” can come from several different layers of the pipeline: the base model, inference settings, the LoRA itself, the checkpoint you selected, video preprocessing/frame sampling, the VAE, or something specific to the model/trainer/runtime.

So the fastest route is usually to find **the first boundary where the quality becomes bad** , rather than changing many training parameters at once.

A useful first pass would be:

1. **Generate the same case with the base model and no LoRA.**
2. **If the base is sharp, sweep the LoRA strength** instead of immediately retraining it.
3. **Compare an early, middle, and late training checkpoint** under exactly the same inference settings.
4. **Inspect what the trainer actually sees after preprocessing** , not just the original MP4 files.

Those four checks already separate a surprising number of failure modes.

For example, Diffusers supports varying adapter weight with [`set_adapters()` / LoRA scaling](https://huggingface.co/docs/diffusers/main/using-diffusers/loading_adapters), so something roughly like

```plaintext
base only
LoRA 0.25
LoRA 0.50
LoRA 0.75
LoRA 1.00

```

with the **same prompt / input image / seed / resolution / frame count / sampler settings** is a much more informative experiment than immediately trying another rank.

If the blur increases smoothly with LoRA strength, that points in a different direction from a case where even the base model is blurry.

A small symptom map might be:

| What you see | I would look at first |
| --- | --- |
| Base model is already blurry | inference settings / VAE / model / runtime |
| Base is sharp, LoRA makes it blurry | LoRA checkpoint / scale / loading path / training |
| Early checkpoint is OK, later ones degrade | training progression / overtraining / effective update strength |
| Individual frames are soft everywhere | preprocessing / VAE / spatial quality / training |
| Still areas are sharp but moving areas smear | FPS / frame sampling / motion distribution / temporal path |
| Training validation looks good, normal inference looks bad | validation-vs-inference mismatch / runtime / LoRA loading |
| First frame is strange, later frames blur | I2V conditioning / VAE / model-specific denoising path |

That does not identify the root cause by itself, but it reduces the search space quickly.

> **A slightly more complete decision tree**
>
> I would probably debug it in approximately this order:
> 
> ```plaintext
> START
> |
> +-- Is the base model sharp with the same input/settings?
> | |
> | +-- NO
> | | -> Do not retrain the LoRA yet.
> | | Check inference / VAE / scheduler / CFG / resolution /
> | | frame count / runtime / base-model limitations.
> | |
> | +-- YES
> | |
> | +-- Does lowering LoRA strength reduce the blur?
> | |
> | +-- YES
> | | -> Adapter/checkpoint/training becomes more likely.
> | |
> | +-- NO
> | -> Check preprocessing, VAE, load path,
> | runtime and model-specific behavior too.
> |
> +-- Was an early checkpoint sharper than a later one?
> | |
> | +-- YES
> | | -> Training progression is a strong clue.
> | |
> | +-- NO
> | -> Look earlier in the pipeline.
> |
> +-- Does the trainer-visible / decoded training data already look wrong?
> | |
> | +-- YES
> | | -> Fix data/preprocessing before tuning LoRA parameters.
> | |
> | +-- NO
> | -> Continue to adapter/inference/model-specific checks.
> |
> +-- Does trainer validation look good but external inference look bad?
> |
> +-- YES
> | -> Match the two inference paths before retraining.
> |
> +-- NO
> -> Then start tuning training parameters one at a time.
> 
> ```
> 
> The important part is that this is a **branching problem** , not really a single “best LoRA configuration” problem.

> **Video preprocessing is worth checking very early**
>
> One particularly easy mistake with video training is assuming:
> 
> > “My source videos look good, therefore my training data looks good.”
> 
> The model usually does not see the source MP4 exactly as you see it.
> 
> Depending on the trainer, there may be:
> 
> - scene splitting
> - frame extraction
> - FPS conversion
> - frame truncation
> - resizing
> - random/center cropping
> - aspect-ratio buckets
> - temporal buckets
> - VAE encoding
> - cached latents
> - caption preprocessing
> 
> before the training step even starts.
> 
> This is why I like the workflow in the [LTX-Video-Trainer dataset preparation guide](https://github.com/Lightricks/LTX-Video-Trainer/blob/main/docs/dataset-preparation.md).
> 
> It can preprocess the dataset into cached latents and then **decode those latents back into videos for verification** with `--decode-videos`.
> 
> That is an excellent debugging idea even if you are not using LTX:
> 
> > inspect the representation immediately before the model learns from it.
> 
> If that decoded/transformed version is already softer, badly cropped, temporally strange, or missing important motion, changing LoRA rank will not repair the real problem.
> 
> FPS handling is also more subtle than it sounds.
> 
> For example, [musubi-tuner’s dataset documentation](https://github.com/kohya-ss/musubi-tuner/blob/main/docs/dataset_config.md) explicitly says that when `source_fps` is specified, frames can be skipped to reach the model FPS, and that this skipping **does not consider the image content**. It recommends checking the result with:
> 
> ```plaintext
> --debug_mode video
> 
> ```
> 
> If `source_fps` is not specified there, all frames are used regardless of the original video’s frame rate.
> 
> So even something apparently simple like “I trained on 30 FPS video” does not fully describe the temporal data the model received.
> 
> Also check:
> 
> - Are the clips dominated by camera motion rather than subject motion?
> - Do the source clips themselves contain motion blur?
> - Are there many near-duplicate clips?
> - Are captions describing the important motion as well as appearance?
> - Are long scenes being squeezed into a very small number of sampled frames?
> - Are important subjects repeatedly near crop boundaries?
> - Is the training resolution much smaller than the source resolution?
> - Are frames/resolutions being silently truncated to model-supported shapes?
> 
> For a beginner, I would treat **viewing the actual transformed training samples** as a standard step, not an advanced debugging trick.

> **Make training validation and normal inference comparable**
>
> Another thing I would check before retraining is whether the output you call “bad” is being generated under the same conditions as the trainer’s validation output.
> 
> Try to match as many of these as possible:
> 
> ```plaintext
> base model / revision
> prompt
> input image/video
> seed
> height / width
> number of frames
> frame rate
> number of inference steps
> scheduler / sampler
> CFG / guidance settings
> LoRA scale
> LoRA loading method
> precision / quantization
> VAE configuration
> 
> ```
> 
> This matters because a trainer’s built-in validation loop is not necessarily identical to the pipeline you later use in ComfyUI, another UI, or a standalone Diffusers script.
> 
> Hugging Face’s [Finetrainers validation dataset format](https://github.com/huggingface/finetrainers/blob/main/docs/dataset/README.md), for example, allows validation cases to explicitly specify things such as:
> 
> - `num_inference_steps`
> - `height`
> - `width`
> - `num_frames`
> - `frame_rate`
> 
> That is a good habit in general: make a **small fixed validation set** and stop changing it while comparing checkpoints.
> 
> Something like:
> 
> ```plaintext
> validation/
> prompt_A + seed_A
> prompt_B + seed_B
> prompt_C + seed_C
> 
> ```
> 
> and then compare:
> 
> ```plaintext
> base
> checkpoint_early
> checkpoint_middle
> checkpoint_late
> 
> ```
> 
> under identical inference settings.
> 
> The goal is not to produce a beautiful benchmark. It is just to stop three variables from changing at once.
> 
> There are also real examples where runtime integration mattered. For example, [this Wan 2.2 SwarmUI issue](https://github.com/mcmonkeyprojects/SwarmUI/issues/996) reported LoRA generations that were blurry/noisy in one path while appearing sharp through another backend with similar settings.
> 
> That issue is not evidence that your problem has the same cause; it is just a useful reminder that:
> 
> > blurry output after loading a LoRA does not automatically prove that LoRA training itself failed.

> **Check that the LoRA is actually attached where you think it is**
>
> If your trainer uses PEFT underneath, another cheap sanity check is to verify the adapter structure itself.
> 
> The current [PEFT troubleshooting guide](https://huggingface.co/docs/peft/main/developer_guides/troubleshooting) recommends checking the number of trainable parameters:
> 
> ```python
> model.print_trainable_parameters()
> 
> ```
> 
> and inspecting adapter layers with:
> 
> ```python
> model.get_layer_status()
> 
> ```
> 
> or inspecting the module hierarchy.
> 
> One subtle point from the PEFT documentation is that if **none** of your configured `target_modules` match, PEFT raises an error, but if **some match and some do not** , the unmatched entries can be silently skipped.
> 
> That means “the training ran successfully” does not necessarily prove that the adapter targeted exactly the layers you intended.
> 
> This becomes especially relevant with newer or unusual transformer architectures, or when copying a LoRA config from a different model family.
> 
> So before spending another long training run, I would verify:
> 
> ```plaintext
> expected target modules
> ==
> actual adapter modules
> 
> ```
> 
> and make sure the trainable parameter count is in the range you expected.

> **Trainer settings are not portable vocabulary**
>
> I would also be careful when copying recipes between trainers.
> 
> A setting with the same friendly name can have different semantics in:
> 
> - Diffusers training examples
> - Finetrainers
> - musubi-tuner
> - diffusion-pipe
> - SimpleTuner
> - AI Toolkit
> - other GUI wrappers
> 
> Things such as:
> 
> ```plaintext
> repeats
> epochs
> steps
> FPS
> frame count
> bucket
> cache
> resume
> validation
> LoRA alpha
> network dimension
> timestep range
> 
> ```
> 
> are not necessarily interchangeable just because their labels look similar.
> 
> Even video extraction itself can differ substantially.
> 
> That is why I would start from a **maintained example for your exact base model and task** , get that baseline running, save the exact config and software versions, and only then change one thing.
> 
> Hugging Face’s [Finetrainers](https://github.com/huggingface/finetrainers) is useful here because it has model-specific examples rather than presenting video LoRA as one universal recipe.
> 
> For example, there are separate model guides/examples for:
> 
> - [CogVideoX](https://github.com/huggingface/finetrainers/blob/main/docs/models/cogvideox.md)
> - [HunyuanVideo](https://github.com/huggingface/finetrainers/blob/main/docs/models/hunyuan_video.md)
> - [LTX-Video](https://github.com/huggingface/finetrainers/blob/main/docs/models/ltx_video.md)
> 
> If your actual model has a maintained trainer from its authors or a well-maintained model-specific implementation, I would normally use that as the first baseline rather than translating settings from an unrelated LoRA tutorial.
> 
> A useful reproducibility habit is to keep something like:
> 
> ```plaintext
> base model + exact revision
> trainer + commit/version
> dataset config
> training config
> seed
> checkpoint numbers
> validation prompts
> inference config
> 
> ```
> 
> next to every run.
> 
> That sounds boring, but once you have three LoRA attempts it becomes much more useful than memory.

> **Only after those checks would I tune rank / LR / steps**
>
> Rank, learning rate, alpha, optimizer, timestep sampling and training duration absolutely can matter.
> 
> I just would not make them the **first** explanation for blur.
> 
> A good order is:
> 
> ```plaintext
> 1. establish a known-working pipeline
> 2. inspect processed data
> 3. establish fixed validation
> 4. compare checkpoint progression
> 5. verify adapter structure
> 6. then tune one training variable at a time
> 
> ```
> 
> For example:
> 
> ```plaintext
> Run A: baseline
> Run B: lower learning rate
> Run C: baseline LR, lower rank
> Run D: baseline again, fewer steps
> 
> ```
> 
> is much easier to interpret than:
> 
> ```plaintext
> new LR + new rank + new optimizer + new captions + different resolution
> 
> ```
> 
> in one run.
> 
> There are model-specific published starting points, but they should stay model-specific.
> 
> For example, the [Diffusers CogVideoX training guide](https://huggingface.co/docs/diffusers/training/cogvideox) gives concrete observations about rank, steps, dataset size and learning rate from its experiments, but the guide also explicitly notes that the testing was **not exhaustive** and recommends finding settings appropriate for your data.
> 
> That is probably the right way to treat most “recommended LoRA settings”:
> 
> > a starting point for the same model/trainer regime, not a law of LoRA training.
> 
> The checkpoint sweep is especially useful.
> 
> If:
> 
> ```plaintext
> 500 steps -> sharp but weak concept
> 1000 steps -> good
> 2000 steps -> blurry / oversaturated / distorted
> 
> ```
> 
> then you have learned much more than you would from only evaluating the final checkpoint.
> 
> At that point, reducing effective training strength becomes a much more plausible next experiment.

> **The exact model matters more than it may seem**
>
> Video models are not all variations of the same architecture.
> 
> For example, Wan 2.2 can use a **two-stage denoising setup** with separate high-noise and low-noise transformers.
> 
> The current [Diffusers Wan documentation](https://huggingface.co/docs/diffusers/api/pipelines/wan) exposes:
> 
> ```plaintext
> transformer
> transformer_2
> boundary_ratio
> 
> ```
> 
> and also documents loading LoRA weights into the second denoiser with:
> 
> ```plaintext
> load_into_transformer_2=True
> 
> ```
> 
> So for Wan 2.2, questions such as:
> 
> ```plaintext
> Which denoiser was trained?
> Which denoiser is receiving the LoRA at inference?
> Where is the timestep/noise boundary?
> 
> ```
> 
> can be more important than asking whether rank 32 or 64 is better.
> 
> Likewise, if you are using CogVideoX, I would follow the CogVideoX-specific training/inference path before importing Wan or image-LoRA recipes.
> 
> There is even an open [CogVideoX1.5-5B-I2V report](https://github.com/zai-org/CogVideo/issues/797) where the base/zero-shot output had normal color, while applying a trained LoRA produced an overexposed first frame followed by blurry frames.
> 
> That does **not** mean your case has the same cause—the issue is not a general explanation for LoRA blur—but it is a nice example of why the _shape of the symptom_ matters.
> 
> “Blurry” is more useful if you can distinguish:
> 
> ```plaintext
> every frame is spatially soft
> motion smears but static regions stay sharp
> first frame is abnormal
> quality progressively collapses over time
> colors/contrast blow out
> identity is learned but fine detail disappears
> 
> ```
> 
> Those point toward different branches.

> **A practical beginner route through the LoRA ecosystem**
>
> If you are still learning LoRA itself, I would separate two things:
> 
> ### 1. Learn the adapter mechanics
> 
> PEFT is a good reference for what LoRA is actually targeting and saving:
> 
> [PEFT documentation](https://huggingface.co/docs/peft/index)
> 
> and especially:
> 
> [PEFT troubleshooting](https://huggingface.co/docs/peft/main/developer_guides/troubleshooting)
> 
> You do not need to understand every internal detail, but it helps to know what:
> 
> ```plaintext
> rank
> alpha
> target_modules
> trainable parameters
> adapter weights
> 
> ```
> 
> actually refer to.
> 
> ### 2. Learn diffusion LoRA mechanics
> 
> The [Diffusers LoRA training guide](https://huggingface.co/docs/diffusers/training/lora) is useful for seeing the basic adapter/training structure in a relatively transparent form.
> 
> It is not a universal video-LoRA recipe, but it gives context for what the trainer is doing.
> 
> ### 3. Move to a trainer/example for your exact video model
> 
> Good entry points include:
> 
> - [Hugging Face Finetrainers](https://github.com/huggingface/finetrainers)
> - [LTX-Video-Trainer](https://github.com/Lightricks/LTX-Video-Trainer)
> - [musubi-tuner dataset guide](https://github.com/kohya-ss/musubi-tuner/blob/main/docs/dataset_config.md) if you are in its supported ecosystem
> 
> The LTX dataset guide is particularly educational even outside LTX because it walks through:
> 
> ```plaintext
> scene splitting
> -> captioning
> -> preprocessing
> -> latent caching
> -> decoding for verification
> -> training
> 
> ```
> 
> which is a good mental model for video fine-tuning in general.
> 
> I would learn the **workflow** first, and only then collect model-specific “tips”.
> 
> That avoids the common trap where you know that someone online used:
> 
> ```plaintext
> rank = X
> lr = Y
> steps = Z
> 
> ```
> 
> but not what their dataset, model, preprocessing or trainer actually did.

If you want a very small default workflow, mine would be:

```plaintext
A. Reproduce a maintained example for the exact model/task.

B. Save the exact trainer version + config.

C. Verify the processed training samples.

D. Make 2-4 fixed validation cases.

E. Save several checkpoints, not only the final one.

F. Compare base vs LoRA at several adapter strengths.

G. If quality degrades during training, then tune LR/rank/steps
   one variable at a time.

```

That should give you much more information per training run than randomly searching for “professional LoRA settings”.

And if you post the **base model** , whether this is **T2V or I2V** , the **trainer/version** , and whether the **base-only output is sharp under the same inference settings** , people can probably narrow the tree much further without needing your entire project configuration.