# 🤗Accelerate

**URL:** https://discuss.huggingface.co/c/accelerate/18.md

[Latest](https://discuss.huggingface.co/latest.md) · [Categories](https://discuss.huggingface.co/categories.md)

---

## [About the 🤗Accelerate category](https://discuss.huggingface.co/t/about-the-accelerate-category/5559)

<div class="topic-metadata">

**Author:** [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Replies:** 1\
**Last updated:** [February 20, 2022, 7:57pm UTC](https://discuss.huggingface.co/t/about-the-accelerate-category/5559 "2022-02-20T19:57:27Z")

</div>

This category is for any question related to the Accelerate library. You can also file an issue.

---

## [FastLoRA v4.2 — Fine-tuning library that never crashes (pip installable)](https://discuss.huggingface.co/t/fastlora-v4-2-fine-tuning-library-that-never-crashes-pip-installable/174423)

<div class="topic-metadata">

**Author:** [@omurberaisik](https://discuss.huggingface.co/u/omurberaisik)\
**Replies:** 0\
**Last updated:** [March 20, 2026, 8:41am UTC](https://discuss.huggingface.co/t/fastlora-v4-2-fine-tuning-library-that-never-crashes-pip-installable/174423 "2026-03-20T08:41:11Z")

</div>

Hi everyone! I built FastLoRA, a drop-in alternative for fine-tuning LLMs. bash pip install "fastlora\[full\]" Why I built it: Unsloth kept breaking on install and crashing mid-training Wanted something that just …

---

## [BitsandBytes conflict with Accelerate](https://discuss.huggingface.co/t/bitsandbytes-conflict-with-accelerate/150275)

<div class="topic-metadata">

**Author:** [@tyrleng](https://discuss.huggingface.co/u/tyrleng)\
**Replies:** 7\
**Last updated:** [March 6, 2026, 2:51am UTC](https://discuss.huggingface.co/t/bitsandbytes-conflict-with-accelerate/150275 "2026-03-06T02:51:55Z")

</div>

I’m running inference on a custom VLM derived model. Inference works fine when using the weights in their bfloat16 precision. However, when I try defining a BitsandBytes config, I receive errors that I suspect is due to…

---

## [FSDP Auto Wrap does not work using \`accelerate\` in Multi-GPU Setup](https://discuss.huggingface.co/t/fsdp-auto-wrap-does-not-work-using-accelerate-in-multi-gpu-setup/105729)

<div class="topic-metadata">

**Author:** [@micron-dickywoo](https://discuss.huggingface.co/u/micron-dickywoo)\
**Replies:** 2\
**Last updated:** [December 31, 2025, 8:17am UTC](https://discuss.huggingface.co/t/fsdp-auto-wrap-does-not-work-using-accelerate-in-multi-gpu-setup/105729 "2025-12-31T08:17:10Z")

</div>

Introduction Hi, I’m trying to fine-tune LLama3-8B with the accelerate package using FSDP and flash-attention. I followed the official guide to config my accelerate with FSDP setting (Fully Sharded Data Parallel). Howev…

---

## [Accelerate + Gemma2 + FSDP](https://discuss.huggingface.co/t/accelerate-gemma2-fsdp/103852)

<div class="topic-metadata">

**Author:** [@cinjonr](https://discuss.huggingface.co/u/cinjonr)\
**Replies:** 4\
**Last updated:** [December 28, 2025, 3:16am UTC](https://discuss.huggingface.co/t/accelerate-gemma2-fsdp/103852 "2025-12-28T03:16:21Z")

</div>

I have a pretty standard accelerate setup with Gemma2-9B. I am trying to finetune it on an 8-gpu A100 node with the following config, using per\_gpu batch size 1 and a total size of 8. However, I can only get up to contex…

---

## [Error on fsdp2 trainer with fsdp\_cpu\_ram\_efficient\_loading = True](https://discuss.huggingface.co/t/error-on-fsdp2-trainer-with-fsdp-cpu-ram-efficient-loading-true/171753)

<div class="topic-metadata">

**Author:** [@enmixkesi](https://discuss.huggingface.co/u/enmixkesi)\
**Replies:** 1\
**Last updated:** [December 19, 2025, 6:00am UTC](https://discuss.huggingface.co/t/error-on-fsdp2-trainer-with-fsdp-cpu-ram-efficient-loading-true/171753 "2025-12-19T06:00:07Z")

</div>

Hi all, I had a quick question. There’s an error when I try to train my model using fsdp2 with the setting: fsdp\_cpu\_ram\_efficient\_loading = True. It will raise error: \[rank0\]: Traceback (most recent call last): \[rank0…

---

## [Confusion with sequence parallelism](https://discuss.huggingface.co/t/confusion-with-sequence-parallelism/171630)

<div class="topic-metadata">

**Author:** [@shon-ot](https://discuss.huggingface.co/u/shon-ot)\
**Replies:** 3\
**Last updated:** [December 16, 2025, 11:54pm UTC](https://discuss.huggingface.co/t/confusion-with-sequence-parallelism/171630 "2025-12-16T23:54:50Z")

</div>

Hi everyone, I’m training a model using SFTTrainer with 4 GPUs and I’m confused about the number of optimization steps reported and in general how sequence parallel works in accelerate. Here’s my configuration: trainin…

---

## [Accelerate consumes more memory](https://discuss.huggingface.co/t/accelerate-consumes-more-memory/170588)

<div class="topic-metadata">

**Author:** [@DhineshR](https://discuss.huggingface.co/u/DhineshR)\
**Replies:** 4\
**Last updated:** [November 19, 2025, 5:18am UTC](https://discuss.huggingface.co/t/accelerate-consumes-more-memory/170588 "2025-11-19T05:18:52Z")

</div>

Hey Folks, I’m trying to fine-tune qwen3-0.6B on gsm8k data using Trainer and Accelerate on a single GPU that has ~23GB memory. Here is my training script. When I launch the script with python \<script.py\>, the training…

---

## [Bug when using gradient accumulation with accelerate](https://discuss.huggingface.co/t/bug-when-using-gradient-accumulation-with-accelerate/170057)

<div class="topic-metadata">

**Author:** [@enmixkesi](https://discuss.huggingface.co/u/enmixkesi)\
**Replies:** 1\
**Last updated:** [November 7, 2025, 1:59pm UTC](https://discuss.huggingface.co/t/bug-when-using-gradient-accumulation-with-accelerate/170057 "2025-11-07T13:59:49Z")

</div>

in GradientState, the plugin\_kwargs will only update when the the kwats of gradient\_accumulation\_plugin is different default GradientAccumulationPlugin kwargs , and in GradientAccumulationPlugin,the default kwargs is num…

---

## [Load\_state() with custom objects and scheduler](https://discuss.huggingface.co/t/load-state-with-custom-objects-and-scheduler/169759)

<div class="topic-metadata">

**Author:** [@rogercc](https://discuss.huggingface.co/u/rogercc)\
**Replies:** 1\
**Last updated:** [November 1, 2025, 12:34am UTC](https://discuss.huggingface.co/t/load-state-with-custom-objects-and-scheduler/169759 "2025-11-01T00:34:39Z")

</div>

Hi! I am having some issues understanding why I can’t resume training corectly with this code: optimizer = optim.AdamW(params, lr=args.learning\_rate) scheduler = get\_linear\_schedule\_with\_warmup(optimizer, num\_warmup\_s…

---

## [What is the best way to save the state of a model and optimizer, when the model has 2 LoRas?](https://discuss.huggingface.co/t/what-is-the-best-way-to-save-the-state-of-a-model-and-optimizer-when-the-model-has-2-loras/169458)

<div class="topic-metadata">

**Author:** [@rogercc](https://discuss.huggingface.co/u/rogercc)\
**Replies:** 3\
**Last updated:** [October 30, 2025, 11:56pm UTC](https://discuss.huggingface.co/t/what-is-the-best-way-to-save-the-state-of-a-model-and-optimizer-when-the-model-has-2-loras/169458 "2025-10-30T23:56:00Z")

</div>

I have seen in PEFT they recommend using model.save\_pretrained(), but I am using Accelerate and I also want to save the optimizer state, and I wonder if I can just use accelerator.save\_state(). this is my (pseudo)code: …

---

## [How does Accelerate ensure uniqueness of data samples across GPUs?](https://discuss.huggingface.co/t/how-does-accelerate-ensure-uniqueness-of-data-samples-across-gpus/44058)

<div class="topic-metadata">

**Author:** [@mdrpanwar](https://discuss.huggingface.co/u/mdrpanwar)\
**Replies:** 3\
**Last updated:** [October 30, 2025, 10:25pm UTC](https://discuss.huggingface.co/t/how-does-accelerate-ensure-uniqueness-of-data-samples-across-gpus/44058 "2025-10-30T22:25:44Z")

</div>

Hi, I have the following questions about the inner workings of Accelerate. If there is an existing document answering these, please link to it. When using DDP, how does Accelerate sample the data from the dataloader …

---

## [Loss spike when resuming from FSDP SHARDED\_STATE\_DICT checkpoint (possible optimizer-state mismatch)](https://discuss.huggingface.co/t/loss-spike-when-resuming-from-fsdp-sharded-state-dict-checkpoint-possible-optimizer-state-mismatch/160974)

<div class="topic-metadata">

**Author:** [@mjkmain](https://discuss.huggingface.co/u/mjkmain)\
**Replies:** 4\
**Last updated:** [October 13, 2025, 8:22am UTC](https://discuss.huggingface.co/t/loss-spike-when-resuming-from-fsdp-sharded-state-dict-checkpoint-possible-optimizer-state-mismatch/160974 "2025-10-13T08:22:19Z")

</div>

Hi Accelerate team — first off, thank you for the phenomenal work on the library and its FSDP integration. I’m running large-scale multi-node jobs and have hit what looks like a systematic resume-instability whenever I …

---

## [Do I need to divide the loss by num\_processes when I set split\_batches True?](https://discuss.huggingface.co/t/do-i-need-to-divide-the-loss-by-num-processes-when-i-set-split-batches-true/168374)

<div class="topic-metadata">

**Author:** [@enyoukai](https://discuss.huggingface.co/u/enyoukai)\
**Replies:** 1\
**Last updated:** [September 14, 2025, 12:07pm UTC](https://discuss.huggingface.co/t/do-i-need-to-divide-the-loss-by-num-processes-when-i-set-split-batches-true/168374 "2025-09-14T12:07:13Z")

</div>

I have the following code: accelerator\_dataloader\_config = DataLoaderConfiguration( dispatch\_batches=True, split\_batches=True, ) accelerator = Accelerator(log\_with="wandb", dataloader\_config=accelerator\_dataloa…

---

## [Perform knowledge distillation using accelerate](https://discuss.huggingface.co/t/perform-knowledge-distillation-using-accelerate/33078)

<div class="topic-metadata">

**Author:** [@allanjie](https://discuss.huggingface.co/u/allanjie)\
**Replies:** 1\
**Last updated:** [August 14, 2025, 10:05am UTC](https://discuss.huggingface.co/t/perform-knowledge-distillation-using-accelerate/33078 "2025-08-14T10:05:26Z")

</div>

I’m working on knowledge distillation from a relatively large model 11B to 2B model, for example, under DDP environment. Usually, we will do the following to prepare the model model, optimizer = accelerator.prepare(mod…

---

## [How to Setup Deferred Init with Accelerate + DeepSpeed?](https://discuss.huggingface.co/t/how-to-setup-deferred-init-with-accelerate-deepspeed/81512)

<div class="topic-metadata">

**Author:** [@jasonkrone](https://discuss.huggingface.co/u/jasonkrone)\
**Replies:** 6\
**Last updated:** [August 11, 2025, 2:11am UTC](https://discuss.huggingface.co/t/how-to-setup-deferred-init-with-accelerate-deepspeed/81512 "2025-08-11T02:11:01Z")

</div>

I’d like to defer initialization of my model until after DeepSpeed finishes sharding to avoid running OOM. Any tips on how to set that up? I found the init\_empty\_weights method to help put the model on the “meta” device…

---

## [How to get the grad norm of a deepspeed-zero3 model after accelerator.prepare()](https://discuss.huggingface.co/t/how-to-get-the-grad-norm-of-a-deepspeed-zero3-model-after-accelerator-prepare/73107)

<div class="topic-metadata">

**Author:** [@Hannibal046](https://discuss.huggingface.co/u/Hannibal046)\
**Replies:** 2\
**Last updated:** [July 23, 2025, 2:38am UTC](https://discuss.huggingface.co/t/how-to-get-the-grad-norm-of-a-deepspeed-zero3-model-after-accelerator-prepare/73107 "2025-07-23T02:38:02Z")

</div>

Hi, I want to ask how to get the grad norm of a prepared model especially with DeepSpeed-Zero3. The model.parameters() is empty. Thanks!

---

## [Problem with full-finetuning on cluster](https://discuss.huggingface.co/t/problem-with-full-finetuning-on-cluster/160571)

<div class="topic-metadata">

**Author:** [@bartMarek](https://discuss.huggingface.co/u/bartMarek)\
**Replies:** 1\
**Last updated:** [June 25, 2025, 2:54am UTC](https://discuss.huggingface.co/t/problem-with-full-finetuning-on-cluster/160571 "2025-06-25T02:54:15Z")

</div>

I am trying to fully fine-tune the model, giving audio + text + image as input. The script works only if the fine-tuning mode is lora (~800k parameters). Otherwise, I got multiple different types of errors. Note: Due to…

---

## [Transformers Trainer + Accelerate FSDP: How do I load my model from a checkpoint?](https://discuss.huggingface.co/t/transformers-trainer-accelerate-fsdp-how-do-i-load-my-model-from-a-checkpoint/61585)

<div class="topic-metadata">

**Author:** [@mgerstgrasser](https://discuss.huggingface.co/u/mgerstgrasser)\
**Replies:** 3\
**Last updated:** [June 22, 2025, 6:05pm UTC](https://discuss.huggingface.co/t/transformers-trainer-accelerate-fsdp-how-do-i-load-my-model-from-a-checkpoint/61585 "2025-06-22T18:05:34Z")

</div>

Hi all, I’ve fine-tuned a Llama2 model using the transformers Trainer class, plus accelerate and FSDP, with a sharded state dict. Now my checkpoint directories all have the model’s state dict sharded across multiple .di…

---

## [NCCL Timeout Accelerate Load From Checkpoint](https://discuss.huggingface.co/t/nccl-timeout-accelerate-load-from-checkpoint/33953)

<div class="topic-metadata">

**Author:** [@zpn](https://discuss.huggingface.co/u/zpn)\
**Replies:** 2\
**Last updated:** [June 20, 2025, 7:40am UTC](https://discuss.huggingface.co/t/nccl-timeout-accelerate-load-from-checkpoint/33953 "2025-06-20T07:40:31Z")

</div>

Hi all, I had a quick question. I’m having issues when I try to resume from a checkpoint when using an IterableDataset. It will get to the first accelerate.backward step, then fail with an NCCL timeout like \[E ProcessGr…

---

## [Not seeing memory benefit to accelerate/FSDP2](https://discuss.huggingface.co/t/not-seeing-memory-benefit-to-accelerate-fsdp2/158039)

<div class="topic-metadata">

**Author:** [@hpcpony](https://discuss.huggingface.co/u/hpcpony)\
**Replies:** 2\
**Last updated:** [June 18, 2025, 3:49pm UTC](https://discuss.huggingface.co/t/not-seeing-memory-benefit-to-accelerate-fsdp2/158039 "2025-06-18T15:49:22Z")

</div>

TL;DR Why doesn’t Acclerate/FSDP seem to be doing much of anything to reduce memory in the following? I’m trying to get some hands-on and learn how to run large models across multiple nodes and/or GPUs. I’m starting wit…

---

## [DistributedSampler with Accelerate](https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787)

<div class="topic-metadata">

**Author:** [@zaryabmakram](https://discuss.huggingface.co/u/zaryabmakram)\
**Replies:** 1\
**Last updated:** [June 10, 2025, 4:15pm UTC](https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787 "2025-06-10T16:15:45Z")

</div>

I have noticed that using torch.utils.data.distributed.DistributedSampler as the sampler in the dataloader, accelerate changes its length after calling accelerate.prepare(). Following is how Im setting up the dataloade…

---

## [Where can I find the full list of parameters for the Accelerate yaml config?](https://discuss.huggingface.co/t/where-can-i-find-the-full-list-of-parameters-for-the-accelerate-yaml-config/158053)

<div class="topic-metadata">

**Author:** [@alckasoc](https://discuss.huggingface.co/u/alckasoc)\
**Replies:** 3\
**Last updated:** [June 5, 2025, 2:48am UTC](https://discuss.huggingface.co/t/where-can-i-find-the-full-list-of-parameters-for-the-accelerate-yaml-config/158053 "2025-06-05T02:48:40Z")

</div>

Is there a full list of parameters for the yaml config file? I can’t find it. There’s one on the CLI: The Command Line. What are the differences between: the plugin specifying the config via a yaml via flags Are they …

---

## [Synchronizing State, Trainer and Accelerate](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255)

<div class="topic-metadata">

**Author:** [@donb](https://discuss.huggingface.co/u/donb)\
**Replies:** 2\
**Last updated:** [May 22, 2025, 4:47pm UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255 "2025-05-22T16:47:29Z")

</div>

Using Trainer, and it appears that if I load any class from accelerate, the Trainer doesn’t perform its accelerate magic behind the scenes, meaning I get an error like this: \[rank1\]: File "/opt/code/repos/MyProject/.v…

---

## [\[RuntimeError\] DPOTrainer - "element 0 of tensors does not require grad and does not have a grad\_fn" on 8x A100 GPUs](https://discuss.huggingface.co/t/runtimeerror-dpotrainer-element-0-of-tensors-does-not-require-grad-and-does-not-have-a-grad-fn-on-8x-a100-gpus/156010)

<div class="topic-metadata">

**Author:** [@xmriz](https://discuss.huggingface.co/u/xmriz)\
**Replies:** 1\
**Last updated:** [May 20, 2025, 2:02pm UTC](https://discuss.huggingface.co/t/runtimeerror-dpotrainer-element-0-of-tensors-does-not-require-grad-and-does-not-have-a-grad-fn-on-8x-a100-gpus/156010 "2025-05-20T14:02:24Z")

</div>

Hi all, I’m encountering a critical issue when running DPOTrainer on a multi-GPU A100 server (8x A100 40GB) using trl==0.17.0 and transformers==4.51.3. The training fails on all ranks with the same RuntimeError during t…

---

## [Reproduce SFTTrainer with Accelerate and Pytorch](https://discuss.huggingface.co/t/reproduce-sfttrainer-with-accelerate-and-pytorch/155746)

<div class="topic-metadata">

**Author:** [@Totototo](https://discuss.huggingface.co/u/Totototo)\
**Replies:** 0\
**Last updated:** [May 18, 2025, 11:39am UTC](https://discuss.huggingface.co/t/reproduce-sfttrainer-with-accelerate-and-pytorch/155746 "2025-05-18T11:39:24Z")

</div>

Hello everyone, SFTTrainer containing lots (and lots..) of parameters, I was aiming to see if I can reproduce its results with pure pytorch and accelerate. I have come up with a first good version but don’t exaclty rep…

---

## [11B model gets OOM after using deepspeed zero 3 setting with 8 32G V100](https://discuss.huggingface.co/t/11b-model-gets-oom-after-using-deepspeed-zero-3-setting-with-8-32g-v100/80748)

<div class="topic-metadata">

**Author:** [@Charlie911](https://discuss.huggingface.co/u/Charlie911)\
**Replies:** 2\
**Last updated:** [April 26, 2025, 1:12pm UTC](https://discuss.huggingface.co/t/11b-model-gets-oom-after-using-deepspeed-zero-3-setting-with-8-32g-v100/80748 "2025-04-26T13:12:33Z")

</div>

Can anyone help me with one-node distributed training? The idea of my custom model is to merge some layers of 2 7B llama models like MOE models do. So that yields an 11B model. I’m loading the model in torch.float16 exce…

---

## [Multi-gpu inference llama-3.2 vision with QLoRA](https://discuss.huggingface.co/t/multi-gpu-inference-llama-3-2-vision-with-qlora/150685)

<div class="topic-metadata">

**Author:** [@shon-ot](https://discuss.huggingface.co/u/shon-ot)\
**Replies:** 4\
**Last updated:** [April 25, 2025, 4:12pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-llama-3-2-vision-with-qlora/150685 "2025-04-25T16:12:54Z")

</div>

Hello :slight\_smile: After fine-tuning meta-llama/Llama-3.2-11B-Vision-Instruct I run into a weird error while running inference with multi-gpu. This is how I loads the model: bnb\_config = BitsAndBytesConfig( load…

---

## [How to work with meta tensors?](https://discuss.huggingface.co/t/how-to-work-with-meta-tensors/60461)

<div class="topic-metadata">

**Author:** [@atyshka](https://discuss.huggingface.co/u/atyshka)\
**Replies:** 1\
**Last updated:** [April 16, 2025, 7:23pm UTC](https://discuss.huggingface.co/t/how-to-work-with-meta-tensors/60461 "2025-04-16T19:23:42Z")

</div>

I’m trying to quantize a CodeLlama model to int8 using smoothquant. I have the circular issue that I need to calibrate the model in fp16, but I want to quantize it in the first place because it won’t fit in GPU memory. I…

---

## [Issues with Dataset Loading and Checkpoint Saving using FSDP with HuggingFace Trainer on SLURM Multi-Node Setup](https://discuss.huggingface.co/t/issues-with-dataset-loading-and-checkpoint-saving-using-fsdp-with-huggingface-trainer-on-slurm-multi-node-setup/149290)

<div class="topic-metadata">

**Author:** [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\
**Replies:** 1\
**Last updated:** [April 7, 2025, 2:54pm UTC](https://discuss.huggingface.co/t/issues-with-dataset-loading-and-checkpoint-saving-using-fsdp-with-huggingface-trainer-on-slurm-multi-node-setup/149290 "2025-04-07T14:54:18Z")

</div>

Hello, I’m training a model using a SLURM multi-node setup with accelerate, HuggingFace Trainer, FlashAttention2, and FSDP. I’m working on a shared filesystem, which is mounted on all nodes so that every process can acc…

[Next page](https://discuss.huggingface.co/c/accelerate/18.md?page=1)