# ValueError: Using fsdp only works in distributed training

**URL:** <https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810>\
**Category:** 🤗Transformers\
**Created:** [June 19, 2023, 3:33pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810 "2023-06-19T15:33:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bright-mb](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@bright-mb](https://discuss.huggingface.co/u/bright-mb)\
**Post date:** [June 19, 2023, 3:33pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/1 "2023-06-19T15:33:35Z")

</div>

I am getting an error when launching the TRL trainer with fsdp. Attaching complete configuration below:

```auto
from transformers import TrainingArguments

output_dir = model_output_dir
per_device_train_batch_size = 6
gradient_accumulation_steps = 8
optim = "paged_adamw_32bit"
save_steps = 50
save_total_limit=3
logging_steps = 10
learning_rate = 2e-4
max_grad_norm = 0.3
warmup_ratio = 0.03
lr_scheduler_type = "cosine_with_restarts"
max_steps = 8000
group_by_length = True
do_fsdp_training = "full_shard"

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    save_total_limit=save_total_limit,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=True,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    # group_by_length=group_by_length,
    lr_scheduler_type=lr_scheduler_type,
    report_to = "tensorboard",
    fsdp= do_fsdp_training
)

math_qa_data_collator = DataCollatorWithPadding(
    tokenizer,
    return_tensors = "pt"
)

from trl import SFTTrainer

max_seq_length = 1024

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_arguments,
    peft_config=peft_config,
    train_dataset=math_qa_dataset["train"],
    formatting_func = dataset_formatting_func,
    max_seq_length=max_seq_length,
    data_collator=math_qa_data_collator,
    packing=True,
)

```

**Stacktrace Snippet**

```auto
File /usr/local/lib/python3.8/dist-packages/transformers/trainer.py:455, in Trainer. __init__ (self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    451 raise ValueError(
    452 "Using --fsdp xxx together with --deepspeed is not possible, deactivate one of those flags."
    453 )
    454 if not args.fsdp_config["xla"] and args.parallel_mode != ParallelMode.DISTRIBUTED:
--> 455 raise ValueError("Using fsdp only works in distributed training.")
    457 # dep_version_check("torch>=1.12.0")
    458 # Would have to update setup.py with torch>=1.12.0
    459 # which isn't ideally given that it will force people not using FSDP to also use torch>=1.12.0
    460 # below is the current alternative.
    461 if version.parse(version.parse(torch. __version__ ).base_version) < version.parse("1.12.0"):

ValueError: Using fsdp only works in distributed training.

```

---

<div class="post-metadata">

**Author:** ![Iamexperimenting](https://avatars.discourse-cdn.com/v4/letter/i/839c29/32.png) [@Iamexperimenting](https://discuss.huggingface.co/u/Iamexperimenting)\
**Post date:** [April 4, 2024, 8:07pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/2 "2024-04-04T20:07:20Z")

</div>

@nielsr @muellerzr can you please help us here? I’m also facing the same issue.

I’m trying gemma model by following this example [https://huggingface.co/google/gemma-7b/blob/main/examples/example\_fsdp.py](https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py)

However, I’m using Nvidia GPU

```auto
import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model_id = "google/gemma-7b"

# Load the pretrained model and tokenizer.
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map={"":0})

# Set up PEFT LoRA for fine-tuning.
lora_config = LoraConfig(
    r=8,
    target_modules=["k_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

# Load the dataset and format it for training.
data = load_dataset("Abirate/english_quotes", split="train")
max_seq_length = 1024

# Set up the FSDP config. To enable FSDP via SPMD, set xla_fsdp_v2 to True.
fsdp_config = {"fsdp_transformer_layer_cls_to_wrap": [
        "GemmaDecoderLayer"
    ]}

# Finally, set up the trainer and train the model.
trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    args=TrainingArguments(
        per_device_train_batch_size=64, # This is actually the global batch size for SPMD.
        num_train_epochs=100,
        max_steps=-1,
        output_dir="./output",
        optim="adafactor",
        logging_steps=1,
        dataloader_drop_last = True, # Required for SPMD.
        fsdp="full_shard",
        fsdp_config=fsdp_config,
    ),
    peft_config=lora_config,
    dataset_text_field="quote",
    max_seq_length=max_seq_length,
    packing=True,
)

trainer.train()

```

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [June 3, 2024, 1:26pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/3 "2024-06-03T13:26:00Z")

</div>

Hi,

How many GPUs do you have? You need at least 2 as FSDP is meant for distributed training.

---

<div class="post-metadata">

**Author:** ![LidorPrototype](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@LidorPrototype](https://discuss.huggingface.co/u/LidorPrototype)\
**Post date:** [June 16, 2024, 5:58pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/4 "2024-06-16T17:58:22Z")

</div>

Hello, I’m also trying with Gemma and I have 4 GPUs, and it still wont work for me

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [June 17, 2024, 5:08am UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/5 "2024-06-17T05:08:42Z")

</div>

@LidorPrototype I think you need to init a config file first, as explained in [the docs](https://huggingface.co/docs/peft/accelerate/fsdp) for FSPD training. Also there is an example script [here](https://github.com/huggingface/peft/tree/main/examples/sft)

---

<div class="post-metadata">

**Author:** ![LidorPrototype](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@LidorPrototype](https://discuss.huggingface.co/u/LidorPrototype)\
**Post date:** [June 18, 2024, 2:03pm UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/6 "2024-06-18T14:03:00Z")

</div>

I followed those docs, and created my fsdp config file, and again when I tried to run my trainer with `trainer.train()` I got that error `ValueError: Using fsdp only works in distributed training.`

Here is my TrainingArguments:

```auto
training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    optim="paged_adamw_8bit", # paged_adamw_32bit
    learning_rate=0.0002, # 0.001
    fp16=True,
    max_grad_norm=0.3,
    warmup_ratio=0.03,
    lr_scheduler_type="constant",
    logging_steps=LOGS,
    save_steps=SAVES,
    max_steps=EPOCHS,
    disable_tqdm=True,
    fsdp="full_shard",
    fsdp_config="fsdp_config.json"
)

```

---

<div class="post-metadata">

**Author:** ![antonpolishko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antonpolishko/32/29772_2.png) [@antonpolishko](https://discuss.huggingface.co/u/antonpolishko)\
**Post date:** [July 24, 2024, 2:43am UTC](https://discuss.huggingface.co/t/valueerror-using-fsdp-only-works-in-distributed-training/43810/7 "2024-07-24T02:43:25Z")

</div>

Having the same issue with Gemma and 8x Nvidia H100 GPUs. Code is based on the same [example\_fsdp.py](https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py)

Accelerator test returns success:

```auto
> accelerate test
...
Test is a success! You are ready for your distributed training!

```

The same script works on TPUs using GCP, with added

```auto
fsdp_config = {
    "fsdp_transformer_layer_cls_to_wrap": [
        "GemmaDecoderLayer"
    ],
    "xla": True,
    "xla_fsdp_v2": True,
    "xla_fsdp_grad_ckpt": True
}

```