# Problem with full-finetuning on cluster **URL:** \ **Category:** 🤗Accelerate\ **Created:** [June 24, 2025, 9:20pm UTC](https://discuss.huggingface.co/t/problem-with-full-finetuning-on-cluster/160571 "2025-06-24T21:20:16Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![bartMarek](https://avatars.discourse-cdn.com/v4/letter/b/e99b99/32.png) [@bartMarek](https://discuss.huggingface.co/u/bartMarek)\ **Post date:** [June 24, 2025, 9:20pm UTC](https://discuss.huggingface.co/t/problem-with-full-finetuning-on-cluster/160571/1 "2025-06-24T21:20:16Z")
I am trying to fully fine-tune the model, giving audio + text + image as input. The script works only if the fine-tuning mode is lora (~800k parameters). Otherwise, I got multiple different types of errors. `Note: Due to the computational constraints, I created a case called hybrid, when I unfreeze embeddings and LoRA (1.5B)` The problem exists only in the cluster with `torchrun` and docker. I could agree that it might be the problem if I used multiple GPUs, but I rewrote everything to try it on a single one. ```auto def configure_model_for_training(model, args): """Configure model for training based on finetuning mode and modalities""" torch.cuda.empty_cache() gc.collect() if args.finetuning_mode == 'full': print("Enabling full finetuning") for param in model.parameters(): if param not in model.model.embed_tokens_extend.audio_embed.parameters(): param.requires_grad = True elif args.finetuning_mode == 'lora': print("LoRA") else: print("Test") if args.use_audio and hasattr(model, 'model') and hasattr(model.model, 'embed_tokens_extend'): if hasattr(model.model.embed_tokens_extend, 'audio_embed'): for param in model.model.embed_tokens_extend.audio_embed.parameters(): param.requires_grad = True print("Enabled gradients for audio embedding layer") if args.use_image and hasattr(model, 'model') and hasattr(model.model, 'embed_tokens_extend'): if hasattr(model.model.embed_tokens_extend, 'image_embed'): for param in model.model.embed_tokens_extend.image_embed.parameters(): param.requires_grad = True print("Enabled gradients for image embedding layer") print("\nTrainable parameters:") for name, param in model.named_parameters(): if param.requires_grad: print(f" - {name}") model = create_model( model_path, use_flash_attention=args.use_flash_attention, ) model.train() configure_model_for_training(model, args) run_name = f"{train_dataset.name}_{args.finetuning_mode}_{args.model_name_or_path}_{args.number_of_train_users}_{args.num_train_epochs}-{args.max_steps}_{args.number_of_eval_users}_{args.batch_size}_{args.learning_rate}_{args.wd}" output_dir = Path(args.output_dir) / run_name output_dir.mkdir(parents=True, exist_ok=True) training_args = TrainingArguments( num_train_epochs=args.num_train_epochs, max_steps=args.max_steps, per_device_train_batch_size=args.batch_size_per_gpu, gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}, gradient_accumulation_steps=gradient_accumulation_steps, optim='adamw_torch', adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-7, learning_rate=args.learning_rate, weight_decay=args.wd, max_grad_norm=1.0, lr_scheduler_type='linear', warmup_steps=50, logging_steps=10, output_dir=str(output_dir), save_strategy='no', save_total_limit=5, save_only_model=True, bf16=bf16, fp16=fp16, remove_unused_columns=False, report_to='wandb' if not args.no_wandb else None, run_name=run_name, deepspeed=None, disable_tqdm=not args.tqdm, dataloader_num_workers=4, ddp_find_unused_parameters=True ) trainer = Trainer( model=model, args=training_args, data_collator=multi_modal_qa_collate_fn, train_dataset=train_dataset, eval_dataset=eval_dataset, callbacks=callbacks ) # Training logger.info(f"Starting {train_dataset.name} training with {args.finetuning_mode} finetuning") trainer.train() ``` I got multiple types of errors (only applied for the test & full mode): 1. with this code I got: `[rank0]: RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations. [rank0]: Parameter at index 1329 with name model.embed_tokens_extend.audio_embed.encoder.encoders.23._checkpoint_wrapped_module.layer_norm.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration.` 2. gradient\_checkpointing = False, then: AttributeError: ‘SiglipEncoder’ object has no attribute ‘\_gradient\_checkpointing\_func’. Did you mean: ‘gradient\_checkpointing’? 3. ddp\_find\_unused\_parameters=False but then I got: `Parameter indices which did not receive grad for rank 0: 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 1331 1332 1333 1334 1340 1342 1344 1346 1348 1350 1352 1354 1356 1358 1360 1362 1364 1366 1368 1370 1372 1374 1376 1378 1380 1382 1384 1386 1388 1390 1392 1394 1396 1398 1400 1402 1404 1406 1408 1410 1412 1414 1416 1418 1420 1422 1424 1426 1428 1430 1432 1434 1436 1438 1440 1442 1444 1446 1448 1450 1452 1454 1456 1458 1460 1462 1464 1466 1468 1470 1472 ...` I call the script: `torchrun --nproc_per_node=1 $@` setup: - Python3.12 / Python3.10 (I had dockers with both, original from nvidia) - transformers 4.46.3 - torch2.6 / torch 2.3 - datasets - both dockers work well for LoRA - datasets==3.5.0 - peft==0.15.2 - accelerate==1.6.0 Note: I tried also with higher versions of transformers and accelerate. Now, I am asking myself the same question I always ask myself before going to bed: what did I do wrong this time? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 25, 2025, 2:54am UTC](https://discuss.huggingface.co/t/problem-with-full-finetuning-on-cluster/160571/2 "2025-06-25T02:54:15Z")
I wonder if there is a compatibility issue between `gradient_checkpointing` and DDP… > > > Hello! > > I found that when I try to enable gradient\_checking with model.train()…, a runtime error will be raised. > > Hope someone will give me some feedback to solve it. > > Thanks! > > > \- Minimal PoC Code > > \`\`\`python > \# grad\_checking.py > import torch > import transformers > from accelerate import Accelerator > from transformers import AutoModel, AutoTokenizer > > \# seems not related to a specific model, longformer/gpt2 can also lead to the crash > model\_path = "facebook/opt-125m" > > accelerator = Accelerator() > > x = \['whoami', 'hello world'\] > y = \['i am who', 'world hello'\] > tokenizer = AutoTokenizer.from\_pretrained(model\_path, local\_files\_only=True) > x = tokenizer(x, padding=True, return\_tensors='pt') > y = tokenizer(y, padding=True, return\_tensors='pt') > > model = AutoModel.from\_pretrained(model\_path, local\_files\_only=True) > model.gradient\_checkpointing\_enable() > model = accelerator.prepare(model) > > \# when gradient\_checkpointing is enabled, this line below will lead to the error > model.train() > > x\_embed = model(\*\*x, return\_dict=True).last\_hidden\_state\[:, -1, :\] > y\_embed = model(\*\*y, return\_dict=True).last\_hidden\_state\[:, -1, :\] > logits = torch.matmul(x\_embed, y\_embed.t()) > loss = transformers.models.clip.modeling\_clip.clip\_loss(logits) > accelerator.backward(loss) > \`\`\` > > command line to launch the script: > \`TORCH\_DISTRIBUTED\_DEBUG=DETAIL accelerate launch grad\_checking.py\` > > \`\`\` > RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the \`forward\` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use \_set\_static\_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple \`checkpoint\` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use \_set\_static\_graph() as a workaround if your module graph does not change over iterations. > Parameter at index 190 with name decoder.layers.11.fc2.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration. > \`\`\` > > > \- Environment > > \`\`\`bash > \- \`Accelerate\` version: 0.9.0 > \- Platform: Linux-5.4.32-1-tlinux4-0001-x86\_64-with-glibc2.2.5 > \- Python version: 3.8.3 > \- Numpy version: 1.22.4 > \- PyTorch version (GPU?): 1.11.0+cu113 (True) > \- \`Accelerate\` default config: > - compute\_environment: LOCAL\_MACHINE > - distributed\_type: MULTI\_GPU > - mixed\_precision: fp16 > - use\_cpu: False > - num\_processes: 1 > - machine\_rank: 0 > - num\_machines: 1 > - main\_process\_ip: None > - main\_process\_port: None > - main\_training\_function: main > - deepspeed\_config: {} > - fsdp\_config: {} > \`\`\`