# Using gradient\_accumulation\_steps does not give the same results

**URL:** <https://discuss.huggingface.co/t/using-gradient-accumulation-steps-does-not-give-the-same-results/31992>\
**Category:** 🤗Accelerate\
**Created:** [February 18, 2023, 7:14pm UTC](https://discuss.huggingface.co/t/using-gradient-accumulation-steps-does-not-give-the-same-results/31992 "2023-02-18T19:14:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\
**Post date:** [February 18, 2023, 7:14pm UTC](https://discuss.huggingface.co/t/using-gradient-accumulation-steps-does-not-give-the-same-results/31992/1 "2023-02-18T19:14:26Z")

</div>

I am using the summarization [example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization_no_trainer.py)

this is my script for traing using 3 gpus  
`accelerate launch --config_file='./accelerate.yaml' run_summarization_no_trainer_test.py --seed=42 --preprocessing_num_workers=1 --weight_decay='0.001' --output_dir="samsum_exp/t5/base_3" --per_device_train_batch_size=1 --per_device_eval_batch_size=1 --num_train_epochs=10 --model_name_or_path='t5-base' --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch' --dataset_name="samsum" --source_prefix "summarize: " --gradient_accumulation_steps=2`

using 6 gpus

`accelerate launch --config_file='./accelerate.yaml' run_summarization_no_trainer_test.py --seed=42 --preprocessing_num_workers=1 --weight_decay='0.001' --output_dir="samsum_exp/t5/base_6" --per_device_train_batch_size=1 --per_device_eval_batch_size=1 --num_train_epochs=10 --model_name_or_path='t5-base' --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch' --dataset_name="samsum" --source_prefix "summarize: " --gradient_accumulation_steps=1`

The resuled metrics are as in figure below:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0aebcda070976c4f686ebd4505d38b0a71a7912f.png)

According to this [answe](https://discuss.huggingface.co/t/question-about-gradient-accumulation-step-in-trainer/9876) It supposed to get identical result but it is not.