# Multi-gpu training does not optimize as expected **URL:** \ **Category:** Beginners\ **Created:** [November 10, 2023, 2:59am UTC](https://discuss.huggingface.co/t/multi-gpu-training-does-not-optimize-as-expected/61738 "2023-11-10T02:59:39Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![nghosh](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@nghosh](https://discuss.huggingface.co/u/nghosh)\ **Post date:** [November 10, 2023, 2:59am UTC](https://discuss.huggingface.co/t/multi-gpu-training-does-not-optimize-as-expected/61738/1 "2023-11-10T02:59:39Z")
I am a bit confused about the behavior of multi-gpu training using accelerate. I am training gpt2 small on the MNLI dataset using the run\_glue.py example script. I expected the following to be equivalent 1. Training with 1 GPU, batch size per device 16, gradient accumulation steps 8, for max\_steps 200 (green curve in image) 2. Training with 8 GPUs, batch size per device 16, gradient accumulation steps 1, for max\_steps 200 (orange curve in image) In both cases I am training with the same constant learning rate. However the single gpu setting (1) seems to perform significantly better than the multi-gpu setting (2). How can I set the multi-gpu training to be “equivalent” to the multi-gpu setting? Is there something I am missing about how these two settings behave? Any help is much appreciated! ![Screenshot 2023-11-09 at 6.53.30 PM](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/e/3eb46e01cafa01a91a72fc0395f1ed402cd50a93.png) ---
**Author:** ![buaa42wxy](https://avatars.discourse-cdn.com/v4/letter/b/a183cd/32.png) [@buaa42wxy](https://discuss.huggingface.co/u/buaa42wxy)\ **Post date:** [February 26, 2024, 3:44am UTC](https://discuss.huggingface.co/t/multi-gpu-training-does-not-optimize-as-expected/61738/2 "2024-02-26T03:44:38Z")
condition 1: 16\*8 per GPU condition 2: 16\*1 per GPU seems learning rate must be 8x