# Training fails on multiple gpu throwing cuda runtime errors

**URL:** <https://discuss.huggingface.co/t/training-fails-on-multiple-gpu-throwing-cuda-runtime-errors/23836>\
**Category:** 🤗Transformers\
**Created:** [September 30, 2022, 2:46pm UTC](https://discuss.huggingface.co/t/training-fails-on-multiple-gpu-throwing-cuda-runtime-errors/23836 "2022-09-30T14:46:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![adibm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adibm/32/5390_2.png) [@adibm](https://discuss.huggingface.co/u/adibm)\
**Post date:** [September 30, 2022, 2:46pm UTC](https://discuss.huggingface.co/t/training-fails-on-multiple-gpu-throwing-cuda-runtime-errors/23836/1 "2022-09-30T14:46:36Z")

</div>

I am fine-tuning a GPT2LMHeadModel. When I run the code on a single GPU, it works, but when I run it on multiple GPUS, I get the following error. I have used multiple GPUS in my code base before, but I get this error after I made some changes to the data format.

File “/project/src/trainer.py”, line 71, in train  
pre\_trainer.train()  
File “/project/envs/lib/python3.9/site-packages/transformers/trainer.py”, line 1409, in train  
return inner\_training\_loop(  
File “/project/envs/lib/python3.9/site-packages/transformers/trainer.py”, line 1651, in \_inner\_training\_loop  
tr\_loss\_step = self.training\_step(model, inputs)  
File “/project/envs/lib/python3.9/site-packages/transformers/trainer.py”, line 2363, in training\_step

tr\_loss\_step = self.training\_step(model, inputs)  
File “/project/envs/lib/python3.9/site-packages/transformers/trainer.py”, line 2363, in training\_step  
loss.backward()  
File “project/envs/lib/python3.9/site-packages/torch/\_tensor.py”, line 396, in backward  
torch.autograd.backward(self, gradient, retain\_graph, create\_graph, inputs=inputs)  
File “/project/envs/lib/python3.9/site-packages/torch/autograd/ **init**.py”, line 173, in backward  
Variable.\_execution\_engine.run\_backward( # Calls into the C++ engine to run the backward pass  
RuntimeError: unique\_by\_key: failed to synchronize: cudaErrorECCUncorrectable: uncorrectable ECC error encountered  
│  
Set the environment variable HYDRA\_FULL\_ERROR=1 for a complete stack trace.