# RuntimeError: arguments are located on different GPUs **URL:** \ **Category:** 🤗Transformers\ **Created:** [October 23, 2020, 8:21am UTC](https://discuss.huggingface.co/t/runtimeerror-arguments-are-located-on-different-gpus/1704 "2020-10-23T08:21:09Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](https://discuss.huggingface.co/u/anon47807654)\ **Post date:** [October 23, 2020, 8:21am UTC](https://discuss.huggingface.co/t/runtimeerror-arguments-are-located-on-different-gpus/1704/1 "2020-10-23T08:21:09Z")
Hi all, I’m facing these issue in these days but I haven’t found a solution. I’m using the simpletransformers, a wrapper for your library, the main code of the model that I’m trying to train, that is T5, is [here](https://github.com/ThilinaRajapakse/simpletransformers/blob/master/simpletransformers/t5/t5_model.py). I’ve trained the model correctly and I would like to continue the training for more epochs, so I’ve loaded the model simply using: ```auto model = T5Model("/PATHtoCHECKPOINT") ``` Then I’ve started the training with: ```auto model.train_model(train_df, eval_data=eval_df) ``` The training starts and the first observation is that the loss spikes at a higher value w.r.t the value of the last checkpoint, as shown in the image: ![W&B Chart 21_10_2020, 14_56_37](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2552ce5c7aa6948f7b83aaf4955dcc722e569c90.png) After some steps the loss starts to decrease and then this error is thrown: ```auto Traceback (most recent call last): File "TINIA_doubleTraining.py", line 112, in model.train_model(train_df, eval_data=eval_df) #,args={'num_train_epochs': 5, 'learning_rate':2e-5}) #args=model_args, sacreBLEU=sacreBLEU) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/simpletransformers/t5/t5_model.py", line 165, in train_model global_step, training_details = self.train( File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/simpletransformers/t5/t5_model.py", line 418, in train results = self.eval_model( File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/simpletransformers/t5/t5_model.py", line 613, in eval_model result = self.evaluate(eval_dataset, output_dir, verbose=verbose, silent=silent, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/simpletransformers/t5/t5_model.py", line 673, in evaluate outputs = model(**inputs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply output.reraise() File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in replica 1 on device 5. Original Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker output = module(*input, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply output.reraise() File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in replica 0 on device 0. Original Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker output = module(*input, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/datadrive/data/T5/transformers/src/transformers/modeling_t5.py", line 1169, in forward encoder_outputs = self.encoder( File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/datadrive/data/T5/transformers/src/transformers/modeling_t5.py", line 711, in forward inputs_embeds = self.embed_tokens(input_ids) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward return F.embedding( File "/home/ubuntu/anaconda3/envs/nlp_venv/lib/python3.8/site-packages/torch/nn/functional.py", line 1814, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629395347/work/aten/src/THC/generic/THCTensorIndex.cu:403 ``` I’m using AWS EC2 with 8 GPUs, so I’ve tried the same code with a machine with 1 GPU only and the code starts, has this peaks at the beginning of each epoch but I don’t get any error, therefore the problem is related to the DataParallel. Any suggestion? Thanks! ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 24, 2020, 8:26am UTC](https://discuss.huggingface.co/t/runtimeerror-arguments-are-located-on-different-gpus/1704/2 "2020-10-24T08:26:52Z")
Would be better if you create an issue in `simpletransformers` repo. ---
**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](https://discuss.huggingface.co/u/anon47807654)\ **Post date:** [October 24, 2020, 8:48am UTC](https://discuss.huggingface.co/t/runtimeerror-arguments-are-located-on-different-gpus/1704/3 "2020-10-24T08:48:42Z")
@valhalla thanks for your reply! I have done it before writing on the forum, the author has answered that he has just used the save/load pretrained from your library so the problem might be related to this methods.