# Inferences with DataParallel

**URL:** <https://discuss.huggingface.co/t/inferences-with-dataparallel/14773>\
**Category:** Beginners\
**Created:** [February 15, 2022, 11:46pm UTC](https://discuss.huggingface.co/t/inferences-with-dataparallel/14773 "2022-02-15T23:46:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![crsegerie](https://avatars.discourse-cdn.com/v4/letter/c/41988e/32.png) [@crsegerie](https://discuss.huggingface.co/u/crsegerie)\
**Post date:** [February 15, 2022, 11:46pm UTC](https://discuss.huggingface.co/t/inferences-with-dataparallel/14773/1 "2022-02-15T23:46:34Z")

</div>

tldr: This is an attempt at using DataParallel class with Huggingface, But I still can’t figure it out. Could you give me some examples ?

Hello, I would like to use my two GPU to make inferences with DataParallel. So I adapted a script which works well on one gpu, but I’m stuck with an error:

```auto
from torch.nn.parallel import DataParallel
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM # type: ignore

device = "cuda:0"
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distill-pegasus-cnn-16-4")
model = AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distill-pegasus-cnn-16-4")

model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

batch = tokenizer(
      df_chunk["abstract"].to_list(),
      truncation=True,
      padding="longest",
      max_length=max_length_abstract,
      return_tensors="pt",
  ).to(device)
  translated = model(**batch) # <--- error here
  translated = model(decoder_input_ids=batch["input_ids"], **batch)
  print("len token batch", batch["input_ids"].shape)
  tldrs = tokenizer.batch_decode(translated, skip_special_tokens=True)

```

‘DataParallel’ object has no attribute ‘generate’.

So I replaced the faulty line by the following line using the **call** method of PyTorch models :

`translated = model(**batch)` but now I get the following error:

> **error**
>
> ValueError: Caught ValueError in replica 0 on device 0.  
> Original Traceback (most recent call last):  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/torch/nn/parallel/parallel\_apply.py”, line 61, in \_worker  
> output = module(\*input, \*\*kwargs)  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in \_call\_impl  
> return forward\_call(\*input, \*\*kwargs)  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/transformers/models/pegasus/modeling\_pegasus.py”, line 1390, in forward  
> outputs = self.model(  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in \_call\_impl  
> return forward\_call(\*input, \*\*kwargs)  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/transformers/models/pegasus/modeling\_pegasus.py”, line 1250, in forward  
> decoder\_outputs = self.decoder(  
> File “/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in \_call\_impl  
> return forward\_call(\*input, \*\*kwargs)  
> File "/scratch/scampo01/condaenvs/torchA100\_v2/lib/python3.9/site-

packages/transformers/models/pegasus/modeling\_pegasus.py", line 1014, in forward  
raise ValueError(“You have to specify either decoder\_input\_ids or decoder\_inputs\_embeds”)  
ValueError: You have to specify either decoder\_input\_ids or decoder\_inputs\_embeds

So finally I replaced the faulty line with

`translated = model(decoder_input_ids=batch["input_ids"], **batch)`

And this seems to work, until the tokenizer cannot decode the outputs and I have to modify the end of the script like that to obtain something:

```auto
    tldrs = tokenizer.batch_decode(translated, skip_special_tokens=True)
    tldrs = tokenizer.batch_decode(
        torch.argmax(translated.logits, dim=2), skip_special_tokens=True
    )

```

But my results and this code is so messy that it is obvious that I missed something. I didn’t find many (any?) examples on how to use DataParallel with Huggingface models for inferences. Could you give me some pointers and tell me what is wrong in my script?

Thank you

---

<div class="post-metadata">

**Author:** ![PahaII](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pahaii/32/4819_2.png) [@PahaII](https://discuss.huggingface.co/u/PahaII)\
**Post date:** [May 19, 2022, 1:07am UTC](https://discuss.huggingface.co/t/inferences-with-dataparallel/14773/2 "2022-05-19T01:07:27Z")

</div>

I happen to come across the same question. Did you find a more elegant way to solve it? It seems that if you replace `model.generate(batch["input_ids"])` with `model(decoder_input_ids=batch["input_ids"],**batch)` and `tldrs = tokenizer.batch_decode(torch.argmax(translated.logits, dim=2))`, then you are performing argmax decoding. You cannot directly use packaged decoding strategy such as beam search in `model.generate()`, which is not convenient…

---

<div class="post-metadata">

**Author:** ![liorcam](https://avatars.discourse-cdn.com/v4/letter/l/3ec8ea/32.png) [@liorcam](https://discuss.huggingface.co/u/liorcam)\
**Post date:** [May 31, 2022, 2:55pm UTC](https://discuss.huggingface.co/t/inferences-with-dataparallel/14773/3 "2022-05-31T14:55:22Z")

</div>

since the the model variable is a DataParrallel instance, it attributes are accessible with model.module  
have you tried  
translated = model.module(\*\*batch)  
?

---

<div class="post-metadata">

**Author:** ![nomadrp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nomadrp/32/18318_2.png) [@nomadrp](https://discuss.huggingface.co/u/nomadrp)\
**Post date:** [March 15, 2024, 3:23pm UTC](https://discuss.huggingface.co/t/inferences-with-dataparallel/14773/4 "2024-03-15T15:23:22Z")

</div>

`accelerator.unwrap_model(model)` worked instead of `model.module`.