# HuggingFace summarization training example notebook raises two warnings when run on multi-GPUs

**URL:** <https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788>\
**Category:** 🤗Transformers\
**Created:** [June 29, 2022, 2:11pm UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788 "2022-06-29T14:11:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](https://discuss.huggingface.co/u/AndreaSottana)\
**Post date:** [June 29, 2022, 2:11pm UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/1 "2022-06-29T14:11:23Z")

</div>

Hello,  
I am running an example summarization training task taken from [here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/chapter7/section5_pt.ipynb) (official HuggingFace example) on a multi-GPU machine, using the following versions: `torch==1.11.0+cu113` and `transformers==4.20.1`. The only difference is that instead of using `google/mt5-small` as model I am using `facebook/bart-base`

I am getting two warnings. I believe they are raised when the model is trying to gather results from multiple but I’m struggling to understand whether they are normal or an indication that something isn’t working with the code. The first warning is

> Parameter ‘function’=\<function preprocess\_function at 0x7f797c2de0e0\> of the transform datasets.arrow\_dataset.Dataset.\_map\_single couldn’t be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won’t be showed.

The other warning is

> venv/lib/python3.10/site-packages/torch/nn/parallel/\_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.

Could someone tell me if these warnings are likely the result of something not working properly with the code and failing silently (code runs without raising any exceptions)?  
For reference I copy below a small reproducible example of the code generating the warnings., adapted from the official HuggingFace notebook linked above.

```auto
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, DataCollatorForSeq2Seq, Seq2SeqTrainer
from datasets import load_dataset

model_checkpoint = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

english_dataset = load_dataset("amazon_reviews_multi", "en")

def preprocess_function(examples):
    model_inputs = tokenizer(examples["review_body"], max_length=512, truncation=True)
    # Set up the tokenizer for targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["review_title"], max_length=30, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_datasets = english_dataset.map(preprocess_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(
    english_dataset["train"].column_names
)
features = [tokenized_datasets["train"][i] for i in range(2)]
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)
data_collator(features)

batch_size = 8
num_train_epochs = 8
# Show the training loss with every epoch
logging_steps = len(tokenized_datasets["train"]) // batch_size
model_name = model_checkpoint.split("/")[-1]

args = Seq2SeqTrainingArguments(
    output_dir=f"{model_name}-finetuned-amazon-en-es",
    evaluation_strategy="epoch",
    learning_rate=5.6e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=num_train_epochs,
    predict_with_generate=True,
    logging_steps=logging_steps,
    push_to_hub=False,
)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer
)

trainer.train()
trainer.evaluate()

```

---

<div class="post-metadata">

**Author:** ![Colorful](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Colorful](https://discuss.huggingface.co/u/Colorful)\
**Post date:** [July 16, 2022, 9:56am UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/2 "2022-07-16T09:56:33Z")

</div>

I have the same issue, do you resolve it?

---

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](https://discuss.huggingface.co/u/AndreaSottana)\
**Post date:** [July 16, 2022, 10:08am UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/3 "2022-07-16T10:08:10Z")

</div>

Not yet unfortunately ☹  
Code seems to run fine but the warnings are still there and I’m not sure what should be done about them

---

<div class="post-metadata">

**Author:** ![Colorful](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Colorful](https://discuss.huggingface.co/u/Colorful)\
**Post date:** [July 17, 2022, 6:59am UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/4 "2022-07-17T06:59:32Z")

</div>

Hello, I find that this problem may be caused by trainer. When I used script run\_glue.py, I got the UserWarning. But when I used script run\_glue\_no\_trainer.py, everything is ok.

---

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](https://discuss.huggingface.co/u/AndreaSottana)\
**Post date:** [July 18, 2022, 11:24am UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/5 "2022-07-18T11:24:57Z")

</div>

Thanks for pointing this out,. Unfortunately I am not using their training script but a custom one. What I would like to understand is why there is this issue and what underlining code part is causing it. Knowing that run\_glue.py raises the warning but run\_glue\_no\_trainer.py does not, may be helpful if you plan to use those, but does not really address my query as to why this is happening. For this reason I will keep the question open

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 17, 2022, 3:38pm UTC](https://discuss.huggingface.co/t/huggingface-summarization-training-example-notebook-raises-two-warnings-when-run-on-multi-gpus/19788/6 "2022-08-17T15:38:22Z")

</div>

did this help you? [Using Transformers with DistributedDataParallel — any examples?](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775)