# MT5 Fine Tuning - KeyError: 'source\_ids'

**URL:** <https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257>\
**Category:** 🤗Transformers\
**Created:** [April 1, 2021, 12:57pm UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257 "2021-04-01T12:57:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ozcangundes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ozcangundes/32/2346_2.png) [@ozcangundes](https://discuss.huggingface.co/u/ozcangundes)\
**Post date:** [April 1, 2021, 12:57pm UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/1 "2021-04-01T12:57:32Z")

</div>

Hi,

I am trying to fine tune MT5 for multitask question answer and question generation similar to @valhalla model. I prepared the dataset by using datasets library as follows:

```python
    train_dataset = Dataset.from_pandas(pd.DataFrame(generate_data(mode="train")))
    valid_dataset = Dataset.from_pandas(pd.DataFrame(generate_data(mode="valid")))

    processor = DataProcessor(
        tokenizer,
        max_source_length=data_args.max_source_length,
        max_target_length=data_args.max_target_length
    )

    
    train_dataset = processor.process(train_dataset)
    valid_dataset = processor.process(valid_dataset)

    columns = ["source_ids", "target_ids", "attention_mask"]

```

However, when I try to train my model as below:

```python
!python3 run_multi.py \
    --model_name_or_path google/mt5-small \
    --model_type mt5 \
    --tokenizer_name_or_path mt5_qg_tokenizer \
    --output_dir mt5-small-multi \
    --train_file_path data/train_data_qa_qg_mt5.pt \
    --valid_file_path data/valid_data_qa_qg_mt5.pt \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --gradient_accumulation_steps 2 \
    --learning_rate 1e-4 \
    --num_train_epochs 2 \
    --seed 42 \
    --do_train \
    --do_eval \
    --logging_steps 100 \
    --prediction_loss_only True

```

it says `KeyError: 'source_ids'`

I am sure the dataset has “source\_ids” field.

```auto
train_dataset=torch.load(r"train_data_qa_qg_mt5.pt")
train_dataset
> Dataset({
    features: ['attention_mask', 'source_ids', 'source_text', 'target_ids', 'target_text', 'task'],
    num_rows: 3449
})

```

What might cause this?

The versions of the libraries are:  
`transformers == 4.4.2`  
`datasets == 1.5.0`

Thank you for the reply in advance.

---

<div class="post-metadata">

**Author:** ![ozcangundes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ozcangundes/32/2346_2.png) [@ozcangundes](https://discuss.huggingface.co/u/ozcangundes)\
**Post date:** [April 1, 2021, 1:28pm UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/2 "2021-04-01T13:28:14Z")

</div>

I found out that my data collator takes only “attention mask” as inputs. I do not know where the other fields disappear 😔

```auto
class T2TDataCollator():
    def __init__ (self, tokenizer, mode='training'):
        self.tokenizer = tokenizer

        self.mode = mode

    def __call__ (self, batch: List) -> Dict[str, torch.Tensor]:
        """
        Take a list of samples from a Dataset and collate them into a batch.
        Returns:
            A dictionary of tensors
        """

        input_ids = torch.stack([example['source_ids'] for example in batch])
        target_ids = torch.stack([example['target_ids'] for example in batch])
        attention_mask = torch.stack([example['attention_mask'] for example in batch])
        .....

```

The error is thrown in this part.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 1, 2021, 3:54pm UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/3 "2021-04-01T15:54:08Z")

</div>

Is your script `run_multi` using `Trainer`? By default the `Trainer` removes any column that is not in your model signature (like “source\_ids”), so you should pass `--remove_unused_columns False` in your command.

---

<div class="post-metadata">

**Author:** ![ozcangundes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ozcangundes/32/2346_2.png) [@ozcangundes](https://discuss.huggingface.co/u/ozcangundes)\
**Post date:** [April 2, 2021, 7:14am UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/4 "2021-04-02T07:14:44Z")

</div>

yes I was using `Trainer`. But I solved the problem. I was loading dataset via `Datasets` library, when I replaced it with `nlp.load_dataset`, it worked seamlessly. But thank you for the response, I did not `pass --remove_unused_columns False`

---

<div class="post-metadata">

**Author:** ![Sahajtomar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sahajtomar/32/906_2.png) [@Sahajtomar](https://discuss.huggingface.co/u/Sahajtomar)\
**Post date:** [May 7, 2021, 10:51am UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/5 "2021-05-07T10:51:17Z")

</div>

Hi, I have custom data and I am not able to load dataset with nlp.load\_dataset. With dataset library I am getting same error. Could you tell me how you resoolved it

---

<div class="post-metadata">

**Author:** ![ozcangundes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ozcangundes/32/2346_2.png) [@ozcangundes](https://discuss.huggingface.co/u/ozcangundes)\
**Post date:** [May 7, 2021, 11:12am UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/6 "2021-05-07T11:12:33Z")

</div>

Hi. I got this error when using `Datasets` library. Just changed it as nlp, then code worked. So,

instead of

```auto
from datasets import Dataset

```

I used

```auto
from nlp import Dataset

```

You can find the related script in [here](https://github.com/ozcangundes/multitask-question-generation/blob/65c80092c1fa21659e37105ba858eef2d2f628d5/prepare_data.py#L117)

---

<div class="post-metadata">

**Author:** ![Sahajtomar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sahajtomar/32/906_2.png) [@Sahajtomar](https://discuss.huggingface.co/u/Sahajtomar)\
**Post date:** [May 7, 2021, 11:59am UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/7 "2021-05-07T11:59:36Z")

</div>

Hi, Thank you for sharing the code. I tried that, error is still there. Also I am using transformers V4.5.1. Any leads?

---

<div class="post-metadata">

**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](https://discuss.huggingface.co/u/sbmaruf)\
**Post date:** [July 17, 2021, 9:05am UTC](https://discuss.huggingface.co/t/mt5-fine-tuning-keyerror-source-ids/5257/8 "2021-07-17T09:05:41Z")

</div>

> [@sgugger](#):
>
> remove\_unused\_columns

I faced the same issue. Follow @sgugger 's suggestion.