# Key error: 0 in DataCollatorForSeq2Seq for BERT

**URL:** <https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260>\
**Category:** Beginners\
**Created:** [June 24, 2021, 1:02pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260 "2021-06-24T13:02:03Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arcticweasel](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@Arcticweasel](https://discuss.huggingface.co/u/Arcticweasel)\
**Post date:** [June 24, 2021, 1:02pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/1 "2021-06-24T13:02:03Z")

</div>

Hello everyone,

I am trying to fine-tune a german BERT2BERT model for text summarization unsing `bert-base-german-cased` and want to use dynamic padding. However, when calling `Trainer.train()` I receive an error, that tensors cannot be created and I should use padding. I was able to trace this error back to my DataCollator. The code I used is the following:

First, I define the function to tokenize my data and do so using the map function.

> tokenizer = BertTokenizerFast.from\_pretrained(“bert-base-german-cased”)  
> tokenizer.bos\_token = tokenizer.cls\_token  
> tokenizer.eos\_token = tokenizer.sep\_token
> 
> max\_input\_length = 512  
> max\_target\_length = 128
> 
> def prepro\_bert2bert(samples):  
> model\_inputs = tokenizer(samples[“text”], max\_length = max\_input\_length, truncation = True)
> 
> ```
> with tokenizer.as_target_tokenizer():
> labels = tokenizer(samples["description"], max_length = max_target_length, truncation = True)
>       
> samples["input_ids"] = model_inputs.input_ids
> samples["attention_mask"] = model_inputs.attention_mask
> samples["decoder_input_ids"] = labels.input_ids
> samples["decoder_attention_mask"] = labels.attention_mask
> samples["labels"] = labels.input_ids.copy()
> 
> return samples
> 
> ```
> 
> traindata = Dataset.from\_pandas(traindata)
> 
> tokenized\_traindata = traindata.map(prepro\_bert2bert, batched = True, remove\_columns = [“text”, “description”, “\__index\_level\_0_\_”])  
> tokenized\_traindata.set\_format(columns = [“labels”, “input\_ids”, “attention\_mask”, “decoder\_input\_ids”, “decoder\_attention\_mask”])

My tokenized\_traindata looks like the following:

> Dataset({  
> features: [‘attention\_mask’, ‘decoder\_attention\_mask’, ‘decoder\_input\_ids’, ‘input\_ids’, ‘labels’],  
> num\_rows: 7986  
> })

Then I instantiate my bert2bert model and my DataCollator:

> bert2bert = EncoderDecoderModel.from\_encoder\_decoder\_pretrained(“bert-base-german-cased”, “bert-base-german-cased”)  
> data\_collator = DataCollatorForSeq2Seq(tokenizer, model = bert2bert, padding = “longest”)

Lastly, I form batches from my training data and want to use the data\_collator

> samples = tokenized\_traindata[:8]  
> batch = data\_collator(samples)

This returns the following error message

> * * *
> 
> KeyError Traceback (most recent call last)  
> in   
> ----\> 1 batch = data\_collator(samples)  
> 2 {k: v.shape for k, v in batch.items()}
> 
> > ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\data\data\_collator.py in **call** (self, features)  
> > 271  
> > 272 def **call** (self, features):  
> > → 273 labels = [feature[“labels”] for feature in features] if “labels” in features[0].keys() else None  
> > 274 # We have to pad the labels before calling `tokenizer.pad` as this method won’t pad them and needs them of the  
> > 275 # same length to return tensors.
> > 
> > KeyError: 0

Unfortunately, I do not know where to look further for a solution. I hope someone may has a suggestion where to look or how to solve this. Thank you very much in advance!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 24, 2021, 3:00pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/2 "2021-06-24T15:00:35Z")

</div>

This is because the `datasets` library returns a slice of the dataset as a dictionary with lists for each key. The data collator however expects a list of dataset elements, so a list of dictionaries. Practically, I think you need to do:

```auto
samples = [tokenized_traindata[i] for i in range(8)]
batch = data_collator(samples)

```

---

<div class="post-metadata">

**Author:** ![Arcticweasel](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@Arcticweasel](https://discuss.huggingface.co/u/Arcticweasel)\
**Post date:** [June 24, 2021, 3:19pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/3 "2021-06-24T15:19:38Z")

</div>

Thank you for your reply. I tried your suggestion and now get the first error message again:

> samples = [tokenized\_traindata[i] for i in range(8)]  
> batch = data\_collator(samples)

> * * *
> 
> ValueError Traceback (most recent call last)  
> ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization\_utils\_base.py in convert\_to\_tensors(self, tensor\_type, prepend\_batch\_axis)  
> 698 if not is\_tensor(value):  
> → 699 tensor = as\_tensor(value)  
> 700
> 
> ValueError: expected sequence of length 40 at dim 1 (got 47)
> 
> During handling of the above exception, another exception occurred:
> 
> ValueError Traceback (most recent call last)  
> in   
> ----\> 1 batch = data\_collator(samples)
> 
> ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\data\data\_collator.py in **call** (self, features)  
> 283 )  
> 284  
> → 285 features = self.tokenizer.pad(  
> 286 features,  
> 287 padding=self.padding,
> 
> ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization\_utils\_base.py in pad(self, encoded\_inputs, padding, max\_length, pad\_to\_multiple\_of, return\_attention\_mask, return\_tensors, verbose)  
> 2678 batch\_outputs[key].append(value)  
> 2679  
> → 2680 return BatchEncoding(batch\_outputs, tensor\_type=return\_tensors)  
> 2681  
> 2682 def create\_token\_type\_ids\_from\_sequences(
> 
> ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization\_utils\_base.py in **init** (self, data, encoding, tensor\_type, prepend\_batch\_axis, n\_sequences)  
> 202 self.\_n\_sequences = n\_sequences  
> 203  
> → 204 self.convert\_to\_tensors(tensor\_type=tensor\_type, prepend\_batch\_axis=prepend\_batch\_axis)  
> 205  
> 206 @property
> 
> ~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization\_utils\_base.py in convert\_to\_tensors(self, tensor\_type, prepend\_batch\_axis)  
> 713 “Please see if a fast version of this tokenizer is available to have this feature available.”  
> 714 )  
> → 715 raise ValueError(  
> 716 "Unable to create tensor, you should probably activate truncation and/or padding "  
> 717 “with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.”
> 
> ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.

Do you have any suggestions as to why this happens again? Thank you very much in advance for your response and time!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 24, 2021, 3:24pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/4 "2021-06-24T15:24:08Z")

</div>

I think it’s compaining because the `decoder_attention_mask` and `decoder_input_ids` aren’t all of the same length. You should let the `DataCollatorForSeq2Seq` create them for you.

---

<div class="post-metadata">

**Author:** ![Arcticweasel](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@Arcticweasel](https://discuss.huggingface.co/u/Arcticweasel)\
**Post date:** [June 24, 2021, 3:29pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/5 "2021-06-24T15:29:10Z")

</div>

Thank you again! So I should not tokenize the labels and original text at the same time in my `prepro_bert2bert` function? How would I approach that?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 24, 2021, 3:30pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/6 "2021-06-24T15:30:56Z")

</div>

You definitely should tokenize your labels, but remove the lines

```auto
samples["decoder_input_ids"] = labels.input_ids
samples["decoder_attention_mask"] = labels.attention_mask

```

which the data collator will ser properly from the labels and the models (there is a shift right to apply in some models).

---

<div class="post-metadata">

**Author:** ![Arcticweasel](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@Arcticweasel](https://discuss.huggingface.co/u/Arcticweasel)\
**Post date:** [June 24, 2021, 3:42pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/7 "2021-06-24T15:42:12Z")

</div>

Thank you very much! This solved my problem. 🙂

---

<div class="post-metadata">

**Author:** ![Arcticweasel](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@Arcticweasel](https://discuss.huggingface.co/u/Arcticweasel)\
**Post date:** [June 24, 2021, 4:01pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/8 "2021-06-24T16:01:21Z")

</div>

Sorry to bother you again. ☹ I now wanted to implement what I learned in my training loop, but now get a value error that I must specify either input\_ids or inputs\_embeds. Do I need to specify anything else, so that the decoder\_input\_ids and decoder\_attention\_mask from the collator get passed into the model? My training arguments look like this:

> batch\_size = 2  
> learning\_rate = 2e-5  
> epochs = 1
> 
> training\_args = Seq2SeqTrainingArguments(output\_dir = “model\_checkpoints/”,  
> evaluation\_strategy = “epoch”,  
> learning\_rate = learning\_rate,  
> per\_device\_train\_batch\_size = batch\_size,  
> per\_device\_eval\_batch\_size = batch\_size,  
> save\_total\_limit = 3,  
> num\_train\_epochs = epochs,  
> predict\_with\_generate = True,  
> fp16 = True)

> trainer = Seq2SeqTrainer(bert2bert,  
> training\_args,  
> train\_dataset = tokenized\_traindata,  
> eval\_dataset = tokenized\_valdata,  
> data\_collator = data\_collator,  
> tokenizer = tokenizer,  
> compute\_metrics = compute\_metrics)

Thank you again for your time and patience!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 24, 2021, 4:05pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/9 "2021-06-24T16:05:41Z")

</div>

Pinging @patrickvonplaten since I’m not super familiar with this BERT2BERT archittecture.

---

<div class="post-metadata">

**Author:** ![TopRightExit](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/toprightexit/32/2802_2.png) [@TopRightExit](https://discuss.huggingface.co/u/TopRightExit)\
**Post date:** [September 28, 2021, 11:21pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/10 "2021-09-28T23:21:13Z")

</div>

Hi @Arcticweasel ,

I ran into the same problem with this, requiring to specify input\_ids or inputs\_embeds. I thought that input\_ids is already in tokenized\_traindata after preprocessing. Have you found a solution?

Cheers!

---

<div class="post-metadata">

**Author:** ![waxef](https://avatars.discourse-cdn.com/v4/letter/w/b77776/32.png) [@waxef](https://discuss.huggingface.co/u/waxef)\
**Post date:** [March 13, 2024, 8:42pm UTC](https://discuss.huggingface.co/t/key-error-0-in-datacollatorforseq2seq-for-bert/7260/11 "2024-03-13T20:42:18Z")

</div>

I was having a similar issue and landed on this thread.

I’m trying to understand why the `DataCollatorForSeq2Seq` is not padding the batch inputs similar to `DataCollatorWithPadding`?

Do I need to specify `padding=True` in the tokenizer? (only then seems to be working)  
When adding `padding=True` as arguement to the `DataCollatorForSeq2Seq` it doesn’t work?

MWE:

```auto
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, DataCollatorForSeq2Seq, Seq2SeqTrainer
from datasets import load_dataset

model_checkpoint = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

english_dataset = load_dataset("McAuley-Lab/Amazon-Reviews-2023", "raw_review_All_Beauty")

def preprocess_function(examples):
    model_inputs = tokenizer(examples["text"],)
    # Set up the tokenizer for targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["title"])

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_datasets = english_dataset.map(preprocess_function, batched=True)

tokenized_datasets = tokenized_datasets.remove_columns(['rating', 'title', 'text', 'images', 'asin', 'parent_asin', 'user_id', 'timestamp', 'helpful_vote', 'verified_purchase'])

tokenized_datasets['full'][:2].keys()
# dict_keys(['input_ids', 'attention_mask', 'labels'])

features = [tokenized_datasets["full"][i] for i in range(2)]

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

featuers_padded = data_collator(features)

len(features_padded[0]['input_ids'])
# 37

len(features_padded[1]['input_ids'])
# 69

len(features_padded[0]['labels'])
# 53

len(features_padded[1]['labels'])
# 82

```

Any idea what’s going on here?