# KeyError: 'input\_ids'. when training BERT with Trainer **URL:** \ **Category:** Models\ **Created:** [November 19, 2020, 12:50pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124 "2020-11-19T12:50:42Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mickeymnemonic](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mickeymnemonic](https://discuss.huggingface.co/u/mickeymnemonic)\ **Post date:** [November 19, 2020, 12:50pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/1 "2020-11-19T12:50:42Z")
greetings fam just curious if anyone provide insight on the key error message (KeyError: ‘input\_ids’.) i go to train my pretrained BertForMaskedLM model (using code: trainer\_BERT.train()) via the huggingface Trainer on my Dataset object. not sure if it has to do with my creation of the dataset or how i am calling my model for training tho any insights are appreciated!! a detailed view of my code and the key error is available at the link below. thank you mick > KeyError Traceback (most recent call last) in ----\> 1 trainer\_BERT.train() 2 trainer.save\_model(“./models/royalBERT”) ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in train(self, model\_path, trial) 755 self.control = self.callback\_handler.on\_epoch\_begin(self.args, self.state, self.control) 756 → 757 for step, inputs in enumerate(epoch\_iterator): 758 759 # Skip past any already trained steps if resuming training ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in **next** (self) 361 362 def **next** (self): → 363 data = self.\_next\_data() 364 self.\_num\_yielded += 1 365 if self.\_dataset\_kind == \_DatasetKind.Iterable and \ ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in \_next\_data(self) 401 def \_next\_data(self): 402 index = self.\_next\_index() # may raise StopIteration → 403 data = self.\_dataset\_fetcher.fetch(index) # may raise StopIteration 404 if self.\_pin\_memory: 405 data = \_utils.pin\_memory.pin\_memory(data) ~/anaconda3/lib/python3.7/site-packages/torch/utils/data/\_utils/fetch.py in fetch(self, possibly\_batched\_index) 45 else: 46 data = self.dataset[possibly\_batched\_index] —\> 47 return self.collate\_fn(data) ~/anaconda3/lib/python3.7/site-packages/transformers/data/data\_collator.py in **call** (self, examples) 193 ) → Dict[str, torch.Tensor]: 194 if isinstance(examples[0], (dict, BatchEncoding)): → 195 examples = [e[“input\_ids”] for e in examples] 196 batch = self.\_tensorize\_batch(examples) 197 if self.mlm: ~/anaconda3/lib/python3.7/site-packages/transformers/data/data\_collator.py in (.0) 193 ) → Dict[str, torch.Tensor]: 194 if isinstance(examples[0], (dict, BatchEncoding)): → 195 examples = [e[“input\_ids”] for e in examples] 196 batch = self.\_tensorize\_batch(examples) 197 if self.mlm: KeyError: ‘input\_ids’ ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 19, 2020, 3:42pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/2 "2020-11-19T15:42:54Z")
The line ```auto tokenizerBERT(unlabelled_dataset['paragraphs'], padding=True, truncation=True) ``` is not stored anywhere, so you’re passing to the Trainer a dataset that hasn’t been tokenized. ---
**Author:** ![mickeymnemonic](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mickeymnemonic](https://discuss.huggingface.co/u/mickeymnemonic)\ **Post date:** [November 19, 2020, 3:56pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/3 "2020-11-19T15:56:15Z")
ok, thank you. so simply store as a variable such as below? ```auto tokenizerBERT = tokenizerBERT(unlabelled_dataset['paragraphs'], padding=True, truncation=True) ``` the reason i didn’t store it was because when i did it i could no longer save the output w/ the save\_pretrained method such as shown below. tokenizerBERT.save\_pretrained(‘tokenizers/pytorch/labelled/BERT/’) any thoughts? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 19, 2020, 3:59pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/4 "2020-11-19T15:59:07Z")
If you store your datasets in the tokenizer it won’t work either. You need to store it in the variable you will send to `Trainer` a your `train_dataset`. Also I don’t know what `Dataset` class you are using, but tokenizer probably can’t take it directly. ---
**Author:** ![mickeymnemonic](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mickeymnemonic](https://discuss.huggingface.co/u/mickeymnemonic)\ **Post date:** [November 19, 2020, 7:01pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/5 "2020-11-19T19:01:20Z")
i’m definitely missing something…curious to hear your views on what that is. based on the HF documentation i thought it would be possible to simply pass an in memory dataframe: unlabelled\_dataset = Dataset.from\_pandas(unlabelled\_corpus) #create Dataset object for tokenization take the relevant paragraphs column from the dataframe that i want to finetune the pretrained model on. this column has a few sentences or a paragraph on each row of the dataframe. and then pass it to the pretrained tokenizer after intialization: tokenizerBERT = BertTokenizerFast.from\_pretrained(‘bert-base-uncased’) #BERT model tokenization & check encodingBERT = tokenizerBERT(unlabelled\_dataset[‘paragraphs’], padding=True, truncation=True) initializing the data collator w/ reference to the pretrained tokenizer: data\_collator\_BERT = DataCollatorForLanguageModeling(tokenizer=tokenizerBERT, mlm=True, mlm\_probability=0.15) and then initializing the model, training arguements and trainer before calling it would do the trick: model\_BERT = BertForMaskedLM.from\_pretrained(‘bert-base-uncased’) training\_args\_BERT = TrainingArguments( output\_dir=’./BERT’, num\_train\_epochs=10, evaluation\_strategy=‘steps’, warmup\_steps=10000, weight\_decay=0.01, per\_device\_train\_batch\_size=64, ) trainer\_BERT = Trainer( model=model\_BERT, args=training\_args\_BERT, data\_collator=data\_collator\_BERT, train\_dataset=encodingBERT, ) trainer\_BERT.train() trainer.save\_model(’./models/royalBERT’) the features of the loaded dataset.from\_pandas are provided below. i only am looking to train the model on the paragraphs column hence why i pass that column in the called pretrained tokenizer. {‘index’: Value(dtype=‘string’, id=None), ‘filename’: Value(dtype=‘string’, id=None), ‘page-id’: Value(dtype=‘string’, id=None), ‘paragraph-id’: Value(dtype=‘string’, id=None), ‘paragraphs’: Value(dtype=‘string’, id=None), ‘ **index\_level\_0** ’: Value(dtype=‘int64’, id=None)} ---
**Author:** ![lovelmark](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@lovelmark](https://discuss.huggingface.co/u/lovelmark)\ **Post date:** [July 21, 2021, 12:42pm UTC](https://discuss.huggingface.co/t/keyerror-input-ids-when-training-bert-with-trainer/2124/6 "2021-07-21T12:42:10Z")
A KeyError means the key you gave pandas isn’t valid. Before doing anything with the data frame, use print(df.columns) to see [column exist](http://net-informations.com/ds/pd/exists.htm) or not. `print(df.columns)` I was getting a similar kind of error in one of my codes. Turns out, that particular index was missing from my data frame as I had dropped the empty dataframe 2 rows. If this is the case, you can do df.reset\_index(inplace=True) and the error should be resolved.