# Error in pad() function of transformers/tokenization\_utils\_base.py

**URL:** <https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254>\
**Category:** 🤗Transformers\
**Created:** [September 23, 2024, 9:45pm UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254 "2024-09-23T21:45:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![toddglidden](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@toddglidden](https://discuss.huggingface.co/u/toddglidden)\
**Post date:** [September 23, 2024, 9:45pm UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254/1 "2024-09-23T21:45:16Z")

</div>

Hello!

I’ve been going through the fine-tuning part of this tutorial: [Getting Started with Sentiment Analysis using Python (huggingface.co)](https://huggingface.co/blog/sentiment-analysis-python)

I was able to make it work using the IMDB dataset that the tutorial covered. I tried to do the same thing with the airline tweets (osanseviero/twitter-airline-sentiment) dataset and I’m getting an error when running the trainer.train() function:

* * *

ValueError Traceback (most recent call last)  
 in \<cell line: 2\>()  
1 # Train the model  
----\> 2 trainer.train()

8 frames  
/usr/local/lib/python3.10/dist-packages/transformers/tokenization\_utils\_base.py in pad(self, encoded\_inputs, padding, max\_length, pad\_to\_multiple\_of, return\_attention\_mask, return\_tensors, verbose)  
3477 # The model’s main input name, usually `input_ids`, has be passed for padding  
3478 if self.model\_input\_names[0] not in encoded\_inputs:  
 → 3479 raise ValueError(  
3480 “You should supply an encoding or a list of encodings to this method "  
3481 f"that includes {self.model\_input\_names[0]}, but you provided {list(encoded\_inputs.keys())}”

ValueError: You should supply an encoding or a list of encodings to this method that includes input\_ids, but you provided [‘label’]

Code can be seen here: [toddpglidden/fine\_tuning\_distilbert\_airline\_tweets.ipynb at main · toddpglidden/toddpglidden (github.com)](https://github.com/toddpglidden/toddpglidden/blob/main/fine_tuning_distilbert_airline_tweets.ipynb)

Any thoughts?

Thanks!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 24, 2024, 1:48am UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254/2 "2024-09-24T01:48:04Z")

</div>

> You should supply an encoding or a list of encodings to this method that includes input\_ids, but you provided [‘label’]

I don’t often use transformers directly, so sorry if I’m off the mark.  
Maybe you simply chose or used the wrong tokenizer.  
According to the error above, you’re getting an error when data is passed in the wrong format, not the content.

> **[Preparing Text Data for Transformers: Tokenization, Mapping and Padding](https://medium.com/@lokaregns/preparing-text-data-for-transformers-tokenization-mapping-and-padding-9fbfbce28028)**
>
> From now, we won’t just be making predictions but training our own models on our datasets as well.

> **[Tokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 24, 2024, 1:55am UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254/3 "2024-09-24T01:55:01Z")

</div>

I thought it was, but when I searched for it, it seems to be an error that appears in some other, weirder place. It seems to be an error of a type that is not allowed when something extra is written.  
Deleting a few lines may fix it.

> [@Pre-training: ValueError: You should supply an encoding or a list of encodings to this method that includes input\_ids, but you provided \[\]](https://discuss.huggingface.co/t/pre-training-valueerror-you-should-supply-an-encoding-or-a-list-of-encodings-to-this-method-that-includes-input-ids-but-you-provided/40114/2):
>
> You are loading the data directly, You need to encode the data before sending to the model. Looks like you are directly sending the data without any encoding to the model.

> <https://stackoverflow.com/questions/78031519/how-to-resolve-valueerror-you-should-supply-an-encoding-or-a-list-of-encodings>

> <https://github.com/huggingface/datasets/issues/5765>
>
> \### Describe the bug
> 
> Following is my code that I am trying to run, but facing a…n error (have attached the whole error below):
> 
> My code:
> 
> \`\`\`
> from collections import OrderedDict
> import warnings
> 
> import flwr as fl
> import torch
> import numpy as np
> 
> import random
> from torch.utils.data import DataLoader
> 
> from datasets import load\_dataset, load\_metric
> 
> from transformers import AutoTokenizer, DataCollatorWithPadding
> from transformers import AutoModelForSequenceClassification
> from transformers import AdamW
> \#from transformers import tokenized\_datasets
> 
> 
> warnings.filterwarnings("ignore", category=UserWarning)
> \# DEVICE = torch.device("cuda:0" if torch.cuda.is\_available() else "cpu")
> 
> DEVICE = "cpu"
> 
> CHECKPOINT = "distilbert-base-uncased" # transformer model checkpoint
> 
> 
> def load\_data():
> """Load IMDB data (training and eval)"""
> raw\_datasets = load\_dataset("yhavinga/imdb\_dutch")
> raw\_datasets = raw\_datasets.shuffle(seed=42)
> 
> # remove unnecessary data split
> del raw\_datasets\["unsupervised"\]
> 
> tokenizer = AutoTokenizer.from\_pretrained(CHECKPOINT)
> 
> def tokenize\_function(examples):
> return tokenizer(examples\["text"\], truncation=True)
> 
> # random 100 samples
> population = random.sample(range(len(raw\_datasets\["train"\])), 100)
> 
> tokenized\_datasets = raw\_datasets.map(tokenize\_function, batched=True)
> tokenized\_datasets\["train"\] = tokenized\_datasets\["train"\].select(population)
> tokenized\_datasets\["test"\] = tokenized\_datasets\["test"\].select(population)
> 
> # tokenized\_datasets = tokenized\_datasets.remove\_columns("text")
> # tokenized\_datasets = tokenized\_datasets.rename\_column("label", "labels")
> 
> tokenized\_datasets = tokenized\_datasets.remove\_columns("attention\_mask")
> tokenized\_datasets = tokenized\_datasets.remove\_columns("input\_ids")
> tokenized\_datasets = tokenized\_datasets.remove\_columns("label")
> tokenized\_datasets = tokenized\_datasets.remove\_columns("text\_en")
> 
> # tokenized\_datasets = tokenized\_datasets.remove\_columns(raw\_datasets\["train"\].column\_names)
>     
> data\_collator = DataCollatorWithPadding(tokenizer=tokenizer)
> trainloader = DataLoader(
> tokenized\_datasets\["train"\],
> shuffle=True,
> batch\_size=32,
> collate\_fn=data\_collator,
> )
> 
> testloader = DataLoader(
> tokenized\_datasets\["test"\], batch\_size=32, collate\_fn=data\_collator
> )
> 
> return trainloader, testloader
> 
> 
> def train(net, trainloader, epochs):
> optimizer = AdamW(net.parameters(), lr=5e-4)
> net.train()
> for \_ in range(epochs):
> for batch in trainloader:
> batch = {k: v.to(DEVICE) for k, v in batch.items()}
> outputs = net(\*\*batch)
> loss = outputs.loss
> loss.backward()
> optimizer.step()
> optimizer.zero\_grad()
> 
> 
> def test(net, testloader):
> metric = load\_metric("accuracy")
> loss = 0
> net.eval()
> for batch in testloader:
> batch = {k: v.to(DEVICE) for k, v in batch.items()}
> with torch.no\_grad():
> outputs = net(\*\*batch)
> logits = outputs.logits
> loss += outputs.loss.item()
> predictions = torch.argmax(logits, dim=-1)
> metric.add\_batch(predictions=predictions, references=batch\["labels"\])
> loss /= len(testloader.dataset)
> accuracy = metric.compute()\["accuracy"\]
> return loss, accuracy
> 
> 
> def main():
> net = AutoModelForSequenceClassification.from\_pretrained(
> CHECKPOINT, num\_labels=2
> ).to(DEVICE)
> 
> trainloader, testloader = load\_data()
> 
> # Flower client
> class IMDBClient(fl.client.NumPyClient):
> def get\_parameters(self, config):
> return \[val.cpu().numpy() for \_, val in net.state\_dict().items()\]
> 
> def set\_parameters(self, parameters):
> params\_dict = zip(net.state\_dict().keys(), parameters)
> state\_dict = OrderedDict({k: torch.Tensor(v) for k, v in params\_dict})
> net.load\_state\_dict(state\_dict, strict=True)
> 
> def fit(self, parameters, config):
> self.set\_parameters(parameters)
> print("Training Started...")
> train(net, trainloader, epochs=1)
> print("Training Finished.")
> return self.get\_parameters(config={}), len(trainloader), {}
> 
> def evaluate(self, parameters, config):
> self.set\_parameters(parameters)
> loss, accuracy = test(net, testloader)
> return float(loss), len(testloader), {"accuracy": float(accuracy)}
> 
> # Start client
> fl.client.start\_numpy\_client(server\_address="localhost:8080", client=IMDBClient())
> 
> 
> if \_\_name\_\_ == "\_\_main\_\_":
> main()
> \`\`\`
> 
> Error:
> \`\`\`
> Traceback (most recent call last):
> File "client\_2.py", line 136, in \<module\>
> main()
> File "client\_2.py", line 132, in main
> fl.client.start\_numpy\_client(server\_address="localhost:8080", client=IMDBClient())
> File "/home/saurav/.local/lib/python3.8/site-packages/flwr/client/app.py", line 208, in start\_numpy\_client
> start\_client(
> File "/home/saurav/.local/lib/python3.8/site-packages/flwr/client/app.py", line 142, in start\_client
> client\_message, sleep\_duration, keep\_going = handle(
> File "/home/saurav/.local/lib/python3.8/site-packages/flwr/client/grpc\_client/message\_handler.py", line 68, in handle
> return \_fit(client, server\_msg.fit\_ins), 0, True
> File "/home/saurav/.local/lib/python3.8/site-packages/flwr/client/grpc\_client/message\_handler.py", line 157, in \_fit
> fit\_res = client.fit(fit\_ins)
> File "/home/saurav/.local/lib/python3.8/site-packages/flwr/client/app.py", line 252, in \_fit
> results = self.numpy\_client.fit(parameters, ins.config) # type: ignore
> File "client\_2.py", line 122, in fit
> train(net, trainloader, epochs=1)
> File "client\_2.py", line 76, in train
> for batch in trainloader:
> File "/home/saurav/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in \_\_next\_\_
> data = self.\_next\_data()
> File "/home/saurav/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 692, in \_next\_data
> data = self.\_dataset\_fetcher.fetch(index) # may raise StopIteration
> File "/home/saurav/.local/lib/python3.8/site-packages/torch/utils/data/\_utils/fetch.py", line 52, in fetch
> return self.collate\_fn(data)
> File "/home/saurav/.local/lib/python3.8/site-packages/transformers/data/data\_collator.py", line 221, in \_\_call\_\_
> batch = self.tokenizer.pad(
> File "/home/saurav/.local/lib/python3.8/site-packages/transformers/tokenization\_utils\_base.py", line 2713, in pad
> raise ValueError(
> ValueError: You should supply an encoding or a list of encodings to this method that includes input\_ids, but you provided \['text'\]
> \`\`\`
> 
> \### Steps to reproduce the bug
> 
> Run the above code.
> 
> \### Expected behavior
> 
> Don't know, doing it for the first time.
> 
> \### Environment info
> 
> \- \`datasets\` version: 1.12.1
> \- Platform: Linux-5.4.0-58-generic-x86\_64-with-glibc2.29
> \- Python version: 3.8.10
> \- PyArrow version: 11.0.0

---

<div class="post-metadata">

**Author:** ![toddglidden](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@toddglidden](https://discuss.huggingface.co/u/toddglidden)\
**Post date:** [September 24, 2024, 3:54pm UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254/4 "2024-09-24T15:54:51Z")

</div>

It’s the same tokenizer/model combination that worked on the IMDB data that the tutorial used.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 24, 2024, 9:51pm UTC](https://discuss.huggingface.co/t/error-in-pad-function-of-transformers-tokenization-utils-base-py/108254/5 "2024-09-24T21:51:50Z")

</div>

> Published February 2, 2022

That is probably due to this date.

HF is the front line of development, for better or worse, so they only look forward. They don’t really look at the sides or the rear.  
It’s quite common for backward compatibility of functions to disappear in a few months. future warnings are out there, but few people read them. And functions that people don’t use frequently are rather commonly buggy.

Maybe the tutorial is wrong now, give site:huggingface.co to Google and find a new one.  
I’m sure some non-HF know-how would be useful, but Google’s search results are useless these days…  
I know I sound like an old man, but Google wasn’t this useless in the past.