# Warning when adding compute\_metrics function to Trainer

**URL:** <https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782>\
**Category:** 🤗Transformers\
**Created:** [February 21, 2021, 11:04am UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782 "2021-02-21T11:04:57Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [February 21, 2021, 11:04am UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/1 "2021-02-21T11:04:57Z")

</div>

When I add a custom compute\_metrics function to the Trainer, I get the warning “Not all data has been set. Are you sure you passed all values?” at each evaluation step.  
This warning is defined in the finalize function of the class trainer\_pt\_utils.DistributedTensorGatherer:

if self.\_offsets[0] != self.process\_length:  
logger.warn(“Not all data has been set. Are you sure you passed all values?”)

This is my compute\_metrics function:

def compute\_metrics(eval\_pred):  
preds, labels = eval\_pred  
preds = np.argmax(preds, axis=1)

```
accuracy = round(accuracy_score(labels, preds),3)
micro_f1 = round(f1_score(labels, preds, average = "micro"),3)
macro_f1 = round(f1_score(labels, preds, average = "macro"),3)

return {"Accuracy": accuracy, "Micro F1": micro_f1, "Macro F1": macro_f1}

```

The additional metrics are successfully returned. So what does this warning mean? Any help would be much appreciated. 🤗

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [February 21, 2021, 3:02pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/2 "2021-02-21T15:02:55Z")

</div>

Could you share the code/script you use so we can reproduce on our side? It seems to indicate that not all the predictions where used for the metric computation (or it could jsut be a bug).

---

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [February 22, 2021, 10:59am UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/3 "2021-02-22T10:59:30Z")

</div>

Thanks! Indeed this is the case. The last values of preds, which is passed to the compute\_metrics function, were all equal to -100, the padding index.

Here is a reduced version of my setup that produces arrays with padded values only:

> **Code**
>
> import torch  
> import torch.nn as nn  
> from torch.utils.data import Dataset  
> from transformers import TrainingArguments, Trainer  
> import numpy as np  
> from sklearn.metrics import accuracy\_score, f1\_score
> 
> class CustomDataset(Dataset):
> 
> ```
> def __init__ (self):
>     
> self.input = [[0.,0.,0.,1.],[1.,0.,0.,1.],[1.,1.,1.,0.],[0.,0.,0.,1.],
> [1.,0.,0.,1.],[1.,1.,1.,0.],[1.,0.,0.,1.],[1.,1.,1.,0.]]
> self.labels = [1,1,0,1,1,0,1,1]
> self.n_tokens = 4
> self.n_labels = 2
>     
> def __len__ (self):
>     
> return(len(self.input))
> 
> def __getitem__ (self, idx):
>     
> input_dict = {"inputs": torch.tensor(self.input[idx]),
> "label_ids": torch.tensor(self.labels[idx])}
> return input_dict
> 
> ```
> 
> class MyTrainer(Trainer):  
> def compute\_loss(self, model, inputs, return\_outputs = False):  
> labels = inputs[“labels”].long()  
> logits = model(inputs[“inputs”])  
> loss\_function = nn.CrossEntropyLoss()  
> loss = loss\_function(logits,labels)  
> return (loss, logits) if return\_outputs else loss
> 
> def compute\_metrics(eval\_pred):  
> preds, labels = eval\_pred  
> print(preds)  
> preds = np.argmax(preds, axis=1)
> 
> ```
> accuracy = accuracy_score(labels, preds)
> micro_f1 = f1_score(labels, preds, average = "micro")
> macro_f1 = f1_score(labels, preds, average = "macro")
> 
> return {"Accuracy": accuracy, "Micro F1": micro_f1, "Macro F1": macro_f1}
> 
> ```
> 
> dataset = CustomDataset()
> 
> n\_tokens = dataset.n\_tokens  
> n\_hidden = 2  
> n\_labels = dataset.n\_labels
> 
> model = nn.Sequential(  
> nn.Linear(n\_tokens,n\_hidden),  
> nn.ReLU(),  
> nn.BatchNorm1d(n\_hidden),  
> nn.Linear(n\_hidden,n\_hidden),  
> nn.ReLU(),  
> nn.BatchNorm1d(n\_hidden),  
> nn.Linear(n\_hidden, n\_labels))
> 
> train\_dataset = eval\_dataset = dataset
> 
> args = TrainingArguments(output\_dir = “example”,  
> report\_to=[],  
> num\_train\_epochs = 3,  
> per\_device\_train\_batch\_size =2,  
> per\_device\_eval\_batch\_size = 1,  
> evaluation\_strategy = “steps”,  
> logging\_steps =4)
> 
> trainer = MyTrainer(  
> model = model,  
> args=args,  
> train\_dataset=train\_dataset,  
> eval\_dataset=eval\_dataset,  
> compute\_metrics=compute\_metrics  
> )
> 
> trainer.train()

Also I found that the shape of preds changes for different values of per\_device\_eval\_batch\_size . If I set eval\_batch\_size=2 in this setup, my compute\_metrics function fails at argmax(axis =1), because the array is one-dimensional.

Generally, preds,

> **as it is used here,**
>
> preds = preds\_gatherer.finalize() if not prediction\_loss\_only else None  
> metrics = self.compute\_metrics(EvalPrediction(predictions=preds, label\_ids=label\_ids))

should simply be the (concatenated) model output, did I understand this correctly?

---

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [February 25, 2021, 12:12pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/4 "2021-02-25T12:12:36Z")

</div>

Hi Sylvain,  
the Code is in the answer above, I missed to directly reply to your comment.

Some additional information:  
In my full setup I am running this model for different datasets with different numbers of classes. The number of empty rows in the preds array ranges from 30%-90%. And here is the interesting relationship that I found: The number of empty rows divided by the number of classes was always 188. Not sure yet, what this means…

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [February 25, 2021, 2:58pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/5 "2021-02-25T14:58:25Z")

</div>

I can reproduce the warning. Will investigate more when I have a bit of time.

---

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [February 25, 2021, 3:27pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/6 "2021-02-25T15:27:29Z")

</div>

Thank you! 🤗  
Trainer would be very helpful for this part of my research pipeline, among other things because of the convenient hyperparameter search integration.

---

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [March 2, 2021, 11:50am UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/7 "2021-03-02T11:50:00Z")

</div>

So I changed the toy dataset (=train=eval dataset) to 16 examples and set the train and eval batch sizes to their default of 8.

> **Code**
>
> import torch  
> import torch.nn as nn  
> from torch.utils.data import Dataset  
> from transformers import TrainingArguments, Trainer  
> import numpy as np  
> from sklearn.metrics import accuracy\_score, f1\_score
> 
> class CustomDataset(Dataset):
> 
> ```
> def __init__ (self):
>     
> self.input = [[1.,0.,0.,0.],[0.,1.,0.,0.],[0.,0.,1.,0.],[0.,0.,0.,1.],
> [1.,0.,0.,0.],[0.,1.,0.,0.],[0.,0.,1.,0.],[0.,0.,0.,1.],
> [1.,0.,0.,0.],[0.,1.,0.,0.],[0.,0.,1.,0.],[0.,0.,0.,1.],
> [1.,0.,0.,0.],[0.,1.,0.,0.],[0.,0.,1.,0.],[0.,0.,0.,1.]]
> self.labels = [1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0]
> self.n_tokens = 4
> self.n_labels = 2
>     
> def __len__ (self):
>     
> return(len(self.input))
> 
> def __getitem__ (self, idx):
>     
> input_dict = {"inputs": torch.tensor(self.input[idx]),
> "label_ids": torch.tensor(self.labels[idx])}
> return input_dict
> 
> ```
> 
> class MyTrainer(Trainer):  
> def compute\_loss(self, model, inputs, return\_outputs = False):  
> labels = inputs[“labels”].long()  
> logits = model(inputs[“inputs”])  
> loss\_function = nn.CrossEntropyLoss()  
> loss = loss\_function(logits,labels)  
> return (loss, logits) if return\_outputs else loss
> 
> def compute\_metrics(eval\_pred):  
> preds, labels = eval\_pred  
> print(preds)  
> preds = np.argmax(preds, axis=1)
> 
> ```
> accuracy = accuracy_score(labels, preds)
> micro_f1 = f1_score(labels, preds, average = "micro")
> macro_f1 = f1_score(labels, preds, average = "macro")
> 
> return {"Accuracy": accuracy, "Micro F1": micro_f1, "Macro F1": macro_f1}
> 
> ```
> 
> dataset = CustomDataset()
> 
> n\_tokens = dataset.n\_tokens  
> n\_hidden = 2  
> n\_labels = dataset.n\_labels
> 
> model = nn.Sequential(  
> nn.Linear(n\_tokens,n\_hidden),  
> nn.ReLU(),  
> nn.BatchNorm1d(n\_hidden),  
> nn.Linear(n\_hidden,n\_hidden),  
> nn.ReLU(),  
> nn.BatchNorm1d(n\_hidden),  
> nn.Linear(n\_hidden, n\_labels))
> 
> train\_dataset = eval\_dataset = dataset
> 
> args = TrainingArguments(output\_dir = “example”,  
> report\_to=[],  
> num\_train\_epochs = 3,  
> per\_device\_train\_batch\_size =8,  
> per\_device\_eval\_batch\_size = 8,  
> evaluation\_strategy = “steps”,  
> logging\_steps =4)
> 
> trainer = MyTrainer(  
> model = model,  
> args=args,  
> train\_dataset=train\_dataset,  
> eval\_dataset=eval\_dataset,  
> compute\_metrics=compute\_metrics  
> )
> 
> trainer.train()

Here is what happens, when comparing the output of trainer.predict and trainer.model:

preds1 = trainer.model(dataset[0:16][“inputs”]).detach().numpy()  
preds2 = trainer.predict(dataset)[0]

preds2 == np.concatenate((preds1[1:8], preds1[9:16], [[-100.,-100],[-100.,-100]]))  
Output:  
array([[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True],  
[True, True]])

**So what got lost here is the first instance of both the two batches.**

I changed it to a 3-class data set and the same two rows were missing, so in this small example there is no indication of the relationship between number of missing values and number of classes here that I saw in the full setup.

Edit:  
I had to run the line preds1 = trainer.model(dataset[0:16][“inputs”]).detach().numpy() twice to get these results. The first call of trainer.model gives different results than subsequent ones.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 2, 2021, 9:23pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/8 "2021-03-02T21:23:58Z")

</div>

I figured out your problem: the `Trainer` is optimized for the Transformers models so it expects the model outputs to always be a tuple, with the loss first and logits second if labels are provided, or just the logits if no labels are provided.

Therefore, in you custom `compute_loss` of your subclass of `Trainer`, the last line should be:

```auto
    return (loss, (loss, logits)) if return_outputs else loss

```

---

<div class="post-metadata">

**Author:** ![chelone](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@chelone](https://discuss.huggingface.co/u/chelone)\
**Post date:** [March 3, 2021, 8:58am UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/9 "2021-03-03T08:58:30Z")

</div>

Thanks so much!!!

May I suggest to add this information to the Trainer documentation, either at the subclassing Trainer example at the beginning or at the compute\_loss part.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 3, 2021, 1:49pm UTC](https://discuss.huggingface.co/t/warning-when-adding-compute-metrics-function-to-trainer/3782/10 "2021-03-03T13:49:36Z")

</div>

Sure! If you want to make a PR with this info that would be great! For now there is just a note in the `Trainer` docstring that is rather vague, so anything more useful is more than welcome.