# Sample evaluation script on custom dataset

**URL:** <https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654>\
**Category:** Beginners\
**Created:** [December 9, 2021, 8:22am UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654 "2021-12-09T08:22:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 9, 2021, 8:22am UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/1 "2021-12-09T08:22:58Z")

</div>

Hey, I have a custom dataset. can you send a sample script to get the accuracy on such a dataset? I was going through examples and I couldn’t get a code that does that. Can someone send me a resource?

my dataset is of the format-  
premise , hypothesis, label(0 or 1)  
and my model is deberta

Thanks

@lewtun

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 9, 2021, 9:10am UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/2 "2021-12-09T09:10:43Z")

</div>

side note: i am using a zero-shot version of deberta- [NDugar/ZSD-microsoft-v2xxlmnli · Hugging Face](https://huggingface.co/NDugar/ZSD-microsoft-v2xxlmnli)

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [December 9, 2021, 1:20pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/3 "2021-12-09T13:20:51Z")

</div>

Hey @NDugar, the simplest way to do this would be with the `accuracy` metric in the `datasets` library, e.g.

```python
from datasets import load_metric

# You might need to install scikit-learn
metric = load_metric("accuracy")
results = metric.compute(references=[0, 1], predictions=[0, 1])
results
# {'accuracy': 1.0}

```

As you can see in this example, the main bit of work you need to do is create the arrays for the `references` (i.e. ground truth labels) and `predictions` (predicted labels). If you’re using the `Trainer` you can get these arrays easily by calling `Trainer.predict(test_dataset)`

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 9, 2021, 1:56pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/4 "2021-12-09T13:56:37Z")

</div>

In the sample code you wrote where do I mention the model and the dataset?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [December 9, 2021, 2:11pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/5 "2021-12-09T14:11:09Z")

</div>

If you’re after an end-to-end example, you could try adapting the official text classification example [here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb) 🙂

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 9, 2021, 2:13pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/6 "2021-12-09T14:13:42Z")

</div>

ok thank you.

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 9, 2021, 5:32pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/8 "2021-12-09T17:32:43Z")

</div>

Hey, I don’t think this code is useful for evaluation. is it possible you have another code for this purpose?

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 10, 2021, 8:19pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/9 "2021-12-10T20:19:34Z")

</div>

```auto
The following columns in the evaluation set don't have a corresponding arg
ument in `DebertaV2ForSequenceClassification.forward` and have been ignored
: premise, id, hypothesis.
*****Running Evaluation*****
  Num examples = 187
  Batch size = 4
100%|██████████████████████████████████████| 47/47 [00:03<00:00, 13.15it/s]

```

Hey,I was able to run the evaluation in a different way but I am not getting the results in the end or in the logs. How do I get the result? Also it seems to ignore the key columns what do I do? @lewtun

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [December 13, 2021, 3:45pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/10 "2021-12-13T15:45:44Z")

</div>

Hey @NDugar if you’re using the `Trainer` my suggestion would be to run `Trainer.predict(your_test_dataset)` so you can get all the predictions. Then you should be able to feed those into the `accuracy` metric in a second step (or whatever metric you’re interested in).

If you’re still having trouble, I suggest providing a minimal reproducible example, as explained [here](https://huggingface.co/course/chapter8/3?fw=pt)🙂

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 13, 2021, 6:02pm UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/11 "2021-12-13T18:02:28Z")

</div>

oh um small question what is the difference between `Trainer.predict(your_test_dataset)` and `Trainer.evaluate(your_test_dataset)` ? I think that predict might solve my problem. i was using evaluate and getting errors previously.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [December 14, 2021, 8:30am UTC](https://discuss.huggingface.co/t/sample-evaluation-script-on-custom-dataset/12654/12 "2021-12-14T08:30:20Z")

</div>

`Trainer.predict()` will return the model’s predicted labels for your dataset, while `Trainer.evaluate()` will go one step further and compute the loss + anything else you’ve defined in the `compute_metrics()` function