# Inference time gets slower as dataset size increase

**URL:** <https://discuss.huggingface.co/t/inference-time-gets-slower-as-dataset-size-increase/32389>\
**Category:** 🤗Transformers\
**Created:** [February 23, 2023, 4:54pm UTC](https://discuss.huggingface.co/t/inference-time-gets-slower-as-dataset-size-increase/32389 "2023-02-23T16:54:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![hernandez](https://avatars.discourse-cdn.com/v4/letter/h/bbe5ce/32.png) [@hernandez](https://discuss.huggingface.co/u/hernandez)\
**Post date:** [February 23, 2023, 4:54pm UTC](https://discuss.huggingface.co/t/inference-time-gets-slower-as-dataset-size-increase/32389/1 "2023-02-23T16:54:30Z")

</div>

Hello.

I have a custom dataset and trained distilBERT model on that dataset. I would like to do inference. So on my inference .py file. I load the model from the default weights of distilibert (from the hub). Then, if in my directory there are my optimized weights, I load on top of the model my weights (It should basically overwrite the standard weights I get no errors). Then I instantiate the Trainer and I call the predict function for work with batches in the same way of the Training part.

Now I have a very larga dataset, and it take a lot of time for doing inference. But something interesting is the following:

Subset of 5000 samples are inferred in 36 seconds with batch of size 64  
Subset of 50000 samples are inferred in 12.36 minutes with batch size 64

so a 21x factor having a 10x size dataset. Do someone here know the reason?

I’m doing it on simple laptop with a 3060RTX gpu

UPDATE  
This behavior seems to be correlated with the type of dataset. As I tested with another dataset of the same size and it is 8 times faster. So I suspect it is something related with the dataset, maybe the Tokenizer? The complexity of the sentences in the other dataset? I’m using fast Tokenizer pre-trained