# Looking for tool class to do predictions

**URL:** <https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399>\
**Category:** 🤗Transformers\
**Created:** [October 7, 2020, 9:45am UTC](https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399 "2020-10-07T09:45:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![PhilipMay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philipmay/32/2324_2.png) [@PhilipMay](https://discuss.huggingface.co/u/PhilipMay)\
**Post date:** [October 7, 2020, 9:45am UTC](https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399/1 "2020-10-07T09:45:13Z")

</div>

Hi,  
I am looking for a tool class to do predictions with BERT models. Like the `Trainer` class just for prediction time. There is much more to do than just doing `logits = model(input_ids)`. When you have large input lists of text you have to do this in batches or you get an OOM from your GPU and there is some more advanced stuff like smart batching (sorting the sentences by token length and batch short sentences with other short sentences to have short batches with minimal padding).

Is there something like this?

Thanks  
Philip

---

<div class="post-metadata">

**Author:** ![stefan-it](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-it/32/20175_2.png) [@stefan-it](https://discuss.huggingface.co/u/stefan-it)\
**Post date:** [October 7, 2020, 9:53am UTC](https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399/2 "2020-10-07T09:53:01Z")

</div>

Hi Philip,

maybe the HF Pipelines implementation could help:

> <https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb>

It should also support e.g. multiple sentences (`List[str]`) as inputs and you could use of course own trained and fine-tuned models. Detailed implementation can be found here:

> <https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py>

To use it for text classification/sentiment analysis:

```auto

nlp_sentence_classif = pipeline('sentiment-analysis')
nlp_sentence_classif('Such a nice weather outside !')

```

and it also outputs the score:

```auto
[{'label': 'POSITIVE', 'score': 0.9997656}]

```

I like using Pipelines especially for testing NER models 🤗

---

<div class="post-metadata">

**Author:** ![PhilipMay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philipmay/32/2324_2.png) [@PhilipMay](https://discuss.huggingface.co/u/PhilipMay)\
**Post date:** [October 7, 2020, 10:48am UTC](https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399/3 "2020-10-07T10:48:44Z")

</div>

> [@stefan-it](#):
>
> I like using Pipelines especially for testing NER models

I bet! 😉

Thanks for the answer.

---

<div class="post-metadata">

**Author:** ![PhilipMay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philipmay/32/2324_2.png) [@PhilipMay](https://discuss.huggingface.co/u/PhilipMay)\
**Post date:** [October 9, 2020, 12:22pm UTC](https://discuss.huggingface.co/t/looking-for-tool-class-to-do-predictions/1399/4 "2020-10-09T12:22:30Z")

</div>

@stefan-it although I see no support for batching. At least not in the `TextClassificationPipeline`.

When I follow the naive approach:

```python
model = AutoModelForSequenceClassification.from_pretrained(model_dir)
tokenizer = AutoTokenizer.from_pretrained(model_dir)
pipeline = TextClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    device=0,
)
pipeline(unlabeled_text_list)

```

I am getting an CUDA OOM if the list (unlabeled\_text\_list) is long…