# Recommended way to perform batch inference for generation

**URL:** <https://discuss.huggingface.co/t/recommended-way-to-perform-batch-inference-for-generation/4211>\
**Category:** 🤗Transformers\
**Created:** [March 6, 2021, 10:18pm UTC](https://discuss.huggingface.co/t/recommended-way-to-perform-batch-inference-for-generation/4211 "2021-03-06T22:18:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [March 6, 2021, 10:18pm UTC](https://discuss.huggingface.co/t/recommended-way-to-perform-batch-inference-for-generation/4211/1 "2021-03-06T22:18:28Z")

</div>

I want to perform inference for a large number of examples. Inference is relatively slow since `generate` is called a lot of times for my use case (using rtx 3090). I wanted to ask what is the recommended way to perform batch inference, I’m using CTRL.  
This [post](https://github.com/huggingface/transformers/issues/3021) from @patrickvonplaten directs towards `test_batch_generation` method of GPT2 for variable sized sequences, but it doesn’t seem like batched because we call `generate` twice for both items in `sentences` list. Also, I don’t see the same method defined for CTRL. Also came across this [GH issue](https://github.com/huggingface/transformers/issues/3021) which expects users to make it a whole full length sequence, so the output is also a one single vector which has the output for individual sequences that were concatenated. This approach is a bit specific and may not work as I’d expect.

Can anyone suggest a decent approach to perform batched generation with variable sized inputs ? Ideally, I’d like to pass a list containing my inputs (just like `sentences`) and call generate once for 3-4 inputs.