# Best practices for improving text generation speed?

**URL:** <https://discuss.huggingface.co/t/best-practices-for-improving-text-generation-speed/17367>\
**Category:** Beginners\
**Created:** [April 30, 2022, 9:59pm UTC](https://discuss.huggingface.co/t/best-practices-for-improving-text-generation-speed/17367 "2022-04-30T21:59:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivnle](https://avatars.discourse-cdn.com/v4/letter/i/b5a626/32.png) [@ivnle](https://discuss.huggingface.co/u/ivnle)\
**Post date:** [April 30, 2022, 9:59pm UTC](https://discuss.huggingface.co/t/best-practices-for-improving-text-generation-speed/17367/1 "2022-04-30T21:59:52Z")

</div>

Aside from buying a faster/larger GPU, are there any best practices for increasing text generation speed? Specifically, I’d like to use GPT-2 (of various sizes) to generate a large set of text (5,000 examples, 1,000 BPE tokens each).

My initial research shows there are a few options:  
(a) [DeepSpeed for inference](https://www.deepspeed.ai/tutorials/inference-tutorial/)  
(b) [batched generation](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517)  
(c) [fp16 inference](https://huggingface.co/docs/transformers/performance#fp16-inference). Although I’m not sure how to do this outside of Trainer. I could call `model.half()` but it’s not clear to me if that’s the right way to go about this.

Any advice is appreciated!