# Scaling batch inference for Longformer model

**URL:** <https://discuss.huggingface.co/t/scaling-batch-inference-for-longformer-model/28140>\
**Category:** Models\
**Created:** [December 19, 2022, 8:59pm UTC](https://discuss.huggingface.co/t/scaling-batch-inference-for-longformer-model/28140 "2022-12-19T20:59:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![benhamner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/benhamner/32/13019_2.png) [@benhamner](https://discuss.huggingface.co/u/benhamner)\
**Post date:** [December 19, 2022, 8:59pm UTC](https://discuss.huggingface.co/t/scaling-batch-inference-for-longformer-model/28140/1 "2022-12-19T20:59:56Z")

</div>

Does anyone have experience running Longformer for inference at scale (millions of docs)?

I’m interested in:

- What GPU architecture + software library would maximize throughput in batches per node?
- If GPU cloud costs were taken into account, would the setup that maximizes cost efficiency be different than the one that maximizes throughput?