# Shortformer: Better Language Modeling using Shorter Inputs

**URL:** <https://discuss.huggingface.co/t/shortformer-better-language-modeling-using-shorter-inputs/3007>\
**Category:** Research\
**Created:** [December 31, 2020, 10:02am UTC](https://discuss.huggingface.co/t/shortformer-better-language-modeling-using-shorter-inputs/3007 "2020-12-31T10:02:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![FL33TW00D](https://avatars.discourse-cdn.com/v4/letter/f/e0b2c6/32.png) [@FL33TW00D](https://discuss.huggingface.co/u/FL33TW00D)\
**Post date:** [December 31, 2020, 10:02am UTC](https://discuss.huggingface.co/t/shortformer-better-language-modeling-using-shorter-inputs/3007/1 "2020-12-31T10:02:40Z")

</div>

Interesting paper focusing on shorter context windows and improving training speed!

> **[shortformer.pdf](https://ofir.io/shortformer.pdf)**
>
> 349.75 KB