# Longformer for text summarization **URL:** \ **Category:** Beginners\ **Created:** [July 26, 2020, 8:47am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478 "2020-07-26T08:47:30Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![kbmr812](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@kbmr812](https://discuss.huggingface.co/u/kbmr812)\ **Post date:** [July 26, 2020, 8:47am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/1 "2020-07-26T08:47:30Z")
Hello! 🙂 Does anyone know how to summarize long documents/news articles using the Longformer library? I am aware that using T5, the token limit is 512. I would really appreciate any help in this area! Thank you 🙂 ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 26, 2020, 8:55am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/2 "2020-07-26T08:55:51Z")
Hi, it’s possible to use Longformer for summerization, the way its done now, is taking BART model and then replacing it’s self attention with longformer sliding window attention so that it can take longer sequences. Check this two issues, [first](https://github.com/huggingface/transformers/issues/4406), [second](https://github.com/allenai/longformer/issues/28), and this [branch](https://github.com/allenai/longformer/tree/encoderdecoder) of longformer repo ---
**Author:** ![kbmr812](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@kbmr812](https://discuss.huggingface.co/u/kbmr812)\ **Post date:** [July 26, 2020, 10:42am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/3 "2020-07-26T10:42:57Z")
Hi, I followed the example from the [branch](https://github.com/allenai/longformer/tree/encoderdecoder) of longformer repo, but it seems that the final output is a tensor instead of words/text. How can I convert it into words? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 26, 2020, 11:29am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/4 "2020-07-26T11:29:28Z")
Note that, these models are not yet fine-tuned for long summarization, you’ll need to fine-tune them yourself or wait till someone does that. And yes, the model returns a tensor, to generate text you’ll need to use the `generate` method. Here’s nice [blog post](https://huggingface.co/blog/how-to-generate) on the generate method ---
**Author:** ![guilleVENT](https://avatars.discourse-cdn.com/v4/letter/g/d9b06d/32.png) [@guilleVENT](https://discuss.huggingface.co/u/guilleVENT)\ **Post date:** [January 22, 2021, 11:02am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/5 "2021-01-22T11:02:34Z")
the links are broken 😕 ---
**Author:** ![kmfoda](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kmfoda/32/42122_2.png) [@kmfoda](https://discuss.huggingface.co/u/kmfoda)\ **Post date:** [November 30, 2021, 8:59am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/6 "2021-11-30T08:59:14Z")
Hey @valhalla. Hope you’re well. In your earlier comment here you mention that Longformer for summarisation takes the BART model and replaces it’s self attention. I was under the impression that [this model](https://huggingface.co/allenai/longformer-base-4096) was based off ROBERTA. Can you confirm if there is a long former model based off BART and if so where it is on the hub? ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [November 30, 2021, 9:10am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/7 "2021-11-30T09:10:29Z")
Hi, LongFormer is an encoder-only Transformer (similar to BERT/RoBERTa), it only has a different attention mechanism, allowing it to be used on longer sequences. The author also released LED (LongFormer Encoder Decoder), which is a seq2seq model (like BART, T5) but with LongFormer as encoder, hence allowing it to be used to summarize long documents for instance, or translate long texts. Weights are on the hub: [Models - Hugging Face](https://huggingface.co/models?other=led) ---
**Author:** ![kmfoda](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kmfoda/32/42122_2.png) [@kmfoda](https://discuss.huggingface.co/u/kmfoda)\ **Post date:** [November 30, 2021, 9:45am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/8 "2021-11-30T09:45:12Z")
Amazing that’s very helpful thank you. I can see on that link that Allen AI’s LED model is based of bart-base which is ideal. If I were to look to try and convert bart-large to a LED would this [notebook](https://colab.research.google.com/drive/1k2e5HRvnjeEcuL9GFBXkN7P-zSa1rkMl?authuser=1) still be the right approach or is this for encoder-only models? ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [November 30, 2021, 10:55am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/9 "2021-11-30T10:55:16Z")
> [@kmfoda](#): > > notebook I don’t have access to this notebook. ---
**Author:** ![kmfoda](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kmfoda/32/42122_2.png) [@kmfoda](https://discuss.huggingface.co/u/kmfoda)\ **Post date:** [November 30, 2021, 11:59am UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/10 "2021-11-30T11:59:07Z")
okay no problem. Thanks for your help! ---
**Author:** ![NightMachinery](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nightmachinery/32/3366_2.png) [@NightMachinery](https://discuss.huggingface.co/u/NightMachinery)\ **Post date:** [August 6, 2022, 4:54pm UTC](https://discuss.huggingface.co/t/longformer-for-text-summarization/478/11 "2022-08-06T16:54:15Z")
Any updates? Can someone share the relevant code snippet from that private notebook publicly?