# Summarization on long documents

**URL:** <https://discuss.huggingface.co/t/summarization-on-long-documents/920>\
**Category:** 🤗Transformers\
**Created:** [August 29, 2020, 5:57pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920 "2020-08-29T17:57:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [August 29, 2020, 5:57pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/1 "2020-08-29T17:57:00Z")

</div>

Hi to all!

I am facing a problem, how can someone summarize a very long text? I mean very long text that also always grows. It is a concatenation of many smaller texts. I see that many of the models have a limitation of maximum input, otherwise don’t work on the complete text or they don’t work at all.

So, what is the correct way of using these models with long documents.

A code snippet with an example of how to handle long documents with the existing models would be perfect to start with!

Thank you!

@sshleifer

---

<div class="post-metadata">

**Author:** ![pratikbhavsar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pratikbhavsar/32/52544_2.png) [@pratikbhavsar](https://discuss.huggingface.co/u/pratikbhavsar)\
**Post date:** [September 4, 2020, 6:38am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/2 "2020-09-04T06:38:45Z")

</div>

You can try extractive summarisation followed by abstractive. In the extractive step you choose top k sentences of which you choose top n allowed till model max length.

Another way is to use successive abstractive summarisation where you summarise in chunk of model max length and then again use it to summarise till the length you want. This method will be super expensive.

You can also combine first + second method.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 7, 2020, 9:11pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/3 "2020-09-07T21:11:35Z")

</div>

Yeah I have never done either, but the second method would be very easy to code.  
for each document: split it into groups of ~500 words, generate 15 word summaries, blindly combine the summaries.

There is also lots of ongoing research into using Longformer for summarization, but I’m not quite sure where that stands. @valhalla or @patrickvonplaten may know.

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [September 8, 2020, 3:52pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/4 "2020-09-08T15:52:24Z")

</div>

Thank you all!

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [September 13, 2020, 5:58pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/5 "2020-09-13T17:58:09Z")

</div>

Hi @pratikbhavsar @sshleifer !

I want to implement the first approach but I have a question.

Here is my function for combining the top K sentences from the extractive summarization.

```
def concat_sentences_till_max_length(top_n_sentences, max_length):
  text = ''

  for s in top_n_sentences:
      if len(text + " " + s) <= max_length:
          text = text + " " + s

  return text

```

However, I don’t know how to the get the max input length of the abstractive summarizer.

I am using a summarization pipeline and I want to make it work for any transformers model.

Is the following the correct way to do it?

```
pipe = pipeline("summarization", model=model, framework='pt')
document = concat_sentences_till_max_length(top_n_sentences, pipe.model.config.max_length)
```

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 13, 2020, 7:09pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/6 "2020-09-13T19:09:38Z")

</div>

I think it’s harder to control lengths through `pipeline`.

- I would look at the run\_eval.py [logic](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_eval.py#L31)
- `model_max_length` is in token space not character space – `len(text) != len(tokenizer(text))`

---

<div class="post-metadata">

**Author:** ![dipanjanS](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@dipanjanS](https://discuss.huggingface.co/u/dipanjanS)\
**Post date:** [September 18, 2020, 5:08am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/7 "2020-09-18T05:08:46Z")

</div>

Just found out about the forums, so far I’ve mostly been in the GH Issues page. Glad to see some useful content here!

This is what I have been using currently, mostly chunking and generating summaries, but definitely open to hearing some good approaches on this.

```python
# generate chunks of text \ sentences <= 1024 tokens
def nest_sentences(document):
  nested = []
  sent = []
  length = 0
  for sentence in nltk.sent_tokenize(document):
    length += len(sentence)
    if length < 1024:
      sent.append(sentence)
    else:
      nested.append(sent)
      sent = []
      length = 0

  if sent:
    nested.append(sent)
  return nested

# generate summary on text with <= 1024 tokens
def generate_summary(nested_sentences):
  device = 'cuda'
  summaries = []
  for nested in nested_sentences:
    input_tokenized = bart_tokenizer.encode(' '.join(nested), truncation=True, return_tensors='pt')
    input_tokenized = input_tokenized.to(device)
    summary_ids = bart_model.to(device).generate(input_tokenized,
                                      length_penalty=3.0,
                                      min_length=30,
                                      max_length=100)
    output = [bart_tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]
    summaries.append(output)
  summaries = [sentence for sublist in summaries for sentence in sublist]
  return summaries

```

Had originally posted it [here](https://github.com/huggingface/transformers/issues/4224)

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 18, 2020, 1:54pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/8 "2020-09-18T13:54:47Z")

</div>

This is great! You can try distilbart-cnn/distilbart-xsum variants to see which type of summaries you like more!

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [September 18, 2020, 6:56pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/9 "2020-09-18T18:56:30Z")

</div>

Hi @dipanjanS!

This is great!

Let’s say now that I use AutoTokenizer and AutoModel to load agnostically a tokenizer and a model. How can I know the max input length? Instead of using 1024 as a hardcoded value how can I know the value for a specific model? Maybe reading it from some configuration?

Any help would be appreciated.

Thanks!

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 7:31pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/10 "2020-09-18T19:31:11Z")

</div>

You can get the max length using

```python
tokenizer.model_max_length

```

---

<div class="post-metadata">

**Author:** ![dipanjanS](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@dipanjanS](https://discuss.huggingface.co/u/dipanjanS)\
**Post date:** [September 18, 2020, 8:06pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/11 "2020-09-18T20:06:46Z")

</div>

Sure will check this out some time!

---

<div class="post-metadata">

**Author:** ![dipanjanS](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@dipanjanS](https://discuss.huggingface.co/u/dipanjanS)\
**Post date:** [September 18, 2020, 8:09pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/12 "2020-09-18T20:09:53Z")

</div>

@echatzikyriakidis thanks. The code was created for showcasing in a workshop so definitely it can be made more configurable etc. You can check the code if needed in a recent workshop I gave [here](https://github.com/dipanjanS/nlp_workshop_odsc_europe20/tree/master/notebooks/Module_04_NLP%20Applications%20-%20Deep%20Transfer%20Learning).

I think as @valhalla mentioned we can use that attribute as an argument \ config for the function to reuse it across different models.

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [September 19, 2020, 10:04am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/13 "2020-09-19T10:04:28Z")

</div>

This is great! @dipanjanS@valhalla

Thank you!

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [September 20, 2020, 9:55am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/14 "2020-09-20T09:55:00Z")

</div>

@dipanjanS @valhalla I think it should be sent = [sentence] instead of sent = [].

Whenever a chunk is ready the current sentence is skipped right now.

Maybe this is the fix:

```
        if length < 1024:
            sentences.append(sentence)
        else:
            nested.append(sentences)
            sentences = [sentence]
            length = len(sentence)
```

---

<div class="post-metadata">

**Author:** ![spate141](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spate141/32/14117_2.png) [@spate141](https://discuss.huggingface.co/u/spate141)\
**Post date:** [November 16, 2020, 11:33pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/15 "2020-11-16T23:33:30Z")

</div>

@echatzikyriakidis May I ask which extractive model/approach are you using? Historically we have used the Gensim version of PageRank but looks like they are removing it from the next release. Just looking for any viable alternatives.

Thanks!

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [November 17, 2020, 7:19am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/16 "2020-11-17T07:19:45Z")

</div>

HI @spate141!

Checkout python module **sumy** it has many alternatives.

---

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [November 17, 2020, 11:58am UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/17 "2020-11-17T11:58:17Z")

</div>

Has anyone tried with Longformer / Reformer models now available?

They definitely need some fine-tuning but I have high expectations these methods will be much better than chunking.

---

<div class="post-metadata">

**Author:** ![spate141](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spate141/32/14117_2.png) [@spate141](https://discuss.huggingface.co/u/spate141)\
**Post date:** [November 17, 2020, 2:29pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/18 "2020-11-17T14:29:11Z")

</div>

Thanks, I’ll check it out.

---

<div class="post-metadata">

**Author:** ![spate141](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spate141/32/14117_2.png) [@spate141](https://discuss.huggingface.co/u/spate141)\
**Post date:** [November 17, 2020, 8:11pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/19 "2020-11-17T20:11:17Z")

</div>

Just out of curiosity, which summarizer you think would work best on plain text documents? In few of my tests I find `LsaSummarizer` to be most interesting one. Thoughts?

---

<div class="post-metadata">

**Author:** ![echatzikyriakidis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echatzikyriakidis/32/154_2.png) [@echatzikyriakidis](https://discuss.huggingface.co/u/echatzikyriakidis)\
**Post date:** [November 17, 2020, 8:49pm UTC](https://discuss.huggingface.co/t/summarization-on-long-documents/920/20 "2020-11-17T20:49:06Z")

</div>

I really don’t know. Maybe you can try them all and see the results?

What test did you accomplish and found that LsaSummarizer is better?

[Next page](https://discuss.huggingface.co/t/summarization-on-long-documents/920.md?page=2)