# Finetuning summarization model using long text data

**URL:** <https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010>\
**Category:** Beginners\
**Created:** [March 12, 2024, 4:43am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010 "2024-03-12T04:43:13Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![RajMalkar](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@RajMalkar](https://discuss.huggingface.co/u/RajMalkar)\
**Post date:** [March 12, 2024, 4:43am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/1 "2024-03-12T04:43:13Z")

</div>

Hello, I want to fine tune pszemraj/led-base-book-summary model on my custom data of Bank Regulatory Document (15-20 pages) but the documents is well above the input token limit I can truncate it but I believe that it will cause a lot of loss of information. Can anyone suggest the right way to fine-tune using long document.

---

<div class="post-metadata">

**Author:** ![itacdonev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/itacdonev/32/23560_2.png) [@itacdonev](https://discuss.huggingface.co/u/itacdonev)\
**Post date:** [March 12, 2024, 8:21am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/2 "2024-03-12T08:21:10Z")

</div>

I had similar problem. What I did was to add `[EOS]` token (or appropriate token for your model) at the end of each document and then when tokenizing the documents used the following setup:

```auto
trx_text_encoded = tokenizer(
    text = trx_text,
    max_length=tokenizer.model_max_length,
    truncation=True,
    padding="max_length",
    return_overflowing_tokens=True,
    stride=12,
    return_tensors="pt"
)

```

The `return_overflowing_tokens` will “chunk” up extra long text into sequences and the `stride` will create overlapping tokens at the beginning of sequences.

---

<div class="post-metadata">

**Author:** ![RajMalkar](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@RajMalkar](https://discuss.huggingface.co/u/RajMalkar)\
**Post date:** [March 12, 2024, 1:37pm UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/3 "2024-03-12T13:37:00Z")

</div>

Thank you so much for your detailed response! Your solution using the tokenizer setup is incredibly helpful. I’m planning to implement it.

I have one more question: could you please elaborate on the significance of adding the [EOS] token at the end of each document? I want to understand its role in the fine-tuning process.

---

<div class="post-metadata">

**Author:** ![itacdonev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/itacdonev/32/23560_2.png) [@itacdonev](https://discuss.huggingface.co/u/itacdonev)\
**Post date:** [March 13, 2024, 8:21am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/4 "2024-03-13T08:21:30Z")

</div>

The idea with the `[EOS]` token was to “tell the model” that there are more than one document as input. So basically, to sort of separate where one document ends and the other begins and all in between is for one document. I thought this could be helpful since the documents are “disjoint” in essence.

---

<div class="post-metadata">

**Author:** ![RajMalkar](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@RajMalkar](https://discuss.huggingface.co/u/RajMalkar)\
**Post date:** [March 14, 2024, 5:00am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/5 "2024-03-14T05:00:39Z")

</div>

Thank you that was incredibly helpful and provided me with valuable insight.

---

<div class="post-metadata">

**Author:** ![Kamaljp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kamaljp/32/15013_2.png) [@Kamaljp](https://discuss.huggingface.co/u/Kamaljp)\
**Post date:** [March 20, 2024, 12:08am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/6 "2024-03-20T00:08:39Z")

</div>

Thanks @itacdonev this helped me also.

Didn’t know about the addtional KWargs like stride, return\_overflowing\_tokens. Could you please help me , how did you learn about this?

I am curating the code snippets like the one you have shared in Transformers Universe space. I want to provide additional context there. [Transformers Universe - a Hugging Face Space by Kamaljp](https://huggingface.co/spaces/Kamaljp/transformers_universe)

---

<div class="post-metadata">

**Author:** ![itacdonev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/itacdonev/32/23560_2.png) [@itacdonev](https://discuss.huggingface.co/u/itacdonev)\
**Post date:** [March 20, 2024, 6:23am UTC](https://discuss.huggingface.co/t/finetuning-summarization-model-using-long-text-data/77010/7 "2024-03-20T06:23:50Z")

</div>

Two main sources that I consulted first:

1. [HF documenatation](https://huggingface.co/docs/transformers/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase. __call__ )
2. [HF Causal lanugage modeling - preparing datasets (YT video)](https://huggingface.co/docs/transformers/en/tasks/language_modeling#preprocess)

Then I tried the code on a very simple example like couple of sentences with `max_length=4` to see it in action and whether it is behaving as expected.

Hope this help.