# Why split sequences into shorter chunks when pretraining llm

**URL:** <https://discuss.huggingface.co/t/why-split-sequences-into-shorter-chunks-when-pretraining-llm/50892>\
**Category:** Beginners\
**Created:** [August 16, 2023, 3:36am UTC](https://discuss.huggingface.co/t/why-split-sequences-into-shorter-chunks-when-pretraining-llm/50892 "2023-08-16T03:36:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChloeYang](https://avatars.discourse-cdn.com/v4/letter/c/b38774/32.png) [@ChloeYang](https://discuss.huggingface.co/u/ChloeYang)\
**Post date:** [August 16, 2023, 3:36am UTC](https://discuss.huggingface.co/t/why-split-sequences-into-shorter-chunks-when-pretraining-llm/50892/1 "2023-08-16T03:36:56Z")

</div>

In the Transformers doc about language modeling, [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling)

we need to do data preprocess below:

- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.

This kind of processing will generate many incomplete sentences. Additionally, I noticed that the text used in the examples is mostly documents, such as wikis, which might not have a significant impact. So, when the text is in a question-and-answer format, like a question followed by an answer, will this chunking operation still be necessary?