# Generate() and automatic truncation of context

**URL:** <https://discuss.huggingface.co/t/generate-and-automatic-truncation-of-context/91995>\
**Category:** 🤗Transformers\
**Created:** [June 13, 2024, 1:08pm UTC](https://discuss.huggingface.co/t/generate-and-automatic-truncation-of-context/91995 "2024-06-13T13:08:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![DonThomasitos](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@DonThomasitos](https://discuss.huggingface.co/u/DonThomasitos)\
**Post date:** [June 13, 2024, 1:08pm UTC](https://discuss.huggingface.co/t/generate-and-automatic-truncation-of-context/91995/1 "2024-06-13T13:08:47Z")

</div>

Hey guys, I have a question regarding language modeling:

I have a decoder-only model (Llama) and want to generate a sequence. During the sequence generation, I want to have a continuously “full” context, equivalent to the model’s maximum context size during training. To my understanding, this would involve truncating the first token after every single token generation.

However, I don’t find this functionality in `generate().` Instead, the MaxLengthCriteria assumes that for the generation of N new tokens, my context length is at max `(CONTEXT_LEN - N)`.

The “manual” approach would be to generate one token at a time with generate() and then cut the first token in the sequence every time. But I’m sure this can be done more elegantly?

Thank you.