# Tokenizer behaviour with pipeline

**URL:** <https://discuss.huggingface.co/t/tokenizer-behaviour-with-pipeline/48969>\
**Category:** 🤗Tokenizers\
**Created:** [August 1, 2023, 10:49am UTC](https://discuss.huggingface.co/t/tokenizer-behaviour-with-pipeline/48969 "2023-08-01T10:49:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![LukeFiasco](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@LukeFiasco](https://discuss.huggingface.co/u/LukeFiasco)\
**Post date:** [August 1, 2023, 10:49am UTC](https://discuss.huggingface.co/t/tokenizer-behaviour-with-pipeline/48969/1 "2023-08-01T10:49:28Z")

</div>

Hey all,

I am trying to use one of the Llama2 models to generate text (summarise docs). This means that I am dealing with long texts that need to be truncated. Upon using the Pipeline API I have discovered some weird behaviour.

```auto
tokenizer = AutoTokenizer.from_pretrained(
  local_model_dir, 
  padding_side="right", 
  model_max_length=MAX_LENGTH, 
  truncation=True
)

# error in source, context should be 4k
upldated_config = LlamaConfig(max_position_embeddings=MAX_POS_EMBEDDINGS)

# 8 bit precision to fit GPU -> maybe try 4 bits?
model = AutoModelForCausalLM.from_pretrained(
    local_model_dir,
    torch_dtype=torch.float16,
    trust_remote_code=True,
    device_map="auto",
    load_in_8bit=True,
    config=upldated_config
)

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    return_full_text=False,
)

```

it seems to me that when I call `pipeline(long_input)` the input text is not correctly truncated. This is checked by calling `pipeline.pre_process(long_input)` which results in the following message:

> Token indices sequence length is longer than the specified maximum sequence length for this model (12351 \> 3896).

I have come to understand that truncation is only done when calling `tokenizer(long_input, truncation=True, max_length=MAX_LENGTH)` However when i try to pass truncation to the Pipeline like this `pipeline(long_input, truncation=True)`. It also throws as error:

> The following model\_kwargs are not used by the model: [‘truncation’] (note: typos in the generate arguments will also show up in this list)

So this all begs the question: How do i make truncation happen while using a pipeline? To me it seems inuitive that when Truncation=True while initiating the tokenizer, and adding it to the pipeline. All downstream texts should be tokenized.