# Need to preprocess text inputs before tokenizer?

**URL:** <https://discuss.huggingface.co/t/need-to-preprocess-text-inputs-before-tokenizer/18929>\
**Category:** Models\
**Created:** [June 10, 2022, 4:34am UTC](https://discuss.huggingface.co/t/need-to-preprocess-text-inputs-before-tokenizer/18929 "2022-06-10T04:34:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![tupk](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@tupk](https://discuss.huggingface.co/u/tupk)\
**Post date:** [June 10, 2022, 4:34am UTC](https://discuss.huggingface.co/t/need-to-preprocess-text-inputs-before-tokenizer/18929/1 "2022-06-10T04:34:18Z")

</div>

I test by adding text input into tokenizer, and summary, but output is not good, maybe need preprocess input text before summarization  
my question is: “Do I need to preprocess text inputs before tokenizer??? if yes, how to process it ?”

```auto
tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")
model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv", attention_type="original_full")

model = model.to(device)
inputs = tokenizer(text_ip, return_tensors='pt', truncation=True).to(device)
prediction = model.generate(**inputs) #max_length output is 256
prediction = tokenizer.batch_decode(prediction)
print(prediction)

```

Here is outputs:

```auto
['<s> the problem of machine learning is to find a way to learn from data.<n> this paper studies the problem of finding a way to learn a way to learn a way to learn a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to learn a way to learn.<n> we study the problem of finding a way to']

```

I test another input text, the output is:

```auto
['<s> we report a study of the optical response of a high-@xmath0 two - dimensional ( 2d ) pyrochlore compound, la@xmath1sr@xmath2cacu@xmath3o@xmath4, by means of time - of - flight spectroscopy.<n> we find that the 2d optical response of the compound is strongly dependent on the lattice constant, @xmath5, and the lattice temperature, @xmath6. for @xmath7,<n> the 2d optical response is very sensitive to the variation of @xmath5 with @xmath6. for @xmath8,<n> the 2d optical response becomes insensitive to the variation of @xmath5 with @xmath6.<n> these results suggest a new mechanism for the optical response of high-@xmath0 2d pyrochlore compounds. <n> high-@xmath0 2d pyrochlore compounds, such as la@xmath1sr@xmath2cacu@xmath3o@xmath4 ( lsco ) and srtio@xmath2 ( srtio@xmath2']

```