# Pipeline's Tokenizer vs training tokenizer

**URL:** <https://discuss.huggingface.co/t/pipelines-tokenizer-vs-training-tokenizer/4240>\
**Category:** Beginners\
**Created:** [March 8, 2021, 2:13pm UTC](https://discuss.huggingface.co/t/pipelines-tokenizer-vs-training-tokenizer/4240 "2021-03-08T14:13:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](https://discuss.huggingface.co/u/theudster)\
**Post date:** [March 8, 2021, 2:13pm UTC](https://discuss.huggingface.co/t/pipelines-tokenizer-vs-training-tokenizer/4240/1 "2021-03-08T14:13:39Z")

</div>

Hello, I am trying to create a pipeline from a trained model. From what I understand I need to provide a tokenizer so that my new input will be tokenised. I guess, it should look like this;

```
from transformers import pipeline, AutoModel
model_name = "TestModel"

model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer, return_all_scores=True)

```

My question is where do the other steps from th tokenisation process take place, like the padding and truncation. During training, my sequences where processed as follows;

```
train_encodings = tokenizer(seq_train, truncation=True, padding=True,
          max_length=1024, return_tensors="pt")

```

Is that no longer needed?

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](https://discuss.huggingface.co/u/david-waterworth)\
**Post date:** [March 8, 2021, 11:01pm UTC](https://discuss.huggingface.co/t/pipelines-tokenizer-vs-training-tokenizer/4240/2 "2021-03-08T23:01:11Z")

</div>

The `pipeline` does the tokenisation for you, that’s why you have to pass in a trained model and it’s tokeniser. Basically, as I understand it the pipeline implementations simply reduce the amount of code you have to write for common use cases.