# Should I use .map(processor) or define tokenizer=processor?

**URL:** <https://discuss.huggingface.co/t/should-i-use-map-processor-or-define-tokenizer-processor/61313>\
**Category:** 🤗Transformers\
**Created:** [November 7, 2023, 3:15am UTC](https://discuss.huggingface.co/t/should-i-use-map-processor-or-define-tokenizer-processor/61313 "2023-11-07T03:15:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Diegulio](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/diegulio/32/20973_2.png) [@Diegulio](https://discuss.huggingface.co/u/Diegulio)\
**Post date:** [November 7, 2023, 3:15am UTC](https://discuss.huggingface.co/t/should-i-use-map-processor-or-define-tokenizer-processor/61313/1 "2023-11-07T03:15:14Z")

</div>

I am working in a vision use case. I have the processor:

```auto
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')

```

Then I process my dataset as follow:

```auto
def apply_processor(example):
  example['pixel_values'] = processor(example['image'].convert("RGB"), return_tensors="pt").pixel_values.squeeze()
  return example

processed_dataset = pet_dataset.map(apply_processor)

```

Considering this, should I **also** add the `tokenizer = processor` in the transformers.Trainer ? If not, which one is the best option, doing the map/transform/etc or doing the tokenizer=processor?

Thanks in advance!