# What does Wav2Vec2Tokenizer do?and what is the difference between it and Wav2Vec2FeatureExtractor?

**URL:** <https://discuss.huggingface.co/t/what-does-wav2vec2tokenizer-do-and-what-is-the-difference-between-it-and-wav2vec2featureextractor/39687>\
**Category:** Beginners\
**Created:** [May 12, 2023, 8:26pm UTC](https://discuss.huggingface.co/t/what-does-wav2vec2tokenizer-do-and-what-is-the-difference-between-it-and-wav2vec2featureextractor/39687 "2023-05-12T20:26:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bassel1](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@Bassel1](https://discuss.huggingface.co/u/Bassel1)\
**Post date:** [May 12, 2023, 8:26pm UTC](https://discuss.huggingface.co/t/what-does-wav2vec2tokenizer-do-and-what-is-the-difference-between-it-and-wav2vec2featureextractor/39687/1 "2023-05-12T20:26:05Z")

</div>

I know my question seems basic but I am talking specifically in audio data. I know what tokenization is in text data which dividing the text into tokens(word, characters or subwords).  
I was checking the hugging face documentation [wav2vec](https://huggingface.co/transformers/v4.3.3/model_doc/wav2vec2.html#wav2vec2tokenizer) but I did not understand the tokenization in the context of audio.  
I also used `Wav2Vec2FeatureExtractor` which normalizes the data and I found out that its output is the same as `Wav2Vec2Tokenizer`’s output  
For example:

```auto
from transformers import Wav2Vec2Tokenizer, Wav2Vec2Model
from datasets import load_dataset
import soundfile as sf

tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2Model.from_pretrained("facebook/wav2vec2-base-960h")

def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

input_values = tokenizer(ds["speech"][0], return_tensors="pt").input_values # Batch size 1
hidden_states = model(input_values).last_hidden_state

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
i= feature_extractor(ds["speech"][0], return_tensors="pt", sampling_rate=16000)

```

`i` equals `input_values` in this example.  
What is the difference between `Wav2Vec2FeatureExtractor` and `Wav2Vec2Tokenizer`?