# Wav2Vec2 pretraining feature extraction during preprocessing as welll as training

**URL:** <https://discuss.huggingface.co/t/wav2vec2-pretraining-feature-extraction-during-preprocessing-as-welll-as-training/23818>\
**Category:** 🤗Transformers\
**Created:** [September 30, 2022, 8:02am UTC](https://discuss.huggingface.co/t/wav2vec2-pretraining-feature-extraction-during-preprocessing-as-welll-as-training/23818 "2022-09-30T08:02:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ujan](https://avatars.discourse-cdn.com/v4/letter/u/6de8d8/32.png) [@Ujan](https://discuss.huggingface.co/u/Ujan)\
**Post date:** [September 30, 2022, 8:02am UTC](https://discuss.huggingface.co/t/wav2vec2-pretraining-feature-extraction-during-preprocessing-as-welll-as-training/23818/1 "2022-09-30T08:02:56Z")

</div>

I’m looking at the wav2vec2 pretraining example given in the [transformer repository](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py). This is a preprocessing step :

```auto
with accelerator.main_process_first():
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            num_proc=args.preprocessing_num_workers,
            remove_columns=raw_datasets["train"].column_names,
            cache_file_names=cache_file_names,
        )

```

The prepare\_dataset function is this :

```auto
def prepare_dataset(batch):
        sample = batch[args.audio_column_name]

        inputs = feature_extractor(
            sample["array"], sampling_rate=sample["sampling_rate"], max_length=max_length, truncation=True
        )
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(inputs.input_values[0])

        return batch

```

Where `feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(args.model_name_or_path)`

Wav2Vec2FeatureExtractor uses 1-D conv to extract features as is done in wav2vec2. But why is this being done as a preprocessing step? Furthermore during training the Wav2Vec2Model has yet another feature extractor which seemingly does the same thing but on the already extracted features?

```auto
class Wav2Vec2Model(Wav2Vec2PreTrainedModel):
    def __init__ (self, config: Wav2Vec2Config):
        super(). __init__ (config)
        self.config = config
        self.feature_extractor = Wav2Vec2FeatureEncoder(config)
...
...
...
extract_features = self.feature_extractor(input_values)

```

Should’nt there be a single feature extraction step with 1-D conv done only during training? What am I missing?

Edit : Wav2Vec2FeatureExtractor simply uses Wav2Vec2FeatureEncoder. So they are identical.

---

<div class="post-metadata">

**Author:** ![Ujan](https://avatars.discourse-cdn.com/v4/letter/u/6de8d8/32.png) [@Ujan](https://discuss.huggingface.co/u/Ujan)\
**Post date:** [October 1, 2022, 2:59pm UTC](https://discuss.huggingface.co/t/wav2vec2-pretraining-feature-extraction-during-preprocessing-as-welll-as-training/23818/2 "2022-10-01T14:59:45Z")

</div>

I was looking at a deprecated feature extractor class with the same name. This is the feature extractor class being used : [transformers/feature\_extraction\_wav2vec2.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L31)