# Need help with speech Encoder-Decoder **URL:** \ **Category:** Beginners\ **Created:** [April 15, 2024, 6:04pm UTC](https://discuss.huggingface.co/t/need-help-with-speech-encoder-decoder/81895 "2024-04-15T18:04:49Z")\ **Posts on this page:** 1\ **Page:** 1

**Author:** ![Yousif720](https://avatars.discourse-cdn.com/v4/letter/y/5fc32e/32.png) [@Yousif720](https://discuss.huggingface.co/u/Yousif720)\ **Post date:** [April 15, 2024, 6:04pm UTC](https://discuss.huggingface.co/t/need-help-with-speech-encoder-decoder/81895/1 "2024-04-15T18:04:49Z")

I’ve been trying to get this snippet to run for days but keep running into the same error so I’ve come here for help. I know there exists a Wav2Vec2-Bert model but I’m trying to learn how to finetune/train a model for educational purposes. I’ve tried setting “padding = True” everywhere, tried to batch them, tried setting “dispatch\_batches=False” and “split\_batches=True” but no matter what I do, I always get the tensor error when it tries to compute any metrics. Any help is appreciated! (I’m using transformers-4.39.3, tokenizers-0.15.2, and accelerate-0.29.2) from transformers import AutoTokenizer, Wav2Vec2Processor, SpeechEncoderDecoderModel from datasets import load\_dataset, load\_metric, Dataset from transformers import Trainer, TrainingArguments import torch import numpy as np from dataclasses import dataclass, field from typing import Any, Dict, List, Optional, Union encoder\_id = “facebook/wav2vec2-base-960h” decoder\_id = “google-bert/bert-base-uncased” processor = Wav2Vec2Processor.from\_pretrained(encoder\_id) tokenizer = AutoTokenizer.from\_pretrained(decoder\_id) model = SpeechEncoderDecoderModel.from\_encoder\_decoder\_pretrained(encoder\_id, decoder\_id) model.config.decoder\_start\_token\_id = tokenizer.cls\_token\_id model.config.pad\_token\_id = tokenizer.pad\_token\_id num\_samples = 10 dataset = load\_dataset(“librispeech\_asr”, split=“train.clean.100”, streaming = True, trust\_remote\_code=True) das = list(dataset.take(num\_samples)) ds = Dataset.from\_dict({‘audio’: [da[‘audio’] for da in das], ‘text’: [da[‘text’] for da in das]}) #ds = load\_dataset(“hf-internal-testing/librispeech\_asr\_demo”, “clean”, split=“validation”) wer\_metric = load\_metric(“wer”) def compute\_metrics(pred): pred\_logits = pred.predictions pred\_ids = torch.argmax(pred\_logits, dim=-1) ``` pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id pred_str = processor.batch_decode(pred_ids, group_tokens = False) label_str = processor.batch_decode(pred.label_ids, group_tokens=False) wer = wer_metric.compute(predictions=pred_str, references=label_str) return {"wer": wer} ``` @dataclass class DataCollatorCTCWithPadding: ``` processor: Wav2Vec2Processor padding: Union[bool, str] = True def __call__ (self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]: input_features = [{"input_values": feature["input_values"]} for feature in features] label_features = [{"input_ids": feature["labels"]} for feature in features] batch = self.processor.pad( input_features, padding=self.padding, return_tensors="pt", ) with self.processor.as_target_processor(): labels_batch = self.processor.pad( label_features, padding=self.padding, return_tensors="pt", ) labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100) batch["labels"] = labels return batch ``` data\_collator = DataCollatorCTCWithPadding(processor=processor, padding=True) def prepare\_dataset(batch): audio = batch[“audio”] batch[“input\_values”] = processor(audio[“array”], sampling\_rate=audio[“sampling\_rate”]).input\_values[0] batch[“input\_length”] = len(batch[“input\_values”]) with processor.as\_target\_processor(): batch[“labels”] = processor(batch[“text”]).input\_ids return batch train\_dataset = ds.map(prepare\_dataset, remove\_columns=ds.column\_names) for param in model.encoder.parameters(): param.requires\_grad = False training\_args = TrainingArguments( output\_dir=“YousifTest”, per\_device\_train\_batch\_size=4, split\_batches=True, gradient\_accumulation\_steps=2, evaluation\_strategy=“epoch”, num\_train\_epochs=1, gradient\_checkpointing=True, fp16=True, save\_steps=400, eval\_steps=1, logging\_steps=1, learning\_rate=3e-4, save\_total\_limit=2, max\_steps=24, ) trainer = Trainer( model=model, data\_collator=data\_collator, args=training\_args, compute\_metrics=compute\_metrics, train\_dataset=train\_dataset, eval\_dataset=train\_dataset, ) trainer.train() trainer.save\_model(“Test2”) Traceback (most recent call last): | 0/2 [00:00\