# Evaluating performance before and after fine-tuning

**URL:** <https://discuss.huggingface.co/t/evaluating-performance-before-and-after-fine-tuning/146665>\
**Category:** Beginners\
**Created:** [March 20, 2025, 3:12pm UTC](https://discuss.huggingface.co/t/evaluating-performance-before-and-after-fine-tuning/146665 "2025-03-20T15:12:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![itskavya](https://avatars.discourse-cdn.com/v4/letter/i/6de8d8/32.png) [@itskavya](https://discuss.huggingface.co/u/itskavya)\
**Post date:** [March 20, 2025, 3:12pm UTC](https://discuss.huggingface.co/t/evaluating-performance-before-and-after-fine-tuning/146665/1 "2025-03-20T15:12:23Z")

</div>

Hi, could somebody please guide me on the correct manner in which I should assess the performance of a fine-tuned model such as Whisper? Should I calculate the WER that Whisper gives before fine-tuning on my entire dataset (train+validation+test), and then compare this with the performance on the test set after fine-tuning? I’m asking this because the performance after fine-tuning (Whisper large) gives a high WER, so I would like to compare the performance and check how much it is improving. I would appreciate your input!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 20, 2025, 5:44pm UTC](https://discuss.huggingface.co/t/evaluating-performance-before-and-after-fine-tuning/146665/2 "2025-03-20T17:44:00Z")

</div>

[https://pubs.aip.org/asa/jel/article/4/2/025206/3267247/Evaluating-OpenAI-s-Whisper-ASR-Performance](https://pubs.aip.org/asa/jel/article/4/2/025206/3267247/Evaluating-OpenAI-s-Whisper-ASR-Performance)  
[There seem to be several github repositories and papers on comparing using Whisper’s Evaluate](https://www.bing.com/search?q=Whisper+evaluate+compare).

If you use the Hugging Face library, it would look something like this (I think there are some typos because it’s from Hugging Chat…)

* * *

To evaluate the performance of a fine-tuned Whisper model, follow these steps to calculate the Word Error Rate (WER) before and after fine-tuning:

### Step-by-Step Explanation

1. **Install Necessary Libraries**

2. **Import Libraries**

3. **Load and Prepare Your Dataset**

4. **Define the Whisper Model and Tokenizer**

5. **Fine-Tune the Model**

6. **Calculate WER for Base and Fine-Tuned Models**

7. **Evaluate the Base Model**

8. **Evaluate the Fine-Tuned Model**

9. **Compare the Results**

### Final Answer

Here is the complete code to assess the performance of a fine-tuned Whisper model by calculating the WER before and after fine-tuning:

```python
import os
import evaluate
from datasets import load_dataset, Audio
import whisper
from whisper.model import WhisperForConditionalGeneration
from whisper.tokenizer import WhisperTokenizer
from torch.utils.data import DataLoader
from transformers import TrainingArguments, Trainer

# Load datasets
train_dataset = load_dataset('my_custom_dataset', split='train')
val_dataset = load_dataset('my_custom_dataset', split='validation')
test_dataset = load_dataset('my_custom_dataset', split='test')

# Initialize model and tokenizer
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")
tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-large")

# Set up training arguments
training_args = TrainingArguments(
    output_dir='whisper-finetuned',
    num_train_epochs=30,
    learning_rate=1e-4,
    per_device_train_batch_size=8,
    gradient_checkpointing=True,
)

# Initialize and train the model
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)
trainer.train()

# Function to calculate WER
def calculate_wer(current_model, dataset):
    wer = evaluate.load("wer")
    references = []
    hypotheses = []
    for example in dataset:
        audio = example['audio']['array']
        audio = whisper.pad_or_trim(audio)
        mel = whisper.log_mel_spectrogram(audio).to(device="cuda")
        # Transcribe
        options = whisper.decoding.DecodingOptions()
        result = current_model.decode(mel, options)
        transcription = result.text
        # Get reference
        reference = example['text']
        # Append to lists
        hypotheses.append(transcription)
        references.append(reference)
    # Calculate WER
    wer_score = wer.compute(references=references, hypotheses=hypotheses)
    return wer_score

# Evaluate base model
base_wer = calculate_wer(model, test_dataset)
print(f"Base model WER: {base_wer:.2f}%")

# Evaluate fine-tuned model
fine_tuned_model = WhisperForConditionalGeneration.from_pretrained("whisper-finetuned")
fine_tuned_wer = calculate_wer(fine_tuned_model, test_dataset)
print(f"Fine-tuned model WER: {fine_tuned_wer:.2f}%")

```

This code will guide you through fine-tuning the Whisper model and evaluating its performance using WER, allowing you to compare the improvement achieved through fine-tuning.