# Embedding evaluation

**URL:** <https://discuss.huggingface.co/t/embedding-evaluation/134706>\
**Category:** Beginners\
**Created:** [January 7, 2025, 8:06pm UTC](https://discuss.huggingface.co/t/embedding-evaluation/134706 "2025-01-07T20:06:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Felix-K404](https://avatars.discourse-cdn.com/v4/letter/f/34f0e0/32.png) [@Felix-K404](https://discuss.huggingface.co/u/Felix-K404)\
**Post date:** [January 7, 2025, 8:06pm UTC](https://discuss.huggingface.co/t/embedding-evaluation/134706/1 "2025-01-07T20:06:59Z")

</div>

How does one even evaluate a fine-tuned model? I don’t want to evaluate it during training to keep things modular and since things take a while. I have been using a Triplet dataset for embedding that basically has a question, positive example and a negative example.  
This is my code for evaluating so far:

```auto
model = SentenceTransformer(model_path, device='cuda')
model.eval()
ds = load_dataset('json', data_files=dataset_path ,split='train')
ds = ds.select(range(1000))

dev_evaluator = TripletEvaluator(
    anchors=ds["question"],
    positives=ds["positive_example"],
    negatives=ds["negative_example"],
    batch_size= 64,
    show_progress_bar= True,
    main_distance_function= SimilarityFunction.COSINE
)
print ("Beginning Evaluation...")
evaluation_score = dev_evaluator(model)
print(evaluation_score["cosine_accuracy"])

```

However there are a few problems with the code. First of all it somehow runs over all the data three times for some reason, and the second and third time are quite a bit slower than the first time.  
Additionally i dont know if its just me but i struggled to even find an example for the main\_distance\_function, or even an example for a simple evaluation like this.  
Any tips what im doing wrong or if i can somehow optimize things further with my GPU ?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 8, 2025, 2:28am UTC](https://discuss.huggingface.co/t/embedding-evaluation/134706/2 "2025-01-08T02:28:39Z")

</div>

Is the batch size too large and the data not fitting into the GPU’s VRAM, causing a slowdown?

---

<div class="post-metadata">

**Author:** ![Felix-K404](https://avatars.discourse-cdn.com/v4/letter/f/34f0e0/32.png) [@Felix-K404](https://discuss.huggingface.co/u/Felix-K404)\
**Post date:** [January 8, 2025, 1:44pm UTC](https://discuss.huggingface.co/t/embedding-evaluation/134706/3 "2025-01-08T13:44:01Z")

</div>

Shouldn’t be the case. The GPU got 12 GB and the data isn’t that large either.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 8, 2025, 2:05pm UTC](https://discuss.huggingface.co/t/embedding-evaluation/134706/4 "2025-01-08T14:05:42Z")

</div>

Hmm. Anyway, there are no documents…😓

> **[Training and Finetuning Embedding Models with Sentence Transformers v3](https://huggingface.co/blog/train-sentence-transformers)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> an example for the main\_distance\_function

Maybe these.

> <https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/util.py#L111>

from [sentence-transformers/sentence\_transformers/similarity\_functions.py at master · UKPLab/sentence-transformers · GitHub](https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/similarity_functions.py)