# Use custom text encoder in CLIP

**URL:** <https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090>\
**Category:** 🤗Transformers\
**Created:** [January 22, 2024, 1:21pm UTC](https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090 "2024-01-22T13:21:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ThreeBlessings](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@ThreeBlessings](https://discuss.huggingface.co/u/ThreeBlessings)\
**Post date:** [January 22, 2024, 1:21pm UTC](https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090/1 "2024-01-22T13:21:17Z")

</div>

Hi guys!  
I want to train a CLIP model for Ukrainian and would like to replace a text encoder with pretrained Ukrainian text encoder. I found a couple of discussions about CLIP implemented in Flax for the [Spanish](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084) and [Korean](https://discuss.huggingface.co/t/koclip-pretraining-clip-on-korean/7280) languages and a research papers about replacing CLIP text encoder for other languages (AltCLIP), but they all kinda reimplement CLIP for that.

Is there an easy way to initialize a CLIPModel but with custom text encoder (available on hf hub)? Same question for replacing the tokenizer in the CLIPProcessor.

I would like something like this:

```auto
model = CLIPModel(text_encoder=my_text_encoder, image_encoder=my_image_encoder)
processor = CLIPProcessor(tokenizer=my_tokenizer, image_processor=my_image_processor)

```

---

<div class="post-metadata">

**Author:** ![syrineM](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@syrineM](https://discuss.huggingface.co/u/syrineM)\
**Post date:** [January 22, 2024, 1:57pm UTC](https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090/2 "2024-01-22T13:57:29Z")

</div>

Hi, the solution i found is to use VisionTextDualEncoder class rather than CLIPModel class of transformers library where you can customize the vision and text encoders : " This model can be used to align the vision-text embeddings using CLIP like contrastive image-text training and then can be used for zero-shot vision tasks such image-classification or retrieval."  
Not sure all text encoders architectures are compatible but worth a try !

---

<div class="post-metadata">

**Author:** ![ThreeBlessings](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@ThreeBlessings](https://discuss.huggingface.co/u/ThreeBlessings)\
**Post date:** [January 22, 2024, 2:07pm UTC](https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090/3 "2024-01-22T14:07:00Z")

</div>

Thank you so much! This looks like exactly what I need! Another thing I didn’t mention was that it’s actually for SigLIP model and as I see `VisionTextDualEncoder` only uses CLIP loss for now. I will create a PR to make it work with SigLIP loss as well 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [January 23, 2024, 2:07am UTC](https://discuss.huggingface.co/t/use-custom-text-encoder-in-clip/70090/4 "2024-01-23T02:07:04Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.