# Jointly train two-stage models using Trainer

**URL:** <https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020>\
**Category:** 🤗Transformers\
**Created:** [October 23, 2021, 4:15pm UTC](https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020 "2021-10-23T16:15:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![SodaSu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sodasu/32/4874_2.png) [@SodaSu](https://discuss.huggingface.co/u/SodaSu)\
**Post date:** [October 23, 2021, 4:15pm UTC](https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020/1 "2021-10-23T16:15:52Z")

</div>

Hi all,

I want to train a two-stage model containing model1 and model2, in which model2 takes model1’s result as input. For now, I could train model2 properly through Huggingface Seq2SeqTrainer, but I have no clue of how to jointly train model1 and model2 through the Huggingface Trainer. Could someone give me some advice? Thank you very much

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 25, 2021, 12:13pm UTC](https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020/2 "2021-10-25T12:13:19Z")

</div>

I don’t think this is possible. For suc a use case, you should definitely check out [Accelerate](https://github.com/huggingface/accelerate) and use your custom training loop.

---

<div class="post-metadata">

**Author:** ![badranx](https://avatars.discourse-cdn.com/v4/letter/b/d6d6ee/32.png) [@badranx](https://discuss.huggingface.co/u/badranx)\
**Post date:** [January 12, 2023, 7:13pm UTC](https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020/3 "2023-01-12T19:13:53Z")

</div>

Shouldn’t a custom model work with the trainer? a simple model that inherits PreTrainedModel and contains the two models with a custom forward method.

I’m new and researching the possibility of doing customized models within the Huggingface ecosystem. Am I wasting my time?

Example:  
[https://stackoverflow.com/questions/70814490/uploading-models-with-custom-forward-functions-to-the-huggingface-model-hub](https://stack_overflow_question)

---

<div class="post-metadata">

**Author:** ![shamanez](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shamanez/32/17994_2.png) [@shamanez](https://discuss.huggingface.co/u/shamanez)\
**Post date:** [March 2, 2023, 11:48am UTC](https://discuss.huggingface.co/t/jointly-train-two-stage-models-using-trainer/11020/4 "2023-03-02T11:48:02Z")

</div>

I have a similar case as well—a model with two towers. Basically, two models were initialized with the from\_pretrained method.

I can confirm that the trainer fails when using multiple GPUs.

If we want to do that, we need to follow the CLIP implementation, where re-inventing the wheel.

@sgugger So, do you suggest doing this stuff with the accelerator?