# Manual pipeline parallelization with DeepSpeed

**URL:** <https://discuss.huggingface.co/t/manual-pipeline-parallelization-with-deepspeed/29099>\
**Category:** DeepSpeed\
**Created:** [January 7, 2023, 6:07am UTC](https://discuss.huggingface.co/t/manual-pipeline-parallelization-with-deepspeed/29099 "2023-01-07T06:07:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sheldon-spock](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@sheldon-spock](https://discuss.huggingface.co/u/sheldon-spock)\
**Post date:** [January 7, 2023, 6:07am UTC](https://discuss.huggingface.co/t/manual-pipeline-parallelization-with-deepspeed/29099/1 "2023-01-07T06:07:04Z")

</div>

Hello folks!

I am looking to finetune a pipeline and I have a stack of models (say A, B and C) in the pipeline, one of which (B) is an LLM (GPT-J) that occupies a lot of memory.

I want to split model B on 2 GPUs and add the other models in the pipeline on these GPUs in order (I.e. A and half of B on GPU\_0 ; other half of B on GPU\_1 and C on GPU\_1).

Does DeepSpeed allow manual model parallelization, i.e. can I decide which model goes on what GPU? If not, what would be the best way to achieve pipeline parallelization with a split model?

Thanks!