# Model Parallism

**URL:** <https://discuss.huggingface.co/t/model-parallism/82911>\
**Category:** DeepSpeed\
**Created:** [April 21, 2024, 1:02pm UTC](https://discuss.huggingface.co/t/model-parallism/82911 "2024-04-21T13:02:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pkr7098](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pkr7098/32/15553_2.png) [@pkr7098](https://discuss.huggingface.co/u/pkr7098)\
**Post date:** [April 21, 2024, 1:02pm UTC](https://discuss.huggingface.co/t/model-parallism/82911/1 "2024-04-21T13:02:28Z")

</div>

Hi, I want pre-train LLaMA-7B on my multi-gpu system. However, because of the huge model, I face OOM. I already read several method in this forum and HuggingFace documents, but they use from\_pretrained which cannot be used in my experiment since I have to training from scratch.

What I want is to split model (not DDP or DP) and allocate each layers in different GPU, i.e., Model Parallism. Is there any way or solution to resolve this problem? Thank you.