# Parallelizing huggingface models

**URL:** <https://discuss.huggingface.co/t/parallelizing-huggingface-models/47923>\
**Category:** DeepSpeed\
**Created:** [July 24, 2023, 1:04pm UTC](https://discuss.huggingface.co/t/parallelizing-huggingface-models/47923 "2023-07-24T13:04:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshatkatoch](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@akshatkatoch](https://discuss.huggingface.co/u/akshatkatoch)\
**Post date:** [July 24, 2023, 1:04pm UTC](https://discuss.huggingface.co/t/parallelizing-huggingface-models/47923/1 "2023-07-24T13:04:19Z")

</div>

Hi,

I have 4 gpus each with 24gb of memory and would like to parallelize the gpt neo models (2.7B and 1.3B) and train them on text data I have. I would also like to parallelize and train models like llama that have 7 billion parameters. I am unsure of how to do this in deepspeed since there is a lot of information and not a lot of straightforward implementations. Please let me know of how this can be done.

Thanks