# Model \*and\* data parallelism when training on multiple GPUs?

**URL:** <https://discuss.huggingface.co/t/model-and-data-parallelism-when-training-on-multiple-gpus/136835>\
**Category:** Amazon SageMaker\
**Created:** [January 22, 2025, 9:13am UTC](https://discuss.huggingface.co/t/model-and-data-parallelism-when-training-on-multiple-gpus/136835 "2025-01-22T09:13:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicoberk](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nicoberk/32/26983_2.png) [@nicoberk](https://discuss.huggingface.co/u/nicoberk)\
**Post date:** [January 22, 2025, 9:13am UTC](https://discuss.huggingface.co/t/model-and-data-parallelism-when-training-on-multiple-gpus/136835/1 "2025-01-22T09:13:18Z")

</div>

Hi there!

I want to train a model from scratch using a dataset of ~35M online comments. Given the computational costs, I hope to set up the training as efficiently as possible, including parallelization.

Following [this guide](https://huggingface.co/docs/sagemaker/en/train#distributed-training), I see that I can use both model and data parallelism. I understand that if I run training on, let’s say, 4 different GPUs, I’d probably want BOTH parallelized processing of the data and training of the model, correct? However, the guide discusses them as alternative options.

I will try to combine the options described in the guide using

```auto
distribution={
    "smdistributed": {
        "modelparallel": smp_options,
        'dataparallel':{ 'enabled': True }
        },
    "mpi": mpi_options
}

```

However, before I burn large amounts of money, it would be great to get some feedback on whether this makes sense. Any general intuition to think about this would also be greatly appreciated.