# Load a large model to multipe, specific GPUs (without CUDA\_VISIBLE\_DEVICES)

**URL:** <https://discuss.huggingface.co/t/load-a-large-model-to-multipe-specific-gpus-without-cuda-visible-devices/126768>\
**Category:** 🤗Transformers\
**Created:** [November 22, 2024, 11:34am UTC](https://discuss.huggingface.co/t/load-a-large-model-to-multipe-specific-gpus-without-cuda-visible-devices/126768 "2024-11-22T11:34:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maximal](https://avatars.discourse-cdn.com/v4/letter/m/ccd318/32.png) [@Maximal](https://discuss.huggingface.co/u/Maximal)\
**Post date:** [November 22, 2024, 11:34am UTC](https://discuss.huggingface.co/t/load-a-large-model-to-multipe-specific-gpus-without-cuda-visible-devices/126768/1 "2024-11-22T11:34:59Z")

</div>

Hi,

So I need to load multiple large models in a single script and control which GPUs they are kept on. For example, lets say I want to load one LLM on the first 4 GPUs and the another LLM on the last 4 GPUs. If I pass “auto” to the device\_map, it will always use all GPUs. I cannot use CUDA\_VISIBLE\_DEVICES since I need all of them to be visible in the script.

For example, what would be the correct arguemnt to device map to load LLama 3.1 on GPUs 0,1,2,3?

```auto
    pipeline = transformers.pipeline(
        "text-generation",
        model="meta-llama/Llama-3.1-70B-Instruct",
        model_kwargs={"torch_dtype": torch.bfloat16},
        device_map=llm_device,
        token=ACCESS_TOKEN,
    )

```