# Multi-GPU LLM inference data parallelism (llama)

**URL:** <https://discuss.huggingface.co/t/multi-gpu-llm-inference-data-parallelism-llama/57949>\
**Category:** Beginners\
**Created:** [October 9, 2023, 11:16pm UTC](https://discuss.huggingface.co/t/multi-gpu-llm-inference-data-parallelism-llama/57949 "2023-10-09T23:16:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![JerryL](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@JerryL](https://discuss.huggingface.co/u/JerryL)\
**Post date:** [October 9, 2023, 11:16pm UTC](https://discuss.huggingface.co/t/multi-gpu-llm-inference-data-parallelism-llama/57949/1 "2023-10-09T23:16:18Z")

</div>

Hi, I’ve been looking this problem up all day, however, I cannot find a good practice for running multi-GPU LLM inference, information about DP/deepspeed documentation is so outdated.  
I just want to do the most naive data parallelism with Multi-GPU LLM inference (llama).

My code is based on some very basic llama generation code:

```auto
model = AutoModelForCausalLM.from_pretrained(
    llama_model_id,
    config=config,
    torch_dtype=torch.float16,
    load_in_4bit=True,
    device_map='auto',
    use_safetensors=False,
)
tokenizer = AutoTokenizer.from_pretrained(llama_model_id)

model_input = tokenizer(data_processed["input"], return_tensors='pt', add_special_tokens=False, padding=True).to('cuda')
model.eval()
with torch.no_grad():
  output = model.generate(**model_input, max_new_tokens=300,
        temperature=0,
        top_p = 0.95,
        top_k= 5)

```

The machine is with 4\*RTX3090, and I’m trying to do Data Parallelism, where I want data to be sent to 4 GPUs separately and gathered afterward. However, I don’t see how to implement it with `model.generate()` specifically.  
I’m trying to wrap it up with ‘DataParallel’ from torch but get this error `'DataParallel' object has no attribute 'config'`, look like the model from huggingface cannot be warpped up with DataParallel. I also tried with finetuning evaluation mode `Trainer.predict()` and set `predict_with_generate=True`, however, I’m not able to set parameters like `temperature` and the results are different from `model.generate()`

I’m doing the naive way: Spilt my data into 4 splits, for each one, I run it on 1 GPU, and use a for loop to batchinize the data in each split. This is actually much faster than doing model parallelism(device\_map=“auto”).

However, the same model is loaded 4 times from disk and the memory is exploding and it’s extremely inconvenient. Is there a way to implement the simplest DataParallel with huggingface models? Any ideas would be appreciated!

---

<div class="post-metadata">

**Author:** ![sp-ananth](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@sp-ananth](https://discuss.huggingface.co/u/sp-ananth)\
**Post date:** [October 25, 2023, 6:04am UTC](https://discuss.huggingface.co/t/multi-gpu-llm-inference-data-parallelism-llama/57949/2 "2023-10-25T06:04:26Z")

</div>

> [@JerryL](#):
>
> running multi-GPU LLM inference

Perhaps this will help: [LLM Multi-GPU Batch Inference With Accelerate | by Victor May | Medium](https://medium.com/@mayvic/llm-multi-gpu-batch-inference-with-accelerate-edadbef3e239)