# Data Parallel Multi GPU Inference **URL:** \ **Category:** 🤗Accelerate\ **Created:** [May 26, 2023, 10:58am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069 "2023-05-26T10:58:45Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [May 26, 2023, 10:58am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/1 "2023-05-26T10:58:45Z")
Found the following statement: `You don’t need to prepare a model if it is used only for inference without any kind of mixed precision` in `accelerate.Accelerator.prepare()` documentation: [Accelerator](https://huggingface.co/docs/accelerate/v0.19.0/en/package_reference/accelerator#accelerate.Accelerator.prepare) In data-parallel multi-gpu inference, we want a model copy to reside on each GPU. How can we achieve that without passing the model through `prepare()` ? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [May 26, 2023, 12:27pm UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/2 "2023-05-26T12:27:19Z")
> [@varadhbhatnagar](#): > > In data-parallel multi-gpu inference, we want a model copy to reside on each GPU. How can we achieve that without passing the model through `prepare()` ? You just move the model to the device. Check out the new distributed inference tutorial, and install accelerate from dev to make use of the new API if you want to do `split_by_processes`. Otherwise pass your `dataloader` to Accelerator.prepare and do `model.to(state.device)`: > **[Distributed Inference with 🤗 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/distributed_inference)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Using the DDP wrapper on your model is only relevant when you want to update the gradients (that’s what it’s designed there for), so inference just load the model on the device normally ---
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [May 29, 2023, 5:54am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/3 "2023-05-29T05:54:18Z")
Thanks @muellerzr for your reply. Is there any benefit in using `split_between_processes()` over `accelerate.Accelerator().prepare()` on a `dataloader`? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [May 29, 2023, 5:30pm UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/4 "2023-05-29T17:30:30Z")
It’s useful if you don’t want to make a `DataLoader`, or have things that can’t go in there easily (like prompts in that example) ---
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [May 30, 2023, 9:29am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/5 "2023-05-30T09:29:19Z")
Understood, thanks @muellerzr ! ---
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [September 6, 2023, 6:45am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/6 "2023-09-06T06:45:14Z")
Hi @muellerzr, is there a way I can do distributed inference using model sharding (FSDP) ? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [September 6, 2023, 1:48pm UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/7 "2023-09-06T13:48:30Z")
> [@varadhbhatnagar](#): > > distributed inference using model sharding (FSDP) ? Is there a reason you want to do so instead of using `device_map`/big model inference? This can help narrow down my recommendation ---
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [September 7, 2023, 5:59am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/8 "2023-09-07T05:59:22Z")
@muellerzr My model size is very close to the total GPU memory and from what I understood in [this](https://huggingface.co/docs/accelerate/usage_guides/big_modeling) article, I cannot run batches in parallel on all GPUs if I use `device_map="auto"`. I was wondering if it’s possible to do inference in FSDP style, i.e. the model layers get sharded across all GPUs and layers get exchanged on demand so that I can process batches in parallel ? Is there an end to end example (in the Example Zoo) that I can refer to? ---
**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\ **Post date:** [September 11, 2023, 3:45am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/9 "2023-09-11T03:45:04Z")
@sgugger @muellerzr Any thoughts on this? ---
**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](https://discuss.huggingface.co/u/smangrul)\ **Post date:** [September 15, 2023, 9:58am UTC](https://discuss.huggingface.co/t/data-parallel-multi-gpu-inference/41069/10 "2023-09-15T09:58:23Z")
Hello @varadhbhatnagar, you can use FSDP for distributed inference as long as you aren’t using the `generate` method as FSDP is incompatible with `generate` (mentioned in the docs here: [Fully Sharded Data Parallel (huggingface.co)](https://huggingface.co/docs/accelerate/usage_guides/fsdp#a-few-caveats-to-be-aware-of)). For example, the docs here show it [Fully Sharded Data Parallel (huggingface.co)](https://huggingface.co/docs/accelerate/usage_guides/fsdp#how-it-works-out-of-the-box). The example `nlp_example.py` also computes metrics on eval set which should mimic the distributed inference.