# How to deploy larger model inference on multiple machine with multiple GPU？

**URL:** <https://discuss.huggingface.co/t/how-to-deploy-larger-model-inference-on-multiple-machine-with-multiple-gpu/52067>\
**Category:** 🤗Transformers\
**Created:** [August 25, 2023, 2:59am UTC](https://discuss.huggingface.co/t/how-to-deploy-larger-model-inference-on-multiple-machine-with-multiple-gpu/52067 "2023-08-25T02:59:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![leonard0](https://avatars.discourse-cdn.com/v4/letter/l/b38774/32.png) [@leonard0](https://discuss.huggingface.co/u/leonard0)\
**Post date:** [August 25, 2023, 2:59am UTC](https://discuss.huggingface.co/t/how-to-deploy-larger-model-inference-on-multiple-machine-with-multiple-gpu/52067/1 "2023-08-25T02:59:58Z")

</div>

I want use llama2-70b-hf for infrence， the total model about 133GB， Now I have 4 machines， each have 4 GPU cards， each GPU card has 16GB memory，and 4 machines are connected by IB，  
the question is how to deploy these model？

---

<div class="post-metadata">

**Author:** ![enochlev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/enochlev/32/22267_2.png) [@enochlev](https://discuss.huggingface.co/u/enochlev)\
**Post date:** [December 19, 2023, 7:03pm UTC](https://discuss.huggingface.co/t/how-to-deploy-larger-model-inference-on-multiple-machine-with-multiple-gpu/52067/2 "2023-12-19T19:03:55Z")

</div>

You need a LLM engineer for this.

You wont be able to load 70b especially if all 4 machines are separate…  
I would try some libraries such as [GitHub - InternLM/lmdeploy: LMDeploy is a toolkit for compressing, deploying, and serving LLMs.](https://github.com/InternLM/lmdeploy) which is the fastest one I saw, or [GitHub - lm-sys/FastChat: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.](https://github.com/lm-sys/FastChat/tree/main) which may be easier to deploy as for separate machines at once. You spin up a worker on every machine.  
or with torchserve make sure your quantize the models

I would suggest llama-13b or llama-7b quantized to 8-bit. Keep in mind that you need about 2gb of VRAM for every parallel request