# Multi-GPU inference with LLM produces gibberish **URL:** \ **Category:** 🤗Transformers\ **Created:** [April 7, 2023, 4:04pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "2023-04-07T16:04:24Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![LucasWeber](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasweber/32/28128_2.png) [@LucasWeber](https://discuss.huggingface.co/u/LucasWeber)\ **Post date:** [April 7, 2023, 4:04pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/1 "2023-04-07T16:04:25Z")
Hey, I am currently trying to run inference on “huggyllama/llama-7b”. I am using the following minimal script: ```python from transformers import pipeline checkpoint = "huggyllama/llama-7b" p = pipeline("text-generation", checkpoint, torch_dtype=torch.bfloat16, device_map="auto") print(p("hi there")) ``` I run it with torchrun: ```bash NCCL_P2P_DISABLE='1' torchrun --nproc_per_node --master_port=13833 run_llama_hf.py ``` With a single GPU, I get reasonable outputs. **Output single GPU** : ‘hi there, I’m a newbie to this forum and I’m looking for some help’ As soon as I use multiple GPUs, I get: **Output multi GPU** : ‘hi there header driv EUannotuta voor measurements shooting variableslowea grayŌbestįbinding’ **My setup** : 2 x NVIDIA a30 (24g vRAM) ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f9f31d025d5a60aa8c0e131b517e9db313512445.jpeg) - `transformers` version: 4.28.0.dev0 - `accelerate` version: 0.18.0.dev0 - Platform: Linux-4.18.0-305.3.1.el8\_4.x86\_64-x86\_64-with-glibc2.28 - Python version: 3.9.16 - Huggingface\_hub version: 0.13.2 - PyTorch version (GPU?): 2.0.0+cu117 (True) Also: As soon as I run the 13B model, I run out of vRAM (which shouldn’t be the case, if the model was loaded in parallel). Any ideas about what might be the problem appreciated 🤗. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [April 7, 2023, 5:35pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/2 "2023-04-07T17:35:20Z")
This code cannot be run with `torchrun`. `device_map="auto"` will use your two GPUs for the generation ---
**Author:** ![LucasWeber](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasweber/32/28128_2.png) [@LucasWeber](https://discuss.huggingface.co/u/LucasWeber)\ **Post date:** [April 8, 2023, 1:28pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/3 "2023-04-08T13:28:01Z")
Thanks a lot for your reply. I have been using plain `python` and `accelerate launch` before, but with the same gibberish output. EDIT: I don’t know if related, but I had similar issues with native LLaMA on multi-machine runs before (see [Torchrun distributed running does not work · Issue #201 · facebookresearch/llama · GitHub](https://github.com/facebookresearch/llama/issues/201)), which was due to wrong assignment of `LOCAL_RANK` and (global) `RANK` in the original repo. ---
**Author:** ![LucasWeber](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasweber/32/28128_2.png) [@LucasWeber](https://discuss.huggingface.co/u/LucasWeber)\ **Post date:** [May 2, 2023, 2:19pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/4 "2023-05-02T14:19:55Z")
There seems to be some deeper problem (it appears as if it has to do with some interaction of the hardware and the drivers and the latest version of transformers/tokenizers). We got in contact with NVIDIA about this. Since it has only indirectly to do with transformers, this can be closed. ---
**Author:** ![rameshveer](https://avatars.discourse-cdn.com/v4/letter/r/54ee81/32.png) [@rameshveer](https://discuss.huggingface.co/u/rameshveer)\ **Post date:** [May 29, 2023, 2:44pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/5 "2023-05-29T14:44:48Z")
Hi Team, Any updates on this issue… still facing similar gibberish output when used with multiple GPU’s. any idea why this occurs. Thanks, Ramesh. ---
**Author:** ![Alchemy5](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@Alchemy5](https://discuss.huggingface.co/u/Alchemy5)\ **Post date:** [June 27, 2023, 9:43pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/6 "2023-06-27T21:43:42Z")
Running into same issue, help would be appreciated! ---
**Author:** ![LucasWeber](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasweber/32/28128_2.png) [@LucasWeber](https://discuss.huggingface.co/u/LucasWeber)\ **Post date:** [July 4, 2023, 12:34pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/7 "2023-07-04T12:34:01Z")
@Alchemy5 and @rameshveer What type of GPUs do you use? (Out of curiosity, since I am thinking that this is an interplay of transformer/tokenizers and the used GPU hardware). ---
**Author:** ![zxia545](https://avatars.discourse-cdn.com/v4/letter/z/9fc348/32.png) [@zxia545](https://discuss.huggingface.co/u/zxia545)\ **Post date:** [July 5, 2023, 8:21am UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/8 "2023-07-05T08:21:35Z")
Just wondering does anyone found the solution? ---
**Author:** ![Dragon777](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@Dragon777](https://discuss.huggingface.co/u/Dragon777)\ **Post date:** [November 16, 2023, 7:29am UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/9 "2023-11-16T07:29:21Z")
Running into same issue, help would be appreciated! 4 V100 : everything is good 8 V100: output consisted of nonsense charaters: \n\n…\t\tt ---
**Author:** ![LucasWeber](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasweber/32/28128_2.png) [@LucasWeber](https://discuss.huggingface.co/u/LucasWeber)\ **Post date:** [November 17, 2023, 11:19am UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/10 "2023-11-17T11:19:02Z")
For me, it was an issue of NCCL in the end. We had to deactivate ACS on the HPC on which I was working and the problem was resolved (see: [Troubleshooting — NCCL 2.19.3 documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs)). It interfered with the communication between the GPUs. @Dragon777 : Is the general setup somehow different in both cases? If the eight GPUs are on different nodes of your HPC and the 4 GPUs in the first case are not, I could imagine that something is going wrong with the inter-node communication. I think the NCCL performance test is a good tool for diagnosing the problem: [GitHub - NVIDIA/nccl-tests: NCCL Tests](https://github.com/NVIDIA/nccl-tests) ---
**Author:** ![Buggod](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@Buggod](https://discuss.huggingface.co/u/Buggod)\ **Post date:** [March 28, 2024, 7:36pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/11 "2024-03-28T19:36:34Z")
Can any one teach me how to use 2 GPUs to run inference? Accelarator can’t detect my GPUs. ![Screenshot 2024-03-28 at 3.36.25 PM](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/9/49ed35d0e4435f606123c48c5c737db0a2c467fa.png) ---
**Author:** ![capnchat](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/capnchat/32/31430_2.png) [@capnchat](https://discuss.huggingface.co/u/capnchat)\ **Post date:** [March 28, 2024, 8:34pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/12 "2024-03-28T20:34:55Z")
When I was inferencing with falcon-7b and mistral-7b-v0.1, I was getting gibberish until I adjusted my generation\_config as below: generation\_config.repetition\_penalty = 1.2 generation\_config.no\_repeat\_ngram\_size = 2 generation\_config.early\_stopping = True later on we switched to the instruct version of mistral (mistral-7b-instruct-v0.2), and then these settings had to be removed, but perhaps playing with these options will help! Good luck! ---
**Author:** ![Buggod](https://avatars.discourse-cdn.com/v4/letter/b/35a633/32.png) [@Buggod](https://discuss.huggingface.co/u/Buggod)\ **Post date:** [March 28, 2024, 8:54pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/13 "2024-03-28T20:54:26Z")
Thank you so much, this sounds soo complicated. ---
**Author:** ![soerendip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/soerendip/32/14075_2.png) [@soerendip](https://discuss.huggingface.co/u/soerendip)\ **Post date:** [September 27, 2024, 12:17pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/14 "2024-09-27T12:17:17Z")
I am facing this on the a single node with two Nvidia A40’s. It works on a single A100, and using two A100’s produces an out of memory error, strangely. On the two A40’s, I get output like this: “missionaries เพราะ выращиconditionallyेक trình顔を stříฤศจ st”. ```auto text_generator = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=max_tokens, eos_token_id=tokenizer.eos_token_id) ``` ```auto def load_model_and_tokenizer(model_id, quantize, bnb_config): """Load the Hugging Face model and tokenizer.""" HF_TOKEN = os.getenv('HF_TOKEN') # Load tokenizer tokenizer = AutoTokenizer.from_pretrained(model_id, token=HF_TOKEN) #tokenizer.pad_token = tokenizer.eos_token # Clear GPU cache gc.collect() torch.cuda.empty_cache() # Model loading arguments model_kwargs = { 'device_map': "auto", 'token': HF_TOKEN, 'low_cpu_mem_usage': True } if quantize: model_kwargs['quantization_config'] = bnb_config model = AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs) return model, tokenizer ``` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 28, 2024, 4:21am UTC](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904/15 "2024-09-28T04:21:28Z")
> **[huggingchat/chat-ui ¡ \[MODELS\] Discussion](https://huggingface.co/spaces/huggingchat/chat-ui/discussions/372#66e525b32c78909bafb196b6)** > > Here we can discuss about HuggingChat available models. The pattern of huge models producing unintelligible output in a multi-GPU environment seems similar to the symptoms of this post. If it seems to appear in the HF official service, it may or may not be a bug in the HF library itself, but it may be an unresolved issue.