# Abnormal generation after multi GPU

**URL:** <https://discuss.huggingface.co/t/abnormal-generation-after-multi-gpu/173050>\
**Category:** 🤗Transformers\
**Created:** [February 3, 2026, 6:49am UTC](https://discuss.huggingface.co/t/abnormal-generation-after-multi-gpu/173050 "2026-02-03T06:49:32Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 3, 2026, 12:39pm UTC](https://discuss.huggingface.co/t/abnormal-generation-after-multi-gpu/173050/4 "2026-02-03T12:39:28Z")

</div>

If relatively easy case:

* * *

## Background: what you’re doing vs other “multi-GPU” modes

Your code uses **model sharding** via a `device_map` (weights split across multiple GPUs inside **one Python process** ). That is different from **distributed inference** (many processes, usually one per GPU, splitting prompts/batches). Accelerate documents these as different approaches: _device\_map / big-model inference_ vs _split prompts across processes_. ([Hugging Face](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling "Big Model Inference"))

This distinction matters because a common failure mode is mixing “sharded model” with “multi-process launch” incorrectly.

* * *

## What “gibberish generation” usually indicates

Output like repetitive, low-information characters (your screenshot) typically comes from **numerical corruption** during forward/generation (wrong device transfers, broken inter-GPU communication, NaNs/Infs, or a buggy kernel path), not from decoding parameters.

There are many public reports of “single GPU OK, multi-GPU gibberish” with `device_map` sharding. ([GitHub](https://github.com/huggingface/transformers/issues/21720 "Multi-GPU inference using accelerate giving inaccurate/gibberish results on RTX 4090s · Issue #21720 · huggingface/transformers · GitHub"))

* * *

## The most likely causes in _your_ exact setup (InternVL2.5 + manual `device_map`)

### 1) Launching with `torchrun` / multi-process while also sharding with `device_map`

In a well-known Hugging Face thread showing the same symptom, the model is sharded with `device_map="auto"` and launched with `torchrun`; a Hugging Face maintainer states **this cannot be run with `torchrun`** in that configuration. ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))

**Fix**

- Run single-process:

- If using `accelerate`, ensure it’s one process:

If you want multi-process throughput, do **not** shard with `device_map`; instead replicate the model per process.

* * *

### 2) Transformers version mismatch (InternVL2.5 explicitly requires a minimum)

InternVL2.5’s model card explicitly says: **“Please use transformers\>=4.37.2 to ensure the model works normally.”** ([Hugging Face](https://huggingface.co/OpenGVLab/InternVL2_5-8B "OpenGVLab/InternVL2\_5-8B · Hugging Face"))

**Fix**

```bash
pip install -U "transformers>=4.37.2" accelerate

```

* * *

### 3) Inter-GPU transport problems (PCIe ACS / P2P / NCCL)

This is the _most_ common root cause when:

- single GPU is fine,
- multi-GPU “runs” but produces nonsense.

In the same Hugging Face “gibberish” thread, the original poster later reports the issue was **NCCL** , fixed by **deactivating ACS** because it interfered with GPU communication. ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))

NVIDIA’s NCCL docs include explicit instructions for disabling ACS (via `setpci`) when it breaks P2P/GPU Direct behavior. ([NVIDIA Docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html "Troubleshooting — NCCL 2.29.1 documentation"))  
NVIDIA also notes that **P2P not being functional is usually tied to ACS being enabled** (and gives BIOS/kernel mitigation suggestions). ([GitHub](https://github.com/NVIDIA/nccl/issues/631 "Question about nccl p2p disable · Issue #631 · NVIDIA/nccl"))

**Fast diagnostic**  
Run once with P2P disabled:

```bash
NCCL_P2P_DISABLE=1 python your_script.py

```

- If the output becomes normal → you’ve almost certainly hit a P2P/ACS/IOMMU/topology issue.
- Next step is to follow your platform’s recommended way to disable ACS/IOMMU (often BIOS + kernel params) or use NCCL’s documented ACS procedure. ([NVIDIA Docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html "Troubleshooting — NCCL 2.29.1 documentation"))

* * *

### 4) FlashAttention path issues (`use_flash_attn=True`)

InternVL examples enable `use_flash_attn=True`, but if your FlashAttention build / CUDA / driver stack is off, it can lead to numerical instability that looks like garbage output.

**Fix / isolation test**  
Load with FlashAttention off:

```python
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=False, # test
    trust_remote_code=True,
    device_map=device_map
).eval()

```

If this fixes it, keep FlashAttention off until you align `flash-attn` + CUDA + driver versions.

* * *

### 5) Your `split_model()` can create invalid layer keys (important to harden)

InternVL’s published `split_model()` (the one you copied) does **not** stop when `layer_cnt == num_layers`; with enough GPUs it can assign non-existent layers (e.g., `layers.32`, `layers.33` for a 32-layer model). The official snippet shows the same loop structure. ([Hugging Face](https://huggingface.co/OpenGVLab/InternVL2_5-8B "OpenGVLab/InternVL2\_5-8B · Hugging Face"))

Depending on library versions, that can be harmless or can cause subtle dispatch issues.

**Fix: make the mapping bounded**

```python
def split_model_safe(model_name):
    device_map = {}
    world_size = torch.cuda.device_count()
    num_layers = {'InternVL2_5-8B': 32}[model_name]

    num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))
    plan = [num_layers_per_gpu] * world_size
    plan[0] = math.ceil(plan[0] * 0.5)

    layer_cnt = 0
    for i, n in enumerate(plan):
        for _ in range(n):
            if layer_cnt >= num_layers:
                break
            device_map[f'language_model.model.layers.{layer_cnt}'] = i
            layer_cnt += 1
        if layer_cnt >= num_layers:
            break

    # Keep entry/exit + vision on GPU0 (matches InternVL rationale)
    device_map['vision_model'] = 0
    device_map['mlp1'] = 0
    device_map['language_model.model.tok_embeddings'] = 0
    device_map['language_model.model.embed_tokens'] = 0
    device_map['language_model.model.rotary_emb'] = 0
    device_map['language_model.model.norm'] = 0
    device_map['language_model.lm_head'] = 0
    device_map['language_model.output'] = 0
    device_map[f'language_model.model.layers.{num_layers - 1}'] = 0
    return device_map

```

* * *

## Concrete debugging checklist (quick → deep)

### A) Confirm you’re in the “supported” config

1. **Single-process run** (no `torchrun` with multiple procs). ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))

2. **Transformers \>= 4.37.2**. ([Hugging Face](https://huggingface.co/OpenGVLab/InternVL2_5-8B "OpenGVLab/InternVL2\_5-8B · Hugging Face"))

3. Print device map after load:

### B) Isolate kernel vs comms vs mapping

Run these toggles **one at a time** :

1. **Disable FlashAttention** (`use_flash_attn=False`)

2. **Disable NCCL P2P**

3. Use `split_model_safe()` (bounded layers)

### C) If P2P disable fixes it

You’re in the “ACS/P2P topology” bucket.

- Follow NVIDIA NCCL troubleshooting guidance for checking/disabling ACS. ([NVIDIA Docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html "Troubleshooting — NCCL 2.29.1 documentation"))
- Consider running NCCL performance/tests; the HF thread explicitly recommends NCCL tests for diagnosing interconnect problems. ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))

* * *

## Why this fits your symptom better than “prompt / decoding”

- Your generation is deterministic (`do_sample=False`), and the prompt is simple.

- Similar “gibberish” reports happen even with plain text-only LLMs when sharded across GPUs. ([GitHub](https://github.com/huggingface/transformers/issues/21720 "Multi-GPU inference using accelerate giving inaccurate/gibberish results on RTX 4090s · Issue #21720 · huggingface/transformers · GitHub"))

- The strongest real-world fixes reported are:

* * *

## If you want the fastest “most likely fix”

1. Upgrade Transformers to `>=4.37.2`. ([Hugging Face](https://huggingface.co/OpenGVLab/InternVL2_5-8B "OpenGVLab/InternVL2\_5-8B · Hugging Face"))
2. Ensure you’re running **one process** (plain `python`, not `torchrun`). ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))
3. Try `NCCL_P2P_DISABLE=1`. If it fixes output, pursue ACS/P2P remediation per NCCL docs. ([Hugging Face Forums](https://discuss.huggingface.co/t/multi-gpu-inference-with-llm-produces-gibberish/35904 "Multi-GPU inference with LLM produces gibberish - Transformers - Hugging Face Forums"))
4. If not, disable FlashAttention and use the bounded `split_model_safe()`.

---

_[View the full topic](https://discuss.huggingface.co/t/abnormal-generation-after-multi-gpu/173050)._