# Helsinki-NLP throws StopIteration within DataParallel

**URL:** <https://discuss.huggingface.co/t/helsinki-nlp-throws-stopiteration-within-dataparallel/174223>\
**Category:** Intermediate\
**Created:** [March 12, 2026, 1:17pm UTC](https://discuss.huggingface.co/t/helsinki-nlp-throws-stopiteration-within-dataparallel/174223 "2026-03-12T13:17:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bogolese](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bogolese/32/53040_2.png) [@bogolese](https://discuss.huggingface.co/u/bogolese)\
**Post date:** [March 12, 2026, 1:17pm UTC](https://discuss.huggingface.co/t/helsinki-nlp-throws-stopiteration-within-dataparallel/174223/1 "2026-03-12T13:17:43Z")

</div>

Hoping this is the correct forum for this:

Python 3.12.3  
torch==2.10.0  
transformers=5.3.0

My aim is to use Helsinki-NLP/opus-mt-ru-en and Helsinki-NLP/opus-mt-zh-en with two GPUs in batches, which I’ve been able to do for a couple years until I upgraded to Ubuntu 24, hence Python 3.12.

It now fails with a StopIteration exception within the Module. For instance, at this line in the Module in the forward method:

```auto
outputs = self.model.generate(**inputs)

```

If, at that point any attempt to interrogate self.model.device, which is what happens downstream at line 2502 within GenerateMixin.generate() in transforers/generation/utils.py, the exception is thrown.

Dr. Google recommends converting to using DistributedDataParallel, but that seems to me a good bit of work for a process running on a single machine without any guarantee of sucess.

I may have to resort to hand-jamming threads, which I had to do for the ModernBERT models some time back.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 13, 2026, 1:02am UTC](https://discuss.huggingface.co/t/helsinki-nlp-throws-stopiteration-within-dataparallel/174223/2 "2026-03-13T01:02:38Z")

</div>

In this case, Transformers v5 has triggered the problem becoming apparent, but it might not be the cause itself.🤔

* * *

## Bottom line

I would treat this as a **`DataParallel` + `generate()` integration failure**, not as a Helsinki-NLP-specific regression and not as strong evidence that Python 3.12 or Ubuntu 24 broke Marian. The most likely immediate cause is:

1. `generate()` needs to know the model’s device.
2. In current Transformers, `ModuleUtilsMixin.device` is implemented as `return next(param.device for param in self.parameters())`.
3. PyTorch has public issue history showing that inside `nn.DataParallel` replica `forward` calls, `self.parameters()` can be empty.
4. If those two facts meet, `StopIteration` is the natural outcome. ([GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py "transformers/src/transformers/modeling\_utils.py at main · huggingface/transformers · GitHub"))

## Why your theory makes sense technically

The key detail is the current Transformers implementation of `.device`. It does not do anything fancy there; it simply asks the module for its parameters and takes the device of the first one. If no parameter is yielded, that line fails immediately with `StopIteration`. ([GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py "transformers/src/transformers/modeling\_utils.py at main · huggingface/transformers · GitHub"))

That lines up unusually well with PyTorch’s `DataParallel` behavior. PyTorch issue **#49828** shows a minimal example where, inside `forward`, `list(self.parameters())` becomes `[]` under `nn.DataParallel`. That is not a theoretical concern; it is a public repro on PyTorch’s tracker. So your description — “it blows up when something downstream interrogates `self.model.device` inside `generate()`” — is mechanically very plausible. ([GitHub](https://github.com/pytorch/pytorch/issues/49828 ".parameters() is empty in forward when using DataParallel · Issue #49828 · pytorch/pytorch · GitHub"))

There is also older Transformers issue history showing the same general failure shape: code paths using `next(self.parameters())` under `DataParallel` can raise `StopIteration`. An old XLNet issue shows exactly that pattern in forward execution under `DataParallel`. ([GitHub](https://github.com/huggingface/transformers/issues/3936 "Pytorch 1.5 DataParallel · Issue #3936 · huggingface/transformers · GitHub"))

## Why this is probably not really “a Marian problem”

Everything about the failure points to the wrapper architecture rather than the translation model family.

Public Hugging Face issue history shows several recurring classes of wrapper-related problems around generation:

- `generate()` on a `DataParallel`-wrapped model not being workable in practice. ([GitHub](https://github.com/huggingface/transformers/issues/18974 "'DistributedDataParallel' object has no attribute 'generate'"))
- seq2seq examples breaking under `DataParallel`, with the issue disappearing when restricted to one visible GPU. ([GitHub](https://github.com/huggingface/transformers/issues/22571 "seq2seq examples can't handle DataParallel · Issue #22571 · huggingface/transformers · GitHub"))
- pipeline and other high-level APIs also not reliably supporting a `DataParallel`-wrapped object. ([GitHub](https://github.com/huggingface/transformers/issues/35747 "`pipeline` AttributeError with `torch.nn.DataParallel` · Issue #35747 · huggingface/transformers · GitHub"))

That pattern is much broader than Marian. It says: **generation-oriented Hugging Face code and `DataParallel` have been a rough combination for years.** Your translation models are just another place where that rough edge is surfacing. ([GitHub](https://github.com/huggingface/transformers/issues/18974 "'DistributedDataParallel' object has no attribute 'generate'"))

## Why it may have worked for years and then failed after your upgrade

That is believable, and it still does not point to Python 3.12 as the root cause.

`DataParallel` is one of those APIs where small changes in PyTorch internals, wrapper behavior, generation logic, or device-resolution timing can change whether a latent bug becomes visible. PyTorch still documents `DataParallel` as a replicated-per-forward, single-process, multi-thread abstraction, and still recommends `DistributedDataParallel` instead. Hugging Face has also continued evolving generation internals in v5 and around that period more generally, so a formerly lucky code path can become unlucky without the model itself changing. ([PyTorch Documentation](https://docs.pytorch.org/docs/stable/generated/torch.nn.DataParallel.html "DataParallel — PyTorch 2.10 documentation"))

So my reading is:

- your **upgrade likely exposed** the problem,
- but the **design weakness was already there**.

## Why `DataParallel` is the wrong fit for your exact workload

Your workload is unusually clear:

- one machine,
- two GPUs,
- inference only,
- models that fit comfortably on a single GPU,
- goal is **throughput** on batched translation.

That is almost the textbook case for **one process per GPU with one ordinary model replica per process**.

PyTorch’s own comparison explains why:

- `DataParallel` is **single-process, multi-threaded** ,
- `DistributedDataParallel` is **multi-process** ,
- `DataParallel` pays thread/GIL overhead,
- it also pays per-iteration replication overhead,
- and scattering/gathering adds more overhead,
- so DDP is usually faster even on a single machine. ([PyTorch Documentation](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html "Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.11.0+cu130 documentation"))

That point matters even more for `generate()`. Translation generation is not just “run one forward once.” It is an iterative control loop that keeps checking model state, generation configuration, and device placement. That is exactly the kind of path where wrapper abstractions tend to leak. ([GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py "transformers/src/transformers/modeling\_utils.py at main · huggingface/transformers · GitHub"))

## Should you switch to DDP?

Directionally, yes. Practically, the cleaner answer is:

**Use the DDP architecture idea, but not necessarily the DDP wrapper.**

For pure inference, the important part is not gradient synchronization. The important part is:

- one process owns GPU 0,
- one process owns GPU 1,
- each process loads a normal unwrapped model,
- each process translates its shard of the batch,
- you merge outputs back in original order.

That gets you the benefit PyTorch wants you to have — multi-process, one replica per GPU — without forcing you into training-style distributed ceremony you do not need. PyTorch’s DDP tutorial and Hugging Face Accelerate’s distributed-inference guide both point in that direction. Accelerate specifically documents `split_between_processes()` for exactly this sort of multi-GPU inference sharding. ([PyTorch Documentation](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html "Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.11.0+cu130 documentation"))

## Why I would not “hand-jam threads”

I would hand-jam **processes** , not threads.

Threads are the wrong escape hatch here because `DataParallel` is already the thread-based solution. PyTorch’s own comparison explicitly frames `DataParallel` as single-process, multi-threaded, with GIL contention and replication overhead, while the preferred alternative is multi-process. If you replace `DataParallel` with your own thread orchestration, you are staying on the same side of the architectural boundary that PyTorch is already telling you to leave. ([PyTorch Documentation](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html "Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.11.0+cu130 documentation"))

So if you want a robust “I control it myself” implementation, the right homemade version is:

- worker process on GPU 0,
- worker process on GPU 1,
- persistent model instance per worker,
- input queue,
- output queue,
- batch sharding by index.

That is much closer to the officially recommended direction than a thread pool.

## A subtle but important caveat about DDP wrappers

There is one thing Dr. Google often leaves out: **wrapping the model in `DistributedDataParallel` and then calling `model.generate(...)` on the wrapper is not always smooth either**. Hugging Face has a public issue where a DDP-wrapped model raised `AttributeError: 'DistributedDataParallel' object has no attribute 'generate'`. ([GitHub](https://github.com/huggingface/transformers/issues/18974 "AttributeError: 'DistributedDataParallel' object has no attribute 'generate' · Issue #18974 · huggingface/transformers · GitHub"))

That is why my real recommendation is not “blindly wrap in DDP.” It is:

- adopt the **process-per-GPU design** ,
- keep each process’s model as a **plain model object** for inference,
- call normal `model.generate(...)` inside that process,
- use a launcher or helper library only for process orchestration.

For inference, that is usually simpler and more reliable than leaning on wrapper methods.

## What I would do in your exact setup

For `Helsinki-NLP/opus-mt-ru-en` and `Helsinki-NLP/opus-mt-zh-en`, on a 2-GPU single machine, I would do this:

1. **Drop `nn.DataParallel` entirely** for inference.
2. Keep one ordinary model replica per GPU.
3. Split each incoming batch into two shards.
4. Run `generate()` independently in each process.
5. Reassemble outputs in input order.
6. Keep models loaded persistently so you do not repay load cost on every request. ([PyTorch Documentation](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html "Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.11.0+cu130 documentation"))

If you want the least boilerplate, use **Accelerate distributed inference**. If you want maximum explicit control and easy debugging, use plain Python multiprocessing or `torchrun` with one process per GPU. Either route is more appropriate than `DataParallel` for this workload. ([Hugging Face](https://huggingface.co/docs/accelerate/usage_guides/distributed_inference "Distributed inference"))

## What I would not do

I would not:

- spend much more time trying to patch around `self.model.device` under `DataParallel`,
- assume downgrading Python will fix it,
- assume downgrading Transformers will fix it,
- rely on `model.module.generate(...)` as the long-term answer,
- replace `DataParallel` with a custom thread-based design. ([PyTorch Documentation](https://docs.pytorch.org/docs/stable/generated/torch.nn.DataParallel.html "DataParallel — PyTorch 2.10 documentation"))

The reason is simple: all of those keep you too close to the wrapper behavior that is already failing.

## My actual diagnosis, stated plainly

If I had to summarize your case in one sentence:

> `generate()` is touching model device state; Transformers currently resolves `.device` by iterating parameters; `DataParallel` replicas can expose no parameters inside `forward`; so `StopIteration` under `DataParallel` is a believable architectural failure, not an especially convincing sign that Marian or Python 3.12 is broken. ([GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py "transformers/src/transformers/modeling\_utils.py at main · huggingface/transformers · GitHub"))

## Recommendation

For your case, I would make this call:

- keep the Marian/SentencePiece fix separate,
- stop using `nn.DataParallel` for translation inference,
- move to **one process per GPU** ,
- use **plain unwrapped models** in those processes,
- shard batches explicitly,
- use **Accelerate** or plain multiprocessing depending on how much control you want. ([Hugging Face](https://huggingface.co/docs/accelerate/usage_guides/distributed_inference "Distributed inference"))

This also fits the broader current state of the Transformers stack and the separate Marian/backend diagnosis already established here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [March 15, 2026, 10:35am UTC](https://discuss.huggingface.co/t/helsinki-nlp-throws-stopiteration-within-dataparallel/174223/3 "2026-03-15T10:35:54Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.