# How to do model.generate() in evaluation steps with Trainer + FSDP?

**URL:** <https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141>\
**Category:** 🤗Transformers\
**Created:** [March 30, 2023, 9:59pm UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141 "2023-03-30T21:59:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![weiqis](https://avatars.discourse-cdn.com/v4/letter/w/898d66/32.png) [@weiqis](https://discuss.huggingface.co/u/weiqis)\
**Post date:** [March 30, 2023, 9:59pm UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141/1 "2023-03-30T21:59:39Z")

</div>

Hi,

I’m training a large GPT2 based causal language model on multiple GPUs using pytorch’s FullyShardedDataParallel (FSDP) strategy. I enabled FSDP in HuggingFace `Trainer` by passing  
the following arguments:

```auto
    "fsdp": "full_shard auto_wrap"
    "fsdp_config": {
        "fsdp_transformer_layer_cls_to_wrap": ["GPT2Block"]
    }

```

With FSDP, the model can be distributed into multiple GPUs with shards and it is successfully trained. Now I want to add an evaluation step to the trainer. I don’t just want to compute the perplexity or accuracy score by getting the argmax of each logit. I want to do an end-to-end evaluation by calling the model’s [`generate` method](https://huggingface.co/docs/transformers/v4.27.2/en/main_classes/text_generation#transformers.GenerationMixin.generate) and generate outputs autoregressively. I couldn’t figure out a way to call `model.generate`, or equivalent methods in the evaluation step. Below are what I have tried.

To do this custom evaluation, I subclassed the `Trainer` class:

```auto
class CustomTrainer(Trainer):
    def evaluate(
            self,
            eval_dataset = None,
            ignore_keys = None,
            metric_key_prefix: str = "eval",
        ):

        # only take one example for illustration
        input_ids = torch.tensor([self.eval_dataset[0]['input_ids']]).to(f"cuda:{self.args.local_rank}")
        output = self.model.generate(input_ids)
        return {"my_fancy_metric": 1.0}

```

If I don’t have the `.to(f"cuda:{self.args.local_rank}")` part, I will get an error message saying:

```auto
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu! (when checking argument for argument index in method wrapper__index_select)

```

This is understandable, since the `input_ids` tensor is on cpu and the models are distributed on different GPUs. But after adding `.to(f"cuda:{self.args.local_rank}")`, I got:

```auto
RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.

```

I also tried calling the `pipeline("text-generation")` with text input but got the same behavior.

So how can I properly call the `model.generate` method in evaluation steps with Trainer and FSDP?

---

<div class="post-metadata">

**Author:** ![luyunan](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@luyunan](https://discuss.huggingface.co/u/luyunan)\
**Post date:** [July 24, 2023, 6:20am UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141/3 "2023-07-24T06:20:31Z")

</div>

Do you have any update on this? I also meet the same problem.

---

<div class="post-metadata">

**Author:** ![mgerstgrasser](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@mgerstgrasser](https://discuss.huggingface.co/u/mgerstgrasser)\
**Post date:** [December 2, 2023, 4:43am UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141/4 "2023-12-02T04:43:30Z")

</div>

I’ve just run into this problem too, in case anyone has figured this out in the mean time? @luyunan @weiqis

---

<div class="post-metadata">

**Author:** ![uygnef](https://avatars.discourse-cdn.com/v4/letter/u/49beb7/32.png) [@uygnef](https://discuss.huggingface.co/u/uygnef)\
**Post date:** [January 12, 2024, 3:49am UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141/5 "2024-01-12T03:49:52Z")

</div>

The model is on CPU。try model.cuda()

---

<div class="post-metadata">

**Author:** ![psr-ai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/psr-ai/32/32607_2.png) [@psr-ai](https://discuss.huggingface.co/u/psr-ai)\
**Post date:** [October 8, 2024, 3:59pm UTC](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141/6 "2024-10-08T15:59:32Z")

</div>

Have you tried FSDP.summon\_full\_params? Make sure you use model.eval before so that the model is able to fit in one GPU for inference.