# A potential in-place operation that caused an RuntimeError

**URL:** <https://discuss.huggingface.co/t/a-potential-in-place-operation-that-caused-an-runtimeerror/3276>\
**Category:** 🤗Transformers\
**Created:** [January 19, 2021, 8:56am UTC](https://discuss.huggingface.co/t/a-potential-in-place-operation-that-caused-an-runtimeerror/3276 "2021-01-19T08:56:02Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Randool](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@Randool](https://discuss.huggingface.co/u/Randool)\
**Post date:** [January 19, 2021, 8:56am UTC](https://discuss.huggingface.co/t/a-potential-in-place-operation-that-caused-an-runtimeerror/3276/1 "2021-01-19T08:56:02Z")

</div>

Hi, I’m using transformers to generate sentences containing gradients using `model.generate` with the modification of removing `@torch.no_grad()` ahead of `def generate(...):`, since the current version (4.2.1) of `model.generate` doesn’t support keeping gradient.

And because I set `do_sample=True` and `num_beams>1` in `generate`, the return type is `BeamSampleEncoderDecoderOutput`. According to the documents, the `scores` of `BeamSampleEncoderDecoderOutput` consists of log softmax scores for each vocabulary token and the sum of log softmax of previously generated tokens in this beam.

What I want to do is gathering the non-inf value from the last beam and applying gradient descent to train the network later. The key pseudo-code is:

```auto
outputs = self.generate(input_ids, ..., **model_kwargs)
# The type of outputs is BeamSampleEncoderDecoderOutput
scores = outputs.scores
last_step_score = scores[-1]
last_step_score = last_step_score[torch.where(last_step_score != -float('inf'))]
last_step_score = last_step_score[::num_beams]

```

However, when I run the program, I receive an error:

```auto
RuntimeError:
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16, 50265]], which is output 0 of LogSoftmaxBackward, is at version 17; expected version 0 instead.
Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

```

which means that there is an in-place operation in `generate`, and I guess the in-place operation lies in `BeamSearchScorer.finalize`, but I can’t figure out what source code to change to make it viable.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 19, 2021, 9:16pm UTC](https://discuss.huggingface.co/t/a-potential-in-place-operation-that-caused-an-runtimeerror/3276/2 "2021-01-19T21:16:24Z")

</div>

cc @patrickvonplaten