# Could I inference the Encoder-Decoder model without specify "decoder\_input\_ids"?

**URL:** <https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811>\
**Category:** 🤗Transformers\
**Created:** [April 27, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811 "2021-04-27T20:12:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![louis2889184](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@louis2889184](https://discuss.huggingface.co/u/louis2889184)\
**Post date:** [April 27, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811/1 "2021-04-27T20:12:57Z")

</div>

I’m using Encoder-Decoder model to train a translation task, while partial of the data are unlabeled.

For labeled data, I can use the following codes to do the inference and compute the loss,

```auto
# model is composed of EncoderDecoder architecture
# source_data and target_data are processed by tokenizer beforehand
batch = {
    "inputs_idx": source_data["inputs_idx"],
    "attention_mask": source_data["attention_mask"],
    "decoder_input_ids": target_data["inputs_idx"]
    "decoder_attention_mask": target_data["attention_mask"],
    "labels": target_data["inputs_idx"].clone()
}

output = model(**batch)
supervised_loss = output["loss"]

```

Besides the supervised loss, I also want to compute some unlabeled loss over the predicted logits of unlabeled source data, such as,

```auto
batch = {
    "inputs_idx": source_data["inputs_idx"],
    "attention_mask": source_data["attention_mask"],
}

output = model(**batch)

unsupervised_loss = some_loss_func(output["logits"])

```

However, I can not do the inference without specifying “decoder\_input\_ids”, the decoder will produce error about

```auto
You have to specify either input_ids or inputs_embeds

```

So far, I assign `source_data["idx"]` for `decoder_input_ids` to avoid the issue, but I feel like it is incorrect cause it will bring inconsistency in inference between labeled and unlabeled data. So, I am wondering how should I do inference for unlabeled data correctly.

---

<div class="post-metadata">

**Author:** ![yurii](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@yurii](https://discuss.huggingface.co/u/yurii)\
**Post date:** [April 29, 2021, 11:33am UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811/2 "2021-04-29T11:33:01Z")

</div>

Hi  
during inference use `output = model.generate(**batch)` instead of `output = model(**batch)`

Also during training:  
decoder\_input\_ids != target\_data[“inputs\_idx”]  
labels = target\_data[“inputs\_idx”]  
and decoder\_input\_ids = shift\_to\_right(target\_data[“inputs\_idx”]) - this action is performed automatically in library code, so you can simply omit decoder\_input\_ids argument

---

<div class="post-metadata">

**Author:** ![louis2889184](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@louis2889184](https://discuss.huggingface.co/u/louis2889184)\
**Post date:** [April 30, 2021, 6:44pm UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811/3 "2021-04-30T18:44:29Z")

</div>

@yurii , thanks for the reply.

I think I confuse others by using the term “inference.” Here I am doing is to “forward” the model without using `decoder_input_ids` and `labels`, cause I’d like to compute some unsupervised loss on unlabeled data. Plus, I don’t want to break the auto-grad graph, so I think `model.generate()` is not a good choice for my case?

Could you show me where the code snippet or document about automatically doing the shift\_to\_right thing? I could’t find it myself. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![yurii](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@yurii](https://discuss.huggingface.co/u/yurii)\
**Post date:** [April 30, 2021, 7:03pm UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811/4 "2021-04-30T19:03:25Z")

</div>

In case of training conditional model (e.g BartForConditionalGeneration), when decoder\_input\_ids is absent it will be created automatically by right-shift of labels:

> <https://github.com/huggingface/transformers/blob/8d43c71a1ca3ad322cc45008eb66a5611f1e017e/src/transformers/models/bart/modeling_bart.py#L1283-L1287>

In case of training bare model (e.g BartModel), when decoder\_input\_ids is absent it will be created automatically by right-shift of input\_ids:

> <https://github.com/huggingface/transformers/blob/8d43c71a1ca3ad322cc45008eb66a5611f1e017e/src/transformers/models/bart/modeling_bart.py#L1147-L1152>

By the way, try  
`"inputs_ids": source_data["inputs_ids"]`  
instead of  
`"inputs_idx": source_data["inputs_idx"]`

---

<div class="post-metadata">

**Author:** ![louis2889184](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@louis2889184](https://discuss.huggingface.co/u/louis2889184)\
**Post date:** [May 1, 2021, 1:24am UTC](https://discuss.huggingface.co/t/could-i-inference-the-encoder-decoder-model-without-specify-decoder-input-ids/5811/5 "2021-05-01T01:24:57Z")

</div>

I see. I used EncoderDecoderModel before, not the Bart Model, so there is no this feature there. Besides, their behaviors seems different. Now, I’d try to use Bart to see how it goes. I personally think EncoderDecoderModel should also be able to forward without “decoder\_input\_ids” and “labels”. Anyways, thank you for sharing that.