# Converting PyTorch model to TorchScript, ValueError: You have to specify either decoder\_input\_ids or decoder\_inputs\_embeds

**URL:** <https://discuss.huggingface.co/t/converting-pytorch-model-to-torchscript-valueerror-you-have-to-specify-either-decoder-input-ids-or-decoder-inputs-embeds/43050>\
**Category:** Beginners\
**Created:** [June 13, 2023, 9:28am UTC](https://discuss.huggingface.co/t/converting-pytorch-model-to-torchscript-valueerror-you-have-to-specify-either-decoder-input-ids-or-decoder-inputs-embeds/43050 "2023-06-13T09:28:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![VIArchitect](https://avatars.discourse-cdn.com/v4/letter/v/f04885/32.png) [@VIArchitect](https://discuss.huggingface.co/u/VIArchitect)\
**Post date:** [June 13, 2023, 9:28am UTC](https://discuss.huggingface.co/t/converting-pytorch-model-to-torchscript-valueerror-you-have-to-specify-either-decoder-input-ids-or-decoder-inputs-embeds/43050/1 "2023-06-13T09:28:29Z")

</div>

I’m converting the SwitchTransformer model from HuggingFace to TorchScript.

But I encountered the following error message

`ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds`.

In some other QnAs in StackOverflow, it said that Enc-Dec model cannot be converted at once and it should be divided into encoder and decoder, then it can be converted, but I still cannot understand. The model itself is integrated at the function level.

Does that mean that I should divide it making a new function?

Here are the two links that I referred to.

- [python - ValueError: You have to specify either decoder\_input\_ids or decoder\_inputs\_embeds - Stack Overflow](https://stackoverflow.com/questions/65140400/valueerror-you-have-to-specify-either-decoder-input-ids-or-decoder-inputs-embed)
- [python - how to convert HuggingFace's Seq2seq models to onnx format - Stack Overflow](https://stackoverflow.com/questions/66109084/how-to-convert-huggingfaces-seq2seq-models-to-onnx-format/66117248#66117248)

Also, in the answer in the above link, someone said that the converted model doesn’t have `generate` function, then how can I make an inference?

Here’s the code that I’m using now.

```auto
from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
from transformers import AutoTokenizer, SwitchTransformersConfig
import torch

torch.set_printoptions(threshold=10_000)

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "google/switch-base-8", resume_download=True, torchscript=True)

input_text = "A <extra_id_0> walks into a bar a orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(0)
# To use GPU, add .to(0)
input_ids = tokenizer(input_text, return_tensors="pt", padding=True).input_ids.to(0)

# To use GPU, add args device_map="auto"
model = SwitchTransformersForConditionalGeneration.from_pretrained(
    "google/switch-base-8",
    device_map="auto",
    resume_download=True, torch_dtype=torch.bfloat16,
    torchscript = True,
    )

# This is for TorchScript
model.eval()

model = torch.jit.trace(model, (input_ids))

outputs = model.generate(
    input_ids,
    decoder_start_token_id=0,
    bos_token_id=2,
    )

```