# Inference problem after loading a fine tuned T5 model for seq2seq method in question answering

**URL:** <https://discuss.huggingface.co/t/inference-problem-after-loading-a-fine-tuned-t5-model-for-seq2seq-method-in-question-answering/44802>\
**Category:** Beginners\
**Created:** [June 28, 2023, 7:28am UTC](https://discuss.huggingface.co/t/inference-problem-after-loading-a-fine-tuned-t5-model-for-seq2seq-method-in-question-answering/44802 "2023-06-28T07:28:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Venom321](https://avatars.discourse-cdn.com/v4/letter/v/97f17d/32.png) [@Venom321](https://discuss.huggingface.co/u/Venom321)\
**Post date:** [June 28, 2023, 7:28am UTC](https://discuss.huggingface.co/t/inference-problem-after-loading-a-fine-tuned-t5-model-for-seq2seq-method-in-question-answering/44802/1 "2023-06-28T07:28:00Z")

</div>

spiece.model file is not present upon saving the model after training .

I used this

python run\_seq2seq\_qa.py --model\_name\_or\_path t5-small --dataset\_name squad\_v2 --context\_column context --question\_column question --answer\_column answers --do\_train --do\_eval --per\_device\_train\_batch\_size 12 --learning\_rate 3e-5 --num\_train\_epochs 1 --max\_seq\_length 384 --doc\_stride 128 --output\_dir /trained\_model/debug\_seq2seq\_squad/ --max\_train\_samples 50 --max\_eval\_samples 50

to run the run\_seq2seq\_qa.py from transformers git examples but cannot be used for inferencing.

Files saved after training :

generation\_config.json  
config.json  
all\_results.json  
train\_results.json  
special\_tokens\_map.json  
eval\_results.json  
tokenizer.json  
tokenizer\_config.json  
runs  
pytorch\_model.bin  
training\_args.bin  
trainer\_state.json

code used :  
from transformers import T5ForConditionalGeneration, T5Tokenizer

def run\_t5\_squad\_example(model\_dir, question, context):  
# Load the T5 model and tokenizer  
model = T5ForConditionalGeneration.from\_pretrained(model\_dir)  
tokenizer = T5Tokenizer.from\_pretrained(model\_dir)

```
# Prepare the input
input_text = f"question: {question} context: {context}"
input_ids = tokenizer.encode(input_text, return_tensors="pt")

# Generate the answer
output = model.generate(input_ids)
answer = tokenizer.decode(output[0], skip_special_tokens=True)

return answer

```

model\_directory = “t5\_model”  
print(type(model\_directory))  
question = “What is the capital of France?”  
context = “Paris is the capital of France.”

answer = run\_t5\_squad\_example(model\_directory, question, context)  
print(“Answer:”, answer)

ERROR received:

\<class ‘str’\>  
┌───────────────────── Traceback (most recent call last) ─────────────────────┐  
│ C:\Users\admin\Desktop\True\_Vision\NLP\Question\_Answering\question-answerin │  
│ g-transformers-git\model\_testing.py:23 in │  
│ │  
│ 20 question = “What is the capital of France?” │  
│ 21 context = “Paris is the capital of France.” │  
│ 22 │  
│ \> 23 answer = run\_t5\_squad\_example(model\_directory, question, context) │  
│ 24 print(“Answer:”, answer) │  
│ 25 │  
│ 26 from transformers import T5ForConditionalGeneration, T5Tokenizer │  
│ │  
│ C:\Users\admin\Desktop\True\_Vision\NLP\Question\_Answering\question-answerin │  
│ g-transformers-git\model\_testing.py:6 in run\_t5\_squad\_example │  
│ │  
│ 3 def run\_t5\_squad\_example(model\_dir, question, context): │  
│ 4 │ # Load the T5 model and tokenizer │  
│ 5 │ model = T5ForConditionalGeneration.from\_pretrained(model\_dir) │  
│ \> 6 │ tokenizer = T5Tokenizer.from\_pretrained(model\_dir) │  
│ 7 │ │  
│ 8 │ # Prepare the input │  
│ 9 │ input\_text = f"question: {question} context: {context}" │  
│ │  
│ C:\Users\admin\Desktop\env\lib\site-packages\transformers\tokenization\_util │  
│ s\_base.py:1825 in from\_pretrained │  
│ │  
│ 1822 │ │ │ else: │  
│ 1823 │ │ │ │ logger.info(f"loading file {file\_path} from cache at │  
│ 1824 │ │ │  
│ \> 1825 │ │ return cls.\_from\_pretrained( │  
│ 1826 │ │ │ resolved\_vocab\_files, │  
│ 1827 │ │ │ pretrained\_model\_name\_or\_path, │  
│ 1828 │ │ │ init\_configuration, │  
│ │  
│ C:\Users\admin\Desktop\env\lib\site-packages\transformers\tokenization\_util │  
│ s\_base.py:1988 in \_from\_pretrained │  
│ │  
│ 1985 │ │ │  
│ 1986 │ │ # Instantiate tokenizer. │  
│ 1987 │ │ try: │  
│ \> 1988 │ │ │ tokenizer = cls(\*init\_inputs, \*\*init\_kwargs) │  
│ 1989 │ │ except OSError: │  
│ 1990 │ │ │ raise OSError( │  
│ 1991 │ │ │ │ "Unable to load vocabulary from file. " │  
│ │  
│ C:\Users\admin\Desktop\env\lib\site-packages\transformers\models\t5\tokeniz │  
│ ation\_t5.py:154 in **init** │  
│ │  
│ 151 │ │ self.\_extra\_ids = extra\_ids │  
│ 152 │ │ │  
│ 153 │ │ self.sp\_model = spm.SentencePieceProcessor(\*\*self.sp\_model\_kw │  
│ \> 154 │ │ self.sp\_model.Load(vocab\_file) │  
│ 155 │ │  
│ 156 │ @staticmethod │  
│ 157 │ def _eventually\_correct\_t5\_max\_length(pretrained\_model\_name\_or\_pa │  
│ │  
│ C:\Users\admin\Desktop\env\lib\site-packages\sentencepiece\_init_.py:905 │  
│ in Load │  
│ │  
│ 902 │ │ raise RuntimeError('model\_file and model\_proto must be exclu │  
│ 903 │ if model\_proto: │  
│ 904 │ │ return self.LoadFromSerializedProto(model\_proto) │  
│ \> 905 │ return self.LoadFromFile(model\_file) │  
│ 906 │  
│ 907 │  
│ 908 # Register SentencePieceProcessor in _sentencepiece: │  
│ │  
│ C:\Users\admin\Desktop\env\lib\site-packages\sentencepiece\_init_.py:310 │  
│ in LoadFromFile │  
│ │  
│ 307 │ │ return \_sentencepiece.SentencePieceProcessor\_serialized\_mode │  
│ 308 │ │  
│ 309 │ def LoadFromFile(self, arg): │  
│ \> 310 │ │ return \_sentencepiece.SentencePieceProcessor\_LoadFromFile(se │  
│ 311 │ │  
│ 312 │ def \_EncodeAsIds(self, text, enable\_sampling, nbest\_size, alpha, │  
│ 313 │ │ return \_sentencepiece.SentencePieceProcessor\_\_EncodeAsIds(se │  
└─────────────────────────────────────────────────────────────────────────────┘  
TypeError: not a string

model directory is not a string.

i have checked that the model directory is of string type .