# How do I do inference using the GPT models on TPUs?

**URL:** <https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644>\
**Category:** 🤗Transformers\
**Created:** [December 8, 2021, 4:53pm UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644 "2021-12-08T16:53:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![NightMachinery](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nightmachinery/32/3366_2.png) [@NightMachinery](https://discuss.huggingface.co/u/NightMachinery)\
**Post date:** [December 8, 2021, 4:53pm UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/1 "2021-12-08T16:53:15Z")

</div>

I have tried the following, but it did not work:

```bash
!pip uninstall -y torch
!pip uninstall -y torchvision
!pip uninstall -y torchtext
!pip uninstall -y torchaudio

!pip install transformers \
  cloud-tpu-client==0.10 \
  datasets \
  torchvision \
  torchaudio \
  librosa \
  jiwer \
  parsivar \
  num2fawords \
  torch==1.9.0 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl

!pip install datasets transformers[sentencepiece]

```

```auto
import torch_xla
import torch_xla.core.xla_model as xm

import torch
from torch import nn, optim
from torchvision import transforms, datasets
from torch.optim import Adam
import torch.nn.functional as F

import torch_xla
import torch_xla.core.xla_model as xm
import torch_xla.debug.metrics as met
import torch_xla.distributed.parallel_loader as pl
import torch_xla.distributed.xla_multiprocessing as xmp
import torch_xla.utils.utils as xu

device = xm.xla_device()

from transformers import pipeline, AutoTokenizer, GPT2LMHeadModel
tokenizer = AutoTokenizer.from_pretrained('flax-community/gpt2-medium-persian')
model = GPT2LMHeadModel.from_pretrained('flax-community/gpt2-medium-persian')
model = model.to(device)
generator = pipeline('text-generation', model, tokenizer=tokenizer, config={'max_length':100})

```

But it errors:

```auto
out = generator('در مورد پدر هری پاتر شک هایی وجود دارد.')

```

```auto
Setting `pad_token_id` to `eos_token_id`:5 for open-end generation.
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-18-f6da4d1d2479> in <module>()
----> 1 get_ipython().magic("timeit out = generator('در مورد پدر هری پاتر شک هایی وجود دارد.')")

19 frames
<decorator-gen-52> in timeit(self, line, cell)

<magic-timeit> in inner(_it, _timer)

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2041 # remove once script supports set_grad_enabled
   2042 _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2043 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2044 
   2045 

RuntimeError: torch_xla/csrc/aten_xla_bridge.cpp:69 : Check failed: xtensor 
***Begin stack trace***
	tensorflow::CurrentStackTrace()
	torch_xla::bridge::GetXlaTensor(at::Tensor const&)
	torch_xla::AtenXlaType::index_select(at::Tensor const&, long, at::Tensor const&)
	c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, long, at::Tensor const&), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, long, at::Tensor const&> >, at::Tensor (at::Tensor const&, long, at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&)
	at::Tensor::index_select(long, at::Tensor const&) const
	at::native::embedding(at::Tensor const&, at::Tensor const&, long, bool, bool)
	torch_xla::AtenXlaType::embedding(at::Tensor const&, at::Tensor const&, long, bool, bool)
	c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, long, bool, bool), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, long, bool, bool> >, at::Tensor (at::Tensor const&, at::Tensor const&, long, bool, bool)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, long, bool, bool)
	at::embedding(at::Tensor const&, at::Tensor const&, long, bool, bool)
	
	_PyMethodDef_RawFastCallKeywords
	_PyCFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	
	_PyObject_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	
	_PyObject_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallDict
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	
	_PyObject_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallDict
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	PyEval_EvalCode
	
	_PyMethodDef_RawFastCallKeywords
	_PyCFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyObject_Call_Prepend
	PyObject_Call
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallDict
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallDict
	_PyEval_EvalFrameDefault
	_PyEval_EvalCodeWithName
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallKeywords
	_PyEval_EvalFrameDefault
***End stack trace***
Input tensor is not an XLA tensor: torch.LongTensor

```

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [December 8, 2021, 6:41pm UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/2 "2021-12-08T18:41:28Z")

</div>

The `pipeline` function does not support TPUs, you will have to manually pass your batch through the model (after placing it on the right XLA device) and then post-process the outputs.

---

<div class="post-metadata">

**Author:** ![NightMachinery](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nightmachinery/32/3366_2.png) [@NightMachinery](https://discuss.huggingface.co/u/NightMachinery)\
**Post date:** [December 8, 2021, 8:37pm UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/3 "2021-12-08T20:37:14Z")

</div>

Are there any examples of doing this in the docs or somewhere?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [December 8, 2021, 8:42pm UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/4 "2021-12-08T20:42:51Z")

</div>

You should check the [task summary](https://huggingface.co/docs/transformers/task_summary).

---

<div class="post-metadata">

**Author:** ![saeed899](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@saeed899](https://discuss.huggingface.co/u/saeed899)\
**Post date:** [October 13, 2024, 6:10am UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/5 "2024-10-13T06:10:05Z")

</div>

@sgugger  
any update on this?  
seems `pipeline` still has problem with TPU/XLA

---

<div class="post-metadata">

**Author:** ![saeed899](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@saeed899](https://discuss.huggingface.co/u/saeed899)\
**Post date:** [October 13, 2024, 6:15am UTC](https://discuss.huggingface.co/t/how-do-i-do-inference-using-the-gpt-models-on-tpus/12644/6 "2024-10-13T06:15:02Z")

</div>

@NightMachinery did you solved? can you please share the solution?