# Efficient detokenization method

**URL:** <https://discuss.huggingface.co/t/efficient-detokenization-method/3421>\
**Category:** 🤗Transformers\
**Created:** [January 28, 2021, 6:29pm UTC](https://discuss.huggingface.co/t/efficient-detokenization-method/3421 "2021-01-28T18:29:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](https://discuss.huggingface.co/u/berkayberabi)\
**Post date:** [January 28, 2021, 6:29pm UTC](https://discuss.huggingface.co/t/efficient-detokenization-method/3421/1 "2021-01-28T18:29:50Z")

</div>

Tokenizer has **call** function nad it accepts List[List[string]] so one can tokenize multiple samples at once efficiently. I am looking for a function that reverts this behavior. There are convert\_ids\_to\_tokens() and convert\_tokens\_to\_string() but they only accept List[int] and List[string]. I do not want to iterate over the sample and convert them back one by one.

Is there an efficient method for this? I am using T5

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 28, 2021, 8:25pm UTC](https://discuss.huggingface.co/t/efficient-detokenization-method/3421/2 "2021-01-28T20:25:28Z")

</div>

There is the `decode` method that could help.

---

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](https://discuss.huggingface.co/u/berkayberabi)\
**Post date:** [January 28, 2021, 10:01pm UTC](https://discuss.huggingface.co/t/efficient-detokenization-method/3421/3 "2021-01-28T22:01:55Z")

</div>

are you sure that it can take multiple sentences? from the documentation, it seems to me that it only accepts one sample.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 28, 2021, 10:04pm UTC](https://discuss.huggingface.co/t/efficient-detokenization-method/3421/4 "2021-01-28T22:04:00Z")

</div>

For multiple sentences there is its counterpart `batch_decode`.