# Issue with Decoding in HuggingFace

**URL:** <https://discuss.huggingface.co/t/issue-with-decoding-in-huggingface/15699>\
**Category:** 🤗Tokenizers\
**Created:** [March 14, 2022, 5:17pm UTC](https://discuss.huggingface.co/t/issue-with-decoding-in-huggingface/15699 "2022-03-14T17:17:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashutoshsaboo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ashutoshsaboo/32/4183_2.png) [@ashutoshsaboo](https://discuss.huggingface.co/u/ashutoshsaboo)\
**Post date:** [March 14, 2022, 5:17pm UTC](https://discuss.huggingface.co/t/issue-with-decoding-in-huggingface/15699/1 "2022-03-14T17:17:38Z")

</div>

Hello! Is there a way to batch\_decode on a minibatch of tokenized text samples to get the actual input text, but with sentence1 and sentence2 as separated? What I mean is that: currently batch\_decode returns the required text but with a whole lot of special tokens by default (PAD, CLS, SEP etc etc). I know there is the skip\_special\_tokens param ([Utilities for Tokenizers](https://huggingface.co/docs/transformers/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.batch_decode)) which can help remove these unwanted tokens, but unfortunately a by-product of that is also that the special SEP token is also removed - which means in the returned special token free text there’s no way to split and get decoded sentence1 and sentence2 as separate sentences and both are concatenated.

Is there some way to clear these other unwanted tokens (PAD, CLS etc) but leave SEP in the batch\_decode (or if there’s any alternative method already available for this use-case?) - so we can get the decoded sentence1 and sentence2 separately back? Can someone please help if possible?

@lewtun: I came across many of your insightful posts/answers in the community. If you could please help out with the above if possible, that’d be so helpful and awesome! 😄

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [March 18, 2022, 2:56pm UTC](https://discuss.huggingface.co/t/issue-with-decoding-in-huggingface/15699/2 "2022-03-18T14:56:02Z")

</div>

Hey @ashutoshsaboo I am not aware of a built in method to achieve what you want, but can’t you just slide out the first and last tokens from the `batch_decode()` function?

Here’s a simple example to show what I mean:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
inputs = tokenizer("My name is Alice", "My name is Bob")
outputs = tokenizer.batch_decode(inputs["input_ids"])
# Returns ['My', 'name', 'is', 'Alice', '[SEP]', 'My', 'name', 'is', 'Bob']
outputs[1:-1]

```

If that’s not what you’re after, perhaps you can share an example of the inputs and the desired outputs?

---

<div class="post-metadata">

**Author:** ![ashutoshsaboo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ashutoshsaboo/32/4183_2.png) [@ashutoshsaboo](https://discuss.huggingface.co/u/ashutoshsaboo)\
**Post date:** [March 24, 2022, 11:22am UTC](https://discuss.huggingface.co/t/issue-with-decoding-in-huggingface/15699/3 "2022-03-24T11:22:41Z")

</div>

First and Last tokens take care of only CLS and last SEP (if it exists). What about PAD token which can be dynamically padded in a batch? Any easy native way to strip those too? @lewtun

Of-course, you could do with python string operations (stripping of PAD tokens might cause redundant spaces too which you might have to strip again) - I’m currently using `re` for the same. But I initially thought this would be a quite common use-case and would ideally have a native method to do this. Alas, not unfortunately!

If a filter of tokens to strip can be passed to batch\_decode that should have done the job ideally.