# Batch tokenize (split into tokens, without processing)

**URL:** <https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238>\
**Category:** 🤗Tokenizers\
**Created:** [October 28, 2023, 10:27am UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238 "2023-10-28T10:27:36Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sifal](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@Sifal](https://discuss.huggingface.co/u/Sifal)\
**Post date:** [October 28, 2023, 10:27am UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238/1 "2023-10-28T10:27:36Z")

</div>

Is there a function that does tokenizer.tokenize(‘text’) except on a batch? (ie return tokens rather than ids)

---

<div class="post-metadata">

**Author:** ![AIGeekProgrammer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aigeekprogrammer/32/20072_2.png) [@AIGeekProgrammer](https://discuss.huggingface.co/u/AIGeekProgrammer)\
**Post date:** [October 28, 2023, 12:48pm UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238/2 "2023-10-28T12:48:17Z")

</div>

You mean like this?

```auto
tokens = tokenizer.tokenize("This is the extraction of tokens.")
print(tokens)
['This', 'is', 'the', 'extraction', 'of', 'token', '##s', '.']

```

---

<div class="post-metadata">

**Author:** ![Sifal](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@Sifal](https://discuss.huggingface.co/u/Sifal)\
**Post date:** [October 28, 2023, 1:17pm UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238/3 "2023-10-28T13:17:51Z")

</div>

yes but for a batch of sequences, like tokenizer.batch\_tokenize(batch\_size \* sequences)  
returns a batch of tokenized sequences ( not the ids , just the split tokens)

---

<div class="post-metadata">

**Author:** ![AIGeekProgrammer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aigeekprogrammer/32/20072_2.png) [@AIGeekProgrammer](https://discuss.huggingface.co/u/AIGeekProgrammer)\
**Post date:** [October 28, 2023, 1:34pm UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238/4 "2023-10-28T13:34:10Z")

</div>

AFAIK, the tokenizer does not have a built-in method for processing a batch of sequences into tokens.  
You can achieve this by using a list comprehension:

```auto
sequences = [["This is the extraction of tokens.", 
              "This is the second sentence"]]
tokenized_sequences = [tokenizer.tokenize(sequence) for sequence in sequences]
print(tokenized_sequences)

```

---

<div class="post-metadata">

**Author:** ![Sifal](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@Sifal](https://discuss.huggingface.co/u/Sifal)\
**Post date:** [October 28, 2023, 1:43pm UTC](https://discuss.huggingface.co/t/batch-tokenize-split-into-tokens-without-processing/60238/5 "2023-10-28T13:43:21Z")

</div>

That’s what I’ve been using and it’s been causing major bottleneck issues, and I need the tokens to pass them to another function