# Tokenizer taking extremely long time to train

**URL:** <https://discuss.huggingface.co/t/tokenizer-taking-extremely-long-time-to-train/1875>\
**Category:** 🤗Tokenizers\
**Created:** [November 3, 2020, 4:29pm UTC](https://discuss.huggingface.co/t/tokenizer-taking-extremely-long-time-to-train/1875 "2020-11-03T16:29:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![wjs20](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@wjs20](https://discuss.huggingface.co/u/wjs20)\
**Post date:** [November 3, 2020, 4:29pm UTC](https://discuss.huggingface.co/t/tokenizer-taking-extremely-long-time-to-train/1875/1 "2020-11-03T16:29:20Z")

</div>

I am trying to tokenize some amino acid sequences using a BPE tokenizer, but it hasn’t finished training after several hours.

There are ~500,000 sequences in a .txt file formatted like so:  
MKTLLLTLVVVTIVCLDLGYTLKCHNTQLPFIYNTCPEGKNLCFKATLKFPLKFPVKRGCAATCPRSSSLVKVVCCKTDKCN  
MALFRKKDKYIRINPNRSRIESAPQAKPEVPDELFSKCPACKVILYKNDLGLEKTCQHCSYNFRITAQERRALTVDEGSFEELFTGIET  
ADNRRPIWNLGHMVNALKQIPTFLXDGANA

I have included the code I used below

```auto
##Import Data

import pandas as pd
import numpy as np

url = 'https://www.uniprot.org/uniprot/?query=reviewed:yes&format=tab&columns=sequence'

seqs = pd.read_csv(url)

seqs_arr = seqs.to_numpy()

#take 1% sample of larger dataset for testing 
n_rows = seqs_arr.shape[0]
rand_ind = np.random.choice(n_rows, size=5000, replace=False)

#seqs_arr_small = seqs_arr[rand_ind, :].shape

np.savetxt('seqs_arr_small.txt', seqs_arr_small, fmt='%s')

## Train tokenizer

# We won't need TensorFlow here
!pip uninstall -y tensorflow
# Install `transformers` from master
!pip install git+https://github.com/huggingface/transformers
!pip list | grep -E 'transformers|tokenizers'
# transformers version at notebook update --- 2.11.0
# tokenizers version at notebook update --- 0.8.0rc1

%%time 
from pathlib import Path

from tokenizers import ByteLevelBPETokenizer

paths = 'seqs_arr_small.txt'

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()

# Customize training
tokenizer.train(files=paths, vocab_size=52000, min_frequency=3, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

```

---

<div class="post-metadata">

**Author:** ![gweidart](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gweidart/32/43593_2.png) [@gweidart](https://discuss.huggingface.co/u/gweidart)\
**Post date:** [March 19, 2025, 8:49am UTC](https://discuss.huggingface.co/t/tokenizer-taking-extremely-long-time-to-train/1875/2 "2025-03-19T08:49:15Z")

</div>

What BPE tokenizer are you using?

Maybe check out [rs-bpe](https://github.com/gweidart/rs-bpe/tree/main). It consistently outperforms tiktoken and tokenizers.