# Error creating custom pre\_tokenizer **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [January 1, 2025, 5:15pm UTC](https://discuss.huggingface.co/t/error-creating-custom-pre-tokenizer/133861 "2025-01-01T17:15:20Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Kevin1807](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@Kevin1807](https://discuss.huggingface.co/u/Kevin1807)\ **Post date:** [January 1, 2025, 5:15pm UTC](https://discuss.huggingface.co/t/error-creating-custom-pre-tokenizer/133861/1 "2025-01-01T17:15:20Z")
```auto class IngenxPreTokenizer(PreTokenizer): def __init__ (self): super(). __init__ () self.base_tokenizer = IngenxTokenizer() def pre_tokenize(self, pretok: PreTokenizedString): # text = pretok.normalized processed = self.base_tokenizer.process_text(pretok) normalized_tokens = [] current_offset = 0 for token in processed: token_len = len(token) normalized_tokens.append(( token, (current_offset, current_offset + token_len) )) current_offset += token_len + 1 pretok.tokens = normalized_tokens return pretok class IngenxTokenTrainer: def __init__ (self,df,size_dataset =240340,vocab_size=150000,min_freq = 5,batch_size=1000): self.tokenizer = IngenxTokenizer() self.df = df self.size_dataset = size_dataset self.vocab_size = vocab_size self.min_freq = min_freq self.batch_size=1000 self.special_tokens = ["<|unk|>","<|pad|>","", "<|var|>","","<|val|>","<|val|>", "<|func|>","<|func|>","<|op|>","" ] self.training_corpus = self.preprare_dataset() def preprare_dataset(self): X2 = np.random.choice(len(self.df), size=self.size_dataset, replace=False) training_texts = [f"{self.df.iloc[i]['problem']} {self.df.iloc[i]['solution']}" for i in X2] return training_texts def get_training_corpus(self): dataset = self.training_corpus with tqdm(total=len(dataset), desc="Processing training corpus", unit="batch") as pbar: for start_idx in range(0, len(dataset), self.batch_size): batch = dataset[start_idx : start_idx + self.batch_size] pbar.update(len(batch)) yield batch def train_tokenizer(self): tokenizer = Tokenizer(BPE()) tokenizer.pre_tokenizer = PreTokenizer.custom(IngenxPreTokenizer) # tokenizer.pre_tokenizer = Whitespace() trainer = BpeTrainer( vocab_size=self.vocab_size, min_frequency=self.min_freq, special_tokens=self.special_tokens ) tokenizer.train_from_iterator(self.get_training_corpus(),trainer=trainer, length=len(self.training_corpus)) tokenizer.save("ingenx_tokenizer.json") return tokenizer ``` when i run this code i got this error ```auto Exception Traceback (most recent call last) in () ----> 1 a.train_tokenizer() in train_tokenizer(self) 37 special_tokens=self.special_tokens 38 ) ---> 39 tokenizer.train_from_iterator(self.get_training_corpus(),trainer=trainer, length=len(self.training_corpus)) 40 tokenizer.save("ingenx_tokenizer.json") 41 return tokenizer Exception: TypeError: IngenxPreTokenizer.pre_tokenize() missing 1 required positional argument: 'pretok' ``` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [January 1, 2025, 11:44pm UTC](https://discuss.huggingface.co/t/error-creating-custom-pre-tokenizer/133861/2 "2025-01-01T23:44:19Z")
There may be a workaround. > > > Hey, > > I want to train a Tokenizer that operates on a custom PreTokenizer. I tr…ied a mix of \[this documentation post\](https://huggingface.co/docs/tokenizers/pipeline) and \[this example\](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom\_components.py). My resulting code looks like this: > > \`\`\`python > class GlyLESPreTokenizer: > def \_\_init\_\_(self, \*args, \*\*kwargs): > pass > > def \_\_new\_\_(cls, \*args, \*\*kwargs): > return super().\_\_new\_\_(cls) > > def glyles\_split(self, iupac: str): > iuapc = iupac.strip().replace(" ", "") > token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}"))) > GlyLESParser(token).start() > idx = 0 > output = \[\] > for i in range(1, len(token.tokens) - 2): > txt = str(token.tokens\[i\].text) > output.append((txt, (idx, idx + len(txt)))) > idx += len(txt) > return output > > def pre\_tokenize\_str(self, input\_: str): > return self.glyles\_split(input\_) > > > iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc" > > \# This returns a list of 33 token > GlyLESPreTokenizer().pre\_tokenize\_str(iupac) > > \# This however only returns a list with one token that is the entire input string. > pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre\_tokenize\_str(iupac) > \`\`\` > > The final idea is to use it in such setting: > \`\`\`python > tokenizer = Tokenizer(models.Model()) > tokenizer.normalizer = normalizers.Strip() > tokenizer.pre\_tokenizer = pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()) > \`\`\` > > Can someone help me to understand how to use the \`pre\_tokenizers.PreTokenizer.custom\` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from \[GlyLES\](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer. > > Thank you for any help, comment, or feedback in advance. > Roman ---
**Author:** ![Kevin1807](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@Kevin1807](https://discuss.huggingface.co/u/Kevin1807)\ **Post date:** [January 2, 2025, 6:39am UTC](https://discuss.huggingface.co/t/error-creating-custom-pre-tokenizer/133861/3 "2025-01-02T06:39:59Z")
thank you for the reply ,in this repo they are talking about using tokenizer i want train a tokenizer and i dont think there is method like > pre\_tokenized=True in train\_from\_iterator ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [January 2, 2025, 8:08am UTC](https://discuss.huggingface.co/t/error-creating-custom-pre-tokenizer/133861/4 "2025-01-02T08:08:01Z")
It doesn’t seem wrong… Most of the tokenizer library is written in Rust, so the Python parts that were added afterwards might as well not inherit from classes. ```py # class IngenxPreTokenizer(PreTokenizer): # class IngenxPreTokenizer(): ``` >